Leaderboards
OpenSAT20_KWS_Progress_Set
Updated: 2020-11-24 15:48:16 -0500
|
PROGRESS SET |
|
|
ASSOC. TEST SET |
TEAM |
TWV |
Best |
Last |
TWV |
Date&Time |
Team_002 |
0.6280 |
X |
X |
0.7116 |
Mon Aug 3 23:39:39 UTC 2020 |
Team_008 |
0.5171 |
X |
X |
0.5859 |
Sun Aug 16 15:16:08 UTC 2020 |
Team_0011 |
0.5023 |
X |
|
0.4239 |
Fri Aug 14 11:42:50 UTC 2020 |
Team_0011 |
0.5018 |
|
X |
0.4274 |
Fri Aug 14 11:45:28 UTC 2020 |
Team_0014 |
-0.417 |
X |
X |
-0.523 |
Sun Aug 16 18:10:37 UTC 2020 |
OpenSAT20_KWS_Test_Set
Updated: 2020-11-27 15:08:31 -0500
|
TEST SET |
|
|
ASSOC. PROGRESS SET |
|
TEAM |
TWV |
Best |
Last |
TWV |
Date&Time |
Team_002 |
0.7116 |
X |
X |
0.6280 |
Mon Aug 3 23:39:39 UTC 2020 |
Team_008 |
0.5994 |
X |
|
0.4970 |
Sat Aug 15 22:11:22 UTC 2020 |
Team_008 |
0.5859 |
|
X |
0.5171 |
Sun Aug 16 15:16:08 UTC 2020 |
Team_0011 |
0.4274 |
X |
X |
0.5018 |
Fri Aug 14 11:45:28 UTC 2020 |
Team_0014 |
-0.523 |
X |
X |
-0.417 |
Sun Aug 16 18:10:37 UTC 2020 |
OpenSAT20_ASR_Progress_Set
Updated: 2020-11-27 15:06:17 -0500
|
PROGRESS SET |
|
|
ASSOC. TEST SET |
|
TEAM |
WER |
Best |
Last |
WER |
Date&Time |
Team_0017 |
8.5 |
X |
X |
13.3 |
Sat Aug 15 02:52:31 UTC 2020 |
Team_002 |
14.0 |
|
X |
9.5 |
Sat Aug 15 04:19:02 UTC 2020 |
Team_002 |
14.0 |
X |
|
9.4 |
Sat Aug 15 03:48:54 UTC 2020 |
Team_008 |
14.0 |
X |
|
9.9 |
Sat Aug 15 22:12:03 UTC 2020 |
Team_008 |
14.9 |
|
X |
10.9 |
Sun Aug 16 19:34:02 UTC 2020 |
Team_007 |
18.6 |
X |
|
12.7 |
Tue Aug 11 10:07:43 UTC 2020 |
Team_007 |
18.8 |
|
X |
12.7 |
Tue Aug 11 15:44:52 UTC 2020 |
Team_0020 |
19.2 |
X |
X |
28.8 |
Mon Aug 10 12:03:34 UTC 2020 |
Team_006 |
22.9 |
X |
|
14.4 |
Thu Jul 30 08:32:03 UTC 2020 |
Team_006 |
26.0 |
|
X |
17.9 |
Fri Aug 14 10:54:04 UTC 2020 |
Team_0014 |
107.4 |
X |
X |
107.5 |
Sun Aug 16 10:48:37 UTC 2020 |
OpenSAT20_ASR_Test_Set
Updated: 2020-11-27 15:17:51 -0500
|
TEST SET |
|
|
ASSOC. PROGRESS SET |
|
TEAM |
WER |
Best |
Last |
WER |
Date&Time |
Team_0017 |
13.3 |
X |
X |
8.5 |
Sat Aug 15 02:52:31 UTC 2020 |
Team_002 |
14.0 |
|
X |
9.5 |
Sat Aug 15 04:19:02 UTC 2020 |
Team_002 |
14.0 |
X |
|
9.4 |
Sat Aug 15 03:48:54 UTC 2020 |
Team_008 |
14.0 |
X |
|
9.9 |
Sat Aug 15 22:12:03 UTC 2020 |
Team_008 |
14.9 |
|
X |
10.9 |
Sun Aug 16 19:34:02 UTC 2020 |
Team_007 |
18.6 |
X |
|
12.7 |
Tue Aug 11 10:07:43 UTC 2020 |
Team_007 |
18.8 |
|
X |
12.7 |
Tue Aug 11 15:44:52 UTC 2020 |
Team_006 |
22.9 |
X |
|
14.4 |
Thu Jul 30 08:32:03 UTC 2020 |
Team_006 |
26.0 |
|
X |
17.9 |
Fri Aug 14 10:54:04 UTC 2020 |
Team_0020 |
28.8 |
X |
X |
19.2 |
Mon Aug 10 12:03:34 UTC 2020 |
Team_0014 |
107.5 |
X |
X |
107.4 |
Sun Aug 16 10:48:37 UTC 2020 |
OpenSAT20_SAD_Progress_Set
Updated: 2020-11-25 11:03:02 -0500
|
PROGRESS SET |
|
|
ASSOC. TEST SET |
|
TEAM |
DCF |
Best |
Last |
DCF |
Date&Time |
Team_002 |
0.0440 |
X |
X |
0.0699 |
Sun Aug 16 09:49:38 UTC 2020 |
Team_0024 |
0.0443 |
|
X |
0.0699 |
Mon Aug 17 03:00:06 UTC 2020 |
Team_0024 |
0.0443 |
X |
|
0.0689 |
Sun Aug 16 21:45:30 UTC 2020 |
Team_007 |
0.0914 |
X |
|
0.1122 |
Sat Aug 15 19:01:27 UTC 2020 |
Team_007 |
0.1135 |
|
X |
0.1729 |
Sat Aug 15 19:32:27 UTC 2020 |
Team_0014 |
0.1506 |
X |
X |
0.2116 |
Sun Aug 16 13:42:08 UTC 2020 |
OpenSAT20_SAD_Test_Set
Updated: 2020-11-27 15:16:39 -0500
|
TEST SET |
|
|
ASSOC. PROGRESS SET |
|
TEAM |
DCF |
Best |
Last |
DCF |
Date&Time |
Team_002 |
0.0600 |
X |
|
0.0536 |
Sat Aug 15 04:48:06 UTC 2020 |
Team_0024 |
0.0678 |
X |
|
0.0448 |
Fri Aug 14 01:02:21 UTC 2020 |
Team_002 |
0.0699 |
|
X |
0.0440 |
Sun Aug 16 09:49:38 UTC 2020 |
Team_0024 |
0.0699 |
|
X |
0.0443 |
Mon Aug 17 03:00:06 UTC 2020 |
Team_007 |
0.1095 |
X |
|
0.1200 |
Tue Jul 21 16:31:21 UTC 2020 |
Team_007 |
0.1729 |
|
X |
0.1135 |
Sat Aug 15 19:32:27 UTC 2020 |
Team_0014 |
0.2116 |
X |
X |
0.1506 |
Sun Aug 16 13:42:08 UTC 2020 |
- OpenSAT20 has completed for 2020 -
Welcome to the NIST 2020 Open Speech Analytic Technologies Evaluation (OpenSAT20)
OpenSAT20 supports the public safety communications domain with ASR, SAD and KWS.
Summary
OpenSAT20 is the second in the OpenSAT Series for speech analytic systems evaluations. OpenSAT provides an opportunity for participants to compare their system performance against a pool of systems performances for each task and is intended to encourage cross-learning among developers.
Tasks and Domains
System Tasks |
Data Domains |
Automatic Speech Recognition (ASR)
Speech Activity Detection (SAD)
Keyword Search (KWS)
|
Public safety communications (PSC)
|
Simulated Public Safety Communications (PSC)
The OpenSAT20 evaluation will include the ASR, SAD, and KWS tasks for simulated first responder public safety communications. Participants can participate with one, two, or all three tasks.
The simulated public safety communications dataset was created by LDC and funded by the Department of Homeland Security (DHS) to advance first responder assistant technologies in first responder noisy and stressed environments including its effects on speech. The public safety communications dataset includes simulated first responder communications with and without the Lombard effect in speech and moments in speech with expression of urgency. The audio also includes low-level and loud background sounds typical in first responder events.
The simulated first responder communications is intended to be a precursor in advancing assistive technologies for real-world operational communications. Real-world operational data is expected to be made available in future evaluations
Milestone |
Date |
Registration opens |
April 13th |
Development/Training data |
Available for download until July 31st |
Evaluation data (contains a combined progress data set and a test data set in one file) |
Available for download until July 31st |
Last date to upload Evaluation data results to NIST server |
August 16th |
NIST Virtual Workshop |
September 16th |
Participation
OpenSAT20 is open to the public. All organizations, ie., universities, government institutions, corporations or businesses, are invited to apply their technologies to the OpenSAT20 data and submit their system results to the OpenSAT evaluation server. The evaluation is open worldwide. Participating in the evaluation includes attendance/participation in a workshop that follow the evaluation. Participation is free for the evaluation and the workshop. NIST does not provide funds to participants.
Participation Logistics
Each participant must create an account on this web platform. After creating an account, each participant will either create a new Team or join an existing Team.
After registering and having an LDC data license agreement approved, participants will be able to participate in the OpenSAT20 Evaluation. Most of the data will be accessed from LDC and some of the data from this site.
Participants will submit tar.gz files of their system’s output to the NIST OpenSAT scoring server using this web site.
Go to the
Register tab for registration instructions and to register for OpenSAT20.
2020 Tentative Schedule
Milestone
30MAR
Registration opens and Development and training data released
1MAY
Evaluation data released
30JUN
Last date to upload system output from evaluation data to NIST server
Registration Instructions
If you already have an account, login
here or at the top of the page. To create an account and register, follow the steps below.
To Create an Account:
1- Click the "Sign up" button at the top of the page.
2- Provide a valid e-mail address and create a password in the “Sign up” form.
(After entering your email address and creating a password and clicking “Sign up” on the “OpenSAT20 Sign up” online form, a confirmation email will be sent to that email address).
3- Click “Confirm my account” in the e-mail sent to you. (A log-in page will display with your email address and created password already entered.)
4- Click “Log in”. (A dashboard for your account will display with Registration Steps.)
5- Complete the steps in the dashboard to complete an account creation.
6- Registration is completed when steps 1-5 are completed.
When you are notified by email from LDC that your License Agreement is approved, you can then access the data.
Creating a Team or Joining a Site and Team
When joining OpenSAT, a participant can either create a Site, or join an existing Site and create a Team, or join an existing Team. A participant can be a member of multiple teams.
Each participant, Site, and Team will have its own password. The creator(s) of the Site and Team creates those passwords respectively.
The NIST Agreement
Check the “I acknowledge that I have read and accepted the OpenSAT20 Terms and Conditions” box and then click the “Update the License Agreement” button.
The Data License Agreement
Site creator is required to agree to the LDC terms in order to access data for that site. Read the LDC license agreement and accept the terms by uploading the signed license agreement form. Participants cannot download data until LDC approves your uploaded signed LDC license agreement.
The Dashboard
The dashboard is the personalized page for each participant. To access the dashboard at any time, click the "Dashboard" at the top right of the screen. This is where you can make submissions and view results.
System Output Submission Instructions
Each submission must be associated with a Site, Team and Task.
Multiple systems may be created for each Task with a submission for each system.
Submit system output for validation checking or scoring following these steps:
1- Prepare for Submission
- System output must be in the format described in the Evaluation Plan for the task that was performed (SAD, KWS, or ASR).
- Have the .tgz or .zip file ready per Appendix IV in the OpenSAT20 Evaluation Plan and also shown below these steps.
2- Go to Dashboard. (Select "Dashboard" on the top right of the page.)
- In "Submission Management", click the Task that represents the system output.
- Click "Create new Submission" located at the upper right of the dashboard. A “Submission Name” page will display.
- Select the Data Domain from the drop down
- Enter a system identifier in “Name”.
- Click “Submit”. A “Submissions” page will display.
- On the “Submissions” page, click “Upload”. The “New Submission” page displays.
- Click the "Choose File". Choose the .tgz or .zip file to upload.
- Click "Submit".
- A submission ID is automatically created for the submission.
- The “Scoring” button on the “Submissions” page displays “submitted” until the scoring server completes scoring and then it changes to “Done”.
- When “Done” is displayed, click “Scoring” button for a Scoring Run Report.
- Click “View Submission” to see Submission information.
3- View Submission Results
- To see a Scoring Run Report, click the “Scoring” button, after “submitted” changes to “Done” on the button.
- To see information about the submission click the “View Submission” button.
Below is Appendix VI from the OpenSAT20 Evaluation Plan:
Appendix IV- SAD, KWS, and ASR - System Output Submission Packaging
- Each submission shall be an archive file named as "SysLabel".tgz or "SysLabel".zip.
- Submit a separate .tgz or .zip file for each system output (e.g., a separate .tgz or .zip file for Primary, Contrastive1, and Contrastive2 systems).
- "SysLabel" shall be an alphanumeric [a-zA-Z0-9] that is a performer-assigned identifier for their submission.
- There should be no parent directory when the submission file is untarred. The tar command should be: > tar MySystemSubmissionFile.tgz or
> tar MySystemSubmissionFile.zip respectively.
Prior to uploading the submission file to the NIST scoring server, performers will be asked for information about the submission. The scoring server will attach the following information to the submission filename to catorigize and uniquely identify the submission:
Field |
Information |
Method |
TeamID |
[Team] |
obtained from login information |
Task |
{SAD | ASR | KWS} |
select from drop-down menu |
SubmissionType |
{primary | contrastive} |
select from drop-down menu |
SubmissionType |
{primary | contrastive} |
select from drop-down menu |
Training Condition |
{unconstrained} |
default - hard-coded |
EvalPeriod |
{2019} |
default - hard-coded |
DatasetName |
{PSC | VAST | Babel} |
select from drop-down menu |
Date |
{YYYYMMDD} |
obtained from NIST scoring server at submission date |
TimeStamp |
{HHMMSS} |
obtained from NIST scoring server at submission time |
Below is an example of a resulting filename:
NIST_ASR_primary_uncontrained_2019_PSC_20190415_163026_ MySystemSubmissionFile.tgz
The NIST scoring server will perform a validation check on each system output submission for conforming to the submission format required for each task.
Submission of a system description conforming to the system description guidelines in Appendix V is required before receiving the system’s score and ranking results in the Evaluation phase.
Overview
Challenge
For additional information about DIHARD III, including registration, schedule, data, tasks, and scoring, please consult the official
DIHARD III website.
Workshop
The results of the challenge will be presented at a post-evaluation workshop, to be collocated with Interspeech 2020 in Shanghai, China on October 25th, 2020. All participants are invited to submit 2 page extended abstracts describing their submissions and results on the DIHARD III development and evaluation sets. Provisions will be made for remote participation to accommodate those unable to travel to Shanghai due to COVID-19 related disruptions. For additional details, please see the workshop website.
Contact
For more information about
DIHARD III, please join the
mailing list or contact the organizers via email at
dihardchallenge@gmail.com.