NIST 2020
Open Speech Analytic Technologies Evaluation
(OpenSAT20)


Leaderboards

OpenSAT20_KWS_Progress_Set

Updated: 2020-11-24 15:48:16 -0500
PROGRESS SET ASSOC. TEST SET
TEAM TWV Best Last TWV Date&Time
Team_002 0.6280 X X 0.7116 Mon Aug 3 23:39:39 UTC 2020
Team_008 0.5171 X X 0.5859 Sun Aug 16 15:16:08 UTC 2020
Team_0011 0.5023 X 0.4239 Fri Aug 14 11:42:50 UTC 2020
Team_0011 0.5018 X 0.4274 Fri Aug 14 11:45:28 UTC 2020
Team_0014 -0.417 X X -0.523 Sun Aug 16 18:10:37 UTC 2020

OpenSAT20_KWS_Test_Set

Updated: 2020-11-27 15:08:31 -0500
TEST SET ASSOC. PROGRESS SET
TEAM TWV Best Last TWV Date&Time
Team_002 0.7116 X X 0.6280 Mon Aug 3 23:39:39 UTC 2020
Team_008 0.5994 X 0.4970 Sat Aug 15 22:11:22 UTC 2020
Team_008 0.5859 X 0.5171 Sun Aug 16 15:16:08 UTC 2020
Team_0011 0.4274 X X 0.5018 Fri Aug 14 11:45:28 UTC 2020
Team_0014 -0.523 X X -0.417 Sun Aug 16 18:10:37 UTC 2020



OpenSAT20_ASR_Progress_Set

Updated: 2020-11-27 15:06:17 -0500
PROGRESS SET ASSOC. TEST SET
TEAM WER Best Last WER Date&Time
Team_0017 8.5 X X 13.3 Sat Aug 15 02:52:31 UTC 2020
Team_002 14.0 X 9.5 Sat Aug 15 04:19:02 UTC 2020
Team_002 14.0 X 9.4 Sat Aug 15 03:48:54 UTC 2020
Team_008 14.0 X 9.9 Sat Aug 15 22:12:03 UTC 2020
Team_008 14.9 X 10.9 Sun Aug 16 19:34:02 UTC 2020
Team_007 18.6 X 12.7 Tue Aug 11 10:07:43 UTC 2020
Team_007 18.8 X 12.7 Tue Aug 11 15:44:52 UTC 2020
Team_0020 19.2 X X 28.8 Mon Aug 10 12:03:34 UTC 2020
Team_006 22.9 X 14.4 Thu Jul 30 08:32:03 UTC 2020
Team_006 26.0 X 17.9 Fri Aug 14 10:54:04 UTC 2020
Team_0014 107.4 X X 107.5 Sun Aug 16 10:48:37 UTC 2020

OpenSAT20_ASR_Test_Set

Updated: 2020-11-27 15:17:51 -0500
TEST SET ASSOC. PROGRESS SET
TEAM WER Best Last WER Date&Time
Team_0017 13.3 X X 8.5 Sat Aug 15 02:52:31 UTC 2020
Team_002 14.0 X 9.5 Sat Aug 15 04:19:02 UTC 2020
Team_002 14.0 X 9.4 Sat Aug 15 03:48:54 UTC 2020
Team_008 14.0 X 9.9 Sat Aug 15 22:12:03 UTC 2020
Team_008 14.9 X 10.9 Sun Aug 16 19:34:02 UTC 2020
Team_007 18.6 X 12.7 Tue Aug 11 10:07:43 UTC 2020
Team_007 18.8 X 12.7 Tue Aug 11 15:44:52 UTC 2020
Team_006 22.9 X 14.4 Thu Jul 30 08:32:03 UTC 2020
Team_006 26.0 X 17.9 Fri Aug 14 10:54:04 UTC 2020
Team_0020 28.8 X X 19.2 Mon Aug 10 12:03:34 UTC 2020
Team_0014 107.5 X X 107.4 Sun Aug 16 10:48:37 UTC 2020



OpenSAT20_SAD_Progress_Set

Updated: 2020-11-25 11:03:02 -0500
PROGRESS SET ASSOC. TEST SET
TEAM DCF Best Last DCF Date&Time
Team_002 0.0440 X X 0.0699 Sun Aug 16 09:49:38 UTC 2020
Team_0024 0.0443 X 0.0699 Mon Aug 17 03:00:06 UTC 2020
Team_0024 0.0443 X 0.0689 Sun Aug 16 21:45:30 UTC 2020
Team_007 0.0914 X 0.1122 Sat Aug 15 19:01:27 UTC 2020
Team_007 0.1135 X 0.1729 Sat Aug 15 19:32:27 UTC 2020
Team_0014 0.1506 X X 0.2116 Sun Aug 16 13:42:08 UTC 2020

OpenSAT20_SAD_Test_Set

Updated: 2020-11-27 15:16:39 -0500
TEST SET ASSOC. PROGRESS SET
TEAM DCF Best Last DCF Date&Time
Team_002 0.0600 X 0.0536 Sat Aug 15 04:48:06 UTC 2020
Team_0024 0.0678 X 0.0448 Fri Aug 14 01:02:21 UTC 2020
Team_002 0.0699 X 0.0440 Sun Aug 16 09:49:38 UTC 2020
Team_0024 0.0699 X 0.0443 Mon Aug 17 03:00:06 UTC 2020
Team_007 0.1095 X 0.1200 Tue Jul 21 16:31:21 UTC 2020
Team_007 0.1729 X 0.1135 Sat Aug 15 19:32:27 UTC 2020
Team_0014 0.2116 X X 0.1506 Sun Aug 16 13:42:08 UTC 2020

Contact Us

For any information about the OpenSAT Evaluation (data, evaluation code, etc.) please email: opensat_poc@nist.gov
- OpenSAT20 has completed for 2020 -
Welcome to the NIST 2020 Open Speech Analytic Technologies Evaluation (OpenSAT20)
OpenSAT20 supports the public safety communications domain with ASR, SAD and KWS.
Summary
OpenSAT20 is the second in the OpenSAT Series for speech analytic systems evaluations. OpenSAT provides an opportunity for participants to compare their system performance against a pool of systems performances for each task and is intended to encourage cross-learning among developers.

Tasks and Domains

System Tasks Data Domains
Automatic Speech Recognition (ASR)
Speech Activity Detection (SAD)
Keyword Search (KWS)
Public safety communications (PSC)

Simulated Public Safety Communications (PSC)

The OpenSAT20 evaluation will include the ASR, SAD, and KWS tasks for simulated first responder public safety communications. Participants can participate with one, two, or all three tasks.

The simulated public safety communications dataset was created by LDC and funded by the Department of Homeland Security (DHS) to advance first responder assistant technologies in first responder noisy and stressed environments including its effects on speech. The public safety communications dataset includes simulated first responder communications with and without the Lombard effect in speech and moments in speech with expression of urgency. The audio also includes low-level and loud background sounds typical in first responder events.

The simulated first responder communications is intended to be a precursor in advancing assistive technologies for real-world operational communications. Real-world operational data is expected to be made available in future evaluations

OpenSAT20 Timeline

Milestone Date
Registration opens April 13th
Development/Training data Available for download until July 31st
Evaluation data (contains a combined progress data set and a test data set in one file) Available for download until July 31st
Last date to upload Evaluation data results to NIST server August 16th
NIST Virtual Workshop September 16th
Participation
OpenSAT20 is open to the public. All organizations, ie., universities, government institutions, corporations or businesses, are invited to apply their technologies to the OpenSAT20 data and submit their system results to the OpenSAT evaluation server. The evaluation is open worldwide. Participating in the evaluation includes attendance/participation in a workshop that follow the evaluation. Participation is free for the evaluation and the workshop. NIST does not provide funds to participants.

Participation Logistics
Each participant must create an account on this web platform. After creating an account, each participant will either create a new Team or join an existing Team.

After registering and having an LDC data license agreement approved, participants will be able to participate in the OpenSAT20 Evaluation. Most of the data will be accessed from LDC and some of the data from this site.

Participants will submit tar.gz files of their system’s output to the NIST OpenSAT scoring server using this web site.

Go to the Register tab for registration instructions and to register for OpenSAT20.

2020 Tentative Schedule

Milestone
2020
30MAR
Registration opens and Development and training data released
1MAY
Evaluation data released
30JUN
Last date to upload system output from evaluation data to NIST server
TBDAUG
NIST workshop


Registration Instructions
If you already have an account, login here or at the top of the page. To create an account and register, follow the steps below.
To Create an Account:
1- Click the "Sign up" button at the top of the page.
2- Provide a valid e-mail address and create a password in the “Sign up” form. (After entering your email address and creating a password and clicking “Sign up” on the “OpenSAT20 Sign up” online form, a confirmation email will be sent to that email address).
3- Click “Confirm my account” in the e-mail sent to you. (A log-in page will display with your email address and created password already entered.)
4- Click “Log in”. (A dashboard for your account will display with Registration Steps.)
5- Complete the steps in the dashboard to complete an account creation.
6- Registration is completed when steps 1-5 are completed.
When you are notified by email from LDC that your License Agreement is approved, you can then access the data.
Creating a Team or Joining a Site and Team
When joining OpenSAT, a participant can either create a Site, or join an existing Site and create a Team, or join an existing Team. A participant can be a member of multiple teams.
Each participant, Site, and Team will have its own password. The creator(s) of the Site and Team creates those passwords respectively.
The NIST Agreement
Check the “I acknowledge that I have read and accepted the OpenSAT20 Terms and Conditions” box and then click the “Update the License Agreement” button.
The Data License Agreement
Site creator is required to agree to the LDC terms in order to access data for that site. Read the LDC license agreement and accept the terms by uploading the signed license agreement form. Participants cannot download data until LDC approves your uploaded signed LDC license agreement.
The Dashboard
The dashboard is the personalized page for each participant. To access the dashboard at any time, click the "Dashboard" at the top right of the screen. This is where you can make submissions and view results.
System Output Submission Instructions
Each submission must be associated with a Site, Team and Task.
Multiple systems may be created for each Task with a submission for each system.
Submit system output for validation checking or scoring following these steps:
1- Prepare for Submission
  • System output must be in the format described in the Evaluation Plan for the task that was performed (SAD, KWS, or ASR).
  • Have the .tgz or .zip file ready per Appendix IV in the OpenSAT20 Evaluation Plan and also shown below these steps.
2- Go to Dashboard. (Select "Dashboard" on the top right of the page.)
  • In "Submission Management", click the Task that represents the system output.
  • Click "Create new Submission" located at the upper right of the dashboard. A “Submission Name” page will display.
    • Select the Data Domain from the drop down
    • Enter a system identifier in “Name”.
    • Click “Submit”. A “Submissions” page will display.
  • On the “Submissions” page, click “Upload”. The “New Submission” page displays.
    • Click the "Choose File". Choose the .tgz or .zip file to upload.
    • Click "Submit".
    • A submission ID is automatically created for the submission.
    • The “Scoring” button on the “Submissions” page displays “submitted” until the scoring server completes scoring and then it changes to “Done”.
    • When “Done” is displayed, click “Scoring” button for a Scoring Run Report.
    • Click “View Submission” to see Submission information.
3- View Submission Results
  • To see a Scoring Run Report, click the “Scoring” button, after “submitted” changes to “Done” on the button.
  • To see information about the submission click the “View Submission” button.
Below is Appendix VI from the OpenSAT20 Evaluation Plan: Appendix IV- SAD, KWS, and ASR - System Output Submission Packaging
  • Each submission shall be an archive file named as "SysLabel".tgz or "SysLabel".zip.
  • Submit a separate .tgz or .zip file for each system output (e.g., a separate .tgz or .zip file for Primary, Contrastive1, and Contrastive2 systems).
  • "SysLabel" shall be an alphanumeric [a-zA-Z0-9] that is a performer-assigned identifier for their submission.
  • There should be no parent directory when the submission file is untarred. The tar command should be: > tar MySystemSubmissionFile.tgz or > tar MySystemSubmissionFile.zip respectively.
Prior to uploading the submission file to the NIST scoring server, performers will be asked for information about the submission. The scoring server will attach the following information to the submission filename to catorigize and uniquely identify the submission:
Field Information Method
TeamID [Team] obtained from login information
Task {SAD | ASR | KWS} select from drop-down menu
SubmissionType {primary | contrastive} select from drop-down menu
SubmissionType {primary | contrastive} select from drop-down menu
Training Condition {unconstrained} default - hard-coded
EvalPeriod {2019} default - hard-coded
DatasetName {PSC | VAST | Babel} select from drop-down menu
Date {YYYYMMDD} obtained from NIST scoring server at submission date
TimeStamp {HHMMSS} obtained from NIST scoring server at submission time

Below is an example of a resulting filename:
NIST_ASR_primary_uncontrained_2019_PSC_20190415_163026_ MySystemSubmissionFile.tgz

The NIST scoring server will perform a validation check on each system output submission for conforming to the submission format required for each task.
Submission of a system description conforming to the system description guidelines in Appendix V is required before receiving the system’s score and ranking results in the Evaluation phase.
Overview
For the first time OpenSAT will be partnering with Linguistic Data Consortium (LDC) in hosting the Third DIHARD Speech Diarization Challenge (DIHARD III). All DIHARD III evaluation activities (registration, results submission, scoring, and leaderboard display) will be conducted through web-interfaces hosted by OpenSAT.

Challenge
For additional information about DIHARD III, including registration, schedule, data, tasks, and scoring, please consult the official DIHARD III website.

Workshop

The results of the challenge will be presented at a post-evaluation workshop, to be collocated with Interspeech 2020 in Shanghai, China on October 25th, 2020. All participants are invited to submit 2 page extended abstracts describing their submissions and results on the DIHARD III development and evaluation sets. Provisions will be made for remote participation to accommodate those unable to travel to Shanghai due to COVID-19 related disruptions. For additional details, please see the workshop website.

Contact
For more information about DIHARD III, please join the mailing list or contact the organizers via email at dihardchallenge@gmail.com.
Scoring and Validation tools: F4DE and SCTK
Here is the Github links for SCTK and F4DE:
SCTK: https://github.com/usnistgov/SCTK for the Automatic Speech Recognition (ASR)
F4DE: https://github.com/usnistgov/F4DE for the Keyword Search (KWS)

For the Speech Activity Detection (SAD), the tools (Validation & Scoring) are available in the Dashboard after registration and approval of the license agreement.
FAQ


Questions? Email questions to opensat_poc@nist.gov