The SQuAD Q&A benchmark measures the time to a F1 score of .75 or greater on the SQuAD dataset.
To do well on SQuAD2.0, systems must not only answer questions when possible, but also determine when no answer is supported by the paragraph and abstain from answering. SQuAD2.0 is a challenging natural language understanding task for existing models
The current fastest time, as of June 10th 2019, is 18 mins 46 seconds.
What will the fastest time on the DAWNBench SQuAD Question & Answer benchmark be by the end of 2019?
- This question will be resolved by the publicly reported DAWNBench page.
- The measurement is in minutes.