Submission

Result Submission

Result submission is organized by track and must remain consistent with the registered team information for that track. Before the first submission, complete team registration through the Registration page.

Track 1 Format

Track 1 submissions currently follow the jsonl format shown below.

{"question_id": "gigaspeech_0_1", "answer": "A"}
{"question_id": "gigaspeech_0_2", "answer": "B"}

Note:

  • The question_id should be the same as the ones in the test set.
  • Only submit ONE file containing the results of all the subsets.
  • Direct API calls are strictly prohibited: Test data must not be used in any API-related process, including preprocessing, intermediate processing steps, inference, result prediction, or evaluation. APIs may only be used for model training purposes.

Track 2 Format

For each sample, the input contains a json file and video frames sampled at 0.5 fps:

files for each question
1  question.json
2  0.5.png  // the frame sampled at 0.5 sec
3  1.0.png
4  ...
5  16.5.png // the last frame of the video
question.json
1  {
2    "question_id": "IM9NZMRfLd4.1.mp4",
3    "question": "What kind of items are sold in the store?"
4  }

Your system should resemble the following python pseudocode:

inference pseudocode
1  input(question)
2
3  for current_time in range(0.5, video_length, 0.5):
4      # READING FRAMES FROM t > current_time IS STRICTLY PROHIBITED NOW
5      input(f"{current_time}.png")
6
7      if respond_now:
8          output(f"Model response at t={current_time}s")

You should submit a jsonl file, one line for each sample. Each line is a json object formatted as follow: 

{
  "question_id": "IM9NZMRfLd4.1.mp4",
  "model_response_list": [
    {"time": 1.0, "content": "Model response at t=1.0s"},
    {"time": 5.5, "content": "Model response at t=5.5s"}
  ]
}

Upload Requirements

Google Drive upload link: Track 1 || Track 2
  • For each team, upload the jsonl result file to the designated Google Drive submission folder.
  • The file name must use the Team ID, exactly the same as the registration.
  • Case-sensitive consistency is required.
  • Any violation will result in an error when dealing with the submission file and the submission will be considered invalid.
  • The same upload links and requirements for Phase 2.

Leaderboard Updates

  • Each Monday and Friday, organizers will download the latest submissions.
  • Organizers will then run evaluation and update the leaderboard.

Track leaderboards: