Submission

Track 1 Format

Track 1 submissions currently follow the jsonl format shown below.

{"question_id": 0, "answer": "A"}
{"question_id": 1, "answer": "B"}

Track 2 Format

For each sample, the input contains a json file and video frames sampled at 0.5 fps:

files for each question
1  question.json
2  0.5.png  // the frame sampled at 0.5 sec
3  1.0.png
4  ...
5  16.5.png // the last frame of the video

question.json
1  {
2    "question_id": "IM9NZMRfLd4.1.mp4",
3    "question": "What kind of items are sold in the store?"
4  }

Your system should resemble the following python pseudocode:

inference pseudocode
1  input(question)
2
3  for current_time in range(0.5, video_length, 0.5):
4      # READING FRAMES FROM t > current_time IS STRICTLY PROHIBITED NOW
5      input(f"{current_time}.png")
6
7      if respond_now:
8          output(f"Model response at t={current_time}s")

You should submit a jsonl file, one line for each sample. Each line is a json object formatted as follow:

{
  "question_id": "IM9NZMRfLd4.1.mp4",
  "model_response_list": [
    {"time": 1.0, "content": "Model response at t=1.0s"},
    {"time": 5.5, "content": "Model response at t=5.5s"}
  ]
}

Upload Requirements

• For each team, upload the jsonl result file to the designated Google Drive submission folder.
• The file name must use the Team ID, exactly the same as the registration.
• Case-sensitive consistency is required.
• Any violation will result in an error when dealing with the submission file and the submission will be considered invalid.

Leaderboard Updates

• Each Monday and Friday, organizers will download the latest submissions.
• Organizers will then run evaluation and update the leaderboard.

Track leaderboards:

• Track 1 Leaderboard
• Track 2 Leaderboard