Track 1 - EmpathyEval
Evaluation of Contextualized Affective Speech
This track evaluates whether omni-models can identify the most appropriate spoken response by combining contextual understanding with affective and paralinguistic reasoning, advancing AI capabilities for empathetic perception in speech.
Task Description
Given a textual context and an audio utterance, with a set of candidate audio responses, select the response that is most empathetic.
Evaluation compares participant predictions against human annotations, so the benchmark emphasizes human-aligned response selection rather than literal semantic overlap alone.
Subtasks
Task 1: Context-Variant
The surrounding conversational context changes, indicating different situations, and the model must determine which candidate response best fits the specific situation.
Data Example (Audio and textual context)
|
Context 1
During a bachelorette weekend, their friends surprised them with a spa package that included a manicure, and they’ve never been to a nail salon before. |
Context 2
After losing a fantasy football bet, the agreed penalty was to get a bright, glittery manicure and keep it for a week. |
|
Utterance Audio
|
|
|
Candidate Responses
A
B
✓
|
Candidate Responses
A
B
✓
|
Task 2: Tone-Variant
The model must rely on vocal and paralinguistic cues of the utterance to infer the inner state or emotion of the user, and identify which response is emotionally appropriate for the given utterance.
Data Example (Audio and textual context)
|
Context
While helping my mom clear out the guest room before her knee surgery, I pulled out a dusty box labeled "1996-2002" packed with camp Polaroids and birthday party prints. I brought the stack home to sort through after dinner a few nights later. |
|
|
Utterance Audio: tone 1
|
Utterance Audio: tone 2
|
|
Candidate Responses
A
✓
B
|
Candidate Responses
A
✓
B
|
Evaluation Metrics
- Accuracy: for each correctly predicted item, the accuracy score increases by 1.
- Grouped bonus: if all items in a context-variant or tone-variant group are predicted correctly, the bonus score increases by 1.
- Final Score:
(Accuracy + Bonus) / (#data + #group).
Dataset
Phase 2 Test Set:
1) multi-context_gigaspeech, 2) multi-context_meld, and 3) multi-emotions_emovdb
Phase 1 Test Set:
1) multi-context_gigaspeech, 2) multi-context_meld, and 3) multi-emotions_emovdb
Please download the training set for each subtask from Hugging Face:
1) empatheticDialogue_n_multi-emotion and 2) empatheticDialogue_t_multi-context
Leaderboard (Phase 1)
The leaderboard will report context-variant and tone-variant results separately, Accuracy / Bonus / Final Score, together with the weighted average of the final scores.
| Rank | Team | Context-variant | Tone-variant | Avg. |
|---|---|---|---|---|
| 1 | 111 | 343 / 113 / 0.786 | 114 / 24 / 0.920 | 457 / 137 / 0.814 |
| 2 | HumOmni_Nexus | 358 / 125 / 0.833 | 98 / 11 / 0.727 | 456 / 136 / 0.811 |
| 3 | smalltry | 345 / 118 / 0.798 | 105 / 15 / 0.800 | 450 / 133 / 0.799 |
| 4 | tryanderror2 | 326 / 104 / 0.741 | 102 / 12 / 0.760 | 428 / 116 / 0.745 |
| 5 | 19191 | 313 / 100 / 0.712 | 109 / 19 / 0.853 | 422 / 119 / 0.741 |
| 6 | Lenormand_Team | 303 / 103 / 0.700 | 106 / 16 / 0.813 | 409 / 119 / 0.723 |
| 7 | WhereIsAI-Lingnan | 289 / 84 / 0.643 | 108 / 19 / 0.847 | 397 / 103 / 0.685 |
| 8 | HumOmni_H | 256 / 63 / 0.550 | 79 / 4 / 0.553 | 335 / 67 / 0.551 |
| (baseline) | Qwen2.5-Omni-7B | 189 / 32 / 0.381 | 60 / 3 / 0.420 | 249 / 35 / 0.389 |
| (baseline) | Qwen2.5-Omni-3B | 182 / 31 / 0.367 | 66 / 3 / 0.460 | 248 / 34 / 0.386 |
Public scores are announced after the evaluation process. Current scores are submissions by Jun 29, 20:00 GMT+8.
Awards
| Rank | Team | Affiliation |
|---|---|---|
| 1 | 111 | Hong Kong Polytechnic University |
| 2 | tryanderror2 | Hong Kong Polytechnic University |
| 3 | HumOmni_Nexus | Lingnan University |