† The first five authors contributed equally.
SEATauBench adapts τ²-bench to Thai, Vietnamese, Indonesian, Mandarin Chinese, and Filipino across Airline, Retail, and Telecom. Four settings separate dialogue, tool understanding, and domain reasoning while keeping the underlying tasks and scoring aligned.
Multilingual language models can hold a conversation, but can they complete a customer-service task when the tools, policies, and data also change language? Agent capabilities in Southeast Asian languages remain poorly understood, and English-only evaluation gives an incomplete picture.
Four settings isolate different language surfaces. S1 is the English reference; S2 localizes dialogue, S3 tool schemas, and S4 the full environment. Each row is a separate condition.
| Setting | Dialogue | Tool schemas | DB & policy |
|---|---|---|---|
| S1 · EnglishThe reference environment | English | English | English |
| S2 · InteractionChange the conversation | L2 | English | English |
| S3 · ToolsChange the tool interface | English | L2 | English |
| S4 · Full domainChange the full environment | L2 | L2 | L2 |
L2 is Vietnamese, Thai, Indonesian, Mandarin Chinese, or Filipino. S3 keeps dialogue in English.
The same customer request can expose different language demands. The example below follows a retail hat exchange through an English environment and a fully localized Thai environment. Compare the conversation and tool responses to see which surfaces change; the accompanying scores summarize benchmark conditions, rather than the outcome of this single illustrated exchange.
One task. Two language environments.
You are a helpful agent...
I want to change the color of my ordered hat
get_order_details[SC250526]"""Get the status and details of an order...{'order_id': 'SC250526', 'item': 'hat','price': 25, 'color': 'red', ...}คุณเป็นเจ้าหน้าที่ที่ให้ความช่วยเหลือ...
ฉันต้องการเปลี่ยนสีของหมวกที่สั่งซื้อไป
get_order_details[SC250526]"""ตรวจสอบสถานะและรายละเอียดคำสั่งซื้อ...{'order_id': 'SC250526', 'item': 'หมวก','price': 25, 'color': 'สีแดง', ...}The adaptation pipeline separates what the agent sees from what the benchmark executes. Offline translation adapts the language while preserving identifiers and executable structure; runtime mappings connect localized tool arguments and responses to canonical values. Automated checks and native-speaker review then assess preservation and translation quality, with approved corrections recorded in a separate reviewed release.
A glossary guides offline translation. Identifiers, executable tokens, and structure are preserved across task descriptions, policies, schemas, and databases.
Runtime mappings normalize localized tool arguments for execution and localize returned content. Scoring compares canonical outcomes.
Automated preservation checks and native-speaker review assess translation quality. Approved corrections are recorded in a separate reviewed release.
Localization must preserve the task the agent is asked to complete.
Changing the conversation language alone preserves much of the English performance, while full localization exposes a larger task-success gap. The five panels below compare Thai, Vietnamese, Indonesian, Chinese, and Filipino across S1 (English baseline), S2 (L2 interaction), S3 (L2 tools), and S4 (L2 domain). Blue circles show mean task success (pass@1), and red squares show repeated-run robustness (ρ³); the connected series summarize each setting, while the individual points show model averages.
Task success and robustness reveal different aspects of multilingual performance. In the plot below, the radar charts compare pass@1 and ρ³ across English and the five target languages, while the domain panels compare English and non-English means for Airline, Retail, and Telecom. pass@1 measures mean task success; ρ³ relates success across three trials to pass@1, so higher robustness does not by itself mean that more tasks are solved. The gaps vary by model, language, and domain, showing why English results alone give an incomplete picture.
Paper v2 snapshot05 September 2026
Explore success and consistency under the same evaluation conditions. The leaderboard starts with S4, the fully localized setting, across all five target languages and three domains; use the filters to compare other selections. Each score is an equal-weight average of the selected language × domain cells: pass@1 measures mean task success, while pass² and pass³ measure success in all two or three trials, respectively. ρ³ is pass³ divided by pass@1 within each published cell and then averaged separately, so ranking by consistency can produce a different leader from ranking by success.
Published results · no live submissions
Swipe the table to compare all four metrics →
| Rank | Model | pass@1 | pass² | pass³ | ρ³ |
|---|---|---|---|---|---|
| 01 | Kimi-K2.515 published results | 60.9% | 46.6% | 37.7% | 59.6% |
| 02 | GPT-5-mini15 published results | 56.9% | 43.3% | 36.3% | 62.3% |
| 03 | Qwen3-235B15 published results | 35.9% | 27.7% | 23.8% | 62.9% |
Equal-weight macro-average over selected language × domain cells.
3 models · 45 published results · ranked by pass@1
Table 19 reports all three models for S3. The paper’s main-text aggregate discussion describes two. This leaderboard uses the explicit appendix values.
pass@1 mean task success · pass² / pass³ success across repeated trials · ρ³ consistency relative to pass@1.
Look beneath the averages to see where performance differs. Each row below reports one model, language, and domain under the selected localization setting, with its published success and consistency scores. These rows follow the leaderboard filters, and the source links identify the paper table behind each result.
| Model | Language | Domain | pass@1 | pass² | pass³ | ρ³ | Source |
|---|---|---|---|---|---|---|---|
| GPT-5-mini | Vietnamese | Airline | 54.0% | 40.0% | 32.0% | 59.3% | Table 21 |
| GPT-5-mini | Thai | Airline | 59.3% | 45.3% | 36.0% | 60.7% | Table 21 |
| GPT-5-mini | Indonesian | Airline | 60.0% | 54.0% | 50.0% | 83.3% | Table 21 |
| GPT-5-mini | Mandarin Chinese | Airline | 60.7% | 49.3% | 44.0% | 72.5% | Table 21 |
| GPT-5-mini | Filipino | Airline | 56.0% | 43.3% | 38.0% | 67.9% | Table 21 |
| Qwen3-235B | Vietnamese | Airline | 48.7% | 46.0% | 44.0% | 90.3% | Table 21 |
| Qwen3-235B | Thai | Airline | 46.0% | 41.3% | 38.0% | 82.6% | Table 21 |
| Qwen3-235B | Indonesian | Airline | 48.0% | 44.0% | 42.0% | 87.5% | Table 21 |
| Qwen3-235B | Mandarin Chinese | Airline | 50.0% | 47.3% | 46.0% | 92.0% | Table 21 |
| Qwen3-235B | Filipino | Airline | 47.3% | 42.0% | 40.0% | 84.6% | Table 21 |
| Kimi-K2.5 | Vietnamese | Airline | 56.0% | 42.7% | 34.0% | 60.7% | Table 21 |
| Kimi-K2.5 | Thai | Airline | 54.7% | 38.7% | 30.0% | 54.8% | Table 21 |
| Kimi-K2.5 | Indonesian | Airline | 60.0% | 48.7% | 42.0% | 70.0% | Table 21 |
| Kimi-K2.5 | Mandarin Chinese | Airline | 66.0% | 50.7% | 44.0% | 66.7% | Table 21 |
| Kimi-K2.5 | Filipino | Airline | 60.0% | 48.7% | 40.0% | 66.7% | Table 21 |
| GPT-5-mini | Vietnamese | Retail | 58.5% | 42.7% | 33.3% | 56.9% | Table 21 |
| GPT-5-mini | Thai | Retail | 32.5% | 17.5% | 12.3% | 37.8% | Table 21 |
| GPT-5-mini | Indonesian | Retail | 54.1% | 39.2% | 32.5% | 60.1% | Table 21 |
| GPT-5-mini | Mandarin Chinese | Retail | 59.6% | 45.0% | 36.8% | 61.7% | Table 21 |
| GPT-5-mini | Filipino | Retail | 48.8% | 33.3% | 26.3% | 53.9% | Table 21 |
| Qwen3-235B | Vietnamese | Retail | 28.9% | 20.5% | 14.9% | 51.6% | Table 21 |
| Qwen3-235B | Thai | Retail | 17.8% | 14.3% | 13.2% | 74.2% | Table 21 |
| Qwen3-235B | Indonesian | Retail | 24.3% | 17.0% | 14.0% | 57.6% | Table 21 |
| Qwen3-235B | Mandarin Chinese | Retail | 26.6% | 18.1% | 14.9% | 56.0% | Table 21 |
| Qwen3-235B | Filipino | Retail | 26.0% | 18.7% | 15.8% | 60.8% | Table 21 |
| Kimi-K2.5 | Vietnamese | Retail | 56.7% | 39.2% | 26.8% | 47.3% | Table 21 |
| Kimi-K2.5 | Thai | Retail | 32.7% | 19.0% | 11.4% | 34.9% | Table 21 |
| Kimi-K2.5 | Indonesian | Retail | 43.3% | 28.7% | 20.2% | 46.7% | Table 21 |
| Kimi-K2.5 | Mandarin Chinese | Retail | 60.7% | 45.0% | 36.6% | 60.3% | Table 21 |
| Kimi-K2.5 | Filipino | Retail | 44.4% | 29.2% | 21.9% | 49.3% | Table 21 |
| GPT-5-mini | Vietnamese | Telecom | 56.4% | 42.7% | 36.0% | 63.8% | Table 21 |
| GPT-5-mini | Thai | Telecom | 59.9% | 44.7% | 36.0% | 60.1% | Table 21 |
| GPT-5-mini | Indonesian | Telecom | 49.7% | 32.5% | 24.6% | 49.5% | Table 21 |
| GPT-5-mini | Mandarin Chinese | Telecom | 76.0% | 65.2% | 58.8% | 77.4% | Table 21 |
| GPT-5-mini | Filipino | Telecom | 68.1% | 54.7% | 47.4% | 69.6% | Table 21 |
| Qwen3-235B | Vietnamese | Telecom | 38.3% | 21.3% | 14.9% | 38.9% | Table 21 |
| Qwen3-235B | Thai | Telecom | 28.4% | 14.3% | 7.9% | 27.8% | Table 21 |
| Qwen3-235B | Indonesian | Telecom | 42.7% | 29.5% | 21.1% | 49.4% | Table 21 |
| Qwen3-235B | Mandarin Chinese | Telecom | 33.0% | 20.5% | 14.9% | 45.2% | Table 21 |
| Qwen3-235B | Filipino | Telecom | 33.0% | 20.5% | 14.9% | 45.2% | Table 21 |
| Kimi-K2.5 | Vietnamese | Telecom | 69.9% | 52.9% | 40.4% | 57.8% | Table 21 |
| Kimi-K2.5 | Thai | Telecom | 69.3% | 52.3% | 43.0% | 62.0% | Table 21 |
| Kimi-K2.5 | Indonesian | Telecom | 79.8% | 68.1% | 58.8% | 73.7% | Table 21 |
| Kimi-K2.5 | Mandarin Chinese | Telecom | 86.3% | 76.3% | 68.4% | 79.3% | Table 21 |
| Kimi-K2.5 | Filipino | Telecom | 74.3% | 58.2% | 47.4% | 63.8% | Table 21 |
Table 20 evaluates GPT-5-mini and Qwen3-235B with mixed-language tool specifications. Mix-2 uses English + Thai; Mix-3 adds Vietnamese; Mix-4 adds Indonesian; Mix-5 adds Chinese. These 24 results are separate from the main leaderboard and do not respond to its filters. Compare mixtures within the same model and domain to see how success and consistency vary as the tool interface includes more languages; the mixture labels describe tool specifications, rather than the conversation language.
| Model | Mixture | Domain | pass@1 | pass² | pass³ | ρ³ | Source |
|---|---|---|---|---|---|---|---|
| GPT-5-mini | Mix-2 | Airline | 68.0% | 56.0% | 48.0% | 70.6% | Table 20 |
| GPT-5-mini | Mix-3 | Airline | 66.0% | 56.0% | 48.0% | 72.7% | Table 20 |
| GPT-5-mini | Mix-4 | Airline | 64.0% | 52.0% | 48.0% | 75.0% | Table 20 |
| GPT-5-mini | Mix-5 | Airline | 64.0% | 51.3% | 44.0% | 68.8% | Table 20 |
| Qwen3-235B | Mix-2 | Airline | 44.0% | 28.0% | 20.0% | 45.5% | Table 20 |
| Qwen3-235B | Mix-3 | Airline | 41.3% | 26.7% | 20.0% | 48.4% | Table 20 |
| Qwen3-235B | Mix-4 | Airline | 41.3% | 28.7% | 24.0% | 58.1% | Table 20 |
| Qwen3-235B | Mix-5 | Airline | 41.3% | 26.7% | 22.0% | 53.3% | Table 20 |
| GPT-5-mini | Mix-2 | Retail | 66.7% | 54.1% | 46.5% | 69.7% | Table 20 |
| GPT-5-mini | Mix-3 | Retail | 66.7% | 53.8% | 45.6% | 68.4% | Table 20 |
| GPT-5-mini | Mix-4 | Retail | 65.2% | 48.5% | 38.6% | 59.2% | Table 20 |
| GPT-5-mini | Mix-5 | Retail | 69.6% | 54.7% | 44.7% | 64.2% | Table 20 |
| Qwen3-235B | Mix-2 | Retail | 59.6% | 45.6% | 38.6% | 64.8% | Table 20 |
| Qwen3-235B | Mix-3 | Retail | 53.5% | 39.8% | 32.5% | 60.7% | Table 20 |
| Qwen3-235B | Mix-4 | Retail | 58.5% | 44.7% | 37.7% | 64.4% | Table 20 |
| Qwen3-235B | Mix-5 | Retail | 57.0% | 40.1% | 29.8% | 52.3% | Table 20 |
| GPT-5-mini | Mix-2 | Telecom | 66.4% | 54.1% | 48.2% | 72.6% | Table 20 |
| GPT-5-mini | Mix-3 | Telecom | 69.6% | 55.8% | 49.1% | 70.5% | Table 20 |
| GPT-5-mini | Mix-4 | Telecom | 67.0% | 55.0% | 49.1% | 73.3% | Table 20 |
| GPT-5-mini | Mix-5 | Telecom | 65.2% | 52.0% | 46.5% | 71.3% | Table 20 |
| Qwen3-235B | Mix-2 | Telecom | 29.8% | 15.5% | 8.8% | 29.5% | Table 20 |
| Qwen3-235B | Mix-3 | Telecom | 39.2% | 21.3% | 14.0% | 35.7% | Table 20 |
| Qwen3-235B | Mix-4 | Telecom | 36.5% | 18.1% | 10.5% | 28.8% | Table 20 |
| Qwen3-235B | Mix-5 | Telecom | 34.2% | 18.1% | 11.4% | 33.3% | Table 20 |
Success across repeated trialspass@1 is mean task success. pass² and pass³ measure success across two and three trials, respectively; they are not success on any one attempt.
Consistency is a different questionρ³ = pass³ / pass@1 for each published cell. We average reported ρ³ values separately. High robustness can coexist with low overall task success.
Source: paper v2, Tables 17–19 and 21. Equal weight per language/domain cell, not per individual task. Values retain the paper’s precision; ties at the displayed precision share a rank. Qwen3-235B denotes Qwen3-235B-A22B-Instruct-2507.
Scores describe the frozen experiment snapshot, not the later reviewed translation release. S3’s appendix lists three models although the main-text aggregate comparison describes two. Rankings here are computed from the appendix cells and may differ from rounded narrative summaries. The benchmark uses 50 Airline tasks and 114 tasks each in Retail and Telecom, with three trials per task. The user simulator is Qwen3-235B-A22B-Instruct; natural-language assertions use GPT-4.1. Download the complete source dataset.
Pronounced “si-tau-bench,” like sitaw, the Filipino word for string beans.