THETOOLDECATHLON: BENCHMARKINGLANGUAGE AGENTS FORDIVERSE, REALISTIC,ANDLONG- HORIZONTASKEXECUTION
G
Shared by Gary on Oct 31, 2025 • 03:49 AM UTC
NobleBlocks IDNB3194197A
Abstract
Real-world language agents must handle complex, multi-step workflows across diverse applications. For instance, an agent may manage emails by coordinating with calendars and file systems, or monitor a production database like BigQuery to detect anomalies and generate reports following a standard ope...
Subject
Highlights
-TOOLATHLONspans 32 real-world applications and 604 tools across 108 tasks.
-The majority of tools are sourced from MCP servers, with several open-source implementations refined by the authors.
-The benchmark uses realistic initial environment states drawn from real software, such as Canvas courses and financial spreadsheets.
-The best-performing model Claude-4.5-Sonnet achieves 38.6% accuracy with 20.2 tool calling turns on average; among open weights, DeepSeek-V3.2-Exp reaches 20.1%.
NobleBlocks' AI Take
Finding related papers...
Discussions
(0)No comments yet
Be the first to share your thoughts!