LLM Function Calls Don't Scale: Code Orchestration Is Simpler, More Effective
Feeding the full output of tool calls back into LLMs is costly and slow. This article argues that output schemas, enabling structured data retrieval, allow LLMs to orchestrate processing via generated code – a simpler and more effective approach. Traditional methods, where tool outputs are fed back to the LLM as messages for next-step determination, work well with small datasets but fail with real-world scale (e.g., large JSON blobs from Linear and Intercom MCP servers). The article proposes code execution as a fundamental data processing method, using variables as memory, and code to orchestrate multiple function calls for scalable data processing, overcoming the cost, speed, and potential data loss issues of LLMs handling large datasets. This necessitates secure, stateless AI runtime environments, currently in early development.
Read more