Are you looking for smarter insights delivered directly to your inbox? Subscribe to our weekly newsletters for essential updates on enterprise AI, data, and security.
The Importance of Interoperability Standards
The adoption of interoperability standards, such as the Model Context Protocol (MCP), offers enterprises valuable insights into the functioning of agents and models beyond their isolated environments. However, many existing benchmarks do not effectively capture real-world interactions with MCP. To address this gap, Salesforce AI Research has developed a new open-source benchmark called MCP-Universe. This benchmark aims to track large language models (LLMs) as they interact with MCP servers in real-world scenarios, asserting that it will provide a more accurate representation of real-time interactions between models and the tools that enterprises utilize.
Insights from Initial Testing
In its preliminary tests, Salesforce found that models like OpenAI’s recently released GPT-5 demonstrate strong capabilities but still fall short in real-life applications. According to Salesforce, “Existing benchmarks predominantly focus on isolated aspects of LLM performance, such as instruction following, math reasoning, or function calling, without providing a comprehensive assessment of how models interact with real-world MCP servers across diverse scenarios.” MCP-Universe captures model performance through various metrics, including tool usage, multi-turn tool calls, long context windows, and expansive tool spaces. It is founded on existing MCP servers with access to actual data sources and environments.
Challenges in AI Scaling
Power limitations, rising token costs, and inference delays are significantly impacting enterprise AI. Join our exclusive salon to learn how leading teams are transforming energy into a strategic advantage, architecting efficient inference for real throughput gains, and unlocking competitive ROI with sustainable AI systems. Secure your spot to stay ahead: https://bit.ly/4mwGngO
Junnan Li, the director of AI research at Salesforce, shared with VentureBeat that many models still encounter limitations that hinder their effectiveness in enterprise tasks. He identified two major challenges: long context issues, where models may lose track of information or struggle with reasoning when faced with lengthy or complex inputs, and unknown tool challenges, where models often cannot seamlessly adapt to unfamiliar tools or systems as humans can. Li emphasized the importance of not relying solely on a single model to power agents but instead utilizing a platform that integrates data context, enhanced reasoning, and trust guardrails to meet the needs of enterprise AI.
MCP-Universe Compared to Other Benchmarks
MCP-Universe joins other proposed benchmarks based on MCP, such as MCP-Radar from the University of Massachusetts Amherst and Xi’an Jiaotong University, as well as MCPWorld from the Beijing University of Posts and Telecommunications. It builds on MCPEvals, which Salesforce released in July, primarily focusing on agents. The key distinction between MCP-Universe and MCPEvals lies in the evaluation method; while MCPEvals uses synthetic tasks, MCP-Universe assesses how well each model performs a series of tasks that closely resemble those undertaken by enterprises.
Salesforce designed MCP-Universe to encompass six core domains utilized by enterprises: location navigation, repository management, financial analysis, 3D design, browser automation, and web search. They accessed 11 MCP servers to create a total of 231 tasks.
Detailed Breakdown of Domains
The location navigation domain emphasizes geographic reasoning and the execution of spatial tasks, utilizing the Google Maps MCP server. The repository management domain examines codebase operations, connecting to the GitHub MCP to expose version control tools such as repository search, issue tracking, and code editing. Financial analysis is linked to the Yahoo Finance MCP server to assess quantitative reasoning and decision-making in financial markets. The 3D design domain evaluates the use of computer-aided design tools through the Blender MCP. Browser automation, connected to Playwright’s MCP, tests browser interactions, while the web searching domain employs the Google Search MCP server and the Fetch MCP to evaluate “open-domain information seeking,” structured as a more open-ended task.
Salesforce reported that it had to create new MCP tasks that reflect genuine use cases. For each domain, they developed four to five types of tasks that researchers believe LLMs can easily complete. For instance, researchers assigned models a goal involving route planning, which included identifying optimal stops and locating the final destination.
Evaluation Methodology
Each model is assessed based on its task completion. Li and his team chose to follow an execution-based evaluation paradigm rather than the more common LLM-as-a-judge system. They noted that the LLM-as-a-judge paradigm “is not well-suited for our MCP-Universe scenario, since some tasks are designed to use real-time data, while the knowledge of the LLM judge is static.”