Visa's Agentic Commerce Benchmark Finds AI Shopping Agents Fail 42.8% of Runs

TL;DR

Visa Agentic Commerce Benchmark launched November 5 with first-round numbers worth pinning above the desk of anyone shipping a shopping agent. Visa ran 22 models from 12 labs through 15 buying scenarios where the customer withholds most of what matters; the models passed 57.2% of runs, and in 54% of the failed runs the agent bought something anyway. Every model got three attempts per scenario and passed all three just 34.2% of the time. The test, the replays and a whitepaper are public.

How a run passes

Each scenario is a conversation with a shopper who wants a TV, a laptop or a personal care appliance and volunteers only part of the brief. In the TV case the customer mentions a brand a relative recommended and a third-party membership, then stops talking. Budget, format compatibility, HDR10+ support for family videos and gaming needs stay hidden until the agent digs.

Passing has two parts: uncover every requirement the task defines, then make the correct buy-or-walk call, including declining when nothing fits. A lucky guess that lands on the right TV without eliciting the requirements fails the run. Visa calls this the first agentic commerce benchmark that requires both.

It is 20 questions with a purchase at the end: you earn the buy button before you press it.

The scoreboard

The overall pass rate was 57.2%. The best models passed 82.2% of attempts, the worst 28.9%, a 53-point spread. Visa's release keeps the models anonymous; Fortune's Jeff John Roberts reported that agents built on Google's Gemini (the Aragon 4 version) and Meta's Making (the Wonder version) tied for the lead at 82%. A Claude agent came third at 76%, an OpenAI agent sixth at 64%.

Questions beat money

The strongest signal in the dataset is elicitation. The five models that uncovered the most hidden buying factors through conversation passed 70.2% of runs; the five that uncovered the fewest passed 44.4%. Good questioning moved the score 25.8 points, more than anything else Visa measured.

Spending more per run helped less. The five costliest models passed 67.1% of runs against 50.2% for the five cheapest, a 16.9-point gap. Paying up is an option, asking better questions is cheaper.

The consistency hole

Every model got three attempts at each scenario. Across all models and scenarios, all three attempts passed 34.2% of the time, and models that had cleared a scenario often failed it on the retry. An agent that cannot reproduce yesterday's good decision behaves more like a slot machine than an assistant.

The failure mode that should worry merchants most: 54% of failed runs still ended in a purchase. Either the agent recommended something that did not meet the requirements it knew about, or it bought before finding all of them.

Why a card network is grading your agent

Visa's Trust Index, published in September, found 72% of US consumers had used an AI assistant while only 23% trusted AI to handle payment transactions for them. That gap is the business case for this benchmark. "AI agents have the potential to power the next era of commerce, but trust is key to making it work," Rubail Birwadker, Visa's global head of growth, said in the announcement. He told Fortune: "We're not there yet from a trust or technology perspective. But we'll soon get into a world where consumers can trust and delegate."

The benchmark sits beside Visa Intelligent Commerce, the company's agentic-payments portfolio, though Visa says the research stands separate from the products. It has shared the benchmark with model providers for feedback, and plans to add scenarios and product categories, then extend testing to checkout, payment and order placement, the part where Visa earns its toll. Mastercard is building agent-payment tooling of its own.

What to steal if you build agents

• Invest in the interview before the catalog search. The gap between good and bad question-askers, 70.2% versus 44.4%, dwarfs every other effect in the dataset.
• Grade your agent on repeats. If frontier models pass all three attempts 34.2% of the time, a single-shot demo is marketing material, not a reliability number.
• Ship the no-purchase path as a feature. The pass rule rewards declining when nothing fits, and the agents that bought anyway failed the customer twice.
• Copy the format. Fifteen scenarios with known hidden requirements, a conversation, and a buy-or-walk verdict is a tidy template for your own evals, and the replays are public.
• Question the assumption that the priciest API wins. The 16.9-point spread between cost tiers is real; the 25.8-point spread from asking better questions costs nothing.

Caveats

• Visa runs the benchmark and sells agentic-commerce infrastructure, so it profits from the trust work the results argue for. Its own FAQ stresses the results show "promising capabilities" with the top models at 82.2%.
• Fifteen scenarios with three runs each is a small sample, and it covers product discovery only. Checkout, payment and order placement are untested until the next rounds.
• The model names come from Fortune's reporting; the release itself keeps all 22 anonymous.

Key Takeaways

• Visa's Agentic Commerce Benchmark, launched November 5, ran 22 models from 12 labs through 15 shopping scenarios: 57.2% overall pass rate, 82.2% at the top, 28.9% at the bottom.
• A pass requires finding every hidden customer requirement and making the correct buy-or-no-buy call; Visa says it is the first agentic commerce benchmark to require both.
• Per Fortune, agents on Google's Gemini (Aragon 4) and Meta's Making (Wonder) tied for first at 82%; a Claude agent scored 76% and an OpenAI agent 64%.
• Question-asking separated the field most: the top five elicitors passed 70.2% against 44.4% for the bottom five, outweighing the cost effect (67.1% versus 50.2%).
• Consistency is the weak point: all three attempts passed only 34.2% of the time, and 54% of failed runs still ended in a purchase.
• Visa's Trust Index frames the stakes: 72% of US consumers have used an AI assistant, 23% trust one with payments. Checkout and payment testing comes next.

Sources: Visa press release, Visa Payments Frontier, Fortune, The Next Web, PYMNTS, Visa Trust Index