Henkan News ·
Agentic commerceReliabilityVisaBenchmarkReturnsProduct data

Visa tested 22 AI models on real shopping tasks: 42.8% of runs failed, and in 54% of those the AI bought anyway

Alexandre Suon·7 min read·
54% of failed AI agent shopping runs still end in a purchase, according to Visa's benchmark
Figure: Visa Agentic Commerce Benchmark, press release of November 5, 2026.

On 5 November, Visa published a benchmark that puts AI agents through realistic shopping scenarios. They have to work out what the customer needs, then buy the right product or buy nothing. The results, what they say about agent reliability, and two direct consequences for an e-commerce director: returns and product data.

In this articleShow all sectionsHide
  1. 30-second summary
  2. 1. The announcement
  3. 2. The numbers
  4. 3. The failures
  5. 4. Consistency
  6. 5. Returns
  7. 6. Product data
  8. Key terms
  9. Sources

30-second summary

ShowHide ⌄
  • On 5 November 2026, Visa launched the "Visa Agentic Commerce Benchmark": 22 AI models from 12 labs, tested on 15 shopping scenarios (a TV, a laptop, a personal care appliance), with three attempts per scenario.
  • To pass, the AI has to uncover every customer requirement through conversation, then buy the right product, or buy nothing if nothing fits. Models failed 42.8% of runs.
  • In 54% of failed runs, the AI still made a purchase. The best model passed 82.2% of its attempts, the worst 28.9%.
  • The most telling number is about consistency: a scenario is passed on all three attempts only 34.2% of the time. The same agent, facing the same customer, does not always give the same answer.
  • For a retailer, two consequences: track the return rate of agent-placed orders, and publish complete product data, because the agent only knows what you write down.

Section 1 · The announcement

Visa puts AI agents in front of realistic customers, with one simple rule: buy right or buy nothing

On 5 November 2026, Visa published the "Visa Agentic Commerce Benchmark", a test of how well AI models shop on a customer's behalf. Visa describes it as the first agentic commerce benchmark to require both discovering every customer requirement and making a correct purchase decision.

  • 22 AI models from 12 labs in this first round.
  • 15 realistic shopping scenarios, such as buying a TV, a laptop or a personal care appliance.
  • Three attempts per model and per scenario.

The setup is close to a real sale. The customer does not say everything up front. The AI has to ask the right questions to find the hidden requirements. Then it buys a product that meets them, or buys nothing if none does. A run only passes if both conditions are met.

Benchmark, in plain words A benchmark is a standardised test: the same tasks, given to several models, scored with the same rules. It lets you compare models and track how they improve from one version to the next.

Section 2 · The numbers

Almost one run in two fails, and the gap between models is huge

Exhibit 1 · Pass rate of shopping runs
% of runs, 22 models, 15 scenarios, 3 attempts each
Best model82.2%Average across models57.2%Worst model28.9%
Source : Visa, "Visa Agentic Commerce Benchmark", press release of 5 November 2026

On average, models pass 57.2% of runs. Put the other way, they fail 42.8% of the time. The best reaches 82.2%, the worst 28.9%. Which model sits behind an agent makes a big difference.

Exhibit 2 · What separates the stronger models from the weaker ones
Group of modelsPass rateCompared with
The 5 that uncover the most hidden requirements70.2%44.4% for the 5 that uncover the least
The 5 most expensive per run67.1%50.2% for the 5 cheapest
Source : Visa, "Visa Agentic Commerce Benchmark", press release of 5 November 2026

Models that ask the right questions do clearly better. That is the core of the test: an agent that does not understand the need ends up picking at random among plausible products.

Section 3 · The failures

In 54% of failed runs, the AI places an order anyway

This is the number a retailer should look at first: in 54% of failed runs, the AI still bought something. It missed part of the need, or it should have held back, and it placed an order all the same.

Across all runs, that is close to one run in four (42.8% × 54%, about 23%, Henkan calculation) ending in a purchase the customer would not have approved.

“An agent that picks the wrong product is an order today and a return in ten days.”

Why "buy nothing" matters A good salesperson knows how to say "we don't have what you need". Visa's test scores that too. An agent that always buys something pushes order volume up, but not necessarily the sales that stick.

Section 4 · Consistency

A scenario is passed on all three attempts only 34.2% of the time

Each model had three attempts per scenario. Across all models and scenarios, all three attempts passed just 34.2% of the time. The same agent, given the same request, can succeed once and fail the next time.

Exhibit 3 · Passing once, passing every time
% of runs or scenarios, all models combined
Runs passed, on average57.2%Scenarios passed on all 3 attempts34.2%
Source : Visa, "Visa Agentic Commerce Benchmark", press release of 5 November 2026

For a retailer, this is the most awkward number. You can manage an average pass rate. Behaviour that changes from one attempt to the next is much harder to manage: two customers with the same need can end up with two different products.

Visa has not tested payment yet. The release says it will add more scenarios and product categories, then extend testing to checkout, payment and order placement. The benchmark has also been shared with leading model providers.

Section 5 · Returns

Track the return rate of agent-placed orders, not just their volume

If an agent gets it wrong and orders anyway, the mistake reaches you as a return. It costs logistics and customer service time, and sometimes leaves a customer disappointed with the brand when the agent was at fault.

Exhibit 4 · The path of an order an agent got wrong
  1. 1
    The customer's requestThe customer describes the need to the agent, without spelling everything out.
  2. 2
    The orderThe agent missed some requirements and buys a plausible product. In your reports, it counts as a sale.
  3. 3
    The returnThe product doesn't fit. Ten days later, the sale becomes a return.

Order volume from agents will say little about what they are worth. You need to be able to isolate these orders and track their return rate, and return reasons, next to those of regular orders.

Section 6 · Product data

The agent only knows what you publish: an incomplete product page makes it guess

The models that uncover the most requirements do best. But understanding the customer is only half the job: the agent then has to match that need to your products. All it has for that is what you publish. If compatibility, dimensions or return conditions are missing, it guesses.

  1. Get ready to track agent-placed orders: a source flag on orders, then return rate and return reasons compared with regular orders.
  2. List the criteria that decide whether a product fits in your key categories (size, compatibility, use, power) and check they are written on every product page, as readable text.
  3. Spell out when a product does not fit: incompatibilities, usage limits, conditions. That is what allows an agent to decide not to buy.
  4. Keep delivery and return conditions up to date and visible on the product page, not only in the terms and conditions.
Exhibit 5 · What the agent looks for, and what happens if it can't find it
Product dataCustomer questionIf the information is missing
Compatibility"Will it work with my setup?"The agent assumes it will
Dimensions"Will it fit in my living room?"The agent chooses from the general description
Use and limits"Is it right for my case?"The agent buys a plausible product
Return conditions"What if it doesn't work out?"The customer finds out afterwards
Source : Henkan grid based on the Visa benchmark results
Henkan's point of view. Visa's benchmark gets one thing right: it measures what matters to a merchant, buying right or buying nothing, and not just the ability to complete a purchase. What strikes us most at Henkan is consistency. When a scenario is passed on all three attempts only a third of the time, an agent cannot be managed like a stable channel. There is no need to wait for agents to pay before preparing. Two workstreams are useful now: measure returns on agent-placed orders from the very first ones, and complete product pages with the criteria that decide whether a product fits. That second one helps human customers too.

Key terms

ShowHide ⌄
Agentic commerce
Purchases made by an AI agent on the customer's behalf: search, comparison, cart and payment.
Benchmark
A standardised test that gives the same tasks to several models so they can be compared under the same rules.
Run
One attempt by a model at a scenario. In Visa's test, each model made three runs per scenario.
Hidden requirement
A need the customer does not state up front and that the agent has to uncover by asking questions.
Return rate
The share of orders sent back by customers. Tracked by source, it shows whether a channel brings sales that stick.

Sources

ShowHide ⌄
  1. Visa, "Visa Releases First-of-Its-Kind Industry Benchmark for AI Shopping; Almost Half of Test Runs Fail", press release, 5 November 2026
  2. The same release distributed by Business Wire, 5 November 2026
Alexandre Suon
Alexandre SuonManaging Partner, Henkan & Partners

Alexandre co-founded and runs Henkan & Partners. He decodes one e-commerce story every day on LinkedIn.

Follow Alexandre ↗