Visa tested 22 AI models on real shopping tasks: 42.8% of runs failed, and in 54% of those the AI bought anyway

On 5 November, Visa published a benchmark that puts AI agents through realistic shopping scenarios. They have to work out what the customer needs, then buy the right product or buy nothing. The results, what they say about agent reliability, and two direct consequences for an e-commerce director: returns and product data.
In this articleShow all sectionsHide
30-second summary
ShowHide ⌄
- On 5 November 2026, Visa launched the "Visa Agentic Commerce Benchmark": 22 AI models from 12 labs, tested on 15 shopping scenarios (a TV, a laptop, a personal care appliance), with three attempts per scenario.
- To pass, the AI has to uncover every customer requirement through conversation, then buy the right product, or buy nothing if nothing fits. Models failed 42.8% of runs.
- In 54% of failed runs, the AI still made a purchase. The best model passed 82.2% of its attempts, the worst 28.9%.
- The most telling number is about consistency: a scenario is passed on all three attempts only 34.2% of the time. The same agent, facing the same customer, does not always give the same answer.
- For a retailer, two consequences: track the return rate of agent-placed orders, and publish complete product data, because the agent only knows what you write down.
Section 1 · The announcement
Visa puts AI agents in front of realistic customers, with one simple rule: buy right or buy nothing
On 5 November 2026, Visa published the "Visa Agentic Commerce Benchmark", a test of how well AI models shop on a customer's behalf. Visa describes it as the first agentic commerce benchmark to require both discovering every customer requirement and making a correct purchase decision.
- 22 AI models from 12 labs in this first round.
- 15 realistic shopping scenarios, such as buying a TV, a laptop or a personal care appliance.
- Three attempts per model and per scenario.
The setup is close to a real sale. The customer does not say everything up front. The AI has to ask the right questions to find the hidden requirements. Then it buys a product that meets them, or buys nothing if none does. A run only passes if both conditions are met.
Benchmark, in plain words A benchmark is a standardised test: the same tasks, given to several models, scored with the same rules. It lets you compare models and track how they improve from one version to the next.
Section 2 · The numbers
Almost one run in two fails, and the gap between models is huge
On average, models pass 57.2% of runs. Put the other way, they fail 42.8% of the time. The best reaches 82.2%, the worst 28.9%. Which model sits behind an agent makes a big difference.
| Group of models | Pass rate | Compared with |
|---|---|---|
| The 5 that uncover the most hidden requirements | 70.2% | 44.4% for the 5 that uncover the least |
| The 5 most expensive per run | 67.1% | 50.2% for the 5 cheapest |
Models that ask the right questions do clearly better. That is the core of the test: an agent that does not understand the need ends up picking at random among plausible products.
Section 3 · The failures
In 54% of failed runs, the AI places an order anyway
This is the number a retailer should look at first: in 54% of failed runs, the AI still bought something. It missed part of the need, or it should have held back, and it placed an order all the same.
Across all runs, that is close to one run in four (42.8% × 54%, about 23%, Henkan calculation) ending in a purchase the customer would not have approved.
“An agent that picks the wrong product is an order today and a return in ten days.”
Why "buy nothing" matters A good salesperson knows how to say "we don't have what you need". Visa's test scores that too. An agent that always buys something pushes order volume up, but not necessarily the sales that stick.
Section 4 · Consistency
A scenario is passed on all three attempts only 34.2% of the time
Each model had three attempts per scenario. Across all models and scenarios, all three attempts passed just 34.2% of the time. The same agent, given the same request, can succeed once and fail the next time.
For a retailer, this is the most awkward number. You can manage an average pass rate. Behaviour that changes from one attempt to the next is much harder to manage: two customers with the same need can end up with two different products.
Visa has not tested payment yet. The release says it will add more scenarios and product categories, then extend testing to checkout, payment and order placement. The benchmark has also been shared with leading model providers.
Section 5 · Returns
Track the return rate of agent-placed orders, not just their volume
If an agent gets it wrong and orders anyway, the mistake reaches you as a return. It costs logistics and customer service time, and sometimes leaves a customer disappointed with the brand when the agent was at fault.
- 1The customer's requestThe customer describes the need to the agent, without spelling everything out.
- 2The orderThe agent missed some requirements and buys a plausible product. In your reports, it counts as a sale.
- 3The returnThe product doesn't fit. Ten days later, the sale becomes a return.
Order volume from agents will say little about what they are worth. You need to be able to isolate these orders and track their return rate, and return reasons, next to those of regular orders.
Section 6 · Product data
The agent only knows what you publish: an incomplete product page makes it guess
The models that uncover the most requirements do best. But understanding the customer is only half the job: the agent then has to match that need to your products. All it has for that is what you publish. If compatibility, dimensions or return conditions are missing, it guesses.
- Get ready to track agent-placed orders: a source flag on orders, then return rate and return reasons compared with regular orders.
- List the criteria that decide whether a product fits in your key categories (size, compatibility, use, power) and check they are written on every product page, as readable text.
- Spell out when a product does not fit: incompatibilities, usage limits, conditions. That is what allows an agent to decide not to buy.
- Keep delivery and return conditions up to date and visible on the product page, not only in the terms and conditions.
| Product data | Customer question | If the information is missing |
|---|---|---|
| Compatibility | "Will it work with my setup?" | The agent assumes it will |
| Dimensions | "Will it fit in my living room?" | The agent chooses from the general description |
| Use and limits | "Is it right for my case?" | The agent buys a plausible product |
| Return conditions | "What if it doesn't work out?" | The customer finds out afterwards |
Key terms
ShowHide ⌄
- Agentic commerce
- Purchases made by an AI agent on the customer's behalf: search, comparison, cart and payment.
- Benchmark
- A standardised test that gives the same tasks to several models so they can be compared under the same rules.
- Run
- One attempt by a model at a scenario. In Visa's test, each model made three runs per scenario.
- Hidden requirement
- A need the customer does not state up front and that the agent has to uncover by asking questions.
- Return rate
- The share of orders sent back by customers. Tracked by source, it shows whether a channel brings sales that stick.


