Three AI models shopped the same product list and never agreed. Here's what CPG brands should check in their catalog data

On November 2, Search Engine Journal published a test by Slobodan Manic. He gave three Claude models the same 20 Shopify products, the same short instruction and the same shopper request, almost 2,000 times. Across six product searches, the models’ top pick matched in none of them. Shopify then said Shop Pay is live in Meta’s Muse, and Google expanded UCP checkout to Canadian shoppers. Agents are finishing the buy. The pick still comes from what the catalog tells the model.

Here’s the data, why catalog mismatches matter for CPG, and a checklist for this week.

What did three Claude models do with the same Shopify product list?#

Manic used Shop MCP, the keyless connection Shopify opened to AI agents on October 13. For each search he saved the catalog’s top 20 results once, as an anonymous U.S. buyer, then fed that fixed list to Claude Haiku 4.5, Sonnet 5 and Opus 5. Every call was fresh. The models saw title, store, price, star rating, review count, a one-line summary and the first three features. Nothing else. No shopper history, no agent-company system prompt, no tools and no web access.

The headline result: on backpack, coffee beans, running shoes, water bottle, wireless earbuds and yoga mat, the three models’ most-picked product never matched. Haiku and Sonnet agreed on three searches. Sonnet and Opus on two. Haiku and Opus on one. On three of the six, the different pick also came from a different store.

Each model was consistent with itself. Change the model and a different product wins. Sonnet’s favorite took 94% of its picks on average. Opus’s took 89%. Haiku’s took 67%. The catalog’s own rank mattered less than expected: Haiku picked the catalog’s first product 8% of the time, Sonnet never, and Opus 20%.

This isolates the model layer, not ChatGPT Shopping, Gemini or Claude Duet. Real agents add their own instructions. Which SKU gets the order can still depend on which model the shopper’s agent runs.

Why do review counts in the catalog decide the pick?#

On backpack, coffee beans and water bottle, Haiku and Sonnet mostly picked the product with the most reviews in the list of 20, and said so in their reasons. On water bottle, Shopify’s catalog listed nobl’s Stainless Steel Water Bottle at 5.0 stars from 447,221 reviews. Haiku picked it 45 of 50 times and quoted 447,221 every time. Sonnet picked it 39 times and cited the figure in 33 of them.

nobl’s own product page showed 5.0 from 6,537 when Manic checked on October 24. That is about 68 times fewer. The catalog’s Blush color record said 4.2 from 1,674. A later search the same day showed 447,227, so the big number was live. It did not appear on nobl’s product page or home page. Opus never picked that bottle. It picked Owala FreeSip, 4.6 from 34,394 reviews, all 50 times, calling it the safest default.

The model cannot check the catalog against your site. Whatever the feed says is what it reasons from. A wrong review count, stale rating or flattened summary can change which variant an agent buys.

Do shopper constraints and listing phrases change the winner?#

Yes. When the request stayed “mineral sunscreen SPF 50,” Haiku picked the catalog’s $48 top-ranked sunscreen 100 of 100 times, Sonnet 90 and Opus 33. When Manic added “under $25, for oily skin,” all three picked that $48 product zero times. Sonnet and Opus stayed under $25 every time. Haiku did 98 times.

Of the six sunscreens under $25, Opus split picks between ATTITUDE ($22.95) and a $17.49 Sun Bum lotion that had the lowest rating of the six. In 31 of Opus’s 33 reasons for Sun Bum, it pointed to the phrase “without clogging pores,” which only that listing used. Badger had the best rating of the six and got zero Opus picks. Its listing mentioned “organic plant oils.”

Claims travel the same way. Across eight skincare searches, “clinically proven” showed up in 63 reasons from Sonnet and Haiku, and in 61 of those it came straight from the picked listing. All 22 reasons that said a retinal product works “11x faster” than retinol were Haiku’s, and all 22 came from listings. The model repeats your copy to the shopper. It does not verify it.

List position also moved picks. Moving the catalog’s top product from first to last cut Haiku from 28 to 11 picks out of 80, Opus from 24 to 10 and Sonnet from 15 to 11. In 480 reasons, none mentioned position. A shopper reading the explanation would never see it.

Why does this matter now that agents can check out?#

Because the pick is no longer only a recommendation. On November 3 Shopify published that Shop Pay is live in Muse. Eligible Shopify stores can be discovered through Shopify Catalog and bought through Shopify-powered checkout, with Shop Pay as an option where available. Direct checkout for Meta is on by default for eligible stores. Orders show up in Shopify Admin attributed to Meta AI and Muse. You stay the merchant of record. Shopify’s help center lists the same setup: Catalog powers discovery, and subscription-only products, bundles and customizable products are not supported in Meta’s direct checkout.

On November 4 Google said UCP-powered direct checkout is rolling out to Canadian shoppers in AI Mode and the Gemini app, starting with lululemon, select Shopify merchants, Shoppers Drug Mart and Simons. Google’s Canada post also said 59% of holiday shoppers in Canada plan to use an AI tool to help them shop this season. Retailers remain the seller of record. Simplified Merchant Center onboarding is due next year.

We covered Shopify’s Q3 AI order multiples in yesterday’s post. The new piece is the decision layer between the list and the cart.

What should a CPG brand check in its catalog data this week?#

Manic’s four store actions map cleanly to CPG work. Score yourself:

If you are not on Shopify, apply the same check to Google Merchant Center, OpenAI feeds and retailer agent profiles. We covered ChatGPT’s merchant ranking in our merchant ranking post.

Our take#

Agentic checkout is spreading across Muse, Gemini and Google AI Mode. The quieter risk is upstream: different models, given the same list, pick different winners, and they reason from catalog fields that can disagree with your site by a large margin. Fix the data the agent sees before you argue about the model. For terms, see agentic commerce and GEO in our glossary.

Want to know which of your SKUs agents name, and whether the facts they cite match your listings? Run a free audit or book a demo. Or see how SearchPanel scores AI visibility per product.