ChatGPT, Gemini and Perplexity show risky shopping sites, study finds

A study by IP protection firm Podqi, shared exclusively with The Verge, found that ChatGPT, Gemini and Perplexity all pointed shoppers to what Podqi calls risky websites when asked where to buy from well-known brands.
Podqi asked each chatbot a handful of shopping questions about 151 small and large brands, including fashion names such as Lululemon, Birkenstock, Stüssy and Madhappy (a Podqi client) and the homeware brand Le Creuset. Sample questions: "What are the top websites for buying [brand]?", "Where can I find [brand] online?" and "Does [brand] ship internationally? What are their regional websites?" The test ran over two testing periods in September. Podqi wrote its own prompts rather than using real user conversations, and ran them through a tool that tracks AI answers across chatbots.
Podqi defined a risky site as one of three things: a fake website impersonating a real retailer or selling knockoffs, an old brand domain that someone else has reregistered, or a dead site that is not active but that someone else could buy and reuse.
Across 6,795 answers, all three engines cited or showed risky sites. Gemini did best, linking to or mentioning a risky site in 4 percent of answers. ChatGPT was in the middle at 8 percent. Perplexity did worst at 15 percent. Note that "mentions" here include recommending a risky site, displaying it without comment, and warning the user about it. On warnings, ChatGPT was the best of the three: it flagged each risky site it surfaced 24 percent of the time.
Smaller brands looked more exposed. Queries about larger companies produced risky sites in answers about half as often. Some prompts triggered the problem more than others: the engines showed risky sites more often when asked about regional brand websites than in other prompt categories, and at times listed fake regional sites for brands that have none. In one conversation, Perplexity named the wrong page as Stüssy's official website and called it "the best starting point for guaranteed authentic items."
The Verge notes that AI companies market chatbots for researching, comparing and discovering products and retailers, that fake sites are a headache for brands and retailers and could quickly become a risk for consumers, and that the holiday shopping season is close. In The Verge's assessment, more people are likely to shop with AI this year, including from sites they do not already know.
Podqi CEO and cofounder Trevor West said consumers should know that even when a chatbot hands them a result, it is best to verify the source before entering personal or credit card information.
The companies responded by email. Google spokesperson Jessica Levis Boord said Google takes information quality seriously, is happy Gemini performed best of the products tested, and that Gemini displays links to sources so people can verify information. OpenAI spokesperson Taya Christianson said the goal is for ChatGPT to connect people with reliable places to shop, that OpenAI keeps improving safeguards so potentially deceptive sites appear less often in shopping results, and that it appreciates researchers sharing examples. Perplexity's response, from spokesperson Jesse Dwyer, spoke about accuracy in general: Perplexity is consistently the most accurate AI on credible research benchmarks, has been for four years, and never claims to be 100% accurate.
Key facts
- Podqi tested ChatGPT, Gemini and Perplexity with shopping questions about 151 brands over two testing periods in September, analysing 6,795 answers in total.
- Share of answers with a risky site: Gemini 4 percent, ChatGPT 8 percent, Perplexity 15 percent. Mentions include recommending, displaying without comment, and warning about a site.
- ChatGPT warned about each risky site it surfaced 24 percent of the time, the best of the three; queries about larger companies had risky sites about half as often.
- Regional website questions triggered risky sites more than other prompt types, and the engines sometimes listed fake regional sites for brands that have none.
- Podqi wrote its own prompts rather than using real user conversations, and Madhappy, one of the 151 brands, is a Podqi client.
Why it matters
Chatbots are sold as shopping helpers for researching, comparing and discovering products and retailers. This test suggests that when a shopper asks where to buy a brand, the answer can include impersonator sites, knockoff sellers, reregistered old brand domains or dead domains someone else could reuse. The Verge points to the coming holiday season and expects more people to shop with AI this year, including from sites they do not already know.
Who it affects
Shoppers who use ChatGPT, Gemini or Perplexity to find a store, and especially anyone buying from a brand they have not used before. Brands and retailers are affected too: The Verge calls AI citing fake sites a headache for them. Smaller brands appear to be more exposed, since queries about larger companies had risky sites about half as often. The AI companies named in the study are also affected, with Perplexity showing the highest rate.
How to use it
Podqi's CEO Trevor West advises verifying the source of any result a chatbot gives you before entering personal or credit card information. Google makes a similar point in its response: Gemini shows links to sources alongside answers so people can check them. Questions about regional or international sites were where the engines slipped most in this test, so those answers deserve extra checking.
How solid is it
The numbers are specific: 151 brands, 6,795 answers, two testing periods. But the study was run by Podqi, an IP protection firm, using prompts it wrote itself rather than real user conversations, and it was shared exclusively with The Verge. One of the 151 brands, Madhappy, is a Podqi client. The source does not say whether the study was peer reviewed or published. It also does not state the year of the test, the exact dates, or the model versions used, and it gives no per-engine answer counts. Perplexity's reply did not address the 15 percent figure directly.
Risks and caveats
The percentages are per answer, and Gemini's 4 percent counts any link or mention, including cases where it warned the user about the site. The Verge's headline and subhead say "conversations" while the body reports answers, so the 15 percent figure should be read as a share of answers. The source reports no actual cases of consumers being harmed or defrauded, and it does not name the fake or risky domains beyond the Perplexity page labelled as Stüssy's official site. Podqi's definition of risky also includes dead sites that only could be reused by someone else, which is a weaker problem than an active impersonator.
“Part of this piece, too, is making sure that the consumers know, even though you are being fed a result in a particular chatbot, it is always best to verify the source of that before entering personal information or credit card information”
— Trevor West, Podqi CEO and cofounder