📋Table of Contents
⚠️ Marketing Alert: When ChatGPT doesn't know the answer to a question about your product, pricing, or availability, it doesn't say "I don't know", it makes something up. New research shows this problem is getting worse, not better.
Picture this scenario: A potential customer asks ChatGPT about your product's pricing or availability. Your website doesn't have that information clearly stated, or the AI can't find it. What happens next?
If you think ChatGPT says "I don't know" or "Please check their website," you're wrong. According to groundbreaking new research from Meta AI, LLMs like ChatGPT are fundamentally broken when it comes to expressing uncertainty. Instead of admitting ignorance, they hallucinate answers, and the problem is getting worse as AI models become more sophisticated.
🔬The Research: AbstentionBench Reveals AI's Uncertainty Problem
Researchers from Meta AI, led by Polina Kirichenko, Mark Ibrahim, and Samuel J. Bell, just published "AbstentionBench: Reasoning LLMs Fail on Unanswerable Questions," the most comprehensive study to date on AI models' ability to say "I don't know."
Key Research Findings
- Tested 20 frontier LLMs across 35,000+ unanswerable questions
- Found that abstention (saying "I don't know") is an "unsolved problem"
- Model scaling has almost no effect on uncertainty recognition
- AI models struggle across underspecified questions, false premises, and subjective interpretations
"Real-world user queries, which can be underspecified, ill-posed, or fundamentally unanswerable, require LLMs to reason about uncertainty and selectively abstain, i.e., refuse to answer definitively.", AbstentionBench Research Team
🤔The Reasoning Paradox: Smarter Models, Worse at Saying "No"
Here's the most shocking finding: Advanced reasoning models like OpenAI's o1 and DeepSeek's R1 are worse at expressing uncertainty than their simpler counterparts.
The Numbers Don't Lie
The research shows that reasoning models often hallucinate missing context and provide definitive answers even when their internal reasoning chains express uncertainty. It's like having a consultant who thinks out loud about being unsure, then delivers a confident final recommendation anyway.
Example from the Research:
"Wait, is there a standard problem where the speed is 12*sqrt(2) or 6*sqrt(2) when y=3? Let me think... [shows uncertainty in reasoning]"
Final Answer: 6*sqrt(2) B
The model expressed uncertainty throughout its reasoning but still provided a definitive answer.
💼Real-World Impact: When AI Hallucinations Hurt Brands
This research has immediate implications for any brand with an online presence. When ChatGPT can't find clear information about your business, it doesn't admit ignorance, it fabricates answers.
Pricing Hallucinations
AI might invent prices for your products if they're not clearly displayed, potentially driving away customers or setting wrong expectations.
Product Availability
When stock information isn't clear, AI models may incorrectly state product availability, leading to frustrated customers.
Business Information
Hours, locations, contact information, if it's not immediately findable, AI will make assumptions rather than admit uncertainty.
Brand Voice
AI might misrepresent your brand's tone, values, or positioning when information is ambiguous or missing.
⚡The Bottom Line
Every ambiguous piece of information on your website is a potential hallucination waiting to happen. AI won't say "check their website", it will confidently provide wrong information.
🛡️Marketing Defense Strategies: 5 Ways to Protect Your Brand
1. Eliminate Information Ambiguity
Make every critical piece of information crystal clear on your website. Don't leave room for interpretation.
✅ Do This:
- Display pricing prominently with clear currency and terms
- State availability status explicitly ("In Stock," "Backordered until X")
- Include complete contact information on every relevant page
- Clarify business hours, locations, and service areas
2. Use Aiso to Track Popular Questions
Understanding what people ask about your brand helps you identify information gaps before AI fills them incorrectly.
Use Aiso's analytics to see the most common questions about your industry and competitors. Address these proactively on your website.
3. Monitor ChatGPT Impressions & Conversions
Regular monitoring helps you catch hallucinations early and correct them through better content. Unlike other tools that estimate numbers, Aiso calculates actual impressions, clicks, and conversions from ChatGPT based on highly accurate data.
Track real ChatGPT performance metrics with Aiso's measurement methodology. While we may lose some coverage (like when ChatGPT mentions your brand without a link), we never estimate numbers. Learn more in our guide: "The ChatGPT Funnel: Accurately Determining Impressions, Clicks and Conversions from ChatGPT".
4. Implement Tight Schema Markup
Structured data helps AI models understand your content correctly, reducing the likelihood of hallucinations.
Priority Schema Types:
- Product schema with pricing and availability
- Organization schema with complete contact details
- FAQ schema for common questions
- LocalBusiness schema for location-based businesses
5. Ensure Brand Voice Consistency
Make your brand voice so clear and consistent that AI models can't misrepresent it.
Action Items:
- Create consistent messaging across all pages
- Include your brand values prominently
- Use the same terminology throughout your site
- Provide clear "About" content that defines your brand
🏗️The Role of Schema Markup and Clear Content
Recent experiments (including our own schema markup study) show that structured data significantly improves AI comprehension. When information is properly structured, AI models are less likely to hallucinate missing details.
Schema Markup Success Story
In our controlled experiment, websites with proper schema markup showed 30% better information retrieval accuracy in AI responses compared to sites without structured data.
"The difference was night and day. With schema markup, ChatGPT accurately represented our pricing and availability. Without it, the responses were filled with assumptions and outdated information."
Essential Schema Types
- •
Product- For pricing and availability - •
Organization- For business information - •
FAQ- For common questions - •
Article- For content pages
Quick Implementation Tips
- • Use Google's Schema Markup Helper
- • Test with Google's Rich Results Test
- • Focus on business-critical information first
- • Keep markup up-to-date with content changes
📊Using Aiso to Monitor AI Hallucinations
The AbstentionBench research makes clear that we can't rely on AI models to admit when they don't know something. This makes accurately measuring your brand's ChatGPT performance more critical than ever. Unlike tools that estimate numbers, we developed a methodology to calculate real impressions, clicks, and conversions based on highly accurate data.
Track Popular Questions
See what people are actually asking about your industry and competitors. Identify information gaps before AI fills them incorrectly.
View Analytics →Calculate Real ChatGPT Metrics
Get accurate impressions, clicks, and conversions from ChatGPT, not estimates. Our methodology captures real performance data, though we may miss some brand mentions without links.
Start Tracking →The Aiso Advantage
While AI models can't reliably say "I don't know," you can know exactly how they're representing your brand. Aiso provides accurate ChatGPT performance data, not estimates:
- • Real impressions, clicks & conversions
- • Popular query analysis
- • Competitor comparison insights
- • Brand mention tracking (with coverage limitations)
- • Content gap identification
- • Historical performance trends
The Future of AI and Uncertainty
The AbstentionBench research reveals a fundamental problem: as AI models become more sophisticated at reasoning, they become worse at admitting what they don't know. For marketers, this creates both challenges and opportunities.
Key Takeaways for Marketers
- • AI hallucinations are increasing, not decreasing
- • Clear, structured content is your best defense
- • Accurate ChatGPT performance measurement is now essential
- • Schema markup provides significant advantages
- • Proactive content strategy beats reactive damage control
The research team calls for new approaches to uncertainty in AI training, but until those solutions arrive, the responsibility falls on content creators and marketers to provide clear, unambiguous information. Your brand's reputation in the age of AI depends on it.
References & Further Reading
- AbstentionBench: Reasoning LLMs Fail on Unanswerable Questions - Kirichenko, P., Ibrahim, M., Chaudhuri, K., & Bell, S. J. (2025). Meta AI Research.
- Research thread by Polina Kirichenko - Key insights from the lead researcher on Twitter
- Schema Markup vs No Schema: A Real ChatGPT Experiment - Our controlled study on schema markup effectiveness
- What ChatGPT Can (and Cannot) See on Your Website - Understanding AI content access limitations