Comparing AI assistants: test the product, not just the model name

Name the product and mode being compared

A consumer assistant, a model API and an application built around that API are different test subjects. Search tools, system instructions and conversation history can change an answer even when the model name looks familiar. Record the interface, visible model, plan, date, market and enabled tools for every test.

This replaces the article's old 2025 feature table. The earlier parameter-count and market-share entries did not provide enough source detail to support a purchasing or testing decision. They should not be treated as specifications for current products.

Separate search-enabled tests from no-search tests

OpenAI describes ChatGPT Search as a workflow that can rewrite a request into one or more searches. Google's Gemini API grounding documentation exposes search queries and supporting source metadata when available. Anthropic documents a web search tool with source citations. These are product-specific capabilities, not evidence that all three retrieve or rank sources in the same way.

Run a no-search brand-recognition check separately from a searched buying-advice check. In the first, score whether the assistant knows the brand and describes it correctly. In the second, inspect the cited pages and whether the recommendation meets the buyer's requirements. An API response does not establish what a consumer account would display.

What to preserve when comparing assistants

Test subject	Record before comparing
ChatGPT	The product interface, visible model and search setting. Do not combine ordinary chat and a multi-step research session into one score.
Gemini	Whether the run is in the consumer app or API, and whether Google Search grounding is enabled. Do not label it a Google AI Mode result.
Claude	The interface and enabled tools. Keep web-search citations separate from content supplied through a connector or uploaded document.
Perplexity	The interface, selected mode, source restrictions and whether the run uses an API. Save the actual result rather than inferring a proprietary model size.
DeepSeek	The provider-hosted service or particular deployment, the model version and any external search tools. A model family name alone does not describe the retrieval setup.

Use buyer tasks that could change a decision

Use a question your business can genuinely answer. Keep the relevant budget, location and exclusion criteria. Add a follow-up that changes suitability, such as a smaller budget or a required integration, and check whether the assistant updates the shortlist correctly.

Keep observed customer prompts separate from questions generated for experimental coverage. Repeat each condition and save the full answer. A brand-free answer belongs in the denominator; a failed request belongs in the error log. Do not compare one lucky answer on one platform with an average of many runs on another.

Score the evidence before assigning a winner

For each valid response, record whether the brand is named, recommended and described accurately. Check whether the cited page supports the specific claim. A flattering recommendation with the wrong price or product fit is not a good result.

Report rates by product and mode, with run counts. Also report failed requests and missing citations. Avoid a single composite score unless the weighting reflects a decision you have defined in advance. A customer-support task and a product-discovery task need not have the same winner.

Allocate monitoring by your customers, not a global traffic chart

Use your observed conversations, qualified-lead feedback and referral analytics to decide which assistants and markets deserve more testing. A general web-traffic share measures a different population from buyers in a particular product category. Where evidence is sparse, use a small balanced pilot and report the uncertainty rather than inventing precise weights.

Our sampling methodology describes the limits of observed conversation data. The AI-search metrics guide covers answer-level denominators. Neither establishes incremental sales: connect completed outcomes separately before increasing the content budget.