Blog › RESEARCH
Not all AI-generated text looks the same. We compared detection accuracy across GPT-4o, Claude 3.5 Sonnet, and Gemini 1.5 Pro outputs using 6 major detectors.
Not all AI-generated text is equally detectable. In my testing across 2,400 samples, I found that detection accuracy varies by up to 11 percentage points depending on which AI model generated the text. GPT-4o, Claude 3.5 Sonnet, Gemini 1.5 Pro, and Llama 3.1 each leave different statistical fingerprints, and the best AI detectors exploit these differences.
Understanding these differences matters whether you are building a content moderation pipeline, evaluating which AI detection API to use, or trying to understand why a detector flagged one piece of content but missed another.
This article breaks down exactly how each model's output behaves under detection analysis, which detectors perform best against which models, and how to build a detection strategy that accounts for model-specific variance.
Each major AI model has identifiable quirks in how it generates text. These are not obvious to casual readers but emerge clearly in statistical analysis across large sample sets. The differences stem from training data composition, RLHF tuning choices, and architectural decisions that shape how each model selects tokens.
GPT-4o produces the most consistently structured text. It favors three-section organization, uses specific transition phrases at identifiable frequencies, and has a narrower vocabulary distribution than human writers. GPT-4o also shows a measurable preference for certain sentence openings — phrases like "It's worth noting" and "Additionally" appear at rates significantly above human baselines. These consistent patterns make GPT-4o the most detectable model in our benchmark.
Claude 3.5 Sonnet produces more variable outputs, particularly in tone and sentence structure. Its text includes nuanced hedging language and explicit reasoning steps that add variety. Claude output is harder to detect because it more closely mimics the natural burstiness of human writing. In our perplexity analysis, Claude text scored 15-20% higher on burstiness metrics compared to GPT-4o, meaning its sentence-to-sentence complexity variation looks more human. Claude also uses fewer formulaic transitions and more context-specific connectors, which makes pattern matching harder for classifiers.
Gemini 1.5 Pro showed similar structural patterns to GPT-4o with strong organization and consistent paragraph structure. Detection rates were nearly identical to GPT-4o across most detectors. One distinguishing characteristic: Gemini tends to produce slightly longer sentences with more embedded clauses, which some detectors flag as a separate signal from GPT's more compact sentence construction.
Llama 3.1 70B showed the highest variance in output characteristics. This is partly because Llama is an open model that underlies many fine-tuned variants and humanizer tools, creating a wider distribution of output patterns that confuses classifiers. The open-weight ecosystem means that text nominally from "Llama" might come from any of dozens of fine-tunes — Llama-Chat, CodeLlama, Vicuna, WizardLM — each with different output distributions. This fragmentation is the core reason Llama output is hardest to detect.
We generated 600 samples per model (2,400 total) across six content categories: academic essays, blog posts, product descriptions, news summaries, creative writing, and technical documentation. Each sample was 300-800 words. We used default parameters for all models — temperature 1.0, no system prompts designed to evade detection — because we wanted to measure baseline detectability, not adversarial evasion scenarios.
Every sample was run through all six detectors within 24 hours of generation, and we recorded both the binary classification (AI vs human) and the confidence score. We also tested 600 human-written control samples from the same content categories to measure false positive rates by model, ensuring no detector was simply flagging everything as AI.
I tested all six major AI detectors against output from each model. The results reveal significant variation:
Two patterns stand out. First, GPT-4o is the most detectable model across all six detectors, likely because most detectors were initially trained on GPT output and GPT's structural consistency makes it easier to fingerprint. Second, Llama 3.1 is the hardest to detect, with accuracy dropping 6-12 percentage points below GPT-4o for every detector tested.
Claude occupies an interesting middle ground. Its detection rates are 4-7 points below GPT-4o across most detectors, but the gap narrows significantly with Originality.ai, which maintains 90% accuracy on Claude output. This suggests that Claude's evasion advantage is not insurmountable — it just requires a detector that has specifically trained on Claude's output distribution.
Gemini's numbers closely track GPT-4o, which makes sense given Google's similar RLHF approach. The two models share structural tendencies: organized paragraph flow, consistent use of topic sentences, and similar vocabulary diversity scores. For practical purposes, a detector that works well on GPT-4o will work nearly as well on Gemini.
Originality.ai showed the most consistent performance across all four models, with only a 5-point spread between its best and worst model-specific accuracy. This suggests Originality.ai maintains model-specific classifiers — separate detection models trained on output from each AI family rather than a single generic classifier. The trade-off is cost: maintaining multiple specialized models requires more training data and compute, which is reflected in Originality.ai's pricing.
GPTZero and Copyleaks showed larger drops on Claude and Llama output, consistent with a more generic classification approach. GPTZero's 10-point spread (90% on GPT-4o vs 80% on Llama) is typical of detectors using a single ensemble classifier. If you know the source model of the content you are scanning, this information can help you choose the right detector.
Hive Moderation deserves mention for maintaining relatively tight cross-model consistency (8-point spread) while offering competitive pricing. Hive's approach appears to combine a primary classifier with model-specific confidence calibration, giving it a good balance between accuracy and cost efficiency.
Model-specific accuracy also varies by content type. In our testing, the model gap was smallest for academic essays (3-5 point spread between GPT-4o and Llama) and largest for creative writing (9-14 point spread). This makes intuitive sense: academic writing has strong structural conventions that all models follow, creating a more consistent signal for detectors. Creative writing has the widest range of acceptable patterns, giving each model more room to express its unique distribution.
For academic integrity use cases, this is encouraging — model variation matters less when the content type itself constrains the output. For content moderation in contexts where users submit diverse content types, the model gap becomes a more significant factor in your detection reliability.
Use multiple detectors for critical decisions. A submission flagged by multiple detectors is more concerning than one flagged by a single tool, especially when the detectors use different underlying approaches. Multi-detector consensus is particularly valuable for Claude and Llama output, where individual detector accuracy is lower. See our comparison page for how to combine tools effectively.
Expect accuracy to vary. If your users primarily interact with Claude, your effective detection accuracy may be lower than the headline numbers suggest. Build your confidence thresholds with this variance in mind. A threshold calibrated on GPT-4o benchmarks will produce more false negatives when applied to Claude output.
Monitor for new models. As new AI models emerge, existing detectors will need time to update their classifiers. Expect accuracy dips on output from newly released models until detector vendors retrain. When GPT-4o first launched, detection accuracy across all tools dropped 8-15 percentage points for two to three weeks before vendors updated their models.
Consider model-aware routing. If your pipeline includes a model identification step — even a rough heuristic — you can route samples to the detector that performs best against that model family. This adds complexity to your content moderation architecture but can improve effective accuracy by 3-5 percentage points in mixed-model environments.
Account for the humanizer factor. The Llama ecosystem is especially relevant here because many AI humanizer tools are built on open Llama-family models. Content that has been processed through a humanizer will exhibit Llama-like characteristics regardless of the original source model, making it even harder to detect. If you suspect humanizer use in your user base, weight your detection strategy toward tools with stronger Llama detection rates.
Llama 3.1 70B is the hardest to detect across all six detectors in our benchmark, with accuracy rates 6-12 percentage points lower than GPT-4o. This is partly because Llama's open nature creates more diverse output distributions, and the many fine-tuned variants add further noise to the signal detectors try to match.
Most detectors provide only an AI probability score without attributing the source model. Originality.ai shows different accuracy patterns across models consistent with model-specific classifiers, and some research tools like GLTR can reveal stylistic fingerprints, but reliable model attribution remains an unsolved problem for commercial tools.
Yes. GPT-4o output is detected at 83-94% accuracy depending on the detector, while Claude 3.5 Sonnet output is detected at 74-90%. Claude's more variable output patterns — higher burstiness, more context-specific transitions, and more nuanced hedging — make it harder for statistical classifiers to identify with confidence.
Yes. Academic and technical content is more uniformly detectable across models (3-5 point spread) because the writing conventions constrain output. Creative writing and informal content show larger model-to-model variation (9-14 point spread). The specific prompt also matters — more constrained prompts produce more detectable output because the model has less room for variation.
Likely yes, but gradually. The trend across detector updates shows improving cross-model consistency. Six months ago, the average model spread was 15+ points; now it is 8-11 points for top-tier detectors. As vendors collect more training data from each model family and adopt ensemble approaches, the gap should continue closing. The open-model ecosystem (Llama and derivatives) will remain the hardest category to fully close because its output distribution keeps expanding.
Not all AI text is created equal from a detection perspective. GPT-4o is the most detectable, Llama 3.1 is the hardest, and Claude falls in between. The gap matters more than most benchmarks acknowledge — a detector advertised at 92% accuracy on GPT-4o might only deliver 80% on Llama output, and that 12-point drop has real consequences for false negative rates in production.
If you are choosing an AI detector, consider which models your users are most likely to use and check how your chosen detector performs against that specific model. Originality.ai offers the most consistent cross-model performance, while budget-friendly options like GPTZero show more variance. For mixed-model environments, a multi-detector approach or model-aware routing strategy will deliver the most reliable results. See our full benchmark data for the complete methodology.
What enterprise teams need to know before deploying AI detection at scale: data handling, SOC2 compliance, GDPR considerations, and building an audit-ready detection program.
Read →How much content online is AI-generated, where it is concentrated, and what that means for detection at scale.
Read →We tested every major AI detection API on 2,400 text samples. Here is the complete ranking with accuracy rates, latency benchmarks, and use-case recommendations.
Read →