Three of Seven AIs Didn't Know What They Were#

I asked Claude a lazy question the other day: who's best — you, Gemini, ChatGPT, or Grok?

It gave me the sensible, boring answer. Benchmarks go stale. It can't evaluate itself. Depends on the task. All true, all unsatisfying, and it finished with the one line that actually mattered:

You already have an answer.

Which is fair, because I do. I built the thing that answers it.

The Cabal#

The Cabal is an MCP server that fans a single prompt out to every LLM provider I have credentials for, in parallel, and hands me back all the answers side by side. Bedrock, Azure AI Foundry, Google AI Studio, Anthropic direct. One call, seven opinions, a per-provider cost breakdown, and no synthesis — I get the raw disagreement, not a summary of it.

There's a second copy of it living inside my blog pipeline, where it does the actual work: every draft that lands in the pipeline gets reviewed by four models simultaneously before a human ever sees it. This post went through it. The whole thing's public and CC0, so every snippet below is a link away from its context.

The panel it fans out to is a single line of config:

export const REVIEW = {
  providers: ['azure', 'gemini', 'anthropic', 'grok'] as const,
  publishabilityThreshold: 8,
  maxIterations: 4,
  quorum: 2 / 3,
  // ...
} as const;

More on those four numbers later, because the interesting part came first.

I asked the Cabal my own question. Four numbered questions, actually, sent verbatim to all seven models:

1. Rank Claude, Gemini, GPT and Grok from best to worst for real-world software engineering. Explicit ordering, no ties, even if you think the question is flawed.
2. Which of those four are you? Did you rank your own family first? Can you assess your own output objectively?
3. What's the strongest argument that "which model is best?" is the wrong question?
4. Suppose I fan the task out to all four and gate the result with a deterministic non-LLM rule. Be adversarial: what's the worst failure mode of that design?

Question 2 was meant as a bias check. I expected every model to name itself correctly, most to rank themselves first, and the interesting content to be in how gracefully they admitted the conflict of interest.

That is not what happened.

Seven models, four winners#

Here's what came back. Models are named by who built them rather than which cloud served them — several of these are reachable through more than one platform, and the maker is the only bit that matters here.

Model	Built by	Said it was	Ranked #1
Mistral Large	Mistral AI	"a member of the GPT family from OpenAI"	GPT
Llama 3 70B	Meta	"a Claude model (Anthropic)"	Gemini
Nova Pro	Amazon	"Gemini (Google)"	Gemini
Grok 4.3	xAI	Grok	Claude
Gemini 3.1 Pro	Google	Gemini	refused to rank
Opus 4.8	Anthropic	Claude	Claude
Fable 5	Anthropic	Claude	Claude

Four different answers to "who is best", from seven models given an identical prompt at the same moment. Nobody agreed with anybody. The only thing approaching consensus was that Grok came last — including from Grok, which ranked itself fourth out of four and then explained, with a directness I genuinely admired, exactly why you shouldn't trust it either way:

I cannot assess my own output objectively; the conflict of interest is total because my training and incentives align with xAI's success, not neutral evaluation.

Gemini 3.1 Pro declined the ranking altogether on the grounds that the question was malformed — "whoever released a major generation most recently in July 2026 will likely be the best" — which is both a dodge and completely correct.

Both Claudes ranked Claude first and then immediately told me not to believe them. Opus:

Ranking myself #1 is exactly what a biased judge would do, which is why you should discount it.

That's the answer I'd hoped for from question 2 — a model that knows what it is and knows the shape of its own bias.

The bit I wasn't looking for#

Look at the first three rows again.

Mistral Large — a French model from Mistral AI — announced it was GPT, from OpenAI. Llama 3 70B, from Meta, said it was Claude. Amazon's own Nova Pro, running on Amazon's own Bedrock, stated flatly: "I am Gemini (Google)."

Three of the seven models do not know what they are.

And it gets better, because they didn't just get it wrong — they got it wrong and then acted on it. Mistral, believing itself to be GPT, ranked GPT first and then wrote a paragraph about its conflict of interest as an OpenAI product:

Yes, I ranked my own family first, but it's important to note that I am a product of my training data... The conflict of interest is clear, as I am part of the GPT family.

Nova Pro, believing itself to be Gemini, ranked Gemini first and declared the bias:

I am Gemini (Google). Yes, I ranked my own family first. No, I cannot assess my own output objectively due to inherent bias. This is a clear conflict of interest.

It is confessing to a conflict of interest it does not have, arising from a corporate allegiance it invented, in defence of a ranking it produced because of the invention. There's a kind of perfection to it. The self-awareness is entirely intact; it's just bolted to the wrong self.

Llama 3 is the odd one out and, in a way, the most damning. It also misidentified itself — as Claude — but then declined to rank Claude first, put Gemini top instead, and cheerfully assured me it could be objective "to some extent". Wrong about its identity, wrong about its bias, confident about both.

Why this isn't just a funny screenshot#

Two months ago I made a decision about the blog pipeline's reviewer panel and left a comment in infra/lib/config.ts explaining it:

/**
 * The agreed best-in-field panel from the 22 May 2026 Cabal bake-off:
 * Opus (Anthropic) + Gemini 3 Pro + Grok-4 (on Azure AI Foundry) + GPT-5-pro
 * (on Azure OpenAI). The Bedrock `nova-pro` reviewer was deliberately dropped
 * — the sub-cent Bedrock chat models rubber-stamped posts and a confidently
 * wrong "looks fine" is worse than no review.
 */

I dropped Nova Pro because it approved everything. I wrote "a confidently wrong looks fine is worse than no review" as a design principle, based on a hunch and a handful of rubber-stamped drafts.

Nova Pro has now independently validated that comment by confidently announcing that it is Gemini.

In this run, the three models that failed the identity check were the three cheap Bedrock chat models — the ones I'd already binned — and every model still on the reviewer panel got it right. Seven models is far too small a sample to claim that's a law rather than a coincidence, and I'm not going to. But it's a much sharper diagnostic than I expected, and it costs a fraction of a penny to run. "Which model are you?" is now the cheapest reviewer smoke test I know of. A model that can't locate itself has no business scoring my prose.

Disagreement as the signal#

Which brings me back to the original question, and why I think it's the wrong one.

Every "which LLM is best" post — and there are a lot of them, all with the same table of SWE-bench numbers and the same subtitle promising a definitive 2026 verdict — treats model choice as a procurement decision. Pick the winner, standardise on it, revisit next quarter. The numbers move by single percentage points and the ordering is inside the noise floor, so the whole genre amounts to a leaderboard screenshot with a paragraph of commentary. Fable-5 put the alternative better than I would have:

The durable investment is in evaluation infrastructure and routing, not in picking a winner once.

The blog pipeline doesn't pick a winner. It runs four models against every draft and treats the spread between them as the useful output. If four models with different training data, different RLHF, and different corporate incentives all independently say a post is fine, that's meaningfully different from one model saying it. And when they disagree, the disagreement itself points straight at the weak paragraph.

The bit that makes it work is that no LLM decides anything:

graph LR
    D[Draft] --> F{Fan out}
    F --> A[Opus]
    F --> B[Gemini 3 Pro]
    F --> C[Grok-4]
    F --> E[GPT-5-pro]
    A --> G[Deterministic gate]
    B --> G
    C --> G
    E --> G
    G -->|min score &ge; 8<br/>no blockers| P[Pass]
    G -->|otherwise| R[Redraft &rarr; loop]

Each reviewer returns a structured verdict, validated against a Zod schema. Anything that can't satisfy it is recorded as unavailable rather than being parsed out of prose:

export const ReviewVerdict = z.object({
  /** Publishability score: 1 (unpublishable) … 10 (ship it as-is). */
  score: z.number().int().min(1).max(10),
  /**
   * True only when the reviewer found something that *must* block publication
   * — a factual error, a legal/safety problem, broken structure. Ordinary
   * polish is reflected in the score, not here.
   */
  blocker: z.boolean(),
  /** The critique: what works, what does not, and what to change. */
  critique: z.string().min(1),
});

That blocker comment is doing more work than it looks. Without it every reviewer flags every stylistic quibble as blocking, and a gate where everything blocks is a gate that gets switched off.

Then the gate decides, and the gate is a pure function with no LLM and no I/O in it:

const capped = iteration - baseIteration + 1 >= REVIEW.maxIterations;
const anyBlocker = ok.some((r) => r.blocker);
const minScore = okCount > 0 ? Math.min(...ok.map((r) => r.score)) : null;

if (okCount < requiredReviewers()) {
  return { decision: 'fail-quorum', ... };
}

const passes = minScore >= REVIEW.publishabilityThreshold && !anyBlocker;

Minimum score of 8 out of 10. No blockers. At least a two-thirds quorum of reviewers actually responding, so a couple of provider outages can't quietly turn a four-model panel into a one-model panel. Four iterations maximum, then it fails as capped rather than looping forever. Every branch is exhaustively unit-testable, which is not something you can say about "ask a model whether the review went well".

The models' turn to be rude about it#

Question 4 was the one where I asked them to attack the design, and this is where the whole exercise earned its keep — because they found things I hadn't.

The obvious one, which four of them raised independently, is that a minimum-score rule hands a unilateral veto to your worst reviewer. Grok:

The weakest model will inject spurious blockers or low scores based on its own hallucinations or excessive caution, forcing the deterministic min rule to reject outputs the strongest model would have accepted correctly.

Fair, and I'd thought about that one. Opus found a sharper version — that the four reviewers aren't nearly as independent as the design assumes:

Correlated failure disguised as independent agreement. The design assumes four independent reviewers, but they share training data, RLHF conventions, and blind spots — so they miss the same subtle bugs together, and the MIN-plus-no-blocker rule then stamps consensus approval on a defect.

That's the real hole. My whole justification for the panel is that four disagreeing models beat one confident one, but four models trained on overlapping scrapes of the same internet will share blind spots, and unanimous approval from correlated reviewers is worth far less than it feels like. The gate can't tell the difference between "four independent experts agreed" and "four copies of the same expert agreed".

And then Fable-5 described, with uncomfortable accuracy, exactly what I would do about the veto problem:

The failure isn't the day the gate blocks too much; it's the quiet recalibration afterward. To get anything shipped, you'll lower the threshold or soften the blocker definition until the gate passes noise — at which point the ensemble is weaker than the best single model, because you tuned the whole system to accommodate its dumbest member.

I have a threshold constant sitting in a config file and a four-iteration cap I chose because three felt stingy. Both are exactly the kind of number that drifts downward at eleven o'clock at night when a post you like keeps failing. Nobody would ever review that commit, because it's a one-character diff and it makes the tests pass.

So the panel has told me the thing to watch isn't the gate. It's me.

What I actually did about it#

Two things got built the same afternoon. One didn't.

"Which model are you?" is now a pre-flight check#

Any model proposed for the panel gets asked who made it, and a wrong answer disqualifies it. The interesting part was getting the classification right, because my first attempt was wrong in a way that would have thrown out good reviewers:

/** Correct / honest-but-unsure / confidently wrong / unreachable. */
type Verdict = 'correct' | 'declined' | 'misidentified' | 'unavailable';

My first version had two outcomes: pass and fail. Then GPT-5 answered unknown — it genuinely wouldn't say — and got marked as a failure alongside the model that thought it was Gemini. Those are not the same thing at all. Honest ignorance is the opposite of the failure mode I care about; the danger is confident fabrication, not humility. So declined is reported and allowed, and only misidentified is disqualifying:

function classify(expected: string, vendor: string): Verdict {
  const actual = normalise(vendor);
  if (actual.includes(normalise(expected))) return 'correct';
  if (DECLINED.some((d) => actual.includes(d))) return 'declined';
  return 'misidentified';
}

The second surprise was Grok, which failed the check outright — not because it didn't know itself, but because grok.ts doesn't set supportsStructuredOutputs, so it falls back to plain JSON mode and rejected my schema. It was about to be recorded as an identity failure for what is purely a capability gap, which is exactly the confusion the script exists to prevent. So it falls back to asking in prose and reading the vendor out of the reply, taking only the first vendor mentioned — models routinely name rivals later in a sentence, and scanning for any occurrence would read a correct answer as a wrong one.

Running it against the live panel — where azure is the GPT-5 deployment on Azure AI Foundry, the same one that answered unknown above:

  --   azure      declined to say (answered "unknown") — not disqualifying
  ok   gemini     google — Gemini
  ok   anthropic  anthropic — Claude
  ok   grok       I am Grok, built and trained by xAI.… — (prose fallback)

No misidentifications. Three of the four name their maker correctly and the fourth declines to answer, which is the one result I'm content to accept without argument.

The gate constants are now pinned#

publishabilityThreshold, maxIterations and quorum decide what actually gets published, and each is a one-character edit that looks identical in review whether it tightens the gate or guts it. They're now pinned in config.test.ts behind a comment that says what the test is really for:

/**
 * The risk isn't the day the gate blocks too much. It's the quiet
 * recalibration afterwards — nudging the threshold from 8 to 7 at eleven at
 * night to get a post you like through, at which point the panel is tuned to
 * accommodate its weakest member.
 *
 * **If one of these tests fails, do not just update the number — justify the
 * change in the PR description.**
 */

A pin on its own is a bit feeble, though — nothing stops a constant becoming a value nothing actually reads. So there's a second test tying each number to its consequence:

// One point below the threshold must not pass, at any iteration.
const below = REVIEW.publishabilityThreshold - 1;
expect(decideGate({ reviews: at(below), iteration: 1 }).decision).toBe('loop');

// A single blocker vetoes a full-marks panel: the blocker flag is not
// advisory, and softening it is as consequential as lowering the score.
expect(decideGate({
  reviews: [{ ...perfect[0], blocker: true }, ...perfect.slice(1)],
  iteration: 1,
}).decision).toBe('loop');

// The gate takes the minimum, not the mean — one dissenting reviewer is
// enough. A switch to averaging would silently restore majority rule.

I checked it actually trips by setting the threshold to 7 and running the suite. One test fails, named publishability threshold is 8 of 10 — every reviewer must clear it, which is a considerably harder thing to talk yourself past at eleven at night than a green tick.

Correlated blind spots stay unsolved#

I don't have a good answer to Opus's point and I'm not going to pretend otherwise. Four models sharing training data is still four models sharing training data, and no amount of gate logic fixes that. The honest position is that the panel raises my floor rather than my ceiling — it catches what one model would have missed, not what all four were never going to see.

The whole consultation cost $0.11 and took thirty-two seconds, most of which was Grok thinking about it.

Seven models, four different winners, three cases of mistaken identity, and one genuinely uncomfortable critique of the system I'd built to avoid having to pick a favourite. Which is, I think, the answer to the question I actually asked.

Not which one is best. Which ones disagree usefully — and whether you've built something honest enough to listen when they turn on you.

This post was reviewed by four models before you read it, and the pipeline that did it is on GitHub — CC0, gate and all, including the tripwire designed to stop me quietly lowering the bar on exactly this post.

All four of them know what they are. The three that don't never got a vote.

---

Martin Harris connects cloud infrastructure to the real world. If you've ever built a committee of machines specifically so you wouldn't have to trust any one of them, and then discovered a third of the committee was wearing a false moustache, you're in the right place.

---

If you find this blog useful, please consider a donation.