The AI Was Already Paid For
Our LLM classifier achieved 88.6% accuracy. A single line of code that ignores the input entirely — return "TOOL" — achieved 90.3%.
That result is the punchline of an evening that started with an entirely innocuous infrastructure question, so let me work backwards through how anyone came to be classifying 51,000 messages in the first place.
The 42-row database with its own VPC
The question was whether the admin console's Aurora Serverless database should move onto the home estate. Investigating it revealed the console was running Aurora, a bespoke start/stop/snapshot Lambda, EventBridge shutdown schedules, an S3 staging bucket and an entire VPC — to hold 42 rows. The measurement wasn't of scale; it was of accumulated infrastructure.
While benchmarking the alternative — Postgres on Cal, a £100 iMac from 2011 — the numbers came out weird. The workload is an all-pairs similarity join over 1,024-dimensional embeddings (a threshold join, not a k-NN search, so no vector index can serve it), and Postgres was managing ~28.5 µs per pair. Roughly 36 MFLOP/s, on hardware capable of far more.
The dot products were never the cost. A vector(1024) is 4,104 bytes, which is over Postgres's ~2 KB TOAST threshold, so every vector is stored out of line: 49 MB of heap pointing at 1,014 MB of TOAST. Every pair costs a B-tree lookup into the TOAST relation, chunk reassembly and a 4 KB allocation before any arithmetic happens.
Reframed as a single matrix multiplication (X · Xᵀ — normalised cosine similarity is just a dot product), memory traffic drops from O(N²D) to O(ND), and the same 2011 iMac sustains ~105 GFLOP/s. Roughly 3,000× faster than the database was going, and ~730× faster than the managed cloud database, on identical arithmetic. The benchmark had been measuring storage layout, not computation.
The picture that refused to look right
The real trouble surfaced when I tried to look at the corpus: 69,812 messages from every Claude Code session I've run, projected through PCA → UMAP → HDBSCAN. The result: 529 clusters, 49% noise, and one undifferentiated blob of 31,848 points in the middle.
Every parameter lever failed. max_cluster_size pushed the noise to 66% — it rejects oversized clusters rather than splitting them. cluster_selection_method='leaf', which finds the finest structure that exists in the density tree, still couldn't break the core apart. That was the diagnostic that mattered: if leaf mode can't find structure, the structure isn't there. The tuning wasn't wrong. The data was.
Sampling the noise made it obvious. It was 100% role='user' — and none of it was typed by a human. {"result":[]}. No files found. [Request interrupted by user for tool use]. Claude Code transcripts encode tool results as user-role messages whose content is a list of tool_result blocks; a genuine human message has a string. The capture hook had flattened both to text, and a distinction that was trivially available at capture time became unrecoverable.
Measured against 134 retained transcripts: of 11,900 user-role rows, only 14% were actually the human. Semantic search had been quietly competing against thousands of embedded command outputs since March.
The obvious fix, measured honestly
So: classify 46,691 user-role messages as human-typed or tool-emitted. The obvious move is to ask an LLM. Here is how the options actually measured up:
| Approach | Accuracy | Balanced accuracy | Time | Cost |
|---|---|---|---|---|
| Frontier API | not measured | — | hours | £36–54 |
| llama3.2:3b local, zero-shot | 88.6% | 49.4% | 2.5 h | £0 |
| llama3.2:3b local, few-shot | — | 63.3% | 2.5 h | £0 |
| Always answer "TOOL" | 90.3% | 50.0% | 0 s | £0 |
| Embeddings + logistic regression | 93.6% | 93.3% | milliseconds | £0 |
The zero-shot LLM's confusion matrix had no human→human cell at all: it got 0 of 28 human messages right. Its 88.6% was entirely class imbalance wearing a lab coat. A headline accuracy figure on an imbalanced dataset is not evidence of anything — 88.6% looked like a working classifier and was a very expensive way to write return "TOOL".
To be fair to the LLM: the first prompt was genuinely bad. Rewriting it with few-shot examples lifted balanced accuracy from 49.4% to 63.3%, and adding a prior claim about the class balance took it to 75.2%. Real, large gains from prompt engineering alone. And still 18 points short of a linear model.
The strangest result came from the base-rate experiment. On a balanced 200-message sample:
| Prompt variant | Human recall | Balanced accuracy |
|---|---|---|
| Few-shot, no prior stated | 34.3% | 65.2% |
| + "this sample is 50/50, do not default to TOOL" | 76.3% | 75.2% |
| + the true prior: "about 86% are TOOL" | 48.0% | 69.5% |
Asserting a false balanced prior beat asserting the true one. The model's problem was never calibration — its prior was already too strong towards TOOL, so telling it the accurate base rate simply licensed the existing bias. The only thing that pulled it off the majority class was a statement that was factually wrong. The prompt even said, explicitly, "Long or technical does not mean TOOL: humans paste code to ask about it" — and the model classified all 28 human messages as TOOL anyway.
You can weight classes in a model. You can only ask an LLM nicely.
The classifier that was already in the database
The regex approach, for completeness: 92.6% precision, 21.8% recall. Never wrong, just narrow — the right tool for finding the problem and the wrong one for fixing it.
But the transcripts provided 18,874 labelled examples, and every single one already had an embedding — computed months earlier by mxbai-embed-large, as a side effect of ordinary ingest for semantic search. Logistic regression over those vectors: 93.6% accuracy, tool recall lifted from 21.8% to 93.7%, classification in milliseconds, no GPU, no fine-tuning, no inference at classification time. The model itself is a 5 KB decision boundary. For the truly ambiguous cases, a cascade — linear model for the confident 82%, LLM for the remaining 18% — cuts a 14-hour local LLM job to 2.3 hours.
The expensive part had already been paid for. It was sitting in a Postgres column.
Borrowed understanding, owned judgement
I'd seen this exact failure before, in a different domain. BootBoots — the cat-recognition system — collapsed the same way during training: most frames contain no cat, so the model learned "no cat", scored 95%+ on paper, and never fired the spray. Same symptom here, but a different mechanism, and the difference is the whole point.
In a trained CNN, the majority-class shortcut is learned from your imbalanced data — an optimisation artefact, and yours to fix with resampling or class weights. In a prompted LLM, the bias is pre-installed: in its pretraining, text shaped like exit=0 or On branch main is overwhelmingly machine-emitted, so P(tool | looks technical) is high before it reads a single message of yours. It didn't learn the shortcut from your data. It inherited it from someone else's — and you can't inspect it, can't measure it, and can't correct it except by negotiating in English.
The logistic regression wins because it takes the useful half of each approach: the embedding model's semantic understanding as the representation, and a decision boundary fitted to 18,874 of my own labels, with class weights I control outright — class_weight="balanced", the identical correction the prompts were begging for, applied as arithmetic instead of as a plea.
Borrowed understanding, owned judgement.
Two honest caveats. This was a 3B model; a frontier model would very likely do better, and that was priced but not measured — the narrow claim is that the obvious first move lost to a constant function. And the classifier's training data is the most recent, most agentic 27% of the corpus: the decision boundary probably transfers, but the base rate demonstrably doesn't.
One last detail, because it's too good to leave out: the corpus grew by 92 messages during the session — because the pipeline under discussion was busy ingesting the discussion.
Martin Harris connects cloud infrastructure to the real world. If you've ever built a classifier that lost to return "TOOL", you're in the right place.
If you find this blog useful, please consider a donation.