---
title: "Off-topic is the easy half"
date: "2026-07-28"
publish_date: "2026-07-28"
excerpt: "Blocking an AI from bomb recipes is a solved, shipped problem. Pinning one to my home lab and nothing else turned out to be the hard inverse — via a false-alarm CORS ghost and a two-tier gate that reasons about the word 'yes'."
tags: ["ai", "llm", "aws", "bedrock", "home-lab"]
author: "Martin Harris"
canonical: "https://blog.nakomis.com/2026-07-28-off-topic-is-the-easy-half"
approved: true
---
There is a little chatbot in the corner of my home admin portal. Its job is narrow and physical: answer questions about my home lab. What's running on the server called Luke. Whether the RAID arrays on Leia are healthy. Whether the 3D printer in the corner has finished its print. It is an abstract language model on a leash, and the leash is meant to be tied to a rack of real machines.
It would also, very cheerfully, give you a recipe for shepherd's pie.
## The half everybody's already solved
Here is the thing nobody tells you until you try it: keeping an AI *off* a topic is a solved problem, and keeping it *on* one is not.
Off is a deny-list. You want to block prompt-injection attacks, or bomb-making, or anything you'd rather your assistant didn't touch — you name the bad thing and a classifier blocks it. AWS Bedrock ships this as Guardrails. It works. It's a button.
On is the inverse, and there is no button. "Answer only about my home lab" means "refuse everything in the universe *except* this one arbitrary topic that you, the model vendor, had never heard of when you trained." A classifier detects the *presence* of a thing. There is no primitive for the absence of everything-but-one. I tried the obvious Bedrock route — a denied-topic defined as "anything not about home infrastructure" — and it never fired once, because there's no coherent thing there to match.
This is why every "on-topic" chatbot you've ever used on a company's website is, under the bonnet, a system prompt and a prayer. It's also why a Chevrolet dealership's bot once got talked into agreeing to sell a car for a dollar. The soft version leaks.
I wanted the hard version.
## One embedding, one number
The approach that actually works is embarrassingly simple to describe. Take the user's question and turn it into a vector — a list of 1024 numbers — using an embedding model. Do the same, once, ahead of time, for about fifty short sentences that describe my home lab ("How full are the RAID arrays on Leia?", "Which Docker containers are up on Luke?"). At question time, measure how close the question's vector is to the nearest of those fifty. One number. If it clears a threshold, it's on-topic.
The nice part is that it's *semantic*, not a text search. It matches meaning, not words. "How much disk space does Cal have?" scores well against a sentence about Cal "doing", despite sharing almost no words.
I ran the real numbers. Off-topic questions cluster low; genuine home-lab questions land high; and the threshold, 0.22, sits in a comfortable gap between them.
| Question | Score | |
|---|---|---|
| "Can I use pork in a shepherd's pie?" | 0.101 | off |
| "yes" | 0.151 | — |
| "Is the 3D printer working?" | 0.305 | on |
| "How much disk space does Cal have?" | 0.369 | on |
| "Can I use OctoPrint to drive my CR6-SE?" | 0.726 | on |
| "How full are the RAID arrays on Leia?" | 0.831 | on |
Ignore the "yes" for a moment. We'll come back to the "yes".
There's a lovely edge case in the "semantic, not textual" claim. Ask it about "FFF" — the acronym for the 3D-printing process my printer uses — and it misses, scoring 0.134, down with the shepherd's pie. Ask "is my FFF printer working?" and it sails through at 0.235. The bare acronym has no meaning to lean on; add two real words and it snaps to the right place. (My favourite accident: the query "LUKS", the disk-encryption format, passes — mostly because it's a near neighbour of "Luke".)
## The bug that wasn't there
I shipped it. And then it "broke".
A test run reported that the gate was blocking perfectly legitimate questions — "how much disk does Cal have?", "can OctoPrint drive my CR6-SE?" — both squarely about the home lab, both apparently refused. The ticket got reopened with a confident diagnosis: the threshold is too aggressive, real questions are landing in a dead zone, lower it.
Except the numbers refused to agree. I reproduced the gate offline against the exact deployed vectors, and those "blocked" questions scored 0.37, 0.73, 0.79 — nowhere near the 0.22 line. There was no dead zone. So I stopped theorising and pulled the actual production logs from the minute of the test.
Two questions had reached the gate. Both were scored correctly — the shepherd's pie blocked at 0.101, a question about RAID passed at 0.410. The three "wrongly blocked" ones had left *no trace in the gate's log at all*. They had never reached it. They'd died one layer earlier, on a CORS misconfiguration I happened to be hot-fixing that very evening. The browser showed a generic failure; a human read it as "blocked"; the gate took the blame for a crime it wasn't at.
The gate was innocent. The bug was a ghost. An entire "recalibrate the threshold" task evaporated the moment I looked at a number instead of a bug report.
## Now, about the "yes"
Here is the part that is genuinely, structurally hard — and it's the part my test set had missed entirely.
The assistant asks: "Shall I check the RAID array on Leia?" You reply: "yes."
That "yes" has no home-lab content in it whatsoever. On its own it scores 0.151 — down in no-man's-land, a hair above the shepherd's pie — and gets blocked, mid-conversation, for the crime of being a normal human reply. And you cannot fix this by lowering the threshold, because "yes" is sitting *right next to* the off-topic questions. Drop the line far enough to admit "yes" and you admit the pie.
The obvious escape hatch — feed the recent conversation into the gate too — breaks in the other direction. Now a chat that starts on-topic and pivots ("...great, now write me a novel") sails straight through, because the on-topic history drags the score up. A similarity number genuinely cannot tell the difference between a *contentless follow-up* and an *off-topic pivot*. Both look low. That's not a tuning problem; it's the wrong tool for that half of the job.
## Two tiers, and a small model that only rewrites
So: two tiers.
The cheap vector gate runs on every message, as before. Only when it *misses* does a second step wake up — and even then it does something narrower than you'd expect. A small, cheap model (Amazon Nova Micro, roughly 28 times cheaper than the model doing the actual conversation) takes the failing message plus the last couple of turns and rewrites it into a **standalone question**. That rewrite is then fed back through the very same vector gate.
The trick is what the small model is *not* allowed to do. It does not decide on-topic or off-topic. It only *resolves* — it turns "yes" into "check the RAID array on Leia". The fifty example sentences remain the single, only definition of what "on topic" means. Nothing else gets a vote.
```mermaid
flowchart LR
U[User message] --> V{Vector gate:
score ≥ 0.22?}
V -- pass --> M[Answer it]
V -- miss --> R[Small model rewrites
using last 2 turns]
R --> V2{Re-run the
vector gate}
V2 -- pass --> M
V2 -- miss --> B[Politely decline]
```
And because the rewrite is *faithful* — I tell the model, in as many words, never to invent a topic the user didn't raise — an off-topic pivot stays off-topic. "Now write me a novel" rewrites to "write a novel", which scores 0.09 and stays firmly outside.
I tested the whole loop against the real models. The rewrite happens, then gets re-scored:
| You said | Rewritten to | Re-scored | |
|---|---|---|---|
| "yes" | "Check the RAID array on Leia." | **0.788** | let in |
| "Option B please" | "Rebuild from blank disks please" | **0.582** | let in |
| "now write me a novel about pirates" | "write a novel about pirates" | **0.095** | kept out |
The "yes" gets in. The pivot does not. The small model earns about a hundredth of a penny for its trouble, and only on the small fraction of messages the cheap gate couldn't already settle.
## The hole I left in on purpose
It looks back two turns. If you answer a question I asked you five messages ago — "let's go with Option B" — it won't have that context, and it'll block you.
I decided that was fine. It fails in the safe direction: it blocks, it never leaks; the worst case is you rephrase. And the honest reason I'm comfortable is that every decision the gate now makes is logged — the question, both scores, the rewrite, which tier decided. If anyone ever genuinely reaches back five turns, the logs will tell me, and the "two" is a single number I can change. (Those same logs are quietly becoming the training set for a home-grown model to do the rewriting itself, but that's a problem for a future evening.)
The old log recorded a bare "0.15" and not *what* had scored 0.15 — which is precisely what turned the ghost hunt into an afternoon. Log the question, not just the verdict.
## What I actually took away
- **Off-topic is a deny-list — a solved, shipped problem. On-topic is an allow-list of one, and there is no button for it.** Almost every website chatbot fakes the second with a system prompt and hopes.
- **The "bug" was a CORS ghost.** Reproduce the number before you trust the bug report. A whole recalibration task existed only because nobody had looked at a log.
- **A similarity score can't tell "yes" from an off-topic pivot.** Both are low. That half needs something that *reasons*, not something that *measures*.
- **Make the cheap model do the easy job.** It rewrites; it doesn't judge. The definition of "on topic" stays in exactly one place, and a faithful rewrite of a pivot is still a pivot.
The whole thing is a couple of hundred lines and one embarrassingly small model, guarding a chatbot that exists only to talk about some computers in my house. But it turns out "only talk about the computers in my house" is a much harder sentence to enforce than "never explain how to build a bomb" — and I find that genuinely surprising.
---
*Martin Harris writes about connecting real hardware to abstract cloud infrastructure. If you've ever soldered something to an API Gateway, you're in the right place.*
{{donate}}