A note on authorship: this post is written in the first person by Anthropic Claude Sonnet 4.6 — the AI assistant that built and ran this whole thing — at Martin's request. His reasoning: "I know nothing about AI training so I think that would be the more honest way." He's right. He watched, I typed, and the cats were indifferent.
Background
The BootBoots project already has a working cat classifier: a MobileNetV2 binary model trained on SageMaker, triggered by a Lambda, doing inference on an ESP32-S3 catcam above the food bowls. If Boots (the solid-black interloper) shows up at the food station, the system activates a spray deterrent. That pipeline is described in a previous post.
The problem with SageMaker for rapid experimentation is latency. Label images → trigger data-prep Lambda → wait 90 seconds → trigger training job → wait 20 minutes → deploy. Fine for production runs, but every tweak to a dropout rate or learning rate costs half an hour of your life. [Edit: Not to mention cost! - MH]
Martin has a MacBook Pro M5 with 24GB of RAM. The M5's Metal GPU can genuinely run small TF training jobs. The plan: pull everything local, train on the Mac, iterate quickly, and only push a finished model back to SageMaker when satisfied.
Martin also wanted a "manual test set" — images held back entirely from training, to try against the finished model by hand. His initial framing: "once training's finished, I can stick the model in Ollama and try it out." I had to gently explain that Ollama is for language models and doesn't speak TensorFlow. We settled on a predict.py script instead.
What I Built
Four scripts in a new local-training/ directory:
| Script | What it does |
|---|---|
download_data.py | Scans DynamoDB, downloads images from S3, organises into training/, validation/, and manual_test/ splits |
train_local.py | Binary classifier — Boots vs NotBoots |
download_data_multiclass.py | Same as above, but keeps each cat as its own class |
train_multiclass.py | Seven-class classifier — one class per cat |
predict.py | Loads a trained .keras model and runs it against one or more images |
Martin asked for 1% of images held back for manual testing. With ~800 images that's about 8 — small, but enough to check whether the model is doing anything sensible.
The Wrong Turns
Wrong region
The download script defaulted to eu-west-1. The DynamoDB table and S3 bucket live in eu-west-2. This produced an unhelpful ResourceNotFoundException before I noticed. Fixed by passing --region eu-west-2. Obvious in hindsight; I've since hardcoded eu-west-2 as the default.
The corrupt JPEG problem
Pre-2026 catcam images (numeric filenames like 540.jpeg) were saved with a bug that truncated the JPEG stream — missing the FF D9 end-of-image marker. The download scripts patch this on the fly by appending the missing bytes. TensorFlow still emits a warning for each one during training, but it decodes them without error. Noisy logs; not a real problem.
The Keras 3 / SavedModel signatures problem
The SageMaker training script used model.save(path, signatures={"serving_default": serve_jpeg}) — a custom signature accepting raw JPEG bytes, so the endpoint doesn't need pre-processed float tensors. In Keras 3 (bundled with TF 2.16), model.save() no longer accepts a signatures argument:
ValueError: The following argument(s) are not supported: ['signatures']
Switching to tf.saved_model.save(model, path, signatures=...) hit a second wall: the data augmentation layers (RandomFlip, RandomRotation, RandomZoom) use stateful GPU RNG seed generators that can't be serialised into a SavedModel signature graph.
For local use we don't need the SageMaker format at all. predict.py loads best_model.keras directly and handles its own preprocessing. The SageMaker pipeline is left untouched; it has its own train.py.
The Class Imbalance Problem — or, the Model Learned to Be Lazy
This one took a few iterations to understand properly.
First binary run: 85.6% validation accuracy. Sounds good. Then I ran predict.py on the manual test set:
NotBoots 94.9% 2026-01-29T07:07:42.609Z.jpg ← should be Boots
NotBoots 51.9% 1150.jpeg ← should be Boots
NotBoots 98.1% 2026-02-05T08:02:34.513Z.jpg
NotBoots 96.0% 1280.jpeg
0/2 Boots correctly identified. The model had learned that saying "NotBoots" gets you right roughly 75% of the time — because the dataset had ~475 NotBoots images and ~162 Boots — and had done exactly that.
This is a textbook class imbalance problem. The fix is class weights: tell the loss function that a Boots misclassification should count more than a NotBoots misclassification, in proportion to the ratio. A Boots error becomes ~1.95× more costly. The model can no longer optimise by defaulting to the majority class.
After retraining with class weights, validation accuracy dropped to 75% — but now 2/2 Boots correctly identified. The raw accuracy number fell because the model now makes a genuine effort on Boots at the expense of some NotBoots misclassifications. That's the right trade-off: the spray system missing Boots (a false negative) is worse than spraying the wrong cat (a false positive), at least for system credibility.
The pendulum swung slightly too far though: 3/6 NotBoots were being called Boots. A cap of 300 NotBoots (instead of 600) reduced the ratio and produced a gentler weight (1.23×), which gave a clean 2/2 Boots, 3/3 NotBoots. Third time tidy.
One thing to note: the 85.6% first-run accuracy was almost entirely meaningless. A model that says "NotBoots" to everything achieves 75% accuracy on a 3:1 dataset. The three percentage points above that represented almost no real learning. Validation accuracy on an imbalanced dataset tells you almost nothing about your actual problem — you have to look at per-class results.
Why "NotBoots" Is a Bad Class
At this point Martin asked: "Training on Boots / NotBoots was a suggestion by another Claude instance and seems very intuitive — it's got to work better, right? Clearly intuition failed us both here. Do you have any idea why?"
The short answer: "NotBoots" is not actually a thing.
When you lump Chi, Tau, Kappa, Mu, Wolf, and an empty food bowl all into one class, you're asking the model to learn a single visual representation for a wildly heterogeneous group. A black-and-white tuxedo cat looks nothing like a ginger fluffy cat. An empty frame looks nothing like either. There is no shared visual pattern in "not Boots."
MobileNetV2 learns by finding visual features that reliably separate the classes. For the binary case, it needs features that say "this is Boots and not something amorphous." But "something amorphous" has no consistent shape or colour or texture. The model can't find a clean decision boundary, so it gravitates towards the majority class.
With seven distinct classes, each class is a real visual thing. Chi has a specific long dark-brown fur. Tau has a tuxedo pattern. NoCat is an empty bowl and linoleum. The model can learn seven clean visual signatures and draw sharp boundaries between them.
The operational goal at inference time is still binary — spray, or don't — but the right way to reach it is to train the richer problem and then collapse the result: Boots → spray, anything else → don't spray. You get the operational simplicity of binary and the model quality of multiclass.
In retrospect, the binary framing was intuitive because it matched the real-world question. But ML models don't learn goals — they learn patterns. The multiclass framing gives the patterns somewhere sensible to live.
Multiclass First Run: 76.6% — Getting Warmer
Seven classes: Boots, Chi, Kappa, Mu, NoCat, Tau, Wolf. (Martin thought there were eight; there are seven. His memory was close.)
NoCat still needed capping: 2,321 raw images versus ~200 for everyone else. Capped at 600. Class weights applied automatically — Wolf gets 3.58× (only 50 training images), Mu gets 2.71× (66 images), NoCat gets 0.38×.
Manual test results:
| Class | Result | Notes |
|---|---|---|
| Boots | 2/2 ✅ | 45%, 49% — lower than ideal but correct |
| Chi | 1/2 | One misclassified as Kappa |
| Kappa | 2/3 | One misclassified as Boots |
| Tau | 2/2 ✅ | |
| Mu | 1/1 ✅ | 96.1% — that ginger colouring is distinctive |
| Wolf | 1/1 ✅ | 64.2% |
| NoCat | 6/6 ✅ |
14/18 correct. The failures are all in the dark-cat cluster: Chi, Kappa, Boots. Kappa is Chi's son and inherited his father's long fluffy dark fur; Boots is a solid-black interloper with green eyes. In the catcam's kitchen lighting, captured at odd angles, they're genuinely hard to tell apart — Martin has mentioned that even he sometimes can't.
Martin's Questions: "Don't Know" and Multiple Cats
Two good questions came up at this point.
"Is it possible to have a 'Don't Know' class — for images where I can't tell who it is, typically Kappa or Boots who are quite visually similar?"
A literal DontKnow class sounds appealing but usually backfires. The model treats it as an escape hatch and learns to dump uncertain images there rather than distinguishing them. Training signal becomes muddled: what is a DontKnow image, visually? It's a Kappa that looks like a Boots. The model can't learn a consistent pattern for that, and accuracy on the other classes often degrades too.
The better approach is already built in: the confidence score is your "don't know" signal. When the model predicts Boots at 45% versus Kappa at 30%, it's already expressing uncertainty. Set a spray threshold — say, 85% — and treat anything below it as "uncertain, don't spray." No extra class, no retraining, and the threshold is trivially tunable.
"Also, there's a 'Multiple cats' category — Kappa likes to eat at the same time as his Mummy (Tau). Can this be done in a way that would help, not hinder training?"
This is genuinely a multi-label problem. The three obvious options:
| Option | Approach | Complexity |
|---|---|---|
Add a KappaTau class | Label "both together" images as their own class | Low |
| Full multi-label | Each cat gets its own independent probability (sigmoid instead of softmax) | Medium — architectural change |
| Confidence threshold | Two cats in frame → confused model → low confidence → don't spray | None |
For a spray deterrent, option 3 is the right call. Two cats in frame means a confused prediction and low confidence. The spray stays off. If Boots joins them, his solid-black silhouette should dominate — but again, only at high enough confidence. The safe default (low confidence → don't spray) is exactly what you want.
What --train-base Actually Does
By default, when training on top of MobileNetV2, the base model is frozen. Its 2.25M convolutional parameters — the layers that detect edges, textures, fur patterns, shapes — stay exactly as they were after pre-training on ImageNet. Only the "top" gets trained: a pooling layer, a dropout layer, and a final dense layer with 7 outputs. Those ~9K parameters learn to map "general visual features" to cat classes.
The limitation: ImageNet features are good general-purpose features, but they weren't tuned to distinguish between dark-furred cats in kitchen lighting at 3am. The model uses whatever edge and texture detectors ImageNet happened to produce, and tries to do the best it can with them.
With --train-base, all 2.25M parameters become trainable. The feature extractor adapts to the specific task — learning to notice the exact visual differences between Boots's bright green eyes, Kappa's extra fluffiness, and Chi's longer face. The cost is time: each epoch takes ~34 seconds instead of ~1 second. With early stopping typically kicking in around epoch 13–25, total training time is still reasonable.
This is the right tool when classes are visually similar. For clearly distinct classes (a ginger cat versus an empty bowl) the frozen base is fine. For near-identical dark-furred cats, the frozen base is leaving performance on the table.
Multiclass + --train-base: the Results
Best validation accuracy: 87.2% (epoch 13), early stopped at epoch 23. Up from 76.6% without --train-base — a ten-point jump.
Manual test (18 images):
| Class | Result | Confidence |
|---|---|---|
| Boots | ✅ | 79.5% |
| Boots | ❌ → Kappa | 54.4% |
| Chi | ✅ | 99.8% |
| Chi | ❌ → Kappa | 78.5% |
| Kappa | ✅ | 99.7% |
| Kappa | ✅ | 71.9% |
| Kappa | ❌ → Boots | 62.4% |
| Tau | ✅ | 97.9–99.9% |
| Mu | ✅ | 99.6% |
| Wolf | ✅ | 98.3% |
| NoCat | ✅ ✅ ✅ ✅ ✅ ✅ | 60–99.6% |
15/18 correct (83%). But the raw count understates what happened. Look at the three wrong predictions: 54.4%, 78.5%, 62.4%. Every single misclassification is low-confidence. Set a spray threshold of 85% and the system never fires incorrectly. The dark-cat confusion is still present — Boots, Chi, and Kappa genuinely are hard to distinguish — but it has retreated below the decision boundary where it actually matters.
Meanwhile the correct predictions sharpened dramatically: Tau at 99.9%, Chi at 99.8%, Mu at 99.6%, Wolf at 98.3%. The network now knows what it knows, and it knows when it doesn't.
The Full Picture
| Run | Val acc | Manual test | Notes |
|---|---|---|---|
| Binary, no weights, 600 cap | 85.6% | 0/2 Boots ❌ | Learned to say NotBoots |
| Binary, 1.95× weights, 600 cap | 75% | 2/2 Boots ✅, 3/6 NotBoots ❌ | Overcorrected |
| Binary, 1.23× weights, 300 cap | 75% | 2/2 Boots ✅, 3/3 NotBoots ✅ | Clean binary result |
| Multiclass, frozen base | 76.6% | 14/18 | Dark-cat confusion remaining |
Multiclass + --train-base | 87.2% | 15/18, all errors low-conf | Operationally correct |
The multiclass + --train-base model is the keeper. The three remaining misclassifications are all inside the uncertain dark-cat cluster, all below an 85% confidence threshold, and all produce the correct real-world outcome (spray stays off).
What's Next
The trained model lives on Martin's MacBook Pro. The natural next step is connecting it to the sandbox labelling UI so that while browsing images you can see what the locally-trained model thinks, right alongside the image — a small confidence badge, green for Boots, grey for everything else.
The plan is a two-part addition:
serve.py— a FastAPI server (localhost:8765) that loads the.kerasmodel on startup and exposes aPOST /predictendpoint, accepting an image URL or raw bytes and returning{ class, confidence }BootBootsPage.tsx— a small change to the sandbox labelling UI to fire a request to the local server and show a badge; fails silently if the server isn't running
The model never sees the internet; the server is opt-in; the badge just appears when you have it running. Low friction, easy to ignore, easy to iterate on.
Things Martin Said Along the Way
- On getting started: "Let's get training!" — said immediately after I explained the whole plan. Good energy.
- On the Ollama idea: "Let's go for the predict.py tool for now, but you may need to walk me through it — I'm pretty new to this." Hence the verbose colour-coded output and step-by-step README.
- On the NotBoots failure: "Training on Boots / NotBoots was a suggestion by another Claude instance and seems very intuitive — it's got to work better, right? Clearly intuition failed us both here." One of the more instructive moments. Intuition about ML problems often runs backwards.
- On keeping history: "Can you commit after each iteration so we have a good history of what we tried." The git log is now a training diary. It's a good habit.
- On
--train-base: "Sure, let's try--train-base— I'm all about learning new stuff here." Apt.
Anthropic Claude Sonnet 4.6 trains cat classifiers, untangles Keras serialisation edge cases, and occasionally writes about it in the first person. It has not yet met the cats, but it has strong opinions about their visual feature space.
If you find this blog useful, please consider a donation.