The Night conversation-memory Grew Up (and Remembered a Secret)
conversation-memory started life months ago as "a quick little pg database to dump messages in". Every Claude Code session I run gets captured into it; an MCP server (Model Context Protocol) lets any future Claude search the lot. It lived in a Postgres container on my Mac, and that was fine.
By the end of one evening it was: Claude Code hook → local SQLite outbox → launchd Rust forwarder → SQS (with an S3 claim-check for anything over 256 KB). A Rust consumer on Cal (a 2011 iMac) decrypts, embeds via Ollama and inserts into pgvector on Luke (the LAN-only Postgres host, reachable only on the LAN). Everything age-encrypted client-side before it leaves the machine. Auth via IAM Roles Anywhere. The home network accepts no inbound connections at all — every hop is outbound-only, meeting in the middle at SQS.
The evening started with a two-line Ansible variable question.
The variable that pulled the thread
The capture config had two settings that could silently disagree: an AWS profile variable and a hardcoded SSM parameter path. Coupling them into a single convmem_env variable was the sort of five-minute tidy-up you do while the kettle boils. Except that looking at it properly exposed something bigger: the whole estate was pointed at the sandbox AWS account. Not by decision — by accident of history. The quick little database had grown production responsibilities while its infrastructure was still on a learner's permit.
So the tidy-up became a migration, and the migration found the next problem: the home certificate authority — which everything's mTLS depends on — had only ever been bootstrapped in sandbox. Real cross-estate security, resting on a throwaway environment. The bootstrap script got generalised, guarded by account ID (names can lie; account IDs don't) and a deliberate CONFIRM=create-prod-ca speed bump, and the production CA came into existence — with the KMS-wrapped key's modulus checked against the certificate, because trust-but-verify applies doubly to your own scripts.
Then it turned out the cert-issuing portal had never worked. Three separate bugs: a negative-serial bug (generate a serial whose leading byte is ≥ 0x80 and Go's parser rejects the certificate); a DynamoDB attribute-name mismatch that orphaned every cert after signing; and a JWT secret derived from Math.random() that rotated on every deploy, invalidating everything in flight. Three separate bugs in a service whose job is trust. Certificates are issued via a break-glass script for now — still HMAC-verified, just honest about the missing UI.
Mid-migration, every GitHub PR creation started returning 500s. Diagnosis by empty-commit test pinned it as GitHub's problem seven minutes before githubstatus.com admitted the incident. Some evenings the universe joins in. Once GitHub recovered and the playbook finally ran, the local estate pushed back.
The estate fights back
Luke's OS drive runs read-only in steady state, and three consecutive playbook tasks hard-failed on unguarded writes — a logical volume creation, then /etc/fstab, then the new backup units — each surfacing as [Errno 30], each looking like a storage problem rather than a filesystem-mode one. The eventual decision: stop scattering remount windows through the playbook and consolidate to one rw window per host. Read-only root is a discipline, and the discipline is mostly about your own tooling.
Then a question that should be asked of every migration, asked just in time: where do the backups go afterwards? The Mac — the machine being decommissioned — had nightly S3 backups. Luke, the new database of record, had none. The system's most durable component was the one being switched off. A Luke-side backup timer went in before a single row moved.
The M5 blitz
With the pipeline live, the backlog needed embedding, and Cal — that 2011 iMac — was managing about 0.6 embeddings a second. ETA: ninety minutes. This MacBook's M5 was sitting right there, still in service for one last job before its final wipe.
The lovely thing about the queue-based design is what it made the fix cost: nothing. Spin up a second consumer on the M5 — reuse the already-issued client cert, cargo build --release, point it at the same queue — and let the two machines compete for messages. No coordination protocol, no partitioning, no leader. SQS hands each message to whoever asks first; ON CONFLICT in the insert makes duplicate delivery harmless. The M5 pegged its GPU at ~8 embeddings a second and drained the backlog in about five minutes, while Cal plodded along beside it, blissfully unaware it was in a race.
Competing consumers: the textbook pattern, running across a 2011 iMac and an M5 MacBook, arbitrated by a cloud queue neither can be reached from.
The poison message
One message went to the dead-letter queue after six consecutive Ollama 500s. The easy read: transient failure, redrive it. But six identical failures doesn't smell transient, so it got decrypted and inspected.
It was 845 characters of drive serials, file paths and runs of = — a grep of RAID conflict markers from the very session doing the migration. The pipeline had choked on a memory of itself. Token-dense text like that blows past the embedding model's 512-token context even at modest character counts, and the existing 800-character truncation bounded the wrong unit. Characters are not tokens. The fix: call /api/embeddings with an appropriate num_ctx in the options, letting the model's own tokeniser enforce the model's limit rather than a character count guessed at the application layer.
(The DLQ redrive path thereby got its first end-to-end test, entirely by accident. Cheapest integration test I've ever run.)
And because "what if a consumer eats a message and dies?" deserved a real answer: the design was audited to five independent copies of any in-flight message, with the consumer deleting from SQS only after the Postgres commit. A crash mid-process means redelivery, and redelivery means ON CONFLICT DO NOTHING.
The secret
Then the test that mattered. I told the Claude in my home-servers session a secret — something true, told to no one else. A minute later, a completely separate Claude session, different repo, different context window, was asked what it was.
It gave the textbook answer first — sessions are isolated, it couldn't possibly know — then caught itself, remembered it had a conversation-memory tool, searched, and read the secret back verbatim. Its closing observation is the honest thesis of the whole build: a secret told to one Claude on this machine is effectively told to all of them.
The secret had been captured by a hook, queued in SQLite, encrypted, shipped through SQS, decrypted on a 2011 iMac, embedded by a local model, inserted into Postgres on a different machine — and surfaced through an MCP query from a session that didn't exist when it was spoken. Four machines and a cloud, there and back, in about a minute.
One coda: the retrieval displayed the timestamp as 21:37 and the other Claude called it "an hour ago". It was two minutes old. The data was right — UTC, correct offset, stored properly — and the presentation misread UTC as local. Storage layer right, display layer lying. Even the victory lap had a timezone bug in it.
What the secret was
I'm not going to tell you, obviously — it wouldn't be much of a secret twice over. It was something true, said to nobody else, and chosen deliberately: a real secret, entrusted to the one system in the house guaranteed to keep it, precisely to see whether it would hand it back.
What I can tell you is that this post was reconstructed by querying conversation-memory for the session that built conversation-memory. The system's first real job was remembering its own birth — and one secret, which is staying between me and the database.
Martin Harris connects cloud infrastructure to the real world. If you've ever told a database a secret to see whether it loved you back, you're in the right place.
If you find this blog useful, please consider a donation.