---
title: "How to Map a Pile of Text Nobody Has Read"
date: "2026-08-06"
publish_date: "2026-09-20"
excerpt: "A step-by-step guide to turning any corpus into a named, explorable 3D map — embeddings, UMAP, HDBSCAN, and a 3B model that will lie to you unless you make it prove itself. With a CC0 tool, a downloadable public-domain dataset, and the story of the thesaurus I published as a Victorian cookbook."
tags: ["machine-learning", "umap", "hdbscan", "embeddings", "postgres", "ollama", "how-to"]
author: "Martin Harris"
repos:
  - name: "corpusmap"
    url: "https://github.com/nakomis/corpusmap"
canonical: "https://blog.nakomis.com/2026-08-06-how-to-map-a-pile-of-text-nobody-has-read"
approved: true
---

# How to Map a Pile of Text Nobody Has Read

![A dark bookshelf lit by a single shaft of light, picking out the gilt spines of the eight public-domain books that make up the demo corpus: On the Origin of Species and The Voyage of the Beagle by Charles Darwin, Moby Dick by Herman Melville, Twenty Thousand Leagues Under the Sea by Jules Verne, Pride and Prejudice by Jane Austen, The Adventures of Sherlock Holmes by Arthur Conan Doyle, The Book of Household Management by Mrs Beeton, and Meditations by Marcus Aurelius](images/how-to-map-a-pile-of-text-nobody-has-read-header.png)

Every Claude Code session I've ever run gets archived. Transcripts, tool calls, the lot — about 76,000 messages at the last count, sitting in a Postgres database I've named `claude_chats`. It is, in the most literal sense, a written record of two years of my working life.

I have read approximately none of it.

The database isn't neglected, mind. It already has a reader that gets through every word: the [conversation-memory MCP](https://blog.nakomis.com/2026-05-15-giving-claude-a-persistent-memory) embeds each message so Claude can search two years of sessions semantically, which is the whole reason `claude_chats` exists. That reader never tires and never skims, and it is extremely well served. It also runs on silicon.

The problem is the other audience. For those of us who run on meat, 76,000 rows behind a vector index is not something you can *look* at. It answers questions, but only the ones you already knew to ask, and it will never volunteer that a third of last spring was the same argument with different variable names. Anyone with a support desk, a mailing list, a scraper or a feedback form has the same pile and the same gap: storage solved, comprehension not. This post is about the second half — organising and presenting a corpus so a human can see the shape of it. By the end you'll have turned a table of text into a set of named clusters and a three-dimensional map you can fly through: writing, about as human and untidy an artefact as exists, converted into coordinates you can walk around.

Everything here is runnable. I've packaged the pipeline as a CC0 tool called [corpusmap](https://github.com/nakomis/corpusmap), and rather than make you point it at your own data to see whether it works, there's [a downloadable dataset](https://github.com/nakomis/corpusmap/releases/latest) of 15,534 passages from eight out-of-copyright books. Same commands, same output, no personal data, no API key.

## What you'll need

Docker, Python 3.10 or newer, and [Ollama](https://ollama.com). No cloud account. Nothing in this pipeline calls a paid API — the largest model involved is 3 billion parameters and runs on a laptop.

```bash
git clone git@github.com:nakomis/corpusmap.git && cd corpusmap
python3 -m venv .venv && .venv/bin/pip install -e '.[fast]'
docker compose up -d          # Postgres + pgvector on port 5433
ollama pull nomic-embed-text  # 274 MB — turns text into vectors
ollama pull llama3.2:3b       # 2 GB — writes the labels
```

## Step 1: Get some text into a table

The tool wants one thing: a table with an id, some text, and a [pgvector](https://github.com/pgvector/pgvector) column. Every column name is configurable, so if you already have such a table, point at it and skip to step 2.

For the demo corpus, there's a builder script:

```bash
export DSN=postgresql://corpusmap:corpusmap@127.0.0.1:5433/corpusmap
.venv/bin/python datasets/build_gutenberg.py --dsn "$DSN" --reset
```

That downloads eight books from Project Gutenberg and chops them up. It takes about fifteen seconds, and two of the decisions inside it are worth stealing.

**Eight books, not one.** My first instinct was a single novel — it's cleaner, it's one download, and it's what "split a book into sentences" suggests. It's also useless. One novel is one author, one register, one subject, so the map comes out as a single undifferentiated smear and you learn nothing about whether your pipeline works. A deliberately mixed shelf — Austen next to Darwin next to Mrs Beeton's *Book of Household Management* — gives you clusters you can check by eye. If the recipes don't separate from the marine biology, the problem is your code, not the world.

**Passages, not sentences.** A sentence is usually too short to have a topic. "He said nothing." belongs to no cluster in any corpus that has ever existed. The builder glues consecutive sentences together up to 600 characters, never crossing a paragraph boundary — roughly a short paragraph, which is about the smallest unit that reliably means something on its own.

There's also a third decision that only announced itself after I looked at the output. Project Gutenberg wraps every text in a licence header and footer, and those are long, formulaic, and nearly identical across all eight books. Left in place, they form their own enormous, perfectly coherent cluster about the Project Gutenberg Literary Archive Foundation. The pipeline worked flawlessly and told me something I did not want to know. Strip the boilerplate.

## Step 2: Turn the text into vectors

```bash
.venv/bin/corpusmap embed --dsn "$DSN" --model nomic-embed-text --workers 8
```

An embedding is a "meaning fingerprint" — a fixed-length list of numbers, 768 of them for this model, where documents about similar things end up with similar numbers. Everything downstream operates on the fingerprints and never looks at the text again until it's time to write labels.

15,534 passages took 117 seconds at about 130 a second. If you'd rather not wait at all, the 59 MB dump on the release page already has them.

Two properties of embedding models bite people, so here they are up front:

**Context is short, and truncation is silent.** `nomic-embed-text` handles 8,192 tokens, but `mxbai-embed-large` handles 512. Go past the limit and the excess is dropped without complaint, so a long document gets embedded on its opening paragraph and you never find out. `corpusmap embed` truncates explicitly at a length you choose, on the grounds that if text is going to be thrown away it should at least be you throwing it.

**Dimensions aren't comparable.** A 768-number nomic vector and a 1024-number mxbai vector cannot share a column, and changing model means re-running everything downstream. Pick one before you embed 75,000 things.

The embed step is resumable — it selects on `WHERE embedding IS NULL` and commits per batch. That's not a nicety. Embedding a real corpus takes long enough that something *will* interrupt it, and starting from zero each time is how people give up on this.

## Step 3: Squash 768 numbers down to 3

```bash
.venv/bin/corpusmap run --dsn "$DSN" --facet-col category \
                        --meta-cols title,author,chapter \
                        --out build --prefix gutenberg --llm llama3.2:3b
```

That single command does the rest. Here's what's happening inside it.

### Normalise

Scale every vector to unit length, so only its *direction* matters. Think of a compass: you care which way the needle points, not how long it is. Skip this and a document can look distinctive purely for having larger numbers — which is a fact about how much text it contained, not about what it said.

### PCA: 768 → 50

A fast, crude cull. If nearly every document has about the same value in slot 738, that slot distinguishes nothing, so throw it away. PCA keeps the fifty directions carrying the most variation. It's lossy, but what it discards is precisely what nobody differed on, and it makes the next stage several times faster.

### UMAP: 50 → 3

The clever stage, and the misunderstood one.

UMAP doesn't try to preserve everything. It asks one narrow question — *who are each point's fifteen nearest neighbours?* — and finds a three-dimensional arrangement that keeps those neighbours together.

Think of seating a wedding. You cannot satisfy every relationship at once, so you settle for a weaker guarantee: everyone sits near their actual friends. Some tables end up next to each other for no reason at all.

Which brings me to the one thing I'd write on the wall:

> **A cluster's membership is meaningful. The distance between two clusters is not.**

Two blobs at opposite ends of the view are not "more different" than two adjacent ones. The map is trustworthy up close and meaningless at long range. This is true of every UMAP plot you have ever seen, including the ones with confident arrows drawn across them, and it is the single most common way these maps get over-read.

One deliberate slowdown: `random_state` is fixed, which forces UMAP single-threaded. That costs real time on a large corpus and I'd do it again — a map that rearranges itself every time you rebuild it is impossible to talk about, and "the cluster in the top left" needs to still be there tomorrow.

## Step 4: Find the clumps

Every document is now a dot in 3D space. HDBSCAN looks for dense clumps of them.

![An aerial night photograph of a music festival, with tight bright knots of people gathered into distinct groups and scattered stragglers drifting through the dark spaces between them](images/how-to-map-a-pile-of-text-nobody-has-read-2.png)

Picture an aerial night photo of a festival. The tight knots of people are groups; the stragglers drifting between them are not a group, they're stragglers. HDBSCAN is comfortable saying "this point belongs to nothing", which k-means structurally cannot, and that willingness to shrug is most of why it's the right tool.

**Expect to throw away a lot.** 42% of the demo corpus comes back as noise. My chat corpus was 48%. That's not a failure — about half of most corpora genuinely is one-off material with no topic to belong to, and a tool that pretended otherwise would be lying to make its output look tidier.

Then there's the setting nobody mentions. On my real corpus, one fit produced a single cluster of **16,890 points** — 37% of everything, in one lump, that no label could describe. My first assumption was a bug. It isn't. HDBSCAN's default *excess of mass* selection keeps the most **stable** clusters, and a broad featureless bulk is stable precisely *because* it has no internal structure to split along. It is the most boring possible group and therefore the most robust one.

The standard remedy is `cluster_selection_method='leaf'`. I measured it against simply capping the cluster size, on 45,183 documents:

| Selection | Clusters | Noise | Largest | Useful |
|---|---|---|---|---|
| `eom`, min 20 | 240 | 25.3% | 16,959 | 37.1% |
| `leaf`, min 20 | 334 | 59.2% | 318 | 40.8% |
| `eom`, min 20, **max 2000** | 294 | 50.6% | **1,011** | **49.4%** |

The cap wins on every axis, which was the surprise. Its cost is honest rather than hidden: noise roughly doubles, because the blob's points don't become good clusters — they become scatter, which is what they always were.

## Step 5: Give the clusters names

A coloured blob labelled "cluster 223" tells you nothing. This stage is where the whole exercise becomes usable, and it's where I'd expected to lean hardest on the LLM. That turned out to be exactly backwards.

**Labels come from c-TF-IDF, not from the model.** Class-based TF-IDF asks: which terms are frequent in this cluster and rare in the rest of the corpus? That is *precisely* the question a label answers. It's deterministic, free, and runs over the whole corpus in seconds. Whereas an LLM asked to name a cluster has to be shown a sample — and a sample of a 3,000-document cluster is a guess about the other 2,990.

So the model gets demoted to a second pass that turns terms into readable English. It renames; it does not decide. Turn it off and you get raw terms: uglier, but never wrong about what the cluster contains.

### The bit that makes it work

A small model asked to name a cluster it does not understand **does not say so**. It produces a confident generic phrase. Over 1,460 clusters with `llama3.2:3b` the failure was completely consistent and always in the same direction:

```
ddr3, sata, fpga     ->  "Software Engineering Chat Log"
great, nah, mornin   ->  "Software engineering meeting discussion"
spacex, spcx, hedge  ->  "Software Engineering Chat Log"
```

None of those is a paraphrase. They're inventions, and each one is *worse* than the terms it replaced, because it reads as authoritative whilst saying nothing. The cluster about SpaceX and hedging is not a software engineering chat log.

The fix costs nothing and needs no second model: **keep the label only if at least one of the cluster's own top terms actually appears in it.** Long terms match on their first four characters at a word boundary, so "testing" grounds "tests" and "cognito" grounds "Cognito". Short terms have to match as whole words — I originally used a plain substring test and it cheerfully let `sata` ground "Satanic verses", which is exactly the kind of accidental agreement the check exists to refuse.

It also helps enormously to give the model an explicit way to decline. The prompt ends with *"if the keywords do not suggest a coherent topic, reply with exactly: UNCLEAR"*. Without that escape hatch it invents rather than abstains.

One more prompt-level lesson, this one embarrassing. An earlier version of my prompt opened with "extracted from a cluster of messages in a software engineering chat log" — and then I spent a while blaming the model for answering "Software Engineering Chat Log". I had handed it the generic answer and was cross that it used it.

## The output

The demo run takes about 45 seconds end to end. Here are the largest clusters it found across all eight books:

```
  1045  thou, unto, doth
   980  elizabeth, darcy, bennet
   860  Whaling and Whalemen
   672  boil, butter, put
   398  elizabeth, lizzy, dear
   340  Birds of South America and Patagonia
   305  Sherlock Holmes stories and characters
   275  Journey of Captain Nemo
   257  Moby Dick Crew and Captain
   228  Evolutionary Extinction of Species
```

Nobody told it that recipes or marine biology existed. `boil, butter, put` is Mrs Beeton; `Birds of South America and Patagonia` is Darwin on the Beagle; the whaling and the Austen households found their own corners.

The entries still showing raw terms are the ones where the grounding check rejected the model's suggestion, and the most interesting cluster in the list is the biggest one. `thou, unto, doth` is Marcus Aurelius — and note what it clustered on. Not subject matter. **Register.** The 1634 translation's archaic English pulled those passages together because of *how* they're written, not what they're about. That's a real and genuinely interesting property of the corpus, and a fluent invented label would have papered straight over it.

## Step 6: Actually look at it

A JSON file is a rubbish place to end. The run finishes by writing **one self-contained HTML file** — no server, no build step, no CDN, no network access of any kind. Open it from `file://`, or send it to someone.

![The corpusmap viewer open in a browser at a file:// URL, needing no server: a dark canvas of coloured point clouds, each cluster captioned with its name — Birds of South America, Cooking Recipes and Ingredients, Journey of Captain Nemo, Species Evolution and Extinction — with a hover card open over one sphere showing the passage it stands for, attributed to Meditations by Marcus Aurelius, and a panel reporting 9,071 points in 56 clusters](images/how-to-map-a-pile-of-text-nobody-has-read-viewer.png)

That constraint is the whole reason it's raw WebGL rather than three.js. Vendoring 600 KB of scene graph to draw a point cloud with fixed positions and no scene to graph is a poor trade. It's one buffer and one draw call, and the spheres are *imposters*: flat points whose surface normal is reconstructed in the fragment shader, so they light like balls without a single instanced mesh.

Hover any sphere and you get the passage it stands for:

> *"In this same New Bedford there stands a Whaleman's Chapel, and few are the moody fishermen, shortly bound for the Indian Ocean or Pacific, who fail to make a Sunday visit to the spot."*
> **Moby Dick** · Herman Melville · CHAPTER 7. The Chapel.
> Whaling and its Culture · 763 points

That hover card is the feature I'd fight to keep. "What *is* that dot" is the first question anyone asks of a point cloud and the one thing a scatter plot fundamentally cannot answer — and the moment you can read the actual text, you stop trusting the map on faith and start checking it. Which, as you'll see in a moment, matters.

Whatever you pass to `--meta-cols` rides along. For the demo corpus that's `title,author,chapter`. Gutenberg plain text has no page numbers — pagination belongs to a printed edition and doesn't survive the conversion — so the chapter heading is the only structural landmark on offer, and it turns out to be present for 97–100% of passages in every book.

A few things that took a while to get right:

- **Cluster names are DOM elements, not WebGL text.** Text in a point cloud needs a font atlas, per-glyph quads, and a shader fighting the depth buffer. An absolutely-positioned `div` needs a projected centroid and nothing else, and it stays crisp at every zoom because it's never rasterised into a texture.
- **Labels are placed by greedy collision rejection, largest cluster first.** Without it the dense middle is an unreadable pile of overlapping text, hiding exactly the structure the labels exist to explain. Zoom in and smaller topics appear as the centroids separate.
- **Clicking dims the other clusters rather than hiding them.** A cluster floating in empty space has lost the context that made a map worth drawing.
- **Coordinates are quantised to int16** before being inlined. UMAP output is a *relative* layout with no meaningful units, so 1/65,536 of the extent is far below what you or the renderer can resolve — and it takes a node from ~40 bytes of JSON to 12 bytes of binary.

The demo map is 4.8 MB, almost all of it hover-card text, and you can [download it from the release](https://github.com/nakomis/corpusmap/releases/latest) and open it without installing anything at all.

## The bit where I embarrassed myself

I wrote the first version of this post with a takeaway bullet that read *"look at your data before you trust your pipeline"*. Then I published a dataset containing Roget's Thesaurus, labelled as Mrs Beeton's *Book of Household Management*.

![A leather-bound Victorian cookery book and a thesaurus standing side by side on a dark shelf, so alike in their bindings that only the lettering tells them apart](images/how-to-map-a-pile-of-text-nobody-has-read-4.png)

Project Gutenberg ebook 10681 is the thesaurus. Mrs Beeton is 10136 — which I had confidently assigned to *The Voyage of the Beagle*, which is actually 944 and wasn't in the corpus at all. Two ids wrong, in a list of eight I'd typed from memory.

Here's the part worth your attention: **nothing looked broken.** The pipeline ran clean. The clusters were tight. I read the output, saw recipes separating from marine biology, and wrote "genuinely good separation" — because there *were* recipes, in a facet labelled `science-travel`, and there *was* a well-formed cluster called `lat, phr, est` sitting in `reference-cookery` which I skimmed straight past as harmless noise. It wasn't noise. It was Latin abbreviations, from a thesaurus, filed under cookery.

A wrong id is invisible in the output. It doesn't crash, it doesn't warn, it doesn't produce anything that looks like an error. It just quietly makes your dataset a lie, and every downstream number correct-but-meaningless.

Two fixes, both cheap:

The builder now asserts that each download's `Title:` **and** `Author:` lines match what the table claims, and aborts if not. The author check matters as much as the title, because titles aren't unique — "Meditations" is Marcus Aurelius here and Descartes elsewhere, and a title-only check would have waved either through.

And the thesaurus is gone rather than recategorised, because it was the wrong *shape* of text regardless of the label. It's word lists, not prose. There's no topic in a thesaurus entry for an embedding to find, which is precisely why its clusters came out as strings of abbreviations.

I'd caught this in about ten seconds if the viewer had existed when I built the dataset. That's not a coincidence — it's the argument for step 6.

## The bug that only showed up because two of us ran it

Late on, I compared my editor's screenshot against my own run and the numbers disagreed. Same corpus, same seed, same code: 51 clusters and 9,553 documents kept on one machine, 53 and 8,974 on mine.

My first assumption was library versions. It wasn't. Re-running on *my own* machine, hours later, reproduced *their* numbers rather than my earlier ones — same database, nothing touched in between.

The cause is one missing clause:

```sql
SELECT id, embedding FROM documents WHERE embedding IS NOT NULL
```

No `ORDER BY`. Postgres is free to return rows in physical heap order, and that is neither id order nor stable — `corpusmap embed` runs an `UPDATE` against every row, which rewrites tuples and reshuffles the heap, and autovacuum can move them again later. **UMAP and HDBSCAN both depend on the order their input arrives in**, so the fixed `random_state` I was so pleased about was guaranteeing nothing at all. The seed pins the algorithm's randomness; it does not pin your data's arrival order.

Adding `ORDER BY id` makes consecutive runs byte-identical, and changes the answer again — 56 clusters, 9,071 kept — because the ordered arrangement is simply a third, and now stable, result.

What makes this worth your attention isn't the bug, it's how it presented. There was no error, no warning, no anomaly. Every run looked healthy. Three different maps, each internally consistent and perfectly plausible, and the only reason I found out is that two people ran the same thing and bothered to compare. If you take one habit from this post, make it that one: **a result you have only produced once is not a result you have checked.**

## The thing that will actually cost you time

On my real corpus: 520 seconds of mathematics, and **seven to eleven hours** of a 3B model writing five-word labels, one cluster at a time, 1,634 times.

Labelling is 98% of the runtime. Every instinct I had about making this faster — a GPU, cuML, a bigger box — optimises the other 1.3%. I costed the GPU option properly and it came out both slower *and* dearer, because a twenty-second computation sits behind a three-and-a-half-minute pull of a 9 GB RAPIDS image.

So `corpusmap` attacks the part that matters, cheapest first:

1. **Cache by term set.** Facets overlap heavily — the `all` view contains every point in all the others — so the same topic gets named repeatedly within a single run. Keying a dict on the term tuple makes every repeat free. In the demo run, 145 distinct term sets covered 145 clusters across 8 facets.
2. **`--llm-min-cluster-size`.** Half of all clusters sit near the size floor and cost exactly as much to name as a 1,100-point one. The free c-TF-IDF fallback is fine for those.
3. **`--llm-workers`.** The calls are independent. Making them serially was leaving a straight multiple on the table.

## One last trap: facets don't compose

`--facet-col category` fits a **separate projection for every value** of that column. That's not tidiness, it's necessity, and it's the mistake I'd most expect a reader to make.

UMAP fits its layout to whichever points are present. So you cannot fit the whole corpus once and then hide dots client-side to get a per-category view — you'd be looking at the full corpus's geometry with holes punched in it, which is a different picture and a misleading one. Each facet needs its own fit.

What you *can* do, and should, is fetch the vectors once and subset them in memory. The fetch is the expensive part and the filtering is free. (Fetching is slower than you'd think, incidentally: pgvector serialises vectors as *text* on the wire, so a 768-dimension vector arrives as roughly 10 KB of ASCII that then has to be parsed. Installing the optional `pgvector` Python package registers a binary adaptor and skips the parsing entirely — hence the `[fast]` extra in the install command.)

## What I'd tell someone starting this tomorrow

- **The mathematics is not the hard part, and it is not the slow part.** Naming things is. Budget accordingly, and don't buy a GPU for the 1.3%.
- **c-TF-IDF beats an LLM at deciding what a cluster is about**, because it can read all of it and the model can only read a sample. Use the model to write English, not to make the judgement.
- **A small model has no way of telling you it doesn't know.** It will hand you fluent, authoritative, entirely invented labels. Making it ground every answer in evidence you already have costs about ten lines and is the difference between output you can use and output you can't.
- **Half your corpus probably has no topic**, and a tool that admits this is more useful than one that forces every document into a bucket.
- **Cluster membership is meaningful; the distance between clusters is not.** Do not measure across a UMAP plot, and be sceptical of anyone who does.
- **A fixed seed is not reproducibility.** Anything upstream that can reorder your data — an unordered SQL query, a directory listing, a set iteration — will move the output whilst every run still looks healthy.
- **Look at your data before you trust your pipeline.** Mine worked perfectly and produced a beautiful, coherent, completely useless cluster about the Project Gutenberg Literary Archive Foundation — and then, a fortnight's worth of confidence later, an equally beautiful set of clusters describing a thesaurus I believed was a Victorian cookbook. A clean run is not evidence of a correct input.
- **Build the thing that lets you look.** Every hour spent on the viewer paid for itself the first time I hovered a sphere and read something that had no business being there.

The tool is [on GitHub under CC0](https://github.com/nakomis/corpusmap) — do whatever you like with it. The [demo corpus](https://github.com/nakomis/corpusmap/releases/latest) is a `pg_dump` you can restore in one command, so nothing in this post has to be taken on trust.

## A last word from the corpus

Hovering spheres at random in the finished map, I landed on this — one passage of *Meditations*, sitting in the cluster whose only label the model would accept was `thou, unto, doth`:

> XX. If anybody shall reprove me, and shall make it apparent unto me, that in any either opinion or action of mine I do err, I will most gladly retract. For it is the truth that I seek after, by which I am sure that never any man was hurt; and as sure, that he is hurt that continueth in any error, or ignorance whatsoever.

Marcus Aurelius, roughly 180 AD, in Meric Casaubon's 1634 English. Which is a better summary of the last two sections than anything I wrote: a thesaurus filed as a cookbook and a map that quietly rearranged itself both survived because nobody had yet shown me I was in error.

The map is worth building for the clusters. It turned out to be worth building for that, too.


---

*Martin Harris connects cloud infrastructure to the real world. If you've ever built a machine to tell you what you had written, and it calmly reported that you had published a thesaurus as a Victorian cookbook, you're in the right place.*


{{donate}}