---
title: "The Embeddings File That Was Going to Eat My Lambda"
date: "2026-05-15"
publish_date: "2026-08-15"
excerpt: "The blog's semantic search worked fine at seven posts. The maths said it would stop working somewhere around 150. Here's what I changed before it got there."
tags: ["aws", "lambda", "dynamodb", "rag", "python", "embeddings", "serverless"]
author: "Martin Harris"
canonical: "https://blog.nakomis.com/2026-05-15-scaling-rag-binary-embeddings-dynamodb"
approved: true
---

# The Embeddings File That Was Going to Eat My Lambda

![A wave of folded spreadsheet paper looming over a small server shed while a figure calmly folds its edge into an origami crane](./images/scaling-rag-binary-embeddings-dynamodb-header.png)

[blog.nakom.is](https://blog.nakom.is) has a chat assistant and a semantic search UI. They're powered by a RAG pipeline: posts are chunked, embedded, and stored in S3; the Lambda loads everything at cold start, runs cosine similarity against the user's query, and returns the best matches.

![A tiny serverless function box straining under an enormous swollen translucent data blob about to crush it, conceptual, dramatic cold blue lighting](images/scaling-rag-binary-embeddings-dynamodb-1.png)

It worked. Then I looked at the growth curve.

Seven posts, 162 chunks, a `blog-embeddings.json` of 3,568 KB. Extrapolating linearly: around 150 posts the cold-start download time becomes actively annoying. Around 400 posts, a 256 MB Lambda runs out of memory before the first query is answered.

I had time to fix it before it became urgent, which is the best time to fix anything.

## What the original code was doing

The first-generation `ingest-blog.py` was simple and honest about it. It split each post into paragraphs by hand — track the current heading, flush on blank lines, skip code fences, merge short paragraphs upward into the next — and stored every chunk as a full JSON object:

```python
records.append({
    "id":          f"{slug}:{i}",
    "post_slug":   slug,
    "post_title":  title,
    "post_date":   post_date,
    "post_url":    post_url,
    "post_tags":   tags,
    "heading":     heading,
    "text":        para_text,
    "embedding":   vector,      # a list of 1024 floats
})
```

`embedding` is a list of 1,024 floats. JSON-serialised, each float takes roughly 17 bytes — `0.0123456789012345` plus a comma. Multiply by 1,024 dimensions and each chunk carries about 17 KB of embedding data, written out as decimal text.

The Lambda loaded the whole file, parsed it with `JSON.parse`, and kept the result in memory for the lifetime of the container. That works fine at small scale. It's also how you end up with a 250 MB cold-start download at 500 posts.

## Switching chunkers

Before dealing with the storage format, I looked at the chunker itself. The hand-rolled version had a gap: it skipped code blocks entirely, on the theory that raw code produces poor embedding signal. That's true in isolation. The problem is that code blocks in a post about CloudFront CDK stacks or ESP32 firmware are often the point — the surrounding prose only makes sense with them, and they only make sense with the surrounding prose.

The fix: use LangChain's `MarkdownTextSplitter`, which is `RecursiveCharacterTextSplitter.from_language(Language.MARKDOWN)` under the hood. It knows about Markdown structure — headings, code fences, paragraphs — and splits on them in priority order, falling back to sentences then words. More importantly, it supports overlap: consecutive chunks share 80 characters across their boundary, so a concept introduced at the end of one chunk also appears at the start of the next.

The hand-rolled version had no overlap. A sentence that happened to fall on a paragraph boundary could end up in neither chunk's embedding neighbourhood.

```python
from langchain_text_splitters import MarkdownTextSplitter

CHUNK_SIZE    = 700  # characters
CHUNK_OVERLAP = 80

_splitter = MarkdownTextSplitter(chunk_size=CHUNK_SIZE, chunk_overlap=CHUNK_OVERLAP)

def chunk_body(body: str) -> list[str]:
    return _splitter.split_text(body)
```

Forty lines of hand-rolled logic replaced by four. The 80-character overlap was the thing the custom code had been missing all along.

![A towering stack of paper compressing down into a single slim glowing crystal sliver](images/scaling-rag-binary-embeddings-dynamodb-2.png)

## Binary-encoding the embeddings

A 1,024-dimensional float32 embedding is exactly 4,096 bytes of binary data — 1,024 × 4 bytes per `float`. JSON represents the same data as roughly 17,000 bytes of decimal text.

The fix is to pack the floats as binary and base64-encode the result:

```python
import base64
import struct

def encode_embedding(vector: list[float]) -> str:
    """Pack a float list as a base64-encoded Float32 binary string."""
    return base64.b64encode(struct.pack(f"{len(vector)}f", *vector)).decode()
```

`struct.pack` with the `f` format code writes IEEE 754 single-precision floats. Base64 encoding adds ~33% overhead, so 4,096 bytes of binary becomes ~5,500 bytes of base64. Compare that to ~17,000 bytes of JSON decimal text — a 68% reduction per embedding.

On the Lambda side, decoding is one line:

```typescript
function decodeEmbedding(b64: string): Float32Array {
    const buf = Buffer.from(b64, 'base64');
    return new Float32Array(buf.buffer, buf.byteOffset, buf.byteLength / 4);
}
```

`Float32Array` gives you the same typed array you'd get from the Bedrock response natively. No precision loss — base64 of binary float32 is a lossless round-trip.

For 7 posts: 3,568 KB of JSON decimal arrays becomes 893 KB of base64-encoded binary. 75% smaller.

## Splitting the data across S3 and DynamoDB

The binary encoding fixes the per-embedding size. It doesn't fix the core architecture problem: the Lambda downloads everything at cold start, including all the chunk text.

The chunk text is only needed after the cosine scan. At cold start you need the embeddings; at query time you need the text for the top K results — never all of it at once.

The new design splits the data across two stores:

**S3** (`blog-embeddings.json`, loaded at cold start): chunk ID, binary embedding, post slug, and tags. Nothing else.

```python
new_s3_records.append({
    "id":        chunk_id,      # e.g. "my-post:3"
    "post_slug": slug,
    "post_tags": tags,
    "embedding": encode_embedding(vector),
})
```

**DynamoDB** (`blog-chunks` table, keyed on chunk ID): all the text and metadata — title, date, URL, excerpt, heading, the chunk text itself.

```python
new_ddb_records.append({
    "id":           chunk_id,
    "post_slug":    slug,
    "post_title":   title,
    "post_date":    post_date,
    "post_url":     post_url,
    "post_excerpt": post_excerpt,
    "heading":      heading,
    "text":         chunk_text,
})
```

At query time: load embeddings from S3 at cold start, run cosine scan to get the top K chunk IDs, then fetch those specific records from DynamoDB with a single `BatchGetItem` call:

```typescript
const response = await ddb.send(new BatchGetItemCommand({
    RequestItems: {
        [table]: {
            Keys: ids.map(id => ({ id: { S: id } })),
        },
    },
}));
```

`BatchGetItem` fetches up to 100 items in one request. For TOP_K = 4, that's one DynamoDB call at query time, adding around 3–5 ms to the warm search latency. Cold start time drops dramatically because the S3 file carries only IDs and embeddings — not the chunk text.

## The numbers

At 7 posts, comparing old to new:

| | Old | New |
|---|---|---|
| S3 file size | 3,568 KB | 893 KB |
| Cold start download | ~35 ms | ~9 ms |
| Warm search | ~1 ms | ~4–6 ms |
| DynamoDB cost | — | ~$0/month |

The slight warm-search regression — text retrieval used to be free because everything was in memory — is completely offset by the cold-start improvement, and the gap widens with every post added.

The scaling projection that motivated this whole exercise:

| Posts | Old S3 file | New S3 file |
|---|---|---|
| 7 | 3.5 MB | 0.9 MB |
| 50 | ~25 MB | ~6 MB |
| 200 | ~100 MB | ~25 MB |
| 500 | ~255 MB | ~63 MB |

At 500 posts, the old architecture runs a 256 MB Lambda out of memory. The new architecture is at 63 MB — fine on a 128 MB Lambda with headroom to spare.

## The alternatives I didn't use

OpenSearch Serverless was the obvious AWS-native answer. It has proper approximate nearest-neighbour search, scales on demand, and integrates cleanly with the rest of the stack. The minimum cost is $700/month — four OCUs at $0.24/OCU-hour, billed continuously whether anyone queries it or not. Not suitable for a personal blog.

Pinecone Serverless is the other obvious answer. Pay-per-query, generous free tier. The catch: the free tier is US-East-1 only. My Lambda runs in eu-west-2. Cross-region Bedrock embedding calls already add latency; adding cross-region vector queries on top would make every search noticeably slower. Standard tier starts at around $50/month minimum.

The S3 + DynamoDB combination costs effectively nothing. DynamoDB on-demand at $0.125 per million reads: at 10,000 searches per month with 4 `BatchGetItem` reads each, that's about $0.005/month. Storage for a few thousand chunks is cents per year. Both services are already in the stack. There's no new infrastructure to manage.

The linear cosine scan is the part that looks obviously wrong from a computer science perspective — O(n) over all embeddings for every query, rather than approximate nearest-neighbour search. In practice it doesn't matter yet. Even at 10,000 chunks the scan takes well under 10 ms. ANN search exists to avoid scanning millions of vectors; it's overkill at the scale of a personal blog.

## The incremental ingest

One thing the old script didn't have: any concept of whether a post had changed since last time.

The new version maintains a SHA-256 manifest in S3 (`ingest-manifest.json`) that records the hash of each post at last ingest. On each run, only posts whose content has changed — or that are new — get re-embedded. Unchanged posts are skipped; their existing DynamoDB records and S3 embedding entries are preserved as-is.

The manifest is just a dict of slug → hash, written to S3 after each successful run. For a blog that adds a post every few weeks and rarely edits old ones, this means most ingest runs do nothing, which is the correct amount of work.

## What I actually learned

The paragraph splitter not having overlap was a design decision that seemed reasonable and was quietly wrong. Overlap is the thing that makes retrieval robust near chunk boundaries — without it, you're hoping that important context never falls at the end of a paragraph. It always does eventually.

The binary embedding encoding is a straightforward engineering trade-off that should probably be the default: `struct.pack` on the write side, `Float32Array` on the read side, 75% smaller files with no loss of precision. The only reason to use JSON float arrays is that they're readable in a text editor, which stops mattering the moment you have more than a handful of them.

The S3 + DynamoDB split isn't a new idea — it's just the standard pattern for separating hot data (embeddings, fetched on every cold start) from cold data (text, fetched only for the top K results). It took looking at the growth curve to make the separation feel necessary rather than premature.

---

*Martin Harris writes about connecting real hardware to abstract cloud infrastructure. If you've ever soldered something to an API Gateway, you're in the right place.*

{{donate}}