---
title: "Two HTTPS Calls, One Crash, and a Paused MQTT Connection"
date: "2026-05-15"
publish_date: "2026-07-31"
excerpt: "The first cat inference worked perfectly. The second one crashed the device. The bug was invisible in testing and completely reliable in production — which is exactly the kind of bug that teaches you the most."
tags: ["esp32", "embedded", "mqtt", "tls", "bootboots", "iot", "aws"]
author: "Martin Harris"
canonical: "https://blog.nakomis.com/2026-05-15-heap-contention-mqtt-https-esp32"
approved: true
---

# Two HTTPS Calls, One Crash, and a Paused MQTT Connection

![Two glowing data streams squeezing across a narrow bridge while a third waits frozen above a flickering circuit city](./images/heap-contention-mqtt-https-esp32-header.png)

BootBoots is the cat monitoring system I've been building — ESP32-CAM above the food bowls, a SageMaker model doing the inference, AWS API Gateway on the receiving end. When Boots (the black interloper, not welcome at the food station) shows up, the system fires a deterrent — a brief burst from a water sprayer mounted above the bowl. The short version: a microcontroller takes a photo, signs it with SigV4 (HTTPS to API Gateway with IAM auth), POSTs it, and acts on the result.

![Extreme macro of a tiny ESP32 circuit board at night, two glowing translucent memory blocks colliding and fragmenting](images/heap-contention-mqtt-https-esp32-1.png)

It worked. First inference: clean. Image uploaded, response received, deterrent decision made.

Second inference: the device crashed.

## The shape of the bug

Not a graceful error. Not an HTTP 500. A hard crash — a Guru Meditation Error — mid-upload, partway through the second TLS handshake. The mbedTLS allocator returned null, and the downstream null-dereference triggered a CPU exception (LoadProhibited), which is what the Guru Meditation actually reports. If you're searching your own serial output, look for the failed allocation in the mbedTLS layer followed by the exception fault — the two appear close together and together tell the full story.

The combination — works in testing, fails in production, second time only — is a signature. It points at one thing: something the lab setup didn't have that production did, holding onto a resource the second operation needed.

That something was the MQTT connection.

## What the heap was doing

The ESP32 has around 520KB of SRAM on the datasheet, but only roughly 320KB is DRAM accessible as heap, and after the WiFi stack loads, the practical ceiling is lower still. That sounds comfortable until you account for what's already living there: the WiFi stack, the MQTT client and its payload buffer (configured to 2048 bytes via `MQTT_MAX_PACKET_SIZE`), TLS session state for the IoT Core connection, whatever the application is doing in the main loop. By the time the device is up and running with a live MQTT connection to AWS IoT Core, the available heap is already smaller than it looks on paper.

More importantly, it's *fragmented*. Because the MQTT TLS allocations are long-lived and non-movable, they pin the heap — the largest free block is smaller than the total free figure suggests.

When an HTTPS connection opens, the mbedTLS stack needs to allocate its SSL buffers. On our build — Arduino-ESP32 core 2.x / ESP-IDF 4.4, PubSubClient 2.8, WiFiClientSecure with mbedTLS, `MBEDTLS_SSL_IN_CONTENT_LEN` and `MBEDTLS_SSL_OUT_CONTENT_LEN` both set to 8192 in sdkconfig — this comes to roughly 18KB of contiguous memory per connection, with the remainder beyond the two 8KB record buffers accounted for by record-expansion overhead and handshake buffers. The key word is contiguous. Fragmentation doesn't just reduce the total free memory; it carves it into pieces. The allocator might have 50KB free in aggregate but be unable to hand you 18KB as a single block because no individual gap is that large.

In the lab, running only the inference path, the heap was clean. One thing used SSL: the HTTPS connection itself. It allocated, ran, freed, and the heap recovered.

In production, the MQTT stack had been running continuously since boot. It maintains its own TLS session to AWS IoT Core — a `WiFiClientSecure` instance with the device certificate loaded, a `PubSubClient` on top, plus a 2KB payload buffer. Held open. Not released between inference calls. By the time the second HTTPS inference tried to allocate its SSL buffers, the heap was already carved up around the MQTT connection's memory, and `malloc` returned null.

The device didn't handle that gracefully. It crashed.

## Why the first call worked

This confused me for a moment. If the MQTT connection was always there, why did the first inference succeed?

Timing. At cold boot, the heap is clean. The MQTT connection establishes and settles. At that point there's one large occupant — the MQTT TLS session — but the remaining free space is still relatively contiguous. The first HTTPS inference gets in, allocates its buffers, runs, and finishes.

Then it frees those buffers. Now the heap has a hole where the HTTPS buffers were, flanked by the MQTT session's allocations that have no intention of moving. The largest free block after the first HTTPS free is smaller than at boot. By the second call, the allocator can't find enough contiguous space.

This is why `heap_caps_get_largest_free_block(MALLOC_CAP_8BIT)` matters more than total free heap: it tells you the number the TLS allocator actually cares about, and it trends downward over runtime in a way that the total free figure obscures.

## The fix

The fix is three lines wrapping every HTTPS inference operation in `CaptureController`:

```cpp
// Pause MQTT (full teardown) to free SSL memory before making HTTPS request
_awsAuth->pauseMqtt();

CatCamHttpClient httpClient;
response = httpClient.postImage(image, _apiHost, _apiPath, _awsAuth);

_awsAuth->resumeMqtt();
```

This pattern appears in every code path that uploads to AWS — the main inference call, training captures, and the quick PIR-triggered detection path. All three pause before the HTTPS upload and resume after.

The critical detail is what `pauseMqtt()` actually does. It forwards to `MqttService::pause()`, which performs a full teardown — not a loop pause or a keep-alive halt. The key deletions are:

```cpp
void MqttService::pause() {
    if (!_initialized) return;  // Idempotent: safe to call twice
    if (_client) {
        _client->disconnect();
        delete _client;
        _client = nullptr;
    }
    if (_wifiClient) {
        _wifiClient->stop();
        delete _wifiClient;
        _wifiClient = nullptr;
    }
    if (_responseSender) {
        delete _responseSender;
        _responseSender = nullptr;
    }
    _connected = false;
    _initialized = false;  // Will need to reinit on resume
    SDLogger::getInstance().infof("MQTT: Paused, free heap: %d, largest block: %d",
        ESP.getFreeHeap(),
        heap_caps_get_largest_free_block(MALLOC_CAP_8BIT));
}
```

The `WiFiClientSecure` instance — the one holding the TLS session — is explicitly deleted. The `PubSubClient` and the response sender go with it. The heap is genuinely recovered, not deferred. The early return on `!_initialized` makes the call idempotent, so calling `pause()` twice or `resume()` without a prior `pause()` is safe.

After a pause/resume cycle, the heap logs look like this:

```
MQTT: Paused,  free heap: ~187KB, largest block: ~62KB
MQTT: Resumed, free heap: ~164KB, largest block: ~41KB
```

The free heap drops on resume because MQTT is back — the HTTPS call happened in between, and the recovery was what mattered. The largest-block figure is what to watch: before pause it may be too small for a TLS allocation; after pause it opens up. The ~23KB recovered by teardown is larger than the 18KB SSL buffer figure because it includes the `WiFiClientSecure` object itself, the `PubSubClient`, the cert buffers, and the response sender on top of the mbedTLS record buffers.

`resume()` does the reverse: reconstructs all three objects from stored member variables (certificate pointers and endpoint config retained outside the deleted objects), then defers the actual TCP reconnection to the next `handle()` call. By the time `resumeMqtt()` returns, the objects exist but the TCP socket hasn't been opened yet. The reconnect happens in the background on the next main-loop tick.

Before calling `pause()`, all inflight QoS 1 publishes should be flushed — call `client.loop()` until the outbox is empty, or track your inflight count and wait for it to reach zero. All MQTT activity in BootBoots is confined to the main loop and quiesces naturally before teardown, so there's no risk of a callback firing while the pointers are null, but if your architecture is different, add the explicit drain step.

Because this is a full TCP teardown, the broker sees a genuine disconnect. To ensure commands published during the pause aren't dropped, we connect with `cleanSession=false` (a persistent session), so AWS IoT Core retains our subscriptions and queues QoS 1 messages while we're offline. We resubscribe on resume as a belt-and-braces measure, but the persistent session is what actually guarantees delivery. AWS IoT Core persistent session queues are bounded by both message count and total payload size, and messages expire after a configurable period — check the service limits documentation if your pause windows are longer or your command rate is higher than ours. The cat isn't getting away.

We also added backoff on the reconnect path so that repeated HTTPS failures don't cause rapid flapping against AWS IoT Core's connection-rate limits.

We also considered reducing mbedTLS buffer sizes, allocating TLS buffers from PSRAM, or reusing a single TLS client. On the ESP32-CAM, external PSRAM is typically accessible via `MALLOC_CAP_8BIT`, so mbedTLS allocations can land there — but the higher bus latency noticeably slows TLS handshakes, and the bus contention with the camera made it less attractive than simply freeing the MQTT session. The teardown-around-HTTPS was the simplest and most reliable option given our constraints. (ESP-IDF's mbedTLS memory buffer allocator, which lets you back TLS allocations from a static pool, is another path worth knowing about for deeper fragmentation control.)

## The class boundary

The MQTT connection lives in `MqttService`, accessed via `AWSAuth`. The comment in `AWSAuth.h` is explicit:

```cpp
// Set MQTT service - will be paused/resumed around SSL requests to free memory
void setMqttService(class MqttService* mqttService) { _mqttService = mqttService; }
```

And the comment in `getCredentialsWithRoleAlias` is equally blunt:

```cpp
// Note: caller is responsible for pausing/resuming MQTT around this call.
```

The credential refresh — itself an HTTPS call — also needs the heap headroom that MQTT was consuming. The pause wraps that too.

The design made this separation necessary. Authentication concerns live in `AWSAuth`. Inference logic lives in `CaptureController`. When inference needed to temporarily give back memory held by the auth layer's MQTT connection, the interface needed two methods: `pauseMqtt()` and `resumeMqtt()`. Just enough exposure of the auth layer's lifecycle to let inference cooperate around a constrained shared resource.

## Why "it worked in testing" is so dangerous here

Lab testing of embedded firmware is, by necessity, simplified. You test the happy path. You test components in isolation. Your integration tests start from a clean state: fresh boot, nothing else running, the specific function under test with nothing competing for resources.

Production doesn't look like that. Production is a device that's been running for hours. It has an MQTT connection that's been alive since boot. Its heap has been allocated and partially freed a dozen times. The allocator's internal state reflects all of that history.

The fragmentation bug was invisible in testing because testing never generated that history. You can't easily reproduce "what does the heap look like after two hours of production operation" — you have to run the real thing, observe what happens, and have enough instrumentation to catch the failure when it occurs.

The crash itself was the first signal. There was no heap instrumentation in place when it happened — the Guru Meditation Error was the diagnostic. After identifying the cause, I added heap logging (both total free and largest free block) around each pause and resume so subsequent runs made the recovery visible. More useful than the total free figure is `heap_caps_get_largest_free_block(MALLOC_CAP_8BIT)`: the largest contiguous free block, and the number that actually determines whether a TLS allocation will succeed. Log both at the start of every HTTPS call if you want to see the trajectory before it kills you. (ESP-IDF's heap tracing is also worth knowing about for deeper diagnosis.)

## The physical/abstract join

Every part of this bug lives at the collision between physical and abstract.

On the physical side: a chip with fixed silicon. A specific number of bytes of SRAM. An allocator that has to carve it up in real time, leaving scars with every allocation cycle.

On the abstract side: two cloud connections sharing that fixed silicon. MQTT to AWS IoT Core. HTTPS to API Gateway. Each demanding its own TLS session state, its own certificate buffers, its own heap headroom.

AWS IoT Core has no concept of a microcontroller's heap. API Gateway doesn't know it's competing with an MQTT client for contiguous 18KB blocks. The collision is entirely local, entirely physical, and entirely predictable once you know to look for it. The fix — fully disconnecting one cloud connection to make room for the other — is the embedded equivalent of swapping a process out to free RAM.

The cloud is patient. That patience on the backend is exactly what lets you be aggressive about freeing and recapturing RAM on the device.

## What I'd do differently

**Test with the full stack running from the start.** The temptation is to test components in isolation — faster iteration, cleaner failures. But for anything that touches shared memory on a constrained device, isolation is the enemy of realism. Run the full system the way production runs it, from the first integration test.

**Log the largest free block before the thing breaks.** `heap_caps_get_largest_free_block(MALLOC_CAP_8BIT)` tells you the number that matters for TLS allocations, not just total free heap. Log it alongside total free at the start of every HTTPS call. If it's trending down over time, you'll see the trajectory before it kills you.

**Make the resource sharing explicit, in both directions.** `pauseMqtt()`/`resumeMqtt()` works because the inference code knows enough to call it. That knowledge lives only in the developer's head and in a source code comment. A more defensive design would wrap the pair in an RAII guard — this is future work rather than what's currently implemented, but the pattern would look something like:

```cpp
struct ScopedMqttPause {
    AWSAuth* _auth;
    explicit ScopedMqttPause(AWSAuth* auth) : _auth(auth) {
        if (_auth) _auth->pauseMqtt();
    }
    ~ScopedMqttPause() {
        if (_auth) _auth->resumeMqtt();
    }
};

// Usage:
{
    ScopedMqttPause guard(_awsAuth);
    response = httpClient.postImage(image, _apiHost, _apiPath, _awsAuth);
}
```

Resume is then guaranteed even on error paths, rather than relying on every caller to remember. The current explicit calls work fine at our scale, but the footgun is real if the number of call sites grows.

---

*Martin Harris writes about connecting real hardware to abstract cloud infrastructure. If you've ever soldered something to an API Gateway, you're in the right place.*

{{donate}}