Skip to content

PATTERN Cited by 1 source

Byte-space arithmetic fold

Intent

Perform Unicode case-folding (or similar character transformations) directly on the UTF-8 byte representation without decoding to a codepoint and re-encoding — turning the fold into a single integer addition on the raw bytes read as a little-endian u32.

Mechanism

let word = u32::from_le_bytes(next_four_bytes) & length_mask;
let folded = word.wrapping_add(BYTE_DELTA[i]);  // the fold as one add
write_u32_le(dst, folded);
dst += utf8_len(folded_lead_byte);

Each fold run has a precomputed BYTE_DELTA constant — the difference between the source character's UTF-8 byte representation and the folded character's, both read as little-endian u32. Adding this delta to the source bytes produces the correct folded bytes directly.

Why it's fast

Every other Unicode case-folder inspected (ICU, Go's unicode, Rust's regex, CPython, glibc) follows the path: decode UTF-8 → lookup codepoint → apply fold → re-encode UTF-8. This pattern skips both the decode and the encode, operating entirely in byte space. The fold becomes a single wrapping_add — no conditional per-byte logic, no branch on output length.

Constraint

Requires well-formed, shortest-form UTF-8 input. The byte-delta arithmetic only produces correct results when the source encoding is canonical (minimal bytes per codepoint). Overlong encodings would break the length_mask and delta arithmetic. In Rust this is guaranteed by &str/String types; callers with raw bytes must validate first.

Handling length changes

Most folds preserve UTF-8 byte length. Two Unicode outliers grow (U+023A: 2→3 bytes, U+023E: 2→3 bytes). The pattern handles this naturally: write_u32_le always writes 4 bytes, but the cursor advances by only the folded character's length (1–4). The allocation reserves 1.5× input size (+4 bytes headroom) to accommodate growth without capacity checks.

Novelty claim

The GitHub team believes this byte-space arithmetic approach is genuinely new — no other production folder they examined operates in byte space. It's why the casefold crate can outrun even a pre-computed HashMap<u32, u32> lookup — the hash map still has to decode its key and encode its result.

(Source: sources/2026-07-31-github-dont-stop-early-case-folding-source-code-at-memory-speed)

Seen in

Last updated · 607 distilled / 1,851 read