From Compiler Bug to Working Exploit

In previous posts, we covered what the compiler does when you break the rules: null checks compiled away, overflow checks removed, strict aliasing producing wrong data, infinite loops from missing volatile.

You might have read those and thought: “Okay, the compiler removes some checks. But how does that actually become an exploit?”

That is exactly the gap this post closes. We take three compiler-assisted bugs from the previous series and walk through how an attacker turns each one into a working attack, step by step.

From Null Check Removal to Kernel Privilege Escalation

The bug (recap): CVE-2009-1897. The Linux kernel dereferenced a pointer before checking if it was null. The compiler saw the prior dereference, concluded the pointer must be non-null, and removed the null check. do_privileged_op() ran unconditionally.

How the attacker exploited it:

Step 1 — Map memory at address zero. On Linux at the time, an unprivileged process could call mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_FIXED|MAP_ANONYMOUS, -1, 0) to map a page at virtual address zero. This is normally reserved, null pointer dereferences are supposed to crash. With this call, they do not.

Step 2 — Place a fake kernel object at address zero. The attacker fills the mapped page with a carefully crafted data structure, specifically something that looks like a valid kernel object when the dereferenced null pointer is used.

Step 3 — Trigger the vulnerable function with a null pointer. The kernel dereferences address zero, reading the attacker’s fake object. The null check that was supposed to stop execution is gone. do_privileged_op() runs with the attacker’s data as input.

Step 4 — Escalate privileges. The attacker overwrites kernel security structures, credentials, capabilities, security policy, granting root access.

What the attacker needed: one mmap call, one crafted data structure, and the knowledge that the null check was gone.

From Overflow Check Removal to Heap Overflow

The bug (recap): A signed integer is used for a size calculation. The compiler removes the overflow check because signed overflow is undefined behavior. malloc receives a tiny (wrapped) value. The buffer is far too small.

How the attacker exploited it:

Consider a real-world pattern from network protocol parsers:

int parse_message(uint8_t *data, int count, int item_size) {
    int total = count * item_size;     // signed - UB if overflow
    if (total < 0) return ERROR;       // compiler removes this
    uint8_t *buf = malloc(total);      // gets wrapped tiny value
    memcpy(buf, data, count * item_size); // copies full size - overflow
    process(buf);
}

The attacker sends a packet with count = 0x10000 and item_size = 0x10000. The multiplication produces 0x100000000, which wraps to 0 in a 32-bit signed integer. The overflow check is gone. malloc(0) returns a valid (but zero-sized) pointer. memcpy copies 0x100000000 bytes starting from buf, writing far beyond the allocation, across the entire heap.

Step 1 — Identify the overflow. The attacker sends increasing values and observes behavior changes, a sign the allocation is unexpectedly small.

Step 2 — Heap grooming. The attacker shapes the heap, placing a target object adjacent to where buf will land.

Step 3 — Trigger the overflow. memcpy writes past buf and corrupts the adjacent object.

Step 4 — Trigger the corrupted path. A follow-up request exercises the corrupted object, calling through an overwritten function pointer.

What the attacker needed: knowledge of the integer types and a way to observe allocation behavior. The compiler removed the only check that would have stopped step 3.

From Strict Aliasing to Silent Data Corruption

The bug (recap): A raw byte buffer is cast to a struct pointer. The compiler, assuming the two pointer types cannot alias, caches a field read from before the function call. The bounds check passes on the stale value. The handler is called with the real (attacker-controlled) value.

How the attacker exploited it:

This is subtler — the attacker does not need to corrupt memory. They just need the bounds check to pass while the handler receives a value larger than expected.

void process_packet(uint8_t *raw) {
    Packet *pkt = (Packet *)raw;      // strict aliasing violation
    uint32_t len = pkt->length;       // compiler may use cached value
    if (len > MAX_SIZE) return;       // check passes on cached (small) value
    handle_data(pkt->data, len);      // called with real (large) value from memory
}

Step 1 — Establish a baseline. The attacker sends packets with small, valid lengths until the compiler caches one in a register.

Step 2 — Send the malicious packet. length = 0xFFFFFFFF in the raw bytes. The compiler reads len from the cached register, bounds check passes. handle_data reads the real value from memory: 0xFFFFFFFF. It processes 4GB of data.

Step 3 — OOB read. handle_data reads far beyond the packet buffer, leaking cryptographic keys or session tokens from adjacent heap memory.

Structurally identical to Heartbleed, a bounds check that passes, followed by a read using the real, unchecked value.

What the attacker needed: a way to send crafted lengths and observe the response. The compiler’s aliasing assumptions did the rest.

The Common Exploitation Structure

Across all three cases, the exploitation structure is identical:

Phase 1 — Reconnaissance: identify the missing check. Send boundary values, observe behavior.

Phase 2 — Primitive acquisition: use the bypassed check to acquire a primitive, heap overflow for corruption, OOB read for information disclosure.

Phase 3 — Impact: privilege escalation, code execution, credential theft.

Compiler-assisted bugs make Phase 1 harder (check visible in source, absent in binary) and Phase 2 more reliable (the overflow happens consistently, not just on specific inputs).

Why Source-Level Review Is Not Enough

A code reviewer looking at each of these functions would see a null check, an overflow check, a bounds check. They would mark the code as reviewed and move on.

The vulnerability does not exist in the source code. It exists in the compiled binary, after the optimizer has processed the UB assumptions.

This is why binary analysis and formal verification are not optional for high-assurance software. The source code and the binary are two different programs, and the security properties of one do not guarantee the security properties of the other.

Formal verification that accounts for actual C semantics, including what the compiler is permitted to do with undefined behavior, catches these cases before the binary is ever produced. The null check that the compiler will remove is flagged at analysis time. The overflow check that will be eliminated is identified before it disappears.

The attacker finds the binary. The formal analyzer finds the source. Only one of them can find it before shipping.