30 - Moving beyond the wall

Concept node: see the DAG and glossary entry 30.

At 100 million creatures with 24 bytes of hot data each, the working set is 2.4 GB. At a billion, 24 GB. Most desktops have 16-64 GB of RAM. The simulator can no longer hold its world and its history and the OS and whatever else and operate at speed.

The fix is streaming: only the relevant slice of the world is in memory at any one time; the rest lives on disk and is read on demand.

The shape:

#![allow(unused)]
fn main() {
struct StreamingWorld {
    in_memory: Window,    // a small contiguous range of recent state
    disk:      Archive,   // the rest, append-only on disk
}
}

A window of recent state lives in memory, indexed for cheap query. Older state lives on disk in append-only chunks; it is read into the window when a query needs it.

This pattern shows up wherever this scale matters:

• Time-series databases (Prometheus, InfluxDB): recent metrics in RAM; older series compressed and disk-resident.
• Game replay systems: the last 30 seconds replayable from a memory ring; the full match streamed from a server.
• Event-sourced systems: recent state cached; the full event log on disk; replay reconstructs.
• Database write-ahead logs: append to log; flush to data files; the data files become disk-resident; recent log + memory hold the active set.

For the simulator, streaming entails three architectural shifts:

The log is the canonical state. The world’s tables are derivable from the log. If the log is complete and durable, every other in-memory representation is reconstructible. This is the structural framing of §37 - The log is the world: the log is not a record of state, it is the state.

Persistence is serialisation of tables. A snapshot is the world’s current SoA, written as a stream of (entity, key, value) triples - the same shape it has in memory. Recovery is reading the triples back. There is no separate domain model; serialisation is transposition, not translation. This is §36.

Storage is a cost like any other. Reading from disk costs bandwidth and IOPS, just as reading from RAM costs cache-line loads. Storage systems with bandwidth (bytes per second) and IOPS (operations per second) limits must be counted against the tick budget. SQLite, network sockets, distributed file systems - all are storage systems with their own cost profiles. This is §38.

Cleanup amortises the write cost. The cleanup system you built in §22 already batches in-memory mutations to avoid mid-tick races. At streaming scale, the same pattern earns its keep again, for a second reason: it batches disk writes. Without batching, 10 000 individual mutations per tick would mean 10 000 disk writes - at 100 µs per write, a full second of I/O per tick, far over budget. With cleanup, those 10 000 mutations become one durable batch per tick: a handful of disk pages flushed sequentially to the log. One syscall, one trip through the block layer, one (or a few) DMA transfers - versus 10 000 of each. The cost is amortised across the batch, not paid per row. The mechanics - page cache, vectored I/O, fsync semantics - belong to §38; the gradient is what matters here. The architecture you assembled in §22 was already the streaming architecture in miniature; this section just lets you spell it out at scale.

The simulator at streaming scale is no longer a process running in memory; it is a pipeline between a memory window and a durable log, with the systems running on whatever slice of the world is currently mounted. Every read might fault to disk; every write is buffered into the next cleanup’s batch.

The transition from in-memory to streaming is the largest architectural shift in the book. Below this wall, the simulator is a single-process program with its working state in RAM. Above it, the simulator is closer to a database with its working state on disk and a small in-memory hot path. The techniques are different; the discipline is the same - layout, working set, ownership, determinism - applied at a different scale.

This wall is where most projects either re-architect or quietly accept slower-than-target performance. The book points at the wall and names the techniques; it does not pretend the techniques are free.

Exercises

1. Compute your streaming threshold. Estimate your simulator’s per-creature footprint at full SoA. Divide your machine’s RAM (the half you can spare for the simulator) by that footprint. The result is roughly the N at which the simulator hits the streaming wall.
2. Predict the cost. A disk read is ~100 µs (NVMe SSD), ~200-500 µs (SATA SSD), or ~10 ms (spinning disk). At a 33 ms tick budget, how many disk reads can a tick afford? How many might a system want to make?
3. Snapshot a small world. Write a function that serialises your simulator’s current state to a single file (one file, no schema gymnastics, just write the columns). Read it back into a fresh world. Confirm the simulator continues running indistinguishably.
4. A windowed log. Implement an append-only log with a fixed in-memory window. Older entries go to disk; new entries always go to memory. Verify queries inside the window are fast; queries outside the window pay the disk cost.
5. Log-as-world. With the windowed log from exercise 4, reconstruct creature state at an earlier tick by replaying the log over the most recent snapshot whose tick is ≤ the requested one. Compare query speed to the in-memory case.
6. (stretch) Document your bound. Write down, for your simulator, the largest N you can run while staying inside a 33 ms tick budget. Include footprint, cache regime, and any disk-bound cost. Above this N, the simulator needs the streaming architecture.

Reference notes in 30_streaming_wall_solutions.md.

What’s next

You have closed Scale. The next phase is Concurrency, starting with §31 - Disjoint write-sets parallelize freely. The simulator is about to start running on more than one thread.