If registers are a carpenter’s workbench, cache memory is the shelf right behind them, close enough to reach without walking to the warehouse, but able to hold far more than the workbench itself. Cache memory is the single biggest reason modern CPUs can run at multi-gigahertz speeds without spending most of their time waiting on painfully slow main memory. This article walks through why cache exists, how the L1/L2/L3 hierarchy is structured, what makes caches fast, and how multiple CPU cores manage to keep their individual caches consistent with each other through cache coherency protocols.
The Problem Cache Solves: The Memory Wall
To understand cache, you have to understand the gap it’s bridging. Over the past several decades, CPU clock speeds and execution capability have improved dramatically faster than DRAM (main memory) latency has improved. A modern CPU core can execute an instruction in under a nanosecond, while a round trip to main memory can take on the order of 100 nanoseconds. If every single instruction had to wait on main memory, the CPU would spend the overwhelming majority of its time idle, a phenomenon often referred to as the “memory wall.”
Cache memory solves this by exploiting two well-documented patterns in how real programs access memory:
- Temporal locality: If a program accesses a particular memory address, it’s likely to access that same address again soon (think of a loop counter or a frequently called function).
- Spatial locality: If a program accesses a particular memory address, it’s likely to access nearby addresses soon after (think of iterating through an array).
Cache exploits both patterns by keeping recently used data, and the data physically near it, in small, extremely fast on-chip memory, dramatically reducing how often the CPU has to wait for slow main memory.
The Cache Hierarchy: L1, L2, and L3
Rather than one single cache, modern CPUs implement a multi-level hierarchy, trading off size against speed at each level.
| Level | Typical Size (per core, modern desktop/server) | Typical Latency | Shared or Private |
|---|---|---|---|
| L1 (split I-cache/D-cache) | 32-64 KB each | ~4-5 cycles | Private per core |
| L2 | 256 KB – 2 MB | ~10-20 cycles | Private per core (mostly) |
| L3 (Last Level Cache) | 8-64+ MB | ~30-50 cycles | Shared across all cores |
| Main Memory (DRAM) | Gigabytes | ~100-300+ cycles | Shared across entire system |
L1 cache sits closest to the core and is typically split into two separate caches: an L1 instruction cache (L1i) holding recently fetched instructions, and an L1 data cache (L1d) holding recently accessed data. Splitting them allows the CPU to fetch an instruction and access data simultaneously without contention, which matters enormously for pipelined execution. L1 is small, on the order of tens of kilobytes, because it must be extremely fast, and speed and size trade off directly against each other in circuit design.
L2 cache is larger and slightly slower, acting as a buffer between the very fast but tiny L1 and the shared, larger L3. On most modern designs, L2 is still private to each core (though a handful of designs have experimented with shared L2), and holds both instructions and data unified into a single cache rather than split.
L3 cache, often called the Last Level Cache (LLC), is shared across all the cores on a chip. Its large size, sometimes tens of megabytes on server chips, lets it hold enough data that useful information from one core’s working set can benefit other cores too, and it acts as the last line of defense before a memory access has to go all the way out to DRAM.
Beyond L3, some modern architectures have begun introducing an “L4” cache, either as a large on-package eDRAM chip or as part of newer 3D-stacked cache designs (AMD’s 3D V-Cache being a notable production example), further pushing back the point at which the CPU must reach out to slow main memory.
How Cache Actually Works: The Basics
A cache doesn’t store data indexed by arbitrary keys; it stores fixed-size chunks called cache lines (commonly 64 bytes on modern x86 and ARM processors), each tagged with the memory address it corresponds to. When the CPU needs to read from or write to memory, it first checks whether the relevant cache line is already present, a cache hit, or not, a cache miss.
On a cache hit, the data is returned immediately from the fast cache, no need to touch slower levels or main memory at all.
On a cache miss, the CPU must fetch the needed cache line from the next level down (L1 miss goes to L2, L2 miss goes to L3, L3 miss goes all the way to DRAM), and it typically fetches the entire 64-byte line, not just the single byte or word requested, because spatial locality suggests nearby bytes will likely be needed soon too.
This is also why cache misses are categorized into distinct types by computer architects: compulsory misses (the very first access to a block, unavoidable), capacity misses (the cache simply isn’t big enough to hold everything the program is actively using), and conflict misses (a cache placement policy limitation causes eviction even though the cache overall has room, which ties directly into cache mapping techniques covered in more detail elsewhere).
Inclusive vs. Exclusive Cache Hierarchies
Cache hierarchies can be designed as inclusive, where data present in L1 is guaranteed to also be present in L2 and L3 (simplifying coherency checks, at the cost of some wasted capacity), or exclusive, where a cache line exists at only one level at a time (maximizing effective total capacity, at the cost of more complex movement between levels when data is evicted or promoted). Some designs use a hybrid “non-inclusive, non-exclusive” policy that doesn’t strictly guarantee either property. AMD and Intel have historically made different choices here across product generations, and this design decision has measurable effects on effective cache capacity and coherency traffic.
Why Cache Coherency Becomes a Problem
Everything above works fine for a single core, but modern CPUs have many cores, and each core typically has its own private L1 and L2 cache. This creates an obvious problem: if Core A reads a memory address into its L1 cache, and then Core B writes a new value to that same memory address, Core A’s cached copy is now stale. If Core A keeps using its outdated cached value without knowing it’s stale, the program’s behavior becomes silently incorrect. This is the cache coherency problem, and it must be solved in hardware, transparently to software, or multithreaded programming would become vastly harder than it already is.
Cache coherency ensures that all cores in a system observe a consistent, correctly ordered view of memory, even though each core is working from its own private, potentially stale-looking cache. The two dominant families of techniques used to enforce this are snooping protocols and directory-based protocols, and the specific state machine governing individual cache lines is most commonly MESI or one of its variants.
MESI: The Foundational Coherence Protocol
MESI stands for Modified, Exclusive, Shared, Invalid, the four states a cache line can be in on any given core.
| State | Meaning |
|---|---|
| Modified (M) | This core has the only copy, and it has been written to (dirty); it differs from main memory |
| Exclusive (E) | This core has the only copy, and it matches main memory (clean, unshared) |
| Shared (S) | This cache line may also exist in other cores’ caches, and all copies match main memory |
| Invalid (I) | This cache line does not contain valid data |
When a core wants to read a memory location, and no other core has it cached, the line is loaded in the Exclusive state. If another core also reads that same location, both copies transition to Shared. If a core wants to write to a Shared line, it must first broadcast an invalidation message to all other cores holding that line, forcing their copies to Invalid, before it can transition its own copy to Modified. This broadcasting is what “snooping” refers to: every cache controller listens (snoops) on a shared bus or interconnect for relevant transactions from other cores.
MOESI and Beyond
MOESI adds a fifth state, Owned (O), which allows a core to have a dirty (modified) line that is also shared with other cores, without immediately writing that dirty data back to main memory. In plain MESI, if another core wants to read a Modified line, the owning core must write the data back to memory first, and then both copies become Shared. In MOESI, the modified data can be forwarded core-to-core directly, with the “Owned” core still responsible for eventually writing it back, avoiding a potentially unnecessary trip through main memory. AMD processors have historically used MOESI, while many Intel designs have used MESI or MESIF (which adds a “Forward” state to optimize which of several Shared copies responds to a new read request).
Snooping vs. Directory-Based Coherence
Snooping protocols work well when all cores share a common bus or interconnect that every cache controller can observe, but this approach doesn’t scale gracefully to systems with many cores or multiple sockets, since broadcast traffic grows with the number of cores. Directory-based coherence solves this scalability problem by maintaining a centralized (or distributed) directory that tracks which cores have a copy of each cache line and in what state. Instead of every core snooping every transaction, a core only needs to consult the directory to find out who currently holds a given line, dramatically reducing broadcast traffic in large multi-socket server systems.
Real-World Performance Implications
Cache behavior has an outsized, often surprising, effect on real-world software performance:
- Loop tiling / blocking: Numerical code (like matrix multiplication) is often restructured to process data in small blocks that fit within cache, dramatically reducing cache misses compared to naive iteration over huge arrays.
- False sharing: A subtle multithreading bug where two threads modify different variables that happen to sit on the same 64-byte cache line. Even though the variables are logically unrelated, the cache coherence protocol forces the line to bounce between cores’ caches on every write, tanking performance. This is purely a cache-line-granularity artifact, not a true data race.
- Cache-friendly data structures: Arrays generally outperform linked lists for sequential access precisely because array elements are contiguous in memory and benefit from spatial locality, while linked list nodes are scattered and cause frequent cache misses.
- NUMA effects: On multi-socket systems, accessing memory attached to a remote socket is significantly slower than accessing local memory, and cache coherency traffic between sockets adds further latency, which is why NUMA-aware programming and thread/memory placement matters for large server workloads.
Common Misconceptions
Misconception 1: Bigger cache is always better. Larger caches have higher latency due to more complex addressing and longer wire delays, so cache hierarchies are carefully tuned tradeoffs, not simply “as big as possible.”
Misconception 2: Cache coherency is a software responsibility. Coherency at the hardware cache-line level is handled entirely by the CPU’s coherence protocol, transparently to software. What software (specifically, the memory model and synchronization primitives of a programming language) does need to manage separately is memory ordering and visibility semantics for concurrent programs, which is related to but distinct from raw cache coherence.
Misconception 3: A cache miss means going straight to RAM. A miss at one level simply means checking the next level down; only an L3 (or L4, if present) miss actually requires a trip to main memory.
Misconception 4: More cores always scale performance linearly. Shared L3 cache capacity and coherence traffic both become contention points as core counts increase, meaning real-world scaling is often sublinear, especially for memory-intensive workloads.
Conclusion
The L1/L2/L3 cache hierarchy exists to bridge the enormous and ever-widening speed gap between CPU cores and main memory, exploiting temporal and spatial locality to keep the data a program actually needs close at hand. As core counts have grown, keeping each core’s private cache consistent with every other core’s view of memory has become a first-class hardware engineering problem, solved through coherence protocols like MESI and MOESI that quietly, invisibly, and constantly negotiate ownership of every cache line in the system. Understanding this hierarchy, and the coherence machinery underneath it, is essential not just for computer architects but for any programmer who wants to understand why some code runs ten times faster than other, seemingly equivalent code.
