When profiling low-level memory performance on modern Intel architectures like Skylake-X (SKX), microarchitectural behaviors can sometimes appear almost mystical. A prime example is observing that translation lookaside buffer misses—specifically MEM_INST_RETIRED.STLB_MISS_LOADS—drop from 1000 down to near zero only when crossing a specific cache line (cache line 52) within a 4KB page.

Understanding the Phenomenon

In microarchitectural experiments measuring STLB (Second-Level TLB) load misses across sequential 4KB pages, developers have observed that issuing a cache-line crossing access at cache line 52 causes STLB misses on subsequent pages to vanish:

MAPPED PAGES = 1000, TARGET_CACHELINE = 51, STLB_LOAD_MISS = 1000
MAPPED PAGES = 1000, TARGET_CACHELINE = 52, STLB_LOAD_MISS = 2
MAPPED PAGES = 1000, TARGET_CACHELINE = 53, STLB_LOAD_MISS = 1000

Why does this happen only at cache line 52? Does Skylake-X have a dedicated "next-page prefetcher" hardcoded to offset ~3392 bytes, or is this the side effect of an existing prefetch engine interacting with the Page Miss Handler (PMH)?

1. Addressing the Core Fallacy: Independent vs. Sequential Runs

Before diving into the hardware mechanics, it is vital to clarify how the benchmark functions. In benchmark loops structured like this:

for (int i = 0; i < 63; i++) {
    flush_tlb_range(d.map_base, map_size);
    d.target_cacheline = i;
    perf_measure(..., kernel, &d);
}

Each run of kernel tests only one cache line offset across all pages. When running with target_cacheline = 53, the code accesses byte 63 of line 53 and byte 0 of line 54. It never touches line 52 during that run.

Therefore, the fact that lines 53 through 62 produce 1000 misses does not mean line 52 is a threshold that stops working; it means that touching lines 53–62 in isolation fails to trigger the prefetch mechanism in the same way.

2. The Math Behind Cache Line 52: The L2 Streamer Lookahead Window

A standard 4KB page consists of exactly 64 cache lines (numbered 0 to 63):

  • Line 0: Bytes 0 – 63
  • Line 52: Bytes 3328 – 3391
  • Line 63: Bytes 4032 – 4095
  • Line 64: Byte 4096 (Start of the next 4KB page)

Notice the distance between cache line 52 and the start of the next page: 64 - 52 = 12 cache lines.

In Intel Skylake microarchitecture, the L2 Streamer Prefetcher monitors sequential line accesses. Once it detects a stream, it dynamically adjusts its prefetch distance (lookahead depth). The nominal initial lookahead distance for forward streaming in the L2 prefetcher is typically 12 to 16 cache lines.

What happens at Cache Line 52:

  1. The test accesses line 52 and then crosses into line 53.
  2. The L2 Streamer registers a forward stream and initiates prefetch requests ahead by its lookahead window: 52 + 12 = 64.
  3. Cache line 64 resides on page i + 1.
  4. To issue the line prefetch for line 64, the processor must translate the address. Because line 64 belongs to the next virtual page, this triggers a speculative translation request to the Page Miss Handler (PMH).
  5. The PMH performs a page table walk and loads the Page Table Entry (PTE) into the STLB before the program instruction for page i + 1 is executed.
  6. When iteration i + 1 runs, its load hits the pre-warmed STLB entry, reducing misses from 1000 to ~0.

3. Why Don't Lines 53 to 62 Trigger the Same Prefetch?

If prefetching ahead by 12 lines crosses the page boundary, why wouldn't an access at line 53 (which projects to 53 + 12 = 65) also prefetch page i + 1?

Intel CPUs implement strict boundary restrictions on speculative prefetchers to prevent spurious page faults and segmentation violations:

  • Page Boundary Clamping: The standard L2 Streamer is designed not to cross 4KB page boundaries under normal conditions. When a stream hits or approaches the end of a page, requests beyond the boundary are typically discarded or clamped.
  • Exact Boundary Hand-off / Trigger Threshold: Hardware designs that support speculative boundary crossing (or TLB prefetching) often sample at an exact trigger distance prior to the boundary (e.g., 768 bytes / 12 lines prior) to initiate the next page walk in parallel without overflowing internal prefetch queues.
  • Stream Training Latency: In this micro-benchmark, each page only receives two single-byte loads. At line 53 or higher, the streamer might recognize the access too close to the page termination, leading the prefetcher pipeline to squash requests targeting beyond the page limit.

4. Verifying the Mechanism via Model-Specific Registers (MSRs)

To definitively prove that the L2 Streamer / hardware prefetcher causes this behavior rather than an isolated STLB logic rule, you can disable individual prefetchers using the IA32_MISC_ENABLE and MSR_PREFETCH_CONTROL (0x1A4) registers on Linux.

# Read current MSR 0x1A4 (requires msr-tools)
sudo rdmsr -p 0 0x1a4

# Disable L2 Streamer and L2 Spatial prefetchers (Bits 0 and 1)
sudo wrmsr -p 0 0x1a4 0x3

The prefetch control bits on Skylake-X are:

  • Bit 0: Disable L2 Hardware (Streamer) Prefetcher
  • Bit 1: Disable L2 Adjacent Cache Line (Spatial) Prefetcher
  • Bit 2: Disable DCU (L1) Streamer Prefetcher
  • Bit 3: Disable DCU (L1) IP Prefetcher

If you re-run the benchmark with Bit 0 set, the anomalous STLB miss drop at cache line 52 should disappear, confirming that the pre-population of the STLB was an artifact of the L2 Streamer triggering the Page Miss Handler.

Summary

The sudden drop in STLB misses at cache line 52 is not a specialized, hardcoded page-crossing prefetcher rule. Rather, it is the mathematical convergence of an access crossing from line 52 to 53 and the L2 Streamer's 12-cache-line lookahead window (52 + 12 = 64), which precisely lands on the first cache line of the next 4KB page, driving speculative address translation via the Page Miss Handler.