A trillion-parameter mixture-of-experts model cannot keep all its weights in RAM, so every serving system builds a custom cache for them. This paper deleted that cache, memory-mapped the weights, and let Linux's page cache take over: the untuned kernel matched the custom cache on hit rate, kept its hit rate through workload shifts that break the custom design, and made a production engine 1.1× faster by changing one flag.
A mixture-of-experts (MoE) model splits its feed-forward weights into hundreds of "experts" per layer; for each token, a small router picks a handful of them. The model is enormous on disk but touches only a slice per token. The production model studied here has a 1.45 TB expert pool, and each decoding step reads about 29 GB of it — on a machine with 480 GB of RAM. The weights cannot all stay resident, so something must decide which ones stay in memory. That component is a cache, whether its authors call it one or not.
The field's consensus design does this in user space: track how often each expert fires, pin the hottest ones in RAM, and read everything else from SSD with caching disabled. The engine the authors instrument passes the "don't cache this" flag on every expert read, on the theory that a cache smaller than the working set only churns.
The alternative deletes code instead of writing it: memory-map the whole pool and let the OS page cache — the disk cache Linux already keeps for all file reads — be the expert tier. The kernel's defaults (admit on miss, evict the least-recently-used page) become the caching policy.
This works because of what expert traffic looks like at scale. On small MoE models, a few experts are hot most of the time, so ranking experts by frequency works. As models grow to hundreds of experts per layer, that hot set thins out: in the trillion-parameter model, the top 8 experts per layer carry only 13–16% of the traffic, versus 34–42% in a small one. The hot set also drifts — the experts that fire on prose barely overlap with the ones that fire on code — so a frequency table learned on one workload is mostly wrong on the next. What persists is recency: 80% of expert reads repeat something used within the last 64 tokens. Least-recently-used eviction, the kernel's default, is a direct bet on recency.
The measurements agree. At equal memory, the untuned page cache served 75.3% of expert reads from RAM. A custom cache pinned by a perfect frequency table — computed from the very trace being replayed, information no real deployment has — served 74.6%. On a different workload, the pinned cache drops to 21–34% while the page cache stays at 70–71%. A pin set tuned on one workload is stale on the next.
The end-to-end win is real but modest. Enabling cache admission — removing that one "don't cache" flag — makes a healthy production engine 1.09–1.10× faster at decode, with token-identical outputs. The raw I/O ceiling underneath is larger: repeat reads went from 13.1 to 68.9 GB/s, a 5.3× difference, but a full decoding step spends most of its time on work other than re-reads. The custom pinned cache keeps a residual 1.09× speed edge at equal memory — not from better caching decisions (the two designs read almost identical bytes from disk) but because each access through the kernel pays lookup and reclaim overhead.
Two caveats. First, everything below the end-to-end test is trace replay: recorded router decisions replayed against the storage tier, with compute excluded by design. Second, the study covers two machines, one kernel line, and one workload family; the authors state it is a characterization, not a general law. They also found a measurement trap: the standard trick for simulating a small-memory machine (locking away most of RAM with a "balloon") interacts badly with the modern Linux reclaim algorithm and can overstate disk traffic by 2× — relevant to anyone who benchmarks caches this way.
If you serve large MoE models: memory-map the weights, remove the direct-I/O and don't-cache flags from expert reads, and put model-specific logic where the kernel has no information — admission. The router knows the next layer's experts about 65% of the time; passing that to the kernel as a readahead hint (an advisory "you will want these bytes soon") helped a little, while synchronously prefetching them did not help at all. Do not reimplement eviction. In production, watch one ratio: when the kernel's reclaim daemon — the thread that evicts pages — scans many pages for every page it frees, the cache is thrashing.
The lesson transfers beyond MoE: before you build a user-space cache, benchmark the one the OS already ships at equal memory, measured with block-device counters rather than application logs. The kernel's thirty years of eviction engineering cost one mmap call, survive restarts, and are shared across processes. Spend the custom effort on admission, not eviction.