Kaiyang Zhao

Over the past few decades, a confluence of factors including i) the proliferation of memory-intensive applications, ii) the rapid expansion of memory capacity towards the terabyte scale, and iii) the slowing of DRAM technology improvements, has made memory scaling difficult and established memory as the critical bottleneck in modern data center computing. The memory bottleneck manifests in multiple dimensions, including access performance, hardware costs, and power consumption.

First, the virtual memory abstraction is under increasing strain. As memory capacity grows while the Translation Lookaside Buffer (TLB) size remains stagnant, virtual address translation overhead becomes more severe; internal profiling at hyperscalers reveals that approximately 20% of CPU cycles are performing page walks due to TLB misses. Furthermore, this overhead will worsen due to the inherent limits of TLB size scaling, the introduction of additional page table levels, hardware virtualization for VMs, the vast memory capacity enabled by heterogeneous memory, and the reliance on the page-level security checks of confidential computing. Without more efficient virtual memory designs, memory will be too slow to scale.

Second, the hardware costs and power consumption of memory have skyrocketed. DRAM now accounts for nearly a quarter of rack power consumption and nearly half of the total cost of ownership (TCO) of a typical data center server. Looking forward, these costs will continue to escalate due to the increasing demands of memory-intensive applications. Such high costs hinder the data center build-out essential for supporting the insatiable demand of AI applications, and increase the operating costs of existing data centers.

This thesis addresses the multifaceted memory efficiency challenge through the co-design of operating systems (OS) and computer architecture.

To improve memory access performance, this thesis presents two works that reduce address translation overhead. Contiguitas creates abundant physical memory contiguity by grouping unmovable allocations in the OS and introducing hardware extensions to migrate previously unmovable pages used for device I/O. Contiguitas then utilizes the newly found physical memory contiguity to allocate more huge pages for applications and reduce the address translation overhead, yielding up to 18% performance improvement for Meta's production workloads. More importantly, the large amount of physical memory contiguity provided by Contiguitas sets up a solid foundation for future work on address translation. Second, Learned Virtual Memory (LVM) replaces the rigid structure of radix page tables with learned page table structures that are tailored to applications’ virtual address spaces. LVM leverages the regularity of application virtual address spaces to build learned indexes for address translation, and balances the size, depth, and accuracy of the learned indexes through a cost model tailored for address translation. LVM enables low-cost virtual address space changes, supports variable page sizes, and is affordable in the hardware. LVM reduces page walk overhead by an average of 44% for a set of memory-intensive applications and achieves a 2–27% speedup in execution.

To reduce the cost and power consumption of memory, this thesis presents two works that enable the practical deployment of CXL memory in modern data centers. Squeezed by the high cost of DRAM and the need for memory capacity, data center operators have explored replacing part of memory with cheaper but slower CXL memory backed by reused last-generation DRAM, along with SSD swap, as offloading tiers. Pensieve introduces a three-tier memory system to transparently manage pages across DRAM, CXL memory, and SSDs, and aims to maximize DRAM savings by utilizing cheaper memory tiers while maintaining application performance. It manages the system through a single-input, single-output control loop, is robust against workload access pattern changes, and leverages CXL memory as an intermediate tier to observe the hotness trends of pages. Pensieve overcomes the capacity limitations of CXL memory and the performance limitations of SSD swap. Pensieve safely offloads 33% of memory on average, substantially outperforming standalone DRAM+CXL (21%) and DRAM+SSD (18%) setups, while keeping application performance degradation below 5%. Equilibria addresses the challenges of deploying tiered memory in today's multi-tenant environments. It ensures fair memory sharing across memory tiers and mitigates noisy-neighbor effects through automatic fair-share determination, flexible fair-share policies, and thrashing mitigation. Equilibria consistently helps workloads meet their service-level objectives (SLOs) while avoiding performance interference, improving performance over the state-of-the-art Linux solution by up to 52% and 1.7× for Meta's production workloads and open-source benchmarks, respectively.

Together, these works show that through a comprehensive set of novel architectural and operating system techniques that work together, the memory efficiency in modern data centers can be significantly improved from the perspectives of performance, hardware costs, and power. This thesis lays the foundation for memory systems that can scale with evolving application demands and hardware trends, while opening a broad set of directions for future research in computer architecture and operating systems. Several of the works presented in this thesis have already been upstreamed into Linux and deployed across production data centers on millions of servers, helping scale memory systems for years to come.