keepcalculator

06 Numbers and Powers

NUMA Non Uniform Memory Access Explained

Card No. 06·71008 · Updated · Standard formula, shown below

NUMA stands for Non-Uniform Memory Access, a computer memory design used in multiprocessor systems where the time to access memory depends on its location relative to the processor. In a NUMA system, each CPU has its own local memory, and accessing that local memory is faster than accessing memory attached to another CPU. This article explains how NUMA works, why it matters, and how it compares to other architectures.

Background: Why NUMA Exists

Early multiprocessor systems used a symmetric multiprocessing (SMP) design, where all processors shared a single memory bus and a single memory controller. This is called Uniform Memory Access (UMA) because every CPU has the same access time to all memory. UMA is simple to program, but it has a serious bottleneck: as you add more CPUs, they all compete for the same bus, and performance degrades quickly. By the late 1990s, systems with more than four or eight CPUs became impractical with UMA.

NUMA solves this by giving each CPU its own memory controller and a slice of physical memory. This removes the single shared bus, allowing systems to scale to dozens or even hundreds of cores. The trade-off is that memory access times are no longer uniform—hence the name. The first mainstream NUMA systems were from vendors like Sequent and SGI in the 1990s, and today every multi-socket x86 server (AMD EPYC, Intel Xeon) uses NUMA. Even some single-socket chips, like AMD’s Zen architecture, use a NUMA-like design internally with chiplets.

How NUMA Works

In a NUMA system, the physical memory is divided into zones, each attached to a specific CPU (or a group of CPUs called a NUMA node). Each node has its own memory controller. When a CPU accesses memory that is attached to its own node, it uses its local memory controller and gets low latency. When it accesses memory attached to another node, it must send a request over an interconnect (such as Intel’s Ultra Path Interconnect or AMD’s Infinity Fabric) to the remote node’s memory controller. This remote access has higher latency and lower bandwidth.

The operating system and the hardware work together to manage this. The OS tries to allocate memory on the same node as the thread that will use it, a policy called “first-touch” allocation. When a thread first touches a page of memory, the OS assigns it to that thread’s node. If a thread migrates to a different CPU later, the memory it was using stays on the old node, leading to remote access. Some systems use automatic migration or page balancing to fix this.

NUMA also applies to I/O. Each node has its own PCIe controllers, so a device like an NVMe SSD or a GPU may be attached to a specific node. Accessing that device from a CPU on another node is slower. This is why you often see “NUMA node” information in tools like lscpu on Linux or Task Manager on Windows.

NUMA vs. UMA vs. MPSoC

To understand NUMA, it helps to compare it to other designs:

  • UMA (Uniform Memory Access): All CPUs share one memory bus and one memory controller. Access time is the same for all CPUs, but the bus becomes a bottleneck. Used in older multi-core laptops and small servers.
  • NUMA (Non-Uniform Memory Access): Each CPU or group of CPUs has its own local memory. Access is faster locally, slower remotely. Used in all modern multi-socket servers and some high-end desktops.
  • MPSoC (Multi-Processor System-on-Chip): A single chip that integrates multiple CPUs (often heterogeneous, like ARM big.LITTLE) and memory. These can be UMA or NUMA depending on the design. The term is more about integration than memory architecture.

The key difference is that UMA is simpler to program because memory access time is predictable. NUMA requires software to be aware of the topology to get the best performance. However, NUMA’s scalability advantage is so significant that it is the standard for any serious server.

When to Use NUMA

NUMA is not a choice you make; it is a property of the hardware. You use NUMA whenever you use a multi-socket server or a modern chiplet-based CPU. However, you can influence how well NUMA works:

  • Virtualization: When running virtual machines, you should pin each VM to a NUMA node to avoid cross-node memory traffic. Most hypervisors (VMware, KVM, Hyper-V) do this automatically, but you can tune it.
  • Databases: Large databases like Oracle or SQL Server have NUMA-aware settings. For example, Oracle recommends setting CPU_COUNT and DB_BLOCK_SIZE based on NUMA topology.
  • High-performance computing: MPI applications often bind processes to specific NUMA nodes to minimize remote memory access. Tools like numactl on Linux let you specify which nodes to use.

For general-purpose desktop use, NUMA is mostly transparent. The OS handles it, and you rarely need to worry about it. The main exception is if you are building a dual-socket workstation for heavy workloads like video editing or 3D rendering—then you might want to ensure your software is NUMA-aware.

Common Problems and Misconceptions

One common misconception is that NUMA is a performance *penalty* that you should avoid. In reality, NUMA is what makes large systems possible. The remote access penalty is real, but it is much smaller than the bottleneck of a shared bus in a large UMA system.

Another issue is “NUMA thrashing” or “cache line bouncing.” If two threads on different nodes frequently read and write the same memory location, the cache coherence protocol forces them to exchange data over the interconnect, which is slow. This is not unique to NUMA, but it is more costly because the interconnect is slower than a shared bus.

Misconfiguration is a common problem. For example, if you install memory modules incorrectly (not populating all channels for each node), you can create an unbalanced NUMA topology, where one node has more memory than another. This can lead to memory being allocated on the wrong node. Also, some BIOS settings like “NUMA optimization” or “Node Interleaving” can change the behavior. Node interleaving makes memory access uniform by spreading addresses across all nodes, but it increases average latency. It is usually better to leave it disabled and let the OS use first-touch.

How to Check NUMA Topology

If you want to see the NUMA layout of your system, there are simple tools:

  • Linux: Run lscpu to see the NUMA node count and CPU-to-node mapping. Run numactl --hardware to see memory ranges for each node.
  • Windows: Open Task Manager, go to the Performance tab, and click on Memory. You will see a “NUMA” section if your system has multiple nodes. You can also use the wmic command or PowerShell to query NUMA info.
  • macOS: Macs with Intel Xeon processors (like the Mac Pro) support NUMA, but Apple’s own silicon (M1/M2) uses a different architecture. You can use sysctl hw.ncpu and look at the hw.memsize to infer, but it is not as straightforward.

Understanding your topology helps you make informed decisions about thread pinning and memory allocation in performance-critical applications.

Costs and Trade-offs

The main cost of NUMA is complexity. Hardware must include an interconnect that can handle remote memory requests, and the cache coherence protocol must work across nodes. This adds cost to the motherboard and CPU. Software must be NUMA-aware to achieve peak performance, which requires effort from OS developers and application programmers.

For the end user, the cost is mostly in the price of multi-socket systems, which are more expensive than single-socket ones. However, the performance scaling you get is worth it for many workloads. For example, a dual-socket server with 64 cores total can outperform a single-socket 32-core server by almost 2x in memory-bandwidth-intensive tasks, even with the NUMA overhead.

Frequently Asked Questions

Is NUMA good or bad?

NUMA is neither good nor bad; it is a necessary design for scaling beyond a few cores. It is good because it allows systems to have many CPUs without a shared bus bottleneck. It is “bad” only in that it requires software to be aware of the topology to get the best performance. For most users, the OS handles it automatically.

How do I disable NUMA?

You usually cannot disable NUMA on modern hardware, but you can make it behave like UMA by enabling “Node Interleaving” in the BIOS. This spreads memory addresses evenly across all nodes, making access times uniform (but higher on average). It is not recommended because it reduces bandwidth and increases latency for all accesses.

What is a NUMA node?

A NUMA node is a group of CPUs and their local memory. In a dual-socket system, each socket is typically one NUMA node. On AMD EPYC, each chiplet may be its own node. The OS sees each node as a separate memory domain.

Does NUMA affect gaming?

For gaming, NUMA has little impact because games typically run on a single CPU socket or on a single chiplet. The OS scheduler handles it well. The only time you might notice is if you have a dual-socket system and the game uses many threads that span nodes, but that is rare.

What is the difference between NUMA and SMP?

SMP (Symmetric Multiprocessing) refers to a system where all CPUs are treated equally and share all memory. UMA is a type of SMP. NUMA is also a type of SMP, but it is not uniform because each CPU has its own local memory. So NUMA is a subset of SMP, not a separate category.

How do I make my application NUMA-aware?

On Linux, you can use numactl to launch a program with specific node bindings. For example, numactl --cpunodebind=0 --membind=0 ./myapp forces the app to run on node 0 and allocate memory there. In code, you can use the libnuma library to control memory allocation and CPU affinity.

Related

Method: the standard formula for this calculation, rounded as each result states. See how the calculators work or report an error.

More cards in this drawer

The whole Numbers and Powers drawer →