A large in-memory database can map its heap at a different virtual address every time it starts, translate that heap using 2 MiB pages, and store some physical frames close to one CPU socket and others close to another.

These behaviors come from three different mechanisms:

They often appear together in performance investigations, but they solve different problems.

Huge pages reduce translation overhead by mapping more bytes with one page-table entry and TLB entry.

Non-uniform memory access, or NUMA, exposes that some physical memory is faster for a given CPU to reach than other memory.

Address Space Layout Randomization, or ASLR, varies virtual addresses across executions so attackers cannot reliably predict where code and data will appear.

Huge pages concern mapping size, NUMA concerns physical topology, and ASLR concerns virtual placement.

Keeping those dimensions separate makes their interactions much easier to reason about.

Paging normally uses a base page size. A common base size on x86-64 Linux is:

Many processors also support larger translation units. Common x86-64 examples are:

Other architectures support different sizes.

A page larger than the base size is broadly called a large page or huge page. Linux uses huge page in several specific interface names, but the general idea is the same:

With 4 KiB pages, a 2 MiB region needs:

With one 2 MiB page:

The application still reads and writes ordinary byte addresses. The difference is how many address bits form the page offset and how much memory one translation covers.

Huge pages mainly reduce address-translation overhead.

Greater TLB reach

Assume a TLB can hold 64 entries.

With 4 KiB pages:

With 2 MiB pages:

The exact processor can have separate TLB structures and capacities for different page sizes, so this arithmetic is a conceptual upper bound. The increase in coverage is still substantial.

A workload scanning a large, densely used memory region can experience fewer TLB misses when each cached translation covers more data.

Smaller page tables

A 1 GiB virtual region requires:

Fewer leaf entries reduce page-table memory and the number of lower-level table nodes needed for a dense region.

Fewer mapping units

Operations that establish, inspect, or fault in a genuinely huge-page mapping can work with fewer translation units. This can reduce some page-fault and page-table-management overhead.

The benefit is strongest when the application uses most of each huge page. A large sparse region does not automatically benefit from allocating large physical units for mostly untouched bytes.

A multilevel page table normally walks to a leaf entry that maps a base page.

For a larger page, an entry at a higher level can terminate the walk early:

The offset is wider for the huge page:

because:

The virtual and physical ranges must satisfy the alignment requirements for the chosen size. A 2 MiB mapping begins at a suitable 2 MiB boundary, not at an arbitrary byte.

Larger translation units trade fine granularity for lower translation overhead.

Internal waste

If an application actively uses only 64 KiB of a privately allocated 2 MiB huge page:

Whether all that space is truly wasted depends on later use and the implementation, but the allocation granularity can increase physical-memory consumption for sparse workloads.

Physical contiguity

A 2 MiB huge page normally requires one aligned, physically contiguous 2 MiB range. Finding that range becomes harder after physical memory is fragmented.

The kernel may compact memory by moving eligible base pages together. Compaction consumes CPU time and can add latency.

Larger fault and copy granularity

Materializing a huge page can require more allocation and zeroing work than a base-page fault.

A write to a shared huge page creates a copy-on-write challenge. Copying the complete huge page is expensive, while splitting it into base pages adds page-table and invalidation work. Kernels choose according to state and policy.

Coarser reclaim

Reclaiming or migrating a larger unit can move more data than the workload currently needs. A system may split a huge page so smaller parts can be managed independently.

Allocation failure

Enough free bytes in total do not guarantee one aligned contiguous huge frame. Explicit huge-page allocation can fail while base-page allocation still succeeds.

Huge pages are therefore an optimization, not a universally better page size.

Linux supports explicitly managed huge pages through the hugetlb subsystem.

The kernel maintains a pool of huge pages, commonly reserved by an administrator. Applications can use mechanisms such as hugetlbfs files or MAP_HUGETLB mappings.

Conceptually:

This request can fail if the configured huge-page pool lacks enough suitable pages. The exact size selection and reservation interfaces depend on architecture and system configuration.

Explicit huge pages provide predictable huge-page backing, but they require operational planning:

They are common in specialized high-performance workloads that prefer explicit capacity management over automatic promotion.

Transparent Huge Pages, or THP, let the Linux kernel use huge translations for eligible ordinary mappings without requiring every application to allocate from an explicit pool.

The kernel can:

Applications continue using normal anonymous mappings. They can provide advice for eligible ranges:

or discourage huge-page use:

Advice is not the same as a guarantee. Alignment, access pattern, fragmentation, kernel configuration, and current memory state affect the result.

THP can improve throughput for large dense heaps by reducing TLB misses. It can also create latency from compaction, promotion, splitting, or large first-touch work. Latency-sensitive systems should measure both steady-state throughput and tail latency under realistic memory pressure.

System-wide THP policy is commonly visible at:

Possible modes and exact behavior depend on the kernel. Do not change a production-wide setting solely from a generic rule; application hints and measured workload behavior often provide a safer basis.

Check the base page size:

Inspect the explicit huge-page pool:

Typical fields include:

For one process, /proc/<pid>/smaps can reveal page-size and huge-page accounting:

Interpret the fields per mapping rather than summing repeated lines without context.

A large anonymous region with nonzero AnonHugePages has at least some memory backed through transparent huge-page mappings. A mapping can contain a mixture of huge pages and base pages, so its virtual size alone does not reveal the translation granularity throughout.

On a small single-socket machine, software can often treat RAM access cost as approximately uniform.

Large multiprocessor systems commonly use non-uniform memory access, or NUMA. CPUs and memory controllers are grouped into NUMA nodes.

A CPU reaches memory attached to its own node through a local path. Reaching memory attached to another node crosses an interconnect and is usually slower or offers less aggregate bandwidth.

The operating system still presents one physical address space. A pointer does not encode “local” or “remote” in an application-visible way. Locality depends on:

The same physical page can be local to CPUs on one node and remote to CPUs on another.

A common default NUMA policy allocates an anonymous physical page near the CPU that first writes it.

Suppose a program:

The first-touch result can be:

The malloc() call alone does not determine placement because untouched virtual pages may have no physical frames. First access triggers physical allocation.

A parallel initialization can improve placement:

This works when later workers continue using the same partitions. If threads migrate or access every partition uniformly, placement needs a different strategy.

First-touch is a common policy, not an application guarantee on every system. Explicit memory policy, automatic balancing, file-page placement, or resource boundaries can change the outcome.

Linux exposes several conceptual placement policies.

Local or default allocation

Prefer memory near the CPU currently executing, subject to available capacity and system policy.

Preferred node

Try one node first, but allow fallback to other nodes if necessary.

Binding

Restrict allocation to a specified set of nodes.

Binding provides predictability but can cause allocation failure within the allowed nodes even when memory exists elsewhere.

Interleaving

Distribute allocations across a set of nodes:

Interleaving spreads bandwidth and avoids concentrating all memory on the initializer's node. It also means many accesses are remote unless work is partitioned to match placement.

The numactl tool can inspect and launch programs with policies:

Example commands on a system that actually has nodes 0 and 1:

Node numbers are machine-specific. Inspect the topology before applying a policy.

CPU affinity and memory policy are related but separate. Pinning a thread to node 1 does not automatically migrate all pages it previously allocated on node 0.

An operating system can observe access patterns and try to improve locality by migrating tasks or pages.

Conceptually:

On detecting repeated remote access, the kernel can move the page closer to the accessing CPU, or move the thread closer to its pages.

Migration costs memory bandwidth and CPU time. A page actively used from both nodes has no placement that is local to every CPU. Moving it back and forth can be worse than leaving it remote to some users.

Automatic balancing is therefore heuristic. It helps workloads whose access patterns are stable enough to learn, but explicit data partitioning can be more predictable for high-performance applications.

Large pages make migration more complicated because moving one huge page transfers more data and requires suitable contiguous space at the destination. The kernel can split pages when finer control is worthwhile, losing some huge-page benefits.

Display CPU-to-node topology:

With numactl installed:

This shows nodes, CPUs, memory sizes, and a relative distance matrix where supported.

Inspect one process:

This can summarize memory allocation across nodes.

For mapping-level detail:

Entries can show a virtual mapping and counts such as:

meaning resident pages from that mapping are distributed across nodes 0 and 1.

Measurements should be correlated with CPU placement. A page distribution is not good or bad by itself. It is good when the threads using those pages run near them or when deliberate interleaving supplies the desired aggregate bandwidth.

NUMA issues often appear as performance variance rather than correctness failures.

Single-threaded initialization

One startup thread first-touches a large heap, placing it on one node. Later worker threads on other nodes perform remote accesses.

Thread migration

A thread builds a cache while running on node 0, then the scheduler moves it to node 1. Its cache remains physically on node 0 unless balancing or explicit migration moves it.

Shared global structures

Threads on every socket frequently update one shared data structure. No single placement is local to everyone, and cache-coherence traffic can dominate alongside remote-memory latency.

Strict binding

A service bound to one node exhausts that node's memory and fails or reclaims heavily while other nodes have free capacity.

Container boundaries

CPU and memory-node restrictions can differ inside a container or cgroup. A process should inspect the topology and policy visible within its actual resource boundary.

NUMA tuning begins with ownership: identify which threads use which data, then align CPU placement, page placement, and data partitioning.

Virtual addresses become valuable to an attacker who can exploit a memory-corruption bug. If the attacker knows exactly where executable code, libraries, stacks, or control data reside, constructing a reliable exploit is easier.

Address Space Layout Randomization, or ASLR, varies the virtual placement of major regions across process executions.

Regions commonly affected include:

Two runs can therefore show:

The program's logic is unchanged. Only the virtual layout differs.

ASLR is a defense in depth. It does not repair an out-of-bounds write, prevent all pointer disclosure, or encrypt memory. It makes a successful exploit less predictable.

Shared libraries are normally position independent so the loader can place them at different virtual bases.

The main executable also needs suitable position-independent code if its own code and globals are to move freely. Such a binary is a position-independent executable, or PIE.

Build one explicitly with a GCC- or Clang-style toolchain:

Without PIE, an executable can still receive random stack, heap, library, and mapping locations, but its main code may remain at a fixed link address on systems that use fixed-position executables.

PIE uses relative addressing and runtime relocation techniques similar to position-independent shared libraries. The loader chooses a base and adjusts private relocation data as required while keeping executable pages suitable for sharing.

Toolchain defaults vary. Inspect the produced file instead of assuming a build is or is not PIE:

A PIE commonly appears as an ELF shared-object-style type even though it is an executable program.

ASLR does not continuously scramble a pointer while a process runs.

When the operating system and loader construct the address space, they choose randomized locations that satisfy alignment, range, and collision constraints.

Once a region is mapped:

The physical frame behind a virtual page can still change through ordinary virtual-memory management without changing the pointer.

This separates two mechanisms:

A child created with fork() inherits the parent's virtual layout. If it later calls exec(), the new program receives a newly constructed layout and fresh randomization according to system policy.

Process A can map a library page at one randomized virtual address while Process B maps it elsewhere:

Physical sharing is based on the backing file page, not equal virtual addresses.

Position-independent code is what lets the same instruction bytes execute correctly at both locations. Process-specific symbol addresses live in private relocation structures when needed.

ASLR therefore does not require duplicate physical code pages. Virtual placement and physical sharing remain independent.

This program prints addresses from three common regions:

Compile as PIE and run it several times:

On a system with ASLR enabled, the exact addresses commonly change between executions.

Linux exposes its broad randomization policy through:

The meaning of values is Linux-specific. A nonzero setting enables some level of randomization, while the most complete normal mode randomizes additional regions. System-wide security policy should not be changed merely to make one debugging session easier.

Debuggers and controlled test environments can provide per-process techniques for reproducible layouts. Production deployments should preserve the expected randomization policy.

An absolute pointer from one run may identify unrelated or unmapped memory in another run.

A crash report should preserve:

Instead of treating:

as a permanent library address, symbolize it as:

The module-relative offset remains meaningful when the same build loads at another randomized base.

ASLR also explains why copying a raw pointer from a production log and using it in a new debugger execution is rarely useful without the original mapping layout.

An information leak that reveals current mappings can weaken ASLR. Randomization is not a substitute for fixing memory-safety bugs and protecting pointer-bearing diagnostics.

Consider a 2 GiB in-memory index.

Virtual placement

ASLR and the virtual-memory allocator choose a process address such as:

Another run can place the index elsewhere.

Translation granularity

The region can use:

if alignment, physical memory, and policy permit.

Physical topology

The frames can reside across NUMA nodes:

depending on first touch or explicit policy.

Changing one dimension does not automatically fix another:

Performance tuning should identify which dimension is actually limiting the workload.

For a large memory-intensive service:

Do not infer huge-page use from mapping size. Inspect page-size accounting.

Do not infer NUMA locality from CPU affinity alone. Inspect page placement.

Do not infer disabled ASLR from one address that happened to repeat. Inspect binary type, process mappings, and system policy.

Optimization goals can conflict. Reserving explicit huge pages reduces memory available to base-page workloads. Strict NUMA binding improves locality until one node runs out of memory. Aggressive huge-page promotion improves TLB reach but can add compaction latency.

The correct configuration depends on whether the service values throughput, tail latency, memory density, startup time, or predictable allocation.

Huge pages map larger virtual and physical units, increasing TLB reach and reducing page-table entries for dense memory regions. Their tradeoffs include internal waste, contiguous physical allocation, compaction, coarser faults, copy-on-write, migration, and reclaim. Linux provides both explicitly reserved hugetlb pages and automatically managed Transparent Huge Pages.

NUMA makes physical-memory access cost depend on the relationship between the executing CPU and the page's node. First-touch placement, CPU affinity, binding, interleaving, automatic balancing, and data partitioning determine whether a workload mostly accesses local or remote memory.

ASLR randomizes virtual placement across executions, while PIE and position-independent libraries allow code to move and remain shareable. It is a security defense in depth, not memory safety. These three mechanisms are independent dimensions: huge pages choose translation size, NUMA chooses physical location, and ASLR chooses virtual location.