[#2 - Memory]

Reserving memory

LPVOID reserved_address = VirtualAlloc(nullptr, arena->capacity, MEM_RESERVE, PAGE_NOACCESS);
arena->address = (unsigned char*)reserved_address;

The snippet above illustrates how to reserve a block of virtual address space on Windows. The returned address is automatically aligned to the operating system’s allocation granularity. The syntax is inherently more complex than standard library functions such as malloc().

Typically, there is no need for concern regarding allocation granularity because the operating system is simply requested to provide a base address wherever space permits within the virtual address space. It only becomes critical when attempting to force a reservation at a specific memory address; in such cases, it must be understood that the supplied address will be rounded down to the nearest allocation boundary.

By bypassing standard allocation functions, the operating system API allows for the reservation of massive blocks of contiguous virtual memory. This guarantees the memory will always appear contiguous within the virtual address space. However, this virtual contiguity does not reflect the underlying physical RAM layout.

Physical vs. Virtual Memory

Physical RAM is managed by the operating system using page tables, which the CPU’s Memory Management Unit (MMU) uses to translate virtual addresses into physical hardware addresses. It is standard for operating systems to utilise 4-kilobyte pages, as they offer an optimal balance for general-purpose computing. While the operating system can utilise much larger memory pages, they introduce significant trade-offs:

• Large pages can be unpredictable to allocate due to memory fragmentation.
• Specialised and predictable hardware can leverage them to reduce Translation Lookaside Buffer (TLB) overhead and maximise performance.

To determine the exact page size and allocation granularity a specific system uses for VirtualAlloc(), GetSystemInfo() must be called to inspect the SYSTEM_INFO structure. For example, if the custom memory allocator is designed only around Intel/AMD default 4-kilobyte pages, trying to run the specialised code on CPU that only supports 16-kilobyte pages might yield unexpected results. In hardcore performance programming, tailoring memory management for each CPU is not unheard of.

Finally, VirtualAlloc() is designed for large, coarse-grained reservations. For smaller, routine allocations, HeapAlloc() is generally more appropriate as it lets you manage a smaller heap for allocations.

Committing memory

LPVOID writable_address = VirtualAlloc(arena->address + arena->offset, chunk->capacity, MEM_COMMIT, PAGE_READWRITE);
chunk->address = (unsigned char*)writable_address;
arena->offset += chunk->capacity;

This code segment is similar to the previous example; however, it instructs the operating system to commit a segment from the previously reserved block. Flags such as PAGE_READWRITE dictate that the application requires both read and write access to this specific chunk of virtual memory.

It must be noted that the software perspective and the hardware reality differ at this stage. Due to the memory management techniques of modern operating systems, the physical RAM might not be actively mapped to the virtual addresses until the application attempts to read from or write to that memory for the first time.

This mechanism is known as demand paging, which defers physical memory allocation until it is strictly necessary.

Slicing paged memory

uint64_t address_offset = (chunk->offset + slice->alignment - 1) & ~(slice->alignment - 1);
slice->address = chunk->address + address_offset;
chunk->offset = address_offset + slice->capacity;

Once the chunk memory address is obtained, it can be partitioned into smaller slices. The alignment calculation may appear cryptic, but omitting it may carry some consequences.

This is because CPUs read memory in fixed-size hardware blocks, such as 32-byte or 64-byte cache lines. If data is not cleanly aligned, the hardware might perform additional work. Proper alignment is a fundamental architectural rule and should not be ignored when managing memory manually.

As a simplified boundary-crossing example, consider placing a 4-byte variable at offset 61:

• It will occupy bytes 61, 62, 63, and 64.
• Because the hardware fetches memory in discrete 64-byte chunks (bytes 0 through 63, then 64 through 127), the variable is physically severed across a cache line boundary.
• To read or write that single variable, the CPU is forced to fetch the first chunk, fetch the second chunk, and stitch the data together behind the scenes.
• Some CPUs might handle unaligned data, but this should not be relied upon

One might ask why all data is not simply aligned and padded to a 64-byte boundary to avoid this entirely. The reason is memory efficiency; padding a single 1-byte character to a 64-byte boundary would waste 63 bytes of space.

The bitmask operation in the code prevents both unaligned access and excessive waste. It automatically advances the starting address forward until it lands on a safe, cleanly aligned boundary suited for the specific data type. It should be noted that this specific mathematical operation requires the target alignment to be a strict power of two and not equal to zero.

False-sharing

uint64_t address_offset = (chunk->offset + CPU_CACHE_ALIGNMENT - 1) & ~(CPU_CACHE_ALIGNMENT - 1);
chunk->offset = address_offset + size;

False-sharing can become an issue when two threads access memory residing in the same chunk. If a chunk’s address is not aligned to the CPU’s cache line size, independent data can end up sharing the same cache line. When threads attempt to read and write to their respective slices of this shared line, it causes massive performance degradation.

An extreme example involves an 8-byte memory chunk containing two 4-byte variables, A and B. Both sit perfectly within the same 64-byte cache line boundary. If Thread 1 is assigned to Variable A and Thread 2 is assigned to Variable B, the threads operate without apparent issue. However, the CPU detects a conflict. Behind the scenes, the hardware cache coherency system interprets that both cores are fighting over ownership of the exact same 64-byte cache line, constantly invalidating it across the cores.

This is resolved by padding each variable to its own 64-byte boundary. While Variable A now wastes 60 bytes of space, the memory cost is a highly preferable alternative to the performance penalty. This approach solves two problems: it eliminates false-sharing, and it assists in unmasking bugs if Thread 1 accidentally writes beyond its allowed capacity.

Advanced Considerations: Poisoning & Prefetching

• Memory Poisoning: If a thread does write beyond its bounds, an underlying bug requires fixing. Consequently, the padding between memory slices is commonly “poisoned” (filled with a specific byte pattern, like 0xDEADBEEF) during debugging to catch these exact memory overruns.

• The Spatial Prefetcher: The benefit of additional padding extends beyond catching bugs; it prevents issues caused by modern CPUs’ spatial prefetchers. Because most high-performance architectures aggressively load adjacent cache lines, they can inadvertently drag cores back into a conflict over memory. To completely eliminate this risk across different hardware targets, memory is ideally padded in multiples of the CPU cache line size.

Struct Sizes and Arrays

It must be noted that aligning the base memory address is only partially sufficient when allocating contiguous blocks, such as arrays. If a data structure’s exact size is not a perfect multiple of the target alignment - for example, a 48-byte structure - subsequent elements within a tightly packed array will inevitably cross cache line boundaries. This reintroduces the risk of false-sharing and subsequent performance degradation.

To maintain strict hardware isolation across an entire block of memory, the allocation stride itself must be padded to the nearest multiple of the CPU cache line size. In the case of a 48-byte structure, the allocation size must be explicitly rounded up to 64 bytes, ensuring every sequential element lands firmly on a safe boundary.

Guard Pages

There is also a possibility that two memory chunks are separately committed by two distinct VirtualAlloc() calls. If these chunks happen to sit directly next to each other in the virtual address space, and Thread 1 and Thread 2 are reading and writing near that shared boundary, there might still be concerns regarding performance or memory safety.

How does this interact with the hardware? Because VirtualAlloc() operates by the rules of the operating system, it aligns allocations to the default page size, which is a perfect multiple of the CPU cache line size. Furthermore, modern hardware prefetchers are specifically designed to stop at the page boundary. This effectively eliminates the risk of hardware-level false-sharing across this divide. However, the risk of a thread sequentially reading or writing past its intended bounds and corrupting the neighbouring thread’s memory remains a critical issue.

The Ultimate Poison

How can this be solved? Separate the chunks with a default-sized page of uncommitted virtual memory. Because the virtual address space is managed at the operating system level, this gap acts as an impassable wall. Not only does this guarantee total isolation between the memory chunks, but it also provides the ultimate poison. If a thread accidentally oversteps its bounds and touches that uncommitted page, the operating system will immediately trigger an access violation, catching the bug instantly.

Decommit & Release

VirtualFree(chunk->address, chunk->capacity, MEM_DECOMMIT);
VirtualFree(arena->address, 0, MEM_RELEASE);

Finally, once the game engine concludes its use of the memory management system, a single VirtualFree() call can immediately destroy the entire block of virtual memory addresses. (When using MEM_RELEASE, the size parameter must be exactly 0).

If only a specific portion of the reserved chunks requires decommitting - where the physical memory is returned to the operating system, but the virtual address space remains reserved for future use - the MEM_DECOMMIT flag must be used instead. This operation requires pointing the function directly at the specific chunk address alongside the corresponding size of the memory to be decommitted.

Final Thoughts

Whilst VirtualAlloc() is restricted to Windows, the underlying concepts of memory management, hardware alignment, and paging apply to every modern operating system. Understanding these mechanics is what bridges the gap between writing general applications and building high-performance engines.

For more technical information visit:

https://learn.microsoft.com/en-us/windows/win32/api/memoryapi/nf-memoryapi-virtualalloc