C++ memory management in a game engine: who eats the frame?
Calling new inside the frame produced 13 ms spikes in the profiler; after a frame arena and object pooling the peak dropped to 7.9 ms. Here is how it went.
A three-millisecond spike in the profiler
Late last winter I was profiling the combat scene in our own engine. Frame time sat around 9 ms on average, but every few seconds a 13 ms frame showed up. Nothing stuttered visibly on screen, yet at a 60 FPS target those spikes were noticeable. We looked at rendering first, because everyone looks there first, and render time turned out to be almost constant frame to frame.
The culprit was a single line inside DamageNumberSystem: a new DamageLabel() for every hit. During a wave with 400 hits per second that meant six or seven heap allocations per frame. Most of them took 60 ns, but once in a while one crossed 40 microseconds and ruined the whole frame. A graph of averages never shows this, because 399 of those 400 allocations are cheap.
Not every spike like this comes from allocation, so the first job was separating the causes. We dropped in a counter over every operator new call inside the frame, and the frames where that counter peaked lined up one to one with the frames where frame time peaked. After that there was nothing left to argue about.
This is exactly why C++ memory management in a game engine is its own topic: the average cost does not matter, the worst case does. Inside a 16.6 ms budget a single tail latency misses the frame, and the player feels it in their hands before you see it in the numbers.
Why new/delete is dangerous inside a frame
A general-purpose allocator is not deterministic. malloc walks free lists looking for a fitting block, takes a lock, and sometimes asks the operating system for fresh pages. In that last case the cost jumps from nanoseconds to microseconds, and no amount of reading the code tells you which frame it will happen in. On Windows most of the spikes we measured coincided exactly with page faults.
The second problem is fragmentation. Over a long session, thousands of short-lived allocations in mixed sizes punch holes through the address space, and after a while a request of the same size costs more than it used to. In a 40-minute test session the same code path ended up taking twice as long as it did in the first minute of play. On console it is worse, because you do not have the same virtual memory slack there.
Third, in a multithreaded engine every allocation site is a hidden synchronization point. When four WorkerThread instances allocate at the same time, the allocator's internal lock serializes them. You do not see this as one clear stall block in the profiler; you see small delays spread everywhere, which is why it gets noticed so late.
The fourth point is measurement itself. The cost of a general-purpose allocator never gathers in one place, it spreads over hundreds of call sites, and none of them looks big enough on its own to draw attention. That is why memory management problems surface when you measure the total, not when you stare at any single system.
Frame arena: one reset per frame
An arena allocator, also called a linear allocator, is a simple idea: reserve one big block at startup, bump a cursor forward on every allocation, and never free individually. At the end of the frame you move the cursor back to zero and you are done. Our FrameArena opens with 8 MB, and an allocation costs one alignment step plus one add. We sized it at twice the measured peak usage and write the used byte count into telemetry at the end of every frame.
The rule is that only things which do not outlive the frame go into the arena: the visibility list, the temporary DrawCommand array, the pair list from the physics broadphase, the text geometry the UI builds for that frame. No pointer that crosses a frame boundary may come from the arena, or you will overwrite it next frame, and that bug is silent.
Reset() does not run destructors. That is why we only put trivially destructible types in the arena. Anything else needs a separate destructor list, which eats part of the speed advantage back; in practice we never went down that road. A static_assert at the allocation site enforces the rule at compile time.
Each worker thread got its own arena. That removed even the last atomic operation from the allocation path, and the hidden contention between threads disappeared completely. An arena that is never shared does not need a lock either. Four arenas add up to 32 MB, a trivial price next to what we gained in frame time.
Object pooling for fixed-size objects
The arena is useless for objects that outlive the frame. Projectiles, audio sources and particle emitters live for seconds and die in arbitrary order. For those we use a pool allocator: one array of equally sized blocks plus a free list pointing at the empty ones. Because the block size is fixed, fragmentation stops being a problem at all.
The nice part of a memory pool is that the free list can live inside the blocks themselves. A free block is not in use, so you write the address of the next free block into its first eight bytes; no separate data structure, no separate allocation. That makes Acquire() and Release() constant time and almost branch free.
We never hand out raw pointers to pooled objects. Every projectile gets a 32-bit index plus a generation counter; when the projectile dies the generation increments and any stale handle held elsewhere becomes invalid on its own. That turns use-after-free from a crash into a silent failure you can assert on before it does damage.
For ProjectilePool we picked a capacity of 4096; peak usage in the densest wave we measured was 2,870. When the pool fills up we do not grow it, we recycle the oldest projectile instead. Growing mid-frame would defeat the reason we introduced the arena and the pool in the first place. We revisit the capacity once per release, using the peak value telemetry reports back.
Cache locality, alignment and false sharing
The real win from changing allocators is not the allocation time, it is that the objects end up next to each other in memory. The 2,870 projectiles from the pool are contiguous, the update loop reads 64-byte cache lines in order, and the hardware prefetcher kicks in. When we scattered the same number of projectiles with new, the cache miss rate tripled.
Keeping the hot data small is the other half. Shrinking the Projectile struct from 96 bytes to 48 fit two projectiles per cache line and dropped the update loop from 0.9 ms to 0.6 ms. Moving cold fields such as the mesh pointer and the sound id into a parallel array was enough. The only thing that changed in that measurement was field layout; the maths stayed identical.
False sharing, on the other hand, is invisible until you measure it. Each of our four worker threads incremented its own counter, but the counters sat on the same cache line, so every write invalidated the line on the other cores. Separating them with alignas(64) bought 1.2 ms on that system, and it was a one-line change.
On the SIMD side alignment is not optional. The arena's Allocate function takes an alignment parameter; we pass 16 for a buffer holding __m128 values and 32 on the AVX path. Even on platforms where unaligned access does not crash, we measured it running measurably slower. Since the arena already returns aligned blocks, that check lives in exactly one place.
Is std::pmr enough, or your own allocator?
The measurement summary: in the combat scene average frame time went from 9.2 ms to 7.1 ms, but the real difference is at the 99th percentile. The peak dropped from 13.4 ms to 7.9 ms and the periodic spikes in the profiler vanished entirely. We repeated the same measurement on two different machines and the ratio held. The whole job took three weeks and fits into two header files.
Most of this you could have done with std::pmr. std::pmr::monotonic_buffer_resource is already an arena, and unsynchronized_pool_resource is already a pool. If you are working with standard containers, measuring first and then plugging a memory resource underneath them covers most of what you need, at close to zero maintenance cost. It also helps that it is a standard interface everyone on the team already recognises.
The cases where you must write your own custom allocator are narrow: hot loops where you do not even want a virtual call, systems where you store 32-bit indices instead of pointers, special memory regions on console hardware, and your own telemetry reporting the frame budget. If none of those four apply to you, stay with pmr.
If you only do one thing, start by counting the allocations inside a frame. Adding a counter to the engine and printing how many allocations each frame makes is half an hour of work, and the number that comes out is usually one nobody on the team guessed. The closer that number gets to zero, the easier everything else becomes.