A GPU particle system in HLSL: past the million mark
I moved particle state fully onto the GPU with StructuredBuffer and DrawProceduralIndirect, went from 40k to a million, and measured what it cost.
The frame that collapsed at forty thousand
On an action prototype I worked on last year, up to thirty explosions could be on screen at once. When the CPU-side particle system reached forty thousand live particles, the profiler showed 11.4 ms on the main thread. Our frame budget was 16.6 ms, and that number did not yet include gameplay logic, animation or physics.
It was obvious where the time went. Every frame we walked forty thousand Particle structs, updated position and velocity, then copied the same data into a vertex buffer. The Mesh.SetVertexBufferData call alone cost 2.1 ms. Burst and the Job System brought the update part down to 3.8 ms, but the copy stayed exactly where it was.
My first attempt was to simply use fewer particles: I halved the count per effect. Frame time dropped to 7.9 ms, but the explosions looked thin and the art team objected, correctly. Cutting the count is not a solution, it is an admission that the problem won.
The real problem was not the cost of the maths, it was the data travelling between CPU and GPU. If only the GPU ever reads a particle's position, there is no reason for that position to live in CPU memory. I moved the whole system to the GPU with an HLSL compute shader; what follows is the technical detail of that move and the numbers it produced.
Keeping state inside a StructuredBuffer
Particle state lives in a single struct: float3 pos, float3 vel, float life, uint seed. That is 32 bytes, so I never had to add padding for alignment. A million particles is 32 MB; because I ping-pong, the two buffers together mean 64 MB of VRAM. The seed field is written once at birth and keeps a particle's randomness deterministic for its whole life.
I read through StructuredBuffer<Particle> and write through RWStructuredBuffer<Particle>. Reading and writing the same buffer in one dispatch is undefined behaviour, because nothing tells you the order in which thread groups will run. In ParticleSystemGPU.cs I swap the two buffer references every frame; no memory is copied, two variables just change places.
Holding the struct at 32 bytes pays off measurably. As an experiment I added a half4 color field, went to 40 bytes, and the update kernel jumped from 0.82 ms to 1.19 ms on an RTX 3060. The colour could already be derived from lifetime, so carrying it in the buffer bought nothing. On the GPU the bottleneck is usually memory bandwidth, not arithmetic.
I allocate with GraphicsBuffer rather than ComputeBuffer. Both work, but GraphicsBuffer lets me mark the same allocation as both a compute target and an indirect draw argument. Using one allocation for two purposes is faster than keeping a second copy, and it removes a class of mistakes.
Why I pick 64 or 128 threads per group
The kernel starts with [numthreads(128, 1, 1)], because the hardware already runs in waves. A warp is 32 threads on NVIDIA, a wave is 64 on AMD. If your group size is not an exact multiple of that, part of the last wave spins for nothing, and that waste is measurable.
I measured the same kernel with a million particles at four sizes: 1.41 ms at 32 threads, 0.91 ms at 64, 0.82 ms at 128, 1.05 ms at 256. Thirty-two is bad because the fixed per-group cost — bounds check, constant buffer reads — repeats far too often. Two hundred and fifty-six is bad because my kernel uses 40 registers, and higher register pressure lowers how many groups fit on an SM at once.
Capacity is rarely an exact multiple of the group size, so the first line of the kernel is if (id.x >= _Capacity) return;. Forget it and the last group writes past the end of the buffer; on Windows you usually get silently wrong results, and sometimes a TDR driver reset. I compute the dispatch count with Mathf.CeilToInt(capacity / 128f), and rounding capacity up to a multiple of 128 makes that check almost free.
A million particles in groups of 128 comes to 7,813 groups, far below the 65,535 limit on a single axis, so I never needed to spread across a second dimension. Past roughly four million you do hit that ceiling and have to bring id.y into play. I did not write that up front; generalising for a case I did not have made the kernel unreadable.
Recycling dead particles with append/consume
A particle that runs out of life has to hand its slot to a newborn one. I do that with AppendStructuredBuffer<uint> _DeadList: the dying particle appends its own index. The spawn kernel pulls indices from the same list through ConsumeStructuredBuffer<uint>. That way I never write a scan loop looking for a free slot.
Append and consume sit on top of an atomic counter, so every call queues up behind the others. If all particles die in the same frame, that counter becomes the bottleneck: in a synthetic test where I killed everything at once, the kernel went from 0.82 ms to 1.26 ms. In real scenes deaths spread out over time and the difference stayed under 0.05 ms, so I left it alone.
The spawn kernel runs as its own dispatch, before the update kernel. With the order reversed, newborn particles got updated once in the same frame and drifted one frame ahead; invisible in motion, but it left a small hole at the centre of each explosion. I pass the spawn count from the CPU as a single int, since gameplay logic decides that number anyway.
There are two classic traps here. The first is allocating the buffer without the ComputeBufferType.Append flag; the shader compiles, runs, the counter stays at zero, and you get no error message at all. The second is resetting the counter in the wrong place: I call SetCounterValue(0) only on the live-index buffer, every frame, right before the dispatch. Apply the same call to the dead list and you erase the free slots you accumulated, and nothing ever spawns again.
DrawProceduralIndirect: never going back to the CPU
The GPU knows how many particles are alive; the CPU does not. Asking for that number with GetData means waiting for the command queue to drain — when I measured it, a single readback added 4–6 ms to the frame. So I draw with Graphics.DrawProceduralIndirect and the count never visits the CPU.
The argument buffer is four uint values: vertex count, instance count, start vertex, start instance. Every frame I call GraphicsBuffer.CopyCount(_aliveIndices, _argsBuffer, 4) to copy the live counter into the second slot. The CPU has no idea what that number is; it only writes a copy command into the queue.
There is no mesh on the vertex side either. I derive the particle index and the corner index from SV_VertexID and build the quad inside the vertex shader, binding _ParticlesOut and _AliveIndices there as StructuredBuffer. Vertex buffer binding, index buffers, mesh updates — all of it disappears.
Here is the measured difference. The CPU system spent 11.4 ms of main thread time on forty thousand particles; the GPU system spends 0.82 ms in compute, 1.6 ms in drawing and 0.05 ms on the main thread for a million. Twenty-five times the particles, roughly eight times less frame time. More importantly the CPU went idle, and we handed that budget to the AI.
Synchronisation traps and where to start
GroupMemoryBarrierWithGroupSync() synchronises only its own group. There is no cross-group synchronisation inside a dispatch, and that is the single most confusing thing when you move to the GPU. If neighbouring particles need to read each other — collision, flocking, a neighbourhood grid — that step has to become a second dispatch.
The second trap is ordering. The order inside _AliveIndices changes every frame, because nothing guarantees which group finishes first. Draw alpha-blended particles in that order and the image flickers frame to frame; I was not the one who spotted it first, it was obvious in a slow-motion capture QA sent over. Two ways out: sort by depth on the GPU (a bitonic sort measured 0.9 ms for a million elements) or switch to additive blending. I took the second one, since most of the effects were additive already.
Your debugging habits have to change too. There are no breakpoints and no Debug.Log. I open a separate RWStructuredBuffer<float4>, write the intermediate values I am suspicious about into it, and read it back only while hunting a bug. That readback costs the same 4–6 ms, so it lives inside a #if DEBUG_PARTICLES block.
Do not try to move the whole system in one go. Take a single effect first — for me it was bullet trails — put it on the GPU with a fixed-capacity buffer, measure the dispatch in RenderDoc, and only then add the dead list and indirect drawing. Picking a million as your capacity up front is pointless too: measure the peak live count in a real scene and take double that, and VRAM and update time both land in the right place.