Game development CI/CD: why nightly builds change everything
Setting up build automation on a small team took five days; the average lifetime of a broken commit dropped from 2.5 days to 9 hours and release days calmed down.
The Friday night build that broke
Last year, on the Friday evening we were due to send a demo to the publisher, my desktop was the only machine that could produce a build. There were four of us on the project and I was the only one who knew how to make one. At 19:40 it died with Shader error in 'Custom/Water': undeclared identifier '_TimeScale'. That line had been committed three days earlier and nobody noticed, because the shader variant never compiled in the editor.
Finding it took 25 minutes, fixing it and rebuilding took another 70. To locate the offending change I had to walk a range of 14 commits by hand. The demo went out at 23:10. The delay was not the bad part; the bad part was that for three days none of us knew whether the project actually compiled.
Our game development CI/CD setup came out of that evening. The goal was never an elegant pipeline: it was answering which commit broke it in one night instead of three days. We started on GitHub Actions because the repository was already there, then moved to Jenkins when we added a second build machine.
What a nightly build actually changes
The only job of an automated build is that a working version of last night's code is waiting when you sit down in the morning. Ours triggers at 03:00, finishes around 06:20 and drops one line into Slack: commit hash, duration, package size, test result. We sent a longer report for two months and I never saw anyone open it, so we went back to one line. It runs on weekends too; finding a broken repository on Monday morning is far better than finding it on Friday evening.
The most concrete win I can measure is this: the average lifetime of a broken commit fell from 2.5 days to 9 hours. The earlier a break is caught, the fewer commits pile on top of it, so you read one diff instead of running git bisect. The cost of fixing it drops too, because whoever wrote the code still remembers the context.
The second win is on the non-programming side. Because the machine produces a playable exe every morning, "it worked on my box" mostly disappeared; the level designer and QA open the same build. Preparing releases by hand used to cost me about 4 hours a week, and those hours came back too. The build we send to the publisher on Friday is now ready on Thursday night, and the only task left for the last day is writing release notes.
Command line builds in Unity and Unreal
The Unity side looks trivial, but the first trap is the -quit flag. In a -batchmode -nographics -quit -executeMethod chain the editor can close before your async work finishes, and the exit code is always 0. We drop -quit and call EditorApplication.Exit(code) at the end of the method instead, so a compile failure actually turns CI red. Routing the log to stdout with -logFile - matters just as much; write it to a file and the error never shows up in the CI interface.
We read the scene list for a build from our own BuildProfile ScriptableObject rather than EditorBuildSettings.scenes. The reason is practical: that list is an asset, it produced merge conflicts constantly, and twice a test scene shipped inside a build without anyone noticing. The same asset carries the define symbols and the version number, so what a build was made from lives in exactly one place.
On Unreal everything runs through RunUAT.bat BuildCookRun. A full pass with -clientconfig=Development -cook -stage -pak -archive takes 48 minutes; with -iterativecooking unchanged assets are skipped and it drops to 11-14 minutes. We do not trust iterative cooking in the nightly run, only in pull request builds. The nightly pass also uses -nocompileeditor, which saves another 6-7 minutes.
Perforce, Git LFS and cache discipline
A 40 GB repository is not what Git was designed for. With Git LFS our .psd and .fbx files turned into pointers, yet a fresh machine still needed 55 minutes for its first clone. Sparse checkout plus lfs.fetchexclude brought that down to 12 minutes. Asking artists to pull only the folders they need did not work; nobody who has to remember a command remembers it.
The real problem is locking, not download time. When two people touch the same Environment_Rocks.fbx there is no such thing as a merge; somebody loses their work. We moved to Perforce not because it is technically superior, but because the artists already knew it and exclusive checkout is the default there. Licences and server upkeep are a real yearly cost, but a cheap one next to the work hours we were losing. My threshold is simple: under 20 GB Git LFS is enough, above it use Perforce.
On caching the numbers are brutal: regenerating a clean Library/ folder takes 26 minutes on our project, while the actual compile takes 9. So Library/ stays on the build machine, but not blindly: Library/PackageCache is wiped whenever the Unity version changes, and once a week we run a fully clean build. Otherwise a bug the cache was hiding shows up months later on release day. We moved that clean build to Sunday night, because we are not giving those 26 minutes back on a workday.
DerivedDataCache plays the same role in Unreal and we keep it on a shared network folder. Shader compilation costs 38 minutes cold and 4 minutes with a warm DDC. Reading over the network is slower than local disk, so the machine keeps a second, local DDC layer as well. Once the folder passed 180 GB we added a job that prunes old entries; when the disk fills the build does not fail, it just quietly gets slower, and it took us two weeks to notice.
Smoke tests and a performance regression gate
By automated testing I do not mean a comprehensive suite. Our SmokeTest scene boots the game, enters the first level from the main menu, replays 30 seconds of recorded input and exits cleanly. Even that caught 6 of the 9 build breaks we had in a year, and reported them at 07:00. The other 3 only showed up on a specific GPU or in the second level; instead of growing the smoke test for those we added two items to the QA morning checklist.
The second gate is performance. We collect frame times in the same scene and look at p99 rather than the mean; the target is 16.6 ms with a p99 threshold of 22 ms. An average hides a single 40 ms hitch, and that hitch is exactly what players complain about. We start measuring after the first 120 frames, because shader warm-up lands in that window and inflates p99 into nonsense.
A single build over the threshold does not turn the pipeline red, two consecutive ones do. Even though the machine runs nothing else and we take the median of three measurements, roughly 6% noise remains; set the threshold tighter than that and within a month the team starts ignoring the alerts. We also keep the artifact of a failing build, so it can be profiled against the previous night instead of disappearing. We keep the p99 of the last 14 nights in a plain CSV file, because the trend tells you more than any single number.
Shipping builds to Steam and TestFlight
Distribution is the easiest step technically and the most irritating one operationally. Uploading to Steam is one line, steamcmd +run_app_build app_build.vdf, but unless you log in manually once on the build machine and keep the ssfn file Steam Guard leaves behind, the pipeline stalls there every night. Nightly builds go only to a password protected internal branch. A 9 GB depot upload takes about 7 minutes, and after the first one only changed chunks go over the wire.
On iOS the chain is xcodebuild -exportArchive followed by xcrun altool. The one rule that matters is binding CFBundleVersion to the CI run number; send the same build number twice and App Store Connect rejects it, with the error arriving by email 20 minutes later. TestFlight processing takes anywhere from 8 to 20 minutes, so it is not ready before your morning coffee. Installing the certificates and provisioning profiles into the machine keychain and adding a security unlock-keychain step was the biggest time sink of that first week.
The whole setup took one person 5 days: 1 day for command line builds, 1 day for the machine and runner, 1.5 days for caching and version control, 1 day for tests, half a day for store uploads. Keep that order. In the first week build a pipeline that only answers "does it compile" and drops the artifact into a folder; leave smoke tests and store uploads for later months. The pipeline needs maintenance of its own, about half a day a month for us, and if you do not budget that line it rots within six months. Nobody builds a complete system on day one, and half a pipeline that runs beats a whole one that does not.