Pipeline · Money

Your build takes four hours. Which four hours?

Ask how long the build takes and you may get three different answers. Measure the working day instead: that is where the money is going.

Ask a team how long the build takes and three people may give you three different numbers.

One means the clean build on CI. One means the incremental cook they run at their desk. The third is thinking of the time between syncing in the morning and getting a version which actually starts. All three are right, which is not especially helpful.

I am less interested in the heroic four-hour number than in how much of an ordinary day is spent waiting. Suppose forty people each lose forty-five minutes to builds, cooks, editor startup and sync trouble. That is thirty engineer-hours every day - nearly four working days disappearing during each working day. The arithmetic gets rude quite quickly.

It also understates the effect. A ten-minute interruption is not only ten minutes. It is enough time to read a message, answer a question and forget why the debugger was open.

Measure the bit people actually wait for

Before changing infrastructure I would record a small set of timings on machines people actually use:

  • a clean and an incremental C++ build;
  • a change in a common header and a change in a leaf file;
  • a full and an incremental cook;
  • editor startup and loading a large production level;
  • sync-to-playable time after a typical team update; and
  • the equivalent CI jobs.

The spread matters as much as the average. If one person can iterate in six minutes and another needs thirty, the second probably does not need a motivational poster about productivity. There is a machine, cache, workspace or process difference to find.

I keep these numbers over time. Otherwise every pipeline discussion becomes a collection of vivid memories: “It was definitely quicker before the engine upgrade” competing with “No, that was before we added the city.” A graph is calmer.

Start with the derived data cache

In an Unreal project, the DDC is an obvious early check because it is both important and quite easy to have while not really benefiting from it.

Is the shared cache actually configured? Are people hitting it? How far away is it? Does an ordinary change invalidate an implausible amount of data? Does the cache retain useful work, or is it being pruned so aggressively that Monday morning begins with everyone reconstructing the world?

A cache-hit rate turns this from folklore into engineering. Low hits may be a configuration problem, unstable inputs or a workflow which creates globally different keys far too often. Merely buying a larger cache does not answer which one.

Shaders have a multiplication habit

Material switches look cheap when viewed one at a time. A few options on a widely used base material can produce a very large permutation set, after which every clean build becomes an opportunity to rediscover combinatorics.

I look for switches which are rarely used, combinations which cannot occur, and base materials which have become universal toolboxes. Reducing permutations can improve compilation, cook size, PSO collection and first-run behaviour at once. It is one of those rare jobs which pays rent in several places.

The useful question is not “Do we have too many shaders?” It is which choices created them and whether the flexibility is still earning its cost.

A cook should know what changed

If an incremental cook behaves like a full cook, I want to know why. Non-deterministic outputs, broad dependencies and build steps which always claim to be dirty are common causes. So is simply cooking far more content than the game can reach.

Unused content is not free because nobody loads it. It still has to be discovered, processed, stored, synced and sometimes submitted for certification. Asset ownership and packaging rules tend to be less exciting than new build hardware, but they often remove more work.

Hardware still matters. Fast local storage, enough memory and appropriately sized CI workers are sensible purchases when the measurements show they are limiting. I just prefer to avoid putting a faster machine underneath a process which needlessly rebuilds everything.

Compilation: the header changed, so the afternoon ended

For C++, compare a change in a leaf file with a change in a commonly included header. If the second causes most of the project to rebuild, include analysis usually finds some rather expensive convenience.

Forward declarations, cleaner module boundaries and moving implementation detail out of headers remain unglamorous and effective. Unity builds and precompiled headers are useful too, although they can conceal dependency trouble until a configuration changes or a platform build stops cooperating.

Distributed compilation is valuable when the work is genuinely parallel. It is not a cure for a dependency graph which declares the whole codebase affected by every edit. Sending unnecessary work to more computers is certainly faster; it is still unnecessary work.

The morning sync is part of the pipeline

Large depots, generated files in source control, unstable integration branches and frequent rebuild requirements can make “get latest and play” the longest build step of all.

This creates an unpleasant loop. Syncing is painful, so people sync less often. Their branches drift further apart, integration becomes riskier, and the next sync is worse. Eventually the team schedules integration days, which is a formidable achievement for a version-control system.

Measure sync-to-playable, not only transfer speed. A small download followed by an hour of local processing is not a fast sync from the user’s point of view.

Keep the gains

Once the obvious work is done, put the important timings into regular CI and make regressions visible. I would rather get a mildly annoying warning when the incremental cook grows by three minutes than rediscover the problem at the next milestone.

Set budgets for the working loop just as you would for frame time or memory. Someone should own them, and exceptions should have an expiry date. Otherwise every sensible short-term compromise becomes permanent archaeology.

The benefit is not only the hours recovered in a spreadsheet. When the game is quick to try, people try smaller changes, test on target more often and throw away bad ideas earlier. Slow iteration quietly changes creative behaviour: it encourages larger guesses and makes each one harder to abandon.

That is why I would not begin by insisting that a four-hour build become forty minutes. I would begin by asking which wait the team meets ten times a day. Fix that one first.

Pipeline trouble rarely needs a grand rewrite.

A performance review can measure the working loop, identify the dominant costs and leave the team with a ranked plan rather than a shopping list.

Robert Troughton

Thirty years in games: engineer, studio founder and former General Manager of Epic Games UK. I now advise the people building, funding and buying games. About Robert.

Previous: Four months of runway. Start with the first fortnight.Next: Your UE5 game runs at 24fps. What now?

Need an independent view?

Start with a free 20-minute fit call. No pitch.