The first two days of a project review

People sometimes ask what actually happens during a project review. The picture in their head is often of someone arriving with an enormous checklist and looking disappointed by lunchtime.

There are lists. But the first two days are mostly about establishing what is true before developing a persuasive theory about it. The latter is much easier and much less useful.

The review begins with one question for whoever commissioned it:

If this review could answer one thing properly, what would you want it to answer?

The answer might be “Can this ship in March?”, “Why has the frame rate collapsed?” or “Do we need to change the technical leadership?” Those are very different reviews. It also helps to ask what has already been tried. There is little value in proudly discovering an option the team tested six months ago.

First, see the game misbehave

Ask for a current build on the hardware it is supposed to ship on, then go to a representative bad area. If performance is part of the concern, collect frame-time numbers, a trace, a GPU capture and memory information. Averages are useful, but the awkward frames usually tell the more interesting story.

Cold-boot the build as well, and play into content not seen before. That approximates the experience of a publisher, platform reviewer or new player rather better than loading the same level for the fifteenth time on a developer machine. Startup, first-use shaders, streaming and forgotten error paths tend to introduce themselves here.

Then time the ordinary working loop. How long from syncing the project to having something playable? How long is an incremental compile, a cook, editor startup or loading the large production level? Ask three people how long the build takes and it is quite common to get three answers, all honestly given. That is a sign the pipeline needs measuring, not that two people are lying.

The repository gets a quick structural look as well. The aim is not to review every line in two days but to find where complexity has collected: large engine changes, unusual module boundaries, third-party dependencies, generated code, script doing jobs nobody expected it to do, and systems which appear to have been extended for years without ever being simplified.

Commit history can be surprisingly informative. It shows where work is concentrated, which areas are effectively owned by one person, and whether the project changes in a steady flow or alarming milestone-shaped avalanches. None of that proves a problem, but it indicates what to ask next.

Then talk to people separately

Six to eight half-hour conversations across engineering, art, design, production and QA suit a medium-sized team. Not just leads. The conversations are confidential and they are not individual performance assessments; if people think they are being secretly scored, very little will be learned.

Useful questions are variations of these:

• What takes longer here than it ought to?
• What are you working around at the moment?
• If you had a free fortnight, what would you fix?
• If one person were ill for a week, where would work stop?
• What was the last thing that made the team change its mind?
• Who makes the final decision when disciplines disagree?

The exact wording matters less than the overlap. If engineering, QA and production independently describe the same release process, that is worth attention. If one group says it is fine and another says it loses two days every milestone, that is worth attention too.

Sometimes the team already knows the central problem and has been unable to say it in a room together. Sometimes everyone agrees on the symptom and has a different explanation. Both are useful findings, although only one saves much detective work.

The plan and the game must eventually meet

By the second day, put the milestone plan beside the build. The questions are what the last few milestones promised and delivered, how much content remains, and which remaining work is production rather than unresolved invention.

“Eight levels left” is not enough. Have the systems been proved? Is the content mostly repetition of a known process, or does each level require a new mechanic? What does done mean? Where is the integration, certification and bug-fixing time? Is any slack visible, even under a false name?

The aim is not to catch production out. Plans decay because information changes, and game development contains a heroic quantity of new information. The important thing is whether the plan changes when reality does.

What to avoid

Avoid prescribing a solution during the first conversation. Early solutions are seductive: every subsequent fact starts arranging itself around them.

Avoid judging an old decision without reconstructing the conditions in which it was made. A technical choice can look absurd three years later and still have been the sensible option when the team had twelve people, no funding certainty and a different game.

By the end of day two there are normally facts, suspicions and disagreements, and the categories should be kept separate. Day three is for chasing the most important uncertainty, estimating the cost of the real problems and beginning to rank them. The team should get a chance to challenge that ranking before it becomes a report.

A studio can run much of this process itself. The difficult part is rarely gathering the information. It is ranking problems when every option affects somebody’s plan, team or previous decision. An outsider has no magic there, but can sometimes say aloud the thing several people already know.