The first two days of a project review

People sometimes ask what actually happens during a project review. I suspect the picture in their head is a man arriving with an enormous checklist and looking disappointed by lunchtime.

I do have lists. I have been doing this too long not to. But the first two days are mostly about establishing what is true before I develop a persuasive theory about it. The latter is much easier and much less useful.

I begin with one question for whoever commissioned the review:

If this review could answer one thing properly, what would you want it to answer?

The answer might be “Can this ship in March?”, “Why has the frame rate collapsed?” or “Do we need to change the technical leadership?” Those are very different reviews. It also helps to ask what has already been tried. There is little value in proudly discovering an option the team tested six months ago.

First, I want to see the game misbehave

I ask for a current build on the hardware it is supposed to ship on, then go to a representative bad area. If performance is part of the concern, I want frame-time numbers, a trace, a GPU capture and memory information. Averages are useful, but the awkward frames usually tell the more interesting story.

I also cold-boot the build and play into content I have not seen. That approximates the experience of a publisher, platform reviewer or new player rather better than loading the same level for the fifteenth time on a developer machine. Startup, first-use shaders, streaming and forgotten error paths tend to introduce themselves here.

Then I time the ordinary working loop. How long from syncing the project to having something playable? How long is an incremental compile, a cook, editor startup or loading the large production level? Ask three people how long the build takes and it is quite common to get three answers, all honestly given. That is a sign the pipeline needs measuring, not that two people are lying.

The repository gets a quick structural look as well. I am not trying to review every line in two days. I am looking for where complexity has collected: large engine changes, unusual module boundaries, third-party dependencies, generated code, script doing jobs nobody expected it to do, and systems which appear to have been extended for years without ever being simplified.

Commit history can be surprisingly informative. It shows where work is concentrated, which areas are effectively owned by one person, and whether the project changes in a steady flow or alarming milestone-shaped avalanches. None of that proves a problem, but it tells me what to ask next.

Then I talk to people separately

For a medium-sized team I might arrange six to eight half-hour conversations across engineering, art, design, production and QA. Not just leads. The conversations are confidential and they are not individual performance assessments; if people think I am secretly scoring them, I will learn almost nothing.

I tend to ask variations of these:

• What takes longer here than it ought to?
• What are you working around at the moment?
• If you had a free fortnight, what would you fix?
• If one person were ill for a week, where would work stop?
• What was the last thing that made the team change its mind?
• Who makes the final decision when disciplines disagree?

The exact wording matters less than the overlap. If engineering, QA and production independently describe the same release process, I pay attention. If one group says it is fine and another says it loses two days every milestone, I pay attention to that too.

Sometimes the team already knows the central problem and has been unable to say it in a room together. Sometimes everyone agrees on the symptom and has a different explanation. Both are useful findings, although only one saves me much detective work.

The plan and the game must eventually meet

By the second day I put the milestone plan beside the build. I want to know what the last few milestones promised and delivered, how much content remains, and which remaining work is production rather than unresolved invention.

“Eight levels left” is not enough. Have the systems been proved? Is the content mostly repetition of a known process, or does each level require a new mechanic? What does done mean? Where is the integration, certification and bug-fixing time? Is any slack visible, even under a false name?

The aim is not to catch production out. Plans decay because information changes, and game development contains a heroic quantity of new information. The important thing is whether the plan changes when reality does.

What I try not to do

I avoid prescribing a solution during the first conversation. Early solutions are seductive: every subsequent fact starts arranging itself around them.

I also try not to judge an old decision without reconstructing the conditions in which it was made. A technical choice can look absurd three years later and still have been the sensible option when the team had twelve people, no funding certainty and a different game.

By the end of day two I normally have facts, suspicions and disagreements. I keep those categories separate. Day three is when I chase the most important uncertainty, estimate the cost of the real problems and begin ranking them. The team should get a chance to challenge that ranking before it becomes a report.

A studio can run much of this process itself, and I think it should. The difficult part is rarely gathering the information. It is ranking problems when every option affects somebody’s plan, team or previous decision. An outsider has no magic there, but can sometimes say aloud the thing several people already know.