← all talks

← → move · D light/dark · . blank

For everyone who reviews, person or machine

Eight Things a Review Comment Owes the Person Reading It

What two months of our reviews showed, and eight fundamentals that hold whoever writes the comment.


Tjakoen Stolk

If you missed the first talk, the short version

most of its work is throwing comments away gates, a skeptic that argues back, a hard cap it checks your fix at the new commit a "done" reply is not evidence an outcome log scores every comment fixed, answered, wrong or ignored The surprise was never accuracy. It was where a comment sat, and how heavy it was.

Mid-September to mid-November, one product

654 review comments on 53 pull requests about half from people, half from AI reviewers cited evidence AI about 90% human about 75% suggested a fix AI 34% human 10% Mostly good. The problems were weight, repetition and missing reasons.

Where the attention actually went

weight 105 blocking items, about 94 from AI 3 requested changes while saying "no blockers" the worst blocking ones gave a rule code and nothing else repetition ~60 near-duplicate pairs on the same PR a re-review restating its own points a second reviewer repeating the first a summary copying its inline comments Only a handful made a true nit block. A handful is enough to make "blocking" a mood.

From my own reviewer's outcome log

authors acted on it · 2,203 scored findings · about 99% right in every row blocking issue 98% inline comment 92% nit 46% review-body note 46% Same reviewer, same accuracy. The label decided whether anyone acted.

Eight fundamentals, whoever writes the comment

  1. 1A finding names its consequence
  2. 2Severity matches the consequence
  3. 3A nit never blocks
  4. 4One issue, one comment
  5. 5Check before you assert
  6. 6Say why, offer a fix
  7. 7Verify the fix
  8. 8Know when to stay silent

Fundamental one of eight

1A finding names its consequence

If you can't say what breaks and for whom, it's a question, not a finding.

Before
"Blocking: INTEGRATION_TESTS_NOT_UPDATED."
After
"The retry path has no integration case, so a regression ships unnoticed. Add one for hold, then retry."

Fundamental two of eight

2Severity matches the consequence

Blocking means users or data get hurt if this merges.

Before
Changes requested, then "No blockers, but..."
After
Post a comment. Test gaps go under "worth doing."

Fundamental three of eight

3A nit never blocks

Authors act on 98% of blockers and 46% of nits. Mixing them spends the first on the second.

Before
A "Steps to Test" placeholder in the list that held the merge
After
"nit: Steps to Test still has the template text."

Fundamental four of eight

4One issue, one comment

Read what is already there before you post.

Before
A second review restated a migration concern already open
After
"+1 on the open migration thread above."

Fundamental five of eight

5Check before you assert

Don't hand the author homework you could have done.

Before
"What about signed-in users?" without reading the branch
After
Read the branch, then drop it or point to the line

Fundamental six of eight

6Say why, offer a fix

A reason, then a fix. Only 1 in 10 human comments had one.

Before
"Could we drop this?"
After
"toBeInstanceOf(Date) passes on any date. Could this freeze time and assert the value?"

Fundamental seven of eight

7Verify the fix

A silently closed thread tells the next reader nothing.

Before
Thread resolved, nothing said
After
"9 of 11 threads fixed at the new commit. Two remain."

Fundamental eight of eight

8Know when to stay silent

Scoped-out work, untouched files, unrelated baselines: a ticket, not this review.

Before
Cleanup asks on a file the change never touched
After
A ticket, linked once, and nothing in the review

From real reviews · fundamentals one and two

1 · A finding needs a consequence

AI reviewer re-reviewCHANGES REQUESTED

Re-review of [PR] since [commit]. Fixed since the last review: 6. New: 1.

Blocking: INTEGRATION_TESTS_NOT_UPDATED

This was a recheck-only pass: new code in this push was not independently reviewed.

2 · Severity matches the consequence

AI reviewerCHANGES REQUESTED

No blockers, but the profile row shows “None” when it hasn’t checked anything, three test gaps would let wrong behaviour pass, and the last four commits are untested.

Findings: 0 block · 1 bug · 8 issue · 5 ask · 6 nit

From real reviews · fundamental three

3 · A nit never blocks

AI reviewerCHANGES REQUESTED

1 bug, 2 issues, 1 nit. One error-code collision, one a11y gap, one date-render mismatch.

…

Not on a diff line: nit: Steps to Test is still the template placeholder.

3 · Label it, so nobody guesses

Human reviewer on a service file

could we drop the export here ? nothing outside this file imports it (0 importers)

From real reviews · fundamentals four and five

4 · One issue, one comment

Human reviewer spec one

wouldn’t it be good to add a case with a null or key-less entry inside [setting list]? the migration filters on setting?.key, but the cases here only cover a missing, null or non array list, so that guard isnt covered.

Human reviewer spec two

same as the [customer] spec, should there be a case with a null or key-less entry inside [setting list] here too?

5 · Check before you assert

Human reviewerQ

when someone is still signed in, does core [login route] let them pick another account here ?

if it is: all good as is. if it isnt: core login sees the session and sends them right back to this page, which auto links again and shows the same refusal.

could we sign out first … and have the spec check that instead of just the href ?

From real reviews · fundamental eight

8 · A serious finding gets a serious first line

Human reviewer on a search query

[playful opener] this runs the regex over the whole users collection with no role filter and no limit, so a short term like “a” or “gmail” pulls back every matching jobseeker and staff id…

On a big tenant that’s a huge $in on every keystroke. Could this filter to employer users and cap the result?

From real reviews · what good already looks like here

a consequence you can test

AI reviewerISSUE (BLOCKING)

Nothing tests this filter. Delete line [N] and every spec still passes, while a representative goes on seeing a job in their list after its delegation has ended.

the reason, then the fix

Human reviewer

toBeInstanceOf(Date) passes on any date. Could this freeze time and assert the cancelledAt value?

user impact, labeled

Human reviewerNOT A BLOCKER

do you think we should show the max codes error here instead of slicing, pasting more than 50 just drops the extras without telling the user, not a blocker for me

a fix verified, in writing

AI reviewerPASS

[commit] settles every point from the last round. AC: 5 of 5 met.

Then one row per requirement, each with its evidence.

Real comments, openers removed. Two by people, two by AI reviewers.

If you are building a reviewer: every finding passes five gates

intentprior artproof exact linenecessity what the ticketasked for already raisedor answered? counted acrossthe service anchored whereit happens does the authorneed it? fail any one and it is dropped, not softened Most of a good reviewer's work is deciding what not to say.

Three more rules, each one sentence long

severity blocking = users or data harmed if merged. Else issue, nit or question. and the verdict has to agree with the text dedupe read the existing threads before posting anything the summary points to inline comments, it never repeats them evidence every claim cites a file and a line, or it is a question a guess phrased as a fact is the expensive kind of wrong

A starting point you can paste into your own prompt

Before posting any finding: - Drop it if the ticket asked for exactly this, if an existing thread already raises it, or if you cannot say what breaks and for whom. - Prove a "convention" by searching the whole service, not one file. - Cite file:line for every claim, or phrase it as a question. - blocking = users or data harmed if merged. Else issue, nit, question. The verdict must agree with the text. - Summaries link to inline comments. They never repeat them.

A prompt is a guess until you measure it

evals from real cases a real miss → a case it must raise a real false positive → a case it must drop rerun them after every change an outcome log every comment scored later: fixed, answered, wrong or ignored read per topic, never one average Without the log, you are guessing. The 98 and 46 came from one.

Keep building

The eight fundamentals are the spec for whatever you build.

Build your reviewer, measure it, and show the rest of us what it got wrong. That is how the next one gets better.


The judgment is human. The typing, by design, is not.

1 / 14 00:00

    Next

    Arrows here drive the main window · T resets the clock · the clock turns accent past 20 minutes