Eight Things a Review Comment Owes the Person Reading It
What two months of our reviews showed, and eight fundamentals that hold whoever writes the comment.
Tjakoen Stolk
Twenty minutes. The shape: what we found, why it matters, eight fundamentals, then how to make your own reviewer follow them.
Say early that this is not about anyone's reviews in particular. Nothing on these slides names a person or a pull request.
The title is the whole argument: every comment spends the author's attention, and a bad one also teaches them to read the next one less carefully.
If you missed the first talk, the short version
Under a minute. The people who saw it get a reminder; the people who did not get enough to follow.
The last line is the bridge. Today is about weight, for everyone, not just the machine.
The full story is in the written version, Seven Hundred Commits to Teach a Reviewer It Was Wrong.
Mid-September to mid-November, one product
Flag it as a snapshot of one product over two months. It is a sample, not a verdict on the team.
Say the good part first and mean it. Accuracy is not where we are losing.
The fix-suggestion gap is the easiest one for people to close, and it comes back at fundamental six.
Where the attention actually went
None of the duplicates came from someone being careless. Each one is a reasonable habit, and they stack on one author.
The AI share of blocking items is not an indictment of AI. It is what happens when a severity rule is not written down, and it is fixable in a prompt.
Do not name or describe a specific pull request if asked. Point them to the pattern instead.
From my own reviewer's outcome log
This is the slide the title is about. Authors trust the label. When it says blocking, they act almost every time.
So every nit dressed up as a blocker spends credibility the next real blocker needed.
Honest limit: this does not prove the label causes the gap, because a nit is also less worth doing. It shows the two move together.
Snapshot as of this week. The counts keep growing.
Eight fundamentals, whoever writes the comment
1A finding names its consequence
2Severity matches the consequence
3A nit never blocks
4One issue, one comment
5Check before you assert
6Say why, offer a fix
7Verify the fix
8Know when to stay silent
This is the map. One slide each, then real comments from our own reviews.
Fundamental one of eight
1A finding names its consequence
If you can't say what breaks and for whom, it's a question, not a finding.
Before
"Blocking: INTEGRATION_TESTS_NOT_UPDATED."
After
"The retry path has no integration case, so a regression ships unnoticed. Add one for hold, then retry."
If you cannot name what goes wrong and for whom, it is not a finding yet. It might be a question.
Fundamental two of eight
2Severity matches the consequence
Blocking means users or data get hurt if this merges.
Before
Changes requested, then "No blockers, but..."
After
Post a comment. Test gaps go under "worth doing."
The verdict and the text have to agree. A request for changes that says no blockers teaches people to ignore the verdict.
Fundamental three of eight
3A nit never blocks
Authors act on 98% of blockers and 46% of nits. Mixing them spends the first on the second.
Before
A "Steps to Test" placeholder in the list that held the merge
After
"nit: Steps to Test still has the template text."
The 98 and 46 from the outcome log are why. Mixing the labels spends the credibility of the first on the second.
Fundamental four of eight
4One issue, one comment
Read what is already there before you post.
Before
A second review restated a migration concern already open
After
"+1 on the open migration thread above."
A summary points at a thread, it never copies it. For a machine reviewer, the better version is often nothing at all.
Fundamental five of eight
5Check before you assert
Don't hand the author homework you could have done.
Before
"What about signed-in users?" without reading the branch
After
Read the branch, then drop it or point to the line
If you have not checked, it is fine to ask. Say it is a question, so nobody mistakes it for a finding.
Fundamental six of eight
6Say why, offer a fix
A reason, then a fix. Only 1 in 10 human comments had one.
Before
"Could we drop this?"
After
"toBeInstanceOf(Date) passes on any date. Could this freeze time and assert the value?"
Could we drop this leaves the author guessing whether it gates the merge. Only 10 percent of human comments carried a fix, and this is the easiest number in the talk to move.
Fundamental seven of eight
7Verify the fix
A silently closed thread tells the next reader nothing.
Before
Thread resolved, nothing said
After
"9 of 11 threads fixed at the new commit. Two remain."
When someone says done, check the code at the new commit and say so in writing.
Fundamental eight of eight
8Know when to stay silent
Scoped-out work, untouched files, unrelated baselines: a ticket, not this review.
Before
Cleanup asks on a file the change never touched
After
A ticket, linked once, and nothing in the review
The hardest one for a good reviewer, because everything on that list is real. It is just not this pull request's business. And a serious finding gets a serious first line.
From real reviews · fundamentals one and two
1 · A finding needs a consequence
AI reviewer re-reviewCHANGES REQUESTED
Re-review of [PR] since [commit]. Fixed since the last review: 6. New: 1.
Blocking:INTEGRATION_TESTS_NOT_UPDATED
This was a recheck-only pass: new code in this push was not independently reviewed.
AI reviewer another pull requestMEDIUM
INADEQUATE_TESTS The [helper] branches for a bare-string array entry and for an object entry with no url (or a null entry) are never exercised…
A regression in how those entries are handled, such as throwing on a null entry, would not be caught.
Add specs where [field] is a JSON array containing a bare string url, an object without url, and null.
2 · Severity matches the consequence
AI reviewerCHANGES REQUESTED
No blockers, but the profile row shows “None” when it hasn’t checked anything, three test gaps would let wrong behaviour pass, and the last four commits are untested.
bug (inline): the profile row shows “None” before it has checked anything.
Worth doing: three test gaps, and the four untested commits.
Left: the rule code is real and so is the gap, but the comment never says what breaks. Flip it to see the same rule written well on another of our pull requests: the branch, the regression it would miss, and the specs to add.
Right: zero blockers, and a request for changes anyway. The author believes the verdict and stops trusting the text.
From real reviews · fundamental three
3 · A nit never blocks
AI reviewerCHANGES REQUESTED
1 bug, 2 issues, 1 nit. One error-code collision, one a11y gap, one date-render mismatch.
…
Not on a diff line: nit: Steps to Test is still the template placeholder.
RewrittenCHANGES REQUESTED
Held for one thing: the error-code collision (inline).
The a11y gap and the date mismatch are issues, not blockers.
nit, holds nothing: Steps to Test is still the template placeholder.
3 · Label it, so nobody guesses
Human reviewer on a service file
could we drop the export here ? nothing outside this file imports it (0 importers)
RewrittenNIT
nit: could we drop the export here? Nothing outside this file imports it (0 importers). Not blocking.
suggestion · illustrative
- export function [helper](…) {
+ function [helper](…) {
Left is a good review with one misplaced line: the template nit sits inside the summary that holds the merge.
Right is a good comment, with evidence even. It only lacks a label, so the author can’t tell whether it gates anything.
From real reviews · fundamentals four and five
4 · One issue, one comment
Human reviewer spec one
wouldn’t it be good to add a case with a null or key-less entry inside [setting list]? the migration filters on setting?.key, but the cases here only cover a missing, null or non array list, so that guard isnt covered.
Human reviewer spec two
same as the [customer] spec, should there be a case with a null or key-less entry inside [setting list] here too?
Rewritten one threadISSUE
Both [customer] specs miss a null or key-less entry inside [setting list]. The migration filters on setting?.key, so that guard isn’t covered in either. One case per spec:
suggestion · illustrative
+ it('skips a null or key-less entry', …)
5 · Check before you assert
Human reviewerQ
when someone is still signed in, does core [login route] let them pick another account here ?
if it is: all good as is. if it isnt: core login sees the session and sends them right back to this page, which auto links again and shows the same refusal.
could we sign out first … and have the spec check that instead of just the href ?
RewrittenQUESTION
I read [login route] at [line]: a signed-in user is sent straight back here, which links again and shows the same refusal.
Could we sign out first, and have the spec assert that instead of the href?
Left: the second comment is a fair cross-reference, but it opens a second thread for one gap. The happy ending is real: the fix landed in both specs, and the re-review said so.
Right: a good instinct, labeled as a question, which is right. But both branches of the “if” were readable, so the reviewer could have walked it and handed over an answer instead of homework. The rewrite assumes that reading; it is illustrative.
From real reviews · fundamental eight
8 · A serious finding gets a serious first line
Human reviewer on a search query
[playful opener] this runs the regex over the whole users collection with no role filter and no limit, so a short term like “a” or “gmail” pulls back every matching jobseeker and staff id…
On a big tenant that’s a huge $in on every keystroke. Could this filter to employer users and cap the result?
RewrittenISSUE (BLOCKING)
This runs the regex over the whole users collection with no role filter and no limit, so a short term like “a” or “gmail” pulls back every matching jobseeker and staff id. On a big tenant that’s a huge $in on every keystroke. Filter to employer users and cap the result.
suggestion · illustrative
- users.find({ email: { $regex: term } })
+ users.find({ role: 'employer', email: { $regex: term } }).limit(20)
The original opened with a few playful words, removed here so nobody is recognized by them. The finding underneath is one of the best in the sample: scope, the input that triggers it, the cost and the fix.
A data exposure that reads like a joke gets treated like one. Warmth is fine on a nit; save it for there. The diff is illustrative, not the real query.
From real reviews · what good already looks like here
a consequence you can test
AI reviewerISSUE (BLOCKING)
Nothing tests this filter. Delete line [N] and every spec still passes, while a representative goes on seeing a job in their list after its delegation has ended.
the reason, then the fix
Human reviewer
toBeInstanceOf(Date) passes on any date. Could this freeze time and assert the cancelledAt value?
user impact, labeled
Human reviewerNOT A BLOCKER
do you think we should show the max codes error here instead of slicing, pasting more than 50 just drops the extras without telling the user, not a blocker for me
a fix verified, in writing
AI reviewerPASS
[commit] settles every point from the last round. AC: 5 of 5 met.
Then one row per requirement, each with its evidence.
Real comments, openers removed. Two by people, two by AI reviewers.
All four are ours, from the same two months. The fundamentals aren’t new; they’re just not every comment yet.
If you are building a reviewer: every finding passes five gates
This is the part I want people to steal. Write each gate down as a rule in your prompt or skill, not as a vibe.
Proof is the one that catches the most. One neighbor in the same file is not a convention; a grep across the service is.
Three more rules, each one sentence long
Severity is the rule that would have removed most of the 105. Tie it to harm, never to how sure the model feels.
Dedupe is cheap: one read of the existing threads before posting. It would have removed most of the sixty pairs.
A starting point you can paste into your own prompt
Say plainly that this is a starting point, not my whole reviewer. It will not make a reviewer good by itself; the next slide is what does.
Share the text in the team channel after the talk rather than asking people to copy it off the wall.
A prompt is a guess until you measure it
The log does not need to be fancy. A file with one row per comment and a column you fill in later is enough to start.
Keep "was it right" and "did anyone act" as two numbers. Fused, an ignored correct comment looks exactly like the reviewer being wrong.
Keep building
The eight fundamentals are the spec for whatever you build.
Build your reviewer, measure it, and show the rest of us what it got wrong. That is how the next one gets better.
The judgment is human. The typing, by design, is not.
Say it straight: the AI workflows people on this team are building are good, and I want more of them, not fewer.
The ask is to keep the loop of teaching going. When your reviewer is wrong, write it down and share it, the way the first talk did.
Take questions.
1 / 14
1 / 1400:00
Next
Arrows here drive the main window · T resets the clock · the clock turns accent past 20 minutes