Review is where small teams stall. Not because nobody can read a diff. Because the first pass is unpaid work that happens after the real job. The author waits. The reviewer opens the PR cold. They comment on naming because the design takes more time than they have.
Adding AI to a review pipeline means the first pass is already there when a human arrives. The human still owns the merge. That is the whole design. Everything else is how you keep the first pass from becoming a second queue.
The pipeline you are aiming at
Write this on a card and stick it next to the deploy checklist.
- Author opens a pull request with a description of the risk.
- CI starts. The model reads the diff, not the whole repo.
- Comments land on the change: bugs, missing tests, auth edges, obvious regressions.
- Author treats the list like a self-review. Fix or dismiss.
- Human reviewer reads the remaining risk and the author’s note.
- Merge, or send it back for a real reason.
If step 4 is skipped, the human inherits a pile of nits and learns to ignore the bot. If step 5 is skipped, you have rubber-stamped a model. Both are how AI in the code review pipeline dies.
What the model is for
Give it a job a tired reviewer would thank you for. Not a job a staff engineer would resent.
Good first-pass work:
- Unhandled error paths in the code you just added.
- A test that is missing for the behavior in the description.
- Authz checks that do not match the new route.
- A migration that cannot run twice.
- A secret or token that showed up in the diff.
Bad first-pass work:
- Rewrite this in a more idiomatic style.
- Consider extracting a helper.
- This function is long.
- Have you thought about scale?
The second list is not always wrong. It is wrong as an automated comment on every PR. Humans can still say those things when they matter. The model should not spend the team’s attention budget on them. Attention is the scarce resource. See when AI review is noise if you already crossed that line.
A one-week experiment
Do not start with “roll out to the org.” Start with one repository and one week.
Day 1. Pick the repo that already has PR review, even if review is slow. Write the seat: trigger on open and on push, scope is the diff, owner is the author. Tell the team in the channel you already use. One message. No all-hands.
Day 2–4. Every new PR gets the first pass. Authors must reply or fix before they ping a human. “LGTM, bot had nothing” is a valid reply. “I ignored it” is also valid if they say why.
Day 5. Look at the last ten PRs. Count comments the author kept. Count comments everyone scrolled past. If the keep rate is low, narrow the job. Turn off style. Turn off files the linter owns. Turn off generated code.
After the week. Either keep the seat or remove it. A half-wired bot that nobody trusts is worse than no bot. The cost is not the subscription. The cost is a team trained to mute review.
How to wire it without a science project
You want comments on the pull request, in the same thread humans use. A Slack dump of “the model said…” will not get acted on. A PDF will not get acted on. The review UI is the queue.
For most small teams that means a GitHub (or GitLab) app that posts inline comments. The author sees them next to the line. The reviewer sees what was already handled. That is the product shape worth paying for.
Mergestorm is the tool we recommend for that loop: run a review on one open PR, read the comments, decide. You do not need to rebuild your host or move off GitHub to try it. If the week-one experiment is the goal, start there, not with a custom agent in Actions.
If you later want the same job inside CI, that is a different seat. Do not collapse them on week one. AI in CI for small teams covers when a check should be a status, not a comment.
Mistakes that look like progress
Requiring the bot to be green. A model will be wrong. If merge is blocked on its feelings, authors will game the prompt or force-push until the bot goes quiet. Keep the human as the gate. Let CI own tests and lint.
Reviewing the prompt instead of the diff. Teams burn a week on the perfect system prompt. The useful work is choosing scope and reading ten real PRs. Tune after you have scars, not before.
One bot, five repos, no owner. The comments show up. Nobody is on the hook to dismiss them. They rot. Install on the repo you ship this week. Name the owner in the PR template: “Author handles AI comments before requesting review.”
Letting the model merge. Auto-merge after a model approval is a different product. It is not a first-pass review pipeline. If you do not already auto-merge on human + CI, do not start with a model in that seat.
What the human still does
The model does not know if this change should exist. It does not know that you are migrating off a vendor, or that the ugly branch is load-bearing. It does not feel the customer. That is the reviewer’s job.
A good human pass after a first-pass model looks like this: skim the AI comments and the author’s replies. Then ask one or two questions about the risk. Then approve or request changes. You should spend less time on “you forgot a null check” and more time on “this cache will be wrong after the backfill.”
If the human pass is still a full line-by-line because you do not trust the bot, the bot is too noisy or the tests are too weak. Fix those. Do not add a second model.
Start with a PR that already exists
Open the oldest non-draft pull request on the repo you ship. If it has been sitting, that is the one. Run a first-pass review on it. Do the author loop. Merge or close.
That is how you add AI to a review pipeline. Not a rollout plan. One PR, a named seat, a human who still merges. If you need the definition of the path this sits on, start with what is an AI code pipeline.