AI engineering
AI-paired engineering, part one: the workflow
Series opener. Where the model helps, where it lies, and how we built a daily rhythm around both.
Part ofai-pairedLoading essay…
Series opener. Where the model helps, where it lies, and how we built a daily rhythm around both.
Part ofai-pairedLoading essay…
A year of pair-programming with an LLM in production has taught us something the demos never did: the model is excellent at one part of the work, average at another, and dangerous at a third. The trick is the workflow that puts each in its proper place. This is part one of three on AI-paired engineering after the demo is over.
Strip the marketing. Day-to-day, the model does three things on our team:
It is bad at three things:
We landed on a four-step pattern, in this order:
1. Spec → 2. First pass → 3. Edit + test → 4. Review
you model you (mostly) model
Notice the bookends. The model's role is in the middle. The human defines the problem and validates the result. Anything else and you are paying for a confident wrong answer.
The spec is a structured document. We use a four-section template:
## Goal
One sentence.
## Inputs
Types, constraints, edge cases worth naming.
## Outputs
Types, what "success" means, what "failure" returns.
## Examples
Three: one happy path, one edge case, one failure case.This is not optional. The spec is what makes the difference between "the model produced something useful" and "the model produced plausible-looking nonsense." Most engineers who say "the model doesn't write good code" skipped step 1.
The model takes the spec and produces a first draft. We do not edit mid-stream. We do not ask follow-up questions. We let it finish, then we read it as a complete proposal.
The rule we landed on: if the first pass is more than 30 % wrong, we do not iterate with the model on it. We rewrite the spec to be tighter and try once more. Iterating with the model on a bad first pass tends to produce three more bad passes that get progressively further from what we want.
We edit the first pass by hand. We write the tests by hand. The model helps with test names but not with the assertions. The reason: the model's tests have a tendency to assert what the code does rather than what the spec said.
After we are done, we paste the diff back into the model with a fresh context and ask "what edge cases did we miss?" The model's hit rate here is around 1 in 5: most of what it suggests is already covered or out of scope, but the 1 that lands is sometimes a real bug.
Three numbers, monthly:
The last number is the one that taught us the most. The model does not remove bugs from the system. It moves them upstream into the spec. That is fine if you treat the spec as the artifact, not the code.
Three patterns we have abandoned:
Letting the model write code, run tests, fix failures, run tests again, in an unattended loop. This works for trivial cases and fails silently on anything important. The failure mode is that the model "fixes" tests by weakening them. We have not found a guardrail that catches this reliably.
Long back-and-forth with the model on the same chunk of code, 10+ turns. This produces code that is structurally weird because the model is averaging across its own earlier suggestions. Better: write a fresh spec, fresh first pass.
The model answers yes. It is a confident speaker. Worse, it sometimes hallucinates issues that do not exist. Replace the open question with a structured one: "list the edge cases in this code that are not covered by the tests below." The structured question gets useful output; the open one does not.
Part two of this series is about spec-driven development specifically: the structure of a spec the model can act on without rework, the anti-patterns, and an actual diff from a feature we shipped this way.
Series page tracks the rest.