This video is about two kinds of tool we use to check code. Formal tools are narrow and exact: they apply fixed rules to specific conditions. These include tests, type checkers and linters. The fuzzy tool is the coding agent itself, used as a reviewer. It can address broad questions, but its verdict can vary even on the same code. We'll see how checks let an agent write, run, read a failure and try again, and when that loop is worth setting up.
Tests, types and linters were developed to help people write software. People make mistakes, forget earlier changes, and cannot hold a whole codebase in mind. If you haven't used a linter before, it is a program that reads your code without running it and reports patterns that are legal but suspicious, such as an import that is never used.
Agents also make mistakes and lose track of earlier changes. Even when an agent is very good at a task, we need a way to catch the errors it makes.
So checks that catch mistakes help people and agents alike, because both of them make mistakes that a check can find.
Now look at what an ordinary session ends with. The agent writes some code and tells you it is done. We still need to check that the change did what we asked.
One candidate is you. You run the program, or read the diff, and work out whether the change did what you asked. That costs your attention on every change.
Then you say what is wrong, and the agent tries again. This is the loop most people are in on their first day with an agent.
Every turn of that loop waits on a person. It runs at human speed, and it stops the moment you look away.
Here is the loop we would rather have. It starts the same way, with the agent writing some code.
Then it runs a check, such as the test suite, the type checker, or the linter. The check needs to give a result the agent can use without waiting for you to inspect the program.
If the check fails, the agent reads the failure. A good failure names which case broke and what was expected, far more specific than a person saying it does not work.
The agent edits the code and runs the check again. Sometimes the check is wrong, so compare an unexpected failure with the requirement before changing either the code or the test. If the same check keeps failing, or the requirement itself is unclear, the agent should stop and ask you rather than keep trying.
When the check passes, the agent can bring you the change together with that result. It can repeat the checking and repair between your reviews, and you then review whether the work meets the task. The check does not make the model cleverer. It tells the agent what failed while the change is still fresh, instead of leaving the defect for a later debugging session.
The checking loop needs feedback the agent can rely on when deciding what to fix. A deterministic check gives the same answer for the same input under the same conditions. Those conditions include anything the check depends on besides the code, such as the clock, random numbers, or an external service. A test that reads the current time or calls a live server can change its answer while the code stays the same.
Tests, type checkers, linters and compilers all work this way. We call them formal tools. Formal here does not mean a mathematical proof. It means a program applies fixed rules, and two properties follow from that. They are narrow: they check only the conditions they were given. And they are exact: each condition either passes or fails, with no judgement call in between.
A formal tool applies its rules independently of how confidently the agent describes its work. That gives the loop a stable result to work from.
The fuzzy tool we referred to earlier is the coding agent itself, or the language model behind it, asked to review your code. It can answer almost any question you ask about the code, which makes it broad.
Remember that a language model is stochastic. It samples its output, so asking the same question about the same code several times can give different answers. Here the code has not changed at all. One run says it is correct, another finds a missing case, and a third only suggests a new name.
The agent also may not read everything or check everything. It can skip a file that matters, or say a case was checked when nothing was run.
So a verdict from the agent can change while the code stays the same. That makes it useful as a reviewer, but not a stable check for the loop.
The two kinds of tool help with different questions. On the left are conditions we have encoded as checks: expected behaviour, compatible types, and project rules. On the right are broader review questions: whether names are clear, the structure makes sense, or an important case is missing.
The formal tool checks the conditions you gave it exactly, and covers nothing else. A test suite for expected outputs does not tell you whether the design is confusing.
The coding agent, used as the fuzzy tool, reaches across both kinds of question, which is why we want it. But its answer can be wrong even where a fixed check was available. Its breadth is useful when it suggests a problem you had not thought to test, like the empty list that one of the reviews found. Once you decide what an empty list should give, that expected result can become a test.
When you can state a condition exactly, check it with a deterministic tool rather than asking the agent. The check gives the same answer on every run. It is also cheaper. Running a local test suite takes seconds of computer time, while a model review is paid for in generated tokens, which costs far more. So run the deterministic checks first, and ask the agent about the questions they leave open, such as naming, structure and missing cases.
Reviewing a change takes your attention each time. What you learned helps with the next review, but you still have to inspect the new work.
An automated check can be reused. Once written, it can run after each change, often in a few seconds. You maintain it when the requirement or the surrounding code changes.
A reusable check reduces what you have to verify by hand after each change. You decide once what matters, and then the check keeps testing it.
Tests are not free. Asking the agent to write them means more work for the agent, which takes more tokens and more time on every task.
Tests are also more code to maintain. When a requirement changes, the tests that describe it have to change too.
These costs are easy to see. The return comes in two parts. A test can find a defect in the change you are making now. And because tests accumulate, it keeps checking that behaviour on every later change.
Take a small to-do app that gains three features: adding items, completing them, and saving and loading the list. After all three, one run of the suite checks each of them, because the tests for the earlier features stay in the repository.
A change can break a feature that used to work. That kind of failure is called a regression. Here, adding save and load broke adding items, and the tests for adding catch it. Nobody has to test the earlier features again by hand. Passing those tests gives you some confidence that what worked before still works. The same holds for a bug you fix. Keep the test that reproduced it, and if the bug comes back, that test fails again.
The old tests cannot see everything. A new feature can interact with an earlier one in ways those tests never tried. Completing an item works, and saving and loading a list works. Now complete an item, save the list, and load it into a fresh app. Select the Active filter, and the completed item shows up among the active ones, because the saved form left out whether each item was done. No existing test did those steps together, so that combination needs a test of its own.
Old tests catch regressions in the behaviour they check. They do not check combinations they never exercise, so when features share state, test them together and keep that test.
A suite of checks buys more than protection against regressions. First, a fresh session can run the suite to check requirements established in earlier sessions. Those checks remain in the repository even when the conversation is gone.
Second, repeatable checks reduce how much behaviour you have to verify by hand when reviewing a change.
There is a case where checks become especially important. An agent will write a program in a language you have never used, but you lose the ability to review it fluently. You can still state expected behaviour and run tests while learning enough to read the code or getting help reviewing it.
None of this means every project should have tests. Think of it as a spectrum of repeated use. At one end, a prototype you delete on Thursday. At the other, a repository three people push to every day.
Checks cost setup, tokens, running time and maintenance. They buy dependability and let the agent do more work unattended. A short-lived prototype reruns its checks only a few times before it is deleted. In a shared repository, every change reruns them.
Ask whether this code has a future: will you or someone else keep changing it? The more often a useful check will run, the more reason there is to build it. Also consider the cost of getting this particular task wrong. A one-off script that converts important data may run only once, but a mistake there is expensive, so it still deserves a careful check.
Formal tools are narrow and exact about specified conditions. Fuzzy tools are broad, but their judgments can vary. Together they cover the checks we can specify and the broader questions we still need to review.
Checks you keep go on testing earlier requirements after every change, so nobody has to recheck them by hand. Their limit is what they never exercise. When features share state, their combination needs a test of its own.
There are three common kinds of formal check. Tests compare behaviour with expectations. Types check that values fit where they are used. Linters check selected patterns and project rules.
At the start, the agent said it was done, and you had to check every change yourself. With checks in place, the agent can run them, read what failed, fix it, and bring you a change that already passes. Your review can then focus on what the checks do not cover.
Happy testing. See you in the next video.