Tests give an agent repeatable checks of its work. This video looks at what those checks cost. A test suite is code that you have to run, review and maintain. Some tests are worth improving or deleting. We'll look at those costs, including brittle tests, flaky tests and mocks. Then we'll deliberately break a rule to see whether the tests detect it, and look at mutation testing, line coverage, and how to decide which tests to keep.
A test suite is code. It gets read, refactored and repaired, just like the program it checks. The tests give us repeatable checks of expected behaviour. To keep that feedback useful, we need to maintain the suite as well as the implementation.
Now change one behaviour. You edit the program, because that is the change you meant to make.
You may also need to change tests that described the old behaviour. One requirement change can therefore mean work in both the program and the suite.
When you estimate a change, include the tests that need updating and review. They are part of the code you maintain.
Follow one test through its life. Writing it happens once, and on the day you write it, that is the whole cost.
After that, the test runs on every change, for as long as it stays in the suite. Someone reads it each time it fails, and updates it each time the behaviour it describes changes. Most of what a test costs comes after it is written. So when you add a test, you are also agreeing to maintain it.
A change-detector test is tied so closely to the implementation that harmless restructuring makes it fail. Here is a test for a shopping cart. It adds a book that costs twelve, and then reads the cart's internal list of items directly. The underscore at the start of the name marks that list as internal to the cart. How the cart stores its items is an internal choice. A user only sees what is in the cart and what it costs.
Now change the cart to keep its items in a dictionary instead of a list. The cart holds the same book, the total is still twelve, and no user could tell the difference.
The test goes red anyway. It was never checking what the cart contains or what it costs. Without meaning to, it was checking that the cart keeps its items in a list. That is a choice made inside the code, and no user can see it.
Google describes tests like this as having negative value. A harmless internal change now requires a test update too, without telling you that any behaviour broke.
Kent Beck, who popularised test-driven development, the red, green, refactor cycle from earlier this week, has a short rule for this. Couple tests to behaviour, and keep them independent of internal structure. The blue bar on the left edge of the program is its public interface, the functions or module interface that callers use. A good test calls the program through it. That does not require every test to open a browser.
The other test depends on one particular internal arrangement, which is what coupling to structure means. It reaches past the public interface to check how the work is organized.
Now reorganize the code while preserving its public behaviour. The test through the public interface still passes. The test tied to the old internal arrangement fails, even though the behaviour is unchanged.
Couple to behaviour and decouple from structure. Tests written this way let you refactor the implementation while continuing to check what its callers need.
Back to the cart. The first test reached into the cart's own list, so a harmless change made it fail. The second test calls the cart's public methods, items and total, the way the rest of the program does. Keep the items internally in a list or in a dictionary, and it passes either way. It fails only when the behaviour changes, which is the failure you want a test to report.
A flaky test sometimes passes and sometimes fails without a change to the code. That makes a failure harder to interpret. In 2016, Google reported flaky results in about one and a half percent of test runs, with about sixteen percent of tests showing some flakiness.
About eighty-four percent of the observed transitions from passing to failing involved a flaky test. A failure often required investigation to distinguish a regression from an unreliable test run.
If rerunning becomes the routine response to a failure, people can stop investigating what went wrong. Nondeterministic tests can undermine trust in the whole test suite.
So repair the cause. If a test depends on the current time or on random input, control them, for example by passing in a fixed clock or a fixed seed. If it waits a fixed number of seconds for something to happen, make it wait for that condition instead. While you work on the fix, you can isolate the test so it stops blocking everyone, and then restore it once it is reliable.
A suite can also grow slow enough that you stop running it during edits. A twenty-minute wait puts distance between the change and its feedback. A slow suite can still be worth keeping. Run a fast set of tests after each edit, and the full suite less often, for example before merging.
Writing a suite by hand takes time, and the work continues when someone reviews and maintains it.
An agent can draft hundreds of tests quickly, perhaps four hundred before lunch. You still need to read them, because a test can pass while checking the wrong thing. It may reach into the code's internals, like the cart test. Or it may check whatever the code does today, bugs included.
Much of the effort now moves from writing tests to reviewing them. Fast generation can leave hundreds of tests to read and maintain, so the amount of generated code is not a measure of what you gained. Ask for tests in groups small enough to review. Read each new test, and look closely at any change to a test you already rely on.
One particular way a generated suite goes wrong is worth its own slide. Real code talks to things outside itself, like a database, a clock, or a payment service.
Tests can replace these external dependencies with mocks. A mock is a stand-in object that a test passes in place of the real one, and it answers however the test tells it to. A fake payment service lets a unit test check payment handling without making a real payment. That is a useful mock.
Here is the harmful case. This test is meant to check that the checkout total includes tax. But it replaces our own tax calculation with a mock, and tells the mock to return one hundred and eighteen.
The assertion then checks for one hundred and eighteen, the number the test itself configured. The real tax calculation never runs, so the test would pass even if that calculation were wrong or deleted.
Now let's look at how to assess the tests themselves. You can trust a test once you have seen it fail for the right reason. In test-driven development, that is the red step: the test is written first, and fails before the code exists. When tests are written after the code, as an agent often writes them, you can still get that evidence. Deliberately remove the rule the test should catch, and run the suite. This function decides whether a new to-do item can be added. The highlighted line rejects an item that is empty after trimming its spaces. Before we touch anything, all forty-two tests pass.
Temporarily remove that line. The program now accepts a blank item, so a test covering that situation should fail.
In this illustrative run, all forty-two tests still pass. None of them tried to add a blank item, so the suite missed the deliberately introduced defect.
So add the missing case. This test tries to add an item made only of spaces to an empty list, and expects the answer to be false. With the rule still removed, the new test fails. That failure is exactly what we wanted to see.
Now restore the rule and run the suite again. All forty-three tests pass. The new test fails without the rule and passes with it, so we know it checks the rule. This tells you something a coverage percentage cannot. This is the red step, done after the fact. When the agent writes tests after the code, ask it to show you each new test failing with the rule removed, and passing once the rule is back.
Doing this by hand for every line would be slow. A tool can make these deliberate changes for you. It makes one small change to the code at a time, and runs the tests against each changed version of the program. Each change is called a mutation, and the whole technique is called mutation testing. For Python, a tool called mutmut does this. For JavaScript and TypeScript, there is Stryker.
A mutation detected by the tests is called killed. One that passes is called survived. A survivor may reveal a gap in the tests, though some mutations leave the behaviour unchanged. Inspect what changed before deciding a new test is needed. In practice, ask the agent to run the tool on the code it just changed, and to bring you the mutations that survived. You then decide, for each one, whether a test is missing.
Google's 2021 paper describes mutation testing used by more than twenty-four thousand developers across more than a thousand projects.
Running every possible mutation at that scale costs too much. Google's system focuses on changed code and filters the mutations, so developers see a small number that are likely to identify useful missing tests.
When the agent writes the code and the tests from one reading of your request, a misunderstanding in that reading can end up in both. The tests then agree with the code, and the suite stays green over a broken feature. Here the requirement says the total includes eighteen percent tax, and the function forgot it. The test computes its expected answer with the same calculation as the function. This is called a tautological test.
The expected answer omits tax in exactly the same way as the function. This test passes with the bug, and would fail if you corrected the function to include tax.
An independent expectation comes from the requirement instead. Two items at fifty each come to one hundred, and eighteen percent tax makes one hundred and eighteen. A test that asserts one hundred and eighteen fails on the current code, and points straight at the missing tax.
A test can also have a name that its assertion does not check. This one is named as though clearing completed items removes them. It builds a list with one completed item and one open item, and clears the completed ones. But its assertion only checks that some result came back. It would pass even if nothing were removed.
Compare the assertion with the behaviour named by the test. Here that means checking that only Call mom is left after clearing. Read the starting state, the action and the expected result to see whether the test covers the behaviour it claims to cover.
Line coverage tells you which executable lines ran during the test suite. It is a measure of which code was exercised, not of what was checked. In one direction it is reliable. If no test ever ran a line, or ever called a function, that code was not tested at all, and adding tests that reach it is worth doing.
This test calls initials on Alan Turing and prints what it returns, with no assertion. Every line of the initials function runs, so its line coverage is one hundred percent.
Now suppose initials returned lowercase a t instead of capital A T. The coverage is still one hundred percent, and the test still passes, because nothing compares the result with an expected value.
Low coverage helps you locate untested code. High coverage shows that many lines ran, but you still need to inspect what the tests checked. Use coverage to guide that inspection.
You don't have to count lines yourself. In Python, a plugin called pytest-cov measures coverage while pytest runs your suite. Add it to the project as a development dependency. Then run pytest with two extra options. The first turns coverage on for the project. The second asks for a report that lists the missing lines.
The report has one row per file. It shows how many statements the file has, how many never ran, and the percentage that did. The last column is the most useful one. It names the lines that no test ran. Here, lines forty-two to forty-seven of the to-do module were never executed.
Those missing lines are where to look first. You can ask the agent to run the report and explain the gaps, but read the report yourself as well. For the lines that did run, go back to the assertions. As we just saw, a line that ran was not necessarily checked.
Deleting or improving a test is a normal maintenance decision. Here are four tests for the initials function. Let's go through them one at a time and decide whether each should be kept, improved or removed.
The first two check the same kind of case, a first name followed by a last name. The second adds no distinct case and would fail for the same reasons, so it can go.
The third also calls initials, but with an empty name. That is a different case with its own expected result, so it stays, even though it calls the same function.
The fourth runs initials and checks nothing. It adds coverage but cannot fail on a wrong answer.
Improve it by adding an expected value. Now a wrong answer makes it fail.
Keep a test that checks a distinct case you care about. Improve one whose assertion misses the behaviour. Remove one that repeats another. For a generated suite, keep the tests whose expectations you understand and whose failures would tell you something useful.
A test suite is worth what it catches. Keep the tests that catch the mistakes you care about, and don't be shy about letting the others go. Thanks for watching, and I'll see you in the next video.