This video is about testing. We'll define what a test actually is and what an assertion adds, run a suite and read its report, and turn a bug into a test before fixing it. Then we'll ask an agent to write tests, and look at the kinds of tests people distinguish. We'll also try writing a test before the implementation.
The word test comes up constantly when we work with coding agents, so let's define it. The tests in this video are programs that run your code on an input you chose in advance. Our example is a function called initials, which takes a name like Alan Turing and returns A-T.
Running the function shows what came back. To judge that result, we also need the answer we expected, written down before we ran anything.
The test compares the two. If they match, the test passes quietly. If they differ, it fails, and the test runner reports the mismatch.
This comparison is a formal tool at work. It asks one narrow question and answers it exactly. The point of a test is that it can fail. When you first write one, watch it fail for the reason you expect. That shows it really checks the behaviour you meant.
Here is a first attempt, in Python with pytest and in TypeScript with vitest. Nothing in this video depends on which language you use. Both versions call initials and print what comes back. If initials returned Z-Z, would either test fail?
Neither one compares the result with anything. It fails only if the call crashes. If initials returned the wrong letters, both tests would still pass.
An assertion adds the missing comparison. It checks the returned value against the expected one, and fails the test when they differ.
Printing the result is useful while you explore or debug. An assertion turns the expected behaviour into a check the runner can evaluate for you.
So how do you actually run your tests? You have probably done this in earlier courses, so here is a short refresher. In Python, run pytest. In TypeScript, run npx vitest run. Each runner finds and runs the tests selected by the project's configuration. If you need more than this refresher, look up the documentation for pytest or vitest.
The runner also returns an exit status. Zero means every test passed, and a non-zero status means something failed. Other commands can act on that status. In this line, the double ampersand runs git push only if pytest returned zero, so this command does not push when a test fails. The agent can use that status to decide whether to continue, then read the failure report to find out what needs fixing.
While you fix one thing you want the run to stop at the first failure. In pytest that's dash x, and in vitest it's dash dash bail one.
Here is a shortened pytest report with one failing test. The first line says which tests the runner selected. It collected two. Next comes one mark per test, a dot for a pass and an F for a failure. For the failure, the report shows the assertion that did not hold, with both values. Here initials returned the two letters in lower case, and the test expected them in upper case. The last lines name the failing test and count the results. The vitest report has the same parts.
A report can also show an error instead of a failure. Here the test file imports a module called names that does not exist, so the runner could not even load the tests.
The two mean different things. A failure is a verdict on the code under test. A collection error is a setup problem, and no test ran, so the report says nothing yet about initials. Fix the setup first.
Now let's use a test to fix a bug. Suppose someone types a name with two spaces between the words, and initials crashes. Splitting on a single space leaves an empty word, and taking its first letter raises an index error.
Before fixing the bug, write the smallest test that reproduces it and check that it fails for the expected reason. Here the test fails with the same index error, so it reproduces the crash. That gives you a concrete case to fix. It also gives the agent something to run in a loop. The agent can edit the code, rerun the test, and keep going until the test passes.
Then you fix the code. Splitting on any run of whitespace drops the empty word. The new test turns green, and the original test still passes.
Once the test passes, keep it in the suite so later runs can detect the same bug returning. This is called a regression test. As you fix bugs, you build a record of cases that must keep working.
Let's look at asking the agent to write tests. A request like this leaves it to decide both the scope and what correct behaviour means.
Suppose it returns forty tests. You now have to read them to find out what they cover, and the request gave you little basis for judging what matters.
A better request does three things. It says what must stay true rather than asking for tests, names the cases you care about, and asks for the smallest failing case first. A request like this one is how you would get the double-space test from the agent.
You decide which properties matter and resolve ambiguity in the requirement. The agent can suggest cases and write the tests, but you should review the expected answers. A specific request makes that review much easier.
Writing the test before the code is a practice with a name. It's called test-driven development, or T-D-D. It asks for a lot of discipline, and many developers found it too demanding to keep up by hand. With an agent, you can ask for each step and check that it happened. Each small behaviour goes through three steps. First, red. You write the test before the code exists, you run it, and you watch it fail. Ask the agent to show you that failing run and explain why it failed. It should fail because the behaviour is missing, not because of a setup error like the missing module we saw earlier.
Second, green. You write the least code that makes that test pass. Not the design you imagine you'll need later, just enough to turn it green. Ask the agent for the passing run and the diff of what it changed.
Third, refactor. Clean up the structure while preserving behaviour. The tests protect the cases they cover as you move things around, which is what makes this way of refactoring safer.
Then the next behaviour starts the cycle again. For initials, one cycle could be the double-space case, and the next could be a name typed in lower case. Each small change starts with a failing test for a behaviour we want. The growing suite preserves those cases while we refactor. We still need to look for inputs and feature combinations we have not tested. A test you have never seen fail is a test you have not tested. Watch it fail for the expected reason before relying on the passing result.
The order matters even more with an agent. If the tests come after the code, the agent can read the implementation and use it to decide what the expected result should be.
If it copies a mistaken behaviour into the expected answer, the test passes while the bug remains. The code and the test then agree because they share the same misunderstanding.
So split the work. Derive the expected answer from the requirement, from what you decided the program should do. You write the failing test, or you read the one the agent proposed and approve it. Then the agent writes only the code that makes it pass, and you read the diff. The agent does the part it's fast at, and you keep the part that decides what correct means.
Writing the test first does not prevent every shared mistake. If the agent misunderstood the request, the test it proposes and the code it writes can both contain the same mistaken expectation. That is why you review the expected answer against the requirement. The reviewed test also gives you a grader for new work. Before the implementation exists, you have specified the answer it must produce for this case.
Expect a particular failure mode here. Give an agent a test it cannot pass, and it has two ways to turn the run green.
The first is to fix the code, which is what you wanted. The second is to change the test, which also turns it green.
Here are two examples. In the first, the test file is untouched, and the code returns A-T for that one input. In the second, the double-space test now uses a single space, so it no longer exercises the bug. Models have been observed removing assertions, hardcoding results or special-casing the test input instead of implementing the general behaviour.
So when the agent reports that the tests pass, read the test diff as well as the implementation diff, and look for special cases in the code. A changed expectation is justified when the requirement changed. If the requirement is the same, a weakened test only makes the failure disappear.
Now that we have a procedure, let's look at how people sort tests. One common grouping is by how much of the system a test exercises. A unit test checks a small piece in isolation, such as one function called directly in memory. A simple unit test like our initials tests can run in a millisecond or less, and its small scope helps you locate a failure.
An integration test puts several pieces together and checks they still work as a group, your code and a real database, say. The database has to be running and set up before the test starts, and a failure has more than one suspect.
An end to end test uses the whole application through its external interface, the way a user or another program reaches it, for example by clicking through the pages in a browser. A browser test needs the server, its data and the browser ready before it can start. It can also fail for something other than a defect in the behaviour you wanted to check, such as an animation still running.
Start with the program's rules, where there are many cases to check and plain function calls make that inexpensive. Add broader tests for the connections between those rules and the interface. Spend testing effort where a mistake is likely or costly, using the smallest test that can catch it.
Two other sorts are worth knowing. These ways of sorting overlap. The double-space regression test is also a unit test, and a small test in the terms we're about to see. Google sorts by what a test may touch rather than by layer. Small is one process, no network, no filesystem. Medium is one machine and localhost. Large may reach another machine.
People also argue about how many tests of each kind a suite should have, and two of the answers have names. One approach is called the testing pyramid. It has many unit tests at the base, fewer integration tests above them, and very few end to end tests at the top. The other is called the testing trophy. Its base is static analysis, which checks the code without running it, as type checkers and linters do. Above that, it spends most of the effort on integration tests.
Neither shape is right for every project. The shape is a choice about where to spend your testing effort. It weighs the risk in each part of the system, the cost of writing and running each kind of test, and how easily those tests break when the code changes.
There are many, many kinds of tests. To get a feeling for the field, let's mention a few. Knowing the names helps you recognize a technique when someone proposes it, and ask the agent for one when you need it. Property-based testing states a rule that must hold and generates inputs trying to break it. Fuzzing does the same, hunting for crashes. Snapshot and golden tests compare against a saved, approved output.
Visual regression tests compare screenshots with saved pictures. Performance and load tests measure speed under different workloads, while soak tests run for longer periods. Contract tests check that services agree on their interface.
Security scanning looks for problems in code, dependencies and exposed secrets. Chaos engineering deliberately disrupts infrastructure to examine recovery. Accessibility checks look at issues such as keyboard access and what a screen reader receives.
Formal verification proves a property over all cases of a model of the system, rather than sampling inputs. Manual exploratory testing is a person trying the system to find problems nobody automated. Each of these kinds answers a specific risk. For example, a parser that accepts input from strangers is a good candidate for fuzzing. It is enough for now to know they exist, and to reach for one when you can name the risk it addresses.
We hope we've convinced you that testing is worth the effort. You can now give the agent a specific behaviour to test, and check whether its test catches the mistake. Try it out. Happy testing!