Test generation is one of the most frequently requested features in automated coding tools, and it is also one of the most frequently oversold. Generating code that calls a function with a valid input and asserts the expected output is not hard. Getting a passing green line in your coverage report is not hard. Writing tests that would actually catch a bug someone will introduce six months from now is a different problem entirely.
We have been building Pylon-Test for about a year. This is what we have learned about the gap between "generates tests" and "generates tests that matter."
The happy path trap
The simplest way to generate a test for a function is to read the function signature, call it with a typical input, and assert on the typical output. This is what most code-generation tools do when you ask for tests. It produces a test that passes immediately and will keep passing because it only exercises the path the function was written for.
Consider a function that takes a list of order items, applies a discount if the total is above a threshold, and returns a final price. A happy-path test calls it with a list of three items totaling above the threshold and asserts the discounted price. That test will never catch a bug where the discount is applied before tax instead of after, or where floating point rounding causes an off-by-one-cent error on prices that end in exactly 0.5, or where an empty list throws an unhandled exception instead of returning 0.
Those are the bugs that matter. They are the ones that end up in production. They happen at edge cases, boundary conditions, and inputs that the original developer did not think about when writing the function.
Coverage gap analysis
Pylon-Test starts from coverage data, not from the function signature. Before generating a single test case, it imports your existing coverage report and identifies which branches in the function under test are not currently covered. A branch is a coverage unit: each side of every if statement, each case in a switch, each edge in a try/except/finally, each early return path.
For each uncovered branch, the agent reasons about what input would be required to reach it. This is a constraint satisfaction problem: given the code logic up to the branch, what conditions on the input values lead execution there? For simple conditional logic, this is straightforward. For functions with multiple nested conditions, it requires tracing through the possible input space more carefully.
The result is a set of test cases targeted at specific gaps in your existing coverage. Each generated test has a comment that explains which branch it is targeting and why that branch matters. The comment is not documentation padding. It is a human-readable explanation of the scenario: "tests that the discount is not applied when the total equals exactly the threshold value, verifying the boundary condition is exclusive not inclusive."
Mutation testing as verification
A test that never fails is not useful. Coverage tools will report a line as covered if the test executes it, but that does not mean the test would fail if the code were broken. An assertion like assert result is not None covers the happy path and passes even if the function returns the wrong value.
We verify generated tests using a form of mutation testing. After generating a test suite, we introduce a set of controlled mutations into the code under test: flip a comparison operator from < to <=, remove an early return, change a constant, negate a conditional. For each mutation, we run the newly generated tests and check whether at least one test fails. If a mutation survives all tests, the tests do not adequately constrain the function's behavior at that point.
When a mutation survives, we flag the corresponding test gap and attempt to generate an additional test case that would catch that specific mutation. This iterative loop runs until either the mutation is killed or we hit a limit on test generation attempts. When we cannot kill a mutation, we report it in the PR body as a test coverage gap that warrants human attention.
This approach has a cost: mutation testing is slow, because it requires running the test suite once per mutation. We run it in a sandboxed environment against a limited set of mutation operators, not an exhaustive mutation space. The goal is not to reach 100% mutation coverage; it is to verify that the generated tests are non-trivially exercising the code.
The test naming and structure problem
Generated tests often have terrible names. test_function_1, test_case_a, test_apply_discount_v2. These names do not communicate what behavior is being tested, which means when the test fails in CI, the developer reading the failure has to open the test to understand what went wrong.
Pylon-Test generates test names using a structured naming convention: test_[function]_[scenario]_[expected_outcome]. The scenario description comes from the branch analysis comment: "when total equals threshold," "when list is empty," "when discount rate is zero." The expected outcome is derived from the assertion: "returns zero," "raises ValueError," "applies no discount." A test named test_apply_discount_when_total_equals_threshold_applies_no_discount tells you what it does without reading the body.
We also enforce test structure: each generated test follows the arrange-act-assert pattern, with explicit setup, a single call to the function under test, and a single assertion. Tests that combine multiple assertions about multiple behaviors are harder to debug when they fail. A test that asserts five things gives you one failure mode that might implicate any of five code paths. We prefer more tests with fewer assertions each.
What Pylon-Test does not do
We do not generate integration tests or end-to-end tests. Pylon-Test targets unit-level coverage gaps in functions and methods. Tests that span service boundaries, involve real database calls, or require live network connections are a different problem, and generating those incorrectly creates more confusion than they remove.
We also do not attempt to generate tests for functions that are tightly coupled to external state, such as functions that read from global variables or functions whose output depends on the current time without an injectable clock. For those, we report the coverage gap and note the testability problem. A test for such a function usually requires refactoring the function first, and the appropriate response to that is to open a refactoring issue rather than generate a test that requires global state manipulation.
Generated tests require human review before merging, like any other generated code. A test suite that passes mutation testing and covers all branches is still code that a human engineer should read. Tests encode assumptions about expected behavior, and those assumptions should be verified by someone who understands the requirements, not blindly accepted because an agent wrote them.