Most demonstrations of AI coding agents run on clean codebases. There is a project with a clear structure, well-named files, 80 percent test coverage, and consistent conventions throughout. The agent performs well. The demo is impressive. Then the same team tries to run the agent on their actual production codebase, which has been in development for eight years and has layers of decisions made by engineers who left three companies ago, and the performance drops to something between disappointing and alarming.
We built Pylon specifically for the messy reality, which means we spent the first year of development learning what "messy" actually means at the code level. The lessons are not obvious from the outside, and I think sharing them is useful both for teams evaluating Pylon and for other people building agents that need to work on real production code.
Legacy codebases are not just old code
The first thing we had to unlearn was treating "legacy codebase" as a synonym for "old code." Old code is actually the easy part. Python 2 syntax is well-defined. jQuery 1.x patterns are well-documented. Rails 4 conventions are known. A tool that understands these patterns can translate them mechanically.
What makes a legacy codebase hard is not the old code itself. It is the accretion layer: the adapters, shims, workarounds, and patches that were added over years to keep the old code running in a changing environment. A Python 2 codebase in 2024 has usually been on life support for several years, which means it has been partially monkey-patched to work with newer infrastructure, has some modules that were quietly rewritten by individuals without formal migration, and has a test suite that was written against specific behavior that may or may not reflect the intended behavior of the original code.
An agent that sees the accretion layer as part of the code to be migrated will produce incorrect output. The accretion layer has to be identified and treated differently: some of it should be removed as part of the migration, some of it should be ported forward, and some of it should trigger a stop-and-report because it encodes decisions that need human review. Distinguishing these three categories requires understanding intent, not just structure.
Implicit conventions are invisible in source files
Every long-lived codebase has conventions that are not written in linting rules or documented in README files. They live in the patterns that repeat across the codebase: how database transactions are structured, how errors are logged versus surfaced to callers, how configuration values are injected versus hardcoded in specific contexts.
These conventions are invisible to an agent that reads individual files in isolation. They only become visible when you analyze patterns across the full codebase: functions that appear in similar positions in multiple modules, imports that always appear together, error handling that takes the same shape across different services.
We spent significant time building the convention inference component of the context graph, which looks for patterns that repeat enough to be intentional rather than coincidental. A pattern that appears 30 or more times in a codebase is almost certainly a convention. When Pylon generates code as part of a job, it samples the convention patterns relevant to the file it is modifying and uses them to calibrate its output. The goal is that the code Pylon writes looks like it was written by a member of the team that built the codebase, not like it was written by a system that read the documentation.
We are not saying convention inference is solved. It is one of the harder ongoing problems in the product. But we have enough signal that we can distinguish between the three or four main error handling conventions in most codebases and apply the right one for a given file.
Test coverage tells you where the risk is
One of the most practically useful insights from working on real codebases is that test coverage is a better risk signal than code complexity. A complex function with comprehensive tests is relatively safe to touch; if the change breaks something, the tests will catch it. A simple function with no tests and high downstream usage is genuinely risky; there is no safety net.
Pylon uses test coverage as a primary input to scope decisions. A job that requires changing a well-covered module proceeds with normal confidence. A job that requires changing an untested module triggers one of two responses: either Pylon-Test runs first to add coverage before the primary job executes, or the primary job proceeds with a stop-on-uncertainty threshold lower than normal, meaning the agent stops and reports rather than pushing through ambiguity.
This is a deliberate coupling of the agent types. Pylon-Test is not just a standalone coverage improvement tool. It is the prerequisite that makes Pylon-Migrate and Pylon-Patch safe to run on untested code. The sequencing matters: add coverage first, then modify, then verify the tests still pass.
The comments tell you more than the code
This one surprised us. In production codebases, inline comments are often the most reliable source of intent information, more reliable than the code itself, because comments are written when the intent is fresh and the code is written under time pressure to match the intent imperfectly.
A comment that says "// do not use connection pool here, legacy session manager is not thread-safe" is invaluable context for an agent considering whether to refactor the connection handling in that file. The code does not encode this constraint explicitly; the comment does. An agent that strips comments before processing code is discarding some of its best available information about why the code is the way it is.
Pylon preserves and reads all comments during context graph construction and job execution. More specifically, it flags comments that contain words like "do not," "never," "legacy," "workaround," "temporary," and "FIXME" as signals of intentional constraints or known technical debt. These flagged comments influence the agent's confidence in making changes to the surrounding code. A function surrounded by constraint comments gets a higher uncertainty threshold than one with no comments.
What we have not figured out yet
Being honest about the limits matters when you are asking people to run an autonomous agent on their production codebase.
We have not solved the problem of codebases where the conventions changed dramatically mid-history. A service where the first half was written using one pattern and the second half was written using a different pattern after a team change does not have a single coherent convention set. Pylon's convention inference finds the most common pattern, which may not be the most recent or the intended future direction. We rely on human configuration in these cases: teams can specify which convention set should govern new code generated by Pylon.
We have not solved the problem of codebases with extensive domain-specific logic embedded in what looks like infrastructure code. A billing service where the tax calculation logic is embedded directly in the database query layer is hard to modify correctly without understanding the tax rules, and the tax rules are not encoded in the code in a form a static analysis tool can reason about. Those jobs stop and report.
The honest summary is that Pylon works well on the category of problems it was designed for: well-defined technical debt with clear success criteria. The boundary between that category and the harder problems is something we learn from every codebase we encounter. Each one teaches us something the clean-benchmark codebases never would.