In late August 2024, a developer at an early-stage fintech company in San Francisco agreed to let us run Pylon on their staging repository. This was not a controlled test environment. It was a real Node.js service they were actively developing, about 45,000 lines, with a mixed pattern of async/await and older callback-based code in some of the payment processing modules. The job we ran was simple: patch an outdated dependency that had a known CVE.
The PR Pylon opened was wrong in two places. This is the story of what went wrong, why it went wrong, and what we changed as a result. I am writing this because I think the details of early failures are more instructive than success stories, and because being specific about what we got wrong is the kind of transparency that either builds trust or ends conversations, and either outcome is honest.
What the job was supposed to do
The CVE was in an older version of a JSON parsing library. The patch required updating the dependency in package.json and package-lock.json, verifying that the updated library's API was backward-compatible with the existing call sites, and updating one test that was asserting on behavior that changed in the new version.
On our internal test repositories, Pylon handled this class of job cleanly. The context graph on a test repo is accurate because we built it with known structure. The call sites are easy to trace. The test changes are predictable.
On the staging repo, Pylon got the dependency update right and the test update right. Where it went wrong was in two call sites it should not have touched at all.
The first error: a missed import alias
The codebase used a module alias in its build configuration: @utils was an alias that resolved to src/utils/index.js. In the import graph, Pylon read all the static imports but did not resolve the alias. So when it saw import { parse } from '@utils', it did not connect this to the parse function being imported from the JSON library two levels down through the re-export chain in src/utils/index.js.
The result: Pylon did not include one of the call sites in its update scope, leaving a code path that was calling the old API signature that no longer existed in the patched version. The test suite did not catch this because the affected code path was only exercised by an integration test that ran against a separate test environment, not in the unit test suite that Pylon ran locally.
When the developer ran the integration test suite, it failed. They caught it before staging deployment. But it was a miss that should not have happened, and the root cause was clear: the context graph's import resolution was not resolving module aliases defined in build configs.
The fix was to read webpack.config.js, tsconfig.json paths, and similar build configuration files as part of repository indexing and incorporate their alias mappings into import edge resolution. We shipped this fix within three days of seeing the failure. It has since expanded to handle Babel module-resolver configs, Vite aliases, and Jest moduleNameMapper entries.
The second error: a generated type file
The second error was more subtle. The repository used a code generation step that produced TypeScript type definitions from an OpenAPI spec. These generated files were committed to the repository. They included type imports from the JSON library being patched.
Pylon updated those type imports as part of the PR, changing the import path to match the patched library's new type export structure. That change was technically correct for the type file. The problem was that the type file was generated code and would be overwritten the next time the code generation step ran, with the old import path. Pylon's PR introduced a change that would be immediately undone by the next npm run generate-types.
The developer noticed this during review. The comment was direct: "these types are generated, if we merge this it'll get reverted immediately." They were right. Pylon had no concept of generated files being different from handwritten files in terms of whether they should be modified.
The fix required adding generated file detection to the indexing step. We now look for a set of indicators: a comment at the top of the file marking it as generated (this is a convention in most code generation tools), a file path that matches common generated file patterns (*.generated.ts, files in __generated__/ directories), and files that appear in gitignore candidates despite being committed. Generated files are now flagged in the context graph and excluded from modification scope by default.
What these errors had in common
Both errors came from the same root cause: the context graph was built from a view of the repository that was less complete than the actual repository. Module aliases and generated file markers are metadata that lives outside the source files themselves, in build configuration and code generation tooling. Our initial indexing step read source files and built the graph from them. It did not read the metadata that governed how those source files fit together.
This is a general lesson that has shaped how we think about indexing ever since. A repository is not just its source files. It is the source files plus the tooling configuration that gives them meaning: build aliases, code generation pipelines, test environment configuration, linting rules that affect what patterns are valid. The context graph needs to model all of this to give the agent an accurate picture of what it can and cannot change.
Why we are writing this now
Both errors were caught by the developer before anything reached production. That is exactly how the PR-first model is supposed to work. The agent makes a mistake; the human reviewer catches it; nothing bad happens outside the review process. The PR was closed without merging, we made the fixes, and the developer ran the job again a week later with a clean result.
We are writing this a year later, not at the time, because we wanted to see whether the fixes held up across more varied codebases before publishing the lessons. They have. The module alias resolution and generated file detection changes have been in production for almost a year and have not regressed. The specific failure modes from that first PR have not recurred in the same form.
New failure modes have appeared, of course. That is the nature of building on real codebases. But the two from that first PR are closed, and the way they are closed is the way we try to close all failures: specific detection, specific fix, and a logged case that future indexing logic is tested against.