The Pylon dashboard has a status that we call "Stopped with report." It is not an error state in the engineering sense; the agent ran successfully and produced output. But the output is not a PR. It is a structured explanation of what the agent found, what it tried, where it got stuck, and what a human engineer would need to do to continue.
We think about this status a lot. An agent that never fails is either doing trivial work or hiding its failures. Pylon operates on real codebases with real complexity, so failures happen. The question is what failure looks like from the engineering team's perspective, and whether the failure is useful or just a dead end.
The three categories of job failure
In practice, job failures cluster into three categories, each with different causes and different appropriate responses.
Scope ambiguity: the job was specified at a level of abstraction where the correct implementation depends on a decision the agent cannot make from the codebase alone. "Refactor the authentication module" is an example. The agent can identify the files in scope, but "refactor" is not a specific enough directive to produce a PR without making product decisions: should the session handling move to a dedicated service? Should the token expiration logic be centralized or stay distributed? These are architectural questions that belong to the engineering team. The agent stops, enumerates the decision points it encountered, and reports them.
Test conflict: the agent produced a diff, ran the test suite against it, and the tests failed in a way that the agent could not resolve without changing behavior. This is different from simply fixing a failing test. Pylon can update a test when the test is testing the wrong thing (for example, a test that was asserting on an implementation detail rather than a contract). But when a test is testing a real contract and the agent's change would violate that contract, the agent stops. The PR is not opened. The report includes the failing test, the diff that caused the failure, and the agent's analysis of why the conflict exists.
Dependency uncertainty: the change the agent wants to make has downstream effects that the agent cannot safely evaluate. This most often happens in codebases with dynamic dispatch patterns, reflection-based plugin systems, or runtime-constructed call graphs. The context graph flags these as uncertain edges. When the agent's planned change would touch a node with high uncertainty edge count, it stops rather than proceeding on the assumption that the uncertain paths are safe.
What a "Stopped with report" report contains
We put significant effort into the failure report format because an opaque "I failed" message is useless. The report has four sections.
First: what was completed before the stop. If the agent analyzed the codebase, built a change plan, and got 60 percent through executing it before encountering the blocker, the report shows what was completed. This is not partial code that got committed; Pylon always works on an ephemeral branch and only opens a PR when the full job is complete. But knowing that the first three files in the plan were clean and the fourth was the problem is useful context for a human continuing the work.
Second: the specific blocker. Not a generic error code but a structured description of what the agent found. For a test conflict: the test name, the failing assertion, the diff line that caused it, and the agent's hypothesis about why the test is failing. For a scope ambiguity: the specific decision points, listed as questions with the options the agent identified.
Third: the agent's recommended next action. This is the most important section for the engineer picking up the job. "Review the `processWebhookEvent` function and decide whether the retry logic should move to the event bus layer or stay in the handler" is actionable. It is also honest about what the agent could not determine.
Fourth: a confidence assessment. Was this a hard blocker (the agent has high confidence that proceeding would break something) or a soft blocker (the agent was uncertain enough to stop but a human might reasonably decide to proceed)? Hard blockers have the agent's reasoning spelled out. Soft blockers include a note that the engineer can override the stop and continue the job with that specific uncertainty acknowledged.
Why we do not guess through ambiguity
There was an internal debate early in building Pylon about the right threshold for stopping versus proceeding with the best available guess. The argument for guessing: an imperfect PR that needs a couple of revisions is still less work than a human starting from scratch. The argument for stopping: an imperfect PR that contains a subtle bug in production code is worse than no PR at all.
We landed on stopping because of what we observed in practice during early testing. When the agent guessed through an ambiguity and the guess was wrong, the resulting PR had a specific failure mode: it looked correct. The diff compiled, the tests passed, the description sounded reasonable. But the behavior was subtly different from the intended behavior in a way that only became apparent in integration testing or production. That failure mode is the worst case for an automated system, because it defeats the human review process that is supposed to catch mistakes.
A PR that fails its CI checks is easy to identify as needing work. A PR that passes all checks but contains a logical error in a code path that tests do not exercise is the kind of thing that slides through review and shows up as an incident. We are not saying that Pylon never produces PRs with bugs; it does, and review is the safety net. But we try hard not to have the agent produce PRs it is not confident in and dress them up in language that makes them look confident.
Failure rate and what it tells us
We track the stop-with-report rate across job types. For Pylon-Patch jobs (CVE patching), the rate is low because the scope is tightly constrained: there is a specific vulnerability, a specific set of files, and a specific patch. For Pylon-Refactor jobs, the rate is higher because refactoring goals are inherently fuzzier and the uncertainty edge problem is more common. For Pylon-Migrate jobs on older codebases, the rate varies significantly based on how consistently the codebase was originally written.
When the stop rate for a specific job type rises noticeably, it usually means either that the context graph is less accurate for a particular language or pattern, or that the job type is being used for more ambiguous requests than it was designed for. Both are signals to us about where to improve the product. Failure is informative when you track it honestly.
The goal is not zero failures. The goal is failures that are useful: that tell the engineer exactly what was found, what was tried, and what they need to do next. A tool that fails informatively and honestly is one that engineers can trust with progressively more complex work over time.