Automated notifications have a credibility problem. Every CI system, every bot, every monitoring tool has trained developers to ignore notifications because most of them are noise. A Slack message that says "Build passed" or "PR opened by pylon-agent" gets glanced at and dismissed. It adds to channel noise without informing any decision.
When we were building the Slack integration for Codepylon, we spent time thinking about what a notification should actually accomplish. The answer we arrived at: a Pylon notification should tell the recipient exactly what they need to decide whether action is needed, without requiring them to open any external link to understand the situation. If they need to click through to find out whether the change is worth reviewing now or can wait until tomorrow, the notification failed.
What we show in the notification
A Pylon Slack notification for a completed job contains five pieces of information, all inline.
First: the ticket reference and title. Not just the ticket ID, but the human-readable title. "Fixed PROJ-847: null pointer in invoice PDF generation" tells the recipient what was worked on. They can tell immediately whether this is in a domain they care about.
Second: the diff size, expressed as files changed and lines changed. "4 files, +82 -14" gives you a sense of the scope of the change. A reviewer scanning their Slack backlog can quickly sort a 4-file patch into the "review in the next hour" bucket versus a 40-file change that will need dedicated time.
Third: test result summary. Did the tests pass, and how many tests ran? "Tests: 847 passed, 0 failed" means the change did not break existing coverage. If tests failed, the notification shows the count of failures and a short description of the first failure. A notification that says "Tests: 2 failed -- InvoicePDFTest: testNullPointerFix" prompts a different response than one that says "Tests: FAIL" and requires you to open CI to find out what broke.
Fourth: the PR link with reviewer request. A direct link to the PR, plus a "Review requested" indicator that shows whether the review request went to a specific person or to a team. If the notification is in a shared channel and the review request went to you specifically, you can see that without clicking.
Fifth: the agent type and principal that ran the job. "Pylon-Migrate / migration-agent" tells a senior engineer at a glance whether this was an automated migration job (lower review priority, probably routine), a security patch (higher urgency), or a feature implementation (requires domain knowledge to review). This matters for triage.
Notification routing
Where the notification goes is as important as what it contains. We support three routing modes.
Channel-based: notifications go to a configured Slack channel, typically a #pylon-jobs or #eng-activity channel. All team members can see the firehose of activity, but no one is personally notified. This works well for teams that do async standups and want a shared activity log.
Reviewer-direct: when the agent selects a reviewer (based on code ownership or explicit configuration), the notification goes to that person as a direct message in addition to the channel. The DM includes everything the channel notification does, plus a clearer call to action: "Review requested by Pylon-Migrate (pylon/migrate/PROJ-847) - 4 files, 82 lines - Tests passed. Review needed."
Initiator-complete: when a developer assigns a job to Pylon and the job finishes, the initiating developer gets a DM confirming the job completed. This is useful when a developer queues several jobs before leaving for the day and wants to check in the morning without scanning the whole channel. Each DM shows job ID, ticket, outcome (PR opened / PR failed / skipped), and next action needed.
The failure notification is the most important one
Most notification design focuses on the success case. The job completed, the PR is open, everything worked. Success notifications inform but rarely require urgent action.
Failure notifications are different. When a Pylon job cannot complete, someone needs to know what happened and make a decision. We designed the failure notification to be the most information-dense notification in the system.
A failure notification contains: the job ID and ticket reference, the specific reason the job stopped (test failures, scope conflict, ambiguous context, dependency conflict), the partial work done (if any files were modified before the stop, we show them), and the recommended next action. "Pylon-Refactor stopped on PROJ-891: 3 of 5 files refactored, stopped on auth_middleware.py (reflection-based routing, manual review needed). Next: open the job log and review auth_middleware.py manually."
That last sentence is the part that takes the most work to get right. We spent time on the logic that determines the recommended next action: is this a case where a human picking up from where the agent stopped is the right path, or is this a case where the job should be re-scoped before trying again, or is this a known limitation that should be logged for future roadmap consideration? The routing matters because a notification that says "please review" when the right action is "please re-scope the ticket" wastes the recipient's time.
What we decided not to include
First iteration of our Slack notifications included a "confidence score" for each completed job, meant to indicate how confident the agent was in the change. We removed it. Confidence scores from automated systems are easy to misread. A 0.85 confidence score on a 4-line bug fix and a 0.85 score on a 200-line migration are very different things, and expressing both as "85% confident" creates false equivalence. Without a reliable way to communicate what the score means for a given job type, it added noise rather than signal.
We also considered including a summary of which parts of the codebase the agent read during context selection. We surfaced this in the Codepylon web UI but not in the Slack notification. The notification is for triage decisions, not for detailed audit review. If you want to know what the agent read, open the job log. The notification should be compact enough to act on in under 30 seconds.
The Slack integration is one of those parts of the product that looks simple from the outside and turns out to have a lot of considered decisions underneath. Getting notified that something happened is trivial. Getting notified in a way that respects the recipient's attention and gives them exactly what they need for the decision at hand is the actual problem.