Post SigNoz alerts to a Mattermost channel
A SigNoz + Mattermost agent flow
SigNoz watches traces, logs, and metrics and raises alerts when a rule trips, but the engineers who own the affected service often spend their day in Mattermost, not the SigNoz console. A platform team running self-hosted observability wants firing alerts to show up in the channel where the rest of incident chatter happens. The agent reads the alert list with signoz_list_alerts, finds the ones now firing that weren't firing last time, and posts each to a Mattermost channel with post_message. SigNoz responds to the call, so you choose the interval and the agent reads then writes on that beat.
The flow
signoz_list_alertsLists alerts in SigNoz.
post_messagePost a message to a channel.
Step by step
- Choose the channel and what counts as firing
Point the agent at one Mattermost channel and decide which alert states matter to you. Resolve the channel ID once with get_channel_by_name so posts have a fixed destination.
- List alerts
The agent calls signoz_list_alerts and reads each alert's name, severity, and current state across the SigNoz instance.
- Find the firing edge
Compare the current firing set to the prior run. Post an alert when it enters firing and, if you want the all-clear, once when it returns to normal, so a sustained alert isn't reposted every poll.
- Post to Mattermost
For each new firing alert the agent calls post_message with the alert name, severity, the service it concerns, and a link, formatted so the channel reads it at a glance.
Tell your agent
Every minute, list SigNoz alerts. For any alert that just entered firing, post to the #observability Mattermost channel with the alert name, severity, affected service, and a link. Post one recovery line per alert when it returns to normal, and stay quiet while it keeps firing.
Setup
This flow needs both servers connected to your agent. Follow each install guide:
Worth knowing
- signoz_list_alerts returns the current state of each alert, so the agent must diff against the last run to find genuine firing and recovery transitions rather than posting on every poll.
- post_message needs a channel ID. Resolve it with get_channel_by_name once and reuse it, since posting to a stale ID fails and the alert is lost.
- A flapping alert can cross the firing edge repeatedly. Add a short grace check, only post if it's still firing on the next poll, so the channel isn't peppered with on-off lines.
Questions
- Why route through an agent when SigNoz has its own notification channels?
- SigNoz channels are wired per alert in its config. This lets one agent read every alert, decide which states and which services post, and write the Mattermost line itself, so you control the routing in one place rather than per rule.
- How does it avoid posting the same alert every minute?
- By remembering which alerts were firing last run. It posts on the edge into firing and the edge back to normal, and stays silent in between, so a long alert is one post plus one recovery.