Post Grafana OnCall alert groups to Mattermost
A Grafana + Mattermost agent flow
Grafana OnCall already pages whoever holds the rotation, but the rest of the team finds out an alert fired only if the on-call engineer remembers to say so. Context lives in the OnCall UI, the discussion happens in Mattermost, and the two don't meet until someone copies a link by hand. This recipe closes that gap: your agent reads the open alert groups from Grafana OnCall and drops a summary into the team channel, so everyone sees the same firing alert the responder does. It calls list_alert_groups filtered to new or acknowledged groups, diffs the group IDs against the previous pass to find the ones that just opened, and posts the title, the integration, and the state with post_message. Grafana replies on request rather than pushing, so you run it on a short loop during incidents and a slower one otherwise.
The flow
list_alert_groupsLists OnCall alert groups with filtering.
post_messagePost a message to a channel.
Step by step
- Filter to the alert groups you care about
Tell the agent which OnCall integration or team to watch and which states count, typically new and acknowledged. Resolved groups stay out so the channel only shows live noise.
- Read the open groups
The agent calls list_alert_groups with that filter and gets back each group's title, the integration that raised it, its current state, and the alert count behind it.
- Find what just opened
Compare the returned group IDs to the set from the last pass. A group that wasn't there before is new; a state change from new to acknowledged is worth a follow-up line if you want the channel to track resolution.
- Post to the team channel
For each new group, call post_message to the Mattermost channel ID with a compact summary: the alert title, the integration, the state, and how many alerts the group rolled up.
Tell your agent
Every two minutes, list Grafana OnCall alert groups in the new or acknowledged state for the production integration. For any group that opened since your last check, post a message to the Mattermost channel 'incidents' with the alert title, the integration name, the state, and the alert count. Skip groups you've already posted.
Setup
This flow needs both servers connected to your agent. Follow each install guide:
Worth knowing
- post_message addresses a channel by its Mattermost channel ID. Resolve the channel once with get_channel_by_name and reuse the ID, and make sure the account behind the server is a member of that channel or the post is rejected.
- list_alert_groups returns grouped alerts, not individual firings. The count is how many alerts rolled into the group, so a single noisy group can represent dozens of underlying events.
- OnCall is one slice of the Grafana server, and many of its data-source query tools are disabled by default. This recipe only needs the OnCall alert-group tools, so confirm those are enabled rather than assuming the whole surface is on.
Questions
- Why route OnCall alerts to Mattermost at all when OnCall already pages?
- Paging reaches one person; the channel reaches the team. This keeps the broader group aware of what's firing and gives them a thread to coordinate in, without anyone hand-copying links out of the OnCall UI.
- Does the post appear the instant an alert fires?
- No. The Grafana server answers when the agent calls it, so the channel trails real time by your polling interval. Tighten the loop during an active incident if you need it close to live.