Send Resend emails when Datadog monitors alert
A Datadog + Resend agent flow
A Datadog monitor trips and the alert fans out to a channel where it joins a hundred others. The signal that matters, the SLO that breached, the queue that's backing up, deserves a real email to a named owner who will look. This recipe has your agent read Datadog monitors in an alerting state and send a focused message to the person who owns that service, with context you decide. It calls search_datadog_monitors filtered to alerting or warning, groups by the tags that identify a team, and sends with Resend's send-email: the monitor name, its current state, the breaching value, and a link. Datadog's server answers calls rather than pushing, so the agent runs on a schedule and the email follows the alert by your interval, aimed at exactly one inbox instead of a firehose.
The flow
search_datadog_monitorsFinds monitors by status, threshold, and alert conditions.
send-emailSends a single transactional email immediately or scheduled, with HTML/text, attachments, CC/BCC, and tags.
Step by step
- Map monitors to owners by tag
Pick the tags that mark ownership, like team or service, and map each to a recipient and a from address. Verify your sending domain in Resend before the first send.
- Find alerting monitors
The agent calls search_datadog_monitors filtered to the alert states you care about, reading each monitor's name, status, the threshold it crossed, and its tags.
- Email the owner
It groups the alerting monitors by ownership tag and calls send-email per group, with a subject naming the monitor and a body holding the state, the value, and a deep link, tagged so Resend can filter them.
- Send on transitions only
Record the state each monitor was in last run. Email when one enters alert and once when it recovers to OK, so a monitor that flaps or stays red doesn't generate an email every poll.
Tell your agent
Every two minutes, search Datadog monitors in Alert or Warn state. Group them by the 'team' tag and, for each newly-alerting monitor, send a Resend email from datadog@acme.com to that team's address, subject '<monitor name>: <status>', body with the breaching value and a link to the monitor, tagged 'datadog'. Send one recovery email per monitor on return to OK, and don't repeat while it stays in alert.
Setup
This flow needs both servers connected to your agent. Follow each install guide:
Worth knowing
- send-email requires a verified Resend domain with SPF and DKIM set. Use a from address on that domain; an unverified domain means the alert silently fails to deliver.
- search_datadog_monitors returns current state, so the agent has to diff against the prior run to find genuine alert and recovery transitions rather than emailing on every poll.
- Datadog monitors can be noisy and flap. Consider a short grace period, only email if the monitor is still alerting on the next poll, and rely on the ownership tag so each email reaches one team, not a broadcast list.
Questions
- Why email when Datadog already has notifications?
- Datadog's notifications go to channels you wire per monitor. This lets one agent read all alerting monitors, route each to the owning team by tag, and write the message itself, so a breach reaches a specific person with the context you chose rather than a shared stream.
- How does it keep from emailing on every run?
- By remembering each monitor's last state. It emails on the edge into alert and the edge back to OK, and stays silent while the state is unchanged, so a persistent alert is one email plus one all-clear.
- Does the email arrive the instant the monitor trips?
- No. The Datadog server responds to the agent's call rather than pushing, so the email trails the alert by your polling interval. A two-minute loop keeps it close without re-reading constantly.