Send Resend emails when a Prometheus query crosses a threshold
A Prometheus + Resend agent flow
Prometheus holds the metric that matters, error rate, queue depth, disk headroom, but turning it into a notification usually means standing up Alertmanager and wiring receivers. A small team wants a named owner to get an email when one specific query crosses a line, without that machinery. The agent runs the PromQL with execute_query, compares the result to your threshold, and if it's breached sends a focused email with Resend's send-email. Prometheus answers the call rather than pushing, so the agent runs the query on a schedule and the email follows a breach by your interval, aimed at one inbox.
The flow
execute_queryExecutes a PromQL instant query against Prometheus.
send-emailSends a single transactional email immediately or scheduled, with HTML/text, attachments, CC/BCC, and tags.
Step by step
- Write the query and the threshold
Pick the PromQL that captures the condition, for instance a five-minute error ratio, and the value that counts as breached. Decide the recipient and verify your sending domain in Resend before the first send.
- Run the query
The agent calls execute_query with your PromQL and reads the scalar or vector it returns, the current value of the metric at query time.
- Email on a breach
If the value crosses the threshold the agent calls send-email with a subject naming the metric, a body holding the value, the threshold, and the time, and a tag so Resend can filter these alerts.
- Email on the edge, not every run
Record whether the last run was breached. Email when the value first crosses, and once when it returns below, so a metric that stays high doesn't generate an email every poll.
Tell your agent
Every minute, run the PromQL `sum(rate(http_requests_total{status=~"5.."}[5m])) / sum(rate(http_requests_total[5m]))` against Prometheus. If it exceeds 0.05, send a Resend email from alerts@acme.com to oncall@acme.com, subject 'Error rate breach', body with the current value, the threshold, and the time, tagged 'prometheus'. Send one all-clear when it drops back below, and don't repeat while it stays high.Setup
This flow needs both servers connected to your agent. Follow each install guide:
Worth knowing
- execute_query returns the value at query time, so the agent has to hold the prior result and compare, emailing on the crossing rather than on every poll while the value stays over the line.
- send-email needs a verified Resend domain with SPF and DKIM. Use a from address on that domain, since an unverified one means the alert silently fails to deliver.
- A range query result is a vector. If your PromQL returns multiple series, decide whether to email per series or aggregate, or the agent may send a confusing multi-value body.
Questions
- Why not just use Alertmanager?
- You can, and for large setups it's the right tool. This is for the case where you want one query watched and one person emailed without running Alertmanager, with the agent reading Prometheus directly and writing the mail.
- Does the email arrive the instant the metric breaches?
- No. Prometheus responds when the agent queries it, so the email trails the breach by your interval. A one-minute loop keeps it close without querying constantly.