We found out the database backup had been failing for nine days when someone needed a two-day-old copy and there wasn't one. The job was in cron. It ran every night. It also exit 1-ed every night, because a disk had filled and the dump couldn't write — and cron did exactly what cron does with a failure, which is nothing you'll ever see. There was an email alert configured, technically. It went to an address that forwarded to a distribution list that everyone had muted in 2023. The lesson wasn't "add alerting." We had alerting. The lesson was that an alert nobody reads is decoration, and the only alert that counted was the one that showed up in the channel we stared at all day.
Why the webhook is the whole thing
A Slack incoming webhook is a URL that turns an HTTP POST into a message in a channel. That's the entire mechanism: curl a JSON body to the URL, Slack posts it. Which means there are exactly two ways to get this wrong — send malformed JSON, or leak the URL — and the script below is built around not doing either.
The jq -n line is the one people skip and regret. The first time an error message contains the path to a file with a " in the name, a hand-built "{\"text\":\"$msg\"}" produces invalid JSON, Slack rejects it with a 400, and now your alerting is the thing that's broken — quietly, of course. Letting jq build the object means the payload is correct for any input.
The trap is what makes it automatic
Wiring slack_alert to trap ... ERR is the difference between alerting you remember to add and alerting that's always on. With set -e, any command that fails and isn't caught by an if or || trips the ERR trap, which posts the failure — with $LINENO telling you exactly which command died — before the script exits. You don't decorate every line with error handling; you set the trap once at the top and the whole script is covered.
What a webhook can't do
A webhook fired from inside the job can only report failures the job is alive to see. If the box is down, cron is misconfigured, or the script never starts, nothing posts — and "nothing posted" looks identical to "everything's fine." That's the gap a dead-man's-switch fills: a service like Healthchecks.io expects a ping by a deadline and alerts when the ping doesn't arrive. The webhook and the external check cover opposite failure modes, which is why serious jobs use both. This script is the first half.
To build alert payloads that carry real data from a response, you'll want jq to construct the JSON, and to only alert on genuine failures you need curl that actually detects them. The full fetch-parse-alert pattern lives in the Shell Scripts That Talk to APIs guide.
Run this script on a real Linux server
Get $200 free credit — DigitalOcean
Get $200 Free →Affiliate link · we earn a commission