The report generator had been dying at 02:14 for eleven nights before anyone noticed, and the way we noticed was worse than the dying. It wrote its CSV straight to the export directory, a downstream loader picked that file up at 02:30, and the loader was perfectly happy with a file that ended mid-row — it loaded 40% of the data and reported success. Eleven days of a revenue dashboard quietly built on partial numbers. The generator's actual bug was a five-minute fix; the part that took a week was re-deriving which of the loaded days were poisoned, because nothing anywhere had failed. And when I finally logged into the box, /tmp had 300-odd orphaned working directories from every previous crash, which was the machine telling me the script had been exiting uncleanly for months and I hadn't listened.
Two habits prevent the whole class of failure: every temporary resource gets a trap that removes it on any exit, and every output file is written somewhere else and moved into place atomically. Neither is more than four lines.
The pattern
Walk the failure paths and watch what happens. generate_report_rows dies halfway: set -e aborts the script, the EXIT trap fires, the partial file is deleted, and $FINAL_PATH still holds yesterday's complete report — the loader at 02:30 gets stale data, which is a monitoring alert, not silent corruption. Someone Ctrl-Cs it, or the box's shutdown sends SIGTERM: same story, the EXIT trap runs on the way down. The script finishes cleanly: mv has already relocated the temp file, rm -f in the trap finds nothing and says nothing, and the exit code passes through untouched because cleanup captured $? before doing anything else.
That local code=$? line deserves a second look, because it's the subtlest bug in the pattern. The trap runs your commands, and those commands set $? like any others. Skip the capture, and a script that failed with exit 3 runs rm -f (which succeeds), and the trap's implicit return status becomes the script's status: 0. Cron sees success. Your alerting sees success. The failure is invisible — which is precisely the disease this whole page is trying to cure.
Temp directories, multiple resources
When the script needs several working files, don't juggle several traps — make one temp directory and remove it whole:
One resource, one trap, one rm -rf — and every intermediate file inherits the cleanup for free. This is also the fix for the 300 orphaned directories: they came from a script that did mkdir /tmp/work.$$ with the cleanup at the bottom of the script, where half the failure paths never reached it. Cleanup that only runs on success isn't cleanup, it's decoration.
Two scoping rules save you a confused hour each. First, EXIT traps belong at the top level — a trap set inside $( ) or ( ) lives and dies with that subshell, and the parent's trap doesn't fire when a subshell exits. Second, registering a second trap ... EXIT replaces the first, it doesn't stack. If two things need cleanup, put both in one function rather than setting two traps.
The one thing trap cannot save you from is kill -9, which no process can catch. That's not a reason to skip the trap — it's the reason the rest of the design is shaped the way it is: mktemp puts orphans where the OS eventually clears them, and atomic mv means even an uncatchable death can't publish a torn file. The trap handles every ordinary death; the architecture handles the extraordinary ones.
This page is one leg of the unattended-script survival kit. set -euo pipefail makes failures stop the script so the trap has something honest to report; timeout bounds a hang so the script dies (and cleans up) instead of blocking forever; flock keeps overlapping runs from stepping on each other's files. The Hardened Cron Wrapper Generator assembles all of it around any command, and the full argument for why cron jobs need this armor is in Bash Scripts That Survive Cron.
Run this script on a real Linux server
Get $200 free credit — DigitalOcean
Get $200 Free →Affiliate link · we earn a commission
The cheapest way to trust this pattern is to break it on purpose: run the script on a scratch droplet, kill it mid-generate, and confirm /tmp is clean and the output file is still yesterday's. The rest of the library is at bashsnippets.xyz — Slack alerts from bash pairs naturally with this one, so the 02:14 death pages you at 02:14 instead of introducing itself through your dashboard.