Skip to content

Make API Requests in Bash with curl (That Actually Fail When the API Does)

bashcurlapihttpdevops
3 min read

Quick Answer

When a bash script calls an API with a plain curl, it trusts the network, and that trust is the bug. curl exits 0 as long as it received any response, so an HTTP 500 error page or a 429 rate-limit body sails through as if it were valid data, and every downstream step quietly processes garbage. The fix is to stop treating curl ran as the request succeeded. Capture the status code separately with -w '%{http_code}', split it from the body, and branch on it: 2xx is success, 5xx and 429 are transient and worth a retry, 4xx is your bug and should fail loudly. Add --connect-timeout and --max-time so a stalled endpoint can never wedge a cron job forever. The wrapper below fetches, checks status, retries transient failures, and returns non-zero when the API is genuinely down.

I pointed a plain curl at an endpoint that answers every request with a 502, the way an API does in the middle of a bad deploy, and saved the result the way a nightly job would:

text
$ curl -s -o page.html https://httpbin.org/status/502; echo "exit=$?" exit=0 $ wc -c < page.html 0

Exit 0, and a zero-byte file where the data should be. Nothing downstream can tell. set -euo pipefail has nothing to trip on, the parser reads an empty file, and whatever gets built from it is quietly wrong until someone notices the numbers. Same endpoint, same box, through the wrapper below:

text
$ ./api-request.sh https://httpbin.org/status/502 ✗ 502 (attempt 1/3) — retrying ✗ 502 (attempt 2/3) — retrying ✗ 502 (attempt 3/3) — retrying ✗ gave up after 3 attempts $ echo $? 1

Three tries, about six and a half seconds, and a non-zero exit that cron and the next step can both see.

Why does curl succeed when the request failed?

curl's exit code answers one question: did I get a response back? It says nothing about what that response was. A 200 with your JSON and a 500 with an error page are both, to curl, a successful round trip. So set -euo pipefail won't save you here — the command didn't fail, it just returned something you didn't want. Any script that treats "curl ran" as "the API returned good data" is one bad deploy on the other end away from silently processing garbage.

The wrapper: check status, time out, retry

bash
#!/bin/bash # Script: api-request.sh # Purpose: Call an HTTP API and FAIL when the API fails — curl exits 0 on HTTP 500 by default, which silently poisons everything downstream # Usage: ./api-request.sh <url> set -euo pipefail CHECK="✓" CROSS="✗" API_URL="${1:?Usage: api-request.sh <url>}" CONNECT_TIMEOUT=5 # seconds allowed to establish the TCP connection MAX_TIME=30 # hard ceiling on the whole request so cron can't wedge MAX_RETRIES=3 # transient 5xx/429 get retried; other 4xx does not RETRY_DELAY=2 # seconds between retries api_get() { local url="$1" local attempt=1 local response http_code body while (( attempt <= MAX_RETRIES )); do # -w appends the status code on its own line after the body, so we can split them. # The || guards the transport-level failures (DNS, refused, timeout) that DO set curl's exit code. if ! response=$(curl -sS \ --connect-timeout "$CONNECT_TIMEOUT" \ --max-time "$MAX_TIME" \ -w $'\n%{http_code}' \ "$url"); then echo "$CROSS transport error on attempt $attempt/$MAX_RETRIES" >&2 attempt=$(( attempt + 1 )); sleep "$RETRY_DELAY"; continue fi http_code="${response##*$'\n'}" # last line is the status code body="${response%$'\n'*}" # everything before it is the body case "$http_code" in 2*) printf '%s' "$body" echo "$CHECK $http_code OK" >&2 return 0 ;; 429|5*) echo "$CROSS $http_code (attempt $attempt/$MAX_RETRIES) — retrying" >&2 attempt=$(( attempt + 1 )); sleep "$RETRY_DELAY" ;; *) echo "$CROSS $http_code — not retryable, this is our request" >&2 echo "$body" >&2 return 1 ;; esac done echo "$CROSS gave up after $MAX_RETRIES attempts" >&2 return 1 } api_get "$API_URL"

A 404 gets one try, because retrying a wrong URL only repeats the mistake, and a good response goes to stdout with the status on stderr:

text
$ ./api-request.sh https://httpbin.org/status/404 ✗ 404 — not retryable, this is our request $ echo $? 1 $ ./api-request.sh https://api.github.com/repos/anguishe/bashsnippets > repo.json ✓ 200 OK $ jq -r .full_name repo.json anguishe/bashsnippets

The body goes to stdout so you can pipe it straight into jq; the status commentary goes to stderr so it doesn't pollute the data. Because api_get returns non-zero when the API is truly down, the caller can do data=$(api_get "$url") || exit 1 and mean it.

Why not just use --fail?

curl --fail is the blunt version of this: it makes curl exit 22 on 4xx/5xx so set -e trips. It's genuinely useful for a quick one-liner. The reason I don't reach for it in a real job is that --fail discards the response body on error — so when the API hands you a 400 with {"error":"missing field x"}, you get the exit code but not the message that tells you what's wrong. Capturing the code and body separately costs three extra lines and gives you something to log. And --fail still doesn't distinguish "retry this" (429/503) from "don't bother" (401/404), which is the distinction that actually keeps a flaky endpoint from paging you.

What this still doesn't cover

This handles a single request. It does not walk paginated results, refresh an OAuth token that expired mid-run, or read a Retry-After header to back off politely (it uses a fixed delay). Those are real and worth adding when the API demands them — see the Shell Scripts That Talk to APIs guide for the fuller pattern. But the status-code check and the timeouts are the two things that turn "exit 0 and an empty file" into "fails tonight, loudly."

Once you've got a clean 2xx body, the next job is reading fields out of it without regex — that's parsing JSON with jq. And when a request finally does fail for real, you want to hear about it: send the failure to Slack.

Raw script, MIT licensed: scripts/bash-curl-api-requests.sh on GitHub

PAID RESOURCE — $9

The Production Bash Toolkit

An operational script system + a 30-function shared library + a 52-page field guide. The production layer the free snippets don't cover.

Get the Toolkit →
curl -O bashlib-starter.sh

Get the bashlib starter

Ten functions I source into every script on my own boxes — strict-mode setup, an ERR trap that names the failing line, lock and timeout wrappers, and cleanup that runs on every exit path. After that, a short email only when something breaks on a real box and I've worked out why, about twice a month. One-click unsubscribe.

BashSnippets logo

Written by Travis

Creator of BashSnippets.xyz

bashsnippets.xyz/about

Related Snippets

Delete Old Log Files

Unmanaged logs fill /var/log until writes fail and services crash. find -mtime deletes .log files older than N days; preview with -print before deleting.

Bash Error Handling with set -euo pipefail

Bash keeps going after a failed command, so a broken cd followed by rm -rf hits the wrong directory. set -euo pipefail stops the script at the first failure.

Kill a Process with pkill and pgrep

ps, grep, copy the PID, kill: four steps every time. pkill stops a process by name in one command, and pgrep -l previews exactly what it will match first.

Frequently Asked Questions

faq — snippet

Why does curl exit 0 even when the API returns a 500 error?

curl's exit code reports the transport, not the HTTP result. If it resolved DNS, opened the connection, and got a complete response back, curl considers its job done and exits 0 — even if that response was a 500 or 404. To make your script react to HTTP status you either add the -f/--fail flag (which makes curl exit 22 on 4xx/5xx but throws the body away) or capture the code yourself with -w '%{http_code}' and branch on it, which is what the wrapper here does.

faq — snippet

What is the difference between --connect-timeout and --max-time?

--connect-timeout caps only the time spent establishing the TCP connection — useful when a host is unreachable and you don't want to wait the full OS default. --max-time caps the entire operation end to end, including the transfer. In a cron job you want both: a short connect timeout so a dead host fails fast, and a hard max-time so a server that accepts the connection but then stalls forever can't leave your job hung until the next run collides with it.

faq — snippet

Which HTTP status codes are safe to retry?

Retry 5xx (server errors) and 429 (rate limited) — those are transient and usually clear on their own. Do not blindly retry 4xx other than 429: a 400, 401, 403, or 404 means your request is wrong, and retrying it just hammers the endpoint with the same broken call. The wrapper retries 429 and 5xx with a delay and fails immediately on other 4xx so you find out it's your bug instead of masking it.

← All Snippets↑ Back to top