Diagnosing a Hung Process
A crashed job leaves you something to read. A hung job leaves you nothing. It does not exit, so no exit code is recorded. It does not error, so nothing lands in the log. The cron entry that should have finished at 02:15 is still sitting there at 09:00 with its output file half written, and the only thing you actually know is that it has not moved.
The instinct is to kill it and run it again. That is also the one action that guarantees you will be back here next week, because everything that would have told you why it hung lives inside the process, and kill -9 throws all of it away. The process is not the problem. It is the evidence.
This is the order to work through, from the cheapest command to the most invasive, with what each answer rules out. Every command below was run on Linux 7.1 / bash 5.3 as an unprivileged user, and where something needs root or extra permissions it says so.
First question: is it hung, or is it slow?
These look identical from the outside and have nothing else in common. The difference is whether the process is burning CPU.
etime is wall-clock age; time is CPU consumed. If time is climbing, the process is working — it may be doing something stupid, like an accidental O(n²) loop over a growing file, but it is not blocked and none of the blocking diagnostics below apply. If etime climbs and time does not move at all, it is genuinely waiting on something, and the rest of this page is about finding out what.
The state letter tells you which kind of stuck
STAT is the important column. From the ps man page, the states you will actually meet:
| Code | Meaning | What it tells you |
|---|---|---|
R | running or runnable | Not blocked. It is slow, not hung — go back a section. |
S | interruptible sleep | Waiting on something, and it will respond to signals. The common case. |
D | uninterruptible sleep (usually I/O) | Blocked in the kernel on I/O. kill -9 will not touch it. |
T | stopped by job control signal | Someone or something sent it SIGSTOP. It is paused, not stuck. kill -CONT resumes it. |
Z | defunct (zombie) | Already dead. It is waiting for its parent to reap it — your problem is the parent, not this. |
Trailing letters are modifiers, not states: N is low priority (niced), < high priority, L has pages locked, s is a session leader, + is in the foreground process group. SN above is "sleeping, niced" — one state, one modifier.
Three of these end the investigation immediately. R means it is not hung. T means it was stopped, most often by a stray SIGSTOP or a debugger that went away, and kill -CONT "$PID" fixes it. Z means the process is already gone and the bug is in whatever forked it and never called wait().
The interesting split is S versus D.
D state is different, and it is why kill -9 sometimes does nothing
An uninterruptible sleep is a process blocked inside a kernel call that cannot be interrupted — classically a read or write against storage that is not answering. An NFS mount whose server went away, a failing disk retrying a sector, a device that has stopped responding.
A process in D cannot be killed. Not with SIGTERM, not with SIGKILL. Signals are delivered when a process returns to user space, and this one never does. kill -9 appears to succeed — kill returns 0, because it queued the signal — and the process stays exactly where it is. Every "kill -9 isn't working" report is this.
There is no command that fixes D state. It clears when the I/O completes or the kernel gives up on the device, or it clears at reboot. What you can do is find out which device, so you fix the real fault instead of fighting the symptom:
If wchan names something like nfs_wait_on_request or an I/O wait, stop looking at the process. The process is fine. The storage under it is not.
What it is blocked on, without root
Two files under /proc answer this and neither needs privileges for your own processes.
wchan is usually enough on its own — hrtimer_nanosleep means it is in a sleep, futex_wait means it is waiting on a lock, anything with sock or tcp in the name means it is waiting on the network.
To turn the syscall number into a name, the header on the box is the authority:
230 is clock_nanosleep on x86_64, which matches the hrtimer_nanosleep above — the process is asleep on a timer, exactly as expected for sleep 300. On a genuinely hung job you are looking for read, write, futex, connect, recvfrom or flock instead, and each points somewhere specific.
/proc/$PID/stack gives the full kernel stack and is the best of these — but it requires root. As an ordinary user it returns Permission denied, so it is a sudo command or nothing.
What it has open
The syscall says what kind of wait. The file descriptors say on what.
Read this against the syscall. Blocked in read with fd 3 pointing at a FIFO means it is waiting on a pipe nobody is writing to. Blocked in flock means another instance holds the lock — which is flock doing its job, not a bug, and the question becomes why the previous run never finished. Blocked in write with the fd on a full filesystem is a disk-space problem wearing a hang costume.
If it is the network
A hung curl, a hung database client and a hung API call all look the same from ps. The socket table separates them:
The queue columns are the diagnosis:
Send-Qlarge and not draining — you have written data the peer is not acknowledging. The far end is gone or wedged; the packets are stacked up locally.Recv-Qlarge — data has arrived that your process has not read. The far end did its job; your process is stuck somewhere else and this is a symptom, not the cause.SYN-SENT— the connection was never established at all. DNS resolved to something that is not answering, or a firewall is dropping rather than rejecting. This is the state that hangs forever with no timeout set, because a dropped packet produces no error, only silence.- No row at all — it is not the network. Go back to
wchan.
That last case is the whole argument for --connect-timeout and --max-time on every curl in a script; a request that hangs in SYN-SENT will otherwise outlive the cron interval that started it. The wrapper that does this properly is in curl API requests that fail correctly.
strace, and why it may refuse
strace -p "$PID" shows syscalls live, and on a hang it usually prints one line and then stops — which is the answer, because that line is the call it is stuck in.
Two things get in the way, and both look like a broken tool:
ptrace_scope of 0 allows attaching to any process you own. 1 — the default on Ubuntu and several others — permits attaching only to direct children, so strace -p on a cron job you did not start fails with Operation not permitted even as the owner. sudo strace -p gets around it. So does raising the sysctl, but do not do that on a production box to debug one process.
Because strace stops the process on every syscall, it is genuinely slow on a busy one. On a hung process this costs nothing — it is not making syscalls, that is the problem.
Kill it in the right order
Once you have the evidence, escalate deliberately. Each signal is a different question.
SIGTERM first, always. A script with a cleanup trap on EXIT removes its temp files and releases its lock on TERM. SIGKILL skips all of that — traps do not run, mktemp files are orphaned, and a lock file written the naive way is left behind to block the next run.
SIGQUIT in the middle is worth knowing: on a JVM it triggers a full thread dump rather than killing anything, which on a hung Java process is the single most useful command available. On most other programs it terminates and writes a core.
And if the process is in D, none of these do anything. That is not the signal failing; it is the state.
Capture everything first
All of the above in one pass, so the evidence survives the kill:
Run that, then kill. It takes five seconds and it is the difference between fixing the cause and rediscovering it next month.
Note the || true on every capture: the process can exit while you are reading /proc, and with set -e a disappearing file would abort the script partway through and leave you with a half-collected report — the one outcome worse than no report. Why that idiom is needed, and what it costs, is in the safe bash script template.
Stop it happening again
Diagnosis is the part you do once. The fix is making the next hang impossible to sit undetected for seven hours.
Bound the runtime so a wedged command dies on its own rather than waiting for a human — timeout sends TERM at a deadline and KILL after a grace period. Set explicit --connect-timeout and --max-time on every network call. Take a lock so a hung run cannot be joined by the next scheduled one and turn one stuck process into forty — flock. And have the job say something when it dies, because a hang you find out about at 02:15 costs minutes and one you find at 09:00 costs the morning: Slack webhook alerts.
The three ways unattended jobs die quietly — overlap, hang, transient failure — and the guard for each are worked through in Bash Scripts That Survive Cron. This page is what to do on the morning you did not have those guards in place yet.