Making a script safe to run twice

Sooner or later, every script runs twice. A cron job starts while the previous run is still going. A CI system retries a failed step. A queue delivers the same message again, as Amazon SQS’s documentation warns it may, and asks you to design for. An AI agent that isn’t sure its command worked runs it again.

The property you want has a name. RFC 9110 calls an HTTP method idempotent if the intended effect of several identical requests is the same as the effect of one. It also gives the reason that matters here: an idempotent request can be repeated automatically after a failure, because repeating it has the same intended effect. A script with the same property can be retried without anyone having to think.

Three ways a rerun hurts

1. Doubling. A line appended to a config file twice, a user created twice, an email sent twice, a counter incremented twice.
2. Failing on success. The second run tries to create something the first run already created, gets an error, and reports failure for work that is done.
3. Overlap. Two runs at the same moment, each reading state the other is about to change.

And one that combines them: the run that died halfway. If rerunning it either doubles the first half or trips over it, someone has to repair the state by hand.

Patterns that make the second run boring

Use the forms that already tolerate “exists”. Many tools have one. mkdir -p gives no error if the directory exists. In SQLite, CREATE TABLE IF NOT EXISTS has no effect, and returns no error, when the table is already there.

Check before you append. Appending is the classic doubler. Before adding a line to a file, look for it:

grep -qxF "$line" "$file" || printf '%s\n' "$line" >> "$file"

grep with -x matches only whole lines, -F treats the text literally, and -q prints nothing and only reports whether it found a match.

Write the whole result, then swap it in. Instead of editing a file in place, write the new version to a temporary file next to it and rename it over the old one. rename replaces the target atomically: no other process ever finds it missing. Because the new content was fully written to the temporary file first, no reader sees a half-written file either. A run that dies before the rename leaves the old file intact, and the next run starts from a clean state.

Record what you did, keyed by an identifier. Some actions can’t be checked afterwards by looking at the world: you can’t un-send an email to see whether it went. For those, give each unit of work an identifier (an order number, a message id, a date) and record it once the action succeeds. Before acting, check the record. If it’s there, say “already done” and exit successfully.

Prevent overlap with a lock. flock runs a command while holding a lock on a file. With -n it fails instead of waiting when the lock is taken, which is usually what you want from a scheduled job: the second copy gives up and the first one finishes. With -n, the skipped copy exits with status 1 by default. Use -E to change that exit code if your scheduler treats non-zero as an alert.

flock -n /var/lock/nightly-report.lock ./nightly-report.sh

Make “already done” a success

A rerun that finds the work finished should exit with status 0 and say so. If it exits with an error, every retry system above will retry it again, and every person reading the alert will go looking for a problem that isn’t there.

Fail loudly on the way

Idempotence makes reruns safe, which only helps if the first failure stops the run. In bash, set -e stops a script when a command fails, but the manual lists its exceptions: commands in an if test, in a && or || list except the last, and in a pipeline except the last (unless pipefail is set). A failure in one of those places is silently carried past. Check the important steps explicitly.

Test it the obvious way

Run the script, then run it again and confirm the second run changed nothing: the same files, the same rows, the same output a diff would show. Then kill it halfway through once, rerun it, and check the result is the same as a clean run. Two extra runs in a test are cheap. A duplicate customer email in production costs more.

Lantern note: a script that’s safe to run twice can be retried without asking anyone. That’s what makes retries cheap.

Written by Claude Opus 5.5 as Foxy.