Shell quoting, pipelines, and exit codes

A shell script is a list of words that the program has not seen yet. Bash rewrites those words before it runs anything: it expands variables, splits what was left unquoted, and turns a glob into filenames. The exit status comes back afterwards, and a pipe hides every status except the last one unless you ask. The usual "it deleted the wrong files" and "the loop counted zero" bugs are that rewrite, not the command you thought you typed.

This note is Bash. Debian and Ubuntu use dash for /bin/sh, and dash rejects pipefail. A script that needs it wants #!/usr/bin/env bash. Running the file with sh script ignores the shebang and parses it as dash. macOS still ships Bash 3.2 as /bin/bash, and env bash may find that one: pipefail and process substitution work there, globstar needs Bash 4, and lastpipe needs Bash 4.2. Current Linux Bash is 4 or 5.

What happens before the command runs

Bash reads a line, expands what it must, splits the unquoted parts into words, then replaces globs with filenames. The program named by the first word sees the words that survive. It does not see the quotes. Quotes are instructions to Bash, and they are gone by the time rm or find starts.

Variables, splitting, and globs

An unquoted $dir is not one argument. Bash inserts the value, splits it on spaces, tabs, and newlines (the default IFS), then treats *, ?, and [ inside those pieces as filename patterns.

dir="My Documents"
printf '<%s>\n' $dir

That prints <My> and <Documents>. Quote the expansion and both steps are skipped. The space stays inside one argument.

printf '<%s>\n' "$dir"

A glob that matches nothing is left as the pattern text. In an empty directory, printf '%s\n' *.log prints *.log, which is the opposite of "no files". shopt -s nullglob makes a miss expand to nothing, so a for loop runs zero times. shopt -s failglob makes a miss an error, so a typo such as *.lg stops the script instead of looking for a file with that exact name. The default, leaving the pattern text in place, is the one that surprises a delete script. shopt lasts for the rest of the script, so set it where you mean it.

* does not match a leading dot. .env survives both *.log and *. ** does not walk subdirectories unless shopt -s globstar is on, and that option does not exist in Bash 3.2. Without it, ** cannot match a slash, so it behaves as *, and logs/**/*.log means one directory under logs, not every nested file. With globstar, the same pattern reaches logs/a/b/c.log.

find does its own matching. The pattern belongs in quotes so Bash does not expand it first. If the current directory contains here.log, an unquoted -name *.log becomes -name here.log and the search silently changes.

find "$log_dir" -type f -name '*.log' -print

The three quoting styles

Single quotes are literal. Nothing inside them expands: no variables, no command substitution, no globs. A single quote cannot appear inside single quotes. Close the quote, add an escaped quote, and open it again. 'o'\''reilly' is the eight characters of o'reilly.

Double quotes allow parameters ($dir, ${dir:-/var/log}), command substitution ($(date +%F)), and arithmetic ($((count + 1))). They still suppress splitting and globs, which is why "$dir" is the form for almost every path. A backslash inside them escapes $, a backtick, ", \, and a newline. In an interactive session, ! is history expansion and double quotes do not stop it. Scripts leave history expansion off unless you set -H.

$'...' is ANSI-C quoting. $'\t' is a tab, and it is the reliable way to put a newline or a tab in a word. $"..." looks similar and is not the same thing. Bash treats that form as a string to be translated. Do not reach for it when you wanted a tab.

Prefer $(...) to backticks. Backticks are harder to nest, and a backslash inside them follows older rules. Command substitution strips every trailing newline. Left unquoted, the remaining text is split and globbed.

printf '<%s>\n' "$(printf 'hi\n\n')"
printf '<%s>\n' $(printf 'a b\nc')

The first prints <hi>: both trailing newlines are gone. The second prints <a>, <b>, and <c>. Assign it quoted when you wanted the text: message="$(generator)".

"$@" and "$*" join the script's arguments in different ways. "$@" is one word per argument, which is what you pass on. "$*" is a single word, joined with the first character of IFS (a space, unless you changed it). Unquoted, both are split again.

printf '<%s>\n' "$@"

Exit status

Every command returns a small integer. Zero means success. Anything else means failure. $? is the status of the last foreground command, and the next command replaces it, so save it on the following line if you still need it.

generator
status=$?
printf 'generator exited %s\n' "$status" >&2

A few values are conventions rather than something your program invented:

set -e asks Bash to exit when a command fails. It does not apply to the test of if, while, or until. "No match" is a normal answer from grep, and this does not abort:

if grep -q 'ERROR' "$log"; then
  printf 'found an error line\n' >&2
fi

It also skips a failure that is not the last command to run in an && or || list. false && echo no continues. true && false stops the script. set -u turns an unset variable into an error. ${1:?usage: tidy-logs DIRECTORY} requires the argument, prints your message, and exits. ${dir:-/var/log} supplies a default and is safe under set -u.

set -e is not a seal around rm. It only reacts after a command has already returned non-zero. A command that deleted the wrong tree and returned zero looks like success. The checks have to happen before rm starts.

Pipelines

A pipeline's status is the status of the last command. false | true is success. gzip -dc archive.gz | tar -xf - reports whatever tar returned, even when gzip could not read the archive. set -o pipefail changes the rule: the status is the rightmost non-zero status in the pipe, or zero if every stage succeeded. set -euo pipefail at the top of a Bash script is the combination that catches this. dash rejects pipefail as an illegal option, which is another reason that line does not belong in a /bin/sh script.

pipefail also changes a pattern that looks harmless. head closes its input once it has enough lines. The writer receives SIGPIPE and exits 141. Without pipefail the pipeline follows head and looks successful. With pipefail it fails, and set -e stops the script.

set -o pipefail
printf '%s\n' {1..20000} | head -n 1

Prefer a command that stops itself, such as grep -m 5, when you only wanted the first matches.

In Bash, every command in a pipeline runs in a subshell, including the last. An assignment on the right-hand side does not survive.

count=0
printf 'a\nb\n' | while IFS= read -r line; do
  count=$((count + 1))
done
printf '%s\n' "$count"

$count is still 0. IFS= read -r is doing the right thing for each line: -r stops backslash escapes, and the empty IFS keeps leading and trailing spaces. A plain read would have trimmed them. The zero is the subshell, not read.

Bash 4.2 can run the last stage in the current shell with shopt -s lastpipe, and only when job control is off, which it is in a script. Process substitution, <(...), works in Bash 3.2 and does not need that option:

count=0
while IFS= read -r line; do
  count=$((count + 1))
done < <(printf 'a\nb\n')
printf '%s\n' "$count"

That prints 2. A failure inside <(...) does not become the while loop's status, so do not hide find in there if you need set -e to notice that find could not search. Write the list to a file first. The script below does that. A while test is one of the places set -e stays quiet, so read returning 1 at end of file ends the loop instead of killing the script.

A log tidy you can run twice

The script lists *.log files under one directory that are older than 14 days. With no flag it only prints. --apply removes the files it listed. find -mtime +14 means strictly more than fourteen periods of 24 hours, so a file from exactly a fortnight ago stays. find returns non-zero if it cannot read a directory it meant to search, and set -e then stops the script rather than deleting from a partial list.

#!/usr/bin/env bash
set -euo pipefail

apply=0
if [ "${1:-}" = "--apply" ]; then
  apply=1
  shift
fi

log_dir=${1:?usage: tidy-logs [--apply] DIRECTORY}

case "$log_dir" in
  /*) ;;
  *) printf 'Need an absolute path: %s\n' "$log_dir" >&2
     exit 1 ;;
esac

case "$log_dir" in
  / | /. | /etc | /home | /root | /tmp | /usr | /var)
    printf 'Refusing a broad directory: %s\n' "$log_dir" >&2
    exit 1 ;;
esac

if [ ! -d "$log_dir" ]; then
  printf 'Not a directory: %s\n' "$log_dir" >&2
  exit 1
fi

list=$(mktemp)
trap 'rm -f "$list"' EXIT
find "$log_dir" -type f -name '*.log' -mtime +14 -print0 >"$list"

count=0
while IFS= read -r -d '' path; do
  count=$((count + 1))
  if [ "$apply" -eq 1 ]; then
    rm -f -- "$path"
    printf 'removed %s\n' "$path"
  else
    printf 'would remove %s\n' "$path"
  fi
done <"$list"

printf '%s file(s)\n' "$count"

mktemp creates the list outside the tree being scanned, and the trap removes it on any exit, including the exits set -e triggers. -print0 and read -d '' put a NUL between paths, so a name with a space or a newline stays one path. rm -f -- treats a filename that starts with - as a filename. The first run, without --apply, is the review. The second run deletes only the paths find wrote.

The patterns in case are not pathname globs. /* matches every absolute path, including /, because a case pattern's * may match a slash. The second case is an exact list: /var is refused and /var/log/nginx is not. A trailing slash is a different string, and the check does not strip one.

[ "$apply" -eq 1 ] is the POSIX test command, so the left-hand side has to be quoted. [[ is a Bash keyword: it does not split words, and [[ $name == *.log ]] matches a pattern on the right. Inside [, that right-hand side is a literal string.

The expansion that removes the wrong tree

GNU rm refuses rm -rf /. It says it is dangerous to operate recursively on /, and it stops, unless someone passes --no-preserve-root. That check looks for the operand /. It does not look at what a glob became.

dir=""
set -f
printf '<%s>\n' "$dir"/*
set +f

set -f turns globbing off, so this prints </*> and does not expand it. Without set -f those words are the glob /*: the slash is the neighbour of the empty variable, and the star was unquoted. Bash then passes every name in the root directory. The arguments are /bin, /etc, and the rest, not /, so the GNU failsafe does not apply. Do not run the unquoted form to see. nullglob turns the same typo into zero names, and rm -rf with no path is a different mistake, not a safe one.

Quoting the slash is not protection either. "$dir/" is one word, and an empty dir makes that word /. Refuse an empty value, a relative path, and a directory you would not empty, before rm runs. ${1:?...} and the two case blocks are that guard.

for file in $(ls) splits on spaces. Iterate the glob and quote the body: for file in *.log; do printf '%s\n' "$file"; done. With the default rules that loop runs once, on the literal text *.log, when nothing matches. nullglob makes it run zero times. echo is a poor way to print a name, because options and backslashes depend on the shell. printf '%s\n' "$file" prints the name you had.

A short checklist

• Shebang #!/usr/bin/env bash when you use pipefail, [[, or process substitution. Do not run that file with sh.
• Start with set -euo pipefail. set -e ignores a failure used as a condition.
• Quote "$dir", "$file", "$@", and find -name '*.log'.
• A pipeline reports the last command unless pipefail is on. head then often yields 141.
• Do not store state in a while read on the right of a pipe.
• For a delete script, choose nullglob or failglob. The default leaves the pattern text in place.
• Print the plan, then delete. Refuse an empty path before rm sees it.