How to build a shell command piece by piece
A long terminal command looks like noise until you see that it is made of the same few parts every time. This guide takes those parts one at a time, then assembles them into commands you can adapt.
The parts of a command
grep -rn --include='*.js' "TODO" src/
| | | | |
| | | | +-- argument: where to look
| | | +-------- argument: what to look for
| | +---------------------- long option with a value
| +--------------------------------- two short options joined
+--------------------------------------- the program
- Program. The first word. The shell finds it by searching the folders listed in
PATH. - Short options. One dash and one letter. Many programs let you join them, so
-r -nand-rnare the same. - Long options. Two dashes and a word, such as
--recursive. Values follow with a space or an equals sign. - Arguments. The things the program acts on: files, folders, patterns.
A lone -- tells most programs that options have ended. It lets you work with a file literally called -draft.txt: rm -- -draft.txt.
Quoting decides what the program receives
The shell splits a line on spaces and expands special characters before the program ever runs. Quotes control that.
| You type | The program receives |
|---|---|
echo $HOME/*.txt | Your home path and every matching file name |
echo "$HOME/*.txt" | Your home path, then a literal /*.txt |
echo '$HOME/*.txt' | Exactly $HOME/*.txt, nothing expanded |
Double quotes still expand variables but stop word splitting and wildcards. Single quotes stop everything. A good default is to put double quotes around every variable, so a file name with a space stays one argument: cp "$src" "$dest".
Joining commands
| Operator | Effect |
|---|---|
a | b | Output of a becomes input of b |
a && b | Run b only if a succeeded |
a || b | Run b only if a failed |
a ; b | Run both, whatever happens |
a > f | Write output to file f, replacing it |
a >> f | Append output to f |
a < f | Read input from f |
a 2> f | Write only error messages to f |
Assembling a real command
Goal: list the ten largest log files under /var/log. Build it in stages and run each stage before you add the next.
- Find the files:
find /var/log -type f -name '*.log' - Get their sizes: add
-exec du -k {} +, which prints size in kilobytes and the path. - Sort by the number, biggest first:
| sort -rn - Keep ten lines:
| head -n 10
find /var/log -type f -name '*.log' -exec du -k {} + | sort -rn | head -n 10
Another example: count how often each HTTP status appears in an access log where the status is the ninth space-separated column.
awk '{print $9}' access.log | sort | uniq -c | sort -rn
uniq -c only merges neighbouring lines, which is why the first sort is needed.
Passing a list to another program
xargs turns lines of input into arguments. It splits on spaces by default, so file names with spaces break it. Pair it with null separators:
find . -name '*.tmp' -print0 | xargs -0 rm --
When you can, the -exec ... {} + form of find avoids xargs entirely.
Safety habits
- Swap a destructive verb for
echoorlsfirst, and check the list before you delete. - Read the manual page:
man tar, ortar --helpfor a summary. - Remember that options differ between GNU tools on Linux and BSD tools on macOS.
sed -iis the classic example. - In scripts, start with
set -euo pipefailso a failed step stops the script instead of running on.
Questions
Why does my command work in the terminal but not in a script?
Scripts may run under a different shell, without your aliases, and from a different folder. Use full paths and a shebang line.
What does tar -czf mean?
Create an archive (c), compress it with gzip (z), and write it to the file name that follows (f). Example: tar -czf site.tar.gz site/.