SpunkArt
Command line and project setup

How to build a shell command piece by piece

A long terminal command looks like noise until you see that it is made of the same few parts every time. This guide takes those parts one at a time, then assembles them into commands you can adapt.

The parts of a command

grep  -rn  --include='*.js'  "TODO"  src/
 |     |          |             |     |
 |     |          |             |     +-- argument: where to look
 |     |          |             +-------- argument: what to look for
 |     |          +---------------------- long option with a value
 |     +--------------------------------- two short options joined
 +--------------------------------------- the program
  • Program. The first word. The shell finds it by searching the folders listed in PATH.
  • Short options. One dash and one letter. Many programs let you join them, so -r -n and -rn are the same.
  • Long options. Two dashes and a word, such as --recursive. Values follow with a space or an equals sign.
  • Arguments. The things the program acts on: files, folders, patterns.

A lone -- tells most programs that options have ended. It lets you work with a file literally called -draft.txt: rm -- -draft.txt.

Quoting decides what the program receives

The shell splits a line on spaces and expands special characters before the program ever runs. Quotes control that.

You typeThe program receives
echo $HOME/*.txtYour home path and every matching file name
echo "$HOME/*.txt"Your home path, then a literal /*.txt
echo '$HOME/*.txt'Exactly $HOME/*.txt, nothing expanded

Double quotes still expand variables but stop word splitting and wildcards. Single quotes stop everything. A good default is to put double quotes around every variable, so a file name with a space stays one argument: cp "$src" "$dest".

Joining commands

OperatorEffect
a | bOutput of a becomes input of b
a && bRun b only if a succeeded
a || bRun b only if a failed
a ; bRun both, whatever happens
a > fWrite output to file f, replacing it
a >> fAppend output to f
a < fRead input from f
a 2> fWrite only error messages to f

Assembling a real command

Goal: list the ten largest log files under /var/log. Build it in stages and run each stage before you add the next.

  1. Find the files: find /var/log -type f -name '*.log'
  2. Get their sizes: add -exec du -k {} +, which prints size in kilobytes and the path.
  3. Sort by the number, biggest first: | sort -rn
  4. Keep ten lines: | head -n 10
find /var/log -type f -name '*.log' -exec du -k {} + | sort -rn | head -n 10

Another example: count how often each HTTP status appears in an access log where the status is the ninth space-separated column.

awk '{print $9}' access.log | sort | uniq -c | sort -rn

uniq -c only merges neighbouring lines, which is why the first sort is needed.

Passing a list to another program

xargs turns lines of input into arguments. It splits on spaces by default, so file names with spaces break it. Pair it with null separators:

find . -name '*.tmp' -print0 | xargs -0 rm --

When you can, the -exec ... {} + form of find avoids xargs entirely.

Safety habits

  • Swap a destructive verb for echo or ls first, and check the list before you delete.
  • Read the manual page: man tar, or tar --help for a summary.
  • Remember that options differ between GNU tools on Linux and BSD tools on macOS. sed -i is the classic example.
  • In scripts, start with set -euo pipefail so a failed step stops the script instead of running on.

Questions

Why does my command work in the terminal but not in a script?

Scripts may run under a different shell, without your aliases, and from a different folder. Use full paths and a shebang line.

What does tar -czf mean?

Create an archive (c), compress it with gzip (z), and write it to the file name that follows (f). Example: tar -czf site.tar.gz site/.