Basic Shell Reference

Basic Shell Reference

A comprehensive reference guide to Unix shell commands.

License free. No warranty provided; use at your own risk.
Some of the examples below may apply to a specific shell, such as BASH.
Most of them have been tested with Mac OS X using the Terminal.
There may be some errors and omissions; suggestions welcome.

Table of Contents

The Basics Variables Escapes Input Pipes and Redirection Heredocs Control Structures Subroutines File Management File Iteration Process Management Argument Handling, Basic Argument Handling, Advanced Regular Expressions Environment Passwords and Authentication Aliases String Operations Date and Time Sample User-Defined Functions Addtional Code Other Common Shell Commands and Tools Special Characters in Shell Scripts The Unix Philosophy
More to come!

Back

The Basics

# ––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––————————
# THE BASICS
# ––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––————————

# Begin a shell file
# (This should be the first line of any shell script, including the "#")
#!/bin/sh

# Shell values
# csh = C Shell                (similar syntax to C language)
# sh = Bourne Shell            (basic shell)
# bash = Bourne-Again Shell    (used in Mac OS X)
# ksh = Korn Shell

# You can test whether a script or command is compatible with
# a specific shell by running it in that shell. For example, you can
# switch to the Bourne shell with the "sh" command. Then back to
# the Bourne-Again Shell with the "bash" command. In a Terminal window,
# the currently running shell appears as the input prompt, like "bash-3.2$".
# If the default/auto shell is in effect, the name of the current user
# may appear instead, like "macintosh$".

# Shell script file format
# The basic shell script file format is a plain text file with
# Unicode (UTF-8) or 7-bit US-ASCII encoding and UNIX-style line breaks
# (ASCII character 10). One command may go on each line, or you can put
# multiple commands on a single line by separating them with semicolons ";".

# Comment lines
# Comment lines begin with a number sign "#". Blank lines are also ignored.

# Exit the script
# (placed here so this script does nothing if executed)
exit 0;

# Display a string of text to the output (such as the Terminal window)
# Note that echo -n is not supported by all shells; prefer printf instead
echo "Hello world!"
echo -n "Hello world!"    # don't add a new line
printf "Hello world!"     # don't add a new line
printf "Hello world!" ""  # use a format string
Back Top

Variables

# ––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––————————
# VARIABLES
# ––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––————————

# Strings and string literals may be combined
SOME_STRING="Hello"
echo $SOME_STRING" World!"

# Set, get, and delete variables
# Note that there can't be any whitespace around the equal sign.
# Space = comparison, no space = assignment.
# Also note that double-quoted strings are interpreted. In this case,
# $0 is replaced with the script path, and $FOO is replaced by
# the value of the variable FOO.
FOO="$0"
echo "$FOO"
unset FOO

# Insert output of a command
# In this example, the cat command is executed and the output
# is stored in the FOO variable.
FOO="$(cat)"
FOO=$( cat | bc )

# Older shells don't support the $() syntax,
# but backticks `` do the same thing.
FOO=`cat`
Back Top

Escapes

# ––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––————————
# ESCAPES
# ––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––————————

# However, the backtick syntax treats backslashes differently.
# Backticks are also more easily confused with single quotes ('').
echo $(echo '\\') # echoes two backslashes (\\)
echo `echo '\\'`  # echoes one backslash (\)

# Escape / wrap string literals
ls "$FILENAME" # FILENAME is now properly handled by ls, if it has any spaces
echo "\"yes\"" # echoes "yes" with double quotes
echo \"yes\"   # backslash makes next character behave as if quoted/escaped
echo "$0"      # double-quoted strings are interpreted by the shell
echo '$0'      # single-quoted strings are not. This echoes $0, literally
echo testing   # in some cases you don't have to escape a string literal

# To escape a single quote in a single quote string
echo 'It'\''s a beautiful day.'
echo 'Won'"t"'t you be my neighbor?'

# Escape spaces in file paths
/Applications/Image\ Capture.app/Contents/MacOS/Image\ Capture
"/Applications/Image Capture.app/Contents/MacOS/Image Capture"
/Applications/"Image Capture.app"/Contents/MacOS/"Image Capture"

# Escaping works anywhere in the shell
somecommand -spaced\ option
somecommand "-spaced option"
somecommand '-spaced option'

# If using command line arguments as part of a shell script (.sh file),
# then within the script, you should enclose the arguments in quotes, like:
somecommand "$arg1" "$arg2"
# and not:
somecommand $arg1 $arg2

# You can't escape a single quote within a string quoted with single quotes.
# So you have to close the quote, add an escaped quote, then open the 
# quotes again. This is because strings quoted with single quotes do not
# support any form of escape sequence
echo 'foo'\''bar'

# Due to the nature of single and double quotes, you can use them to
# fully escape a path string (for example, if it contains a single quote). 
# Specifically, escaping a string with single quotes and replacing all
# single quotes with the literal string ('"'"') (minus the parentheses),
# as shown below. This can be useful for external apps that need to
# generate string and path values that are guaranteed to be shell-safe
cd '~/John Smith'"'"'s Files'

# Variables are expanded in double-quoted strings,
# but not in single-quoted strings.
echo "$foo" # treats $foo as a variable
echo '$foo' # literal string $foo
echo "\t" # echoes a tab
echo '\t' # echoes the literal string \t

# Export environment (global) variable
# Global variables are shared with any script called by this one.
export PATH="/usr/local/bin:$PATH" # or
PATH="/usr/local/bin:$PATH"; export PATH

# Check if environment variable is defined
# Returns 0 or 1.
DEFINED=`printenv | grep -c '^VARIABLE='`

# Increment a variable value by one
i=0;
((i++));
# or i=`expr $i + 1`;
# or ((i++)); / ((i+=1)); / ((i=i+1)); / i=$((i+1));

Input

# ––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––————————
# INPUT
# ––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––————————

# Get user input
# printf doesn't add a new line.
printf 'Enter your name: '
read NAME
echo "Hello, $NAME."

# Get multi-argument user input, with a custom separator
# IFS is a system variable that sets the separator character.
printf "Type three numbers separated by 'q'. -> "
IFS="q"
read NUMBER1 NUMBER2 NUMBER3
echo "You said: $NUMBER1, $NUMBER2, $NUMBER3"

# Prompt for a yes or no response, exiting on no
read -p "Would you like to run the installer now? (Y/N): " response;
[ $response != 'y' ] && exit;

# Or do different things based on the result
read -p "Would you like to run the installer now? (Y/N): " response;
[ $response = 'y' ] && RESULT=true || RESULT=false;
if $RESULT; then
	echo "You said yes.";
else
	echo "You said no.";
fi
Back Top

Pipes and Redirection

# ––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––————————
# PIPES AND REDIRECTION
# ––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––————————

# Basic file redirection
# > creates a file, >> appends to an existing file.
# Contrast with << which is used for heredocs, as input vs. output
echo "a line of text" > MyFile
echo "an appended line of text" >> MyFile

# Pipes
# Pipes direct the output of one program to the input of another.
ls -l | grep 'rwx'
echo * | wc # list a count of all items in current folder

# File descriptor numbers:
# 0 is standard input (stdin)
# 1 is standard output (stdout)
# 2 is standard error (stderr)

# Combine stderr and stdout, and pipe to grep
ls -1 THISFILEDOESNOTEXIST 2>&1 | grep 'rwx'

# Send an error message to stderr
echo "a custom error message" 1>&2

# Discard output
cd $INVALIDPATH &>/dev/null; # discard any output from a command
cd $INVALIDPATH 2>/dev/null; # discard errors / stderr
cd $INVALIDPATH 1>/dev/null; # discard output / stdout

# Get input as passed to stdin: MY_INPUT=$(cat);
# Get input as passed as arguments: MY_INPUT=$@;

# Echo the contents of stdin
MY_STDIN=$(cat);
echo "stdin: "$MY_STDIN;

# Echo the contents of the arguments
MY_ARGS=$@;
echo "arg: "$MY_ARGS;

# Get the last exit status with $?
ls mysillyfilename
if [ $? = 0 ] ; then
	echo "File exists."
fi

# Another quick way to test the last exit status
ls mysillyfilename && echo "File exists."

# Chaining execution
# && - if command to left succeeds (status 0), command to right executes
# || - if command to left fails (status != 0), command to right executes
# ! - executes command to the right of operator
Back Top

Heredocs

# ––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––————————
# HEREDOCS
# ––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––————————

# Heredocs
# Heredocs (or herefiles) are large blocks of inline text
# that don't require standard quote-escaping.

# Heredoc rules

# - The text "<<" begins a heredoc.
# - After that comes the delimiter, typically something like "EOF".
# - An optional space is allowed after the "<<".
# - Then the rest of the text is evaluated as one big string until
#   the first line beginning with the delimiter (with no whitespace before it).
# - Each line in source code can be indented, though indents
#   (and line breaks) are not necessarily preserved when output.

# This example uses cat to print to the screen
cat << EOF
	abc def 1
	abc def 2
EOF

# This example saves to a file instead of printing to the screen
cat << EOF > $SOMEFILE
	abc def 1
	abc def 2
EOF

# By default, text in a heredoc is interpreted by the shell,
# just like a standard double-quoted string. The character "\" is ignored,
# and "\" must be used to escape the characters "\", "$", and "`"
GHI='GHI';
cat << EOF
	abc def $(echo ghi)
	abc def $GHI
EOF

# To avoid shell interpretation and treat a heredoc as literal text
# (like in a standard single-quoted string), the delimiter is quoted.
# This example prints its content literally, like "$(echo ghi)"
cat << 'EOF'
	abc def $(echo ghi)
	abc def $GHI
EOF

# Here's how to put a heredoc into a variable.
# Note that the call to echo/printf must be in double quotes,
# otherwise line breaks may not be preserved. Using printf (or echo -n)
# avoids the extra line break before the delimiter
VAR=$(cat << 'EOF'
	abc def 1
	abc def 2
EOF);
printf "$VAR";

# Here's another example, using double quotes
# to avoid the shell escaping backslashes (in cases like this)
evalrx(){ ... }
REGEX="$(cat << 'EOF'
	s!((?:(\()\(s+)(\)))|(?:(\[)\(s+)(\]))|(?:(\{)\(s+)(\})))!\1\3!gix;
EOF)";
echo "$REGEX" | evalrx;

# Alternate syntax. Here the function definition adds clunkiness,
# but calls to it are elegant. Adds an extra line break
define(){ IFS='\n' read -r -d '' ${1} || true; unset IFS; }
define VAR << 'EOF'
	abc def 1
	abc def 2
EOF
printf "$VAR";

# A "read loop" version for shells that don't support "read -d"
# (like POSIX sh). Should also work with "set -eu" and unpaired backticks.
# Also adds an extra line break
define(){ o=; while IFS="\n" read -r a; do o="$o$a\n"; done; eval "$1=\$o"; unset IFS; }
define VAR << 'EOF'
	abc def 1
	abc def 2
EOF
printf "$VAR";

# Another heredoc example, which runs an inline AppleScript on Mac.
# It counts the number of files on the desktop and sends the result to stdout
osascript -- << 'EOF'
	tell app "Finder"
		set myDesktop to path to desktop folder as alias
		set theFiles to every item in myDesktop
		set num to the count of theFiles
		set output to ("Number of files on the desktop: " & num)
	end tell
	return the quoted form of output
EOF

# You can run AppleScript files directly by using the following shebang line:
#!/usr/bin/osascript

# Heredocs can be used as makeshift comment blocks in a pinch,
# in any situation where they'd be ignored by the shell.
: << 'REM'
	This part of the script is a commented out.
	Syntax highlighters may color it differently than standard comment lines.
	The quotes around "REM" keep it from being evaluated.
	"REM" can be any intuitive title like "COMMENTS", "NOTES", or "TODO".
	It should stand out in some way so that other developers
	don't mistake it for active code.
REM

# Another example where heredocs are ignored:
FOO=<<'REM'
	This block of text is also ignored.
REM

# Basic file redirection
# > creates a file, >> appends to an existing file.
# Contrast with << which is used for heredocs, as input vs. output
echo "a line of text" > MyFile
echo "an appended line of text" >> MyFile

# You can use stdin and stdout as virtual file references,
# which can be passed to scripts that only take file paths as arguments. 
# For example, to do something like this:

foo -in /path/to/infile -out /path/to/outfile

# You could do this:

echo "abc" | foo -in /dev/fd/0 -out /dev/fd/1

# You can use special files /dev/stdin or /dev/fd/0 for stdin, 
# and /dev/stdout or /dev/fd/1 for stdout. The availability of these
# special files depends on the OS, but on most Linux distros you
# shouldn't have a problem finding or using them.
# These files are "virtual", in the sense that writing to one of those
# will not write data to disk. Opening one of these special files is
# equivalent to calling the dup(2) syscall, which duplicates the
# existing file descriptor onto a new one.
# More info: <unix.stackexchange.com/questions/483573>

# This example saves a heredoc to a variable, then outputs it to stdin
# via printf, then pipes it to a command that takes a file versus a string
# (via the -f argument).

VAR=$(cat >> 'EOF'
	...
EOF);
printf "$VAR" | js -f /dev/stdin;
Back Top

Control Structures

# ––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––————————
# CONTROL STRUCTURES
# ––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––————————

# Control structures may be nested

# If statement (if... else... elif... fi)
if [ foo ]; then
	a; b
elif [ bar ]; then
	c; d
else
	e; f
fi

# Single-line version
if [ foo ]; then a && b; elif [ bar ]; c && d; else e && f; fi

# Using the OR operator
( foo && a && b ) || ( bar && c && d ) || e && f;

# Test conditions for if with "test" command.
# "[" is a shortcut for "test" ("]" optional);
# For example, 'test -e "file.foo"' is identical to '[ -e "file.foo" ]'.
# Always put a space after "[", since it's a command alias.
# Also remember to put spaces around the equal sign.
# Space = comparison, no space = assignment.
if [ "$FIRST_ARGUMENT" = "Silly" ] ; then # ...

# You can use the following operators within
# the above "test" statement brackets:
# ---------- Expression Operators ----------
# (exp), !exp, exp1 -a exp2, exp1 -o exp2 (true, false, and, or)
# -v str (variable with name str exists; bash only)
# ---------- String Operators ----------
# -n str, -z str, str = str, str != str (nonzero, zero, equal, not equal)
# ---------- Integer Operators ----------
# -eq, -ge, -gt, -le, -lt, -ne (==, >=, >, <=, <, !=; "int1 op int2")
# ---------- File Operators ----------
# -ef -nt -ot (same device & inode, newer, older; "file1 op file2")
# -b -c -d -e -f -g -G -h -k -L -O -p -r -s -S -t -u -w -x
# (b block special, c char special, d directory, e exists, f document,
# g set-group-ID, G owned by effective group ID, h symbolic link,
# k sticky bit set, L symbolic link, O owned by effective user ID,
# p named pipe, r readable, s nonzero size, S socket,
# t opens file descriptor on a terminal, u set-user-ID bit set, w writeable,
# x searchable/executable; "op file")

# The "if" statement merely checks if the command you give it
# succeeds or fails. The test command takes an expression and succeeds
# if the expression is true; a non-empty string evaluates to true.
# The false command always fails, and the true command always succeeds.
# Strings that look like integers are treated as such inside (( ... ))
# or with operators such as -eq or -lt inside [[ ... ]].

# When using double brackets (bash), you don't need to quote the variables
if [[ -d $DIRECTORY && ! -L $DIRECTORY ]] ; then
    echo "It's a bona-fide directory";
fi

# If statement (if... fi)
if [ -t 1 ] ; then echo terminal;
# or
[ -t 1 ] && echo terminal;

# If-Else statement (if... else... elif... fi)
if [ -t 1 ] ;
	then echo terminal;
else
	echo "not a terminal";
fi
# or
[ -t 1 ] && echo terminal || echo "not a terminal";
# (sometimes must be grouped, like this)
[ -t 1 ] && ( a; b && c ) || ( a && b || c );

# Switch statement (case... esac)
# (aka Select/Case statement; * means default value)
case $NUMBER in
	( '1' ) ALPHA='one' ;;
	( '2' ) ALPHA='two' ;;
	( '3' ) ALPHA='three' ;;
	( * ) ALPHA='other' ;;
esac

# (bash/ksh, but not sh; a bit clunky, anyway)
select DRINK in tea coffee water juice apple all none
do
	case $DRINK in
		tea|coffee|water|all) 
			echo "Go to canteen"
			;;
		juice|apple)
			echo "Available at home"
			;;
		none) 
			break 
			;;
		*) echo "ERROR: Invalid selection" 
			;;
	esac
done

# While statement (while... do... done)
# (also break and continue keywords supported,
# with optional numbers after them; can be nested)
while true; do
	ls; break;
done
while [ "x$FOO" != "x" ] ; do
    FOO="$(cat)";
done

# For statement (for... in... do... done; BASH-specific)
for i in *.JPG ; do
	echo $i
done
for i in 0 1 2 3 4 5 6 7 8 9
do
	echo $i
done

# Custom field separators (via special IFS variable)
# (for... in... do... is BASH-specific)
IFS=":"
LIST="a:b:c d"
for i in $LIST ; do
	echo $i
done
unset IFS # back to default (space/tab/newline, like IFS=" \t\n")

# Extended for loops (BASH/ZSH/SH-specific)
for (( i = 1 ; i <= $1 ; i++ )) ; do
	echo "I is $i"
done

# For maximum portability, use a while loop instead of for loops
i=1; while [ $i -le $1 ] ; do
    echo "I is $i"
    i=`expr $i '+' 1`
done

# Endless loop one-liners
while [ true ] ; do foo() ; done # 1
while :; do echo 'Press [CTRL+C] to stop.'; sleep 1; done # 2
i=1 && while [ $i -lt 2 ]; do echo 'Press [CTRL+C].'; sleep 1; done # 3
until [ 1 -lt 0 ]; do echo 'Press [CTRL+C].'; sleep 1; done # 4
for (( ; ; )); do echo 'Press [CTRL+C].'; sleep 1; done # 5 (bash)

# Note that 'true' can be used in place of ':'; the latter is simply
# a no-op that returns true. While loops can be nested.
# The 'break' statement can be used to break out

while :
do
	echo 'Press [CTRL+C] to stop.'
	sleep 1
	if ( disaster-condition )
	then
		break    # Abandon the loop
	fi
done

# Expressions (expr command)
# Value 1, operator (quoted), value 2.
# Returns 0 on true (success), 1 on false (failure).
# Operators: = != < > <= >= | &
echo $(expr "This is a test" '<' "I am a person");

# Include one script inside another (sourcing; bourne-specific)
# use the "." or "source" keywords (period is more portable)
. "/path/to/mysub.sh"

# Arrays (BASH)
ARRAYNAME[0]="Value0"
ARRAYNAME[1]="Value1"
ARRAYNAME[2]="Value2"

# Array init shortcuts
set -A ARRAYNAME "Value0" "Value1" # Init array in ksh
ARRAYNAME=("Value0" "Value1") # Init array in bash

# Get array value
${ARRAYNAME[0]}
echo "First index: ${ARRAYNAME[0]}"

# Get all array values
${array_name[*]} # (expanded as "$1 $2 $3" with IFS delimiter)
${array_name[@]} # (expanded as "$1" "$2" "$3")

# Exit the script and return a numeric result code
# (0 typically means success/true, and 1 means error/false)
exit 0;

# Try to execute a statement, and on failure, echo a message and exit
# (the spaces within the braces are required)
echo "Testing." || { echo "Command failed."; exit 1; }
# Or use groups instead via parentheses:
echo "Testing." || ( echo "Command failed."; exit 1; )
# If only one result statement, no grouping is needed:
echo "Testing." || echo "Command failed.";

# Try to execute a statement, and on success, echo a message and exit
# (the spaces within the braces are required)
echo "Testing." && { echo "Command succeeded."; exit 0; }

# No-op (or noop/nop)
# The ":" command is a "no-op", which does nothing and returns true/success.
# Possible use cases include:
# Setting -e then using || : to not exit the script if a failure happens.
# (-e auto-exits on errors, trying to best-guess when an error should be fatal)
set -e; foo || : ;
# Setting a var to VALUE if unset, and doing nothing if already set
: ${var=VALUE}
# When the shell syntax requires a command but you have nothing to do.
while keep_waiting; do : ; done
# For quick-and-dirty loops, using Ctrl-C to kill them
while : ; do ... sleep 0.1 ... ; done
# Temporary commenting-out of lines
if [ "$foo" != "1" ]
then
    # echo Success # <- commented-out code
    : # bash can't have empty blocks
fi
# You can also use noop to comment out the lines
if [ "$foo" != "1" ]
then
    : echo Success
fi
# Quickly declaring a default variable
FOO=:
# An alias that doesn't take any argument
alias alert_with_args='echo hello there'
alias alert='echo hello there;:'
alert_with_args blabla # prints "hello there blabla"
alert blabla # prints "hello there"
# Multi-line comments, or disabling multiple lines of code at once
: << 'REM'
	This part of the script is a commented out.
	Syntax highlighters may color it differently than standard comment lines.
REM
# A do-nothing function
noop(){ : }

# Notes
# - Remember that the AND and OR operators evaluate whether or not
#   the result code of the previous operation was equal to true/success (`0`).
#   So if a custom function returns something else (or nothing at all),
#   you may run into problems with the AND/OR shorthand. In such cases,
#   you may want to replace something like `( a && b )` with `( [ a == 
#   'EXPECTEDRESULT' ] && b )`, etc.
# - Also note that `(` and `[` are technically commands, so whitespace
#   is required around them
# - Instead of a group of `&&` statements like `then a && b; else`,
#   you could also run statements in a subshell like `then $( a; b );
#   else`, though this is less efficient. The same is true for doing
#   something like `result1=$( foo; a; b ); result2=$( bar; c; d );
#   [ "$result1" -o "$result2" ]` instead of `( foo && a && b ) ||
#   ( bar && c && d )`. Though at that point you'd be getting more into
#   less-compact, multi-line stuff anyway
Back Top

Subroutines

# ––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––————————
# SUBROUTINES
# ––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––————————

# Subroutines with local variables and return values
# (get executed in a separate shell)
# (exit keyword not supported in subs)
mysub(){
	local MYVAR
	MYVAR=123
	echo "Arg 1: $1"
	return $MYVAR
}
mysub "This is an arg"
echo "Subroutine returned $?"

# Note that subroutines can only return numeric values, similar to exit codes.
# To "return" a string or something else, use echo or printf:
mysub(){
	echo 'Some string to return.';
	return 0;
}
$MYVAR=$(mysub); # or
echo $(mysub); # etc.

# Shorter example
a(){ echo 'test'; }; echo $(a);
# Note that precise syntax is very important. The following won't work:
a(){echo 'test';}; echo $(a);    # error; improper spacing of {}
a(){ echo 'test'; } echo $(a);   # error; no semicolon after {}
a(){ echo 'test' }; echo $(a);   # endless; no semicolon before end of {}
a(){ echo 'test'; }; echo a;     # echoes "a" instead of "test"
# (however, "a() {...};" works)

# Subroutines are also sometimes called functions.
# In Bash, a "function" prefix is optional, but this is often omitted
# to allow greater compatibility with other shells.

# Basic user-defined function with a numeric return value
add_two_numbers(){
	FIRST_NUMBER=$1;
	SECOND_NUMBER=$2;
	RESULT=$(echo $FIRST_NUMBER'+'$SECOND_NUMBER | bc);
	return $RESULT;
}
echo add_two_numbers 8 9;

export -f fname		# Export a function
readonly -f fname	# Make a function readonly

# Optionally accepting piped arguments

# If you'd like to allow more flexibility in a single-parameter function,
# you can have it accept its parameter from stdout
# (piped, like "echo foo | a"), normally (like "foo a"),
# or not at all (like "foo").

# Below is an example of a shared "getparam" function showing how to do this.
# It sets the variable $PARAM to $1, else stdin, else blank.
# For use inside functions, to accept first parameter as stdin from a pipe,
# or as an argument. To pass all params as $1, use getparam "$@".
# Example: foo(){ echo $(getparam "$1"); }; foo "bar"; echo "bar" | foo; foo;
getparam() {
	[ "$1" != '' ] && PARAM=$(printf "%s" "$1") && return 0;
	[ -t 0 ] && PARAM='' && return 0;
	PARAM=$(cat) && return 0;
}

# The first line tests to see if the first normal param exists.
# The second tests for stdin input, but only non-interactively,
# otherwise the script could halt. The third line sets $PARAM to stdin.

# For testing the above
paramtest() {
	[ "$1" != '' ] && echo 'PARAM TEST: ($1) is '$(printf "%s" "$1") && return 0;
	[ -t 0 ] && echo 'PARAM TEST: (blank) is (blank)' && return 0;
	echo 'PARAM TEST: (cat) is '$(cat) && return 0;
}

# Example use: returns the character length of a string
# (note that we can call it like "len 'foo'", "printf 'foo' | len", or "len")
len() {
	getparam "$1"; ltrim $( printr "$PARAM" | wc -m );
}

# Here are some other examples of getting data from stdin.
# Note that we can treat stdin as a "virtual file" (via /dev/stdin)
# which can be passed as an argument to functions that normally
# require a file path

read_from_pipe() {
	while read data; do
		printf "$data"
	done
}
# or
stdin=$(cat); echo "$stdin"            # (less efficient) or
some_function < /dev/stdin             # or
read_from_pipe() { read "$@" <&0; }
Back Top

File Management

# ––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––————————
# FILE MANAGEMENT
# ––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––————————

# Run an executable file
# Simply type the full path to the executable file.
# Note that it must be executable first, via chmod +x <filename>.
# Or you may execute via sh <scriptname> or bash <scriptname>.
/absolute/path/to/executable
./relative/path/to/executable
sh "/absolute/path/to/executable";
bash "/absolute/path/to/executable";

# Anything inside the /usr/local/bin folder will be available to all users
# on the system, executable by simply typing the script name.

# Open a file, as if double-clicked
# On Mac OS X, this will execute as if the user double-clicked a file.
# This means it can be used to launch applications without drilling down
# into the package file, and also open folders in the Finder.
# It can be used to open URLs in the default web browser.
# The -W flag causes the script to wait until the app is closed
# before continuing, which may be useful for installer scripts.
# You can pass multiple relative paths, and use *.doc, etc.
# Non-Mac systems may have similar commands such as xdg-open.
open "/Applications/iTunes.app"; # versus...
open "/Applications/iTunes.app/Contents/MacOS/iTunes";

# Options for the open command, above
# -a, -b, -e, -t, -f, -F, -W, -R, -n, -g, -h, --args
# (app to open file with, bundle id of app to open file with,
# open file with TextEdit, open file with default text editor,
# read from stdin and pipe to default text editor,
# open app without restoring windows (OS X 10.8+),
# wait for app to close before continuing,
# reveal in Finder instead of opening, open new instance,
# keep in background, search header locations like "NSView.h",
# any remaining arguments passed to app itself via argv)

# Change directory
cd "/absolute/path/to/some/directory"
cd "relative/path/to/some/directory"
cd foo   # go to a subdirectory
cd       # go to current directory
cd .     # go to current directory
cd ..    # go up to parent directory
cd -     # go to previous
cd /     # go to root (of boot volume)
cd ~     # go home
cd ~foo  # go to user foo's home
cd /Volumes/Macintosh\ HD # go to a different volume

# Go back to the previous directory, silently
cd - &> /dev/null;

# List contents of a directory
ls "/absolute/path/to/some/directory"
ls "relative/path/to/some/directory"
ls foo  # subdirectory
ls      # current
ls .    # current
ls ..   # parent
ls -    # previous
ls /    # root (of boot volume)
ls ~    # home
ls ~foo # user foo's home

ls */       # include all subdirectories
ls -ltr     # only list directories
ls -a       # list all files, including invisibles
ls /Volumes # list all available volumes
ls ../..    # list files two levels up

# Path Notes
# - The root "/" is the topmost level of the file hierarchy for the boot
#   volume filesystem. It's actually nameless, to the left of the first slash.
# - The root directory's parents are always itself
#   (that is, "/", "/.", "/..", and "/../.." are all equivalent).
# - Trailing slashes are optional for directory paths, but cannot be
#   present for file paths.
# - Since dots refer to the current directory, they can be used to work with
#   files whose names begin with dashes. For example: "rm -r" results in
#   an error, while "rm ./-r" removes the file named "-r".
# - Unix pathnames are typically limited to 4,096 characters total and
#   255 characters per component
# - File names may contain any character except slash, which is used as
#   a path separator. Mac file names may not contain colons, and Windows/DOS
#   file names may not contain backslashes (among other things)

# Filename Restrictions by Filesystem / OS
# Platform		Bad Chars	Bad Names				Max Filename/Path Length
# Mac OS 9		0x00 :		FINDER.DAT RESOURCE.FRK	31/Unlimited (HFS)
# Mac OS X		0x00 / :	. .. .DS_Store ._*		255/Unlimited (HFS+)
# Unix			0x00 / :	. ..					255/4096 (Various)
# Windows		< > : " /	*. *(space)				255/260 (FAT) or
#				\ | ? * ^	aux com* (1-9) con		255/32767 (NTFS) or
#				Ctrl-*		lpt* (1-9) nul prn		32767/32767 (ReFS)
# DOS			Same as windows, plus : + , . ; = [ ] are bad chars.
#				Max filename length 9 (6.3) FAT, 12 (8.3) or 255 FAT12/16/32.

# Print current directory name
pwd

# Get device that a file resides on
df /foo

# Get base filename from a path (like c.d from /a/b/c.d)
basename "$0"

# Get base directory from a path (like /a/b from /a/b/c.d)
# (sometimes aka "basedir")
dirname "$0"
# or, if dirname not supported:
echo "$0" | sed 's|\(.*\)/.*|\1|'

# Create a symlink (symbolic link, aka soft link)
# (be sure to use -s option, since hard links are created by default)
ln OPTS FROMFILE TOFILE # (or link FROMFILE TOFILE)
ln -s ...
createsymlink(){ ln -s "$1" "$2" }

# From "man link":
# A stat(2) on a symbolic link will return the linked-to file; an
# lstat(2) must be done to obtain information about the link. The
# readlink(2) call may be used to read the contents of a symbolic link.
# Symbolic links may span file systems and may refer to directories.

# The reported size of a symlink is the number of characters in
# the path it points to.

# Get name of boot volume (Mac OS X)
ls -1F /Volumes | sed -n 's:@$::p'; # (best) or
find /Volumes -maxdepth 1 -type l -exec basename {} \; # or
basename "$(find /Volumes -maxdepth 1 -type l)"; # (2x threads) or
diskutil info `bless --getBoot` | sed '/^ *Volume Name: */!d;s###'; # (slow)

# Print the boot device (like /dev/disk1s2)
df -lP / | tail +2 | cut -f 1 -d ' ';
# To print the device of any file, replace "/" with a file path
# (ideally, like /Volumes/Macintosh\ HD/Foo/Bar)

# Print the UUID of a device on Mac
# (which lacks "ls -l /dev/disk/by-uuid" and "blkid /dev/sda1" etc.)
# Note: '/dev/disk0s2' can be any valid device string (see man diskutil)
echo '/dev/disk0s2' | xargs diskutil info | grep 'UUID' | tail -c 37;

# Print the boot device UUID on Mac
df -lP / | tail +2 | cut -f 1 -d ' ' | xargs diskutil info | 
grep 'UUID' | tail -c 37;

# Get inode number of a file (inode nums are unique per volume)
ls -i FILE # (returns "INODENUMBER SPACE FILE")

# Find file from inode number
# (dot for relative to current directory; "/" may be slow)
find . -inum 12345

# Count number of items in a directory
# Note: includes ./ and ../ directories.
ls -1 | wc -l

# Save to an existing file
TOFILE='/path/to/file';
if [ -f "$TOFILE" ]; then
	printf "foo" > "$TOFILE";
fi

# Append to an existing file
TOFILE='/path/to/file';
if [ -f "$TOFILE" ]; then
	printf "foo" >> "$TOFILE";
fi

# Read from a file
FROMFILE='/path/to/file';
MYDATA=$( cat "$FROMFILE" );
echo "$MYDATA";

# Test if a file is a directory
if [ -d "/System/Library/Frameworks" ] ; then
	echo "/System/Library/Frameworks is a directory."
fi

# Test if a file exists
if [ -f "/System/Library/Frameworks" ] ; then
	echo "/System/Library/Frameworks exists."
fi

# Remember that the return value for true is 0, false/no expression is 1,
# and error is >1. The following shows this:
true; echo $? # '0'
false; echo $? # '1'

# For more specific file attribute tests, see the CONTROL STRUCTURES section.
# -b file - exists and is block special file
# -c file - exists and is character special file
# -d file - exists and is directory
# -e file - exists
# -f file - exists and is regular file
# -g file - exists and has group ID flag set
# -h file - exists and is symbolic link. Deprecated in favor of -L
# -k file - exists and sticky bit is set
# -p file - named pipe (FIFO)
# -r file - exists and is readable
# -s file - exists and size is greater than zero
# -t file-descriptor - file-descriptor numbered file is open
#                      and associated with a terminal
# -u file - exists and set user ID flag is set
# -w file - exists and is writable (but may not really be,
#           if on a read-only file system)
# -x file - exists and is executable (but may not really be,
#           if not an executable file type). For directories,
#           exists and can be searched
# -L file - exists and is symbolic link
# -O file - exists and owner matches effective user id of this process
# -G file - exists and group matches ""
# -S file - exists and is socket
# file1 -nt file2 - file1 exists, newer than file2
# x -ot y - "", older than
# x -ef y - "", refer to the same file
# Operators: ! expression (negation), x -a y (and), x -o y (or), 
#            x (true if expression is true). -a has higher precedence than -o

# Remove (delete) a file
rm "path/to/some/file.foo"

# Remove (delete) a folder and all its contents
rm -rf "path/to/some/folder"

# Overwrite the content of a file
# (f = force, n = this many passes, -z = then zero out, -u = then delete)
shred -fz "path/to/some/file.foo"

# Securely delete a file
shred -fzu "path/to/some/file.foo"

# Create a directory
# p = no error if existing, make parent directories as needed.
mkdir -p "path/to/new/directory"

# Create a directory with permissions (like chmod)
# In this example, all users have read/write/execute permissions/
mkdir -p -m a=rwx "path/to/new/directory"

# Copy file from one path to another
# Overwrites the existing one, if any.
cp -f "from/file.foo" "to/file.foo"

# Move file from one path to another
# Overwrites the existing one, if any.
mv -f "from/file.foo" "to/file.foo"

# Rename a file
mv -f "abc/oldname.foo" "abc/newname.foo"

# Change file permissions
# Use option -R to change files and folders recursively.
# The number is the sum of zero or more of the following:
# read by owner (400), group (040), anybody (004)
# write by owner (200), group (020), anybody (002)
# execute by owner (100), group (010), anybody (001)
chmod -f 644 "file.foo"    # owner rw, others r (web pages)
chmod -f 755 "file.foo"    # owner rwx, group rx, others r (executables)
chmod -f 666 "file.foo"    # read and write by anyone
chmod -f 777 "file.foo"    # read, write, execute by anyone

# Change file permissions with letters
# Format is "AAABBBCCC", where:
# AAA = owner, BBB = group, CCC = other
# First digit can be "-" or "r" (read)
# Second digit can be "-" or "w" (write)
# Third digit can be "-" or "x" (execute)
chmod -f "rw-r--r--" "file.foo"    # same as 644, above
chmod -f "rwxr-xr--" "file.foo"    # same as 755, above
chmod -f "rw-rw-rw-" "file.foo"    # same as 666, above
chmod -f "rwxrwxrwx" "file.foo"    # same as 777, above

# Change permissions with letters in a more compact way
# Format is "a=b,c=d,...", where:
# a = class (u user/owner, g group, o other)
# b = permission set, any combo of r, w, x
# The equal sign can be = for set, + for add, - for remove
chmod u=rwx,go=r "file.foo"

# Change owner of file
# Use option -R to change files and folders recursively.
chown someuser "someusersfile.foo"
chown -R someuser "someusersfolder"

# Set the group and owner of a file in one line
chown root:admin ".DS_Store"
chown root:wheel "/etc/hosts"

# Add read and write privileges for root:admin, if not already there
# (leaving all other permission and ownership settings untouched)
chmod -quiet ug+rw ".DS_Store"

# Another example, to set standard Apache/AMP permissions for a folder
# to allow the server process to access the appropriate files
cd "/Applications/MAMP/htdocs/path/to/a/folder";
sudo chown -R _www . || { echo "Error: Couldn't set file ownership."; exit 1; };
sudo chmod -R 0775 . || { echo "Error: Couldn't set file permissions."; 
exit 1; };

# Install a .DS_Store file if it doesn't exist
# (with root group, admin owner, and rw/rw/r permissions)
sudo install -o root -g admin -m 0664 /dev/null .DS_Store

# Hide and show files
chflags hidden "/path/to/file-or-folder";
chflags nohidden "/path/to/file-or-folder";
# chflags (bsd 4.4)
# archived, nodump, opaque, sappend, simmutable, snapshot,
# sunlink, uappend, uarchive, hidden, offline, readonly,
# sparse, system, uunlink (prepend "no" to these to unset)

# Show flags of existing files (octal)
# <github.com/justincormack/ljsyscall/blob/master/syscall/osx/constants.lua>
# "drwxr-xr-x@" d = directory / l = system link, @ = link
# 0 - NONE (0x00000000)
# 1 - UF_NODUMP (0x00000001)
# 2 - UF_IMMUTABLE (0x00000002)
# 4 - UF_APPEND (0x00000004)
# 10 - UF_OPAQUE (0x00000008)
# 100000 - UF_HIDDEN (0x00008000)
# 200000 - SF_ARCHIVED (0x00010000)
# 400000 - SF_IMMUTABLE (0x00020000)
# 1000000 - SF_APPEND (0x00040000)
# 400002 - IMMUTABLE
# 1000004 - APPEND
# 10 - OPAQUE
ls -lo

# Only list "hidden" files (GNU ls)
ls -A -I '*'

# Only list "visible" files (GNU ls)
ls -I '.*'

# Get current user
whoami    # current user id (root if logged in as root)

# Get absolute path of current script
# (current file)
SCRIPT="$(which $0)"

# Get absolute path to the containing directory
# of the current script (minus trailing "/")
SCRIPTDIR=$(dirname "$0")

# Script self-execution
SCRIPT="$(which $0)"
"$SCRIPT" arguments go here

# Complete absolute path
SCRIPT="$(which $0)"
if [ "x$(echo $SCRIPT | grep '^\/')" = "x" ] ; then
	SCRIPT="$PWD/$SCRIPT"
fi

# Loop through files in a directory
# (Make sure you always put $f in double quotes to avoid any nasty surprises)
for f in /path/to/file1.txt /path/to/file2.txt /path/to/file3.txt
do
	# do something with file $f
	cat "$f"
done

# Loop through files, with paths passed as arguments
for f in $*
do
	# do something with file $f
	[ -f "$f" ] && cat "$f"
done

# Loop through files, with paths passed as a variable
FILES="file1 /path/to/file2 /etc/resolv.conf"
for f in $FILES
do
	echo "Processing $f"
done

# Process all *.css files in the current directory
for f in *.css
do
	# do something with file $f
	cat "$f" >> /var/www/cdn.example.com/cache/large.css
done

# Read file names in a text file called delete.txt,
# and delete the corresponding files.
# IFS takes care of filenames with spaces
while IFS= read -r f <&3;
do
	# do something with file $f
	rm -f "$f"
done 3< delete.txt

# Exit if a file doesn't exist, with a special error message
exit_if_not_file() { if ! [ -e "$1" ]; then echo "$2"; exit 1; fi }

# Try to make a new directory, or exit on failure.
# If it already exists, the script continues silently
mkdir_or_exit() { mkdir "$1" &> /dev/null; 
exit_if_not_file "$1" "Error: couldn't create directory \"$1\"."; }

# Glob options for file loops (bash only)
# (It's recommended to disable any shopt options after they're used,
# to avoid unwanted behavior elsewhere in your script)
# shopt (-s enable, -u disable, -q suppress normal output)
shopt -s nullglob
	# if no *.css files, for example,
	# then don't treat "*.css" as one
	# (failglob and dotglob are related options)
shopt -s dotglob # handles dot files in globs
shopt -s nocaseglob # case-insensitivity
shopt -s failglob # if no matches, returns error and doesn't execute 
    # the associated command
GLOBIGNORE="foo bar" # ignores items from globs (default is . and ..)
setopt NULL_GLOB # (zsh equivalent of nullglob)
for x in ~(N)*; do; done # (ksh equivalent of nullglob)
Back Top

File Iteration

# ––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––————————
# FILE ITERATION
# ––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––————————

# Iterate over several files using the "find" command

# Example:
# Merges all text files from a directory root "$1" to one file "$2"
find -s "$1" -name "*" -type 'f' -exec cat {} + > "$2" ;

# Synopsis:
# find     command: locates files and evaluates expressions
# -s       option (find): alphabetical order in each directory
# $1       option (find): base dir path (first argument)
# -name *  option (find): "for files whose name equals" (* "anything")
# -exec    option (find): "execute the following command on the file"
# cat      command: get the contents of the file
# {}       token (find): insert current file path here
# +        token (find): pass as many paths as possible per -exec call
# > $2     output (shell): send the output to file with path $2 (2nd arg)
# ;        token (find): end of -exec data (may need escaping for shell)

# Basic "find" usage:
# From <unix.stackexchange.com/questions/321697>

# To run a single command for each file found, use:
find dirname ... -exec somecommand {} \;

# To run multiple commands in sequence for each file found, where
# the second command should only be run if the first command succeeds, use:
find dirname ... -exec somecommand {} \; -exec someothercommand {} \;

# To run a single command on multiple files at once:
find dirname ... -exec somecommand {} +

# If you need to use shell features in the command, such as redirecting
# the output or stripping an extension off the filename or something similar,
# you can make use of the sh -c construct. But:
# - Never embed {} directly in the sh code
# - Don't use {} multiple times, or as part of a longer argument

# To run a single shell command per file, use:
find dirname ... -exec sh -c 'somecommandwith "$1"' find-sh {} \;

# However it will usually give better performance to handle the files
# in a shell loop so that you don't spawn a shell for every single file found:
find dirname ... -exec sh -c 'for f do somecommandwith "$f"; done' find-sh {} +

# Note that 'for f do' is equivalent to 'for f in "$@"; do', and handles
# each of the positional parameters in turn. In other words, it uses each
# of the files found by "find", regardless of any special
# characters in their names.

# Notable options for "find" command

# (see "man find" for more)
# -----
# find [-H/L/P] [-EXdsc] [-f path] path ... [expression]
# find [-H/L/P] [-EXdsc] -f path [path ...] [expression]
# -----
# -s - alphabetical order in each directory
# -x - do not span devices
# -f DIRPATH - hierarchy to traverse
# -d - do not visit dirs before their contents
# -X - format for use with xargs
# -P - act on symlink files themselves (default)
# -L - act on files referenced by symlinks (vs. -P)
# -H - same as -L, but for files specified on command line
# -E interpret regex as extended vs. basic
# -----
# Notable "find" expression syntax:
# -name FOO - files whose name is FOO
# -exec UTILITY [ARGS] ; - execute the command on the file, like cat {} + ;
# -execdir - same as -exec, but executes UTILITY from dir that holds
#            current file, with {} not qualified
# -d -depth N - true if depth of file relative to starting point is N
# -empty - file/dir is empty
# -flags FOO - Flags, tested against using "-" and "+" prefixes.
# -mindepth/maxdepth N - descend at least/most to N dir levels
#                        below command line arguments
# -name PATTERN - match file basenames matching PATTERN. Accepts wildcards
# -ok[dir] [UTILITY ARGS ...] ; - require user confirmation for UTILITY
# -path/wholename PATTERN - true if pathname matches
# -print - print current file pathname to stdout
# -prune - don't descend into current file (no effect if -d specified)
# -regex - matches against whole file path using regex
# -type T - file is of specified type (b block, c char, d dir, 
#           f regular file, l symlink, p FIFO, s socket)
# -----
# "find" expression operators:
# ( exp ), ! exp, -not exp, -false, -true, exp -and exp,
# exp exp (same as -and), exp -or exp.

# Additional "find" notes
# - "If you invoke find from a shell you may need to quote the semicolon
#   if the shell would otherwise treat it as a control operator.
#   If the string ``{}'' appears anywhere in the utility name or the arguments
#   it is replaced by the pathname of the current file. Utility will be
#   executed from the directory from which find was executed.
#   Utility and arguments are not subject to the further expansion of
#   shell patterns and constructs."
# - Use {} + to replace "{}" with as many pathnames as possible for
#   each invocation of UTILITY.
Back Top