Intro

Why know about the shell?

The shell is a useful and flexible environment for controlling the computer and the history of the shell is intricately entwined with the history of unix. Depending on data source, at least two-thirds of the computers powering the internet are based on unix and many estimates are much higher than this.¹ If you want to know and to be able to extract maximum utility from unix-derived operating systems, such as but not limited to Linux, MacOS and FreeBSD, it is worth knowing about the shell, as it is the natural interface to such systems.

The key difference between the shell and the GUI (‘graphical user interface’) that most desktop computer users are familiar with is that the shell is programmable, which is very useful for the automation of routine tasks. Shell scripts—text files which tell the shell what programs to execute and how—can be thought of as computer programs in their own right. Because of this it is also useful as a way to understand certain concepts in programming and can be thought of as a gateway to more powerful programming languages.²

How do I launch the shell?

How to access the shell will depend on your operating system:³

• MasOS: hit ⌘-space, type in terminal, hit enter
• Linux: will be specific to distribution, but e.g. on Ubuntu, hit Alt-F2, enter gnome-terminal hit enter
• Chromebook: enable ‘developer mode’ (instructions for this elsewhere on the internet), hit Ctrl+Alt+T, type shell, hit enter

In the shell

You’re now in the shell. Shell commands generally take the following form:

command [arguments]

Here, arguments are shown in square brackets to denote the fact that they are optional. Once you hit enter, the shell will interpret what you have entered and try to act accordingly.

It is traditional to have the shell print “Hello world!” as a first foray, so let’s do that.

echo 'Hello world!'

Great, we are now using the shell.

What can we do?

Firstly it is worth saying that there are many different of shells. One of the most popular, because it is often the default on linux systems, is bash.⁴

The name of the shell you are using is in the environment variable SHELL. An environment variable is a variable that has a value assigned to it. Some environment variables will be set automatically for us, and we can also create out own. To access the value of the variable, we simply prepend a $ to it. To get the value of SHELL then, we use $SHELL

echo $SHELL

/usr/local/bin/bash

will tell us what shell we are using. For the rest of this guide I will assume you are using bash, as this is broadly speaking the default in linux (including chromebook if you followed the instructions above) and MacOS. If the above command seems to suggest you are running a different shell, you may be able to use bash by simply entering bash (and hitting enter).

Quick warning

The shell assumes you know what you are doing, so if you type in a command to delete the files which are essential for the running of your operating system, it will do it, rather than say “are you sure you want to do that?” . When deleting files on the shell, for instance, there is no undo and this is not reversible. For this reason, I would strongly recommend that you do not enter a command under any circumstances unless you fully understand what it will do.

This applies especially to the following commends which can be dangerous if not used appropriately:

• mv, which is used to move files
• rm, which is used to delete files
• sudo or su, which are used to run commands as the root user i.e. have system level privileges

Built-ins

Certain commands are built into the shell. Other commands are discrete programmes which sit on the disk.⁵

Moving around

First let’s learn how to move around the filesystem.

Note that there is usually some text on the left, which is referred to as the ‘prompt’, this is configurable but usually it will show you what directory you are in and end in a $ sign. In these examples [user@localhost] $ is used to represent the prompt, but don’t worry if your prompt is different.⁶

To follow along, which is recommended: for each of the boxes shown, where you see $, which represents the prompt, you need to type in what is shown on the rest of that line and hit enter. Where there is text in purple box with no dollar sign at start of line that is example output—your output may differ depending on your system.

cd allows us to change directory, and cd on its own will take us to the default directory for the user we are logged in as, referred to as the home directory on unix systems. But before we go anywhere else, let’s find out where we are, with pwd, which stands for ‘print wording directory’. As you may have noticed by now, most frequently used commands are quite brief, to reduce the amount of time you need to spend typing.

cd
pwd

/Users/mike

Note your system will likely say something different, because the default directory for the user, referred to as a home directory, depends on the system and the username of the user.

Let’s make a folder with mkdir and move into it with mv:

mkdir test_dir
cd test_dir/
pwd

/Users/mike/test_dir

You have now made a new folder and changed directory into it.

Creating files and folders

We know how to move around. How do we create files? For now we can create empty files with touch. We can verify we have created this file using ls (for list). ls will tell us what files and folders exist in the current directory. Once we have done that we can clean up by deleting our empty file and our empty test directory with rm and rmdir respectively.

touch empty_file
ls

empty_file

cd ..
rm test_dir/empty_file
rmdir test_dir/
pwd

/Users/mike

Note the use of cd .. here: the .. just means ‘one directory up from the current’ which is why it then changed the working directory from /Users/mike/test_dir to /Users/mike. .. is a helpful shortcut which saves us typing out the full directory path.

Flags

There exists a special form of argument called the flag. Flags are denoted with a single (or sometimes double) dash, followed by a character or string. So they look like -x, where x is just a placeholder for a character or string.

Note that the number of arguments is not fixed, generally we will add flags until the command provides the behaviour we want. Sometimes multiples flags will be used.

Sometimes flags will take an argument after the flag, like this:

command -a x -b y -c z

In this example, a b and c are all flags and x y and z are arguments to those flags respectively.

This begs the question: ‘which flags do I use?‘. Often command -h or command --help will provide detail about what flags and arguments should be provided to the command to achieve the desired behaviour.

If it doesn’t, man command will provide information on the command, by taking you to the ‘man page’. Note that man is short for manual and nothing to do with gender! In the man page, space will scroll down and q will exit and return you to the shell.

Let’s use ls as an example. ls is short for ‘list’ and will print the files and folders in the current directory. Using ls with no flags will just list the directory contents. Note that yours will look different to mine.

pwd

/Users/mike

ls

Applications	Downloads	Mail		Pictures
Desktop		Dropbox		Movies		Public
Documents	Library		Music

So far so good, but we want to change the behaviour of ls, perhaps to add more detail in the description of the files and folders, beyond the file or folder name.

ls is a truly old command from the early days of unix and as such is not that helpful in explaining flags.

ls -h will yield the same output as plain ls and ls --help will tell us that there are very many flags to use but not what they do.

ls --help

ls: illegal option -- -
usage: ls [-ABCFGHLOPRSTUWabcdefghiklmnopqrstuwx1] [file ...]

Thankfully the manpage will come to the rescue: man ls will give us the details of the flags. Use q to quit the manpage and then we are back to the shell.

(Note: if you are using a chromebook, the manpages are not installed. If you are using linux or MacOS, they will be. However, the good news for chromebook users is that ls --help is much more helpful on that system)

By way of example, with the -l flag, ls will provide more detail in a list, and with the -p flag, folders will have a trailing slash added to the folder name, making it easier to identify which are files and which are folders.

ls -l -p

total 6416
drwx------@  3 mike  staff      102 11 Jul  2017 Applications/
drwx------+ 19 mike  staff      646  3 Feb 11:28 Desktop/
drwx------+  5 mike  staff      170 20 May  2018 Documents/
drwx------+ 61 mike  staff     2074  2 Feb 17:29 Downloads/
drwx------@ 20 mike  staff      680  3 Feb 11:34 Dropbox/
drwx------@ 69 mike  staff     2346  2 Jan 20:34 Library/
drwx------   2 mike  staff       68 16 Apr  2018 Mail/
drwx------+  3 mike  staff      102  3 Apr  2017 Movies/
drwx------+  5 mike  staff      170 30 Sep  2017 Music/
drwx------+  4 mike  staff      136  2 Aug  2017 Pictures/
drwxr-xr-x+  5 mike  staff      170  3 Apr  2017 Public/

Files

Before flags we covered files and folders and how to make folders and create empty files. If you are felt confused that is fine … after all, what use is an empty file?

We will get to non-empty files, but first, let’s consider what files are made of. Certain file types, for instance .txt or .html files, when opened or peeked into, will show text.

By way of example, let’s take a peak at the top of the html of the BBC news homepage. I will explain curl and head later, they key is the output for now:

curl -s 'https://www.bbc.co.uk/news' | head -n 10

<!DOCTYPE html>
<html lang="en-GB" class="b-pw-1280 no-touch" id="responsive-news">
  <head>
    <meta charset="utf-8" />
    <meta
      name="viewport"
      content="width=device-width, initial-scale=1, user-scalable=1"
    />
    <meta http-equiv="X-UA-Compatible" content="IE=edge,chrome=1" />
    <meta
      name="google-site-verification"
      content="Tk6bx1127nACXoqt94L4-D-Of1fdr5gxrZ7u2Vtj9YI"
    />
    <link href="//static.bbc.co.uk" rel="preconnect" crossorigin />
    <link href="//m.files.bbci.co.uk" rel="preconnect" crossorigin />
    <link href="//nav.files.bbci.co.uk" rel="preconnect" crossorigin />
  </head>
</html>

Fine, this is HTML, but it is also viewable by us as text. To the computer, all files are just data. To us, when we inspect some files, they look like text, and some look like gibberish. The files that look like text, however, are still data to the computer, they just use a translation scheme—a text encoding—to transform the data into text that it displays and we can then read.⁷

So, data in files can be essentially be arbitrarily structured and encoded, but for now let’s just cover files that contain text data.

When working with files in the shell, one often works with files of this type. This is because many commonly used shell programs work with text data, and they are quick and convenient to use.

Input, output and redirection

The shell outputs text, you input text into the shell, and is primarily used with files that are made up of text data. As text is common to each of these, the shell makes it easy to move text around, so it can be an input or output either on the screen or in a file.

This is called ‘piping’ and ‘redirection’.

The | character is used to take the output of one program and feed it into another program as input. In this way, text data can be filtered through arbitrarily long chains of programs, which makes this feature very powerful.

Pipes were used in the command above with curl, which downloads data from a URL, and head which prints just the top of the data only. To more fully explain the command above:

1. We use the command curl with the flag -s. The -s flag just tells curl to only output the data that is downloads from the URL, which is the argument.
2. curl downloads this data and prints it to stdout. stdout stands for standard output, and if we didn’t include the pipe |, this data would be printed to the terminal
3. The pipe | takes stdout and feeds it into the input—stdin, logically enough—for the next command, in this case head
4. We use the command head with the flag -n 10, which makes head print only the first 10 lines of the input it receives.

Here is another example:

echo '!dlrow olleH' | rev

Hello world!

echo prints the characters in the argument; rev then prints them in reverse order.

“So far so interesting, but we still don’t have any files with contents?!” you may be thinking. Let’s get to that now.

The other way to manipulate input and output is with ‘redirection’, which simply means using a file instead of receiving input or printing output on the screen. To do this we use > to tell the shell to put the output into a new file, or >> to append the output to an existing file.

cd
pwd

/Users/mike

cd tmp/
pwd

/Users/mike/tmp

echo 'egassem desrever a si sihT' > message.txt
echo '!siht si os dna ...' >> message.txt
cat message.txt | rev

This is a reversed message
... and so is this!

Here we have used another command, cat, which prints the contents of the file supplied as an argument. A common way to create or add to files is with redirection.

Similarly, files can be used as input, using <

The below is another way to have cat print the contents of the file

cat < message.txt

egassem desrever a si sihT
!siht si os dna ...

And remembering to pipe into rev:

cat < message.txt | rev

This is a reversed message
... and so is this!

One command which is often ‘piped’ into is less. less is a pager, which allows you to view more than one screen of output by scrolling with the arrow keys (or the space bar). In less you can hit q to quit.

Appendix: standard *nix programs

catdoc reads MS Word files and spits out as plain text
alias alias commands, shell built-in, usually used in .bashrc or similar
apropos / whatis search doc databases for a keyword, useful for finding commands to do things
bg shell built-in to run stopped jobs (stop jobs with ctrl-z) in the background
cal calendar (see also ncal)
cat concatenate and print files
cc / c99 / c11 links to C compiler
cd change directory
chmod change file modes / permissions
chown change file owner and/or group
clear clear terminal
cp copy files
curl used to grab files from internet, prints to stdout by default
df show free disk space
diff describes differences between two files
dmesg print kernel messages
du show disk usage in a directory
echo print to terminal, useful in printing environment variables e.g. echo $HOME
exit exit terminal
fc fix command; allows you to fix last command in $EDITOR and then runs it; somewhat dangerous
fg shell built-in to run stopped jobs (stop jobs with ctrl-z) in the foreground
file tells you the format of a file
find file find Swiss army knife
fold wrap lines (see also fmt)
fortune fortune cookie
grep outputs lines that match one or more patterns
head view top of file
history built-in, shows command history
hostname print or set hostname
ifconfig shows IP address and other info
indent formats C source code
jobs built-in to show jobs
join join lines of two files on a common field
killall kill processes with a given name
leave reminds you to leave the terminal after a certain amount of time
less pager; pipe through this to view output a page at a time; space scrolls through; q to quit
ln create symbolic link
look look up a partially spelt word
locate locate a file—will require running
updatedb or similar to update the file database
ls list directory contents
man show manual page for a command; useful if you are not sure how to use it as will usually include examples as well as info on flags. Use -t flag for postscript output
mkdir make directory
mount mount filesystem (generally this is taken care of automatically on ubuntu and the like)
mv move or rename file
nc netcat; like cat only for TCP/IP
netstat display network connections
nice control process priority
ping see if host is up (may not work if host is firewalled)
popd / pushd easily return to current directory when done
ps list processes
rev reverses the input to stdin and prints to stdout
rm delete files, obviously be careful with this
rmdir delete empty folder
scp copy with ssh
sed streams editor, allows modification of text e.g. substitution on the fly
sort sorts lines
ssh secure shell (see also ssh-copy-id which can copy keys across servers securely)
su change to different user mid session
sudo execute as root mid session
tail view bottom of file
tar file archiver, often used to certain flags to decompress compressed tar archives e.g. .tar.gz
tee view output and also print stdout to file
type prints alias or source of (shell) function
tmux screen multiplexer vim text editor w3m shell-based browser
wc counts words/characters in a file
which / whereis locate a file in $PATH