Navigating the filesystem
File manipulation
Wildcards, redirection and piping
Variables and loops
File properties and permissions
Scripts
And more...
Communication encrypted!
To access one of the login nodes (remotely) from a Unix machine:
ssh <username>@legion.rc.ucl.ac.uk
Log on using UCL username/password
On Windows (e.g. Desktop@UCL) you can use PuTTY.
[user@host ~]$
On Legion the command prompt has the form [<username>@<host name> <present directory>]$
<user name> is the Unix user name (your UCL user ID).
<host name> is the name of the computer that you are accessing.
<present directory> is the directory that the user is currently in.
Other systems you log into may have a different form of prompt.
Read evaluate print loop
User types a command and presses enter
The shell reads this command, and evaluates it
The shell then prints any output to the screen and returns the command prompt to the user
[user@host ~]$ git clone https://github.com/tcouch/shell-training.git
[user@host ~]$ ls
Scratch shell-training
[user@host ~]$ ls shell-training
animals data docs scripts
[user@host ~]$ ls -a
. .bash_logout Scratch
.. .bashrc shell-training
.bash_history .emacs .ssh
man ls
for a list of options you can use with ls (q to quit)[user@host ~]$ ls -a
. .cshrc shell-training
.. .config .ssh
.bash_history .emacs
[user@host ~]$ pwd
/home/username
[user@host ~]$
[user@host ~]$ pwd
/home/username
[user@host ~]$ cd shell-training
[user@host shell-training]$ pwd
/home/username/shell-training
[user@host shell-training]$ cd ..
[user@host ~]$ cd /
[user@host /]$
/home/alice
../home/alice
There are many ways of writing the path to a directory.
Every directory has a unique absolute path: ls /home/username/shell-training/
Relative paths depend on your current location: ls shell-training/
This relative path begins with the home alias and works from anywhere: ls ~/shell-training/
This path works too:
cd /bin/../tmp/../home/username/shell-training
cd ~
cd sh
and then press the tab key?The shell-training folder has the following structure:
~/shell-training
|--animals
|--data
|--docs
|--scripts
cd
on its own do? Try typing it from several different locations.cd -
do? Try typing it from several different locations.[user@host ~]$ mkdir a_directory
[user@host ~]$ ls
a_directory Scratch shell-training
[user@host ~]$ mkdir b_directory c_directory
[user@host ~]$ ls
a_directory b_directory c_directory Scratch shell-training
[user@host ~]$ mkdir shell-training/a_directory
[user@host ~]$ ls shell-training
a_directory animals data docs scripts
[user@host ~]$ nano a_file
There are many other (better but more complex) text file editors on the system such as vim, emacs and nedit.
Use the one you feel most comfortable with.
[user@host ~]$ nano a_script.sh
echo "Creating new directory..."
mkdir new_directory
ls
echo "Finished"
[user@host ~]$ source a_script.sh
Creating new directory...
a_directory a_script.sh c_directory Scratch
a_file b_directory new_directory shell-training
Finished
[user@host ~]$ cd shell-training
[user@host shell-training]$ source ../a_script.sh
[user@host shell-training]$ source ../a_script.sh
Creating new directory...
a_directory animals data docs new_directory scripts
Finished
Write a script that will do the following steps: a. Create a new directory called cake inside your home directory b. Use an absolute path to create a directory inside cake called "Cheesecake" c. Change into the Cheesecake directory d. Use a relative path to create another directory inside cake called "Battenberg" e. Return to the home directory and list the contents of cake
The final set of directories should look like this:
~/cake
|--Cheesecake
|--Battenberg
Read the man page for mkdir to find out what the -p option does. Use it to create the following set of directories with a single command:
~/bread
|--focaccia
|--naan
[user@host shell-training]$ cd docs
[user@host shell-training]$ ls
123.dat abc.txt cake.txt def.txt r2d2.bot xyz.txt
abcde.txt ab.txt cheesecake.txt food.txt some-maths.txt
[user@host docs]$ wc abc.txt def.txt xyz.txt
10 52 168 abc.txt
7 61 394 def.txt
11 99 589 xyz.txt
28 212 1151 total
wc
counts lines, words and bytes for each file[user@host docs]$ wc *.txt
13 119 683 abcde.txt
10 52 168 abc.txt
8 83 454 ab.txt
26 206 1332 cake.txt
11 105 656 cheesecake.txt
7 61 394 def.txt
13 85 561 food.txt
6 12 56 some-maths.txt
11 99 589 xyz.txt
105 822 4893 total
*
is a wildcard that matches zero or more characters[user@host docs]$ ls a*.txt
abcde.txt abc.txt ab.txt
*
is a wildcard that matches zero or more characters.[user@host docs]$ ls ???.txt
abc.txt def.txt xyz.txt
?
is also a wildcard. It matches a single character.[user@host docs]$ ls [fx]*
food.txt xyz.txt
[user@host animals]$ less birds.txt
[user@host animals]$ grep bat mammals.txt
whiskered bat, myotis mystacinus
natterer's bat, myotis nattereri
daubenton's bat, myotis daubentonii
leisler's bat, nyctalus leisleri
brown long-eared bat, plecotus auritus
Command | Action |
---|---|
head | visualise the first 10 lines of a file |
tail | visualise the last 10 lines of a file |
cat | print file contents to the terminal screen |
sdiff | visualise and compare two files side-by-side |
(Use "man <command>" to see more information)
Use grep to search through all of the files in the animal directory to find animals with the word red in their name. Read the man page to find out how to make grep only select the word red, and not just any word containing r-e-d.
[user@host ~]$ cp a_file copy_of_a_file
[user@host ~]$ ls
a_directory a_script.sh bread cakes.sh copy_of_a_file Scratch
a_file b_directory cake c_directory new_directory shell-training
[user@host ~]$ mv a_file control.in
[user@host ~]$ ls
a_directory b_directory cake c_directory copy_of_a_file Scratch
a_script.sh bread cakes.sh control.in new_directory shell-training
[user@host ~]$ mv control.in a_directory
[user@host ~]$ ls a_directory
control.in
[user@host ~]$ rm a_directory/control.in
[user@host ~]$ rm a_directory
rm: cannot remove ‘a_directory/’: Is a directory
[user@host ~]$ rm -r a_directory
[user@host ~]$
[user@host ~]$ alias zap='rm -ri'
[user@host ~]$ zap *_directory
The docs directory contains several files ending in ".txt". Write a backup script that will create a backups folder and copy each of these files there.
Rather than having the output of a command printed to the screen, we can send it to be written to a file instead.
[user@host ~]$ echo hello > hello.txt
[user@host ~]$ echo hello again >> hello.txt
[user@host ~]$ cat hello.txt
hello
hello again
>>
appends the output to the end of an existing file.>
will overwrite any existing content.[user@host ~]$ cd shell-training/animals
[user@host animals]$ head -n 5 mammals.txt
common pipistrelle, pipistrellus pipistrellus
whiskered bat, myotis mystacinus
natterer's bat, myotis nattereri
daubenton's bat, myotis daubentonii
leisler's bat, nyctalus leisleri
-n
with head and tail to print n lines from the start or end of a fileWhat about printing a number of lines from the middle of a file? There's no mid
command!
An inefficient solution:
>
to redirect what would normally be printed to the screen to a file instead[user@host shell-training]$ head -n 15 mammals.txt > temp.txt
[user@host shell-training]$ tail -n 5 temp.txt
european hedgehog, erinaceus europaeus
pygmy shrew, sorex minutus
wood mouse, apodemus sylvaticus
house mouse, mus domesticus
brown rat, rattus norvegicus
What if you wanted to do this for 1000 files? What if your work flow involves several intermediate steps? That's a lot of temporary files!
We can use a pipe to redirect the output from one command and make it the input for another command:
[user@host shell-training]$ head -n 15 mammals.txt | tail -n 5
european hedgehog, erinaceus europaeus
pygmy shrew, sorex minutus
wood mouse, apodemus sylvaticus
house mouse, mus domesticus
brown rat, rattus norvegicus
[user@host shell-training]$ head -n 15 mammals.txt | tail -n 5 | sed 's/mouse/elephant/g'
european hedgehog, erinaceus europaeus
pygmy shrew, sorex minutus
wood elephant, apodemus sylvaticus
house elephant, mus domesticus
brown rat, rattus norvegicus
You can chain any number of programs together to achieve your goal:
This allows you to build up fairly complex workflows within one command-line.
Use the wget command to download Alice's Adventures in Wonderland: https://www.gutenberg.org/files/11/11.txt
Use less to read the file, search for specific words, and step through the results
Use sed to replace every instance of Alice with your own name, and redirect the result to a new file.
Using a combination of head and tail, find lines 325-335
There are three streams of communication between a program and its environment:
[user@host docs]$ wc *.txt not_a_file > txt_list 2> txt_list_err
2>
redirects any error messages created by a commandls -l
ls -l > list_of_files
You can also redirect standard input to a command, using <
to send the contents of a file in place of command line input.
[user@host docs]$ bc < some-maths.txt
3.14285714285714285714
9.99
16.66666666666666666666
10.312567
[user@host ~]$ var1='hello'
[user@host ~]$ echo $var1 world!
hello world!
[user@host ~]$ myname="John Smith"
[user@host ~]$ echo "Hello my name is $myname. Nice to meet you."
Hello my name is John Smith. Nice to meet you.
[user@host ~]$ echo 'Hello my name is $myname. Nice to meet you.'
Hello my name is $myname. Nice to meet you.
[user@host ~]$ fruit=orange
[user@host ~]$ echo "I love eating $fruits."
I love eating .
[user@host ~]$ echo "I love eating ${fruit}s."
I love eating oranges.
[user@host ~]$ two=2
[user@host ~]$ result=$(( $two + 2 ))
[user@host ~]$ echo $result
4
[user@host ~]$
Run commands inside $( ) and assign the wrapped command to a variable
[user@host ~]$ ls
a_directory a_file
[user@host ~]$ dir_contents=$( ls )
[user@host ~]$ echo $dir_contents
a_directory a_file
What is the output of this command?
[user@host ~]$ echo $PATH
Note the structure: <path1>:<path2>:<path3>
PATH is an environment variable which Bash uses to search for commands typed on the command line without a full path.
Use the command env to discover more environment variables.
[user@host ~]$ for i in first second third
> do
> echo $i iteration
> done
first iteration
second iteration
third iteration
[user@host ~]$ for (( i=1 ; i<=5 ; i++ ))
> do
> echo iteration$i
> done
iteration1
iteration2
iteration3
iteration4
iteration5
(Note how it is possible to create number labels)
[user@host ~]$ ls -l
total 368
-rw-r--r-- 1 course1 ucaac2 167546 Jan 25 2014 11.txt
-rw-r--r-- 1 course1 ucaac2 72 Jun 6 12:34 a_script.sh
drwxr-xr-x 4 course1 ucaac2 4096 Jun 6 12:43 bread
drwxr-xr-x 4 course1 ucaac2 4096 Jun 6 12:41 cake
...
drwxr-xr-x 4 course1 ucaac2 4096 Jun 6 12:43 bread
drwxr-xr-x - File type and permissions
4 - Number of links to the file
course1 - User name of file owner
ucaac2 - Group to which the file belongs
4096 - size of file in bytes
Jun 6 - last change date
12:43 - last change time
bread - file name
There are three user security classifications that apply to the ownership of a file:
user: the individual user that has ownership of the file
group: a group of users to which that user belongs
others: all other users (not owner or in that group)
Each of these has three file access classifications:
read: permission to read the file
write: permission to write the file
execute: permission to execute (run) the file
drwxr-xr-x has four fields:
d | The file is a directory |
rwx | The user has read, write and execute access |
r-x | The group has read and execute access |
r-x | Others have read and execute access |
Permissions | read | write | execute |
---|---|---|---|
rwx | yes | yes | yes |
rw- | yes | yes | no |
r-- | yes | no | no |
r-x | yes | no | yes |
--- | no | no | no |
[user@host ~]$ ls -l
drwxr-xr-x 4 course1 ucaac2 4096 Jun 6 12:43 bread
[user@host ~]$ chmod go-rx bread
[user@host ~]$ ls -l
drwx------ 2 user rcops 4096 2009-12-08 07:31 a_directory
How did the permissions definition go-rx work?
chmod [ugoa][+/-][rwx] file
ugoa - user, group, other, all
+/- - add/remove
rwx - read, write, execute
If you don't specify u,g,o or a, default is ALL (so chmod +x makes file executable for everyone).
[user@host ~]$ groups
ucaac2 legon020
The data folder contains 200 files. Each file is named according to a type of measurement (A or B), and a location (1-100) e.g. A_21, B_56 etc. The scripts folder contains a python script which takes the names of an A and a B file as arguments e.g. scripts/calculate_score.py data/A_1 data/B_1 This will calculate a score based on the data in the two files and print it to standard output along with the name of the files used.
Make calculate_score.py executable
Use a for loop to run through the data files corresponding to each location and generate a score
Modify the for loop to save the scores to a file
Use the sort command to find the location with the highest score
So far we've been using source to run our scripts, but it's also possible to make them executable - just like commands.
#!/bin/bash
# This is a very simple hello world script.
echo "Hello, world!"
[user@host ~]$ chmod u+x hello_world.sh
[user@host ~]$ ls -l hello-world.sh
-rwxr--r-- 1 user ccaas0 30 Mar 31 17:10 hello_world.sh
[user@host ~]$ ./hello_world.sh
hello world!
If you want to be able to make your script work like a command, you need the directory it is in to be in your PATH
[user@host ~]$ mkdir ~/scripts
[user@host ~]$ PATH=$PATH:$HOME/scripts
[user@host ~]$ export PATH
You can control your script's behaviour with arguments you pass to it when you run it.
[user@host ~]$ ./script.sh var1 var2
Within the script: $1 contains "var1" $2 contains "var2"
The script looks like this:
#!/bin/bash
echo The first argument is $1
echo The second argument is $2
echo And together they make ${1}${2}
And here it is in use:
[user@host ~]$ ./var-script green house
The first argument is green
The second argument is house
And together they make greenhouse
You can now control the number of times a for loop iterates by including a number as an argument when you call it. Write a script which will create as many numbered directories as you want when you run it.
Create a bash script called hi which will use the USER environment variable to say hello to you
Make this script executable
Create a /scripts directory in your home directory
Add this directory to the PATH environment variable
Move the new script there and try running it as a command
Remember those hidden files in your home directory? Some of them are actually pretty useful.
Both of these are already set up for you on Legion, but you can improve them as you like.
Close PuTTY and log back in.
Now edit .bashrc to report $SHLVL when you start a new instance of bash
[user@host ~]$ tar -czvf work.tgz work
work/
work/program/
work/calculations/
work/calculations/control.in
work/workfile
Use the ssh command to login to other systems you have access to
[user@host ~]$ ssh <username>@aristotle.rc.ucl.ac.uk
To Aristotle:
[user@host ~]$ scp work.tgz <username>@aristotle.rc.ucl.ac.uk:
...
Password:
work.tgz 100% 213 0.2KB/s 00:00
[user@host ~]$ tar -xzvf work.tgz
work/
work/program/
work/calculations/
work/calculations/control.in
work/workfile
From Aristotle:
[user@host ~]$ scp <username>@aristotle.rc.ucl.ac.uk:~/work.tgz .
...
Password:
work.tgz 100% 340 0.3KB/s 00:00
Use tar to create an archive of the shell-training directory
Use scp to copy the archive to Aristotle: aristotle.rc.ucl.ac.uk
Extract the archive file on Aristotle
Use scp to copy a file from Aristotle to your home directory
A process is in the:
foreground when it is interacting with the user via an interface (usually the shell).
background if it is running without interacting with the user.
suspended if it is neither interacting nor running.
To run a process in the background: add the symbol "&" at the end of the command line.
To send a foreground process to the backgound: press Ctrl+z and then execute the command "bg"
To bring a background process to the foreground: execute the command "fg"
Use the jobs command
[user@host ~]$ sleep 60 &
[1] 24991
[user@host ~]$ jobs
[1]+ Running sleep 60 &
Or ps for more information
[user@host ~]$ ps
PID TTY TIME CMD
24467 pts/6 00:00:00 bash
24991 pts/6 00:00:00 sleep
24996 pts/6 00:00:00 ps
Use the kill command with the PID:
[user@host ~]$ kill 24991
[1]+ Terminated sleep 60
Or use the job number:
[user@host ~]$ kill %1
[1]+ Terminated sleep 60
Q: What happens if you want to:
Log out and turn off your PC?
Go home and continue working?
Freely create and delete shell sessions within one terminal?
A: Use a terminal multiplexer like GNU screen
ssh user@aristotle.rc.ucl.ac.uk
(or use Putty)
screen
The screen will clear and you will be presented with a new prompt.
Do something in that shell (e.g.) "ls".
Press Ctrl-a, then press c
You'll be given another shell (what screen calls a "window")!
Create new "windows" with Ctrl-a, c
Ctrl-a, a switches between this window and your last one.
Windows are numbered 0->N, Ctrl-a, number (e.g. Ctrl-a, 3) to switch to a particular one.
NOTE: make sure you log into the same login node that you started screen on
screen -d - forces detach of a screen that's running (so you can screen -r it). Handy if your SSH connection drops.
screen -ax - forces attach to a screen that's attached to from somewhere else.
You can configure screen in lots of useful ways, including changing the keyboard shortcuts.
For more see man pages, or online documentation: (http://www.gnu.org/software/screen/)
[user@host ~]$ seq 1 5
1
2
3
4
5
[user@host ~]$ seq 1 2 9
1
3
5
7
9
[user@host ~]$ seq -f %03g 1 2 9
001
003
005
007
009
Created with "ln"
Inode table keeps track of hard links
Deleting a file = "unlinking" it
Can only be used inside a single file system
[user@host ~]$ ln -s ~/some_project/2012/part531 ~/current_project
[user@host ~]$ ls -l ~
lrwxr-xr-x 1 user staff 11 10 Oct 17:56
current_project -> /home/user/some_project_2012/part531
Can use relative or absolute paths!
Create using absolute paths to make sure they go where you want
cat <<EOF > child_script.sh
#!/bin/bash
echo The child script says Hi!
EOF
write a parent_script.sh that creates and executes the child_script.sh
write a parent_script.sh that creates and executes 10 different child_script.sh that print out their individual number