Transferring data to and from Legion

Data can be transferred to and from Legion from either the login nodes, the data transfer node or by jobs running on compute nodes.

Transferring modest amounts of data

You can use Legion’s login nodes to transfer modest amounts of data.

Please bear in mind that there is no limit on the number of users that can be logged in to the Login nodes. Their activity may cause your data transfer rates to drop (as yours may affect theirs). If you need to transfer large amounts of data or need more reliable transfer rates, we recommend that you do so within a batch job, as described in the following section.

If you are running scp from your local machine, use legion.rc.ucl.ac.uk as the hostname and you will transfer via the login nodes.

### Dedicated transfer node

Legion provides a dedicated transfer node with ten gigabit network connections to the UCL network and to Legion. To access this node, log in via scp, sftp or ssh to:

login05.external.legion.ucl.ac.uk

Please note that you cannot submit or view the status of your compute jobs on this node - it is only available for data transfer.

Importing compressed data using login05

A good way to transfer data into Legion is through the use of a compressed (zip) file.

We will use scp to securely copy the file work.tgz across the network to Legion:

[ccaaxxx@login06 Scratch]$ scp ccaaxxx@socrates.ucl.ac.uk:~/work.tgz .
...
Password:
work.tgz         100%  340     0.3KB/s   00:00

Note: remote machine (in this case Socrates) must be running SSH server.

If you are using Desktop@UCL you will need to map the T: drive to access the Unix filestore, files on the T: drive are visible in socrates.

Click on the Windows Start button and find go to Utilities > Mount Unix FileStore (T Drive).

After this you should be presented with a window stating the T drive has been successfully mapped.

So, from your desktop:

[you@desktop ~]$ scp work.tgz ccaaxxx@login05.external.legion.ucl.ac.uk:Scratch

Exporting compressed data from Legion

From Legion to Socrates:

[ccaaxxx@login06 Scratch]$ scp work.tgz ccaaxxx@socrates.ucl.ac.uk:~/
...
Password:
work.tgz         100%  340     0.3KB/s   00:00

From Legion to your desktop:

[you@desktop ~]$ scp ccaaxxx@login05.external.legion.ucl.ac.uk:Scratch/work.tgz

Archiving and Compression

We can use the tar function to compress and extract compressed files

[ccaaxxx@login06 Scratch]$ tar -zcvf work.tgz work
work/
work/program/
work/calculations/
work/calculations/control.in
work/workfile

[ccaaxxx@login06 Scratch]$ ls
work.tgz work

tar -zcvf - archives and compresses directory trees and files: c - create archive, z - compress, v - verbose, f - in the following file

Extracting files from compressed archives

[ccaaxxx@login06 Scratch]$ ls
work.tgz
[ccaaxxx@login06 Scratch]$ tar -zxvf work.tgz
work/
work/program/
work/calculations/
work/calculations/control.in
work/workfile
[ccaaxxx@login06 Scratch]$ ls
work.tgz work

tar -zxvf - extracts and uncompresses directory trees and files: x - extract archive, z - uncompress, v - verbose, f - from the following file