Loughborough University
Leicestershire, UK
LE11 3TU
+44 (0)1509 222222
Loughborough University

IT Services : High Performance Computing

File Transfer


Introduction

Please be aware that files on hydra are not backed up.

Using secure copy (SCP and SFTP)

Files can be transferred between the cluster systems and your desktop computer with secure copy (SCP) or using SSH file transfer protocol (SFTP).

Transfers both to and from hydra are possible.

To use scp to transfer a file to hydra (where file and username are to be replaced by the file you want to transfer, with your username): scp file username@hydra.lboro.ac.uk:

     
username@hydra's password:  
file    100%     678    0.7KB    00:00     
      

Note that if you are logged into a system with the same username as the only on hydra you may omit username@ above. This is the case on most university systems. In the above case the file is transferred to the home directory. It is possible to copy directories with the -r flag: scp -r directory username@hydra: where directory is replaced with the name of the directory to transfer.

To use scp to transfer a file from hydra: scp username@hydra.lboro.ac.uk:file .

username@hydra's password:   
file    100%     678    0.7KB    00:00
      

In the above example the file file is in the home directory on hydra and . means transfer to the current directory on the local machine.

Sftp can be used using: sftp username@hydra.lboro.ac.uk

Connecting to hydra.lboro.ac.uk... username@hydra.lboro.ac.uk's
password:
sftp > put file
sftp > exit     
      

Linux and Macintosh users have these commands installed by default. Windows users may wish to install the putty client which also installs windows versions of these commands as pscp and psftp.

PuTTY Download .

Using rsync over secure shell

An alternative is to use rsync which keeps a copy on hydra in sync with a copy on a local disk. rsync -vuarP directory username@hydra.lboro.ac.uk: synchronises a directory called directory on your local machine with one called directory on hydra. This is a good pattern for transferring a dataset of multiple files, which may have changed, to hydra. The special feature of rsync is that it only transfers files that have changed, and so if only a 1MB file in a 1GB data set has changed, only 1MB is transferred. If partial file rsync is used, even the whole of a 1MB file need not be transferred, just part of it.

The synchronisation may also be used in the other direction: rsync -vuarP username@hydra.lboro.ac.uk:directory directory which synchronises the remote directory with a local one called directory . Because hydra is not backed up this is useful for enabling you to bring back your important results from hydra to the main Loughborough file store which is backed up. You may wish to use a cron job to do this periodically, provided you have set up suitable authentication to hydra using ssh keys which allow you to login via ssh without entering a password.

Alternatively you may wish to look at the section on Organising Work which looks at how to handle reproducible results, but also discusses the use of revision control systems to handle synchronisation, although it takes additional thought to do so.

Linux and Macintosh users can easily install versions from their favourite package managers if they don't already have it installed. However, there is not currently a good, simple, free command-line version for Windows that supports secure transfer and so users must check to see if such tools that are available meet their requirements for rsync over an already mounted filestore for Windows. A version of similar synchronisation tool called unison is available for windows but it is not a simple single-file installation at present.

Graphical tools

A number of graphical tools that do one or more of the above exist.

Linux and Macintosh users may wish to look at grsync. Whilst FileZilla used to be recommended it is no longer as the installer now comes packaged with additional software that is not advisable to install.

Windows users may wish to look at WinSCP which can handle transfers and synchronisation of directories in a way similar to rsync.

A version of grsync exists for windows. grsync download .

Mounting the filestore locally

An alternative to the above is mounting the storage on hydra to your local machine, information on which can be found in the help pages on storage.

Once the filestore is mounted you may simply copy files to and from the filestore on hydra using standard commands or graphical tools. You may also use tools such as rsync to synchronise directories without the need to use secure shell options to connect remotely but be aware that the work of checking what has changed will now be placed on your local machine rather than hydra.

Future Additions

Further guidance will be added here on how users may automate rsync to backup work from hydra to other storage. Also guidance will be added on how to combine this with revision control systems to allow tracking of data as it changes so that the results from a particular experimental run can be more easily tracked.