Linux Data

All Linux servers and desktops in GEO domain have transparent access to NFS users profiles and research storage volumes with help of NIS, YP and Automount.

You can get list of mount points available at your computer:

ypcat -k auto.master
  /data yp:auto.data   
  /software yp:auto.software
  /Volumes yp:auto.volumes
  /Users/home yp:auto.home

It means that information about /data mount point is available via auto.data binding, /software - via auto.software and so forth.

ypcat -k auto.data
  gis -fstype=nfs resstoconfs0001.res.vuw.ac.nz:/GeoColridge_01/gis
  sabine -fstype=nfs      resstoconfs0001.res.vuw.ac.nz:/GeoSabine_01
  hawea -fstype=nfs       resstoconfs0001.res.vuw.ac.nz:/GeoHawea_01

First column is share alias, second - mount options, third - path to network share

To make the share appear you just need to get there using any command e.g. ls, cd, df, :

ls /data/gis

To save network resources all volumes will automatically unmount after 5 minutes timeout.

Transfer data#

You might need to work with large data sets on research storage. Uploading or downloading them can take a long time. It’s important to choose the proper tools. Please avoid of using any GUI file managers: their indexing and progress services may result to massive performance degradation on data transfer.

For data location. Home profiles are mapped to /Users/home/, research storage - to /Volumes/ and /data/. Please use ypcat on the server to find out where the particular volume is located.

SSH#

To transfer some small amount of data (few files) you might simply run

scp my_local_file USERNAME@sgees001.geo.vuw.ac.nz:/path/to/destination/

Otherwise please use a more reliable instruments, like rsync. In very short, syntax is rsync [FLAGS] <FROM> <TO>. For more details please refer to documentation.

A typical rsync command might look as:

rsync -av --progress --log-file=rsync.log \
/media/portable_hdd/field_trip_data/ \
USERNAME@sgees001.geo.vuw.ac.nz:/Volumes/GeoPhysics_42/users-data/USERNAME/field_trip_data/

NFS#

Once mapped, NFS volume appears as a regular directory, then you can use any tools for copying or moving your data across.

Globus#

https://www.globus.org/

Gigabytes, terabytes, petabytes—research data is large and distributed. Globus lets you efficiently, securely, and reliably transfer data directly between systems separated by an office wall or an ocean. Focus on your research and offload your data transfer headaches to Globus.

Get unified access to your research data, across all systems, using any existing identity. Laptop? HPC cluster? Cloud storage? Tape archive? Access them all using just a web browser. Data stored at a different institution? At a supercomputing facility? All you need is your campus login.

Globus client. You need to install and configure a copy of client software on each end.#

Linux GUI (and CLI, see the docs): https://downloads.globus.org/globus-connect-personal/linux/stable/globusconnectpersonal-latest.tgz

Windows GUI: https://downloads.globus.org/globus-connect-personal/windows/stable/globusconnectpersonal-latest.exe

MacOS GUI: https://downloads.globus.org/globus-connect-personal/mac/stable/globusconnectpersonal-latest.dmg

CLI (Python, any O/S): https://github.com/globus/globus-cli

Linux version doesn’t require root privileges:

wget https://downloads.globus.org/globus-connect-personal/linux/stable/globusconnectpersonal-latest.tgz
tar xf globusconnectpersonal-latest.tgz
cd globusconnectpersonal-3.2.2
./globusconnect

Web interface (dashboard):#

https://app.globus.org/ => Use your existing organizational login => pick up ‘Victoria University of Wellington’ => Continue =>

Here you can start/finish transfer and monitoring the process, configure connections e.t.c…

Documentation: https://docs.globus.org/ #

It’s a bit disorganized. Good start point is: https://docs.globus.org/how-to/get-started/

…then read the following:

https://www.globus.org/globus-connect-personal

https://docs.globus.org/how-to/globus-connect-personal-linux

RSYNC daemon#

We’ve got rsyncd running on several servers - this is really helpful when you need to transfer a bunch of files. Please contact computing system administrator for more details.

VPN#

If you’re outside the campus, you still may connect to any of the servers with VPN

Remote Session with X2Go#

X2Go is a Remote Desktop solution, which some vendors vaguely call Remote Control. This is not to be confused with Microsoft Remote Desktop Connection, which is a competing Remote Desktop solution and protocol.

With X2Go, you mainly access Linux computers. The Linux computer you are accessing is the X2Go Server. (There are ways to connect to a Windows computer as well; some of them are only available with the Linux X2Go Client.)

You can connect from a computer running Linux, Windows or Mac OS X. This computer is the X2Go Client.

SGEES Linux machines already have X2Go client installed. For Windows or MacOS download and install client here

All details about x2Go connection setup are here

Auto mount NFS share#