Transferring files to QCDOC servers

Homepage

• Introduction
• ssh Tunneling
  • Creating an ssh Tunnel Manually
  • Using the tunnel Script
  • Example: transferring files from FNAL to BNL or JLAB
• Transferring files to FNAL using kerberized clients
Future Plans
• Available File Systems

Introduction

To access BNL QCDOC computing resources (including copying files to/from QCDOC servers) you must use the QCDOC ssh gateways. One way (2-hop) to transfer files is to first copy them into the gateways and then transfer them to the appropriate QCDOC file system.

If the remote site does not use a gateway, files may be transferred to BNL in one hop if you execute the scp command on the BNL host. For example, you can transfer files from Columbia (qcdochost4.phys.Columbia.edu) to the US QCDOC front-end node if you run the scp command on the qcdochostb host:

qcdochostb:>scp stratos@qcdochost4.phys.columbia.edu:/home/stratos/ker* .

You can avoid typing passwords/passphrases if you have your ssh keys set up. Instructions on how to do this can be found at http://www.bnl.gov/cybersecurity/ssh_unix_protocol2.asp

ssh Tunneling

Another way to transfer files from remote hosts is via ssh tunneling (port forwarding). This way you can transfer files directly to the QCDOC file systems (in one hop) without first copying them onto the ssh gateways. An ssh tunnel can be setup manually or you can use an ssh tunneling script as described below.

NOTE : If you are using a windows machine, you can still create an ssh tunnel to the QCDOC front-end nodes. Detailed instructions can be found here.

Creating an ssh tunnel manually

There are many instructions on the web on how to set up ssh port forwarding. Here is an example: To set up an ssh tunnel from a remote (outside BNL) host, such as qcdochost4.phys.columbia.edu, to the US DOE front-end node qcdochostb.qcdoc.bnl.gov do:

qcdochost4:>ssh -N -f -L 2123:qcdochostb.qcdoc.bnl.gov:22 user@ssh.qcdoc.bnl.gov

[BNL security Banner removed to conserve space]

user@ssh.qcdoc.bnl.gov's password: 

You can choose any port on your local host as long as it is not a privileged port (here 2123, should be greater than 1023). To transfer files to qcdochostb.qcdoc.bnl.gov via the tunnel (notice the capital –P in the scp command):

qcdochost4:>scp -P 2123 kerbuf.* user@localhost:/home/user/.

[BNL security Banner removed to conserve space]

user@qcdochostb's password:

You can set qcdcohostb as an alias to localhost by editing the /etc/hosts file on the remote host (you can have more than one aliases for localhost). For example, on qcdochost4.phys.columbia.edu the first line in /etc/hosts is:

               127.0.0.1 loopback localhost qcdochostb 

This way you can use qcdochostb in place of localhost in the above ssh command:

qcdochost4:>scp -P 2123 kerbuf.* user@qcdochostb:/home/user/.

NOTE: The ssh tunnel will remain open until you close it. When you are done with the file transfers,

find the process ID of the ssh tunnel and kill it.

Using the tunnel script

Instead of opening the tunnel manually, you can use a Perl script (by Steve Adler) that can:

1. Create a tunnel
2. ssh/scp via the tunnel (tssh, tscp)
3. Close the tunnel when done (tclose).

The web site with instructions about the tunnel script is at:

http://www.phy.bnl.gov/cybersecurity/tunnel.html

The above script has been installed on the QCDOC front-end node.

Installing the tunnel script

Download and untar the script on the machine from which you access BNL or JLab. Run the ./install.pl script to install it. If you are not a root user, put the binary in your own area:

qcdochost4:> ./install.pl
Enter the directory where you want tunnel installed [/usr/local/bin]: /home/user/tunnel
 

Every user needs to create a config file at $HOME/.tunnel/config on the remote host. In this file you will put a line such as:

qcdochost4:> more .tunnel/config
qcdochostb.qcdoc.bnl.gov:2010:ssh.qcdoc.bnl.gov:user:qcdochostb:bnl
 

NOTE 1: You may have several lines in your config file, one per remote host.

NOTE 2: The fields in the config line: qcdochostb.qcdoc.bnl.gov is the destination host at BNL

            (qcdi01.jlab.org at JLAB), 2010 is the local port that will be used by the tunnel

            (you can use any port greater than 1024), ssh1.bnl.gov is one of the ssh gateways

            at BNL (one at JLAB is login.jlab.org), user is the username, qcdochostb is an alias to localhost

            (in /etc/hosts), and the last field is unique abbreviation of you choosing.

NOTE 3: If localhost has not being aliased to qcdochostb in /etc/hosts, then in the fifth field on

            the above line you should use localhost, instead of qcdochostb.

NOTE 4: ssh records RSA host public keys for the hosts you connect to in .ssh/known_hosts or

            .ssh2/hostkeys. If localhost has not being aliased in /etc/hosts, the destination host is known

            to ssh as localhost. If you already have an RSA host key for localhost in the above files,

            ssh may not allow you to go through the tunnel because it thinks the host identification

            has been changed. In this case, you should remove or rename the the known_hosts or hostkeys

            entry before running tscp.

Patching the tunnel script

To make the tunnel script compatible with both Linux and AIX, you need to comment-out line 316 on the tunnel.pl script and replace it with the following:

# open(PSFLDS,"ps -efw|");
open(PSFLDS,"ps -e -o user,pid,ppid,ppid,etime,tty,time,args |");
 

The un-patched script works fine on Linux, but on AIX tclose cannot find the open ssh tunnel and thus doesn't close it.

Using the tunnel script

You can then then use tscp the same way you use scp: 

qcdochost4:> tscp kerbuf.* user@bnl:/home/stratos/.

Firing up tunnel to qcdochostb.qcdoc.bnl.gov through ssh1.bnl.gov using port 2010

user@ssh1.bnl.gov's password:
user@qcdochostb's password:

Once the tunnel has been opened, it will stay open so subsequent tscp commands use the tunnel and require entering your password only once, just as conventional scp.

When you are finished with the tunnel, you should close it with the tclose command. If you don't, the tunnel remains open and the port is tied up. 

qcdochost4:> tclose
Port to check is: 2010

Tunnel through port 2010 is active
Closing tunnel on port 2010 going to qcdochostb.qcdoc.bnl.gov through ssh1.bnl.gov

Example: Transferring files from FNAL to BNL or JLAB

To transfer files from FNAL to BNL or JLAB you should tunnel (either manually or using the tunnel script) through the BNL ssh gateways or the JLAB gateways. Here is an example of the .tunnel/config script:

lqcd.fnal.gov:> more .tunnel/config
qcdochostb.qcdoc.bnl.gov:2010:ssh1.bnl.gov:user:localhost:qcdochostb  qcdi01.jlab.org:2271:login.jlab.org:user:localhost:qcdi01

The fields in the config file are explained above. You can then use tscp to push or pull files from the remote sites:

lqcd.fnal.gov:> tscp user@qcdochostb:notes .

              [BNL banner removed]

Password:

user@localhost's password:

You will be asked for your gateway password first and then for your password on the remote host. When you are done using the tunnel, remember to close it:

lqcd.fnal.gov:> tclose

Transferring files to FNAL using kerberized clients

Kerberos clients have been installed on the USDOE front-end host. To transfer files to FNAL you first need to obtain a kerberos ticket (using kinit) and then use rcp or fscp to transfer files from/to FNAL:

qcdochostb:>kinit
Password for user@FNAL.GOV:

qcdochostb:>klist
Ticket cache: FILE:/var/krb5/security/creds/krb5cc_pid
Default principal: user@FNAL.GOV

Valid starting Expires Service principal
05/20/05 15:38:16 05/21/05 15:38:08 krbtgt/FNAL.GOV@FNAL.GOV
 

qcdochostb:>rcp testFile user@lqcd.fnal.gov:.

qcdochostb:>rcp user@lqcd.fnal.gov:testFile testFile.bak
 

ssh and scp on qcdochostb are not kerberized. Instead, Don Holmgren has patched ssh with the FNAL patches and installed the kerberized versions fssh and fscp. SO, after kinit, you can use fscp to transfer files to FNAL.

If you use a non-kerberized utility (ssh on qcdochostb is not kerberized) you will be prompted for your password. If you hit enter at the password prompt, you will be prompted with a cryptocard challenge:

qcdochostb:>ssh lqcd.fnal.gov
user@lqcd.fnal.gov's password:

Red Hat Linux release 7.1 (Seawolf)
Kernel 2.4.25 on a 4-processor i686

Press ENTER and compare this challenge to the one on your display: [22569055]
Enter the displayed response:
 

For more on using FNAL kerberos, see: http://lqcd.fnal.gov/kerberos_notes.html

The kerberos 5 package can be obtained from NCSA: http://www.ncsa.uiuc.edu/UserInfo/Consulting/Security/Software.html

Future Plans

Secure copy (scp) is currently the only tool that has been tested and working in transferring files to and from QCDOC systems. scp, however, is certainly not the most efficient way to transfer files. In addition, remote users come from sites with a variety of security requirements, restrictions and file copying tools. We plan to investigate the possibility of deploying additional tools to meet the needs of remote users. Such tools include:

• Storage Resource Managers (SRMs): Grid middleware components whose function is to provide dynamic space allocation and file management.
   

Available File Systems

/home

• Each user has a normal home directory /home/username, also known as $HOME.
• User home directories are backed up (daily incremental back ups).
• They are not accessible from the QCDOC machine partitions and thus you cannot read or write to them from an application running on the machine.
• Currently no quota restrictions have been implemented, although we may do so in the near future restricting the available space to a few GB per user (more on request if justified).

/host

• A /host filesystem is a local filesystem on the front-end nodes.
• A /host/username directory is created as soon as a user account is created.
• The /host partition is NFS mounted by the QCDOC machine partition and thus an application running on the QCDOC machine can read and write from/to the /host partition.
• The /host partition in not backed up.
• Currently no quota restrictions have been implemented.
• /host is also known as /qdata to comply with the USQCD standards.

Homepage

Last Modified: September 3, 2008

One of ten national laboratories overseen and primarily funded by the Office of Science of the U.S. Department of Energy (DOE), Brookhaven National Laboratory conducts research in the physical, biomedical, and environmental sciences, as well as in energy technologies and national security. Brookhaven Lab also builds and operates major scientific facilities available to university, industry and government researchers. Brookhaven is operated and managed for DOE’s Office of Science by Brookhaven Science Associates, a limited-liability company founded by Stony Brook University, the largest academic user of Laboratory facilities, and Battelle, a nonprofit, applied science and technology organization.

Privacy and Security Notice  | Contact Web Services for help