Accessing data using HPSS
(Burt Holzman)
This page was last updated at 08/04/04 02:11 PM
The HPSS system at RCF
is where we store all of our raw data and basic reconstructed data (DSTs). It consists of a tape robot, some servers (ours is rmds02), and software to access
files on tape. Be careful when accessing HPSS. Besides the
obvious possibility of marking our files for deletion, you could also overload the system and compromise our ability to
sink data (or Star's).
All PHOBOS data is stored in one of four "classes of service" (COS): Raw,
Recon, Analysis, Archive.
To see the allocations, look here.
We access this data using two accounts:
phobsink: this account is used to sink data (R/W) to the Raw COS. (discouraged!)
phobreco: this account is used for reconstruction on the CRS: input from
Raw COS (R/O), output to Recon COS.
There are three ways to get data from HPSS:
- Log onto any RCF machine.
- You should put large datafiles on some subdirectory of $SCR
- Check whether someone has already downloaded your file:
find_scratch <filename>
If this returns a value, you should make a hard link to that file:
ln <find_scratch_filename>
If find_scratch returns nothing, you should continue as below.
- hsi
- cd ~phobreco/pr00/recon (or other directory where your data
is in HPSS. HPSS has an entirely separate filesystem from the one on RCF).
- now you can ls and get files
to your heart's content like regular ftp
(warning: try to avoid ls in big directories if
you can avoid it -- it is extremely time-consuming!)
- bye & logout.
3. Direct pftp using the phobreco account
(Working, but not recommended unless needed)
- Log onto any RCF machine as phobreco.
- pftp hpss.rcf.bnl.gov 2121
- cd ~phobsink/engrun/data (or other directory as above)
- now you can ls and get files
to your heart's content like regular ftp
- ls -l outfile will make a file outfile
with a complete directory listing
- bye & logout.
This page is maintained by Burt Holzman
Last Edited: Wednesday, August 04, 2004 14:11