BNL Scientific Data and Computing Center Projects
Mass Storage as a Service
As the volume of data collected by scientific experiments explodes, there will be a need to provide a place to economically store the data, in the short and long term. As the volume of data is expected to vastly exceed what can be provided by disk based storage solutions, the obvious candidate for providing this service is one backed by magnetic tape. At the RHIC and Atlas Computing Facility (RACF), the HPSS (High Performance Storage System) system has been providing mass storage services to the experiments at the Relativistic Heavy Ion Collider and the Large Hadron Collider for over 15 years. This system utilizes HPSS software (developed by IBM and the Department of Energy’s National Laboratories), magnetic tape cartridges and tape drives, and robotic tape libraries to provide near line access to as much as 60 petabytes of existing data. With the latest generation tape drives and tape cartridges, the currently deployed HPSS system can potently store hundreds of PB of data.
From a purely technical perspective, the RACF HPSS system is clearly capable of storing the expected deluge of data that will be generated by scientific instruments going forward; however, the details on how the HPSS storage capacity is provided to experimenters are less clear. Currently, the RACF HPSS system is utilized by only a few organizations with a small number of clearly defined patterns for storing and retrieving data. Dedicated equipment and awareness of the capabilities of the HPSS system allow current groups to efficiently and effectively utilize the system. Moving forward, the expectation is that a centrally provided mass storage service will be needed by multi-experiment facilities like NSLS-II and the Center for Functional Nanomaterials (CFN) with their exponentially larger number of groups and individuals with widely varying access requirements and varying levels of technical expertise. In light of this changing environment, how short and long term storage is presented to users is critical to the utility of the service. As part of this research it will be necessary to get an understanding of what users expect out of a mass storage service and how they expect to interact with the system. Among the key issues that need to be understood are data ingest rates (files per hour, GB/sec, peak and sustained), performance variability, service utilization windows, shared vs dedicated resources, file transfer mechanisms for data import and export, read/write ratios, life time and life cycle of the data placed into the system, access authorization requirements, service availability requirements, time locality of data access, file access “temperature” and “temperature” distribution (hot/warm/cold representing access frequency), and data confidentiality requirements. In addition to getting information from potential users, it is also necessary to provide them with information on the characteristics of the back end HPSS system so that they can more effectively utilize the system.
A tight working relationship will be established between prominent stakeholders from NSLS-II beamlines and CFN, and the CSI/BNL Scientific Data and Computing Center project team to conduct the information exchange necessary to provide the data points needed to address the above listed questions. From the standpoint of providing mass storage services to a large and diverse community of new users, there are many issues that need to be addressed. Although HPSS is a fully functional mass storage system, how the system is configured, how users access the system and how their data is organized in the system are crucial in making the mass storage service sustainable in the short and long term. In addition to the “native” capabilities of the HPSS system, it is likely that additional capabilities will need to be added to support the usage patterns and expectations of new users of the mass storage service. Identifying these critical new capabilities and making them available will require development effort and more likely than not, the deployment of new systems to support these capabilities.
In light of the open questions with providing mass storage services, both from the service provider and consumer perspective, the goal of this investigation is to identify and deploy the changes that are needed to transform the RACF HPSS system into an effective mass storage system for the broader scientific community.
