BNL Home

Data Management and Software Engineering Group

XDC Mission and DMSE Capabilities

The Data Management and Software Engineering Group (DMSE) manages the ARM External Data Center (XDC) which identifies sources and acquires data, called "external data", to augment the data generated within the ARM Climate Research Facility.

The DMSE also provides comprehensive data lifecycle support for the ARM Facility from instrument deployment to data discovery. This support includes: instrument deployment and mentorship, data collection, data integration, data quality support, database support, web application development, and metadata management.

ARM Metadata Management

The DMSE has been involved with designing metadata databases and defining and implementing policies and procedures for maintaining metadata for over 20 years. Accurate and self-consistent metadata are crucial for maintaining the growing archive of ARM data and for facilitating the discovery and retrieval of those data for scientific analysis. In recent years the DMSE has been given a leadership role for the ARM metadata management.

External Data Processing and Acquisition

Processing: The XDC provides collection, development, operation and management for ARM external data products.

Products: The XDC collects a wide variety of data types, including numerical weather prediction model output, satellite, and surface mesonet data, from a number of different external institutions.  External data provide additional measurements and/or extend the spatial or temporal coverage for ARM sites. Examples include the Rapid Refresh (RAP) numerical weather forecast model dataset, which is used in products to enhance and augment data collected from ARM sites. Spatial coverage is increased by many of the XDC datasets over the ARM sites, such as the global satellite data set from the Ozone Monitoring Instrument (OMI). Data products are also provided from a variety of data networks and organizations, which expand ARM’s user community through data intercomparison and collaboration. Examples of organizations which provide data to the XDC include NASA AERONET, UCAR COSMIC, NCEP, ECMWF and the Oklahoma Mesonet. For more information on available data streams see XDC data set descriptions.

External Data Acquisition:  The scientific need and the priorities for acquiring, processing and archiving the external data-streams are established by the ASR  Working Groups and consider such factors as:

  • Availability and accessibility in other archives
  • Acquisition cost
  • Ease of use of the native data-format

If this analysis determines that significant value to the ARM data users will be added, the XDC will develop an automated process to acquire the data from the data provider, and reformat the data as necessary into an ARM-compatible data format. The processed data and the native format data are sent to the ARM permanent archive at ORNL.