About the Author

John Haggerty is a physicist at Brookhaven National Laboratory, and is serving as the PHENIX Run Coordinator for Run 9.

Ten Years Before the DAQ

By John Haggerty

Some of the DAQ group celebrating 5 kHz of event assembly after a meeting at the University of Colorado last summer hosted by Jamie Nagle. Jamie Nagle and his student, Matt Wysocki, made some crucial software improvements which allowed PHENIX to exceed 5 kHz of event assembly speed for the first time.

One of the defining characteristics of PHENIX is its ability to record data at high rates. That ability has extended the physics reach of PHENIX by allowing it to record more collision events, and more detailed information about the events. In the Au+Au run that ended recently, we were able to record up to 5000 events per second, which is the highest recording rate we have ever achieved. The result of this was writing data at about 600 Mbyte per second, which is almost one CD's worth of data every second. Since 1997, I have been the "DAQ Coordinator" in PHENIX, and had the singular honor of working with a dedicated band of PHENIX collaborators from Brookhaven, Columbia University, Iowa State University, Oak Ridge National Laboratory, and other PHENIX institutions, and this is some of the story of those years.

When Sam Aronson, the PHENIX Project Manager, appointed me DAQ Coordinator in the last century, the landscape of electronics and computing was quite different. A trinity of PHENIX collaborators, Glenn Young of Oak Ridge National Laboratory, Cheng-Yi Chi of Columbia University, and Leo Paffrath of Brookhaven, had pioneered a vision of the "Front End Electronics" which was very fast, very compact, and where data was transmitted entirely through a relatively small number of high speed fiber optic cables to PHENIX computers. With Bill Sippach at Columbia designing the "Data Collection Modules" that are the first stop for data sent up from the collision hall, Chuck Britton and Bill Bryan at Oak Ridge, Chuck Pancake at Stony Brook, and Paul O'Connor at Brookhaven were among the many collaborators who made this vision a reality with their engineering artistry. The "Front End Electronics" or FEE is the electronics that resides in the experimental hall and takes the tiny signals from the particle detectors, amplifies them, and digitizes them so they can be read by computers. What made this scheme possible were advances in optical data transmission and the design for PHENIX of nearly a dozen custom "Application Specific Integrated Circuits" (ASIC's) which are custom electronic chips designed by those artistes of engineering. It was a sophisticated and somewhat daring approach at the time, but one that has proven crucial to PHENIX. The experiments starting up at the LHC at CERN are using many of the same techniques.

The other sea change taking place in the data acquisition world was the use of relatively inexpensive computers connected by high speed networks. After much discussion, Brian Cole and the Columbia group designed a network based on a network standard called ATM for connecting PC's running Window-NT, which at first were quite standard Dell desktop computers. For the computers that would control the system and record the data, the Brookhaven group used Sun computers running the Solaris operating system. This combination of technologies was successfully used for data taking in the "Engineering Run" in 2000, and the first PHENIX physics run in 2001.

Physicists can always imagine a way to make things better, or faster, or collect more data, and the DAQ group in PHENIX has, through seven physics runs, pushed and prodded and coerced the data acquisition system to ever higher performance. Shortly after Run 1 ended, we began the process of moving everything we could to the Linux operating system, encouraged by Martin Purschke and Steve Adler. The network was upgraded to the new, faster Gigabit Ethernet standard, and new, faster, and more compact computers were purchased, which David Winter from Columbia used to port a big part of the online code that assembled data into complete events to Linux. One of the key features of the Front End Modules called multi-event buffering, which allows an event to be saved in the memory of the FEM while another one is being digitized, was brought into operation, and this increased the fraction of the time the electronics was sensitive to collisions.

Figure 1. Livetime as a function of event rate for PHENIX runs. The livteime was limited by the speed of the Front End Electronics until the multievent buffering feature (simultanueous digitization and readout) was activated in Run 5, and event assembly was sped up in Run 7. (enlarge)

The figure illustrates the progress in data acquisition over those years. You can see how as we improved our networking and software we have been able to record more events per second. The "livetime" is the fraction of the time that we were able to record a collision when there was a collision to record, and is a measure of how fast the electronics can digitize signals from the detectors; you can see the surge in livetime when we activated multi-event buffering in run 5.

Work today continues forward, because data acquisition has to keep up with and exceed what the accelerator can throw at us, and RHIC has had its own success story in increasing the luminosity over these years. New detectors in PHENIX bring with them new electronics, and software can always be improved. The one thing that has been the same over my years as DAQ Coordinator is the dedication, persistence, and skill of the DAQ group, which has been a real privilege to work in. Martin Purschke is taking over that position in PHENIX, with Steve Boose and Ed Desmond, both of whom received BNL Engineering Awards for their work on PHENIX, as his deputies, and I can only hope that they have as much fun as I did and they leave some good stuff to do when I get back.

It has been a team effort, so let me close by listing the present and past collaborators who have worked in the DAQ group:

Steve Adler, Sotiria Batsouli, Sergey Belikov, Abby Bickley, Steve Boose, Cheng-Yi Chi, Mickey Chiu, Brian Cole, Ed Desmond, Justin Frantz, Tony Frawley, John Haggerty, Ali Hanks, Jiangyong Jia, Jiamin Jin, Sean Kelly, John Lajoie, Felix Matathias, Jamie Nagle, Chris Pinkenburg, Martin Purschke, Peter Steinberg, Carla Vale, David Winter, Chris Witzig, Matt Wysocki, Glenn Young, and Chun Zhang