General Information

Top of Page

Computing at Brookhaven National Laboratory

New York Blue/P and BNL Blue Gene/Q Parallel Supercomputers

The New York Blue machine is an 18 rack Blue Gene/L and a 2 rack Blue Gene/P massively parallel supercomputer based on the IBM system-on-chip technology. The BNL Blue Gene/Q is a 1 rack IBM Blue Gene/Q (DD2 revision hardware and software) massively parallel supercomputer. Only BNL employees and BNL collaborators will be granted accounts on these machines.

Blue Gene Machines

Blue Gene is not a typical parallel computer. It is meant for codes that scale well into hundreds or even thousands of processors. The processors in Blue Gene are slower than those on the BNL Linux clusters. So, unless your code scales well and so can benefit from using a lot of processors, you may be better off running it with fewer processors on e.g. a standard Linux cluster.

  • How to login to Blue Gene computer resources

New York Blue/L

Please note, New York Blue/L has been decommissioned and these web pages have not yet been updated to reflect its removal from service. The New York Blue/P is now one rack and remains in service. The BNL Blue Gene/Q also remains in service.

It consists of 18432 (18 x 1024) dual-processor Compute Nodes (Blue Gene chip) with each Compute Node having two standard 700 MHz PowerPC440 processors (a total of 36864). The two processors (cores) on a chip share a 1 GB of DDR memory.

The 18 racks are arranged in six (6) rows with three (3) racks each making up a 48x24x16 3D torus. The first rack on the first row (row 0) is designated as R00, while the third one on the sixth row (row 5) is R52. More...

In addition to the Compute Nodes, there are dedicated I/O nodes. Each I/O node provides the dedicated hardware that serves the operating system tasks to a group of compute nodes. The I/O node with the compute nodes that it serves make up a group that is referred to a pset. In New York Blue, the pset ratio (the ratio of I/O nodes to compute nodes) can be one of the following: 1:16, 1:32, 1:64 and 1:128.

New York Blue is subdivided into partitions (a.k.a blocks). Each partition has a specific size (number of nodes), type (Mesh or Torus), and pset ratio (one of the ratios mentioned above). Partitions can be predefined or created by users dynamically according to their job needs. A complete list of predefined partitions and naming conventions can be found in the User's Guide. The pset ratio varies throughout the machine, having one of the four pset ratios mentioned above.

There are 512 compute nodes on a midplane.

New York Blue partitions have access to about 360 TB (usable) General Parallel File System (GPFS) file systems that provide scalable, high performance I/O. The Pseries cluster that serves the GPFS file systems to the Blue Gene I/O nodes (IONs), the Front-End Node (FEN) and the Service Node (SN) is made up of 14 p575 8-way GPFS NSD servers running SUSE SLES 9. Each GPFS server connects to the functional network via 10-GigE FC links. Individual SATA disks are housed in seven SA29500 Data Direct Networks (DDN) disk systems racks.

New York Blue has one Front-End Node (FEN) that provides a direct user interface to the machine partitions. Users compile their application codes and submit their jobs on the FEN. The FEN is a Pseries p570, 8-way server with 64 GB of RAM running SuSE Linux Enterprise Server 9 (SLES 9). The New York Blue FEN, with the hostname fen.bluegene.bnl.gov, can be accessed remotely via the gateways (known as ssh.bluegene.bnl.gov outside BNL and as ssh.bluegene.bnl.local inside the BNL campus network).

Jobs on New York Blue partitions are scheduled using Loadleveler and are run using mpirun. The User's Guide provides examples on how to compile application code and run interactive and batch jobs. In the future we plan to only allow batch jobs.

New York Blue/P

New York Blue/P is a Blue Gene/P, the second generation of the Blue Gene series of supercomputers. It is a one rack (2048 compute nodes) Blue Gene/P. Threaded programs can be run on its compute nodes. Each of its compute chips contains four PowerPC 450 processor cores, running at 850 MHz.

BNL Blue Gene/Q

It consists of a 1 rack IBM Blue Gene/Q (DD2 revision hardware and software) massively parallel supercomputer.

IBM Blue Gene Supercomputer