High Performance Computing Group

Overcoming the Computational Bottlenecks of Particle-resolved Direct Numerical Simulation with High Performance Computing and Machine Learning

Particle-resolved direct numerical simulations (PR-DNS) play a crucial role in simulating the aerosol-cloud-precipitation system that is recognized to be the key in forecasting weather and climate (changes). Despite its importance, considerable challenges exist due to its high computational cost, especially in a large domain size. The challenges are mainly driven by solving multiphysical partial differential equations (PDEs) and the large number of particles to be tracked. This project, lead by Brookhaven Lab’s EBNN, will address the computational bottlenecks of the PR-DNS models through two research thrusts. Thrust 1 is addressing the effective use of large-scale heterogeneous high performance computing (HPC) systems, where issues related to performance, scalability and portability of the existing algorithms and software will be addressed, while Thrust 2 includes the investigation and development of novel machine learning (ML)-based PDE solvers to further speed up PR-DNS simulations. Here, ML-based solvers will be enhanced to address multiphysical procedures, super-resolution domains, etc., that are critical to PR-DNS. PR-DNS simulations on HPC systems will be used to contrast and validate the ML-based results in terms of performance (e.g., time-to-solution) and accuracy with the goal of identifying situations where ML-based PDE solvers are applicable and have the overall advantage.

Outcomes from these two thrusts will help overcome the computational challenges posed by PR-DNS and allow for simulating microphysical processes in a domain several orders of magnitude larger than currently feasible. The goal is to increase the PR-DNS simulation domain from the current 1×1×1 cubic meters to 10×10×10 cubic meters, significantly improving fundamental understanding of microphysical processes in a complex system. As a result, the PR-DNS framework will help fill in key knowledge gaps that are essential and urgent for improving predictions of climate, weather, and renewable energy, e.g., solar and wind. In addition, Brookhaven Lab is actively planning to build a large experimental facility to study clouds (cloud chamber). The new PR-DNS framework will benefit this large long-term investment with its ability to run longer simulations and understand the evolution of many particles (aerosol and cloud droplets) systems in a turbulent environment, as well as support the simulation of larger domains that can provide the truth for other models where microphysical processes are parameterized. The ML-based solver also can benefit broader DOE scientific simulations where PDE-solving is at the center, such as molecular dynamics, computational fluid dynamics, and quantum chromodynamics.

CSI’s Meifeng Lin leads the HPC aspects of this project, while Lingda Li is heading the machine learning effort.

Publications

Zhang T, L Li, V Lopez-Marrero, et al. Digital Twin of PR-DNS: Accelerating Dynamical Fields with Neural Operators in Particle-Resolved Direct Numerical Simulation. Authorea. July 09, 2023. DOI: 10.22541/essoar.168889862.29246354/v1.