LANS Seminar

Seminar Title: Fast coarse grid solvers for the solution of exascale Poisson equations

Speaker: Thilina Ratnayaka, Post-Doctoral Research Appointee, High-Performance Computing and Numerical Linear Algebra, ALCF, Argonne National Laboratory

Date/Time: November 27, 2024/ 10:30 AM-11:30 AM
Location: See Meeting URL on the cels-seminars website which will require an Argonne login.

Description: p-multigrid (pMG) is a widely used multilevel preconditioner for high-order spectral element and finite element methods (SEM/FEM). pMG consists of a series of nested meshes with decreasing polynomial orders starting from a fine mesh with a higher order (usually, p=7–8) to a coarse mesh with a lower order (usually, p=1). In the SEM, which is essentially a nodal-based matrix-free finite element method, finer level relaxations are performed with fast tensor product operations that are largely local and require only nearest neighbor communication. However, solving the coarse grid problem, which is essential for fast convergence, requires global all-to-all communication since the inverse of coarse operator is completely dense. The size of the coarse problem, which comprises p=1 elements on an unstructured mesh, is approximately equal to the number of elements in the SEM mesh. With exascale platforms, it is not uncommon to have more than a billion elements, so the communication-intensive coarse-grid problem presents a formidable computational challenge. Fast coarse solvers for pMG are essential for high performance SEM. The author presents a scalable two-level Schwarz method for the solution of the coarse problem that targets GPU-based exascale systems. The levels consist of a fully local problem in the p=1 space that is solved in parallel and which requires only a pair of near-neighbor data exchanges pre- and post-solve. This local solve is followed (either additively or multiplicatively) by a reduced-space solve on a very coarse system having approximately 4P unknowns, where P is the number of MPI ranks (i.e., GPUs). This reduced-space problem is solved cooperatively by all the processes. A novel contribution of this work is the introduction of a structured, non-nested, space for the reduced-space problem which enables communication-free interpolation between the p=1 and reduced spaces. We design the reduced space such that it is small enough to be solved using a fast, communication-minimal, sparse direct method while still retaining reasonable approximation properties compared to algebraic multigrid (AMG). Our sparse direct solver completes in a time that is comparable to a single all-reduce. We demonstrate the effectiveness of the proposed two level Schwarz method by comparing it to the state of the art AMG solver found in BoomerAMG by doing a series of experiments using NekRS, a highly scalable incompressible Navier-Stokes solver on GPUs. Results are presented on the Frontier exascale supercomputer at Oak Ridge Leadership Computing Facility.

Bio: Thilina Ratnayaka is a post-doctoral research appointee at ALCF specializing in high-performance computing and numerical linear algebra. He received his Ph.D. in Scientific Computing at the University of Illinois, where he was advised by Prof. Paul Fischer. He worked on improving the performance of Nek5000 and nekRS on cutting-edge computing architectures.

Please note that the meeting URL for this event can be seen on the cels-seminars website which requires an Argonne login.