Asynchronous task based Eulerian-Lagrangian parallel solver for combustion applications

Multiphysics applications often require the use of intimately coupled solvers. The application studied here makes use of an Eulerian solver to model fluid flow and combustion and a Lagrangian solver to model spray droplets. These are then implemented within one code to solve gas turbine combustion problems. However, large scale simulations where the flow and spray are within the same computational process can be expensive as the parallel solution does not scale well due to the poor load balancing of the spray particles. This is overcome by an asynchronous task-based Eulerian-Lagrangian (ATEL) approach where separate computational processes are used so that each solver can use an appropriate technique to partition the problem. Previously, this has been shown to overcome the load balancing problem but was restricted to a single computational node where shared memory could be used to transfer data. This work expands the methodology to work on large scale HPC facilities using a combination of shared memory and high speed interconnect to transfer data. The parallel methodology exploits one-sided shared memory communication when the corresponding processes are located within a computer node, otherwise it falls back to a conventional pair of send/receive. Also an hierarchical partitioning procedure is proposed that ensures that groups of parallel subdomains with high connectivity are placed on a compute node. Results are shown for two combustor cases: the DLR generic single sector combustor with an injection process that resembles a prefilming airblast atomiser which is found in many modern civil aircraft engines and a bluff-body swirl burner with a single source of fuel injection resembling a pressure atomiser. Both single sector and three sector combustor configurations have been used to carry out the performance studies. All performance cases have been tested with three different solver configurations: a) base-line Eulerian-Lagrangian solver b) ATEL and c) the baseline Eulerian solver without spray. The unstructured grids varied from 7M cells to 84M cells. In all cases the ATEL solution with flow, combustion and spray scaled identically to when solving flow and combustion alone. In fact, due to the memory bandwidth limitations of multicore processors, reducing the number of cores allocated to the flow and combustion to allocate some cores for the spray, hardly affected the computational speed of the flow solution, and due to the overlap of the spray calculation meant that the coupled Eulerian-Lagrangian solution could be achieved at almost no cost penalty to the Eulerian on its own. The choice of how to split the cores across the two solvers was considered by proposing a simple model to estimate the cost of each solver. Timing measurements show, that for the cases considered, the overall computational time is only weakly sensitive to this choice.