FESTIM using all cores of a shared machine (MUMPS / OpenMP threading) and how to limit it

User Support
1

Hi all,

First time posting!! My colleague and I have been working with FESTIM2 for the past few weeks and ran into a behaviour on a shared server that I could not find documented anywhere on the forum, so I am writing it up here in case it saves someone else the debugging time.

Running a FESTIM case built on HydrogenTransportProblemDiscontinuous the process ended up using every CPU available on the machine rather than one. Nothing failed and the results were correct, but the run quietly slowed down the other calculations my colleagues were running on the same cluster.

This occurs because the default petsc_options of HydrogenTransportProblemDiscontinuous (and that of HydrogenTransportProblem as well) use MUMPS as pc_factor_mat_solver_type for the matrix factorisation. The parallelization is not something FESTIM directly asks for, it comes from the threaded BLAS / OpenMP layer underneath MUMPS.

Fix

Capping the thread count before launching solves it:


export OMP_NUM_THREADS=1

export OPENBLAS_NUM_THREADS=1

export MKL_NUM_THREADS=1

export BLIS_NUM_THREADS=1

In our case OMP_NUM_THREADS=1 on its own was sufficient.

Before we found this we had worked around the problem by launching runs inside a tmux server with restricted CPU usage. That works too, but the environment variables are much cleaner.

Alternatively, another solver could be chosen, as the problem classes accept a petsc_options argument, so the solver and preconditioner can be set directly from your script.

Speed analysis

Not only was it found that parallelization slowed down the other server’s processes but it can paradoxically end up slowing down the FESTIM code itself. For instance, the problem we were dealing with was a 1D case with 1000 degrees of freedom iterated over 800,000 timesteps. Making an analysis of the time it took to run as a function of the threads used, one ends up with the following figure:

image
image567×346 20.9 KB

The explanation we found plausible is that a problem this small produces very small matrices in the MUMPS factorization, so the time it takes to synchronize the multiple threads is larger than the arithmetic itself. Since that factorization happens at every timestep that overhead is paid 800,000 times.

Thanks to @remidm for the quick response and to Jørgen for proposing the solution.

For reference, my setup: FESTIM 2.2rc1, dolfinx 0.11.0, installed via conda-forge.

Cheers

2

Thanks for sharing this @alevazcor !