Disclaimer : This post is not finished and has NOT been fully reviewed. Feedback are welcome, but no arguments (yet) please!
Controls should be simple. But what if the system is really -really- complex ? … like an High Performance Research software ?
Introduction
The direct controls of HPC softwares are usually hard to master : cabalistic inputs files; ninja-like commands and loads of jargon. This is because their users are experts who need a raw access to the guts of the code.
Here we lay down the user experience (UX) of several HPC codes, so you can see how they can look alike, or differ… and make our opinion.
General UX
Walberla
WaLBerla is a Massively Parallel Framework to Lattice Boltzman methods. The “easy way” is to control this numerical platform through [lbmpy](https://pypi.org/project/lbmpy/, a code generation library controlled in Python.
To sum up, you need 3 files to set up your simulation :
-
a
.cppfile in which you build the simulation using waLBerla. An executable will be generated from this file. -
a
.pyfile using waLBerla python extensions (where you specify the stencil and the collision method). When you compile waLBerla, this script will build c++ libraries that you need in your .cpp file to set up the simulation.
This is for the creation of the executable. Then :
- a
.prmcontaining information about the geometry, the fluid, the boundary conditions, the initial conditions… The executable generated from the .cpp file will read the parameters contained in the .prm file in order to build the set up and run the simulation.
Once those three files are written, you have to compile waLBerla (c++ libraries are built from the .py file and an executable is generated from the .cpp file). Then, you just have to call the executable while specifying the .prm file :
mpirun -np $nbp /scratch/coop/donze/walberla/build/WFC_LAMINAIRE/CodegenLbmLamEx /scratch/coop/donze/walberla/WFC_LAMINAIRE/CodegenLbmLamEx.prm
waLBerla is using non-dimensional quantities. Therefore, before running a simulation, one must non-dimentionalize the quantities (forcing terms, viscosity, …).
AVBP
AVBP is a Massively Parallel solver for Reactive Flows using the Large Eddy Simulation of Navier-Stokes equations. Refer to the official AVBP website for an exhaustive overview. There is also a specific post on setting up an AVBP relatively complex run.
AVBP uses a single executable, and need for most simulations
- An ASCII, Keywords based
run.paramsfile to control numerics and models. - An ASCII, Keywords base
asciiBoundfile to declare boundary conditions. - A Binary
solutBound.h5file to store the targets of boundary conditions. - A Binary
mesh.h5file to store the unstructured mesh and patch definition.
No recompilation is needed before running the case..
mpirun -np $nbp $AVBP_HOME/HOSTS/$AVBP_HOST/AVBPEXEC/avbp_v7.8_Kraken
Installation
Walberla
The first thing to do is installing lbmpy and pystencils libraries to your Python environment : lbmpy, pystencils.
pip install lbmpy
pip install pystencils
Then, install waLBerla on your computer and compile the code using the following instructions :
# create a separate build directory
cd walberla
mkdir build
cd build
# set up the build system
cmake ..
# modify the configuration if necessary
ccmake . # or alternatively "cmake-gui . "
# compile the code
make -j 4
When setting up the build system, you need to set the variable “WALBERLA_BUILD_WITH_CODEGEN” to on. We’ll use codegen tools to set up the stencil and the collision method we’ll be using during the simulation.
For additional information about waLBerla, report to the website.
AVBP
AVBP is stored on CERFACS’s forge . If you have the credentials:
git clone (thecerfacsforge):cfd-apps/avbp.git
Then, compile the code :
cd avbp
module load avbp
make
The mesh
Walberla
In waLBerla, the domain is first partitioned into blocks, where each block holds its own field. This has the advantage that the domain can easily be distributed to multiple processes, where every process can have one or more local blocks.
In the .prm file :
DomainSetup
{
blocks < 1, 1, 1>;
dx 100;
cellsPerBlock < 110, 10, 31>;
periodic < 1, 0, 1 >;
}
As you can notice, the periodicity conditions are specified here in the Domain setup section (not in the Boundary conditions section).
Finally, when building waLBerla with the CMAKE configuration WALBERLA_BUILD_WITH_OPENMESH enabled, one can use its capabilities for loading, storing and manipulating meshes. It’s even possible to manipulate triangle meshes. However, we only used quad meshes which is the most commonly used in LBM. You’ll find here more information about meshes manipulation
AVBP
To generate a simple mesh based on the test case domain, we’re using HIP (for more information, follow the instructions in our HIP quick-start manual). Here a the script to generate an Hexa mesh (Cubes, as a matter of fact) :
ge 0 0 11000 2700 1100 270
copy 3d 0 1000 100 z
se bc-or 1 4 4 1
se bc-te 4 Inlet
se bc-te 2 Outlet
se bc-te 3 Periodic_left
se bc-te 1 Periodic_right
se bc-te 5 Wall
se bc-te 6 Symmetry
se bc-ty Inlet l00
se bc-ty Outlet u00
se bc-ty Periodic_left u01
se bc-ty Periodic_right l01
wr hd -7 windfarm_quad_fine
As we can notice, we also need to specify the boundary conditions. We’ll be using those information later in order to generate the boundary condition file. After running the hip script, 3 files are generated : a mesh file, a xmf file, a asciibound file (used to generate the BC file).
The initial solution
Walberla
Here, we used the method described in the tutorial available on waLBerla’s website.
The density is also defined here and will remain the same and waLBerla uses incompressible LBM equations.
To initialize the fields, we are using the lbm::initializer::PdfFieldInitializer class with lbm::initializer::ExprSystemInitFunction which is capable of parsing mathematical expressions from the parameter file to set up complex initial flows. We’ll not dive into the details of the C++ code behind the initializer as it’s using quite complex C++ coding techniques.
The most important thing you need to know is that the required parameters and expressions for initializing the density and velocity are defined in the parameter file. The block is called ShearFlowSetup. For rho, u_x, u_y and u_z, mathematical expressions can be specified which may include the variables x, y, z for a cell’s global position and n_x, n_y, n_z representing the number of cells in each direction. These expressions will be evaluated for each domain cell.
For example, using the following parameters, we can in define a parabolic initial speed profile (the same profile we’re supposed to get after running a laminar Poiseuille test case) :
ShearFlowSetup
{
rho 1.1720;
u_x (23.2 * 0.05/ 10) * (( 2 * y / n_y ) - ( y * y / (n_y * n_y) ));
u_y 0;
u_z 0;
noise_magnitude 0.005;
noise_seed 42;
}
AVBP
Use the tool makesolution to generate your initial solution, and check your initial solution using Paraview.
The Boundary Conditions
Walberla
In the .prm file :
Boundaries
{
Border { direction S; walldistance -1; flag NoSlip; }
Border { direction N; walldistance -1; flag FreeSlip; }
}
If you prefer having a NoSlip condition at the lower boundary, you may write :
Boundaries
{
Border { direction S, N; walldistance -1; flag NoSlip; }
}
AVBP
The first thing to do here is to make sure the asciibound file generated after running the hip script is fully completed. In our case, we need to fill the Wall and Symmetry boundary conditions (WALL_LAW_ADIAB and SYMMETRY). Once completed, use the tool makesolubound to generate the Boundary Conditions.
The controls
Walberla
A model in LBM consists in a stencil and a collision method. To generate this model, we’ll use LBMPY : one of waLBerla’s codegen tools. The first thing to do is writing a python script specifying the stencil and the collision method we’ll be using. We can also specify a forcing term during the method definition (CF code bellow).
stencil = 'D3Q19'
omega = sp.Symbol('omega')
layout = 'fzyx'
trt_params = {'stencil': stencil,
'method': 'trt',
'force': (0.000427, 0, 0),
'relaxation_rate': omega}
You can find here more information about the creation of LBM kernels.
The power of LBMPY is that during the compilation, c++ libraries will be generated from this python script. The next step will be to use those libraries in order to set up our simulation.
The simulation is set up in waLBerla using c++ script that extracts information from a .prm file (containing information about the geometry, the fluid, the boundary conditions, the initial conditions…) and the libraries built using lbmpy. Most of the code we used for this test case was found on waLBerla’s website (tutorials section).
In the .prm file :
Fluid
{
viscosity 0.09182;
}
Omega, which is the collision operator main parameter, is then calculated from the viscosity. In the .cpp file (main file) :
const real_t viscosity = fluid.getParameter< real_t >("viscosity", real_c(1.8));
const real_t omega = lbm::collision_model::omegaFromViscosity(viscosity);
Viscosity is actually the only “fluid parameter” we need to specify (in addition to density which is defined in the Field Initialization) as it’s directly linked to the collision parameter omega. The models used in waLBerla use the isothermal hypothesis. Therefore, you don’t have yo specify thermal parameters for the fluid.
Stability issues
The collision frequency omega which is the main ingredient of the LBM (SRT/TRT,…) exhibits a theoretical upper bound of 2 that is related with the positiveness of the molecular kinematic viscosity. Thus, stability problems arise as the collision frequency approaches to this limiting value.
In LBM, quantities are non-dimensional. Moreover, the default value for the time-step is 1. Therefore, if you observe stability issues, a first solution would be to modify this default value. Of course, you’ll have renormalize the quantities afterwards.
trt_params = {'stencil': stencil,
'method': 'trt',
'force': (0.000427, 0, 0),
'relaxation_rate': omega,
'timestep' : 0.5}
AVBP - The run.params
Fill the run.params according to the documentation on the AVBP run params help.
Now you can use the tool checkrun to keep track of your setup sanity.
Here is the run-control section of our run.params file :
$RUN-CONTROL
solver_type = ns
diffusion_scheme = FE_2delta
simulation_end_iteration = 100000000
mixture_name = AIR
reactive_flow = no
equation_of_state = pg
LES_model = smago
prandtl_turb = 0.60000000D+00
schmidt_turb = 0.60000000D+00
convection_scheme = LW
CFL = 0.70000000D+00
Fourier = 0.10000000D+00
compute_chemical_timestep = no
artificial_viscosity_model = colin
artificial_viscosity_2nd_order = 0.50000000D-01
artificial_viscosity_4th_order = 0.50000000D-02
clip_species = no
wall_clock_time_limit = 11.5d+0
$end_RUN-CONTROL
Concluding remarks
The user experience of HPC softwares should not be compared blindly to commercial codes, because this user experience crudeness serves a purpose : these controls are meant for experts, and must evolve a lot with the minimal additional cost.
With years, HPC software specializes in an application field, allowing the emergence of more mature user interaction. waLBerla is for now a bleeding edge numerical testbed for LBM methods. AVBP was a similar numerical testbed for parallel computation in the 90ies, its UX eventually improved since with a specialization to Large Eddy Simulation of combustion flows.