Reading time : 7 minutes
(A super nintendo controller, Photo Devin Berko) Fortran Namelists is a I/O standard since 1991, contemporary to the SuperNintendo Controller. Less fun to play with, and very limited, Namelists are often shunned. Moreover, making your own I/O parser is very tempting. Let’s try them out completely, for once
Experimenting namelists on a real-life software.
The solver AVTP is a in-house Cerfacs thermal solver, written in Fortran. The input files were in custom ASCII files, and had to be refreshed. For example, the main control file was written this way (blank lines did matter):
'../MESH/mesh.mesh.h5' ! Mesh file
'../MESH/mesh.asciiBound' ! Ascii Boundary file
'../SOLUTBOUND/mesh.solutBound.h5' ! Boundary solution file
'./SOLUT/init.h5' ! inital solution
'./SOLUT/mesh' ! output files
'./TEMPORAL/' ! temporal evolution directory
1.0d0 ! Reference length | scales coordinates X by X/reflen
200 ! Number of iterations
400 ! Number of elements per group (typically of order 100)
1 ! Preprocessor: skip (0), use (1) & write (2) & stop (3)
-1 ! Interactive details of convergence (1) or not (0)
100 ! Prints convergence every x iterations
1 ! Type of output control
10 ! Store solution every x iterations
2 ! Spatial scheme - ivisc : 1 = four delta operator ; 2 = 2 delta operator
2 ! istoreadd
1 ! Steady state (0) or (1) unsteady calculation or (2) fixed dt or (3) freq calculation
-1 ! Temporal scheme identification (explicit or implicit)
1000 1e-6 ! Scheme description
400.d0 ! Fourier parameter for viscous time-step
Different formats were considered:
- The AVBP 7.X Parser, keyword based, human oriented, with possibly non-hashable keys (Read more on AVBP non-hashable keys here).
- Fortran Namelists. Keyword based, present sice Fortran 77, standard built-in since Fortran 90. Look a lot alike AVBP 7.X Parser.
- JSON fortran, a JSON parser. Perfect for computer data exchange but not really human oriented, needing a Fortran 2008 compiler.
- TOML-f, a TOML parser, good for both human and machine, but needing a Fortran 2008 compiler.
- Do not change format (always an option!)
The AVBP 7.X Parser is 28800 lines of Fortran, half the size of the full AVTP thermal solver sources. It would have be a huge duplication (unless we also move the AVBP parser to a common library, and therefore impact AVBP for AVTP). JSON and TOML were tempting, but quite far from the habits of the large end-users basis. We decided to try out Fortran Namelists, and see how far we could go.
Fortran Namelists being a standard, their conversion back and forth with others languages are already available. For python, we used f90nml.
Limitation of namelists.
First some resources :
- An introduction to Namelists - part of the JULES documentation
- F77 Namelists - the original reference
- IBM Namelists from F95 to 2003 - beware of IBMs extensions
- Reading only some values from anamelist, A Fortran-Lang discussion that sums up the frustrations of namelists.
The prominent limitation with namelists is the static aspect. Once compiled, the code will allocate a fixed number of degrees of freedom once-for-all. Technically speaking, its structure is immutable. Since Fortran 2003, the size of a variable can be dynamic in namelists, but the structure is not.
This static aspect is bad news, sure, but not a total pushback. We just have to cope with two constraints:
Optional arguments
In most serialized format (e.g. YAML) an optional value is inexistent if missing:
order:
type_: pizza
tomato: true
mozzarella: true
This data will know nothing about dressing.
order:
type_: gnocchi
dressing: sugo di noce
This data will know nothing about tomato and mozzarella parameters.
Oppositely, A namelist I/O implies all the options are present in memory, whatever the case. Indeed the namelist declaration will be :
namelist \order\ type_, tomato, mozzarella, dressing
We can ask the user, while muting the parsing errors:
&order
type_ = 'pizza'
tomato = 'yes'
mozzarella = 'no'
/
This lloks like dynamic. However, the Fortran code will still know about dressing with its default value if any, or garbage if not. Therefore the developer must keep all the d.o.f. in mind, since all will co-exists at all times.
Dynamic blocks
The second problem, Harder to deal with, is when the number of blocks changes. For example, in YAML or JSON, you can declare a list of objects such as this one:
- type = 'pizza'
tomato = 'yes'
mozzarrella = 'no'
- type = 'pizza'
tomato = 'yes'
mozzarrella = 'yes'
- type = 'pizza'
tomato = 'no'
mozzarrella = 'no'
The number of elements is not bounded, we could have 0,1, 4 or 66 elements in the list. With static namelists we still can give the impression of a dynamic choice to the end user :
&
&customer001
type = 'pizza'
tomato = 'yes'
mozzarrella = 'no'
/
&customer002
type = 'pizza'
tomato = 'yes'
mozzarrella = 'yes'
/
&customer003
type = 'pizza'
tomato = 'no'
mozzarrella = 'no'
/
However the developer must pre-declare, for example, 30 blocks from customer001 to customer030, and read the number of activated customers somewhere else. The two downsides are:
- There can be a mismatch between the number of elements activated and the ones that are actually declared.
- The code must be re-compiled if a situation requires a bigger list (here 34 customers)
A full example
Here is an example on how to actually implement a namelist. It was inspired from an example from Programming In Modern Fortran, adding a trick to find problematic input lines.
The namelist declaration is limited to the inside of read_some_parameters(), and is not visible from the signature. The opening and closing are outsourced to open_inputfile(file_path, file_unit, iostat) and close_inputfile(file_path, file_unit, iostat) to have a common ground for user feedback. These subroutines are not speficic to namelists by the way.
!! This module will show a simple namelist reader
!! Its takes from Modern Fortran Porgramming
!! https://cyber.dabamos.de/programming/modernfortran/namelists.html
!!
!! We added a little trick in the namelist loader,
!! found on http://degenerateconic.com/namelist-error-checking/
!! to make it point out , if the readin fails,
!! the line in the input file that let to a failure.
program main
!! Assuming we want to read some parameters.
!! No namelist aspect should be seen here
use, intrinsic :: iso_fortran_env, only: stderr => error_unit
implicit none
!integer :: int_
character(len=32) :: type_
character(len=32) :: tomato
character(len=32) :: mozzarella
character(len=32) :: dressing
! Read from file.
call read_some_parameters('short.nml', type_, tomato, mozzarella, dressing)
! Output some values.
print '(2a)', 'type_: ', type_
print '(2a)', 'tomato: ', tomato
print '(2a)', 'mozzarella: ', mozzarella
print '(2a)', 'dressing: ', dressing
contains
subroutine read_some_parameters(file_path, type_, tomato, mozzarella, dressing)
!! Read some parmeters, Here we use a namelist
!! but if you were to change the storage format (TOML,or home-made),
!! this signature would not change
character(len=*), intent(in) :: file_path
character(len=32), intent(out) :: type_
character(len=32), intent(out) :: tomato
character(len=32), intent(out) :: mozzarella
character(len=32), intent(out) :: dressing
!integer, intent(out) :: type_
integer :: file_unit, iostat
! Namelist definition===============================
namelist /ORDER/ &
type_ , &
tomato, &
mozzarella, &
dressing
type_ = "undefined"
tomato = "undefined"
mozzarella = "undefined"
dressing = "undefined"
! Namelist definition===============================
call open_inputfile(file_path, file_unit, iostat)
if (iostat /= 0) then
!! write here what to do if opening failed"
return
end if
read (nml=ORDER, iostat=iostat, unit=file_unit)
call close_inputfile(file_path, file_unit, iostat)
if (iostat /= 0) then
!! write here what to do if reading failed"
return
end if
end subroutine read_some_parameters
!! Namelist helpers
subroutine open_inputfile(file_path, file_unit, iostat)
!! Check whether file exists, with consitent error message
!! return the file unit
character(len=*), intent(in) :: file_path
integer, intent(out) :: file_unit, iostat
inquire (file=file_path, iostat=iostat)
if (iostat /= 0) then
write (stderr, '(3a)') 'Error: file "', trim(file_path), '" not found!'
end if
open (action='read', file=file_path, iostat=iostat, newunit=file_unit)
end subroutine open_inputfile
subroutine close_inputfile(file_path, file_unit, iostat)
!! Check the reading was OK
!! return error line IF not
!! close the unit
character(len=*), intent(in) :: file_path
character(len=1000) :: line
integer, intent(in) :: file_unit, iostat
if (iostat /= 0) then
write (stderr, '(2a)') 'Error reading file :"', trim(file_path)
write (stderr, '(a, i0)') 'iostat was:"', iostat
backspace(file_unit)
read(file_unit,fmt='(A)') line
write(stderr,'(A)') &
'Invalid line : '//trim(line)
end if
close (file_unit)
end subroutine close_inputfile
end program main
You can try this code with the short.nml example (do not forget the carriage return after \) :
&order
tomato = 'yes'
type_ = 'pizza'
mozzarella = 'no'
! dressing = 'bah'
/
In the original example, the namelist included arrays and derived types. hic sunt dracones : some gfortran versions had issues when parsing derived types (see dedicated thread). But should you really ask for inputs mapping exactly the way memory is structured? Are you asking to much from the poor end-user?
Application to our solver
The Main input file
The AVTP code is not asking for a huge variety of d.o.f, because the Heat conduction modeling is rather simple. The main input file looks like this:
&hpc_debug
ncell_group = 400
preprocessor = 1
/
&input_control
asciibound_file = './MESH/bloup.asciibound'
initial_solution_file = './INIT/bloup.init.h5'
material_database = 'material_database.dat'
mesh_file = './MESH/bloup.mesh.h5'
probe_file = './bloup_record.dat'
solutbound_file = './SOLUTBOUND/bloup.solutbound.h5'
tab_species = 'FER_varie'
/
&output_average
save_average = 'no'
save_average_freq = 10
save_average_name = './SOLUT/av'
save_average_out = 3000
/
&output_control
save_solution = 'yes'
save_solution_additional = 'minimum'
save_solution_name = './SOLUT/bloup'
save_solution_time = 0.0001
save_temporal = 'yes'
save_temporal_balance = 'yes'
save_temporal_dirname = './TEMPORAL/'
save_temporal_iteration = 1
/
&preproc
ichoice_nml = 1
iolcomm_nml = 0
iorder_nml = -1
ippnode_nml = 1
ireorder_nml = 1
itghost_nml = 0
ndum_nml = 0
/
&run_control
diffusion_scheme = 'FE_2delta'
fourier = 5
implicit_conv_crit = 1e-06
implicit_nb_it = 1000
number_of_species = 1
simulation_end_time = 0.001
solver_type = 'implicit_CG'
/
The boundary conditions
Again AVTP setups can comply well replicated number of blocks. As the number of active blocks is read elsewhere while loading the CAD (the mesh), there is no additional variable needed.
&patch001
patch_name_nml = 'outlet'
boundary_condition = 'WALL_ISOT'
target_origin = 'solutbound'
/
&patch002
patch_name_nml = 'inlet'
boundary_condition = 'WALL_ISOT'
target_origin = 'solutbound'
/
&patch003
patch_name_nml = 'wallup'
boundary_condition = 'WALL_ISOT'
target_origin = 'solutbound'
/
&patch004
patch_name_nml = 'walldown'
boundary_condition = 'WALL_ISOT'
target_origin = 'solutbound'
/
&patch005
patch_name_nml = 'walldroite'
boundary_condition = 'WALL_ISOT'
target_origin = 'solutbound'
/
&patch006
patch_name_nml = 'wallgauche'
boundary_condition = 'WALL_ISOT'
target_origin = 'solutbound'
/
&patch007
patch_name_nml = 'perfo'
boundary_condition = 'WALL_FLUX'
target_origin = 'solutbound'
/
(To see how many patches are currently in you AVTP source see SOURCES/COMMON/commonbl_master.h)
A key/value parser, by blocks, Like the AVBP parser
To the end user accustomed to AVBP software, the main visual difference is the markup of the blocks, and quotes around strings:
# Namelist style (AVTP 3.0, Thermal setup)
&input_control
asciibound_file = './MESH/bloup.asciibound'
initial_solution_file = './INIT/bloup.init.h5'
material_database = 'material_database.dat'
mesh_file = './MESH/bloup.mesh.h5'
probe_file = './bloup_record.dat'
solutbound_file = './SOLUTBOUND/bloup.solutbound.h5'
tab_species = 'FER_varie'
/
# AVBP Parser style (AVBP 7.8, CFD Setup)
$INPUT-CONTROL
mesh_file = ../../../../Documents/MESH_C3SM/TRAPVTX/trappedvtx.mesh.h5
asciibound_file = ./SOLUTBOUND/combu.asciibound
asciibound_tpf_file = None
initial_solution_file = ./INIT/combu.init.h5
mixture_database = ./mixture_database.dat
probe_file = ./combu_probe.dat
solutbound_file = ./SOLUTBOUND/combu.solutbound.h5
species_database = ./species_database.dat
$end_INPUT-CONTROL
Adding validation and much more
One could reasonably argue:
Ok but my custom parser does sooo much more, it can check the validity of the inputs and give precise information to fix the problem. And Fortran namelist are not that smart don’t they?
Good point. Namelists will complain if the type does not match, that is all. Moreover, the error raised by a misspelled keyword is sometimes vague. But that is OK because namelists deals with I/O, not validation. Let’s use a real validator…
Move to python
First how we can load the namelist in python? Note this code is universal to any namelist:
import f90nml
from opentea.noob.asciigraph import nob_asciigraph
nml = f90nml.read("./run.params") # the namelist object
nml_dict = nml.todict() # translate to dict
print(nob_asciigraph(nml_dict)) # show output
As you can see, all types are well casted:
┣ hpc_debug (OrderedDict)
┃ ┣ ncell_group (int) : 400
┃ ┗ preprocessor (int) : 1
┣ input_control (OrderedDict)
┃ ┣ asciibound_file (str) : ./MESH/bloup.asciibound
┃ ┣ initial_solution_file (str) : ./INIT/bloup.init.h5
┃ ┣ material_database (str) : material_database.dat
┃ ┣ mesh_file (str) : ./MESH/bloup.mesh.h5
┃ ┣ probe_file (str) : ./bloup_record.dat
┃ ┣ solutbound_file (str) : ./SOLUTBOUND/bloup.solutbound.h5
┃ ┗ tab_species (str) : FER_varie
┣ output_average (OrderedDict)
┃ ┣ save_average (str) : no
┃ ┣ save_average_freq (int) : 10
┃ ┣ save_average_name (str) : ./SOLUT/av
┃ ┗ save_average_out (int) : 3000
┣ output_control (OrderedDict)
┃ ┣ save_solution (str) : yes
┃ ┣ save_solution_additional (str) : minimum
┃ ┣ save_solution_name (str) : ./SOLUT/bloup
┃ ┣ save_solution_time (float) : 0.0001
┃ ┣ save_temporal (str) : yes
┃ ┣ save_temporal_balance (str) : yes
┃ ┣ save_temporal_dirname (str) : ./TEMPORAL/
┃ ┗ save_temporal_iteration (int) : 1
┣ preproc (OrderedDict)
┃ ┣ ichoice_nml (int) : 1
┃ ┣ iolcomm_nml (int) : 0
┃ ┣ iorder_nml (int) : -1
┃ ┣ ippnode_nml (int) : 1
┃ ┣ ireorder_nml (int) : 1
┃ ┣ itghost_nml (int) : 0
┃ ┗ ndum_nml (int) : 0
┗ run_control (OrderedDict)
┣ diffusion_scheme (str) : FE_2delta
┣ fourier (int) : 5
┣ implicit_conv_crit (float) : 1e-06
┣ implicit_nb_it (int) : 1000
┣ number_of_species (int) : 1
┣ simulation_end_time (float) : 0.001
┗ solver_type (str) : implicit_CG
Now we can translate this dictionary to a JSON file, infer online a SCHEMA, and tinker it to our taste. You can read a full explanation on Using the JSON SCHEMA standard for scientific applications. SCHEMA can be lenghty so let us see a small part:
type: object
properties:
run_control:
type: object
properties:
control_kind:
type: string
default: time
stepper:
type: string
default: fourier
solver_type:
type: string
default: implicit_CG
diffusion_scheme:
type: string
default: FE_2delta
enum: [FE_2delta, FE_4delta]
implicit_nb_it:
type: integer
default: 1000
minimum: 100
(...)
Now how do we make a validator out of this? We use jsonschema, a general purpose validator, to write these 5 statements, applicable for any namelist:
import f90nml
import yaml
import jsonschema
from opentea.noob.asciigraph import nob_asciigraph
nml = f90nml.read("./run.params")
nml_dict = nml.todict()
with open("./schema.yml", "r") as fin:
schema = yaml.load(fin, Loader=yaml.SafeLoader)
jsonschema.validate(nml, schema)
Now if you run this small script on a bad input, e.g. diffusion_scheme=FE_4_delta (Spurious _ here), you get the following error.
Failed validating 'enum' in schema['properties']['run_control']['properties']['diffusion_scheme']:
{'default': 'FE_2delta',
'enum': ['FE_2delta', 'FE_4delta'],
'type': 'string'}
On instance['run_control']['diffusion_scheme']:
'FE_4_delta'
Takeaway
The static aspect of a Fortran Namelist is a regular pushback, at first. But if you accept the constraints, it can be also a very simple framework to deal with afterwards. In addition, because the structure is a reliable standard, smart people already made some I/O helpers for Python, a language more suited to stunts on dynamic structures and string operations. This enable the most mainstream and advanced techniques of data validation and automatic documentation on a Fortran input.
So re-think about it twice : do you really -really- need a dynamic structure for your inputs values?
Namelists are for you if:
- You want a solid parser for free, even without the power Fortran 2008
- the set of combinations for your d.o.f. is humanly manageable
- you can live with unused variables
Namelists are not for you if:
- The set of combinations for your d.o.f. is exponentially large
- Unused variables is a terrible sin
- Your designed the ultimate parser yourself
- You already moved to TOML-f