Reading time : 7 minutes

heat

(A super nintendo controller, Photo Devin Berko) Fortran Namelists is a I/O standard since 1991, contemporary to the SuperNintendo Controller. Less fun to play with, and very limited, Namelists are often shunned. Moreover, making your own I/O parser is very tempting. Let’s try them out completely, for once

Experimenting namelists on a real-life software.

The solver AVTP is a in-house Cerfacs thermal solver, written in Fortran. The input files were in custom ASCII files, and had to be refreshed. For example, the main control file was written this way (blank lines did matter):

'../MESH/mesh.mesh.h5'                    ! Mesh file
'../MESH/mesh.asciiBound'                 ! Ascii Boundary file
'../SOLUTBOUND/mesh.solutBound.h5'        ! Boundary solution file
'./SOLUT/init.h5'                         ! inital solution
'./SOLUT/mesh'                            ! output files
'./TEMPORAL/'                             ! temporal evolution directory

1.0d0     ! Reference length  | scales coordinates X by X/reflen 

200        ! Number of iterations

400       ! Number of elements per group (typically of order 100)
1         ! Preprocessor: skip (0), use (1) & write (2) & stop (3)

-1        ! Interactive details of convergence (1) or not (0)
100       ! Prints convergence every x iterations
1         ! Type of output control
10        ! Store solution every x iterations

2         ! Spatial scheme - ivisc : 1 = four delta operator ; 2 = 2 delta operator
2         ! istoreadd
1         ! Steady state (0) or (1) unsteady calculation or (2) fixed dt or (3) freq calculation

-1         ! Temporal scheme identification (explicit or implicit)
1000 1e-6  ! Scheme description
400.d0     ! Fourier parameter for viscous time-step

Different formats were considered:

  • The AVBP 7.X Parser, keyword based, human oriented, with possibly non-hashable keys (Read more on AVBP non-hashable keys here).
  • Fortran Namelists. Keyword based, present sice Fortran 77, standard built-in since Fortran 90. Look a lot alike AVBP 7.X Parser.
  • JSON fortran, a JSON parser. Perfect for computer data exchange but not really human oriented, needing a Fortran 2008 compiler.
  • TOML-f, a TOML parser, good for both human and machine, but needing a Fortran 2008 compiler.
  • Do not change format (always an option!)

The AVBP 7.X Parser is 28800 lines of Fortran, half the size of the full AVTP thermal solver sources. It would have be a huge duplication (unless we also move the AVBP parser to a common library, and therefore impact AVBP for AVTP). JSON and TOML were tempting, but quite far from the habits of the large end-users basis. We decided to try out Fortran Namelists, and see how far we could go.

Fortran Namelists being a standard, their conversion back and forth with others languages are already available. For python, we used f90nml.

Limitation of namelists.

First some resources :

The prominent limitation with namelists is the static aspect. Once compiled, the code will allocate a fixed number of degrees of freedom once-for-all. Technically speaking, its structure is immutable. Since Fortran 2003, the size of a variable can be dynamic in namelists, but the structure is not.

This static aspect is bad news, sure, but not a total pushback. We just have to cope with two constraints:

Optional arguments

In most serialized format (e.g. YAML) an optional value is inexistent if missing:

order:
   type_: pizza
   tomato: true
   mozzarella: true

This data will know nothing about dressing.

order:
    type_: gnocchi
    dressing: sugo di noce

This data will know nothing about tomato and mozzarella parameters.

Oppositely, A namelist I/O implies all the options are present in memory, whatever the case. Indeed the namelist declaration will be :

namelist \order\ type_, tomato, mozzarella, dressing

We can ask the user, while muting the parsing errors:

&order
    type_ = 'pizza'
    tomato = 'yes'
    mozzarella = 'no'
/

This lloks like dynamic. However, the Fortran code will still know about dressing with its default value if any, or garbage if not. Therefore the developer must keep all the d.o.f. in mind, since all will co-exists at all times.

Dynamic blocks

The second problem, Harder to deal with, is when the number of blocks changes. For example, in YAML or JSON, you can declare a list of objects such as this one:

- type = 'pizza'
  tomato = 'yes'
  mozzarrella = 'no'
- type = 'pizza'
  tomato = 'yes'
  mozzarrella = 'yes'
- type = 'pizza'
  tomato = 'no'
  mozzarrella = 'no'

The number of elements is not bounded, we could have 0,1, 4 or 66 elements in the list. With static namelists we still can give the impression of a dynamic choice to the end user :

&

&customer001
    type = 'pizza'
    tomato = 'yes'
    mozzarrella = 'no'
/

&customer002
    type = 'pizza'
    tomato = 'yes'
    mozzarrella = 'yes'
/

&customer003
    type = 'pizza'
    tomato = 'no'
    mozzarrella = 'no'
/

However the developer must pre-declare, for example, 30 blocks from customer001 to customer030, and read the number of activated customers somewhere else. The two downsides are:

  • There can be a mismatch between the number of elements activated and the ones that are actually declared.
  • The code must be re-compiled if a situation requires a bigger list (here 34 customers)

A full example

Here is an example on how to actually implement a namelist. It was inspired from an example from Programming In Modern Fortran, adding a trick to find problematic input lines.

The namelist declaration is limited to the inside of read_some_parameters(), and is not visible from the signature. The opening and closing are outsourced to open_inputfile(file_path, file_unit, iostat) and close_inputfile(file_path, file_unit, iostat) to have a common ground for user feedback. These subroutines are not speficic to namelists by the way.

!! This module will show a simple namelist reader
!! Its takes from Modern Fortran Porgramming 
!! https://cyber.dabamos.de/programming/modernfortran/namelists.html
!!
!! We added a little trick in the namelist loader,
!! found on http://degenerateconic.com/namelist-error-checking/
!! to make it point out , if the readin fails,
!! the line in the input file that let to a failure.

program main
    !! Assuming we want to read some parameters.
    !! No namelist aspect should be seen here
    use, intrinsic :: iso_fortran_env, only: stderr => error_unit
    implicit none

    !integer :: int_
    character(len=32) :: type_
    character(len=32) :: tomato
    character(len=32) :: mozzarella
    character(len=32) :: dressing

    ! Read from file.
    call read_some_parameters('short.nml', type_, tomato, mozzarella, dressing)
    ! Output some values.
    print '(2a)',    'type_: ', type_
    print '(2a)',    'tomato: ', tomato
    print '(2a)',    'mozzarella: ', mozzarella
    print '(2a)',    'dressing: ', dressing

contains
    subroutine read_some_parameters(file_path, type_, tomato, mozzarella, dressing)
        !! Read some parmeters,  Here we use a namelist 
        !! but if you were to change the storage format (TOML,or home-made), 
        !! this signature would not change

        character(len=*),  intent(in)  :: file_path
        character(len=32), intent(out) :: type_
        character(len=32), intent(out) :: tomato
        character(len=32), intent(out) :: mozzarella
        character(len=32), intent(out) :: dressing
        !integer, intent(out) :: type_
        integer                        :: file_unit, iostat

        ! Namelist definition===============================
        namelist /ORDER/ &
            type_ , &
            tomato, &
            mozzarella, &
            dressing
        type_ = "undefined"
        tomato = "undefined"
        mozzarella = "undefined"
        dressing = "undefined"
        ! Namelist definition===============================

        call open_inputfile(file_path, file_unit, iostat)
        if (iostat /= 0) then
            !! write here what to do if opening failed"
            return
        end if

        read (nml=ORDER, iostat=iostat, unit=file_unit)
        call close_inputfile(file_path, file_unit, iostat)
        if (iostat /= 0) then
            !! write here what to do if reading failed"
            return
        end if
    end subroutine read_some_parameters

    !! Namelist helpers

    subroutine open_inputfile(file_path, file_unit, iostat)
        !! Check whether file exists, with consitent error message
        !! return the file unit
        character(len=*),  intent(in)  :: file_path
        integer,  intent(out) :: file_unit, iostat

        inquire (file=file_path, iostat=iostat)
        if (iostat /= 0) then
            write (stderr, '(3a)') 'Error: file "', trim(file_path), '" not found!'
        end if
        open (action='read', file=file_path, iostat=iostat, newunit=file_unit)
    end subroutine open_inputfile

    subroutine close_inputfile(file_path, file_unit, iostat)
        !! Check the reading was OK
        !! return error line IF not
        !! close the unit
        character(len=*),  intent(in)  :: file_path
        character(len=1000) :: line
        integer,  intent(in) :: file_unit, iostat

        if (iostat /= 0) then
            write (stderr, '(2a)') 'Error reading file :"', trim(file_path)
            write (stderr, '(a, i0)') 'iostat was:"', iostat
            backspace(file_unit)
            read(file_unit,fmt='(A)') line
            write(stderr,'(A)') &
                'Invalid line : '//trim(line)
        end if
        close (file_unit)   
    end subroutine close_inputfile

end program main

You can try this code with the short.nml example (do not forget the carriage return after \) :

&order
    tomato = 'yes'
    type_ = 'pizza'
    mozzarella = 'no'
!    dressing = 'bah'
/

In the original example, the namelist included arrays and derived types. hic sunt dracones : some gfortran versions had issues when parsing derived types (see dedicated thread). But should you really ask for inputs mapping exactly the way memory is structured? Are you asking to much from the poor end-user?

Application to our solver

The Main input file

The AVTP code is not asking for a huge variety of d.o.f, because the Heat conduction modeling is rather simple. The main input file looks like this:

&hpc_debug
    ncell_group = 400
    preprocessor = 1
/

&input_control
    asciibound_file = './MESH/bloup.asciibound'
    initial_solution_file = './INIT/bloup.init.h5'
    material_database = 'material_database.dat'
    mesh_file = './MESH/bloup.mesh.h5'
    probe_file = './bloup_record.dat'
    solutbound_file = './SOLUTBOUND/bloup.solutbound.h5'
    tab_species = 'FER_varie'
/

&output_average
    save_average = 'no'
    save_average_freq = 10
    save_average_name = './SOLUT/av'
    save_average_out = 3000
/

&output_control
    save_solution = 'yes'
    save_solution_additional = 'minimum'
    save_solution_name = './SOLUT/bloup'
    save_solution_time = 0.0001
    save_temporal = 'yes'
    save_temporal_balance = 'yes'
    save_temporal_dirname = './TEMPORAL/'
    save_temporal_iteration = 1
/

&preproc
    ichoice_nml = 1
    iolcomm_nml = 0
    iorder_nml = -1
    ippnode_nml = 1
    ireorder_nml = 1
    itghost_nml = 0
    ndum_nml = 0
/

&run_control
    diffusion_scheme = 'FE_2delta'
    fourier = 5
    implicit_conv_crit = 1e-06
    implicit_nb_it = 1000
    number_of_species = 1
    simulation_end_time = 0.001
    solver_type = 'implicit_CG'
/
The boundary conditions

Again AVTP setups can comply well replicated number of blocks. As the number of active blocks is read elsewhere while loading the CAD (the mesh), there is no additional variable needed.

&patch001
    patch_name_nml = 'outlet'
    boundary_condition = 'WALL_ISOT'
    target_origin = 'solutbound'
/

&patch002
    patch_name_nml = 'inlet'
    boundary_condition = 'WALL_ISOT'
    target_origin = 'solutbound'
/

&patch003
    patch_name_nml = 'wallup'
    boundary_condition = 'WALL_ISOT'
    target_origin = 'solutbound'
/

&patch004
    patch_name_nml = 'walldown'
    boundary_condition = 'WALL_ISOT'
    target_origin = 'solutbound'
/

&patch005
    patch_name_nml = 'walldroite'
    boundary_condition = 'WALL_ISOT'
    target_origin = 'solutbound'
/

&patch006
    patch_name_nml = 'wallgauche'
    boundary_condition = 'WALL_ISOT'
    target_origin = 'solutbound'
/

&patch007
    patch_name_nml = 'perfo'
    boundary_condition = 'WALL_FLUX'
    target_origin = 'solutbound'
/

(To see how many patches are currently in you AVTP source see SOURCES/COMMON/commonbl_master.h)

A key/value parser, by blocks, Like the AVBP parser

To the end user accustomed to AVBP software, the main visual difference is the markup of the blocks, and quotes around strings:

# Namelist style (AVTP 3.0, Thermal setup)

&input_control
    asciibound_file = './MESH/bloup.asciibound'
    initial_solution_file = './INIT/bloup.init.h5'
    material_database = 'material_database.dat'
    mesh_file = './MESH/bloup.mesh.h5'
    probe_file = './bloup_record.dat'
    solutbound_file = './SOLUTBOUND/bloup.solutbound.h5'
    tab_species = 'FER_varie'
/

# AVBP Parser style (AVBP 7.8, CFD Setup)

$INPUT-CONTROL
  mesh_file = ../../../../Documents/MESH_C3SM/TRAPVTX/trappedvtx.mesh.h5
  asciibound_file = ./SOLUTBOUND/combu.asciibound
  asciibound_tpf_file = None
  initial_solution_file = ./INIT/combu.init.h5
  mixture_database = ./mixture_database.dat
  probe_file = ./combu_probe.dat
  solutbound_file = ./SOLUTBOUND/combu.solutbound.h5
  species_database = ./species_database.dat
$end_INPUT-CONTROL

Adding validation and much more

One could reasonably argue:

Ok but my custom parser does sooo much more, it can check the validity of the inputs and give precise information to fix the problem. And Fortran namelist are not that smart don’t they?

Good point. Namelists will complain if the type does not match, that is all. Moreover, the error raised by a misspelled keyword is sometimes vague. But that is OK because namelists deals with I/O, not validation. Let’s use a real validator

Move to python

First how we can load the namelist in python? Note this code is universal to any namelist:

import f90nml
from opentea.noob.asciigraph import nob_asciigraph

nml = f90nml.read("./run.params") # the namelist object
nml_dict = nml.todict() # translate to dict
print(nob_asciigraph(nml_dict)) # show output

As you can see, all types are well casted:

┣ hpc_debug (OrderedDict)
┃  ┣ ncell_group (int) : 400
┃  ┗ preprocessor (int) : 1
┣ input_control (OrderedDict)
┃  ┣ asciibound_file (str) : ./MESH/bloup.asciibound
┃  ┣ initial_solution_file (str) : ./INIT/bloup.init.h5
┃  ┣ material_database (str) : material_database.dat
┃  ┣ mesh_file (str) : ./MESH/bloup.mesh.h5
┃  ┣ probe_file (str) : ./bloup_record.dat
┃  ┣ solutbound_file (str) : ./SOLUTBOUND/bloup.solutbound.h5
┃  ┗ tab_species (str) : FER_varie
┣ output_average (OrderedDict)
┃  ┣ save_average (str) : no
┃  ┣ save_average_freq (int) : 10
┃  ┣ save_average_name (str) : ./SOLUT/av
┃  ┗ save_average_out (int) : 3000
┣ output_control (OrderedDict)
┃  ┣ save_solution (str) : yes
┃  ┣ save_solution_additional (str) : minimum
┃  ┣ save_solution_name (str) : ./SOLUT/bloup
┃  ┣ save_solution_time (float) : 0.0001
┃  ┣ save_temporal (str) : yes
┃  ┣ save_temporal_balance (str) : yes
┃  ┣ save_temporal_dirname (str) : ./TEMPORAL/
┃  ┗ save_temporal_iteration (int) : 1
┣ preproc (OrderedDict)
┃  ┣ ichoice_nml (int) : 1
┃  ┣ iolcomm_nml (int) : 0
┃  ┣ iorder_nml (int) : -1
┃  ┣ ippnode_nml (int) : 1
┃  ┣ ireorder_nml (int) : 1
┃  ┣ itghost_nml (int) : 0
┃  ┗ ndum_nml (int) : 0
┗ run_control (OrderedDict)
   ┣ diffusion_scheme (str) : FE_2delta
   ┣ fourier (int) : 5
   ┣ implicit_conv_crit (float) : 1e-06
   ┣ implicit_nb_it (int) : 1000
   ┣ number_of_species (int) : 1
   ┣ simulation_end_time (float) : 0.001
   ┗ solver_type (str) : implicit_CG

Now we can translate this dictionary to a JSON file, infer online a SCHEMA, and tinker it to our taste. You can read a full explanation on Using the JSON SCHEMA standard for scientific applications. SCHEMA can be lenghty so let us see a small part:

type: object
properties:
  run_control:
    type: object
    properties:
      control_kind:
        type: string
        default: time
      stepper:
        type: string
        default: fourier
      solver_type:
        type: string
        default: implicit_CG
      diffusion_scheme:
        type: string
        default: FE_2delta
        enum: [FE_2delta, FE_4delta]
      implicit_nb_it:
        type: integer
        default: 1000
        minimum: 100
(...)

Now how do we make a validator out of this? We use jsonschema, a general purpose validator, to write these 5 statements, applicable for any namelist:

import f90nml
import yaml
import jsonschema
from opentea.noob.asciigraph import nob_asciigraph
nml = f90nml.read("./run.params")
nml_dict = nml.todict()
with open("./schema.yml", "r") as fin:
    schema = yaml.load(fin, Loader=yaml.SafeLoader)
jsonschema.validate(nml, schema)

Now if you run this small script on a bad input, e.g. diffusion_scheme=FE_4_delta (Spurious _ here), you get the following error.

Failed validating 'enum' in schema['properties']['run_control']['properties']['diffusion_scheme']:
    {'default': 'FE_2delta',
     'enum': ['FE_2delta', 'FE_4delta'],
     'type': 'string'}

On instance['run_control']['diffusion_scheme']:
    'FE_4_delta'

Takeaway

The static aspect of a Fortran Namelist is a regular pushback, at first. But if you accept the constraints, it can be also a very simple framework to deal with afterwards. In addition, because the structure is a reliable standard, smart people already made some I/O helpers for Python, a language more suited to stunts on dynamic structures and string operations. This enable the most mainstream and advanced techniques of data validation and automatic documentation on a Fortran input.

So re-think about it twice : do you really -really- need a dynamic structure for your inputs values?

Namelists are for you if:

  • You want a solid parser for free, even without the power Fortran 2008
  • the set of combinations for your d.o.f. is humanly manageable
  • you can live with unused variables

Namelists are not for you if:

  • The set of combinations for your d.o.f. is exponentially large
  • Unused variables is a terrible sin
  • Your designed the ultimate parser yourself
  • You already moved to TOML-f

Like this post? Share on: TwitterFacebookEmail


Antoine Dauptain is a research scientist focused on computer science and engineering topics for HPC.

Keep Reading


Published

Category

Pitch

Tags

Stay in Touch