2 Changes between OASIS3-MCT_4.0 and OASIS3-MCT_3.0

Different developments were realised to improve the parallel performance of OASIS3-MCT_4.0. These developments are detailed in (Valcke et al 2018) .

• A new communication method, using the remapping weights to define the intermediate mapping decomposition, offers a significant gain at run time, especially for high-resolution cases running on a high number of tasks, thanks to reduced communication. However, as expected, the new method takes longer to initialize, partly due to the fact that the mapping weight file has to be read twice but also due to the extra cost for the initialization of the different MCT routers. That initialization cost is largely mitigated by an upgrade to MCT 2.10.beta1 which reduces the penalty to few seconds. Generally, it should be worth the extra initial cost to speed up the run time. See $NMAPDEC in section 3.2.

• The hybrid MPI+OpenMP parallelisation of the SCRIP library (previously fully sequential) leads to great improvement in the calculation of the remapping weights. The results obtained here show a reduction in the weight calculation time of 2 to 3 orders of magnitude with the new parallel SCRIP library for high-resolution grids. Details are available in (Piacentini et al 2018). The test_interpolation environment (see section ) gives a practical example on how to use OASIS3-MCT_4.0 to pre-calculate (i.e. in a separate job prior to the “real” simulation) the remapping weight and address file.

  Thanks to some preliminary work, few bugs were fixed, in particular in the bounding box definition of the grid cells. This solves an important bug observed in the Pacific near the equator for the bilinear and bicubic interpolations for Cartesian grids.

  However, given these modifications, one cannot expect to get exactly the same results for the interpolation weight-and-adress remapping files with this new parallel SCRIP version as compared to the previous SCRIP version in OASIS3-MCT_3.0. We checked in many different cases that the interpolation error is smaller or of the same order than before. We also observed that the parallelisation does not ensure bit reproducible results when varying the number of processes or threads.

• The new methods introduced in the global CONSERV operation reduce its calculation costs by one order of magnitude while still ensuring an appropriate level of reproducibility. This removes the bottleneck foreseen at high resolution with this important, and in few cases still unavoidable, global operation. These new methods are detailed in section 4.4 .

The other new features offered by OASIS3-MCT_4.0 are the following:

• Support for bundled coupling fields.

  Bundled fields is now supported in the oasis_put and oasis_get interfaces to allow easier coupling of multi-level or other related fields via a single namcouple coupling definition and a single call to oasis_put or oasis_get. Further details are provided in sec 2.2.7

• Automatic coupling restart writing

  An optional argument write_restart was added to the oasis_put routine. This argument is false by default but if it is explicitly set to true in the code, a coupling restart file will be written for that field only for that coupling timestep, saving the data that exists at the time of the call (see section 2.2.7).

• Exact consistency between the number of weights and fields

  Exact consistency is now required between number of weights fields in the coupling restart file and the arrays passed as arguments to the oasis_put routine. For example, for a 2nd order conservative remapping (CONSERV SECOND), 3 weights are needed and 3 fields must be provided as arguments: the value of the field, its gradient with respect to the longitude and its gradient with respect to the latitude. For a first order conservative remapping (CONSERV FIRST), only one weight and one field are needed. Using a weight file with 3 weights for a first order conservative remapping is no longer allowed.

• Upgrade in the namcouple configuration file

  The namcouple reading routine was cleaned up including a refactoring of the gotos and continue statements, addition of few reusable routines including an abort routine, removal of some dead code, addition of support for blank lines (which are now considered comments), removal of requirement that keywords start at character 2 on a line, removal of requirement for $END in the namcouple, and updates to some error messages.

• Other new functionalities with corresponding new namcouple keywords (see section 3.2)
  • $NUNITNO: specifies the minimum and maximum unit numbers to be used for input and output files in the coupling layer.

  • $NMATXRD: indicates the method used to read mapping weights, either orig or ceg. In both methods, the weights are read in chunks by the model master task. With the orig option, the weights are then broadcast to all other tasks and each task then saves the weights that will be applied to its grid points. With the ceg option, the master task reads the weights and then identifies to which other task each weight should be sent. A series of exchanges are then done with each other task involving just the weights needed by that other task. The orig method sends much more data but is more parallel, while the ceg method does most of the work on the master task but less data is communicated.

  • $NWGTOPT: indicates how to handle bad interpolation weights.

  • $NNOREST : if true, OASIS3-MCT will initialise any variable that normally requires a coupling restart file with zeros if that file does not exist.