[v5.0.x] bug in the execution of iom_init in ORCA2_ICE_PISCES and X3_ORCA2_ICE_PISCES

Configurations Global
,
1

I compiled ORCA2_ICE_PISCES using the Intel OneAPI compilers 2024.1.0 on Leonardo CINECA, and in particular “mpiifx”; I used these flags in the cpp_ORCA2_ICE_PISCES.fcm
bld::tool::fppkeys key_si3 key_top key_xios key_xios3 key_qco key_vco_1d3d key_RK3

I compiled XIOS3 without the flag --use_oasis oasis3_mct

I ran NEMO using 4 MPI processes, together with XIOS in attached mode.
I obtained this error from each of the 4 processes

In file “extract_domain.cpp”, function “virtual void xios::CExtractDomain::checkValid(CDomain *)”, line 63 → Extract is wrongly defined, please check the values : ‘ni’ (1), ‘nj’ (148), ‘ibegin’ (49), ‘jbegin’ (0)

I added some printout line into the code and I understood that the error originates during the execution of the set_grid_znl function in iom.F90, and more precisely, it is caused by the call to the iom_set_zoom_domain_attr function, called with the “ptr” string assigned to the cdid argument.

I turned lines 2700 and 2702 in the iom.F90 file into comments (the calls to the iom_set_zoom_domain_attr function and to the iom_update_file_name function) and everithing seemed to work well.

I did all these steps to allow XIOS3 to work properly, and it seems that it works: before these changes I obtained votemper and vosaline 2D maps with values multiplied by 10; now I have
the tos (sea surface temperature) and sos (sea surface salinity) fields in the ORCA2_5d_00010101_00021231_grid_T.nc file, containing the “ocean grid T variables”.

For what concerns X3_ORCA2_ICE_PISCES, I compiled it with the same compiler and the following keys in the cpp_X3_ORCA2_ICE_PISCES.fcm file:

bld::tool::fppkeys key_si3 key_top key_xios key_qco key_vco_1d3d key_xios3

I also commented the same two function calls in the iom.F90 file (putting it in the MY_SRC directory).
However I find a different error that I still have to investigate further.

In file “domain.cpp”, function “void xios::CDomain::checkDomain()”, line 975 → [ id = nemo__domain_undef_id_63 , context = 'nemo ] The global domain is badly defined, ‘nj_glo’ attribute should be strictly positive so ‘nj_glo = 0’ is invalid.

This error appeared when I ran nemo and XIOS3 in attached mode using 4 MPI processes and also when I tried to run nemo and XIOS3 using respectively 4 MPI processes and 12 MPI processes. In both cases the message is repeated 4 times.

I don’t know what is the diagnostic that I am not writing on the output files when I comment the two functions above. I found this issue wrong index returned by dom_ngb (#603) · Issues · NEMO Workspace / Nemo · GitLab that should be related to this problem, however I see that the corrections are already included in my source code.

Do you have any suggestion?

2

MORE ACCURATE EXPLANATION
The XIOS code (compiled without the OASIS3-MCT libraries) hangs during a function call in src/node/extract_domain.cpp.
The function where it hangs has the following signature:

void CExtractDomain::checkValid(CDomain* domainSrc)

The hang occurs when execution enters the following if block, which checks that the extent of the extracted domain does not exceed the extent of the global domain.

if (ibegin < 0 || jbegin < 0 || iend > ni_glo - 1 || jend > nj_glo - 1)
  ERROR("CExtractDomain::checkValid(CDomain* domainSrc)",
    << "Extract is wrongly defined, "
    << "please check the values : 'ni' (" << ni.getValue() << "), 'nj' (" << nj.getValue() << "), "
    << "'ibegin' (" << ibegin.getValue() << "), 'jbegin' (" << jbegin.getValue() << ")");

Debug printouts have demonstrated that the error occurs when NEMO initializes the input/output manager (iom), during the call to iom_init_closedef, when this function calls xios_close_context_definition through the branch where llrstw=false.

The values printed before entering the if block, using the following statement, are shown below:

std::cerr << "MYDEBUG extract=" << this->getId() << " source=" << domainSrc->getId()
 << " ibegin=" << ibegin << " jbegin=" << jbegin << " iend=" << iend << " jend=" << jend
 << " ni_glo=" << ni_glo << " nj_glo=" << nj_glo << std::endl;

MYDEBUG
extract=nemo__extract_domain_undef_id_2
source=nemo__domain_undef_id_19
ibegin="49" jbegin="0"
iend=49 jend=147
ni_glo=180 nj_glo=148

Numerically, there should be no error, and the program should not enter the if block.

Analysis of the standard error output shows that the error arises because an extraction is attempted from a subdomain that has already been extracted from the global domain and contains only one column. Each process attempts to extract the same domain twice, and the second time the following values are reported:

MYDEBUG
extract=nemo__extract_domain_undef_id_3
source=nemo__domain_undef_id_63
ni_glo=1 nj_glo=148

I have demonstrated that only two files use "grid_ptr_...", both defined in file_def_nemo-oce.xml:

file id="file11" name_suffix="_diaptr2D" description="zonal mean variables"
file id="file12" name_suffix="_diaptr3D" description="zonal mean variables"

Disabling both files makes the error disappear. Enabling either file individually makes the error reappear. As for the fields, the error persists even when only one field in a single file is enabled.

Therefore, the error involves all three grids:

  • "grid_ptr_T_2D" (the only grid used by all fields in file 11)
  • "grid_ptr_T_3D" (used in file 12)
  • "grid_ptr_W_3D" (used in file 12)