[v4.2.x] Coupled mode NEMO-OASIS-XIOS freezes at `prism_initxios.x` task

Versions v4.2.x
,
1

I use xios trunk (Revision: 2320) with NEMO trunk (Revision: 15770) in coupled mode.
I also use the last official version of oasis3-mct_4.0

The coupled model freezes after printing

 (oasis_unitget)        9999
 (oasis_unitget)        9999
 (oasis_unitget)        9999
 starting wrf task            0  of            1
 ---> prism_initxios.x           0

I tried to change call_oasis_enddef to false in iodef.xml. But it resulted in segmentation fault

 (oasis_unitget)        9999
 (oasis_unitget)        9999
 (oasis_unitget)        9999
 starting wrf task            0  of            1
---> prism_initxios.x           0
-> info : CServer : Register new Context : nemo_server
-> info : Register new Context : nemo
forrtl: severe (174): SIGSEGV, segmentation fault occurred
Image              PC                Routine            Line        Source             
xios_server.exe    0000000000C9A82A  for__signal_handl     Unknown  Unknown
libpthread-2.17.s  00002B36B2DFA630  Unknown               Unknown  Unknown
libmpi.so.12.0.0   00002B36B19D1071  Unknown               Unknown  Unknown
libmpi.so.12.0.0   00002B36B19D2C47  PMPI_Iprobe           Unknown  Unknown
xios_server.exe    000000000064FD3C  Unknown               Unknown  Unknown
xios_server.exe    0000000000650690  Unknown               Unknown  Unknown
xios_server.exe    0000000000904D26  Unknown               Unknown  Unknown
xios_server.exe    0000000000653185  Unknown               Unknown  Unknown
xios_server.exe    000000000044EBB9  Unknown               Unknown  Unknown
xios_server.exe    0000000000CF7D96  Unknown               Unknown  Unknown
libc-2.17.so       00002B36B352F555  __libc_start_main     Unknown  Unknown
xios_server.exe    000000000044EACF  Unknown               Unknown  Unknown

What could be the possible reason for the model hanging?
I would appreciate any suggestions.

2

This forum is probably not the best way to reach OASIS community or at least an experienced audience in running coupled models with it. Actually NEMO does not even seem to be the cause of the deadlock or the crash of your runs.
I suggest you to send a message to OASIS mailing list (oasis_users@lists.enes.org) ou to post a message to their users forum on
Oasis Forum

3

I am not an XIOS specialist, but I had the same issue with a recent XIOS_2.5 revision (#2018) and found some work-around by commenting the MPI_Barrier and MPI_Comm_free instructions located immediately after the MPI_Intercomm_create instruction, in both client.cpp and server.cpp. XIOS developers are warned and a better solution hoped.

4

Thank you for the reply.
I did as suggested and now the coupled model freezes after

 starting wrf task            0  of            1
 ---> prism_initxios.x           0
-> info : CServer : Register new Context : nemo_server
-> info : Register new Context : nemo
-> info : update step : 0 timestep 300s
-> info : CClientBuffer: allocated 2 x 73907148 bytes for server 0 with a maximum of 16 buffered events

I checked that the model freezes when xios_close_context_definition() is called in SUBROUTINE iom_init in NEMOGCM.

According to

it seems that there is an issue with compiler.

Can you please tell me which compiler did you use?

5

We now compile both xios and nemo with intel-compilers-2021.4.0.
The compiling options were also changed to -O2 (instead of -Ofast) and march=native (instead of march=core-avx2).
This helped on our system.