Hi,
I am trying to upgrade from using XIOS2 to XIOS3 and initially just trying to get (a minor variation on) the X3_ORCA2_ICE_PISCES demonstrator configuration working.
I have successfully compiled XIOS3-trunk (revision: 2825) and a custom configuration based on X3_ORCA2_ICE_PISCES in NEMO branch_5.0.1 (#6f645ba8). The only change is that I do not includePISCES and remove the corresponding *pisces.xml files and references to them in context_nemo.xml. I usually run a custom ORCA2_ICE configuration in this way without any issues (with XIOS2) so presumably that doesn’t matter here. If relevant, our system has OpenMPI v4.1.1 and GCC v8.5.0.
Running the job with XIOS in detached mode on 12 processors and NEMO on 32 seems to correctly initialise: I get the usual init. outputs in ocean.output and various xios_server_* outputs, but the simulation fails with “signal 11 (Segmentation fault)”. I get the same error changing to attached mode. I also get the same behaviour trying all this with NEMO main branch and branch_5.0. I have tried also a simpler output setup with just ice variables, with NEMO on 8 processors and 3 XIOS servers for the 2 iwriter and 1 igatherer—no difference.
I tried setting the info_level to 100 in iodef.xml, which doesn’t seem to give much insight beyond confirming that the XIOS servers are initialised.
I tried recompiling with debugging flags and this gave a bit more insight from the back trace: the segmentation fault occurs after the following series of subroutine calls:
nemo_gcm → stp_MLF → iom_init_closedef → xios_close_context_definition
So, it looks like the segmentation fault is arising from within XIOS but seems unlikely to be a bug there or in NEMO since these are recent versions of each and clearly others are using XIOS3 without problems.
Please does anyone have any suggestions as to what might be causing this? I’m at a loss so any ideas would be welcome!