MPI error: `'xios::CNetCdfException' ... NetCDF: Parallel operation on file opened for non-parallel access`

Dear all,

Hello, I met some problems when running ORCA2_ICE_PISCES.
When I try to use MPI I receive the following message (details are below):

"terminate called after throwing an instance of 'xios::CNetCdfException'
  what():  Error when calling function: nc_create_par(fileName.c_str(), cMode, comm, info, &ncId)
NetCDF: Parallel operation on file opened for non-parallel access" 

Does anyone know how to fix it?

There was no E R R O R flag in ocean.output and time.step was 76.

My arch file

%NCDF_HOME           /usr/lib/x86_64-linux-gnu
%HDF5_HOME           /usr/lib/x86_64-linux-gnu/hdf5/serial
%XIOS_HOME           /media/cmlws/Data1/hjc/xios-2.5
%OASIS_HOME          /not/defined

#%NCDF_INC            -I%NCDF_HOME/include -I%HDF5_HOME/include
%NCDF_INC            -I%NCDF_HOME -I%HDF5_HOME/include
%NCDF_LIB            -L%NCDF_HOME -lnetcdff -lnetcdf -lstdc++
%XIOS_INC            -I%XIOS_HOME/inc
%XIOS_LIB            -L%XIOS_HOME/lib -lxios -L/usr/lib/gcc/x86_64-linux-gnu/9 -lstdc++

%OASIS_INC           -I%OASIS_HOME/build/lib/mct -I%OASIS_HOME/build/lib/psmile.MPI1
%OASIS_LIB           -L%OASIS_HOME/lib -lpsmile.MPI1 -lmct -lmpeu -lscrip

%CPP                 /usr/bin/cpp-9 -Dkey_nosignedzero
%CPPFLAGS            -P -traditional

%FC                  /usr/bin/mpif90
%FCFLAGS             -fdefault-real-8 -funroll-all-loops -cpp -fcray-pointer -ffree-line-length-none -g -O0 -fbacktrace -fbounds-check
%FFLAGS              %FCFLAGS
#%LD                  /usr/bin/mpif90 -Wl,-rpath=$HOME/INSTALL/lib:/usr/lib
%LD                  /usr/bin/mpif90
%LDFLAGS             -L/usr/lib/x86_64-linux-gnu
%FPPFLAGS            -P -C -traditional
%AR                  ar
%ARFLAGS             -rs
%MK                  make
%USER_INC            %XIOS_INC %OASIS_INC %NCDF_INC
%USER_LIB            %XIOS_LIB %OASIS_LIB %NCDF_LIB

%CC                  gcc
%CFLAGS              -O0 -fbacktrace

Summarized mpirun log :

Fortran runtime warning: Namelist object 'clname' truncated on read.
At line 103 of file /media/cmlws/Data1/hjc/NEMO/r4.0.6/cfgs/ORCA_testing3/BLD/ppsrc/nemo/tideini.f90 (unit = 16, file = 'namelist_ref')

Backtrace for this error:
terminate called after throwing an instance of 'xios::CNetCdfException'
  what():  Error when calling function: nc_create_par(fileName.c_str(), cMode, comm, info, &ncId)
NetCDF: Parallel operation on file opened for non-parallel access
Unable to create file on parallel file system, given its name:
and its creation mode In the composed creation mode

Program received signal SIGABRT: Process abort signal.

#47  0x55dccd27cb17 in __sbcssm_MOD_sbc_ssm
        at /media/cmlws/Data1/hjc/NEMO/r4.0.6/cfgs/ORCA_testing3/BLD/ppsrc/nemo/sbcssm.f90:218
#47  0x5629c49ceb17 in __sbcssm_MOD_sbc_ssm
        at /media/cmlws/Data1/hjc/NEMO/r4.0.6/cfgs/ORCA_testing3/BLD/ppsrc/nemo/sbcssm.f90:218
#46  0x55cdb1a5c001 in __iom_MOD_iom_p2d
        at /media/cmlws/Data1/hjc/NEMO/r4.0.6/cfgs/ORCA_testing3/BLD/ppsrc/nemo/iom.f90:1693
#47  0x55cdb0e4bb17 in __sbcssm_MOD_sbc_ssm
        at /media/cmlws/Data1/hjc/NEMO/r4.0.6/cfgs/ORCA_testing3/BLD/ppsrc/nemo/sbcssm.f90:218
#48  0x5629c497f679 in __sbcmod_MOD_sbc
        at /media/cmlws/Data1/hjc/NEMO/r4.0.6/cfgs/ORCA_testing3/BLD/ppsrc/nemo/sbcmod.f90:452
#49  0x5629c49ffc02 in __step_MOD_stp
        at /media/cmlws/Data1/hjc/NEMO/r4.0.6/cfgs/ORCA_testing3/BLD/ppsrc/nemo/step.f90:138
#50  0x5629c495cb01 in __nemogcm_MOD_nemo_gcm
        at /media/cmlws/Data1/hjc/NEMO/r4.0.6/cfgs/ORCA_testing3/BLD/ppsrc/nemo/nemogcm.f90:192
#51  0x5629c4957f11 in nemo
        at /media/cmlws/Data1/hjc/NEMO/r4.0.6/cfgs/ORCA_testing3/WORK/nemo.f90:18
#52  0x5629c4957f4a in main
        at /media/cmlws/Data1/hjc/NEMO/r4.0.6/cfgs/ORCA_testing3/WORK/nemo.f90:11

1693 line in iom.f90: CALL xios_send_field(cdname, pfield2d)

   SUBROUTINE iom_p2d( cdname, pfield2d )
      CHARACTER(LEN=*)            , INTENT(in) ::   cdname
      REAL(wp),     DIMENSION(:,:), INTENT(in) ::   pfield2d
      CALL xios_send_field(cdname, pfield2d)
   END SUBROUTINE iom_p2d

the last part of ocean.output:

fld_read: var dust kt =       76 (   4.7188 days), Y/M/D = 0001/01/05, records b/a:   0012/  0001 (days  -15.5000/  15.5000)
       it_offset is :            0
   fld_read: var riverdic kt =       76 (   4.7188 days), Y/M/D = 0001/01/05, records b/a:   0001/  0002 (days    2.5000/   7.5000)
       it_offset is :            0
   fld_read: var riverdoc kt =       76 (   4.7188 days), Y/M/D = 0001/01/05, records b/a:   0001/  0002 (days    2.5000/   7.5000)
       it_offset is :            0
   fld_read: var riverdin kt =       76 (   4.7188 days), Y/M/D = 0001/01/05, records b/a:   0001/  0002 (days    2.5000/   7.5000)
       it_offset is :            0
   fld_read: var riverdon kt =       76 (   4.7188 days), Y/M/D = 0001/01/05, records b/a:   0001/  0002 (days    2.5000/   7.5000)

Thank you in advance.

Best regards,

Hwa-Jin Choi

Hi Hwa-Jin, The clue is probably this:

%HDF5_HOME          /usr/lib/x86_64-linux-gnu/hdf5/serial

You will need to use an HDF5/NetCDF4 combination that has been compiled with parallel IO support if you want to use the full functionality of XIOS.

The details should be under:
https://forge.ipsl.jussieu.fr/nemo/chrome/site/doc/NEMO/guide/html/install.html

From memory, it means making sure HDF5 was built with --enable-parallel.

Thank you very much for your answer.
I recompiled the HDF5/NetCDF4 with MPI.
Successfully done!!! :heart_eyes:
Thank you again for your help.

Best regards,

Hwa-Jin Choi