We recently upgraded our simulation from NEMO 5.0-beta to NEMO 5.0. Without making any other changes, we noticed a significant increase in memory usage with version 5.0 compared to 5.0-beta.
For reference, here’s a comparison of memory consumption:
NEMO 5.0 (left) vs. NEMO 5.0-beta (right)
Each peak corresponds to a simulation day. While NEMO 5.0-beta never exceeded 3.75 GB, NEMO 5.0 reaches 15 GB of RAM under the same conditions.
We are using XIOS2/trunk, RK3, and MPICH (4 parallel processes in this example).
Which configuration did you use?
Do you get similar results when you switch off xios by removing the key_xios at the compilation stage or by setting all enabled= to enabled=".false." in file_def-nemo*xml` files?
Which too did you use to get NEMO memory use?
Dear Sebastien, thank you for your reply and sorry for the wait.
Indeed, after de-activating XIOS as you explained and leaving all the rest untouched, the RAM peaks are gone and the memory profile is similar to the one we had in nemo 5.0-beta! So it seems to be an issue related to XIOS. Is it possible that in the transition between nemo 5.0-beta and 5.0-stable something happened that caused this problem?
Regarding your questions:
- We are running NEMO inside a Docker container, so for the RAM profiling we are using the built-in tools from Docker.
- We use a custom configuration based on “ORCA2_ICE_PISCES”. We just remove the key_top and key_si3 (and of course the key_xios for this test). We tried with and without RK3, it seems to have in impact on the RAM
I have no real/good explanation for the changes you observe between nemo 5.0-beta and nemo 5.0-stable… I can propose :
Do you have exactly the same outputs volume (variables, output frequency) and the same use of XIOS (attached/detached mode, one_file/multiple _file, mapping of the XIOS processes among the NEMO processes if you are in detached mode)?
I made changes in XIOS interface between nemo 5.0-beta and nemo 5.0-stable but, “of course”, they should be transparents…
I recently (April 8th) fixed these changes, again this should be transparent… but could you test NEMO version cdc14dd3 or the head of the branch_5.0?
Could you try XIOS3/trunk instead of XIOS2/trunk?
XIOS3 is much more efficient and uses much less memory than XIOS2.
XIO3 compiles in the same way than XIOS2.
To use XIOS3 in NEMO, you must
(1) compile NEMO with the key_xios3, in addition to the key_xios.
(2) in iodef.xml, you must replace oasis_codes_id by clients_codes_id (or simply remove this line as you are not using oasis!).
ps: Happy to know RK3 has no impact on the problem!
