Hi,
I’m running an eORCA025 simulation and chose a decomposition (and tested several others) from the list provided using ln_listonly = .true. .
Although mpp_init tells:
mpp_init:
~~~~~~~~
==> you use the best mpi decomposition
some useless land subdomains are kept “in the lower left corner” of the grid (mpirank.png attached) although the area is land only; I double-checked the variable bottom_level (used to identify land in mppini.F90) and can confirm it is zero all over Antarctica.
Why is this the case? And how would I get rid of those?
The reason for which I’m worrying about this is that these useless subdomains appear as high values in “deviation from MPI average of the total elapse time spent on full_code” (timing_tsum_allmpi_full_code.png attached) provided by the new timing output in NEMO 5.0 (with which I’m not yet familiar).
Thanks!
Franziska
1 Like
First question :
NEMO and XIOS MPI decompositions are not the same. XIOS domain decomposition is always done only over j, to write full lines which is much more efficient. Each XIOS process therefore handles a certain number of NEMO lines. XIOS must have at least a few NEMO points to deal with in the lines it handles. It crashes if all the points have been removed (because of unnecessary sub-domains) in the lines it processes. To avoid this type of situation, we keep the 1st subdomain of each line of subdomains that would be completely removed.
To make it short, because of some XIOS constrain, me must keep at least 1 active subdomain per line of subdomains.
Second question :
This plot shows you the relative speed of each MPI domain in % of the mean value. You are right the process 0 (located at the bottom left) is almost twice slower than the mean value (> 90%). I suspect (hope) this difference explained by the initialization parts which mostly concern the proc 0. You should have 2 files timing_tsum_allmpi_*.nc, for example: timing_tsum_allmpi_t1_t100.nc et timing_tsum_allmpi_t7_t96.nc. The first one contains the total time spent on all the time steps, the second one exclude the firsts and the last time steps. This can make a real difference if you run a limited number of time steps. Note that because of the communications you more of less force all the processes to run at the same pace. You should not see large différences between the processus in this plot.
1 Like
Thank you Sebastien for your quick response and clarification on domain decomposition and restrictions from XIOS.
I can confirm that the initialisation part is responsible for the high values when taking into account the “full code”, however it is not always process 0 which takes most of the time (here I just picked timing from another year in the simulation). Nevertheless, only looking at the time stepping and further excluding the first and last couple of time steps reduces the load on these “mandatory for XIOS only” subdomains to below the rest of the ocean.
Thanks again!
Franziska
Thanks Franziska for posting this question (and Sébastien for the reply). I had encountered the same situation in a eORCA1 experiment, where a land subdomain was kept in the lower left corner of the grid even though I was using “the best mpi decomposition”. Now I finally understand why 
