Start NEMO 5.0 simulation from NEMO 4.2.2 restarts (including AGRIF)

1

Hi,

I’d like to start a nested simulation in NEMO 5.0 (using RK3) based on NEMO 4.2.2 restarts. Although I haven’t tried, this should be fine without AGRIF but fails with AGRIF using dynspg_ts because of missing variables [uv]b2_i_b and [uv]n_adv. Is it safe to initialse these variables with zeros instead of reading the fields, similarly as done for other variables (e.g. sshbb_e etc.).

Changing lines 1143 ff in dynspg_ts.F90 from

#if defined key_agrif
            ! Read time integrated fluxes
            IF ( .NOT.Agrif_Root() ) THEN
               CALL iom_get( numror, jpdom_auto, 'ub2_i_b'  , ub2_i_b(:,:), cd_type = 'U', psgn = -1._wp )
               CALL iom_get( numror, jpdom_auto, 'vb2_i_b'  , vb2_i_b(:,:), cd_type = 'V', psgn = -1._wp )
            ELSE
               ub2_i_b(:,:) = 0._wp   ;   vb2_i_b(:,:) = 0._wp   ! used in the 1st update of agrif
            ENDIF
# if defined key_RK3
            CALL iom_get( numror, jpdom_auto, 'un_adv'      ,  un_adv(:,:), cd_type = 'U', psgn = -1._wp )
            CALL iom_get( numror, jpdom_auto, 'vn_adv'      ,  vn_adv(:,:), cd_type = 'V', psgn = -1._wp )
# endif
#endif

to


#if defined key_agrif
            ! Read time integrated fluxes
            IF ( .NOT.Agrif_Root() ) THEN
               IF( iom_varid( numror, 'ub2_i_b', ldstop = .FALSE. ) > 0 ) THEN 
                  CALL iom_get( numror, jpdom_auto, 'ub2_i_b'  , ub2_i_b(:,:), cd_type = 'U', psgn = -1._wp )
                  CALL iom_get( numror, jpdom_auto, 'vb2_i_b'  , vb2_i_b(:,:), cd_type = 'V', psgn = -1._wp )
               ELSE   
                  ub2_i_b(:,:) = 0._wp   ;   vb2_i_b(:,:) = 0._wp
               ENDIF  
            ELSE
               ub2_i_b(:,:) = 0._wp   ;   vb2_i_b(:,:) = 0._wp   ! used in the 1st update of agrif
            ENDIF
# if defined key_RK3
               IF( iom_varid( numror, 'un_adv', ldstop = .FALSE. ) > 0 ) THEN  
                  CALL iom_get( numror, jpdom_auto, 'un_adv'      ,  un_adv(:,:), cd_type = 'U', psgn = -1._wp )
                  CALL iom_get( numror, jpdom_auto, 'vn_adv'      ,  vn_adv(:,:), cd_type = 'V', psgn = -1._wp )
               ELSE      
                  un_adv(:,:) = 0._wp   ;   vn_adv(:,:) = 0._wp  
               ENDIF      
# endif
#endif

seems to work technically but is this a way to go or what am I “losing/missing” when doing so? Is there another way to start a nested NEMO 5.0 simulation using RK3 from NEMO 4.2.2 restarts?

Thank you!
Franziska

2

Hi Franziska,
This has been (hopefully) clarified in this related issue.
It should be ok starting from this commit.

Let me know,

Jérôme

3

Hi Jérôme,

Thank you for your response.

Unfortunately using the commit you shared doesn’t solve the issue but produces segmentation faults which seem to come from dynspg_ts.F90.

In my particular case here I start a configuration with a two-level nest from restarts of a configuration with only the first level nest and hot-start the secondary nest. In dynspg_ts.F90 of the commit you shared there is no “ELSE” part for IF( ln_rstart ) (line 1136) , which is present in tag 5.0 (but not in branch_5.0*) and I think this causes the segmentation fault. However, re-implementing this into your version brings me back to the initial issue of [uv]b2_i_b and [uv]n_adv not being present in the restarts I try to use.

* which brings me to the side-question: which version do I get when downloading the code following the instructions in the release notes git clone --branch 5.0 https://forge.nemo-ocean.eu/nemo/nemo.git nemo_5.0 ? Is this the tagged version which doesn’t change or a branch which evolves?

Best wishes
Franziska

4

which version do I get when downloading the code following the instructions in the release notes git clone --branch 5.0 https://forge.nemo-ocean.eu/nemo/nemo.git nemo_5.0

Doing this, you get the tag, not the branch that evolves and that will (well is supposed to) contain bugfixes (such as the one we are talking about).
Be sure to make your tests with the latest version of the branch.
We agree that you use RK3, right ? [uv]b2_i_b and [uv]n_adv should be useless in that case. Using MLF, and setting ln_1st_euler = .true. (on all grids for security), should also make these fields useless at initialization. But I may have missed something.

5

Why a ELSE would solve the problem ? It means you don’t have ln_rstart=T for all grids ?

6

Correct. I have ln_rstart=F for the inner-most grid and ln_init_chfrpar = T.

7

Good to know to get the tagged version! And I did the testing today in the commit you shared.
Right, I use RK3. If these variables are useless in this case, removeing the respective lines in the code I’m using otherwise should be fine? So I would not need to initialize the fields with zeros (as I suggested initially) but simply not read them?

8

ok. The mixture of ln_rstart=T/F among grids leaves me a bit uncertain of what it can do… It makes me think that I should add a check that it is consistent for all grids :slight_smile:

9

There has been two commits concerning the initialization with agrif. So be sure not to use just one commit, but the branch as it is today.

10

That’s what I did, I used the branch as of today.

11

Could you please give the location of the segmentation fault (with debug options) ?

12

The error message is:

forrtl: severe (174): SIGSEGV, segmentation fault occurred
Image              PC                Routine            Line        Source
libpthread-2.28.s  0000145B55637D10  Unknown               Unknown  Unknown
nemo               0000000000F5B6FE  lbclnk_mp_load_pt        2756  lbclnk.f90
nemo               0000000000F5994E  lbclnk_mp_sub_loo        2698  lbclnk.f90
nemo               0000000000F59420  lbclnk_mp_lbc_lnk        2353  lbclnk.f90
nemo               00000000007FDBCC  agrif_all_update_         241  agrif_all_update.f90
nemo               00000000007FC6AC  agrif_all_update_         179  agrif_all_update.f90
nemo               0000000000985C6D  agrif_util_mp_agr        1135  modutil.f90
nemo               0000000000425A2E  nemogcm_mp_nemo_g         274  nemogcm.f90
nemo               00000000004251C8  MAIN__                     44  nemo.f90
nemo               000000000042518D  Unknown               Unknown  Unknown
libc-2.28.so       0000145B54F077E5  __libc_start_main     Unknown  Unknown
nemo               00000000004250AE  Unknown               Unknown  Unknown

with line 241 in agrif_all_update.f90

      CALL lbc_lnk( 'finalize_lbc_for_agrif', ssh(:,:,  Kmm_a), 'T', 1._wp, &
           &                                  ssh(:,:,  Kbb_a), 'T', 1._wp, &
           &                                  uu_b(:,:, Kmm_a), 'U',-1._wp, &
           &                                  uu_b(:,:, Kbb_a), 'U',-1._wp, &
           &                                  vv_b(:,:, Kmm_a), 'V',-1._wp, &
           &                                  vv_b(:,:, Kbb_a), 'V',-1._wp, &
           &                                  ub2_i_b(:,:), 'U',-1._wp,     &
           &                                  vb2_i_b(:,:), 'V',-1._wp      )
13

Thanks. There’s indeed a bug. One should not perform mpp exchange for [uv]b2_i_b arrays with RK3, and not on the root grid for MLF (they are not allocated). The correct code from this line should be:

           CALL lbc_lnk( 'finalize_lbc_for_agrif', ssh(:,:, Kmm_a), 'T', 1._wp, &
           &                                       ssh(:,:, Kbb_a), 'T', 1._wp, &
           &                                      uu_b(:,:, Kmm_a), 'U',-1._wp, &
           &                                      uu_b(:,:, Kbb_a), 'U',-1._wp, &
           &                                      vv_b(:,:, Kmm_a), 'V',-1._wp, &
           &                                      vv_b(:,:, Kbb_a), 'V',-1._wp  )
# if ! defined key_RK3
           CALL lbc_lnk( 'finalize_lbc_for_agrif', ub2_b(:,:),   'U',-1._wp,    &
           &                                       un_bf(:,:),   'U',-1._wp,    &
           &                                       vb2_b(:,:),   'V',-1._wp,    &
           &                                       vn_bf(:,:),   'V',-1._wp.    )
           IF (.NOT.Agrif_root()) THEN
              CALL lbc_lnk( 'finalize_lbc_for_agrif', ub2_i_b(:,:), 'U',-1._wp,    &
              &                                       vb2_i_b(:,:), 'V',-1._wp     ) 
           END IF
#endif

I will correct this in branch_5.0 and let you know.
This said, if you pass in this code, it means that you have a zoom with cyclic boundaries, right ? Could you tell me more about your setup ?

14

Thank you Jérôme!
I actually don’tr have cyclic boundaries in the zooms*. The configuration is basically an update of INALT60 (respective paper) with eORCA025 as global parent grid, a primary nest at 1/20° resoultion (similar to INALT20 but extended southward) and a secondary nest at 1/100° around southern Africa.

* addition: the first nest, however crosses the east-west folding line

15

ok. Well, you’re in a special case (with a zoom crossing a parent east-west cyclic boundary), this is what I meant. This case does require extra mpp links on parent grid to feed ghost points there (agrif update is not done on cyclic ghost points). Good to know that this functionnality (not really fully tested, I admit) is usefull.
Talking about this, there may be a (small) issue in such a case with “interface separation” which is defined here. “Interface separation” implies agrif update from parent is not performed at the closest points to the dynamical boundary of the nest (see debreu et al 2012 for explanation). This tends to reduce artefacts near the interface as can be appreciated in the VORTEX test case. I’m not 100% sure, but it is likely that in your case, it also prevents from an update of parent near its west/east folding line which should not be the case. You can still disable this by setting #undef DECAL_FEEDBACK in place of #define DECAL_FEEDBACK in agrif_oce_update.F90. It should not be a real problem anyway. Solving these tiny details and, more generally, an assessment of agrif related numerical choices (interpolation/update schemes) is underway. But I will communicate on the forum about it.

16

I can confirm that correcting the code as suggested solves the initial issue. Thanks!

17

I will have a look into this and let you know.

18

ok. Commited here.

19

I didn’t dig deeper into the code but ran a short experiment with #undef DECAL_FEEDBACK in agrif_oce_update.F90 and compared the solutions to an otherwise identical run after a few days.
They only differ marginally everywhere - as expected. The plot here shows absolute values of SST difference from the two experiments on a logarithmic scale (to actually see something) for a small region around the North-Eastern corner of the nest. Black lines mark the nest boundaries, white line at 73°E the east-west cyclic boundary.

sst_parent_diff_day5_log
sst_parent_diff_day5_log819×707 117 KB

The nest boundaries show up with higher differences, while the cyclic boundary on the parent grid does not. Patterns are similar for velocities. From this I would conclude that the parent receives an update at its east-west cyclic boundary in both versions.

If you agree on this, which version would you then recommend to use? From what you say above, I assume #define DECAL_FEEDBACK (which I used before and was quite happy with the dynamics along the boundaries - in line with what you expect from that option).

20

Great, thank you Jérôme!