Triple-nested configuration (v4.2.2) crashes with NaN values in child grid 2 at an early timestep when T-S restoring is enabled in the parent grid only. Crash occurs before parent has completed its first time step.
Configuration:
- Parent grid:
- ln_tsd_dmp = .true.
- ln_tradmp = .true. (uses resto.nc created by DMP_TOOLS)
- resto.nc timescale: tested both small and arbitrarily large timescales
- Child grids 1-3:
- No tracer restoring enabled
Error:
stp_ctl: |ssh| > 20 m or |U| > 10 m/s or S <= 0 or S >= 100 or NaN encounter in the tests
kt 14 |ssh| max NaN at i j 965 202 found in 0 MPI tasks, spread out among ranks 65 to -1
kt 14 |U| max NaN at i j k 549 285 1 found in 0 MPI tasks, spread out among ranks 65 to -1
kt 14 Sal min NaN at i j k 963 324 1 found in 0 MPI tasks, spread out among ranks 65 to -1
kt 14 Sal max NaN at i j k 208 114 1 found in 0 MPI tasks, spread out among ranks 65 to -1
NaN values occur throughout almost all variables in the abort file produced.
The model is being ramped from rest, so the initial values are all very small and show no signs of instability until the crash.
it : 1 |ssh|_max: 0.5000078499421523D+00 |U|_max: 0.5338898005352797D-04 S_min: 0.3499989810560336D+02 S_max: 0.3500001285309981D+02
it : 2 |ssh|_max: 0.5000164101501530D+00 |U|_max: 0.1219715804226504D-03 S_min: 0.3499979842825845D+02 S_max: 0.3500002559036318D+02
it : 3 |ssh|_max: 0.5000241919902291D+00 |U|_max: 0.2007283073258841D-03 S_min: 0.3499969810847775D+02 S_max: 0.3500003817566927D+02
it : 4 |ssh|_max: 0.5000307985849506D+00 |U|_max: 0.2781563096504587D-03 S_min: 0.3499959747797033D+02 S_max: 0.3500005055493870D+02
it : 5 |ssh|_max: 0.5000363839264022D+00 |U|_max: 0.3499295682801734D-03 S_min: 0.3499949986983838D+02 S_max: 0.3500006264493426D+02
it : 6 |ssh|_max: 0.5000410924357340D+00 |U|_max: 0.4193647019755068D-03 S_min: 0.3499940316930564D+02 S_max: 0.3500007432192935D+02
it : 7 |ssh|_max: 0.5000451649475746D+00 |U|_max: 0.4912666774895735D-03 S_min: 0.3499930643551589D+02 S_max: 0.3500008540301214D+02
it : 8 |ssh|_max: 0.5000480818224758D+00 |U|_max: 0.5673734445764815D-03 S_min: 0.3499921203204125D+02 S_max: 0.3500009562561567D+02
it : 9 |ssh|_max: 0.5000511145806703D+00 |U|_max: 0.6419888437460982D-03 S_min: 0.3499911523845655D+02 S_max: 0.3500010464247536D+02
it : 10 |ssh|_max: 0.5000515944586033D+00 |U|_max: 0.7111520307252764D-03 S_min: 0.3499902345408617D+02 S_max: 0.3500011199481795D+02
it : 11 |ssh|_max: 0.5000535615172085D+00 |U|_max: 0.7768700260074612D-03 S_min: 0.3499893177127268D+02 S_max: 0.3500011721277296D+02
it : 12 |ssh|_max: 0.5000592256553120D+00 |U|_max: 0.8429974057367995D-03 S_min: 0.3499884352021075D+02 S_max: 0.3500012026764173D+02
it : 13 |ssh|_max: 0.5001562815194808D+00 |U|_max: 0.9087450479251793D-03 S_min: 0.3499875426339223D+02 S_max: 0.3500012175010772D+02
it : 14 |ssh|_max: NaN |U|_max: NaN S_min: NaN S_max: NaN’
DMP_TOOLS namelist:
&nam_dmp_create
cp_cfg = ‘’
cp_cfz = ‘’
jp_cfg = 2
lzoom = .false.
ln_full_field = .true.
ln_med_red_seas = .false.
ln_old_31_lev_code = .false.
ln_coast = .false.
ln_zero_top_layer = .false.
ln_custom = .false
nn_hdmp = 0
pn_surf = 1e12 ! large to test near-zero coefficients
pn_bot = 1e12 ! large to test near-zero coefficients
pn_dep = 40
jperio = 0
/
&nam_zoom_dmp
lzoom_n = .false.
lzoom_e = .false.
lzoom_w = .false.
lzoom_s = .false.
/
Are there any known issues using AGRIF together with tracer restoring in this way? Can I use DMP_TOOLS as is, or do I need some special treatment for nested configurations?
Thank you in advance for any responses!
David