# \[v4.2.x\] MPI error at open boundary **URL:** \ **Category:** v4.2.x\ **Tags:** BDY\ **Created:** [6 October 2021 15:00 UTC](https://nemo-ocean.discourse.group/t/v4-2-x-mpi-error-at-open-boundary/131 "2021-10-06T15:00:54Z")\ **Posts on this page:** 13\ **Page:** 1
**Author:** ![amoulin](https://avatars.discourse-cdn.com/v4/letter/a/aeb1de/32.png) [@amoulin](https://nemo-ocean.discourse.group/u/amoulin)\ **Post date:** [6 October 2021 15:00 UTC](https://nemo-ocean.discourse.group/t/v4-2-x-mpi-error-at-open-boundary/131/1 "2021-10-06T15:00:54Z")
Dear all, The actual CMEMS Mediterranean sea configuration is running with NEMO v3.6 and we are now testing the 4.2\_RC. Simulations with closed boundaries are running but when we open the Atlantic boundaries several problems are coming. We appreciate any comment that might help us. Here is a picture of the actual CMEMS Medsea configuration and the boundary options defined in the 3.6 version: ![MEDSEA_cfg](https://global.discourse-cdn.com/free1/uploads/nemo_ocean/original/1X/943cadaa83e02d0b7004230851ce66b99daa7990.jpeg) 1. We have a problem with domain decomposition. We tried several options for `jpni`/`jpnj` and the cases with `jpni>1` gave this error: ```nohighlight Abort(738313486) on node 71 (rank 71 in comm 0): Fatal error in PMPI_Recv: Message truncated, error stack: PMPI_Recv(173): MPI_Recv(buf=0x48cbb0d0, count=655, MPI_DOUBLE_PRECISION, src=69, tag=4, comm=0x84000006, status=0x1) failed (unknown)(): Message truncated ``` Example: `jpni=1` / `jpnj=72` / `nn_comm=1` / `ln_bdy=.true.` / OK `jpni=2` / `jpnj=36` / `nn_comm=1` / `ln_bdy=.false.` / OK `jpni=2` / `jpnj=36` / `nn_comm=1` / `ln_bdy=.true.` / `Fatal error in PMPI_Recv: Message truncated, error stack` We tried to define the boundary with segments (in case the _coordinate.bdy.nc_ was wrong) but the same error appears when the northern and/or southern Atlantic boundaries are opened. There is no problem with the western boundary. If it can help, this error does not appear with the trunk revision 14657. Would you have suggestions for the choice of the domain decomposition or any idea about this error? 1. If we follow the run using the decomposition with `jpni=1`, the model blow-up after a few times step. The model blows up only with the `orlanski_npo` option for `dyn3d`. The problem is not visible for `dyn3d='frs'` or `'orlanski'`. It seems an MPI issue at point 1 and jpj-1 (see picture with 48 processors). ![orlanski_npo_blowup](https://global.discourse-cdn.com/free1/uploads/nemo_ocean/original/1X/1105c98ea01b2368c8982babd716965ed1287722.png) We are using the same _coordinates.bdy_ and `nambdy_dta` files as for the CMEMS Medsea configuration with NEMO 3.6, could it be a problem? 1. Another point is about the Dardanelles strait. Could someone explain to us why the Orlanski option cannot be used anymore when boundaries are on the interior of the computational domain ( see the message in `bdyini`: `' Orlanski is not safe when the open boundaries are on the interior of the computational domain’`)? Aimie ---
**Author:** ![smasson](https://avatars.discourse-cdn.com/v4/letter/s/eb9ed0/32.png) [@smasson](https://nemo-ocean.discourse.group/u/smasson)\ **Post date:** [6 October 2021 15:47 UTC](https://nemo-ocean.discourse.group/t/v4-2-x-mpi-error-at-open-boundary/131/2 "2021-10-06T15:47:52Z")
Which value do you use for `nn_hls`? I am currently tracking a bug somewhere in BDY. I have it with some domain decomposition and `nn_hls = 2`. @jchanut also has the error with `nn_hls = 1` but for me it is ok… Regarding orlanski, as far as I am remembering, it is using diagonal points (like `(i+1,j+j)` or `(i-1,j-j)`) which are quite a nightmare to deal with in MPI and is not working if the MPI domain decomposition passes just along (or at 1 point of) your BDY. ---
**Author:** ![amoulin](https://avatars.discourse-cdn.com/v4/letter/a/aeb1de/32.png) [@amoulin](https://nemo-ocean.discourse.group/u/amoulin)\ **Post date:** [6 October 2021 15:54 UTC](https://nemo-ocean.discourse.group/t/v4-2-x-mpi-error-at-open-boundary/131/3 "2021-10-06T15:54:36Z")
I am using `nn_hls=1`. `nn_hls=2` does not change anything for the domain decomposition error and seems to solve the MPI error with `orlanski_npo` but gave quite different results at the boundary in comparison with NEMO v3.6. ---
**Author:** ![clem](https://avatars.discourse-cdn.com/v4/letter/c/6a8cbe/32.png) [@clem](https://nemo-ocean.discourse.group/u/clem)\ **Post date:** [6 October 2021 17:24 UTC](https://nemo-ocean.discourse.group/t/v4-2-x-mpi-error-at-open-boundary/131/4 "2021-10-06T17:24:51Z")
Do you use the very last revision of the trunk? I tried an intermediate revision (like 5 revisions back?) which crashed. ---
**Author:** ![amoulin](https://avatars.discourse-cdn.com/v4/letter/a/aeb1de/32.png) [@amoulin](https://nemo-ocean.discourse.group/u/amoulin)\ **Post date:** [7 October 2021 08:04 UTC](https://nemo-ocean.discourse.group/t/v4-2-x-mpi-error-at-open-boundary/131/5 "2021-10-07T08:04:52Z")
I have just updated the revision to the last one and I have the same issue. ---
**Author:** ![smasson](https://avatars.discourse-cdn.com/v4/letter/s/eb9ed0/32.png) [@smasson](https://nemo-ocean.discourse.group/u/smasson)\ **Post date:** [7 October 2021 15:13 UTC](https://nemo-ocean.discourse.group/t/v4-2-x-mpi-error-at-open-boundary/131/6 "2021-10-07T15:13:08Z")
Please try [r15345](https://forge.ipsl.jussieu.fr/nemo/changeset/15345), hopefully it should solve the problem at least for the non-orlanski BDY ---
**Author:** ![amoulin](https://avatars.discourse-cdn.com/v4/letter/a/aeb1de/32.png) [@amoulin](https://nemo-ocean.discourse.group/u/amoulin)\ **Post date:** [8 October 2021 08:52 UTC](https://nemo-ocean.discourse.group/t/v4-2-x-mpi-error-at-open-boundary/131/7 "2021-10-08T08:52:58Z")
Unfortunately, the MPI error is still here when I activate the `ln_bdy`. ---
**Author:** ![smasson](https://avatars.discourse-cdn.com/v4/letter/s/eb9ed0/32.png) [@smasson](https://nemo-ocean.discourse.group/u/smasson)\ **Post date:** [8 October 2021 16:05 UTC](https://nemo-ocean.discourse.group/t/v4-2-x-mpi-error-at-open-boundary/131/8 "2021-10-08T16:05:20Z")
yes, it is better but there is still potential bug when the BDY rim is located on a corner of an MPI subdomain. I am working on it… ---
**Author:** ![smasson](https://avatars.discourse-cdn.com/v4/letter/s/eb9ed0/32.png) [@smasson](https://nemo-ocean.discourse.group/u/smasson)\ **Post date:** [14 October 2021 08:30 UTC](https://nemo-ocean.discourse.group/t/v4-2-x-mpi-error-at-open-boundary/131/9 "2021-10-14T08:30:05Z")
please try [r15368](https://forge.ipsl.jussieu.fr/nemo/changeset/15368), it should work… ---
**Author:** ![amoulin](https://avatars.discourse-cdn.com/v4/letter/a/aeb1de/32.png) [@amoulin](https://nemo-ocean.discourse.group/u/amoulin)\ **Post date:** [14 October 2021 10:18 UTC](https://nemo-ocean.discourse.group/t/v4-2-x-mpi-error-at-open-boundary/131/10 "2021-10-14T10:18:07Z")
The [15368] revision solves the issue. I have no more MPI errors with bdy. Thanks a lot! ---
**Author:** ![Franziska](https://avatars.discourse-cdn.com/v4/letter/f/8797f3/32.png) [@Franziska](https://nemo-ocean.discourse.group/u/Franziska)\ **Post date:** [14 October 2021 10:35 UTC](https://nemo-ocean.discourse.group/t/v4-2-x-mpi-error-at-open-boundary/131/11 "2021-10-14T10:35:09Z")
I also had problems with different domain decompositions and was following this issue. I can confirm that the problem is solved with revision 15368. Thank you very much @smasson! ---
**Author:** ![smasson](https://avatars.discourse-cdn.com/v4/letter/s/eb9ed0/32.png) [@smasson](https://nemo-ocean.discourse.group/u/smasson)\ **Post date:** [14 October 2021 10:48 UTC](https://nemo-ocean.discourse.group/t/v4-2-x-mpi-error-at-open-boundary/131/12 "2021-10-14T10:48:52Z")
yes, it took me a while but I think (hope) I finally got it! 😅 ---
**Author:** ![afstyles](https://avatars.discourse-cdn.com/v4/letter/a/b4bc9f/32.png) [@afstyles](https://nemo-ocean.discourse.group/u/afstyles)\ **Post date:** [8 November 2021 10:11 UTC](https://nemo-ocean.discourse.group/t/v4-2-x-mpi-error-at-open-boundary/131/13 "2021-11-08T10:11:15Z")
I have a similar issue for a different domain in UKMO/NEMO\_4.0.4\_mirror @14075. When a corner is required for BDY I get the exact same problem as the OP. I have tried to introducing the fix in [15368] to r4.0.4 but I have had to substitute some variables to allow for differences between the versions: ```auto nn_hls = 1 Nis0 --> 1 + nn_hls Nie0 --> jpi - nn_hls Njs0 --> 1 + nn_hls Nje0 --> jpj - nn_hls ``` This implementation does seem to fix the corner issue with BDY, however I am concerned about possible issues I might have overlooked. **Any suggestions on how to properly apply the fix in [15368] to NEMO 4.0.4 would be greatly appreciated.**