Sync CUDA streams before MPI calls

I don't claim to fully understand why an explicit stream synchronization
is required before a MPI call (maybe because MPI calls are executed in
their own stream?), but starting from CUDA 5.7, `interpolate` fails if
this is not done (under certain conditions, unsure which exactly).

Adding a synchronization was also advice I got when debugging a similar
bug a few months ago.

Author: Sbozzolo

NEWS.md

@@ -4,6 +4,14 @@ ClimaCore.jl Release Notes
 main
 -------
 
+
+v0.14.29
+-------
+
+ - Fix empty interpolation with certain versions of CUDA+MPI
+   [2256](https://github.com/CliMA/ClimaCore.jl/pull/2256).
+
+
 v0.14.28
 -------
 

Project.toml

@@ -1,7 +1,7 @@
 name = "ClimaCore"
 uuid = "d414da3d-4745-48bb-8d80-42e94e092884"
 authors = ["CliMA Contributors <clima-software@caltech.edu>"]
-version = "0.14.28"
+version = "0.14.29"
 
 [deps]
 Adapt = "79e6a3ab-5dfb-504d-930d-738a2a938a0e"

src/Remapping/Remapping.jl

@@ -15,6 +15,7 @@ import ..DataLayouts,
     ..Fields,
     ..Hypsography
 import ClimaCore.Utilities: half
+import ClimaCore.Spaces: cuda_synchronize
 
 using ..RecursiveApply
 

src/Remapping/distributed_remapping.jl

@@ -751,6 +751,7 @@ function _collect_interpolated_values!(
     index_field_end::Int;
     only_one_field,
 )
+    cuda_synchronize(ClimaComms.device(remapper.comms_ctx))   # Sync streams before MPI calls
     if only_one_field
         ClimaComms.reduce!(
             remapper.comms_ctx,