The MFE (run with 3 processes)
from firedrake import *
mesh = UnitSquareMesh(1, 1)
vom = VertexOnlyMesh(mesh, [[0.7, 0.7]])
print(f"{mesh.comm.rank}: {vom.cell_set.size = }")
R = FunctionSpace(vom, "R", 0)
f = Function(R)
f.interpolate(666)
print(f"{mesh.comm.rank}: {f.dat.data_ro = }")prints
0: vom.cell_set.size = np.int64(0)
1: vom.cell_set.size = np.int64(1)
2: vom.cell_set.size = np.int64(0)
0: f.dat.data_ro = array([0.])
1: f.dat.data_ro = array([666.])
2: f.dat.data_ro = array([0.])
When f.dat.data_ro should report 666 on all ranks.
This issue is occurring because the op2.Global underlying f is only getting touched on the single rank that has an owned point. Other ranks have zero sized parloops and so the value never gets set.
To corroborate this, the problem goes away if one does f.interpolate(666, access=op2.INC). This is because at the end of the parloop we do a reduction, so contributions from all ranks are included.
The MFE (run with 3 processes)
prints
When
f.dat.data_roshould report 666 on all ranks.This issue is occurring because the
op2.Globalunderlyingfis only getting touched on the single rank that has an owned point. Other ranks have zero sized parloops and so the value never gets set.To corroborate this, the problem goes away if one does
f.interpolate(666, access=op2.INC). This is because at the end of the parloop we do a reduction, so contributions from all ranks are included.