[Users] Synchronization and MoL

[Ian Hinder]

On Tue, Sep 19, 2006 at 01:50:40PM -0500, Erik Schnetter wrote:
> On Sep 19, 2006, at 12:34:07, Ian Hinder wrote:
> 
> >Hi,
> >
> >When using MoL with Carpet, in which situations should I 'SYNC' my
> >grid functions?  In the MoL documentation, it says that MoL handles
> >all SYNCing across processors,
> 
> Does it?  I don't think so.  I don't see any synchronisation  
> happening in MoL.  I think the real statement is:
> 
> MoL does the correct thing, independent of whether you sync the state  
> vector or the RHS of the state vector.  But the user has to sync at  
> least one of these.

Line 455 of CactusBase/MoL/doc/documentation.tex:

  Points to note. The thorn routine {\tt Thorn\_CalcRHS} does not need
  to know and in fact should definitely not know where precisely in the
  MoL step it is. It just needs to know that it is receiving {\it some}
  intermediate data stored in the GFs ${\bf q}$ and that it should
  return the RHS ${\bf L}({\bf q})$. All the book-keeping to ensure that
  it is passed the correct intermediate state at that the GFs contain
  the correct data at the end of the MoL step will be dealt with by the
  MoL thorn and the flesh. Also the synchronization of grids across
  separate processors will be dealt with by the MoL thorn, the driver
  the flesh.

To me that suggests that I do not have to worry about SYNCing.
Looking at CactusExamples/WaveMoL/schedule.ccl I see:

schedule WaveMoL_CalcRHS in MoL_CalcRHS
{
  LANG: C
} "Register RHS calculation for MoL"

with no SYNC line.  Also, in CarpetExtra/WaveTolMoL/schedule.ccl:

SCHEDULE WaveToyMoL_CalcRHS IN MoL_CalcRHS
{
  LANG: Fortran
} "Calculate RHS for MoL"

Now, there are SYNC statements in MoL_PostStep for the boundary
condition routine, and these SYNC the evolved variables.  Is this the
reason that there is no SYNC after the calculation of the RHS
variables, as the evolved variables will be SYNCed later?

> You have two choices; you can either sync the state vector or its  
> RHS.  This is somewhat independent of where you apply the physical  
> (outer) boundary condition.  In our multi-block code, we apply the  
> boundary conditions to our RHS via penalties, and we also sync the RHS.
> 
> Using the boundary interface is another issue; the boundary interface  
> also applies symmetry and inter-patch boundary conditions.  We  
> therefore also use the boundary interface to apply boundary  
> conditions onto the RHS.
> 
> In principle, you can mix and match applying the triple  
> (synchronisation, boundary interface, physical boundaries) to either  
> the state vector or the RHS.  We chose to apply all to the RHS for  
> consistency reasons, but you can do otherwise.

Do you mean the whole triple can either be applied to the RHS or to
the state vector, or that I can choose either for each element of the
triple?

For example, can I set my interior RHSs in MoL_CalcRHS (with no SYNC
statement), apply my physical boundary condition to the RHS in
MoL_RHSBoundaries and synchronize the RHS variables with a SYNC line
there.  At this point all the RHSs are correct except in the symmetry
zones.  The MoL update then happens and updates the state vector.  I
then call the Boundary interface in MoL_PostStep on the state vector,
and all the symmetry boundary conditions are applied.  This seems
consistent; it is correct?

Another question: does 'SYNC' include prolongation?  Or is that
handled somewhere else entirely?

-- 
Ian Hinder
hinder at gravity.psu.edu
http://www.gravity.psu.edu/~hinder