[Users] Synchronization and MoL

[Erik Schnetter]

On Sep 19, 2006, at 14:24:35, Ian Hinder wrote:

> On Tue, Sep 19, 2006 at 01:50:40PM -0500, Erik Schnetter wrote:
>> On Sep 19, 2006, at 12:34:07, Ian Hinder wrote:
>>
>>> Hi,
>>>
>>> When using MoL with Carpet, in which situations should I 'SYNC' my
>>> grid functions?  In the MoL documentation, it says that MoL handles
>>> all SYNCing across processors,
>>
>> Does it?  I don't think so.  I don't see any synchronisation
>> happening in MoL.  I think the real statement is:
>>
>> MoL does the correct thing, independent of whether you sync the state
>> vector or the RHS of the state vector.  But the user has to sync at
>> least one of these.
>
> Line 455 of CactusBase/MoL/doc/documentation.tex:
>
>   Points to note. The thorn routine {\tt Thorn\_CalcRHS} does not need
>   to know and in fact should definitely not know where precisely in  
> the
>   MoL step it is. It just needs to know that it is receiving {\it  
> some}
>   intermediate data stored in the GFs ${\bf q}$ and that it should
>   return the RHS ${\bf L}({\bf q})$. All the book-keeping to ensure  
> that
>   it is passed the correct intermediate state at that the GFs contain
>   the correct data at the end of the MoL step will be dealt with by  
> the
>   MoL thorn and the flesh. Also the synchronization of grids across
>   separate processors will be dealt with by the MoL thorn, the driver
>   the flesh.

This statement can be easily misunderstood.  That is, it is wrong.

> To me that suggests that I do not have to worry about SYNCing.
> Looking at CactusExamples/WaveMoL/schedule.ccl I see:
>
> schedule WaveMoL_CalcRHS in MoL_CalcRHS
> {
>   LANG: C
> } "Register RHS calculation for MoL"
>
> with no SYNC line.  Also, in CarpetExtra/WaveTolMoL/schedule.ccl:
>
> SCHEDULE WaveToyMoL_CalcRHS IN MoL_CalcRHS
> {
>   LANG: Fortran
> } "Calculate RHS for MoL"
>
> Now, there are SYNC statements in MoL_PostStep for the boundary
> condition routine, and these SYNC the evolved variables.  Is this the
> reason that there is no SYNC after the calculation of the RHS
> variables, as the evolved variables will be SYNCed later?

You are correct.

>> You have two choices; you can either sync the state vector or its
>> RHS.  This is somewhat independent of where you apply the physical
>> (outer) boundary condition.  In our multi-block code, we apply the
>> boundary conditions to our RHS via penalties, and we also sync the  
>> RHS.
>>
>> Using the boundary interface is another issue; the boundary interface
>> also applies symmetry and inter-patch boundary conditions.  We
>> therefore also use the boundary interface to apply boundary
>> conditions onto the RHS.
>>
>> In principle, you can mix and match applying the triple
>> (synchronisation, boundary interface, physical boundaries) to either
>> the state vector or the RHS.  We chose to apply all to the RHS for
>> consistency reasons, but you can do otherwise.
>
> Do you mean the whole triple can either be applied to the RHS or to
> the state vector, or that I can choose either for each element of the
> triple?

I think you can mix and match, that is, you can choose either for  
each element of the triple independently.  Of course, this may not  
work for specific boundary conditions, e.g. when a boundary condition  
takes tangential derivatives on the boundary.

> For example, can I set my interior RHSs in MoL_CalcRHS (with no SYNC
> statement), apply my physical boundary condition to the RHS in
> MoL_RHSBoundaries and synchronize the RHS variables with a SYNC line
> there.  At this point all the RHSs are correct except in the symmetry
> zones.  The MoL update then happens and updates the state vector.  I
> then call the Boundary interface in MoL_PostStep on the state vector,
> and all the symmetry boundary conditions are applied.  This seems
> consistent; it is correct?

Yes, this seems correct.  However, there does not seem to be a reason  
for doing so -- you may as well apply the boundary conditions to the  
RHS, simplifying the logic and ensuring that  all RHS grid points are  
correct.  There is a group MoL_rhs_boundary or so for that effect.

> Another question: does 'SYNC' include prolongation?  Or is that
> handled somewhere else entirely?

It does.  However, MoL switches off prolongation at certain times, so  
that you need buffer zones.

-erik

-- 
Erik Schnetter <schnetter at cct.lsu.edu>

My email is as private as my paper mail.  I therefore support encrypting
and signing email messages.  Get my PGP key from www.keyserver.net.



-------------- next part --------------
A non-text attachment was scrubbed...
Name: PGP.sig
Type: application/pgp-signature
Size: 186 bytes
Desc: This is a digitally signed message part
Url : http://www.cactuscode.org/pipermail/users/attachments/20060919/de5f7f32/attachment.bin