Cactus Communication Infrastructure
-----------------------------------

Author: Tom Goodale
Date:   April 2000

Why
---

Currently the Cactus Flesh has no communication layer of its own.  All 
communication is dealt with by the driver thorns - PUGH, PAGH, ...

This of course localises things nicely, and since the driver is the 
only thing which knows the actual layout of the grid across processors, 
it makes sense. 

On the other hand there are some things outside of the driver which need 
communication - e.g. IO and the IsoSurfacer.  We could insist that every 
driver provides things like GetSlice, GetVolume, GetLine which gets the 
appropriate data, and for IO perhaps this may be the best way to go.  
Alternatively we could provide some sort of Cactus Communication 
Infrastructure (CCI) which would be a set of functions the flesh guarantees 
to be there (either by overloading or registration) but the functionality of 
which is provided by Thorns.

This way people can write a driver thorn, like PUGH, which would be 
independent of the parallel mechanism, e.g.PVM, MPI, Threads, CORBA, shared 
memory, etc, and then other thorns can provide the actual parallel 
infrastructure, which could in fact change from machine to machine, and even
from time to time depending on machine loads, etc.

Basically this splits the driver and communication layers, and allows thorns 
some measure of access to parallelism in a flexible way ('Though they should 
still only use this directly if they are declared 'global' in the scheduler, 
otherwise they _will_ break with AMR.)

What
----

Ok, so what functionality do we need in the CCI ?

We could just duplicate the functions available in another layer, such as 
PVM or MPI, effectively writing these over some other layer, or we could
divorce ourselves from the current implementations and standards, and come 
up with the minimum we need, and add functionality as and when we need it.

So, as I see it, we need, roughly

1) Basic stuff: Start, abort, processor numbers
2) Blocking and non-blocking sends
3) Barrier
4) Scatter, Gather, Reduce

Do we need communication groups ?

We probably want more than this in the long run, but in the short term the
above is enough to get us going.

1)

I guess as a minimum we need.

CCI_Init
CCI_MyTask
CCI_NTasks
CCI_Abort
CCI_End

2)

Do we want send and receive buffers ?  Or do we want to just tell the CCI 
to send so many bytes of data, with some particular stride, to some other 
process ?  We probably want to tell it the type of data to allow it to do
byte-swapping during the transfer - should we allow user-defined types too ?

PVM packs data into a message buffer, and then a send is sent. The data can 
either be copied into a working buffer, or sent from where it is in memory.

MPI sends data directly from where it is in memory.  It does allow user 
derived types which allow the data areas to be non-contiguous.

Other things 
 - message tags
 - probe function to ask if a message has arrived 
 - ...

3)

CCI_Barrier

Do we need anything else ?


4)

CCI_Broadcast
CCI_Gather
CCI_Scatter
CCI_Reduce


How
---

This can be implemented as a thorn to begin with, and once we are really
happy with it, it can move into the flesh if we think it is necessary (there 
are advantages, but it may lead to bloat, which should be avoided 8-).