Multi-Model Requirements and Preliminary Design Notes ----------------------------------------------------- Author: Tom Goodale Date: April 2002 Status: Feedback requested Introduction: ------------ There is a requirement, for certain problems, to solve different physical problems coupled just at boundaries, or on different overlapping grids, in the same Cactus run. Some example cases are 1) Aerodynamic calculations where one has a multi-block grid structure around an aircraft body or wing, perhaps with unstructured meshes covering the stores 2) Aeroelastic calculations which have one (or more) grid(s) doing aerodynamic calculations and another doing elasticity calculations. 3) Climate modelling calculations where there is, for example, ocean and atmospheric GCMs, ice models, etc. 4) Cauchy-Characteristic Matching where there is a cauchy code calculating the interior of a problem and a characteristic code doing the outer-region. Current Status: -------------- These can be done in Cactus 4.0 by use of Grid Arrays (GAs). Each model could work on a separate set of GAs, with a coordinate system associated with them so that CCTK_Interp could be used to interpolate between them whenever necessary. This works, but has several drawbacks 1) Must have a seperate Grid thorn for each model, rather than having several instantiations of the same one. Ditto for any other common application stuff. (IO shouldn't have a problem.) 2) Driver has no knowledge of how models are coupled, so can't do sensible load balancing. 3) Models may run at different wall-clock rates, so each needs its own physical time variable and must implement logic to couple models at appropriate times. [ASIDE: by physical time here I mean the time the simulation thinks has passed, which has nothing to do with actual wall-clock time.] There are almost certainly other drawbacks too, however it is perfectly do-able with the current infrastructure, albeit not as easily as we'd like. Planned Status: -------------- In 4.1 Cactus will be extended to allow full multi-model capabilities. The idea is that people should be able to develop a set of application thorns just as they currently do, for each model. Each model should be able to use GFs and standard physics infrastructure stuff such as base grid thorns, timestep thorns, etc. When each model has been independently tested, they should be able just change their parameter file to activate both models at the same time, just specifying how the models interact. Each model would then have its own private version of all info specific to itself. Thus: 1) Each model will have its own cGH containing information on its local grid. 2) Parameters, active thorns/implementations/etc will all be on a per-model basis - this means all CCTK_ functions must take a cGH as a parameter, not all do in 4.0 - this means a change to the parameter system so that DECLARE_CCTK_PARAMETERS picks up the current model's parameters 3) A thorn may be active in more than one model at once, with each model having its own set of Grid Variables - this means a change to the parameter file syntax 4) Interp and Reduce will be extended to allow selection of models involved in the calculation. 5) Drivers may be able to run different models on different sets of processors depending on how tightly couple the models are. Implementation Notes: -------------------- Goals: 1) People doing single model calculations should be minimally affected 2) Model developers should be able to develop and test models independently, and coupling into a multi-model situation should not require modifications to the thorns involved in any one model Notes: The following are just the current ideas of how some of the changes may look... 1) Parameter file syntax. For single-model runs there should be no change. For multi-model runs each new model's parameters will be in a block Model "" { ActiveThorns = "..." ... } Anything not in such a block will be implicitly assigned to the "Default" model. A multi-model aware driver could be activated at the default level, or a seperate activation could be done in each model. Each model could have a different driver. 2) Parameter implementation Currently parameters are stored in global structures which map to Fortran common blocks. However this has always been abstracted by the DECLARE_CCTK_PARAMETERS macro as thorns are not supposed to care how these things are implemented. In order to do multi-model stuff the only way to continue using common blocks would be to have an arbitrary maximum number of models and then have each parameter being an array, and then to pass the model-number into the routine. This would be efficient, but ugly, and, as stated, introduces an arbitrary maximum. A cleaner solution would be to mandate that DECLARE_CCTK_PARAMETERS should always come after any local routine declarations, and the have the macro expand to something like CCTK_REAL foo CCTK_INT bar CCTKi_MyThornPars(cctkGH,foo,bar) where the number of function calls and arguments would be some balance between the speed of a function call and how good the compiler does with large numbers of arguments. It could be one call per parameter or one call for all parameters, as extreme cases. Note that this is slightly cleaner than now, as any accidental changes a thorn might make to a parameter would not be propagated. We would need to have the cctkGH passed to all routines wanting to use CCTK_DECLARE_PARAMETERS, which is not currently the case.