Generic Hyperslab API for CCTK arrays ------------------------------------- Author: Gabrielle Allen, Tom Goodale, Thomas Radke comments & suggestions by Erik Schnetter and Jonathan Thornberg Date: October 2001 Version: $Id: Hyperslab.txt,v 1.8 2003/01/30 12:31:08 tradke Exp $ Initial version: http://www.cactuscode.org/Development/Specs/Hyperslab_v1.txt Final Note: =========== The complete specification of the CCTK Hyperslab API as described in this final draft is now documented in the ThornGuide chapter of thorn PUGHSlab in the CactusPUGH arrangement. Please refer to this document for any details about the API specification and its current implementation by thorn PUGHSlab. =========================================================================== Why --- Many I/O thorns output data from distributed CCTK array variables. If output is done by only one processor, it needs to collect the data from other processors. This ties the I/O thorn to the driver since it needs to know about data distribution, ghostzones, etc. A clean way of separating the I/O code from the driver is to use a thorn which provides a generic interface to extract the distributed data and return it to the I/O processor. This interface can also provide more features such as downsampling, datatype conversions, or hyperslab selections. A hyperslab is just a subset of a global CCTK array, with its own dimension, origin, direction, and extents. Specification ------------- In the following, a generic hyperslab API is specified. In general, a hyperslab get/put operation is done in a 3-level scheme: 1) At first a hyperslab mapping is calculated by Hyperslab_Define*MappingBy*(). There are routines for defining a mapping by either physical coordinates or by index points. There are also routines which define a processor- local or a global hyperslab resp. 2) With such a mapping, hyperslabs can be extracted/distributed by one or more calls to Hyperslab_Get*()/Hyperslab_Put*(). There are routines for getting/putting a hyperslab from/to a single variable or from/to a list of variables. 3) Once all hyperslabs are done, the hyperslab mapping is freed by a call to Hyperslab_FreeMapping(). If Hyperslab_Get*()/Hyperslab_Put*() gets passed a mapping for a global hyperslab, a global, synchronuous operation will be performed (ie. it must be called in sync by every processor). All input arguments must be consistent between processors. All routine parameters are given as CCTK datatypes so that the hyperslab API can be defined by CCTK function aliases. Different hyperslab thorns can then provide functions with the same names, and other thorns using these can be independent of the actual active hyperslab thorn. ============================================================================== Prototypes for routines to create a hyperslab mapping: ------------------------------------------------------ An M-dimensional hyperslab subspace mapped into an N-dimensional space can be specified either by coordinates on the physical grid or by index points on the underlying computational grid. CCTK_INT Hyperslab_DefineGlobalMappingByIndex ( const cGH *GH, CCTK_INT vindex, CCTK_INT hdim, const CCTK_INT *direction /* vdim*hdim */, const CCTK_INT *origin /* vdim */, const CCTK_INT *extent /* hdim */, const CCTK_INT *downsample /* hdim */, CCTK_INT table_handle, t_hslabConversionFn conversion_fn, CCTK_INT *hsize /* hdim */); CCTK_INT Hyperslab_DefineGlobalMappingByPhys ( const cGH *GH, CCTK_INT vindex, CCTK_INT hdim, const char *coord_system_name, const CCTK_INT *direction /* vdim*hdim */, const CCTK_REAL *origin /* vdim */, const CCTK_REAL *extent /* hdim */, const CCTK_INT *downsample /* hdim */, CCTK_INT table_handle, t_hslabConversionFn conversion_fn, CCTK_INT *hsize /* hdim */); CCTK_INT Hyperslab_DefineLocalMappingByIndex ( const cGH *GH, CCTK_INT vindex, CCTK_INT hdim, const CCTK_INT *direction /* vdim*hdim */, const CCTK_INT *origin /* vdim */, const CCTK_INT *extent /* hdim */, const CCTK_INT *downsample /* hdim */, CCTK_INT table_handle, t_hslabConversionFn conversion_fn, CCTK_INT *hsize_local, /* hdim */ CCTK_INT *hsize_global, /* hdim */ CCTK_INT *hoffset_global /* hdim */); CCTK_INT Hyperslab_DefineLocalMappingByPhys ( const cGH *GH, CCTK_INT vindex, CCTK_INT hdim, const char *coord_system_name, const CCTK_INT *direction /* vdim*hdim */, const CCTK_REAL *origin /* vdim */, const CCTK_REAL *extent /* hdim */, const CCTK_INT *downsample /* hdim */, CCTK_INT table_handle, t_hslabConversionFn conversion_fn, CCTK_INT *hsize_local, /* hdim */ CCTK_INT *hsize_global, /* hdim */ CCTK_INT *hoffset_global /* hdim */); * Parameters: - const cGH *GH pointer to the CCTK grid hierarchy. - CCTK_INT vindex the index of a CCTK variable which will be used as a template in the following hyperslab get/put operation to denote the arrays' domain decomposition (dimensionality and distribution over processors). (must match the domain decomposition of all input CCTK variables given by the argument in subsequent calls to Hyperslab_Get*()/Hyperslab_Put*()) - CCTK_INT hdim the dimension of the hyperslab to get/put. (must be a positive integer and less or equal to the dimension of ) - const CCTK_INT *direction const CCTK_INT *origin const CCTK_INT *extent const CCTK_INT *downsample -------------------------- const char *coord_system_name const CCTK_INT *direction const CCTK_REAL *origin const CCTK_REAL *extent const CCTK_INT *downsample arguments describing the mapping of the hyperslab to get/put. The hyperslab location is identified by its origin (lower left corner), the direction vectors starting from the origin and spanning the hyperslab in the N-dimensional space, and its extents (size of the hyperslab in every direction). There are direction vectors (one for every hyperslab axis) with elements. The direction vectors are given in index points and must not be a linear combination of each other. The dimension array has to be passed as "direction[vdim_index + hdim_index*vdim]", so is the fastest changing dimension. The origin and extent can be given in either physical coordinates or grid points - for the first case a coordinate system needs to be given to do the mapping onto the underlying grid. The downsampling parameter denotes the downsampling factors for the hyperslab to be extracted. They are given in terms of grid points in every hyperslab direction. The downsampling parameter is optional - if NULL is passed here, no downsampling will be applied. - CCTK_INT table_handle a table handle might be passed in to provide additional information to the hyperslab get/put routines about the hyperslab mapping. For example, there could be a tolerance parameter for hyperslabs not rectangular to the underlying grid. For grid points which offset from the direction vectors, the tolerance would then specify a (plus/minus) offset for the directions saying which points should still be included in the hyperslab space. Another example could be whether to do interpolation between grid points or not. Passing a table handle is optional, an invalid (negative) table handle denotes no additional table information. Remark: thorn PUGHSlab will not evaluate any table information. - t_hslabConversionFn conversion_fn reference to a user-defined datatype conversion function. If this reference is given (not NULL) the Hyperslab_Get*() routine will use it in favour of any implicite datatype conversion routine, if conversion is requested. - CCTK_INT *hsize CCTK_INT *hsize_local reference to a size array with elements to be set by the Hyperslab_Define*MappingBy*() routines. The resulting size of the hyperslab to be extracted is set according to the hyperslab extents and downsampling parameters chosen. With this information, one can allocate memory of appropriate size and pass as user-provided hyperslab data buffers into subsequent calls to Hyperslab_Get*(). - CCTK_INT *hsize_global reference to a size array with elements to be set by the Hyperslab_DefineLocalMappingBy*() routine. This array holds the sizes of the corresponding global hyperslab. It is set according to the local hyperslab extents and downsampling parameters chosen. ( can be passed as a NULL pointer if no information about the global hyperslab size is needed.) - CCTK_INT *hoffset_global reference to an offset array with elements to be set by the Hyperslab_DefineLocalMappingBy*() routine. This array holds the offsets of the local hyperslab into the corresponding global hyperslab. It is set according to the local hyperslab extents and downsampling parameters chosen. ( can be passed as a NULL pointer if no information about a hyperslab offsets is needed.) * Return Code: (according to the Cactus Coding Conventions) 0 for success, negative for some error condition (to be defined by the thorn implementing the Hyperslab_Define*MappingBy*() routines) Typedef describing the prototype of a datatype conversion function: ------------------------------------------------------------------- typedef void (*t_hslabConversionFn) (CCTK_INT nelems, CCTK_INT stride, CCTK_INT from_type, CCTK_INT to_type, const void *from, void *to); Users can provide their own datatype conversion function and pass a reference to it to the Hyperslab_Get*() routines. A datatype conversion function gets passed the number of elements to convert (), and a stride between adjacent elements (), a pointer to the data to convert (), a pointer to the conversion target buffer (), and the corresponding CCTK data types for the data (). Prototypes of generic hyperslab get/put routines ------------------------------------------------ Every set of hyperslab get/put routines consists of two functions: one which operates on a single hyperslab, and another which gets/puts hyperslabs for a list of variables. Depending on the actual hyperslab implementation it might be more efficient to use the list functions rather than doing a single get/put several times in a sequence, if the hyperslab parameters are identical for all variables. CCTK_INT Hyperslab_Get (const cGH *GH, CCTK_INT mapping_handle, CCTK_INT proc, const CCTK_INT vindex, const CCTK_INT timelevel, const CCTK_INT hdatatype, void *hdata); CCTK_INT Hyperslab_GetList (const cGH *GH, CCTK_INT mapping_handle, CCTK_INT num_arrays, const CCTK_INT *procs /* num_arrays */, const CCTK_INT *vindices /* num_arrays */, const CCTK_INT *timelevels /* num_arrays */, const CCTK_INT *hdatatypes /* num_arrays */, void *const *hdata /* num_arrays */, CCTK_INT *retvals /* num_arrays */); CCTK_INT Hyperslab_Put (const cGH *GH, CCTK_INT mapping_handle, CCTK_INT proc, const CCTK_INT vindex, const CCTK_INT timelevel, const CCTK_INT hdatatype, const void *hdata); CCTK_INT Hyperslab_PutList (const cGH *GH, CCTK_INT mapping_handle, CCTK_INT num_arrays, const CCTK_INT *procs /* num_arrays */, const CCTK_INT *vindices /* num_arrays */, const CCTK_INT *timelevels /* num_arrays */, const CCTK_INT *hdatatypes /* num_arrays */, const void *const *hdata /* num_arrays */, CCTK_INT *retvals /* num_arrays */); * Parameters: - const cGH *GH pointer to the CCTK grid hierarchy. * Parameters: - const cGH *GH pointer to the CCTK grid hierarchy. - CCTK_INT mapping_handle handle for the hyperslab mapping as returned by a previous call to Hyperslab_Define*MappingBy*(). - CCTK_INT num_arrays total number of input arrays to get/put a hyperslab from/to. (must be a positive integer and match the number of array elements in the arguments following) - CCTK_INT proc const CCTK_INT *procs (list of) processors which will receive/provide the hyperslab(s) in a subsequent call to Hyperslab_Get*() or Hyperslab_Put*(). For Hyperslab_Get*(), there may be either exactly one processor providing the hyperslab data (then its processor ID must be given here), or all all processors will get the extracted hyperslab data (a negative processor ID is given, or procs[] is passed as a NULL pointer). For Hyperslab_Put*(), there may only be one processor providing the hyperslab data to be distributed to all others. - CCTK_INT vindex const CCTK_INT *vindices (list of) CCTK variables to get/put a hyperslab from/to. (variables are given by their CCTK indices; their domain decomposition must match the template variable as given in a previous call to Hyperslab_Define*MappingBy*() and coded in the mapping) - CCTK_INT timelevel const CCTK_INT *timelevels timelevel(s) to get/put a hyperslab from/to. (each timelevel matches its entry in the vindex/vindices[] parameter) If timelevels[] is passed as a NULL pointer then all timelevels default to 0 (the current timelevel). - CCTK_INT hdatatype const CCTK_INT *hdatatypes (list of) datatypes to convert the hyperslab data to/from. Thorns providing a hyperslab implementation might define implicite datatype conversion functions for some CCTK datatypes. These would convert the input array datatypes to some output array datatype. The calling routine can also supply a user-defined conversion function when creating the hyperslab mapping. (a negative output type or passing a NULL pointer for the hdatatypes argument indicates no type conversion) - (const) void *hdata (const) void *const *hdata (list of user-supplied) buffers to store the extracted hyperslabs for each input variable (for a get operation) or to take the hyperslab data from (for a put operation). This argument is only evaluated on processors which actually receive/ provide hyperslab data. - CCTK_INT *retvals user-provided array of CCTK_INT values to store the return codes for a get/put operation of an individual variable from the given list in Hyperslab_*List() If retvals[] is passed as a NULL pointer then no return codes for individual hyperslab operations are passed back. * Return Code: (according to the Cactus Coding Conventions) 0 for success, negative for some error condition (to be defined by the thorn implementing the Hyperslab_Get*()/Hyperslab_Put*() routines)