SFTPD-HDF5: Globus Striped Ftp Server with HDF5 plugin
Cactus uses the HDF5 (Hierarchical
Data Format version 5) as its standard I/O library and binary file
format for the output of multi-dimensional datasets.
Cactus output files generated by the corresponding HDF5 I/O methods can
then be imported by any standard visualization program which supports the
HDF5 file format.
For the OpenDX Visualization Toolkit we
provide an OpenDXutils package with various import
modules to read datasets from local HDF5 files into an OpenDX network.
One of these modules, ImportHDF5,
is also prepared to import datasets from HDF5 datafiles which are located on
certain remote FTP servers.
On the client side it uses the GridFtp Virtual
File Driver of the HDF5 library to transparently access
HDF5 files over the network.
This effectively enables you to visualize remote datasets
directly, without having to stage them to a local filesystem first.
In order to serve individual requests to datasets in remote HDF5 files,
the FTP daemon running on the remote FTP server has to provide
commands for partial remote file access. One server which implements such
extended FTP operations is the wuftpd server with GridFTP patches,
as distributed with the Globus toolkit. This server offers extended get/put
commands which, besides the filename, take an offset and a length as
parameters to access a contiguous sequence of bytes in a remote file space.
Higher-level operations such as reading an individual dataset in a remote HDF5
file are then executed as a sequence of such raw remote reads.
The drawback of this simple data protocol is that hyperslab requests which
access non-contiguous regions in the remote file space (eg. due to
downsampling) may result in a large number of individual raw remote read
transactions, increasing the overall latency of the single HDF5 read call.
SFTPD-HDF5 is another, more efficient implementation of an FTP server which can
be used for remote HDF5 file access. It is based on the Globus Striped Ftp
Server (SFTPD) and has been enhanced by a plugin which can execute HDF5 read
requests directly (without translating them into lower-level transactions).
The following sections describe how to obtain and install the SFTPD-HDF5 server,
and how to use it for remote HDF5 file access.
Downloading and Installing
The SFTPD-HDF5 server is contained in the SFTPD-HDF5 package which can be
downloaded
here.
It contains the required source files (with a few fixes) from the original
SFTPD distribution, the HDF5 plugin source code, a makefile to build the
frontend and backend executables, as well as a README file with some technical
information and software licensing conditions.
Note that the original SFTPD server is distributed under the Globus license
(see the file GLOBUS_LICENSE for details) whereas the enhanced
SFTPD-HDF5 server itself is licensed under the GPL General Public License
(see the file COPYING for details).
Please be aware of this licensing issue when using the SFTPD server.
If you don't need the SFTPD-HDF5 functionality to access remote HDF5 files
you should download and install the original SFTPD server instead
which is available via anonymous cvs:
setenv CVSROOT :pserver:anonymous@cvs.globus.org:/home/globdev/CVS/gsi_clients
cvs login # password is 'anon'
cvs checkout striped_server
In order to build the SFTPD-HDF5 server you should follow the steps below.
As a prerequisite, you will also need to have Globus with the
globus_ftp_control library installed on your machine as well as HDF5
(please refer to the Globus
Toolkit installation pages and
Installing and Compiling
HDF5 resp. for installation details).
The environment variable GLOBUS_LOCATION must be set to point to your
Globus installation directory. The environment variable GLOBUS_FLAVOR
must be set to specify a valid globus build flavor which should be used to
build the SFTPD-HDF5 server.
The environment variable HDF5_LOCATION must be set to point to your
HDF5 installation directory.
# get the SFTPD-HDF5 package and unpack it in some scratch directory
tar xzf SFTPD-HDF5-1.0.tar.gz
# set the necessary environment variables
setenv GLOBUS_LOCATION <my_globus_location> # eg. /usr/local/apps/globus/
setenv GLOBUS_FLAVOR <my_globus_flavor> # eg. gcc32
setenv HDF5_LOCATION <my_hdf5_location> # eg. /usr/local/apps/hdf5/
# build and install frontend and backend
cd SFTPD-HDF5
make
The GNUMakefile which comes with the SFTPD-HDF5 package will build the
SFTPD-HDF5 frontend and backend executables using the appropriate compiler and
linker flags for Globus programs, and install them in the designated Globus
location.
Support and Copyright
Thomas Radke is the author and
maintainer of the SFTPD-HDF5 package, with acknowledgements to Andrei Hutanu and
Andre Merzky from the Zuse-Institut Berlin
for their valuable ideas on the design of the HDF5 plugin.
The development work has been supported
by the Deutsches Forschungsnetz Verein through
the GriKSL project under contract TK 602 -
AN 200.
Please report bugs and send any comments to the
maintainer of the SFTPD-HDF5 package.
The software in the SFTPD-HDF5 package is available under the GNU General
Public License. Please refer to the files README and COPYING
in the SFTPD-HDF5 package for the full copyright notice and distribution
conditions.
In addition to the conditions in the GNU General Public License,
the author strongly suggests using this software for non-military purposes
only.
Last modified: $Header: /cactus/CactusWebSite/VizTools/SFTPD-HDF5.html,v 1.2 2004/01/22 16:34:21 tradke Exp $
|
![[MAINIMAGE]](../images/development.jpg)
