HYDALP
Hydrology of Alpine and High Latitude Basins

Project Reference: ENV4-CT96-03634

Author: Graham Glendinning, IMGI
Date: 28 August 1997

Approved by manager of WP 110: R Caves Date: 27 August 1997

Approved by coordinator of WP 100: H Rott Date: 11 August 1997

Approved by project coordinator: H Rott Date: 11 August 1997

This document was produced under the terms and conditions of Contract ENV4-CT96-03634 for the European Union, DG XXII.

Hard copies to:

Soft copies are available via ftp from:

Server: ftp.sai.org
Directory: /ext/sca/HYDALP/private/common/Deliverables/RI112
Files: RI112.doc

1. Introduction *

2. Current Usage *

3. Recommendations *

4. Information Formats *

5. Information Storage and Transfer *

The aim of Task 112, the definition of software standards, is the choice of software usage for the HydAlp project, and a specification of software formats and documentation for use by all partners.

This report firstly outlines the current usage of project partners. Recommendations are then given for the specific hardware and software usage within the project. The following issues are considered; operating platforms, media types, word processing packages, image processing, programming languages, visualisation, drawing packages, GIS, and databases.

One of the problems with formulating software standards is that a common package will most often be the lowest common denominator of packages on offer. Hence, greater functionality at some institutes may need to be traded for compatibility between all institutes. When there is no package common to all institutes, communication between different packages becomes the crucial issue. An attempt has been made to draw upon the common usage of packages where possible, but it was often necessary to accept more than one standard with set porting routines.

Formats for program headers and in code documentation are suggested. The software consultancy, DIBAG, will write a program for general documentation from the information contained in program headers. For this reason, the software header information should be complete and self-explanatory. This automatic documentation program will accept all programming languages used in the project.

During the lifetime of the project, there will need to be an element of quality control and review of software. This is discussed in section 3.5.7.

Standards are given for transfer and compression of files between HydAlp members.

A general data format is outlined, for which the data files flowing between HydAlp modules are defined. These files themselves are to be defined in RI110/225, Data Flowlines.

For the purpose of standardising software usage, a matrix was compiled of current hardware and software usage of all HYDALP partners. This matrix comprised the following headings:

• Operating platforms
• Media types
• Word Processing
• Image Processing
• Programming Languages
• Visualisation
• Drawing
• GIS
• Databases

The results are listed below in Table 1. The highlighted rows in the table indicate recommended choices.

Table 2-1: Usage matrix.

There are two accepted computational platforms to be used within the project. These are UNIX workstations (Solaris, SUNOS, Silicon Graphics) and PCs running Windows (3.1, 95 or NT). All code written for the UNIX platforms should be at least portable between the different UNIX types.

Remote sensing data should be ordered / transferred onto CDROM (preferable) or EXABYTE and transferred by FTP (see Section 4.4). Other data will be stored on the individual institute’s own hard disk space and backed up regularly using their own normal methods.

As decided at the kick-off meeting, all reports must be written in Word 6. Every institute has access to Word 6 and experience of use. Attempts to use Word 7 and a converter to Word 6 have been successful, but the direct export from Word 7 to Word 6 loses some formatting information. Thus documents must always be checked in Word 6 before distribution. This situation may need to be reviewed if Word 7 supersedes Word 6 as the standard word processor at ALL institutes. For those who use Word 7, superior conversion software may be found at CEO on the ftp site under public/common/CompUtils/Word_converters, and information on it at http://www.microsoft.com/Officefreestuff/word/dlpages/wrd6ex32.htm.

PCI EASI/PACE 6.1 is the recommended image processing system. Additional modules for this may be compiled with help files in FORTRAN or C. PCI runs on UNIX platforms at every institute. With every institute working with this software, imagery, vector information and bitmaps may be transferred easily between institutes as PCI EASI/PACE image files (.PIX format).

Software modules to be developed in HydAlp, or to be made available to the project, will be written in a number of different languages according to application and user. For most scientific applications, Fortran 77 (or 90) and ANSI C (or C ++) will be utilised. These should be written as stand alone modules. Java will be used for the final demonstration product, and for intermediate tools. Its ability to call Fortran and C subroutines will be utilised. Other code and formats, such as shellscripts and IDL /PVWAVE *.pro modules will also be utilised. Java is suitable for coding new programs since it is both a powerful language (like C++) and it is highly portable. It is possible to have a main Java program and to call code written in other languages. It is not possible to call Java routines from C or FORTRAN.

The use of well commented, modular code with indenting of embedded loops is highly recommended. Good inline documentation is also critical. All program files (including subroutines) must begin with header information (see below).

Each program should be stored as source code, with README files and MAKE files sufficient for successful compilation and example input and output files. All subroutines should be written as separate files. Where there is more than one file, for purposes of dissemination all relevant files should be zipped into a single file, containing all of the required files for complete documentation, compilation and a test run. In the case of different subroutine files, the header information should be included in every program file. Where a program may be of use to more than one institute, a definitive version of the program should be stored at the CEO FTP site.

Errors should be captured and lead to a logical notification procedure. Log files should be generated by each program. These should contain the details of inputs and run parameters such that the run could be repeated if required.

File name endings should be such that the language is overtly recognisable, as shown in the below table:

Table 1: File extensions

All program files (including subroutines) must begin with a software header. To minimise the problems associated with using different programming languages, headers for all languages in use must be defined in a standard way. The only differences in the styles of headers should be in the format used for commenting in the various languages. Allowed comment characters are listed below (with families of languages grouped together i.e. Fortran, PCI Fortran, Fortran 90):

Table 3: Comment Characters

All lines of a software header should be regarded as comments by the compiler.

The software header consists of software keywords and descriptive text associated with the keywords. All of the text following a keyword, before the next new line that starts with another keyword, is associated with that software keyword. New lines that do not begin with a keyword are admissible within the descriptive text. New lines that begin with a keyword signify the end of the previous descriptive block. The software header section ends with the keyword HEADER_END, which has no associated text.

The software keywords are as follows:

• FILENAME (self explanatory name)
• TITLE (title of program or subroutine)
• AUTHOR (author)
• INSTITUTE (author's institute)
• VERSION (which version of program)
• MODIFICATIONS (dates of original and revisions)
• LANGUAGE (programming language)
• PURPOSE (description of software purpose)
• MEHODS (description of software methods)
• INPUTS (list of input data)
• OUTPUTS (list of output data)
• COMPILATION (instructions for compilation)
• USAGE (how the program is executed by the user)
• LIBRARIES (list of required libraries)
• BUGS (known of; new ones should of course be reported)
• REFERENCES (e.g. for the algorithm)
• HEADER_END (signifies start of code)

These keywords are associated with strings, except HEADER_END, which signifies the start of the actual code.

Additional keywords will be defined if required.

An example program with software header keywords is given below:

All software developed within the project must be verified and tested. Verification and testing must be conducted by at least one person other than the original author(s) of the program. All bugs should be reported to the authors and an upgrade issued. Upgrades must also be verified and tested. Then a final version may be disseminated. Software verification and testing, including timescales, should be co-ordinated at the WP manager level. Hopefully the majority of bugs will be rooted out during testing, and any coming to light afterwards must be reported to the authors and all users. A new upgrade should then be tested and issued.

For programs specific to a package (such as PCI programs written in Fortran or C), there may well also be a format for including package help information. This should be fully utilised.

For Java, there is an API Documentation Generator called "Javadoc". By using a simple syntax in the header documentation of the Java-classes, the generator is able to create automatic documentation in HTML-format. A further Java program is to be developed by DIBAG to extract this simple header information from C, Fortran, Java, shellscripts and IDL files.

The preferred visualisation tool for other than image processing is IDL (or PVWAVE). IDL enables repeated procedures to be carried out easily, and is more powerful than a simple spreadsheet package. It is also available for both PC and UNIX environments. Plots may be output from IDL in Computer Graphics Metafile (CGM) or PostScript (PS) format, both of which may be converted to Windows Meta File (WMF) format or read directly (via EPS for PS) into Word.

Excel may be used instead of IDL for those who wish, though it is more time consuming as it is harder to program to do repeated tasks on different data sets, and thus share methods between institutes. SMHI’s chosen display format will be Excel, as they have no access to IDL.

Most institutes use PC based drawing packages (see Table 2-1). WMF is the chosen format for storing drawings. WMF files may be further edited if required. For institutes preferring Mac or Unix based drawing packages, EPS is the preferred storage format. EPS files are not easily edited, but can be imported directly into Word (as can WMF).

The chosen geographical information systems are ARCINFO, and ArcView. It is also possible to utilise the GIS capabilities of PCI EASI/PACE.

No specification is made regarding the use of databases at each institute. The format of data supplied by the database is laid out in RI110/225.

Image data files should be stored in the .PIX format for PCI files. Raw image data files require processing to reach .PIX format, and their format is specified by the supplier (see RI232 and RI233). Images for inclusion in reports should be stored in JPEG format. Word files containing large numbers of images should use symbolic links to incorporate images. Images for display on the WWW should be degraded e.g. for the final public product (as discussed at the kick-off meeting). This can be accomplished using a simple averaging filter or the lossy compression facilities of JPEG.

The general form for data to be used in the HYDALP project (below the level of specific DBMS or GIS packages) is given here. This will be a common format for all such data, and has been defined to be as open as possible in structure. This general format enables the coding of general input and output routines for tools in the HYDALP project.

The file structure consists of general heading keywords, specific data keywords and data. These keywords are given below.

Where the actual data is defined by the COLUMN keywords, eg.

Other column keywords are also defined. The units may be defined using the column keyword, column.#.units, which is associated with a string. Parameters will also have default units, so column.#.units is not obligatory. The remote sensing data format includes the column.#.code keyword, as an identifier of hydrological units.

After this list of keywords, the data given by the COLUMN keywords is given, encapsulated by data-table identifiers, as given below.

More than one set of tabulated data may be included in one file (thus using the same general headers), by including further data after the TABULATED_DATA_END. The next table is then defined by redefining the COLUMN keywords, and given between further data-table identifiers.

A sample meteorological data file is given, and an example remote sensing data file per hydrological unit. The full file formats in the HydAlp project are defined in RI110/225, Data Flowlines.

The meteorological column parameters are defined with their default units.

Date_Time DD.MM.YY HH:MM date and time

Temp_Mean degrees C mean temperature

Precip mm (per time unit) water equivalent precipitation (per unit area)

Snow_Height mm(per time unit) height of snow

Pet mm (per time unit) potential evapotranspiration

Humidity percent relative humidity

Windspeed m/s windspeed

The hydrological column parameters are defined with their units:

Date_Time DD.MM.YY HH:MM date and time

Runoff m³/s runoff (average value per timestep)

Temp_Water °C Water temperature (seldom used)

Keywords describing the generation of the remote sensing product and additional information about sensor, image characteristic, map projection, etc. These keywords are not used at the moment, but they provide additional information about the product.

The choice of compression routine for the large files to be used in the HYDALP project is of some import. It is recommended that the ZIP utility, by INFO-ZIP, be utilised. This is a cross platform utility, and has the advantage of being totally compatible with PKZIP on PCs. Compression is encouraged for all sizeable transfers.

The INFO-ZIP software may be downloaded from:

Data may be passed between groups by way of the WWW, by CDROM and by Exabyte. WWW transfer should be done as much as possible through the FTP site at CEO. Email file attachments may be attempted, using standard software such as Netscape mail, Unix Mail or Pegasus Mail. Problems arising from mailing should be reported to the sender if not soluble, and the file placed at the CEO FTP site. It should be emphasised that the CEO FTP site is the preferred method of transfer, and obligatory for files larger than 10MB but less than 50MB. Very large files (>50MB), should be posted on CDROM ISO9660 (readable by UNIX and PCs) format (preferable) or Exabyte, to reduce downloading problems. They may also be downloaded at night to and from the CEO site.

The CEO site should be managed by placing files under the following directories:

ext/sca/HYDALP/private/name for data from name to another party, or

ext/sca/HYDALP/private/common for data to all project members.

All files under common should be placed in a logical directory structure.