@ Copyright, CSIRO Australia
Use of this Web site and information available from it is subject to our
Legal Notice and Disclaimer
This paper was originally published in the Proceedings of the Land AVHRR Workshop, 9th Australasian Remote Sensing and Photogrammetry Conference, Sydney, July 1998.
Common AVHRR Processing Software (CAPS)
Peter Turner and Harvey Davies,
CSIRO Atmospheric Research,
Private Bag 1, Aspendale, Vic. 3195, Australia
Paul Tildesley and Chris Rathbone,
CSIRO Marine Research,
GPO Box 1538, Hobart 7001
The intention of the CAPS project is to promote the production of standard calibrated and geolocated AVHRR (Advanced Very High Resolution Radiometer) data products. This goal is to be achieved by including CSIRO's existing best AVHRR processing methodologies into a new coherent software package called CAPS and encouraging its use. CAPS has been designed to allow the efficient implementation of standard processing algorithms for AVHRR data that are contained in the NOAA satellite HRPT (High Resolution Picture Transmission). CAPS reads generic HRPT data files, and converts radiometer counts to radiances, reflectances and brightness temperatures. CAPS can then geolocate the calibrated data and transform them to a map projection. CAPS includes support for channel 3a and the solar reflectance channel dual slope calibration procedure for the new NOAA 15 AVHRR. CAPS has been implemented as an extension to the scripting language Tcl/Tk developed by Ousterhout (1994, 1998). CAPS uses HDF (Hierarchical Data Format, NCSA, 1998) to store results. CAPS is designed to be portable, flexible, extendable, efficient and is freely available to the intended user. CAPS presently runs under Unix and a PC version will soon be available.
CAPS comprises a set of commands added to Tcl/Tk environment. These commands include AVHRR calibration and navigation functions and the "nap" command. Figure 1 shows the structure of the CAPS system. The CAPS and NAP (Numeric Array Processor) libraries have been added to Tcl/Tk and are accessed through the Tcl interpreter interface. Tk commands provide an easy means of building GUI interfaces. The "CAPS" section of Figure 1 represents the CAPS parameter interface that manages input to, and output from, the algorithms that perform file I/O, calibration and navigation. The "NAP" section represents the numeric expression parser that calls the array processing functions of NAP. HDF I/O is accessed through both CAPS and NAP. All these functions are compiled into one program. The Tcl interpreter provides control structures, symbol tables and text-processing facilities. Tcl scripts can be written to allow batch processing. Tcl commands can be entered interactively, either directly or via a Tk GUI. All CAPS and array processing functions are performed efficiently using compiled C code.
The CAPS algorithms are designed to process AVHRR data from NOAA 7 to NOAA 15 and beyond. The NOAA 15 AVHRR instrument includes the new 1.6 m m channel (3a). Channel 3a replaces the data stream from thermal channel 3 during the day. In the new AVHRR instrument the old thermal channel 3 is called channel 3b.
In Australia most HRPT data has been archived in near HRPT form. In particular, the Australian Satellite Data Archive (ASDA, Turner et al., 1996) format is now used by the Australian Bureau of Meteorology and is being adopted more widely. The ASDA format consists of HRPT data preceded by a descriptive header written in Parameter Value Language (PVL, CCSDS, 1992). CAPS provides facilities to read ASDA format using the PVL library (Jones, 1998) and a range of other HRPT variants. CAPS can write to, and read from, HDF files using the Scientific Data Set (SDS) sub-format.
Calibration of the solar reflectance channels (AVHRR channels 1, 2 and 3a) is performed using the method proposed by Mitchell (1998). Mitchell recasts the standard NOAA NESDIS method in terms of responsivity and space count rather than gain and intercept. The results from the two methods are identical, but the Mitchell approach is more physically based and allows the orbit-related variation in solar irradiance to be easily introduced into the calculation. CAPS will convert AVHRR solar reflectance channel counts into radiances and reflectance factors. CAPS supports the use of both pre-launch and in-flight time dependent calibration values as well as the dual slope gain characteristic for the new NOAA 15 AVHRR. The calculation of the various radiances and reflectance factors is performed efficiently by using look-up tables.
The thermal IR channels, AVHRR 3, 3b, 4 and 5, are converted to radiances and brightness temperatures using the linear radiance correction method described by Rao et al. (1993). The initial conversion to radiance has been recast in terms of responsivity and space count in a similar way to the solar reflectance calibration. The thermal IR channels are calibrated in-flight, resulting in channel responsivities that vary as a function of received line number. Processing efficiency is achieved by parameterising the AVHRR count to radiance formulae for each line of each AVHRR channel.
The accurate geolocation of AVHRR pixels is problematic. Inaccuracies in orbit elements and small variations in satellite attitude can lead to navigation errors of several kilometres. Version 1.1 of CAPS includes the Brouwer-Lyddane (see Lyddane, 1963) and Clift orbit propagators. The Brouwer-Lyddane propagator uses the NOAA T-BUS orbital elements and is able to navigate within approximately 3 pixels. The Clift model uses a combination of an advanced orbit predictor and coastline fitting to produce orbit elements and partial spacecraft attitude information. The Clift orbit predictor then uses the orbit elements to navigate within approximately 1 pixel. The orbit modelling to produce the orbit elements is not part of CAPS and is performed at CSIRO Marine Research in Hobart.
The orbit models are normally applied only once every few lines of AVHRR data. The position of the central pixel in a line is calculated using the orbit model and satellite-Earth geometry. The satellite velocity information derived for the central pixel is then used to locate sampled pixels within the line. The resultant sampled grid can be linearly interpolated to locate any pixel in the scene. Such interpolation is used by various CAPS commands and facilitated within NAP by a form of generalised subscripting.
CAPS can remap data in the satellite coordinate system to map projection page coordinates. The procedure involves generating an inverse grid, which is a mapping from page coordinates to satellite coordinates (line and pixel). The warp command uses the inverse-grid to transform data to the required map projection. The warp command provides a choice of either linear interpolation or nearest neighbour data selection.
The control and execution of algorithms for reading HRPT files, AVHRR calibration, navigation and remapping is performed using the CAPS command parameter interface. CAPS commands have the form
caps <sub-command name> <parameter name> = <value name> …
Braces "{}" can be used to separate the "name=value" assignments across lines. All CAPS commands have input and output parameters that are defined in a data dictionary. CAPS parameter names are intended to convey the type of information contained. For example, a parameter name, radiance_avhrr_3b can correctly be concluded to be radiance derived from AVHRR channel 3b. There are approximately twenty CAPS commands needed to implement the current CAPS functions. It is not possible to describe all the commands here. Instead, an example is given that shows how the explicit parameter specification for one CAPS command (caps tircal) is made. The facilities provided by the CAPS parameter interface to simplify the explicit parameter specification are then illustrated by further examples.
The CAPS command caps tircal which converts thermal counts to radiances can be called from the Tcl environment with explicit parameter and value assignments in the following way: In this hypothetical example we have chosen a prefix that relates to satellite number (n15) and orbit number (14456), however there is no restriction on the prefix name.
caps tircal {
_caps_tircal_in_avhrr_3b=_n15_14456_avhrr_3b
_caps_tircal_in_responsivity_avhrr_3b=_n15_14456_responsivity_avhrr_3b
_caps_tircal_in_spaceCount_avhrr_3b=_n15_14456_spaceCount_avhrr_3b
_caps_tircal_in_spaceRadiance_avhrr_3b=_n15_14456_spaceRadiance_avhrr_3b
_caps_tircal_in_lrc_avhrr_3b=_n15_14456_lrc_avhrr_3b
_caps_tircal_out_radiance_avhrr_3b=_n15_14456_radiance_avhrr_3b
}
The names on the left of the equal signs are parameter names associated with tircal and are defined in the CAPS data dictionary. These names are unique to each CAPS command because, as can be seen, the command name forms part of the parameter name. In practice _caps_tircal can be omitted from the parameter names and, in most cases, so can the in and out. The names on the right can either be the names of Numeric Array Objects (NAOs, see section 5), text or references to these. CAPS uses a systematic naming convention (which can be overridden) that relates the command parameter name to the name of the data array. This convention allows the above example to be written as
set _caps_valuePrefix _n15_14456
caps tircal {
avhrr_3b
responsivity_avhrr_3b
spaceCount_avhrr_3b
spaceRadiance_avhrr_3b
lrc_avhrr_3b
radiance_avhrr_3b
}
The equals signs and value names on the right have been omitted and the parameter name prefix removed. The input array names are generated within the parameter interface by prefixing the short form of the parameter names by the value of the special Tcl variable _caps_valuePrefix. Therefore, in the above example, the input and output data array names are _n15_14456_avhrr_3b, _n15_14456_radiance_avhrr_3b, etc.. The lrc in lrc_avhrr_3b is an abbreviation for linear radiance correction. CAPS uses a dependency table that defines the input parameters needed to generate a specified output. The use of the dependency table enables the example to be further simplified, to
caps tircal {
radiance_avhrr_3b
}
or even
caps tircal radiance_avhrr_3b
Where several AVHRR channels are to be processed they can be specified individually:
caps tircal {
radiance_avhrr_3b
radiance_avhrr_4
radiance_avhrr_5
}
or a regular expression can be used to match the desired channels:
caps tircal radiance_avhrr_(3b|4|5)
The use of a data dictionary and dependency table and adoption of a naming convention reduces a quite complex specification of input and output parameters to a relatively simple expression involving only the names of the required output quantities. The names of parameters are chosen so that the names of output data from a command will match those of the input data requirements of subsequent commands. Thus, most input parameter specifications can be omitted. The names and values of all parameters are available during and on completion of the process.
NAP is an independent sub-system of CAPS which provides the infrastructure for storing and processing arrays of binary data. NAP is object-oriented and data are stored in numeric array objects (NAOs). Most CAPS data are stored in NAOs and can thus be processed using the generic facilities of NAP as well as CAPS-specific commands. NAP is also array-oriented and supports a wide range of array operations, including many mathematical operators and functions. NAP uses C notation where possible, but there is also notation derived from Fortran, APL and its successor, J. The following example illustrates some of the basic features of NAP. The input command lines begin with the standard Tcl prompt "% ".
% nap "x = {2.5 5 2}"
nao#0
% nap "y = x * x"
nao#3
% $y
6.25 25 4
NAP uses standard Tcl variables whose string values are NAO IDs consisting of the string "nao#" followed by one or more digits. The NAO IDs are uniqely defined by the system. The standard Tcl command set can be used to display the values of Tcl variables. So we can display the values of the above Tcl variables x and y by:
% set x nao#0 % set y nao#3
Every NAO has an associated (object-oriented) command whose name is the NAO's
ID. Thus we can execute the NAO nao#3 by:
% nao#3 6.25 25 4
which is equivalent to
% $y 6.25 25 4
It is usually more convenient to use names rather than IDs.
NAP provides array-oriented operations and functions in addition to those of standard arithmetic. The infrastructure provided by NAP has increased the flexibility and simplified the design of CAPS.
CAPS is designed to process large data arrays in memory. However, every computer has a finite amount of physical memory, and once this limit is reached the computer will swap data onto disk. Swapping is disastrous to performance. To overcome this problem, CAPS includes facilities to allow looping through a sequence of commands to process the data in pieces called chunks. Figure 2 shows a schematic of the "chunking" process that is implemented using the caps_process command, which is actually just a Tcl procedure that translates tcl commands into more detailed Tcl commands. In the example shown in Figure 2, chunks of a large input data file are read, calibrated, remapped to a standard geographic projection and written to a file. The rounded boxes represent either individual commands (get_hrpt and put_hdf) or groups of commands forming a sub-process (solcal and remap). The "nao" boxes represent unique NAOs that are constantly being created and destroyed during the processing. If a command is unable to process all its input data then it can save the remainder in a NAO for processing during the next loop. This is illustrated in Figure 2 by the remap process, which takes data in line and pixel coordinates and transforms them to a geographic projection. This transformation process will nearly always be unable to process all the data in one chunk without reference to data in the next chunk. The caps_process command sets a flag when input has ceased, and continues to loop until all participating commands have completed.
CAPS software has been written according to the specifications in the Tcl/Tk Engineering Manual (Ousterhout, 1996). Code is written in C, yacc, lex and m4. Yacc, lex and m4 components are translated into C using standard utilities. All CAPS and NAP code is managed using CVS version control. The CAPS executable code is checked for memory leaks and corruption using Purify (Rational software, 1998) and Lint for coding inconsistencies before a new version of CAPS is released. The Tcl "test" facility is used to apply an extensive battery of tests after every compilation. This includes comparison of calibration and navigation results with those from existing calibration and navigation systems.
CAPS is an efficient extendable system that provides state-of-the-art NOAA and CSIRO methodology to calibrate and navigate AVHRR data. Much of the flexibility of the system has been derived by utilising Tcl/Tk, and providing the NAP system as an extension to Tcl/Tk. CAPS is freely available and should contribute to the production of improved AVHRR data products.
CCSDS, 1992, http://ccsds.org/
Jones, R., 1998: PVL Library, http://www.bom.gov.au/climate/satellite/
Lyddane, R.H., 1963: Small eccentricities or inclinations in the Brouwer theory of the artificial satellite, The Astronomical Journal, 68, 555-558.
Mitchell, R.M., 1998: Calibration Status of NOAA AVHRR Solar Reflectance Channels, http://www.dar.csiro.au/pub/programs/atproc/earth_obs/avhrr_cal/calwatch/calwatch.html
NCSA, 1998: HDF Home Page, http://hdf.ncsa.uiuc.edu/
Ousterhout, J.K., 1994: Tcl and the Tk Toolkit, Addison-Wesley, Reading, Massachusetts.
Ousterhout, J.K., 1996: Tcl Engineering Manual: Conventions for writing C code in Tcl core, http://sunscript.sun.com/techcorner/
Ousterhout, J.K., 1998: Tcl/Tk Home Page, http://www.scriptics.com
Rao, C.R.N., J.T. Sullivan, C.C. Walton, J.W. Brown and R.H. Evans, 1993: Nonlinearity Corrections for the Thermal Infrared Channels of the Advancced Very High Resolution Radiometer: Assessment and Recommendations. NOAA Technical Report NESDIS 69, U.S. Department of Commerce, June 1993.
Rational Software, 1998: Purify, http://www.pureatria.com/index.html
Turner, P.J., B.A. Berzins, R.T. Jones and P.C. Tildesley, 1996: Australian Satellite Data Archive Format, CSIRO Division of Atmospheric Research, June 1996. http://www.dar.csiro.au/pub/programs/atproc/earth_obs/caps/index.html
(under construction)
Turner, P.J., H.L. Davies, P. Tildesley, C. Rathbone, 1998: Common AVHRR Processing Software, http://www.dar.csiro.au/pub/programs/atproc/earth_obs/caps/index.html
|
@ Copyright, CSIRO Australia |