Argo pressure correction audits


Auditing the Argo GDAC archive to improve standardisation of pressure correction and of meta data,
and to measure and reduce pressure errors



Introduction

Pressure data from different types and generations of Argo floats requires different treatments to make correct use of surface pressure readings. The Argo community has worked to understand and agree on what treatments are required by which floats, and also on standard elements in the metadata allowing third parties to carry out or verify the surface pressure corrections.

Since August 2010 we have run assessments of the global archive to determine how well the metadata identifies float type (as it relates to pressure correction), and the extent to which the agreed pressure correction is being applied. We have used this process to provide input to efforts to standardise the metadata and the treatments, and to provide feedback to PIs and national DACs so they can modify files where required.

During this period there has been steady refinement of metadata standards, and improvement in implementation of metadata and pressure correction. This has lead to bias reduction and improved accuracy of the pressure dataset, which in turn potentially improves global climate signal estimates.

Method

All R and D files except those grey-listed are exaimined. Key fields in meta and tech files are scanned for standard strings. To increase the likelihood of identifying standard strings we have ignored Case and the placement of spaces. Tables of equivalent strings have been created to assist in some cases. Arguably even fields such as PROJECT_NAME and PI_NAME should have standard entries or codes, to assist matching and aggregation of statistics.

It seemed the critical piece of metadata for determining pressure correction is the Surface Pressure Offset name ("SPO name") in the TECHNICAL_PARAMETER_NAME field. By constructing a table prescribing the relationship between this name and the type of pressure correction we can uniquely identify the required correction by reading just this one metadata field. This is desirable because it should be simple and foolproof, but also because other existing fields, such as PLATFORM_MODEL, SENSOR_MODEL, SENSOR_SERIAL_NUMBER, are mostly poorly populated or very far from standardised. However, we already have exceptions where the float type (SOLO) is also needed in order to determine the correction, and also where one SPO name is associated with two possible treatments.

Defects which we have sought to eliminate include:

  • missing or empty tech or meta files
  • files corrupted or of deprecated format
  • multiple SPO names associated with any one profile
  • SPO name changing during the life of a float
  • The value associated with the SPO name is also decoded. It is regarded as being valid if it is present and lies between -100 and +100. The series of SPO values for each float is pre-processed in the recommended manner for R and D files, omitting however the recommended manual assessment for delayed-mode. Where the pressure correction method is able to be determined we apply that correction to the contents of the PRES field. The result is compared with the existing apparent pressure correction, deduced from the difference between PRES and PRES_ADJUSTED.

    Missing ADJUSTED data

    Many profiles have some missing data in the PRES_ADJUSTED or PRES_ADJUSTED_QC field (where there are corresponding values for PRES and PRES_QC). This is much more common in R-files but also occurs in D-files. This is different to cases where pressure data has been assessed and flagged as bad. It is obviously serious as the user community is being deprived of this data. This condition is highlighted on audit plots and listings, and also quantified in the audit Pressure Correction summary table. Obviously a knowledgable user could apply the pressure correction for themselves but first they need to correctly identify the correction appropriate to that float.

    There are many variations to this fault. Technically, ADJUSTED fields only have to be populated in R-files if an adjustment to the raw data is required [but if populated for ANY parameter they are then required to be populated for ALL parameters]. However, since users will prefer to access ADJUSTED fields it may be warranted to populate these even if just with a copy of the raw data? Some Apex floats have missing PRES_ADJUSTED fields - which is not acceptable because this data requires correction,

    The actual algorithm to detect this condition is (as of Nov 2011):

        if PRES data and (empty PRES_ADJUSTED or empty PRES_ADJUSTED_QC or 
           ( all PRES_ADJUSTED==Fillvalue and any PRES_ADJUSTED_QC<4 ))
    
    The final pair of conditions is to allow ADJUSTED fields filled with Fillvalue if and only if the QC values indicate the data has been intentionally flagged as bad. A potential fault with this test is if the data values have been overwritten with Fillvalue where QC=3 [I don't know whether or not this is accepted practice.]

    A great many cases arise because PRES_ADJUSTED_QC appears to be empty when decoded with Matlab str2num function, due to the presence on a value (such as 0) which does not translate to a numeral. In future, such cases will be listed and reported to the DACs.

    A less common fault is where the difference between PRES and PRES_ADJUSTED varies throughout the profile. (SOLO floats may have the deepest pressure value adjusted to correct the average for that bin. Obviously we do not regard that as an error.) Previously we just reported the floats with this error but we now identify the individual profiles so that it can be more easily traced and corrected.

    Comparison of our correction and that performed by the DAC

    The correction performed by the DAC can differ from our estimated correction for several reasons:

  • the wrong SPO name has been used
  • the wrong correction has been carried out for the given SPO name
  • we are not fully informed of the requirements of the particular float type, so more dialogue and updating of the SPO name table is required
  • the PI or DM operator has made a judgement or manual intervention to provide a better result than our automated system can achieve. This accounts for many of the discrepancies. For example, discrepancies are detected with the CSIRO dataset even though the software is almost identical. However, this should not be just assumed to be cause of discrepancies.
  • different SP spike removal or gap-filling algorithms have been used
  • the correction has been done wrongly or not at all
  • The level of agreement is quantified by counting the number of profiles where:

  • difference is less than .1 dbar ("exact agreement")
  • 0.1 < difference < 1 dbar ("slight disagreement")
  • difference greater than 1 dbar ("strong disagreement")
  • Where the DAC determines for a given float that any discrepancies are due solely to more correct judgements applied during DM processing, we record the float details in a list. That float is then excluded from discrepancy statistics and the float's plot is identified with a distinctive colour. This has so far been applied for a number of floats managed by John Gilson.

    For float types where two possible pressure correction treatments are accepted, we test against both treatments and notify where the secondary correction is used, or where neither correction matches that performed by the DAC.

    TNPD

    Assessment of Truncated Negative Pressure Drift (TNPD) Apex floats is best done by visual inspection by a knowledgable operator. However an algorithm is used which attempts to identify the TNPD condition despite spikes and other bad SP values. It cannot always be correct, and certainly cannot match an expert's assessment of the point of onset of significant negative drift. However the resulting bulk statistics do indicate the level of skill and compliance with the TNPD detection and notification methods.

    Different levels of TNPD are recorded by adjusting values in PRES_ADJUSTED_ERROR, PRE_ADJUSTED_QC and strings in SCIENTIFIC_CALIB_COMMENTS. These strings are sometimes intended to notify TNPD but do not precisely match the standard string. Such cases of "imperfect comments" are being steadily corrected throughout the archive.

    Reporting

    Each audit is reported through a collection of webpages. These include tables analysing the metadata, with statistics for each DAC, and listings showing occurences of a range of conditions for each float, with a link to a surface pressure correction plot for each float. Tables also quantify the pressure correction and TNPD detection and reporting for each DAC. The main page of each audit also carries a commentary of significant changes or findings in that audit.


     

    Legal Notice and Disclaimer | Copyright | Privacy | Page author: Jeff Dunn | Contact: Jeff.Dunn@csiro.au