Sections below: Float counts | Meta Files | Tech files | Plots & listings | Surface Pressure | TNPD


Argo file format and Surface Pressure correction compliance Audit


Argo Technical and Meta file compliance Audit

This page is automatically created by code which scans all files in the Argo GDAC.

Previous versions can be accessed via the CSIRO DMQC page

The audit purpose and method is described here

Compliance summary of this audit


Specific comments on this audit, based on Coriolis GDAC at 4 March 2012

Format errors in tech and meta files

A new set of lists is provided which identify tech or meta files which have defects. These are not listed for floats which have no recorded profiles. The lists can be accessed from the penultimate table on the Plots Index Tables page.

TNPD

Some reduction in the use of not-quite-right TNPD COMMENT strings at MEDS.

Some improvement overall in agreement with the audit estimate of TNDP occurence. It is not possible to fully automate a test for this so we do not expect total agreement. However the ratio of absent to present TNPD reports for each DAC should be somewhat indicative. Most DACs have "absent" rates of between 4% and 16%. KMA has a rate of 30% and JMA's is 142%. Is there something about the JMA floats that I don't know? If not, then this appears to be a place for big improvement.

Surface Pressure Parameter names

AOML have changed surface pressure parameter names for over 600 floats, to conform to the latest agreed naming schemes.

Three groups of invalid names are still seen at the GDAC: PRESSURE_OFFSET_dBar, PRES_OffsetAdjustment_dBar, PRES_DescentStart

Surface Pressure Correction

Little change is apparent when counting the number of floats with or without any SP adjustments different to our assessed adjustment, however when counting individual profiles a further improvement is evident. So new processing is being done very well.

Changes in profile counts where "good" indicates surface pressure adjustment agrees with audit software estimate to within 1.5dbar: 

  •  RT:  good +12.4%    bad: + 0.7%                     Overall Bad/Good ratio: 0.35%

  •  DM:  good +5.7%     bad: -9.8% (all at Coriolis)    Overall Bad/Good ratio:  1.1%


      • All profiles exhibiting any of the following faults are listed for each DAC. These lists are accessed from the table at the bottom of the Plots Index Tables. AOML and KORDI wil need to separate out the many "P ADJ not populated" reports in order to efficiently see the other reports.

    missing PRES_ADJUSTED

    There is a decrease in the number of floats missing PRES_ADJUSTED data. For RT files this might be in part due to changes to surface pressure parameter names in over 600 AOML floats so they are now regarded as auto-correcting and hence not requiring the ADJUSTED profiles. This also explains an apparent decrease in the incidence of individual RT profiles lacking values in PRES_ADJUSTED.

    Where RT PRES_ADJUSTED is unpopulated it will probably not be repaired until DM processing replaces the files, so this issue will not reduce immediately, but we should significant progress in 6 months? It is important that this issue is addressed so that it new profiles are not affected.

    Several DACs have made progress, including JMA who have almost completely removed this condition, however new cases are found at BODC and KORDI. Please let me know if any of these cases are legitimate rather than errors.

    "Truncating" float but negative surface pressure

    These curiousities can just arise from rounding errors, or because float is mislabeled as Truncating, but there also appear to be other causes? 65 profiles in various DACs.

    varying pressure correction in one profile

    A small number (161) cases exist where the apparent SP adjustment changes throughout the profile. MEDS has repaired some of these, AOML and Coriolis have acquired some.

    format faults in PRES profile and QC fields

  • fillval in PRES_ADJUSTED where QC<=3: ........... 3023 cases, mostly KORDI
  • space character (value 32) in ADJUSTED_QC: .......... 216 cases, Coriolis & KMA
  • bad PRES_QC, maybe space character: .............1700 cases, mostly KORDI
  • null character (value 0) in PRES_ADJUSTED_QC: ......... 3 cases, different DACs


    Jeff Dunn, Mar 15 2012 CSIRO


    • This analysis was based on GDAC as at 04 March 2012

    Float counts in DACs

    DAC details [explanations]
    DAC nameAOMLBODCCORIOLISCSIROGTSJMAMEDSCSIOINCOISKMAKORDI
    # floats4036378130651121101317110251147119
    # Projects9652112212222
    # PIs18667111652342
    # Models14450311854532

    Floats in multiple DACS
    DACAOMLBODCCORIOLISCSIROGTSJMAMEDSCSIOINCOISKMAKORDI
    # floats4036378130651121101317110251147119
    dups in GTS--15--------
    other dups--1-----1--


    Meta files

    Meta files [explanations]
    DAC nameAOMLBODCCORIOLISCSIROGTSJMAMEDSCSIOINCOISKMAKORDI
    # floats4036378130651121101317110251147119
    # meta files4036378130651121101317110251138119
    broken A-----------
    broken B--14--------
    Bad launch info1--------10-

    PLATFORM_MODEL
    The strings found in the PLATFORM_MODEL fields
    Strings have been converted to lower case and blanks removed to assist match-ups
    field contentsAOMLBODCCORIOLISCSIROGTSJMAMEDSCSIOINCOISKMAKORDITOTAL
    Empty Field---------9-9
    apex------2--9993194
    provor--146----19-2665
    palace_n/a17----------17
    solo_sbe1521----------1521
    apex_sbe1730------784--1812
    soloof_fsi28----------28
    solo_fsi272----------272
    apexir_sbe248------12---260
    soloir_sbe115----------115
    solo2ir_sbe84----------84
    apex_seascan10----------10
    apex_n/a2----------2
    solo_sbe41cp1----------1
    provor_seascan1----------1
    provor_n/a6----------6
    provor_sbe1------12---13
    apex-sbe-350-495-852279-22139-2236
    martec-fsi-21---------21
    -sbe-2---------2
    martec-sbe-5---------5
    provor ctf2--61--------61
    apex sbe apf7--91--------91
    provor ctf3--38--------38
    provor cts2--160--------160
    apex sbe apf8--204--------204
    nemo--86--------86
    nemo 2.5--4--------4
    provor cts3--177-2---5--184
    apex sbe 41 apf8--7--------7
    arvor iridium--6--------6
    provor cts3 do--16--------16
    arvor--35----8---43
    provor cts3 4.82--20--------20
    provor cts3 v4.8--6--------6
    provor cts3_4.84--1--------1
    provor cts3-do--15--------15
    nemo with sbe 41--11--------11
    apex sbe41 apf8--45--------45
    apex apf9a f/w w--6--------6
    apex apf9a f/w--55--------55
    apex-apf9a--73--------73
    provor cts3 do 9--14--------14
    apex apf8 rev da--4--------4
    apex-apf8c--12--------12
    apex apf8 rey da--2--------2
    provor t--30--------30
    apex sbe apf8b s--3--------3
    apex sbe apf8b--14--------14
    apex sbe apf8 s--10--------10
    apex sbe apf8 sn--11--------11
    apex sbe apf8 2--1--------1
    apex sbe apf8--1--------1
    apex sbe-43f apf--3--------3
    apex apf9a f/w s--3--------3
    apex-apf9--19--------19
    apex-sbe-apf8c--3--------3
    provor bioargo--1--------1
    apex-apf9a f/w--9--------9
    apex-sbe-apf9a--6--------6
    apex-apf9i--4--------4
    arvor-a3--2--------2
    apex sbe-41-alac--3--------3
    apex sbe-1-alace--1--------1
    apex sbe -41-ala--2--------2
    apex sbe-41 ap--2--------2
    apex sbe-41 apf--8--------8
    apex sbe-41 a--1--------1
    provor ct--5--------5
    apex sbe -41 apf--1--------1
    r1-sbe---10-2-----12
    palace-----4-----4
    alace_r1-fsi-----2-----2
    r1-sbe profiler-----1-----1
    r1-td-----9-----9
    provor-sbe-----14131----172
    ninja-tsk-----3-----3
    ninja-sbe-----15-----15
    tem. profiler-----1-----1
    apex-sbe-aandera-----7-----7
    provor-sbe cts-3-----14-----14
    nemo-sbe-----12-----12
    arvor-sbe-----27-----27
    apex-fsi-----3-----3
    pops-sbe-----3-----3
    apex-td profiler-----4-----4
    pops-provor-sbe-----1-----1
    p-alace------4----4
    apex-sbe/optode------1----1
    provor_fsi--------2--2

    Basic MODEL
    Grouping PLATFORM_MODEL fields by stanard float types
    Basic typeAOMLBODCCORIOLISCSIROGTSJMAMEDSCSIOINCOISKMAKORDITOTAL
    Unknown-2---74--9-22
    apex1990350604495-86628290225138935133
    solo2021----------2021
    provor8-55862155311226-26824
    nemo--101--12-----113
    ninja-----18-----18
    martec-26---------26
    palace17----4-----21
    arvor--43--27-8---78
    r1---10-12-----22

    PLATFORM_MODEL field - APF9 Status [explanations]
    AOMLBODCCORIOLISCSIROGTSJMAMEDSCSIOINCOISKMAKORDITOTAL
    Not APEX2046287021622353520269263145
    APEX not APF9--408--------408
    APEX unknown199035021495-86628290225138934550
    APEX APF9--175--------175
    Manual list*303--12-------315

    SENSOR pressure field
    * Sensor names converted to lower case and blanks removed to assist match-ups
    *AOMLBODCCORIOLISCSIROGTSJMAMEDSCSIOINCOISKMAKORDITOTAL
    Empty Field-3835--5317---51165
    druck67--434----182--683
    mp40-c-2000-g14----------14
    n/a1566-87--------1653
    sbe41383-3--581-264--997
    sbe1----------1
    2900psia1540-90----76---1706
    sbe41cp453-18--108-85--592
    sstd10----------10
    2000dbar1----------1
    1500psia1----------1
    sbe-41-346---------346
    ectp202mar-21---------21
    sbe-41v3.0-4---------4
    sbe-41cp-4---------4
    sbe41cp-2900psia--28--------28
    sbe41cp?2900psia--112-2------114
    sbe41cp2900psia--5--------5
    xxxx--62--------62
    none--6--------6
    3000psia--8--------8
    41--23--------23
    41do--1--------1
    sbe41do--3--------3
    sbbe41v3.0--8--------8
    1600psia--3--------3
    paine---38-------38
    amatek---19-------19
    paine3000psia---2----21--23
    kistler---17-------17
    "druck"---1-------1
    straingauge_sbe41-----285-----285
    straingauge-----26-----26
    straingauge_sbe41cp-----89-----89
    straingauge_ctd-f01-----3-----3
    sbe41,2900psias/n4867-----1-----1
    sbe41,3000psias/n2657701518-----1-----1
    sbe37-----2-----2
    fsi41--------2--2
    endrow--------37--37
    sbecp41---------124114238
    sbe41v3.0---------14-14

    SENSOR pressure Maker field
    * names converted to lower case and blanks removed to assist match-ups
    *AOMLBODCCORIOLISCSIROGTSJMAMEDSCSIOINCOISKMAKORDITOTAL
    Empty Field-3836--9----5853
    micron14----------14
    paines92----------92
    sbe961-31---3172691381141596
    n/a96-81--------177
    druck2770-239-2--68---3079
    2900psia67----------67
    kistler13-4----16---33
    seascan11----------11
    ametec12----------12
    seabird-3549511----240--1114
    fsi-21---1--2--24
    martec--5--------5
    sea-birdelectronics--4--------4
    xxxx--64--------64
    manufacturerdruck--6--------6
    ametek--7--------7
    none--1--------1
    nke--1--------1
    paine--4--------4
    sea-birdelectronics,inc._druck-----637-----637
    sea-birdelectronics,inc.-----158-----158
    sea-birdelectronics,inc.(druck)-----60-----60
    sea-birdelectronics,inc._kistler-----153-----153
    falmouthscientific,inc.-----4-----4
    sea-birdelectronics,inc._paine-----13-----13
    sea-birdelectronics,inc._ametek-----60-----60
    thetsurumi-seikico.,ltd.-----3-----3
    notspecified-----1-----1
    sea-birdelectronics,inc-----1-----1
    sea-birdelectronics,inc._-----1-----1

    SENSOR pressure Serial Number field
    S/NAOMLBODCCORIOLISCSIROGTSJMAMEDSCSIOINCOISKMAKORDITOTAL
    # floats4036378130651121101317110251147119
    No decodable number43091194-220511919121711

    tech files

    Tech files [explanations]
    FormatAOMLBODCCORIOLISCSIROGTSJMAMEDSCSIOINCOISKMAKORDITOTAL
    No profiles14918157-165661732271
    Missing------------
    Old-------7---7
    Old - Bad------------
    New388736012915042108531297245130878000
    New, no Surf Pres------------
    New, Bad format4-99--214----128
    V 2.3388736012915042108531297245130878000

    Surface Pressure Offset parameter name
    AOMLBODCCORIOLISCSIROGTSJMAMEDSCSIOINCOISKMAKORDITOTAL
    Old format
    Other1--1-6131-2841
    surface_pressure-1---------1
    surface_offset------------
    pressure_sensor_offset------------
    start_surface_pressure------------
    pressure_offset_(counts)1----------1
    pressure_offset_(dbar)-------7---7
    pres_offsetadjustment_dbar------------
    pressure_offset_dbar------------
    pres_surfacecomplement_dbar------------
    New format
    Missing27-1084-264----169
    pres_surfaceoffsettruncatedplus5dbar_dbar1052245414227-67521544139104-3115
    pres_surfaceoffsettruncated_dbar-----2-----2
    pres_surfaceoffsetnottruncated_dbar100789185275-21365459726872089
    pres_surfaceoffsetnottruncatedplus5dbar_dbar------------
    pres_surfaceoffsetcorrectednotreset_1dbarresolution_dbar-19294--8-----321
    pres_surfaceoffsetcorrectednotreset_1cbarresolution_dbar690----122-----812
    pres_surfaceoffsetbeforereset_dbar1014----------1014
    pres_surfaceoffsetbeforeresettruncated_dbar------------
    pres_surfaceoffsetafterreset_dbar------------
    pres_surfaceoffsetbeforereset_1dbarresolution_dbar-5168-213--4--192
    pres_surfaceoffsetbeforereset_1cbarresolution_dbar-21313-31-85--180
    pres_surfaceoffsetafterreset_1cbarresolution_dbar------------
    pres_surfaceoffsetbeforereset_5cbarresolution_dbar------------
    pres_surfaceoffsetafterreset_5cbarresolution_dbar------------
    pres_surfaceoffsetbeforereset_4mbarresolution_dbar------------
    pres_surfaceoffsetafterreset_4mbarresolution_dbar------------
    pres_descentstart_dbar29----------29
    pres_offsetadjustment_dbar96----------96
    pres_surfaceoffsettruncatedplus3dbar_dbar-----1-----1
    pres_surfaceoffsetnottrancated_dbar-----1-----1
    pressure_offset_dbar------28----28
    Multiple SP names per profile1027----------1027
    SP name changes thru float lifetime------------
    % ok SP values [explanation]9696959910097979999100--

    Surface Pressure Offset parameter name vs model
    apexsoloprovornemoninjamartecpalacearvorr1UnknownAll models
    Other7133-------41
    surface_pressure1---------1
    surface_offset-----------
    pressure_sensor_offset-----------
    start_surface_pressure-----------
    pressure_offset_(counts)-1--------1
    pressure_offset_(dbar)--7-------7
    pres_offsetadjustment_dbar-----------
    pressure_offset_dbar-----------
    pres_surfacecomplement_dbar-----------
    Missing19121310115--1-8161
    pres_surfaceoffsettruncatedplus5dbar_dbar3103-------12-3115
    pres_surfaceoffsettruncated_dbar---------2-
    pres_surfaceoffsetnottruncated_dbar19181271812--4-912088
    pres_surfaceoffsetnottruncatedplus5dbar_dbar-----------
    pres_surfaceoffsetcorrectednotreset_1dbarresolution_dbar--301--19-1--321
    pres_surfaceoffsetcorrectednotreset_1cbarresolution_dbar-682127-3-----812
    pres_surfaceoffsetbeforereset_dbar-1014--------1014
    pres_surfaceoffsetbeforeresettruncated_dbar-----------
    pres_surfaceoffsetafterreset_dbar-----------
    pres_surfaceoffsetbeforereset_1dbarresolution_dbar--184--5-3--192
    pres_surfaceoffsetbeforereset_1cbarresolution_dbar--106----72-2178
    pres_surfaceoffsetafterreset_1cbarresolution_dbar-----------
    pres_surfaceoffsetbeforereset_5cbarresolution_dbar-----------
    pres_surfaceoffsetafterreset_5cbarresolution_dbar-----------
    pres_surfaceoffsetbeforereset_4mbarresolution_dbar-----------
    pres_surfaceoffsetafterreset_4mbarresolution_dbar-----------
    pres_descentstart_dbar12-----17---29
    pres_offsetadjustment_dbar-96--------96
    pres_surfaceoffsettruncatedplus3dbar_dbar--------1-1
    pres_surfaceoffsetnottrancated_dbar1---------1
    pressure_offset_dbar--28-------28

    Plots and listings analysing the Surface Pressure Offset correction for all floats for each DAC are accessed through the Plots Index Tables. Separate tables link to all floats, or only floats with apparent SP correction anomalies, or only floats with apparent TNPD.

    Note: The following analyses were based on assumptions about behaviour of different types of floats, where "type" was here primarily defined by the Surface Pressure Offset parameter name. Some of these assumptions will be wrong, and some types are not analysed at all because I am unsure of their behaviour. It is probably necessary to also take basic float model (eg Apex or SOLO) into account to fully determine the pressure correction treatment required for a float, but it would be better if this could be fully determined by just the parameter name. My assumptions are recorded in this table.

    Surface Pressure Offset

    We have attempted to test for correct Surface Pressure (SP) correction of profile pressure.

    The difference between PRES and PRES_ADJUSTED is the DAC-applied pressure correction (assuming PRES is the raw P as reported by the float.)

    Correct application of this test depends on being able to deduce the correct float type and P adjustment from the meta and tech files. However this is not always possible due to missing or non-standard entries in these files.

    Our computed pressure correction is 

         p_calc = PRES - (SP - adjustment)
    where adjustment is the known SP adjustment for a given float type (usually 5dbar). For Apex floats the SP reported with a profile actually relates to the previous profile, because it was recorded just as the float begins to descend. RT profile pressure can only be adjusted with the SP reported with the same profile, but in DM profiles the correct SP is used (ie for profile N, 
         p_calc(N) = PRES(N) - (SP(N+1) - adjustment)
    So, in the provided plots , for floats which do not auto-correct, the "Calculated Correction" (CC) is 
         For DM profiles:     CC = SP(N+1) - adjustment
     For RT profiles:     CC = SP(N) - adjustment
    For this reason, CC will differ from Raw Surface Pressure by the "offset", and will be displaced by one profile for DM profiles. The "DAC adjustment" should however agree with the CC.

    Interpretting the plots: For complete agreement in pressure correction, the "DAC Adjustment" (blue squares), should agree with the CC (red stars). Missing "DAC Adjustment" symbols can occur where there is no PRES_ADJUSTED profile (eg if QC=4). Missing CC symbols can occur where the DAC has provided an SP estimate (possibly by interpolation) but the automated process has not [due to different processing or because the tech files are not quite up-to-date.]

    Disagreement is likewise visible where "MPP minus CC" (cyan diamond) does not overlay "Min Corrected Pressure" (magenta square). The latter quantity is just the near-surface PRES_ADJUSTED value.

    Note that the raw SP symbols (blue diamond) are shown exactly as found in the tech files. That is, their x-coordinate is the cycle number corresponding to that value in the tech files.

    A light gray background is used where disagreement is between 0.1 and 1.0 dbar. This occurs frequently due to DAC smoothing of the SP series, or for other reasons.

    A dark gray background indicates disagreement greater than 1.0 dbar. Some of these cases are due to deficiencies in our automatic procedures, but some others appear to be related to processing faults at the DACs.

    An olive green background behind the CC value indicates an alternative correction method (such as allowed for SOLO floats) gives better agreement with the DAC value than the primary method. The CC value shown is then as derived by the alternative method. For example, SOLO floats are auto-correcting, so the primary method is "no correction". However, a secondary method - the normal SP(n+1) correction - may be applied in DM if there is evidence of strong drift.

    A very pale green background indicates that the DAC has explained how their adjustment is correct, or pointed out that the CSIRO audit software has made a mistake. Any disagreement in these profile is ignored when collating statistics.

    Extra information in upper frame on plots
    Name Colour/Where Meaning
    P Adj QCtop halfprofile minimum PRES_ADJUSTED_QC for each DM profile, value indicated by scale on left of frame
    DMLower middleDM processed profiles
    No P ADJLower middlePRES_ADJUSTED or PRES_ADJUSTED_QC profiles not fully populated
    TNPD CommentNear bottom Has correct pressure SCIENTIFIC_CALIB_COMMENT for a TNPD profile
    P ERR 2.4bottomprofile min PRES_ADJUSTED_ERROR==2.4
    P ERR 20bottomprofile min PRES_ADJUSTED_ERROR==20

    See the table which summarizes the results of this pressure correction audit for all DACs.

    Lists: for each DAC a text file (accessable through the Plots Index Tables) lists the result of checking the surface pressure correction for every float. The floats are grouped by PI. The lists give the number of DM profiles, total number of profiles, assumed surface pressure offset, and counts of profiles which agree exactly or disagree with my automated correction. Please note that in many cases the disagreement is because your DAC correction is a better choice than my automated one.

    Format example: 

         5900033 DM 264/295 SPoffs=5 Diffs(dbar) =0[271] <1 [0] >1 [ 1 *]

    Of 295 profiles for this float, 264 have been DM processed. 271 profiles exactly agree with my automated correction, 0 disagree by a small amount, but 1 disagrees by more than 1 dbar.

    TNPD

    Truncated Negative Drifting Pressure (TNPD) is a term used to describe Apex floats which truncate negative surface pressure values (all firmware prior to APF9) and the pressure sensor develops a strong negative drift. The usual cause of strong negative drift in these sensors is the "microleak" fault associated with Druck sensors. The pressure in profiles affected by TNPD cannot be corrected for surface pressure offset.

    The first step in detecting TNPD is to determine if a float is of the type where such an fault might occur. We have attempted to do this using the tech and meta files alone, but this is not always possible.

    The next step is to find where the reported SP (corrected for 5dbar adjustment if necessary) has "flat spots" on zero, indicating that negative values are being truncated. Previously a float with 80% SP=0 was declared to be TNPD. We now say that any susceptible float is in a TNPD state after the last believable positive SP value. This is because the strong negative drift condition is typically monotonic, so if such a condition has commenced then typical variability will not give rise to small positive SP values.

    The main difficulty of this approach [assuming the basic assumptions are correct] is determining when isolated positive values are natural perturbations about a small negative value, or when they are just sensor error spikes. In the present analysis I have used (for the first time) the crude test that a positive value > 1dbar which has 2 zero values immediately either side of it, is a spike that can be ignored. [This differs to the standard despiking process used to prepare the SP series for pressure correction because here we just want a simple way of assessing where an SP series is devoid of true +ve values.] Obviously this is very imperfect - there is no substitute for skilled DM analysis, and in very many cases the DAC will be right and our algorithm will be wrong.

    Another deficiency of our algorithm is that is [apparently] extremely unlikely that microleaking will start after years of correct operation. For example, if SP is 0 for the last 5 of 200 profiles then the algorithm says TNPD commenced at profile 196. The time of onset of TNPD is examined in this set of TNPD plots. That page also includes plots which examining whether, for different sensor types, a large negative drift can be followed by recovery to plausible positive SP values.

    "Start of TNPD" in the surface pressure plots provided indicates where TNPD appears to commences, according to the above algorithm, if that is at least 6 months before the last DM profile. If the condition is met with less than 6 months of subsequent DM profiles, then 'Potential TNPD' is displayed. Where some of the float series has been DM processed, extra information is provided for the DM profiles in an extra frame at the top of the figure.

    See the table which summarizes the results of this TNPD treatment audit for all DACs.

    Lists: for each DAC two text files (accessable through the Plots Index Tables) list the result of checking for TNPD. The floats are grouped by PI. The "short" list gives the cycle number ("pn") where TNPD appears to start, and counts of profiles which appear to require the TNPD comment string and do not have it ("missed"), or do have the comment string ("good"), or have a comment string which is not precisely the standard string ("imperfect"). The list also notes where we have NOT detected TNPD but have found the TNPD comment string.

    		 
    Legal Notice and Disclaimer | Copyright | Privacy | Page author: Jeff Dunn | Contact: Jeff.Dunn@csiro.au