CTD Metadata Development Notes

CalCOFI CTD data are available from 1993 to present. Currently, a revision of the CTD data file formats are underway. Data quality code columns are being added to the CTD+bottle csvs offered as our final processed data. In addition to processing recent cruises, older cruises are being re-processed to update the data file formats. The re-processed CTD measurements should remain virtually unchanged but the column formats are being updated to include data quality codes (QC) for each core measurement. Data quality codes flag questionable or bad data and allow queries to filter out data using these criteria.

• CTD.csv: Bottle-corrected CTD+Bottle Index with QC

Older information about Metadata development is described below:

Starting with CalCOFI 0907M2, the BTLvsCTD program (developed by SIO-CalCOFI) generates several Metadata-friendly csvs. With Seasave Data Processing software v7.19 and later, sensor calibration coefficients are embedded as xml data in the .hdr file.  BTLvsCTD.exe parses the .hdr for all stations into three csvs: hdr.csv, xmlcoeff.csv, & span.csv.

• The .hdr csv includes the cast information such as date, time, line, sta, lat & lon, sensor configuration, and all Seasave data processing modules and coefficients applied.
• The span.csv is a tabulation of the min and max values for each sensor, extracted from the .hdr file. This file/table will be useful for automating basic data-quality controls and sensor behavior assessments, allowing data processors to query data ranges for validity.
• The last file is xmlcoeff.csv, a tabulation of the coefficients for sensor.

Starting with 0907, the first cruise to use dual SBE43 O2 sensors so the data tables will have these columns by default. Earlier cruises, using single O2 sensors will auto-fill the tables, leaving the second O2 columns blank. 
Since sensor installation may vary from cruise to cruise and sensors may occupy different voltage channels, the voltage column headers "V1" "V2" "V3"... are replaced by "TransV" "FluorV" "AltV"... whenever possible.

Complete FinalQC zip archives (YY-YYMMCC_CTDFinalQC.zip) include CTD data .asc files with both column headers - numeric voltages V0-V7 and relabeled voltages. This archives the Seasoft .asc product unchanged and adds BtlVsCTD Metadata-friendly .asc files. CalCOFI tries to keep the sensor channels consistent unless deploying test or ancillary sensors. On cruises such as 1210NH or 1611SR, the secondary oxygen was replaced with a RINKO Oxygen Optode so the voltage columns on those cruises are different. Typical sensor configuration:

• V0 = Transmissometer
• V1 = Fluorometer
• V2 = PAR
• V3 = Altimeter
• V4 = Primary Oxygen
• V5 = Secondary Oxygen
• V6 = ISUS Nitrate
• V7 = pH
• V15 = Surface PAR, when made available by the vessel; SIO's was retired

BTLvsCTD.exe's main module also generates bottle-corrected upcast and downcast csvs, merging all the individual station up and down csvs into two files.  A dbcoeff.csv file is generated tabulating the cruise-ave bottle vs CTD coefficients as well as the station regression coefficients (each station's CTD data are regressed against bottle samples from that station, reducing the station-to-station variability witnessed using the cruise-ave coefficients).  If bottle data are not available then only cruise-ave coefficients are used to correct the CTD data.

BTLvsCTD.exe has a module to automatically relabel the v0 v1 v2... to the sensor voltage in all .asc files. (I plan on using the hdr to automate this but currently the table in BTLvsCTD tabulating voltages vs sensor is used.)

BTLvsCTD.exe has a file renaming module that renames all dYYMM###.asc & .hdr, uYYMM###.asc & .hdr files to web-format YYMM_LLLLSSSS_###d.asc & .hdr, YYMM_LLLLSSSS_###u.asc & u.hdr. Please refer to the CTD Data Processing web page for more information on BtlVsCTD and CTD file formats.

The CTD final data database is finally finished.

The three zip files below contain the final bottle-corrected, Seasoft-processed CTD data from CalCOFI cruises from 1993-2019. These have been reprocessed into the current 84-col CTD csv format adopted by CalCOFI using data-quality codes-flags. 2020+ cruise csvs should import into these tables without issue. SIO-CalCOFI uses Microsoft Access as their relational database and that is the format distributed.

Download:

• 2019-2010 Microsoft Access Database File - final CTD data with bottle data from CalCOFI cruise 2010-2019
• 2009-2000 Microsoft Access Database File - final CTD data with bottle data from CalCOFI cruise 2000-2009
• 1993-1999 Microsoft Access Database File - final CTD data with bottle data from CalCOFI cruise 1993-1999

Please note - these files are distributed in decade-based databases because MS Access has a 2gb limit on table size. A complete CTD database should be imported into a relational database like MySQL that does not have this limitation. Exporting these individual databases in sql format from MS Access would facilitate the migration to a non-Access database. MS Access does allow linking separate databases so this method could be used to keep these data in MS Access tables for queries.

Archives are below - these are the individual cruise csvs used to create the above database files. For the complete 1993-2019 csv table visit this webpage

The files on this web page are database-import-ready csvs which combine 1m bin-averaged CTD sensor data with rosette bottle data. The addition of data-quality code columns required older (than 2013) archived cruises be reformatted with data code columns added. Currently, the original CTD+bottle data 62-column data files (YY-YYMMSS_CTDFinal.zip) are being reprocessed to insure consistency of the csv format. Data codes (see the CTD Database Index for key) are added to the QC columns to flag bad or questionable CTD sensor data. These data-quality codes may be used to exclude bad or questionable data during queries.  Bad, missing or questionable data values may be changed to -99.0 or NaN, plus the number of columns has increased from 62 to 82 columns.

CTD data archives labeled with "QC" (YY-YYMMSS_FinalQC.zip) or "DB" (YY-YYMMSS_FinalDB.zip) are the latest format. Individual database-ready csvs will be available for each cruise so users can update an existing database on whatever platform they prefer. The addition of these new columns should not change the data values presented in the earlier formats although bad data may be nulled (NaN or -99.0). Additionally, when a sensor primary measurement is data coded "9", any data derived from it will change to "NaN" or "-99.0" - estimated NO3 from the ISUS sensor, for example. Refer to the CTD Database Index and the CTD.csv with QC Index pages for columns and descriptions. As older CTD data are reprocessed to add the QC columns and data codes, this table listing database-ready csvs will be updated.

Additionally, filenames have or will change to match the current CTD data distribution format. CTD.csvs processed with IEH-formatted bottle data (all cruises prior to 2012) will have an "I" denoting "IEH" in the filenames before "D.csv" or "U.csv": e.g. 20-1104SH_093301200_049Id.csv.
Preliminary versions of newer database csvs are available by request (This email address is being protected from spambots. You need JavaScript enabled to view it.) but this page only lists final versions.

Two versions of the CTD final data for each cruise (YY-YY is year, MM is month, SS is ship code):

• YY-YYMMSS_CTDFinalDB = two (all upcasts merged & all downcasts merged) CTD.csvs. Example: 20-1708SR_CTDFinalDB includes the downcast bottle-corrected CTD data csv 20-1708SR_CTDBTL_001-073D & upcast bottle-corrected CTD data csv 20-1708SR_CTDBTL_001-073U. Please note the "001-073" refers to the cast numbers and can be different for each cruise. When different bottle-correction coefficients are applied to specific casts, there will be separate downcast & upcasts csvs for each group. For example, 20-1708SR_CTDBTL_001-025D would be downcast data for casts 001-025 & 20-1708SR_CTDBTL_026-073D is downcast data for casts 026-073.
• YY-YYMMSS_CTDFinalQC is the complete CTD+Bottle data archive: our standard CTD plus bottle-corrected CTD final data file distributed on Cruise Summary pages and Recent CTD Data page. It is also made available here since it includes the FinalDB csvs plus much more such as plots, CTD asc & hdr files and metadata. The individual station CTD.csvs, in same database-ready format, are in the csvs-plots subdirectory.