Data management

In November 2009, there was a EGO2009 meeting in Cyprus, organized by Pierre Testor from LODY-C.

Among other topics, there were discussion on data-management.
A key issue of data management is to build a system close to Argo (profiling floats); a "brother" of Argo data management.

A first version of an EGO DAC (Data Assembly Centre) is available at:

It contains NetCDF OceanSITES data files, one per deployment.

Data are stored as vertical profiles; descent and ascent are split in 2 profiles.

A GoogleEarth index of Glider data is available at:

 

EGO2009 Data management session report

The data management discussions focused on 3 main issues: data format, metadata and technical data.

Data format

  • Different data formats were briefly presented: NetCDF, ASCII ODV, Medatlas, XML. HDF was also mentioned.
    During the discussion, a consensus was formed on NetCDF for data and a metadata subset.
    For the parameter dictionary, CF convention and SeaDataNet parameter discovery dictionary were presented. CF is now open on bio-geo-chemical parameters. SeaDataNet maintains a table of correspondence between the 2 dictionaries.
    A NetCDF variable has a name and a standard_name attribute. An harmonious way to proceed is to use CF standard_name attribute and SeaDataNet parameter discovery names.
  • Different data format will circulate among EGO (raw data (sbd), matlab files, ascii files, NetCDF for data set under construction), but ultimately, when the data set is “completed” it should be distributed in a standard way from EGO.
  • It is not envisaged to ask manufacturers to standardize data transmission from gliders.
    The telecommunication bandwidth and price is such an issue that it has to be used as efficiently as possible (a standardization would limit the efficiency of data transmission).
  • As a starting point, OceanSITES user's manual can circulate among EGO participants.

Meta-data

  • The users should find a subset of metadata in the data file (Who, what, were, when, how). The NetCDF global attributes can handle this kind of subset.
    But, in a NetCDF file, the global attributes is a flat list of metadata.
  • A comprehensive and detailed metadata may be handled in SensorML files.
    EGO may take advantage of the SensorML description of OceanSITES mooring deployments.

Technical data

  • They have to be archived for each deployment.
  • Command data: not a priority, but should also be archived along technical data (eg : pitch control commands).

 

Conclusion

An EGO data management group could be setup to propose a global data and metadata flow.

The data management should be very close to Argo data management; like a “brother” of Argo data management.

A NetCDF implementation for glider observations

Workshop on glider data-management in Brest, December 2012

How to provide glider data to operational oceanography users

How to provide glider data to operational oceanography users