id	summary	reporter	owner	description	type	status	priority	milestone	component	version	resolution	keywords	cc
122	Rationalise missing data values	Dave Offiler	Dave Offiler	"Currently, ropp_io_init() intialises the ROPP structures with hard-coded missing data values (MDV). These values vary with the parameter, and were designed to fit the field widths of the ROPP text formatted files, whilst being clearly 'special', out-of-range values. As such the MDV can be -9.0, -99.0, -999.0 etc. Testing for MDV is currently a hit-and-miss affair because of this parameter dependency and also because files not created with ROPP may have completely different MDVs. (This is the case with GFZ ROPP-format text files today, where for instance Temp_sigma MDV in ROPP is -999.0 but GFZ set missing values to -9.0.) 

Definition and use of MDVs needs to be rationalised by implementing the following requirements:

1) The only place where MDVs are actually defined is in the hard-coded ropp_io_init() routine. MDVs should be more openly defined, for instance as new structures complementing the valid range structures. In principle, the MDV default values so defined could be over-ridden by the user.

2) The default MDVs should be documented, e.g. in the ROPP Interface File Format document parameter tables.

3) Use of MDVs for initialisation and testing should then use the structure values and not assumed hard-coded values.

4) Create a user-callable routine specifically to check for valid parameter values (using the range structures) and to substute the parameter-dependent MDV (from the above MDV structures) for out-of-range parameter values. NB some parameters come in sets and may need to be checked in combination; it may not be appropropriate to check them individually - e.g. POD (X,Y,Z) is all zero for POD MDV, but zero is valid for any single POD component.

5) Call this check routine after reading or before writing a file, or before thinning a profile.
It should be possible for a user to optionally suppress this default behaviour.

6) MDVs should be protected from units conversions.

7) All parameter attributes should be consistently saved to the netCDF file and read back.

The above should be implemented in roughly that order of prority. Some (e.g. 1,2,3) should be targetted for V1.1 release; the others could be implemented in later v1.x releases depending on the complexity of the implementaton, resources and pressure to release v1.1, e.g. for critical bug fixes elsewhere.

For later releases, consideration should be given to the possibility of defining a single common (parameter-independent) MDV for internal and netCDF file use, reserving parameter-dependent MDVs only for text-formatted I/O. The check routine could have an option flag in the argument list
to switch between the two MDV sets.

Indeed, we could (and eventually should) go futher and remove direct support for text files from the generic ROPP read/write routines, and perhaps provide only a simple, stand-alone text/netCDF converter. At some point, the text file should be deprecated and then become unsupported entirely in favour of the netCDF format. Text in CDL for manual inspection can be provided by ncdump (and ncgen can re-import CDL)."	enhancement	closed	normal	3.0	ropp_io	1.0	fixed	Missing data value, range check