_FillValue attribute for variables?

[Huw Lewis]

At ROPP-4, the value of ropp_MDFV is now output as a global attribute in the netCDF. Choosing a value of -99999999.9 means that missing float values are actually being output as -1E8 due to rounding.

All double and float missing values should really have the same missing value. If the variable attribute _FillValue is set, then perhaps we can set different output FillValues for floats and doubles. It may also be possible then for missing data to be output when viewing the file as '-' rather than the full numerical value.

To be investigated/fixed....

[Huw Lewis]

Choice of -99999000.0_wp gives a value which is represented by the same number in both float and double precision. This is still a large enough negative value to cover all valid ranges of ROPP variables.

Still need to investigate options for defining _FillValue explicitly for each variable. Ticket remains open.

[Huw Lewis]

First part of ticket (representation of missing values) closed by milestone 4.1 deadline.

Milestone for outstanding part of this ticket (defining _FillValue attribute for each output variable) moved to ROPP-6 (unlikely to fall within scope of ROPP-5 developments).

[Dave Offiler]

Comment by Dave: not sure that it is necessary or desirable to have per-variable !_FillValue. The new global value (-99999000.0[_wp]) does the job. Do we have examples where this value is not appropriate? (Actually, there is one: GNSS (X,Y,Z) positions in which all 3 components set to zero is interpreted as the position vector is 'missing')

[Dave Offiler]

Related issue to this: if a parameter saved in a ROPP netCDF file is invalid (out of valid range, but not actually set 'missing' to _FillValue), within ropp_io_getvar() the pre-check before units conversion will pass the test for ropp_MDTV, and the (invalid) value will be unit-converted. This may (and in demonstrated cases does) bring the converted value into valid range, so that the range check at the higher ropp_io_read() level will pass the bad value as valid.

1) The low-level read needs not only to check for 'missing' but also for 'invalid' before unit conversion. That or unit conversion is done at a higher level, after range checking (or even as part of that procedure).

2) Unit conversion should also convert the parameters valid max/min range values and update the new units too. This didn't appear to be happening the test case (pressure hPa->Pa)

[Ian Culverwell]

User Florian Ladstädter has resurrected this issue by raising it as helpdesk query 83

(​http://www.grassaf.org/view_admin_enquiry.php?id=83)

User Stig Syndergaard has spoken in favour:

Hi,



my 25 ører:



I support Florian's suggestion. Bringing the format closer to CF-standard would be a good reason in itself. It would also display missing values as a '-' when using ncdump, which I find is a nice feature and makes it easy to spot missing values when looking at ncdumps.



-Stig

So we should probably do something about it.

[Ian Culverwell]

Address at ROPP7.0

[Dave Offiler]

Non-trivial change. Not for v7.0

[Dave Offiler]

Not considered a priority. This ticket a linkage to the general issue of checking validity and units conversion - see for instance #312 (which while a specific issue, has implications for general ROPP netCDF I/O) - and this ticket could be wrapped up in that wider context.