Rview/change valid ranges

[Dave Offiler]

Some of the 'valid ranges' tested in ropp_io_rangecheck() may be too strict for some parameters for which there is no clear-cut threshold for valid/not valid. The philosophy of the range check should be to set as 'missing' physically unrealistic values, not to knock out merely bad/noisy data.

a) Review all parameter default valid ranges as defined in ropp_io_types.f90 for meeting 'physically unrealistic' criteria and change as appropriate.

b) Change the values for Excess Phase (+/- 10,000m) & BA (-0.01 to 1.0) as suggested by Stig (see emails below) for v1.2.

c) Include an explicit review of the ranges in Josep's beta testing 'shopping list' for any updates in the v2.0 release version.

NB: Stig/Martin have been informed how to over-ride the built-in defaults, so they can apply whatever range limits they wish for the current v1.1. The changes here can be applied to the defaults in the next & subsequent release versions.

From Stig:

Dave,

I had a look at some more BA data, and talked to Martin about what
we think the range check in ROPP 1.2 should/could be. We suggest 
to have wide margins for a couple of reasons:

1) we don't think the ROPP range check should be like a QC check. 
A range check is okay to avoid e.g., format overflow - but does not 
necessarily have to screen out bad data. Data can anyways be 
pretty bad, but still within physically expected ranges, so strict
limits won't catch all the bad data anyways.

2) If data are indeed bad, it is nice to have the full information
in the NetCDF files and not have the data masked by -99999.9 
values.

3) We don't really know how large BA (and phases) can become once
the open loop data starts flowing - theoretically BA can become 
infinite, although that can't happen in practice - 
but how big then?

So here is the suggestion:

 -0.01 < BA < 1.0
 
Martin also noted that the excess phases have a range between 0m 
and 1500m. That also seems way too narrow. L1 and L2 excess phases
can become negative even if they by definition starts at zero for
a setting occ because the ionosphere can make them dip below 
zero before they start increasing in the neutral atmosphere. 
For rising occs, I could imagine EUM to set the first data point 
to zero (they don't do that yet, In fact most EUM phases are way 
out of range right now - but COSMIC do it), in which case rising
phases would decrease to become negative in the neutral atmosphere. 

Anyways, 1500m is too little - it can easily reach 4000m (for 
COSMIC open loop data that happens most of the time).

So to give some margin, a suggestion there could be:

 -10000m < excess phase < 10000m.

I'm a bit concerned that there may be more range checks around in
the ROPP that are too strict. Maybe we should consider review all
range checks in connection with the release of ROPP 1.2. Perhaps
specifically ask beta-testers to look at these. What do you think?

Best regards,
Stig

And in reply:

Hi Stig,

I agree that the purpose of the ROPP rangecheck is to throw out
'physically invalid' values, not merely 'bad' ones. The obvious
things are like latitudes, which have to lie between +/- 90 degr,
but others like BA are not so clearcut.

The range values for the various parameters have a long history -
back to the CLIMAP project of the early 2000's, and they haven't
been systematically reviewed since - Chris just took the values
from the CLIMAP format document (forerunner of the ROPP test format)
when he wrote the v1.0 ropp_io_types.f90 code, and so these are 
the (default) values tested with the new ropp_io_rangecheck()
routine in v1.1

We can change the BA & Phase limits for v1.2 as you suggest, and ask
Josep review all the values as part of his beta testing 'contract'.
Meanwhile, if anyone has suggestions to revise the current default
values, we'll be pleased to take them into account before the beta
testing starts.

Dave

[Huw Lewis]

Implemented change described under (b) for v1.2 release. See [1573]. Ticket remains open to cover more detailed review, and outcomes from v2.0 beta-testing.

[Dave Offiler]

Stig has consulted with Sergey on most extrere BA values ever seen, and the outcome is that a BA range of -0.01 to +0.1 should be more than sufficient:

Dave,

I talked to Sergey Sokolovskiy when I was in Boulder last month. The largest BA from 
COSMIC that he has been able to detect was slightly less than 0.05 radians. So it seems 
that the 6x10**-2 is just narrowly okay, although, I'd still like to keep it more open 
in the ROPP NetCDF files, just in case. If we (or somebody) ever see a valid BA larger 
then 6x10**-2 in the NetCDF files, we could then consider if a change in the BUFR 
template is necessary.

The -10**-3 on the negative end should be okay. My suggestion of -0.01 for ROPP 1.2 
included a wide margin. In practice I haven't seen a valid

L1 or L2 BA less than -0.0003 or so (that's not to say that it could not happen).

Bottom line: The limits in the BUFR seems okay (at least until we become wiser one day).

-Stig

Updated upper range default value to +0.1 (from 1.0 in [1573] - original was 0.06) for v1.2. See [1642].

NB: This lower limit is compatible with the BUFR offset (minimum negative value), but the new upper limit is a bit above the bit width capacity (~0.07) - hence Stig's reference to BUFR templates.

[(none)]

Milestone 2.0 deleted

[Huw Lewis]

Valid ranges of all variables reviewed in context of BUFR limits. See #145.

Ticket closed as fixed.