Handling of missing values in input phase

[kmk]

This sprang from the testing of ROPP on the ROSA data.

There are two elements to it. First: the code uses the dividing point between GO and WO to double as also the dividing line between two different ways of handling low-quality L2 data. Below 25 km low-quality (or missing) L2 data is replaced by a synthetic L2 signal extrapolated from L1. Above 25 km low-quality L2 data is replaced by a heavily-smoothed version of the input L2 signal. It is not very clear or reasonable that this dividing line between two regimes for handling L2 should always be the same as the GO-WO dividing line. The two don't have much to do with each other. Secondly, and probably more seriously, we discovered during ROSA data processing that the code does not correctly catch and handle missing values in the L2 excess phase. On the contrary they are treated as valid data with the value -9999. In the WO regime this will not cause drastic problems, because a missing value will be flagged as bad L2 data and L2 will be replaced by a synthetic L2. Also for GRAS data this will be caught as data gaps in the special GRAS preprocessing routine. However without a special preprocessing routine data in the GO regime will go crazy because the L2 signal will merely be replaced by a heavily smoothed version of the same signal – still including the -9999 value(s).

I think Dave's attittude was that ROPP should maybe more stringently prohibit data input from an unknown source. Right now it just throws a warning

[kmk]

Example of problematic GRAS input data

[kmk]

I just realized that this issue actually exists with GRAS data as well. Which makes it somewhat more serious. The GRAS preprocessing routine fills in data gaps but does not do anything to extend the L2 signal below the lowest data record. Therefore the issue with ROSA data described above is fully reproducible with GRAS data. The issue only appears when the L2 signal terminates close to or above the 25 km line (there is some special handling of the transition so in reality the issue may possibly be seen with L2 signals terminating down to about 20 km. This is true for something like 1 % of GRAS profiles. The code will detect the problem by reporting an extremely high L2 score (so e.g. in DMI processing we definitely catch these in QC) but the code will output an L2 bending angle profile that is very very bad and this problem will propagate to raw and optimized bending angles and on to refractivity.

Se attached file. Run ropp_pp_occ_tool on this to see a terrible output profile. If you run ropp_pp_occ_tool with hmax_wo set to 35 km instead of the 25 km default the output profile is much better (because all the missing L2 data is then replaced with extrapolated data)

[Ian Culverwell]

Despite the 'major' importance tab, nobody is running around screaming about this, so move to ROPP8.0. Certainly no time to do anything about it for ROPP7.0.

[Ian Culverwell]

Don't know when I'm going to have time to devote to this. Changing milestone accordingly.

[Stig Syndergaard]

Should be revisited in light of modifications at DMI to accommodate Metop reprocessing. Moving ticket to DMI development. Changing milestone and owner.