Thinning short profiles

[Dave Offiler]

From Axel von Engeln 15:30 19/6/2007:

Previous versions would not thin when less than the requested thinned
levels were given in the unthinned data. So, assuming I want 247 levels
back, but only have 120 in the unthinned data, the thinner would return 120
levels, without modifications to the data. This is of course one of those
annoying extreme cases and should not happen under normal circumstances,
but we currently have this kind of data in some Christian's offline
processor.

Anyway, for batch processing this is not very useful, so what I do now is
to return the 247 levels, but set them all to missing. That was the easy
way to implement this kind of trouble, the more accurate would be to use
the 120 levels and actually thin them were they are providing data. I leave
this update to you if you can bother (as said, should not really happen in
reality if all works well).

Axel reports that a few GRAS profiles can be generated with only a small number of vertical samples (say 120). The SAMPLE thinning method obviously does not sub-sampling if the user requests (or defaults) to a max. no of output samples greater than the input - ie it would output the origianl 120 samples. The interpolation methods behave in the same way; any profile with less than the (nominal) 247 samples is unthinned and will also output the 120 samples unchanged.

Axel requests that the 120 samples be mapped onto the 247 fixed levels (thus in effect increasing data volume rather than decreasing it, which latter was the objective of doing the thinning!). However, in this case, mapping all profiles to a common set of fixed levels is the objective for validation purposes. Axel has made a local fix to fill all 247 samples with missing data in these cases.

I consider the existing behavour to be expected - this is not a bug but a 'feature change' request, and a non-urgent minor issue to be investigated when resources allow.

[Dave Offiler]

I've now had a chance to look into this. The interpolation, in principle, can be done to map (say) 120 input samples to (say) 247 output levels. The problem arises in putting these 247 levels back into the ROPP structure which has been allocated (on file read) array sizes of only 120.

Implementing the requested change would mean one of:

a) deallocating all affected profile arrays and re-creating them larger before copying the thinned arrays, or

b) creating a whole new ROPP structure, copy non-profile data to it, then the thinned (bigger!) profiles or

c) knowing the required no. of fixed levels when reading the file, and pre-allocating arrays for the larger of the two.

Since any of these solutions would require significant amount of work, and Axel admits that for normal quality GRAS data, such short profiles ought to be very rare, I suggest not implementing this request - at least not before ROPP-2.

I leave it open for review again when GRAS Level 1b is declared operational by EUM; if short profiles do indeed 'go away' or a rare enough to be ignored as now, then this ticket should be closed as a 'won't fix'.

[(none)]

Milestone 2.0 deleted

[Huw Lewis]

Updates to the ropp thinner [1834] and rangecheck [1856] mean that a (empty) data structure with the required number of levels is created if no thinning occurs (e.g. if there are fewer data points in a profile than the required number of thinned levels). Rangecheck now writes out data if there is a valid height range, so the required output levels are always output. Note that short profiles are not 'thinned' by this change, but the output number of levels should be maintained.

Fixed for v2.0 requirement (number of output levels = number of thinned levels).

Ticket remains open and moved to v3.0 as reminder to look at thinner implementation for these cases in more detail.

[Huw Lewis]

GRAS Level1b data now operational. No further issues with thinning short profiles have been flagged up, and fix at v2 already ensures number of output levels = number of thinned levels irrespective of size of input observations.

Close ticket as fixed.