Too many levels in BUFR for MetDB

[Dave Offiler]

GFZ data in ROPP v1.0 (TEXT) is provided with 350-400 levels (samples) in the Level 1b, Level 2a and Level 2b sections. MetDB can only handle up to 379 on extraction due to fixed array sizes. Higher nos. of levels can be ingested, but not extracted again from MetDB - and in fact prevent smaller profiles (e.g. from UCAR) from being extracted too.

Solution: limit no. of levels in BUFR for MetDB to no more than 375.

[Dave Offiler]

While awating a 'proper' thinning implementation, ropp2bufr merely sub-sampled the input profile to ~200, but assuming the input was hi-res (of order 5000 samples), sampled 1-in-N triggered by there being >400 samples (ie >2x the desired number). For profiles <=400, no thinning was done. This allowed GFZ data 350-400 samples to go through unthinned. Also, only the Level 1b profile was potentially thinned, assuming Level 2a,b profiles would already be lo-res.

The thinning algorithm in ropp2bufr has now been modified to:

1. allow a max. levels (MLev) value on the command line with new -p switch (default: 375)
2. 'select 1-in-N' when (NLev > MLev*2)
3. 'reject 1-in-N' when (MLev < NLev <= MLev*2)
4. don't thin when (NLev <= MLev)

This thinning is applied to all three sections containing profiles.

Diagnostic output, help text and man page (ropp2bufr.1) updated to reflect the new -p switch and its application.

GFZ ROPP files now being encoded with default 375 max. levels; tested ok in MetDB extract and being used as from 22 Feb.

[(none)]

Milestone 0.9 deleted