Support for long BUFR messages

[Dave Offiler]

Martin is attempting to decode full-res BA data from EUM, but is getting error messages from both the ROPP tool bufr2ropp and the generic BUFR decoder decbufr and requests this be investigated so that DMI can compare EUM's full-res and thinned BA BUFR products.

[Dave Offiler]

Sean reports that sample BUFR file is not decodable by ECMWF either.

Error message reported by both decbufr and bufr2ropp is code #3 (end-of-message delimiter '7777' not found), but inspection of file in nedit shows that '7777' is present.

Martin's sample BUFR file is >308,900 bytes long. This is significantly larger than the decbufr tool internal character/array sizes, so probably not enough of the message is being read in.

1) Attempted to raise declared character max. length of messages & max. array size for expanded descriptors from 25,000 to 500,000 bytes (new GTS max bulletin length) and 20,000 to 200,000 descriptors, respectively in decbufr. Still gives error code #3.

2) Looked into MetDB BUFR library code: -low-level C routine just reads a byte stream; no limits here. -intermediate-level bufrread.f, which extracts just a BUFR message (BUFR...7777) from the input byte stream works by reading in 1 or 2 150,000-byte buffers, as necessary. This leads to a max. supported message length of 300,000 - Cf EUM message of >308,000 bytes. Raised buffer size in bufrread.f90 from 150,000 to 250,000 bytes (max. bulletin/message length 500,000). Rebuilt BUFR package with ifort; decbufr tool can now decode the sample file with no reported errors. (NB not attempted here to validate the decoded values.)

Documented change in BUFR package CHANGELOG and rebuilt bufrpack.zip as Release Version 17.4 and emailed to Martin to test at DMI.

Successfully built & tested package with usual set of compilers under ROPP_ARC buildpack script.

[Dave Offiler]

Now for bufr2ropp...

Similarly raised max. message length and descriptor array size in bufr2ropp.f90 as per decbufr and re-run make.

Original error gone, but still no successful decode. Use of -d switch reveals that the message is being rejected as 'non-RO' because >1 observation in the message. A run of decbufr with -dh switch confirms that sample file contains 2 obs. in this one message - not supported by ROPP.

(Commenting out this test in the notRO() subroutine allows the message to go on to be decoded, but due to the interleaved nature of the decoded array parameters, bufr2ropp just gets out of sync and the output netCDF values will be wrong.)

At this point, the MetDB BUFR library package can support long messages (up to 500,000 bytes containing up to 200,000 expanded descriptors), and the ROPP decoder can likewise support long messages. But ROPP remains limited to decoding messages containing one - and only one - occultation per BUFR message. This was a design decision based on stated user requirements when the BUFR template was being developed; changing bufr2ropp.f90 to support >1 occ/msg could be done but would not be trivial.

This limitation (and UR) has been noted to EUM in the past, together with the 'won't fix' decision, so this is as far as long message support is going to go for the foreseeable future.

Martin's original problem therefore not entirely solved; recommend that EUM stick to UR of one occ. per message.

Updated decoder committed as [1552].