Port to use ECMWF BUFR library

[Dave Offiler]

ROPP-3 (and all previous releases) interface with the Met Office ('MetDB') BUFR library. This was chosen mainly because it was already in use for several similar internal and export applications (and was therefore well-known to the author); was/is (reasonably) well supported on a variety of platforms/compilers; was/is well validated since the same code is used operationally for the MetDB.

The downsides are:

• the code is not open source, but has a restrictive licence, and it is not encouraged to be 'given away' except to registered users. We do distribute the package as a stand-alone dependency of ROPP via the GARF website, which at least audits user downloads.

• while the MetDB decoder function handles BUFR Edition 4, the encoder does not (yet) support this edition, which has been in use since 2005; edition 3 is supposed not to be used after November 2012.

• with the MetDB Replacement project (potentially being replaced for NWP use by ECMWF's ODB system), the continued long-term support and development of the MetDB library is not guaranteed (though certain to be continued for a few more years yet).

• the MetDB interface has only a 32-bit float argument to the encoder & decoder. With GPSRO Impact Parameter values requiring high precision (80km to nearest 0.1m), we experience truncation errors at the 0.5m level.

• the MetDB encoder crashes when attempting to encode particularly long (but within WMO guidelines) BUFR messages. (The decoder works ok with long messages, though). Not an issue for GPSRO unless there is a requirement to encode many full-resolution profiles into one BUFR message; our continued policy is 'one profile is one message'.

At the time that the MetDB package was selected for ROPP development, other implementations were available (notably ECMWF's) but with even more restrictive conditions, with an unknown interface and dubious support prospects. Several of the downsides for the MetDB package noted above were also yet to surface...

In the last couple of years, the ECMWF library has been available under a GPL licence, is demonstrably supported, and is already being used for NWP SAF deliverable code (AAPP). The latest release (000380) also fully supports BUFR Edition 4 (to meet WMO FM94-XIII regulations).

So ROPP-4 would be a good opportunity to port the ropp2bufr & bufr2ropp tools to use the ECMWF API library. As a period of backward compatibility will be required, as users may not initially wish to change their dependency libraries, we need to be able to interface to either library (at build time). This should be possible by extending the existing ropp_io configure script which detects if the (MetBD) BUFR library is present or not (and skips the encoder/decoder builds if not. The application code will need two parallel API-level interfaces to be chosen at build time.

[Dave Offiler]

Changeset [2253] is a working port of the BUFR encoder (ropp2bufr_ec.f90 and associated files).

This version (v4.0b) has so far only been validated using ../data/gfz_test.nc against ropp2bufr v3.0 (ie as released), and requires testing against other input files (esp. GRAS). This version has BUFR Section 0/1 settings the same as for V3.0 (notable BUFR Edition 3, Local Table Version 1 and Data Sub Type 255. The ported version gives identical BUFR Section 0-3.

There are two non-identical issues:

1) The value coded into the IP header for the bulletin length is 15665 in v3 and 155675 in v4. This is because v4 explicitly adds in the length of the IPH itself (10 bytes). This needs re-checking - the FROST technote seems to suggest v4 is correct, but v3 is obviously working without complaint from the FROST guys.

2) A few (perhaps a dozen) Level 2 profile pressure values are exactly 10Pa different (smaller) in v4 against v3 when both dumped using decbufr (using MetDB library decoder). Investigation shows this to very likely be a rounding error, as BUFR only encodes to the nearest 0.1hPa (10Pa), and different rounding behaviour can be expected for REAL*4 (v3) and REAL*8 (v4) implementations for the 'Values' working array.

Things to do:

1) Implemented look-up of ICAO codes via CT001033 locally for ropp2bufr_ec.f90 since this functionality is built into the MetDB BUFR library and not available in the ECMWF one. This would better be implemented as an extension of the roppbufrcodes.nl namelist given the limited number of known codes involved.

2) More thorough validation using as a minimum the GRAS & UCAR test netCDF files and with a larger sample of GRAS data from DMI.

3) Reset encoder to use BUFR Edition 4, Local Table version to 0 (or 255, whichever the Expert Group on WMO codes meeting next week recommends) and Data Sub-type to 14 ('GPSRO') assuming this submission is accepted by the same meeting. Check expected coding of headers.

3) Port decoder to ECMWF BUFR and validate likewise.

4) Tidy code, finalise build set up, document, merge to trunk for Test Folder checking.

[Dave Offiler]

Changeset [2259] implements ToDo item (1).

roppbufrcods.nl - and default tables in ConvertCodes() - now include look-up for the ICAO code in the ORGcodes namelist/arrays. Called within !ConvertBUFRtoROPP(), results from the look-up are passed via the main program to EncodeBUFR() for insertion of the originating centre code into Section 1 and the ICAO code into the ARH (if one is required).

As the encoding is now run at the processing centres (DMI & GFZ), the requirement to modify the originating centre code in Section 1 and ICAO code in the ARH has disappeared, so the -c option has been removed and the codes are derived only from the data. This implies that the originating centre code, derived from the text-based ROPP code in the netCDF file, is the same in Section 1 header (and ARH) as in the encoded Section 4 data. In the event that another centre (e.g. !MetO) encode RO data processed elsewhere (e.g. GFZ), then the namelist tables should be edited appropriately.

Man page for ropp2bufr also updated to reflect changes to the encoder to date.

[Dave Offiler]

The ECMWF port, testing & documentation will not now be complete in time for v4.0 release. As there are other changes (notably OCC code) which will also not make it into v4.0, it is agreed to hold these lower-priority changes over for a v4.1 minor update release in Q1/2010.

To (a) make it simpler to continue the BUFR port from v4.0 to v4.1, and (b) introduce users to the forthcoming (if temporary until ROPP-5) support for both MetDB and ECMWF BUFR libraries, the v4.0 release should support the v18.2 MetDB BUFR build, which builds the object library as libmetdbbufr.a instead of libbufr.a (which unfortunately is the name also used by ECMWF).

Part of changeset [2268] implements the ropp_io module configure/build system to use the new name. In case users do not install the v18.2 BUFR package, changeset [2276] provides a modified buildpack script which will copy an existing, pre-v18.2 object library file to the new name prior to building ropp_io.

[Dave Offiler]

News since this Ticket was raised:

1) The MetDB BUFR library code at v19 has been "refreshed" and has been re-written completely in Fortran 95. It now supports encoding of BUFR Edition 4. The MetDB replacement project is still some years off from implementation, so the MetDB code will continue to have support for now. Hence some of the reasons to move away from this package are less valid.

2) The ECMWF package is now at version 000387, and compiler/platform support is more flexible for combinations not supported 'out-of-the-box'.

The ROPP ropp2bufr & bufr2ropp tools have been re-written to separate common code as new Fortran modules (one for encode, one for decode), with the main application code containing only BUFR library specific (API) code. Some code from the MetDB package 'extra' tools (e.g. GTS header generation) have been re-used in modified form to be compatible with both BUFR libraries.

First implementation used Edition 3 and other header settings from ROPP-4; this was tested to demonstrate bit-identical BUFR files and equality of decoded netCDF values between ROPP-4.1 and the two flavours of ROPP-5.

Secondly, set up both ROPP-5 to encode Ed.4 and update some of the Section 1 values to reflect corrected settings advised by IPET-DRC in the past year or so. It is noted that MetDB BUFR still inserts pad-to-even-number bytes, and ECMWf dies not, hence EC files are 2 bytes smaller than MetDB ones. This is of no consequence, and apart from this, the BUFR remains identical (but now different, in Section 1 only, from ROPP-4.1).

Finally, the ECMWF build system, which uses make include files, has been integrated into the ROPP build package and buildpack updated to use these files, including some OS and/or compiler dependencies.

The ECMWF (000387) + MetO (19.0) packages have been built and both tested manually (with ropp_io 'make test') on Linux, AIX and Cygwin with all the usual compilers.

Closing this Ticket. Should any bug emerge in full Test Folder testing, a new ticket should be raised.