Opened 3 years ago

Closed 3 years ago

#699 closed task (fixed)

Errors in RO BUFR descriptors

Reported by: Ian Culverwell Owned by: Ian Culverwell
Priority: major Milestone: 11.0
Component: ropp_io Version: 10.0
Keywords: BUFR Cc:

Description

Two inconsistencies in the ROM SAF RO BUFR document (current version, vn 2.5, June 2019 is attached) have been discovered by Jeff Ator at NOAA and Doug Hunt of UCAR:

  1. The Master Version Table Number in Table 3 (octet 14 of BUFR Section 1) is incorrectly set to 12. It should be set to 35, since new satellite IDs (under descriptor 0-01-007 in Table B) and new satellite instruments (under descriptor 0-02-019 in Table B) have been added for Spire and GeoOptics data, but these did not become fully valid at the WMO level until version 35 of BUFR Table B.
  1. Bits 8-10 of the RO quality flag in Table 9 (descriptor 0 33 039) have been given the same meanings that the equivalent variable, PCD, has been given in ROPP. But these meanings have not yet been endorsed by WMO. https://github.com/wmo-im/BUFR4/blob/master/BUFRCREX_CodeFlag_en_33.csv shows that they are currently 'reserved', i.e. undefined.

See https://trac.romsaf.org/wiki/ticket/332 for further details.

The ROM SAF BUFR document has been updated at r6588.

These changes, permanent in the first case and (we hope) temporary in the second, will affect ropp2bufr and bufr2ropp and their ecodes cousins. These should be fixed urgently.

Attachments (2)

bfrPrf_2020-09-28T20-55-40Z.41739859.100.G27.bufr (41 bytes ) - added by olewis 3 years ago.
bfrPrf_2020-09-28T20-59-21Z.41811642.107.E05.bufr (41 bytes ) - added by olewis 3 years ago.

Download all attachments as: .zip

Change history (16)

comment:1 by olewis, 3 years ago

To update the Master version table the values assigned is changed in the ropp_io/bufr/ropp2bufr_mod.f90 file from the current value of 12 to 35.

For ropp2bufr_eccodes the current version of eccodes does not work with Master Table 35. By changing the value to 35 and running ropp2bufr_eccodes the following output is given.

---------------------------------------------------------------------
                    ROPP to BUFR (ecCodes) Encoder
---------------------------------------------------------------------

INFO (from ropp2bufr_eccodes):  Reading  ROPP data from ../../../ROPP_n699_origeccodes/ropp_io/data/ropp_test.nc
INFO (from ropp2bufr_eccodes):  Encoding profile    1 : OC_20090817215807_META_G027_DMI_
 
WARNING (from ConvertCodes):  ROPP-BUFR codes NAMELIST file roppbufrcodes.nl could not be found.
       Using default look-up tables
ECCODES ERROR   :  unable to find definition file sequence.def in bufr/tables/0/wmo/35/sequence.def::bufr/tables/0/local/0/94/0/sequence.def
Definition files path="/data/users/olewis/ROPP_n699_origec_BUILD/ifort/share/eccodes/definitions"
ECCODES ERROR   :  unable to get hash value for sequences
ECCODES ERROR   :  set: unexpandedDescriptors Hash array no match

With the current version of eccodes used (eccodes-2.12.5-Source) the latest Master table available is 31. To get the Master Table 35 the version required is eccodes-2.19, the current most up to date version of eccodes is eccodes-22. To build these newer versions requires CMake3.

For the MO Bufr when you update the Master table to 35 the conversion passes and the value is included in the decoded BUFR file. However it states that the value is not valid.

-------------------------------------------------------
          Met Office Generic BUFR Decoder
-------------------------------------------------------

Reading bufr_mo_test_gfortran.bfr

 ===== Message no. 1 =====

 BUFR Section 0
  Length of section:            8 octets
  Total Message length:     14742 octets
  BUFR Edition:                 4

 BUFR Section 1
  Length of section:           22 octets
  BUFR Master Table:            0 (Meteorology)
  Originating Centre:          94 (Copenhagen)
  Originating Sub-Centre:       0 (none)
  Update Sequence No:           0 (Original)
  Data  Category:               3 (Vertical soundings (satellite))
  Inter sub-Category:          50 (Radio Occultation sounding)
  Local sub-Category:          14
  Master Table version:        35 (INVALID VERSION VALUE)
  Local  Table version:         0 (not used)
  Message time:          21:58:08 17-Aug-2009

On looking into the MO BUFR package it has been found that the current version being used is bufr-24.0.2a and this contains Master tables 20-26. The next available MO bufr package v25 contains upto master table 31.

comment:2 by olewis, 3 years ago

I have built an updated version of eccodes using cmake3. The version used is eccodes-2.22 which contains the tables for MasterTable 35. This has required updating some of the configuration files to enable building using cmake3. I have tested this new version with Spire files and this is detailed in ticket #702 as this involved updating the namelist file.

The change has been tested with a a few Spire bufr files which have been attached to this ticket.

To test the Spire bufr file has been converted to an ropp file using the bufr2ropp routine and then this file is converted back to a bufr file using ropp2bufr and the orignal bufr file is compared to the new bufr file.

To decode the bufr files the eccodes dump_bufr routine was used.

With Spire file "bfrPrf_2020-09-28T20-58-34Z.41754962.101.G32.bufr" there were two differences in the final outputted bufr and the original bufr. First there was the masterTableVersionNumber which was 12 from the orignal Spire bufr and is now 35 as the change has aimed to do

"key" : "masterTablesVersionNumber",
      "value" : 35

Secondly there is a noncoordinatePressure value. In the Spire file this appears as 1010 hPa but at a geopotential height of -1000 (at the very end of file, ordered ascending) which in the outputted bufr file the pressure is reset to NULL. This behavior seems to be consistent across the Spire files with the very upper levels having values substituted with null.

The netcdf ropp files produced by the control without the new namelist and produced by the new namelist were also compared the results of this are shown below for the same spire bufr file as above. With new namelist

// global attributes:
		:title = "ROPP Radio Occultation data" ;
		:institution = "SPIRE Spire Global, Inc." ;
		:Conventions = "CF-1.0" ;
		:format_version = "ROPP I/O V1.1" ;
		:processing_centre = "SPIRE Spire Global, Inc." ;
		:processing_software = "UNKNOWN" ;
		:processing_date = "2021-07-19 10:08:42.351" ;
		:pod_method = "UNKNOWN" ;
		:phase_method = "UNKNOWN" ;
		:bangle_method = "UNKNOWN" ;
		:refrac_method = "UNKNOWN" ;
		:meteo_method = "UNKNOWN" ;
		:thin_method = "UNKNOWN" ;
		:software_version = "V00.001" ;
		:_FillValue = -99999000. ;
data:

 occ_id =
  "OC_20200928205835_SP3U_G032_SPIR" ;

 gns_id =
  "G032" ;

 leo_id =
  "SP3U" ;

 stn_id =
  "UNKN" ;

Control

// global attributes:
		:title = "ROPP Radio Occultation data" ;
		:institution = "UNKNOWN" ;
		:Conventions = "CF-1.0" ;
		:format_version = "ROPP I/O V1.1" ;
		:processing_centre = "UNKNOWN" ;
		:processing_software = "UNKNOWN" ;
		:processing_date = "2021-07-19 10:08:41.885" ;
		:pod_method = "UNKNOWN" ;
		:phase_method = "UNKNOWN" ;
		:bangle_method = "UNKNOWN" ;
		:refrac_method = "UNKNOWN" ;
		:meteo_method = "UNKNOWN" ;
		:thin_method = "UNKNOWN" ;
		:software_version = "V00.001" ;
		:_FillValue = -99999000. ;
data:

 occ_id =
  "OC_20200928205835_UNKN_G032_UNKN" ;

 gns_id =
  "G032" ;

 leo_id =
  "UNKN" ;

 stn_id =
  "UNKN" ;

The final comparison is between the bufr outputted by the current setup without changing the namelist or updating the eccodes build. The differences between the spire bufr files created by bufr2ropp followed by ropp2bufr are shown below. New output

{
      "key" : "bufrHeaderCentre",
      "value" : 178
    },
{
      "key" : "masterTablesVersionNumber",
      "value" : 35
    },
{
          "key" : "satelliteIdentifier",
          "value" : 269,
          "units" : "CODE TABLE"
        },
        [

          {
            "key" : "satelliteInstruments",
            "value" : 530,
            "units" : "CODE TABLE"
          },
          [

            {
              "key" : "centre",
              "value" : 178,
              "units" : "Common CODE TABLE C-1"
            },
{
 "key" : "geopotentialHeight",
 "value" : -1000,
 "units" : "gpm"
                           },
{
 "key" : "nonCoordinatePressure",
 "value" : null,
 "units" : "Pa"
                           }
  

Control

{
      "key" : "bufrHeaderCentre",
      "value" : 65535
    },
{
      "key" : "masterTablesVersionNumber",
      "value" : 12
    },

{
          "key" : "satelliteIdentifier",
          "value" : null,
          "units" : "CODE TABLE"
        },
        [

          {
            "key" : "satelliteInstruments",
            "value" : null,
            "units" : "CODE TABLE"
          },
          [

            {
              "key" : "centre",
              "value" : null,
              "units" : "CODE TABLE"
            },
{
    "key" : "geopotentialHeight",
    "value" : -1000,
    "units" : "gpm"
  },
  {
    "key" : "nonCoordinatePressure",
    "value" : null,

Both handle the odd value at the end of the file in the same way.

comment:3 by olewis, 3 years ago

I now understand the pressure value at the end of the file is actually the Spire satellite id. Currently there is no designated area within the BUFR for storing this extra information and so a pressure coordinate has been used. The satellite id is multiplied by 10 to conform to the expected values of the pressure coordinate. So in the above example commented on the satellite id would 100.

I am not sure yet how we can handle this in the ropp2bufr and bufr2ropp routines.

comment:4 by olewis, 3 years ago

With the Met Office MetDb BUFR library this change also works.

The current MetDB library does not contain Master Table Version 35 but by updating the roppbufrcodes.nl namelist the MO bufr conversion code can also work with version 35. The only difference in the decoded bufr from tests run on building ropp_io is the Master version table number changing from 12 to 35.

As for the eccodes this has also been tested with the Spire bufr files and the same results are seen with the MO bufr as was seen with eccodes.

Original MO decoded bufr output (v12)

-------------------------------------------------------
          Met Office Generic BUFR Decoder
-------------------------------------------------------

Reading /data/users/olewis/ropp_n699_results/bufr_mo_control_test_ifort.bfr

 ===== Message no. 1 =====

 BUFR Section 0
  Length of section:            8 octets
  Total Message length:     14224 octets
  BUFR Edition:                 4

 BUFR Section 1
  Length of section:           22 octets
  BUFR Master Table:            0 (Meteorology)
  Originating Centre:       65535 (INVALID CENTRE CODE VALUE)
  Originating Sub-Centre:       0 (INVALID CENTRE CODE VALUE)
  Update Sequence No:           0 (Original)
  Data  Category:               3 (Vertical soundings (satellite))
  Inter sub-Category:          50 (Radio Occultation sounding)
  Local sub-Category:          14
  Master Table version:        12 (2 November 2005)
  Local  Table version:         0 (not used)
  Message time:          20:58:36 28-Sep-2020

 BUFR Section 2
  Section 2 not present

 BUFR Section 3
  Length of section:           10 octets
  No. of obs. in Section 4:     1
  Message contains observed data
  Data is not compressed
  Descriptor   1: 310026

 BUFR Section 4
  Length of data:           14180 octets

 BUFR Section 5
  OK

 Data in Section 4:

 SATELLITE IDENTIFIER                                  Code 001007    - - - - -
 SATELLITE INSTRUMENTS                                 Code 002019    - - - - -
 IDENTIFICATION OF ORIGINATING/GENERATING CENTRE       Code 001033    - - - - -

decoded bufr (bufr2ropp ropp2bufr on Spire bufr file)

-------------------------------------------------------
          Met Office Generic BUFR Decoder
-------------------------------------------------------

Reading /data/users/olewis/ropp_n699_results/roppbufr_mo_test_ifort.bfr

 ===== Message no. 1 =====

 BUFR Section 0
  Length of section:            8 octets
  Total Message length:     14224 octets
  BUFR Edition:                 4

 BUFR Section 1
  Length of section:           22 octets
  BUFR Master Table:            0 (Meteorology)
  Originating Centre:         178 (Reserved)
  Originating Sub-Centre:       0 (ZZZZ Reserved)
  Update Sequence No:           0 (Original)
  Data  Category:               3 (Vertical soundings (satellite))
  Inter sub-Category:          50 (Radio Occultation sounding)
  Local sub-Category:          14
  Master Table version:        35 (INVALID VERSION VALUE)
  Local  Table version:         0 (not used)
  Message time:          20:58:36 28-Sep-2020

 BUFR Section 2
  Section 2 not present

 BUFR Section 3
  Length of section:           10 octets
  No. of obs. in Section 4:     1
  Message contains observed data
  Data is not compressed
  Descriptor   1: 310026

 BUFR Section 4
  Length of data:           14180 octets

 BUFR Section 5
  OK

 Data in Section 4:

 SATELLITE IDENTIFIER                                  Code 001007          269
 SATELLITE INSTRUMENTS                                 Code 002019          530
 IDENTIFICATION OF ORIGINATING/GENERATING CENTRE       Code 001033          178

comment:5 by Ian Culverwell, 3 years ago

ecCodes 2.20.1 (and other releases) can be found at

https://confluence.ecmwf.int/display/ECC/Releases

comment:6 by Ian Culverwell, 3 years ago

(Owen actually used ecCodes vn 2.22.0.)

comment:7 by Ian Culverwell, 3 years ago

The first job is to try to build the new ecCodes release with the ad-hoc fix to suppress voluminous output that we introduced in ROPP-10.0, and see if it still works.

It doesn't. After downloading eccodes-2.22.0-Source and applying ropp_io/tools/eccodes_patch to it, I find that the ropp_io core test output is flooded with > 2000 messages like

GRIB_LOG_DEBUG :  encode_double_value: softwareIdentification. Value (1.7e+38) out of range (minAllowed=0, maxAllowed=16383). Setting it to missing value
GRIB_LOG_DEBUG :  encode_double_value: percentConfidence. Value (1.7e+38) out of range (minAllowed=0, maxAllowed=127). Setting it to missing value
GRIB_LOG_DEBUG :  encode_double_value: DistanceFromEarthCentreInDirectionOf0DegreesLongitude. Value (1.7e+38) out of range (minAllowed=-1.07374e+07, maxAllowed=1.07374e+07). Setting it to missing value
GRIB_LOG_DEBUG :  encode_double_value: DistanceFromEarthCentreInDirection90DegreesEast. Value (1.7e+38) out of range (minAllowed=-1.07374e+07, maxAllowed=1.07374e+07). Setting it to missing value
GRIB_LOG_DEBUG :  encode_double_value: DistanceFromEarthCentreInDirectionOfNorthPole. Value (1.7e+38) out of range (minAllowed=-1.07374e+07, maxAllowed=1.07374e+07). Setting it to missing value
GRIB_LOG_DEBUG :  encode_double_value: absolutePlatformVelocityFirstComponent. Value (1.7e+38) out of range (minAllowed=-10737.4, maxAllowed=10737.4). Setting it to missing value
GRIB_LOG_DEBUG :  encode_double_value: absolutePlatformVelocitySecondComponent. Value (1.7e+38) out of range (minAllowed=-10737.4, maxAllowed=10737.4). Setting it to missing value
GRIB_LOG_DEBUG :  encode_double_value: absolutePlatformVelocityThirdComponent. Value (1.7e+38) out of range (minAllowed=-10737.4, maxAllowed=10737.4). Setting it to missing value
GRIB_LOG_DEBUG :  encode_double_value: DistanceFromEarthCentreInDirectionOf0DegreesLongitude. Value (1.7e+38) out of range (minAllowed=-1.07374e+08, maxAllowed=1.07374e+08). Setting it to missing value
GRIB_LOG_DEBUG :  encode_double_value: DistanceFromEarthCentreInDirection90DegreesEast. Value (1.7e+38) out of range (minAllowed=-1.07374e+08, maxAllowed=1.07374e+08). Setting it to missing value
GRIB_LOG_DEBUG :  encode_double_value: DistanceFromEarthCentreInDirectionOfNorthPole. Value (1.7e+38) out of range (minAllowed=-1.07374e+08, maxAllowed=1.07374e+08). Setting it to missing value
GRIB_LOG_DEBUG :  encode_double_value: absolutePlatformVelocityFirstComponent. Value (1.7e+38) out of range (minAllowed=-10737.4, maxAllowed=10737.4). Setting it to missing value
GRIB_LOG_DEBUG :  encode_double_value: absolutePlatformVelocitySecondComponent. Value (1.7e+38) out of range (minAllowed=-10737.4, maxAllowed=10737.4). Setting it to missing value
GRIB_LOG_DEBUG :  encode_double_value: absolutePlatformVelocityThirdComponent. Value (1.7e+38) out of range (minAllowed=-10737.4, maxAllowed=10737.4). Setting it to missing value
GRIB_LOG_DEBUG :  encode_double_value: bearingOrAzimuth. Value (1.7e+38) out of range (minAllowed=0, maxAllowed=655.35). Setting it to missing value
GRIB_LOG_DEBUG :  encode_double_value: bendingAngle. Value (1.7e+38) out of range (minAllowed=-0.001, maxAllowed=0.00948575). Setting it to missing value
GRIB_LOG_DEBUG :  encode_double_value: firstOrderStatistics. Value (1.7e+38) out of range (minAllowed=0, maxAllowed=63). Setting it to missing value
GRIB_LOG_DEBUG :  encode_double_value: bendingAngle. Value (1.7e+38) out of range (minAllowed=-0.001, maxAllowed=0.00948575). Setting it to missing value
GRIB_LOG_DEBUG :  encode_double_value: firstOrderStatistics. Value (1.7e+38) out of range (minAllowed=0, maxAllowed=63). Setting it to missing value
.
.
.

In other words, GRIB_LOG_DEBUG diagnostics are no longer suppressed by default, as they apparently were in eccodes-2.12.5-Source. This is probably enough to deter users from using this ecCodes library.

Ask ECMWF if there's anything we can do to suppress the GRIB_LOG_DEBUG output.

comment:8 by Ian Culverwell, 3 years ago

Shahram Najm from ECMWF helped a lot. For the earlier library, eccodes-2.12.5-Source, error messages were produced by lines like

grib_context_log(c, GRIB_LOG_ERROR, "encode_double_array: %s. Value (%g) out of range (minAllowed=%g, maxAllowed=%g)."

and so they could be suppressed by replacing the above by

grib_context_log(c, GRIB_LOG_DEBUG, "encode_double_array: %s. Value (%g) out of range (minAllowed=%g, maxAllowed=%g)."

But at eccodes-2.22.0-Source the error messages are produced differently, by lines like

fprintf(stderr,
  "ECCODES WARNING :  encode_double_array: %s. Value (%g) out of range (minAllowed=%g, maxAllowed=%g)."

This means that the messages always appear on stderr. It turns out that the way to suppress these is to replace the above by

grib_context_log(c, GRIB_LOG_DEBUG,
  "GRIB_LOG_DEBUG :  encode_double_array: %s. Value (%g) out of range (minAllowed=%g, maxAllowed=%g)."

When we make his change to eccodes-2.22.0-Source/src/grib_accessor_class_bufr_data_array.c, we lose all the unnecessary GRIB_LOG_DEBUG messages when we test ropp_io, e.g.:

Testing BUFR encode/decode [EUM netCDF4 files, ecCodes library] ...


==============================================================================
1. eum2bufr_eccodes: test eum2bufr_eccodes by comparing against reference file
==============================================================================


--------------------------------------------
1a. Converting EUM netCDF4 file -> BUFR file
--------------------------------------------

../tools/eum2bufr_eccodes  ../data/eum_test.n4  -o eum_test_l.bfr

---------------------------------------------------------------------
                  EUMETSAT to BUFR (ecCodes) Encoder
---------------------------------------------------------------------

INFO (from eum2bufr_eccodes):  Reading  EUM data from ../data/eum_test.n4
INFO (from ropp_io_read_eumdata):  Format Version 11.0
INFO (from eum2bufr_eccodes):  Encoding profile    1 : OC_20120909000057_META_G015_EUME


-------------------------------------------------
1b. Comparing output BUFR file and reference file
-------------------------------------------------

Using cmp to compare eum_test_l.bfr and ../data/eum_test_l.bfr_ref
 
 *** No differences found in test BUFR files
 
************************************
*** eum2bufr_eccodes  test: PASS ***
************************************

Good. First thing solved.

comment:9 by Ian Culverwell, 3 years ago

Note that we have problems installing eccodes from an existing tarball. We get error messages like

CMake Error at cmake_install.cmake:41 (file):
  file INSTALL cannot copy file
  "/data/users/idculv/ROPP/ropp_src/branches/dev/Share/ROPP110_prototype/eccodes-2.22.0-Source/tmpbuild/eccodes_ecbuild_config.h"
  to "/usr/local/include/eccodes_ecbuild_config.h": Permission denied.

which is presumably because we have

include dir    : [/usr/local/include]

in the log file.

This is true even if we replace

make install

by

make install prefix=$PREFIX

in all the MCSs.

Easiest just to add a comment to the release notes saying that we advise a fresh checkout of eccodes each time.

comment:10 by Ian Culverwell, 3 years ago

The current ROPP-11 prototype works OK with eccodes-2.22.0 and with

  INTEGER, PARAMETER :: VerMasTable = 12      ! Table version number (12)

in ropp_io/bufr/ropp2bufr_mod.f90.

If we change this to

  INTEGER, PARAMETER :: VerMasTable = 35      ! Table version number (35)

then:

  • mobufr has problems in testing because
    1. Encoding ROPP netCDF -> BUFR
    ===============================
    
    ---------------------------------------------------------------------
                      ROPP to BUFR (MetDB) Encoder
    ---------------------------------------------------------------------
    
    INFO (from ropp2bufr):  Reading  ROPP data from ../data/ropp_test.nc
    INFO (from ropp2bufr):  Encoding profile    1 : OC_20090817215807_META_G027_DMI_
    INFO (from ropp2bufr):  Total of  14742 bytes written to bufr_test.bfr
    INFO (from ropp2bufr):  Generated 1 BUFR message to bufr_test.bfr
    
    
    2. Comparing BUFR with reference file
    =====================================
    
    ../data/ropp_test.bfr bufr_test.bfr differ: char 22, line 1
     *** Differences found in test BUFR files (continuing test)
    

although it formally it passes the test:

3. Decoding BUFR --> ROPP netCDF
================================


---------------------------------------------------------------------
                  BUFR (MetDB) to ROPP Decoder
---------------------------------------------------------------------

INFO (from bufr2ropp):  Reading bufr_test.bfr
INFO (from bufr2ropp):  Decoded profile    1 : OC_20090817215807_META_G027_DMI_
INFO (from bufr2ropp):  Writing bufr_test.nc
INFO (from bufr2ropp):        1 new profile saved


4. Comparing CDL dump with reference file
=========================================

/project/ukmo/rhel7/R/R-3.6.3/bin/ncdump ../data/ropp_testb.nc > ropp_test.cdl
/project/ukmo/rhel7/R/R-3.6.3/bin/ncdump bufr_test.nc > bufr_test.cdl

To check results, compare ropp_test.cdl and bufr_test.cdl
(Ignore the difference in file names and processing_date value)

*** Differences found; please review bufr_test.dif to check significance.
    Differences of +/-1 in the last decimal place can be expected and
    these are not considered to be significant.

but only because the Master Table version is not encoded in ROPP netCDF files. (Byte 22 of a BUFR message is indeed the Master Table Version.)

  • ecbufr has more serious problems, because
    1. Encoding ROPP netCDF -> BUFR
    ===============================
     
    ERROR (from EncodeBUFR):  Error generating BUFR message
    
    ---------------------------------------------------------------------
                        ROPP to BUFR (ecbufr) Encoder
    ---------------------------------------------------------------------
    
    INFO (from ropp2bufr):  Reading  ROPP data from ../data/ropp_test.nc
    INFO (from ropp2bufr):  Encoding profile    1 : OC_20090817215807_META_G027_DMI_
     open error on /data/users/idculv/ROPP/data/bufr/B0000000000000035000.TXT
     
    WARNING (from ropp2bufr):  No profiles were encoded or written to the BUFR file
    
    *** Failed to generate a BUFR file from netCDF
    
    ***********************
    *** BUFR test: FAIL ***
    ***********************
    

(We only have /data/users/idculv/ROPP/data/bufr/B0000000000000007000.TXT to /data/users/idculv/ROPP/data/bufr/B0000000000000014000.TXT (and most of them are empty).)

  • ecCodes still has problems:
    ==============================================================================
    1. eum2bufr_eccodes: test eum2bufr_eccodes by comparing against reference file
    ==============================================================================
    
    
    --------------------------------------------
    1a. Converting EUM netCDF4 file -> BUFR file
    --------------------------------------------
    
    ../tools/eum2bufr_eccodes  ../data/eum_test.n4  -o eum_test_l.bfr
    
    ---------------------------------------------------------------------
                      EUMETSAT to BUFR (ecCodes) Encoder
    ---------------------------------------------------------------------
    
    INFO (from eum2bufr_eccodes):  Reading  EUM data from ../data/eum_test.n4
    INFO (from ropp_io_read_eumdata):  Format Version 11.0
    INFO (from eum2bufr_eccodes):  Encoding profile    1 : OC_20120909000057_META_G015_EUME
    
    
    -------------------------------------------------
    1b. Comparing output BUFR file and reference file
    -------------------------------------------------
    
    Using cmp to compare eum_test_l.bfr and ../data/eum_test_l.bfr_ref
    eum_test_l.bfr ../data/eum_test_l.bfr_ref differ: char 22, line 1
     
     *** Differences found in test BUFR files
     
    ************************************
    *** eum2bufr_eccodes  test: FAIL ***
    ************************************
    

(This is more of a problem for ecCodes than it was for mobufr because the former compares test and ref with cmp, whereas the latter passes the output through bufr2ropp and then cmps the CDL dumps of those, and Master Table Version is not part of the ROPP files. We should probably test eum2bufr_eccodes the same way.)

We could probably fix the ecCodes problem by updating the reference file. But the ecbufr problem will remain, and there's no way of solving it because the ecbufr lib is now frozen.

Suggested solutions:

  • Leave Master Table Version as 12 in ROPP-11, but include commented-out Version 35 code;
  • Explain in the Release Notes that users can use Master Table 35 code, but they will need a recently updated BUFR lib to use it (e.g. ecCodes 2.22.0), and that doing so will cause a failure in the eccodes core tests;
  • Replace ecCodes 2.15.5 by 2.22.0 as the recommended ecCodes version;
  • Longer term (ROPP-11.1): test ropp2bufr_mobufr, ropp2bufr_ecbufr and ropp2bufr_eccodes separately, as part of the general ropp_io core test revision (see #273). This will allow us to use different reference files for each exec. Then, if we can include the Master Table version as a variable in the mobufr/ecbufr/eccodes codes themselves, rather than as a parameter in the ropp2bufr module, we can use different Master Tables for each BUFR library. This would be the most straightforward to update in future.

comment:11 by Ian Culverwell, 3 years ago

Code changes to support use of ecCodes_2.22.0 in ROPP-11 have been implemented at r6904; add note to ropp2bufr_mod.f90 advising use of VerMasTable = 35 if the user's BUFR library can handle it, at r6905; Change Log updated at r6906; Release Notes updated at r6907.

comment:12 by Ian Culverwell, 3 years ago

There are connections to this work at #273 and #688, and links from them to this ticket have been made.

comment:13 by Ian Culverwell, 3 years ago

eccodes_patch has been updated at r6908 to handle the ecCodes_2.22.0 version of grib_accessor_class_bufr_data_array.c.

comment:14 by Ian Culverwell, 3 years ago

Resolution: fixed
Status: newclosed

I cannot do any more on this now. Further work will take place in #688 and #273. Closing ticket.

Note: See TracTickets for help on using tickets.