Opened 4 years ago
Closed 3 years ago
#699 closed task (fixed)
Errors in RO BUFR descriptors
Reported by: | Ian Culverwell | Owned by: | Ian Culverwell |
---|---|---|---|
Priority: | major | Milestone: | 11.0 |
Component: | ropp_io | Version: | 10.0 |
Keywords: | BUFR | Cc: |
Description
Two inconsistencies in the ROM SAF RO BUFR document (current version, vn 2.5, June 2019 is attached) have been discovered by Jeff Ator at NOAA and Doug Hunt of UCAR:
- The Master Version Table Number in Table 3 (octet 14 of BUFR Section 1) is incorrectly set to 12. It should be set to 35, since new satellite IDs (under descriptor 0-01-007 in Table B) and new satellite instruments (under descriptor 0-02-019 in Table B) have been added for Spire and GeoOptics data, but these did not become fully valid at the WMO level until version 35 of BUFR Table B.
- Bits 8-10 of the RO quality flag in Table 9 (descriptor 0 33 039) have been given the same meanings that the equivalent variable, PCD, has been given in ROPP. But these meanings have not yet been endorsed by WMO. https://github.com/wmo-im/BUFR4/blob/master/BUFRCREX_CodeFlag_en_33.csv shows that they are currently 'reserved', i.e. undefined.
See https://trac.romsaf.org/wiki/ticket/332 for further details.
The ROM SAF BUFR document has been updated at r6588.
These changes, permanent in the first case and (we hope) temporary in the second, will affect ropp2bufr and bufr2ropp and their ecodes
cousins. These should be fixed urgently.
Attachments (2)
Change history (16)
comment:1 by , 3 years ago
comment:2 by , 3 years ago
I have built an updated version of eccodes using cmake3. The version used is eccodes-2.22 which contains the tables for MasterTable 35. This has required updating some of the configuration files to enable building using cmake3. I have tested this new version with Spire files and this is detailed in ticket #702 as this involved updating the namelist file.
The change has been tested with a a few Spire bufr files which have been attached to this ticket.
To test the Spire bufr file has been converted to an ropp file using the bufr2ropp routine and then this file is converted back to a bufr file using ropp2bufr and the orignal bufr file is compared to the new bufr file.
To decode the bufr files the eccodes dump_bufr routine was used.
With Spire file "bfrPrf_2020-09-28T20-58-34Z.41754962.101.G32.bufr" there were two differences in the final outputted bufr and the original bufr. First there was the masterTableVersionNumber which was 12 from the orignal Spire bufr and is now 35 as the change has aimed to do
"key" : "masterTablesVersionNumber", "value" : 35
Secondly there is a noncoordinatePressure value. In the Spire file this appears as 1010 hPa but at a geopotential height of -1000 (at the very end of file, ordered ascending) which in the outputted bufr file the pressure is reset to NULL. This behavior seems to be consistent across the Spire files with the very upper levels having values substituted with null.
The netcdf ropp files produced by the control without the new namelist and produced by the new namelist were also compared the results of this are shown below for the same spire bufr file as above. With new namelist
// global attributes: :title = "ROPP Radio Occultation data" ; :institution = "SPIRE Spire Global, Inc." ; :Conventions = "CF-1.0" ; :format_version = "ROPP I/O V1.1" ; :processing_centre = "SPIRE Spire Global, Inc." ; :processing_software = "UNKNOWN" ; :processing_date = "2021-07-19 10:08:42.351" ; :pod_method = "UNKNOWN" ; :phase_method = "UNKNOWN" ; :bangle_method = "UNKNOWN" ; :refrac_method = "UNKNOWN" ; :meteo_method = "UNKNOWN" ; :thin_method = "UNKNOWN" ; :software_version = "V00.001" ; :_FillValue = -99999000. ; data: occ_id = "OC_20200928205835_SP3U_G032_SPIR" ; gns_id = "G032" ; leo_id = "SP3U" ; stn_id = "UNKN" ;
Control
// global attributes: :title = "ROPP Radio Occultation data" ; :institution = "UNKNOWN" ; :Conventions = "CF-1.0" ; :format_version = "ROPP I/O V1.1" ; :processing_centre = "UNKNOWN" ; :processing_software = "UNKNOWN" ; :processing_date = "2021-07-19 10:08:41.885" ; :pod_method = "UNKNOWN" ; :phase_method = "UNKNOWN" ; :bangle_method = "UNKNOWN" ; :refrac_method = "UNKNOWN" ; :meteo_method = "UNKNOWN" ; :thin_method = "UNKNOWN" ; :software_version = "V00.001" ; :_FillValue = -99999000. ; data: occ_id = "OC_20200928205835_UNKN_G032_UNKN" ; gns_id = "G032" ; leo_id = "UNKN" ; stn_id = "UNKN" ;
The final comparison is between the bufr outputted by the current setup without changing the namelist or updating the eccodes build. The differences between the spire bufr files created by bufr2ropp followed by ropp2bufr are shown below. New output
{ "key" : "bufrHeaderCentre", "value" : 178 }, { "key" : "masterTablesVersionNumber", "value" : 35 }, { "key" : "satelliteIdentifier", "value" : 269, "units" : "CODE TABLE" }, [ { "key" : "satelliteInstruments", "value" : 530, "units" : "CODE TABLE" }, [ { "key" : "centre", "value" : 178, "units" : "Common CODE TABLE C-1" }, { "key" : "geopotentialHeight", "value" : -1000, "units" : "gpm" }, { "key" : "nonCoordinatePressure", "value" : null, "units" : "Pa" }
Control
{ "key" : "bufrHeaderCentre", "value" : 65535 }, { "key" : "masterTablesVersionNumber", "value" : 12 }, { "key" : "satelliteIdentifier", "value" : null, "units" : "CODE TABLE" }, [ { "key" : "satelliteInstruments", "value" : null, "units" : "CODE TABLE" }, [ { "key" : "centre", "value" : null, "units" : "CODE TABLE" }, { "key" : "geopotentialHeight", "value" : -1000, "units" : "gpm" }, { "key" : "nonCoordinatePressure", "value" : null,
Both handle the odd value at the end of the file in the same way.
by , 3 years ago
Attachment: | bfrPrf_2020-09-28T20-55-40Z.41739859.100.G27.bufr added |
---|
by , 3 years ago
Attachment: | bfrPrf_2020-09-28T20-59-21Z.41811642.107.E05.bufr added |
---|
comment:3 by , 3 years ago
I now understand the pressure value at the end of the file is actually the Spire satellite id. Currently there is no designated area within the BUFR for storing this extra information and so a pressure coordinate has been used. The satellite id is multiplied by 10 to conform to the expected values of the pressure coordinate. So in the above example commented on the satellite id would 100.
I am not sure yet how we can handle this in the ropp2bufr and bufr2ropp routines.
comment:4 by , 3 years ago
With the Met Office MetDb BUFR library this change also works.
The current MetDB library does not contain Master Table Version 35 but by updating the roppbufrcodes.nl namelist the MO bufr conversion code can also work with version 35. The only difference in the decoded bufr from tests run on building ropp_io is the Master version table number changing from 12 to 35.
As for the eccodes this has also been tested with the Spire bufr files and the same results are seen with the MO bufr as was seen with eccodes.
Original MO decoded bufr output (v12)
------------------------------------------------------- Met Office Generic BUFR Decoder ------------------------------------------------------- Reading /data/users/olewis/ropp_n699_results/bufr_mo_control_test_ifort.bfr ===== Message no. 1 ===== BUFR Section 0 Length of section: 8 octets Total Message length: 14224 octets BUFR Edition: 4 BUFR Section 1 Length of section: 22 octets BUFR Master Table: 0 (Meteorology) Originating Centre: 65535 (INVALID CENTRE CODE VALUE) Originating Sub-Centre: 0 (INVALID CENTRE CODE VALUE) Update Sequence No: 0 (Original) Data Category: 3 (Vertical soundings (satellite)) Inter sub-Category: 50 (Radio Occultation sounding) Local sub-Category: 14 Master Table version: 12 (2 November 2005) Local Table version: 0 (not used) Message time: 20:58:36 28-Sep-2020 BUFR Section 2 Section 2 not present BUFR Section 3 Length of section: 10 octets No. of obs. in Section 4: 1 Message contains observed data Data is not compressed Descriptor 1: 310026 BUFR Section 4 Length of data: 14180 octets BUFR Section 5 OK Data in Section 4: SATELLITE IDENTIFIER Code 001007 - - - - - SATELLITE INSTRUMENTS Code 002019 - - - - - IDENTIFICATION OF ORIGINATING/GENERATING CENTRE Code 001033 - - - - -
decoded bufr (bufr2ropp ropp2bufr on Spire bufr file)
------------------------------------------------------- Met Office Generic BUFR Decoder ------------------------------------------------------- Reading /data/users/olewis/ropp_n699_results/roppbufr_mo_test_ifort.bfr ===== Message no. 1 ===== BUFR Section 0 Length of section: 8 octets Total Message length: 14224 octets BUFR Edition: 4 BUFR Section 1 Length of section: 22 octets BUFR Master Table: 0 (Meteorology) Originating Centre: 178 (Reserved) Originating Sub-Centre: 0 (ZZZZ Reserved) Update Sequence No: 0 (Original) Data Category: 3 (Vertical soundings (satellite)) Inter sub-Category: 50 (Radio Occultation sounding) Local sub-Category: 14 Master Table version: 35 (INVALID VERSION VALUE) Local Table version: 0 (not used) Message time: 20:58:36 28-Sep-2020 BUFR Section 2 Section 2 not present BUFR Section 3 Length of section: 10 octets No. of obs. in Section 4: 1 Message contains observed data Data is not compressed Descriptor 1: 310026 BUFR Section 4 Length of data: 14180 octets BUFR Section 5 OK Data in Section 4: SATELLITE IDENTIFIER Code 001007 269 SATELLITE INSTRUMENTS Code 002019 530 IDENTIFICATION OF ORIGINATING/GENERATING CENTRE Code 001033 178
comment:7 by , 3 years ago
The first job is to try to build the new ecCodes release with the ad-hoc fix to suppress voluminous output that we introduced in ROPP-10.0, and see if it still works.
It doesn't. After downloading eccodes-2.22.0-Source and applying ropp_io/tools/eccodes_patch to it, I find that the ropp_io
core test output is flooded with > 2000 messages like
GRIB_LOG_DEBUG : encode_double_value: softwareIdentification. Value (1.7e+38) out of range (minAllowed=0, maxAllowed=16383). Setting it to missing value GRIB_LOG_DEBUG : encode_double_value: percentConfidence. Value (1.7e+38) out of range (minAllowed=0, maxAllowed=127). Setting it to missing value GRIB_LOG_DEBUG : encode_double_value: DistanceFromEarthCentreInDirectionOf0DegreesLongitude. Value (1.7e+38) out of range (minAllowed=-1.07374e+07, maxAllowed=1.07374e+07). Setting it to missing value GRIB_LOG_DEBUG : encode_double_value: DistanceFromEarthCentreInDirection90DegreesEast. Value (1.7e+38) out of range (minAllowed=-1.07374e+07, maxAllowed=1.07374e+07). Setting it to missing value GRIB_LOG_DEBUG : encode_double_value: DistanceFromEarthCentreInDirectionOfNorthPole. Value (1.7e+38) out of range (minAllowed=-1.07374e+07, maxAllowed=1.07374e+07). Setting it to missing value GRIB_LOG_DEBUG : encode_double_value: absolutePlatformVelocityFirstComponent. Value (1.7e+38) out of range (minAllowed=-10737.4, maxAllowed=10737.4). Setting it to missing value GRIB_LOG_DEBUG : encode_double_value: absolutePlatformVelocitySecondComponent. Value (1.7e+38) out of range (minAllowed=-10737.4, maxAllowed=10737.4). Setting it to missing value GRIB_LOG_DEBUG : encode_double_value: absolutePlatformVelocityThirdComponent. Value (1.7e+38) out of range (minAllowed=-10737.4, maxAllowed=10737.4). Setting it to missing value GRIB_LOG_DEBUG : encode_double_value: DistanceFromEarthCentreInDirectionOf0DegreesLongitude. Value (1.7e+38) out of range (minAllowed=-1.07374e+08, maxAllowed=1.07374e+08). Setting it to missing value GRIB_LOG_DEBUG : encode_double_value: DistanceFromEarthCentreInDirection90DegreesEast. Value (1.7e+38) out of range (minAllowed=-1.07374e+08, maxAllowed=1.07374e+08). Setting it to missing value GRIB_LOG_DEBUG : encode_double_value: DistanceFromEarthCentreInDirectionOfNorthPole. Value (1.7e+38) out of range (minAllowed=-1.07374e+08, maxAllowed=1.07374e+08). Setting it to missing value GRIB_LOG_DEBUG : encode_double_value: absolutePlatformVelocityFirstComponent. Value (1.7e+38) out of range (minAllowed=-10737.4, maxAllowed=10737.4). Setting it to missing value GRIB_LOG_DEBUG : encode_double_value: absolutePlatformVelocitySecondComponent. Value (1.7e+38) out of range (minAllowed=-10737.4, maxAllowed=10737.4). Setting it to missing value GRIB_LOG_DEBUG : encode_double_value: absolutePlatformVelocityThirdComponent. Value (1.7e+38) out of range (minAllowed=-10737.4, maxAllowed=10737.4). Setting it to missing value GRIB_LOG_DEBUG : encode_double_value: bearingOrAzimuth. Value (1.7e+38) out of range (minAllowed=0, maxAllowed=655.35). Setting it to missing value GRIB_LOG_DEBUG : encode_double_value: bendingAngle. Value (1.7e+38) out of range (minAllowed=-0.001, maxAllowed=0.00948575). Setting it to missing value GRIB_LOG_DEBUG : encode_double_value: firstOrderStatistics. Value (1.7e+38) out of range (minAllowed=0, maxAllowed=63). Setting it to missing value GRIB_LOG_DEBUG : encode_double_value: bendingAngle. Value (1.7e+38) out of range (minAllowed=-0.001, maxAllowed=0.00948575). Setting it to missing value GRIB_LOG_DEBUG : encode_double_value: firstOrderStatistics. Value (1.7e+38) out of range (minAllowed=0, maxAllowed=63). Setting it to missing value . . .
In other words, GRIB_LOG_DEBUG
diagnostics are no longer suppressed by default, as they apparently were in eccodes-2.12.5-Source. This is probably enough to deter users from using this ecCodes library.
Ask ECMWF if there's anything we can do to suppress the GRIB_LOG_DEBUG
output.
comment:8 by , 3 years ago
Shahram Najm from ECMWF helped a lot. For the earlier library, eccodes-2.12.5-Source, error messages were produced by lines like
grib_context_log(c, GRIB_LOG_ERROR, "encode_double_array: %s. Value (%g) out of range (minAllowed=%g, maxAllowed=%g)."
and so they could be suppressed by replacing the above by
grib_context_log(c, GRIB_LOG_DEBUG, "encode_double_array: %s. Value (%g) out of range (minAllowed=%g, maxAllowed=%g)."
But at eccodes-2.22.0-Source the error messages are produced differently, by lines like
fprintf(stderr, "ECCODES WARNING : encode_double_array: %s. Value (%g) out of range (minAllowed=%g, maxAllowed=%g)."
This means that the messages always appear on stderr. It turns out that the way to suppress these is to replace the above by
grib_context_log(c, GRIB_LOG_DEBUG, "GRIB_LOG_DEBUG : encode_double_array: %s. Value (%g) out of range (minAllowed=%g, maxAllowed=%g)."
When we make his change to eccodes-2.22.0-Source/src/grib_accessor_class_bufr_data_array.c, we lose all the unnecessary GRIB_LOG_DEBUG
messages when we test ropp_io, e.g.:
Testing BUFR encode/decode [EUM netCDF4 files, ecCodes library] ... ============================================================================== 1. eum2bufr_eccodes: test eum2bufr_eccodes by comparing against reference file ============================================================================== -------------------------------------------- 1a. Converting EUM netCDF4 file -> BUFR file -------------------------------------------- ../tools/eum2bufr_eccodes ../data/eum_test.n4 -o eum_test_l.bfr --------------------------------------------------------------------- EUMETSAT to BUFR (ecCodes) Encoder --------------------------------------------------------------------- INFO (from eum2bufr_eccodes): Reading EUM data from ../data/eum_test.n4 INFO (from ropp_io_read_eumdata): Format Version 11.0 INFO (from eum2bufr_eccodes): Encoding profile 1 : OC_20120909000057_META_G015_EUME ------------------------------------------------- 1b. Comparing output BUFR file and reference file ------------------------------------------------- Using cmp to compare eum_test_l.bfr and ../data/eum_test_l.bfr_ref *** No differences found in test BUFR files ************************************ *** eum2bufr_eccodes test: PASS *** ************************************
Good. First thing solved.
comment:9 by , 3 years ago
Note that we have problems installing eccodes from an existing tarball. We get error messages like
CMake Error at cmake_install.cmake:41 (file): file INSTALL cannot copy file "/data/users/idculv/ROPP/ropp_src/branches/dev/Share/ROPP110_prototype/eccodes-2.22.0-Source/tmpbuild/eccodes_ecbuild_config.h" to "/usr/local/include/eccodes_ecbuild_config.h": Permission denied.
which is presumably because we have
include dir : [/usr/local/include]
in the log file.
This is true even if we replace
make install
by
make install prefix=$PREFIX
in all the MCSs.
Easiest just to add a comment to the release notes saying that we advise a fresh checkout of eccodes each time.
comment:10 by , 3 years ago
The current ROPP-11 prototype works OK with eccodes-2.22.0 and with
INTEGER, PARAMETER :: VerMasTable = 12 ! Table version number (12)
in ropp_io/bufr/ropp2bufr_mod.f90.
If we change this to
INTEGER, PARAMETER :: VerMasTable = 35 ! Table version number (35)
then:
- mobufr has problems in testing because
1. Encoding ROPP netCDF -> BUFR =============================== --------------------------------------------------------------------- ROPP to BUFR (MetDB) Encoder --------------------------------------------------------------------- INFO (from ropp2bufr): Reading ROPP data from ../data/ropp_test.nc INFO (from ropp2bufr): Encoding profile 1 : OC_20090817215807_META_G027_DMI_ INFO (from ropp2bufr): Total of 14742 bytes written to bufr_test.bfr INFO (from ropp2bufr): Generated 1 BUFR message to bufr_test.bfr 2. Comparing BUFR with reference file ===================================== ../data/ropp_test.bfr bufr_test.bfr differ: char 22, line 1 *** Differences found in test BUFR files (continuing test)
although it formally it passes the test:
3. Decoding BUFR --> ROPP netCDF ================================ --------------------------------------------------------------------- BUFR (MetDB) to ROPP Decoder --------------------------------------------------------------------- INFO (from bufr2ropp): Reading bufr_test.bfr INFO (from bufr2ropp): Decoded profile 1 : OC_20090817215807_META_G027_DMI_ INFO (from bufr2ropp): Writing bufr_test.nc INFO (from bufr2ropp): 1 new profile saved 4. Comparing CDL dump with reference file ========================================= /project/ukmo/rhel7/R/R-3.6.3/bin/ncdump ../data/ropp_testb.nc > ropp_test.cdl /project/ukmo/rhel7/R/R-3.6.3/bin/ncdump bufr_test.nc > bufr_test.cdl To check results, compare ropp_test.cdl and bufr_test.cdl (Ignore the difference in file names and processing_date value) *** Differences found; please review bufr_test.dif to check significance. Differences of +/-1 in the last decimal place can be expected and these are not considered to be significant.
but only because the Master Table version is not encoded in ROPP netCDF files. (Byte 22 of a BUFR message is indeed the Master Table Version.)
- ecbufr has more serious problems, because
1. Encoding ROPP netCDF -> BUFR =============================== ERROR (from EncodeBUFR): Error generating BUFR message --------------------------------------------------------------------- ROPP to BUFR (ecbufr) Encoder --------------------------------------------------------------------- INFO (from ropp2bufr): Reading ROPP data from ../data/ropp_test.nc INFO (from ropp2bufr): Encoding profile 1 : OC_20090817215807_META_G027_DMI_ open error on /data/users/idculv/ROPP/data/bufr/B0000000000000035000.TXT WARNING (from ropp2bufr): No profiles were encoded or written to the BUFR file *** Failed to generate a BUFR file from netCDF *********************** *** BUFR test: FAIL *** ***********************
(We only have /data/users/idculv/ROPP/data/bufr/B0000000000000007000.TXT to /data/users/idculv/ROPP/data/bufr/B0000000000000014000.TXT (and most of them are empty).)
- ecCodes still has problems:
============================================================================== 1. eum2bufr_eccodes: test eum2bufr_eccodes by comparing against reference file ============================================================================== -------------------------------------------- 1a. Converting EUM netCDF4 file -> BUFR file -------------------------------------------- ../tools/eum2bufr_eccodes ../data/eum_test.n4 -o eum_test_l.bfr --------------------------------------------------------------------- EUMETSAT to BUFR (ecCodes) Encoder --------------------------------------------------------------------- INFO (from eum2bufr_eccodes): Reading EUM data from ../data/eum_test.n4 INFO (from ropp_io_read_eumdata): Format Version 11.0 INFO (from eum2bufr_eccodes): Encoding profile 1 : OC_20120909000057_META_G015_EUME ------------------------------------------------- 1b. Comparing output BUFR file and reference file ------------------------------------------------- Using cmp to compare eum_test_l.bfr and ../data/eum_test_l.bfr_ref eum_test_l.bfr ../data/eum_test_l.bfr_ref differ: char 22, line 1 *** Differences found in test BUFR files ************************************ *** eum2bufr_eccodes test: FAIL *** ************************************
(This is more of a problem for ecCodes than it was for mobufr because the former compares test and ref with cmp
, whereas the latter passes the output through bufr2ropp and then cmp
s the CDL dumps of those, and Master Table Version is not part of the ROPP files. We should probably test eum2bufr_eccodes the same way.)
We could probably fix the ecCodes problem by updating the reference file. But the ecbufr problem will remain, and there's no way of solving it because the ecbufr lib is now frozen.
Suggested solutions:
- Leave Master Table Version as 12 in ROPP-11, but include commented-out Version 35 code;
- Explain in the Release Notes that users can use Master Table 35 code, but they will need a recently updated BUFR lib to use it (e.g. ecCodes 2.22.0), and that doing so will cause a failure in the eccodes core tests;
- Replace ecCodes 2.15.5 by 2.22.0 as the recommended ecCodes version;
- Longer term (ROPP-11.1): test ropp2bufr_mobufr, ropp2bufr_ecbufr and ropp2bufr_eccodes separately, as part of the general ropp_io core test revision (see #273). This will allow us to use different reference files for each exec. Then, if we can include the Master Table version as a variable in the mobufr/ecbufr/eccodes codes themselves, rather than as a parameter in the
ropp2bufr
module, we can use different Master Tables for each BUFR library. This would be the most straightforward to update in future.
comment:11 by , 3 years ago
comment:12 by , 3 years ago
comment:13 by , 3 years ago
eccodes_patch has been updated at r6908 to handle the ecCodes_2.22.0 version of grib_accessor_class_bufr_data_array.c.
comment:14 by , 3 years ago
Resolution: | → fixed |
---|---|
Status: | new → closed |
To update the Master version table the values assigned is changed in the ropp_io/bufr/ropp2bufr_mod.f90 file from the current value of 12 to 35.
For ropp2bufr_eccodes the current version of eccodes does not work with Master Table 35. By changing the value to 35 and running ropp2bufr_eccodes the following output is given.
With the current version of eccodes used (eccodes-2.12.5-Source) the latest Master table available is 31. To get the Master Table 35 the version required is eccodes-2.19, the current most up to date version of eccodes is eccodes-22. To build these newer versions requires CMake3.
For the MO Bufr when you update the Master table to 35 the conversion passes and the value is included in the decoded BUFR file. However it states that the value is not valid.
On looking into the MO BUFR package it has been found that the current version being used is bufr-24.0.2a and this contains Master tables 20-26. The next available MO bufr package v25 contains upto master table 31.