Default BUFR ARH?

[Dave Offiler]

Rune has been testing the DRI Candidate Release v4.1 and reports an anomaly in the v4.1 ropp2bufr tool wrt the currently operational v3.0 version:

I've run the GRAS SAF production on a days occulations using 
ROPP-3 and ROPP-4.1 . The are some differences in the output netcdf
files that I guess is due to going from float to doubles. I've
tried to compare output bufr files generated using ropp2bufr using
input netcdf ('dis'-)files.

When I run the old ropp2bufr on both the old and new dis-files
there is no difference between the output bufr files. It is the
same when I run the new version on both files. But the bufr output
files of ropp2bufr version 3,0 and the ropp2bufr version 4.1 doesn't compare well.

Looking at the binary dump it seems that the GTS header data is now
added as default ? I can see the 'IUTL14 EKMI 050812' header in the
bufr  output files.

The intent is that -g or -gi switches are required to generate any abbreviated routing header (ARH) (default: no ARH generated), and -c nn (nn=BUFR code for originating centre) to set the ICAO code in the header (default nn=74 for 'EGRR').

So code needs checking to see why this non-default default ARH is being generated. From memory I don't recall any deliberate change to the way headers are generated between v3.0 and v4.0, and onlt DateTime changes have been made to v4.1.

Since the ARH Rune quotes are correct and required for operational BUFR, this is consisderd a minor anomoly, though ought to be closed for v4.1 release.

[Dave Offiler]

Rune also notes binary differences in the BUFR when no ARH is generated, and has provided the input netCDF file, BUFR generated with v3.0 ('old') and v4.1 ('new') ropp2bufr versions and decbufr dumps of both.

Inspection of the dump files reveals (in email reply to Rune):

Lets first look at the headers (I've highlighted the differences in the 'new' 
dump with <<<<<< and a minor issue with +++++):


old_bufr.txt:
=============

 BUFR Section 0
  Total BUFR Message length: 12594 octets
  Edition: 3

 BUFR Section 1
  Length of section:         18 octets
  BUFR Master Table:          0 (TABLE MASTERTABLE NOT LOADED)
  Originating Centre:        94 (TABLE ORIGCENTRES NOT LOADED)
  Originating Sub-Centre:     0 (none)
  Update Sequence No:         0 (Original)
  Data  Category:             3 (TABLE DATACATEGORY NOT LOADED)
  Inter sub-Category:       255 (not present in Edition 3)
  Local sub-Category:       255
  Master Table version:      12 (Reserved)
  Local  Table version:       1 
  Message time:        00:09:00 05-Jul-2010

 BUFR Section 2
  Length of section: 0 octets
  Section 2 not present

 BUFR Section 3
  Length of section: 10 octets
  No. of observations in Section 4: 1       
  Message contains observed data
  Data is not compressed
  Descriptor   1: 310026

 BUFR Section 4
  Length of data: 12554 octets


new_bufr.txt
============

 BUFR Section 0
  Total BUFR Message length: 15588 octets <<<<<<<
  Edition: 3

 BUFR Section 1
  Length of section:         18 octets
  BUFR Master Table:          0 (TABLE MASTERTABLE NOT LOADED) ++++
  Originating Centre:        94 (TABLE ORIGCENTRES NOT LOADED) ++++
  Originating Sub-Centre:     0 (none)
  Update Sequence No:         0 (Original)
  Data  Category:             3 (TABLE DATACATEGORY NOT LOADED) ++++
  Inter sub-Category:       255 (not present in Edition 3)
  Local sub-Category:        14            <<<<<<<<<
  Master Table version:      13 (Reserved) <<<<<<<<< +++++
  Local  Table version:       0 (not used) <<<<<<<<<
  Message time:        00:09:00 05-Jul-2010


 BUFR Section 2
  Length of section: 0 octets
  Section 2 not present

 BUFR Section 3
  Length of section: 10 octets
  No. of observations in Section 4: 1       
  Message contains observed data
  Data is not compressed
  Descriptor   1: 310026

 BUFR Section 4
  Length of data: 15548 octets <<<<<<<<<

<<<<<< The differences in total and Section 4 data lengths, I'll come to later, but the 
Section 1 differences are easily explained; the WMO Expert Group on Table-Driven Codes 
clarified some ambiguities in the BUFR rules & tables at their meeting late last year and 
ruled that:

1) when a local table is not used (as is the case here), the value in this field should
   be zero, not one (DWD had previously insisted it should be 1, but I'd never thought
   this to be logical and the EG-TDC agreed with me!).

2) The (proposed new) Local Data Sub-Category 14 (=GPSRO) in place of 255 ('other'/
   'unknown').
   Strictly speaking, this value is currently in 'validation' and by the book 
   shouldn't be used for operational messages on the GTS until 'pre-operational' - 
   I was pre-empting  this getting approval by now, but since I don't think this stage 
   has been reached yet, I have just now reverted the value back to 255 until it is 
   allowed to be used (hopefully by ROPP v5) - so the v4.1 release version of
   bufr2ropp will be the same as v3.0 (=255) for this parameter.

3) The Master table version is updated to 13 from 12 reflecting the first time
   the GPSRO template was officially published in the BUFR table updates for 
   November 2007. Master Table 12 was temporary, and should have been changed
   long ago! (The current Master tables are at version 14, but the BUFR header 
   value is only changed if the new tables impact the data in this message, which
   isn't the case here.)

So the header differences are fully explained (because they don't usually change once 
defined in the BUFR tables the values are hard-coded in the f90). Hence you shouldn't 
expect bit-exactness between the v3 and v4 BUFR files.

As noted, diff (2) is now reverted until we get confirmation we can use a specific value 
for GPSRO - the next EG-TDC meeting is likely to be around October if it follows the pattern 
of recent years.

++++ The fact that several header parameters have notes about tables not loaded
suggests that while the v18.2/18.2 version of decbufr is being used here, it isn't finding
these run-time files. Can you check that when you installed BUFR v18.2/3 it installed to the
same run-time tables directory as the previous version (via BUFR_LIBRARY which might have been
changed before or during the new version installation). You should have files like MASTERTABLE,
DATACATEGORY and ORIGCENTRE (not ORIGCENTRES - that's a typo in the decbufr tool) in 
BUFR_LIBRARY when decbufr is run. Note that not having these files on decode doesn't present 
any problems to the data decoded - only the lack of decode text for the dump itself - but the 
encoder uses ORIGCENTRE for the ICAO code look-up in the ARH.

Finally, the different message lengths, which is confined to Section 4 (the actual GRAS data).
This is simple to explain: 'new' has a Level 2b (P,T,q) profile and 'old' doesn't. If you used
the same input file in both cases, and it contains valid L2b data, then why v4.1 encoded it and 
v3.0 didn't needs further investigation...

Finally-finally, in both old and new dumps, the Originating Centre code is 94 (which would 
decode to Copenhagen if the ORIGCENTRE table file had been found in BUFR_LIBRARY). The encoder
makes no assumptions (like: this is GRAS data, so it must be from Copenhagen), so 94 must have
been specified externally[1]. So for both encoders, at least '-c 94' must have been used on 
the command line, whether explicitly or implicitly via an alias. (neither raw BUFR file has
an ARH, so a -g option was not used in these cases). 

[1] I've re-checked that the ropp2bufr.f90 source code defines the default centre to be 
74/EGRR (Exeter), so the internal defaults must have been over-ridden when you did the 
encoding in both cases (assuming the source code defaults haven't been changed at DMI!)

So the only anomaly to be checked is why a L2b profile appears in v4.1 but not in v3.0 
(I'm sure v3.0 should encode L2b if present & valid - we didn't add this at v4.0). 
I'll continue to look at this aspect.

The originating centre code of 94 I can only explain by your encoder using the -c option,
but I'll check what I get with your sample file.

The header value for data sub-category is reverted to 255 with [2562].

Discussing the L2c data, Huw suggests that this may be a consequence of the change to always force 247 (or whatever) levels for the interpolation methods, even if the input no. of levels is smaller (including none). In the case of L2b%Npoints=0, a nominal set of geopotential heights will be generated with P,T,q set invalid/missing and Npoints=247. Since the BUFR encoder only skips a section if Npoints=0, it will encode 247 levels with missing data (but seemingly valid heights). This is indeed what is seen in Rune's 'new' dumps, so this is a likely reason for the difference.

If this is confirmed by running the same v3.0 and v4.0/v4.1 versons on Rune's netCDF file, then since the forced 247-level output is appropriate for netCDF output, but not for BUFR, the encoder should be modified test for all P,T,q missing, and reset Npoints=0 for any data section to suppress useless output and restore previous behaviour.

[Dave Offiler]

In ropp2bufr.f90/ConvertROPPtoBUFR, added a section before L1b, L2a & L2b are converted to test for the whole profile key parameters set missing (except height coordinates), viz:

L1b : BA_L1, BA_L2 and BA
L2a : N
L2b : P, T, and q

If count of missing samples is equal to ..%Npoints, reset ..%Npoints to zero, thus skipping the conversion for that Level using the existing code tests. A (diagnostic) message is written if this happens.

Tested on Rune's netCDF file, and correctly skips the L2b dummy profile generated by the LOG-247 thinner, generating an identical number of BUFR octets (12594) as produced by the v3.0 and v4.0 encoders.

Run t_ropp2bufr in ropp_io/tests - FAIL. Reference datasets (notably ropp_test.bfr) regenerated to reflect reverted Local Data Sub-Category value (see BUFR header item (2) above). Re-run t_ropp2bufr - PASS.

Changeset [2564]. Will provide Rune with updated f90 & bfr files for him to confirm expected outputs.

NB original issue of 'default' ARH is yet to be explained, so this Ticket remains open for now.

[Dave Offiler]

Rune has replied today:

Hi Dave

Just had to add that I have made a mistake concerning the GTS headers.
These are included both in the refenrence and on the test system so there is no difference.

/Rune

So closing this ticket.