Missing data on decoding test TerraSAR-X BUFR

[Dave Offiler]

Torsten Schmidt at GFZ has used the ROPP-2 (and also ROPP-3) gfz2ropp and ropp2bufr tools to encode some test RO data from TerraSAR-X. Reports back from NCAR are that the BUFR contains unexpected 'missing data'. Torsten has provided a day's worth of original .dat/.dsc files, the converted ROPP netCDF and encoded BUFR files, plus some background emails with NCEP (not attached to this ticket).

Torsten's direct emails quoted here:

Hi Dave,

in addition to the mail in the morning I send you for your analysis
some  more data for day 08.206. DAT and DSC contain our ascii files
which are the input for the gfz2ropp program. NC contains the 
resulting netcdf files whereas BUFR are the same data as I already
send you in the first mail.

Some comments to the gfz2ropp program. I had to make some small
changes because the output from Georg Beyerle was not exactly
compatible with the GRACE data. But this is not the reason for the
missing values NOAA sees.

The changes are in
line 493: comment out this line
line 499-502: my output goes to 'OUTPUT/', therefor "datfile(8:9)"
instead of datfile(1:2)
line 502: change of TX to TS
line 521/522: comment out. The program was interrupted at this line. 
I don't know why.
line 526/527: inserted, because Georg didn't wrote the azimuth
angle in the file header, but in the dsc file.

I've attached the version I've changed.

Just to remember, I used ROPP Vers.3 for the results attached.

Tomorrow I'm out of office for the whole day and back on Monday. 
So I hope you have all informations you need.

Maybe you get in contact directly with Greg Greg.Krasowski@noaa.gov)
if you have already results tomorrow?

Thank you very much, Dave.
Cheers,   Torsten

Offiler, Dave wrote:

> Hi Torsten,
> 
> Could you just try adding -p0 (0=zero) to the ropp2bufr command?
> This forces no thinning. Also -d to output more diagnostics. 
> Do you get the same problem with GRACE-A or is it just TSRX?
> 
> I have a meeting in a few minutes, but I'll have a look at your files 
> this afternoon.
> 
> Dave
> --
> David Offiler   Head Satellite Active Sensing Group
> Met Office   FitzRoy Road   Exeter   EX1 3PB   United Kingdom
> Tel: +44 (0)1392 88 6298    Fax: +44 (0)1392 88 5681
> E-mail: dave.offiler@metoffice.gov.uk   http://www.metoffice.gov.uk
> 
> -----Original Message-----
> From: Torsten Schmidt [mailto:tschmidt@gfz-potsdam.de]
> Sent: 10 December 2009 08:57
> To: Offiler, Dave
> Cc: Greg Krasowski; Lidia Cucurull; Jens Wickert; Georg Beyerle
> Subject: [Fwd: Re: TerraSAR-X RO data in BUFR format]
> 
> Hi Dave,
> 
> as you know we are working on the TerraSAR-X (open-loop) processing 
> and hope to have an operational software version soon.
> 
> As a preliminary dataset Georg Beyerle had prepared data from a period 
> covering some months in 2008. This dataset I've converted to bufr 
> (same procedure as with the operational GRACE data) using the ROPP software.
> 
> The problem is that NOAA sees a lot of 'strange things'. I don't know 
> what's about Sean. He got the same dataset but I've no response from 
> him until now.
> 
> The 'strange things' are related to, e.g., missing values in the 
> decoded data.
> 
> What I've done so far was the application of the ROPP Vers.2 software 
> just in the way I use it for the GRACE data (since February this year).
> 
> Because the 'strange things' happened with Vers.2 I installed 
> yesterday ROPP Vers.3 (ropp_utils-3.0, ropp_io-3.0) and also the new 
> bufrpack-18.1.
> 
> Then I converted one day of TSX data (08.206) by application of the 
> following programs: (1) gfz2ropp for the conversion of the dat and dsc 
> files to netcdf and (2) ropp2bufr ('ropp2bufr ncfile -c 78'), i.e., 
> without thinning of the data. There were now error message as well as 
> any warning. The results you find attached in the TSX_08.206.tar.gz 
> archive.
> 
> What I've also done for validation purposes is the conversion back to 
> netcdf with the bufr2ropp program. I can't find any missing values in 
> the back-converted files, but Greg from NOAA does. His error output is 
> also attached.
> 
> Dave, could you please give a comment to that. I'm not familiar with 
> bufr data so I think you are the person that could (hopefully) solve 
> the problem.
> 
> Thank you very much.
> 
> Bets regards,
> Torsten

Of note is that it seems that TSRX data in the DAT/DSC is written a bit differently from how CHAMP/GRACE-A is written, which means some modification to gfz2ropp anyway.

Not critically urgent, as TSRX data is not yet being processed in NRT, but it needs to be fully supported in the next few months, and ideally the problem reported here investigated and (if the problem is found to be in one of the ROPP tools) fixed asap to support ongoing testing.

First thing to do is to rebuild ROPP_IO tools using (ideally) the same F90/C compilers as used at GFZ (awaiting this info from Torsten...)

[Dave Offiler]

After a full analysis of sample DAT/DSC, netCDF and BUFR files provided by Torsten (GFZ), plus an example BUFR file from Greg (NCEP), the conclusion is that there is absolutely nothing wrong with the output from the ROPP BUFR encoder. A gfz2ropp-->ropp2bufr-->bufr2ropp cycle, and ncdump'ing the netCDF files to cdl shows identical output (apart from the usual non-significant file name, processing time and a few last-digit rounding errors).

Almost all of the alleged 'missing' data is indeed missing - and is expected to be missing, as these parameters (e.g. BA/N/T error, SH) are not in the GFZ files to start with! The exception to 'almost' is that profile lat/long (per level) is still claimed to be missing when decoded by NCEP, but are present and valid in the BUFR decoded with the MetDB kernel library (whether using the generic decbufr tool or ROPP bufr2ropp tool).

The problem must therefore lie in NCEP's decoder. TBC, but it seems they are attempting to decode 3 sets of BA per level - the GFZ DAT files only contain the iono-corrected BA, not L1 or L2, so there is only one 'frequncy' replication for GFZ data.

It appears that NCEP have never before decoded GFZ BUFR (CHAMP or GRACE-A), only CHAMP from UCAR, which (a) contains the parameters that are missing from GFZ and (b) contain the L1 and L2 as well as corrected BA, so they may well be making false assumptions in their decoding sequence.

While we will continue to provide advice to Greg in solving his problem, there is no evidence at all that there is any problem with the ROPP-generated BUFR, so this ticket is hereby closed as 'no problem here'.

The issue of the gfz2ropp tool needing modification to correctly convert TerraSAR-X DAT/DSC files is separately dealt with under Ticket #194.

[Dave Offiler]

Greg has come back. Apparently, it's not all profiles which have missing data in the BUFR, only some. Having re-dumped all 62 of Gregs set of profiles there are indeed some with missing lat_tp/lon_tp. My sampling clearly skipped over these, and thinking all lat/lon values were claimed to be missing (and were shown not to be), problem not ours.

Re-encoding using v3.0 ropp2bufr with -p0 switch (suppress all thinning) 'solves' the missing lat/lons. I suspect (but need to test) that any of the non-sample methods will work correctly, and we have a regression bug in the default sampling-to-no-more-than-[375] code.

This Ticket re-opened to investigate this part of the thinner.

Meanwhile, Torsten should be advised re-encode TerraSAR-X data for NCEP using -p0 (or apply the log-247 thinner file)

Greg's email below:

Hi Dave,

You recall that I was trying to find out why I'm seeing missing latitude and longitude 
values in various replication sequences of the sample TerraSAR-X BUFR data provided by GFZ,
whereas you and GFZ were not seeing any of these missing values. I'm finally getting 
around to looking at the three output files (bufr01, bufr31, and bufr62) you provided in 
your last message to me from December. I was wondering. Did the bufr01 file contain
everything in the entire message number 1 of the file dated 07/24/08 that I provided 
or was it just the first subset of that message? The latitudes and longitudes (and e
verything else, even other parameters with missing values) match with what I got through
the end of the first subset. However, I see two more subsets after the first subset and 
in the second subset under message number 1, I see in my file an initial latitude/longitude 
of 69.37100/-29.13200, but then in all of the replications for that subset, the Lat/Lon 
is missing. If you still have my sample BUFR file available, could you send an output 
file (or several output files, depending on the size) of the other two subsets under 
message number 1 at the beginning of that BUFR file, so that I can continue to compare 
that message number? If you need another copy of my sample BUFR file, let me know, and 
I will resend it to you.

I also noticed that the bufr31 output file is actually the first subset of message 
number 11, while the bufr62 output file contains the first and only subset of message 
number 1 from a different tank (11/03/08). It appears the data from these first 
subsets all match. If we can just verify the subsequent subsets under message 
number 1 of the tank from 07/24/08, we should be able to figure everything out soon.

Thanks in advance for continuing to help us in this matter, Greg Krasowski