Missing data on decoding test TerraSAR-X BUFR

[Dave Offiler]

Torsten Schmidt at GFZ has used the ROPP-2 (and also ROPP-3) gfz2ropp and ropp2bufr tools to encode some test RO data from TerraSAR-X. Reports back from NCAR are that the BUFR contains unexpected 'missing data'. Torsten has provided a day's worth of original .dat/.dsc files, the converted ROPP netCDF and encoded BUFR files, plus some background emails with NCEP (not attached to this ticket).

Torsten's direct emails quoted here:

Hi Dave,



in addition to the mail in the morning I send you for your analysis
some  more data for day 08.206. DAT and DSC contain our ascii files
which are the input for the gfz2ropp program. NC contains the 
resulting netcdf files whereas BUFR are the same data as I already
send you in the first mail.



Some comments to the gfz2ropp program. I had to make some small
changes because the output from Georg Beyerle was not exactly
compatible with the GRACE data. But this is not the reason for the
missing values NOAA sees.



The changes are in

line 493: comment out this line

line 499-502: my output goes to 'OUTPUT/', therefor "datfile(8:9)"

instead of datfile(1:2)

line 502: change of TX to TS

line 521/522: comment out. The program was interrupted at this line. 
I don't know why.

line 526/527: inserted, because Georg didn't wrote the azimuth
angle in the file header, but in the dsc file.



I've attached the version I've changed.



Just to remember, I used ROPP Vers.3 for the results attached.



Tomorrow I'm out of office for the whole day and back on Monday. 
So I hope you have all informations you need.



Maybe you get in contact directly with Greg Greg.Krasowski@noaa.gov)
if you have already results tomorrow?



Thank you very much, Dave.



Cheers,   Torsten



Offiler, Dave wrote:

> Hi Torsten,
> 
> Could you just try adding -p0 (0=zero) to the ropp2bufr command?
> This forces no thinning. Also -d to output more diagnostics. 
> Do you get the same problem with GRACE-A or is it just TSRX?
> 
> I have a meeting in a few minutes, but I'll have a look at your files 
> this afternoon.
> 
> Dave
> --
> David Offiler   Head Satellite Active Sensing Group
> Met Office   FitzRoy Road   Exeter   EX1 3PB   United Kingdom
> Tel: +44 (0)1392 88 6298    Fax: +44 (0)1392 88 5681
> E-mail: dave.offiler@metoffice.gov.uk   http://www.metoffice.gov.uk
> 
> -----Original Message-----
> From: Torsten Schmidt [mailto:tschmidt@gfz-potsdam.de]
> Sent: 10 December 2009 08:57
> To: Offiler, Dave
> Cc: Greg Krasowski; Lidia Cucurull; Jens Wickert; Georg Beyerle
> Subject: [Fwd: Re: TerraSAR-X RO data in BUFR format]
> 
> Hi Dave,
> 
> as you know we are working on the TerraSAR-X (open-loop) processing 
> and hope to have an operational software version soon.
> 
> As a preliminary dataset Georg Beyerle had prepared data from a period 
> covering some months in 2008. This dataset I've converted to bufr 
> (same procedure as with the operational GRACE data) using the ROPP software.
> 
> The problem is that NOAA sees a lot of 'strange things'. I don't know 
> what's about Sean. He got the same dataset but I've no response from 
> him until now.
> 
> The 'strange things' are related to, e.g., missing values in the 
> decoded data.
> 
> What I've done so far was the application of the ROPP Vers.2 software 
> just in the way I use it for the GRACE data (since February this year).
> 
> Because the 'strange things' happened with Vers.2 I installed 
> yesterday ROPP Vers.3 (ropp_utils-3.0, ropp_io-3.0) and also the new 
> bufrpack-18.1.
> 
> Then I converted one day of TSX data (08.206) by application of the 
> following programs: (1) gfz2ropp for the conversion of the dat and dsc 
> files to netcdf and (2) ropp2bufr ('ropp2bufr ncfile -c 78'), i.e., 
> without thinning of the data. There were now error message as well as 
> any warning. The results you find attached in the TSX_08.206.tar.gz 
> archive.
> 
> What I've also done for validation purposes is the conversion back to 
> netcdf with the bufr2ropp program. I can't find any missing values in 
> the back-converted files, but Greg from NOAA does. His error output is 
> also attached.
> 
> Dave, could you please give a comment to that. I'm not familiar with 
> bufr data so I think you are the person that could (hopefully) solve 
> the problem.
> 
> Thank you very much.
> 
> Bets regards,
> Torsten

Of note is that it seems that TSRX data in the DAT/DSC is written a bit differently from how CHAMP/GRACE-A is written, which means some modification to gfz2ropp anyway.

Not critically urgent, as TSRX data is not yet being processed in NRT, but it needs to be fully supported in the next few months, and ideally the problem reported here investigated and (if the problem is found to be in one of the ROPP tools) fixed asap to support ongoing testing.

First thing to do is to rebuild ROPP_IO tools using (ideally) the same F90/C compilers as used at GFZ (awaiting this info from Torsten...)

[Dave Offiler]

After a full analysis of sample DAT/DSC, netCDF and BUFR files provided by Torsten (GFZ), plus an example BUFR file from Greg (NCEP), the conclusion is that there is absolutely nothing wrong with the output from the ROPP BUFR encoder. A gfz2ropp-->ropp2bufr-->bufr2ropp cycle, and ncdump'ing the netCDF files to cdl shows identical output (apart from the usual non-significant file name, processing time and a few last-digit rounding errors).

Almost all of the alleged 'missing' data is indeed missing - and is expected to be missing, as these parameters (e.g. BA/N/T error, SH) are not in the GFZ files to start with! The exception to 'almost' is that profile lat/long (per level) is still claimed to be missing when decoded by NCEP, but are present and valid in the BUFR decoded with the MetDB kernel library (whether using the generic decbufr tool or ROPP bufr2ropp tool).

The problem must therefore lie in NCEP's decoder. TBC, but it seems they are attempting to decode 3 sets of BA per level - the GFZ DAT files only contain the iono-corrected BA, not L1 or L2, so there is only one 'frequncy' replication for GFZ data.

It appears that NCEP have never before decoded GFZ BUFR (CHAMP or GRACE-A), only CHAMP from UCAR, which (a) contains the parameters that are missing from GFZ and (b) contain the L1 and L2 as well as corrected BA, so they may well be making false assumptions in their decoding sequence.

While we will continue to provide advice to Greg in solving his problem, there is no evidence at all that there is any problem with the ROPP-generated BUFR, so this ticket is hereby closed as 'no problem here'.

The issue of the gfz2ropp tool needing modification to correctly convert TerraSAR-X DAT/DSC files is separately dealt with under Ticket #194.