grib2ropp tool

[Huw Lewis]

GRAS SAF helpdesk enquiry (45):

Dear ROPP Team,

    I have some difficulty in convertting model data (GRIB format)
 to ROPP format background file, which stop the use of ROPP_1DVAR
 in our work.  Would ROPP Team develope the convert tool in the 
future version? 

Kind regards,
Bi Yanmeng

See ​http://www.grassaf.org/view_enquiry.php?id=45

A grib2ropp tool would be a useful addition to the ropp_io toolset, if practical without too many added dependencies. Michael Gorbunov's OCC package includes a GRIB sub-directory for reading in GRIB. Kent thinks this is without using an external library. He simply checked the definitions and then wrote a reader to read these binary files. So in order not to bring in too many external libraries this approach could maybe be considered (if we can agree it is a good idea to have this functionality in ROPP).

[Huw Lewis]

Comment by email from Huw:

As a point of detail, I would actually be proposing that the GRIB/ sub-directory is included 
'as-is' in a distribution (equivalent to the way the bufr libraries are used or Chris' PES 
interfaces to netCDF). I don't propose any 'recoding' of those routines.

Comment by email from Dave:

As a further point of detail, as you know, the BUFR library (MetO and in future ECMWF) are not 
packaged with ROPP, but are kept quite separate, for the user to download and install before 
ROPP - as is the Unidata netCDF package. 

I would envisage a similar GRIB2 library package; the one most people in the Met Office use is 
from NOAA.

I'm assuming that Michael's code is more like Chris' netCDF intermediate higher-level interface 
to provide 'insulation' between the programmer and the low-level library API, and doesn't 
actually include the GRIB(2) library code itself. As such, it's more question of does this 
interface support GRIB2 (if there are any differences in the GRIB2 API), rather than _is_ it 
GRIB2. So the next question is which low-level GRIB library package does Michael's code 
interface with?

[Dave Offiler]

I now consider this to be in the 'nice to have' but low priority category and little or not effort to be put into it during CDOP-1. As an intermediate 'solution' perhaps DMI could advise users wanting this facility how they (DMI) handle ECMWF background fields in GRIB2.

[Ian Culverwell]

There was a direct request at the ROPP6.0 DRI for this to be given a higher priority. This led to a formal recommendation to the Steering Group "that the Project Team includes a grib2ropp tool by the end of 2012 in an updated v6.1 delivery".

So I'm changing the milestone to 6.1

[Ian Culverwell]

txt2ropp.html

[Ian Culverwell]

Hallgeir Wilhelmsen at DMI suggested that the best place to start may be a couple of fortran 90 scripts written by Hans Gleisner, also of DMI. These have the advantage of already being linked to ROPP. Hallgeir downloaded these, and some example grib1 and grib2 data, ​here. I made a copy, containing local changes, ​here.

The scripts see (hgl_script/grib2txt and hgl_script/txt2ropp) worked OK, but only with grib_api_1.9.9, which I downloaded from ​the ECMWF releases page. This is the same version that Hans used. The latest release, 1.9.16, did not seem to hold the right longitudes in the file (although it did, according to the grib_ls tool?). Under discussion with ECMWF.

I made some notes at grib2txt.html and txt2ropp.html (attached).

[Ian Culverwell]

Incorporate Hans' scripts into ropp_io/tools/grib2bgrasc and ropp_io/tools/bgrasc2ropp. Ascii format means fortran namelist (in the standard). Man pages included. Base build logic on the absence/existence of grib_f_open_file. If it is, we need to append -lgrib_api_f90 -lgrib_api to the list of libraries - see ropp_io/configure.ac.

Include some very coarse (4 deg resn) GRIB data, downloaded from the ECMWF MARS archive, for testing purposes; write tests for both modules. This requires use of new routines nml_diff.f90 and nc_diff.f90, which might prove useful in future (see #273).

DO wrote mini-configure scripts to build GRIB_API. Suggest distributing grib_api_1.9.9 as a dependency, with the distribution, not least because 1.9.16 doesn't (yet) work, as noted above.

Allow time interpolation.

Tests OK. Passed to DMI (HW+HGL). They seem happy:

> Ian.
> Very nice!
> 
> I have finally found time to build it and I have run the script once. 
> I actually had some trouble building it all.
> After reproducing an error too many times, I asked Rune for help. All 
> he needed to do was to look at the script while building, and the 
> problem was gone! Magic! Probably me doing something stupid before?
> 
> I will make some comparisons tomorrow, but from what I have just seen, 
> this really looks useful! The output even contains values in the 
> geop-variable! And it takes 1/2 the time of what our python script 
> did, to process a profile. That will save us several days of 
> processing time.
> 
> I would like to ask for a way to avoid the required file extension. If 
> not we need to rename all our grib files. Our files are just called 
> 000, 006, 012, 018. I do not know why do not have an extension (before 
> my time). Would this be possible?
> 
> Will do some small tests tomorrow. More there!
> 
> But this looks very promising!
> Kind regards,
> Hallgeir.

I removed the restriction on filenames ending .grib etc., and other changes as requested by them. There remains an occasional difference between the results of Hans' code and the NRT processing:

> > >> Some files differ more than the others in pressure(more
> > than 5hPa) :
> > >> filename | max diff press
> > >> bgr20121120_082216_M02_1130470735_N0019_XXXX.nc 14.667020024 
> > >> bgr20121120_200745_M02_1100470784_N0019_XXXX.nc 6.6980577906 
> > >> bgr20121120_175123_M02_2290470774_N0019_XXXX.nc 9.0856112403 
> > >> bgr20121120_052451_M02_2010470723_N0019_XXXX.nc 5.62557083472 
> > >> bgr20121120_160606_M02_2120470767_N0019_XXXX.nc 10.0490500496
> > >>
> > >>
> > >> Some profiles differ more than others in temp (more than 1e-2).
> > >> filename | max diff temp
> > >> bgr20121120_082548_M02_2010470735_N0019_XXXX.nc 2.87976835221 
> > >> bgr20121120_082216_M02_1130470735_N0019_XXXX.nc 0.726499397092 
> > >> bgr20121120_224420_M02_1200470795_N0019_XXXX.nc 0.552076263783 
> > >> bgr20121120_103053_M02_1220470744_N0019_XXXX.nc 0.190988559729 
> > >> bgr20121120_223916_M02_1320470794_N0019_XXXX.nc 0.024822884818
> > >>

Don't understand these differences. But the ROPP implementation agrees with a stand-alone version of Hans' code. So:

Hi Hallgeir,

The latest revision (r3429) of 

root/ropp_src/branches/dev/Share/GRIB2ROPP 

contains:

1)  fixes to the incorrect lon business;

2) better checks on the "gribiness" of the input files to grib2bgrasc, so you don't 
have to call them "*.grb" anymore; and

3) a --no-ranchk flag to bgrasc2ropp.

There are other changes like a reduction in size and number of the test files for
these routines, and a better test method. 

grib2bgrasc ran OK on the example files you sent me, and in each of those cases
agreed as closely as numerical precision allows with the results of Hans's program
get_ecmwf_profs.x. 

I suspect the scientifically significant differences from the operational processing
that you sometimes see are the result of some interpolation difference.  It's 
difficult to say without running the operational code (which I don't want to install
here!) If you could point me to the files that produced

bgr20121120_082216_M02_1130470735_N0019_XXXX.nc, 

which has big press and temp diffs, I could at least compare my tool with Hans's.
(Or perhaps you've already done it?)

Kind regards,
Ian.

[Ian Culverwell]

No further problems arose. If the occasional difference between ROPP and DMI NRT processing is ever resolved, the ticket can be reopened. Closing it now.