eum2ropp and eum2bufr tools

[Ian Culverwell]

Axel is working on readers for EUM data in netcdf-4 format.

From: Axel Von Engeln [mailto:Axel.VonEngeln@eumetsat.int] 
Sent: 18 January 2012 09:57
To: kbl@dmi.dk; kmk@dmi.dk; Culverwell, Ian
Cc: Yago Andres; Christian Marquardt; Riccardo Notarpietro

Subject: RE: Rosa2ropp

Hi Kent, Kjartan, Ian,

I am currently updating my ROPP 5.1 version to be able to read
netCDF4. I am half way through, so I can read it, but still have to
get from the current EUMETSAT format to the internal ROPP format.
We want to use this then to convert our format into BUFR, using the
ECMWF BUFR library, thus allowing us to provide the offline data
also in BUFR format. I am currently adding tools for eum2ropp and
soon eum2bufr.

I am only doing the basics and would like to leave the rest to the
ROPP team, if you agree. If all goes according to plan then I'm
able to check in my modified version later this week and provide
you with a summary what I did and where work is still needed.

Does that sound okay, or did you already start with a netCDF4
reader?

Cheers,
         Axel

This ticket will hold further details.

[frae]

Taken directly from the svn commit message, revision 3281, branch as given below. Note the formatting might look a bit odd since it was ASCII in the svn commit:

This update of the ROPP 5.1 distribution allows to read EUMETSAT netCDF4 formats, as they are currently distributed. This format will be further refined in the coming weeks and I already posted several issues I found to Christian. Hence this should not be seen as a full fledged version, rather one that can be used by the GRAS SAF for further development.

It is available in repository:

​https://svn.grassaf.org/ropp/ropp_src/branches/dev/Share/AvE_ROPP51netCDF4/

Tools: ======

The following tools have been added:

• eum2ropp converts netCDF4 EUMETSAT files to ROPP format, not all

fields are included yet. Command line switches allow to read either the thinned data, or the high_res data from the level1B group. Very limited level1A reading is so far included, there is a command line switch, but for some reason the few fields are not correctly read. Might be an issue with the units or something else. Man pages are partly updated, but please recheck.

• eum2bufr converts netCDF4 EUMETSAT files to BUFR. Uses the ECMWF

BUFR library, no attempt has been made to include the Met Office one.

Issues: =======

I identified the following issues:

• use of the vlist is no longer possible with netCDF4, this might have to be updated (I liked it since it allowed to have variables in the netCDF file that are not in ROPP, but are kept in the e.g. thinning). Since the netCDF4 file has groups, with different number of variables, the simple solution as currently implemented is not supported for netCDF4 (3 should still work(.

• I included FIXME comments in the code where I essentially did a quick hack to get some issues resolved. They can be improved and should thus be checked.

• I did not fully update the header sections, or manual pages. Need to be checked.

• as said above, the level1A data reading is not working at the moment.

• I got the code working on Linux, using ifort and gcc, with the updated configure files. I did not try to compile on other platforms. I noticed that the hdf compilation had a virtual memory error, but I got around that by using the -02 optimisation only.

Added / Modified Files: =======================

• Configuration:

• compilers used:

gcc --version:

gcc (SUSE Linux) 4.3.2 [gcc-4_3-branch revision 141291]

ifort --version

ifort (IFORT) 11.1 20091130

• Libraries used:

hdf5-1.8.8 netcdf-4.1.3.

• Compilation/Configuration:

• configure/hdf_configure_ifort_linux working ifort / gcc version
• configure/netcdf_configure_ifort_linux updated for hdf path

• ropp_io/configure.ac: Update to search for hdf library

• ropp_io/tools/Makefile.am: eum2ropp, eum2bufr additions, only

for ECMWF BUFR library.

• ropp_io/ncdf/Makefile.am: ncdf_getgroupid.f90 file added.

• ropp_io/build/Makefile.am: new files added, but I am sure that

this needs to be checked again, e.g. for distribution!

• all Makefile.in are a result of changes in Makefile.am! They might

not all be required for the update to work. configure changes since configure.ac needed updates. aclocal.m4 is also likely changed due to other updates.

• NetCDF4

• ropp_io/ncdf/ncdf_getgroupid.f90: core routine to get the group ID

of a netCDF4 file. Is included in several other ncdf files now before the call for the variable ID is performed:

• ropp_io/ncdf/ncdf_isvar.f90
• ropp_io/ncdf/ncdf_getnrec.f90
• ropp_io/ncdf/ncdf_getatt.f90
• ropp_io/ncdf/ncdf_getsize.f90
• ropp_io/ncdf/ncdf_getvar.f90
• ropp_io/ncdf/ncdf_putvar.f90

NOTE: Some of these files are generated with m4 calls (e.g. ncdf_putvar,

hence the underlying changes are made in the corresponding m4 files:

• ropp_io/ncdf/ncdf_putgetvar_acode.m4
• ropp_io/ncdf/ncdf_putgetvar_scode.m4
• ropp_io/ncdf/ncdf_putgetvar_schar.m4
• ropp_io/ncdf/ncdf_putgetatt_acode.m4
• ropp_io/ncdf/ncdf_putgetatt_scode.m4
• ropp_io/ncdf/ncdf_putgetatt_schar.m4

Some m4 changes are trigger changes putvar routines. BUT NOTE: No writing of netCDF4 is possible!

• in order to get a variable from a group use: ncdf_getvar('/group1/group2/group3/varname', variable)

• in order to get an attribute from the root group, or sub-groups: ncdf_getatt('satellite', readstr) ncdf_getatt('/group1/group2/group3/satellite', readstr)

• in order to get an attribute from a variable (and NOTE this is not too nice, but needs other code modifications to make it nicer): ncdf_getatt('/level_1b/units', readstr, 'utc_start_absdate') hence this will for all variables in the level_1b group be the same call of '/level_1b/units', and only the variable name will change.

• BUFR

• restructuring of the code, thus large parts of the ropp2bufr are now in a separate routine, to allow more flexibility and avoid code duplicates.

• ropp_io/tools/ropp2bufr_ec.f90: large parts of this are now moved ...
• ropp_io/bufr/ropp2bufr_mod.f90: ... into this file
• ropp_io/tools/eum2bufr_ec.f90: added this for eum2bufr tool

• ROPP

I noticed the following issues:

• bangle sigma ranges seem rather small.
• time_offset can be negative, since the geo-referencing time is generated from orbits alone. But in netCDF format, it is considered only positive.
• we use NaN to indicate missing data in netCDF4, but this sometimes seems to leads to trouble with the bufr encoder. I now have a call in the reading routine to set all these values to ROPP missing. This is not very nice. But as said, I am not always observing this behaviour.

Tests: ======

I ran 2 simple tests, more a certainly required:

• convert netCDF4 routines to ROPP format with my IDL tool and converted them also with the eum2ropp tool. Found no differences in the generated files for bending angles, impact parameters. I did not go through all fields, but most seem to be working.
• ran an eum2ropp, used the ROPP file to run through bufr2ropp, and compared it to one that I generated myself from IDL. Seems to be giving the same result. However needs further verification.

[frae]

Revision 3282:

Preparation for a central installation at EUMETSAT. In particular removed the compilation of ropp_utils/common/typeSizes.f90 since this module is also provided by netCDF and is different to the one used in netCDF 4.

[Ian Culverwell]

Axel has made some further modifications, prompted in part by an observation by Harald Anlauf of DWD, reported as Helpdesk query ​121

Here is Axel's commit message for Changeset 3323 for ropp_src/branches/dev/Share/AvE_ROPP51netCDF4:

r3323 | frae | 2012-05-31 14:19:27 +0000 (Thu, 31 May 2012) | 27 lines

Updated in order to include some missing bufr data into the eum2bufr
converter, as noted by H. Anlauf, DWD.

While implementing, I noticed that the GNSS and LEO satellite position
and velocities when using bufr are actually included in the usual level
1a dimension, however with this dimension set to 1. I am unsure how this
works when more than 1 value is provided in the level 1a data, which one
is picked for the bufr data? Suggest to avoid this in a next release and
introduce a dimension in the netCDF 3 file that identifies reference
data, e.g. lat, lon of location, position & velocities, azimuth angle at
reference point. This new dimension would always be set to one and assures
that the correct data is used for bufr encoding and not one from a full
array of level 1a data.

Otherwise introduced a -B or -b flag to eum2ropp tool to get around this
problem. So it is now only possible to do the -b (results in dim_lev1a set
to 1), or to set the -l (which results in the full filling up of level 1a
data fields). Setting both will lead to an error.

Also note that this includes the updates Kjartan did. These can do with
some more modification to simplify them, which I did not do.

Several more FIXMEs are included, some of which can only be properly
sorted out once the EUMETSAT format has been adapted.

ROPP7.0 tasks building up ...

[kmk]

I checked in some further modifications. Now read-in of the 1a block is working and the eum2ropp tool therefore generates a ropp file with level 1a data included. I followed the old grasrs2ropp routine in that I combine the raw sampling and the closed loop recods into one record in the output file. There is a FIXME to suggest that ideally there should be a flag to decide which record(s) to read from input (closed loop, raw sampling or open loop / downsampled raw sampling)

[kmk]

No doubt my code can be simplified by some further modification as suggested by Axel above

[Ian Culverwell]

After quite a bit of effort, we have a set of files that work OK. Some points:

1. Define HAVE_HDF5 during the configuration step, according to whether netcdf4 is defined or not. This is then used in the build steps to define which tools are built and tested.
2. Need to introduce a netCDF-3 version of the modified ncdf_getgroupid.f90, which returns the groupid of a variable. (The netCDF-3 version returns the variable id.) The one to be used depends on the value of HAVE_HDF5.
3. Need to add a dummy variable to ropp_io_read_ncdf_get_eumdata or some compilers can't disambiguate tell the difference between it and ropp_io_read_ncdf_get_otherdata.
4. There's a nasty inconsistency between ifort12 and ifort11 in the output L1 phase. Finally traced to precision problems in subroutine accumulate_phase (which I don't like one bit). Solve by relaxing the test on the difference between test result and reference, but needs to be re-examined.
5. Sundry minor complaints about non-standard fortran (like BYTE, WHERE, ISNAN etc) which trip up some compilers (notably nagfor), but all patched up OK.

Eventually driven through the test folder OK (I made new tests on eum2ropp and eum2bufr).

No further problems, so closing the ticket.