Support latest dependency packages

[Dave Offiler]

Since ROPP-6 (Feb'12), all the dependency packages - MetO BUFR, ECMWF BUFR & netCDF - have been updated. Some (ECBUFR, netCDF) have changed the detail of their build systems, such that buildpack would not work. This ticket is a marker that if the current 3rd party libraries are to be supported by the buildpack & mini-script ROPP_BUILD system, then some effort is needed to achieve this aim.

[Dave Offiler]

1) MOBUFR

Release v20.0.0 (with my extras) includes MetDB kernel package v20.0.00 which is in late beta, but the production version can be expected to be released before ROPP-7. The kernel package has little in the way of code changes (the most obvious to a user is that the target for BUFR_LIBRARY no longer requires a trailing '/' and the run-time table names are lower-cased and prefixed with 'bufr_'), but provides the latest WMO BUFR tables (V17, Nov 2012). The build system is unchanged from v19.1.x - no changes to buildpack required.

2) ECBUFR

The current release is 000389. The most obvious change is in the name of the tarball/directory: bufrdc_<ver> instead of bufr_<ver>. Internally, the build system is significantly different. The package now provides several Makefile.in files which the install script edits on the the fly based in interactive user inputs to generate final Makefile files, which are then run via a master Makefile. The latter also drives a test script.

I have re-created the Makefile generation in buildpack (to avoid the interactive Q&A). Running this builds the library OK, but the test programs all fail during linking with many missing for_* references. These appear to be low-level ifort library routines, so something is preventing the linker from finding the ifort shared libraries at this point. Building and running one of the test tools separately (but still within buildpack) works OK.

For the moment it seems safe to ignore this problem, but it will need correcting at some point. NB: only tested with ifort so far. A similar linker problem may or may not occur with other compilers.

3) netCDF

The latest release is V4.2.1.1. As from V4.2, the netcdf-<ver> tarball contains only the core C-based library code; the Fortran (and other language bindings) is now in a separate tarball netcdf-fortran-<ver> (V4.2), and the two are built alone (obviously the core library first).

In principle, we could support this new system as well as the v4.1.x all-in-one, but it would be somewhat messy and be essentially obsolete come release of ROPP-7. For this development, I plan on just supporting the V4.2 release.

In buildpack the main netCDF build remains the same (but replacement mini-scripts are needed), and a new package netcdff is supported (which needs a new set of mini-scripts). Users will need to build netcdf or netcdf4 as now, then netcdff (the latter builds and tests against both classic and HDF5 versions).

As of now, only mini-scripts for ifort/linux are in place. Scripts for other compilers and platforms are yet to be created.

4) HDF5

The latest release is v1.8.10-patch1 (ROPP currently uses v1.8.8). This latest To Be Tested. No indication that buildpack or mini-scripts will need any changes here.

5) Zlib

The latest release is v1.2.7 (ROPP currently uses v1.2.5). This latest To Be Tested. No indication that buildpack or mini-scripts will need any changes here.

6) SQLite

The new ROPP_GG module will require SQLite support. Like Zlib, it's more than likely that most Linux distros will have the object library (static and/or shared) pre-built in the system /lib (or /usr/lib), but do not always have the .h development headers installed. Hence buildpack should support SQLite also. The latest release as at today is v3.7.15.2

Updated for items 2 & 3 (but ifort only) committed as [3600]

[Dave Offiler]

Confirm build & self-test of complete stack Zlib/HDF5/netCDF4/netCDF-Fortran for ifort (v12) with the latest releases (but not yet with ROPP)

Added build support for SQLite3 (most compilers)

Committed as [3602]

[Dave Offiler]

Found problems in failed configure of ropp_io when checking for netCDF-4 group routine. Not had a chance to track this one down before v7.0 freeze, so we have decded to stay with v4.1.x for this release. Putting milestone back to v7.1

[Dave Offiler]

The MetO BUFR package is currently at v20.0.2; this has been tested with ROPP branch do_exitcodes; no changes to any ROPP code/scripts are required, so this version should be used for ROPP-8.

To be noted that ECMWF no longer supply a standalone BUFR package, but a new emos package bundles BUFR, GRIB, grid interpolation and other stuff all into one tarball. This might have attraction for some users, but only provides much wider scope for build failures on components we have no interest in for ROPP. We don't have time now to sort out trying to build this new package on all the platforms expected of ROPP, so I suggest we stick with bufr_000387 (even if it is now unsupported by ECMWF) because we know it works (excepting gcc on Cygwin - see #276) for ROPP-8.

Ditto netCDF - suggest sticking with the last v4.1.3 release for ROPP-8. I have successfully built the two parts of v4.2.x for use with the GWV package and have mini-config scripts for ifort_linux (only). Scripts for all the other platforms would need to be generated plus support in buildpack. Suggest include in following ROPP release, at which time I would recommend dropping build support for netCDF-classic mode and single-tarball netCDF v4.1.x; support only v4.2.x (in two parts) and in full netCDF-4/HDF5 guise only (this mode is needed for the eum2ropp tool anyway, and would be the necessary default to support a future ROPP-netCDF4 grouped format a la EUM style).

[Dave Offiler]

Agreed with Ian to stick with HDF5 v1.8.8 which we know is compatible with netCDF-4.1.3 (current HDF5 release is v1.8.13). [To note that testing ROPP-7.1 on home OpenSUSE 13.1 worked with system-installed HDF5-1.8.11 and locally-installed netCDF-4.2.]

Having decided on dependency package versions for ROPP-8, This ticket effectively closed, but leaving open as an ongoing action for ROPP-9 (or perhaps next minor release).

[Ian Culverwell]

External dependency packages available and recommended for use with ROPP9.0:

zlib-1.2.8
hdf5-1.8.16 
netcdf-c-4.4.0
netcdf-fortran-4.4.3
bufr-24.0.2
bufr_000387
grib_api-1.14.5

The build system has been developed to build and run with these. Note the splitting of the netCDF libraries into Core (C) and Fortran libs.

As this is a recurring task the ticket has been rolled over into ROPP10.0.

[Ian Culverwell]

External dependency packages fully tested, supported and distributed with ROPP-10.0:

zlib-1.2.11.tar.gz
hdf5-1.10.6.tar.gz
netcdf-c-4.7.3.tar.gz
netcdf-fortran-4.5.2.tar.gz
bufr-25.0.2a.tar.gz
bufr_000387a.tar.gz
eccodes-2.12.5-Source.tar.gz
grib_api-1.14.5-Source.tar.gz
sofa_f-20190722.tar.gz

As this is a recurring task the ticket has been rolled over into ROPP11.0 (although it might not actually be done).

[Ian Culverwell]

ROPP-11 uses same external dependency packages as ROPP-10, except for

eccodes-2.22.0-Source.tar.gz

rather than

eccodes-2.12.5-Source.tar.gz

which has been included for reasons described in #699.

[Ian Culverwell]

Re-assigning this rolling ticket to ROPP-12.