ROPP8.0 dependency issues

[Ian Culverwell]

ROPP8.0 beta reviewer Dave Offiler (UKMO) says

This is where the 'fun' starts. The netCDF ./configure fails to locate the 
statically-built HDF5 libraries, whatever combination of flags I tried in the
netcdf4_configure_ifort_linux (or other compiler) mini-script to point it to
$PREFIX/lib64 (NB prefix/lib64 on OpenSUSE, not prefix/lib).

So tried re-building HDF5 as shared (simply change --disable-shared to 
--enable-shared) which got netCDF4 ./configure going. However, since HDF5 libs
are in a non-standard location, LD_LIBRARY_PATH needs this extra path for
run-time shared-lib location. This could be added to the mini-script(s) for the
build testing, but users woud stil have to set it (e.g. in their .profile) 
before running any tools accessing any (ROPP or EUM) netCDF files.

But then HDF5 barfs on finding my system-installed HDF5 .h file (v1.8.12) 
before the local one, and this is inconsistent with the local HDF5 libs 
(1.8.14). This causes netCDF reading tools to fail (reporting it as not a 
netCDF file).

Will try later with a local build of HDF5 v1.8.12....

NB: in principle it should be possible for netCDF to pick up a system-installed
HDF5, except that ROPP ./configure wants the Fortran API (which exists but only
a gfortran pre-built one). Not sure why, as the F90-API isn't used by ROPP 
otherwise. Can the ROPP_IO ./configure drop the 'requirement' for HDF5-Fortran
so as to allow a system-installed C-lib to be used?

Is this anything to do with ROPP?

[Ian Culverwell]

Updated 10/12/2014:

a) Dependencies

ZLIB 1.2.8   [OS-installed version]
HDF5 v1.8.8  [recommended for use with ROPP and provided on the ROPP-beta 
              website]
HDF5 v1.8.13 [OS-installed version]
HDF5 v1.8.14 [latest release downloaded from GitHub]

netCDF4 v4.1.3 in full (HDF5) mode.


This is where the 'fun' starts!

HDF5 v1.8.8 fails to build with gcc (all of ifort, gfortran & g95)
    CC     h5diff.lo
  h5diff.c: In function 'h5diff':
  h5diff.c:647:5: error: expected expression before '/' token
       //int i1, i2;
       ^
However, it does successfully build with suncc (sunf95), so I continued to build
netcdf-4 with HDF5 (v1.8.8) support using sunf95/suncc (OK) and ROPP_IO (also
OK, including the extra EUMESAT netCDF4 eum2ropp & eum2bufr tools). The same
ROPP_FM errors noted in (1) of course appear again.

Since HDF5 v1.8.8. is is now a very old version, and maybe the current gcc is 
stricter than our RHEL6 gcc, I then tried the latest HDF5 release (v1.8.14),
which builds OK with gcc.

But then the netCDF ./configure fails to locate the statically-built HDF5 
libraries, whatever combination of flags I tried in the 
netcdf4_configure_ifort_linux (or other compiler) mini-script to point it to
$PREFIX/lib64 (NB <prefix>/lib64 on OpenSUSE, not <prefix>/lib).

So tried re-building HDF5 as shared (simply change --disable-shared to 
--enable-shared) which got netCDF4 ./configure going. However, since HDF5 libs
are in a non-standard location, LD_LIBRARY_PATH needs this extra path for
run-time shared-lib location. The following line was added to buildpack
  
  export LD_LIBRARY_PATH=$ROPP_ROOT/$compiler/lib:$ROPP_ROOT/$compiler/lib64:$LD_LIBRARY_PATH

to allow the build & test tools to run, but users would still have to set
something similar (e.g. in their .profile) before directly running any tools 
which access (ROPP or EUM) netCDF files.

It also proved possible to remove the locally-built HDF5 libraries, etc, and 
let netCDF4 pick up the system-built version (v.18.13) installed from the 
OpenSUSE repository. ROPP_IO netCDF4 files tested sucessfully with all 
available compilers.

Noticed that the ROPP_IO ./configure when testing for HDF5 support includes 
references to the HDF5 Fortran API libraries - not sure why, as the F90-API 
isn't used by ROPP (or netCDF4). Can the ROPP_IO ./configure drop this redundant
'requirement' for HDF5-Fortran for the next release?

[Ian Culverwell]

While undertaking pre-operational testing of ROPP8.0, Santi Oliveras (IEEC) noted that

I've used latest HDF5 src package 1.8.14
and I've modified configure settings in 
"netcdf4_configure_gfortran_linux" and 
"ropp_configure_gfortran_linux" 
adding this option:
LIBS="-ldl"

Look at this link:
http://www.unidata.ucar.edu/support/help/MailArchives/netcdf/msg11847.html

After this change, NetCDF4 recognises HDF5 and passed
all tests.

checking for library containing H5Fflush... -lhdf5

Later I build ROPP with a successful complete 
installations, even the "grib2bgrasc" functionality. In 
this case I've noticed that I have set LD_LIBRARY_PATH 
to the ROPP/gfortran/lib path.

After, I tested to do the same build resetting
LD_LIBRARY_PATH variable, and in this case grib2bgrasc 
fails. Other tests worked perfectly.

I attach you my logs.

[Ian Culverwell]

ropp_logs.tgz

[Ian Culverwell]

Santi adds:

I've just wanted to finish this installation in a 
internal machine.

I've noticed that to fix the GRIB library problem, I 
had to set the LIBS="-ldl" option in ROPP Build 
configuration, and set correctly the LD_LIBRARY_PATH 
enviroment variable.

I attach different ropp_io instasllation executions.

In this case, I've used HDF5 v1.8.8

internal machine later clarified to mean:

just inside DMI LAN infraestructure; so it uses ifort
compiler with DMI licence.
In fact, it was the machine with Local IP 172.25.2.34 
corresponding to grassaf3.dmi.dk (Ubuntu 8.04)

[Ian Culverwell]

build_ropp_io_results.tgz

[Ian Culverwell]

dependency_table.txt

[Ian Culverwell]

dependency_table.png

[Ian Culverwell]

I don't know or care about any of this rubbish.

At ROPP9.0 we have an entirely new set of external dependencies, and corresponding mini-configure scripts, and they work OK in practice. The only necessary bodges are (as seen in buildpack_all):

      # annoying bodge 1:
      if [[ $compiler = "ifort14" || $compiler = "ifort16" || $compiler = "nagfor52" || $compiler = "nagfor60" || $compiler = "pgf11" || $compiler = "gfortran" ]] && [[ $module = "netcdff" ]];then
        banner "MANUAL INSTALL" >> ${ODIR}/${compiler}/buildpack_${module} 2>&1
        cd netcdf-fortran-4.4.3
        make install >> ${ODIR}/${compiler}/buildpack_${module} 2>&1
        cd ..
      fi

      # annoying bodge 2:
      if [[ $compiler = "nagfor52" || $compiler = "nagfor60" ]] && [[ $module = "grib" ]];then
        banner "MANUAL INSTALL" >> ${ODIR}/${compiler}/buildpack_${module} 2>&1
        cd grib_api-1.14.5-Source
        make install >> ${ODIR}/${compiler}/buildpack_${module} 2>&1
        cd ..
      fi

The shared library problem is solved by setting

export LD_LIBRARY_PATH=$ROPP_SRC/lib_link:$LD_LIBRARY_PATH

in .profile, and setting

    rm -f lib_link
    ln  -sf  ${ROPP_ROOT}/${compiler}/lib/  lib_link

in buildpack_all (attached).

What a mess. Thanks for this legacy, Dave.

For what their worth, Dave's ROPP9.0 external dependency table and notes are attached.

[Ian Culverwell]

I'm closing this ticket. I expect we'll get further problems from ROPP9.0 in this regard, but I hope they'll be someone else's problem.