This test runs fine with ifort15 at ROPP9.0:

idculv@eld037:> test_1dvar_iono.sh                                                                                                                 
 
Running t_1dvar_iono_bangle (1DVAR L1 and L2) ...

----------------------------------------------------------------------
                    ROPP 1DVAR File Comparison Tool
----------------------------------------------------------------------

... (from ropp_1dvar_compare):  Comparing anl20090401_000329_M02_2030337800_N0007_YYYY_iono.nc and ../data/anl20090401_000329_M02_2030337800_N0007_YYYY_iono_reference.nc: the results of running test t_1dvar_iono_bangle (1DVAR L1 and L2)
... (from ropp_1dvar_compare):  Both files contain    1 profiles
... (from ropp_1dvar_compare):  No significant differences between anl20090401_000329_M02_2030337800_N0007_YYYY_iono.nc and ../data/anl20090401_000329_M02_2030337800_N0007_YYYY_iono_reference.nc
****************************
**********  PASS  **********
****************************
... examine t_1dvar_iono_bangle.log for details

There are tiny (<~1e-12) differences between the result and the reference (ifort12)?

(This is with a slightly more recent version of ifort15 (v15.0.2 20150121) than Dave's (v15.0.1 20141023 ).)

I rebuilt ROPP80_prototype with ifort15 (note that there were no ifort15 mini config scripts at ROPP8.0, so this was officially an unsupported compiler at that release, so I don't know why Dave was testing it), and again it was OK (apart from the same optimisation issue dealt with in #392).

I suspect it is a memory issue. Retrievals using L1 and L2 need an obs vector that is twice as long as usual. Axel had the same problems, which were fixed by setting ulimit -S -s unlimited. A note to this effect was added to the ROPP8.0 Release Notes.

My default stack size limit (ulimit -s) is 10240. If I reduce this to 1024, I get a seg fault with test_1dvar_iono.sh:

idculv@eld037:> test_1dvar_iono.sh
 
Running t_1dvar_iono_bangle (1DVAR L1 and L2) ...
test_1dvar_iono.sh: line 54: 17521 Segmentation fault      (core dumped) ./$EXEC ${EXTRA_CMD} -d -y $OBFILE1 --obs-corr $OBCOV -b $BGFILE1 --bg-corr $BGCOV -c $CONFIG -o $OFILE >> $LOGFILE

but the other 1dvar tests still run OK, eg

idculv@eld037:> test_1dvar_COSMIC_bangle.sh
 
Running t_1dvar_COSMIC_04_bangle (1DVAR COSMIC; default) ...
****************************
**********  PASS  **********
****************************
... examine t_1dvar_COSMIC_04_bangle.log for details
 
 
Running t_1dvar_COSMIC_04comp_bangle (1DVAR COSMIC; comp factors) ...
****************************
**********  PASS  **********
****************************
... examine t_1dvar_COSMIC_04comp_bangle.log for details
 
 
Running t_1dvar_COSMIC_04newop_bangle (1DVAR COSMIC; new interp) ...
****************************
**********  PASS  **********
****************************
... examine t_1dvar_COSMIC_04newop_bangle.log for details

Some experimentation on my linux desktop suggests we need a stack size of at least 9300 kilobytes. We could therefore put a check on ulimit -s in the script, and bail out with a 'not run' result if the stack size is likely to be too small. Defer until ROPP10.0.

Something like

if [[ $(ulimit -s) != "unlimited" ]] ; then
  if [[ $(ulimit -s) -lt 10000 ]] ; then
    echo "Stack size $(ulimit -s) insufficient to run test ... increase to at least 10000 KiB and rerun"
    exit
  fi
fi

would probably do it.

(While we're there, fix the typo in the robodoc-compliant header of test_1dvar_iono.sh:

#****s* fm/test_1dvar_iono.sh

should read

#****s* 1dvar/test_1dvar_iono.sh

)

We run into the same problem with the new laptops, which have a stack size of 8192 kB. The problem has been fixed by adding a line

ulimit -S -s unlimited  # This test needs a stack size over 8192 Kibytes.

to the top of ropp_1dvar/tests/test_1dvar_iono.sh.

Closing ticket.