User tests dependence on IDL

[Huw Lewis]

The dependence on idl for verifying the test output is considered to be a limitation of the automatic test scripts at present. Consideration of more generic output and validation tools will be considered for future ROPP releases.

e.g. For external ROPP-4 beta testing by Luiz Sappucci, the build tests would run ok, but no output could be displayed since IDL was not available.

We should consider more simplified text based PASS/FAIL output (e.g. as in ropp_io user tests), and then provide idl examples as part of planned 'ropp_apps' extensions.

[Huw Lewis]

Suggestion:

Provide a '-t' type test flag for each stand-alone tool. Each user 'make tests' call would then run the stand-alone tools for each module with the '-t' flag (invoking default input data, in-line result comparisons, pass/fail output to user). If available, results may be plotted using IDL, but there should be no dependence on IDL to enable the user to determine pass/fail of a test.

This approach leads to consistency of implementation across all user tests, and ensures tests are performed with the latest available stand-alone tool.

[Ian Culverwell]

ropp_utils doesn't contain any tests.

ropp_io/test routines make no calls on IDL, except for t_roppthin.sh, which appears to be obsolete. (It executes the IT-IO-03 test.) No need to update.

ropp_pp/tests/t_ropp_pp calls idl 4 times if available; otherwise they skip the checks. But the IDL routines do more than plot the data: they calculate stats, interpolate onto common levels, and generate jpeg/gif/png files. So avoiding IDL would be difficult.

ropp-fm/tests/test_fm_GRAS.sh and test_fm_2D.sh call IDL, if available. But these are visual checks, so if we bypass IDL we'll need to replace with something similar, which again is unlikely to be universally available.

ropp_1dvar/tests/test_1dvar.sh, test_1dvar_GRAS.sh and test_1dvar_MO.sh call IDL if available, which then does processing and necessary plotting. Hard to avoid it or something like it.

Conclusion: "Make test" will not fail if IDL is not available. It may fail to carry out some tests, but these currently involve the visual comparison of one result with another, standard, reference. Even if we could boil this down to a numerical comparison of the two datasets (as we do for ropp_io) would anyone really feel that a "physical" ROPP module had been properly tested without some visual evidence of the sensibleness of the results?

Recommendation: Close the ticket, as the routines won't fail if IDL is absent. The larger question of the investigation into the generality and utility of the ROPP make tests, and other such questions, falls more naturally under the "ropp_apps" discussion of ROPP ticket #167.