id	summary	reporter	owner	description	type	status	priority	milestone	component	version	resolution	keywords	cc
273	"Overhaul the ""test"" aspects of the build system"	Ian Culverwell	Ian Culverwell	"The ROPP6.0 beta reviewer described the voluminous output of text and graphics during the ""make test"" part of the build procedure as a ""tedious adventure"".  He made several suggestions for improvement:

   1. Introduction of environment variable $ROPP_SHOW_PLOTS=[TRUE | FALSE], to suppress pop-up output if desired;
   1. Use of one graphics output format only (preferably eps);
   1. Concatenation of (better annotated) reference and test plots into single file;
   1. Storage in same directory as build_ropp.log file;
   1. Clear announcement if any test graphics files are missing; 
   1. Inclusion of test summary table/list, inc pass/fail status.

He also additionally recommended to provide a complete list of tests for all test groups or modules, containing the

   1. exact test name or test identifier, the
   1. path/name of the script used to generate the test, the
   1. name of the REFERENCE data/plot file, and the
   1. name of the TEST output data/plot file.

This list could be provided either as a tests.README in the ./tests/ sub-directory or as a list or table in the *.pdf documentation for all test groups or modules.

He also thought it was noticeable that the annotation of graphics is incomplete: it would be an improvement to include a small table or list on each plot (i.e., on the REFERENCE and TEST plots) indicating the following information:
   1. the exact test name or test identifier,
   2. the path/name of the script used to generate the plot,
   3. a clear/prominent indication, if the plot is a REFERENCE or a TEST plot,
   4. the exact name of the plot graphics file, and the plot generation date (in YYYY-MM-DDHH:MM:SSS.S format)

Finally, he felt that the output of the final build and test message
{{{
------------------------------------------------------------------
Done. Check messages to confirm successful build & installation
------------------------------------------------------------------
}}} 
is not completely satisfying. An automatic creation of a test summary list or table, indicating the test result status, would be welcomed.

We agree that the “make test” aspects of building ROPP need a “make over”. The beta reviewer's ideas resonate with similar ones we were having in connection with the testing of changes for potential inclusion in the official ROPP release, maybe my means of a “light” ROPP test folder, or some such.



"	enhancement	closed	normal	11.1	ROPP (all)	7.1	fixed	make test