Cross reference ROPP test folder tests with SRD requirements

[Ian Culverwell]

In RID 003 ROPP-10 DRR reviewer Carlos Vicente said, in connection with the ROPP Test Plan,

It is said that not all Tests have a corresponding entry in the SRD.
Does that mean that some tests are performed, not to cover a 
requirement but because they were needed to assure ROPP
functionalities?

He means:

Each test has a test procedure ID, but not all Test Procedure IDs 
have a corresponding entry in the SRD [RD.10], since several 
additional tests are introduced to assure full functionality of the 
ROPP package.

in the Test Plan.

We said:

Yes, exactly. The full statement says:

"Each test has a test procedure ID, but not all Test Procedure IDs 
have a corresponding entry in the SRD [RD.10], since several 
additional tests are introduced to assure full functionality of the 
ROPP package."

Please advise if we can make this any clearer.

(Incidentally, I discovered several outdated references to ROPP-9, 
CDOP-2 etc in the Test Plan.  I have brought those up-to-date.)

He said:

If the full functionality of the package is not captured by Software
requirements, then the ROPP is under-specified in the SRD.

Are these functionalities required/requested anywhere?

Closed with Action 002, targetted at ROPP-11:

PT to cross check the requirements in the SRD and the tests confirming 
that all the required functionalities for ROPP are captured on the 
SRD and tested.

[Ian Culverwell]

romsaf_srd_v52.pdf

[Ian Culverwell]

romsaf_ropp_testplan_v100.pdf

[Ian Culverwell]

Discussed this with Kent on 6/11/2020. We agreed, for ROPP-11, to:

1. Extend the Software Deliverables section and table GRM_16 of the PRD to include all the testing of ROPP that is currently undertaken.
2. Remove the ASSIM.* refs from the SRD. (These date from early in the GRAS SAF, when ROPP was largely a 1dvar retrieval tool.)
3. Replace the SRD references by PRD ones in the Test Plan. (Ditto for the Test Folder html pages, which echo these SRD cross-references.)
4. Replace references to the "Test Folder" by "test system" (or similar) in the Test Plan, since the ROM SAF will be moving away from the Test Folder in CDOP-4.

1 and 2 will need SG approval. Maybe 3 too.

[Ian Culverwell]

romsaf_prd_v34.pdf

[Kent Bækgaard Lauritsen]

Addition to point 1:

Introduce PRD-08-nn requirements for each of the ROPP modules (and list the main algorithms that a given module shall contain).

Extra point:

5. Introduce requirements related to #compilers and #platforms which must be tested. This is a response to the following SG action:

SG25-Act-04: The SG tasks the PT to update PRD-08-03 and SeSp-08-03 related to ROPP testing based on the response in document SG25-Doc-04a to action SG24-Act-01: (i) All tests must be carried out on one compiler (probably ifort). (ii) All ‘compile’ and all ‘run’ tests must be performed for ifort and gfortran, the two most popular compilers. (iii) ‘compile’ tests: Must be performed on linux and Cygwin, the two most popular platforms, on at least one common compiler (probably gfortran); Must be performed on at least 5 compilers; (iv) ‘run’ tests: Should be performed on the remaining ‘compile test’ compilers; And subsequently present the updated PRD and SeSp documents to the SG for approval.

[Ian Culverwell]

In Table GRM-16 of the PRD I suggest that

Characteristics and Methods - Routines for handling RO data (input/output, pre-processing, forward
                              modelling, corrections, assimilation)

be replaced by

Characteristics and Methods - Routines for handling RO data, including:
                              * Low-level utility routines (geodesy calculations, date/time conversion, coordinate transformations, error messages etc);
                              * I/O routines (support for intrinsic ROPP netCDF data structure, and its conversion to and from BUFR, tools to read UCAR, GFZ and EUMETSAT Leve1b data in their proprietary formats, ability to process data from the missions listed below, tool to extract background profiles from GRIB files, range-checking, profile thinning etc);
                              * Pre-processing routines (tools to generate L1 and L2 channel bending angles from excess phase and amplitude, and to compute ionospheric corrected bending angle and refractivity profiles from L1 and L2 bending angles, 1D and 2D wave optics propagation codes etc);
                              * Forward modelling routines (tools to generate refractivities and bending angles from background profiles produced by ECMWF and Met Office models, 1D refractivity operator, 1D and 2D bending angle operators, tangent linear and adjoint counterparts for use in 1dvar etc);
                              * 1D-Var routines (tools to retrieve solution profiles from ECMWF and Met Office models and refractivity and bending angle profiles, quality control, minimisers, solution diagnostics etc);
                              * 'Applications' routines (tools to diagnose tropopause height and boundary layer height from profiles of a variety of observational and model variables).

I think the next row of the table could be telescoped a bit, e.g.:

Operational Satellite Input Data - Metop-A, B, C (GRAS), Metop-SG,
                                   COSMIC, COSMIC-2,
                                   CHAMP, GPS/MET,
                                   GRACE, GRACE-FO, 
                                   TerraSAR-X, TanDEM-X,
                                   Oceansat-2/ROSA,
                                   Megha-Tropiques, 
                                   PAZ,
                                   FY-3 (GNOS)

The rest is OK, but I think

Verification/Validation Methods - Test Folder

should be replaced by

Verification/Validation Methods - Test Folder, i.e. a set of tests comprising:
                                 * Coding and compilation testing (compliance to coding standards; tests of basic functionality ('core tests') on at least 6 commonly available compilers and at least two platforms);
                                 * Module testing for the I/O module (test reading and writing of data is within tolerances, test unit conversion, etc);
                                 * Integration testing for the pre-processing module (test Abel transform, ionospheric correction, excess phase processing, wave optics propagators);
                                 * Integration testing for the foward modelling module (test computed refractivities and bending angles from ECMWF and Met Office background temperature, water vapour and pressure, against independently generated profiles, including at least one test of a full day of data);
                                 * Integration testing for the 1D-Var module (test the propriety of the input data, sensitivity to assumed errors, retrievals using real observational data from a variety of sources, including at least one test of a full day of data);
                                 * Integration testing for the applications module (calculated TPH and PBLHs compared against independently calculated reference values);
                                 * Regression testing (for each module, the results of one test are closely compared against those of the immediately preceding ROPP release);
                                 * Portability testing (check that at least one compiler passes as many tests as the default compiler);
                                 * Timing testing (check that code runs in reasonable times on different compilers and platforms);
                                 * Documentation testing (check that user guides and reference manuals are clear and correct).
                                 * All 'compile', 'run' and 'regression' tests shall be carried out on one compiler (probably either gfortran or ifort).
                                 * All 'compile' and all 'run' tests must be performed with at least two compilers (probably ifort and gfortran).
                                 * All 'compile' tests must be performed on at least two platforms (probably linux and Cygwin).
                                 * All 'compile' tests must be performed with at least five compilers.

Just suggestions - to be discussed if necessary.

The ROPP test plan has been updated to refer to the PRD rather than the SRD, and to talk about Test System rather than Test Folder, at r6921 and r6923.

The ROPP test folder top level html page has been updated to refer to the PRD rather than the SRD at r6922.

[Ian Culverwell]

Kent says we should mention iono retrieval code, so perhaps replace

                              * Forward modelling routines (tools to generate refractivities and bending angles from background profiles produced by ECMWF and Met Office models, 1D refractivity operator, 1D and 2D bending angle operators, tangent linear and adjoint counterparts for use in 1dvar etc);
                              * 1D-Var routines (tools to retrieve solution profiles from ECMWF and Met Office models and refractivity and bending angle profiles, quality control, minimisers, solution diagnostics etc);

above by

                              * Forward modelling routines (tools to generate refractivities and bending angles from background profiles produced by ECMWF and Met Office models, 1D refractivity operator, 1D and 2D bending angle operators, routines to calculate electron density and L2-L1 bending angle profiles for idealised ionospheres, tangent linear and adjoint counterparts for use in 1dvar etc);
                              * 1D-Var routines (tools to retrieve solution profiles from ECMWF and Met Office models and refractivity and bending angle profiles, tools to retrieve electron density profiles from L2-L1 bending angle differences, quality control, minimisers, solution diagnostics etc);

And Sentinel-6 isn't mentioned in the list of supported satellites, so perhaps we should also replace

Operational Satellite Input Data - Metop-A, B, C (GRAS), Metop-SG,
                                   COSMIC, COSMIC-2,
                                   CHAMP, GPS/MET,
                                   GRACE, GRACE-FO, 
                                   TerraSAR-X, TanDEM-X,
                                   Oceansat-2/ROSA,
                                   Megha-Tropiques, 
                                   PAZ,
                                   FY-3 (GNOS)

above, by

Operational Satellite Input Data - Metop-A, B, C (GRAS), Metop-SG,
                                   COSMIC, COSMIC-2,
                                   CHAMP, GPS/MET,
                                   GRACE, GRACE-FO, 
                                   TerraSAR-X, TanDEM-X,
                                   Oceansat-2/ROSA,
                                   Megha-Tropiques, 
                                   PAZ,
                                   FY-3 (GNOS),
                                   Sentinel-6

[Ian Culverwell]

This has been completed in response to Action 002 arising from ROPP-10 DRR. Changes committed at r6921, r6962 and r6963.