Occasional problems with ropp_pp_occ

[Ian Culverwell]

Santi Oliveras (IEEC) noticed that occasionally ropp_pp_occ_tool stops running. He supplied the attached 3 files (and their paired navigation bit files), which I ran through the attached test1.sh.

ropp_pp_occ_tool runs for me, in the sense that it completes (in 15-20 secs), although the output is pants, eg:

----------------------------------------------------------------------
               ROPP Occultation Pre-processor Tool
----------------------------------------------------------------------

INFO:  Reading configuration file stalled_processes/cosmic_pp.cf.

INFO:  Processing profile    1 of      1
INFO:  Reading input data file stalled_processes/ropp_atmPhs_C001.2011.192.01.14.G09_2010.2640_nc.

INFO:  (OC_20110711011445_C001_G009_UCAR) 

...:  Occultation point: Lat=    55.67, Lon=    83.43
INFO (from ropp_pp_preprocess):  COSMIC data preprocessing
INFO (from ropp_pp_preprocess):  Reading lost carrier flag data from input file
INFO (from ropp_pp_preprocess):  Reading external NDM data from file stalled_processes/gpsBit_C001.2011.192.01.14.G09_2010.2640_txt
INFO (from ropp_pp_preprocess):  COSMIC data: openloop preprocessing
... (from ropp_pp_preprocess):  Navbit correllation area:           62.4           66.4
... (from ropp_pp_preprocess):  Max. navbit correlation:    0.983 at    0
... (from ropp_pp_cutoff):    L1 aperture:  16388.670  FFT points:  1024
... (from ropp_pp_cutoff):    L1 aperture:    835.244  FFT points:    64
... (from ropp_pp_cutoff):    L2 aperture:    946.200  FFT points:    64
... (from ropp_pp_cutoff):    L2 badness between  15.0 and  50.0 km: -Infinity
... (from ropp_pp_cutoff):    Click at time(    2) =   -0.475
... (from ropp_pp_cutoff):    Click at time(    2) =   -0.475
... (from ropp_pp_cutoff):    Click at time(    2) =   -0.475
INFO:  Retrieving bending angle profiles by GEOMETRIC OPTICS 

...:  Smoothed bending angle.
...:  Pmin =   2000.000 Pmax = 132457.749
...:  ws_go_smooth      143
...:  Full resolution bending angle.
...:  Pmin =   Infinity Pmax =  -Infinity
...:  ws_go_full        0
 
WARNING:  Cannot process profile (full): Pmin > Pmax
INFO:  Retrieving bending angle profiles by WAVE OPTICS 

...:  WM =   143
...:  W =     0
...:  WL =     0
 
WARNING:  Cannot process profile (wo): Pmin > Pmax
 
WARNING:  Cannot process profile: Invalid bending angles computed 

INFO:  Writing to output file ropp_atmPhs_C001.2011.192.01.14.G09_2010.2640_out.nc

I suspect this is associated with the presence of missing data (specifically, phase_L2=-999) in the source files. In which case, #287 refers.

There's also a mismatch between the size of the data in the nc file (dim_lev1a = 6199) and the size of the navbit file (which has 7500 elements). Could this be significant?

[Ian Culverwell]

Despite concerns over the navbit file compatibility, I tested the sensitivity to missing data by removing the missing phase_L2 points from the file and retrying (with the old navbit file). (ncks to find the missing data points and to extract the non-missing parts to two files, ncpdq to make dim_lev1a their record dimension, ncrcat to glue them together, ncpdq to make dim_unlim the record dimension).

It works:

----------------------------------------------------------------------
               ROPP Occultation Pre-processor Tool
----------------------------------------------------------------------

INFO:  Reading configuration file stalled_processes/cosmic_pp.cf.

INFO:  Processing profile    1 of      1
INFO:  Reading input data file stalled_processes/ropp_atmPhs_C001.2011.192.01.14.G09_2010.2640_nomissing_nc.

INFO:  (OC_20110711011445_C001_G009_UCAR) 

...:  Occultation point: Lat=    55.67, Lon=    83.43
INFO (from ropp_pp_preprocess):  COSMIC data preprocessing
INFO (from ropp_pp_preprocess):  Reading lost carrier flag data from input file
INFO (from ropp_pp_preprocess):  Reading external NDM data from file stalled_processes/gpsBit_C001.2011.192.01.14.G09_2010.2640_nomissing_txt
INFO (from ropp_pp_preprocess):  COSMIC data: openloop preprocessing
... (from ropp_pp_preprocess):  Navbit correllation area:           62.4           66.4
... (from ropp_pp_preprocess):  Max. navbit correlation:    0.983 at    0
INFO (from ropp_pp_cutoff):  Cut-off (bangle/impact criterion). Keep data        1 to     5303
... (from ropp_pp_cutoff):    L1 aperture:  16193.990  FFT points:  1024
... (from ropp_pp_cutoff):    L1 aperture:    825.322  FFT points:    64
... (from ropp_pp_cutoff):    L2 aperture:    934.960  FFT points:    64
... (from ropp_pp_preprocess):    L2 badness between  15.0 and  50.0 km:    13.826
... (from ropp_pp_preprocess):    Click at time(   45) =    0.385
... (from ropp_pp_preprocess):    Click at time(   54) =    0.565
... (from ropp_pp_preprocess):    Click at time(   55) =    0.585
INFO:  Retrieving bending angle profiles by GEOMETRIC OPTICS 

...:  Smoothed bending angle.
...:  Pmin =   2000.000 Pmax = 132457.749
...:  ws_go_smooth      122
...:  Full resolution bending angle.
...:  Pmin =   2000.000 Pmax = 132564.653
...:  ws_go_full      122
INFO:        5303 data points in output between 6387.2km and 6517.8km 


INFO:  Retrieving bending angle profiles by WAVE OPTICS 

...:  WM =   122
...:  W =    82
...:  WL =   -40
... (from ropp_pp_DCT):  Channel 1. Hi-res grid size =  524288
INFO (from ropp_pp_DCT):  Complex field filtering 

... (from ropp_pp_DCT):  DPW =    250.000  CFW =    11.2
... (from ropp_pp_DCT):  DP0 =   2526.556
... (from ropp_pp_DCT):  Pmin =   2526.556 Pmax=  25000.000
... (from ropp_pp_DCT):  Pmin =   2519.228 Pmax=  25000.000
... (from ropp_pp_DCT):  Channel 2. Hi-res grid size =  262144
INFO (from ropp_pp_DCT):  Complex field filtering 

... (from ropp_pp_DCT):  DPW =    250.000  CFW =    17.4
... (from ropp_pp_DCT):  DP0 =  15839.010
... (from ropp_pp_DCT):  Pmin =  15839.010 Pmax=  25000.000
... (from ropp_pp_DCT):  Pmin =  15838.889 Pmax=  25000.000
... (from ropp_pp_DCT):    ICW =  1  Pmin =   2519.228
INFO:        5303 data points in output between 6387.2km and 6517.8km 

...:  Pmin =   2519.228 Pmax = 150000.000
...:  Standard grid size nbi =   1476
INFO:  Correcting bending angle profile by STATISTICAL OPTIMISATION 

...:  Global MSIS search:   Month =  8 Lat =   70. Lon =  180.
...:    2-parameter model:ob fit: From     20017. to     56112. No. data =    362
...:    RF        =  0.0014,  0.9984
INFO:        1252 data points in output between 6387.7km and 6512.8km 

INFO:  Retrieving refractivity profile by LIN ABEL TRANSFORM with STAT OPT 

INFO:  Writing output altitude scales with respect to EGM96 GEOID
INFO:  Computing dry temperature 

INFO:  Writing to output file ropp_atmPhs_C001.2011.192.01.14.G09_2010.2640_nomissing_out.nc

And the bangle and refrac profiles look OK too.

So I think this missing data portions of the profile explain it. In which case, we can probably close this ticket as the issue is addressed in #287.

[Ian Culverwell]

Asked Santi if he was happy with this:

Hi Santi,

I think it is the missing data that is causing the problems. When I remove it from
one of the profiles (attached), ropp_pp_occ_tool runs OK and gives sensible
refractivities and bending angles. 

See https://trac.romsaf.org/ropp/ticket/375 for some discussion.  If you agree that
the missing data is the cause of your problems (ie the attached dataset works OK for
you), then I'll close the ticket as the issue is being addressed in another one
(#287).

Regards,
Ian.

No reply after 3 months so closing ticket.