Introduction of Lev2a Tdry variable

[Ian Culverwell]

Kent Bækgaard Lauritsen at DMI has suggested the following development:

Hi Dave, Ian

As we've discussed at previous PT meetings DMI uses the dry temperature. 
It can be derived by the ROPP code and is then stored in the temp variable 
in the level 2b block. Currently its value is stored in our 'atm' file that 
contains the BA and REF values.

We foresee two potential problems with this approach:

- in our 'wet' files (output from the 1dvar) there will be the T and q 
variables. A user having both atm and wet files may then get confused 
about these two different temperature values stored in a variable with 
the same name. Even though it may be described clearly in the PUM, 
we know that in reality not all users will read this document carefully.

- we would also like to 'forward model' the model T,q,P values to not 
only BA and REF but also to Tdry. Since the background files (the 'bgr' files) 
already has a temperature stored in the T variable in the level 2b block, 
Tdry cannot be stored there.


We think that the optimal solution for us would be to have an extra field 
in the level 2a block and store Tdry there. We think this would be a 
logical choice since Tdry is on the same levels as the refractivity.


DMI would therefore like to suggest the following:

1. DMI will do the necessary coding needed to introduce a Tdry variable in 
the level 2a block.

2. DMI will write a new routine for the forward modelling of T,q into Tdry 
using the forward modelling present in ROPP_PP.
This should also be part of the routine for forward modelling a BG file 
into BA and REF.

3. Met Office will make the final polishing of this code and update the 
documentation.


The DMI contributions will be done by Kjartan and Stig.
They will be in close contact with the ROPP DT in order that they 
follow as closely as possible the ROPP logic.

Please let us know your comments to this suggestion.


Kent

Sounds good to me. Timescales? Any further comments?

[Ian Culverwell]

Dave Offiler responds:

Kent,


As you know, this issue has some history! At the last 
review (for v3.0, I think), we agreed to add Bangle_opt 
(optimised bending angles), but at that time, there 
wasn't a convincing case to add Tdry, partly as there 
was a working alternative (separate files).


Since there now seems to be a realistic use-case to 
include Tdry, this would be a good time to revisit the 
issue. 


I agree we should go about this in a structured way, 
starting with discussion at the next PT.

I suggest DMI presents the use-case(s) for Tdry (I 
struggle to think why anyone wants to use Tdry and 
speaking personally, I think its use should be actively 
discouraged!) and make the case for adding as 
a new core parameter in ROPP netCDF (ie why the 
separate file method is no longer acceptable. This case 
('reason for proposed change') can be summarised in the 
new Ticket). We would then (post-PT) need to overview 
the necessary code changes (it's not a matter of just 
adding Tdry to the netCDF data structure; every bit of 
ROPP which anywhere handles Temp would have to have a 
parallel Tdry code, from I/O to range checking to 
thinning and more, as well as your new operator), and 
then agree who does what (which could be as you 
propose).


Dave

[Ian Culverwell]

Some discussion with Stig:

Hi Stig,

Thanks again.  

I've taken a look at these changes and they look fairly straightforward.

I know Dave was worried about the amount of "support" work that might have been 
needed to ensure Tdry would be properly processed wherever it appeared throughout 
the code.   Have you done much processing with ropp_fm or ropp_1dvar on the "Tdry-
inclusive files" produced by these routines?  I would hope that since Tdry is a 
lev2a object, with a new name, "downstream" processes should leave it alone. 

One strategic point: why have you removed the output_tdry option?  I just wonder 
whether some users might want backwards compatibility in their output.  Why did you 
not just keep the option but change the default to .true.?

We can talk about it at the ROPP GG2 on Friday if you wish. 

Thanks again,
Ian.

PS: Kjartan's changes to ropp_pp_openloop did make it into ROPP5.1 (r3003), but not 
ROPP5.0 (r2930) which your branch started from. The change to ropp_pp_fourier_filter 
wasn't , but then as far as I can see from the ROPP tickets it's never been raised 
as an issue.  (It's not mentioned in #237, which concerns the ropp_pp_openloop 
change.) If it was mentioned and I neglected to do it, I apologise. If I didn't, 
I'll raise a ropp ticket on it, so that the issue is recorded. 


> -----Original Message-----
> From: Culverwell, Ian
> Sent: 28 September 2012 15:48
> To: 'Stig Syndergaard'
> Cc: Culverwell, Ian
> Subject: RE: Tdry in ROPP
> 
> Hi Stig,
> 
> Thanks very much for this.  I'll take a closer look next week and get 
> back to you.
> 
> Have a good weekend,
> Ian. 
> 
> > -----Original Message-----
> > From: Stig Syndergaard [mailto:ssy@dmi.dk]
> > Sent: 28 September 2012 15:23
> > To: Culverwell, Ian
> > Cc: Kent Baekgaard Lauritsen
> > Subject: Re: Tdry in ROPP
> > 
> > Hi Ian,
> > 
> > Here is a list of the changes I made back then. It was made
> on top of
> > ROPP 5.1, but anyways the branch and revision number is
> given at the
> > end of the email, so hopefully not difficult to sort out. 
> All of these
> > changes were also merged to our dmi_trunk last year.
> > You are welcome to give me a call if/when you start looking
> into the
> > details for implementation in the official ROPP.
> > I'll be happy to help sort out things if needed.
> > 
> > 1) ropp_pp_occ_tool and ropp_pp_invert_tool:
> > - Now putting dry temperature in level2a instead of 2b (new
> dry_temp
> > variable).
> > - Calculating dry temperature before reducing output range (fix to 
> > avoid unfortunate extrapolation above the highest data point that 
> > resulted in bad initialization of hydrostatic integration).
> > - Skipping option output_tdry; will now always output dry
> temperature.
> > 
> > 2) ropp_pp_invert_tool:
> > - Fix to allow it to run with diagnostics output (it broke when 
> > setting that option).
> > - Now gracefully terminates when there are missing values in the 
> > input.
> > This is only a temporary fix, and a problem that needs further 
> > attention. The thing is that we won't be able to use 
> > ropp_pp_invert_tool on bending angles with missing values, which we 
> > basically always get from Eumetsat in NRT.
> > 
> > 3) ropp_pp_read_config.f90 and ropp_pp_types.f90:
> > - Changes to skip the output_tdry option
> > 
> > 4) gras_pp.cf, cosmic_pp.cf, champ_pp.cf, default_pp.cf, and
> > grassaf_invert.cf:
> > - Changes to skip the output_tdry option
> > 
> > 5) grassaf_invert.cf:
> > - Changes to be in accordance with the GPAC NRT procesing.
> > 
> > 6) ropp_pp.f90 and ropp_pp_tdry.f90:
> > - Skipping Zmax as argument; Zmax not needed after rearranging the 
> > calculation of dry temperature and output range reduction in 
> > ropp_pp_occ_tool and ropp_pp_invert_tool; i.e., always
> initialize dry
> > temperature from highest point in profile, including the model 
> > profile.
> > 
> > 7) ropp_pp_tdry.f90:
> > - Call to GravityGC under 7.1 now with correct argument (GCLat)
> > - Fixed mistake regarding epsilon in formula in function
> FTZP Both of
> > these had only very little influence on the calculated dry 
> > temperature; code now consistent with OCC code.
> > 
> > 8) tangent_point.f90 and occ_point.f90:
> > - Fixed bug in calculation of azimuth angle.
> > 
> > 9) ropp_pp_openloop.f90 and ropp_pp_fourier_filter.f90 Patches from 
> > Kjartan that didn't make it to ROPP 5.1.
> > 
> > 10) ropp_io_types.f90, ropp_io_read_ncdf_get.f90, 
> > ropp_io_write_ncdf_def.f90, ropp_io_write_ncdf_put.f90, 
> > ropp_io_ascend.f90, ropp_io_shrink.f90, ropp_io_free.f90, 
> > ropp_io_init.f90, ropp_io_assign.f90, ropp_io_rangecheck.f90, 
> > test2ropp.f90, gfz2ropp.f90, ropp_io_thin.f90:
> > - Included dry_temp variable
> > 
> > Actual code changes that I made in my branch can be seen at 
> > https://trac.grassaf.org/ropp/log/ropp_src/branches/dev/Share/
> > ssy_5.0_SU?rev=3166
> > 
> > 
> > Best regards,
> > -Stig
> > 
> > On 2012-09-28 09:37, Culverwell, Ian wrote:
> > > Hi Stig,
> > >
> > > I was asked to contact you at the last ROPP GG meeting to
> > discuss the
> > > feasibility of getting DMI's Tdry work into ROPP6.1, which
> > is scheduled
> > > for release at the end of the year.  Could you give me some
> > details of
> > > what's already been done please?
> > >
> > > Many thanks,
> > > Ian.
> > 
> > --
> > __________________________________________________________________
> > Stig Syndergaard, PhD
> > Danish Meteorological Institute              Phone: +45 39 157 408
> > Lyngbyvej 100                                  Fax: +45 39 157 461
> > DK-2100 Copenhagen                              E-mail: ssy@dmi.dk
> > __________________________________________________________________
> > 
> > 

And his reply:

Hi Ian,

On 2012-10-01 16:04, Culverwell, Ian wrote:
> Hi Stig,
>
> Thanks again.

No problem!

>
> I've taken a look at these changes and they look fairly straightforward.
>
> I know Dave was worried about the amount of "support" work that might 
> have been needed to ensure Tdry would be properly processed wherever it
> appeared throughout the code.   Have you done much processing with
> ropp_fm or ropp_1dvar on the "Tdry-inclusive files" produced by these 
> routines?  I would hope that since Tdry is a lev2a object, with a new 
> name, "downstream" processes should leave it alone.

Joe has been using our dmi_trunk (in which Tdry has been for almost a 
year) for his 1dvar work, and he has not encountered any troubles. And he does see
Tdry in his files although he is not using it. So I think it works as it should, but
it probably should be checked more systematically.

>
> One strategic point: why have you removed the output_tdry option?  I 
> just wonder whether some users might want backwards compatibility in 
> their output.  Why did you not just keep the option but change the 
> default to .true.?

I don't remember exactly, I think I thought it didn't make much sense to exclude it
if it was to become a more official parameter on par with the other 2a variables. If
the output_Tdry option was kept but set to false, I suppose the fields would still
be written, just with missing values. 
So backwards compatibility would be lost anyways (unless I was to do something extra
in the I/O module to avoid writing the fields if all missing). Maybe that's why I
did it.

But I also made another change around the same time (also in our dmi_trunk, but not
sure if I did it directly in the dmi_trunk or in my own branch - I can find out)
that included new options in the .cf files to exclude whole levels (one or more of
1a, 1b, 2a). That seems more useful, since especially lev1a makes output files quite
big, and for some purposes we didn't need it and were re-reading and re-writing ROPP
files after using the occ tool to get rid of level 1a. So that's why I made that
optional in the occ tool. Maybe that was already in my thoughts when I decided to
skip the output_tdry option.

>
> We can talk about it at the ROPP GG2 on Friday if you wish.
>

Talk to you Friday. But also feel free to call me at other times if you want to talk
about the issues (Wednesday - Friday; I'm not here Mondays and Tuesdays).


> Thanks again,
> Ian.
>
> PS: Kjartan's changes to ropp_pp_openloop did make it into ROPP5.1 
> (r3003), but not ROPP5.0 (r2930) which your branch started from. The 
> change to ropp_pp_fourier_filter wasn't , but then as far as I can see 
> from the ROPP tickets it's never been raised as an issue.  (It's not 
> mentioned in #237, which concerns the ropp_pp_openloop change.) If it 
> was mentioned and I neglected to do it, I apologise. If I didn't, I'll 
> raise a ropp ticket on it, so that the issue is recorded.

I remember talking to Kjartan about this, and I think he concluded that the change
was not strictly necessary, only if one wanted to match the output from Michaels OCC
tool exactly. I'm cc'ing Kjartan, may he can say more.

Kind Regards,
-Stig
>

[Ian Culverwell]

Successfully merged into ropp6.1 prototype. There was some complication arising from the fact that Stig's branch was based on ropp5.0, not 5.1, so some of his changes had already been implemented. In summary:

1) Included fourier_filter fix, but not the openloop fix, as this had already been done at 5.1;
2) Left output_tdry as an option, but defaulted to .true., so that existing
   users could continue setting output_tdry in their config files;
3) Avoided tangent point bug-fix - ask Stig to compare with CB's fix #233;
4) Included fix in ropp_pp_tdry to guard against refrac < 0
5) Include Tdry etc in test datasets to enable ROPP to pass the "make test" tests.

[Ian Culverwell]

After a bit of effort, the Tdry changes have been incorporated into the ROPP6.1 devt branch. Only real issue: one of the checks against entirely missing L1 and L2 data (as for UCAR datasets) was tripping up our test folder which had some missing data. Moving the check after the bit that "shrinks" the profile by removing the missing parts fixes it.

UG documentation kindly updated by Stig - no problems.

No further problems, so closing the ticket.