Add support for GNOS

[Dave Offiler]

We may soon be able to gain access to Chinese RO data from the GNOS instrument on FY-3C which was launched earlier this year. Intitially a batch of offline data (likely in ROPP netCDF format) and perhaps eventually in BUFR over the GTS.

Currently the ROPP BUFR encoder/decoder does not support this mission, so the convertcodes.f90 and roppbufrcodes.nl files need to be updated to include the appropriate mappings.

A code for FY-3C (522) is ready for formal approval and can be expected to be included in the next version of operational tables (V23, Nov 2014?).

Unfortunately, an instrument code for GNOS, and a satellite series code for Beidou (the Chinese GPS) do not appear to have been even proposed, so for now, assumed values will have to be included. The logical sequence of existing codes would suggest 404 for Beidou and 956 for GNOS, but both need to be formally proposed and accepted by WMO before these can be used operationally (on the GTS). Since this is unlikely to happen until after the release of ROPP-8, if different values are defined, then a patched namelist file can be provided and the F90 code updated for the following release.

This ticket is to provide provisional support in ROPP-8 by adding mappings for:

• Satellite ID ('FY3C'[*] <-> 522)
• Satellite series ID (Beidou - 'B'[*] <-> 404[])
• Satellite instrument ID ('GNOS' <-> 956[])
• Originating Centre (CMA[*] <-> 38)

[*] TBC when example netCDF files are available [] assumed BUFR codes TBC when WMO publish actual codes

[Dave Offiler]

Entries in namelist and convert routine updated as above in my do_exitcodes working branch.

Tested by:

1. Additional code entries inserted into run-time BUFR table bufr_codefig and 'local' entry '000' added as sub-centre to '038' (Beijing) in bufr_origcentre file. While these entries are not necessary to the encode/decode, they do assist in checking the decode dumps have re-mapped to the expected entries.

2. Using ncdump, create cdl file from /data/ropp_test.nc; edit the cdl file and change the institution and originating_centre IDs to "CMA", gns_id first letter to "B" and leo_id to "FY3C"; use ncgen to generate a new netCDF file ropp_test1.nc.

3. Encode to BUFR using the pre-built ropp2bufr; decode with decbufr -d and confirm that the satellite ID is displayed as 'missing data' flag.

4. Copy modified namelist to $BUFR_LIBRARY; re-run step 3; confirm originating centre (including BUFR Section 1), satellite, instrument and classification (series) IDs are as expected. Validated namelist.

5. Rebuild convertcodes() and ropp2bufr (and other IO tools) with make locally.

6. Temporarily rename namelist and re-run step 4 but with new ropp2bufr. Warning message about missing namelist and use of internal table; confirm decoded originating centre (including BUFR Section 1), satellite, instrument and classification (series) IDs are as expected. Validates internal table.

NB. Only tested with default ifort and with provisional codes (BUFR and/or ROPP netCDF IDs) in a hacked sample ROPP netCDF file. Needs re-checking against a real GNOS file when one (or more) is available. However, should the details need to change after release of ROPP-8, a final patched namelist file can be provided to users.

Ideally, the modified BUFR library run-time files supporting (nominal) FY-3C/GNOS/Beidou should be included in the MetO BUFR package to be used with ROPP-8. The MetDB Team have it on their task list to generate the latest (V22) BUFR tables compatible with the package from the WMO XML files, but I'm told this will not be done for some while yet. In any case, the formal entry for FY-3C is not due in the operational tables until V23 and the other entries later still. However, the actual binary encode/decode process doesn't require these new code-table entries; they are only used for interpreting the decoded dumps.

[Dave Offiler]

Changes committed as [4277]. Leavign ticket open until reviewed & accepted for ROPP-8.

[Ian Culverwell]

Merged in Dave's changes at [4283].

Not sure this is entirely working. We now get:

  Originating Centre:          38 (Beijing (RSMC))
 SATELLITE IDENTIFIER                                  Code 001007          522
 SATELLITE INSTRUMENTS                                 Code 002019          956
 IDENTIFICATION OF ORIGINATING/GENERATING CENTRE       Code 001033 BABJ BEIJING
 SATELLITE CLASSIFICATION                              Code 002020          404

instead of

  Originating Centre:           0 (WMO Secretariat)
 SATELLITE IDENTIFIER                                  Code 001007    - - - - -
 SATELLITE INSTRUMENTS                                 Code 002019    - - - - -
 IDENTIFICATION OF ORIGINATING/GENERATING CENTRE       Code 001033            0
 SATELLITE CLASSIFICATION                              Code 002020    - - - - -

But the COSMIC data in #376 gave:

  Originating Centre:          60 (NCAR, Boulder CO)
 SATELLITE IDENTIFIER                                  Code 001007     COSMIC-6
 SATELLITE INSTRUMENTS                                 Code 002019 CHAMP GPS so
 IDENTIFICATION OF ORIGINATING/GENERATING CENTRE       Code 001033         NCAR
 SATELLITE CLASSIFICATION                              Code 002020          GPS

Why haven't we got nice readable words instead of codes?

[Ian Culverwell]

... because I hadn't updated bufr_codefig and bufr_origcentre. When I do so, I get

  Originating Centre:          38 (Beijing (RSMC))
 SATELLITE IDENTIFIER                                  Code 001007        FY-3C
 SATELLITE INSTRUMENTS                                 Code 002019      GNOS RO
 IDENTIFICATION OF ORIGINATING/GENERATING CENTRE       Code 001033 BABJ BEIJING
  Originating Centre:          38 (Beijing (RSMC))
  Originating Sub-Centre:       0 (Beijing (China))
 SATELLITE CLASSIFICATION                              Code 002020       BEIDOU

which makes more sense. bufr_codefig and bufr_origcentre attached.

[Dave Offiler]

Updated BUFR tarball to include modified bufr_codefig and bufr_origcentre files. Package tarball bufr-20.0.3.tar.gz copied to $ROPP_ARC and built for all Linux compilers[*] using buildmobufr_ropp. This tarball should be used as the (MO)BUFR dependency for ROPP-8.

[*] NB: the sunf95 compiler again has problems finding it's own .mod files, so the BUFR package fails to build on ROPP_ARC. There seems to be a problem with sunf95 accessing files on the hardware (or controller) hosting /data/nwp1 which we've seen before; the package builds OK with sunf95 under HOME or LOCALDATA, so there's no fundamental problem with the code, the compiler or the compile options. The solution is to build with sunf95 on any partition except /data.

[Ian Culverwell]

bufr-20.0.3 seems to work OK: it generates

 SATELLITE IDENTIFIER                                  Code 001007        FY-3C
 SATELLITE INSTRUMENTS                                 Code 002019         GNOS
 SATELLITE CLASSIFICATION                              Code 002020       BEIDOU
 VERTICAL SIGNIFICANCE (SATELLITE OBSERVATIONS)        Code 008003      SURFACE
 IDENTIFICATION OF ORIGINATING/GENERATING CENTRE       Code 001033 BABJ BEIJING
  Originating Centre:          38 (Beijing (RSMC))
  Originating Sub-Centre:       0 (Beijing (China))

although I notice that there isn't an entry like

 522 FY-3C                                                       !1.x

at about line 120 in bufr-20.0.3/MetDB_BUFR20.0.01/bufr_codefig, as there is in my attached version of bufr-19.1.2/metdb/CODEFIG (assuming these to be corresponding files). Does this matter? The output above looks OK, but should such an entry be there for completeness anyway?

[Ian Culverwell]

Turns out it's bufr-20.0.3/extra/bufr_codefig that it uses (probably), and that's got the required entries

522 FY-3C

and

  404 BEIDOU

Rerunning after deleting bufr-20.0.3/extra/bufr_codefig/CODEFIG gives same as before, so problem solved.

[Dave Offiler]

Ian confirms GNOS support is working, so closing this Ticket.

However, it should be noted that this support is provisional upon confirmation of character codes to be included in the ROPP netCDF source files that ropp2bufr inputs, and the assumed WMO numeric codes being adopted operationally. If any code is not the assumed value, a replacement roppbufrcodes.nl file can be provided; the default values hard-coded in the source code can be updated for a subsequent release.

[Ian Culverwell]

Reopening ticket for ROPP10.0 because Axel von Engeln (EUM) says:

I recently had some issue on how to identify the BeiDou 
/ Compass constellation. E.g. for GPS we use G and then 
the PRN, for Galileo E and for GLONASS R. For our 
occultation simulator we actually use C for COMPASS. 
But it seems you/ROPP are using B:

ropp-9.0/ropp_io/bufr/roppbufrcodes.nl:GNSlist = "U",      "G", "R", "E", "B"     ! [GPS, GLONASS, Galileo, Beidou]

As far as I understood from Yago, C is actually the one 
used in the IGS community, not B. Is the B here a BUFR 
thing or could you update that in ROPP10?

[Ian Culverwell]

Yago Andres has kindly sent me the attached IGS standard files rinex303.pdf and sp3d.pdf, which show Beidou <--> 'C' (for 'COMPASS', the basename of the individual satellites in the Beidou constellation, I think).

[Ian Culverwell]

rinex303.pdf

[Ian Culverwell]

sp3d.pdf

[Ian Culverwell]

From rinex303.pdf:

9.3 BDS Satellite System Code
The satellite system code for BeiDou navigation satellite System (BDS) has been defined as “C”,
see Figure 1.

[Ian Culverwell]

rinex303_fig1.png

[Ian Culverwell]

And from sp3d.pdf:

Each identifier will consist of a letter followed by a 
2-digit integer between 01 and 99. For example, "Gnn" for 
GPS satellites, "Rnn" for GLONASS satellites, "Lnn"
for Low-Earth Orbiting (LEO) satellites, “Snn” for 
Satellite-Based Augmentation System (SBAS) satellites, 
"Enn" for Galileo satellites, "Cnn" for BeiDou Navigation 
Satellite System (BDS) satellites, “Inn” for the
Indian Regional Navigation Satellite System (IRNSS) 
satellites, and "Jnn" for the Japanese Quasi-Zenith 
Satellite System (QZSS) satellites.

So the grounds for using 'C' for Beidou seem sound.

[Ian Culverwell]

Given the ties between this ticket and #488 it makes sense to do this development in https://trac.romsaf.org/ropp/browser/ropp_src/branches/dev/Share/ic_gnos.

[Ian Culverwell]

A_IUTA14BAWX160008_C_BAWX_20170616025049.bin

[Ian Culverwell]

Mi has sent the attached example GNOS/FY-3C BUFR file A_IUTA14BAWX160008_C_BAWX_20170616025049.bin. She also says

   The bufr codes are listed as follows.
   522                FY3C   SATELLITE ID
   523                FY3D   SATELLITE ID
   958	              GNOS   INSTRUMENT ID
   404	              BDS (BeiDou Navigation Satellite System)

[Ian Culverwell]

Since they're chosen a code for FY-3D, we should probably incorporate support for this now.

[Ian Culverwell]

This requires a change to $BUFR_TABLES/bufr_codefig. We also need to make sure roppbufrcodes.nl is copied to $BUFR_TABLES. (Why do we need this namelist? You have to define things in ropp_io/bufr/convertcodes.f90 anyway, but then they're overwritten by the namelists in $BUFR_TABLES/roppbufrcodes.nl, which you have to remember to overwrite, or include in the BUFR tarball somehow. Surely the namelist under $BUFR_TABLES should be the default, which can then be overwritten by the user-controllable convertcodes.f90 version?)

Be that as it may, changes to account for Beidou and FY-3D identifiers, as tested in the attached test3.sh file, seem to work OK. On Mi's example bufr file, bufr2ropp produces

 occ_id =
  "OC_20170616000842_FY3C_G013_CMA_" ;

 gns_id =
  "G013" ;

 leo_id =
  "FY3C" 

and ropp2bufr on this produces

 SATELLITE IDENTIFIER                                  Code 001007        FY-3C
 SATELLITE INSTRUMENTS                                 Code 002019         GNOS
 IDENTIFICATION OF ORIGINATING/GENERATING CENTRE       Code 001033           38
 SATELLITE CLASSIFICATION                              Code 002020          GPS

(which matches the original file).

When we artificially set Lcode = 523 and Gclass = 404 in ropp_io/bufr/convertcodes.f90, bufr2ropp produces

 occ_id =
  "OC_20170616000842_FY3D_C013_CMA_" ;

 gns_id =
  "C013" ;

 leo_id =
  "FY3D" ;

and ropp2bufr on this produces

 SATELLITE IDENTIFIER                                  Code 001007        FY-3D
 SATELLITE INSTRUMENTS                                 Code 002019         GNOS
 IDENTIFICATION OF ORIGINATING/GENERATING CENTRE       Code 001033           38
 SATELLITE CLASSIFICATION                              Code 002020       BEIDOU

So that seems OK. (Can also pass this last BUFR file through bufr2ropp and get the same as the input netCDF file.)

[Ian Culverwell]

test3.sh

[Ian Culverwell]

Changes committed at r5526. The revised bufr_codefig file, which, like the new roppbufrcodes.nl, needs to be installed (by copying or inclusion in the BUFR tarball) under $BUFR_TABLES, is attached to this ticket.

[Ian Culverwell]

bufr_codefig

[Ian Culverwell]

roppbufrcodes.nl_ROSA

[Ian Culverwell]

There's another namelist in $BUFR_TABLES, roppbufrcodes.nl_ROSA, which is also attached to this ticket. Obviously this is to facilitate reading of MT-ROSA data. There are also tickets to allow ROPP to read KOMPSAT5 data (#494) and GRACE-FO data (#507). Suggest closing this ticket, and return to the question of the right way to do this in #494 or #507.

[Ian Culverwell]

bfr20180101_000016_M01_2010657500_N0023_XXXX.bin

[Ian Culverwell]

bfr20180101_000016_M01_2010657500_N0023_XXXX.log

[Ian Culverwell]

bfr20180101_000016_M01_2010657500_N0023_XXXX.decbufr

[Ian Culverwell]

As part of the general look at BUFR (en)coding, some more diagnostics have been added to ropp_io/bufr/bufr2ropp_mod.f90 at r5548. These allow the ROPP user to easily examine the contents of the file without running it through (something like) decbufr. The diags are activated by use of the -d switch to bufr2ropp.

When run on the attached SAF-generated BUFR file bfr20180101_000016_M01_2010657500_N0023_XXXX.bin it generates bfr20180101_000016_M01_2010657500_N0023_XXXX.log, which should be compared to the decbufr output in bfr20180101_000016_M01_2010657500_N0023_XXXX.decbufr. A clear snapshot of the BUFR file has obviously been produced.

[Ian Culverwell]

bfrPrf_KOM5.2018.182.00.04.G27_0001.0007_bufr

[Ian Culverwell]

bfrPrf_KOM5.2018.182.00.04.G27_0001.0007_log

[Ian Culverwell]

bfrPrf_KOM5.2018.182.00.04.G27_0001.0007_decbufr

[Ian Culverwell]

The same is true of the KOMPSAT-5 BUFR file bfrPrf_KOM5.2018.182.00.04.G27_0001.0007_bufr, which was downloaded from the CDAAC website. The stdout in bfrPrf_KOM5.2018.182.00.04.G27_0001.0007_log should be compared to the results of running decbufr, in bfrPrf_KOM5.2018.182.00.04.G27_0001.0007_decbufr. Again, the -d option provides a more readable diagnostic of the content of the BUFR file.

[Ian Culverwell]

A_IUTA14BAWX160008_C_BAWX_20170616025049_new.bin

[Ian Culverwell]

A bit of binary jiggery-pokery on A_IUTA14BAWX160008_C_BAWX_20170616025049.bin converts it to A_IUTA14BAWX160008_C_BAWX_20170616025049_new.bin, which has the codes for FY-3D (sat id=523) and Beidou (sat class = 404). Attached to ticket.

When run though ROPP9.0 control, this gives

gns_id[0--4]="B013" 
leo_id[0--4]="UNKN" 
occ_id[0--32]="OC_20170616000842_UNKN_B013_CMA_" 

and, on conversion back to bufr,

 SATELLITE IDENTIFIER                                  Code 001007    - - - - -
 SATELLITE INSTRUMENTS                                 Code 002019    - - - - -
 SATELLITE CLASSIFICATION                              Code 002020       BEIDOU

The ic_gnos prototype gives

TEST:
gns_id[0--4]="C013" 
leo_id[0--4]="FY3D" 
occ_id[0--32]="OC_20170616000842_FY3D_C013_CMA_" 

and

 SATELLITE IDENTIFIER                                  Code 001007        FY-3D
 SATELLITE INSTRUMENTS                                 Code 002019         GNOS
 SATELLITE CLASSIFICATION                              Code 002020       BEIDOU

All looks OK.

[Ian Culverwell]

Changes to recognize the FY-3D BUFR codes committed at r5550.

[Ian Culverwell]

T_IUTF14_C_DEMS_20180223075850_G05_ROSA_MT1.bin

[Ian Culverwell]

Note that the ICAO codes provided by the data generators can be found with a binary dump:

idculv@eld037:> od -c T_IUTF14_C_DEMS_20180223075850_G05_ROSA_MT1.bin |head -5 
0000000   I   U   T   F   1   4       D   E   M   S       2   3   0   7
0000020   5   8  \r  \r  \n   B   U   F   R  \0 034 361 004  \0  \0 026
0000040  \0  \0 034  \0  \0  \0  \0 003   2 016  \f  \0  \a 342 002 027
0000060  \a   :   2  \0  \0  \t  \0  \0 001 200 312 032  \0 034 306  \0
0000100   n  \b 370 340 020 310 021   ~   "   \ 375   a 250 020  \0 177

DEMS is right there, after the ARH (IUT[A-L]14) (which also needs checking, to make sure it doesn't share the EUMETSAT CAF anomalies).

[Ian Culverwell]

Attaching the MGTP file deleted the text that should have gone before it. Here it is:

Try the same thing on Jignesh's example Megha-Tropiques file T_IUTF14_C_DEMS_20180223075850_G05_ROSA_MT1.bin, which is attached.

ROPP9.0 gives

gns_id[0--4]="G005" 
leo_id[0--4]="UNKN" 
occ_id[0--32]="OC_20180223075850_UNKN_G005_UNKN" 

and then

  Originating Centre:       65535 (INVALID CENTRE CODE VALUE)
  Originating Sub-Centre:       0 (INVALID CENTRE CODE VALUE)
 SATELLITE IDENTIFIER                                  Code 001007    - - - - -
 SATELLITE INSTRUMENTS                                 Code 002019    - - - - -
 IDENTIFICATION OF ORIGINATING/GENERATING CENTRE       Code 001033    - - - - -

ic_gnos branch says

gns_id[0--4]="G005" 
leo_id[0--4]="MGTP" 
occ_id[0--32]="OC_20180223075850_MGTP_G005_ISRO"

and then

  Originating Centre:          28 (New Delhi (RSMC))
  Originating Sub-Centre:       0 (none)
 SATELLITE IDENTIFIER                                  Code 001007 Megha-Tropiq
 SATELLITE INSTRUMENTS                                 Code 002019         ROSA
 IDENTIFICATION OF ORIGINATING/GENERATING CENTRE       Code 001033           28

All looking good, so changes committed to repository at r5551.

[Ian Culverwell]

bfrPrf_KOM5.2018.182.00.04.G27_0001.0007_bufr

[Ian Culverwell]

Ditto for the example KOMPSAT5 dataset bfrPrf_KOM5.2018.182.00.04.G27_0001.0007_bufr, downloaded from the CDAAC website, and attached to this ticket.

ROPP9.0 gives

gns_id[0--4]="G027" 
leo_id[0--4]="UNKN" 
occ_id[0--32]="OC_20180701000150_UNKN_G027_UCAR" 

and then

 SATELLITE IDENTIFIER                                  Code 001007    - - - - -
 SATELLITE INSTRUMENTS                                 Code 002019    - - - - -

whereas ic_gnos says

gns_id[0--4]="G027" 
leo_id[0--4]="KOM5" 
occ_id[0--32]="OC_20180701000150_KOM5_G027_UCAR" 

and then

 SATELLITE IDENTIFIER                                  Code 001007    KOMPSAT-5
 SATELLITE INSTRUMENTS                                 Code 002019         IGOR

which looks OK. (KOMPSAT-5 uses an IGOR RO receiver as part of its AOPOD payload.)

[Ian Culverwell]

Changes committed at r5552.

[Ian Culverwell]

The example GRACE-A bufr file GA-AI-3-NRT+2011_052_1100_G23_zdif_006.bin, produced by running the source GFZ files GA-AI-3-NRT+2011_052_1100_G23_zdif_006.dat and GA-AI-3-NRT+2011_052_1100_G23_zdif_006.dsc through gfz2ropp and ropp2bufr, is attached to this ticket. Both ROPP9.0 and ic_gnos handle it OK:

gns_id[0--4]="G023" 
leo_id[0--4]="GRAA" 
occ_id[0--32]="OC_20110221105950_GRAA_G023_GFZ_" 

and then

SATELLITE IDENTIFIER                                  Code 001007      GRACE A
SATELLITE INSTRUMENTS                                 Code 002019 CHAMP GPS so

[Ian Culverwell]

GA-AI-3-NRT+2011_052_1100_G23_zdif_006.bin

[Ian Culverwell]

GA-AI-3-NRT+2011_052_1100_G23_zdif_006.dat

[Ian Culverwell]

GA-AI-3-NRT+2011_052_1100_G23_zdif_006.dsc

[Ian Culverwell]

GA-AI-3-NRT+2011_052_1100_G23_zdif_006.nc

[Ian Culverwell]

Ditto for the pseudo GRACE-B bufr file GB-AI-3-NRT+2011_052_1100_G23_zdif_006.bin, which is generated from GA-AI-3-NRT+2011_052_1100_G23_zdif_006.nc by means of

ncap2 -s'occ_id(:,0:31)="OC_20110221105950_GRAB_G023_GFZ_";leo_id(:,0:3)="GRAB"' GA-AI-3-NRT+2011_052_1100_G23_zdif_006.nc -o GB-AI-3-NRT+2011_052_1100_G23_zdif_006.nc

$ROPP_ROOT/ropp_src/branches/dev/Share/ic_gnos/ropp_io/tools/ropp2bufr -d GB-AI-3-NRT+2011_052_1100_G23_zdif_006.nc -o GB-AI-3-NRT+2011_052_1100_G23_zdif_006.bin.

This produces

gns_id[0--4]="G023" 
leo_id[0--4]="GRAB" 
occ_id[0--32]="OC_20110221105950_GRAB_G023_GFZ_" 

and then

 SATELLITE IDENTIFIER                                  Code 001007      GRACE B
 SATELLITE INSTRUMENTS                                 Code 002019 CHAMP GPS so

with both ROPP9.0 and ic_gnos.

[Ian Culverwell]

GB-AI-3-NRT+2011_052_1100_G23_zdif_006.nc

[Ian Culverwell]

GB-AI-3-NRT+2011_052_1100_G23_zdif_006.bin

[Ian Culverwell]

For the pseudo GRACE-C (FO) file GC-AI-3-NRT+2011_052_1100_G23_zdif_006.bin, generated in an analagous way to the GRACE-B file, ROPP9.1 gives

gns_id[0--4]="G023" 
leo_id[0--4]="UNKN" 
occ_id[0--32]="OC_20110221105950_UNKN_G023_GFZ_" 

and then

 SATELLITE IDENTIFIER                                  Code 001007    - - - - -
 SATELLITE INSTRUMENTS                                 Code 002019    - - - - -

whereas ic_gnos gives

gns_id[0--4]="G023" 
leo_id[0--4]="GRAC" 
occ_id[0--32]="OC_20110221105950_GRAC_G023_GFZ_" 

and then

 SATELLITE IDENTIFIER                                  Code 001007 GRACE C (GRA
 SATELLITE INSTRUMENTS                                 Code 002019 Triple-G (GP

(Or at least, it gives the latter if we add

  001007 SATELLITE IDENTIFIER
...
803 GRACE C (GRACE-FO)                                          !1.x
804 GRACE D (GRACE-FO)                                          !1.x

and

  002019 SATELLITE INSTRUMENTS                                    !1.12
...
104 Triple-G (GPS, Galileo, GLONASS)                             !1.x

to $BUFR_TABLES/bufr_codefig.)

[Ian Culverwell]

GC-AI-3-NRT+2011_052_1100_G23_zdif_006.nc

[Ian Culverwell]

GC-AI-3-NRT+2011_052_1100_G23_zdif_006.bin

[Ian Culverwell]

And likewise for GD-AI-3-NRT+2011_052_1100_G23_zdif_006.bin: ROPP9.1 gives

gns_id[0--4]="G023" 
leo_id[0--4]="UNKN" 
occ_id[0--32]="OC_20110221105950_UNKN_G023_GFZ_" 

and

 SATELLITE IDENTIFIER                                  Code 001007    - - - - -
 SATELLITE INSTRUMENTS                                 Code 002019    - - - - -

whereas ic_gnos gives

gns_id[0--4]="G023" 
leo_id[0--4]="GRAD" 
occ_id[0--32]="OC_20110221105950_GRAD_G023_GFZ_" 

and then

 SATELLITE IDENTIFIER                                  Code 001007 GRACE D (GRA
 SATELLITE INSTRUMENTS                                 Code 002019 Triple-G (GP

[Ian Culverwell]

GD-AI-3-NRT+2011_052_1100_G23_zdif_006.nc

[Ian Culverwell]

GD-AI-3-NRT+2011_052_1100_G23_zdif_006.bin

[Ian Culverwell]

Checked that all is well if we copy ropp_io/bufr/roppbufrcodes.nl to $BUFR_TABLES, so that we use this namelist rather than the one inside ropp_io/bufr/convertcodes.f90. OK. Changes committed at r5553.

[Ian Culverwell]

Check that the namelists match. In convertcodes.f90:

Intrinsic namelists
 &GNSCODES
 GNSLIST = UGRECU,
 GNSCODE =  2147483647,         401,         402,         403,         404,  2147483647
 /
 &LEOCODES
 LEOLIST = UNKNOERSCHMPSUNSSACCGRAAGRABGRACGRADC001C002C003C004C005C006METAMETBMETCTSRXTDMXPAZEOSATCNOFMGTPFY3CFY3DKOM5C2E1C2E2C2E3C2
 E4C2E5C2E6C2P1C2P2C2P3C2P4C2P5C2P6UNKNUNKNUNKN,
 LEOCODE =  2147483647,          40,          41,         800,         820,         722,         723,         803,         804,
          740,         741,         742,         743,         744,         745,           4,           3,           5,          42,
           43,          44,         421,         786,         440,         522,         523,         825,         750,         751,
          752,         753,         754,         755,         724,         725,         726,         727,         728,         729,
  3*2147483647,
 INSCODE =  2147483647, 6*102, 2*104, 6*102, 3*202, 3*103,         287,         102,         287, 2*958,         103,
  12*104, 3*2147483647
 /
 &ORGCODES
 ORGLIST = UNKNOWN DMI     GFZ     METO    UCAR    NESDIS  EUMETSATCMA     ISRO    UNKNOWN UNKNOWN ,
 ORGCODE =  2147483647,          94,          78,          74,          60,         160,         254,          38,          28,
  2*2147483647,
 SUBCODE =  2147483647,           0,         173, 8*0,
 ORGNAME =                                    (ROM SAF)                          Helmholtz Centre, Potsdam          Met Office, Exete
 r                 Boulder                            Washington                         Darmstadt                          Beijing  
                           New Delhi                                                                                                ,
 ICAOCODE        = ZZZZEKMIEDZWEGRRKWBCKNESEUMSBAWXDEMSZZZZZZZZ
 /
 &BGDCODES
 BGDLIST = UNKNOWN ECMWF   DMI     METO    NCEP    CMA     ISRO    NONE    UNKNOWN UNKNOWN UNKNOWN ,
 BGDCODE =  2147483647,          98,          94,          74,           7,          38,          28, 4*2147483647
 /

In roppbufrcodes.nl:

Extrinsic namelists
 &GNSCODES
 GNSLIST = UGRECU,
 GNSCODE =    -9999999,         401,         402,         403,         404,  2147483647
 /
 &LEOCODES
 LEOLIST = UNKNOERSCHMPSUNSSACCGRAAGRABGRACGRADC001C002C003C004C005C006METAMETBMETCTSRXTDMXPAZEOSATCNOFMGTPFY3CFY3DKOM5C2E1C2E2C2E3C2
 E4C2E5C2E6C2P1C2P2C2P3C2P4C2P5C2P6UNKNUNKNUNKN,
 LEOCODE =    -9999999,          40,          41,         800,         820,         722,         723,         803,         804,
          740,         741,         742,         743,         744,         745,           4,           3,           5,          42,
           43,          44,         421,         786,         440,         522,         523,         825,         750,         751,
          752,         753,         754,         755,         724,         725,         726,         727,         728,         729,
  3*2147483647,
 INSCODE =    -9999999, 6*102, 2*104, 6*102, 3*202, 3*103,         287,         102,         287, 2*958,         103,
  12*104, 3*2147483647
 /
 &ORGCODES
 ORGLIST = UNKNOWN DMI     GFZ     METO    UCAR    NESDIS  EUMETSATCMA     ISRO    UNKNOWN UNKNOWN ,
 ORGCODE =    -9999999,          94,          78,          74,          60,         160,         254,          38,          28,
  2*2147483647,
 SUBCODE =    -9999999,           0,         173, 8*0,
 ORGNAME =                                    (ROM SAF)                          Helmholtz Centre, Potsdam          Met Office, Exete
 r                 Boulder                            Washington                         Darmstadt                          Beijing  
                           New Delhi                                                                                                ,
 ICAOCODE        = ZZZZEKMIEDZWEGRRKWBCKNESEUMSBAWXDEMSZZZZZZZZ
 /
 &BGDCODES
 BGDLIST = UNKNOWN ECMWF   DMI     METO    NCEP    CMA     ISRO    NONE    UNKNOWN UNKNOWN UNKNOWN ,
 BGDCODE =    -9999999,          98,          94,          74,           7,          38,          28, 4*2147483647
 /

Tkdiff shows no difference, except for the different missing data indicators (2147483647 and -9999999). This doesn't seem to cause any problems.

Note that to get this match, I had to change, at r5554, the first SUBcode from 000 to NVIND in convertcodes.f90.

(For future reference, do this with some simple print statements in convertcodes.f90:

  print*, 'Intrinsic namelists'
  write(*, NML=GNScodes)
  write(*, NML=LEOcodes)
  write(*, NML=ORGcodes)
  write(*, NML=BGDcodes)

.)

[Ian Culverwell]

ROPP IO UG updated to reflect new LEOs at r5555.

[Ian Culverwell]

Test folder testing revealed a bug, which has been fixed at r5560.

[Ian Culverwell]

The example FY-3D file, in 'pseudo-UCAR' format, FY3D_GNOSX_GBAL_L1_20180609_2352_AEG09_MS.NC (attached), says

fileStamp = "G002.2018.160.23.52.G09" ;

CMA are therefore using G002 to identify FY-3D. The UCAR data reading routines therefore need updating to account for this, which they have been at r5577.

[Ian Culverwell]

FY3D_GNOSX_GBAL_L1_20180609_2352_AEG09_MS.NC

[Ian Culverwell]

It should also be noted that the given example FY-3D file has variables called xGnss etc instead of xGps etc, as expected (and laid down in the standard at ​https://cdaac-www.cosmic.ucar.edu/cdaac/cgi_bin/fileFormats.cgi?type=atmPhs. These need to be renamed (eg using ncrename) before ucar2ropp can read the file. Likewise the badness global attribute bad, which is set to

// global attributes:
...
		:bad = 0 ;

when it is assumed in the code to be

// global attributes:
...
		:bad = "0" ;

The change in type from character to integer is enough to crash ucar2ropp, and therefore also needs to be amended (eg with ncatted) before using the dataset.

(Note, however, that the latest CDAAC atmPrf file description (​https://cdaac-www.cosmic.ucar.edu/cdaac/cgi_bin/fileFormats.cgi?type=atmPrf) defines bad as an integer:

bad
    Description:Badness flag. 1 = Profile flunked quality control, 0 = Profile passed QC
    Data Type: int

But these are really atmPhs files, of course, and there's no prescription for bad on their description page, ​https://cdaac-www.cosmic.ucar.edu/cdaac/cgi_bin/fileFormats.cgi?type=atmPhs. So who knows what it's supposed to be? For the moment, ROPP will assume bad to be a character.)

[Ian Culverwell]

Nor is it a bad idea to (manually) change the global attribute

		:center = "NSMC                                                                            " ;

to

		:center = "CMA" ;

as this ensures that the correct processing centre appears in the BUFR files:

  Originating Centre:          38 (Beijing (RSMC))
  Originating Sub-Centre:       0 (Beijing (China))

rather than

  Originating Centre:           0 (WMO Secretariat)
  Originating Sub-Centre:       0 (none)

[Ian Culverwell]

r5578 contains some improvement to the output from the -d option to bufr2ropp:

INFO (from bufr2ropp):  Decoded profile    1 : OC_20180609075349_FY3D_G030_CMA_
... (from bufr2ropp):                        : 07:53UT 09-Jun-2018 (-70.8,-129.5)

becomes

INFO (from bufr2ropp):  Decoded profile    1 : OC_20180609075349_FY3D_G030_CMA_
... (from bufr2ropp):  Occ. time & (lat,lon) : 07:53UT 09-Jun-2018 (-70.8,-129.5)

[Ian Culverwell]

For the record, and for future testing purposes, the attached FY3D_GNOSX_GBAL_L1_20180609_0024_AEG20_MS.NC is a 'bad' profile. It fails to produce any bending angles because

INFO (from ropp_pp_preprocess):  GNOS data preprocessing
INFO (from ropp_pp_preprocess_GNOS):  Minimum valid L2 SLTA =  105.245   km.
 
WARNING (from ropp_pp_cutoff):  Error: data unusable [L2 amplitude]

At least it has recognised that it is GNOS data.

[Ian Culverwell]

FY3D_GNOSX_GBAL_L1_20180609_0024_AEG20_MS.NC

[Ian Culverwell]

Note that a small test of GNOS data has been included in the automatic 'make tests', with the results:

*** Results log of t_pp_occ_gnos_1 (PP occ; GNOS data) ***
./../tools/ropp_pp_occ_tool  -ellipsoid  ../data/ropp_pp_test_gnos.nc  -o ropp_pp_test_gnos.nc  -c ../config/cosmic_pp.cf  -m MSIS  -d

----------------------------------------------------------------------
               ROPP Occultation Pre-processor Tool
----------------------------------------------------------------------

INFO:  Reading configuration file ../config/cosmic_pp.cf.

INFO:  Processing profile    1 of      1
INFO:  Reading input data file ../data/ropp_pp_test_gnos.nc.

 
WARNING (from ropp_io_read_ncdf_get):  'start_time' and yr/mo/dy/hr/mn/sc/ms timestamps differ by 5.000 seconds - using yr/../ms timestamp
INFO:  (OC_20170128033252_FY3C_G015_NSMC) 

...:  Occultation date (dd/mm/yyyy): =  28/01/2017
...:  Occultation time (hh:mm:ss): =  03:32:52
...:  Occultation point (lat,lon): =  (  25.07N, -99.15E)
...:  Undulation (m) =     -0.19558E+02
INFO (from ropp_pp_preprocess):  GNOS data preprocessing
INFO (from ropp_pp_preprocess_GNOS):  Minimum valid L2 SLTA =   27.917   km.
... (from ropp_pp_radiooptic_analysis):    L1 aperture:  15025.470  FFT points:  1024
... (from ropp_pp_radiooptic_analysis):    L1 aperture:    765.769  FFT points:    64
... (from ropp_pp_radiooptic_analysis):    L2 aperture:    867.496  FFT points:    64
... (from ropp_pp_correct_L2):    L2 badness between  15.0 and  50.0 km:   831.493
... (from ropp_pp_correct_L2):    Detected click in L2 phase at index  3854, time =   77.060
... (from ropp_pp_correct_L2):    Detected click in L2 phase at index  3853, time =   77.040
... (from ropp_pp_correct_L2):    Detected click in L2 phase at index  3854, time =   77.060
INFO:  Retrieving bending angle profiles by GEOMETRIC OPTICS 

...:  Smoothed bending angle.
...:  Pmin =   2000.000 Pmax = 120242.348
...:  ws_go_smooth      143
...:  Full resolution bending angle.
...:  Pmin =   2134.257 Pmax = 120255.545
...:  ws_go_full      143
INFO:        5600 data points in output between 6365.1km and 6483.2km 

INFO:  Retrieving bending angle profiles by WAVE OPTICS 

...:  WM =   143
...:  W =    95
...:  WL =   -47
... (from ropp_pp_DCT):  Channel 1. Hi-res grid size =  524288
INFO (from ropp_pp_DCT):  Complex field filtering 

... (from ropp_pp_DCT):  DPW =    250.000  CFW =    13.8
... (from ropp_pp_DCT):  DP0 =   6698.621
... (from ropp_pp_DCT):  Pmin =   6698.621 Pmax=  25000.000
... (from ropp_pp_DCT):  Pmin =   6702.202 Pmax=  25000.000
... (from ropp_pp_DCT):  Channel 2. Hi-res grid size =  262144
INFO (from ropp_pp_DCT):  Complex field filtering 

... (from ropp_pp_DCT):  DPW =    250.000  CFW =    21.5
... (from ropp_pp_DCT):  DP0 =   8677.870
... (from ropp_pp_DCT):  Pmin =   8677.870 Pmax=  25000.000
... (from ropp_pp_DCT):  Pmin =   8677.644 Pmax=  25000.000
... (from ropp_pp_DCT):    ICW =  1  Pmin =   6702.202
INFO:        5600 data points in output between 6365.1km and 6483.2km 

INFO (from ropp_pp_bending_angle_gnos):  Extrapolating GNOS L2 data from L2 - L1
...:  Pmin =   6702.202 Pmax = 150000.000
...:  Standard grid size nbi =   1434
INFO:  Correcting bending angle profile by STATISTICAL OPTIMISATION 

... (from ropp_pp_fit_model_refraction):    2-parameter model:ob fit: From     20002. to     20002. No. data =      1
... (from ropp_pp_fit_model_refraction):    RF        =  0.0000,  1.0000
INFO:        1087 data points in output between 6369.7km and 6478.3km 

INFO:  Retrieving refractivity profile by LIN ABEL TRANSFORM with STAT OPT 

INFO:  Writing output altitude scales with respect to WGS84 ELLIPSOID.
INFO:  Computing dry temperature 

INFO:  Writing to output file ropp_pp_test_gnos.nc

*** Running IDL to plot results
IDL Version 8.2 (linux x86_64 m64). (c) 2012, Exelis Visual Information Solutions, Inc.
Installation number: 404878-1.
Licensed for use by: Met Office

Automatic journal file: 
$HOME/idl_journal_files/idlidculvWedSep2610:40:592018.jou
% Compiled module: IT_PP_01.
Reading original file ../data/ropp_pp_test_gnos.nc
% Loaded DLM: NCDF.
% Compiled module: NCDF_GETVAR.
Reading output file ropp_pp_test_gnos.nc
% Compiled module: INTERPOL.
Impact parameters differ by        0.0000000% - TEST PASSED
Corrected bending angles differ by        0.0000000rad - TEST PASSED
Geometric heights differ by  0.00000% - TEST PASSED
Refractivity differ by  0.00000% - TEST PASSED
% Compiled module: OPEN_JPGFILE.
% Compiled module: CLOSE_JPGFILE.
% Loaded DLM: JPEG.
All tests successful.....TEST PASSED!
*** Finished IDL
*** ropp_pp_test_gnos.jpg can be compared against ../data/ropp_pp_test_gnos_reference.jpg

and

************************** SUMMARY OF ROPP_PP TEST RESULTS ***************************
--------------------------------------------------------------------------------------
|                   Test name    |              Description       |    Run? |  PASS? |
--------------------------------------------------------------------------------------
|                  t_pp_invert_1 |     PP invert; default options |     Run |  PASS  |
|                     t_pp_occ_1 |        PP occ; default options |     Run |  PASS  |
|                t_pp_occ_gnos_1 |              PP occ; GNOS data |     Run |  PASS  |
|                      t_pp_rs_1 | PP raw sampling; default optio |     Run |  PASS  |
|                    t_pp_abel_1 |          PP Abel/Inv; def opts |     Run |  PASS  |
|                t_pp_spectra_1a |    PP spectra; def opt (L1 dt) |     Run |  PASS  |
|                t_pp_spectra_1b |    PP spectra; def opt (L2 dt) |     Run |  PASS  |
|                t_pp_spectra_1c |    PP spectra; def opt (L1 ep) |     Run |  PASS  |
|                t_pp_spectra_1d |    PP spectra; def opt (L2 ep) |     Run |  PASS  |
|                    t_pp_wopt_1 |         PP WOPT; quick options |     Run |  PASS  |
--------------------------------------------------------------------------------------

For ROPP10.0, we should include a test of ~ 1 day of FY-3C (and perhaps FY-3D) data in the test folder.

[Ian Culverwell]

Passes test folder, so close ticket.

[Ian Culverwell]

I thought it was worth double-checking that the new BUFR libraries (bufr-24.0.2a and bufr_000387a) are OK for inclusion with ROPP9.1. They build and run OK on the standard files, but these aren't new, of course. When we run bufr2ropp on the new BUFR files attached to this ticket, we find that the resulting netCDF ROPP files are the same. The only difference is that decbufr fails to find the right names for some of the new codes, so that, for example, on the GRACE-D FO file GD-AI-3-NRT+2011_052_1100_G23_zdif_006.bin we find:

Old MetDB BUFR lib bufr-24.0.2

 SATELLITE IDENTIFIER                                  Code 001007          804
 SATELLITE INSTRUMENTS                                 Code 002019          104
 IDENTIFICATION OF ORIGINATING/GENERATING CENTRE       Code 001033           78

and

New MetDB BUFR lib bufr-24.0.2a

 SATELLITE IDENTIFIER                                  Code 001007 GRACE D (GRA
 SATELLITE INSTRUMENTS                                 Code 002019 Triple-G (GP
 IDENTIFICATION OF ORIGINATING/GENERATING CENTRE       Code 001033           78

So the new tables are doing the right thing.

(I note that support for the IDENTIFICATION OF ORIGINATING/GENERATING CENTRE seems to have disappeared. Earlier (> 5 years old) versions of decbufr translated the code into a name, so that, in the example above, it would have said

 IDENTIFICATION OF ORIGINATING/GENERATING CENTRE       Code 001033 EDZW OFFENBACH

instead of

 IDENTIFICATION OF ORIGINATING/GENERATING CENTRE       Code 001033           78

(See #comment:4 for an example.) I don't think this is associated with my mucking about, since the unadulterated BUFR library does the same thing on an ancient Metop-A file. But it's curious and annoying that this functionality has disappeared.)

[Ian Culverwell]

Check MetDB BUFR with test1b.sh (attached).

[Ian Culverwell]

Checked that the same is true for the two (now obsolete) ECMWF BUFR libraries, bufr_000387 and bufr_000387a. Note that decbufr is a product of the MetDB BUFR library, so we have to use the changed tables from that library when we test this with test2b.sh (attached).

[Ian Culverwell]

test1b.sh

[Ian Culverwell]

test2b.sh