Add ability to read Sentinel-6 / JASON-CS data

[Ian Culverwell]

Axel von Engeln (EUM) writes:

Hi Kent, Ian, Sean, Stig, 


a bit later than envisioned, but here are some example Sentinel-6 L1B 
netCDF files and also one example of a tar package. 

We generated the L1Bs using COSMIC-2 input data, but otherwise 
pretending we do the S6A mission. The data is from 11th Jan 2020, file 
name; variables and attributes should be as in the final product. The 
data processing is fine, though not perfect, so we are not primarily 
looking for a data validation here, but a format one. Some format 
changes have happened in the mean time (and you'll see similar format 
changes for the reprocessing L1Bs we are just setting up). E.g. the 
navbit information is now only provided once, not internal and 
external, there is no closed loop data available for S6, etc. On 
navbits, a flag let's you know which one we used. Hopefully this is 
all rather self-explanatory. I haven't updated the Product Format 
Specification yet, but will do this once we have a final format 
version. 

I ran the data through eum2ropp and eum2bufr of ROPP v10, and that 
flagged no issues (S6A is though currently unknown). This is however 
only on the L1B level, the L1A will need some ROPP updates. Eg the 
eum2ropp -l .. command will not work as implemented now, since there 
is only open loop data available, and no closed loop flags are thus 
set. I suggest you evaluate what is missing / needs to be updated for 
ROPP, or is missing on our side, and we discuss this once you have a 
complete list. The data should though allow to assess whether the 
non-equidistant time stamps are an issue for ROPP L1A to L1B 
processing. 

I also added a tar file, this is how we provide data to the EUM 
archive, as well as to the internal "Data Lake", all data is 
"wrapped" in a tar package, along with an xml manifest. I assume 
you'll get the data in this format or in the corresponding Windows 
zip format. We so far have only a tar file with the latest, real data, 
so you can get acquainted with this tar wrapper. I also did add a zip 
file, but that has a L1B product from earlier testing, so this is just 
a format example of zip files. Do you have any info on what data 
wrapping you'll get, zip or tar or none? And are you interested in 
the xml content? If yes, I could provide a document that describes 
it. But we only wrap / populate the xml for the archive, so all 
relevant info is of course in the netCDF.


The data download link is available to anybody, so you can forward 
this to others. 


Kind regards,


            Axel

[Ian Culverwell]

I forgot Axel's PS:

PS: So far all good on S6A, RO instrument switch on is on Friday...

[Ian Culverwell]

Axel notifies us that there's a mistake in the netCDF files:

Hi all, 

Christian noted one inconsistency in the S6 files we provided. 
We wrote in there a /data/level_1a/measurements group, but that 
will change to /data/level_1a/open_loop to be consistent with 
other data. Sorry for this. We’ll be updating and providing new 
files soon (today or tomorrow). 

I won’t update the tar  /  zip files though, these are just 
packaging examples. 

There is one more change that will come sometime in the future, 
where the reference time for times given in the occultation 
will change from 2000-01-01, to 2020-01-01. But that will be a 
bit later. It would though be good if you can deal with this 
reference time update easily. EPS-SG will also refer to 2020. 

On real S6 data, we have processed the first S6 occultations, 
looking good so far, but still some issues to follow up. 

Cheers,

           Axel

And

Hi all,

as discussed, updated COSMIC-2 netCDF4 L1B files, processed as
they were Sentinel-6. The following changes are done from 
previous version:

- renamed /data/level_1a/measurements to /data/level_1a/open_loop 

- Yaros version string includes a string, for these provided 
examples it is processor_version = "2.0rc6" . Note that this 
will not work with the current ROPP, Leonid did though already 
provided an update (so maybe get in touch with him).

I did not include an updated zip file, as this was only provided 
as an example last time. But a new tar file is also included. 

Let us know if there are any issues.

Cheers,

           Axel

So I've deleted the old netCDF files and uploaded the new ones to this ticket.

[Ian Culverwell]

ropp_yaros_version.patch

[Ian Culverwell]

Leonid Butenko at EUMETSAT has provided the attached patch ropp_yaros_version.patch to fix the processor_version formatting problem described in Axel's last message.

[Ian Culverwell]

To be included in ROPP-11.1 - provided the ropp_pp components of the ROPP-9.1 follow-on changes are also included, since they are needed to process Sentinel-6 data.