Create controlled ATL03 HDF5 test file

[asteiker]

To better control these performance tests, we ought to create stripped down ATL03 files based on information from #23 and #26. The goal is to better control for the complexities in the native data. We can then reduce the noise as we investigate switches in the HDF5 library to increase performance in the cloud.

Tasks

Preview Give feedback

1. Determine key variables, including quality flags, needed for higher level products.

[asteiker]

@andypbarrett to provide key variables, including quality flags, needed for higher level products. Utilize ATBD and earthdata-varinfo to inform what information is needed for a controlled ATL03 HDF5 test file.

[andypbarrett]

I've looked at ATL07 and ATL06. Below are a list of candidate variables. I've left out geophysical corrections that are already included in h_ph. Paths are from ATL03 data dictionary. Variable lists are from table 2 in ATL07 ATBD and table 5-1 in ATL06 ATBD.

11 out of the 13 variables in /gtx/heights are used by one or both of ATL07 and ATL06, and 12 out of 37 variables from /gtx/geolocation are used.

ATL03 Variable	Description	Data type	Data Rate	Source
/gtx/heights/delta_time	Elapsed GPS seconds	Double(:)	Photon Detection Rate	6 + 7
/gtx/heights/dist_ph_across	Distance off RGT	Float(:)	Photon Detection Rate	6 + 7
/gtx/heights/dist_ph_along	Distance from equator crossing	Float(:)	Photon Detection Rate	6 + 7
/gtx/heights/h_ph	Photon WGS84 height	Float(:)	Photon Detection Rate	6 + 7
/gtx/heights/lat_ph	Latitude	Double(:)	Photon Detection Rate	6 + 7
/gtx/heights/lon_ph	Longitude	Double(:)	Photon Detection Rate	6 + 7
/gtx/heights/pce_mframe_cnt	PCE major frame counter	UINT(:)	Photon Detection Rate	6
/gtx/heights/ph_id_channel	Receive channel id	UINT(:)	Photon Detection Rate	6
/gtx/heights/ph_id_pulse	laser pulse counter id	UINT(:)	Photon Detection Rate	6
/gtx/heights/quality_ph	Photon quality	INT	Photon Detection Rate	7
/gtx/heights/signal_conf_ph	Photon signal confidence	INT	Photon Detection Rate	6
/gtx/geolocation/delta_time	Elapsed GPS seconds	DOUBLE(:)	Segment rate	6
/gtx/geolocation/ph_index_beg	Index of first segment photon	INT	Photon Detection Rate	6
/gtx/geolocation/podppd_flag	Geolocation quality	INT	Photon Detection Rate	6
/gtx/geolocation/segment_dist_x	Segment distance from equator	Double(:)	Segment rate	6
/gtx/geolocation/segment_id	along-track segment id	INT(:)	Segment rate	6 + 7
/gtx/geolocation/segment_length	along-track segment length	Double(:)	Segment rate	6 + 7
/gtx/geolocation/segment_ph_cnt	Number of photons	INT(:)	Segment rate	6
/gtx/geolocation/sigma_across	across-track geolocation accuracy	Float(:)	Segment rate	6 + 7
/gtx/geolocation/sigma_along	along-track geolocation accuracy	Float(:)	Segment rate	6 + 7
/gtx/geolocation/sigma_h	height uncertainty	(Float(:)	Segment rate	6
/gtx/geolocation/surf_type	Surface type	INT(:, 5)	Segment rate	7
/gtx/geolocation/velocity_sc	spacecraft xyz velocity	Float(:,3)	Segment rate	6

[asteiker]

We determined that we will grab all variables within the /heights groups for each beam group. A new task was added to actually perform the subsetting using icepyx.

[asteiker]

This may still be helpful to determine core variables needed to create a derived product if we were to reformat to something like geoparquet.