CLONE: Catalogs - Analyze results from CALDP-2024.10.2-2+enchilada delivery

[stscijgbot-hstdp]

Issue HLA-1392 was created on JIRA by Steve Goldman:

This ticket is for testing the the catalogs for the current build, but for ACS data. This ticket is a clone of HLA-1379.

[stscijgbot-hstdp]

Comment by Steve Goldman on JIRA:

The newer version of the images and catalog have slightly different alignment solutions as the newer version had one additional nmatch in the a posteriori fitting. It seems as though the images also had a slightly different background value with a difference of (MDRIZSKY) ~0.003. This may have resulted in a slightly different threshold for acceptable catalog sources. This is a fairly crowded field where, in the newer catalog, many of the sources in the crowded region are lost. 

In the images that I've attached, the segmentation catalog is plotted in blue, the point source catalog in red, and the purply sources are both (overlayed). As you can also see in the screenshot of the image, the lower left-had corner (galaxy center?) is crowded, and that is where many sources were lost in the newer version of the catalog. 

Blinking the images themselves doesn't show any substantial differences in the alignment. The general background, however, does appear slightly different. The difference images looks pretty uniform. 

[stscijgbot-hstdp]

Comment by Steve Goldman on JIRA:

In looking at the trailer files, I found that the psf's calculated for individual images has changed significantly. This is probably related to the photutils changes. 

For example, for image dataset {}jb1f0belq{}, the psf calculated for the new and old code were 3.8321 and 2.9454 (FWHM), respectively. 

[stscijgbot-hstdp]

Comment by Steve Goldman on JIRA:

The old code also includes plenty of errors related to the photutils changes.

New code

2024321075206 INFO src=drizzlepac.haputils.align_utils- Looking for sample PSF in jb1f0belq
2024321075208 WARNING src=astroquery- One or more fit(s) may not have converged. Please check the "flags" column in the output table.
2024321075208 WARNING src=astroquery- One or more fit(s) may not have converged. Please check the "flags" column in the output table.
2024321075208 INFO src=drizzlepac.haputils.align_utils-   Found PSF with FWHM =    3.8321 ```
old code
```java
2024312025509 INFO src=drizzlepac.haputils.align_utils- Looking for sample PSF in jb1f0belq
2024312025510 WARNING src=astroquery- AstropyDeprecationWarning: The DAOGroup class is deprecated and may be removed in a future version.
        Use `photutils.psf.SourceGrouper` instead.
2024312025510 WARNING src=astroquery- AstropyDeprecationWarning: The IterativelySubtractedPSFPhotometry class is deprecated and may be removed in a future version.
        Use `photutils.psf.IterativePSFPhotometry` instead.
2024312025510 WARNING src=astroquery- AstropyDeprecationWarning: The BasicPSFPhotometry class is deprecated and may be removed in a future version.
        Use `photutils.psf.PSFPhotometry` instead.
2024312025510 WARNING src=astroquery- AstropyDeprecationWarning: The _extract_psf_fitting_names function is deprecated and may be removed in a future version.
2024312025510 WARNING src=astroquery- AstropyDeprecationWarning: The get_grouped_psf_model function is deprecated and may be removed in a future version.
2024312025510 WARNING src=astroquery- The fit may be unsuccessful; check fit_info['message'] for more information.
2024312025510 WARNING src=astroquery- AstropyDeprecationWarning: The subtract_psf function is deprecated and may be removed in a future version.
2024312025510 WARNING src=astroquery- AstropyDeprecationWarning: The _extract_psf_fitting_names function is deprecated and may be removed in a future version.
2024312025510 WARNING src=astroquery- AstropyDeprecationWarning: The get_grouped_psf_model function is deprecated and may be removed in a future version.
2024312025510 WARNING src=astroquery- The fit may be unsuccessful; check fit_info['message'] for more information.
2024312025510 WARNING src=astroquery- AstropyDeprecationWarning: The subtract_psf function is deprecated and may be removed in a future version.
2024312025510 WARNING src=astroquery- AstropyDeprecationWarning: The _extract_psf_fitting_names function is deprecated and may be removed in a future version.
2024312025510 INFO src=drizzlepac.haputils.align_utils-   Found PSF with FWHM =    2.9454 ```

[stscijgbot-hstdp]

Comment by Steve Goldman on JIRA:

CORRECTION: I was mistakenly looking at values that were related to alignment and not the catalog creation. More relevant differences may be those in the difference header image attached:

!Screenshot 2024-12-09 at 2.55.57 PM.png!

[stscijgbot-hstdp]

Comment by Steve Goldman on JIRA:

I am seeing an odd shifting between the images. It may be the the new alignment solution is slightly worse. In the newer version of this image dataset, the RMS_RA is better, but the RMS_DEC is worse. My guess is that one or multiple images of this dataset are being smeared a little and that is resulting in more sources being cut. The fewer sources in our newer catalog tend to have a lower concentration index than the older catalog. This seems to me to point to a general decrease in the quality of the catalog sources.

[stscijgbot-hstdp]

Comment by Rick White on JIRA:

Steve Goldman
It looks to me like this visit may be affected by some combination of misaligned filters and misaligned exposures within filters. I don't have a real understanding of what is happening, but I'll show you what I have so far.

Here is a plot that compares the magnitudes (top row) and CI values (bottom row) for the 3 filters in the visit:
!plot1.png|width=100%!

In each panel, the x-axis shows the quantity in the new regression test catalog (from the prearchive_science-photutils test), and the y-axis shows the same quantity in the current ops cache. I matched the sources in the catalog, so we are looking at the same object.

The magnitudes for the f435w and f814w filters look like they agree very well with ops (top left, top right). But the magnitudes for the f555w filter (top center) are systematically fainter in the regression test. That probably indicates that the f555w image is not well aligned with the f814w/f435w filters.

But weirdly, I think only the f555w filter should have been used in the detection image, because that's the only filter that has multiple exposures allowing for CR rejection. And when I compare the total image to the f555w filter, I can see that they are in fact identical. So I really don't understand that.

Then for the CI values (bottom row), the f435w filter looks consistent with the ops version (bottom left), the f555w version has somewhat larger CI values than ops (bottom center), and the f814w filter has much larger CI values.

It is hard to make sense of all of this, which is why I suspect there is a combination of errors in the alignment of exposures (which can only affect the f555w image since it is the only one with multiple exposures) plus mismatches between the f435w/f814w filters and the f555w filter.

[stscijgbot-hstdp]

Comment by Rick White on JIRA:

Steve Goldman At Michele De La Pena 's suggestion, I have looked at the distribution of CI differences for all the datasets in the prearchive_science-photutils regression test sample.

I matched sources in the regression test catalog to sources for the same catalog in the HST cache and calculated the median difference CI(reg test) - CI(cache) for each catalog. The top row of the attached plot shows the CI differences for the point catalogs for acs/wfc, wfc3/uvis, and wfc3/ir. The bottom row shows the CI differences for the segment catalogs.

!reg_CI_prearchive_science-photutils.png|width=100%!

For acs/wfc and wfc3/uvis, there is no sign of a significant change in the CI values. The big spikes in the histogram are at zero, and there is very little scatter. There are a few cases with larger offsets (including dataset hst_11570_0b_acs_wfc_jb1f0b from this ticket), but they are rare.

FYI, the median CI differences for hst_11570_0b_acs_wfc_jb1f0b are:
0.287, 0.095, 0.333 for f435w, f555w, f814w (point catalogs)
0.218, 0.090, 0.269 (segment catalogs)
So that visit by itself accounts for many of the largest offsets for acs/wfc.

For wfc3/ir, there does seem to be something wrong. There are a lot of wfc3/ir point catalogs with relatively large offsets in the CI values (top right). It is interesting that the segment catalogs for wfc3/ir do not show the same effect (bottom right). That makes me think this is not an issue with image/exposure alignment (which would have to affect both catalogs). It seems more likely to be a problem with the source detection for the point catalog for wfc3/ir.

[stscijgbot-hstdp]

Comment by Michele De La Pena on JIRA:

Rick White Thank you yet again for your contribution to our work.  WOW - the WFC3/IR differences certainly stand out (which if pretty much an understatement)!  Is there an easy way for you to identify a dataset or two in the "bad regime" which, hopefully, will make tracking down this issue easier?

[stscijgbot-hstdp]

Comment by Rick White on JIRA:

Michele De La Pena Here is a list of some datasets. They all have shifts of about 0.07 in CI, which puts them in the middle of the bump in the CI difference distribution:

hst_11570_23_wfc3_ir_total_ib1f23
hst_12440_91_wfc3_ir_total_iboa91
hst_12443_1j_wfc3_ir_total_iboe1j
hst_12873_12_wfc3_ir_total_ic1o12
hst_13386_t8_wfc3_ir_total_ica9t8
hst_13614_01_wfc3_ir_total_ichq01
hst_14038_a3_wfc3_ir_total_icsza3
hst_14662_65_wfc3_ir_total_id8065
hst_15096_60_wfc3_ir_total_idho60

I picked out one dataset, hst_12440_91_wfc3_ir_iboa91, for closer examination. An attached image shows a zoomed-in region with the point catalog overplotted. It blinks between the current HST cache version and the regression test version (from the prearchive_science-photutils regression test). The image pixels values are extremely similar, but I can see that the source positions in the point catalog are moving a little. Some of them seem to be moving off the apparent position of the star in the image. The blue arrow marks one with a relatively large shift.

This kind of shift would cause the CI values to get larger.

!hst_12440_91_wfc3_ir_iboa91.gif|width=75%!

[stscijgbot-hstdp]

Comment by Steve Goldman on JIRA:

I think a part of the issue is that the code is using a different sized PSF in the newer version of the build. In the mostly-cos build (for the ACS dataset {}jb1f0b{}) the PSF used for the point source catalog has a FWHM size of 2.07802016375355, in the enchilada build it uses a value of 1.52

 A difference of a clipped version of trailer files is attached. 

!Screenshot 2024-12-11 at 2.56.54 PM.png!

[stscijgbot-hstdp]

Comment by Rick White on JIRA:

Steve Goldman Is the PSF FWHM different for wfc3/ir and the other detectors? It seems like things are working pretty well for wfc3/uvis and acs/wfc. Maybe if we had a table of the PSF FWHM sizes for those 3 detectors (in the cache version and in the regtest version), that would be helpful in understanding the problem.

[stscijgbot-hstdp]

Comment by Steve Goldman on JIRA:

To be honest, I'm not as familiar with this part of the code. I was just trying to read up on what is used for the psfs for catalog creation. Michele may be able to say more about this. It seems like at least for this example, a psf (drz) file is created for the enchilada build, but not the mostly-cos build. I'll try to see what I can find. 

[stscijgbot-hstdp]

Comment by Steve Goldman on JIRA:

It seems that the psfs are only important in identifying sources, and then an aperture is used for actually determining brightness. The aperture values are listed here. I'm tracking down the psf values. 

[stscijgbot-hstdp]

Comment by Steve Goldman on JIRA:

The determination of the psfs seems based on the tinytim psfs. I think the relevant line of code though is Line 1143 of catalog_utils.py. In the enchilada build, the code is not finding any sources that match the theoretical psf model, so it is reverting to a value calculated in align_utils.py which is based on an input value of the fwhm and the pixel scale. So, on second thought, if we are seeing some smearing, this difference in the psfs is likely a consequence of an underlying issue and not the reason for it. 

I'll keep looking. 

[stscijgbot-hstdp]

Comment by Michele De La Pena on JIRA:

As noted, the problems are arising from the updates needed to use Photutils 1.13.  The critical updates were made in the astrometric_utils.py module.  It turns out the changes affect NOT ONLY the alignment, BUT ALSO the source identification for the Point catalog only. 

Larry has also given me some good tips which should be incorporated into the Photutils updates.  Because we should not make last minute changes which we do not have enough time to evaluate properly, I will send an email suggesting we pull the Photutils updates for this build.  We can incorporate the changes in a Quick Fix in January.

Many Thanks to Rick and Steve for their analyses.

[stscijgbot-hstdp]

Comment by Rick White on JIRA:

Michele De La Pena Steve Goldman
Here is one more plot before I pause work on this. For each catalog, I calculate the median offset between sources that match between the regression test and the cache versions of the catalogs. That allows for the astrometry to change a little between versions, while still matching up the correct source pairs.

This plot shows how the CI difference (regression test CI minus cache CI) depends on the distance of the source from the expected position in the reg-test catalog. This is for a single image, and each dot in the image is for one matched source. Sources at {x=0) are close to the expected position, and sources with larger {x} values are offset from the expected position. There are plots for each of the 2 filters in this visit.

The larger CI differences come from sources that are farther away from their expected locations (according to the cache catalog). I think this supports the suggestion that the positions of sources are not accurate for the point catalog, which leads to the CI values being too large.

!hst_12440_91_wfc3_ir_total_iboa91_point-CI.png|width=80%!

This seems to be happening systematically for the wfc3/ir point catalogs. Maybe the issues Steve is seeing in the log files where the PSF size is not what is expected are most common for the wfc3/ir images. So I'm guessing that will be a good place to focus when trying to fix this.

[stscijgbot-hstdp]

Comment by Jennifer Mack on JIRA:

That's curious indeed! I see almost no change between the old and hew HAP drizzled products for iboa91.

The attachment hst_12440drz.png  compares F125W, F160W, Total drizzled images (hst_12440_91_wfc3_ir__iboa91_drz.fits) in the NEW products (top row) and OLD products (bottom row) from the following directories:

█████████████████████████████████████████████████████████████████████████████████████████████████
█████████████████████████████████████████████████████████████████████████████████████████████████████████

Both have FIT-SVM-GSC242 solutions.  The NEW one has fewer matches (66) versus the old one (77) but it has lower fit rms values (RMS_RA,RMS_Dec of ~60 mas versus ~70 mas in the OLD).   Nothing jumped out as unusual about this dataset, so it's interesting to see those concentration index results.   I'll look at more examples in the list from Rick.

 

 

[stscijgbot-hstdp]

Comment by Jennifer Mack on JIRA:

Quick note:   A good diagnostic dataset for WFC3/IR is id8065 with N=400 matches and Gaia eDR3 WCS solutions. 

There are noticeable offsets in the WCS between the numpy and the photutils builds ONLY,  and the RMS_RA and RMS_DEC keywords are a factor of 2 higher in both standard pipeline and in HAP drizzled images.  (That seems consistent with Steve's note about PSF fwhm values being larger. )

HAP:  hst_14662_65_wfc3_ir_total_id8065_drz.fits

F160W Product combines 2 FLT files (SPARs 100) and 1 FLT file (SPARS 11)
HSTDP-2024.3.1-446ebba2e-20241031-prearchive_science-pmap1192/hap_files - Nmatches = 415,  RMS_RA=9, RMS_Dec=10
HSTDP-2024.4.0-13348c188d-20241115-prearchive_science-photutils/hap_files  - Nmatches = 393, RMS_RA=22, RMS_Dec=22

HST:  id8065020_drz.fits

F160W Product combines 2 FLT files (SPARS100)
HSTDP-2024.3.1-446ebba2e-20241031-prearchive_science-pmap1192/science_fits - Nmatches = 372, RMS_RA=7, RMS_Dec=10
HSTDP-2024.4.0-13348c188d-20241115-prearchive_science-photutils/science_fits  - Nmatches = 370, RMS_RA=25, RMS_Dec=25

[stscijgbot-hstdp]

Comment by Jennifer Mack on JIRA:

I did some more testing last night and looked more carefully at some drizzled images with Varun this afternoon. Here is a summary:
 
With the new photutils, the HAP drz PSFs for WFC3/IR look more smeared out than the drzs for standard products. This appears to be due to poorer alignment quality across different associations in the same visit.    When detecting sources in the image for alignment, it may be that the software is convolving the image with a different size gaussian kernel (based on comments from Steve about the larger PSF FWHM).  This could result in larger errors in the position of objects which in turn increases the fit RMS values reported in the header with the new photutils.
 
The fit RMS values are a factor of 2 higher in the drz dataset below between numpy and photutils versions. 
████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████
█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████
 
The increase RMS is apparent for both HST and HAP, but the impact on the combined drz images is more obvious for HAP.  For standard products, point sources in the drz images look nearly identical to the pre-photutils build, and the larger RMS values do not necessarily mean the alignment solution is degraded... just that the error in the fit was larger. 
 
Thus, I don't see any reason to delay the pending software build for standard HST data products.