Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

HLA-1110: aperture keyword from poller to header #1683

Conversation

s-goldman
Copy link
Collaborator

@s-goldman s-goldman commented Oct 25, 2023

Resolves HLA-1110

Closes #1661

This PR adds the aperture keyword to HAPProduct object as well as its child products (TotalProduct, FilterProduct, ExposureProduct). The code changes how the poller file data is used.

Previously the code either 1) used the poller file filenames and pulled the other relevant data from the header keywords 2) used full poller files with all of the relevant data. We have now implemented a third option where if the poller file has two columns, it is assumed that the second column is the aperture keyword. This code will then update the aperture keyword in the output SVM data for only WFPC2.

Checklist for maintainers

  • added entry in CHANGELOG.rst within the relevant release section
  • updated or added relevant tests
  • updated relevant documentation
  • added relevant milestone
  • added relevant label(s)
  • ran regression tests, post a link to the Jenkins job below.
    How to run regression tests on a PR

@s-goldman s-goldman added enhancement Do Not Merge PR which should not be merged labels Oct 25, 2023
@s-goldman s-goldman requested review from mdlpstsci and a team as code owners October 25, 2023 20:58
@s-goldman
Copy link
Collaborator Author

Still need to test inputs as a list, as opposed to polller file, and testing the MVMs.

@codecov
Copy link

codecov bot commented Oct 25, 2023

Codecov Report

Attention: 21 lines in your changes are missing coverage. Please review.

Files Coverage Δ
drizzlepac/hapsequencer.py 58.13% <100.00%> (ø)
drizzlepac/haputils/product.py 65.26% <100.00%> (+0.05%) ⬆️
drizzlepac/haputils/processing_utils.py 74.80% <50.00%> (-0.41%) ⬇️
drizzlepac/haputils/poller_utils.py 51.58% <48.64%> (-0.49%) ⬇️

... and 36 files with indirect coverage changes

📢 Thoughts on this report? Let us know!.

@s-goldman
Copy link
Collaborator Author

Since your waiting on me @mdlpstsci , I'll include you in my current problem-solving. My issue right now is a couple SVM unit tests.

It looks like gather_output_data is not working as intended, and not finding the test data that should be there.

Any thoughts?

(stenv-python-3.10) sgoldman@wovas2023 drizzlepac_dev % pytest tests/hap/test_svm_ibqk07.py 
=================================== test session starts ===================================
platform darwin -- Python 3.10.11, pytest-7.3.1, pluggy-1.0.0
rootdir: /Users/sgoldman/Documents/drizzlepac_dev
configfile: pyproject.toml
plugins: asdf-2.15.0, typeguard-3.0.2, jwst-1.10.2, xdist-3.3.1, ci-watson-0.6.1, anyio-3.7.0
collected 9 items                                                                         

tests/hap/test_svm_ibqk07.py .FFFF....                                                                                                 [100%]

================================================================== FAILURES ==================================================================
______________________________________________________________ test_svm_wcs_ir _______________________________________________________________

gather_output_data = []

    def test_svm_wcs_ir(gather_output_data):
        print("\ntest_svm_wcs_ir.")
        # Get the TDP for this detector
        tdp_files = [files for files in gather_output_data if files.lower().find("ir_total") > -1 and files.lower().endswith("drz.fits")]
    
        # Check the WCS solution is as expected
>       wcsname = fits.getval(tdp_files[0], "WCSNAME", ext=1).upper()
E       IndexError: list index out of range

/Users/sgoldman/Documents/drizzlepac_dev/tests/hap/test_svm_ibqk07.py:184: IndexError
----------------------------------------------------------- Captured stdout setup ------------------------------------------------------------

gather_output_data. Output data files: []
------------------------------------------------------------ Captured stdout call ------------------------------------------------------------

test_svm_wcs_ir.
____________________________________________________________ test_svm_wcs_ir_all _____________________________________________________________

gather_output_data = []

    def test_svm_wcs_ir_all(gather_output_data):
        print("\ntest_svm_wcs_ir_all.")
        # Check the output primary WCSNAME
        ir_files = [files for files in gather_output_data if files.lower().find("_ir_") > -1 and files.lower().endswith("drz.fits")]
    
        wcsnames = [fits.getval(ir, "WCSNAME", ext=1).upper() for ir in ir_files]
>       assert len(set(wcsnames)) == 1, f"WCSNAMES are not all the same for the IR detector: {wcsnames}"
E       AssertionError: WCSNAMES are not all the same for the IR detector: []
E       assert 0 == 1
E        +  where 0 = len(set())
E        +    where set() = set([])

/Users/sgoldman/Documents/drizzlepac_dev/tests/hap/test_svm_ibqk07.py:195: AssertionError
------------------------------------------------------------ Captured stdout call ------------------------------------------------------------

test_svm_wcs_ir_all.
_____________________________________________________________ test_svm_wcs_uvis ______________________________________________________________

gather_output_data = []

    def test_svm_wcs_uvis(gather_output_data):
        print("\ntest_svm_wcs_uvis.")
        # Get the TDP for this detector
        tdp_files = [files for files in gather_output_data if files.lower().find("uvis_total") > -1 and files.lower().endswith("drc.fits")]
    
        # Check the WCS solution is as expected
>       wcsname = fits.getval(tdp_files[0], "WCSNAME", ext=1).upper()
E       IndexError: list index out of range

/Users/sgoldman/Documents/drizzlepac_dev/tests/hap/test_svm_ibqk07.py:204: IndexError
------------------------------------------------------------ Captured stdout call ------------------------------------------------------------

test_svm_wcs_uvis.
___________________________________________________________ test_svm_wcs_uvis_all ____________________________________________________________

gather_output_data = []

    def test_svm_wcs_uvis_all(gather_output_data):
        # Check the output primary WCSNAME
        print("\ntest_svm_wcs_uvis_all.")
        uvis_files = [files for files in gather_output_data if files.lower().find("_uvis_") > -1 and files.lower().endswith("drc.fits")]
    
        wcsnames = [fits.getval(uvis, "WCSNAME", ext=1).upper() for uvis in uvis_files]
>       assert len(set(wcsnames)) == 1, f"WCSNAMES are not all the same for the UVIS detector: {wcsnames}"
E       AssertionError: WCSNAMES are not all the same for the UVIS detector: []
E       assert 0 == 1
E        +  where 0 = len(set())
E        +    where set() = set([])

/Users/sgoldman/Documents/drizzlepac_dev/tests/hap/test_svm_ibqk07.py:215: AssertionError
------------------------------------------------------------ Captured stdout call ------------------------------------------------------------

test_svm_wcs_uvis_all.
============================================================== warnings summary ==============================================================
../../../../opt/anaconda3/envs/stenv-python-3.10/lib/python3.10/site-packages/pkg_resources/__init__.py:121
  /opt/anaconda3/envs/stenv-python-3.10/lib/python3.10/site-packages/pkg_resources/__init__.py:121: DeprecationWarning: pkg_resources is deprecated as an API
    warnings.warn("pkg_resources is deprecated as an API", DeprecationWarning)

../../../../opt/anaconda3/envs/stenv-python-3.10/lib/python3.10/site-packages/pkg_resources/__init__.py:2870
  /opt/anaconda3/envs/stenv-python-3.10/lib/python3.10/site-packages/pkg_resources/__init__.py:2870: DeprecationWarning: Deprecated call to `pkg_resources.declare_namespace('mpl_toolkits')`.
  Implementing implicit namespace packages (as specified in PEP 420) is preferred to `pkg_resources.declare_namespace`. See https://setuptools.pypa.io/en/latest/references/keywords.html#keyword-namespace-packages
    declare_namespace(pkg)

../../../../opt/anaconda3/envs/stenv-python-3.10/lib/python3.10/site-packages/pkg_resources/__init__.py:2870
../../../../opt/anaconda3/envs/stenv-python-3.10/lib/python3.10/site-packages/pkg_resources/__init__.py:2870
../../../../opt/anaconda3/envs/stenv-python-3.10/lib/python3.10/site-packages/pkg_resources/__init__.py:2870
../../../../opt/anaconda3/envs/stenv-python-3.10/lib/python3.10/site-packages/pkg_resources/__init__.py:2870
  /opt/anaconda3/envs/stenv-python-3.10/lib/python3.10/site-packages/pkg_resources/__init__.py:2870: DeprecationWarning: Deprecated call to `pkg_resources.declare_namespace('sphinxcontrib')`.
  Implementing implicit namespace packages (as specified in PEP 420) is preferred to `pkg_resources.declare_namespace`. See https://setuptools.pypa.io/en/latest/references/keywords.html#keyword-namespace-packages
    declare_namespace(pkg)

../../../../opt/anaconda3/envs/stenv-python-3.10/lib/python3.10/site-packages/pkg_resources/__init__.py:2870
../../../../opt/anaconda3/envs/stenv-python-3.10/lib/python3.10/site-packages/pkg_resources/__init__.py:2870
  /opt/anaconda3/envs/stenv-python-3.10/lib/python3.10/site-packages/pkg_resources/__init__.py:2870: DeprecationWarning: Deprecated call to `pkg_resources.declare_namespace('stsci')`.
  Implementing implicit namespace packages (as specified in PEP 420) is preferred to `pkg_resources.declare_namespace`. See https://setuptools.pypa.io/en/latest/references/keywords.html#keyword-namespace-packages
    declare_namespace(pkg)

../../../../opt/anaconda3/envs/stenv-python-3.10/lib/python3.10/site-packages/stsci/tools/stpyfits.py:20
../../../../opt/anaconda3/envs/stenv-python-3.10/lib/python3.10/site-packages/stsci/tools/stpyfits.py:20
  /opt/anaconda3/envs/stenv-python-3.10/lib/python3.10/site-packages/stsci/tools/stpyfits.py:20: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
    ASTROPY_VER_GE32 = LooseVersion(astropy.__version__) >= LooseVersion('3.2')

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
========================================================== short test summary info ===========================================================
FAILED tests/hap/test_svm_ibqk07.py::test_svm_wcs_ir - IndexError: list index out of range
FAILED tests/hap/test_svm_ibqk07.py::test_svm_wcs_ir_all - AssertionError: WCSNAMES are not all the same for the IR detector: []
FAILED tests/hap/test_svm_ibqk07.py::test_svm_wcs_uvis - IndexError: list index out of range
FAILED tests/hap/test_svm_ibqk07.py::test_svm_wcs_uvis_all - AssertionError: WCSNAMES are not all the same for the UVIS detector: []
============================================ 4 failed, 5 passed, 10 warnings in 231.02s (0:03:51) ============================================
(stenv-python-3.10) sgoldman@wovas2023 drizzlepac_dev % 

@s-goldman
Copy link
Collaborator Author

Since your waiting on me @mdlpstsci , I'll include you in my current problem-solving. My issue right now is a couple SVM unit tests.

It looks like gather_output_data is not working as intended, and not finding the test data that should be there.

Any thoughts?

(stenv-python-3.10) sgoldman@wovas2023 drizzlepac_dev % pytest tests/hap/test_svm_ibqk07.py 
=================================== test session starts ===================================
platform darwin -- Python 3.10.11, pytest-7.3.1, pluggy-1.0.0
rootdir: /Users/sgoldman/Documents/drizzlepac_dev
configfile: pyproject.toml
plugins: asdf-2.15.0, typeguard-3.0.2, jwst-1.10.2, xdist-3.3.1, ci-watson-0.6.1, anyio-3.7.0
collected 9 items                                                                         

tests/hap/test_svm_ibqk07.py .FFFF....                                                                                                 [100%]

================================================================== FAILURES ==================================================================
______________________________________________________________ test_svm_wcs_ir _______________________________________________________________

gather_output_data = []

    def test_svm_wcs_ir(gather_output_data):
        print("\ntest_svm_wcs_ir.")
        # Get the TDP for this detector
        tdp_files = [files for files in gather_output_data if files.lower().find("ir_total") > -1 and files.lower().endswith("drz.fits")]
    
        # Check the WCS solution is as expected
>       wcsname = fits.getval(tdp_files[0], "WCSNAME", ext=1).upper()
E       IndexError: list index out of range

/Users/sgoldman/Documents/drizzlepac_dev/tests/hap/test_svm_ibqk07.py:184: IndexError
----------------------------------------------------------- Captured stdout setup ------------------------------------------------------------

gather_output_data. Output data files: []
------------------------------------------------------------ Captured stdout call ------------------------------------------------------------

test_svm_wcs_ir.
____________________________________________________________ test_svm_wcs_ir_all _____________________________________________________________

gather_output_data = []

    def test_svm_wcs_ir_all(gather_output_data):
        print("\ntest_svm_wcs_ir_all.")
        # Check the output primary WCSNAME
        ir_files = [files for files in gather_output_data if files.lower().find("_ir_") > -1 and files.lower().endswith("drz.fits")]
    
        wcsnames = [fits.getval(ir, "WCSNAME", ext=1).upper() for ir in ir_files]
>       assert len(set(wcsnames)) == 1, f"WCSNAMES are not all the same for the IR detector: {wcsnames}"
E       AssertionError: WCSNAMES are not all the same for the IR detector: []
E       assert 0 == 1
E        +  where 0 = len(set())
E        +    where set() = set([])

/Users/sgoldman/Documents/drizzlepac_dev/tests/hap/test_svm_ibqk07.py:195: AssertionError
------------------------------------------------------------ Captured stdout call ------------------------------------------------------------

test_svm_wcs_ir_all.
_____________________________________________________________ test_svm_wcs_uvis ______________________________________________________________

gather_output_data = []

    def test_svm_wcs_uvis(gather_output_data):
        print("\ntest_svm_wcs_uvis.")
        # Get the TDP for this detector
        tdp_files = [files for files in gather_output_data if files.lower().find("uvis_total") > -1 and files.lower().endswith("drc.fits")]
    
        # Check the WCS solution is as expected
>       wcsname = fits.getval(tdp_files[0], "WCSNAME", ext=1).upper()
E       IndexError: list index out of range

/Users/sgoldman/Documents/drizzlepac_dev/tests/hap/test_svm_ibqk07.py:204: IndexError
------------------------------------------------------------ Captured stdout call ------------------------------------------------------------

test_svm_wcs_uvis.
___________________________________________________________ test_svm_wcs_uvis_all ____________________________________________________________

gather_output_data = []

    def test_svm_wcs_uvis_all(gather_output_data):
        # Check the output primary WCSNAME
        print("\ntest_svm_wcs_uvis_all.")
        uvis_files = [files for files in gather_output_data if files.lower().find("_uvis_") > -1 and files.lower().endswith("drc.fits")]
    
        wcsnames = [fits.getval(uvis, "WCSNAME", ext=1).upper() for uvis in uvis_files]
>       assert len(set(wcsnames)) == 1, f"WCSNAMES are not all the same for the UVIS detector: {wcsnames}"
E       AssertionError: WCSNAMES are not all the same for the UVIS detector: []
E       assert 0 == 1
E        +  where 0 = len(set())
E        +    where set() = set([])

/Users/sgoldman/Documents/drizzlepac_dev/tests/hap/test_svm_ibqk07.py:215: AssertionError
------------------------------------------------------------ Captured stdout call ------------------------------------------------------------

test_svm_wcs_uvis_all.
============================================================== warnings summary ==============================================================
../../../../opt/anaconda3/envs/stenv-python-3.10/lib/python3.10/site-packages/pkg_resources/__init__.py:121
  /opt/anaconda3/envs/stenv-python-3.10/lib/python3.10/site-packages/pkg_resources/__init__.py:121: DeprecationWarning: pkg_resources is deprecated as an API
    warnings.warn("pkg_resources is deprecated as an API", DeprecationWarning)

../../../../opt/anaconda3/envs/stenv-python-3.10/lib/python3.10/site-packages/pkg_resources/__init__.py:2870
  /opt/anaconda3/envs/stenv-python-3.10/lib/python3.10/site-packages/pkg_resources/__init__.py:2870: DeprecationWarning: Deprecated call to `pkg_resources.declare_namespace('mpl_toolkits')`.
  Implementing implicit namespace packages (as specified in PEP 420) is preferred to `pkg_resources.declare_namespace`. See https://setuptools.pypa.io/en/latest/references/keywords.html#keyword-namespace-packages
    declare_namespace(pkg)

../../../../opt/anaconda3/envs/stenv-python-3.10/lib/python3.10/site-packages/pkg_resources/__init__.py:2870
../../../../opt/anaconda3/envs/stenv-python-3.10/lib/python3.10/site-packages/pkg_resources/__init__.py:2870
../../../../opt/anaconda3/envs/stenv-python-3.10/lib/python3.10/site-packages/pkg_resources/__init__.py:2870
../../../../opt/anaconda3/envs/stenv-python-3.10/lib/python3.10/site-packages/pkg_resources/__init__.py:2870
  /opt/anaconda3/envs/stenv-python-3.10/lib/python3.10/site-packages/pkg_resources/__init__.py:2870: DeprecationWarning: Deprecated call to `pkg_resources.declare_namespace('sphinxcontrib')`.
  Implementing implicit namespace packages (as specified in PEP 420) is preferred to `pkg_resources.declare_namespace`. See https://setuptools.pypa.io/en/latest/references/keywords.html#keyword-namespace-packages
    declare_namespace(pkg)

../../../../opt/anaconda3/envs/stenv-python-3.10/lib/python3.10/site-packages/pkg_resources/__init__.py:2870
../../../../opt/anaconda3/envs/stenv-python-3.10/lib/python3.10/site-packages/pkg_resources/__init__.py:2870
  /opt/anaconda3/envs/stenv-python-3.10/lib/python3.10/site-packages/pkg_resources/__init__.py:2870: DeprecationWarning: Deprecated call to `pkg_resources.declare_namespace('stsci')`.
  Implementing implicit namespace packages (as specified in PEP 420) is preferred to `pkg_resources.declare_namespace`. See https://setuptools.pypa.io/en/latest/references/keywords.html#keyword-namespace-packages
    declare_namespace(pkg)

../../../../opt/anaconda3/envs/stenv-python-3.10/lib/python3.10/site-packages/stsci/tools/stpyfits.py:20
../../../../opt/anaconda3/envs/stenv-python-3.10/lib/python3.10/site-packages/stsci/tools/stpyfits.py:20
  /opt/anaconda3/envs/stenv-python-3.10/lib/python3.10/site-packages/stsci/tools/stpyfits.py:20: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
    ASTROPY_VER_GE32 = LooseVersion(astropy.__version__) >= LooseVersion('3.2')

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
========================================================== short test summary info ===========================================================
FAILED tests/hap/test_svm_ibqk07.py::test_svm_wcs_ir - IndexError: list index out of range
FAILED tests/hap/test_svm_ibqk07.py::test_svm_wcs_ir_all - AssertionError: WCSNAMES are not all the same for the IR detector: []
FAILED tests/hap/test_svm_ibqk07.py::test_svm_wcs_uvis - IndexError: list index out of range
FAILED tests/hap/test_svm_ibqk07.py::test_svm_wcs_uvis_all - AssertionError: WCSNAMES are not all the same for the UVIS detector: []
============================================ 4 failed, 5 passed, 10 warnings in 231.02s (0:03:51) ============================================
(stenv-python-3.10) sgoldman@wovas2023 drizzlepac_dev % 

@mdlpstsci Actually it might be the fact that the test is testing full poller files, which I haven't tested yet.

Copy link
Collaborator

@mdlpstsci mdlpstsci left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just a caution: For complex code, it is good to keep changes straightforward. I would not have inserted the new "aperture" value in the middle of the string which gets split up into tokens.

Please update any and all comment strings throughout the poller_utils.py module to include the new information. Again, I would use a different word than just "aperture" to differentiate it from the original FITS RAW keyword value.

Copy link
Collaborator

@mdlpstsci mdlpstsci left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You should also add an entry in the CHANGELOG.

@mdlpstsci
Copy link
Collaborator

@s-goldman It does not look like the output is on Artifactory. So...I would get the dataset and process it with the original code, keeping all the output. I would then process it with the new code, and see if the differences are valid.

Copy link
Collaborator

@mdlpstsci mdlpstsci left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Important documentation changes.

@mdlpstsci mdlpstsci removed the Do Not Merge PR which should not be merged label Oct 26, 2023
Copy link
Collaborator

@mdlpstsci mdlpstsci left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ready to go!

@s-goldman s-goldman changed the title [WIP] HLA-1110: aperture keyword from poller to header 10 10 23 HLA-1110: aperture keyword from poller to header Oct 26, 2023
@s-goldman s-goldman merged commit f9ed037 into spacetelescope:master Oct 26, 2023
13 of 14 checks passed
@s-goldman s-goldman deleted the HLA-1110_APERTURE_keyword_from_poller_to_header_10_10_23 branch October 26, 2023 20:36
mdlpstsci pushed a commit to mdlpstsci/drizzlepac that referenced this pull request Oct 26, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Update WFPC2 header Keyword 'APERTURE'
2 participants