Allow marking tests Pending when they fail audits #22
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Works on #21. This is a draft, mostly to start the conversation for one way this could work.
This doesn't completely fix the problem that is identified in #21, but it is a step in that direction. The idea is that, if the configuration setting
accessibility_audit_skip_on_error
is set to true, if anaxe
violation is detected, instead of generating an error it will instead mark the test asskipped
with theaxe
output as the message in the skip output.There is a problem with this approach, which is the same problem identified in issue #7 - as soon as a single test is marked skipped or failed, then the rest of the test is skipped. This means that if a test has both failing application logic and failing accessibility validation, the first of those tests to fail will be the one that is reported. For a system test, since any interaction with the page will trigger the accessibility tests, it is likely that the test will be marked as skipped and the application logic failure will be hidden.
This can be solved by running the tests a second time with
accessibility_audit_enabled = false
and comparing the results, but that does require two runs through the test suite.