Retry Plugin Retries All Cucumber Tests Instead of Only Failed Tests

[lnhrdt]

The Cucumber Companion plugin docs describe how to use it with the Test Retry plugin, so I tried using them together. But I've found an issue where retries don’t work as expected in projects with both plugins.

Reproduction

I created a repo to demostrate the issue: https://github.com/lnhrdt/gradle-plugin-cucumber-retry-issue

The repo has three subprojects:

1. just-junit
2. suite
3. cucumber

Each subproject contains two tests (one passing, one failing) and configures the Test Retry plugin to retry failed tests once.

Run this command to reproduce the issue:

./gradlew test --continue

Expected Behavior:

• Passing tests run once in each subproject.
• Failing tests run twice in each subproject.

Observed Behavior:

• just-junit and suite: Passing tests run once and failing tests retry correctly.
• cucumber: Passing tests incorrectly run twice.

Sample Output:

> Task :cucumber:test FAILED

fail > Cucumber > Fail > features.fail.it fails FAILED
    org.opentest4j.AssertionFailedError at JUnitSupport.kt:56

pass > Cucumber > Pass > features.pass.it passes PASSED

fail > Cucumber > Fail > features.fail.it fails FAILED
    org.opentest4j.AssertionFailedError at JUnitSupport.kt:56

pass > Cucumber > Pass > features.pass.it passes PASSED

4 tests completed, 2 failed

> Task :just-junit:test FAILED

FailTest > alwaysFail() FAILED
    org.opentest4j.AssertionFailedError at FailTest.kt:9

PassTest > alwaysPass() PASSED

FailTest > alwaysFail() FAILED
    org.opentest4j.AssertionFailedError at FailTest.kt:9

3 tests completed, 2 failed

> Task :suite:test FAILED

FailTest > alwaysFail() FAILED
    org.opentest4j.AssertionFailedError at FailTest.kt:9

PassTest > alwaysPass() PASSED

FailTest > alwaysFail() FAILED
    org.opentest4j.AssertionFailedError at FailTest.kt:9

3 tests completed, 2 failed

Notes

I suspect this issue is related to the Cucumber Companion plugin but acknowledge it could stem from my project setup or the Test Retry plugin. Let me know if I should report this issue to the Test Retry plugin instead.

[pshevche]

@lnhrdt , thank you for reaching out. This is a limitation of the retry plugin: gradle/test-retry-gradle-plugin#276 (comment). As I mention in that issue, having proper retry support for Cucumber features is on our roadmap, but we don't have a timeline yet.

[lnhrdt]

Thanks @pshevche for the context here that test-retry is the culprit.

After looking through the comments, it seems that gradle/test-retry-gradle-plugin#278 is the issue my team should follow–our projects (and the reproduction project I shared) use JUnit 5, not 4.

I understand you don't have a timeline to share but we'll stay subscribed to those issues for update. We'd get a lot of value out of having this limitation resolved!