How to retry Cucumber scenarios using test-retry-gradle-plugin and JUnit4 runner

[hunterabhi212223]

I am new to using gradle.I have added below in my build.gradle.
plugins {
id 'org.gradle.test-retry' version '1.0.0'
}

test {
retry {
failOnPassedAfterRetry = true
maxFailures = 3
maxRetries = 2
}
}
Here i am getting weird issue. could not run phased build action using gradle distribution gradle 7.0.1. Could not find method retry() for argument admin of type org.gradle.api

Did i missed anything to add or gradle removed test-retry

here i am unsing junit 4
Gradle 7.1

[hunterabhi212223]

Below error i am getting
Could not run phased build action using Gradle distribution 7.0.1.zip. A problem occured evaluatin root project. Could not find method retry() for arguments [Build*****] on task ':admin:test' of type org.gradle.api.tasks.testing.Test

[pshevche]

@hunterabhi212223 , could you try using the latest version of the plugin, i.e., 1.5.8?

[pshevche]

Other than that, your configuration looks correct.

[hunterabhi212223]

With latest version it is working. for passed one also it is running twice. how can i run passed one 1 time and failed one twice
test {
retry {
failOnPassedAfterRetry = true
maxFailures = 3
maxRetries = 1
}
}

here i am using 1.5.6

[pshevche]

It should already work this way so that only failed methods are retried. But it depends on how your class is set up. For example, for parameterized methods, we will retry all iterations and not just failed ones. You need to provide a reproducer of your particular case for us to check why it is not working as expected.

[hunterabhi212223]

you want me to share the scenario?
or any other info

[pshevche]

It would be best if you share your test class/method (or simplified version of it). Also, try it out on a different method and check if the plugin works as expected. Maybe this is a one of the special cases where all methods will be retried.

[hunterabhi212223]

Runner File
@RunWith(Cucumber.class)

my feature file

Scenarion: verify name in google page
Given launch google page
Then Verify title of the page

class file

@then("Verify_title_of_the_page")
public void verify_name() {
//Code
Assert.assertEquals("driver.getTitle(),"Google"
}

[pshevche]

I'll try to take a look at it today. We haven't seen anyone using the plugin for cucumber tests (especially with JUnit 4 runner). I'll need to check what is possible.

[hunterabhi212223]

is there any way that we can tell in hooks if it failed then run the gradle-retry with maxRetry = 1

[pshevche]

@hunterabhi212223 , I looked into the issue today, and everything seems to work as expected. I managed to create a feature file with a passing and flaky scenario, and only the flaky scenario was re-executed. See this test for the detailed setup: https://github.com/gradle/test-retry-gradle-plugin/pull/277/files#diff-fcd808be75c92d4bdb3a7bb939cff581b01d655d67e7768e05bbea991bcba02a. I hope this will help you configure retries in your project. If I misunderstood your configuration, let me know.

[pshevche]

If you have multiple feature files, then also only failing scenarios from failing feature files will be retried.

[petrandreev]

FYI: I was struggling with proposed setup for retries:
https://github.com/cucumber/cucumber-jvm/tree/main/cucumber-junit-platform-engine#using-gradle
Unfortunately the entire feature set gets executed on retry, see the branch:
https://github.com/petrandreev/test-retry-gradle-plugin/blob/petrandreev/more-cucumber-func-tests/plugin/src/test/groovy/org/gradle/testretry/testframework/CucumberJUnit5SuiteFuncTest.groovy

The reason could be that in discovery phase (gradle) JUnit5 implementation decides that all Cucumber features, being not class and not method TestSources shouldRun and the postDiscovery filter, propagated from RetryTestExecuter on fork
doesn't play any role, since with JUnitSuites and Cucumber we have hierarchical discovery, where the fallback relies on children. See also gradle/gradle#20169.

[pshevche]

@petrandreev , let's move this discussion to a dedicated issue, since this one is about assisting @hunterabhi212223 with setting up retries for Cucumber tests using JUnit4. Here it is: #278. You are right, at the moment, the entire feature will be retried. As you said, this is a limiting factor on the Gradle side, which I am not sure we'll be able to work around in this plugin. I documented your findings + added tests for the future feature in this PR: #277. I added full Cucumber support to our roadmap, but I can't provide any schedule at the moment.

[pshevche]

Updated the issue title to make it easier to discover

[hunterabhi212223]

@pshevche Thank you for looking into the issue. I don't understand what it is written in groovy.

Anyways I understood little based on that I have few questions.

1. you are creating a new feature file for failed one and running that feature file
2. if you are running passed one and failed one. Do we get one report or it will display combination of pass and fail report
   Another one is only for failure report

If possible could please mention where to write that code part in gradle. I am completely new and learning from beginning.

[pshevche]

@hunterabhi212223 , no worries, let's figure it out:

1. you are creating a new feature file for failed one and running that feature file

1. Create your feature files regularly as you would normally do in the src/test/resources directory (example). Those can include any number of scenarios.
2. Create your step definitions file in src/test/java directory, also as usual (example).
3. Create your runner file to support executing tests with JUnit4. This allows JUnit to discover your feature files (using the features property) and identify corresponding Java steps (using the glue property) (example).
4. Add retry configuration that you have posted above.

At this point, you can run and retry your Cucumber scenarios using Gradle and this plugin.

2. if you are running passed one and failed one. Do we get one report or it will display combination of pass and fail report

As described in this documentation, any files written by Cucumber will be overwritten retrying. I haven't tested it myself, but this means that on your first run, you would have a report containing all executions (passed + failed). That report will be overwritten on the retry and contain only retried executions.

Does it help?

[hunterabhi212223]

@pshevche
like you mentioned in the above
created a feature file with multiple scenario (src/test/resources)
created SD file (src/test/java/step)
My runner file
@RunWith(Cucumber.class)
@CucumberOptions(features="src/test/resources", strict = true, glue = {src/test/java},
plugin = {}
tags={"@test"},
dryRun = false}

public class RetryFeatureTest {
}

        plugin configured in build.gradle file
        
        Now i executed. i can see passed executed twice, failed executed twice.
        
        please correct me if i did any mistake in the above

[pshevche]

@hunterabhi212223 , which versions of cucumber-junit and junit are you on?

[hunterabhi212223]

cucmber-java 4.8.0
cucumber junit 4.8.0

[pshevche]

And junit:junit itself?

[hunterabhi212223]

test{
useJUnitPlatform()}

[pshevche]

Aha, ok, now this makes sense. Unfortunately, retrying individual scenarios with JUnit Platform does not work at the moment. We will retry all scenarios in a feature. That's the same issue that @petrandreev mentioned above. We will look into it at #278. As for now, the plugin might not be that helpful to you.

[hunterabhi212223]

Any workaround for this as of know
like updating version
junit platform

[pshevche]

It works without JUnit platform, so with plain JUnit 4. If you remove useJUnitPlatform(), retries work as expected.
But there is obviously a reason why you use JUnit platform. If you want to stay on it, then the only workaround is to split feature files to have fewer scenarios. We will still retry all scenarios if one of them fails, but there will be just fewer of those.

[hunterabhi212223]

after removing JunitPlatform() it is working fine. it is showing it is executed on default package.

[pshevche]

Happy to hear that! I'll consider this issue resolved in this case. Feel free to re-open if you face any other issues configuring retries for your project.

[hunterabhi212223]

Thank you for your support. If you find any alternate solution for junitplatform() please post here.
If I upgrade my junit to 5 it will work?

[hunterabhi212223]

I mean junitplatform () with junit 5

[pshevche]

If I upgrade my junit to 5 it will work?

Unfortunately, no. Everything using JUnit platform will not work. Sure, will let you know once we have any updates on supporting Cucumber with JUnit platform.

[hunterabhi212223]

Currently on which version it is working

[pshevche]

Currently on which version it is working

Only if you use JUnit4 without JUnit platform.

[hunterabhi212223]

Okay thank you for info