When test retry fails, it obscures the actual failed assertion

[adam-enko]

Test-retry is enabled in Dokka.

https://github.com/Kotlin/dokka/blob/df672b74de0d47679ba7c73b331de83de3608623/build-logic/src/main/kotlin/dokkabuild.base.gradle.kts#L27-L32

When a test fails, test-retry attempts to retry it, but fails (probably because some Dokka tests use Kotest #93).

Aside from the Kotest compatibility issues, it's inconvenient that the actual test error has been obscured.

All I can see in the build logs is the location of the failed assertion, but I can't see any of the error message.

For example, I would expect to see the withClue message: "files with errors: ${filesWithErrors.joinToString("\n") { " - ${it.toUri()}" }}"

https://github.com/Kotlin/dokka/blob/383f029a89a5ab3b58ec59ac1a8477cd0e179ed5/dokka-runners/dokka-gradle-plugin/src/testFunctional/kotlin/KmpCommonSourceSharedWithDependentsTest.kt#L39-L43

> Task :dokka-gradle-plugin:test FROM-CACHE
> Task :dokka-gradle-plugin:testFunctional

org.jetbrains.dokka.gradle.KmpCommonSourceSharedWithDependentsTest > common source set is propagated to dependents FAILED
    java.lang.AssertionError at KmpCommonSourceSharedWithDependentsTest.kt:42
        Caused by: java.lang.AssertionError at KmpCommonSourceSharedWithDependentsTest.kt:42

105 tests completed, 1 failed
There were failing tests. See the report at: file:///home/runner/work/dokka/dokka/dokka-runners/dokka-gradle-plugin/build/reports/tests/testFunctional/index.html

> Task :dokka-gradle-plugin:testFunctional FAILED

342 actionable tasks: 231 executed, 111 from cache
FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':dokka-gradle-plugin:testFunctional'.
> The following test methods could not be retried, which is unexpected. Please file a bug report at https://github.com/gradle/test-retry-gradle-plugin/issues
     org.jetbrains.dokka.gradle.KmpCommonSourceSharedWithDependentsTest#common source set is propagated to dependents


* Try:
> Run with --info or --debug option to get more log output.
> Get more help at https://help.gradle.org./

* Exception is:
org.gradle.api.tasks.TaskExecutionException: Execution failed for task ':dokka-gradle-plugin:testFunctional'.
...


BUILD FAILED in 9m 16s

Publishing build scan...
https://ge.jetbrains.com/s/jhbii4zkbocee

Build scan: https://ge.jetbrains.com/s/jhbii4zkbocee

[pshevche]

@adam-enko , this is not related to the test retry plugin. Gradle's test reporting does not write the clue message to the console. I checked it on another project with retries disabled and also see that the clue message is not being printed (ref). However, the clue message will be visible in the Build Scan (here or here).

If you'd like to have this behavior in the console report as well, I suggest filing an issue with the Gradle core team: https://github.com/gradle/gradle

[pshevche]

I'll close this issue for now as there is nothing we can do on the test retry plugin side (apart from adding support for Kotest :))