Support running multiple tests per commit #27
Conversation
This makes it easier for callers.
This is working towards allowing multiple tests to be run at the same time.
If multiple tests might have failed, then it's not very meaningful to exit with the return code of the last failing test. Instead, output the retcodes of all of the failing tests in stderr, but just exit with retcode 1.
|
Thanks for writing this! I've been trying it out and my initial impression is that it works great. The only thing I've noticed so far is that it's slightly awkward to use
I usually don't want to proceed to the next commit, since I'm going to have to amend the commit and rerun tests anyway. However, I will often want to proceed to the next test, when one is for unit tests and the other is a linter, and they are fairly independent. |
|
@motlin: Thanks for the comment! That makes sense as a way of working. It would be fairly easy to add another option, but I hate piling on new options if there's a way around it. I wonder if it would be better if all tests run against a commit even if an earlier test fails (i.e., even without If you think that separate options are really needed, do you have a suggestion for what to call them? (For the sake of argument, there is yet another mode that would possibly make sense: When a particular test fails for a commit, then stop running that test for any subsequent commits, while continuing to run all of the other tests against that and subsequent commits. But I doubt that's useful enough to be worth implementing.) |
|
I think you'll want to keep I had an experiment that I never got working that takes every (commit,sha) pair, creates a git worktree for it, and runs the test in the worktree. I wanted it to support parallelism across commits, and it would have had the behavior you're talking about, where it would be possible for the linter test to pass on say 5 commits before the unit tests are even up to the 3rd commit. |
This is a riff on #24. Thanks to @hlovdal for that contribution, from which I borrowed heavily!
Compared to that PR, I have tried to make the new output more consistent and informative without being overwhelming. But I also went through most of the implementation again and revised it quite a bit. I preserved the authorship of commits that retained the flavor of the original even though they are somewhat transformed; @hlovdal let me know if you have any objections!
With this change, you can run multiple tests at the same time by feeding multiple test names to the
--test(or-tor--tests) option, likeThis checks out the commits as before, one by one, but for each commit it runs all of the specified tests. If any of the tests fail, the commit is considered bad. The good/bad status is recorded for each (commit, test) pair, in Git notes named after the individual tests.
The stdout is changed somewhat; now each line has a third field explaining which test failed; for example,
There is one such line per (commit, test) pair.
git test results, when applied to multiple tests, outputs one line per commit of the formThe stderr should mostly be self-explanatory. If a commit is found to be bad, the bad commit and its log messages are printed out as usual. But now, the header says what tests failed and what their individual return codes were:
The last thing output is the overall success or failure of all tests over all commits so that it's obvious whether everything was OK or whether there was a problem:
One other change is that, if
--keep-goingis specified, thengit-testdoesn't try to set its own exit code based on the exit code of the last failing test. This didn't seem like a useful feature, since multiple tests might have failed. But if--keep-goingis not specified, then the old behavior is preserved: the exit code ofgit-testis the same as that of the test that failed (if any).@hlovdal if you are interested to give this a try, let me know what you think. Since you've obviously used this kind of functionality before, you might notice if something important is missing or inconvenient.