tests: run tests based on the change included in the pr #14338

sergiocazzolato · 2024-08-08T21:35:44Z

This change shows the suggested tests based on the changes included in a pr.

There are defined some set of rules which can be used to determine which
tests to run. In this change we define 3 set of rules:

main is used for most of the systems.
nested is used for nested tests systems
trusty is used for ubuntu 14.04 only

In this pr, the suggested tests by spread-filter tool are displayed in the CI but those are not executed (we continue running all the tests). We do this to make sure the suggestions are correct, and once we are confident the tools is working well, the suggested tests will be executed.

…b77e1..4ab8b00afb 4ab8b00afb New spread-filter tool (canonical#53) cdf5cfd47b Remove centos-7 support f3996cc3fa change the spread label 1e309f41c6 change how legacy parameter is determine in remote.pull c43c35f7e3 run remote refresh and wait-for for xenial (skip bionic) 5262d30da7 make sure the test jobs are executed in runners with the spread label cb74259b7a add openstack systems 0b41fd40d3 fix tests.pkgs on arch-linux 558e109793 run fedora-40 spread tests in openstack 6f6187416d fix list implementation b4a5439c9b added more type annotatios to log_helper 58da1e36c3 mypy cleaned 1ff651e680 update wording of remote.pull 18615b1667 just usc scp -O when ssh version is -ge 9 cc68c9868b Added type annotations for log-filter 66f90d10cd Adding -O to scp command to make it compatible in uc24 tests 496cb7b5b3 removing support for centos-8 f2eef30db4 Updated the log helper and log parser 5a375ebf73 Formatting for python utils d3eed3faa5 fix codespell in CODE_OF_CONDUCT.md 18bcca6b14 new log helper d60381fcd9 add run number to filtered filename 5dde2d67b8 consider the tests execution in main 6b9a3aabcc change filtered log name b2756aa579 default file is .filtered.log 500b9dace4 Fix tests workflow 45db26a3d2 fix shellcheck error in log-filter fe45c27b7d create a var to store filter params 5a9b66d7dc filter spread results 51f9b055af New tool used to filter the spread.log output b8d20c1d5b fix snaps.name test with correct siffix spelling f640ac72e3 Add missing test details f0754df304 Filter the error y debug output in log-parser fc10196efd Add suggestions to details 94ac5ffe58 Add details on tests 501578c719 add more checks in os.query to check is-core_xx e8929207ff fix os-query for ubuntu comparing with core 226114641f os.query won't check SPREAD_SYSTEM anymore to compare core systems b89ec98b23 use local variables in os.query tool dacfd81de9 fix is_core functions 1db5214d5f Improve the remote docs (canonical#36) 2e4a3153a2 1 more comment 3a0fc57e1e add explanation about why we check for ( Do | Doing ) 4cf8e635bf fix os.query test after merge b89b4f8647 fix artifacts name d30cee6da0 Merge remote-tracking branch 'upstream/main' 5ef5dcbe8f Tests use artifacts in spread tests (canonical#51) 555c43d2ab Support auto-refresh with Do instead of Doing 96c2b0c19c remove tests support for ubuntu 23.04 (EoL) 74082c0c34 Tests improve remote wait (canonical#49) 5121bfb659 remove support for opensuse leap 15.4 (canonical#48) 30df700d08 Add new systems support (canonical#47) 1f08938925 Support check amazon linux version (canonical#46) 43533bdd97 Change the exit value checking for test formats (canonical#45) 3c88244c04 Update check-test-format to support a dir and a list of files (canonical#44) 510d95f429 add extra check for error in auto-refresh detection function 3289d4031b Try open the log with latin-1 encoding when utf-8 is not working 9db785499f improved how the tools are waiting for system reboot 2a5c4414a3 fix shellcheck errors 5e7b63883d Fixes for osquery and tests pkgs (canonical#43) 4c9145e2ac support reboot waiting for auto-refresh 45768f5188 show changes in unknown status after refresh 8013c30c2a Remove support for ubuntu 22.10 b32b80bf54 Fix remote.rait-for test in bionic 5675c625e9 Enable fedora 38 55f4471957 Support for new oss f2e88b357c New tool used to query spread json reports cacd35ede0 utils/spread-shellcheck: explain disabled warnings (canonical#42) c82afb2dee Support --no-install-recommends parameter when installing dependencies with tests.pkgs b84eea92e2 spread-shellcheck: fix quotes in environment variables (canonical#41) ab1e51c29f New comparison in os-query for core systems (canonical#40) e5ae22a5d4 systemd units can be overwritten 63540b845a Fix error messages in remote pull and push 75e8a426a5 make sure the unit is removed in tests.systemd test 9089ff5c02 Update tests to use the new tests.systemd stop-unit 44ecd5e56a Move tests.systemd stop-units to stop-unit 01a2a83b4b Update tests.systemd to have stop units as systemd.sh 162e93bd35 update tests.systemd CLI options to be the same than retry command 14aa43a405 new feature to re-run failed spread tests (canonical#39) 604cb782db Fix shellcheck in systemd tool bfc71082c8 Update the tests.systemd to allow parameters waiting for service status 8a2d0a99df Adding quiet tool and removing set +-x from tests.pkgs d90935d2a4 A comment explaining about the default values for wait-for 3232c5dba7 Add support for ubuntu 23.04 a7164fba07 remove fedora 35 support, add fedora 37 support 89b9eb5301 Update systems supported 92bb6a0664 Include snap-sufix in the snaps.name tool git-subtree-dir: tests/lib/external/snapd-testing-tools git-subtree-split: 4ab8b00afbfe73017a88c58018c72a2d5beea3be

…port-changes-spread-filter tests/lib/external/snapd-testing-tools/tests/remote.retry/task.yaml

…port-changes-spread-filter

…00afb..e43b9e314d e43b9e314d rename test libraries to avoid static check errors in snapd git-subtree-dir: tests/lib/external/snapd-testing-tools git-subtree-split: e43b9e314d0a078521d0ee3ec115f11c03005c12

…port-changes-spread-filter

…e314d..33aa8d1e53 33aa8d1e53 fix details in remote.retry test git-subtree-dir: tests/lib/external/snapd-testing-tools git-subtree-split: 33aa8d1e53161e740d7ac720d13284fb9ddacb82

This change runs the tests based on the changes in cluded in a pr. There are defined some set of rules which can be used to determine which tests to run. In this change we define 3 set of rules: 1. main is used for most of the systems. 2. nested is used for nested tests systems 3. trusty is used for ubuntu 14.04 only

…d1e53..1d4350a818 1d4350a818 Make sure the tools are at the begining of the path 45c2f45004 address comments 8743ceaf8c tools: fix support for multi-arch packages on apt-based systems (canonical#54) git-subtree-dir: tests/lib/external/snapd-testing-tools git-subtree-split: 1d4350a818c69da0240f4848ba08dc9411293e5a

…lter-spread-tests-based-on-changes

…ad-tests-based-on-changes

pedronis · 2024-08-23T16:32:30Z

tests/lib/spread/rules/main.yaml

+  tests:
+    from:
+      - tests/main/.*
+      - tests/core/.*
+      - tests/completion/.*
+      - tests/cross/.*
+      - tests/regression/.*
+      - tests/smoke/.
+      - tests/unit/.*
+      - tests/upgrade/.*
+    to: [$SELF]
+
+  unit:
+    from: [.*_test.go]
+    to:
+     - tests/unit/go
+     - tests/unit/c-unit-tests-clang
+     - tests/unit/c-unit-tests-gcc
+
+  rest:


is "rest" special and only used if the changed file didn't match any other rule?

right, it is for the other files which are not matching with previous rules.

pedronis · 2024-08-23T16:34:25Z

tests/lib/spread/rules/main.yaml

+      - tests/smoke/.
+      - tests/unit/.*
+      - tests/upgrade/.*
+    to: [$SELF]


so if we change a test we just run that test? what if it's restore is wrong and it will affect other tests?

Right, if the change breaks the restore this could affect following tests

As an alternative we could run:
to: [ $SELF, /path/to/sanity/suite ]

So we make sure we always cover smoke or a sanity suite with any change

I propose that because we have code reviews, and also the prepare-restore helpers make sure the environment is cleaned after each test is executed. So, installed snaps/debs are deleted, test directory is cleaned, it is verified that a big part of the file system has not changed, etc

you probably have more a sense than me how oftern we get test interference problems? anyway on master we will run all the tests?

yes, finally if the there is no pr, we will run everything, in fact I am thinking on running also the nested tests too.

pedronis

lgtm

bboozzoo · 2024-08-27T10:55:54Z

.github/workflows/test.yaml

              done
          fi
          echo RUN_TESTS="$RUN_TESTS"  >> $GITHUB_ENV
+          echo "Suggested tests by spread-filter tool"
+          echo "$SUGGESTED_TESTS"


When doing the changes to run the suggested tests only we should ensure that when executing the CI pipeline on master, we should rather run all tests, such that we don't end up in a situation where only a subset of the whole test suite ever runs.

We must also have a clear path for enabling/disabling this feature on PR bases, eg. through a label.

right, that was also mentioned by Samuele. In the next pr I'll include that check for sure.

bboozzoo · 2024-08-27T11:04:58Z

tests/lib/spread/rules/main.yaml

+    to: [$SELF]
+
+  unit:
+    from: [.*_test.go]


Sorry, I haven't reviewed the filter tool, how does the logic for matching the rules work? Is the resulting set of tests a union of the results from each ruleset? i.e. such that a PR modifying foo_test.go and tests/main/foo will trigger both unit and tests rulesets?

right, in that case it executes both

the filter get all the changes and calculates the tests to be executed for each change and makes an union for all the tests to execute

alfonsosanchezbeato

Thanks, couple of minor comments

alfonsosanchezbeato · 2024-08-30T20:03:08Z

.github/workflows/test.yaml

-            tests: 'tests/nested/...'
+            tests: 'tests/...'


I am a bit confused about this change, it does not match the other nested groups.

you are right, I'll update this, thanks for the review

alfonsosanchezbeato · 2024-08-30T20:10:20Z

tests/lib/spread/rules/main.yaml

+    to: [$SELF]
+
+  unit:
+    from: [.*_test.go]


We have some test-only packages like seedtest, not sure if they need to be considered here or not.

we could consider those files in the future, I would start easy and then go a bit more on details. The idea of this change is to monitor how the tool works before we move to this solution

zyga

Quick pass. I'm sorry for sending so late. I left a few comments but I didn't notice they were in pending state.

zyga · 2024-08-20T07:39:46Z

tests/lib/spread/rules/main.yaml

@@ -0,0 +1,23 @@
+rules:


Can you please leave a comment that links to the syntax / reference of this file.

Sure, I'll add some documentation explaining the files.

zyga · 2024-08-20T07:40:25Z

tests/lib/spread/rules/trusty.yaml

@@ -0,0 +1,7 @@
+rules:
+  all:
+    from: [.*]


What does this rule encode? That all the changes run on trusty? Can you perhaps leave a comment inline in the code?

Sure, I'll add some documentation explaining the files.

README added

zyga · 2024-08-20T07:42:44Z

.github/workflows/test.yaml

@@ -772,6 +799,16 @@ jobs:
          # This step has to be after the cache is restored
          mkdir -p "$TEST_RESULTS_DIR"        

+    - name: Get changed files
+      id: changed-files
+      uses: tj-actions/[email protected]


Not thinking about this much could we perhaps achieve the same thing without a 3rd party dependency using https://docs.github.com/en/actions/writing-workflows/workflow-syntax-for-github-actions#onpushpull_requestpull_request_targetpathspaths-ignore which quites the example:

on: push: paths: - '**.js'

I see this like a filter to determine when to trigger a workflow and the action retrieves the list of changed files in the current change. I checked in the actions store and all the actions to get changes are third party sadly.
The good part of this one used is that seems to be maintained actively.

alfonsosanchezbeato

LGTM, thanks for the changes

) * Squashed 'tests/lib/external/snapd-testing-tools/' changes from 1c8efb77e1..4ab8b00afb 4ab8b00afb New spread-filter tool (canonical#53) cdf5cfd47b Remove centos-7 support f3996cc3fa change the spread label 1e309f41c6 change how legacy parameter is determine in remote.pull c43c35f7e3 run remote refresh and wait-for for xenial (skip bionic) 5262d30da7 make sure the test jobs are executed in runners with the spread label cb74259b7a add openstack systems 0b41fd40d3 fix tests.pkgs on arch-linux 558e109793 run fedora-40 spread tests in openstack 6f6187416d fix list implementation b4a5439c9b added more type annotatios to log_helper 58da1e36c3 mypy cleaned 1ff651e680 update wording of remote.pull 18615b1667 just usc scp -O when ssh version is -ge 9 cc68c9868b Added type annotations for log-filter 66f90d10cd Adding -O to scp command to make it compatible in uc24 tests 496cb7b5b3 removing support for centos-8 f2eef30db4 Updated the log helper and log parser 5a375ebf73 Formatting for python utils d3eed3faa5 fix codespell in CODE_OF_CONDUCT.md 18bcca6b14 new log helper d60381fcd9 add run number to filtered filename 5dde2d67b8 consider the tests execution in main 6b9a3aabcc change filtered log name b2756aa579 default file is .filtered.log 500b9dace4 Fix tests workflow 45db26a3d2 fix shellcheck error in log-filter fe45c27b7d create a var to store filter params 5a9b66d7dc filter spread results 51f9b055af New tool used to filter the spread.log output b8d20c1d5b fix snaps.name test with correct siffix spelling f640ac72e3 Add missing test details f0754df304 Filter the error y debug output in log-parser fc10196efd Add suggestions to details 94ac5ffe58 Add details on tests 501578c719 add more checks in os.query to check is-core_xx e8929207ff fix os-query for ubuntu comparing with core 226114641f os.query won't check SPREAD_SYSTEM anymore to compare core systems b89ec98b23 use local variables in os.query tool dacfd81de9 fix is_core functions 1db5214d5f Improve the remote docs (canonical#36) 2e4a3153a2 1 more comment 3a0fc57e1e add explanation about why we check for ( Do | Doing ) 4cf8e635bf fix os.query test after merge b89b4f8647 fix artifacts name d30cee6da0 Merge remote-tracking branch 'upstream/main' 5ef5dcbe8f Tests use artifacts in spread tests (canonical#51) 555c43d2ab Support auto-refresh with Do instead of Doing 96c2b0c19c remove tests support for ubuntu 23.04 (EoL) 74082c0c34 Tests improve remote wait (canonical#49) 5121bfb659 remove support for opensuse leap 15.4 (canonical#48) 30df700d08 Add new systems support (canonical#47) 1f08938925 Support check amazon linux version (canonical#46) 43533bdd97 Change the exit value checking for test formats (canonical#45) 3c88244c04 Update check-test-format to support a dir and a list of files (canonical#44) 510d95f429 add extra check for error in auto-refresh detection function 3289d4031b Try open the log with latin-1 encoding when utf-8 is not working 9db785499f improved how the tools are waiting for system reboot 2a5c4414a3 fix shellcheck errors 5e7b63883d Fixes for osquery and tests pkgs (canonical#43) 4c9145e2ac support reboot waiting for auto-refresh 45768f5188 show changes in unknown status after refresh 8013c30c2a Remove support for ubuntu 22.10 b32b80bf54 Fix remote.rait-for test in bionic 5675c625e9 Enable fedora 38 55f4471957 Support for new oss f2e88b357c New tool used to query spread json reports cacd35ede0 utils/spread-shellcheck: explain disabled warnings (canonical#42) c82afb2dee Support --no-install-recommends parameter when installing dependencies with tests.pkgs b84eea92e2 spread-shellcheck: fix quotes in environment variables (canonical#41) ab1e51c29f New comparison in os-query for core systems (canonical#40) e5ae22a5d4 systemd units can be overwritten 63540b845a Fix error messages in remote pull and push 75e8a426a5 make sure the unit is removed in tests.systemd test 9089ff5c02 Update tests to use the new tests.systemd stop-unit 44ecd5e56a Move tests.systemd stop-units to stop-unit 01a2a83b4b Update tests.systemd to have stop units as systemd.sh 162e93bd35 update tests.systemd CLI options to be the same than retry command 14aa43a405 new feature to re-run failed spread tests (canonical#39) 604cb782db Fix shellcheck in systemd tool bfc71082c8 Update the tests.systemd to allow parameters waiting for service status 8a2d0a99df Adding quiet tool and removing set +-x from tests.pkgs d90935d2a4 A comment explaining about the default values for wait-for 3232c5dba7 Add support for ubuntu 23.04 a7164fba07 remove fedora 35 support, add fedora 37 support 89b9eb5301 Update systems supported 92bb6a0664 Include snap-sufix in the snaps.name tool git-subtree-dir: tests/lib/external/snapd-testing-tools git-subtree-split: 4ab8b00afbfe73017a88c58018c72a2d5beea3be * Squashed 'tests/lib/external/snapd-testing-tools/' changes from 4ab8b00afb..e43b9e314d e43b9e314d rename test libraries to avoid static check errors in snapd git-subtree-dir: tests/lib/external/snapd-testing-tools git-subtree-split: e43b9e314d0a078521d0ee3ec115f11c03005c12 * Squashed 'tests/lib/external/snapd-testing-tools/' changes from e43b9e314d..33aa8d1e53 33aa8d1e53 fix details in remote.retry test git-subtree-dir: tests/lib/external/snapd-testing-tools git-subtree-split: 33aa8d1e53161e740d7ac720d13284fb9ddacb82 * tests: run tests based on the change included in the pr This change runs the tests based on the changes in cluded in a pr. There are defined some set of rules which can be used to determine which tests to run. In this change we define 3 set of rules: 1. main is used for most of the systems. 2. nested is used for nested tests systems 3. trusty is used for ubuntu 14.04 only * fix actions script (bash) * show the suggested tests instead of running them * Squashed 'tests/lib/external/snapd-testing-tools/' changes from 33aa8d1e53..1d4350a818 1d4350a818 Make sure the tools are at the begining of the path 45c2f45004 address comments 8743ceaf8c tools: fix support for multi-arch packages on apt-based systems (canonical#54) git-subtree-dir: tests/lib/external/snapd-testing-tools git-subtree-split: 1d4350a818c69da0240f4848ba08dc9411293e5a * udpate how suggested tests are displayed * udpate rule names * updated tests to run on nested 24.04 * Adding README in the rules directory

sergiocazzolato added 7 commits August 7, 2024 12:58

Merge commit '07a9e9e95970c06f4f6c886244fca9cd2dd2c8d7' into tests-im…

8bffd16

…port-changes-spread-filter tests/lib/external/snapd-testing-tools/tests/remote.retry/task.yaml

Merge commit 'e7ff6ad63d9e0055bd059f604aa7950041ebf070' into tests-im…

faa2adc

…port-changes-spread-filter

Squashed 'tests/lib/external/snapd-testing-tools/' changes from 4ab8b…

e7ff6ad

…00afb..e43b9e314d e43b9e314d rename test libraries to avoid static check errors in snapd git-subtree-dir: tests/lib/external/snapd-testing-tools git-subtree-split: e43b9e314d0a078521d0ee3ec115f11c03005c12

Merge commit 'e81e15ae66d75f68d359e79b5b2d4f0e03be1caf' into tests-im…

5fae9fa

…port-changes-spread-filter

Squashed 'tests/lib/external/snapd-testing-tools/' changes from e43b9…

e81e15a

…e314d..33aa8d1e53 33aa8d1e53 fix details in remote.retry test git-subtree-dir: tests/lib/external/snapd-testing-tools git-subtree-split: 33aa8d1e53161e740d7ac720d13284fb9ddacb82

sergiocazzolato requested review from bboozzoo, zyga, pedronis and ernestl August 8, 2024 21:35

sergiocazzolato marked this pull request as draft August 8, 2024 21:35

sergiocazzolato requested a review from alfonsosanchezbeato August 8, 2024 21:36

fix actions script (bash)

970ad4d

pedronis added the Needs Samuele review Needs a review from Samuele before it can land label Aug 9, 2024

sergiocazzolato added 4 commits August 14, 2024 11:40

show the suggested tests instead of running them

9a7dcff

Merge commit '9327b3704ff2a764aa64398e2ea2c8ea64981f89' into tests-fi…

f961436

…lter-spread-tests-based-on-changes

udpate how suggested tests are displayed

d0f4749

sergiocazzolato marked this pull request as ready for review August 14, 2024 14:47

sergiocazzolato added 2 commits August 14, 2024 12:46

Merge remote-tracking branch 'upstream/master' into tests-filter-spre…

38830ba

…ad-tests-based-on-changes

udpate rule names

e6de86e

pedronis reviewed Aug 23, 2024

View reviewed changes

sergiocazzolato requested a review from pedronis August 26, 2024 13:44

pedronis approved these changes Aug 26, 2024

View reviewed changes

bboozzoo reviewed Aug 27, 2024

View reviewed changes

sergiocazzolato requested a review from bboozzoo August 27, 2024 18:53

alfonsosanchezbeato reviewed Aug 30, 2024

View reviewed changes

updated tests to run on nested 24.04

29f3805

sergiocazzolato requested a review from alfonsosanchezbeato September 2, 2024 12:08

zyga reviewed Sep 3, 2024

View reviewed changes

alfonsosanchezbeato approved these changes Sep 3, 2024

View reviewed changes

Adding README in the rules directory

882eb38

sergiocazzolato merged commit a65eed3 into canonical:master Sep 4, 2024
49 of 53 checks passed

sergiocazzolato deleted the tests-filter-spread-tests-based-on-changes branch September 4, 2024 14:54

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

tests: run tests based on the change included in the pr #14338

tests: run tests based on the change included in the pr #14338

sergiocazzolato commented Aug 8, 2024 •

edited

Loading

pedronis Aug 23, 2024 •

edited

Loading

sergiocazzolato Aug 26, 2024

pedronis Aug 23, 2024

sergiocazzolato Aug 26, 2024

sergiocazzolato Aug 26, 2024 •

edited

Loading

pedronis Aug 26, 2024

sergiocazzolato Aug 27, 2024 •

edited

Loading

pedronis left a comment

bboozzoo Aug 27, 2024

sergiocazzolato Sep 3, 2024

bboozzoo Aug 27, 2024

sergiocazzolato Aug 27, 2024

sergiocazzolato Aug 27, 2024

alfonsosanchezbeato left a comment

alfonsosanchezbeato Aug 30, 2024

sergiocazzolato Sep 2, 2024

alfonsosanchezbeato Aug 30, 2024

sergiocazzolato Sep 2, 2024

zyga left a comment

zyga Aug 20, 2024

sergiocazzolato Sep 3, 2024

zyga Aug 20, 2024

sergiocazzolato Sep 3, 2024

sergiocazzolato Sep 3, 2024

zyga Aug 20, 2024

sergiocazzolato Sep 3, 2024

alfonsosanchezbeato left a comment

tests: run tests based on the change included in the pr #14338

tests: run tests based on the change included in the pr #14338

Conversation

sergiocazzolato commented Aug 8, 2024 • edited Loading

pedronis Aug 23, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sergiocazzolato Aug 26, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sergiocazzolato Aug 27, 2024 • edited Loading

Choose a reason for hiding this comment

pedronis left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

alfonsosanchezbeato left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

zyga left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

alfonsosanchezbeato left a comment

Choose a reason for hiding this comment

sergiocazzolato commented Aug 8, 2024 •

edited

Loading

pedronis Aug 23, 2024 •

edited

Loading

sergiocazzolato Aug 26, 2024 •

edited

Loading

sergiocazzolato Aug 27, 2024 •

edited

Loading