Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Some of the Retina capture unit tests are flaky #1045

Closed
nddq opened this issue Nov 20, 2024 · 3 comments
Closed

Some of the Retina capture unit tests are flaky #1045

nddq opened this issue Nov 20, 2024 · 3 comments
Assignees
@kamilprz @ibezrukavyi
Labels
area/infra Test, Release, or CI Infrastructure
Milestone
1.0

Comments

@nddq
Copy link
Contributor

nddq commented Nov 20, 2024

Image
Need to investigate why
@nddq nddq added the area/infra Test, Release, or CI Infrastructure label Nov 20, 2024
@anubhabMajumdar anubhabMajumdar added this to the 1.0 milestone Nov 21, 2024
@kamilprz
Copy link
Contributor

kamilprz commented Dec 10, 2024 •
edited
Loading

Hey @nddq , did you run the make test under any specific conditions? I have ran it around 10 times from my end and each time I get all tests passing. Unable to reproduce the failures.

Image

edit: looking back at PRs, there has been at least one related to captures since this issue was made, its possible that fixed the failing tests

@nddq
Copy link
Contributor Author

nddq commented Dec 10, 2024

hey @kamilprz, you can view pass test runs on GitHub Actions. I have not tried to reproduce this locally yet since this seems to only occured in our pipelines

@kamilprz kamilprz mentioned this issue Dec 31, 2024
Merged
7 tasks
@kamilprz
Copy link
Contributor

Example of failing tests - https://github.com/microsoft/retina/actions/runs/12469454230/job/34802618326

github-merge-queue bot pushed a commit that referenced this issue Jan 2, 2025
@kamilprz
fix(test): Add custom comparer to ignore timestamps in capture tests (#…
87d2196 
…1173)

# Description

Previously, in [PR 894](#894),
timestamps were introduced to be able to print capture names as their
jobs were beginning.

This PR addresses an oversight in the testing which compared whether the
timestamp of the test and the timestamp of the test capture were the
same. This was not always the case - particularly if there was some
delay between the beginning of the test and the beginning of the
capture. The change in this PR adds a custom comparer to the `cmp.Diff`
which ignores the timestamp environment variables.

## Related Issue

[1045](#1045)

## Checklist

- [x] I have read the [contributing
documentation](https://retina.sh/docs/contributing).
- [x] I signed and signed-off the commits (`git commit -S -s ...`). See
[this
documentation](https://docs.github.com/en/authentication/managing-commit-signature-verification/about-commit-signature-verification)
on signing commits.
- [x] I have correctly attributed the author(s) of the code.
- [x] I have tested the changes locally.
- [x] I have followed the project's style guidelines.
- [ ] I have updated the documentation, if necessary.
- [x] I have added tests, if applicable.

## Testing Completed

The tests now succeed regardless of whether a delay occurs or not -
causing the timestamps to differ. This was simulated with an artificial
delay with `time.Sleep`.

---

Please refer to the [CONTRIBUTING.md](../CONTRIBUTING.md) file for more
information on how to contribute to this project.

---------

Signed-off-by: Kamil <[email protected]>
@kamilprz kamilprz closed this as completed Jan 2, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@kamilprz kamilprz

@ibezrukavyi ibezrukavyi

Labels
area/infra Test, Release, or CI Infrastructure
Projects
Retina Triage Board
Archived in project
Milestone
1.0
Development

No branches or pull requests
4 participants
@anubhabMajumdar @nddq @kamilprz @ibezrukavyi