Try to reproduce #19179

[serathius]

Please read https://github.com/etcd-io/etcd/blob/main/CONTRIBUTING.md#contribution-flow.
#19179

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: serathius

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [serathius]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 68.78%. Comparing base (9eb85ee) to head (54813cf).
Report is 29 commits behind head on main.

Additional details and impacted files

see 26 files with indirect coverage changes

@@            Coverage Diff             @@
##             main   #19191      +/-   ##
==========================================
+ Coverage   68.77%   68.78%   +0.01%     
==========================================
  Files         420      420              
  Lines       35641    35629      -12     
==========================================
- Hits        24511    24507       -4     
+ Misses       9707     9700       -7     
+ Partials     1423     1422       -1     

---

Continue to review full report in Codecov by Sentry.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 9eb85ee...54813cf. Read the comment docs.

[serathius]

cc @fuweid any recommendation on how to reproduce the #19179

[fuweid]

Hi @serathius

I can't reproduce it in my local. But I used to etcd-dump-logs to get one thing in first three failure pipeline runs

https://prow.k8s.io/view/gs/kubernetes-ci-logs/pr-logs/directory/pull-etcd-robustness-amd64/1877585036438409216
https://prow.k8s.io/view/gs/kubernetes-ci-logs/pr-logs/directory/pull-etcd-robustness-arm64/1877364764741472256
https://prow.k8s.io/view/gs/kubernetes-ci-logs/pr-logs/directory/pull-etcd-robustness-arm64/1877466683589791744

There were only two kinds of requests before compaction panic. One is creation and other is deletion.
Even if Update type has high weight during random pick, there is no update type in first 400+ requests.
So, creation revision has been deleted in first compaction batch before panic. After restart, etcd key index ignores keys which has only one revision for tombstone. And then etcd resumes compaction, there are no such key in index, compaction will delete all the tombstones. However, it should keep these tombstones.

I am still investigating why there is no update before compaction panic. Hope that information helps

[k8s-ci-robot]

@serathius: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
pull-etcd-verify	54813cf	link	true	/test pull-etcd-verify

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

[serathius]

There were only two kinds of requests before compaction panic. One is creation and other is deletion.

I think I can guess the reason, in Kubernetes traffic operations are based on local state that is feed from watch. It tries to balance number of objects within the storage to keep the average. If the watch was very delayed there would be a long time where the traffic would execute only creates as it would not be aware of any objects. When watch would caught up, the traffic would skip random operations and immediately go for deletions, as there were too many objects in storage.

Still no progress on reproduction.

[k8s-ci-robot]

PR needs rebase.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.