Skip to content

Commit

Permalink
feat: autorecover from stuck situations
Browse files Browse the repository at this point in the history
Signed-off-by: Gernot Feichter <[email protected]>
  • Loading branch information
gerrnot committed Aug 6, 2024
1 parent afa9a48 commit bbaa513
Showing 1 changed file with 16 additions and 5 deletions.
21 changes: 16 additions & 5 deletions hips/hip-9999.md
Original file line number Diff line number Diff line change
Expand Up @@ -44,10 +44,14 @@ After implementation, the --timout parameter will be stored in the helm release
have an indirect impact on possible parallel processes.

`helm ls -a` shows two new columns, regular `helm ls` does NOT show those:
- LOCKED TILL
<datetime> calculated by the helm client: k8s server time + timeout parameter value
- SESSION ID
Unique, random session id generated by the client
- `LOCKED TILL`
datetime calculated by the helm client: current time + timeout parameter value
Originally k8s server time was intended as "current time", but since helm exclusively uses the
client time everywhere else, we do not change that via this HIP, such a refactoring would need
to be performed via a separate HIP against the entire codebase.
- `SESSION ID`

Unique, random session id generated by the client.

Furthermore, if the helm client process gets killed (SIGTERM), it tries to clear the LOCKED TILL value,
SESSION ID and sets the release into a failed state before terminating in order to free the lock.
Expand Down Expand Up @@ -88,7 +92,14 @@ None
- [ ] HIP status `accepted'
- [x] Reference implementation
- [x] Test for concurrent upgrade (valid lock should still block concurrent upgrade attempts)
- [ ] Test for kill scenario (forever stuck in pending)
- [x] Test for upgrading from pending state
- [x] Test for upgrading from failed state
- [ ] Decision: Helm ls -> which flag should show the new fields `LOCKED TILL` and `SESSION ID`?
- [ ] Decision: k8s Lease object vs helm relesae secret for storing the `LOCKED TILL` and `SESSION ID`
- [ ] Backwards compatibility check (looking good already)

## References

https://github.com/helm/helm/issues/7476
https://github.com/rancher/rancher/issues/44530
https://github.com/helm/helm/issues/11863

0 comments on commit bbaa513

Please sign in to comment.