-
Notifications
You must be signed in to change notification settings - Fork 8
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
fix: don't cache rejected promises #1238
fix: don't cache rejected promises #1238
Conversation
Current issue: Sometimes our pods would be stuck due to the following error in our logs: "Error: The request to Forest Admin server has timed out while trying to reach https://api.forestadmin.com/liana/v4/permissions/environment". This would make our app unusable and we had to restart the pod to make it work again. Root cause: The code responsible for calling the "https://api.forestadmin.com/liana/v4/permissions/environment" URL, in ActionPermissionService, uses a TTLCache, which keeps returning the timed out promise. Suggested solution: Don't cache rejected promises, so that the following calls will retry until it succeeds.
Another solution would have been to catch the "Time out" error only in ActionPermissionService, and invalidate the cache there, but I thought it would make sense to put this logic in TTLCache. Let me know what you think, since I am not that familiar with this project 😉 |
Hello @BeLi4L, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks a lot for this bugfix - We've been on this issue for weeks now, not being able to reproduce it on our internal agent.
Manually tested by forcing utils/server.ts
to reject via a simulated timeout
if (
!path.includes('ip') &&
!path.includes('hashcheck') &&
!path.includes('oidc') &&
!path.includes('liana/v2/renderings') &&
!path.includes('permissions/users')
) {
const err = new Error();
err.timeout = true;
throw err;
}
Cache is indeed cleared for the specific key on rejected promise 🙌
(I'm still curious to understand why these timeout appears. Default timeout is ~10s, and all of our monitoring systems seems to indicate pretty good response time (P99>200ms), especially on users/environments set of permissions 🤔)
The CI is failling because of unaccessible credentials from external contribution, so I'll bypass it. Thanks a lot 🙏
## [1.12.2](https://github.com/ForestAdmin/agent-nodejs/compare/[email protected]@1.12.2) (2024-12-31) ### Bug Fixes * don't cache rejected promises ([#1238](#1238)) ([161459d](161459d))
## [1.56.2](https://github.com/ForestAdmin/agent-nodejs/compare/@forestadmin/[email protected]...@forestadmin/[email protected]) (2024-12-31) ### Bug Fixes * don't cache rejected promises ([#1238](#1238)) ([161459d](161459d))
## [1.36.2](https://github.com/ForestAdmin/agent-nodejs/compare/@forestadmin/[email protected]...@forestadmin/[email protected]) (2024-12-31) ### Bug Fixes * don't cache rejected promises ([#1238](#1238)) ([161459d](161459d))
🎉 This PR is included in version 1.12.2 🎉 The release is available on Your semantic-release bot 📦🚀 |
🎉 This PR is included in version 1.12.13 🎉 The release is available on npm package (@latest dist-tag) Your semantic-release bot 📦🚀 |
🎉 This PR is included in version 1.56.2 🎉 The release is available on npm package (@latest dist-tag) Your semantic-release bot 📦🚀 |
🎉 This PR is included in version 1.36.2 🎉 The release is available on npm package (@latest dist-tag) Your semantic-release bot 📦🚀 |
The TTL is an option given to // One year cache duration when using events
const DEFAULT_CACHE_DURATION_WITH_EVENTS = 31560000;
// [...]
// When using the event source to refresh cache we set a one year cache duration
copyOptions.permissionsCacheDurationInSeconds = copyOptions.instantCacheRefresh
? DEFAULT_CACHE_DURATION_WITH_EVENTS
: copyOptions.permissionsCacheDurationInSeconds ?? DEFAULT_MINIMUM_CACHE_DURATION * 15; I have no idea why this is done, especially since the |
🎉 This PR is included in version 1.3.29 🎉 The release is available on npm package (@latest dist-tag) Your semantic-release bot 📦🚀 |
# [1.58.0](https://github.com/ForestAdmin/agent-nodejs/compare/@forestadmin/[email protected]...@forestadmin/[email protected]) (2025-01-02) ### Bug Fixes * don't cache rejected promises ([#1238](#1238)) ([161459d](161459d)) * upgrade ssh-tunnel dependency to avoid crash when "no response from server" ([#1228](#1228)) ([83fbf9c](83fbf9c)) ### Features * **segment:** support templating for segment live query ([#1235](#1235)) ([5452a95](5452a95))
## [1.1.44](https://github.com/ForestAdmin/agent-nodejs/compare/@forestadmin/[email protected]...@forestadmin/[email protected]) (2025-01-02) ### Bug Fixes * don't cache rejected promises ([#1238](#1238)) ([161459d](161459d)) * upgrade ssh-tunnel dependency to avoid crash when "no response from server" ([#1228](#1228)) ([83fbf9c](83fbf9c)) ### Features * **segment:** support templating for segment live query ([#1235](#1235)) ([5452a95](5452a95))
## [1.8.9](https://github.com/ForestAdmin/agent-nodejs/compare/@forestadmin/[email protected]...@forestadmin/[email protected]) (2025-01-02) ### Bug Fixes * don't cache rejected promises ([#1238](#1238)) ([161459d](161459d)) * upgrade ssh-tunnel dependency to avoid crash when "no response from server" ([#1228](#1228)) ([83fbf9c](83fbf9c)) ### Features * **segment:** support templating for segment live query ([#1235](#1235)) ([5452a95](5452a95))
## [1.5.9](https://github.com/ForestAdmin/agent-nodejs/compare/@forestadmin/[email protected]...@forestadmin/[email protected]) (2025-01-02) ### Bug Fixes * don't cache rejected promises ([#1238](#1238)) ([161459d](161459d)) * upgrade ssh-tunnel dependency to avoid crash when "no response from server" ([#1228](#1228)) ([83fbf9c](83fbf9c)) ### Features * **segment:** support templating for segment live query ([#1235](#1235)) ([5452a95](5452a95))
## [1.11.4](https://github.com/ForestAdmin/agent-nodejs/compare/@forestadmin/[email protected]...@forestadmin/[email protected]) (2025-01-02) ### Bug Fixes * don't cache rejected promises ([#1238](#1238)) ([161459d](161459d)) * upgrade ssh-tunnel dependency to avoid crash when "no response from server" ([#1228](#1228)) ([83fbf9c](83fbf9c)) ### Features * **segment:** support templating for segment live query ([#1235](#1235)) ([5452a95](5452a95))
## [1.16.4](https://github.com/ForestAdmin/agent-nodejs/compare/@forestadmin/[email protected]...@forestadmin/[email protected]) (2025-01-02) ### Bug Fixes * don't cache rejected promises ([#1238](#1238)) ([161459d](161459d)) * upgrade ssh-tunnel dependency to avoid crash when "no response from server" ([#1228](#1228)) ([83fbf9c](83fbf9c)) ### Features * **segment:** support templating for segment live query ([#1235](#1235)) ([5452a95](5452a95))
# [1.46.0](https://github.com/ForestAdmin/agent-nodejs/compare/@forestadmin/[email protected]...@forestadmin/[email protected]) (2025-01-02) ### Bug Fixes * don't cache rejected promises ([#1238](#1238)) ([161459d](161459d)) * upgrade ssh-tunnel dependency to avoid crash when "no response from server" ([#1228](#1228)) ([83fbf9c](83fbf9c)) ### Features * **segment:** support templating for segment live query ([#1235](#1235)) ([5452a95](5452a95))
## [1.4.12](https://github.com/ForestAdmin/agent-nodejs/compare/@forestadmin/[email protected]...@forestadmin/[email protected]) (2025-01-02) ### Bug Fixes * don't cache rejected promises ([#1238](#1238)) ([161459d](161459d)) * upgrade ssh-tunnel dependency to avoid crash when "no response from server" ([#1228](#1228)) ([83fbf9c](83fbf9c)) ### Features * **segment:** support templating for segment live query ([#1235](#1235)) ([5452a95](5452a95))
## [1.1.18](https://github.com/ForestAdmin/agent-nodejs/compare/@forestadmin/[email protected]...@forestadmin/[email protected]) (2025-01-02) ### Bug Fixes * don't cache rejected promises ([#1238](#1238)) ([161459d](161459d)) * upgrade ssh-tunnel dependency to avoid crash when "no response from server" ([#1228](#1228)) ([83fbf9c](83fbf9c)) ### Features * **segment:** support templating for segment live query ([#1235](#1235)) ([5452a95](5452a95))
## [1.4.2](https://github.com/ForestAdmin/agent-nodejs/compare/@forestadmin/[email protected]...@forestadmin/[email protected]) (2025-01-02) ### Bug Fixes * don't cache rejected promises ([#1238](#1238)) ([161459d](161459d)) * upgrade ssh-tunnel dependency to avoid crash when "no response from server" ([#1228](#1228)) ([83fbf9c](83fbf9c)) ### Features * **segment:** support templating for segment live query ([#1235](#1235)) ([5452a95](5452a95))
🎉 This PR is included in version 1.58.0 🎉 The release is available on npm package (@latest dist-tag) Your semantic-release bot 📦🚀 |
🎉 This PR is included in version 1.1.44 🎉 The release is available on npm package (@latest dist-tag) Your semantic-release bot 📦🚀 |
🎉 This PR is included in version 1.8.9 🎉 The release is available on npm package (@latest dist-tag) Your semantic-release bot 📦🚀 |
🎉 This PR is included in version 1.5.9 🎉 The release is available on npm package (@latest dist-tag) Your semantic-release bot 📦🚀 |
🎉 This PR is included in version 1.11.4 🎉 The release is available on npm package (@latest dist-tag) Your semantic-release bot 📦🚀 |
🎉 This PR is included in version 1.16.4 🎉 The release is available on npm package (@latest dist-tag) Your semantic-release bot 📦🚀 |
🎉 This PR is included in version 1.46.0 🎉 The release is available on npm package (@latest dist-tag) Your semantic-release bot 📦🚀 |
🎉 This PR is included in version 1.4.12 🎉 The release is available on npm package (@latest dist-tag) Your semantic-release bot 📦🚀 |
🎉 This PR is included in version 1.1.18 🎉 The release is available on npm package (@latest dist-tag) Your semantic-release bot 📦🚀 |
🎉 This PR is included in version 1.4.2 🎉 The release is available on npm package (@latest dist-tag) Your semantic-release bot 📦🚀 |
Especially to have the TTLCache fix from ForestAdmin/agent-nodejs#1238
FYI we still had the issue in prod, since we use Maybe it'd be worth it to auto-update that lib too, when forestadmin-client is updated 😉 EDIT: I did the update there, in this PR: ForestAdmin/forest-express#1045 |
Especially to have the TTLCache fix from forestadmin-client: ForestAdmin/agent-nodejs#1238
Current issue:
Sometimes our app would be stuck due to the following error in our logs:
"Error: The request to Forest Admin server has timed out while trying to reach https://api.forestadmin.com/liana/v4/permissions/environment"
.This would make our app unusable (since it can't reach Forest) and we had to restart the pod to make it work again.
Root cause:
The code responsible for calling the "https://api.forestadmin.com/liana/v4/permissions/environment" URL, in ActionPermissionService, uses a TTLCache, which keeps returning the same timed out promise.
Suggested solution:
Don't cache rejected promises, so that the following calls will retry until one succeeds.