Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: don't cache rejected promises #1238

Merged

Conversation

BeLi4L
Copy link
Contributor

@BeLi4L BeLi4L commented Dec 31, 2024

Current issue:
Sometimes our app would be stuck due to the following error in our logs: "Error: The request to Forest Admin server has timed out while trying to reach https://api.forestadmin.com/liana/v4/permissions/environment".
This would make our app unusable (since it can't reach Forest) and we had to restart the pod to make it work again.

Root cause:
The code responsible for calling the "https://api.forestadmin.com/liana/v4/permissions/environment" URL, in ActionPermissionService, uses a TTLCache, which keeps returning the same timed out promise.

Suggested solution:
Don't cache rejected promises, so that the following calls will retry until one succeeds.

Current issue:
Sometimes our pods would be stuck due to the following error in our logs: "Error: The request to Forest Admin server has timed out while trying to reach https://api.forestadmin.com/liana/v4/permissions/environment". This would make our app unusable and we had to restart the pod to make it work again.

Root cause:
The code responsible for calling the "https://api.forestadmin.com/liana/v4/permissions/environment" URL, in ActionPermissionService, uses a TTLCache, which keeps returning the timed out promise.

Suggested solution:
Don't cache rejected promises, so that the following calls will retry until it succeeds.
@BeLi4L
Copy link
Contributor Author

BeLi4L commented Dec 31, 2024

Another solution would have been to catch the "Time out" error only in ActionPermissionService, and invalidate the cache there, but I thought it would make sense to put this logic in TTLCache.

Let me know what you think, since I am not that familiar with this project 😉

@Scra3
Copy link
Member

Scra3 commented Dec 31, 2024

Hello @BeLi4L,
Thank you so much for your contribution! Your code seems to address a critical bug, and I truly appreciate your effort. I will present your contribution to the tech team in the coming weeks, as many of us are currently on holiday.
Once again, thank you for your help—it’s highly valued and greatly appreciated! 🙏

Copy link
Member

@jeffladiray jeffladiray left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks a lot for this bugfix - We've been on this issue for weeks now, not being able to reproduce it on our internal agent.

Manually tested by forcing utils/server.ts to reject via a simulated timeout

      if (
        !path.includes('ip') &&
        !path.includes('hashcheck') &&
        !path.includes('oidc') &&
        !path.includes('liana/v2/renderings') &&
        !path.includes('permissions/users')
      ) {
        const err = new Error();
        err.timeout = true;
        throw err;
      }

Cache is indeed cleared for the specific key on rejected promise 🙌

(I'm still curious to understand why these timeout appears. Default timeout is ~10s, and all of our monitoring systems seems to indicate pretty good response time (P99>200ms), especially on users/environments set of permissions 🤔)

The CI is failling because of unaccessible credentials from external contribution, so I'll bypass it. Thanks a lot 🙏

@jeffladiray jeffladiray merged commit 161459d into ForestAdmin:main Dec 31, 2024
20 of 21 checks passed
forest-bot added a commit that referenced this pull request Dec 31, 2024
@forest-bot
Copy link
Member

🎉 This PR is included in version 1.12.2 🎉

The release is available on [email protected]

Your semantic-release bot 📦🚀

@forest-bot
Copy link
Member

🎉 This PR is included in version 1.12.13 🎉

The release is available on npm package (@latest dist-tag)

Your semantic-release bot 📦🚀

@forest-bot
Copy link
Member

🎉 This PR is included in version 1.56.2 🎉

The release is available on npm package (@latest dist-tag)

Your semantic-release bot 📦🚀

@forest-bot
Copy link
Member

🎉 This PR is included in version 1.36.2 🎉

The release is available on npm package (@latest dist-tag)

Your semantic-release bot 📦🚀

@BeLi4L
Copy link
Contributor Author

BeLi4L commented Dec 31, 2024

(I'm still curious to understand why these timeout appears. Default timeout is ~10s, and all of our monitoring systems seems to indicate pretty good response time (P99>200ms), especially on users/environments set of permissions 🤔)

The TTL is an option given to TTLCache, and comes from the options-validator file:

// One year cache duration when using events
const DEFAULT_CACHE_DURATION_WITH_EVENTS = 31560000;

// [...]

// When using the event source to refresh cache we set a one year cache duration
copyOptions.permissionsCacheDurationInSeconds = copyOptions.instantCacheRefresh
  ? DEFAULT_CACHE_DURATION_WITH_EVENTS
  : copyOptions.permissionsCacheDurationInSeconds ?? DEFAULT_MINIMUM_CACHE_DURATION * 15;

I have no idea why this is done, especially since the instantCacheRefresh naming would hint the opposite 😅

@BeLi4L BeLi4L deleted the fix/dont-cache-rejected-promises branch December 31, 2024 15:36
@forest-bot
Copy link
Member

🎉 This PR is included in version 1.3.29 🎉

The release is available on npm package (@latest dist-tag)

Your semantic-release bot 📦🚀

forest-bot added a commit that referenced this pull request Jan 2, 2025
# [1.58.0](https://github.com/ForestAdmin/agent-nodejs/compare/@forestadmin/[email protected]...@forestadmin/[email protected]) (2025-01-02)

### Bug Fixes

* don't cache rejected promises ([#1238](#1238)) ([161459d](161459d))
* upgrade ssh-tunnel dependency to avoid crash when "no response from server" ([#1228](#1228)) ([83fbf9c](83fbf9c))

### Features

* **segment:** support templating for segment live query ([#1235](#1235)) ([5452a95](5452a95))
forest-bot added a commit that referenced this pull request Jan 2, 2025
## [1.1.44](https://github.com/ForestAdmin/agent-nodejs/compare/@forestadmin/[email protected]...@forestadmin/[email protected]) (2025-01-02)

### Bug Fixes

* don't cache rejected promises ([#1238](#1238)) ([161459d](161459d))
* upgrade ssh-tunnel dependency to avoid crash when "no response from server" ([#1228](#1228)) ([83fbf9c](83fbf9c))

### Features

* **segment:** support templating for segment live query ([#1235](#1235)) ([5452a95](5452a95))
forest-bot added a commit that referenced this pull request Jan 2, 2025
## [1.8.9](https://github.com/ForestAdmin/agent-nodejs/compare/@forestadmin/[email protected]...@forestadmin/[email protected]) (2025-01-02)

### Bug Fixes

* don't cache rejected promises ([#1238](#1238)) ([161459d](161459d))
* upgrade ssh-tunnel dependency to avoid crash when "no response from server" ([#1228](#1228)) ([83fbf9c](83fbf9c))

### Features

* **segment:** support templating for segment live query ([#1235](#1235)) ([5452a95](5452a95))
forest-bot added a commit that referenced this pull request Jan 2, 2025
## [1.5.9](https://github.com/ForestAdmin/agent-nodejs/compare/@forestadmin/[email protected]...@forestadmin/[email protected]) (2025-01-02)

### Bug Fixes

* don't cache rejected promises ([#1238](#1238)) ([161459d](161459d))
* upgrade ssh-tunnel dependency to avoid crash when "no response from server" ([#1228](#1228)) ([83fbf9c](83fbf9c))

### Features

* **segment:** support templating for segment live query ([#1235](#1235)) ([5452a95](5452a95))
forest-bot added a commit that referenced this pull request Jan 2, 2025
## [1.11.4](https://github.com/ForestAdmin/agent-nodejs/compare/@forestadmin/[email protected]...@forestadmin/[email protected]) (2025-01-02)

### Bug Fixes

* don't cache rejected promises ([#1238](#1238)) ([161459d](161459d))
* upgrade ssh-tunnel dependency to avoid crash when "no response from server" ([#1228](#1228)) ([83fbf9c](83fbf9c))

### Features

* **segment:** support templating for segment live query ([#1235](#1235)) ([5452a95](5452a95))
forest-bot added a commit that referenced this pull request Jan 2, 2025
## [1.16.4](https://github.com/ForestAdmin/agent-nodejs/compare/@forestadmin/[email protected]...@forestadmin/[email protected]) (2025-01-02)

### Bug Fixes

* don't cache rejected promises ([#1238](#1238)) ([161459d](161459d))
* upgrade ssh-tunnel dependency to avoid crash when "no response from server" ([#1228](#1228)) ([83fbf9c](83fbf9c))

### Features

* **segment:** support templating for segment live query ([#1235](#1235)) ([5452a95](5452a95))
forest-bot added a commit that referenced this pull request Jan 2, 2025
# [1.46.0](https://github.com/ForestAdmin/agent-nodejs/compare/@forestadmin/[email protected]...@forestadmin/[email protected]) (2025-01-02)

### Bug Fixes

* don't cache rejected promises ([#1238](#1238)) ([161459d](161459d))
* upgrade ssh-tunnel dependency to avoid crash when "no response from server" ([#1228](#1228)) ([83fbf9c](83fbf9c))

### Features

* **segment:** support templating for segment live query ([#1235](#1235)) ([5452a95](5452a95))
forest-bot added a commit that referenced this pull request Jan 2, 2025
## [1.4.12](https://github.com/ForestAdmin/agent-nodejs/compare/@forestadmin/[email protected]...@forestadmin/[email protected]) (2025-01-02)

### Bug Fixes

* don't cache rejected promises ([#1238](#1238)) ([161459d](161459d))
* upgrade ssh-tunnel dependency to avoid crash when "no response from server" ([#1228](#1228)) ([83fbf9c](83fbf9c))

### Features

* **segment:** support templating for segment live query ([#1235](#1235)) ([5452a95](5452a95))
forest-bot added a commit that referenced this pull request Jan 2, 2025
## [1.1.18](https://github.com/ForestAdmin/agent-nodejs/compare/@forestadmin/[email protected]...@forestadmin/[email protected]) (2025-01-02)

### Bug Fixes

* don't cache rejected promises ([#1238](#1238)) ([161459d](161459d))
* upgrade ssh-tunnel dependency to avoid crash when "no response from server" ([#1228](#1228)) ([83fbf9c](83fbf9c))

### Features

* **segment:** support templating for segment live query ([#1235](#1235)) ([5452a95](5452a95))
forest-bot added a commit that referenced this pull request Jan 2, 2025
## [1.4.2](https://github.com/ForestAdmin/agent-nodejs/compare/@forestadmin/[email protected]...@forestadmin/[email protected]) (2025-01-02)

### Bug Fixes

* don't cache rejected promises ([#1238](#1238)) ([161459d](161459d))
* upgrade ssh-tunnel dependency to avoid crash when "no response from server" ([#1228](#1228)) ([83fbf9c](83fbf9c))

### Features

* **segment:** support templating for segment live query ([#1235](#1235)) ([5452a95](5452a95))
@forest-bot
Copy link
Member

🎉 This PR is included in version 1.58.0 🎉

The release is available on npm package (@latest dist-tag)

Your semantic-release bot 📦🚀

@forest-bot
Copy link
Member

🎉 This PR is included in version 1.1.44 🎉

The release is available on npm package (@latest dist-tag)

Your semantic-release bot 📦🚀

@forest-bot
Copy link
Member

🎉 This PR is included in version 1.8.9 🎉

The release is available on npm package (@latest dist-tag)

Your semantic-release bot 📦🚀

@forest-bot
Copy link
Member

🎉 This PR is included in version 1.5.9 🎉

The release is available on npm package (@latest dist-tag)

Your semantic-release bot 📦🚀

@forest-bot
Copy link
Member

🎉 This PR is included in version 1.11.4 🎉

The release is available on npm package (@latest dist-tag)

Your semantic-release bot 📦🚀

@forest-bot
Copy link
Member

🎉 This PR is included in version 1.16.4 🎉

The release is available on npm package (@latest dist-tag)

Your semantic-release bot 📦🚀

@forest-bot
Copy link
Member

🎉 This PR is included in version 1.46.0 🎉

The release is available on npm package (@latest dist-tag)

Your semantic-release bot 📦🚀

@forest-bot
Copy link
Member

🎉 This PR is included in version 1.4.12 🎉

The release is available on npm package (@latest dist-tag)

Your semantic-release bot 📦🚀

@forest-bot
Copy link
Member

🎉 This PR is included in version 1.1.18 🎉

The release is available on npm package (@latest dist-tag)

Your semantic-release bot 📦🚀

@forest-bot
Copy link
Member

🎉 This PR is included in version 1.4.2 🎉

The release is available on npm package (@latest dist-tag)

Your semantic-release bot 📦🚀

BeLi4L pushed a commit to BeLi4L/forest-express that referenced this pull request Jan 8, 2025
Especially to have the TTLCache fix from ForestAdmin/agent-nodejs#1238
@BeLi4L
Copy link
Contributor Author

BeLi4L commented Jan 8, 2025

FYI we still had the issue in prod, since we use forest-express-sequelize, which uses forest-express, which still uses "forestadmin-client" "1.35.0.

Maybe it'd be worth it to auto-update that lib too, when forestadmin-client is updated 😉

EDIT: I did the update there, in this PR: ForestAdmin/forest-express#1045

BeLi4L pushed a commit to BeLi4L/forest-express-sequelize that referenced this pull request Jan 9, 2025
Especially to have the TTLCache fix from forestadmin-client: ForestAdmin/agent-nodejs#1238
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants