Unexpected behavior against misbehaving Observatorium

[esnible]

I had misconfigured my Observatorium with an invalid client ID and OIDC secret in my K8s observatorium-xyz-observatorium-api Secret, while following https://github.com/observatorium/observatorium/blob/main/docs/usage/getting-started.md

token-refresher did not work, as expected, but it was very hard to troubleshoot. Hundreds of pages of token-refresher logs, mostly http: proxy error: dial tcp 127.0.0.1:8443: socket: too many open files.. No Observatorium-api logs.

I had no idea what was going on. It looks like net/http/httputil/reverseproxy.go closes files under defer. The incredible amount of output mislead me to suspect a recursion problem.

Perhaps more logging or rate limiting around rejection would help.