Support jitter in client reconnect logic

[KirkMartinez]

We have about 450 clients which all disconnected simultaneously when our Jenkins master went down.
When we brought it back up, most workers failed to connect due to the thundering herd. We could see from logs that most workers were trying to reconnect at the same time. This feature should alleviate that problem. It adds a new cmdline switch for the swarm client -useJitter (default: off) which uses a retryInterval randomly selected between zero and the retryInterval (as opposed to just using the retryInterval directly). The change still respects the absolute maximum (set by maxRetryInterval) and the selected backoff strategy. Happy to modify this if you have any concerns.

[KirkMartinez]

@basil can you help me get the right people to review this PR? Sorry to @ mention you directly, but there are no OWNERS or CONTRIB for this repo and I'm not sure how to get help. If there is a better way, please let me know and I'll be happy to comply.

[basil]

@basil can you help me get the right people to review this PR? Sorry to @ mention you directly, but there are no OWNERS or CONTRIB for this repo and I'm not sure how to get help. If there is a better way, please let me know and I'll be happy to comply.

Hey Kirk, thanks for your contribution! I'd love to have this be part of the next release. It's unfortunately a bit of a busy time for me, but I will try to review this sometime next week.

[justinseanmartin]

Bump on this, would be really nice to avoid the thundering herd that not staggering these causes.

[basil]

Hi @KirkMartinez, sorry for the delay in reviewing this. I just took a look now, and while overall the change looks fine, I wanted to suggest a possible alternative.

The current change adds a boolean flag to enable jitter. When set, we use a reconnect interval between zero and retryInterval. One flaw with this scheme is that it doesn't allow for the minimum retry interval to be controlled.

May I suggest making the jitter an integer rather than a boolean, and adding it to the value of retryInterval instead of overriding the retryInterval? This way, the user can control the minimum retry interval (by setting retryInterval) and can also control the amount of jitter (by setting the jitter interval). In other words, the actual retry interval would be retryInterval + rand(0, jitterInterval). This should avoid thundering herds while also giving the user control over the minimum interval.

What do you think?

[moshavnik]

I'm a heavy user of the swarm plugin and would love to see this feature merged.
Is there a maintainer who can perform the merge and release the new version?

thanks :)