Worker removes flow_run id if fails to submit.

[jeanluciano]

This PR fixes but where flow runs would be stuck in the AwaitingConcurrency if a concurreny limit was set at the deployment level. The bug can be summarized into:

• would be added to the worker's _submitting_flow_run_ids
• server changes the state to AwaitingConcurrency therefore making ready_to_submit=False
• _submit_scheduled_flow_runs never resubmits flow because it still finds it in _submitting_flow_run_ids
  closes:Deployment concurrency limit leaves jobs stuck in AwaitingConcurrencySlot #16027

Checklist

• This pull request references any related issue by including "closes <link to issue>"
  • If no issue exists and your change is not a small fix, please create an issue first.
• If this pull request adds new functionality, it includes unit tests that cover the changes
• If this pull request removes docs files, it includes redirect settings in mint.json.
• If this pull request adds functions or classes, it includes helpful docstrings.

[codspeed-hq]

CodSpeed Performance Report

Merging #16561 will not alter performance

_{Comparing jean/oss-5929-deployment-concurrency-limit-leaves-jobs-stuck-in (5ddc5a4) with main (660d1f3)}

Summary

✅ 3 untouched benchmarks

[desertaxle]

It'd be good to add a regression test in addition to the suggested changes to the log levels.

[desertaxle]

After looking at this more closely, since we're removing from _submitting_flow_run_ids in both branches, you simplify this by moving the removal outside of the if/else statement like this:

if ready_to_submit:
     ...
else:
     self._release_limit_slot(flow_run.id)
self._submitting_flow_run_ids.remove(flow_run.id)

[desertaxle]

LGTM!