Respect dispatch concurrency limits for clusterdispatch

[sashayakovtseva]

Fixes #1641

[github-actions]

CLA Assistant Lite bot All contributors have signed the CLA ✍️ ✅

[sashayakovtseva]

I have read the CLA Document and I hereby sign the CLA

[jzelinskie]

I believe @vroldanbet might have discovered an issue with this approach. I'll let him chime in whenever he's available (it is the holiday season).

[sashayakovtseva]

Hello,

any news here? what issue have you discovered?

[vroldanbet]

Hey @sashayakovtseva, my apologies for not getting back to this earlier.

There isn't so much of an issue but a concern around the impact of this change. While the unbounded dispatch configuration is not great for the health of the cluster, it does as well fundamentally change the baseline throughput of existing clusters, which may see a regression when this change is deployed in a non-trivial way, until they figure out that this change caps the dispatch concurrency.

I don't have an alternative, to be honest, other than rolling this out and observing the actual impact in practice, and tuning accordingly when required.

[sashayakovtseva]

While the unbounded dispatch configuration is not great for the health of the cluster, it does as well fundamentally change the baseline throughput of existing clusters

I should mention that currently dispatch is bounded with the default value of 50. So unless one doesn't explicitly set dispatch limit < 50 no regression should be observed