Cross-path acknowledgments should be congestion controlled

[michael-eriksson]

QUIC packets that only contain an ACK frame are not congestion controlled. RFC 9000 states that:

Since packets containing only ACK frames are not congestion controlled, an endpoint MUST NOT send more than one such packet in response to receiving an ack-eliciting packet.

When ACK-only packets are sent on another path than where the ack-eliciting packets were received, this requirement is obviously violated. A receiver of a big flow of packets could easily swamp a low-latency, low-bandwidth path by sending the acknowledgments there.

[huitema]

The requirement in RFC 9000 is not about congestion control. It is about not having an infinite cycle of sending an ACK, receiving an ACK in return, ACKing that, etc.

The issue that you describe is also happening in asymmetric links, without using multipath. If the return path is too narrow, it could be congested by too many ACKs. RFC 9000 is silent about that, largely because previous attempts at congestion control of acknowledgements have been "inconclusive". Since ACKs are not immediately acknowledged, the congestion control loop is very loose, and thus very imprecise. You can easily reach a state where congestion control slows the sending of ACKs, which in turn slows the sending of data, etc.

In uni-path scenarios, the best protection against swamping the return path is to control ACK Frequency, which limits the number of ACKs sent per RTT. The big problem there is that the ACK Frequency draft does not mention multipath. It is probably too late to change that draft, so maybe we should write a companion draft, "ACK Frequency for Multipath". Basically, define a PATH_ACK_FREQUENCY frame similar to ACK Frequency but with a Path ID.

[michael-eriksson]

Christian wrote:

The requirement in RFC 9000 is not about congestion control. It is about not having an infinite cycle of sending an ACK, receiving an ACK in return, ACKing that, etc.

Hmm, I think that the quote from RFC 9000 very explicitly mentions congestion control...

But I agree that congestion controlled ACKs can easily lead to infinite ack cycles.

[huitema]

The sentence that you quoted does mention congestion control. The discussion of infinite loops happens a couple paragraphs later -- An endpoint that is only sending non-ack-eliciting packets might choose to occasionally add an ack-eliciting frame to those packets to ensure that it receives an acknowledgment; see Section 13.2.4. In that case, an endpoint MUST NOT send an ack-eliciting frame in all packets that would otherwise be non-ack-eliciting, to avoid an infinite feedback loop of acknowledgments.

The spec does mention congestion control, as you said: Since packets containing only ACK frames are not congestion controlled, an endpoint MUST NOT send more than one such packet in response to receiving an ack-eliciting packet. That mean that if a node receives just one packet, it should send at most one ACK for that specific packet. It also implies that if you receive 2 or 3 packets, it is fine to send 2 or 3 ACKs. For example, if a node receives packets 1 and 2, it would be perfectly fine to send ACK 1-2 on path 1, and a copy of ACK 1-2 on path 2. Which would not address our problem, "avoid saturating a narrow return path".

The solution really is to send fewer ACKs overall, and that means adapting the ACK FREQUENCY draft for multipath.

[mirjak]

I agree that the problem discussed is not specific to a specific path and as such not specific to the multipath extension. I proposed to close the issue with no action.

The only thing we could do is to add a bit of discussion or a note of caution to not overload a path if one path is only used to send ACKs given they are not congestion control. But again this is effectively already a problem if you only have one path and send only ACKs in one direction. So not sure if even makes sense to explicitly say anything.

[mirjak]

@michael-eriksson can we close this issue?

[michael-eriksson]

Well, I don't think so. I can't see how an endpoint can be allowed to send (potentially large) non-congestion controlled traffic on another path than where packet being acknowledged was received.

Maybe we should have a quick question on the wg mailing list for more views on the issue?

[mirjak]

How is this different to the case where you send only ACKs in on direction on the same path?

[michael-eriksson]

1. Most paths are reasonably symmetrical, so there is at least a fraction of the bandwidth available in the other direction. If the paths are separate, one can be "infinitely" smaller than the other.
2. Some paths, like some wireless paths, vary in capacity over time. This will affect both directions, so they will scale (more or less) equally in bandwidth. If the ACKs are sent over a separate path, its capacity can deteriorate without any reduction in the amount of sent ACKs.

[mirjak]

These are areas for experimentation but the basic problem exists in both cases.

[huitema]

I propose closing this issue, because we do have great experience that ACK should not be congestion controlled in general -- whether on one path or another. We may want to open another issue regarding control of ACK frequency for multiple paths.

[mirjak]

Do you mean we should say something in this draft about ACK frequency or would that be a separate draft?

I think the only we thing we can say right now is that the ack frequency is negotiated for all path and if different ack frequencies are need for different path, that would require a new extension. However, the ack frequency after all is a recommendation for a max value and of course the sender could still use different (smaller) delay on each path.

I guess we could add that as a short subsection to the implementation section...?

I that is not something useful to say. I recommend we mark this is as "separate draft" and leave it open for later.

[huitema]

It absolutely should be a separate draft. I started working on a simple "PATH_ACK_FREQUENCY" draft, but quickly realized it is not that simple and that parallel "ships in the night" handling of each path will not necessarily work. For example, one of the motivation for ACK_FREQUENCY is to set a "budget of acknowledgements" so we send the right amount, don't cause CPU overload but also don't cause spurious timers. The "spurious timer" part could be set per path, because it is a function of the RTT of the path. But the "overload" part is a global setting. So, yes it is important, but No we should not try to do it as part of the main multipath draft.

[mirjak]

Do you want to say anything about it in this draft?

[michael-eriksson]

I agree that a separate draft is the right thing to do.

Until it is available, I am strongly against the current (intended Standards Track!) document allowing, or even requiring, the transmission of huge amounts of acknowledgments on paths with completely unknown capacity and load.

[huitema]

I am against adding text for that issue. We received strong feedback in Dublin that the draft was too researchy. If anything, we need to trim it to concentrate on presenting the mechanism and focusing on interop. Dropping half the text would be nice.

Sending tons of packets on a narrow link is stupid. But we have to assume that developers are not stupid, and we cannot add text for each of the many ways in which they can do stupid things.

[mirjak]

We received feedback by one person that wasn't following the discussion super closely. I don't think we have to change our working approach based on this. We knew in advance that there are many open questions and therefore decided to focus on the basic mechanisms to manage paths (and e.g. not include address exchange). I think this is still what the draft does in the normative parts, and I agree we should not invent new normative requirements now without clear evidence. However, for recommendation and known issues we also have the implementation guidance section. And I still think it is correct to mention those issues that we know about there and potentially even give recommendation in cases where we already have some deployment of testing experience. (Again we should do an editorial pass and streamlined the document where possible but I would like to do that after this coming revision when we are don't to basically 0 issues.)

For this issue, I agree that we don't know enough to do a normative change here from RFC9000. Limiting ACK is a new mechanism that we don't have any experience which, e.g. what would be the right limit? However, we could still mention the issue in the implication guidance section to ensure that all implementor are at least aware of this. More concretely we could add one more sentence at the end of section 6.4 on Packet Scheduling which already talks about ack strategies like this:

"As packets that only carries PATH_ACK frames are not congestion controlled, sending only PATH_ACK frames on a path should carefully consider the load induced by these packets, especially if the capacity is unknown on that path."

[michael-eriksson]

Yes, a caveat text like that is a reasonable stop gap measure until a separate document is ready.

However, the draft must also be changed to allow the transmission of PATH_ACK frames (but no other frames) on the same path also after having sent a PATH_ABANDON frame. The current draft forces an endpoint to send all acks of a full speed path on another path with unknown capacity (and even if it's known to be too narrow!).

[mirjak]

This would be a mayor design change as PATH_ABANDON is meant to mean I won't send anything anymore on this path. Also we have a normative requirement that explicitly disallows that in section 3.3.2 (https://quicwg.org/multipath/draft-ietf-quic-multipath.html#ack-after-abandon):

"These PATH_ACK frames MUST be sent on a different path than the path being abandoned."

However, while the doc says "endpoints SHOULD send PATH_ACK frames promptly" (note this is a SHOULD), this doesn't mean you have to send "full speed" (whatever that means...?). You can decide yourself how many and how quickly you send the acks and of course you need to take the path capacity into account. I'd say "promptly" means don't delay them unnecessarily but of course only send them when you can safely do so.

[michael-eriksson]

It is not a major change, it's a very small one.

Sending a PATH_ABANDON frame implies that you won't send any more data on that path. Sending in-path PATH_ACK frames (which of course are non-ack eliciting) is just for the peer's loss detection while the path is being shut down. The peer anyway has to be prepared to receive out-of-order packets from you after having received your PATH_ABANDON frame (especially if you send that on another path), and will trivially handle the acks that you send after the PATH_ABANDON without any implementation change!

With full speed I mean that the peer is sending data to you at the full available bandwidth of the link.

Quoting RFC 9000:

A receiver SHOULD send an ACK frame after receiving at least two ack-eliciting packets.

Most (all?) implementations of course follows this. Having separate ack transmission logic during path abandonment is a huge increase in complexity.

[mirjak]

PATH_ABANDON means that you don't send any new packets using the CID belong to the path ID indicated. This has nothing to do with data or not data. Changing that is a mayor change because current assumptions in path closure won't hold.

Acking every second packet is a fine strategy. If that leads to some ACK losses that is actually not a big problem. Usually the peer should stop sending you packets as soon as it receives the PATH_ABANDON so after one RTT anyway.

Also I would expect that it is rather unlikely that you send a PATH_ABANDON while the other end is currently sending full speed. E.g. you could first send a PATH_STANDBY and then later the PATH_ABANDON. Or if you do that, maybe also spending the effort of implementing a specific ack strategy for this case is worth it, especially if you have local knowledge about these other path. Nothing in the spec prevents you from optimising these corner cases. However, these are all the things where we don't know how this is used and we therefore cannot and should not say more.