[Enhancement] Support replicating already replicated messages and multi-hop forwarding replication topologies

[lhotari]

Search before asking

• I searched in the issues and found nothing similar.

Motivation

Pulsar replication doesn't replicate already replicated messages:

pulsar/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/GeoPersistentReplicator.java

Lines 135 to 141 in 44f9860

	if (msg.isReplicated()) {
	// Discard messages that were already replicated into this region
	cursor.asyncDelete(entry.getPosition(), this, entry.getPosition());
	entry.release();
	msg.recycle();
	continue;
	}

Code was refactored in 3.0.0, this is the code in branch-2.11 (same behavior):

pulsar/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/PersistentReplicator.java

Lines 326 to 332 in ff3018a

	if (msg.isReplicated()) {
	// Discard messages that were already replicated into this region
	cursor.asyncDelete(entry.getPosition(), this, entry.getPosition());
	entry.release();
	msg.recycle();
	continue;
	}

It would be useful to have support for multi-hop replication in Pulsar clusters. It would be useful to have support for star (hub-and-spoke) topology as an alternative to full mesh point-to-point topology. Star topology is useful since instead of having every cluster connected with every other cluster, it's possible to configure geo-replication in a star topology where the number of connections is significantly reduced.

This might also have an impact on blue-green cluster migration. The concern is that already replicated messages would get skipped. This issue isn't confirmed.

Solution

Support multi-hop replication topologies where messages can be forwarded across multiple clusters.

Alternatives

No response

Anything else?

No response

Are you willing to submit a PR?

• I'm willing to submit a PR!

[lhotari]

@nodece @yyj8 I wonder if this is an issue with blue-green cluster migration?

[nodece]

Pulsar replication doesn't replicate already replicated messages.

This is correct. The Pulsar doesn't replicate the replicated message to the other cluster, this can suppress cyclic replication.
Such as that we have three cluster-a, cluster-b, and cluster-c, they are full-mesh.

A message-x is sent to cluster-a, which will replicate to the cluster-b and cluster-c from the cluster-a.

As far as I know, 2.10.x, 3.x, 4.x has the same behavior.

(hub-and-spoke) topology

Do you mean that select a broker as the main, and then use main broker replicates the message to the remote cluster?

I wonder if this is an issue with blue-green cluster migration?

Our project hasn't used the blue-green feature yet :(

[lhotari]

Do you mean that select a broker as the main, and then use main broker replicates the message to the remote cluster?

@nodece For example with clusters A, B, C and D. A would be the hub and there would be cluster connections A-B, B-A, A-C, C-A, A-D and D-A. In this type of star topology, there would be less connections than in a full mesh topology.
Messages produced to a topic in any cluster should eventually be available in all topics in all clusters.

[nodece]

This idea sounds very suitable for message replication, but we have a subscription replication feature. When using a star topology, subscription replication will be complex. I remember that subscribing to replication depends on the order of messages. What do you think?

[lhotari]

This idea sounds very suitable for message replication, but we have a subscription replication feature. When using a star topology, subscription replication will be complex. I remember that subscribing to replication depends on the order of messages. What do you think?

@nodece For subscription replication, it might not make sense to handle subscription replication in a star topology. The reason for this is that I cannot think of a use case where this would make sense. What would even be the expected behavior in that case?

Even in the current subscription replication, it is not supported to have active consumers in different clusters for the same subscription. There won't be errors if there are active consumers on both clusters, but the messages will get randomly delivered to either consumers or both consumers.