Kafka Replication

🌍 Kafka in Multi-Region (Active-Active or Active-Passive) Setup

🔁 Data Replication Flow: Region A ↔ Region B

1. Active-Passive Setup (One-Way Replication)

▶ Use Case:

Region A is active, Region B is backup (DR).

▶ Flow:

1. Producers write to Region A Kafka cluster.

2. MirrorMaker 2 replicates data to Region B.

3. Consumers in Region B are inactive until failover.

Benefits:

• Simpler, more consistent

• Region B can take over if Region A fails (Disaster Recovery)

2. Active-Active Setup (Bi-Directional Replication)

▶ Use Case:

Both Region A and Region B actively handle traffic (e.g., global users).

▶ Flow:

1. Producers in Region A write to Kafka A; producers in Region B write to Kafka B.

2. MirrorMaker 2 (or Confluent Replicator) replicates:

   • Region A → Region B

   • Region B → Region A

3. You must prefix topics to avoid cyclic replication:

   • regionA.orders, regionB.orders

Important Caveats:

• Avoid conflict: Same topic names can cause infinite loops.

• Use deduplicators, topic renaming, or routing rules.

• Maintain global IDs (e.g., UUIDs) to track message uniqueness.

⚙️ Example Kafka Replication Topology

pgsql

+------------------------+

| Kafka Cluster A |

| (Azure - East US) |

+------------------------+

| |

MirrorMaker2 MirrorMaker2

(East → West) (West → East)

| |

+------------------------+

| Kafka Cluster B |

| (Azure - West Europe) |

+------------------------+

🔐 Networking and Security Between Regions

• VNET Peering or ExpressRoute for fast & secure transfer

• TLS Encryption, SASL Auth, ACLs

• Can use Azure Private Link or VPN Gateway

✅ Tips for Successful Replication

Challenge	Solution
Message duplication	Use message deduplication logic via UUIDs
Topic conflicts	Use prefixes or renaming strategy
Replication lag	Monitor with Prometheus/Grafana
Network latency	Use regional edge zones or ExpressRoute
Failover complexity	Implement external DNS + Traffic Manager