Migrate from Apache Kafka to Apache Pulsar — without Coding

Originally published on Medium by Andreas Müller, a well-known messaging expert, creator of SwiftMQ, CEO of swiftmq.com and CTO of flowdirector.io.

I’m not to say that one of these two messaging systems is better than the other. It always depends on the use case. Both compete in the same high-volume event streaming space where Kafka dominates by far. Pulsar is a challenger, and we see small but increasing movements from Kafka to Pulsar.

This article describes a smart way to migrate without writing a single line of code.

Migration Example

Say, you have everything running under Kafka but a new invoicing system that should run under Pulsar and needs the messages from the Kafka billing topic. Your new invoicing clients should receive it from a Pulsar topic because they are connected to that broker. It would be best if you bridge these topics. After your new invoicing works, you can switch off the old one.

Because I’m too lazy to install everything on-premise, I use the cloud for a demonstration. I need:

  • A Kafka cloud provider.
  • A Pulsar cloud provider.
  • An integration cloud provider for the bridging.

Confluent Cloud for Kafka

Confluent Cloud provides managed Kafka clusters. They charge by volume, so it’s cheap for a demonstration but not free. I need to create an account, a cluster, and a billing topic. Then I create and save access credentials and the Kafka client configuration, which I can collect from my account.

Kesque Cloud for Pulsar

Kesque Cloud provides managed Pulsar clusters. They offer a free account that’s suitable for a demonstration. As with Confluent Cloud, I need to create an account, a cluster, and a billing topic. Then I create and save the access token, server- and topic URL.

Flow Director Cloud for the Bridging

Flow Director is a flow-based no-code messaging platform that integrates any messaging system, so Kafka and Pulsar. It provides pre-defined apps for different use cases. One of them is the free Kafka to Pulsar Migration app (look at the “Integration” tab under this link). All I have to do is to install it from the integrated repository.

Flow Director Cloud provides Flow Director as a managed cloud instance. There is a free seven days trial and perfect for a demonstration. I need to create an account, launch the cloud instance, and install the Kafka to Pulsar Migration app from the integrated repository (just one click).

Create the Topic Bridge

The bridge is configured and runs in Flow Director. Look here for detailed documentation. I will concentrate on the most relevant parts.

Flow Director is flow-based. You graphically connect components and have a flow which is then executed. The Kafka to Pulsar Migration app provides three so-called template flows. They are used as a template to create a new flow.

Template Flows

Only the last one — TopicBridge — is of interest. The other two are for testing.

The TopicBridge template flow is copied and saved under TopicBridgeBilling:

Image for post
Copy

I then fill the flow parameters of the new flow with the credentials and connection data saved from Confluent and Kesque:

Flow Parameters
Flow Parameters

That’s it. I only need to activate the flow and have my billing bridge:

Activate the flow.

Monitor the Topic Bridge

To have a live view of my bridge, I use the View option:

Viewing a flow

Here I can see the number of messages traveling between components and the messages itself if I click on a link:

Image for post

Getting Statistics of the Bridge

All flows already have dashboard components attached that produce statistic data streams which can be visualized in dashboards. Creating a dashboard is easy but I skip this here (look at the detailed documentation, please).

The resulting dashboard I created displays an area chart with the messages transferred per minute:

Messages bridged per minute

Summary

Migration from one messaging system to another is not a big deal in today’s world. You neither have to be a Kafka nor a Pulsar specialist. Everything can be done with cloud providers but the same works with on-premise installations. Usually, a migration ends when everything works on the new system, so it’s time-limited. Therefore, it’s not a bad idea to use an integration cloud provider such as Flow Director Cloud that you can cancel when the job is done.