As promised, we continue blogging about a series of new features and capabilities we have been working on lately. For first time readers, here’s a little blurb about us. Kesque is a fully managed data messaging service. We started with Apache Pulsar and built quite a bit of technology and tools around it. With these additions and modifications, we offer a true enterprise-grade data messaging service, as companies have come to expect. Think Kafka, but a lot better.
Speaking of tools that Kesque offers in addition to Pulsar, as you may know, we offer a robust dashboard where customers can easily utilize and manage Pulsar features and more. We are excited to announce that Kesque Dashboard now supports data sources and sinks.
What are Pulsar Sources and Sinks?
Pulsar Sources feed data to Pulsar from other external sources. Typical examples include Debezium source, which tracks changes to databases and feeds them to Pulsar (change data capture or CDC). Think of it as data coming into Pulsar. For data going out of Pulsar there are sinks. Typical sinks can be SQL or NO-SQL database.
Sources and Sinks are key components in the Pulsar IO data streaming pipeline. These battle tested components enable rapid set up for real time data streaming pipelines.
In the screenshot below we show how easy it is to add a Debezium Postgresql source in the Kesque dashboard. Just fill in the configuration details and click on the Add button:
Debezium sink receives database update messages to stream to a Pulsar topic for further processing, analytic, or monitoring purposes. Pulsar IO has built Debezium sinks for Postgresql, MySQL, and MongoDB.
A good example of a sink is a JDBC sink. A JDBC sink is able to push Pulsar messages from one or more topics directly to rows of a table in a JDBC compatible SQL database. Two popular supported databases are Postgresql and MySQL.
Here is what a running JDBC sink looks like in the Kesque dashboard:
From this screen, you can take a look at the topics the sink is reading from, view detailed stats on the sink instances, and view the logs from the running sink.
If you haven’t seen our dashboard yet, there’s no better way to understand what it can do for you than signing up for our free tier. If you like what you see and need more features and/or higher data rate needs, you can easily upgrade to higher-tier packages right from the dashboard. Most of our packages and features are self-served and there’s no long-term commitment. We run our service in AWS, GC, and Azure, whichever you prefer.
Featured image by Bruce Warrington on Unsplash.