Prometheus Metrics Now Supported On All Plans

We have great news to share. The team has been working on a series of new features that we will be blogging about in the next few weeks. Our customers are giving us great feedback on what new capabilities they want to see and we are working around the clock to make them happen. Please let us know what you think

The first exciting new feature that we would like to share with you is Kesque’s support for tenant-based Prometheus metrics. The fact that Prometheus metrics are available on the Starter plan means you can do your POC including the metrics part without blowing your budget. Fully tested and battle-ready, it is up for primetime – several of our customers are already using it, in fact. Kesque’s support for Prometheus couldn’t be any easier and hassle-free. All you need to do is to scrape your tenant’s existing Prometheus metrics and voila, you are done.

Here’s an example scrape config for Prometheus:

- job_name: kesque-service
  honor_timestamps: true
  scrape_interval: 30s
  scrape_timeout: 10s
  metrics_path: /pulsarmetrics
  scheme: https
  static_configs:
  - targets:
    - ueast2.aws.kafkaesque.io:8964
  bearer_token: <token>

Yep, that simple. 

As you are well aware, Prometheus is used for monitoring and metrics collection. Utilizing an HTTP pull model, it stores real-time metrics in a time series database. Prometheus users can also take advantage of their flexible queries and real-time alerting. Although Apache Pulsar supports Prometheus, it is a distributed system so you need to make sure you are collecting metrics from all the components. This is where Kesque continues to add value. We give you a single, simple endpoint to scrape to get all your Pulsar metrics. This is part of our goal to make things hassle-free for our customers to use, so they can focus on running their business instead of wasting time and resources on managing their data pipelines.

Since we have you, let’s talk about a great use case for Prometheus metrics with Pulsar. Let’s say you want to scale up or down the number of consumers (workers) on a shared subscription depending on the message backlog in that subscription. By collecting the subscription backlog metrics using Prometheus you can integrate with an auto-scaling group that will add or remove resources based on how deep the backlog is. If the backlog is getting too deep, the auto-scaler can spin up more resources. If the backlog is shallow, you can scale down resources to save money.

For example, you could export the Pulsar metrics to AWS CloudWatch and use the custom metrics to control an AWS Auto Scaling Group. Or if you are using Kubernetes, you can connect Prometheus to a Horizontal Pod Autoscaler and use the Pulsar custom metrics to spin pods up and down. There are lots of options, depending on how and where you run your applications that connect to Pulsar.

Prometheus metrics add-on is an extra $49/month that can be added to the Starter Plan and above. Just in case you didn’t know, Kesque is a self-served and fully managed service. We have a free tier you can start with and upgrade your package based on your needs.

Featured image by Igor Lepilin on Unsplash


Want to check out Pulsar? Just sign up for the free plan of our managed Pulsar service give it a try. It only takes a minute to get started.