Billions of Events a Day Without Breaking the Bank: How Pex Avoided Cloud-Provider Lock-In and Saved Over 50% With Kesque Managed Cluster Plan

“The Kesque team is fantastic to work with, bottomline! They follow-through on their promises and are always looking at the bottom-line to help us save money, while still achieving our performance goals.”

David Southwell – Head of Infrastructure @ Pex

Highlights

  • Pex, a global music and video search engine, made a strategic decision to move its technology stack from GCP to Azure
  • In GCP they used PubSub as their event bus and had experienced initial implementation pains due to their requirement to process billions of events a day
  • The technology stack powering their search engine needed to stay up-and-running during the transition
  • Using a Managed Cluster plan from Kesque, Pex was able to smoothly transition from GCP to Azure
  • Compared to an equivalent locked-in cloud-provider solution, Pex was able to save over 50% in monthly charges for their eventing infrastructure

The Challenge

When Pex came to Kesque, they were looking to move the technology stack powering their global music and video search engine from GCP to Azure. Pex indexes and fingerprints every piece of audio and video content on over 30 social media platforms. Pex customers, including major music labels and film studios, then upload their audio or video content to find any uses of it online. Pex scans its database of 20B+ assets to find any matches within seconds. They needed to process billions of events a day and they knew it would be a challenge for any cloud service. In GCP they had been using PubSub and had experienced initial implementation pains because of their performance requirements. This time they needed to make a quick and smooth transition.

They were also wary of the lock-in created by using a cloud-provider specific solution. They preferred a multi-cloud solution backed by open source technology. And with a strong engineering team, they wanted to know what was going on inside the messaging technology at the core of their tech stack so they could optimize how they built applications on top of it.

The Solution

The Kesque Managed Cluster plan was the perfect solution for Pex. It consists of a custom-designed Apache Pulsar cluster running in the customer’s Kubernetes environment. The Kesque team provides management, 24/7 monitoring, and support but the customer has easy access to the cluster so they can integrate it with their own tooling.

Since the Managed Cluster plan is customizable, the Kesque team took the performance requirements from Pex and designed a Apache Pulsar cluster to meet their exact performance requirements. Kesque supports most cloud providers and is built on the open-source Apache Pulsar, so if the Pex team ever decides to change cloud providers, they can take their Managed Cluster plan with them. 

Once the Pex team provided the Kubernetes environment, the Kesque team remotely installed Apache Pulsar plus their suite of enterprise-grade monitoring and management tools. During the transition, the teams at Kesque and Pex worked closely together to optimize the performance of the Managed Cluster plan so that Pex was getting the best value possible.

The transition went off without a hitch. 

Today Pex’s Managed Cluster plan processes billions of events a day and it is tightly integrated into their technology stack. If something goes wrong with their Managed Cluster, they have the expert team at Kesque ready to jump in and fix it 24/7. And here’s perhaps the best part. Pex was able to meet their performance requirements, avoid cloud-vendor lock-in, and get excellent support while at the same time saving a significant amount of money. Compared to an equivalent set from the cloud provider (Event Hubs Dedicated, 2 Capacity Units, Professional Direct support) the Managed Cluster plan from Kesque is over 50% less per month! Billions of events a day, without breaking the bank.

PDF Version

Photo by Drew Graham on Unsplash


Want to check out Pulsar? Just sign up for the free plan of our managed Pulsar service give it a try. It only takes a minute to get started.