The world’s leading publication for data science, AI, and ML professionals.

Kafka UI Monitoring Tools (2021 Update)

Exploring some of the most powerful UI monitoring tools for Apache Kafka clusters

Photo by Chris Liverani on Unsplash
Photo by Chris Liverani on Unsplash

Introduction

Apache Kafka is amongst the fastest growing products out there that has been widely adopted by many companies across the globe. If you are using Kafka in production, is very important to be able to monitor and manage the cluster.

This article contains an updated list of the most popular and powerful monitoring tools for Apache Kafka clusters. You can also find an older list of recommendations that I proposed in an article written a few years back.

Overview of UI Monitoring Tools for Apache Kafka Clusters

Specifically, in this article we will cover the following tools and services that can help you manage and monitor the health of Kafka clusters across organisations:

  • Confluent Control Center and Health+
  • Lenses
  • Xinfra Monitor (formerly Kafka Monitor)
  • Cruise Control
  • CMAK (formerly Kafka Manager)

Confluent Control Center and Health+

Source: Confluent Website
Source: Confluent Website

The Confluent Control Center is a web-based tool that offers a user interface that facilitates cluster monitoring and administration. Specifically, you can quickly access an overview of the cluster health, access and observe messages, topics and Schema Registry services as well as executing ksql queries.

Technically, the tool is compromised by components that collect metric data on producers and consumers, Kafka that is used to move the collected metrics and the Control Center application server that is used to analyse stream metrics.


Confluent Health + is another tool that can be used to to manage and monitor the Confluent Platform (it was actually introduced in Confluent Platform version 6.2).

The Health+ Monitoring dashboard in Confluent Control Center - Source: Confluent Website
The Health+ Monitoring dashboard in Confluent Control Center – Source: Confluent Website

Specifically, Health + can help you configure intelligent and rule-based alerts for which you can receive notifications on specific endpoints such as email or Slack. Additionally, you can view all the collected metrics as shown in the figure above as well as integrate Health+ with other monitoring tools such as Prometheus.


Lenses

Lenses offers a complete Kafka UI Tool that enables developers and engineers monitor the health Kafka infrastructure as well as application performance. Additionally, the tool can be used to configure alerts at real-time which facilitate incident management in an effective manner.

Lenses Monitoring Tool - Source: Lenses website
Lenses Monitoring Tool – Source: Lenses website

Lenses Tool also allows troubleshooting through an intuitive UI that lets you query Kafka using SQL.

Source: Lenses website
Source: Lenses website

You can see more details regarding Lenses’ pricing in their official website.


Xinfra Monitor (formerly Kafka Monitor)

Xinfra Monitor is an open-source tool developed by LinkedIn that is used to configure and execute long-running system tests over Kafka clusters. It helps developers and administrators capture bugs or regressions that are typically observed rarely or only after prolonged period of time.

Additionally, it helps you monitor the cluster using end-to-end pipelines that collect metrics such as end-to-end latency, service availability, producers and consumers availability, message loss rate and many more.


Cruise Control

Cruise Control, is another LinkedIn open-source tool that helps run Kafka clusters at large scale and is designed to address scalability issues such as broker death and workload balance.

Kafka Cluster Status - Source: LinkedIn Engineering Blog
Kafka Cluster Status – Source: LinkedIn Engineering Blog

Specifically, the tool allows you to monitor and track resource utilisation, query the cluster state and observe partitions, replicas and distribution. Additionally, you can monitor anomaly detection and configure alerts and self-healing processes as well as re-balancing.

Furthermore, you can also perform traditional administration such as adding, or removing brokers, adjusting replication factor, rebalancing the cluster, fixing offline replicas and triggering preferred leader election.

Kafka cluster load as calculated by Cruise Control - Source: LinkedIn Engineering Blog
Kafka cluster load as calculated by Cruise Control – Source: LinkedIn Engineering Blog

CMAK (formerly Kafka Manager)

CMAK (Cluster Manager for Apache Kafka) is an open-source tool that helps you manage Kafka clusters.

Specifically, the tool can help you manage various cluster, which is quite convenient if -say- you want to monitor clusters in different environments as shown below.

Cluster Management in CMAK - Source: GitHub
Cluster Management in CMAK – Source: GitHub

Additionally, CMAK can help you inspect the state of a cluster, including topics, consumers, offsets, brokers, replicas and partition distribution.

Topic List in CMAK - Source: GitHub
Topic List in CMAK – Source: GitHub

Furthermore, the tool can help you delete topics, reassign partitions, adjust partition number, update topic configuration and perform manual partition assignment through the interactive user interface.

Topic View in CMAK - Source: GitHub
Topic View in CMAK – Source: GitHub

The tool can be configure fairly easily – for more details about how to get it up and running, refer to the relevant documentation section.


Final Thoughts

In this article we covered the most commonly used monitoring tools that can be used to control and manage Apache Kafka clusters efficiently.

Personally, I believe that Confluent – the original creators of Kafka – offer the best tool in the market that also comes at certain cost. This means that there is a chance you may not be able to afford the licensing for this particular monitoring service and thus you need to consider other options covered in this article. Note that Health+ tool has a free tier – For more details regarding its pricing refer to the Confluent’s documentation.

The second best option is probably Lenses, that also requires a paid license but probably at a -significantly- lower cost. If you cannot afford licences, then LinkedIn Cruise Control and CMAK are good enough solutions that will definitely make your life easier when it comes to cluster management and monitoring.


Become a member and read every story on Medium. Your membership fee directly supports me and other writers you read.


You may also like

Kafka No Longer Requires ZooKeeper


How To Fetch Specific Messages in Apache Kafka


How to Programmatically Create Topics in Kafka Using Python


Related Articles