The world’s leading publication for data science, AI, and ML professionals.

Overview of UI Monitoring Tools for Apache Kafka Clusters

What are the best monitoring tools for Apache Kafka?

Photo by Chris Liverani on unsplash.com
Photo by Chris Liverani on unsplash.com

With the increasing demand around Kafka Cluster monitoring and management, a number of open-source and commercial graphical tools have reached the market, offering a variety of administration and monitoring functionalities.

  1. Confluent Control Centre
  2. Lenses
  3. Datadog Kafka Dashboard
  4. Cloudera Manager
  5. Yahoo Kafka Manager
  6. KafDrop
  7. LinkedIn Burrow
  8. Kafka Tool

Confluent Control Centre

Confluent is the company founded by the original creators of Apache Kafka. Confluent Enterprise, is a -more complete- Kafka distribution for production environments. The commercial licence of Confluent Platform comes with Confluent Control Centre which is a management system for Apache Kafka that enables cluster monitoring and management from a User Interface.

Confluent Control Centre
Confluent Control Centre

Confluent Control Center delivers understanding and insight about the inner workings of the Apache Kafka clusters and the data that flows through them. Control Center gives the administrator monitoring and management capabilities through curated dashboards, so that they can deliver optimal performance and meet SLAs for their Apache Kafka clusters.


Lenses

Lenses (ex Landoop) is a company that offers enterprise features and monitoring tools for Kafka Clusters. More precisely, it enhances Kafka with User Interface, streaming SQL engine and Cluster monitoring. It also enables faster monitoring of Kafka data pipelines by providing SQL and Connector visibility into your data flows.

Monitoring and managing streaming data flows with Lenses
Monitoring and managing streaming data flows with Lenses

Lenses works with any Kafka distribution, delivers high quality enterprise features and monitoring, SQL for ALL and self-serviced real-time data access and flows on Kubernetes.

The company also offers Lenses Box, which is a free all-in-one docker that can serve a single broker for up to 25M messages. Note that Lenses Box is recommended for development environments.

Kafka Topics UI

Furthermore, Lenses also offers Kafka Topics UI, which is a web tool for managing Kafka Topics.

Kafka Topics UI by Lenses
Kafka Topics UI by Lenses

Datadog Kafka Dashboard

Kafka Dashboard by Datadog is a comprehensive Kafka Dashboard that displays key metrics for Kafka Brokers, Producers, Consumers and Apache Zookeeper. Kafka deployments often rely on external software which is not part of the Kafka, like Apache Zookeeper. Datadog enables a comprehensive Monitoring on all the layers of your deployment, including software components in your data pipeline which are not part of Kafka as such.

Kafka Dashboard by DataDog
Kafka Dashboard by DataDog

Save time on setup and visualize your Kafka data in minutes with Datadog’s out-of-the-box dashboard.


Cloudera Manager

Kafka in Cloudera Manager is clearly a less rich monitoring tool compared to Confluent, Lenses and Datadog. However, it is very convenient for companies that are already customers of Cloudera and need their monitoring mechanisms under the same platform.

Monitoring Kafka using Cloudera Manager
Monitoring Kafka using Cloudera Manager

_Want to read this story later? Save it in Journal._

Yahoo Kafka Manager

Yahoo Kafka Manager is an open-source managing tool for Apache Kafka clusters. With Kafka Manager, you can:

  • Manage multiple clusters
  • Easy inspection of cluster state (topics, consumers, offsets, brokers, replica distribution, partition distribution)
  • Run preferred replica election
  • Generate partition assignments with option to select brokers to use
  • Run reassignment of partition (based on generated assignments)
  • Create a topic with optional topic configs
  • Delete topics
  • Batch generate partition assignments for multiple topics with option to select brokers to use
  • Batch run reassignment of partition for multiple topics
  • Add partitions to existing topic
  • Update config for existing topic
  • Optionally enable JMX polling for broker level and topic level metrics.
  • Optionally filter out consumers that do not have ids/ owners/ & offsets/ directories in zookeeper.
Topics Overview in Yahoo Kafka Manager
Topics Overview in Yahoo Kafka Manager

KafDrop

KafDrop is an open-source UI for monitoring Apache Kafka clusters. The tool displays information such as brokers, topics, partitions, and even lets you view messages. It is a lightweight application that runs on Spring Boot and requires very little configuration.

Broker Overview in KafDrop. Photo by HomeAdvisor
Broker Overview in KafDrop. Photo by HomeAdvisor

KafDrop 3

Kafdrop 3 is a UI for navigating and monitoring Apache Kafka brokers. The tool displays information such as brokers, topics, partitions, consumers and lets you view messages.

This project is a reboot of Kafdrop 2.x, dragged kicking and screaming into the world of JDK 11+, Kafka 2.x and Kubernetes.


LinkedIn Burrow

LinkedIn Burrow is an open-source monitoring companion for Apache Kafka that provides consumer lag checking as a service without the need for specifying thresholds. It monitors committed offsets for all consumers and calculates the status of those consumers on demand. An HTTP endpoint is provided to request status on demand, as well as provide other Kafka cluster information. There are also configurable notifiers that can send status out via email or HTTP calls to another service. Burrow is written in Go, so before you get started, you should install and set up Go.


Kafka Tool

Kafka Tool is a GUI application for managing and using Apache Kafka clusters. It provides an intuitive UI that allows one to quickly view objects within a Kafka cluster as well as the messages stored in the topics of the cluster. It contains features geared towards both developers and administrators. Using Kafka Tool, you can:

  • View metrics on cluster, broker, topic and consumer level
  • View contents of messages in your partitions and add new messages
  • View offsets of the Kafka consumers, including Apache Storm Kafka spout consumers
  • Show JSON and XML messages in a pretty-printed format
  • Add and drop topics plus other management features
  • Save individual messages from your Kafka partitions to local hard drive
  • Write your own plugins that allow you to view custom data formats

The tool runs on Windows, Linux and Mac OS.


Comparison and Conclusions

If you cannot afford commercial licenses then your options are Yahoo Kafka Manager, LinkedIn Burrow, KafDrop and Kafka Tool. In my opinion, the former is a comprehensive solution that should do the trick for most of the use-cases.

If you are running relatively big Kafka Clusters, then it is worth paying for a commercial license. Confluent and Lenses offer more rich functionality compared to the other monitoring tools we’ve seen in this post and I would highly recommend both of them.

For an updated list of available Kafka UI Monitoring tools, make sure to read my recent article below:

Kafka UI Monitoring Tools (2021 Update)


Become a member and read every story on Medium. Your membership fee directly supports me and other writers you read.


You may also like

Kafka No Longer Requires ZooKeeper

How To Fetch Specific Messages in Apache Kafka


Related Articles