Kafka Cluster Operations: Day-2 Management for Enterprise Deployments

Day-2 Kafka: Beyond the Initial Deployment

Kafka is relatively straightforward to deploy and hard to operate well at scale. The initial deployment — configure brokers, create topics, deploy producers and consumers — is well-documented and tooled. The challenge is everything that comes after: managing consumer group lag as throughput grows, handling broker storage as data accumulates, rebalancing partitions after a broker failure, and maintaining performance as topic counts and partition counts scale beyond initial estimates.

This is a practical guide to the operational discipline that keeps enterprise Kafka clusters healthy over the long term.

Consumer Group Lag: The Primary Health Signal

Consumer group lag — the difference between the latest offset produced to a topic partition and the latest offset consumed by a consumer group from that partition — is the most important Kafka operational metric. Growing lag means consumers are not keeping up with producers; if lag grows without bound, the consumer is effectively non-functional for real-time use cases.

Monitoring Consumer Lag

Kafka Lag Exporter (open source) and AKHQ both provide consumer group lag metrics compatible with Prometheus. The key metrics are:

• kafka_consumer_group_lag: lag per partition per consumer group
• kafka_consumer_group_lag_sum: total lag across all partitions for a consumer group

Alert thresholds depend on your use case latency requirements: for real-time processing, alert when lag exceeds 1,000 messages; for batch processing, the threshold might be 1,000,000. The critical alert should trigger when lag is growing — even a small lag that is consistently increasing indicates a consumer that will eventually fall behind completely.

Diagnosing Consumer Lag

• Consumer is slow: Processing time per message exceeds the produce rate. Profile the consumer for blocking operations, slow downstream calls, or insufficient parallelism.
• Consumer is under-partitioned: More partitions allow more consumer threads; if lag appears evenly across partitions, adding partitions (and consumers) increases throughput. Note that adding partitions to an existing topic requires a partition rebalance.
• Consumer is failing and rebalancing: Repeated consumer group rebalances (due to consumer crashes, long processing times exceeding max.poll.interval.ms) create lag spikes. Look for frequent rebalance log entries in consumer application logs.

Broker Storage Management

Kafka retains messages on disk for a configured retention period (default 7 days for time-based retention, or a configured size limit for size-based retention). For high-throughput topics, broker storage can grow rapidly. Key practices:

• Monitor disk usage per broker and alert at 70% of configured disk capacity, not 90% — Kafka performance degrades before disks are full.
• Use log compaction for changelog topics (state stores, CDC topics) rather than time-based retention. Compacted topics retain only the latest value per key, bounding storage independent of write rate.
• Set topic-level retention configurations (retention.ms, retention.bytes) explicitly. Topics without explicit configuration inherit broker defaults, and broker defaults may be inappropriate for specific use cases.
• Monitor the total number of log segments per broker. The Kafka controller uses partition metadata in memory; very large partition counts (tens of thousands per broker) can create controller performance issues.

Partition Rebalancing

Partition leadership is distributed across brokers but can become unbalanced over time — particularly after a broker failure and recovery, where the recovering broker returns without its previously-led partitions. The Kafka preferred replica election mechanism re-elects preferred leaders, but it must be triggered or configured for automatic preferred leader election.

• Enable auto.leader.rebalance.enable=true and set leader.imbalance.check.interval.seconds=300 to automatically trigger rebalancing.
• Use Cruise Control (open source, by LinkedIn) for data-aware partition rebalancing that balances both leadership count and byte-in/byte-out rates across brokers.
• After a broker failure and recovery, explicitly run a preferred replica election (kafka-preferred-replica-election.sh) to return partition leadership to the recovered broker.

KRaft Mode Operations

Kafka clusters migrated to KRaft mode (ZooKeeper-free, available from Kafka 3.3+) have a different operational model for metadata management. The KRaft controller quorum replaces ZooKeeper; controller health monitoring and quorum majority requirements replace ZooKeeper ensemble health. In KRaft mode, monitor controller quorum health via the kafka.controller:type=KafkaController,name=ActiveControllerCount JMX metric and ensure the controller quorum maintains majority (3 of 5, or 2 of 3 controller nodes, depending on configuration).

OSSeva Operate for Kafka

OSSeva Operate for Kafka clusters includes pre-deployed consumer lag monitoring with per-group alerting, broker storage management with automated retention review, and 24/7 response to partition leadership failures and consumer group incidents. Quarterly reviews include capacity planning for partition and storage growth based on trending data.

Conclusion

Kafka day-2 operations are primarily a monitoring and response discipline. The failure modes are deterministic once you understand the data model; the operational challenge is instrumenting the right metrics, setting thresholds that match your use case requirements, and responding to degradation before it escalates. Teams that invest in this operational foundation find Kafka reliable at enterprise scale; teams that don't encounter the same failure modes repeatedly without a systematic path to resolution.