Kafka and its components
Kafka is considered as the next-gen distributed messaging system. It is a very fast, distributed pub-sub messaging system which by design is highly scalable and durable (http://kafka.apache.org/ ).
Apache Zookeepers ( http://zookeeper.apache.org/ ) are used for managing Kafka brokers. A Kafka brokers cluster can handle millions of messages. Hence along with sufficiently powered hardware, high throughput network is essential as well.
Kafka by default retains 7 days worth of message. Producers/Consumers communicate with each other though the topics. A topic is designed to be consumed by many consumer threads by the way of many partitions. A consumer can connect to partition available on the topic they are interested and read message from there. The message affinity is maintained through the hash of the message itself (i.e. message having the same hash key value will be directed to the same partition)
(From Kafka Documentation )
Among the brokers available one will be a leader broker and others will be replicas. Whenever the leader changes the Kafka rebalances itself among leader/replicas. Automatically balances the consumers during failure as well.
Many consumers can consume from same topic provided each of the consumers identifies itself with unique identification known as groupid (a simple and unique string). This group id is what is used to maintain each consumers read offsets for a topic.
(From Kafka Documentation )
Written in Scala, Kafka has two kinds of consumers APIs – High Level Consumer and Simple Consumer
Simple Consumer APIs provide a low level consumer APIs where much of the ack/offset handling etc needs to be handled by consumer program.. However High Level Consumer APIs is ‘simpler’ (when compared to Simple Consumer) and takes care of the tasks like keeping track of consumed messages among other things. Usually High Level Consumer is sufficient in most cases.
How message processing is done
In our use-case the Producer application has millions of user registered and has equal number of messages getting produced during the peak times. These messages are consumed by the Consumer application with has equal number of consumer threads as that of partitions available on the topic.
As compared to the custom built solution of Async message processing through REST APIs, this mechanism with both sender and receiver components independent of each eathers APIs and configuration makes it much more efficient at the same time making it less resource intensive.
Replication and failover
A Kafka cluster is configured to have a leader and a set of replicas. The replication factor is the the total number of replicas including the leader. Kafka replicates the logs pushed to a topic for each partition across these configurable number of replicas. This can be configured per topic.
With our application running in three AZs (Availability Zone) on AWS, we have 3 replicas configured per topic per zone i.e. 3 Node cluster with a Node in each AZ to support any kind of failover scenario
With JMX enabled Kafka can be monitored with tools such as JConsole or JMC (in the latest version of Java). There are various JMX reporting tools which can be employed to monitor the Kafka broker as well.
Some key properties for monitoring are as below,
- All Topics Message in rate
- Specific Topic Message in rate
- All Topics Bytes in rate
- Specific Topic Bytes in rate
- All Topics Byte out rate
- Specific Topic Byte out rate
- Leader election rate
- Partition counts
- Leader replica counts
- Consumer Message rate
- Consumer lags