kafka_group not working with kafka > 2.1 #315
Description
Using the kafka_group consumer with a Kafka 2.3 cluster results in lost messages, duplicated messages and abnormally high load and low throughput. Using a Kafka 2.1 cluster works well. I am not sure of the reason but I suspect that the consumer offset change (introduced in Kafka 2.2) is not supported by ruby-kafka (due to the compatibility chart). But these are just speculations that made me try out Kafka 2.1, nothing I have confirmed.
The complete plugin config for reference:
<source>
@type kafka_group
brokers log-kafka-bootstrap.log.svc.cluster.local:9093
consumer_group fluentd
topics test
format json
time_source record
time_format "%iso8601"
start_from_beginning false
max_wait_time 5
max_bytes 1048576
username "#{ENV['KAFKA_USERNAME']}"
password "#{ENV['KAFKA_PASSWORD']}"
scram_mechanism sha512
ssl_ca_cert /etc/ssl/certs/kafka/ca.crt
</source>
I am running multiple fluentd instances with the same config.