Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly
Appearance settings

How to tune Kafka UI to handle large messages? #870

Closed Answered by germanosin
TheDukeDK asked this question in Q&A
How to tune Kafka UI to handle large messages? #870
@TheDukeDK TheDukeDK
Feb 26, 2025 · 2 comments · 7 replies
Answered by germanosin Return to top
Discussion options

TheDukeDK
Feb 26, 2025

We have some large messages. Like 5MB. When viewing the messages it takes forever to load. I have added some resources to the pods, running on K8s, to see if this did anything but it doesn't seem to.

Is there some way to tune the app to handle these large messages?

image
You must be logged in to vote
Answered by germanosin Mar 7, 2025

  1. The max-partition-fetch-bytes consumer configuration controls how messages larger than 1MB are handled. It is recommended to increase this value. Reference.

  2. By default, Java in Docker allocates 25% of the available memory for the heap. It is advisable to increase this to 50%. Reference.

  3. I noticed that your instance is allocated 0.5 CPU. When Java detects 1 CPU, it defaults to the Serial GC. In this case, it's recommended to use G1GC (-XX:+UseG1GC).

View full answer

Replies: 2 comments · 7 replies

Comment options

germanosin
Feb 26, 2025
Maintainer

Hi, @TheDukeDK .

Could you share more details?

  • What format do you use for messages?
  • Do you have any backend logs you could share?
You must be logged in to vote
0 replies
Comment options

TheDukeDK
Feb 27, 2025
Author

Messages are compressed JSON. They are quite large, around 5MB per message. Here is a screen recording.

large-message-slow-load.mp4

Logs don't really show anything revealing. I have checked the memory and cpu resources. The seem fine also. So it looks like it is just slow decompressing the messages?

kafka-ui-7749c77f64-2q2jf 2025-02-27 07:47:42,201 INFO  [boundedElastic-70] o.a.k.c.m.Metrics: Metrics reporters closed
kafka-ui-7749c77f64-2q2jf 2025-02-27 07:47:42,204 INFO  [boundedElastic-70] o.a.k.c.u.AppInfoParser: App info kafka.consumer for kafbat-ui-consumer-1740642460618 unregistered
kafka-ui-7749c77f64-2q2jf 2025-02-27 07:48:04,235 DEBUG [parallel-4] i.k.u.s.ClustersStatisticsScheduler: Start getting metrics for kafkaCluster: oms-test3
kafka-ui-7749c77f64-2q2jf 2025-02-27 07:48:04,235 DEBUG [parallel-5] i.k.u.s.ClustersStatisticsScheduler: Start getting metrics for kafkaCluster: shared-test
kafka-ui-7749c77f64-2q2jf 2025-02-27 07:48:04,235 DEBUG [parallel-8] i.k.u.s.ClustersStatisticsScheduler: Start getting metrics for kafkaCluster: products-test
kafka-ui-7749c77f64-2q2jf 2025-02-27 07:48:04,235 DEBUG [parallel-7] i.k.u.s.ClustersStatisticsScheduler: Start getting metrics for kafkaCluster: availability-dev
kafka-ui-7749c77f64-2q2jf 2025-02-27 07:48:04,235 DEBUG [parallel-6] i.k.u.s.ClustersStatisticsScheduler: Start getting metrics for kafkaCluster: tracking-test
kafka-ui-7749c77f64-2q2jf 2025-02-27 07:48:04,242 WARN  [kafka-admin-client-thread | kafbat-ui-admin-1740582514-2] o.a.k.c.a.KafkaAdminClient: [AdminClient clientId=kafbat-ui-admin-1740582514-2] The DescribeTopicPartitions API is not supported, using Metadata API to describe topics.
kafka-ui-7749c77f64-2q2jf 2025-02-27 07:48:04,243 WARN  [kafka-admin-client-thread | kafbat-ui-admin-1740582514-3] o.a.k.c.a.KafkaAdminClient: [AdminClient clientId=kafbat-ui-admin-1740582514-3] The DescribeTopicPartitions API is not supported, using Metadata API to describe topics.
kafka-ui-7749c77f64-2q2jf 2025-02-27 07:48:04,247 WARN  [kafka-admin-client-thread | kafbat-ui-admin-1740582514-1] o.a.k.c.a.KafkaAdminClient: [AdminClient clientId=kafbat-ui-admin-1740582514-1] The DescribeTopicPartitions API is not supported, using Metadata API to describe topics.
kafka-ui-7749c77f64-2q2jf 2025-02-27 07:48:04,266 WARN  [kafka-admin-client-thread | kafbat-ui-admin-1740582514-5] o.a.k.c.a.KafkaAdminClient: [AdminClient clientId=kafbat-ui-admin-1740582514-5] The DescribeTopicPartitions API is not supported, using Metadata API to describe topics.
kafka-ui-7749c77f64-2q2jf 2025-02-27 07:48:04,269 DEBUG [parallel-5] i.k.u.s.ClustersStatisticsScheduler: Metrics updated for cluster: tracking-test
kafka-ui-7749c77f64-2q2jf 2025-02-27 07:48:04,270 WARN  [kafka-admin-client-thread | kafbat-ui-admin-1740582514-4] o.a.k.c.a.KafkaAdminClient: [AdminClient clientId=kafbat-ui-admin-1740582514-4] The DescribeTopicPartitions API is not supported, using Metadata API to describe topics.
kafka-ui-7749c77f64-2q2jf 2025-02-27 07:48:04,275 WARN  [kafka-admin-client-thread | kafbat-ui-admin-1740582514-5] o.a.k.c.a.KafkaAdminClient: [AdminClient clientId=kafbat-ui-admin-1740582514-5] The DescribeTopicPartitions API is not supported, using Metadata API to describe topics.
kafka-ui-7749c77f64-2q2jf 2025-02-27 07:48:04,279 DEBUG [parallel-1] i.k.u.s.ClustersStatisticsScheduler: Metrics updated for cluster: products-test
kafka-ui-7749c77f64-2q2jf 2025-02-27 07:48:04,393 DEBUG [parallel-6] i.k.u.s.ClustersStatisticsScheduler: Metrics updated for cluster: oms-test3
kafka-ui-7749c77f64-2q2jf 2025-02-27 07:48:04,413 DEBUG [parallel-8] i.k.u.s.ClustersStatisticsScheduler: Metrics updated for cluster: availability-dev
kafka-ui-7749c77f64-2q2jf 2025-02-27 07:48:04,836 DEBUG [parallel-6] i.k.u.s.ClustersStatisticsScheduler: Metrics updated for cluster: shared-test
kafka-ui-7749c77f64-2q2jf 2025-02-27 07:48:34,235 DEBUG [parallel-1] i.k.u.s.ClustersStatisticsScheduler: Start getting metrics for kafkaCluster: oms-test3
kafka-ui-7749c77f64-2q2jf 2025-02-27 07:48:34,235 DEBUG [parallel-5] i.k.u.s.ClustersStatisticsScheduler: Start getting metrics for kafkaCluster: products-test
kafka-ui-7749c77f64-2q2jf 2025-02-27 07:48:34,235 DEBUG [parallel-2] i.k.u.s.ClustersStatisticsScheduler: Start getting metrics for kafkaCluster: shared-test
kafka-ui-7749c77f64-2q2jf 2025-02-27 07:48:34,235 DEBUG [parallel-3] i.k.u.s.ClustersStatisticsScheduler: Start getting metrics for kafkaCluster: tracking-test
kafka-ui-7749c77f64-2q2jf 2025-02-27 07:48:34,235 DEBUG [parallel-4] i.k.u.s.ClustersStatisticsScheduler: Start getting metrics for kafkaCluster: availability-dev
kafka-ui-7749c77f64-2q2jf 2025-02-27 07:48:34,242 WARN  [kafka-admin-client-thread | kafbat-ui-admin-1740582514-2] o.a.k.c.a.KafkaAdminClient: [AdminClient clientId=kafbat-ui-admin-1740582514-2] The DescribeTopicPartitions API is not supported, using Metadata API to describe topics.
kafka-ui-7749c77f64-2q2jf 2025-02-27 07:48:34,244 WARN  [kafka-admin-client-thread | kafbat-ui-admin-1740582514-3] o.a.k.c.a.KafkaAdminClient: [AdminClient clientId=kafbat-ui-admin-1740582514-3] The DescribeTopicPartitions API is not supported, using Metadata API to describe topics.
kafka-ui-7749c77f64-2q2jf 2025-02-27 07:48:34,251 WARN  [kafka-admin-client-thread | kafbat-ui-admin-1740582514-5] o.a.k.c.a.KafkaAdminClient: [AdminClient clientId=kafbat-ui-admin-1740582514-5] The DescribeTopicPartitions API is not supported, using Metadata API to describe topics.
kafka-ui-7749c77f64-2q2jf 2025-02-27 07:48:34,254 DEBUG [parallel-6] i.k.u.s.ClustersStatisticsScheduler: Metrics updated for cluster: tracking-test
kafka-ui-7749c77f64-2q2jf 2025-02-27 07:48:34,255 WARN  [kafka-admin-client-thread | kafbat-ui-admin-1740582514-4] o.a.k.c.a.KafkaAdminClient: [AdminClient clientId=kafbat-ui-admin-1740582514-4] The DescribeTopicPartitions API is not supported, using Metadata API to describe topics.
kafka-ui-7749c77f64-2q2jf 2025-02-27 07:48:34,257 WARN  [kafka-admin-client-thread | kafbat-ui-admin-1740582514-1] o.a.k.c.a.KafkaAdminClient: [AdminClient clientId=kafbat-ui-admin-1740582514-1] The DescribeTopicPartitions API is not supported, using Metadata API to describe topics.
kafka-ui-7749c77f64-2q2jf 2025-02-27 07:48:34,257 WARN  [kafka-admin-client-thread | kafbat-ui-admin-1740582514-5] o.a.k.c.a.KafkaAdminClient: [AdminClient clientId=kafbat-ui-admin-1740582514-5] The DescribeTopicPartitions API is not supported, using Metadata API to describe topics.
kafka-ui-7749c77f64-2q2jf 2025-02-27 07:48:34,268 DEBUG [parallel-7] i.k.u.s.ClustersStatisticsScheduler: Metrics updated for cluster: products-test
kafka-ui-7749c77f64-2q2jf 2025-02-27 07:48:34,349 DEBUG [parallel-4] i.k.u.s.ClustersStatisticsScheduler: Metrics updated for cluster: availability-dev
kafka-ui-7749c77f64-2q2jf 2025-02-27 07:48:34,362 DEBUG [parallel-3] i.k.u.s.ClustersStatisticsScheduler: Metrics updated for cluster: oms-test3
kafka-ui-7749c77f64-2q2jf 2025-02-27 07:48:34,711 DEBUG [parallel-3] i.k.u.s.ClustersStatisticsScheduler: Metrics updated for cluster: shared-test
kafka-ui-7749c77f64-2q2jf 2025-02-27 07:49:04,235 DEBUG [parallel-8] i.k.u.s.ClustersStatisticsScheduler: Start getting metrics for kafkaCluster: shared-test
kafka-ui-7749c77f64-2q2jf 2025-02-27 07:49:04,235 DEBUG [parallel-7] i.k.u.s.ClustersStatisticsScheduler: Start getting metrics for kafkaCluster: oms-test3
kafka-ui-7749c77f64-2q2jf 2025-02-27 07:49:04,235 DEBUG [parallel-1] i.k.u.s.ClustersStatisticsScheduler: Start getting metrics for kafkaCluster: tracking-test
kafka-ui-7749c77f64-2q2jf 2025-02-27 07:49:04,235 DEBUG [parallel-3] i.k.u.s.ClustersStatisticsScheduler: Start getting metrics for kafkaCluster: products-test
kafka-ui-7749c77f64-2q2jf 2025-02-27 07:49:04,235 DEBUG [parallel-2] i.k.u.s.ClustersStatisticsScheduler: Start getting metrics for kafkaCluster: availability-dev
kafka-ui-7749c77f64-2q2jf 2025-02-27 07:49:04,243 WARN  [kafka-admin-client-thread | kafbat-ui-admin-1740582514-3] o.a.k.c.a.KafkaAdminClient: [AdminClient clientId=kafbat-ui-admin-1740582514-3] The DescribeTopicPartitions API is not supported, using Metadata API to describe topics.
kafka-ui-7749c77f64-2q2jf 2025-02-27 07:49:04,244 WARN  [kafka-admin-client-thread | kafbat-ui-admin-1740582514-2] o.a.k.c.a.KafkaAdminClient: [AdminClient clientId=kafbat-ui-admin-1740582514-2] The DescribeTopicPartitions API is not supported, using Metadata API to describe topics.
kafka-ui-7749c77f64-2q2jf 2025-02-27 07:49:04,246 WARN  [kafka-admin-client-thread | kafbat-ui-admin-1740582514-1] o.a.k.c.a.KafkaAdminClient: [AdminClient clientId=kafbat-ui-admin-1740582514-1] The DescribeTopicPartitions API is not supported, using Metadata API to describe topics.
kafka-ui-7749c77f64-2q2jf 2025-02-27 07:49:04,253 WARN  [kafka-admin-client-thread | kafbat-ui-admin-1740582514-4] o.a.k.c.a.KafkaAdminClient: [AdminClient clientId=kafbat-ui-admin-1740582514-4] The DescribeTopicPartitions API is not supported, using Metadata API to describe topics.
kafka-ui-7749c77f64-2q2jf 2025-02-27 07:49:04,254 WARN  [kafka-admin-client-thread | kafbat-ui-admin-1740582514-5] o.a.k.c.a.KafkaAdminClient: [AdminClient clientId=kafbat-ui-admin-1740582514-5] The DescribeTopicPartitions API is not supported, using Metadata API to describe topics.
kafka-ui-7749c77f64-2q2jf 2025-02-27 07:49:04,260 WARN  [kafka-admin-client-thread | kafbat-ui-admin-1740582514-5] o.a.k.c.a.KafkaAdminClient: [AdminClient clientId=kafbat-ui-admin-1740582514-5] The DescribeTopicPartitions API is not supported, using Metadata API to describe topics.
kafka-ui-7749c77f64-2q2jf 2025-02-27 07:49:04,262 DEBUG [parallel-7] i.k.u.s.ClustersStatisticsScheduler: Metrics updated for cluster: tracking-test
kafka-ui-7749c77f64-2q2jf 2025-02-27 07:49:04,270 DEBUG [parallel-5] i.k.u.s.ClustersStatisticsScheduler: Metrics updated for cluster: products-test
kafka-ui-7749c77f64-2q2jf 2025-02-27 07:49:04,338 DEBUG [parallel-2] i.k.u.s.ClustersStatisticsScheduler: Metrics updated for cluster: oms-test3
kafka-ui-7749c77f64-2q2jf 2025-02-27 07:49:04,385 DEBUG [parallel-7] i.k.u.s.ClustersStatisticsScheduler: Metrics updated for cluster: availability-dev
kafka-ui-7749c77f64-2q2jf 2025-02-27 07:49:04,654 DEBUG [parallel-7] i.k.u.s.ClustersStatisticsScheduler: Metrics updated for cluster: shared-test
kafka-ui-7749c77f64-2q2jf 2025-02-27 07:49:08,084 WARN  [kafka-admin-client-thread | kafbat-ui-admin-1740582514-1] o.a.k.c.a.KafkaAdminClient: [AdminClient clientId=kafbat-ui-admin-1740582514-1] The DescribeTopicPartitions API is not supported, using Metadata API to describe topics.
kafka-ui-7749c77f64-2q2jf 2025-02-27 07:49:34,235 DEBUG [parallel-1] i.k.u.s.ClustersStatisticsScheduler: Start getting metrics for kafkaCluster: tracking-test
kafka-ui-7749c77f64-2q2jf 2025-02-27 07:49:34,235 DEBUG [parallel-3] i.k.u.s.ClustersStatisticsScheduler: Start getting metrics for kafkaCluster: products-test
kafka-ui-7749c77f64-2q2jf 2025-02-27 07:49:34,235 DEBUG [parallel-7] i.k.u.s.ClustersStatisticsScheduler: Start getting metrics for kafkaCluster: oms-test3
kafka-ui-7749c77f64-2q2jf 2025-02-27 07:49:34,235 DEBUG [parallel-8] i.k.u.s.ClustersStatisticsScheduler: Start getting metrics for kafkaCluster: shared-test
kafka-ui-7749c77f64-2q2jf 2025-02-27 07:49:34,235 DEBUG [parallel-2] i.k.u.s.ClustersStatisticsScheduler: Start getting metrics for kafkaCluster: availability-dev
kafka-ui-7749c77f64-2q2jf 2025-02-27 07:49:34,243 WARN  [kafka-admin-client-thread | kafbat-ui-admin-1740582514-2] o.a.k.c.a.KafkaAdminClient: [AdminClient clientId=kafbat-ui-admin-1740582514-2] The DescribeTopicPartitions API is not supported, using Metadata API to describe topics.
kafka-ui-7749c77f64-2q2jf 2025-02-27 07:49:34,243 WARN  [kafka-admin-client-thread | kafbat-ui-admin-1740582514-3] o.a.k.c.a.KafkaAdminClient: [AdminClient clientId=kafbat-ui-admin-1740582514-3] The DescribeTopicPartitions API is not supported, using Metadata API to describe topics.
kafka-ui-7749c77f64-2q2jf 2025-02-27 07:49:34,247 WARN  [kafka-admin-client-thread | kafbat-ui-admin-1740582514-1] o.a.k.c.a.KafkaAdminClient: [AdminClient clientId=kafbat-ui-admin-1740582514-1] The DescribeTopicPartitions API is not supported, using Metadata API to describe topics.
kafka-ui-7749c77f64-2q2jf 2025-02-27 07:49:34,258 WARN  [kafka-admin-client-thread | kafbat-ui-admin-1740582514-4] o.a.k.c.a.KafkaAdminClient: [AdminClient clientId=kafbat-ui-admin-1740582514-4] The DescribeTopicPartitions API is not supported, using Metadata API to describe topics.
kafka-ui-7749c77f64-2q2jf 2025-02-27 07:49:34,258 DEBUG [parallel-1] i.k.u.s.ClustersStatisticsScheduler: Metrics updated for cluster: tracking-test
kafka-ui-7749c77f64-2q2jf 2025-02-27 07:49:34,259 WARN  [kafka-admin-client-thread | kafbat-ui-admin-1740582514-5] o.a.k.c.a.KafkaAdminClient: [AdminClient clientId=kafbat-ui-admin-1740582514-5] The DescribeTopicPartitions API is not supported, using Metadata API to describe topics.
kafka-ui-7749c77f64-2q2jf 2025-02-27 07:49:34,268 WARN  [kafka-admin-client-thread | kafbat-ui-admin-1740582514-5] o.a.k.c.a.KafkaAdminClient: [AdminClient clientId=kafbat-ui-admin-1740582514-5] The DescribeTopicPartitions API is not supported, using Metadata API to describe topics.
kafka-ui-7749c77f64-2q2jf 2025-02-27 07:49:34,271 DEBUG [parallel-6] i.k.u.s.ClustersStatisticsScheduler: Metrics updated for cluster: products-test
kafka-ui-7749c77f64-2q2jf 2025-02-27 07:49:34,438 DEBUG [parallel-1] i.k.u.s.ClustersStatisticsScheduler: Metrics updated for cluster: oms-test3
kafka-ui-7749c77f64-2q2jf 2025-02-27 07:49:34,448 DEBUG [parallel-5] i.k.u.s.ClustersStatisticsScheduler: Metrics updated for cluster: availability-dev
kafka-ui-7749c77f64-2q2jf 2025-02-27 07:49:34,829 DEBUG [parallel-5] i.k.u.s.ClustersStatisticsScheduler: Metrics updated for cluster: shared-test
kafka-ui-7749c77f64-2q2jf 2025-02-27 07:50:04,235 DEBUG [parallel-7] i.k.u.s.ClustersStatisticsScheduler: Start getting metrics for kafkaCluster: availability-dev
kafka-ui-7749c77f64-2q2jf 2025-02-27 07:50:04,235 DEBUG [parallel-6] i.k.u.s.ClustersStatisticsScheduler: Start getting metrics for kafkaCluster: tracking-test
kafka-ui-7749c77f64-2q2jf 2025-02-27 07:50:04,235 DEBUG [parallel-8] i.k.u.s.ClustersStatisticsScheduler: Start getting metrics for kafkaCluster: products-test
kafka-ui-7749c77f64-2q2jf 2025-02-27 07:50:04,235 DEBUG [parallel-4] i.k.u.s.ClustersStatisticsScheduler: Start getting metrics for kafkaCluster: oms-test3
kafka-ui-7749c77f64-2q2jf 2025-02-27 07:50:04,235 DEBUG [parallel-5] i.k.u.s.ClustersStatisticsScheduler: Start getting metrics for kafkaCluster: shared-test
kafka-ui-7749c77f64-2q2jf 2025-02-27 07:50:04,242 WARN  [kafka-admin-client-thread | kafbat-ui-admin-1740582514-2] o.a.k.c.a.KafkaAdminClient: [AdminClient clientId=kafbat-ui-admin-1740582514-2] The DescribeTopicPartitions API is not supported, using Metadata API to describe topics.
kafka-ui-7749c77f64-2q2jf 2025-02-27 07:50:04,244 WARN  [kafka-admin-client-thread | kafbat-ui-admin-1740582514-3] o.a.k.c.a.KafkaAdminClient: [AdminClient clientId=kafbat-ui-admin-1740582514-3] The DescribeTopicPartitions API is not supported, using Metadata API to describe topics.
kafka-ui-7749c77f64-2q2jf 2025-02-27 07:50:04,250 WARN  [kafka-admin-client-thread | kafbat-ui-admin-1740582514-1] o.a.k.c.a.KafkaAdminClient: [AdminClient clientId=kafbat-ui-admin-1740582514-1] The DescribeTopicPartitions API is not supported, using Metadata API to describe topics.
kafka-ui-7749c77f64-2q2jf 2025-02-27 07:50:04,252 WARN  [kafka-admin-client-thread | kafbat-ui-admin-1740582514-5] o.a.k.c.a.KafkaAdminClient: [AdminClient clientId=kafbat-ui-admin-1740582514-5] The DescribeTopicPartitions API is not supported, using Metadata API to describe topics.
kafka-ui-7749c77f64-2q2jf 2025-02-27 07:50:04,258 WARN  [kafka-admin-client-thread | kafbat-ui-admin-1740582514-5] o.a.k.c.a.KafkaAdminClient: [AdminClient clientId=kafbat-ui-admin-1740582514-5] The DescribeTopicPartitions API is not supported, using Metadata API to describe topics.
kafka-ui-7749c77f64-2q2jf 2025-02-27 07:50:04,261 DEBUG [parallel-8] i.k.u.s.ClustersStatisticsScheduler: Metrics updated for cluster: tracking-test
kafka-ui-7749c77f64-2q2jf 2025-02-27 07:50:04,261 WARN  [kafka-admin-client-thread | kafbat-ui-admin-1740582514-4] o.a.k.c.a.KafkaAdminClient: [AdminClient clientId=kafbat-ui-admin-1740582514-4] The DescribeTopicPartitions API is not supported, using Metadata API to describe topics.
kafka-ui-7749c77f64-2q2jf 2025-02-27 07:50:04,272 DEBUG [parallel-2] i.k.u.s.ClustersStatisticsScheduler: Metrics updated for cluster: products-test
kafka-ui-7749c77f64-2q2jf 2025-02-27 07:50:04,376 DEBUG [parallel-7] i.k.u.s.ClustersStatisticsScheduler: Metrics updated for cluster: availability-dev
kafka-ui-7749c77f64-2q2jf 2025-02-27 07:50:04,435 DEBUG [parallel-7] i.k.u.s.ClustersStatisticsScheduler: Metrics updated for cluster: oms-test3
kafka-ui-7749c77f64-2q2jf 2025-02-27 07:50:04,737 DEBUG [parallel-6] i.k.u.s.ClustersStatisticsScheduler: Metrics updated for cluster: shared-test
kafka-ui-7749c77f64-2q2jf 2025-02-27 07:50:33,990 WARN  [kafka-admin-client-thread | kafbat-ui-admin-1740582514-1] o.a.k.c.a.KafkaAdminClient: [AdminClient clientId=kafbat-ui-admin-1740582514-1] The DescribeTopicPartitions API is not supported, using Metadata API to describe topics.
kafka-ui-7749c77f64-2q2jf 2025-02-27 07:50:34,235 DEBUG [parallel-7] i.k.u.s.ClustersStatisticsScheduler: Start getting metrics for kafkaCluster: availability-dev
kafka-ui-7749c77f64-2q2jf 2025-02-27 07:50:34,235 DEBUG [parallel-6] i.k.u.s.ClustersStatisticsScheduler: Start getting metrics for kafkaCluster: tracking-test
kafka-ui-7749c77f64-2q2jf 2025-02-27 07:50:34,235 DEBUG [parallel-5] i.k.u.s.ClustersStatisticsScheduler: Start getting metrics for kafkaCluster: shared-test
kafka-ui-7749c77f64-2q2jf 2025-02-27 07:50:34,235 DEBUG [parallel-8] i.k.u.s.ClustersStatisticsScheduler: Start getting metrics for kafkaCluster: products-test
kafka-ui-7749c77f64-2q2jf 2025-02-27 07:50:34,235 DEBUG [parallel-4] i.k.u.s.ClustersStatisticsScheduler: Start getting metrics for kafkaCluster: oms-test3
kafka-ui-7749c77f64-2q2jf 2025-02-27 07:50:34,241 WARN  [kafka-admin-client-thread | kafbat-ui-admin-1740582514-2] o.a.k.c.a.KafkaAdminClient: [AdminClient clientId=kafbat-ui-admin-1740582514-2] The DescribeTopicPartitions API is not supported, using Metadata API to describe topics.
kafka-ui-7749c77f64-2q2jf 2025-02-27 07:50:34,243 WARN  [kafka-admin-client-thread | kafbat-ui-admin-1740582514-3] o.a.k.c.a.KafkaAdminClient: [AdminClient clientId=kafbat-ui-admin-1740582514-3] The DescribeTopicPartitions API is not supported, using Metadata API to describe topics.
kafka-ui-7749c77f64-2q2jf 2025-02-27 07:50:34,247 WARN  [kafka-admin-client-thread | kafbat-ui-admin-1740582514-1] o.a.k.c.a.KafkaAdminClient: [AdminClient clientId=kafbat-ui-admin-1740582514-1] The DescribeTopicPartitions API is not supported, using Metadata API to describe topics.
kafka-ui-7749c77f64-2q2jf 2025-02-27 07:50:34,249 WARN  [kafka-admin-client-thread | kafbat-ui-admin-1740582514-5] o.a.k.c.a.KafkaAdminClient: [AdminClient clientId=kafbat-ui-admin-1740582514-5] The DescribeTopicPartitions API is not supported, using Metadata API to describe topics.
kafka-ui-7749c77f64-2q2jf 2025-02-27 07:50:34,250 WARN  [kafka-admin-client-thread | kafbat-ui-admin-1740582514-4] o.a.k.c.a.KafkaAdminClient: [AdminClient clientId=kafbat-ui-admin-1740582514-4] The DescribeTopicPartitions API is not supported, using Metadata API to describe topics.
kafka-ui-7749c77f64-2q2jf 2025-02-27 07:50:34,256 WARN  [kafka-admin-client-thread | kafbat-ui-admin-1740582514-5] o.a.k.c.a.KafkaAdminClient: [AdminClient clientId=kafbat-ui-admin-1740582514-5] The DescribeTopicPartitions API is not supported, using Metadata API to describe topics.
kafka-ui-7749c77f64-2q2jf 2025-02-27 07:50:34,264 DEBUG [parallel-6] i.k.u.s.ClustersStatisticsScheduler: Metrics updated for cluster: tracking-test
kafka-ui-7749c77f64-2q2jf 2025-02-27 07:50:34,270 DEBUG [parallel-5] i.k.u.s.ClustersStatisticsScheduler: Metrics updated for cluster: products-test
kafka-ui-7749c77f64-2q2jf 2025-02-27 07:50:34,348 DEBUG [parallel-4] i.k.u.s.ClustersStatisticsScheduler: Metrics updated for cluster: availability-dev
kafka-ui-7749c77f64-2q2jf 2025-02-27 07:50:34,433 DEBUG [parallel-4] i.k.u.s.ClustersStatisticsScheduler: Metrics updated for cluster: oms-test3
kafka-ui-7749c77f64-2q2jf 2025-02-27 07:50:34,946 DEBUG [parallel-6] i.k.u.s.ClustersStatisticsScheduler: Metrics updated for cluster: shared-test
You must be logged in to vote
7 replies
@germanosin
Comment options

germanosin Mar 5, 2025
Maintainer

@TheDukeDK seems suspicious. Could you provide the CPU/memory configuration for the instance? Is it running in Docker? What is the GC setting? Also, please share the consumer settings, especially the max.partition.fetch.bytes.
@TheDukeDK
Comment options

TheDukeDK Mar 7, 2025
Author

Its running on a K8s cluster. It requests 1Gi of memory with a 2Gi limit and half of a CPU with no limits defined. Tried adjusting these values but it seemed to have no effect.

Also tried tweaking -Xms and -Xmx to the the JAVA_OPTS but it seemed to have no effect so it is configured as below.

JAVA_OPTS: "-Djavax.net.ssl.trustStore=/keystore/keystore.jks -Djavax.net.ssl.trustStorePassword=testtest -Djavax.net.ssl.trustStoreType=jks"

I am not sure where I should find the max.partition.fetch.bytes configuration. But am guessing this is a Kafka Cluster configuration? Our Kafka clusters are Strimzi based so I should be able to tweak this I guess.

apiVersion: apps/v1
kind: Deployment
metadata:
  annotations:
    deployment.kubernetes.io/revision: "121"
    downscaler/original-replicas: "1"
    meta.helm.sh/release-name: kafka-ui
    meta.helm.sh/release-namespace: queues
  creationTimestamp: "2022-07-04T12:11:33Z"
  generation: 131
  labels:
    app.kubernetes.io/instance: kafka-ui
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: kafka-ui
    app.kubernetes.io/version: v1.1.0
    helm.sh/chart: kafka-ui-1.4.11
  name: kafka-ui
  namespace: queues
  resourceVersion: "1370654831"
  uid: 1c2fcc92-7482-42e3-a4e7-515fa21cbf02
spec:
  progressDeadlineSeconds: 600
  replicas: 1
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      app.kubernetes.io/instance: kafka-ui
      app.kubernetes.io/name: kafka-ui
  strategy:
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 25%
    type: RollingUpdate
  template:
    metadata:
      annotations:
        checksum/config: 14d41ee3e105b5b3e6e93b1d4433292c3f9a89e87d6331dd0c9ffdd2a4820966
        checksum/configFromValues: 64a87795660cdd551c2ac54d22a59f130fa35c857ed68aeaf16bd303a31c31c3
        checksum/secret: 01ba4719c80b6fe911b091a7c05124b64eeece964e09c058ef8f9805daca546b
        kubectl.kubernetes.io/restartedAt: "2025-02-26T15:21:37+01:00"
      creationTimestamp: null
      labels:
        app.kubernetes.io/instance: kafka-ui
        app.kubernetes.io/name: kafka-ui
    spec:
      containers:
      - env:
        - name: logging.config
          value: /logging-config/logback-json.xml
        - name: SPRING_CONFIG_ADDITIONAL-LOCATION
          value: /kafka-ui/config.yml
        envFrom:
        - configMapRef:
            name: kafka-ui
        image: ghcr.io/kafbat/kafka-ui:v1.1.0
        imagePullPolicy: IfNotPresent
        livenessProbe:
          failureThreshold: 3
          httpGet:
            path: /actuator/health
            port: http
            scheme: HTTP
          initialDelaySeconds: 60
          periodSeconds: 30
          successThreshold: 1
          timeoutSeconds: 10
        name: kafka-ui
        ports:
        - containerPort: 8080
          name: http
          protocol: TCP
        readinessProbe:
          failureThreshold: 3
          httpGet:
            path: /actuator/health
            port: http
            scheme: HTTP
          initialDelaySeconds: 60
          periodSeconds: 30
          successThreshold: 1
          timeoutSeconds: 10
        resources:
          limits:
            memory: 2Gi
          requests:
            cpu: 500m
            memory: 1Gi
        securityContext: {}
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /keystore
          name: cert-keystore
          readOnly: true
        - mountPath: /logging-config/
          name: kafka-ui-logging-config
          readOnly: true
        - mountPath: /kafka-ui/
          name: kafka-ui-yaml-conf
      dnsPolicy: ClusterFirst
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      serviceAccount: kafka-ui
      serviceAccountName: kafka-ui
      terminationGracePeriodSeconds: 30
      volumes:
      - configMap:
          defaultMode: 420
          name: nemlig-cert-keystore
        name: cert-keystore
      - configMap:
          defaultMode: 420
          name: kafka-ui-logging-config
        name: kafka-ui-logging-config
      - configMap:
          defaultMode: 420
          name: kafka-ui-fromvalues
        name: kafka-ui-yaml-conf
status:
  availableReplicas: 1
  conditions:
  - lastTransitionTime: "2022-07-04T12:11:33Z"
    lastUpdateTime: "2025-02-28T12:29:44Z"
    message: ReplicaSet "kafka-ui-7cc8cc477b" has successfully progressed.
    reason: NewReplicaSetAvailable
    status: "True"
    type: Progressing
  - lastTransitionTime: "2025-02-28T12:47:47Z"
    lastUpdateTime: "2025-02-28T12:47:47Z"
    message: Deployment has minimum availability.
    reason: MinimumReplicasAvailable
    status: "True"
    type: Available
  observedGeneration: 131
  readyReplicas: 1
  replicas: 1
  updatedReplicas: 1
@germanosin
Comment options

germanosin Mar 7, 2025
Maintainer

  1. The max-partition-fetch-bytes consumer configuration controls how messages larger than 1MB are handled. It is recommended to increase this value. Reference.

  2. By default, Java in Docker allocates 25% of the available memory for the heap. It is advisable to increase this to 50%. Reference.

  3. I noticed that your instance is allocated 0.5 CPU. When Java detects 1 CPU, it defaults to the Serial GC. In this case, it's recommended to use G1GC (-XX:+UseG1GC).
Answer selected by TheDukeDK
@TheDukeDK
Comment options

TheDukeDK Mar 10, 2025
Author

Super helpful 😄 !

Thank you very much! ❤️
@hcygeorge
Comment options

hcygeorge May 27, 2025

I am also experiencing issues loading large messages (over 1 MB per message) in the Kafbat UI and came across this discussion. I have increased the container resources and the Java heap size for Kafbat, but it didn't help. Could you tell me where I can find and configure the max.partition.fetch.bytes setting for the Kafbat consumer?
@germanosin
Comment options

germanosin May 27, 2025
Maintainer

@hcygeorge
there is a specific config property consumerProperties for each cluster:

      consumer-properties:
        "max.partition.fetch.bytes": 1048576
@hcygeorge
Comment options

hcygeorge May 29, 2025

Thanks, @germanosin.!
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Category
🙏
Q&A
Labels
None yet
3 participants
@TheDukeDK @germanosin @hcygeorge
Morty Proxy This is a proxified and sanitized view of the page, visit original site.