Monitor Confluent Cloud Clients

Monitor clients on the Confluent Cloud Console

The Clients overview page in Confluent Cloud organizes the clients into the principal groups. You can view the client metrics that are aggregated within a principal. You can also filter for unsupported client versions.

To view and monitor the installed clients on the Clients overview page of the Confluent Cloud cluster:

1. On the Confluent Cloud Console, browse to the environment and the cluster, and click Clients.

2. For a specific principal, click the Producers, Consumers, Consumer lag, or Share groups tabs to see applicable clients in the category.

   For each principal, you can see client metrics.

3. To see consumer lag information, click the Consumer lag tab.

   Consumer lag refers to the delay between the production and consumption of messages in Kafka, which can significantly impact your system’s overall performance.

   For details about monitoring consumer lag, see Monitor Kafka Consumer Lag in Confluent Cloud.

4. Expand one or all principals to see clients for the principal.

   For each producer client, you can see the average request latency, which is the average latency between the client and the Kafka cluster.

   This metric is only available for the clients using Java 3.8+ when the client configuration, enable.metrics.push, has the value true (the default setting).

   For the best practices in the latency monitoring in Confluent Cloud, refer to Optimize Confluent Cloud Clients for Latency.

   Kafka Streams applications emit client metrics beyond request latency. For the list, see Kafka Streams client metrics.

Monitor share groups and share consumers

To monitor and manage share groups and share consumers in the Confluent Cloud Console, select your Cluster, Clients, and then the Share groups tab.

The Share groups tab provides a list of the share groups in your cluster. Each share group provides visibility into the status (Empty or Stable), the number of share consumers in the group, and the number of partitions included in the topics subscribed to by the share group.

To access the share group details, click the Share Group ID. Ensure you have the right permissions to manage the share group. See Manage share group access.

Note

You must have an empty share group to delete share groups, reset offsets, or delete offsets. To empty a share group, stop all share consumers in the share group. This ensures none of the share consumers in the share group are currently subscribed to a topic. If a share group is not empty, you must unsubscribe all the share consumers in the group from all topics to empty it.

Within the Share group details pane, you can manage your share group:

• View Topics and Consumers by navigating between the available tabs.

• To delete the share group, click Delete group in the Overview tab and follow the prompts.

  Note

  You can only delete empty share groups.

  Only users with the DeveloperManage role can delete share groups.

• To reset offsets, click Reset all offsets on the Overview tab and follow the prompts.

  Note

  You can only reset offsets in an empty share groups.

  You can reset offsets to change the position from which the share group starts consuming records. Resetting offsets allows you to consume records produced before the share consumer connected to cloud and joined the share group. You can reset offsets to earliest and latest offsets.

• To delete offsets, click Delete all offsets on the Overview tab and follow the prompts.

  Note

  You can only delete offsets in an empty share groups.

  Only users with the DeveloperManage role can delete offsets.

• To configure share group settings, click the Settings tab. You can edit these share group settings:

  • Isolation level: Controls how to read records written transactionally.

  • Auto offset reset: The strategy to initialize the share-partition start offset.

  • Record lock duration: The record acquisition lock duration in milliseconds for share groups.

  • Heartbeat interval: The heartbeat interval given to the members of a share group.

  • Session timeout: The timeout to detect client failures when using the share group protocol.

Monitor unsupported client versions

On the Clients overview page of the Confluent Cloud Console, you can check if you are using an unsupported client version.

It is recommended that you update your client versions regularly to avoid undesired behavior in your clusters. Some issues that can arise from running unsupported client versions are:

• Security vulnerabilities: Confluent Cloud is regularly updated with security patches and bug fixes. Using an unsupported client version means that you may be missing critical security updates, leaving your applications and data vulnerable to potential security threats or exploits.

• Lack of support: Confluent provides official support and maintenance for supported client versions only. If you encounter issues or bugs while using an unsupported client version, Confluent may not be able to provide assistance or troubleshooting help, leaving you to resolve the issues on your own.

• Missing features and improvements: Newer versions of client libraries often introduce new features, performance improvements, and bug fixes.

• Potential service disruptions: Confluent Cloud may introduce changes or updates that are designed to work with supported client versions. Using an unsupported client version could lead to service disruptions or unexpected behavior when such changes are made.

To ensure a stable, secure, and well-supported experience with Confluent Cloud, it is highly recommended that you use the officially supported client versions for your specific programming language or framework. See the Client versions and support for the supported client versions and their compatibility with Confluent Cloud services and features.​​​​​​​​​​​​​​​​

To monitor clients versions on the Confluent Cloud Console:

1. Select your cluster name.

2. Click Clients, and navigate to the Producer or Consumer tab depending on which clients you would like to view.

3. Select Show clients with traffic only to filter down to only clients that have produced/consumed a message in the last 10 minutes.

4. Principals containing unsupported clients are indicated by a yellow warning icon.

5. Click the dropdown arrow to view unsupported clients associated with this principal.

6. Supported versions are indicated with a green status, and the unsupported versions are indicated with a yellow status.

7. Locate the unsupported version and update the client.

   Some clients report “unknown” as the version because those clients do not emit the ApiVersion.

Kafka Streams client metrics

Confluent Cloud collects a set of Kafka Streams-specific client metrics, beyond the request latency shown in the Confluent Cloud Console, that you can query with the Confluent Cloud Metrics under the io.confluent.kafka.streams namespace. Kafka Streams applications push these metrics to the Kafka brokers through the KIP-714 client metrics push mechanism, the same way that producer and consumer clients do. These applications appear on the Clients page with the Type Stream.

The following Kafka Streams client metrics are available:

Client and thread state

• kafka_streams_client_state: The current state of a Kafka Streams client.

• thread_thread_state: The current state of a stream thread.

• recording_level: The metrics recording level for a Kafka Streams client.

Thread processing

• process_rate: The thread-level average number of records processed.

• process_ratio: The fraction of time the thread spent processing records.

• poll_ratio: The fraction of time the thread spent polling records.

• commit_ratio: The fraction of time the thread spent committing records.

• punctuate_ratio: The fraction of time the thread spent running punctuators.

Processor node

• node_e2e_latency_max: The maximum end-to-end latency of a processor node.

• node_e2e_latency_min: The minimum end-to-end latency of a processor node.

State store

• num_keys: The number of keys in the in-memory state store.

• estimate_num_keys: The estimated number of keys in the RocksDB state store.

• block_cache_usage: The percentage of RocksDB block cache in use.

• size_all_mem_tables_bytes: The total size, in bytes, of all RocksDB memtables.

The set of available client metrics changes over time. For the authoritative, current list, see the Metrics Reference and filter for the io.confluent.kafka.streams namespace.

For more about Kafka Streams metrics, including the JMX metrics that the client exposes, see Kafka Streams Metrics in Confluent Cloud.

Metrics API

The Confluent Cloud Metrics provides programmatic access to actionable metrics for your Confluent Cloud deployment, including server-side metrics for the Confluent-managed services. To retrieve client-side metrics, see Producers and Consumers.

The Metrics API, enabled by default, aggregates metrics at the topic and cluster level. Any authorized user can gain access to the metrics that allow you to monitor overall usage and performance. To get started with the Metrics API, see the Confluent Cloud Metrics documentation.

You can use the Metrics API to query metrics at the following granularities (other resolutions are available if needed):

• Bytes produced per minute grouped by topic

• Bytes consumed per minute grouped by topic

• Max retained bytes per hour over two hours for a given topic

• Max retained bytes per hour over two hours for a given cluster

You can retrieve the metrics easily over the internet using HTTPS, capturing them at regular intervals to get a time series and an operational view of cluster performance. You can integrate the metrics into any cloud provider monitoring tools like Azure Monitor, Google Cloud’s operations suite (formerly Stackdriver), or Amazon CloudWatch, or into existing monitoring systems like Prometheus and Datadog, and then plot them in a time series graph to see usage over time. When writing your own application to use the Metrics API, see the full API specification to use advanced features.

JMX metrics

Kafka applications expose some internal Java Management Extensions (JMX) metrics, and many users run JMX exporters to feed the metrics into their monitoring systems. You can retrieve JMX metrics for your client applications and the services you manage by starting your Kafka client applications with the JMX_PORT environment variable configured. You cannot use JMX metrics for the Confluent-managed services.

There are many Kafka Metrics that are exposed through JMX to provide insight on the performance of your applications.