Open
Description
Description
The Producer and Consumer can use Python logging via the logger configuration option. This works well as long as the program is running. Unfortunately, the last log messages get lost when the program exits. Those messages can only be seen if the logger feature is not used.
How to reproduce
logger.py:
import logging
import sys
from confluent_kafka import Producer, Consumer
import confluent_kafka
logger = logging.getLogger("kafka")
logger.setLevel(logging.DEBUG)
logger.addHandler(logging.StreamHandler())
c = Consumer({
"bootstrap.servers": "not-available-broker",
"group.id": "my-group",
"debug": "msg",
"logger": logger if sys.argv[-1] == "logger" else None,
})
c.poll(0)
c.close()
print("version:", confluent_kafka.version(), "lib-version:", confluent_kafka.libversion())output with logger:
$ ./logger.py logger
INIT [rdkafka#consumer-1] [thrd:app]: librdkafka v2.3.0 (0x20300ff) rdkafka#consumer-1 initialized (builtin.features gzip,snappy,ssl,sasl,regex,lz4,sasl_plain,sasl_scram,plugins,zstd,sasl_oauthbearer,http,oidc, STRIP STATIC_LINKING GCC GXX PKGCONFIG INSTALL GNULD LIBDL PLUGINS ZLIB SSL ZSTD CURL HDRHISTOGRAM SYSLOG SNAPPY SOCKEM SASL_SCRAM SASL_OAUTHBEARER OAUTHBEARER_OIDC CRC32C_HW, debug 0x40)
version: ('2.3.0', 33751040) lib-version: ('2.3.0', 33751295)
output without logger:
$ ./logger.py
%7|1709646389.068|INIT|rdkafka#consumer-1| [thrd:app]: librdkafka v2.3.0 (0x20300ff) rdkafka#consumer-1 initialized (builtin.features gzip,snappy,ssl,sasl,regex,lz4,sasl_plain,sasl_scram,plugins,zstd,sasl_oauthbearer,http,oidc, STRIP STATIC_LINKING GCC GXX PKGCONFIG INSTALL GNULD LIBDL PLUGINS ZLIB SSL ZSTD CURL HDRHISTOGRAM SYSLOG SNAPPY SOCKEM SASL_SCRAM SASL_OAUTHBEARER OAUTHBEARER_OIDC CRC32C_HW, debug 0x40)
%7|1709646389.068|DESTROY|rdkafka#consumer-1| [thrd:app]: Terminating instance (destroy flags none (0x0))
%7|1709646389.068|DESTROY|rdkafka#consumer-1| [thrd:main]: Destroy internal
%7|1709646389.068|DESTROY|rdkafka#consumer-1| [thrd:main]: Removing all topics
%3|1709646389.114|FAIL|rdkafka#consumer-1| [thrd:not-available-broker:9092/bootstrap]: not-available-broker:9092/bootstrap: Failed to resolve 'not-available-broker:9092': Temporary failure in name resolution (after 46ms in state CONNECT)
version: ('2.3.0', 33751040) lib-version: ('2.3.0', 33751295)
Expected behaviour
Both outputs should be similar.
Consumer.close() should really terminate the consumer. The Producer should provide a close() method to terminate it cleanly.
Checklist
Please provide the following information:
- confluent-kafka-python and librdkafka version (
confluent_kafka.version()andconfluent_kafka.libversion()): - Apache Kafka broker version: not relevant
- Client configuration:
{...} - Operating system: Ubuntu 22.04
- Provide client logs (with
'debug': '..'as necessary) - Provide broker log excerpts
- Critical issue