Skip to content

Navigation Menu

Sign in
Appearance settings
tansu-io

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly
Appearance settings

@tansu-io Tansu

Discussion options

shaeqahmed
Nov 20, 2025

Does Tansu support load balancing based on consistent hashing or balancing the incoming connections? What are the options supported and does system do anything to detect and balance hot partitions or to evenly spread the load of overpartitioned topics / large number of topics with a highly skewed load?
You must be logged in to vote

Replies: 1 comment

Comment options

shortishly
Nov 24, 2025
Maintainer

None at present. However, there is work in progress on authentication (#47) that could enable throttling of clients, and routing of requests depending on a QOS (with accompanying ACL/config). There is also a tansu proxy that could be used to route/layer requests:

tansu/tansu-proxy/src/lib.rs

Lines 174 to 200 in ad005c6

let meta = HijackLayer::new(
FrameApiKeyMatcher(MetadataRequest::KEY),
(
FrameRequestLayer::<MetadataRequest>::new(),
MapRequestLayer::new(move |request: MetadataRequest| {
MetadataRequest::default()
.topics(request.topics.map(|topics| {
topics
.into_iter()
.map(|topic| {
MetadataRequestTopic::default()
.name(topic.name)
.topic_id(topic.topic_id.or(Some(NULL_TOPIC_ID)))
})
.collect()
}))
.allow_auto_topic_creation(
request.allow_auto_topic_creation.or(Some(false)),
)
.include_cluster_authorized_operations(
request.include_cluster_authorized_operations,
)
.include_topic_authorized_operations(
request.include_topic_authorized_operations.or(Some(false)),
)
}),
MapResponseLayer::new(move |response: MetadataResponse| {

it currently adapts metadata/find_coordinator responses, but could provide routing instead (requests for topic A all go to broker B).

As all storage/metadata is delegated to PostgreSQL/S3/SQLite/..., it could be used to route client A to SQLite, B to PostgreSQL/etc.

One of the original concepts, was having lots of "small" single broker ephemeral clusters rather than having a single "big" cluster with all the issues of noisy neighbours/etc.

I am also looking at slatedb (#500) for read heavy topics on S3 that might benefit from a layer of caching.

Happy to discuss in more detail if you have particular questions.
You must be logged in to vote
0 replies
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
2 participants
@shaeqahmed @shortishly
Morty Proxy This is a proxified and sanitized view of the page, visit original site.