feat: Capability in dagger to consume from ACL enabled kafka clusters#195
Merged
Meghajit merged 14 commits intoraystack:mainraystack/dagger:mainfrom Oct 17, 2022
mayankrai09:mainCopy head branch name to clipboard
Merged
feat: Capability in dagger to consume from ACL enabled kafka clusters#195Meghajit merged 14 commits intoraystack:mainraystack/dagger:mainfrom mayankrai09:mainCopy head branch name to clipboard
Meghajit merged 14 commits intoraystack:mainraystack/dagger:mainfrom
mayankrai09:mainCopy head branch name to clipboard
Conversation
lavkesh
reviewed
Sep 23, 2022
Meghajit
reviewed
Sep 28, 2022
Meghajit
reviewed
Sep 28, 2022
Meghajit
reviewed
Sep 28, 2022
Meghajit
reviewed
Sep 28, 2022
Meghajit
reviewed
Sep 28, 2022
Member
|
@prakharmathur82 @mayankrai09 This seems like an important but advanced feature, which might be tricky to configure for new users. Hence I was thinking if we should include a separate page under the Maybe titled cc: @ravisuhag |
Meghajit
reviewed
Sep 28, 2022
dagger-core/src/main/java/io/odpf/dagger/core/source/config/StreamConfig.java
Show resolved
Hide resolved
Meghajit
reviewed
Sep 28, 2022
fix: added security page to describe Jaas config
feat: added basic validation for supported kafka acl configuration
Meghajit
reviewed
Oct 17, 2022
fix: added missing security page in sidebar
Meghajit
approved these changes
Oct 17, 2022
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
we are working on a centralized Kafka cluster that would contain data from other entities (Gojek/Gofin..etc). For compliance reasons, the cluster needs to be ACL enabled. Also, there is a need to run daggers that would consume from this ACL-enabled cluster.
We are planning to use
securtiy_protocol: SASL_PLAINTEXT
sasl_mechanism: SCRAM-SHA-512/SCRAM-SHA-256 [We took the decision to use SCRAM for the ease of adding new users]
Apart from these configs we would also need to have jaas-config in the consumer side containing the user credentials. We would be mounting the jaas-config to that particular flink cluster which would be consuming from ACL-enabled Kafka clusters.