Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly
Appearance settings
Discussion options

kdkavanagh
Jul 18, 2025

Is it explicitly required that dictionary values be unique? Currently the at least the C++ API DictionaryMemoTable will deduplicate entries on insert, however I can think of usecases where multiple underlying index values should map to the same dictionary value. Given the O(1) lookup from index->value, this would require tolerating duplicates in the dictionary array.

Example of creating non-unique dicts: #25845

Curious if this is a hard-requirement, or if it's an implementation detail open to change.
You must be logged in to vote
Answered by assignUser Jul 18, 2025

From the Arrow format spec:

Note that a dictionary is permitted to contain duplicate values or nulls:

data VarBinary (dictionary-encoded)
   index_type: Int32
   values: [0, 1, 3, 1, 4, 2]

dictionary
   type: VarBinary
   values: ['foo', 'bar', 'baz', 'foo', null]

The null count of such arrays is dictated only by the validity bitmap of its indices, irrespective of any null values in the dictionary.
Arrow Columnar Format – Dictionary-encoded Layout

(Thanks Docs chat bot!)

So forcing the de-duplication seems to go against the spec and there for is a bug?

View full answer

Replies: 4 comments

Comment options

felipecrv
Jul 18, 2025
Collaborator

Code processing dictionary-encoded arrays should not assume values to be unique.
You must be logged in to vote
0 replies
Comment options

assignUser
Jul 18, 2025
Collaborator

From the Arrow format spec:

Note that a dictionary is permitted to contain duplicate values or nulls:

data VarBinary (dictionary-encoded)
   index_type: Int32
   values: [0, 1, 3, 1, 4, 2]

dictionary
   type: VarBinary
   values: ['foo', 'bar', 'baz', 'foo', null]

The null count of such arrays is dictated only by the validity bitmap of its indices, irrespective of any null values in the dictionary.
Arrow Columnar Format – Dictionary-encoded Layout

(Thanks Docs chat bot!)

So forcing the de-duplication seems to go against the spec and there for is a bug?
You must be logged in to vote
0 replies
Answer selected by kdkavanagh
Comment options

raulcd
Jul 21, 2025
Collaborator

Is there an issue opened to track here? Otherwise, @kdkavanagh can you open an issue, please?
You must be logged in to vote
0 replies
Comment options

kdkavanagh
Jul 21, 2025
Author

Created #47151, thanks
You must be logged in to vote
0 replies
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Category
🙏
Q&A
Labels
None yet
4 participants
@kdkavanagh @felipecrv @raulcd @assignUser
Morty Proxy This is a proxified and sanitized view of the page, visit original site.