Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly

Appearance settings

Commit a6badb7

Browse filesBrowse files
committed
fix: binary format version bump
The binary format of docarray_v2 has been changed in a non-backward-compatiable way. In order to tell the difference between docarray_v1 and docarray_v2, a bump in the wire format version is needed. Signed-off-by: yanlong.wang <yanlong.wang@naiver.org>
1 parent 176fd36 commit a6badb7
Copy full SHA for a6badb7

1 file changed

+11-3Lines changed: 11 additions & 3 deletions

File tree

Expand file treeCollapse file tree
Open diff view settings
Filter options
  • docarray/array/doc_list
Expand file treeCollapse file tree
Open diff view settings
Collapse file

‎docarray/array/doc_list/io.py‎

Copy file name to clipboardExpand all lines: docarray/array/doc_list/io.py
+11-3Lines changed: 11 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -546,11 +546,11 @@ def to_dataframe(self) -> 'pd.DataFrame':
546546
def _stream_header(self) -> bytes:
547547
# Binary format for streaming case
548548

549-
# V1 DocList streaming serialization format
550-
# | 1 byte | 8 bytes | 4 bytes | variable | 4 bytes | variable ...
549+
# V2 DocList streaming serialization format
550+
# | 1 byte | 8 bytes | 4 bytes | variable(docarray v2) | 4 bytes | variable(docarray v2) ...
551551

552552
# 1 byte (uint8)
553-
version_byte = b'\x01'
553+
version_byte = b'\x02'
554554
# 8 bytes (uint64)
555555
num_docs_as_bytes = len(self).to_bytes(8, 'big', signed=False)
556556
return version_byte + num_docs_as_bytes
@@ -597,6 +597,10 @@ def _load_binary_all(
597597
from docarray.utils._internal.progress_bar import _get_progressbar
598598

599599
# 1 byte (uint8)
600+
version_num = int.from_bytes(d[0:1], 'big', signed=False)
601+
if (version_num != 2):
602+
raise ValueError(f'Unsupported version number {version_num} in binary format, expected 2')
603+
600604
# 8 bytes (uint64)
601605
num_docs = int.from_bytes(d[1:9], 'big', signed=False)
602606

@@ -655,6 +659,10 @@ def _load_binary_stream(
655659
with file_ctx as f:
656660
version_numdocs_lendoc0 = f.read(9)
657661
# 1 byte (uint8)
662+
version_num = int.from_bytes(version_numdocs_lendoc0[0:1], 'big', signed=False)
663+
if (version_num != 2):
664+
raise ValueError(f'Unsupported version number {version_num} in binary format, expected 2')
665+
658666
# 8 bytes (uint64)
659667
num_docs = int.from_bytes(version_numdocs_lendoc0[1:9], 'big', signed=False)
660668

0 commit comments

Comments
0 (0)
Morty Proxy This is a proxified and sanitized view of the page, visit original site.