Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly
Appearance settings

Commit 14526db

Browse filesBrowse files
hanxiaoalaeddine-13
hanxiao
and
alaeddine-13
authored
feat: push meta data along with docarray 0.16 (#490)
* feat: push meta data along with docarray * feat: push meta data along with docarray * feat: push meta data along with docarray * feat: push meta data along with docarray * feat: push meta data along with docarray (#491) * feat: push meta data along with docarray * feat: push meta data along with docarray * feat: push meta data along with docarray * feat: push meta data along with docarray * feat: push meta data along with docarray * feat: push meta data along with docarray * feat: push meta data along with docarray * chore: login not required * Revert "chore: login not required" This reverts commit 7b008c1. * refactor: reuse common code * chore: rename client * docs: document cloud operations * chore: login not required * chore: apply suggestions * refactor: reuse code * chore: better naming Co-authored-by: Alaeddine Abdessalem <alaeddine-13@live.fr>
1 parent ea2a7a8 commit 14526db
Copy full SHA for 14526db

11 files changed

+281-48Lines changed: 281 additions & 48 deletions

File tree

Expand file treeCollapse file tree
Open diff view settings
Filter options
Expand file treeCollapse file tree
Open diff view settings
Collapse file

‎.github/workflows/cd.yml‎

Copy file name to clipboardExpand all lines: .github/workflows/cd.yml
+2Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -56,6 +56,8 @@ jobs:
5656
-v -s -m "not gpu" ${{ matrix.test-path }}
5757
echo "::set-output name=codecov_flag::docarray"
5858
timeout-minutes: 30
59+
env:
60+
JINA_AUTH_TOKEN: "${{ secrets.JINA_AUTH_TOKEN }}"
5961
- name: Check codecov file
6062
id: check_files
6163
uses: andstor/file-existence-action@v1
Collapse file

‎.github/workflows/ci.yml‎

Copy file name to clipboardExpand all lines: .github/workflows/ci.yml
+2Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -145,6 +145,8 @@ jobs:
145145
-v -s -m "not gpu" ${{ matrix.test-path }}
146146
echo "::set-output name=codecov_flag::docarray"
147147
timeout-minutes: 30
148+
env:
149+
JINA_AUTH_TOKEN: "${{ secrets.JINA_AUTH_TOKEN }}"
148150
- name: Check codecov file
149151
id: check_files
150152
uses: andstor/file-existence-action@v1
Collapse file

‎docarray/array/mixins/io/pushpull.py‎

Copy file name to clipboardExpand all lines: docarray/array/mixins/io/pushpull.py
+147-12Lines changed: 147 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -1,24 +1,159 @@
1+
import json
12
import os
3+
import os.path
24
import warnings
5+
from collections import Counter
36
from pathlib import Path
4-
from typing import Dict, Type, TYPE_CHECKING, Optional
7+
from typing import Dict, Type, TYPE_CHECKING, List, Optional, Any
58

6-
from docarray.helper import get_request_header, __cache_path__
9+
import hubble
10+
from hubble import Client as HubbleClient
11+
from hubble.client.endpoints import EndpointsV2
12+
13+
14+
from docarray.helper import get_request_header, __cache_path__, _get_array_info
715

816
if TYPE_CHECKING:
917
from docarray.typing import T
1018

1119

20+
def _get_length_from_summary(summary: List[Dict]) -> Optional[int]:
21+
"""Get the length from summary."""
22+
for item in summary:
23+
if 'Length' == item['name']:
24+
return item['value']
25+
26+
1227
class PushPullMixin:
1328
"""Transmitting :class:`DocumentArray` via Jina Cloud Service"""
1429

1530
_max_bytes = 4 * 1024 * 1024 * 1024
1631

32+
@staticmethod
33+
def cloud_list(show_table: bool = False) -> List[str]:
34+
"""List all available arrays in the cloud.
35+
36+
:param show_table: if true, show the table of the arrays.
37+
:returns: List of available DocumentArray's names.
38+
"""
39+
from rich import print
40+
41+
result = []
42+
from rich.table import Table
43+
from rich import box
44+
45+
resp = HubbleClient(jsonify=True).list_artifacts(
46+
filter={'type': 'documentArray'}, sort={'createdAt': 1}
47+
)
48+
49+
table = Table(
50+
title=f'You have {resp["meta"]["total"]} DocumentArray on the cloud',
51+
box=box.SIMPLE,
52+
highlight=True,
53+
)
54+
table.add_column('Name')
55+
table.add_column('Length')
56+
table.add_column('Access')
57+
table.add_column('Created at', justify='center')
58+
table.add_column('Updated at', justify='center')
59+
60+
for da in resp['data']:
61+
result.append(da['name'])
62+
63+
table.add_row(
64+
da['name'],
65+
str(_get_length_from_summary(da['metaData'].get('summary', []))),
66+
da['visibility'],
67+
da['createdAt'],
68+
da['updatedAt'],
69+
)
70+
71+
if show_table:
72+
print(table)
73+
return result
74+
75+
@staticmethod
76+
def cloud_delete(name: str) -> None:
77+
"""
78+
Delete a DocumentArray from the cloud.
79+
:param name: the name of the DocumentArray to delete.
80+
"""
81+
HubbleClient(jsonify=True).delete_artifact(name=name)
82+
83+
def _get_raw_summary(self) -> List[Dict[str, Any]]:
84+
(
85+
is_homo,
86+
_nested_in,
87+
_nested_items,
88+
attr_counter,
89+
all_attrs_names,
90+
) = _get_array_info(self)
91+
92+
items = [
93+
dict(
94+
name='Type',
95+
value=self.__class__.__name__,
96+
description='The type of the DocumentArray',
97+
),
98+
dict(
99+
name='Length',
100+
value=len(self),
101+
description='The length of the DocumentArray',
102+
),
103+
dict(
104+
name='Homogenous Documents',
105+
value=is_homo,
106+
description='Whether all documents are of the same structure, attributes',
107+
),
108+
dict(
109+
name='Common Attributes',
110+
value=list(attr_counter.items())[0][0] if attr_counter else None,
111+
description='The common attributes of all documents',
112+
),
113+
dict(
114+
name='Has nested Documents in',
115+
value=tuple(_nested_in),
116+
description='The field that contains nested Documents',
117+
),
118+
dict(
119+
name='Multimodal dataclass',
120+
value=all(d.is_multimodal for d in self),
121+
description='Whether all documents are multimodal',
122+
),
123+
dict(
124+
name='Subindices', value=tuple(getattr(self, '_subindices', {}).keys())
125+
),
126+
]
127+
128+
items.append(
129+
dict(
130+
name='Inspect attributes',
131+
value=_nested_items,
132+
description='Quick overview of attributes of all documents',
133+
)
134+
)
135+
136+
storage_infos = self._get_storage_infos()
137+
_nested_items = []
138+
if storage_infos:
139+
for k, v in storage_infos.items():
140+
_nested_items.append(dict(name=k, value=v))
141+
items.append(
142+
dict(
143+
name='Storage backend',
144+
value=_nested_items,
145+
description='Quick overview of the Document Store',
146+
)
147+
)
148+
149+
return items
150+
17151
def push(
18152
self,
19153
name: str,
20154
show_progress: bool = False,
21155
public: bool = True,
156+
branding: Optional[Dict] = None,
22157
) -> Dict:
23158
"""Push this DocumentArray object to Jina Cloud which can be later retrieved via :meth:`.push`
24159
@@ -33,6 +168,7 @@ def push(
33168
:param show_progress: if to show a progress bar on pulling
34169
:param public: by default anyone can pull a DocumentArray if they know its name.
35170
Setting this to False will allow only the creator to pull it. This feature of course you to login first.
171+
:param branding: a dict of branding information to be sent to Jina Cloud. {"icon": "emoji", "background": "#fff"}
36172
"""
37173
import requests
38174

@@ -47,11 +183,14 @@ def push(
47183
'name': name,
48184
'type': 'documentArray',
49185
'public': public,
186+
'metaData': json.dumps(
187+
{'summary': self._get_raw_summary(), 'branding': branding},
188+
sort_keys=True,
189+
),
50190
}
51191
)
52192

53193
headers = {'Content-Type': ctype, **get_request_header()}
54-
import hubble
55194

56195
auth_token = hubble.get_token()
57196
if auth_token:
@@ -98,11 +237,9 @@ def _get_chunk(_batch):
98237
yield _tail
99238

100239
with pbar:
101-
from hubble import Client
102-
from hubble.client.endpoints import EndpointsV2
103240

104241
response = requests.post(
105-
Client()._base_url + EndpointsV2.upload_artifact,
242+
HubbleClient()._base_url + EndpointsV2.upload_artifact,
106243
data=gen(),
107244
headers=headers,
108245
)
@@ -133,17 +270,12 @@ def pull(
133270

134271
headers = {}
135272

136-
import hubble
137-
138273
auth_token = hubble.get_token()
139274

140275
if auth_token:
141276
headers['Authorization'] = f'token {auth_token}'
142277

143-
from hubble import Client
144-
from hubble.client.endpoints import EndpointsV2
145-
146-
url = Client()._base_url + EndpointsV2.download_artifact + f'?name={name}'
278+
url = HubbleClient()._base_url + EndpointsV2.download_artifact + f'?name={name}'
147279
response = requests.get(url, headers=headers)
148280

149281
if response.ok:
@@ -183,3 +315,6 @@ def pull(
183315
fp.write(_source.content)
184316

185317
return r
318+
319+
cloud_push = push
320+
cloud_pull = pull
Collapse file

‎docarray/array/mixins/plot.py‎

Copy file name to clipboardExpand all lines: docarray/array/mixins/plot.py
+12-26Lines changed: 12 additions & 26 deletions
Original file line numberDiff line numberDiff line change
@@ -11,6 +11,8 @@
1111

1212
import numpy as np
1313

14+
from docarray.helper import _get_array_info
15+
1416

1517
class PlotMixin:
1618
"""Helper functions for plotting the arrays."""
@@ -37,44 +39,28 @@ def summary(self):
3739
tables = []
3840
console = Console()
3941

40-
all_attrs = self._get_attributes('non_empty_fields')
41-
# remove underscore attribute
42-
all_attrs = [tuple(vv for vv in v if not vv.startswith('_')) for v in all_attrs]
43-
attr_counter = Counter(all_attrs)
42+
(
43+
is_homo,
44+
_nested_in,
45+
_nested_items,
46+
attr_counter,
47+
all_attrs_names,
48+
) = _get_array_info(self)
4449

4550
table = Table(box=box.SIMPLE, highlight=True)
4651
table.show_header = False
4752
table.add_row('Type', self.__class__.__name__)
4853
table.add_row('Length', str(len(self)))
49-
is_homo = len(attr_counter) == 1
5054
table.add_row('Homogenous Documents', str(is_homo))
5155

52-
all_attrs_names = set(v for k in all_attrs for v in k)
53-
_nested_in = []
54-
if 'chunks' in all_attrs_names:
55-
_nested_in.append('chunks')
56-
57-
if 'matches' in all_attrs_names:
58-
_nested_in.append('matches')
59-
6056
if _nested_in:
6157
table.add_row('Has nested Documents in', str(tuple(_nested_in)))
6258

6359
if is_homo:
6460
table.add_row('Common Attributes', str(list(attr_counter.items())[0][0]))
65-
else:
66-
for _a, _n in attr_counter.most_common():
67-
if _n == 1:
68-
_doc_text = f'{_n} Document has'
69-
else:
70-
_doc_text = f'{_n} Documents have'
71-
if len(_a) == 1:
72-
_text = f'{_doc_text} one attribute'
73-
elif len(_a) == 0:
74-
_text = f'{_doc_text} no attribute'
75-
else:
76-
_text = f'{_doc_text} attributes'
77-
table.add_row(_text, str(_a))
61+
62+
for item in _nested_items:
63+
table.add_row(item['name'], item['value'])
7864

7965
is_multimodal = all(d.is_multimodal for d in self)
8066
table.add_row('Multimodal dataclass', str(is_multimodal))
Collapse file

‎docarray/helper.py‎

Copy file name to clipboardExpand all lines: docarray/helper.py
+41-1Lines changed: 41 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -6,7 +6,11 @@
66
import uuid
77
import warnings
88
from os.path import expanduser
9-
from typing import Any, Dict, Optional, Sequence, Tuple, Union
9+
from typing import Any, Dict, Optional, Sequence, Tuple, Union, TYPE_CHECKING
10+
from collections import Counter
11+
12+
if TYPE_CHECKING:
13+
from docarray import DocumentArray
1014

1115
__resources_path__ = os.path.join(
1216
os.path.dirname(
@@ -455,3 +459,39 @@ def _safe_cast_int(value: Union[str, int, float]) -> int:
455459
if isinstance(value, float) and not value.is_integer():
456460
raise ValueError(f"Can't safely cast {value} to an int")
457461
return int(value)
462+
463+
464+
def _get_array_info(da: 'DocumentArray'):
465+
all_attrs = da._get_attributes('non_empty_fields')
466+
# remove underscore attribute
467+
all_attrs = [tuple(vv for vv in v if not vv.startswith('_')) for v in all_attrs]
468+
attr_counter = Counter(all_attrs)
469+
470+
all_attrs_names = set(v for k in all_attrs for v in k)
471+
_nested_in = []
472+
if 'chunks' in all_attrs_names:
473+
_nested_in.append('chunks')
474+
475+
if 'matches' in all_attrs_names:
476+
_nested_in.append('matches')
477+
478+
is_homo = len(attr_counter) == 1
479+
480+
_nested_items = []
481+
if not is_homo:
482+
for n_attributes, n_docs in attr_counter.most_common():
483+
if n_docs == 1:
484+
_doc_text = f'{n_docs} Document has'
485+
else:
486+
_doc_text = f'{n_docs} Documents have'
487+
if len(n_attributes) == 1:
488+
_text = f'{_doc_text} one attribute'
489+
elif len(n_attributes) == 0:
490+
_text = f'{_doc_text} no attribute'
491+
else:
492+
_text = f'{_doc_text} attributes'
493+
_nested_items.append(
494+
dict(name=_text, value=str(n_attributes), description='')
495+
)
496+
497+
return is_homo, _nested_in, _nested_items, attr_counter, all_attrs_names
Collapse file

‎docs/fundamentals/documentarray/serialization.md‎

Copy file name to clipboardExpand all lines: docs/fundamentals/documentarray/serialization.md
+23Lines changed: 23 additions & 0 deletions
  • Display the source diff
  • Display the rich diff
Original file line numberDiff line numberDiff line change
@@ -393,3 +393,26 @@ The maximum size of an upload is 4GB under the `protocol='protobuf'` and `compre
393393

394394
To avoid unnecessary download when upstream DocumentArray is unchanged, you can add `DocumentArray.pull(..., local_cache=True)`.
395395

396+
Furthermore, it is possible to list all `DocumentArray` objects stored on the cloud using:
397+
```python
398+
DocumentArray.cloud_list(show_table=True)
399+
```
400+
401+
```text
402+
You have 1 DocumentArray on the cloud
403+
404+
Name Length Access Created at Updated at
405+
────────────────────────────────────────────────────────────────────────────────
406+
da123 10 public 2022-09-15T07:14:54.256Z 2022-09-15T07:14:54.256Z
407+
408+
['da123']
409+
```
410+
411+
```{tip}
412+
Use parameter `show_table` to show table summarizing information about DocumentArrays in the cloud.
413+
```
414+
415+
It is also possible to delete DocumentArray objects in the cloud using:
416+
```python
417+
DocumentArray.cloud_delete('da123')
418+
```

0 commit comments

Comments
0 (0)
Morty Proxy This is a proxified and sanitized view of the page, visit original site.