Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly
Appearance settings

Commit 52fde78

Browse filesBrowse files
davidbpalaeddine-13
davidbp
and
alaeddine-13
authored
feat: annlite find with filter and no query vector (#401)
* feat: use annlite find with only filter * chore: upgrade annlite version * docs: remove typo Co-authored-by: Alaeddine Abdessalem <alaeddine-13@live.fr>
1 parent aad1e1f commit 52fde78
Copy full SHA for 52fde78

4 files changed

+91-5Lines changed: 91 additions & 5 deletions

File tree

Expand file treeCollapse file tree
Open diff view settings
Filter options
Expand file treeCollapse file tree
Open diff view settings
Collapse file

‎docarray/array/storage/annlite/find.py‎

Copy file name to clipboardExpand all lines: docarray/array/storage/annlite/find.py
+21-1Lines changed: 21 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -8,7 +8,8 @@
88

99
if TYPE_CHECKING:
1010
import numpy as np
11-
from .... import DocumentArray
11+
12+
from .... import DocumentArray
1213

1314

1415
class FindMixin:
@@ -41,3 +42,22 @@ def _find(
4142
)
4243

4344
return match_docs
45+
46+
def _filter(
47+
self,
48+
filter: Dict,
49+
limit: Optional[Union[int, float]] = 20,
50+
only_id: bool = False,
51+
) -> 'DocumentArray':
52+
"""Returns a subset of documents by filtering by the given filter (`Annlite` filter).
53+
54+
:param filter: the input filter to apply in each stored document
55+
:param limit: the number of results to get for each query document in search.
56+
:param only_id: if set, then returning matches will only contain ``id``
57+
:return: a `DocumentArray` containing the `Document` objects that verify the filter.
58+
"""
59+
60+
docs = self._annlite.filter(
61+
filter=filter, limit=limit, include_metadata=not only_id
62+
)
63+
return DocumentArray(docs)
Collapse file

‎docs/advanced/document-store/annlite.md‎

Copy file name to clipboardExpand all lines: docs/advanced/document-store/annlite.md
+56-1Lines changed: 56 additions & 1 deletion
  • Display the source diff
  • Display the rich diff
Original file line numberDiff line numberDiff line change
@@ -55,7 +55,62 @@ The following configs can be set:
5555
Search with `.find` can be restricted by user-defined filters.
5656
Filters can be constructed following the guidelines provided in [the AnnLite source repository](https://github.com/jina-ai/annlite).
5757

58-
### Example of `.find` with a filter
58+
### Example of `.find` with a filter only
59+
60+
61+
Consider you store Documents with a certain tag `price` into annlite and you want to retrieve all Documents
62+
with `price` lower or equal to some `max_price` value.
63+
64+
65+
You can index such Documents as follows:
66+
```python
67+
from docarray import Document, DocumentArray
68+
import numpy as np
69+
70+
n_dim = 3
71+
da = DocumentArray(
72+
storage='annlite',
73+
config={
74+
'n_dim': n_dim,
75+
'columns': [('price', 'float')],
76+
},
77+
)
78+
79+
with da:
80+
da.extend([Document(id=f'r{i}', tags={'price': i}) for i in range(10)])
81+
82+
print('\nIndexed Prices:\n')
83+
for price in da[:, 'tags__price']:
84+
print(f'\t price={price}')
85+
```
86+
87+
Then you can retrieve all documents whose price is lower than or equal to `max_price` by applying the following
88+
filter:
89+
90+
```python
91+
max_price = 3
92+
n_limit = 4
93+
94+
filter = {'price': {'$lte': max_price}}
95+
results = da.find(filter=filter)
96+
97+
print('\n Returned examples that verify filter "price at most 3":\n')
98+
for price in results[:, 'tags__price']:
99+
print(f'\t price={price}')
100+
```
101+
102+
This would print
103+
104+
```
105+
Returned examples that satisfy condition "price at most 3":
106+
107+
price=0
108+
price=1
109+
price=2
110+
price=3
111+
```
112+
113+
### Example of `.find` with query vector and filter
59114

60115
Consider Documents with embeddings `[0,0,0]` up to ` [9,9,9]` where the document with embedding `[i,i,i]`
61116
has as tag `price` with value `i`. We can create such example with the following code:
Collapse file

‎setup.py‎

Copy file name to clipboardExpand all lines: setup.py
+3-3Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -64,15 +64,15 @@
6464
'uvicorn',
6565
'strawberry-graphql',
6666
'weaviate-client~=3.3.0',
67-
'annlite>=0.3.0',
67+
'annlite>=0.3.2',
6868
'qdrant-client~=0.7.3',
6969
'elasticsearch>=8.2.0',
7070
],
7171
'qdrant': [
7272
'qdrant-client~=0.7.3',
7373
],
7474
'annlite': [
75-
'annlite>=0.3.0',
75+
'annlite>=0.3.2',
7676
],
7777
'weaviate': [
7878
'weaviate-client~=3.3.0',
@@ -100,7 +100,7 @@
100100
'jupyterlab',
101101
'transformers>=4.16.2',
102102
'weaviate-client~=3.3.0',
103-
'annlite>=0.3.0',
103+
'annlite>=0.3.2',
104104
'elasticsearch>=8.2.0',
105105
'jina',
106106
],
Collapse file

‎tests/unit/array/mixins/test_find.py‎

Copy file name to clipboardExpand all lines: tests/unit/array/mixins/test_find.py
+11Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -392,6 +392,17 @@ def test_search_pre_filtering(
392392
)
393393
for operator in ['gt', 'gte', 'lt', 'lte']
394394
],
395+
*[
396+
tuple(
397+
[
398+
'annlite',
399+
lambda operator, threshold: {'price': {operator: threshold}},
400+
numeric_operators_annlite,
401+
operator,
402+
]
403+
)
404+
for operator in numeric_operators_annlite.keys()
405+
],
395406
],
396407
)
397408
def test_filtering(storage, filter_gen, operator, numeric_operators, start_storage):

0 commit comments

Comments
0 (0)
Morty Proxy This is a proxified and sanitized view of the page, visit original site.