Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly
Appearance settings

Commit 464f24d

Browse filesBrowse files
samsjaalaeddine-13JohannesMessner
authored
feat: add filter to match method (#378)
* feat: add filter to match method * fix: test function name fix Co-authored-by: AlaeddineAbdessalem <alaeddine-13@live.fr> * fix: delete filtering mention in docstring Co-authored-by: AlaeddineAbdessalem <alaeddine-13@live.fr> * feat: add documentation * fix: apply johannes docs suggestion Co-authored-by: Johannes Messner <44071807+JohannesMessner@users.noreply.github.com> Co-authored-by: AlaeddineAbdessalem <alaeddine-13@live.fr> Co-authored-by: Johannes Messner <44071807+JohannesMessner@users.noreply.github.com>
1 parent fef03f0 commit 464f24d
Copy full SHA for 464f24d

6 files changed

+147-5Lines changed: 147 additions & 5 deletions

File tree

Expand file treeCollapse file tree
Open diff view settings
Filter options
Expand file treeCollapse file tree
Open diff view settings
Collapse file

‎docarray/array/mixins/match.py‎

Copy file name to clipboardExpand all lines: docarray/array/mixins/match.py
+4-1Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
from typing import Optional, Union, Callable, Tuple, TYPE_CHECKING
1+
from typing import Optional, Union, Callable, Tuple, TYPE_CHECKING, Dict
22

33
if TYPE_CHECKING:
44
import numpy as np
@@ -20,6 +20,7 @@ def match(
2020
metric_name: Optional[str] = None,
2121
batch_size: Optional[int] = None,
2222
exclude_self: bool = False,
23+
filter: Optional[Dict] = None,
2324
only_id: bool = False,
2425
use_scipy: bool = False,
2526
device: str = 'cpu',
@@ -50,6 +51,7 @@ def match(
5051
elements. When `darray` is big, this can significantly speedup the computation.
5152
:param exclude_self: if set, Documents in ``darray`` with same ``id`` as the left-hand values will not be
5253
considered as matches.
54+
:param filter: filter query used for pre-filtering
5355
:param only_id: if set, then returning matches will only contain ``id``
5456
:param use_scipy: if set, use ``scipy`` as the computation backend. Note, ``scipy`` does not support distance
5557
on sparse matrix.
@@ -76,6 +78,7 @@ def match(
7678
metric_name=metric_name,
7779
batch_size=batch_size,
7880
exclude_self=exclude_self,
81+
filter=filter,
7982
only_id=only_id,
8083
use_scipy=use_scipy,
8184
device=device,
Collapse file

‎docs/advanced/document-store/annlite.md‎

Copy file name to clipboardExpand all lines: docs/advanced/document-store/annlite.md
+1-1Lines changed: 1 addition & 1 deletion
  • Display the source diff
  • Display the rich diff
Original file line numberDiff line numberDiff line change
@@ -49,7 +49,7 @@ The following configs can be set:
4949

5050
*You can check the default values in [the AnnLite source code](https://github.com/jina-ai/annlite/blob/main/annlite/core/index/hnsw/index.py)
5151

52-
52+
(annlite-filter)=
5353
## Vector search with filter
5454

5555
Search with `.find` can be restricted by user-defined filters.
Collapse file

‎docs/advanced/document-store/qdrant.md‎

Copy file name to clipboardExpand all lines: docs/advanced/document-store/qdrant.md
+1-1Lines changed: 1 addition & 1 deletion
  • Display the source diff
  • Display the rich diff
Original file line numberDiff line numberDiff line change
@@ -133,7 +133,7 @@ print(da.find(np.random.random(D), limit=10))
133133
<DocumentArray (length=10) at 4917906896>
134134
```
135135

136-
136+
(qdrant-filter)=
137137
## Vector search with filter
138138

139139
Search with `.find` can be restricted by user-defined filters. Such filters can be constructed following the guidelines
Collapse file

‎docs/advanced/document-store/weaviate.md‎

Copy file name to clipboardExpand all lines: docs/advanced/document-store/weaviate.md
+1-1Lines changed: 1 addition & 1 deletion
  • Display the source diff
  • Display the rich diff
Original file line numberDiff line numberDiff line change
@@ -169,7 +169,7 @@ print(results[0].text)
169169
Persist Documents with Weaviate.
170170
```
171171

172-
172+
(weaviate-filter)=
173173
## Vector search with filter
174174

175175
Search with `.find` can be restricted by user-defined filters. Such filters can be constructed following the guidelines
Collapse file

‎docs/fundamentals/documentarray/matching.md‎

Copy file name to clipboardExpand all lines: docs/fundamentals/documentarray/matching.md
+22Lines changed: 22 additions & 0 deletions
  • Display the source diff
  • Display the rich diff
Original file line numberDiff line numberDiff line change
@@ -155,6 +155,28 @@ Note that framework is auto-chosen based on the type of `.embeddings`. For examp
155155

156156
By default `A.match(B)` will copy the top-K matched Documents from B to `A.matches`. When these matches are big, copying them can be time-consuming. In this case, one can leverage `.match(..., only_id=True)` to keep only {attr}`~docarray.Document.id`.
157157

158+
### Pre filtering
159+
160+
Both `match` and `find` support pre-filtering by passing a `filter` argument to the method.
161+
162+
Pre-filtering is an advanced approximate nearest neighbors feature that allows to efficiently retrieve the nearest vectors
163+
that respect the filtering condition.
164+
165+
In contrast, post-filtering in the naive approach where you first retrieve the
166+
nearest neighbors and then discard all the candidates that do not respect the filter condition.
167+
168+
````{admonition} Pre-filtering is not available for in-memory backend
169+
:class: caution
170+
By default a DocumentArray will use the in-memory backend which does not support pre-filtering
171+
```
172+
````
173+
174+
You can find example on how to use the pre-filtering here:
175+
176+
- {ref}`ANNLite <annlite-filter>`
177+
- {ref}`Weaviate <weaviate-filter>`
178+
- {ref}`Qdrant <qdrant-filter>`
179+
158180

159181
### GPU support
160182

Collapse file

‎tests/unit/array/mixins/test_match.py‎

Copy file name to clipboardExpand all lines: tests/unit/array/mixins/test_match.py
+118-1Lines changed: 118 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -10,7 +10,7 @@
1010
from scipy.spatial.distance import cdist as scipy_cdist
1111

1212
from docarray import Document, DocumentArray
13-
from docarray.array.storage.weaviate import WeaviateConfig
13+
import operator
1414

1515

1616
@pytest.fixture()
@@ -577,3 +577,120 @@ def test_match_ensure_scores_unique():
577577
for m in query.matches:
578578
assert m.scores['euclidean'].value >= previous_score
579579
previous_score = m.scores['euclidean'].value
580+
581+
582+
numeric_operators_annlite = {
583+
'$gte': operator.ge,
584+
'$gt': operator.gt,
585+
'$lte': operator.le,
586+
'$lt': operator.lt,
587+
'$eq': operator.eq,
588+
'$neq': operator.ne,
589+
}
590+
591+
numeric_operators_weaviate = {
592+
'GreaterThanEqual': operator.ge,
593+
'GreaterThan': operator.gt,
594+
'LessThanEqual': operator.le,
595+
'LessThan': operator.lt,
596+
'Equal': operator.eq,
597+
'NotEqual': operator.ne,
598+
}
599+
600+
601+
numeric_operators_qdrant = {
602+
'gte': operator.ge,
603+
'gt': operator.gt,
604+
'lte': operator.le,
605+
'lt': operator.lt,
606+
'eq': operator.eq,
607+
'neq': operator.ne,
608+
}
609+
610+
611+
@pytest.mark.parametrize(
612+
'storage,filter_gen,numeric_operators,operator',
613+
[
614+
*[
615+
tuple(
616+
[
617+
'weaviate',
618+
lambda operator, threshold: {
619+
'path': ['price'],
620+
'operator': operator,
621+
'valueInt': threshold,
622+
},
623+
numeric_operators_weaviate,
624+
operator,
625+
]
626+
)
627+
for operator in numeric_operators_weaviate.keys()
628+
],
629+
*[
630+
tuple(
631+
[
632+
'qdrant',
633+
lambda operator, threshold: {
634+
'must': [{'key': 'price', 'range': {operator: threshold}}]
635+
},
636+
numeric_operators_qdrant,
637+
operator,
638+
]
639+
)
640+
for operator in ['gte', 'gt', 'lte', 'lt']
641+
],
642+
*[
643+
tuple(
644+
[
645+
'qdrant',
646+
lambda operator, threshold: {
647+
'must': [{'key': 'price', 'value': {operator: threshold}}]
648+
},
649+
numeric_operators_qdrant,
650+
operator,
651+
]
652+
)
653+
for operator in ['eq', 'neq']
654+
],
655+
*[
656+
tuple(
657+
[
658+
'annlite',
659+
lambda operator, threshold: {'price': {operator: threshold}},
660+
numeric_operators_annlite,
661+
operator,
662+
]
663+
)
664+
for operator in numeric_operators_annlite.keys()
665+
],
666+
],
667+
)
668+
def test_match_pre_filtering(
669+
storage, filter_gen, operator, numeric_operators, start_storage
670+
):
671+
n_dim = 128
672+
da = DocumentArray(
673+
storage=storage, config={'n_dim': n_dim, 'columns': [('price', 'int')]}
674+
)
675+
676+
da.extend(
677+
[
678+
Document(id=f'r{i}', embedding=np.random.rand(n_dim), tags={'price': i})
679+
for i in range(50)
680+
]
681+
)
682+
thresholds = [10, 20, 30]
683+
684+
for threshold in thresholds:
685+
686+
filter = filter_gen(operator, threshold)
687+
688+
doc = Document(embedding=np.random.rand(n_dim))
689+
doc.match(da, filter=filter)
690+
691+
assert all(
692+
[
693+
numeric_operators[operator](r.tags['price'], threshold)
694+
for r in doc.matches
695+
]
696+
)

0 commit comments

Comments
0 (0)
Morty Proxy This is a proxified and sanitized view of the page, visit original site.