Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly

Appearance settings

Commit 72aa3b2

Browse filesBrowse files
feat: added support for all Weaviate HNSW configuration parameters (#341)
* feat: added support for all Weaviate HNSW configuration parameters * feat: added support for all Weaviate HNSW configuration parameters * feat: added support for all Weaviate HNSW configuration parameters * feat: added support for all Weaviate HNSW configuration parameters * feat: added support for all Weaviate HNSW configuration parameters * feat: added support for all Weaviate HNSW configuration parameters * docs: document new parameters Co-authored-by: Alaeddine Abdessalem <alaeddine-13@live.fr>
1 parent 02e8724 commit 72aa3b2
Copy full SHA for 72aa3b2

3 files changed

+56-12Lines changed: 56 additions & 12 deletions

File tree

Expand file treeCollapse file tree
Open diff view settings
Filter options
Expand file treeCollapse file tree
Open diff view settings
Collapse file

‎docarray/array/storage/weaviate/backend.py‎

Copy file name to clipboardExpand all lines: docarray/array/storage/weaviate/backend.py
+15Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -33,10 +33,18 @@ class WeaviateConfig:
3333
name: Optional[str] = None
3434
serialize_config: Dict = field(default_factory=dict)
3535
n_dim: Optional[int] = None # deprecated, not used anymore since weaviate 1.10
36+
# vectorIndexConfig parameters
3637
ef: Optional[int] = None
3738
ef_construction: Optional[int] = None
3839
timeout_config: Optional[Tuple[int, int]] = None
3940
max_connections: Optional[int] = None
41+
dynamic_ef_min: Optional[int] = None
42+
dynamic_ef_max: Optional[int] = None
43+
dynamic_ef_factor: Optional[int] = None
44+
vector_cache_max_objects: Optional[int] = None
45+
flat_search_cutoff: Optional[int] = None
46+
cleanup_interval_seconds: Optional[int] = None
47+
skip: Optional[bool] = None
4048

4149

4250
class BackendMixin(BaseBackendMixin):
@@ -120,6 +128,13 @@ def _get_schema_by_name(self, cls_name: str) -> Dict:
120128
'ef': self._config.ef,
121129
'efConstruction': self._config.ef_construction,
122130
'maxConnections': self._config.max_connections,
131+
'dynamicEfMin': self._config.dynamic_ef_min,
132+
'dynamicEfMax': self._config.dynamic_ef_max,
133+
'dynamicEfFactor': self._config.dynamic_ef_factor,
134+
'vectorCacheMaxObjects': self._config.vector_cache_max_objects,
135+
'flatSearchCutoff': self._config.flat_search_cutoff,
136+
'cleanupIntervalSeconds': self._config.cleanup_interval_seconds,
137+
'skip': self._config.skip,
123138
}
124139

125140
return {
Collapse file

‎docs/advanced/document-store/weaviate.md‎

Copy file name to clipboardExpand all lines: docs/advanced/document-store/weaviate.md
+19-11Lines changed: 19 additions & 11 deletions
  • Display the source diff
  • Display the rich diff
Original file line numberDiff line numberDiff line change
@@ -83,17 +83,25 @@ Other functions behave the same as in-memory DocumentArray.
8383

8484
The following configs can be set:
8585

86-
| Name | Description | Default |
87-
|--------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------|----------------------------------------------------|
88-
| `host` | Hostname of the Weaviate server | 'localhost' |
89-
| `port` | port of the Weaviate server | 8080 |
90-
| `protocol` | protocol to be used. Can be 'http' or 'https' | 'http' |
91-
| `name` | Weaviate class name; the class name of Weaviate object to presesent this DocumentArray | None |
92-
| `serialize_config` | [Serialization config of each Document](../../../fundamentals/document/serialization.md) | None |
93-
| `ef` | The size of the dynamic list for the nearest neighbors (used during the search). The higher ef is chosen, the more accurate, but also slower a search becomes. | `None`, defaults to the default value in Weaviate* |
94-
| `ef_construction` | The size of the dynamic list for the nearest neighbors (used during the construction). Controls index search speed/build speed tradeoff. | `None`, defaults to the default value in Weaviate* |
95-
| `timeout_config` | Set the timeout configuration for all requests to the Weaviate server. | `None`, defaults to the default value in Weaviate* |
96-
| `max_connections` | The maximum number of connections per element in all layers. | `None`, defaults to the default value in Weaviate* |
86+
| Name | Description | Default |
87+
|----------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|----------------------------------------------------|
88+
| `host` | Hostname of the Weaviate server | 'localhost' |
89+
| `port` | port of the Weaviate server | 8080 |
90+
| `protocol` | protocol to be used. Can be 'http' or 'https' | 'http' |
91+
| `name` | Weaviate class name; the class name of Weaviate object to presesent this DocumentArray | None |
92+
| `serialize_config` | [Serialization config of each Document](../../../fundamentals/document/serialization.md) | None |
93+
| `ef` | The size of the dynamic list for the nearest neighbors (used during the search). The higher ef is chosen, the more accurate, but also slower a search becomes. | `None`, defaults to the default value in Weaviate* |
94+
| `ef_construction` | The size of the dynamic list for the nearest neighbors (used during the construction). Controls index search speed/build speed tradeoff. | `None`, defaults to the default value in Weaviate* |
95+
| `timeout_config` | Set the timeout configuration for all requests to the Weaviate server. | `None`, defaults to the default value in Weaviate* |
96+
| `max_connections` | The maximum number of connections per element in all layers. | `None`, defaults to the default value in Weaviate* |
97+
| `dynamic_ef_min` | If using dynamic ef (set to -1), this value acts as a lower boundary. Even if the limit is small enough to suggest a lower value, ef will never drop below this value. This helps in keeping search accuracy high even when setting very low limits, such as 1, 2, or 3. | `None`, defaults to the default value in Weaviate* |
98+
| `dynamic_ef_max` | If using dynamic ef (set to -1), this value acts as an upper boundary. Even if the limit is large enough to suggest a lower value, ef will be capped at this value. This helps to keep search speed reasonable when retrieving massive search result sets, e.g. 500+. | `None`, defaults to the default value in Weaviate* |
99+
| `dynamic_ef_factor` | If using dynamic ef (set to -1), this value controls how ef is determined based on the given limit. E.g. with a factor of 8, ef will be set to 8*limit as long as this value is between the lower and upper boundary. It will be capped on either end, otherwise. | `None`, defaults to the default value in Weaviate* |
100+
| `vector_cache_max_objects` | For optimal search and import performance all previously imported vectors need to be held in memory. However, Weaviate also allows for limiting the number of vectors in memory. By default, when creating a new class, this limit is set to 2M objects. A disk lookup for a vector is orders of magnitudes slower than memory lookup, so the cache should be used sparingly. | `None`, defaults to the default value in Weaviate* |
101+
| `flat_search_cutoff` | Absolute number of objects configured as the threshold for a flat-search cutoff. If a filter on a filtered vector search matches fewer than the specified elements, the HNSW index is bypassed entirely and a flat (brute-force) search is performed instead. This can speed up queries with very restrictive filters considerably. Optional, defaults to 40000. Set to 0 to turn off flat-search cutoff entirely. | `None`, defaults to the default value in Weaviate* |
102+
| `cleanup_interval_seconds` | How often the async process runs that “repairs” the HNSW graph after deletes and updates. (Prior to the repair/cleanup process, deleted objects are simply marked as deleted, but still a fully connected member of the HNSW graph. After the repair has run, the edges are reassigned and the datapoints deleted for good). Typically this value does not need to be adjusted, but if deletes or updates are very frequent it might make sense to adjust the value up or down. (Higher value means it runs less frequently, but cleans up more in a single batch. Lower value means it runs more frequently, but might not be as efficient with each run). | `None`, defaults to the default value in Weaviate* |
103+
| `skip` | There are situations where it doesn’t make sense to vectorize a class. For example if the class is just meant as glue between two other class (consisting only of references) or if the class contains mostly duplicate elements (Note that importing duplicate vectors into HNSW is very expensive as the algorithm uses a check whether a candidate’s distance is higher than the worst candidate’s distance for an early exit condition. With (mostly) identical vectors, this early exit condition is never met leading to an exhaustive search on each import or query). In this case, you can skip indexing a vector all-together. To do so, set "skip" to "true". skip defaults to false; if not set to true, classes will be indexed normally. This setting is immutable after class initialization. | `None`, defaults to the default value in Weaviate* |
104+
97105

98106
*You can read more about the HNSW parameters and their default values [here](https://weaviate.io/developers/weaviate/current/vector-index-plugins/hnsw.html#how-to-use-hnsw-and-parameters)
99107

0 commit comments

Comments
0 (0)
Morty Proxy This is a proxified and sanitized view of the page, visit original site.