You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: docs/advanced/document-store/index.md
+40-17Lines changed: 40 additions & 17 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -13,7 +13,8 @@ extend
13
13
benchmark
14
14
```
15
15
16
-
Documents inside a DocumentArray can live in a [document store](https://en.wikipedia.org/wiki/Document-oriented_database) instead of in memory, e.g. in SQLite, Redis. Comparing to the in-memory storage, the benefit of using an external store is often about longer persistence and faster retrieval.
16
+
Documents inside a DocumentArray can live in a [document store](https://en.wikipedia.org/wiki/Document-oriented_database) instead of in memory, e.g. in SQLite, Redis.
17
+
Comparing to the in-memory storage, the benefit of using an external store is often about longer persistence and faster retrieval.
17
18
18
19
The look-and-feel of a DocumentArray with external store is **almost the same** as a regular in-memory DocumentArray. This allows users to easily switch between backends under the same DocArray idiom.
19
20
@@ -24,24 +25,44 @@ from docarray import DocumentArray, Document
24
25
25
26
da = DocumentArray(storage='sqlite', config={'connection': 'example.db'})
26
27
27
-
da.append(Document())
28
+
with da:
29
+
da.append(Document())
28
30
da.summary()
29
31
```
30
32
31
33
```text
32
-
Documents Summary
33
-
34
-
Length 1
35
-
Homogenous Documents True
36
-
Common Attributes ('id',)
37
-
38
-
Attributes Summary
39
-
40
-
Attribute Data type #Unique values Has empty value
Note that `da` was modified inside a `with` statement. This context manager ensures that the the `DocumentArray` indices,
61
+
which allow users to access the `DocumentArray` by position (allowing statements such as `da[1]`),
62
+
are properly mapped and saved to the storage backend.
63
+
This is the recommended default usage to modify a DocumentArray that lives on a document store to avoid
64
+
unexpected behaviors that can yield to, for example, inaccessible elements by position.
65
+
45
66
46
67
Creating, retrieving, updating, deleting Documents are identical to the regular {ref}`DocumentArray<documentarray>`. All DocumentArray methods such as `.summary()`, `.embed()`, `.plot_embeddings()` should work out of the box.
47
68
@@ -136,7 +157,8 @@ The output is:
136
157
0
137
158
```
138
159
139
-
Looks like `db` is not really up-to-date with `da`. This is true and false. True in the sense that `1` is not `0`, number speaks by itself. False in the sense that, the Document is already written to the storage backend. You just can't see it.
160
+
Looks like `db` is not really up-to-date with `da`. This is true and false. True in the sense that `1` is not `0`, number speaks by itself.
161
+
False in the sense that, the Document is already written to the storage backend. You just can't see it.
140
162
141
163
To prove it does persist, run the following code snippet multiple times and you will see the length is increasing one at a time:
Simply put, the reason of this behavior is that certain meta information **not synced immediately** to the backend on *every* operation; it would be very costly to do so. As a consequence, your multiple references to the same backend would look different if they are written in one code block as the example above.
173
+
Simply put, the reason of this behavior is that certain meta information **not synced immediately** to the backend on *every* operation; it would be very costly to do so.
174
+
As a consequence, your multiple references to the same backend would look different if they are written in one code block as the example above.
152
175
153
176
To solve this problem, simply use `with` statement and use DocumentArray as a context manager. The last example can be refactored into the following:
0 commit comments