Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly
Appearance settings

Commit 3f6b942

Browse filesBrowse files
jirimoravcikjirimoravcik
authored
feat: Add documentation for StorageManager and StorageClientManager, open_* methods in Actor (#34)
1 parent 387143c commit 3f6b942
Copy full SHA for 3f6b942

File tree

Expand file treeCollapse file tree

4 files changed

+137
-18
lines changed
Filter options
Expand file treeCollapse file tree

4 files changed

+137
-18
lines changed

‎docs/docs.md

Copy file name to clipboardExpand all lines: docs/docs.md
+52-3Lines changed: 52 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -146,7 +146,23 @@ That’s useful if you want to use the client as a different Apify user than the
146146

147147
#### async classmethod open_dataset(dataset_id_or_name=None, \*, force_cloud=False)
148148

149-
TODO: docs.
149+
Open a dataset.
150+
151+
Datasets are used to store structured data where each object stored has the same attributes,
152+
such as online store products or real estate offers.
153+
The actual data is stored either on the local filesystem or in the Apify cloud.
154+
155+
* **Parameters**
156+
157+
* **dataset_id_or_name** (`str`, *optional*) – ID or name of the dataset to be opened.
158+
If not provided, the method returns the default dataset associated with the actor run.
159+
160+
* **force_cloud** (`bool`, *optional*) – If set to True then the Apify cloud storage is always used.
161+
This way it is possible to combine local and cloud storage.
162+
163+
* **Returns**
164+
165+
An instance of the Dataset class for the given ID or name.
150166

151167
* **Return type**
152168

@@ -156,7 +172,23 @@ TODO: docs.
156172

157173
#### async classmethod open_key_value_store(key_value_store_id_or_name=None, \*, force_cloud=False)
158174

159-
TODO: docs.
175+
Open a key-value store.
176+
177+
Key-value stores are used to store records or files, along with their MIME content type.
178+
The records are stored and retrieved using a unique key.
179+
The actual data is stored either on a local filesystem or in the Apify cloud.
180+
181+
* **Parameters**
182+
183+
* **key_value_store_id_or_name** (`str`, *optional*) – ID or name of the key-value store to be opened.
184+
If not provided, the method returns the default key-value store associated with the actor run.
185+
186+
* **force_cloud** (`bool`, *optional*) – If set to True then the Apify cloud storage is always used.
187+
This way it is possible to combine local and cloud storage.
188+
189+
* **Returns**
190+
191+
An instance of the KeyValueStore class for the given ID or name.
160192

161193
* **Return type**
162194

@@ -166,7 +198,24 @@ TODO: docs.
166198

167199
#### async classmethod open_request_queue(request_queue_id_or_name=None, \*, force_cloud=False)
168200

169-
TODO: docs.
201+
Open a request queue.
202+
203+
Request queue represents a queue of URLs to crawl, which is stored either on local filesystem or in the Apify cloud.
204+
The queue is used for deep crawling of websites, where you start with several URLs and then
205+
recursively follow links to other pages. The data structure supports both breadth-first
206+
and depth-first crawling orders.
207+
208+
* **Parameters**
209+
210+
* **request_queue_id_or_name** (`str`, *optional*) – ID or name of the request queue to be opened.
211+
If not provided, the method returns the default request queue associated with the actor run.
212+
213+
* **force_cloud** (`bool`, *optional*) – If set to True then the Apify cloud storage is always used.
214+
This way it is possible to combine local and cloud storage.
215+
216+
* **Returns**
217+
218+
An instance of the RequestQueue class for the given ID or name.
170219

171220
* **Return type**
172221

‎src/apify/actor.py

Copy file name to clipboardExpand all lines: src/apify/actor.py
+47-3Lines changed: 47 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -478,7 +478,22 @@ def _get_storage_client(self, force_cloud: bool) -> Optional[ApifyClientAsync]:
478478

479479
@classmethod
480480
async def open_dataset(cls, dataset_id_or_name: Optional[str] = None, *, force_cloud: bool = False) -> Dataset:
481-
"""TODO: docs."""
481+
"""Open a dataset.
482+
483+
Datasets are used to store structured data where each object stored has the same attributes,
484+
such as online store products or real estate offers.
485+
The actual data is stored either on the local filesystem or in the Apify cloud.
486+
487+
Args:
488+
dataset_id_or_name (str, optional): ID or name of the dataset to be opened.
489+
If not provided, the method returns the default dataset associated with the actor run.
490+
force_cloud (bool, optional): If set to `True` then the Apify cloud storage is always used.
491+
This way it is possible to combine local and cloud storage.
492+
493+
Returns:
494+
Dataset: An instance of the `Dataset` class for the given ID or name.
495+
496+
"""
482497
return await cls._get_default_instance().open_dataset(dataset_id_or_name=dataset_id_or_name, force_cloud=force_cloud)
483498

484499
async def _open_dataset_internal(self, dataset_id_or_name: Optional[str] = None, *, force_cloud: bool = False) -> Dataset:
@@ -488,7 +503,21 @@ async def _open_dataset_internal(self, dataset_id_or_name: Optional[str] = None,
488503

489504
@classmethod
490505
async def open_key_value_store(cls, key_value_store_id_or_name: Optional[str] = None, *, force_cloud: bool = False) -> KeyValueStore:
491-
"""TODO: docs."""
506+
"""Open a key-value store.
507+
508+
Key-value stores are used to store records or files, along with their MIME content type.
509+
The records are stored and retrieved using a unique key.
510+
The actual data is stored either on a local filesystem or in the Apify cloud.
511+
512+
Args:
513+
key_value_store_id_or_name (str, optional): ID or name of the key-value store to be opened.
514+
If not provided, the method returns the default key-value store associated with the actor run.
515+
force_cloud (bool, optional): If set to `True` then the Apify cloud storage is always used.
516+
This way it is possible to combine local and cloud storage.
517+
518+
Returns:
519+
KeyValueStore: An instance of the `KeyValueStore` class for the given ID or name.
520+
"""
492521
return await cls._get_default_instance().open_key_value_store(key_value_store_id_or_name=key_value_store_id_or_name, force_cloud=force_cloud)
493522

494523
async def _open_key_value_store_internal(self, key_value_store_id_or_name: Optional[str] = None, *, force_cloud: bool = False) -> KeyValueStore:
@@ -498,7 +527,22 @@ async def _open_key_value_store_internal(self, key_value_store_id_or_name: Optio
498527

499528
@classmethod
500529
async def open_request_queue(cls, request_queue_id_or_name: Optional[str] = None, *, force_cloud: bool = False) -> RequestQueue:
501-
"""TODO: docs."""
530+
"""Open a request queue.
531+
532+
Request queue represents a queue of URLs to crawl, which is stored either on local filesystem or in the Apify cloud.
533+
The queue is used for deep crawling of websites, where you start with several URLs and then
534+
recursively follow links to other pages. The data structure supports both breadth-first
535+
and depth-first crawling orders.
536+
537+
Args:
538+
request_queue_id_or_name (str, optional): ID or name of the request queue to be opened.
539+
If not provided, the method returns the default request queue associated with the actor run.
540+
force_cloud (bool, optional): If set to `True` then the Apify cloud storage is always used.
541+
This way it is possible to combine local and cloud storage.
542+
543+
Returns:
544+
RequestQueue: An instance of the `RequestQueue` class for the given ID or name.
545+
"""
502546
return await cls._get_default_instance().open_request_queue(request_queue_id_or_name=request_queue_id_or_name, force_cloud=force_cloud)
503547

504548
async def _open_request_queue_internal(

‎src/apify/storage_client_manager.py

Copy file name to clipboardExpand all lines: src/apify/storage_client_manager.py
+12-4Lines changed: 12 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -7,7 +7,7 @@
77

88

99
class StorageClientManager:
10-
"""TODO: docs."""
10+
"""A class for managing storage clients."""
1111

1212
_config: Configuration
1313

@@ -16,18 +16,26 @@ class StorageClientManager:
1616
_default_instance: Optional['StorageClientManager'] = None
1717

1818
def __init__(self) -> None:
19-
"""TODO: docs."""
19+
"""Create a `StorageClientManager` instance."""
2020
self._config = Configuration.get_global_configuration()
2121
self._client = MemoryStorage(persist_storage=self._config.persist_storage)
2222

2323
@classmethod
2424
def get_storage_client(cls) -> Union[ApifyClientAsync, MemoryStorage]:
25-
"""TODO: docs."""
25+
"""Get the current storage client instance.
26+
27+
Returns:
28+
ApifyClientAsync or MemoryStorage: The current storage client instance.
29+
"""
2630
return cls._get_default_instance()._client
2731

2832
@classmethod
2933
def set_storage_client(cls, client: Union[ApifyClientAsync, MemoryStorage]) -> None:
30-
"""TODO: docs."""
34+
"""Set the storage client.
35+
36+
Args:
37+
client (ApifyClientAsync or MemoryStorage): The instance of a storage client.
38+
"""
3139
cls._get_default_instance()._client = client
3240

3341
@classmethod

‎src/apify/storages/storage_manager.py

Copy file name to clipboardExpand all lines: src/apify/storages/storage_manager.py
+26-8Lines changed: 26 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -15,7 +15,7 @@
1515

1616

1717
class Storage(Protocol[T]):
18-
"""TODO: Docs."""
18+
"""A protocol defining common interface for storage classes."""
1919

2020
@classmethod
2121
def _create_instance(cls, storage_id_or_name: str, client: Union[ApifyClientAsync, MemoryStorage]) -> T: # noqa: U100
@@ -33,16 +33,14 @@ async def _purge_default_storages(client: Union[ApifyClientAsync, MemoryStorage]
3333

3434

3535
class StorageManager:
36-
"""TODO: docs."""
36+
"""A class for managing storages."""
3737

3838
_default_instance: Optional['StorageManager'] = None
3939
_cache: Dict[Type[Storage], Dict[str, Storage]]
40-
_config: Configuration
4140

4241
def __init__(self) -> None:
43-
"""TODO: docs."""
42+
"""Create a `StorageManager` instance."""
4443
self._cache = {}
45-
self._config = Configuration.get_global_configuration()
4644

4745
@classmethod
4846
def _get_default_instance(cls) -> 'StorageManager':
@@ -59,9 +57,23 @@ async def open_storage(
5957
client: Optional[Union[ApifyClientAsync, MemoryStorage]] = None,
6058
config: Optional[Configuration] = None,
6159
) -> T:
62-
"""TODO: docs."""
60+
"""Open a storage of the given class, or return a cached storage object if it was opened before.
61+
62+
Opens a new storage (`Dataset`, `KeyValueStore`, or `RequestQueue`) with the given ID or name.
63+
Returns the cached storage object if the storage was opened before.
64+
65+
Args:
66+
storage_class (Type[Dataset] or Type[KeyValueStore] or Type[RequestQueue]): Class of the storage to be opened.
67+
storage_id_or_name (str, optional): ID or name of the storage to be opened. If omitted, an unnamed storage will be opened.
68+
client (ApifyClientAsync or MemoryStorage, optional): The storage client which should be used in the storage.
69+
If omitted, the default client will be used.
70+
config (Configuration, optional): The actor configuration to be used in this call. If omitted, the global configuration will be used.
71+
72+
Returns:
73+
An instance of the storage given by `storage_class`.
74+
"""
6375
storage_manager = StorageManager._get_default_instance()
64-
used_config = config or storage_manager._config
76+
used_config = config or Configuration.get_global_configuration()
6577
used_client = client or StorageClientManager.get_storage_client()
6678

6779
# Create cache for the given storage class if missing
@@ -93,7 +105,13 @@ async def open_storage(
93105

94106
@classmethod
95107
async def close_storage(cls, storage_class: Type[Storage], id: str, name: Optional[str]) -> None:
96-
"""TODO: docs."""
108+
"""Close the given storage by removing it from the cache.
109+
110+
Args:
111+
storage_class (Type[Dataset] or Type[KeyValueStore] or Type[RequestQueue]): Class of the storage to be closed.
112+
id (str): ID of the storage to be closed.
113+
name (str, optional): Name of the storage to be closed.
114+
"""
97115
storage_manager = StorageManager._get_default_instance()
98116
del storage_manager._cache[storage_class][id]
99117
if name is not None:

0 commit comments

Comments
0 (0)
Morty Proxy This is a proxified and sanitized view of the page, visit original site.