Source Source¶

Source subclasses are responsible for generating an integer value given an open source project’s source URL.

class dffml.source.source.BaseSource(config: Optional[Type[BaseConfig]])[source]¶

Abstract base class for all sources. New sources must be derived from this class and implement the records method.

class dffml.source.source.BaseSourceContext(parent: BaseSource)[source]¶

abstract async record(key: str)[source]¶

Get a record from the source or add it if it doesn’t exist.

Examples

>>> import asyncio
>>> from dffml import *
>>>
>>> async def main():
...     async with MemorySource(records=[Record("example", data=dict(features=dict(dead="beef")))]) as source:
...         # Open, update, and close
...         async with source() as ctx:
...             example = await ctx.record("example")
...             # Let's also try calling `record` for a record that doesnt exist.
...             one = await ctx.record("one")
...             await ctx.update(one)
...             async for record in ctx.records():
...                 print(record.export())
>>>
>>> asyncio.run(main())
{'key': 'example', 'features': {'dead': 'beef'}, 'extra': {}}
{'key': 'one', 'extra': {}}

abstract async records() → AsyncIterator[Record][source]¶

Returns a list of records retrieved from self.src

Examples

>>> import asyncio
>>> from dffml import *
>>>
>>> async def main():
...     async with MemorySource(records=[Record("example", data=dict(features=dict(dead="beef")))]) as source:
...         async with source() as ctx:
...             async for record in ctx.records():
...                 print(record.export())
>>>
>>> asyncio.run(main())
{'key': 'example', 'features': {'dead': 'beef'}, 'extra': {}}

abstract async update(record: Record)[source]¶

Updates a record for a source

Examples

>>> import asyncio
>>> from dffml import *
>>>
>>> async def main():
...     async with MemorySource(records=[]) as source:
...         # Open, update, and close
...         async with source() as ctx:
...             example = Record("one", data=dict(features=dict(feed="face")))
...             # ... Update one into our records ...
...             await ctx.update(example)
...             # Let's check out our records after calling `record` and `update`.
...             async for record in ctx.records():
...                 print(record.export())
>>>
>>> asyncio.run(main())
{'key': 'one', 'features': {'feed': 'face'}, 'extra': {}}

exception dffml.source.source.NoRecordsWithMatchingFeatures[source]¶

Raised when SourcesContext.with_features() was called but no records were yielded.

class dffml.source.source.NoRecordsWithMatchingFeaturesHelper[source]¶

Helper to print a diff of the features requested and the features available in unittest diff format

failureException¶

alias of NoRecordsWithMatchingFeatures

class dffml.source.source.Sources(*args)[source]¶

CONTEXT¶

alias of SourcesContext

SINGLETON¶

alias of BaseSource

class dffml.source.source.SourcesContext(parent: AsyncContextManagerList)[source]¶

async record(key: str)[source]¶

Retrieve and or register record will all sources

async records(validation: Optional[Callable[[Record], bool]] = None) → AsyncIterator[Record][source]¶

Retrieves records from all sources

async update(record: Record)[source]¶

Updates a record for a source

async with_features(features: List[str]) → AsyncIterator[Record][source]¶

Returns all records which have the requested features

class dffml.source.source.SubsetSources(*args: BaseSource, keys: Optional[List[str]] = None)[source]¶

Restricts access to a subset of records during iteration based on their keys.

CONTEXT¶

alias of SubsetSourcesContext

class dffml.source.source.SubsetSourcesContext(parent: AsyncContextManagerList)[source]¶

async records(validation: Optional[Callable[[Record], bool]] = None) → AsyncIterator[Record][source]¶

Retrieves records from all sources

class dffml.source.source.ValidationSources(validation: Callable[[Record], bool], *args: BaseSource)[source]¶

Restricts access to a subset of records during iteration based on a validation function.

CONTEXT¶

alias of ValidationSourcesContext

class dffml.source.source.ValidationSourcesContext(parent: AsyncContextManagerList)[source]¶

async records(validation: Optional[Callable[[Record], bool]] = None) → AsyncIterator[Record][source]¶

Retrieves records from all sources