Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly

Appearance settings
Discussion options

GX Core's ephemeral Data Context (gx.get_context(mode="ephemeral")) is a natural fit for throwaway/cloud sandbox environments — disposable containers, hosted notebooks, and CI runners — because nothing is written to a persistent metadata store. Sharing a small reproducible pattern in case it's useful to others, and to ask whether a short docs note would be welcome.

Minimal pattern (pandas, in-memory)

import great_expectations as gx
import pandas as pd

context = gx.get_context(mode="ephemeral")  # nothing persists between runs
df = pd.DataFrame({"passenger_count": [1, 2, 3, 4]})

batch = (
    context.data_sources.add_pandas("pandas")
    .add_dataframe_asset("trips")
    .add_batch_definition_whole_dataframe("batch")
    .get_batch(batch_parameters={"dataframe": df})
)
result = batch.validate(
    gx.expectations.ExpectColumnValuesToBeBetween(column="passenger_count", min_value=1, max_value=6)
)
print(result.success)

Install fresh each run (pip install great_expectations), validate, and discard the environment on exit — no gx/ project directory committed, no persistent backend.

Running it across sandbox providers

Because GX itself is runtime-agnostic, the same script runs unchanged in any sandbox. I put together a small open-source demo with provider-agnostic runners behind one run(script) interface — verified end-to-end on islo.dev (cloud Firecracker microVM) and a local Apple container, with Daytona implemented and Modal/E2B planned:

https://github.com/zozo123/gx-sandbox-demo

(Disclosure: that's my own demo repo — sharing it as a worked example, not a product.)

Question

Would a short, provider-agnostic note on the Create a Data Context (Ephemeral) page — covering ephemeral mode for CI / disposable compute / no-persistent-filesystem use — be a welcome [DOCS] contribution? Happy to open a small PR if so. Also curious which other sandbox environments people run GX Core in.

You must be logged in to vote

Replies: 2 comments

Comment options

hey @zozo123 , thanks for sharing! Feel free to open a docs PR. The code snippet on the EphemeralDataContext setup page should remain DataContext focused, but you're right, it would be helpful to have some of the additional use cases enumerated.

You must be logged in to vote
0 replies
Comment options

Thanks @joshua-stauffer! Opened #11928 — it adds the use-case enumeration to the Ephemeral Data Context section and leaves the code snippet DataContext-focused as you suggested.

You must be logged in to vote
0 replies
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
2 participants
Morty Proxy This is a proxified and sanitized view of the page, visit original site.