Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly
Appearance settings

Add a proof-of-concept for "Observer-like" batch loading #148

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
bbakerman merged 11 commits into from
May 19, 2024
Merged

Add a proof-of-concept for "Observer-like" batch loading #148

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
11 commits
Select commit Hold shift + click to select a range
6e30220
Add a proof-of-concept for "Observer-like" batch loading
AlexandreCarlton May 12, 2024
2032e33
Merge remote-tracking branch 'origin/master' into observer-batch-load…
AlexandreCarlton May 18, 2024
6b5a732
Eliminate *BatchObserver in favour of Publisher
AlexandreCarlton May 18, 2024
68d7f54
Use internal Assertions over Java's raw assert
AlexandreCarlton May 18, 2024
a3132b7
Remove handling of Throwable passed into onNext
AlexandreCarlton May 18, 2024
fbeffae
Expose `new*DataLoader` methods for *PublisherBatchLoader
AlexandreCarlton May 18, 2024
b2a662d
Copy/tweak original/ DataLoader tests for publisher equivalents
AlexandreCarlton May 18, 2024
0d0b2f8
Rename '*PublisherBatchLoader' to 'BatchPublisher'
AlexandreCarlton May 18, 2024
14002f6
Ensure DataLoaderSubscriber is only called by one thread
AlexandreCarlton May 18, 2024
0f303a8
Document Subscriber#onNext invocation order
AlexandreCarlton May 18, 2024
ce115fd
Merge branch 'reactive-streams-branch' into observer-batch-loader-pro…
bbakerman May 19, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion 2 build.gradle
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,6 @@ def getDevelopmentVersion() {
version
}


def releaseVersion = System.env.RELEASE_VERSION
version = releaseVersion ? releaseVersion : getDevelopmentVersion()
group = 'com.graphql-java'
Expand Down Expand Up @@ -76,6 +75,7 @@ dependencies {
testImplementation 'org.awaitility:awaitility:2.0.0'
testImplementation 'io.projectreactor:reactor-core:3.6.6'
testImplementation 'com.github.ben-manes.caffeine:caffeine:2.9.0'
testImplementation 'io.projectreactor:reactor-core:3.6.6'
}

task sourcesJar(type: Jar) {
Expand Down
22 changes: 22 additions & 0 deletions 22 src/main/java/org/dataloader/BatchPublisher.java
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
package org.dataloader;

import org.reactivestreams.Subscriber;

import java.util.List;

/**
* A function that is invoked for batch loading a stream of data values indicated by the provided list of keys.
* <p>
* The function will call the provided {@link Subscriber} to process the values it has retrieved to allow
* the future returned by {@link DataLoader#load(Object)} to complete as soon as the individual value is available
* (rather than when all values have been retrieved).
* <p>
* <b>NOTE:</b> It is <b>required </b> that {@link Subscriber#onNext(V)} is invoked on each value in the same order as
* the provided keys.
Copy link
Member

@bbakerman bbakerman May 19, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe some more explanatory text here as its so important

*
* @param <K> type parameter indicating the type of keys to use for data load requests.
* @param <V> type parameter indicating the type of values returned
*/
public interface BatchPublisher<K, V> {
void load(List<K> keys, Subscriber<V> subscriber);
}
Copy link
Member

@bbakerman bbakerman May 19, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I like the name now that we have the reactive streams Subscriber in play. The name makes more sense now - its a Publisher of batched keys

Copy link
Member

@bbakerman bbakerman May 19, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

And less interfaces needed

12 changes: 12 additions & 0 deletions 12 src/main/java/org/dataloader/BatchPublisherWithContext.java
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
package org.dataloader;

import org.reactivestreams.Subscriber;

import java.util.List;

/**
* An {@link BatchPublisher} with a {@link BatchLoaderEnvironment} provided as an extra parameter to {@link #load}.
*/
public interface BatchPublisherWithContext<K, V> {
void load(List<K> keys, Subscriber<V> subscriber, BatchLoaderEnvironment environment);
}
268 changes: 268 additions & 0 deletions 268 src/main/java/org/dataloader/DataLoaderFactory.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -278,6 +278,274 @@ public static <K, V> DataLoader<K, V> newMappedDataLoaderWithTry(MappedBatchLoad
return mkDataLoader(batchLoadFunction, options);
}

/**
* Creates new DataLoader with the specified batch loader function and default options
* (batching, caching and unlimited batch size).
*
* @param batchLoadFunction the batch load function to use
* @param <K> the key type
* @param <V> the value type
*
* @return a new DataLoader
*/
public static <K, V> DataLoader<K, V> newPublisherDataLoader(BatchPublisher<K, V> batchLoadFunction) {
return newPublisherDataLoader(batchLoadFunction, null);
}

/**
* Creates new DataLoader with the specified batch loader function with the provided options
*
* @param batchLoadFunction the batch load function to use
* @param options the options to use
* @param <K> the key type
* @param <V> the value type
*
* @return a new DataLoader
*/
public static <K, V> DataLoader<K, V> newPublisherDataLoader(BatchPublisher<K, V> batchLoadFunction, DataLoaderOptions options) {
return mkDataLoader(batchLoadFunction, options);
}

/**
* Creates new DataLoader with the specified batch loader function and default options
* (batching, caching and unlimited batch size) where the batch loader function returns a list of
* {@link org.dataloader.Try} objects.
* <p>
* If it's important you to know the exact status of each item in a batch call and whether it threw exceptions then
* you can use this form to create the data loader.
* <p>
* Using Try objects allows you to capture a value returned or an exception that might
* have occurred trying to get a value. .
*
* @param batchLoadFunction the batch load function to use that uses {@link org.dataloader.Try} objects
* @param <K> the key type
* @param <V> the value type
*
* @return a new DataLoader
*/
public static <K, V> DataLoader<K, V> newPublisherDataLoaderWithTry(BatchPublisher<K, Try<V>> batchLoadFunction) {
return newPublisherDataLoaderWithTry(batchLoadFunction, null);
}

/**
* Creates new DataLoader with the specified batch loader function and with the provided options
* where the batch loader function returns a list of
* {@link org.dataloader.Try} objects.
*
* @param batchLoadFunction the batch load function to use that uses {@link org.dataloader.Try} objects
* @param options the options to use
* @param <K> the key type
* @param <V> the value type
*
* @return a new DataLoader
*
* @see #newDataLoaderWithTry(BatchLoader)
*/
public static <K, V> DataLoader<K, V> newPublisherDataLoaderWithTry(BatchPublisher<K, Try<V>> batchLoadFunction, DataLoaderOptions options) {
return mkDataLoader(batchLoadFunction, options);
}

/**
* Creates new DataLoader with the specified batch loader function and default options
* (batching, caching and unlimited batch size).
*
* @param batchLoadFunction the batch load function to use
* @param <K> the key type
* @param <V> the value type
*
* @return a new DataLoader
*/
public static <K, V> DataLoader<K, V> newPublisherDataLoader(BatchPublisherWithContext<K, V> batchLoadFunction) {
return newPublisherDataLoader(batchLoadFunction, null);
}

/**
* Creates new DataLoader with the specified batch loader function with the provided options
*
* @param batchLoadFunction the batch load function to use
* @param options the options to use
* @param <K> the key type
* @param <V> the value type
*
* @return a new DataLoader
*/
public static <K, V> DataLoader<K, V> newPublisherDataLoader(BatchPublisherWithContext<K, V> batchLoadFunction, DataLoaderOptions options) {
return mkDataLoader(batchLoadFunction, options);
}

/**
* Creates new DataLoader with the specified batch loader function and default options
* (batching, caching and unlimited batch size) where the batch loader function returns a list of
* {@link org.dataloader.Try} objects.
* <p>
* If it's important you to know the exact status of each item in a batch call and whether it threw exceptions then
* you can use this form to create the data loader.
* <p>
* Using Try objects allows you to capture a value returned or an exception that might
* have occurred trying to get a value. .
*
* @param batchLoadFunction the batch load function to use that uses {@link org.dataloader.Try} objects
* @param <K> the key type
* @param <V> the value type
*
* @return a new DataLoader
*/
public static <K, V> DataLoader<K, V> newPublisherDataLoaderWithTry(BatchPublisherWithContext<K, Try<V>> batchLoadFunction) {
return newPublisherDataLoaderWithTry(batchLoadFunction, null);
}

/**
* Creates new DataLoader with the specified batch loader function and with the provided options
* where the batch loader function returns a list of
* {@link org.dataloader.Try} objects.
*
* @param batchLoadFunction the batch load function to use that uses {@link org.dataloader.Try} objects
* @param options the options to use
* @param <K> the key type
* @param <V> the value type
*
* @return a new DataLoader
*
* @see #newPublisherDataLoaderWithTry(BatchPublisher)
*/
public static <K, V> DataLoader<K, V> newPublisherDataLoaderWithTry(BatchPublisherWithContext<K, Try<V>> batchLoadFunction, DataLoaderOptions options) {
return mkDataLoader(batchLoadFunction, options);
}

/**
* Creates new DataLoader with the specified batch loader function and default options
* (batching, caching and unlimited batch size).
*
* @param batchLoadFunction the batch load function to use
* @param <K> the key type
* @param <V> the value type
*
* @return a new DataLoader
*/
public static <K, V> DataLoader<K, V> newMappedPublisherDataLoader(MappedBatchPublisher<K, V> batchLoadFunction) {
return newMappedPublisherDataLoader(batchLoadFunction, null);
}

/**
* Creates new DataLoader with the specified batch loader function with the provided options
*
* @param batchLoadFunction the batch load function to use
* @param options the options to use
* @param <K> the key type
* @param <V> the value type
*
* @return a new DataLoader
*/
public static <K, V> DataLoader<K, V> newMappedPublisherDataLoader(MappedBatchPublisher<K, V> batchLoadFunction, DataLoaderOptions options) {
return mkDataLoader(batchLoadFunction, options);
}

/**
* Creates new DataLoader with the specified batch loader function and default options
* (batching, caching and unlimited batch size) where the batch loader function returns a list of
* {@link org.dataloader.Try} objects.
* <p>
* If it's important you to know the exact status of each item in a batch call and whether it threw exceptions then
* you can use this form to create the data loader.
* <p>
* Using Try objects allows you to capture a value returned or an exception that might
* have occurred trying to get a value. .
*
* @param batchLoadFunction the batch load function to use that uses {@link org.dataloader.Try} objects
* @param <K> the key type
* @param <V> the value type
*
* @return a new DataLoader
*/
public static <K, V> DataLoader<K, V> newMappedPublisherDataLoaderWithTry(MappedBatchPublisher<K, Try<V>> batchLoadFunction) {
return newMappedPublisherDataLoaderWithTry(batchLoadFunction, null);
}

/**
* Creates new DataLoader with the specified batch loader function and with the provided options
* where the batch loader function returns a list of
* {@link org.dataloader.Try} objects.
*
* @param batchLoadFunction the batch load function to use that uses {@link org.dataloader.Try} objects
* @param options the options to use
* @param <K> the key type
* @param <V> the value type
*
* @return a new DataLoader
*
* @see #newDataLoaderWithTry(BatchLoader)
*/
public static <K, V> DataLoader<K, V> newMappedPublisherDataLoaderWithTry(MappedBatchPublisher<K, Try<V>> batchLoadFunction, DataLoaderOptions options) {
return mkDataLoader(batchLoadFunction, options);
}

/**
* Creates new DataLoader with the specified batch loader function and default options
* (batching, caching and unlimited batch size).
*
* @param batchLoadFunction the batch load function to use
* @param <K> the key type
* @param <V> the value type
*
* @return a new DataLoader
*/
public static <K, V> DataLoader<K, V> newMappedPublisherDataLoader(MappedBatchPublisherWithContext<K, V> batchLoadFunction) {
return newMappedPublisherDataLoader(batchLoadFunction, null);
}

/**
* Creates new DataLoader with the specified batch loader function with the provided options
*
* @param batchLoadFunction the batch load function to use
* @param options the options to use
* @param <K> the key type
* @param <V> the value type
*
* @return a new DataLoader
*/
public static <K, V> DataLoader<K, V> newMappedPublisherDataLoader(MappedBatchPublisherWithContext<K, V> batchLoadFunction, DataLoaderOptions options) {
return mkDataLoader(batchLoadFunction, options);
}

/**
* Creates new DataLoader with the specified batch loader function and default options
* (batching, caching and unlimited batch size) where the batch loader function returns a list of
* {@link org.dataloader.Try} objects.
* <p>
* If it's important you to know the exact status of each item in a batch call and whether it threw exceptions then
* you can use this form to create the data loader.
* <p>
* Using Try objects allows you to capture a value returned or an exception that might
* have occurred trying to get a value. .
*
* @param batchLoadFunction the batch load function to use that uses {@link org.dataloader.Try} objects
* @param <K> the key type
* @param <V> the value type
*
* @return a new DataLoader
*/
public static <K, V> DataLoader<K, V> newMappedPublisherDataLoaderWithTry(MappedBatchPublisherWithContext<K, Try<V>> batchLoadFunction) {
return newMappedPublisherDataLoaderWithTry(batchLoadFunction, null);
}

/**
* Creates new DataLoader with the specified batch loader function and with the provided options
* where the batch loader function returns a list of
* {@link org.dataloader.Try} objects.
*
* @param batchLoadFunction the batch load function to use that uses {@link org.dataloader.Try} objects
* @param options the options to use
* @param <K> the key type
* @param <V> the value type
*
* @return a new DataLoader
*
* @see #newMappedPublisherDataLoaderWithTry(MappedBatchPublisher)
*/
public static <K, V> DataLoader<K, V> newMappedPublisherDataLoaderWithTry(MappedBatchPublisherWithContext<K, Try<V>> batchLoadFunction, DataLoaderOptions options) {
return mkDataLoader(batchLoadFunction, options);
}

static <K, V> DataLoader<K, V> mkDataLoader(Object batchLoadFunction, DataLoaderOptions options) {
return new DataLoader<>(batchLoadFunction, options);
}
Expand Down
Loading
Morty Proxy This is a proxified and sanitized view of the page, visit original site.