feat: Support LangChain orchestration client streaming #711

KavithaSiva · May 7, 2025

Context

Closes SAP/ai-sdk-js-backlog#259.

What this PR does and why it is needed

This PR introduces streaming support for Langchain Orchestration client.

Usage:

const orchestrationConfig: LangchainOrchestrationModuleConfig = {
  llm: {
    model_name: 'gpt-4o'
  },
  templating: {
    template: [
      {
        role: 'user',
        content: 'Write a 100 word explanation about {{?topic}}'
      }
    ]
  }
};

const client = new OrchestrationClient(orchestrationConfig);
const stream =  client.stream([], {
  inputParams: {
    topic: 'SAP Cloud SDK and its capabilities'
  },
  signal: controller.signal
});

for await (const chunk of stream) {
 //Access the streamed chunk here
}

packages/langchain/src/orchestration/util.ts

…:SAP/ai-sdk-js into langchain-orchestration-client-streaming

KavithaSiva · May 12, 2025

~~Adding more tests in a follow-up PR.~~

ZhongpinWang

Suggestions for improvement. Also please do implement tests already in this PR to avoid changes later in case they don't work as expected. 😄

.changeset/chilly-steaks-divide.md

packages/langchain/src/orchestration/client.ts

packages/langchain/src/orchestration/orchestration-message-chunk.ts

packages/langchain/src/orchestration/util.ts

sample-code/src/langchain-orchestration.ts

Co-authored-by: Zhongpin Wang <zhongpin.wang@sap.com>

sample-code/src/index.ts

packages/langchain/src/orchestration/util.ts

ZhongpinWang

I see refactoring needed for the big *_streamResponseChunks function. It would make the code more readable and testable.

packages/langchain/src/orchestration/client.ts

sample-code/src/langchain-orchestration.ts

…:SAP/ai-sdk-js into langchain-orchestration-client-streaming

tomfrenken

Looks like a lot was already considered, still there are some pointers from my side left.

I will continue reviewing the tests.

test-util/data/orchestration/orchestration-chat-completion-stream-chunk-response-tool-call.json

test-util/data/orchestration/orchestration-chat-completion-stream-chunks-tool-calls.txt

sample-code/src/server.ts

sample-code/src/langchain-orchestration.ts

packages/orchestration/src/orchestration-stream-chunk-response.test.ts

packages/langchain/src/orchestration/util.ts

tomfrenken · May 22, 2025

packages/langchain/src/orchestration/client.ts

+        generationInfo: { ...tokenIndices }
+      });
+
+      // Notify the run manager about the new token, some parameters are undefined as they are implicitly read from the context.


I would put it as a comment right above this line. Doesn't need to be user-facing, but for us to maintain in the future.

Co-authored-by: Tom Frenken <54979414+tomfrenken@users.noreply.github.com>

KavithaSiva · May 22, 2025

packages/langchain/src/orchestration/__snapshots__/client.test.ts.snap

+            {
+              "delta": {
+                "content": "",
+                "role": "assistantassistantassistantassistantassistantassistantassistantassistantassistantassistantassistant",


Some properties are concatenated multiple times, as concat() in AIMessageChunk does not have these properties. We need to custom handle these properties during concatenation. Will handle this in a follow-up BLI: https://github.com/SAP/ai-sdk-js-backlog/issues/322.

I don't think this is a blocker for the streaming feature. What do the others think?

Agreed, but could also be a contribution to LangChain itself (as part of that follow-up ticket), as they hint as concat being the "standard" way to chain chunks together, and it's clearly flawed.

Changes addressed.

…:SAP/ai-sdk-js into langchain-orchestration-client-streaming

tomfrenken

A few minor comments, but overall LGTM!

packages/orchestration/src/orchestration-stream-chunk-response.test.ts

tomfrenken · May 23, 2025

packages/langchain/src/orchestration/__snapshots__/client.test.ts.snap

+            {
+              "delta": {
+                "content": "",
+                "role": "assistantassistantassistantassistantassistantassistantassistantassistantassistantassistantassistant",


Agreed, but could also be a contribution to LangChain itself (as part of that follow-up ticket), as they hint as concat being the "standard" way to chain chunks together, and it's clearly flawed.

….test.ts Co-authored-by: Tom Frenken <54979414+tomfrenken@users.noreply.github.com>

KavithaSiva and others added 7 commits May 1, 2025 22:54

chore: progress till now

996a818

chore: progress till now

6c9039f

Merge branch 'main' into langchain-orchestration-client-streaming

a84b446

chore: update implementation

a54250f

chore: minor corrections

e0f0e68

chore: add changeset

9a743b6

Merge branch 'main' into langchain-orchestration-client-streaming

d386b30

KavithaSiva commented May 9, 2025

View reviewed changes

packages/langchain/src/orchestration/util.ts Outdated Show resolved Hide resolved

KavithaSiva requested review from tomfrenken and ZhongpinWang May 9, 2025 08:52

KavithaSiva and others added 6 commits May 9, 2025 11:26

chore: change finish_reason storage

8c4a7ec

Merge branch 'langchain-orchestration-client-streaming' of github.com…

989f742

…:SAP/ai-sdk-js into langchain-orchestration-client-streaming

chore: update additonal properties

db8491f

chore: revert type change for additional_properties

fd37e19

Merge branch 'main' into langchain-orchestration-client-streaming

009b508

Merge branch 'main' into langchain-orchestration-client-streaming

ca2c7bc

KavithaSiva marked this pull request as ready for review May 12, 2025 07:57

ZhongpinWang requested changes May 12, 2025

View reviewed changes

Update .changeset/chilly-steaks-divide.md

175e9b9

Co-authored-by: Zhongpin Wang <zhongpin.wang@sap.com>

tomfrenken reviewed May 14, 2025

View reviewed changes

sample-code/src/index.ts Outdated Show resolved Hide resolved

tomfrenken reviewed May 14, 2025

View reviewed changes

packages/langchain/src/orchestration/util.ts Outdated Show resolved Hide resolved

KavithaSiva and others added 3 commits May 14, 2025 17:01

chore: address review comments

dd0a47c

Merge branch 'main' into langchain-orchestration-client-streaming

13f10ce

chore: use convenience functions always

96df564

ZhongpinWang reviewed May 14, 2025

View reviewed changes

packages/langchain/src/orchestration/client.ts Outdated Show resolved Hide resolved

packages/langchain/src/orchestration/client.ts Outdated Show resolved Hide resolved

sample-code/src/langchain-orchestration.ts Outdated Show resolved Hide resolved

KavithaSiva added 4 commits May 15, 2025 10:30

chore: switch to map

5769148

chore: add more convenience functions

643a6bb

chore: address review comments

7503191

chore: simplify function

a6cb6d7

KavithaSiva and others added 8 commits May 22, 2025 11:23

chore: Add snapshot tests instead

40da119

Merge branch 'main' into langchain-orchestration-client-streaming

cc81b03

chore: fix issues after merge conflict

94ceda9

fix: Changes from lint

ad368c1

chore: fix tests

8968eb4

Merge branch 'langchain-orchestration-client-streaming' of github.com…

b2ae5d0

…:SAP/ai-sdk-js into langchain-orchestration-client-streaming

Merge branch 'main' into langchain-orchestration-client-streaming

dcd5550

chore: use mockInference instead of inference with resilience

dd9c34f

KavithaSiva requested review from deekshas8 and ZhongpinWang May 22, 2025 11:41

tomfrenken reviewed May 22, 2025

View reviewed changes

KavithaSiva and others added 6 commits May 22, 2025 14:59

Update packages/langchain/src/orchestration/util.ts

146825f

Co-authored-by: Tom Frenken <54979414+tomfrenken@users.noreply.github.com>

Update packages/langchain/src/orchestration/util.ts

80f1e7d

Co-authored-by: Tom Frenken <54979414+tomfrenken@users.noreply.github.com>

chore: review comment

d08f4a8

chore: Remove test file and adapt

1eac3c7

chore: adapt sample code

9003b02

chore: address review comment

92ea3b3

KavithaSiva commented May 22, 2025

View reviewed changes

KavithaSiva and others added 2 commits May 22, 2025 16:16

chore: improve comment

3df806a

Merge branch 'main' into langchain-orchestration-client-streaming

60a4f44

KavithaSiva requested a review from tomfrenken May 22, 2025 14:20

KavithaSiva added 2 commits May 22, 2025 17:01

chore: add test for testing timeout

b4dae2b

Merge branch 'langchain-orchestration-client-streaming' of github.com…

d87c204

…:SAP/ai-sdk-js into langchain-orchestration-client-streaming

tomfrenken approved these changes May 23, 2025

View reviewed changes

KavithaSiva and others added 3 commits May 23, 2025 11:25

Update packages/orchestration/src/orchestration-stream-chunk-response…

79a3f1a

….test.ts Co-authored-by: Tom Frenken <54979414+tomfrenken@users.noreply.github.com>

chore: fix test

cf0f01e

fix: Changes from lint

69b66e9

KavithaSiva merged commit 7574dd1 into main May 23, 2025
10 checks passed

KavithaSiva deleted the langchain-orchestration-client-streaming branch May 23, 2025 09:43

Search code, repositories, users, issues, pull requests...

feat: Support LangChain orchestration client streaming #711

feat: Support LangChain orchestration client streaming #711

Uh oh!

Conversation

KavithaSiva commented May 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Context

What this PR does and why it is needed

Uh oh!

Uh oh!

KavithaSiva commented May 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ZhongpinWang left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ZhongpinWang left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

tomfrenken left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

tomfrenken May 22, 2025

Choose a reason for hiding this comment

Uh oh!

KavithaSiva May 22, 2025

Choose a reason for hiding this comment

Uh oh!

tomfrenken May 23, 2025

Choose a reason for hiding this comment

Uh oh!

tomfrenken left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

tomfrenken May 23, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

KavithaSiva commented May 7, 2025 •

edited

Loading

KavithaSiva commented May 12, 2025 •

edited

Loading