Supporting AI tools #3007

Apr 24, 2024

antonpirker
Apr 24, 2024

Hey everybody!

Not long ago we added an OpenAI integration to the Sentry SDK. Right now we have PRs for two more AI related integrations open, Anthropic and Langchain.

If you work in the AI sector or are an AI enthusiast, I would like to ask you a couple of questions:

What data from your AI project would you like to see in Sentry?
What data would help to improve your AI pipelines, the training of your models, help you improve your AI project?
What other AI related features would you love to see in Sentry?
Are there any other tools you would like to see automatic instrumentation for?

Let's get a discussion going, so together we can build kick ass AI support into Sentry!

czyber · Apr 28, 2024

smeubank
Apr 28, 2024

After talking with some folks internally and from the community (downloads and stars etc), I believe the most useful platforms for folks would be as follows, grouped by product/service, and then popularity within. ✅ Means we already have a PR in the works for them

11 replies

czyber Apr 30, 2024

Hey hey! Sure I'll give a vector db a spin. If you have any preferences on which to integrate first, I'm open to do any of them. Otherwise I'll have time on the weekend to have a deeper look 👍

antonpirker Apr 30, 2024
Author

@colin-sentry you probably know best what vector db @czyber could look into. You do cohere. Chroma is the one that is included in Langchain, so maybe this is a good candidate? Or should we wait until you do a cohere integration and just copy it and modify it to fit another db?

colin-sentry Apr 30, 2024

I was just going to do the cohere LLM, not their vector DB yet.

We do have the embeddings measured in the OpenAI integration, which is most of the interesting bits for most vector DBs

smeubank Apr 30, 2024

in the future we may have better/more native OTel support in the Python SDK

but until then :) @czyber if you want it might be interesting to work on chromadb, and refer the implementation here for inspiration

https://github.com/traceloop/openllmetry/tree/main/packages/opentelemetry-instrumentation-chromadb

czyber Apr 30, 2024

Sure thing!

Apr 28, 2024

smeubank
Apr 28, 2024

Interesting use cases:

the following I am regurgitating from @czyber 😆 for full transparency 🚀

An interesting metric to observe for LLMs in general could be the output tokens from the LLM categorized by prompts.

Lets take the following example:
I have the use case that I want the LLM to create studying material for students (e.g. task descriptions for math problems),

The prompt is a template, and has some dynamic inputs (such as 'output_language', 'grade', 'topic'). It would be interesting to have
some metrics (maybe even alerts) if the LLM responses deviate from the expectation value of the output_token count.

0 replies

Apr 28, 2024

smeubank
Apr 28, 2024

Hugging Face does provide some support in the way that you can indicate an OTLP endpoint (Text Generation Interface: Distributed Tracing). Which appear to be what openLLMetry takes advantage of here

0 replies

Jun 20, 2024

antonpirker
Jun 20, 2024
Author

Just a quick update:

Our LLM monitoring is now out of Beta: https://docs.sentry.io/product/insights/llm-monitoring/
The Python libraries we support right now can be found here: https://docs.sentry.io/platforms/python/integrations/#ai

0 replies

nsaef · Feb 17, 2025

iloveitaly
Feb 17, 2025

@antonpirker the documentation isn't clear: If I have the py openai library installed, will monitoring automatically be setup or do I need to @ai_track("My AI pipeline") each function which makes an openai call?

2 replies

nsaef Mar 28, 2025

The same goes for LangChain: the way the documentation is written, I would assume it to track my LLM calls automatically. However, the "LLM Monitoring" section only showed up once I explicitely initialized Sentry with the LangChain integration, and it's not creating any traces without using the decorators.

Relevant excerpt from the docs:

The Sentry LLM Monitoring feature relies on the fact that you have an orchestrator (like LangChain) creating pipelines of one or more LLMs (such as gpt-4). [...]

If you're using a provider like OpenAI without an orchestrator like LangChain, you'll need to manually create pipelines with the @ai_track annotation.

smeubank Mar 31, 2025

hi @iloveitaly and @nsaef

thanks for this feedback. We discussed this in planning today, I don't have an ETA, but we did create an issue. Generally i think we should better communicate what auto-instrumentation does from the SDK, and what we expect form users. Feel free to comment any corrections or feedback also on that issue.

getsentry/sentry-docs#13167

smeubank · Apr 28, 2025

invke
Apr 28, 2025

Hi @antonpirker, we are typically using https://github.com/griptape-ai/griptape as our AI framework. Is there plans to support integration there? The current project I'm working on is using OpenAI models (for now) via Griptape, which in turn uses the OpenAI SDK, so I had thought perhaps it would work. However, it doesn't seem to report token usage, has non realistic 0.02ms durations and of course then no cost estimates. I figure perhaps it's just an incompatibility with how Griptape operates around the SDKs.

Did think about them both supporting Open Telemetry but figured I'd check here first as I wasn't immediately sure how that'd work and was time-bound.

1 reply

smeubank Apr 28, 2025

hi @invke

It isn't planned. We are working on OpenTelemetry support, but OpenAI is not necessarily a core feature support of this work, https://github.com/getsentry/projects/issues/68. I don't know that OTel has native support for AI libraries today. There is OpenLLMetry which is not part of OTel, and is 3rd party.

i also created an issue to track this topic, and let the team consider it, and any more context you might want to add to the issue.

#4330

smeubank · Jun 3, 2025

Luke31
Jun 3, 2025

Hi @antonpirker . I am not sure if this thread is still being followed, but some feedback from my side. Sorry for the long post.

We are using OpenAI with Langchain. After following the documentation I found it a bit difficult to start using so I created a PR to update the documentation: getsentry/sentry-docs#13905

Currently two use-cases aren't easily covered by current functionality:

Used tokens by tag

See number of used tokens by custom tag or user-id. Currently we can only see the total used token by project and enviornment. There is no method to further drill down in the AI LLM-Monitoring tab in sentry.

It would be nice to have further filters similar to traces tab:

Quick overview to view multiple input and output prompts

Currently input and output prompts are hard to find from the AI pipeline tab. Multiple clicks and identifying the correct span in the traces view to find the prompt is cumbersome. It is maybe enough for finding a bug but insufficient for frequent use, especially if the prompt is long. The UI is not made to display long prompts.

If the prompt is too long, it is also cut off (maybe configureable?):

When developing an AI pipeline, I'm mainly interested in the data in-and output and prompt in-and output.

Of course the requirements depend on each developer and maybe sentry is not the correct tool for these use-cases.
Thank you!

2 replies

smeubank Jun 6, 2025

hey @Luke31

thanks for this! I will make sure someone gets eyes on this

antonpirker Jun 10, 2025
Author

Hey @Luke31 we are working on a better AI experience at the moment, so stay tuned, it will get better.

For the "tokens used" by tag or user, I am not sure but I asked our backend folks. I will keep you posted!

Search code, repositories, users, issues, pull requests...

Supporting AI tools #3007

Uh oh!

Replies: 7 comments · 16 replies

Uh oh!

Uh oh!

LLM Providers

Framework

Vector DBs

Uh oh!

Uh oh!

antonpirker Apr 30, 2024 Author

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Interesting use cases:

Uh oh!

Uh oh!

Uh oh!

antonpirker Jun 20, 2024 Author

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Used tokens by tag

Quick overview to view multiple input and output prompts

Uh oh!

Uh oh!

antonpirker Jun 10, 2025 Author

antonpirker Apr 30, 2024
Author

antonpirker
Jun 20, 2024
Author

antonpirker Jun 10, 2025
Author