Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly

Appearance settings
Discussion options

I am using Sentry on a Django application served by uWSGI with the follwoing configuration:

[uwsgi]
[...]
harakiri = 20
[...]

That means that if a request last more than 20s, uWSGI will kill the worker.
As the worker is killed, Sentry (configured for Django) is unable to send any data about the request.

I saw some options to retrieve Stacktrace (such as harakiri-verbose, py-tracebacker, uwsgi.after_req_hook). However, nothing (at least recent) regarding implementing is with Sentry.

Is it possible for Sentry to catch the Harakiri signal, and send an Exception or a Message before being killed ?

You must be logged in to vote

Replies: 4 comments · 5 replies

Comment options

hi @TheBlusky

sorry for the late reply here

I have not heard this come up before and not sure that the SDK can manage this out of the box.

Looking around it does seem this has come up for others though. We don't officially support the libraries or methods below, but it may be helpful. Maybe a feature request if more users are interested though! Thanks for writing in.

https://blog.theodo.com/2022/02/logging-uwsgi-alarms-errors-sentry/

https://github.com/futurecolors/raven-harakiri

You must be logged in to vote
5 replies
@TheBlusky
Comment options

Thanks for the answer.

I also already saw those posts, and nothing came out useful.

However, we came with another approch :

  • we added a django middleware that create a thread, and kill the thread when the query is finished
  • the thread wait harakiri time - 2 seconds then capture an exception / error message "Query is too long" and finish the transaction (to send to the performance Sentry database)
  • 2s later, the request will be harakiried

With this, if a query last less than harakiri time - 2 seconds, the middleware do nothing. If not, the query will be sent to sentry.

It's not the cleanest way to handle this issue, but so far, it's working fine, and we've been able to pinpoint where queries were stucked / looped when harakiri occured.

@smeubank
Comment options

getsentry/sentry-docs#6428

opening a docs issue. Worth discussing with team if we would want to add something like this there

Thank you! And if you have any specific code snippet examples, it could make it easier to discuss with the team.

@TheBlusky
Comment options

The following snippet should work with Django and uWSGI :

timeout_delay = None
try:
    import uwsgi

    if uwsgi_harakiri := uwsgi.opt.get("harakiri", False):
        timeout_delay = int(uwsgi_harakiri) - 2
except ImportError:
    pass


def log_before_harakiri(thread_hub, transaction):
    with thread_hub:
        capture_message("Long request")
    if transaction is not None:
        transaction.finish()
    thread_hub.flush()


class TimeoutMiddleware:
    def __init__(self, get_response):
        self.get_response = get_response

    def __call__(self, request):
        if timeout_delay is None:
            return self.get_response(request)
        thread_hub = sentry_sdk.Hub(sentry_sdk.Hub.current)
        transaction = sentry_sdk.Hub.current.scope.transaction
        harakiri_timer = threading.Timer(
            timeout_delay, log_before_harakiri, args=[thread_hub, transaction]
        )
        harakiri_timer.start()
        try:
            return self.get_response(request)
        finally:
            harakiri_timer.cancel()

However, as-is, there is a drawback as request during between harakiri - 2s and harakiris won't be fully sent to Sentry.

@antonpirker
Comment options

Yes, we could add this to the docs. Maybe we should create a "cook book" section, where we put recipies for common use cases?

@smeubank
Comment options

I like it :D

would just like to follow something similar to other frameworks, that could also be transparently used in the includes across uWSGI

also I will put on PM hat: uWSGI Framework Features section might have more SEO bump

Comment options

There is a (kind of) related issue for Gunicorn: #1447
When Gunicorn times out Sentry catches the logging message that is sent by Gunicorn and sends it to Sentry.
A similar thing could be done for uwsgi harakiri.

You must be logged in to vote
0 replies
Comment options

uWSGI can now gracefully kill before Harakiri: https://uwsgi-docs.readthedocs.io/en/latest/Changelog-2.0.22.html

You must be logged in to vote
0 replies
Comment options

I was able to create a Django middleware able to catch Harakiri, and send transaction to sentry before being killed using new gracefull killing feature :

class HarakiriLoggerMiddleware:
    transactions = {}

    @classmethod
    def handle_signal(cls, *args):
        hubs = []
        for (hub, transaction) in HarakiriLoggerMiddleware.transactions.values():
            with hub:
                capture_message("LBH02 - Exiting")
            if transaction is not None:
                transaction.finish()
            if hub not in hubs:
                hubs.append(hub)
        for hub in hubs:
            hub.flush()

    def __init__(self, get_response):
        self.get_response = get_response

    def __call__(self, request):
        hub = sentry_sdk.Hub(sentry_sdk.Hub.current)
        transaction = sentry_sdk.Hub.current.scope.transaction
        t_id = str(uuid.uuid4())
        HarakiriLoggerMiddleware.transactions[t_id] = (hub, transaction)
        try:
            return self.get_response(request)
        finally:
            HarakiriLoggerMiddleware.transactions.pop(t_id)

Adding signal handle in the app config:

class MyappConfig(AppConfig):
    def ready(self):
        signal.signal(
            signal.SIGSYS, HarakiriLoggerMiddleware.handle_signal
        )

And adding the following in uWSGI config :

harakiri = 20
lazy-apps = true
harakiri-graceful-timeout = 5
harakiri-graceful-signal = 31
py-call-osafterfork = true

It might need further testing, but imho this is a good way to do it.

You must be logged in to vote
0 replies
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Category
🙏
Q&A
Labels
Component: Docs updates on getsentry/docs or docstrings Component: SDK Core Dealing with the core of the SDK Integration: Django Integration: WSGI
3 participants
Morty Proxy This is a proxified and sanitized view of the page, visit original site.