fix: disable sentry profiling
What does this merge request do and why?
Error tracking is still enabled.
As mentioned here: https://gitlab.com/gitlab-org/modelops/applied-ml/code-suggestions/ai-assist/-/issues/1589#note_2839024058, we now see the segfault errors in the logs: https://cloudlogging.app.goo.gl/zWtbosNgBauYqjtP6
Click to expand
Fatal Python error: Segmentation fault
Thread 0x00007fdbda0006c0 (most recent call first):
File "/usr/local/lib/python3.12/concurrent/futures/thread.py", line 90 in _worker
File "/usr/local/lib/python3.12/threading.py", line 1012 in run
File "/home/aigateway/app/venv/ai-gateway-M6hW6iiC-py3.12/lib/python3.12/site-packages/sentry_sdk/integrations/threading.py", line 123 in _run_old_run_func
File "/home/aigateway/app/venv/ai-gateway-M6hW6iiC-py3.12/lib/python3.12/site-packages/sentry_sdk/integrations/threading.py", line 130 in run
File "/usr/local/lib/python3.12/threading.py", line 1075 in _bootstrap_inner
File "/usr/local/lib/python3.12/threading.py", line 1032 in _bootstrap
Thread 0x00007fdbdaa006c0 (most recent call first):
File "/usr/local/lib/python3.12/concurrent/futures/thread.py", line 90 in _worker
File "/usr/local/lib/python3.12/threading.py", line 1012 in run
File "/home/aigateway/app/venv/ai-gateway-M6hW6iiC-py3.12/lib/python3.12/site-packages/sentry_sdk/integrations/threading.py", line 123 in _run_old_run_func
File "/home/aigateway/app/venv/ai-gateway-M6hW6iiC-py3.12/lib/python3.12/site-packages/sentry_sdk/integrations/threading.py", line 130 in run
File "/usr/local/lib/python3.12/threading.py", line 1075 in _bootstrap_inner
File "/usr/local/lib/python3.12/threading.py", line 1032 in _bootstrap
Thread 0x00007fdbdb4006c0 (most recent call first):
File "/usr/local/lib/python3.12/concurrent/futures/thread.py", line 90 in _worker
File "/usr/local/lib/python3.12/threading.py", line 1012 in run
File "/home/aigateway/app/venv/ai-gateway-M6hW6iiC-py3.12/lib/python3.12/site-packages/sentry_sdk/integrations/threading.py", line 123 in _run_old_run_func
File "/home/aigateway/app/venv/ai-gateway-M6hW6iiC-py3.12/lib/python3.12/site-packages/sentry_sdk/integrations/threading.py", line 130 in run
File "/usr/local/lib/python3.12/threading.py", line 1075 in _bootstrap_inner
File "/usr/local/lib/python3.12/threading.py", line 1032 in _bootstrap
Thread 0x00007fdbdbe006c0 (most recent call first):
File "/usr/local/lib/python3.12/concurrent/futures/thread.py", line 90 in _worker
File "/usr/local/lib/python3.12/threading.py", line 1012 in run
File "/home/aigateway/app/venv/ai-gateway-M6hW6iiC-py3.12/lib/python3.12/site-packages/sentry_sdk/integrations/threading.py", line 123 in _run_old_run_func
File "/home/aigateway/app/venv/ai-gateway-M6hW6iiC-py3.12/lib/python3.12/site-packages/sentry_sdk/integrations/threading.py", line 130 in run
File "/usr/local/lib/python3.12/threading.py", line 1075 in _bootstrap_inner
File "/usr/local/lib/python3.12/threading.py", line 1032 in _bootstrap
Thread 0x00007fdbe4c006c0 (most recent call first):
File "/usr/local/lib/python3.12/concurrent/futures/thread.py", line 90 in _worker
File "/usr/local/lib/python3.12/threading.py", line 1012 in run
File "/home/aigateway/app/venv/ai-gateway-M6hW6iiC-py3.12/lib/python3.12/site-packages/sentry_sdk/integrations/threading.py", line 123 in _run_old_run_func
File "/home/aigateway/app/venv/ai-gateway-M6hW6iiC-py3.12/lib/python3.12/site-packages/sentry_sdk/integrations/threading.py", line 130 in run
File "/usr/local/lib/python3.12/threading.py", line 1075 in _bootstrap_inner
File "/usr/local/lib/python3.12/threading.py", line 1032 in _bootstrap
Thread 0x00007fdbe56006c0 (most recent call first):
File "/usr/local/lib/python3.12/concurrent/futures/thread.py", line 90 in _worker
File "/usr/local/lib/python3.12/threading.py", line 1012 in run
File "/home/aigateway/app/venv/ai-gateway-M6hW6iiC-py3.12/lib/python3.12/site-packages/sentry_sdk/integrations/threading.py", line 123 in _run_old_run_func
File "/home/aigateway/app/venv/ai-gateway-M6hW6iiC-py3.12/lib/python3.12/site-packages/sentry_sdk/integrations/threading.py", line 130 in run
File "/usr/local/lib/python3.12/threading.py", line 1075 in _bootstrap_inner
File "/usr/local/lib/python3.12/threading.py", line 1032 in _bootstrap
Thread 0x00007fdbe60006c0 (most recent call first):
File "/usr/local/lib/python3.12/concurrent/futures/thread.py", line 90 in _worker
File "/usr/local/lib/python3.12/threading.py", line 1012 in run
File "/home/aigateway/app/venv/ai-gateway-M6hW6iiC-py3.12/lib/python3.12/site-packages/sentry_sdk/integrations/threading.py", line 123 in _run_old_run_func
File "/home/aigateway/app/venv/ai-gateway-M6hW6iiC-py3.12/lib/python3.12/site-packages/sentry_sdk/integrations/threading.py", line 130 in run
File "/usr/local/lib/python3.12/threading.py", line 1075 in _bootstrap_inner
File "/usr/local/lib/python3.12/threading.py", line 1032 in _bootstrap
Thread 0x00007fdbe6a006c0 (most recent call first):
File "/usr/local/lib/python3.12/concurrent/futures/thread.py", line 90 in _worker
File "/usr/local/lib/python3.12/threading.py", line 1012 in run
File "/home/aigateway/app/venv/ai-gateway-M6hW6iiC-py3.12/lib/python3.12/site-packages/sentry_sdk/integrations/threading.py", line 123 in _run_old_run_func
File "/home/aigateway/app/venv/ai-gateway-M6hW6iiC-py3.12/lib/python3.12/site-packages/sentry_sdk/integrations/threading.py", line 130 in run
File "/usr/local/lib/python3.12/threading.py", line 1075 in _bootstrap_inner
File "/usr/local/lib/python3.12/threading.py", line 1032 in _bootstrap
Thread 0x00007fdbe74006c0 (most recent call first):
File "/usr/local/lib/python3.12/concurrent/futures/thread.py", line 90 in _worker
File "/usr/local/lib/python3.12/threading.py", line 1012 in run
File "/home/aigateway/app/venv/ai-gateway-M6hW6iiC-py3.12/lib/python3.12/site-packages/sentry_sdk/integrations/threading.py", line 123 in _run_old_run_func
File "/home/aigateway/app/venv/ai-gateway-M6hW6iiC-py3.12/lib/python3.12/site-packages/sentry_sdk/integrations/threading.py", line 130 in run
File "/usr/local/lib/python3.12/threading.py", line 1075 in _bootstrap_inner
File "/usr/local/lib/python3.12/threading.py", line 1032 in _bootstrap
Thread 0x00007fdbe7e006c0 (most recent call first):
File "/usr/local/lib/python3.12/concurrent/futures/thread.py", line 90 in _worker
File "/usr/local/lib/python3.12/threading.py", line 1012 in run
File "/home/aigateway/app/venv/ai-gateway-M6hW6iiC-py3.12/lib/python3.12/site-packages/sentry_sdk/integrations/threading.py", line 123 in _run_old_run_func
File "/home/aigateway/app/venv/ai-gateway-M6hW6iiC-py3.12/lib/python3.12/site-packages/sentry_sdk/integrations/threading.py", line 130 in run
File "/usr/local/lib/python3.12/threading.py", line 1075 in _bootstrap_inner
File "/usr/local/lib/python3.12/threading.py", line 1032 in _bootstrap
Thread 0x00007fdbf0c006c0 (most recent call first):
File "/usr/local/lib/python3.12/concurrent/futures/thread.py", line 90 in _worker
File "/usr/local/lib/python3.12/threading.py", line 1012 in run
File "/home/aigateway/app/venv/ai-gateway-M6hW6iiC-py3.12/lib/python3.12/site-packages/sentry_sdk/integrations/threading.py", line 123 in _run_old_run_func
File "/home/aigateway/app/venv/ai-gateway-M6hW6iiC-py3.12/lib/python3.12/site-packages/sentry_sdk/integrations/threading.py", line 130 in run
File "/usr/local/lib/python3.12/threading.py", line 1075 in _bootstrap_inner
File "/usr/local/lib/python3.12/threading.py", line 1032 in _bootstrap
Thread 0x00007fdbf16006c0 (most recent call first):
File "/usr/local/lib/python3.12/concurrent/futures/thread.py", line 90 in _worker
File "/usr/local/lib/python3.12/threading.py", line 1012 in run
File "/home/aigateway/app/venv/ai-gateway-M6hW6iiC-py3.12/lib/python3.12/site-packages/sentry_sdk/integrations/threading.py", line 123 in _run_old_run_func
File "/home/aigateway/app/venv/ai-gateway-M6hW6iiC-py3.12/lib/python3.12/site-packages/sentry_sdk/integrations/threading.py", line 130 in run
File "/usr/local/lib/python3.12/threading.py", line 1075 in _bootstrap_inner
File "/usr/local/lib/python3.12/threading.py", line 1032 in _bootstrap
Thread 0x00007fdbf20006c0 (most recent call first):
File "/usr/local/lib/python3.12/threading.py", line 355 in wait
File "/usr/local/lib/python3.12/queue.py", line 171 in get
File "/home/aigateway/app/venv/ai-gateway-M6hW6iiC-py3.12/lib/python3.12/site-packages/snowplow_tracker/emitters.py", line 548 in consume
File "/usr/local/lib/python3.12/threading.py", line 1012 in run
File "/home/aigateway/app/venv/ai-gateway-M6hW6iiC-py3.12/lib/python3.12/site-packages/sentry_sdk/integrations/threading.py", line 123 in _run_old_run_func
File "/home/aigateway/app/venv/ai-gateway-M6hW6iiC-py3.12/lib/python3.12/site-packages/sentry_sdk/integrations/threading.py", line 130 in run
File "/usr/local/lib/python3.12/threading.py", line 1075 in _bootstrap_inner
File "/usr/local/lib/python3.12/threading.py", line 1032 in _bootstrap
Thread 0x00007fdbf2a006c0 (most recent call first):
File "/home/aigateway/app/venv/ai-gateway-M6hW6iiC-py3.12/lib/python3.12/site-packages/sentry_sdk/profiler/transaction_profiler.py", line 740 in run
File "/usr/local/lib/python3.12/threading.py", line 1012 in run
File "/home/aigateway/app/venv/ai-gateway-M6hW6iiC-py3.12/lib/python3.12/site-packages/sentry_sdk/integrations/threading.py", line 123 in _run_old_run_func
File "/home/aigateway/app/venv/ai-gateway-M6hW6iiC-py3.12/lib/python3.12/site-packages/sentry_sdk/integrations/threading.py", line 130 in run
File "/usr/local/lib/python3.12/threading.py", line 1075 in _bootstrap_inner
File "/usr/local/lib/python3.12/threading.py", line 1032 in _bootstrap
Thread 0x00007fdbf34006c0 (most recent call first):
File "/home/aigateway/app/venv/ai-gateway-M6hW6iiC-py3.12/lib/python3.12/site-packages/sentry_sdk/monitor.py", line 58 in _thread
File "/usr/local/lib/python3.12/threading.py", line 1012 in run
File "/home/aigateway/app/venv/ai-gateway-M6hW6iiC-py3.12/lib/python3.12/site-packages/sentry_sdk/integrations/threading.py", line 123 in _run_old_run_func
File "/home/aigateway/app/venv/ai-gateway-M6hW6iiC-py3.12/lib/python3.12/site-packages/sentry_sdk/integrations/threading.py", line 130 in run
File "/usr/local/lib/python3.12/threading.py", line 1075 in _bootstrap_inner
File "/usr/local/lib/python3.12/threading.py", line 1032 in _bootstrap
Thread 0x00007fdbf3e006c0 (most recent call first):
File "/usr/local/lib/python3.12/threading.py", line 355 in wait
File "/home/aigateway/app/venv/ai-gateway-M6hW6iiC-py3.12/lib/python3.12/site-packages/sentry_sdk/_queue.py", line 242 in get
File "/home/aigateway/app/venv/ai-gateway-M6hW6iiC-py3.12/lib/python3.12/site-packages/sentry_sdk/worker.py", line 131 in _target
File "/usr/local/lib/python3.12/threading.py", line 1012 in run
File "/home/aigateway/app/venv/ai-gateway-M6hW6iiC-py3.12/lib/python3.12/site-packages/sentry_sdk/integrations/threading.py", line 123 in _run_old_run_func
File "/home/aigateway/app/venv/ai-gateway-M6hW6iiC-py3.12/lib/python3.12/site-packages/sentry_sdk/integrations/threading.py", line 130 in run
File "/usr/local/lib/python3.12/threading.py", line 1075 in _bootstrap_inner
File "/usr/local/lib/python3.12/threading.py", line 1032 in _bootstrap
Thread 0x00007fdc068006c0 (most recent call first):
File "/usr/local/lib/python3.12/threading.py", line 1012 in run
File "/home/aigateway/app/venv/ai-gateway-M6hW6iiC-py3.12/lib/python3.12/site-packages/sentry_sdk/integrations/threading.py", line 123 in _run_old_run_func
File "/home/aigateway/app/venv/ai-gateway-M6hW6iiC-py3.12/lib/python3.12/site-packages/sentry_sdk/integrations/threading.py", line 130 in run
File "/usr/local/lib/python3.12/threading.py", line 1075 in _bootstrap_inner
File "/usr/local/lib/python3.12/threading.py", line 1032 in _bootstrap
Thread 0x00007fdc07c006c0 (most recent call first):
File "/usr/local/lib/python3.12/selectors.py", line 415 in select
File "/usr/local/lib/python3.12/socketserver.py", line 235 in serve_forever
File "/usr/local/lib/python3.12/threading.py", line 1012 in run
File "/home/aigateway/app/venv/ai-gateway-M6hW6iiC-py3.12/lib/python3.12/site-packages/sentry_sdk/integrations/threading.py", line 123 in _run_old_run_func
File "/home/aigateway/app/venv/ai-gateway-M6hW6iiC-py3.12/lib/python3.12/site-packages/sentry_sdk/integrations/threading.py", line 130 in run
File "/usr/local/lib/python3.12/threading.py", line 1075 in _bootstrap_inner
File "/usr/local/lib/python3.12/threading.py", line 1032 in _bootstrap
Thread 0x00007fdc0ca006c0 (most recent call first):
File "/home/aigateway/app/venv/ai-gateway-M6hW6iiC-py3.12/lib/python3.12/site-packages/googlecloudprofiler/cpu_profiler.py", line 53 in _profile
File "/home/aigateway/app/venv/ai-gateway-M6hW6iiC-py3.12/lib/python3.12/site-packages/googlecloudprofiler/cpu_profiler.py", line 49 in profile
File "/home/aigateway/app/venv/ai-gateway-M6hW6iiC-py3.12/lib/python3.12/site-packages/googlecloudprofiler/client.py", line 293 in _collect_and_upload_profile
File "/home/aigateway/app/venv/ai-gateway-M6hW6iiC-py3.12/lib/python3.12/site-packages/googlecloudprofiler/client.py", line 335 in _poll_profiler_service
File "/usr/local/lib/python3.12/threading.py", line 1012 in run
File "/usr/local/lib/python3.12/threading.py", line 1075 in _bootstrap_inner
File "/usr/local/lib/python3.12/threading.py", line 1032 in _bootstrap
Current thread 0x00007fdc36b5db80 (most recent call first):
File "/usr/local/lib/python3.12/typing.py", line 2344 in get_origin
And the problem is that we have multiple profilers running simultaneously: Sentry Profiler and Google Cloud Profiler. The crash occurs because:
- Multiple profilers are using Python's
sys.setprofile()andthreading.settrace() - Sentry's threading integration wraps
threading.Thread.run() - When threads access typing internals simultaneously during profiling, we get memory corruption
- The
ThreadPoolExecutorworkers are particularly vulnerable because they're created/destroyed frequently
We don't seem to use Sentry Profiling: https://docs.sentry.io/product/explore/profiling/. This is the Sentry page for the project: https://new-sentry.gitlab.net/organizations/gitlab/performance/?project=36. So let's go ahead and disable it according to https://docs.sentry.io/platforms/python/profiling/
Resolves gitlab-org/gitlab#578160 (closed)
Edited by Dylan Griffith