Skip to content

fix: disable sentry profiling

What does this merge request do and why?

Error tracking is still enabled.

As mentioned here: https://gitlab.com/gitlab-org/modelops/applied-ml/code-suggestions/ai-assist/-/issues/1589#note_2839024058, we now see the segfault errors in the logs: https://cloudlogging.app.goo.gl/zWtbosNgBauYqjtP6

Click to expand
Fatal Python error: Segmentation fault
Thread 0x00007fdbda0006c0 (most recent call first):
File "/usr/local/lib/python3.12/concurrent/futures/thread.py", line 90 in _worker
File "/usr/local/lib/python3.12/threading.py", line 1012 in run
File "/home/aigateway/app/venv/ai-gateway-M6hW6iiC-py3.12/lib/python3.12/site-packages/sentry_sdk/integrations/threading.py", line 123 in _run_old_run_func
File "/home/aigateway/app/venv/ai-gateway-M6hW6iiC-py3.12/lib/python3.12/site-packages/sentry_sdk/integrations/threading.py", line 130 in run
File "/usr/local/lib/python3.12/threading.py", line 1075 in _bootstrap_inner
File "/usr/local/lib/python3.12/threading.py", line 1032 in _bootstrap
Thread 0x00007fdbdaa006c0 (most recent call first):
File "/usr/local/lib/python3.12/concurrent/futures/thread.py", line 90 in _worker
File "/usr/local/lib/python3.12/threading.py", line 1012 in run
File "/home/aigateway/app/venv/ai-gateway-M6hW6iiC-py3.12/lib/python3.12/site-packages/sentry_sdk/integrations/threading.py", line 123 in _run_old_run_func
File "/home/aigateway/app/venv/ai-gateway-M6hW6iiC-py3.12/lib/python3.12/site-packages/sentry_sdk/integrations/threading.py", line 130 in run
File "/usr/local/lib/python3.12/threading.py", line 1075 in _bootstrap_inner
File "/usr/local/lib/python3.12/threading.py", line 1032 in _bootstrap
Thread 0x00007fdbdb4006c0 (most recent call first):
File "/usr/local/lib/python3.12/concurrent/futures/thread.py", line 90 in _worker
File "/usr/local/lib/python3.12/threading.py", line 1012 in run
File "/home/aigateway/app/venv/ai-gateway-M6hW6iiC-py3.12/lib/python3.12/site-packages/sentry_sdk/integrations/threading.py", line 123 in _run_old_run_func
File "/home/aigateway/app/venv/ai-gateway-M6hW6iiC-py3.12/lib/python3.12/site-packages/sentry_sdk/integrations/threading.py", line 130 in run
File "/usr/local/lib/python3.12/threading.py", line 1075 in _bootstrap_inner
File "/usr/local/lib/python3.12/threading.py", line 1032 in _bootstrap
Thread 0x00007fdbdbe006c0 (most recent call first):
File "/usr/local/lib/python3.12/concurrent/futures/thread.py", line 90 in _worker
File "/usr/local/lib/python3.12/threading.py", line 1012 in run
File "/home/aigateway/app/venv/ai-gateway-M6hW6iiC-py3.12/lib/python3.12/site-packages/sentry_sdk/integrations/threading.py", line 123 in _run_old_run_func
File "/home/aigateway/app/venv/ai-gateway-M6hW6iiC-py3.12/lib/python3.12/site-packages/sentry_sdk/integrations/threading.py", line 130 in run
File "/usr/local/lib/python3.12/threading.py", line 1075 in _bootstrap_inner
File "/usr/local/lib/python3.12/threading.py", line 1032 in _bootstrap
Thread 0x00007fdbe4c006c0 (most recent call first):
File "/usr/local/lib/python3.12/concurrent/futures/thread.py", line 90 in _worker
File "/usr/local/lib/python3.12/threading.py", line 1012 in run
File "/home/aigateway/app/venv/ai-gateway-M6hW6iiC-py3.12/lib/python3.12/site-packages/sentry_sdk/integrations/threading.py", line 123 in _run_old_run_func
File "/home/aigateway/app/venv/ai-gateway-M6hW6iiC-py3.12/lib/python3.12/site-packages/sentry_sdk/integrations/threading.py", line 130 in run
File "/usr/local/lib/python3.12/threading.py", line 1075 in _bootstrap_inner
File "/usr/local/lib/python3.12/threading.py", line 1032 in _bootstrap
Thread 0x00007fdbe56006c0 (most recent call first):
File "/usr/local/lib/python3.12/concurrent/futures/thread.py", line 90 in _worker
File "/usr/local/lib/python3.12/threading.py", line 1012 in run
File "/home/aigateway/app/venv/ai-gateway-M6hW6iiC-py3.12/lib/python3.12/site-packages/sentry_sdk/integrations/threading.py", line 123 in _run_old_run_func
File "/home/aigateway/app/venv/ai-gateway-M6hW6iiC-py3.12/lib/python3.12/site-packages/sentry_sdk/integrations/threading.py", line 130 in run
File "/usr/local/lib/python3.12/threading.py", line 1075 in _bootstrap_inner
File "/usr/local/lib/python3.12/threading.py", line 1032 in _bootstrap
Thread 0x00007fdbe60006c0 (most recent call first):
File "/usr/local/lib/python3.12/concurrent/futures/thread.py", line 90 in _worker
File "/usr/local/lib/python3.12/threading.py", line 1012 in run
File "/home/aigateway/app/venv/ai-gateway-M6hW6iiC-py3.12/lib/python3.12/site-packages/sentry_sdk/integrations/threading.py", line 123 in _run_old_run_func
File "/home/aigateway/app/venv/ai-gateway-M6hW6iiC-py3.12/lib/python3.12/site-packages/sentry_sdk/integrations/threading.py", line 130 in run
File "/usr/local/lib/python3.12/threading.py", line 1075 in _bootstrap_inner
File "/usr/local/lib/python3.12/threading.py", line 1032 in _bootstrap
Thread 0x00007fdbe6a006c0 (most recent call first):
File "/usr/local/lib/python3.12/concurrent/futures/thread.py", line 90 in _worker
File "/usr/local/lib/python3.12/threading.py", line 1012 in run
File "/home/aigateway/app/venv/ai-gateway-M6hW6iiC-py3.12/lib/python3.12/site-packages/sentry_sdk/integrations/threading.py", line 123 in _run_old_run_func
File "/home/aigateway/app/venv/ai-gateway-M6hW6iiC-py3.12/lib/python3.12/site-packages/sentry_sdk/integrations/threading.py", line 130 in run
File "/usr/local/lib/python3.12/threading.py", line 1075 in _bootstrap_inner
File "/usr/local/lib/python3.12/threading.py", line 1032 in _bootstrap
Thread 0x00007fdbe74006c0 (most recent call first):
File "/usr/local/lib/python3.12/concurrent/futures/thread.py", line 90 in _worker
File "/usr/local/lib/python3.12/threading.py", line 1012 in run
File "/home/aigateway/app/venv/ai-gateway-M6hW6iiC-py3.12/lib/python3.12/site-packages/sentry_sdk/integrations/threading.py", line 123 in _run_old_run_func
File "/home/aigateway/app/venv/ai-gateway-M6hW6iiC-py3.12/lib/python3.12/site-packages/sentry_sdk/integrations/threading.py", line 130 in run
File "/usr/local/lib/python3.12/threading.py", line 1075 in _bootstrap_inner
File "/usr/local/lib/python3.12/threading.py", line 1032 in _bootstrap
Thread 0x00007fdbe7e006c0 (most recent call first):
File "/usr/local/lib/python3.12/concurrent/futures/thread.py", line 90 in _worker
File "/usr/local/lib/python3.12/threading.py", line 1012 in run
File "/home/aigateway/app/venv/ai-gateway-M6hW6iiC-py3.12/lib/python3.12/site-packages/sentry_sdk/integrations/threading.py", line 123 in _run_old_run_func
File "/home/aigateway/app/venv/ai-gateway-M6hW6iiC-py3.12/lib/python3.12/site-packages/sentry_sdk/integrations/threading.py", line 130 in run
File "/usr/local/lib/python3.12/threading.py", line 1075 in _bootstrap_inner
File "/usr/local/lib/python3.12/threading.py", line 1032 in _bootstrap
Thread 0x00007fdbf0c006c0 (most recent call first):
File "/usr/local/lib/python3.12/concurrent/futures/thread.py", line 90 in _worker
File "/usr/local/lib/python3.12/threading.py", line 1012 in run
File "/home/aigateway/app/venv/ai-gateway-M6hW6iiC-py3.12/lib/python3.12/site-packages/sentry_sdk/integrations/threading.py", line 123 in _run_old_run_func
File "/home/aigateway/app/venv/ai-gateway-M6hW6iiC-py3.12/lib/python3.12/site-packages/sentry_sdk/integrations/threading.py", line 130 in run
File "/usr/local/lib/python3.12/threading.py", line 1075 in _bootstrap_inner
File "/usr/local/lib/python3.12/threading.py", line 1032 in _bootstrap
Thread 0x00007fdbf16006c0 (most recent call first):
File "/usr/local/lib/python3.12/concurrent/futures/thread.py", line 90 in _worker
File "/usr/local/lib/python3.12/threading.py", line 1012 in run
File "/home/aigateway/app/venv/ai-gateway-M6hW6iiC-py3.12/lib/python3.12/site-packages/sentry_sdk/integrations/threading.py", line 123 in _run_old_run_func
File "/home/aigateway/app/venv/ai-gateway-M6hW6iiC-py3.12/lib/python3.12/site-packages/sentry_sdk/integrations/threading.py", line 130 in run
File "/usr/local/lib/python3.12/threading.py", line 1075 in _bootstrap_inner
File "/usr/local/lib/python3.12/threading.py", line 1032 in _bootstrap
Thread 0x00007fdbf20006c0 (most recent call first):
File "/usr/local/lib/python3.12/threading.py", line 355 in wait
File "/usr/local/lib/python3.12/queue.py", line 171 in get
File "/home/aigateway/app/venv/ai-gateway-M6hW6iiC-py3.12/lib/python3.12/site-packages/snowplow_tracker/emitters.py", line 548 in consume
File "/usr/local/lib/python3.12/threading.py", line 1012 in run
File "/home/aigateway/app/venv/ai-gateway-M6hW6iiC-py3.12/lib/python3.12/site-packages/sentry_sdk/integrations/threading.py", line 123 in _run_old_run_func
File "/home/aigateway/app/venv/ai-gateway-M6hW6iiC-py3.12/lib/python3.12/site-packages/sentry_sdk/integrations/threading.py", line 130 in run
File "/usr/local/lib/python3.12/threading.py", line 1075 in _bootstrap_inner
File "/usr/local/lib/python3.12/threading.py", line 1032 in _bootstrap
Thread 0x00007fdbf2a006c0 (most recent call first):
File "/home/aigateway/app/venv/ai-gateway-M6hW6iiC-py3.12/lib/python3.12/site-packages/sentry_sdk/profiler/transaction_profiler.py", line 740 in run
File "/usr/local/lib/python3.12/threading.py", line 1012 in run
File "/home/aigateway/app/venv/ai-gateway-M6hW6iiC-py3.12/lib/python3.12/site-packages/sentry_sdk/integrations/threading.py", line 123 in _run_old_run_func
File "/home/aigateway/app/venv/ai-gateway-M6hW6iiC-py3.12/lib/python3.12/site-packages/sentry_sdk/integrations/threading.py", line 130 in run
File "/usr/local/lib/python3.12/threading.py", line 1075 in _bootstrap_inner
File "/usr/local/lib/python3.12/threading.py", line 1032 in _bootstrap
Thread 0x00007fdbf34006c0 (most recent call first):
File "/home/aigateway/app/venv/ai-gateway-M6hW6iiC-py3.12/lib/python3.12/site-packages/sentry_sdk/monitor.py", line 58 in _thread
File "/usr/local/lib/python3.12/threading.py", line 1012 in run
File "/home/aigateway/app/venv/ai-gateway-M6hW6iiC-py3.12/lib/python3.12/site-packages/sentry_sdk/integrations/threading.py", line 123 in _run_old_run_func
File "/home/aigateway/app/venv/ai-gateway-M6hW6iiC-py3.12/lib/python3.12/site-packages/sentry_sdk/integrations/threading.py", line 130 in run
File "/usr/local/lib/python3.12/threading.py", line 1075 in _bootstrap_inner
File "/usr/local/lib/python3.12/threading.py", line 1032 in _bootstrap
Thread 0x00007fdbf3e006c0 (most recent call first):
File "/usr/local/lib/python3.12/threading.py", line 355 in wait
File "/home/aigateway/app/venv/ai-gateway-M6hW6iiC-py3.12/lib/python3.12/site-packages/sentry_sdk/_queue.py", line 242 in get
File "/home/aigateway/app/venv/ai-gateway-M6hW6iiC-py3.12/lib/python3.12/site-packages/sentry_sdk/worker.py", line 131 in _target
File "/usr/local/lib/python3.12/threading.py", line 1012 in run
File "/home/aigateway/app/venv/ai-gateway-M6hW6iiC-py3.12/lib/python3.12/site-packages/sentry_sdk/integrations/threading.py", line 123 in _run_old_run_func
File "/home/aigateway/app/venv/ai-gateway-M6hW6iiC-py3.12/lib/python3.12/site-packages/sentry_sdk/integrations/threading.py", line 130 in run
File "/usr/local/lib/python3.12/threading.py", line 1075 in _bootstrap_inner
File "/usr/local/lib/python3.12/threading.py", line 1032 in _bootstrap
Thread 0x00007fdc068006c0 (most recent call first):
File "/usr/local/lib/python3.12/threading.py", line 1012 in run
File "/home/aigateway/app/venv/ai-gateway-M6hW6iiC-py3.12/lib/python3.12/site-packages/sentry_sdk/integrations/threading.py", line 123 in _run_old_run_func
File "/home/aigateway/app/venv/ai-gateway-M6hW6iiC-py3.12/lib/python3.12/site-packages/sentry_sdk/integrations/threading.py", line 130 in run
File "/usr/local/lib/python3.12/threading.py", line 1075 in _bootstrap_inner
File "/usr/local/lib/python3.12/threading.py", line 1032 in _bootstrap
Thread 0x00007fdc07c006c0 (most recent call first):
File "/usr/local/lib/python3.12/selectors.py", line 415 in select
File "/usr/local/lib/python3.12/socketserver.py", line 235 in serve_forever
File "/usr/local/lib/python3.12/threading.py", line 1012 in run
File "/home/aigateway/app/venv/ai-gateway-M6hW6iiC-py3.12/lib/python3.12/site-packages/sentry_sdk/integrations/threading.py", line 123 in _run_old_run_func
File "/home/aigateway/app/venv/ai-gateway-M6hW6iiC-py3.12/lib/python3.12/site-packages/sentry_sdk/integrations/threading.py", line 130 in run
File "/usr/local/lib/python3.12/threading.py", line 1075 in _bootstrap_inner
File "/usr/local/lib/python3.12/threading.py", line 1032 in _bootstrap
Thread 0x00007fdc0ca006c0 (most recent call first):
File "/home/aigateway/app/venv/ai-gateway-M6hW6iiC-py3.12/lib/python3.12/site-packages/googlecloudprofiler/cpu_profiler.py", line 53 in _profile
File "/home/aigateway/app/venv/ai-gateway-M6hW6iiC-py3.12/lib/python3.12/site-packages/googlecloudprofiler/cpu_profiler.py", line 49 in profile
File "/home/aigateway/app/venv/ai-gateway-M6hW6iiC-py3.12/lib/python3.12/site-packages/googlecloudprofiler/client.py", line 293 in _collect_and_upload_profile
File "/home/aigateway/app/venv/ai-gateway-M6hW6iiC-py3.12/lib/python3.12/site-packages/googlecloudprofiler/client.py", line 335 in _poll_profiler_service
File "/usr/local/lib/python3.12/threading.py", line 1012 in run
File "/usr/local/lib/python3.12/threading.py", line 1075 in _bootstrap_inner
File "/usr/local/lib/python3.12/threading.py", line 1032 in _bootstrap
Current thread 0x00007fdc36b5db80 (most recent call first):
File "/usr/local/lib/python3.12/typing.py", line 2344 in get_origin

And the problem is that we have multiple profilers running simultaneously: Sentry Profiler and Google Cloud Profiler. The crash occurs because:

  1. Multiple profilers are using Python's sys.setprofile() and threading.settrace()
  2. Sentry's threading integration wraps threading.Thread.run()
  3. When threads access typing internals simultaneously during profiling, we get memory corruption
  4. The ThreadPoolExecutor workers are particularly vulnerable because they're created/destroyed frequently

We don't seem to use Sentry Profiling: https://docs.sentry.io/product/explore/profiling/. This is the Sentry page for the project: https://new-sentry.gitlab.net/organizations/gitlab/performance/?project=36. So let's go ahead and disable it according to https://docs.sentry.io/platforms/python/profiling/

Resolves gitlab-org/gitlab#578160 (closed)

Edited by Dylan Griffith

Merge request reports

Loading