doc: Debugging Gitaly

Why is this change being made?

Primarily aimed at Create oncall training for Gitaly (gitlab-org/core-platform-section/discussions#178), also serve as a basis for debugging customer issues.

This is NOT oncall training, but a supporting document for that, as well as for customer issues or just simply debugging while developing.

The main goal is to bring everyone on the same page about what tools are available, and lower their barrier of entry. It's intended both for newcomers and as reference material in day-to-day.

Author and Reviewer Checklist

Please verify the check list and ensure to tick them off before the MR is merged.

• Provided a concise title for this Merge Request (MR)
• Added a description to this MR explaining the reasons for the proposed change, per say why, not just what
  • Copy/paste the Slack conversation to document it for later, or upload screenshots. Verify that no confidential data is added, and the content is SAFE
• Assign reviewers for this MR to the correct Directly Responsible Individual/s (DRI)
  • If the DRI for the page/s being updated isn’t immediately clear, then assign it to one of the people listed in the Maintained by section on the page being edited
  • If your manager does not have merge rights, please ask someone to merge it AFTER it has been approved by your manager in #mr-buddies
  • The when to get approval handbook section explains the workflow in more detail
• For transparency, share this MR with the audience that will be impacted.
  • Team: For changes that affect your direct team, share in your group Slack channel
  • Department: If the update affects your department, share the MR in your department Slack channel
  • Division: If the update affects your division, share the MR in your division Slack channel
  • Company: If the update affects all (or the majority of) GitLab team members, post an update in #whats-happening-at-gitlab linking to this MR
    • For high-priority company-wide announcements work with the internal communications team to post the update in #company-fyi and align on a plan to circulate in additional channels like the "While You Were Iterating" Newsletter

Commits

• doc: Debugging Gitaly

---

content/handbook/engineering/infrastructure/core-platform/systems/gitaly/debug.md

@@ -103,6 +103,25 @@ The query uses [PromQL](https://prometheus.io/docs/prometheus/latest/querying/ba
 - [hyperfine](https://github.com/sharkdp/hyperfine): a performance tool that can benchmarks over time
   - hyperfine can be used together with grpcurl to check the response time of a gPRC call
 
+### strace
+
+`strace(1)` a gitaly process:
+
+```shell
+strace -fttTyyy -s 1024 -o /paht/filename -p $(pgrep -fd, gitaly)
+```
+
+Or wrap a process to make it easy to strace, especially if it then spawns more processes:
+
+```shell
+#!/bin/bash/sh
+echo $(date)" $PPID $@" >> /tmp/gitlab-shell.txt
+exec /opt/gitlab/embedded/service/gitlab-shell/bin/gitlab-shell-orig "$@"
+# strace -fttTyyy -s 1024 -o /tmp/sshd_trace-$PPID /opt/gitlab/embedded/service/gitlab-shell/bin/gitlab-shell-orig
+```
+
+https://gitlab.com/gitlab-com/support/toolbox/strace-parser is useful to make the results more readable.
+
 ## Log analysis
 
 Kibana (Elastic) Dashboards