Kubernetes Runner - large commit messages cause pipeline to fail

MR:

1. Truncate commit message if it breaches size limit (gitlab!208666)

Summary

This was initially reported by a customer who had run into the issue after a large commit - relevant ticket: https://gitlab.zendesk.com/agent/tickets/164991 (internal use).

Jobs created from commits with large messages (300+kib / thousands of lines) fail with: etcdserver: request is too large.

Steps to reproduce

1. Create a .gitlab-ci.yml that does a job of your choosing (a simple echo will work) and ensure it will be executed via a Kubernetes runner
2. Create a new commit for the relevant repo, ensuring the commit is exceptionally long/large (this was tested with a long commit message that had a size of about 300 KiB)
3. Push the commit and watch the generated pipeline for failure

Actual behavior

The job fails with the error message: Job failed (system failure): prepare environment: etcdserver: request is too large.

Expected behavior

The job either:

• A.) Succeeds
• B.) Fails with a more relevant error message (e.g. "The commit message is too large!")

Relevant logs and/or screenshots

job log

Running with gitlab-runner 13.1.1 (6fbc7474)
  on gitlab-runner3-gitlab-runner-5fdf84bfbb-rhmn9 4UqZ-igj
Preparing the "kubernetes" executor
00:00
Using Kubernetes namespace: runner
Using Kubernetes executor with image docker:19.03.1 ...
Preparing environment
00:01
ERROR: Job failed (system failure): prepare environment: etcdserver: request is too large. Check https://docs.gitlab.com/runner/shells/index.html#shell-profile-loading for more information

runner log

Checking for jobs... received                       job=653357498 repo_url=https://gitlab.com/calebw/ci-test2.git runner=4UqZ-igj
ERROR: Job failed (system failure): prepare environment: etcdserver: request is too large. Check https://docs.gitlab.com/runner/shells/index.html#shell-profile-loading for more information  duration=399.093936ms job=653357498 project=14962722 runner=4UqZ-igj
WARNING: Failed to process runner                   builds=0 error=prepare environment: etcdserver: request is too large. Check https://docs.gitlab.com/runner/shells/index.html#shell-profile-loading for more information executor=kubernetes runner=4UqZ-igj

Environment description

config.toml contents

listen_address = ":9252"
concurrent = 10
check_interval = 30
log_level = "info"

[session_server]
  session_timeout = 1800

[[runners]]
  name = "gitlab-runner3-gitlab-runner-5fdf84bfbb-rhmn9"
  output_limit = 4096
  request_concurrency = 1
  url = "https://gitlab.com/"
  token = "redacted"
  executor = "kubernetes"
  [runners.custom_build_dir]
  [runners.cache]
    [runners.cache.s3]
    [runners.cache.gcs]
  [runners.kubernetes]
    host = ""
    bearer_token_overwrite_allowed = false
    image = "ubuntu:18.04"
    namespace = "runner"
    namespace_overwrite_allowed = ""
    privileged = true
    cpu_request = "100m"
    memory_request = "128Mi"
    service_cpu_request = "100m"
    service_memory_request = "128Mi"
    helper_cpu_request = "100m"
    helper_memory_request = "128Mi"
    poll_timeout = 180
    service_account_overwrite_allowed = ""
    pod_annotations_overwrite_allowed = ""
    [runners.kubernetes.pod_security_context]
    [runners.kubernetes.volumes]

Used GitLab Runner version

Running with gitlab-runner 13.1.1 (6fbc7474)
  on gitlab-runner3-gitlab-runner-5fdf84bfbb-rhmn9 4UqZ-igj

Possible fixes