Webservice: make Workhorse stick around for SHUTDOWN_BLACKOUT_SECONDS

What does this MR do?

Webservice: make Workhorse stick around for SHUTDOWN_BLACKOUT_SECONDS

Add a preStop hook because Workhorse goes on vacation at lunchtime.

Workhorse does not respect a graceful shutdown window at this time. As
a precaution, and to address false HTTP 502s, add a preStop `sleep` for
`SHUTDOWN_BLACKOUT_SECONDS` in accordance with what Rails expects, and
what upstreams require.

Changelog: fixed

Related issues

Related to #2943 (closed)

Closes #3367 (closed)

Closes #4405 (closed)

Closes #4025 (closed)

Checklist

See Definition of done.

For anything in this list which will not be completed, please provide a reason in the MR discussion.

Testing

Note: additional testing notes in !3084 (comment 1347484373)

1. Deploy with default values.
2. Delete the Webservice HPA.
3. Manually scale the Webservice Deployment to 3 or 4.
4. Run the following in a separate session: while true; do curl -I https://yourinstance.com/users/sign_in 2>&1 | grep '^HTTP/' ; sleep 0.1 ; done
5. Confirm only 200s in output.

You can also test that the sleep duration is working as expected by redirecting the output of an echo in the prestop:

        lifecycle:
          preStop:
            exec:
              command:
              - /bin/bash
              - -c
              - echo "shutting down after sleeping for $SHUTDOWN_BLACKOUT_SECONDS...">/proc/1/fd/1
                && sleep $SHUTDOWN_BLACKOUT_SECONDS

Then tail the logs:

gitlab-webservice-default-6db57cfb77-v277h gitlab-workhorse shutting down after sleeping for 10...

Required

• Merge Request Title and Description are up to date, accurate, and descriptive
• MR targeting the appropriate branch
• MR has a green pipeline on GitLab.com
• When ready for review, MR is labeled "~workflow::ready for review" per the Distribution MR workflow

Expected (please provide an explanation if not completing)

• Test plan indicating conditions for success has been posted and passes
• Documentation created/updated
• Tests added
• Integration tests added to GitLab QA
• Equivalent MR/issue for omnibus-gitlab opened
• Validate potential values for new configuration settings. Formats such as integer 10, duration 10s, URI scheme://user:passwd@host:port may require quotation or other special handling when rendered in a template and written to a configuration file.