Make kubernetes API retries configurable
What does this MR do?
It allows to specify limit for Kubernetes API calls instead of hardcoded limit in const
Why was this MR needed?
We run our jobs in EKS Kuberentes and use Karpenter for scaling nodes.
When using default limit, our jobs often fail with prepare environment: setting up trapping scripts on emptyDir: error dialing backend: remote error: tls: internal error
When we bumped defaultTries
to 35, issue disappeared. Therefore we made this limit configurable defaulting to old limit, not to enforce higher limit for everyone.
We're running this code on on production since 6.12.2023 and successfully processed more than 6000 jobs since then.
What's the best way to test this MR?
What are the relevant issue numbers?
Merge request reports
Activity
Hey @m.skibicki!
Welcome to our community! We're excited to have you here, and can't wait to review this first MR with you!
Thank you for your contribution to GitLab. Please refer to the contribution documentation for an overview of the process.
When you're ready for a first review, post
@gitlab-bot ready
. If you know a relevant reviewer(s) (for example, someone that was involved in a related issue), you can also assign them directly with@gitlab-bot ready @user1 @user2
.At any time, if you need help, feel free to post
@gitlab-bot help
or initiate a mentor session on Discord. Read more on how to get help.Some contributions require several iterations of review and we try to mentor contributors during this process. However, we understand that some reviews can be very time consuming. If you would prefer for us to continue the work you've submitted now or at any point in the future please let us know.
If you're okay with being part of our review process (and we hope you are!), there are several initial checks we ask you to make:
- The merge request description clearly explains:
- The problem being solved.
- The best way a reviewer can test your changes (is it possible to provide an example?).
- If the pipeline failed, do you need help identifying what failed?
- Check that Go code follows our Go guidelines.
- Read our contributing to GitLab Runner document.
This message was generated automatically. You're welcome to improve it.
- The merge request description clearly explains:
added 1st contribution Community contribution workflowin dev labels
assigned to @m.skibicki
3 Warnings This merge request includes more than 10 commits. Each commit should meet the following criteria: - Have a well-written commit message.
- Has all tests passing when used on its own (e.g. when using git checkout SHA).
- Can be reverted on its own without also requiring the revert of commit that came before it.
- Is small enough that it can be reviewed in isolation in under 30 minutes or so.
If this merge request contains commits that do not meet this criteria and/or contains intermediate work, please rebase these commits into a smaller number of commits or split this merge request into multiple smaller merge requests.
7e4a55e9: The commit subject must contain at least 3 words. For more information, take a look at our Commit message guidelines. 47f9589a: Commits that change 30 or more lines across at least 3 files should describe these changes in the commit body. For more information, take a look at our Commit message guidelines. Reviewer roulette
Changes that require review have been detected! A merge request is normally reviewed by both a reviewer and a maintainer in its primary category and by a maintainer in all other categories.
To spread load more evenly across eligible reviewers, Danger has picked a candidate for each review slot. Feel free to override these selections if you think someone else would be better-suited or use the GitLab Review Workload Dashboard to find other available reviewers.
To read more on how to use the reviewer roulette, please take a look at the Engineering workflow and code review guidelines. Please consider assigning a reviewer or maintainer who is a domain expert in the area of the merge request.
Once you've decided who will review this merge request, mention them as you normally would! Danger does not automatically notify them for you.
Reviewer Maintainer @vshushlin
(UTC+1)
@ajwalker
(UTC+0)
If needed, you can retry the
danger-review
job that generated this comment.Generated by
Dangerrequested review from @ggeorgiev_gitlab
added Category:Runner Core devopsverify grouprunner labels
added sectionci label
mentioned in issue gitlab-org/quality/triage-reports#15257 (closed)
changed milestone to %16.7
added featureenhancement typefeature labels
- Resolved by Michał Skibicki
- Resolved by Michał Skibicki
- Resolved by Michał Skibicki
- Resolved by Michał Skibicki
@m.skibicki A few comments, let me know what you think
These were mostly limitations with my original design.
- Resolved by Michał Skibicki
- Resolved by Michał Skibicki
mentioned in merge request !4529 (closed)
@m.skibicki, it seems we're waiting on an action from you for approximately two weeks.
- Do you still have capacity to work on this? If not, you might want to close this MR and/or ask someone to take over.
- Do you need help in getting it ready? At any time, you can:
- If you're actually ready for a review, you can post
@gitlab-bot ready
.
This message was generated automatically. You're welcome to improve it.
added automation:author-reminded label
added 1 commit
- ac937c55 - Check if the KubernetesConfig instance is nil
added 202 commits
-
1ba30613...9000c3c0 - 195 commits from branch
gitlab-org:main
- 2b82c015 - Make kubernetes API retries configurable
- 023d3914 - Change configuration option name
- 47f9589a - Apply changes suggested from ggeorgiev
- f6dec965 - Format code and add regenerate mockery
- d32825e9 - Check if the KubernetesConfig instance is nil
- 7e4a55e9 - Fix whitespaces
- 5af44ad3 - Correct integration tests
Toggle commit list-
1ba30613...9000c3c0 - 195 commits from branch
@gitlab-bot ready
added workflowready for review label and removed workflowin dev label
Hi
@aqualls
! Please review this documentation merge request. This message was generated automatically. You're welcome to improve it.added documentation twtriaged labels
requested review from @aqualls
@ggeorgiev_gitlab
@aqualls
, this Community contribution is ready for review.- Do you have capacity and domain expertise to review this? If not, find one or more reviewers and assign to them.
- If you've reviewed it, add the workflowin dev label if these changes need more work before the next review.
This message was generated automatically. You're welcome to improve it.
removed automation:author-reminded label
- Resolved by Michał Skibicki
- Resolved by Michał Skibicki
added Technical Writing label
- Resolved by Fiona Neill
@fneill I've left some minor changes here, but I'm not knowledgeable enough about Runner to be a truly helpful reviewer. Sending your way. (Aside: I see a multiple style improvements in
common/config.go
to lines that already exist. Lots of small polish.)
Noting that a large GitLab Ultimate customer is currently facing the
remote error: tls: internal error
error in their Karpenter/EKS runner setup when higher numbers of jobs run at once and that this MR may help them avoid the disruption they are facing (ZD internal link).added customer label
requested review from @ggeorgiev_gitlab
mentioned in issue #37244 (closed)
Thanks @m.skibicki for this contribution. The changes here LGTM, approving from my side.
removed review request for @fneill
mentioned in issue #37331
@vshushlin Can we get a review?
@m.skibicki I'm not really an expert on this, and I see that Georgi already started the review process.
@ggeorgiev_gitlab can you take another look? Looks like all the previous discussions are now resolved.
Thank you @m.skibicki! Adding to merge train!
changed milestone to %16.9
mentioned in issue #37349 (closed)
mentioned in commit 25aa6fe5
@m.skibicki, how was your code review experience with this merge request? Please tell us how we can continue to iterate and improve:
- React with a
or a on this comment to describe your experience. - Create a new comment starting with
@gitlab-bot feedback
below, and leave any additional feedback you have for us in the comment.
Subscribe to the GitLab Community Newsletter for contributor-focused content and opportunities to level up.
Thanks for your help!
This message was generated automatically. You're welcome to improve it.
- React with a
@m.skibicki, congratulations for getting your first MR merged
If this is your first MR against a GitLab project, we'd like to invite and encourage you to self-nominate yourself for
First MR Merged
swag prize here.Thank you again for contributing, what's your next contribution going to be?
This message was generated automatically. You're welcome to improve it.