Topology Service Client ruby gem is integrated in GitLab Rails

The purpose of this issue is to allow using Topology Service in GitLab Rails:

• Topology Service Client is scaffolded
• Topology Service once released published ruby gem with a client interface
• The Topology Service Client is integrated into GitLab Rails bundle. Done here !159778 (merged)
• Document the steps to keep the Topology Service Gem up to date in the Monolith Application. See the comment here for context.
• [ ] The GitLab Rails has easily accessible instance of all exposed Topology Service services. Moved into its own issue: #474660 (closed)

added to epic &13158 (closed)

marked this issue as related to #451051 (closed)

changed the description

@ayufan, Please add a group or category label to identify issue ownership.

You can refer to the Features by Group handbook page for guidance.

If you are unsure about the correct group, please do not leave the issue without a group label, and refer to GitLab's shared responsibility functionality guidelines for more information on how to triage this kind of issue.

This message was generated automatically. You're welcome to improve it.

added Category:Cell grouptenant scale labels

added devopsdata stores sectioncore platform labels

added backend typefeature labels

changed the description

changed milestone to %Backlog

added workflowrefinement label

changed title from Global Service Client ruby gem is integrated in GitLab Rails to Topology Service Client ruby gem is integrated in GitLab Rails

changed the description

changed milestone to %17.1

added milestonep1 label

marked this issue as blocked by #451042 (closed)

@OmarQunsulGitlab @tkuah

My opinion is that Topology Service will have a Ruby Client to interact with Topology Service. However, I would prefer that we "vendor-in" the Gem, and keep updating it every time there's a major change (like in a protobuf).

It will be quite overhead to push the gem to RubyGems on a constant basis. We should be fine with just pushing the gem to vendor/gems/ folder.

WDYT?

Let me know if this works for you. I pushed initial scaffolding: gitlab-org/cells/topology-service@4bc1f87c and claimed a RubyGem.

Ref.: https://rubygems.org/gems/gitlab-topology-service-client

It will be quite overhead to push the gem to RubyGems on a constant basis. We should be fine with just pushing the gem to vendor/gems/ folder.

@ayufan Yes, I agree

@ayufan I don't have experience with Ruby Gems development to give an opinion Of course when I worked on the PoC, I had to change Gemfile to read from a local developed Gem

My question is: How do we keep gitlab-org/gitlab/vendor/gems folder up-to-date with the generated gem from cells/topology-service. Do we copy the files manually every time?

@OmarQunsulGitlab

My question is: How do we keep gitlab-org/gitlab/vendor/gems folder up-to-date with the generated gem from cells/topology-service. Do we copy the files manually every time?

Yes, you "vendor" when you need to update it. Protobuf by design is forward/backward compatible. We can have a simple script in GitLab Rails to pull the folder.

@ayufan @OmarQunsulGitlab I see we have https://gitlab.com/explore/catalog/gitlab-org/components/gem-release which can automate a lot.

But we can vendor while things are still pre 1.0

@ayufan @tkuah my proposal for automating this is as following:

1. [Manual] First step. Integrate the current version of the Topology Service Gem manually. I opened an MR for this: !159778 (merged). I can submit it for review if this suggestion sounds good.
2. [Automation] Every time Topology Service pipeline runs on main branch. Maybe it can be a daily scheduled pipeline or on each update, it can check if the code/version is different between gitlab-org/gitlab:vendor/gems/gitlab-topology-service-client/lib and Topology Service clients/ruby. If there is a difference, then it can automatically open an MR with the diffs, and maybe close existing MR if there is an open one.

What do you think?

@OmarQunsulGitlab

Hmm, that kind of automation seems to be more work compared to publishing a new gem using gem release CI component, then using renovate to bump in GitLab monolith.

Something manual is good to start. As we just need to cp to the vendor/gems directory. Just make sure to add some docs. We can then move towards rubygems.org/renovate automation in the future.

@tkuah True. Let's skip the automation for now then and copy the Gem manually every time the Topology Service API changes. I will add some docs to the Topology Service as well once !159778 (merged) is reviewed and merged

@OmarQunsulGitlab @tkuah I think we might have scripts/update-topology-service-client.bash that would do it, and refer in docs to do it to avoid doing manual cp.

@OmarQunsulGitlab How we publish gitaly? Is there some sort of automation to publish it on a new version tag?

@ayufan by checking quickly the .gitlab-ci.yml of the Gitaly project, I see that they have these two CI/CD jobs that build and publishes the gem: https://gitlab.com/gitlab-org/gitaly/-/blob/master/.gitlab-ci.yml?ref_type=heads#L472-497

They do that on new tags/releases: https://gitlab.com/gitlab-org/gitaly/-/blob/master/doc/PROCESS.md?ref_type=heads#publishing-the-ruby-gem

changed the description