Update com infrastructure to deploy some knowledge graph nodes

Everyone can contribute. Help move this issue forward while earning points, leveling up and collecting rewards.

• Close this issue

By default existing nodes are considered as zoekt-only, we will need to update .com deployment runbooks to deploy at least 2 knowledge graph nodes to test knowledge graph on .com

Update: related to #540786 (closed), this may not be needed - it depends if we decide to set list of services from node side, or if we to automatic categorization/service assignment on Rails side based on existing node attributes.

These options were discussed briefly in this Slack thread:



jprovaznik
jprovaznik  Yesterday at 13:12
@dgruzd
 
@dylan
 , to follow-up on our previous discussion of serving knowledge graph tasks by zoekt-indexer on zoekt nodes: does the approach outlined in this schema make sense to you? https://gitlab.com/gitlab-org/gitlab/-/issues/540786#schema


dgruzd
dgruzd  Yesterday at 13:15
 Initially, we could update the zoekt_nodes records manually to set a specific role without adding it to the heartbeat API


dgruzd
dgruzd  Yesterday at 13:16
Maybe we could even consider automatic role assignment on the rails side. Sending it from the indexer might be a bit challenging for some deployment methods since all nodes are equal


jprovaznik
jprovaznik  Yesterday at 13:18
That's a valid point, that sendign this from zoekt-indexer may not be necessary and just adds extra complexity


jprovaznik
jprovaznik  Yesterday at 13:18
I like the idea of setting this just Rails side 

1

1



dylan
dylan  Yesterday at 13:23
@dgruzd
 I think that seems easy in the short term but I guess it begs the question about how we'd want to manage these long term. Would we have Rails automatically allocating a node to knowledge_graph if the setting is enabled? Would we want some kind of admin APIs? I'm thinking that this configuration is the kind of data that you'd want managed in infrastructure config (e.g. helm chart config) and that managing the total node count via helm while having Rails choose the split between Zoekt and Knowledge Graph might not be ideal for GitLab operators.


dgruzd
dgruzd  Yesterday at 13:30
I think long-term we might want to have some admin UI to manage these roles. I'd prefer not to specify it on the infrastructure level since it introduces quite a bit of friction.
In general, with Zoekt, we've been trying to make it automatic as much as possible to make admin's job easier. Ideally, we'd want to have some custom logic that would dynamically split available nodes if this setting is enabled. If not, we'd use available nodes for both (the default).
I'd agree with infrastructure-based assignment if we had separate resource requests/limits for zoekt and knowledge graph nodes. Maybe we'll come to that eventually


dylan
dylan  Yesterday at 13:38
I'd agree with infrastructure-based assignment if we had separate resource requests/limits for zoekt and knowledge graph nodes. Maybe we'll come to that eventually
Yeah I guess it depends on that. In any case controlling it via Rails console initially doesn't really limit us for what we can do in future.


jprovaznik
jprovaznik  Yesterday at 13:39
Ideally, we'd want to have some custom logic that would dynamically split available nodes if this setting is enabled. If not, we'd use available nodes for both (the default).
There is a scenario when a customer wants to have both separate from the beginning, so this dynamic split should still allow customers to have both separate from the first heartbeat (to make sure none of unwanted type of tasks are scheduled for the node before it's disabled manually).
I can imagine this dynamic split could be based even just on node name or some other existing parameter we already send

1



dylan
dylan  Yesterday at 13:44
even just on node name
I'd probably avoid that as we'll be creating unnecessary technical debt if we have application logic based on how we name our VMs. It might make migrations harder in future and may create a blocker for certain customers to adopt it where they don't have full flexibility to name their nodes.


jprovaznik
jprovaznik  Yesterday at 13:48
That's true. Not sure then how the automatic split should look like. I agree that it's not a blocker for current development phase but I think we will have to clarify this before release


dgruzd
dgruzd  Yesterday at 13:51
A quick note: we probably shouldn't assume that there will only be two roles. Later, we might want to introduce an N+1 role to GitLab Zoekt (I'm still thinking about a good name for our distributed file-based database)

2



jprovaznik
jprovaznik  Yesterday at 13:54
Yes. exactly

Update

The new nodes will be deployed using another Helm Chart Stateful set. Please see this issue: gitlab-org/cloud-native/charts/gitlab-zoekt#30 (closed). So we would need to set values on the helm release to deploy the nodes.

This is to be done on Staging first, then on Production