Technical Breakdown for Monitor: Logging Removal

This is the technical breakdown for Monitor:Logging Deprecation (gitlab-org/gitlab#346485 - closed).

"Logging" today refers to:

Users can install ElasticStack in the gitlab-managed-apps namespace on their Kubernetes cluster, then view the pod logs in GitLab. This has four main parts:

  1. Rendering logs in the UI [/namespace/project/-/logs]
  2. Navigating to the logs in the UI [via metrics dashboard, left nav, environments list]
  3. Enabling GitLab to access pod logs [via certificate & gitlab-managed-apps or cluster integration]
  4. Installing ElasticStack [via cluster management project (/+template)]

All of these parts will be in scope of the removal.

User impact

Looks like the metric we had for logging got (partially?) removed gitlab-org/gitlab#321786 (closed), gitlab-org/gitlab!61872 (merged), gitlab-org/gitlab#357256 (closed). So we won't need to go through deprecating that! But it's also harder to know the type of impact this change will have.

We can estimate usage by number of clusters with ElasticStack installed. That's ~1%, which is ~1500 clusters installations.

Timeline

To guarantee we can effectively turn off the feature in %15.0, we can start by feature flagging logging. This should protect our team's flexibility.

Phase 1 - Add feature flag (%15.0)

  • Introduce feature flag
  • Update documentation to reference flag

Phase 2 - Code removal (anytime after %15.2)

Removing logs

We want to remove any files, routes, UI, etc which allow users to view log data.

Frontend

  • remove link to 'View logs' from metrics dashboard charts
  • remove Monitor > Logs & left nav item
  • remove link to logs from pod markers on Deployments > Environments
  • remove relevant docs
Relevant files
  • app/assets/javascripts/logs/*
  • app/assets/javascripts/environments/components/deploy_board.vue
  • app/assets/javascripts/environments/components/environments_table.vue
  • app/assets/javascripts/monitoring/components/dashboard_panel.vue
  • app/assets/javascripts/monitoring/components/dashboard.vue
  • app/assets/javascripts/vue_shared/components/deployment_instance.vue
  • ee/app/assets/javascripts/clusters/components/environments.vue
  • app/views/projects/logs/empty_logs.html.haml
  • app/views/projects/logs/index.html.haml

Backend

Relevant files
  • app/controllers/projects/logs_controller.rb
  • config/routes/project.rb
  • app/services/pod_logs/kubernetes_service.rb
  • app/services/pod_logs/elasticsearch_service.rb
  • app/services/pod_logs/base_service.rb
  • lib/gitlab/elasticsearch/logs/lines.rb
  • lib/gitlab/elasticsearch/logs/pods.rb
  • .rubocop_todo/layout/line_length.yml
  • app/serializers/environment_entity.rb [Decommission fields, not remove?]
  • app/serializers/cluster_entity.rb [Decommission fields, not remove?]

Removing log configuration

We want to remove any files, routes, UI, etc which allow users to configure their clusters to produce logs.

Frontend

  • remove setting to enable elastic stack from Infrastructure > Kubernetes clusters > cluster > Integrations
  • remove relevant docs
Relevant files
  • app/views/clusters/clusters/_integrations.html.haml
  • app/views/clusters/clusters/_namespace.html.haml

Backend

  • deprecate clusters_integrations_elastic_stack metric
  • remove Elastic Stack routes/controllers, application code
  • drop clusters_applications_elastic_stacks table & clean up database dictionary records [can be done in a post-migration in same release as removes all app code]
  • drop clusters_integration_elasticstack table & clean up database dictionary records [can be done in a post-migration in same release as removes all app code]
  • remove the elastic-stack directory from the cluster management project template
Relevant files
  • app/controllers/clusters/clusters_controller.rb
  • app/helpers/environments_helper.rb
  • app/models/clusters/cluster.rb
  • app/models/clusters/applications/elastic_stack.rb
  • app/models/clusters/applications/elastic_stack.rb
  • app/models/clusters/integrations/elastic_stack.rb
  • app/models/clusters/concerns/elasticsearch_client.rb
  • app/presenters/clusters/integration_presenter.rb
  • app/models/environment.rb
  • app/presenters/clusters/cluster_presenter.rb
  • app/services/clusters/integrations/create_service.rb
  • lib/gitlab/usage_data.rb
  • config/metrics/counts_all/20210216175309_clusters_applications_elastic_stack.yml
  • config/metrics/counts_all/20210916200931_clusters_integrations_elastic_stack.yml
  • .rubocop_todo/layout/line_length.yml
  • db/docs/clusters_applications_elastic_stacks.yml
  • db/docs/clusters_integration_elasticstack.yml
  • lib/gitlab/database/gitlab_schemas.yml

Assumptions & Limitations

  1. With the removal of logging, it makes sense to remove ~"group::configure"'s elasticstack-related features as well.
  2. The logging feature category isn't being removed.
  3. Direct communication with owners of impacted projects is out of scope.
  4. With the number of removals in %15.0, there are decent possibilities engineers could be pulled to other tasks/teams

Issue breakdown

Issue Notes Backend Frontend
Add feature flag for logging UI Either backend or frontend -> M
Delete logging frontend code ✅
Delete logging configuration frontend code ✅
Delete logging backend code ✅
Deprecate logging configuration usage metrics ✅
Delete logging configuration backend code ✅
Drop clusters_applications_elastic_stacks & clusters_integration_elasticstack tables ✅

Related issues:

Edited by Sarah Yasonik