Technical Breakdown for Monitor: Logging Removal
This is the technical breakdown for Monitor:Logging Deprecation (gitlab-org/gitlab#346485 - closed).
"Logging" today refers to:
Users can install ElasticStack in the gitlab-managed-apps
namespace on their Kubernetes cluster, then view the pod logs in GitLab. This has four main parts:
- Rendering logs in the UI [
/namespace/project/-/logs
] - Navigating to the logs in the UI [via metrics dashboard, left nav, environments list]
- Enabling GitLab to access pod logs [via certificate & gitlab-managed-apps or cluster integration]
- Installing ElasticStack [via cluster management project (/+template)]
All of these parts will be in scope of the removal.
User impact
Looks like the metric we had for logging got (partially?) removed gitlab-org/gitlab#321786 (closed), gitlab-org/gitlab!61872 (merged), gitlab-org/gitlab#357256 (closed). So we won't need to go through deprecating that! But it's also harder to know the type of impact this change will have.
We can estimate usage by number of clusters with ElasticStack installed. That's ~1%, which is ~1500 clusters installations.
Timeline
To guarantee we can effectively turn off the feature in %15.0, we can start by feature flagging logging. This should protect our team's flexibility.
%15.0)
Phase 1 - Add feature flag (- Introduce feature flag
- Update documentation to reference flag
%15.2)
Phase 2 - Code removal (anytime afterRemoving logs
We want to remove any files, routes, UI, etc which allow users to view log data.
Frontend
- remove link to 'View logs' from metrics dashboard charts
- remove
Monitor > Logs
& left nav item - remove link to logs from pod markers on
Deployments > Environments
- remove relevant docs
Relevant files
- app/assets/javascripts/logs/*
- app/assets/javascripts/environments/components/deploy_board.vue
- app/assets/javascripts/environments/components/environments_table.vue
- app/assets/javascripts/monitoring/components/dashboard_panel.vue
- app/assets/javascripts/monitoring/components/dashboard.vue
- app/assets/javascripts/vue_shared/components/deployment_instance.vue
- ee/app/assets/javascripts/clusters/components/environments.vue
- app/views/projects/logs/empty_logs.html.haml
- app/views/projects/logs/index.html.haml
Backend
-
cleanup PodLogs metric,gitlab-org/gitlab!84966 (merged)UsageCounters
module - Remove logging routes/controllers, application code
- Nullify REST API fields
Relevant files
- app/controllers/projects/logs_controller.rb
- config/routes/project.rb
- app/services/pod_logs/kubernetes_service.rb
- app/services/pod_logs/elasticsearch_service.rb
- app/services/pod_logs/base_service.rb
- lib/gitlab/elasticsearch/logs/lines.rb
- lib/gitlab/elasticsearch/logs/pods.rb
- .rubocop_todo/layout/line_length.yml
- app/serializers/environment_entity.rb [Decommission fields, not remove?]
- app/serializers/cluster_entity.rb [Decommission fields, not remove?]
Removing log configuration
We want to remove any files, routes, UI, etc which allow users to configure their clusters to produce logs.
Frontend
- remove setting to enable elastic stack from
Infrastructure > Kubernetes clusters > cluster > Integrations
- remove relevant docs
Relevant files
- app/views/clusters/clusters/_integrations.html.haml
- app/views/clusters/clusters/_namespace.html.haml
Backend
- deprecate
clusters_integrations_elastic_stack
metric - remove Elastic Stack routes/controllers, application code
- drop
clusters_applications_elastic_stacks
table & clean up database dictionary records [can be done in a post-migration in same release as removes all app code] - drop
clusters_integration_elasticstack
table & clean up database dictionary records [can be done in a post-migration in same release as removes all app code] - remove the
elastic-stack
directory from the cluster management project template
Relevant files
- app/controllers/clusters/clusters_controller.rb
- app/helpers/environments_helper.rb
- app/models/clusters/cluster.rb
- app/models/clusters/applications/elastic_stack.rb
- app/models/clusters/applications/elastic_stack.rb
- app/models/clusters/integrations/elastic_stack.rb
- app/models/clusters/concerns/elasticsearch_client.rb
- app/presenters/clusters/integration_presenter.rb
- app/models/environment.rb
- app/presenters/clusters/cluster_presenter.rb
- app/services/clusters/integrations/create_service.rb
- lib/gitlab/usage_data.rb
- config/metrics/counts_all/20210216175309_clusters_applications_elastic_stack.yml
- config/metrics/counts_all/20210916200931_clusters_integrations_elastic_stack.yml
- .rubocop_todo/layout/line_length.yml
- db/docs/clusters_applications_elastic_stacks.yml
- db/docs/clusters_integration_elasticstack.yml
- lib/gitlab/database/gitlab_schemas.yml
Assumptions & Limitations
- With the removal of logging, it makes sense to remove ~"group::configure"'s elasticstack-related features as well.
- The
logging
feature category isn't being removed. - Direct communication with owners of impacted projects is out of scope.
- With the number of removals in %15.0, there are decent possibilities engineers could be pulled to other tasks/teams
Issue breakdown
Related issues:
- Monitor:Logging Deprecation (gitlab-org/gitlab#346485 - closed) [Tracking the removal]