clean up logging clusters after not using aliases
follow up on: production#1748 (closed)
-
the same is happening in nonprod -
registry
,sidekiq
-
mailroom
in gstg- created index templates from: gitlab-com/runbooks!2026 (merged)
- stopped fluentd from sending logs:
kubectl -n logging delete daemonset fluentd-elasticsearch
- removed
pubsub-inf-mailroom-gstg
index - created an alias and the first index
- restarted fluentd:
kubectl apply -f fluentd_daemonset.yaml
-
mailroom
in pre- https://gitlab.com/gitlab-cookbooks/gitlab-server/-/merge_requests/212
- followed the same steps as for gstg
-
release-tools
:-
removed all release-tools-*
indices, alias, index template -
adjust the runbooks repo to start using release_tools
instead ofrelease-tools
: gitlab-com/runbooks!2035 (merged) -
adjust client config to send logs to release_tools
(rather than create indices): gitlab-org/release-tools!912 (closed) -
index pattern for release_tools
is already in place -
existing indices use the release_tools-<date>
format, so there's no need to reindex anything -
switch to only managing the delete phase of indices: gitlab-com/runbooks!2059 (merged)
-
-
static-objects-cache
-
configure the cluster to start using an alias: gitlab-com/runbooks!2037 (diffs) -
initialize the alias with the first index -
switch static-objects-cache
from using a suffix to using an alias: https://ops.gitlab.net/gitlab-com/gl-infra/terraform-modules/static-objects-cache/-/merge_requests/22 -
trigger a release -
use the new module in gstg: https://ops.gitlab.net/gitlab-com/gitlab-com-infrastructure/-/merge_requests/1515 - Turns out we save the ES indices to cold storage: https://ops.gitlab.net/gitlab-com/gl-infra/static-objects-cache-logs-exporter/-/blob/master/.gitlab-ci.yml so we can't use aliases. However, the index template with the appropriate ILM policy can still be used.
-
runbooks update: gitlab-com/runbooks!2050 (merged) -
tf module update: simply close https://ops.gitlab.net/gitlab-com/gitlab-com-infrastructure/-/merge_requests/1515
-
-
-
production logging cluster -
registry
,sidekiq
-
mailroom
- added mailroom to list of managed indices in elastic: gitlab-com/runbooks!2026 (merged)
- created index templates
- marked the index as read-only
- cloned the existing index
- stopped fluentd DaemonSet
- removed index
- created an alias and the first index
- restarted fluentd
- fixed index mapping conflicts and confirmed the index is searchable in Kibana: https://log.gprd.gitlab.net/goto/5e435fac1eed61a61ef5275dc897bde0
-
static-objects-cache
:-
prep cluster -
roll out new tf module version
-
-
-
make sure there are no ILM errors firing in the logging clusters
Edited by Michal Wasilewski