Add Elastic Stack cluster integration

Merged Hordur Freyr Yngvason requested to merge add-elasticstack-cluster-integration into master

What does this MR do?

Allows GitLab to continue querying Elasticsearch for pod logs. The main idea is for existing integrations to continue working without user intervention once we stop using the cluster_applications_elastic_stack table in %14.0.

In this first iteration, newly created integrations are expected to expose pod logs in a certain format over http on a service called elastic-stack-elasticsearch-master, on port 9200, in the Kubernetes namespace gitlab-managed-apps.

See #326560 (closed)

Screenshots (strongly suggested)

Form State sync Logs
multiple_forms state_sync_720p logs

Edit: Changed approach to multiple forms. Updated only the "Form" recording to save time, as only that part of the functionality was affected.

Does this MR meet the acceptance criteria?


Availability and Testing


If this MR contains changes to processing or storing of credentials or tokens, authorization and authentication methods and other items described in the security review guidelines:

  • [-] Label as security and @ mention @gitlab-com/gl-security/appsec
  • [-] The MR includes necessary changes to maintain consistency between UI, API, email, or other methods
  • [-] Security reports checked/validated by a reviewer from the AppSec team

Migration output

➜  gitlab git:(add-elasticstack-cluster-integration) ✗ bin/rails db:migrate
== 20210504153354 CreateClustersIntegrationElasticstack: migrating ============
-- create_table(:clusters_integration_elasticstack, {:id=>false})
-- quote_column_name(:chart_version)
   -> 0.0000s
   -> 0.0075s
-- quote_table_name("check_f8d671ce04")
   -> 0.0000s
-- quote_table_name(:clusters_integration_elasticstack)
   -> 0.0000s
-- execute("ALTER TABLE \"clusters_integration_elasticstack\"\nADD CONSTRAINT \"check_f8d671ce04\" CHECK (char_length(\"chart_version\") <= 10)\n")
   -> 0.0009s
== 20210504153354 CreateClustersIntegrationElasticstack: migrated (0.0162s) ===

➜  gitlab git:(add-elasticstack-cluster-integration) ✗ bin/rails db:rollback
== 20210504153354 CreateClustersIntegrationElasticstack: reverting ============
-- drop_table(:clusters_integration_elasticstack, {:id=>false})
   -> 0.0020s
== 20210504153354 CreateClustersIntegrationElasticstack: reverted (0.0105s) ===
Edited by Hordur Freyr Yngvason