ci_pipelines.yaml_errors and pipeline_messages are recording duplicate data

The Problem

It was very helpfully noticed by @mbobin and @fabiopitino that ci_pipelines.yaml_errors and pipeline_messages are recording the same data. They appear to have been doing this for several years

    module Pipeline
      module Chain
        module Helpers
          def error(message, failure_reason: nil)
            sanitized_message = ActionController::Base.helpers.sanitize(message, tags: [])

            pipeline.yaml_errors = sanitized_message if failure_reason == :config_error

            pipeline.add_error_message(sanitized_message)

while our database creaks under the weight of its tables.

Proposal (updated)

  • Update yaml_errors in the Ci::Pipeline model to use error_messages <- workflowin review !178041 (diffs)
  • Remove duplicate collection of errors in the being done via pipeline.yaml_errors <- this is the for stopping the recording of duplicate data - THIS ISSUE AIMS TO SOLVE THIS - it's blocked by the MR up above. Once that is merged, this can be reviewed again (MR is already created and "pre-reviewed")
  • Drop pipeline.yaml_errors column - this will take 3 milestones as per guidelines
  • Update locations for yaml_errors to use error_messages instead in the FE
    • These can be updated as technical debt although some have already been updated:
      • FRONTEND:~~ app/assets/javascripts/ci/pipelines_page/components/pipeline_labels.vue no need to fix, ref is changed here~~ #507283 (closed) -> !179041 (merged) seems like the FE code isn't actually in use and is under a feature flag, so no need to fix here. More info in the MR.

        FRONTEND:~~ app/assets/javascripts/ci/pipeline_details/header/components/header_badges.vue

      • FRONTEND:~~ app/assets/javascripts/ci/merge_requests/utils.js

      • FRONTEND: app/assets/javascripts/ide/stores/modules/pipelines/mutations.js~~ <- this is going away in 18.0 so we can ignore until and is unrelated to the dupe errors.

      • **ANY:**~~app/views/projects/pipelines/show.html.haml~~~

      • BACKEND:~~ app/models/integrations/chat_message/pipeline_message.rb~~

Historical context: Drop one or the other. It appears that we created pipeline_messages to try to take some data off the ci_pipelines table, so the move is probably to drop the yaml_errors column and use the related pipeline messages everywhere in the app.

Edited by Laura Montemayor