Upgrade monitoring and logs ES clusters to 8.7

Planning and coordination issue for Elasticsearch cluster upgrades noted in &958 (closed)

Clusters

The Reliability::Observability team is responsible for four clusters that need to be upgraded

1. monitoring-cluster - Currently at 8.6.2, the monitoring cluster should be upgraded before anything else, and should be the same version or newer as the clusters being monitored (reference point 5 here)
2. gitlab-pre - Currently at 7.17.6
3. gitlab-logs-nonprod - Currently at 7.17.6
4. gitlab-logs-prod - Currently at 7.17.6

For the 7.17.6 clusters, while that version is currently supported, it is the oldest supported version will be EOL on 2023-08-01 and we have an OKR to ensure that we have no components out-of-date.

Tasks

1. Review upgrade documentation
2. Identify/assess any pre-requisite or co-requisite changes
   1. Cluster configuration
   2. ES Plugins
3. Tasks
   1. gitlab-pre
      1. Test upgrade on gitlab-pre ES cluster
      2. Delete gitlab-pre cluster (not currently used or needed for anything)
   2. gitlab-logs-nonprod
      1. Create a clone of gitlab-logs-nonprod cluster and do a test upgrade
      2. Upgrade gitlab-logs-nonprod ES cluster
   3. gitlab-logs-prod
      1. Upgrade gitlab-logs-prod ES cluster moved to https://gitlab.com/gitlab-com/gl-infra/reliability/-/issues/23901

Upgrade process

1. List installed plugins
2. Ensure plugins are compatible with new ES version
3. Use upgrade assistant to check for deprecation warnings
4. Launch upgrade from cloud console