.gitlab/issue_templates/Security developer workflow.md

@@ -9,19 +9,17 @@ Set the title to: `Description of the original issue`
 ## Prior to starting the security release work
 
 - [ ] Read the [security process for developers] if you are not familiar with it.
-- [ ] Mark this [issue as related] to the Security Release tracking issue. You can find it on the topic of the `#releases` Slack channel.
-- [ ] Run `scripts/security-harness` in your local repository to prevent accidentally pushing to any remote besides `gitlab.com/gitlab-org/security`.
+- [ ] Mark this [issue as related] to the Security Release Tracking Issue. You can find it on the topic of the `#releases` Slack channel.
 - Fill out the [Links section](#links):
   - [ ] Next to **Issue on GitLab**, add a link to the `gitlab-org/gitlab` issue that describes the security vulnerability.
-  - [ ] Next to **Security Release tracking issue**, add a link to the security release issue that will include this security issue.
 
 ## Development
 
+- [ ] Run `scripts/security-harness` in your local repository to prevent accidentally pushing to any remote besides `gitlab.com/gitlab-org/security`.
 - [ ] Create a new branch prefixing it with `security-`.
 - [ ] Create a merge request targeting `master` on `gitlab.com/gitlab-org/security` and use the [Security Release merge request template].
-- [ ] Follow the same [code review process]: Assign to a reviewer, then to a maintainer.
 
-After your merge request has been approved according to our [approval guidelines], you're ready to prepare the backports
+After your merge request has been approved according to our [approval guidelines] and by a team member of the AppSec team, you're ready to prepare the backports
 
 ## Backports
 
@@ -49,7 +47,6 @@ After your merge request has been approved according to our [approval guidelines
 | Description | Link |
 | -------- | -------- |
 | Issue on [GitLab](https://gitlab.com/gitlab-org/gitlab/issues) | #TODO  |
-| Security Release tracking issue | #TODO  |
 
 ### Details
 

.gitlab/merge_request_templates/Security Release.md

@@ -13,25 +13,33 @@ See [the general developer security release guidelines](https://gitlab.com/gitla
 ## Developer checklist
 
 - [ ] **On "Related issues" section, write down the [GitLab Security] issue it belongs to (i.e. `Related to <issue_id>`).**
-- [ ] Merge request targets `master`, or `X-Y-stable` for backports.
+- [ ] Merge request targets `master`, or a versioned stable branch (`X-Y-stable-ee`).
 - [ ] Milestone is set for the version this merge request applies to. A closed milestone can be assigned via [quick actions].
 - [ ] Title of this merge request is the same as for all backports.
-- [ ] A [CHANGELOG entry](https://docs.gitlab.com/ee/development/changelog.html) is added without a `merge_request` value, with `type` set to `security`
-- [ ] Assign to a reviewer and maintainer, per our [Code Review process].
+- [ ] A [CHANGELOG entry] is added without a `merge_request` value, with `type` set to `security`
 - [ ] For the MR targeting `master`:
-  - [ ] Ask for a non-blocking review from the AppSec team member associated to the issue in the [Canonical repository](https://gitlab.com/gitlab-org/gitlab). If you're unsure who to ping, ask on `#sec-appsec` Slack channel.
+  - [ ] Assign to a reviewer and maintainer, per our [Code Review process].
   - [ ] Ensure it's approved according to our [Approval Guidelines].
-- [ ] Merge request _must not_ close the corresponding security issue, _unless_ it targets `master`.
+  - [ ] Ensure it's approved by an AppSec engineer.
+    - If you're unsure who should approve, find the AppSec engineer associated to the issue in the [Canonical repository], or ask #sec-appsec on Slack.
+    - Trigger the [`package-and-qa` build]. The docker image generated will be used by the AppSec engineer to validate the security vulnerability has been remediated.
+  - [ ] Merge request _must_ close the corresponding security issue.
+- [ ] For a backport MR targeting a versioned stable branch (`X-Y-stable-ee`)
+  - [ ] Ensure it's approved by a maintainer.
 
 **Note:** Reviewer/maintainer should not be a Release Manager
 
 ## Maintainer checklist
+
 - [ ] Correct milestone is applied and the title is matching across all backports
 - [ ] Assigned to `@gitlab-release-tools-bot` with passing CI pipelines and **when all backports including the MR targeting master are ready.**
 
 /label ~security
 
 [GitLab Security]: https://gitlab.com/gitlab-org/security/gitlab
-[approval guidelines]: https://docs.gitlab.com/ee/development/code_review.html#approval-guidelines
-[Code Review process]: https://docs.gitlab.com/ee/development/code_review.html
 [quick actions]: https://docs.gitlab.com/ee/user/project/quick_actions.html#quick-actions-for-issues-merge-requests-and-epics
+[CHANGELOG entry]: https://docs.gitlab.com/ee/development/changelog.html
+[Code Review process]: https://docs.gitlab.com/ee/development/code_review.html
+[Approval Guidelines]: https://docs.gitlab.com/ee/development/code_review.html#approval-guidelines
+[Canonical repository]: https://gitlab.com/gitlab-org/gitlab
+[`package-and-qa` build]: https://docs.gitlab.com/ee/development/testing_guide/end_to_end/#using-the-package-and-qa-job

Gemfile

@@ -164,7 +164,6 @@ gem 'diff_match_patch', '~> 0.1.0'
 
 # Application server
 gem 'rack', '~> 2.0.9'
-gem 'rack-timeout', '~> 0.5.1'
 
 group :unicorn do
   gem 'unicorn', '~> 5.5'
@@ -174,6 +173,7 @@ end
 group :puma do
   gem 'gitlab-puma', '~> 4.3.3.gitlab.2', require: false
   gem 'gitlab-puma_worker_killer', '~> 0.1.1.gitlab.1', require: false
+  gem 'rack-timeout', require: false
 end
 
 # State machine

Gemfile.lock

@@ -817,7 +817,7 @@ GEM
       rack
     rack-test (1.1.0)
       rack (>= 1.0, < 3)
-    rack-timeout (0.5.2)
+    rack-timeout (0.5.1)
     rails (6.0.3.1)
       actioncable (= 6.0.3.1)
       actionmailbox (= 6.0.3.1)
@@ -1350,7 +1350,7 @@ DEPENDENCIES
   rack-cors (~> 1.0.6)
   rack-oauth2 (~> 1.9.3)
   rack-proxy (~> 0.6.0)
-  rack-timeout (~> 0.5.1)
+  rack-timeout
   rails (~> 6.0.3.1)
   rails-controller-testing
   rails-i18n (~> 6.0)

app/models/merge_request_diff.rb

@@ -493,6 +493,8 @@ def build_external_merge_request_diff_files(rows)
     self.stored_externally = true
 
     rows
+  ensure
+    tempfile&.unlink
   end
 
   def create_merge_request_diff_files(rows)
@@ -503,19 +505,17 @@ def create_merge_request_diff_files(rows)
   end
 
   def build_external_diff_tempfile(rows)
-    pos = 0
+    Tempfile.open(external_diff.filename) do |file|
+      rows.each do |row|
+        data = row.delete(:diff)
+        row[:external_diff_offset] = file.pos
+        row[:external_diff_size] = data.bytesize
 
-    segments = rows.map do |row|
-      segment = row.delete(:diff)
-
-      row[:external_diff_offset] = pos
-      row[:external_diff_size] = segment.bytesize
-      pos += segment.bytesize
+        file.write(data)
+      end
 
-      segment
+      file
     end
-
-    CarrierWaveStringFile.new(segments.join(''), external_diff.filename)
   end
 
   def build_merge_request_diff_files(diffs)

changelogs/unreleased/216458-remove-tempfile-from-external-diff-creation.yml

----
-title: Remove tempfile from external diff creation
-merge_request: 35750
-author:
-type: performance

changelogs/unreleased/217362-move-manage-stage-usage-activity-to-ce.yml

+---
+title: Move manage stage usage activity to CE
+merge_request: 36089
+author:
+type: changed

changelogs/unreleased/drop-old-non-unique-index-on-mr-metrics.yml

+---
+title: Remove non-unique index on `merge_request_metrics.merge_request_id` column
+merge_request: 36170
+author:
+type: changed

changelogs/unreleased/update-rack-timeout.yml

----
-title: Update `rack-timeout` to `0.5.2`
-merge_request: 36071
-author:
-type: changed

danger/roulette/Dangerfile

@@ -12,17 +12,19 @@ MARKDOWN
 
 CATEGORY_TABLE_HEADER = <<MARKDOWN
 
-To spread load more evenly across eligible reviewers, Danger has randomly picked
-a candidate for each review slot. Feel free to
+To spread load more evenly across eligible reviewers, Danger has picked a candidate for each
+review slot, based on their timezone. Feel free to
 [override these selections](https://about.gitlab.com/handbook/engineering/projects/#gitlab)
 if you think someone else would be better-suited, or the chosen person is unavailable.
 
 To read more on how to use the reviewer roulette, please take a look at the
 [Engineering workflow](https://about.gitlab.com/handbook/engineering/workflow/#basics)
 and [code review guidelines](https://docs.gitlab.com/ee/development/code_review.html).
+Please consider assigning a reviewer or maintainer who is a
+[domain expert](https://about.gitlab.com/handbook/engineering/projects/#gitlab) in the area of the merge request.
 
 Once you've decided who will review this merge request, mention them as you
-normally would! Danger does not (yet?) automatically notify them for you.
+normally would! Danger does not automatically notify them for you.
 
 | Category | Reviewer | Maintainer |
 | -------- | -------- | ---------- |
@@ -38,6 +40,7 @@ MARKDOWN
 
 OPTIONAL_REVIEW_TEMPLATE = "%{role} review is optional for %{category}".freeze
 NOT_AVAILABLE_TEMPLATE = 'No %{role} available'.freeze
+TIMEZONE_EXPERIMENT = true
 
 def mr_author
   roulette.team.find { |person| person.username == gitlab.mr_author }
@@ -48,7 +51,7 @@ def note_for_category_role(spin, role)
     return OPTIONAL_REVIEW_TEMPLATE % { role: role.capitalize, category: helper.label_for_category(spin.category) }
   end
 
-  spin.public_send(role)&.markdown_name || NOT_AVAILABLE_TEMPLATE % { role: role } # rubocop:disable GitlabSecurity/PublicSend
+  spin.public_send(role)&.markdown_name(timezone_experiment: TIMEZONE_EXPERIMENT, author: mr_author) || NOT_AVAILABLE_TEMPLATE % { role: role } # rubocop:disable GitlabSecurity/PublicSend
 end
 
 def markdown_row_for_spin(spin)
@@ -73,7 +76,9 @@ if changes.any?
   project = helper.project_name
   branch_name = gitlab.mr_json['source_branch']
 
-  roulette_spins = roulette.spin(project, categories, branch_name)
+  markdown(MESSAGE)
+
+  roulette_spins = roulette.spin(project, categories, branch_name, timezone_experiment: TIMEZONE_EXPERIMENT)
   rows = roulette_spins.map do |spin|
     # MR includes QA changes, but also other changes, and author isn't an SET
     if spin.category == :qa && categories.size > 1 && !mr_author.reviewer?(project, spin.category, [])
@@ -85,9 +90,8 @@ if changes.any?
     markdown_row_for_spin(spin)
   end
 
-  unknown = changes.fetch(:unknown, [])
-
-  markdown(MESSAGE)
   markdown(CATEGORY_TABLE_HEADER + rows.join("\n")) unless rows.empty?
+
+  unknown = changes.fetch(:unknown, [])
   markdown(UNKNOWN_FILES_MESSAGE + helper.markdown_list(unknown)) unless unknown.empty?
 end

db/migrate/20200707071941_drop_old_non_unique_index_on_mr_metrics.rb

+# frozen_string_literal: true
+
+class DropOldNonUniqueIndexOnMrMetrics < ActiveRecord::Migration[6.0]
+  include Gitlab::Database::MigrationHelpers
+
+  DOWNTIME = false
+  INDEX_NAME = 'index_merge_request_metrics'
+
+  disable_ddl_transaction!
+
+  def up
+    remove_concurrent_index_by_name(:merge_request_metrics, INDEX_NAME)
+  end
+
+  def down
+    add_concurrent_index :merge_request_metrics, :merge_request_id, name: INDEX_NAME
+  end
+end

db/structure.sql

@@ -19443,8 +19443,6 @@ CREATE INDEX index_merge_request_diffs_on_merge_request_id_and_id ON public.merg
 CREATE INDEX index_merge_request_diffs_on_merge_request_id_and_id_partial ON public.merge_request_diffs USING btree (merge_request_id, id) WHERE ((NOT stored_externally) OR (stored_externally IS NULL));
-CREATE INDEX index_merge_request_metrics ON public.merge_request_metrics USING btree (merge_request_id);
-
 CREATE INDEX index_merge_request_metrics_on_first_deployed_to_production_at ON public.merge_request_metrics USING btree (first_deployed_to_production_at);
 CREATE INDEX index_merge_request_metrics_on_latest_closed_at ON public.merge_request_metrics USING btree (latest_closed_at) WHERE (latest_closed_at IS NOT NULL);
@@ -23588,5 +23586,6 @@ COPY "schema_migrations" (version) FROM STDIN;
 20200704143633
 20200706005325
 20200706170536
+20200707071941
 \.

doc/administration/monitoring/prometheus/gitlab_metrics.md

@@ -173,6 +173,7 @@ configuration option in `gitlab.yml`. These metrics are served from the
 | `geo_repositories_retrying_verification_count` | Gauge   | 11.2  | Number of repositories verification failures that Geo is actively trying to correct on secondary  | `url` |
 | `geo_wikis_retrying_verification_count`        | Gauge   | 11.2  | Number of wikis verification failures that Geo is actively trying to correct on secondary | `url` |
 | `global_search_bulk_cron_queue_size`           | Gauge   | 12.10 | Number of database records waiting to be synchronized to Elasticsearch | |
+| `global_search_awaiting_indexing_queue_size`   | Gauge   | 13.2  | Number of database updates waiting to be synchronized to Elasticsearch while indexing is paused | |
 | `package_files_count`                          | Gauge   | 13.0  | Number of package files on primary | `url` |
 | `package_files_checksummed_count`              | Gauge   | 13.0  | Number of package files checksummed on primary | `url` |
 | `package_files_checksum_failed_count`          | Gauge   | 13.0  | Number of package files failed to calculate the checksum on primary

doc/api/import.md

@@ -55,7 +55,7 @@ POST /import/bitbucket_server
 
 ```shell
 curl --request POST \
-  --url https://gitlab.example.com/api/v4/import/bitbucket/server \
+  --url https://gitlab.example.com/api/v4/import/bitbucket_server \
   --header "content-type: application/json" \
   --header "PRIVATE-TOKEN: 9koXpg98eAheJpvBs5tK" \
   --data '{

doc/development/dangerbot.md

@@ -128,10 +128,18 @@ Here is a (non-exhaustive) list of the kinds of things Danger has been used for
 at GitLab so far:
 
 - Coding style
-- Database review workflow
-- Documentation review workflow
+- Database review
+- Documentation review
 - Merge request metrics
-- Reviewer roulette workflow
+- Reviewer roulette. Reviewers and maintainers are chosen based on:
+  - Their roles (backend, frontend, database, etc).
+  - Their availability:
+    - No "OOO"/"PTO"/"Parental Leave" in their GitLab or Slack status.
+    - No `:red_circle:`/`:palm_tree:`/`:beach:`/`:beach_umbrella:`/`:beach_with_umbrella:` emojis in GitLab or Slack status.
+  - [Experimental] Their timezone: people for which the local hour is between
+    6 AM and 2 PM are eligible to be picked. This is to ensure they have a good
+    chance to get to perform a review during their current work day. The experimentation is tracked in
+    [this issue](https://gitlab.com/gitlab-org/quality/team-tasks/-/issues/563)
 - Single codebase effort
 
 ## Limitations

doc/development/telemetry/usage_ping.md

@@ -655,6 +655,18 @@ appear to be associated to any of the services running, since they all appear to
 | `projects_prometheus_active`                              | `usage_activity_by_stage`            | `monitor`     |                  | EE      |                                                                            |
 | `projects_with_error_tracking_enabled`                    | `usage_activity_by_stage`            | `monitor`     |                  | EE      |                                                                            |
 | `projects_with_tracing_enabled`                           | `usage_activity_by_stage`            | `monitor`     |                  | EE      |                                                                            |
+| `events`                                                  | `usage_activity_by_stage`            | `manage`      |                  | CE+EE   |                                                                            |
+| `groups`                                                  | `usage_activity_by_stage`            | `manage`      |                  | CE+EE   |                                                                            |
+| `users_created_at`                                        | `usage_activity_by_stage`            | `manage`      |                  | CE+EE   |                                                                            |
+| `omniauth_providers`                                      | `usage_activity_by_stage`            | `manage`      |                  | CE+EE   |                                                                            |
+| `ldap_keys`                                               | `usage_activity_by_stage`            | `manage`      |                  | EE      |                                                                            |
+| `ldap_users`                                              | `usage_activity_by_stage`            | `manage`      |                  | EE      |                                                                            |
+| `value_stream_management_customized_group_stages`         | `usage_activity_by_stage`            | `manage`      |                  | EE      |                                                                            |
+| `projects_with_compliance_framework`                      | `usage_activity_by_stage`            | `manage`      |                  | EE      |                                                                            |
+| `ldap_servers`                                            | `usage_activity_by_stage`            | `manage`      |                  | EE      |                                                                            |
+| `ldap_group_sync_enabled`                                 | `usage_activity_by_stage`            | `manage`      |                  | EE      |                                                                            |
+| `ldap_admin_sync_enabled`                                 | `usage_activity_by_stage`            | `manage`      |                  | EE      |                                                                            |
+| `group_saml_enabled`                                      | `usage_activity_by_stage`            | `manage`      |                  | EE      |                                                                            |
 | `keys`                                                    | `usage_activity_by_stage`            | `create`      |                  |         |                                                                            |
 | `projects_jira_dvcs_server_active`                        | `usage_activity_by_stage`            | `plan`        |                  |         |                                                                            |
 | `service_desk_enabled_projects`                           | `usage_activity_by_stage`            | `plan`        |                  |         |                                                                            |

ee/changelogs/unreleased/217918-follow-up-from-implement-a-switch-to-pause-all-of-the-write-req.yml

+---
+title: Add Prometheus Metrics for Index Pause Sidekiq Queue.
+merge_request: 35871
+author:
+type: changed

ee/lib/ee/gitlab/usage_data.rb

@@ -237,17 +237,13 @@ def usage_activity_by_stage_create(time_period)
         override :usage_activity_by_stage_manage
         def usage_activity_by_stage_manage(time_period)
           super.merge({
-            events: distinct_count(::Event.where(time_period), :author_id),
-            groups: distinct_count(::GroupMember.where(time_period), :user_id),
             ldap_keys: distinct_count(::LDAPKey.where(time_period), :user_id),
             ldap_users: distinct_count(::GroupMember.of_ldap_type.where(time_period), :user_id),
-            users_created: count(::User.where(time_period), start: user_minimum_id, finish: user_maximum_id),
             value_stream_management_customized_group_stages: count(::Analytics::CycleAnalytics::GroupStage.where(custom: true)),
             projects_with_compliance_framework: count(::ComplianceManagement::ComplianceFramework::ProjectSettings),
             ldap_servers: ldap_available_servers.size,
             ldap_group_sync_enabled: ldap_config_present_for_any_provider?(:group_base),
             ldap_admin_sync_enabled: ldap_config_present_for_any_provider?(:admin_group),
-            omniauth_providers: filtered_omniauth_provider_names.reject { |name| name == 'group_saml' },
             group_saml_enabled: omniauth_provider_names.include?('group_saml')
           })
         end
@@ -356,24 +352,12 @@ def distinct_count_service_desk_enabled_projects(time_period)
         end
         # rubocop:enable CodeReuse/ActiveRecord
 
-        def ldap_available_servers
-          ::Gitlab::Auth::Ldap::Config.available_servers
-        end
-
         def ldap_config_present_for_any_provider?(configuration_item)
           ldap_available_servers.any? { |server_config| server_config[configuration_item.to_s] }
         end
 
-        def omniauth_provider_names
-          ::Gitlab.config.omniauth.providers.map(&:name)
-        end
-
-        # LDAP provider names are set by customers and could include
-        # sensitive info (server names, etc). LDAP providers normally
-        # don't appear in omniauth providers but filter to ensure
-        # no internal details leak via usage ping.
-        def filtered_omniauth_provider_names
-          omniauth_provider_names.reject { |name| name.starts_with?('ldap') }
+        def ldap_available_servers
+          ::Gitlab::Auth::Ldap::Config.available_servers
         end
       end
     end

ee/lib/elastic/metrics_update_service.rb

@@ -11,6 +11,9 @@ def execute
 
       initial_gauge = Gitlab::Metrics.gauge(:global_search_bulk_cron_initial_queue_size, 'Number of initial database updates waiting to be synchronized to Elasticsearch', {}, :max)
       initial_gauge.set({}, Elastic::ProcessInitialBookkeepingService.queue_size)
+
+      awaiting_indexing_gauge = Gitlab::Metrics.gauge(:global_search_awaiting_indexing_queue_size, 'Number of database updates waiting to be synchronized to Elasticsearch while indexing is paused.', {}, :max)
+      awaiting_indexing_gauge.set({}, Elastic::IndexingControlService.queue_size)
     end
 
     private

ee/spec/lib/ee/gitlab/usage_data_spec.rb

@@ -364,73 +364,53 @@
         it 'includes accurate usage_activity_by_stage data' do
           stub_config(
             ldap:
-              { enabled: true, servers: ldap_server_config },
-            omniauth:
-              { providers: omniauth_providers }
+              { enabled: true, servers: ldap_server_config }
           )
 
           for_defined_days_back do
             user = create(:user)
-            create(:event, author: user)
-            create(:group_member, user: user)
             create(:key, type: 'LDAPKey', user: user)
             create(:group_member, ldap: true, user: user)
             create(:cycle_analytics_group_stage)
             create(:compliance_framework_project_setting)
           end
 
-          expect(described_class.uncached_data[:usage_activity_by_stage][:manage]).to eq(
-            events: 2,
-            groups: 2,
+          expect(described_class.uncached_data[:usage_activity_by_stage][:manage]).to include(
             ldap_keys: 2,
             ldap_users: 2,
-            users_created: 8,
             value_stream_management_customized_group_stages: 2,
             projects_with_compliance_framework: 2,
             ldap_servers: 2,
             ldap_group_sync_enabled: true,
             ldap_admin_sync_enabled: true,
-            omniauth_providers: ['google_oauth2'],
             group_saml_enabled: true
           )
-          expect(described_class.uncached_data[:usage_activity_by_stage_monthly][:manage]).to eq(
-            events: 1,
-            groups: 1,
+          expect(described_class.uncached_data[:usage_activity_by_stage_monthly][:manage]).to include(
             ldap_keys: 1,
             ldap_users: 1,
-            users_created: 5,
             value_stream_management_customized_group_stages: 2,
             projects_with_compliance_framework: 2,
             ldap_servers: 2,
             ldap_group_sync_enabled: true,
             ldap_admin_sync_enabled: true,
-            omniauth_providers: ['google_oauth2'],
             group_saml_enabled: true
           )
         end
 
-        def omniauth_providers
-          [
-            OpenStruct.new(name: 'google_oauth2'),
-            OpenStruct.new(name: 'ldapmain'),
-            OpenStruct.new(name: 'group_saml')
-          ]
-        end
-
         def ldap_server_config
           {
             'main' =>
-              {
-                'provider_name' => 'ldapmain',
-                'group_base'    => 'ou=groups',
-                'admin_group'   => 'my_group'
-              },
+            {
+              'provider_name' => 'ldapmain',
+              'group_base'    => 'ou=groups',
+              'admin_group'   => 'my_group'
+            },
             'secondary' =>
-              {
-                'provider_name' => 'ldapsecondary',
-                'group_base'    => nil,
-                'admin_group'   => nil
-              }
+            {
+              'provider_name' => 'ldapsecondary',
+              'group_base'    => nil,
+              'admin_group'   => nil
+            }
           }
         end
       end