Skip to content

Remove docker-machine feature flag

Steve Xuereb requested to merge remove-dm-ff into main

What does this MR do?

Remove FF_SKIP_DOCKER_MACHINE_PROVISION_ON_CREATION_FAILURE feature flag

Why was this MR needed?

Remove FF_SKIP_DOCKER_MACHINE_PROVISION_ON_CREATION_FAILURE since it's been enabled by default for a month and no issues were reported. The feature flag was added as a fail safe in case there are problems in GitLab.com. Since we saw positive impact we should never try and provision a machine.

What's the best way to test this MR?

There is no easy way to force a creation failure mid creation, and it's not always reproducible because it needs to be at the moment where the local state exists and remote state doesn't. Below is an attempt at this.

  1. Start docker+machine runner configured like below to provision VMs on GCP.

    config.toml 
    [[runners]]
  name = "docker-machine"
  url = "https://gitlab.com/"
  token = ""
  executor = "docker+machine"
  [runners.docker]
    tls_verify = false
    image = "alpine:3.14"
    privileged = false
    disable_entrypoint_overwrite = false
    oom_kill_disable = false
    disable_cache = false
    volumes = ["/cache"]
    shm_size = 0
  [runners.machine]
    IdleCount = 0
    MachineDriver = "google"
    MachineName = "steveazz-%s"
    MachineOptions = [
      "google-project=group-verify-df9383",
      "google-zone=europe-west4-b",
  ]

  2. Open up the GCP console with the configured project opened.

  3. Start a job for this executor and as soon as you see Waiting for Instance in the runner logs, delete the machine in GCP

    Screenshot_2021-08-17_at_09.14.03

  4. Look at the runner system logs, and check that it failed to create a machine and created a new one

    runner logs 
    {"driver":"google","level":"info","msg":"Creating machine...","name":"runner-rducy8nc-steveazz-1629185747-3a0f80bb","operation":"create","time":"2021-08-17T09:35:48+02:00"}
{"driver":"google","level":"info","msg":"(runner-rducy8nc-steveazz-1629185747-3a0f80bb) Generating SSH Key","name":"runner-rducy8nc-steveazz-1629185747-3a0f80bb","operation":"create","time":"2021-08-17T09:35:48+02:00"}
{"driver":"google","level":"info","msg":"(runner-rducy8nc-steveazz-1629185747-3a0f80bb) Creating host...","name":"runner-rducy8nc-steveazz-1629185747-3a0f80bb","operation":"create","time":"2021-08-17T09:35:49+02:00"}
{"driver":"google","level":"info","msg":"(runner-rducy8nc-steveazz-1629185747-3a0f80bb) Opening firewall ports","name":"runner-rducy8nc-steveazz-1629185747-3a0f80bb","operation":"create","time":"2021-08-17T09:35:49+02:00"}
{"driver":"google","level":"info","msg":"(runner-rducy8nc-steveazz-1629185747-3a0f80bb) Creating instance","name":"runner-rducy8nc-steveazz-1629185747-3a0f80bb","operation":"create","time":"2021-08-17T09:35:49+02:00"}
{"driver":"google","level":"info","msg":"(runner-rducy8nc-steveazz-1629185747-3a0f80bb) Waiting for Instance","name":"runner-rducy8nc-steveazz-1629185747-3a0f80bb","operation":"create","time":"2021-08-17T09:35:50+02:00"}
{"driver":"google","level":"info","msg":"(runner-rducy8nc-steveazz-1629185747-3a0f80bb) Uploading SSH Key","name":"runner-rducy8nc-steveazz-1629185747-3a0f80bb","operation":"create","time":"2021-08-17T09:37:51+02:00"}
{"driver":"google","level":"error","msg":"Error creating machine: Error in driver during machine creation: operation error: {RESOURCE_NOT_FOUND  The resource 'projects/group-verify-df9383/zones/europe-west4-b/instances/runner-rducy8nc-steveazz-1629185747-3a0f80bb' was not found [] []}","name":"runner-rducy8nc-steveazz-1629185747-3a0f80bb","operation":"create","time":"2021-08-17T09:38:20+02:00"}
{"error":"exit status 1","fields.time":152745532205,"level":"error","msg":"Machine creation failed","name":"runner-rducy8nc-steveazz-1629185747-3a0f80bb","time":"2021-08-17T09:38:20+02:00"}
{"level":"warning","lifetime":152745696539,"msg":"Requesting machine removal","name":"runner-rducy8nc-steveazz-1629185747-3a0f80bb","now":"2021-08-17T09:38:20.350354+02:00","reason":"Failed to create","time":"2021-08-17T09:38:20+02:00","used":152745696864,"usedCount":0}
{"level":"warning","lifetime":152841008533,"msg":"Stopping machine","name":"runner-rducy8nc-steveazz-1629185747-3a0f80bb","reason":"Failed to create","time":"2021-08-17T09:38:20+02:00","used":95252257,"usedCount":0}
{"level":"info","msg":"Stopping \"runner-rducy8nc-steveazz-1629185747-3a0f80bb\"...","name":"runner-rducy8nc-steveazz-1629185747-3a0f80bb","operation":"stop","time":"2021-08-17T09:38:20+02:00"}
{"driver":"google","level":"info","msg":"Running pre-create checks...","name":"runner-rducy8nc-steveazz-1629185901-db2c619c","operation":"create","time":"2021-08-17T09:38:21+02:00"}
{"driver":"google","level":"info","msg":"(runner-rducy8nc-steveazz-1629185901-db2c619c) Check that the project exists","name":"runner-rducy8nc-steveazz-1629185901-db2c619c","operation":"create","time":"2021-08-17T09:38:21+02:00"}
{"level":"error","msg":"googleapi: Error 404: The resource 'projects/group-verify-df9383/zones/europe-west4-b/instances/runner-rducy8nc-steveazz-1629185747-3a0f80bb' was not found, notFound","name":"runner-rducy8nc-steveazz-1629185747-3a0f80bb","operation":"stop","time":"2021-08-17T09:38:22+02:00"}
{"error":"exit status 1","level":"warning","lifetime":154651705012,"msg":"Error while stopping machine","name":"runner-rducy8nc-steveazz-1629185747-3a0f80bb","reason":"Failed to create","time":"2021-08-17T09:38:22+02:00","used":1905948841,"usedCount":0}
{"level":"warning","lifetime":154651784143,"msg":"Removing machine","name":"runner-rducy8nc-steveazz-1629185747-3a0f80bb","reason":"Failed to create","time":"2021-08-17T09:38:22+02:00","used":1906027514,"usedCount":0}
{"level":"info","msg":"About to remove runner-rducy8nc-steveazz-1629185747-3a0f80bb","name":"runner-rducy8nc-steveazz-1629185747-3a0f80bb","operation":"remove","time":"2021-08-17T09:38:22+02:00"}
{"level":"info","msg":"WARNING: This action will delete both local reference and remote instance.","name":"runner-rducy8nc-steveazz-1629185747-3a0f80bb","operation":"remove","time":"2021-08-17T09:38:22+02:00"}
{"level":"info","msg":"(runner-rducy8nc-steveazz-1629185747-3a0f80bb) Deleting instance.","name":"runner-rducy8nc-steveazz-1629185747-3a0f80bb","operation":"remove","time":"2021-08-17T09:38:22+02:00"}
{"driver":"google","level":"info","msg":"(runner-rducy8nc-steveazz-1629185901-db2c619c) Check if the instance already exists","name":"runner-rducy8nc-steveazz-1629185901-db2c619c","operation":"create","time":"2021-08-17T09:38:22+02:00"}
{"driver":"google","level":"info","msg":"Creating machine...","name":"runner-rducy8nc-steveazz-1629185901-db2c619c","operation":"create","time":"2021-08-17T09:38:22+02:00"}
{"driver":"google","level":"info","msg":"(runner-rducy8nc-steveazz-1629185901-db2c619c) Generating SSH Key","name":"runner-rducy8nc-steveazz-1629185901-db2c619c","operation":"create","time":"2021-08-17T09:38:22+02:00"}
{"driver":"google","level":"info","msg":"(runner-rducy8nc-steveazz-1629185901-db2c619c) Creating host...","name":"runner-rducy8nc-steveazz-1629185901-db2c619c","operation":"create","time":"2021-08-17T09:38:22+02:00"}
{"driver":"google","level":"info","msg":"(runner-rducy8nc-steveazz-1629185901-db2c619c) Opening firewall ports","name":"runner-rducy8nc-steveazz-1629185901-db2c619c","operation":"create","time":"2021-08-17T09:38:22+02:00"}
{"driver":"google","level":"info","msg":"(runner-rducy8nc-steveazz-1629185901-db2c619c) Creating instance","name":"runner-rducy8nc-steveazz-1629185901-db2c619c","operation":"create","time":"2021-08-17T09:38:23+02:00"}
{"level":"info","msg":"(runner-rducy8nc-steveazz-1629185747-3a0f80bb) Remote instance does not exist, proceeding with removing local reference","name":"runner-rducy8nc-steveazz-1629185747-3a0f80bb","operation":"remove","time":"2021-08-17T09:38:23+02:00"}
{"level":"info","msg":"Successfully removed runner-rducy8nc-steveazz-1629185747-3a0f80bb","name":"runner-rducy8nc-steveazz-1629185747-3a0f80bb","operation":"remove","time":"2021-08-17T09:38:23+02:00"}
{"level":"info","lifetime":156045999827,"msg":"Machine removed","name":"runner-rducy8nc-steveazz-1629185747-3a0f80bb","now":"2021-08-17T09:38:23.650656+02:00","reason":"Failed to create","retries":0,"time":"2021-08-17T09:38:23+02:00","used":3300243540,"usedCount":0}
{"driver":"google","level":"info","msg":"(runner-rducy8nc-steveazz-1629185901-db2c619c) Waiting for Instance","name":"runner-rducy8nc-steveazz-1629185901-db2c619c","operation":"create","time":"2021-08-17T09:38:24+02:00"}

What are the relevant issue numbers?

closes #27749 (closed)

Edited by Steve Xuereb

Merge request reports