Enable 301 Redirects on about.gitlab.com for Marketing (infrastructure project)

moving the discussion from: gitlab-com/www-gitlab-com#3952 (closed) to infrastructure project

• create a yaml file with redirects definitions: www-gitlab-com/data/redirects.yml , start with the following content and use this format:

- sources: /some-old-path/ 
  target: /some-new-path/
- sources:
    - /another-old-path/
    - /another-old-path-as-well/
  target: /some-other-new-path/

• add a CI job in the build stage that validates the yml file (@alejandro):
  • no source path appears twice in the collection
  • no target path appears twice in the collection
  • no target path appears as a source path
  • no redirect loops
• create manually within Fastly (@mwasilewski-gitlab):
  • An edge dictionary named redirects
  • link dictionary with a version of config
  • create redirects with a condition which uses a table, follow docs (the redirects are actually so simple that there shouldn't be a need for VCL snippets)
  • update a 301 redirect using an API call
• write a Ruby script www-gitlab-com/bin/redirects that will:

  • read yml file, parse it into 3 parts: exact matches, simple regexes (literal matches), regexes

  • exact matches (@alejandro):

    • get items from fastly edge dictionary curl -H "$FASTLY_API_TOKEN_STAGING" https://api.fastly.com/service/$FASTLY_SERVICE_ID_STAGING/dictionary/$FASTLY_DICTIONARY_ID_STAGING/items
    • compare with exact matches from yml:
      • if items exist in fastly, but not in yml, delete them from fastly using batch update, e.g. json:

        {
            "items": [
                {
                "op": "delete",
                "item_key": "/src/path1",
                "item_value": ""
                },
                {
                "op": "delete",
                "item_key": "/src/path2",
                "item_value": ""
                }
            ]
        }

  • for all else use batch upsert api call, e.g. json:

        {
            "items": [
                {
                "op": "upsert",
                "item_key": "/src/path1",
                "item_value": "/dst/path1"
                },
                {
                "op": "upsert",
                "item_key": "/src/path2",
                "item_value": "/dst/path2"
                }
            ]
        }

  • literal matches and regexes:

    • generate and upload one recv dynamic VCL snippet and one error dynamic VCL snippet. They need to include all rules inside of them. e.g. recv:

curl -X PUT -s https://api.fastly.com/service/$FASTLY_SERVICE_ID_STAGING/snippet/$FASTLY_VCL_SNIPPET_STAGING -H "$FASTLY_API_TOKEN_STAGING" -H 'Content-Type: application/x-www-form-urlencoded' --data $'content=if ( req.url ~ "^/gitlab-ee" ) {\n error 805 "Permanent Redirect";\n}\nif ( req.url ~ "^/development" ) {\n error 806 "Permanent Redirect";\n}\n';

e.g. error:

curl -X POST -s https://api.fastly.com/service/$FASTLY_SERVICE_ID_STAGING/version/8/snippet -H "$FASTLY_API_TOKEN_STAGING" -H 'Content-Type: application/x-www-form-urlencoded' --data $'content=if ( obj.status == 805 ) {\n set obj.status = 301;\n set obj.response = "Moved Permanently";\n set obj.http.Location = "/pricing/";\n synthetic {""};\n return (deliver);\n}\nif ( obj.status == 806 ) {\n set obj.status = 301;\n set obj.response = "Moved Permanently";\n set obj.http.Location = "/sales";\n synthetic {""};\n return (deliver);\n}\n';

• add a protected env var with an API key to Fastly API tied only to the about.gitlab.com Fastly Service. An env var with a key was already present in www-gitlab-com ci/cd config. I turned it into a protected one and added a key for staging (@mwasilewski-gitlab )
• add a CI job in the build stage that validates the script
• add a CI job in the deploy stage that triggers the script
• after all this work is done, confirm redirects in Fastly are fully operational
• remove redirects from Chef and chef-repo

batch update curl: curl -X PATCH -H 'Content-Type: application/json' -H "$FASTLY_API_TOKEN_STAGING" -d @batch.json "https://api.fastly.com/service/$FASTLY_SERVICE_ID_STAGING/dictionary/$FASTLY_DICTIONARY_ID_STAGING/items"

batch updating and upserts