META: Make Auto DevOps work first go for Elixir Phoenix Applications

In my quest to dogfood Auto DevOps for https://gitlab.com/DylanGriffith/gitlab-lightning I encountered many many issues. It took a very long time to get set up and it's still got some major problems so I wanted to at least document everything here so it may serve also as a guide for anyone else trying to accomplish the same thing and also a TODO list of things that should be improved in Auto DevOps (not all of this is specific to just Phoenix).

TL;DR look at the related issues as they were all impediments for me

Config to get it working

CI/CD Variables:

BUILDPACK_URL=https://github.com/HashNuke/heroku-buildpack-elixir
DB_INITIALIZE=sleep 30 && /bin/herokuish procfile exec mix ecto.create
DB_MIGRATE=/bin/herokuish procfile exec mix ecto.migrate
POSTGRES_VERSION=11.2

Add a elixir_buildpack.config to your repo containing:

hook_post_compile="mix phx.digest"

Add a Procfile to your repo containing:

web: mix phx.server

The following diff applied for .gitlab-ci.yml from https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/25500

Caveates/Workarounds

  • We are using a randomly generated secret_key_base in config/prod.exs since there is no way to pass environment variables into the compile stage of Auto DevOps (https://gitlab.com/gitlab-org/gitlab-ce/issues/54681) . Using a random secret here probably means that probably you will not be able to rely on using the session cookie at all since it will be different on every pod running the app.
  • The host has been hardcoded in config/prod.exs which means it will be the same for all deployed environments (staging, review, production). Again, since environment variables can't be passed in during compile time it doesn't seem easy to make this dynamic or dependent on the environment.
  • Selenium tests won't pass so they need to be disabled by default with mix test (needs https://gitlab.com/gitlab-org/gitlab-ce/issues/53073 )

Rant of events as I tried to install this

  • First couple of deployments failed because the namespace was missing. I tried re-adding the credentials multiple times and it was not creating the namespace in Kubernetes. Finally I tried testing locally and it worked so I tried again on GitLab.com and it worked. The namespace was created. I'm not sure if I was copy/pasting the credentials wrong somehow the first few times (NOTE I was using add existing cluster because I wanted to share it with another project cluster and group level clusters do not support the apps I need and also they don't work on personal namespaces). See build failure https://gitlab.com/DylanGriffith/gitlab-lightning/-/jobs/165631697 . At any rate it would have been really helpful to have https://gitlab.com/gitlab-org/gitlab-ce/issues/55447 or https://gitlab.com/gitlab-org/gitlab-ce/issues/54506

  • Deployment failed the first time saying there was a problem with the domain dylangriffith-gitlab-lightning.. This seemed to indicate that it didn't have the base domain and so I assumed $KUBE_INGRESS_BASE_DOMAIN was not passed to CI. Not really sure how to account for this since the problem seemed to not happen again. I tried testing with a minmal .gitlab-ci.yml file but it seemed all the variables were being passed as expected and even $CI_ENVIRONMENT_URL had the full correct value. See build https://gitlab.com/DylanGriffith/gitlab-lightning/-/jobs/165640550 . This problem seemed to end up resolving itself and I don't know how.

  • After the first deploy failed I ran into a known issue installing after a failed install https://gitlab.com/gitlab-org/gitlab-ce/issues/54760 . I tried using Helm/Tiller locally to find and clean up resources but this did not work for some reason. I then tried working around this by deleting the namespace from Kubernetes and removing then re-adding the project to the cluster to clear everything out but then it still failed https://gitlab.com/DylanGriffith/gitlab-lightning/-/jobs/166096717 . I then noticed I was running tiller client locally so I think this messed up the state so I stopped running it, deleted the namespace and did that all again and I got further through deploy. Now got back to failing because the URL is not right https://gitlab.com/DylanGriffith/gitlab-lightning/-/jobs/166122020 which was caused by me forgetting to set the base domain so had to go through the whole flow again (delete namespace, remove and re-add cluster which actually was something I had to do about 20 times while getting this to work)

  • Ran into an issue deploying DB initialization https://gitlab.com/DylanGriffith/gitlab-lightning/-/jobs/166136663 : Error: Job failed: BackoffLimitExceeded. Debugging I found the problem was my migration command failing with /bin/sh: 1: mix: not found. From this and testing locally with the built docker image I found I could run instead /bin/herokuish procfile exec mix ecto.create to get the correct environment (I was previously trying cd app; mix ecto.create).

  • Ran into issue with above create command ** (Mix) The database for GitlabLightning.Repo couldn't be created: exited in: :gen_server.call(#PID<0.212.0>, {:checkout, #Reference<0.3531457777.1883766785.236814>, true, 15000}, 5000). Troubleshooting by creating a pod to SSH into and try running the command. Strangely it seems to work when I just apply a pod with this same command. I actually get "The database for GitlabLightning.Repo has already been created". Finally I got past this after realising that the pods for initializing the DB are starting up quicker than the DB pod so I added a sleep before ecto.create by setting DB_INITALIZE=sleep 30 && /bin/herokuish procfile exec mix ecto.create. Also increasing the backoffLimit of retries for the initialize job will help with this.

  • After that I ran into an issue with mix phx.digest not being run so the app wouldn't start Could not find static manifest at "/app/_build/prod/lib/gitlab_lightning/priv/static/cache_manifest.json". Run "mix phx.digest" after building your static files or remove the configuration from "config/prod.exs"

  • After adding mix phx.digest as hook_post_compile I got the error The input path "priv/static" does not exist in my build output (https://gitlab.com/DylanGriffith/gitlab-lightning/-/jobs/166216537 ) even though the build did not fail. So I added this directory to the repo with .gitkeep

  • Next error was the container receiving SIGTERM constantly and restarting. Needed to add a Procfile with web: mix phx.server

  • Finally it started deploying but now I am getting Internal server error when performing a search because it is not using the right version of postgres to support the features I'm using so I created https://gitlab.com/gitlab-org/gitlab-ce/issues/58098 as there is no way to configure the postgres version today.

Edited by 🤖 GitLab Bot 🤖