CI/CD UX Scorecard: Use Auto DevOps in an existing project without CI/CD configuration

JTBD

When I have an existing project without CI/CD configuration, I want to use Auto DevOps, so I can use modern devops workflows on my project with minimal effort

Checklist

• Document the current experience of the JTBD, as if you are the user. Capture the screens and jot down observations. Also, apply the following Emotional Grading Scale to document how a user likely feels at each step of the workflow. Add this documentation to the epic's description.
• Use the Grading Rubric to provide an overall measurement that becomes the Benchmark Score for the experience, and add it to the epic's description.
• Once you’re clear about the user’s path, create a click through video that walks through the experience and includes narration of the Emotional Grading Scale and Benchmark Score.
• Post your video to the GitLab Unfiltered YouTube channel, and link to it from the epic's description.
• If your JTBD spans more than one stage group, that’s great! Review your JTBD with a designer from that stage group for accuracy.
• Create an issue to revisit the same JTBD the following quarter to see if we have made improvements. We will use the grades to monitor progress toward improving the overall quality of our user experience.

Emotional Grading Scale

• 😃 Positive: The user’s experience included a pleasant surprise—something they were not expecting to see. The user enjoyed the experience on the screen and could complete the task, effortlessly moving forward without having to stop and reassess their workflow. Emotion(s): Happy, Motivated, Possibly Surprised
• 😐 Neutral: The user’s expectations were met. Each action provided the basic expected response from the UI, so that the user could complete the task and move forward. Emotion(s): Indifferent
• 😡 Negative: The user did not receive the results they were expecting. There may be bugs, roadblocks, or confusion about what to click on that prevents the user from completing the task. Maybe they even needed to find an alternative method to achieve their goal. Emotion(s): Angry, Frustrated, Confused, Annoyed

Current experience of the JTBD

🚀 Overall Benchmark Score: C (Average)

• Workflow needs improvement, but user can still finish completing the task. It usually takes longer to complete the task than it should. User may abandon the process or try again later.
• Frustration: Medium
• Task Completion: Successful but with unnecessary steps
• Steps to Complete Task: Average complexity

📹 Experience baseline video: Watch it on GitLab Unfiltered

• Test project: https://gitlab.com/rverissimo/jtbd-auto-devops

Getting started
I have set-up a project on gitlab.com and want to get started with Auto Devops so I can have a complete workflow with minimal configuration. For context, my project was created from a Ruby on Rails template.
😐 Neutral: On my project details page, I look for indications of Auto DevOps. I see a few options under the first commit section, one of them is to Enable Auto DevOps, as well as Set up CI/CD. I am unsure which one of the options I have to click first, or if they need to follow a specific order so that my set-up succeeds.
😐 Neutral: I hover on the buttons expecting to see an informative tooltip, but nothing happens. I decide to click Enable Auto DevOps first because the button is positioned before Set up CI/CD.
😃 Positive: I click the Enable Auto DevOps button and am taken to a new page: CI/CD Settings. The page loads with a scroll to the section I need to work on, which is already expanded. That's super useful, one click less.
😐 Neutral: I see a checkbox option to set Default to Auto DevOps pipeline. There's a link to the documentation, but I don't want to read it now. I want to keep following the workflow to enable Auto DevOps without having to read anything else.
😐 Neutral: I click the checkbox and an information box is displayed... it reads I need to add a Kubernetes cluster integration to my project for the deployment to work correctly. The message is displayed inside an orange banner, so this time I decide to check the docs before proceeding.
😐❓ Neutral: Under the orange banner I am given a few options for my deployment strategy: Continuous deployment to production, Continuous deployment to production using timed incremental rollout, and Automatic deployment to staging, manual deployment to production. I don't quite understand the difference between them, and the (?) icons positioned next to each label link to a new page. I'd expect that hovering those icons I could read some sort of excerpt that clarifies each strategy. This is not the case.
I click on Learn more about Auto DevOps and am taken to a documentation page.

Following the documentation for set-up
😐❓ Neutral: The documentation states that the Auto DevOps pipeline is enabled by default for all projects. I know that I don't have a .gitlab-ci.yaml, so I really think need to configure this firs.
😐❓ Neutral: I see a link to a quick start guide for how to use Auto DevOps with GitLab.com and a Kubernetes cluster. I click on it to read the documentation. I would expect to be given this option sooner, instead of having to read that other documentation page, since I am configuring this for the first time.
😃 Positive: This documentation is a set-by-step guide to use Auto DevOps. It says I won't need to create a Kubernetes cluster. Yay! This makes me feel that this process shouldn't take long.
😡 Negative: This is weird... the next section in this documentation, Configuring your Google account, tells me that before creating and connecting your Kubernetes cluster to your GitLab project, you need a Google Cloud Platform account. But I just read that I won't need to configure anything on Kubernetes? Why is that? Shouldn't this option be skipped? I don't think this is part of my workflow, so I will skip it for now.
😐 Neutral: The next step is to Create a new project from a template. I already did that, so I will skip this section as well.

Adding a Kubernetes cluster integration
😐 Neutral: Next is Creating a Kubernetes cluster from within GitLab.
😐❗️Neutral: Oh! Now I see that I won't need to create a cluster from Google Cloud Platform, but I need to create one within GitLab. That's more clear. Perhaps that section above should be updated to stress that. Moving on, I follow the set-by-step guide:
😡 Negative: The first thing I notice is that the interface UI of the screenshot in this documentation page is not the same as the one I see in the product. Not sure if this means the documentation is outdated, but it makes me a bit unsure if this guide will help me configure things successfully.
😡 Negative: The documentation says I need to first click Add Kubernetes cluster. That's really interesting. Why wasn't this button positioned before Enable Auto DevOps then? I go back to my project and click Add Kubernetes cluster.
😡 Negative: I am taken to a new page: Add a Kubernetes cluster integration. The documentation says to *Choose Create on Google Kubernetes Engine * but I don't see this option in the application UI. Once again, I notice that the screenshot in the documentation and the application UI are inconsistent. This frustrates me a little bit.
😐❗️Neutral: The option is still there, but in a different interface and presented in a different copy. I click Sign in with Google, and choose my google account I want to use for this setup. I am then taken back to GitLab, but this time I need to enter the details for my Kubernetes cluster.
😐 Neutral: The copy says I need to * Make sure your account meets the requirements to create Kubernetes clusters*. I am 99% sure my account meets those requirements, but I click the link to double check. I am taken to a new page, this time for the Kubernetes documentation on Google Cloud.
😐 Neutral: I click the link to Visit the Kubernetes Engine page in the Google Cloud Platform Console.
😃 Positive: I click on Select a project and there I can see that I am already added to two GitLab projects. I'm glad I don't have to configure anything here! I close the page and move back to entering details for my cluster on GitLab.
😐 Neutral: I name my cluster jtbd-ux, leave the environment scope as is (although I am unsure what that means). I select gitlab-internal in the Google Cloud Platform project dropdown.
😐❓ Neutral: I need to select a Zone, but I am not sure which one. Zone sounds a lot like Time Zone, but I am pretty sure that's not the case because the options I am given don't look like standard time zones.
😐 Neutral: I click the link to learn more https://cloud.google.com/compute/docs/regions-zones/.
😐 Neutral: As I suspected, nothing to do with Time Zones :) I understand that the compute engine needs to ensure that the project have a consistent zone to cluster mapping, and because of that I need to select a specific Zone. I am located in Amsterdam, The Netherlands, and I see an option in the table for Location: Eemshaven, Netherlands - Zones: a, b, c - Region: europe-west4.
😡 Negative: I go back to GitLab so I can select a zone in the dropdown. Ah! Now I notice that GitLab asks me for a Zone, when in reality I need to provide a Region. This label might need to be updated.
😐❓ Neutral: I can also select between europe-west4-a, europe-west4-b, and europe-west4-c. Since I am not sure of the impact this might have, I simple choose europe-west4-a.
😐 Neutral: I won't change the rest of the configuration options. They came pre-populated in the form so I believe they should be some sort of basic set-up GitLab already configured for me. I click the Create Kubernetes cluster button.
😐 Neutral: Once I click the Create Kubernetes cluster button I am taken to a new page: Kubernetes Clusters > jtbd-ux. There is a blue banner that reads: Kubernetes cluster is being created on Google Kubernetes Engine...
😡 Negative:I don't know how long that will take, nor if the page/message will get updated automatically once my cluster is created. I would like to be informed of that. Will I receive a notification via GitLab? Perhaps an email?
😐 Neutral: Going back to the documentation, I see that I *can also see its status on your GCP dashboard. I click the link to visit the dashboard, and filter by my cluster name.
😃 Positive: My cluster is already running! I go back to GitLab.
😃 Positive: Thee blue banner is replaced with a green one, and a successful message (could't screenshot). The banner disappears, and now I can continue with the set-up. The next step is to install some applications on my cluster. I go back to the documentation.
😡 Negative: Once again, an outdated screenshot.

Installing applications on my cluster
😐 Neutral: I first need to install Helm, since this integration is mandatory to install all other apps. I click the Install button.
😃 Positive: The button label changes from Install to Installing. And an animated spinner is also shown. This is quite cool, it helps me get a sense of progress. After some time Help is installed and now I can check all other applications.
😐 Neutral: I go back to the documentation, and it says I need Ingress and Prometheus. There is a banner under Ingress that says Installing Ingress may incur additional costs. Hm... I don't know about that. I am running a cluster within GitLab, so there might be some extra costs for the company for each of those clusters. I go ahead and click again Install on both of them.
😐 Neutral: After installed, Ingress also starts the load of an Endpoint.
😐 Neutral: After loaded, I can now see and copy an IP address. I copy the IP address and continue with the set-up.
😃 Positive: Now that my cluster is ready, I can enable Auto DevOps. I follow the documentation once again:

Enabling Auto DevOps
😐 Neutral: I go back to my project, click to expand the collapsed sidebar, and look for settings > ci/cd in the menu. I'm always confused by these menu options...
😡 Negative: I am taken to the first page where I tried to set-up Auto DevOps. So I was right! There is a right order I need to follow in order to do this installation 😝 If I knew that from the beginning I could have started from a different place.
😡 Negative: Once again, discrepancies between the documentation and the application UI. The docs read: Select Enable Auto DevOps but the application UI shows: Default to Auto DevOps pipeline The docs and screenshots really need an update.
😡 Negative: The documentation tells me to add a base Domain, but I don't see this option in the application.
😐 Neutral: Lastly, it tells me to select the continuous deployment strategy. This is the only point between the docs and the application that are correct. I select the option and click the Save changes button.
😐❗️ Neutral: Two banners are displayed on the top of the page. They are pretty heavy visually, and my first reaction is to dismiss them. I am always surprised that the GitLab alerts/banner don't have a button close to dismiss them...
😐 Neutral:The green banner reads: A new Auto DevOps pipeline has been created, go to Pipelines page for details I click the link to visit the Pipelines page.

Viewing my first pipeline
😐 Neutral:In the pipelines page, I see that my first pipeline has failed.
😐 Neutral: I know that I don't have a gitlab-ci.yaml file, so I go back to the documentation to see what are the next steps. The documentation starts explaining what each job does.
😡 Negative: It states that my pipeline status should be running, but it was not. It failed in the build stage.
😡 Negative: Hmm... the documentation continues to explain how to check a deployed application, but still doesn't rout me anywhere I can see why my pipeline failed or how to proceed.
😐 Neutral: I continue following the documentation and look for the Operations > Environments option in the side menu.
😐 Neutral: I click the option and am taken to the environments page. Not surprisingly, I don't have any deployments yet.
😡 Negative: I feel I can't continue following this documentation because it really explains only the happy path of the installation. My pipeline has failed. What now? I expect some help here as well.
😐 Neutral:I go back to my Pipelines view and click retry in the failed job. The pipeline starts running again.
😐 Neutral: Hm... now I notice the tab Failed Jobs (1). There should be some explanation for why my job did not pass. Oh, there you go: Your Ruby version is 2.5.5, but your Gemfile specified 2.5.1

Fixing my pipeline
😡 Negative: Ok... where I can go to fix that? I don't want to use the command line to fix that. What a bummer! Why would this fail? I used a template provided by GitLab, so shouldn't my pipeline pass?
😐 Neutral: I go back to my project home page to look for a solution.
😐 Neutral: I scroll down and notice there is a file called .ruby-version. I open the file and see the version is set to 2.5.1. I click the button to Edit the file.
😐 Neutral: I change the ruby version from 2.5.5 to 2.5.1, and commit the changes.
😐 Neutral: I go back to the Pipelines page and see a new pipeline running.
😐 Neutral: I click on the pipeline details to see how each stage is doing.
😡 Negative: Still, it fails...
😡 Negative: The job details show the same error:
😐 Neutral: I go back to the project home page, and now I see a file called Gemfile.lock.
😐 Neutral: I click on it and look for the version specified 2.5.1. I update with 2.5.5, commit, and go back to the pipelines page.
😡 Negative: Now I have to wait for the pipeline to run again... This should take a couple of minutes.
😡 Negative: Arghh! It failed once again! I have no idea what I am doing wrong...
😡 Negative: Oh no! That was my own fault. I modified the wrong file 😅 I should have updated the Gemfile, not the Gemfile.lock. Ok, hopefully this will be the last time I have to do this... project details, gemfile, edit... replace 2.5.1 with 2.5.5, commit.
😡 Negative: Back to the pipelines page, another pipeline running... let's way a few minutes. More waiting. I honestly think this template shouldn't come with any of those problems.
😃 Positive: Weehoo! The build stage passed! Now I have to wait until my pipeline succeeds.
😡 Negative: The production step failed 🤦 now let's look into what happened there.
In order to deploy or use Review Apps, KUBE_INGRESS_BASE_DOMAIN variables must be set. From 11.8, you can set KUBE_INGRESS_BASE_DOMAIN in cluster settings or by defining a variable at group or project level. You can also manually add it in .gitlab-ci.yml
😡 Negative: Well, I don't have a .gitlab-ci.yml file. That's quite an obvious fix. I'm starting to feeling frustrated because I did not expect to manually fix so many small things within this project, that was created using a template.
😐 Neutral: Back in my project homepage, I click the plus + icon to create a new file.
😡 Negative: In the new file view, I select the .gitlab-ci.yml option as template, and a new dropdown appears: Apply a GitLab CI Yaml template. I select the Auto-DevOps option and the file get populated with a lot of data. Wow! That's really overwhelming, I honestly don't want to configure any of this in order to set-up my production environment. I have no idea where to go from here. I'm really frustrated...
😡 Negative: Perhaps I can configure this somewhere else. The error log talks about set KUBE_INGRESS_BASE_DOMAIN in cluster settings. I cancel my changes and try to find the cluster settings in the application UI.
😐 Neutral: I find a Kubernetes option under Operations > Kubernetes in the side menu.
😐 Neutral: I click on it and am taken to what seems to be the overview of my clusters.
😐 Neutral: I click on jtbd-ux, and a new page loads with the details. I remember this page, it's where I had to set up the applications running in my cluster. Now I see an option to add a Base domain. I go to my Ingress application, copy the IP number I see there, and paste it inside the Base domain option. I then save my changes.
😐 Neutral: I go back to my last pipeline that has failed, and click on retry in the Production stage.
😡 Negative: This step is taking so long... I reloaded the page but the pipeline is still running... It's taking soooo long! 22 minutes and counting! Now it's on 44 minutes.... I will stop for now and go have dinner. i have no idea when this will be done.
😡 Negative: Oh no! It failed! Why? :( I feel like I did everything right this time! Let me check the job details...
😡 Negative: I'll ask Google for help. So frustrated at this point!
😐 Neutral: The first page in the search results seems to be the one I'm looking for. It's a GitLab doc, so it should help me fix that failed job
I cannot stop thinking that the Auto DevOps quick guide never mentioned I had to go through all those steps! And if it did, it wasn't as visible as it should 😭

Getting help from the team to understand errors
😡 Negative: I decided to ask for internal help in our Slack channels, and Joao promptly jumped into a call to help me. He did a walkthrough of the installation and set-up of Auto DevOps on gitlab.com. He tested my Kubernetes cluster and could see that is was working properly. The problem is the Ruby on Rails template available at GitLab that does not work correctly with Auto Devops. This is a big problem because we explicitly indicate that the user should use our Rails template, which currently does not work seamlessly for Auto DevOps. This thread in the Configure Slack channel (private) discuss that our project templates are often less likely to work with Auto DevOps by default than newly generated projects since herokuish is not trying to keep support for our project templates. If that is a fact, we should indicate this to users, as well as remove the Ruby on Rails template from the Auto DevOps Quickstart Guide in our Documentation. Either this of fix the bug in the template.
😡 Negative: Joao recommends that I use the Node JS Express template for my project. I will need to start the set-up from the beginning with a new project.

Starting over with a new project template
😐 Neutral: I create a new project using the Node JS Express template
😐 Neutral: After spending so much time doing this task initially, I feel that I already know all steps by heart. I click the button in the project home page to enable auto devops and set the deployment strategy to Continuous deployment to production
😐 Neutral: I go to Operations > Kubernetes and click the button to Add a new kubernetes cluster
😐 Neutral: Instead of creating a new cluster, I will add my existing cluster that I used in the previous project. I know the cluster is working fine, so I'll just reuse it and skip the Sign in with Google step. I disable the cluster in the old project, copy/paste the cluster data from one project to another and add it to my new project.
😐 Neutral: Now I can install Helm, Ingress, and Prometheus
😡 Negative: I got an error in the applications while installing Ingress and Prometheus. Maybe it's because I have already installed those using this same cluster in a different project. I go ahead and uninstall those applications from my old project.
😡 Negative: I uninstall the applications and try to install them in the new project, but it still shows an error. Joao helped me once again and said that this is a known bug. There is a workaround, but it requires some extra work. Since I don't want to go into this path, I decided to create a brand new cluster instead.
😡 Negative: It would be nice to know somewhere in the gitlab interface that I can/cannot use the same cluster in different projects, and what would that mean, which errors I can encounter... I feel frustrated again because I read in the documentation that I can use multiple clusters, or use the same cluster in different projects, but that didn't work out.
😐 Neutral: I remove the Kubernetes Cluster integration and start over. This time I sign in with google, and go over all the same steps to configure my cluster...
😐 Neutral: I delete my first cluster, because there was no space available for a new one...
😡 Negative: My cluster was not created and when I look back at the gitlab interface, I don't see any updates. It would have been nice to know what happened here... instead I keep seeing the same message that Kubernetes cluster is being created on Google Kubernetes Engine...
😐 Neutral: I create a new cluster again, and install Helm, Ingress, and Prometheus.
😐 Neutral: Now that Auto DevOps is enabled, and Kubernetes in configured, I can go check my pipeline.
😐 Neutral: I navigate the menu, CI/CD > Pipelines and see that my pipeline has passed! But it only has 2 stages 🤔 I click to run my pipeline again to see what happens
😐 Neutral: Oh, a new page is prompted. I really did not expect to see this. I thought the pipeline would start running automagically. I'm not sure what I should enter in the variables fields, so I'm going to leave it empty and click on Run pipeline.
😃 Positive: Aha! Now I can see all stages: Build, Test, Production, and Performance. This feels good! I'm almost done and soon I'll be able to see my project in a production environment :D
Alright! The pipeline for Test and Production passed. Now it's time to check my environments.
😐 Neutral: I navigate to Operation > Environments and see my newly deployed environment there, as well as pods, information on when it was last updated, and some action buttons. What I want to do now is see my project in a live environment. I click the button to open it.
😃 Positive: Hello, world! I'm so happy 😁 Finally I can see it live! After learning the ropes the whole process feels very seamless, but the errors and bugs in the project templates were really upsetting. I can also check the metrics for my environment. My job here is done!

Recap

• Final project: https://gitlab.com/rverissimo/rayana-jtbd-cicd

• Final Benchmark Score: C (Average) - Workflow needs improvement, but user can still finish completing the task. It usually takes longer to complete the task than it should. User may abandon the process or try again later.

  • Frustration: Medium
  • Task Completion: Successful but with unnecessary steps
  • Steps to Complete Task: Average complexity

• Creating a project from a template: 😐 Neutral / 😡 Negative

• Getting started with CI/CD and Auto DevOps: 😐 Neutral

• Following the documentation: 😡 Negative

• Adding a kubernetes cluster integration: 😡 Negative

• Installing applications on my cluster: 😐 Neutral

• Viewing my pipeline: 😃 Positive

• Viewing my live environment: 😃 Positive

• Troubleshooting: 😡 Negative