As a lead engineer / platform engineer, I want a single place to define the lifecycle and related processes of a service, so GitLab can support me in Day 0, Day 1 and Day 2 operations.
A design example of a filled service list, with no metrics and no deep dive (into an individual service page yet). Search seemed important, but I am also okay cutting it for MVC to get something out to our users.
A design example of an empty service list (no services in this environment).
A journey map of registering a service (potentially not including UI yet, but just an idea of how users would go about this).
@emilybauman You asked how should we start the designs. I see two areas to start working on:
How to register a service - this might be code based at first. Still having a design perspective there would be very valuable.
I recommend starting with the list view that we used for the interviews. We should drop the metrics part, and we can start with a VERY simple list and an environment selector.
@nagyv-gitlab - I think thats great, I was actually leaning on the list view as MVC as well, so glad to be on the same page there. I can work on fixing up a list view based on the outcomes of our interviews and do a bit of exploration into an initial set-up.
I've been working on breaking this down and I think what we'll need for MVC will be the following:
A design example of a filled service list, with no metrics and no deep dive (into an individual service page yet). Search seemed important, but I am also okay cutting it for MVC to get something out to our users.
A design example of an empty service list (no services in this environment).
A journey map of registering a service (potentially not including UI yet, but just an idea of how users would go about this).
We might be able to start with a very simple, microservice oriented
convention: if you don't register any services, we assume you have a single
service in the project, and we can call it by the name of the project.
Most tools do the registration using a combination of the following steps:
Install some agent in the infrastructure (e.g. datadog)
Register a service on the UI. It returns configuration details.
Apply those configuration details to your agent
The agent reaches out to the SaaS provider and populates the service
It would be great to simplify this somehow that customers still understand.
I don't have any ideas yet.
if you don't register any services, we assume you have a single service in the project, and we can call it by the name of the project.
@nagyv-gitlab - I like this to start, we've also seen a few customers we've interviewed who have been doing something along these lines already.
It would be great to simplify this somehow that customers still understand. I don't have any ideas yet.
I think what I can do is fix up a use journey map in FigJam (which has recently replaced mural) and focus some of 16.5 on creating a first iteration of what steps users need to take to register the service. This will give us a place to brainstorm and simplify.
@nagyv-gitlab@shinya.maeda - Thoughts on these steps for a user registering a service (based on Viktor and I's chat this morning around using a backstage approach):
User wants to register a service.
(optional) User reads documentation on how to get started with services.
Overview of service .yml file.
Describe options for entering metadata.
Potential provide a template?
User writes the .yml
What actually gets written here is tbd.
User validates the .yml somehow.
User commits file.
User checks to Service UI to ensure everything is showing up as planned.
We can discuss this more tomorrow during the design pair, but wanted to get it down on paper.
Also just to get this down somewhere - something interesting I was thinking about here for a future iteration that could be cool, would be to auto generate .yml based on what is in the project.
@emilybauman I think having YAML files as APIs makes sense. We can even start implementing without database (PostgreSQL) as we can just read the file from the repository. Starting from stateless entries would simplify the initial iteration.
@emilybauman LGTM. We can likely skip validation first in step 4, and show
an error on the Service UI, or we can make validation work with the GLab
CLI. Whichever is simpler to implement.
What should be part of the YAML? (The conceptual model is needed to answer
this question :))
I would start with:
name
description
kubernetes:
Namespace
Flux resource
The kubernetes properties would set up the cluster UI.
Would it be possible to edit these properties through the UI or it would be
read-only? I think it should be read-only and we should open the service
definition YAML for editing.
We can likely skip validation first in step 4, and show an error on the Service UI, or we can make validation work with the GLab CLI. Whichever is simpler to implement.
@nagyv-gitlab - Agreed. My main purpose of calling this out was the fact the user would probably validate in some way, but how they do that is still up to us (it could me manual on their end).
Would it be possible to edit these properties through the UI or it would be read-only? I think it should be read-only and we should open the service definition YAML for editing.
I was thinking first iteration the service UI would be read-only and the docs can specify that editing must be done in the yaml.
I think the "data" is actually stored after the commit. GitLab will need to
process the yaml and populate the related DB. Followingly, the user can
view the Services UI. I'd move the yellow box under "commits .yml file"
I think the "data" is actually stored after the commit. GitLab will need to process the yaml and populate the related DB. Followingly, the user can view the Services UI. I'd move the yellow box under "commits .yml file"
@nagyv-gitlab - Got it, that makes sense. At first I read this as in data that goes into the input, but I think taking a step back data that goes into the DB makes more sense. I'll adjust.
@emilybauman are you planning on designing the user flow for viewing and managing services after they've been registered? That would be useful to understand in order to provide good feedback on the designs.
are you planning on designing the user flow for viewing and managing services after they've been registered? That would be useful to understand in order to provide good feedback on the designs.
@pedroms - Good callout here! I can design one up this week for ease of understanding the user journey. For some quick context, viewing would happen in the Service Page UI (list and details) and editing services can happen in the .yml or the editor tab we are planning on the service details page. This page still needs a draft design.
@emilybauman thanks for sharing this issue with me on our last 1:1. For transparency, here are my thoughts that I shared with you:
The last two proposed artifacts (journey map and documentation on the services concept) seem to be the most valuable at this moment. These will help facilitate discussions within the team and ensure that we are not too tied to the UI.
For the journey map, if you don't have findings about user emotions, pain points, or opportunities, I suggest you do a simple user flow instead — that's still a valuable visualization.
@pedroms - We've started on the conceptual modal which can be found here. I'll spent a bit more time this week overhauling it, but let's use some time at our sync next week to review!
@nagyv-gitlab Thanks for the recording! That's very helpful. I have a couple of questions:
In the Service details page, we see that release artifacts are shown under the specific service. If I understand correctly, the page is service and environment specific, for example, release artifact A on FancyAPI1-Production. Or did you mean that the environment can be omitted in the page, so that users can see release artifacts across multiple environments under a specific service?
In GitLab infra case, 5% of traffic always goes to canary stage. If we project this to the "Active" section in the Service details page, release artifact v2.0.1 shows rollout: 5% whereas release artifact v2.0.0 shows rollout: 95%, however, users wouldn't have idea which is canary and which is main. Should we also show a metadata or label alongside the active deployments something like rollout: 5%, stage: canary and rollout: 95%, stage: main?
In the last half part, you navigated to the release artifact v2.0.2 page. How would you call this page? At first, I though "Deployment details page" but it's cross environments so it would be more like "Release artifact promotion" page.
In the Service details page, we see that release artifacts are shown under the specific service. If I understand correctly, the page is service and environment specific, for example, release artifact A on FancyAPI1-Production. Or did you mean that the environment can be omitted in the page, so that users can see release artifacts across multiple environments under a specific service?
You understand it correctly. The page is Service+Environment. There should be somewhere an environment selector on the page.
In GitLab infra case, 5% of traffic always goes to canary stage. If we project this to the "Active" section in the Service details page, release artifact v2.0.1 shows rollout: 5% whereas release artifact v2.0.0 shows rollout: 95%, however, users wouldn't have idea which is canary and which is main. Should we also show a metadata or label alongside the active deployments something like rollout: 5%, stage: canary and rollout: 95%, stage: main?
I don't think this is needed. The version and deployment date (either of the two) should provide enough information.
In the last half part, you navigated to the release artifact v2.0.2 page. How would you call this page? At first, I though "Deployment details page" but it's cross environments so it would be more like "Release artifact promotion" page.
Great question! My first pick are "Release status" or "Release management" page.
Thanks for the great overview here @nagyv-gitlab. The recordings are always much appriciated.
Overall these look great, I can take them away and design out something using GitLab components that we can add to the page and better discuss in our design pairing sessions.
Great question! My first pick are "Release status" or "Release management" page.
Could the page simple be Release Details or Release Actions - which goes in line with a lot of our other details pages? We could loop in @phillipwells here for ideas when he is back from PTO since exact naming can always change later on.
There are actually similarities I'm seeing in the "facebook wall" section of the mock that relate back to our deployment details page (Approval actions, comments etc) we could probably take inspiration from as these tested well.
@shinya.maeda@nagyv-gitlab - Based on the above conversations, I wanted to break down exactly what pages this concept would include:
List View (Group)
List View (Project)
Details View
Within the Details View we have the following:
Deployments Tab
Release Artifacts Tab (which when clicked in produces a breadcrumb and a new page - Release Details?)
Would this be good to start? I know we played around with the idea of a feature flags page, settings and events but this could be good for the first iteration.
Cluster UI (might not need a design as we can use the current one, just the tab)
View the service definition YAML + edit button (tab title: Configuration?)
I think the Deployments and Release artifacts tabs are one. A deployment is created out of a release artifact. We can call it Deployments. When clicked in produces a breadcrumb and a new page. If we call it Deployments, I might call it "Deployment flow", otherwise Release details or Release process seem to be good.
A specific release artifact for this service a.k.a "Release status" or "Release management" or "Release Details" or "Release Actions" or "Deployment flow" page. This is cross-environments page and could be part of Details View.
One question from my side is where the environment-name becomes required or optional parameter. It seems we're proposing that Details views and following content don't require it, but cluster and flux resource could be environment specific. Can we elaborate the point?
@emilybauman The first REST API from Shinya point out that we'll need a logic or a design when the user opens the service listing page: which environment to show first?
My idea:
new user, opens the service list page
we show some explanation text and a list of environments to pick from
once an environment was picked, we show the service list
To improve for returning visitors
We can save the picked environment in local storage / cookie to open it automatically the next time
We can make enable "favourite" / "starred" environments (actually, just one), and open it automatically
I don't like this idea as the user is interested in a service list, and we give them an environment list first. Do you have a better idea?
We could try to be smart and open the last updated deployment_tier: production environment?
It's still far from perfect as users might get confused why we change the pre-selected environment.
How would the URL look like for the service page? gitlab.com/<group>/<project>/-/services/?environment=production or gitlab.com/<group>/<project>/-/services/-/environment/production
One question from my side is where the environment-name becomes required or optional parameter. It seems we're proposing that Details views and following content don't require it, but cluster and flux resource could be environment specific. Can we elaborate the point?
I think, even the details view requires an environment to be selected. The page should show the deployments/pipelines affecting a given service in a given environment.
Only the "Release status" view is a cross-environment view.
I don't like this idea as the user is interested in a service list, and we give them an environment list first. Do you have a better idea?
I don't think this flow is particularly bad imo, it seems fairly typical in GitLab in terms of setting up a page. Alternatively, if we can't open the last updated, we could just default to the top environment in the dropdown list with a how to popover, but when I mocked this up it was a bit less clear:
Empty State Setup
Popover Setup
How would the URL look like for the service page?
I think gitlab.com/<group>/<project>/-/services/?environment=production as the other one implies production is a new page.
How would the URL look like for the service page? gitlab.com/<group>/<project>/-/services/?environment=production or gitlab.com/<group>/<project>/-/services/-/environment/production
@nagyv-gitlab The former one. The query param format is TBD. The latter one is fine too but it might not be flexible.
One important feature we have to ensure is providing deterministic and shareable link that everyone sees exactly the same information. For example, Kibana's share direct link feature generates the following query params that contains all of the filtering and grouping conditions:
If we think this in REST API's convention, it would look like this:
I think for the design of this we need to be careful and distinguish REST API paths from the Ruby controller paths that are used in the web browsers. While they are similar, they may differ
We could try to be smart and open the last updated deployment_tier: production environment?
@nagyv-gitlab I wouldn't like that as a user - I like when things are deterministic and I know exactly what I get when I click something.
I don't think this flow is particularly bad imo, it seems fairly typical in GitLab in terms of setting up a page.
Yes, I think it's okay to first choose the environment. Maybe the user could configure their "default" environment? Or it would show the last visited environment (although that's not deterministic, at least it's from the same page as opposed to the "last deployment" happening from "somewhere" (e.g. CI job).).
How would the URL look like for the service page? gitlab.com/<group>/<project>/-/services/?environment=production or gitlab.com/<group>/<project>/-/services/-/environment/production
I think that query parameters are they way to go here. Nothing is inherently wrong with gitlab.com/<group>/<project>/-/services/-/environment/production, but to me it suggests that environment/production is some kind of sub-resource of a service (like with group/project), but it isn't.
There was a question around "what if the query parameter is omitted?" - I think we can just open the default page, which as I proposed earlier in this comment, would be the user-configured default environment or the last visited one - or it shows a page where they should choose the environment. I think all those would be better then showing an error
An alternative for a path to "show all services of an environment" would (naturally for me) be: gitlab.com/<group>/<project>/-/environments/production/-/services. However, that wouldn't focus on the "services" which may not be desired
For transparency adding some comments here from our UX Review Meeting this morning:
Brainstorm - how are users managing Services in context with Deployments?
Define a boring solution, how does this work into our current IA? Can this be shown somewhere in the environments area? Deployments page? What is the smallest thing we can do that has the largest impact.
Map user flow
Validate hypothesis
When setting up a service, think about onboarding and how we can offer customers templates. Can this be done similar to the CI Catalog?
Create a strong docs page that accurately describes services.
Sync with @gdoyle around table information and responsiveness.
Take a look at the information that users wanted about deployments in the GitLab and “the cloud” study Report Brief - Artifact Management Integration - How are users working with Gitlab CI/CD and "the Cloud"? (link in agenda)
Another thing that came out of this review is the need for an elevator pitch for services. I found myself struggling to explain succinctly - and if we want to share this out more broadly, this will be important.
@emilybauman There were no alternative proposals. The schema of the YAML will need more discussion, but we can start with something very simple. (Just a name?)
Review environments: the schema defines a list of environments where the service is present. Does it mean that the schema needs to be edited for every review environment? No. That would be very hard to manage. Assuming that the above "file" is consumed by an API as is, we should be able to consume PATCH requests on the same API or provide a separate API to manage environments of a service. This would be similar to how Deployment and Pod works in K8s. While the user can define a Pod directly, it's often defined in a Deployment (that templates a Replicaset that templates Pods, but let's not get lost in the details). Similarly, we could treat the environments list above as template to add the service to those environments, but the same action can be taken directly through a dedicated API.
Where to store the above "file"? I recommend .gitlab/services/*.yaml where multiple services can be defined. We should support having multiple services defined in a single .yaml file or across multiple files.
Is this a desired state schema? No, it's not. This is a configuration file. At least at this stage, we know nothing on how to deploy a service to an environment what the schema would mean if treated as a desired state description. As a result, modifying elements of it might be possible outside of the file as well. (e.g. the review environments topic above)
@nagyv-gitlab@emilybauman how does this map to a gitlab-ci deployment? would the deployment jobs look like:
environment:name:prodservice:api
or is this solely for gitops-style deployments?
if the former is included, maybe Environments can be optional, as they would be controlled by gitlab CI?
I think ultimately we should stick to our own thing, with the idea of how our keys map to kubernetes. Mostly in the event that as more tools and abstractions are built on k8s to smooth out the developer experience, they might change names or obscure values in ways that make it hard to keep the mapping obvious. In that event, we'd have to provide definitions for all the parameters anyway, and that would be the point to offer mapping ideas over various services.
Use case 1: Show me the environments where a service runs (or will run)
Use case 2: Promote a given release from environment 1 to environment 2
While use case 1 can be served by specifying the environment and service in CI, I think that use case 2 needs an external configuration of the sequence of environments.
As a result, I think about in two steps:
define the service, including the environments where it's expected to deploy to
connect a specific CI/GitOps deployment with a service+environment (this is the closes GitLab has to releases)
The snippet you provided would be used for step 2.