With #29568 (closed) we add the ability to fetch a merged .gitlab-ci.yml (containing all includes rendered as one YAML) via the API. It would also be handy if you could fetch it via the web interface, from the linter.
Make it easier to use this feature for anyone, even folks not comfortable with the API.
Proposal
Add a button that downloads the combined YAML to your computer. This design is using the default small button component with a download left aligned icon.
Further details
Permissions and Security
Documentation
Availability & Testing
What does success look like, and how can we measure that?
What is the type of buyer?
Is this a cross-stage feature?
Links / references
This page may contain information related to upcoming products, features and functionality.
It is important to note that the information presented is for informational purposes only, so please do not rely on the information for purchasing or planning purposes.
Just like with all projects, the items mentioned on the page are subject to change or delay, and the development, release, and timing of any products, features, or functionality remain at the sole discretion of GitLab Inc.
I assigned this issue to @nadia_sotnikova as she will be taking over the work from @v_mishra on the Pipeline Authoring items going further.
@jyavorska this issue was added as a candidate 13.5 today and we are trying to finalise the already existing scope for 13.5 by the end of the week. I dont think this will be realistic and propose to move it back to candidate 13.6.
@nudalova this is also just a single button. Commitment is due in two weeks and I suspect the engineering team is able to weight adding a button between now and then. What do you think?
@jyavorska Thank you for adding the proposed solution.
I understand that it may look like a small issue, however it a Product Designer will still need to validate that it is a correct approach for the problem, collect information around it and manage the proper team collaboration. @v_mishra is working at her capacity to finish the other items for 13.5 this week.
Going further, I propose that we use a process to measure UX capacity to make sure we align on the UX velocity and expectations between UX and PM a bit more upfront. @v_mishra and @nadia_sotnikova will be driving this going further.
As discussed we'll try to get this done targeting the due date of being completely done (with engineering also) by the 13th when the team does commitment.
@jyavorska Am I understanding correctly that we want to make it possible for the user to download a YAML file that contains their full pipelines configuration that goes beyond just the default config .gitlab-ci.yml and contains all the include: pipelines?
When a developer downloads this YAML, do they want to use linter to validate it? If there are other use cases, what are they?
That's correct @nadia_sotnikova. They don't need to re-lint it since it's equivalent to linting without things expanded (the linter expands things), but they would look at the contents to really understand what's happening.
To me it wasn't obvious that the linter expands and validates everything in the background. Perhaps in the future it could be useful to allow the user to somehow expand things directly in the linter or indicate somehow that the includes have also been validated. Unless it's obvious to a developer, which may be the case.
For now a download button seems like a reasonable MVC. I want to make sure we define the behavior of the button for all use cases.
On load we'll automatically validate the gitlab-ci.yml. However, I can also use the CI Linter to validate any CI file, not necessarily the default config of gitlab-ci.yml. Is there any situation that I'm missing where the button Download YAML with merged includes will be confusing because it's unclear what YAML we're talking about?
In other words, does the button have to be dynamic based on which file the user is validating?
The linter does, it's required but on the other hand it may make sense to make it obvious. We should take advantage of your fresh perspective to make improvements like that.
The linter will always fetch the .gitlab-ci.yml in the editor with its includes, so that should be clear.
The linter will always fetch the .gitlab-ci.yml in the editor with its includes, so that should be clear.
Okay, makes sense!
I'm considering two options for the button placement. We can put it next to the header, or at the top of the linter window. I'm leaning towards using a smaller button in the linter window to keep it a close to the context of the file as possible, but wanted to see what you think.
@nadia_sotnikova I have a couple of questions before I can weight this issue:
Reading this thread, I got a bit confused as to whether or not clicking on the "Download YAML with merged include" would parse it right away or not? Because the action is clearly labelled as "download" and not "lint" I would not expect it to do both of these at the same time (maybe it's just a copy issue?)
Secondly, what would be the loading state while it loads? We don't want the user to be able to edit the textarea since it will all get overwritten anyway once the loading is over, and if we download and then parse it, the loading can be quite long. We might need a clear loading state that's not just a spinner on the button for example. Do you have any insights on this?
Also, I agree that it looks nice inside the grey box Great work!
As far as I understood, the Download YAML with merged includes will download your .gitlab-ci.yml with merged includes, regardless of what's in the linter input. By default we populate the linter input with .gitlab-ci.yml, so that makes sense.
However, it is true that the user can also validate any CI yaml there, and may expect to only download the yaml with includes for the file they're validating. I had a question about it here, wondering whether the button behavior should be dynamic based on the content of the linter, or if it should always pull .gitlab-ci.yml.
If the button only downloads .gitlab-ci.yml regardless of what's in the linter input, then it should be enough to disable the button while it's processing.
@jyavorska can you help clarify the intended behavior here?
@nadia_sotnikova@f_caplette we should just add a button to call the API endpoint that @fabiopitino is adding to download the complete YAML and not change the way that endpoint works at all, which is that it takes a user defined input and processes it, but does not run it.
Fabio hasn't shipped his part yet, it's part of 13.4, but he could confirm if you're wondering how it works.
@jyavorska Just to confirm, even if the user pastes a different CI file (not .gitlab-ci.yml) into the linter and validates it, clicking Download YAML with merged includes should download .gitlab-ci.yml with merged includes? So it will not pull anything from the linter input.
Secondly, what would be the loading state while it loads? We don't want the user to be able to edit the textarea since it will all get overwritten anyway once the loading is over, and if we download and then parse it, the loading can be quite long. We might need a clear loading state that's not just a spinner on the button for example. Do you have any insights on this?
A few clarifying questions.
What do you mean by it will all get overwritten anyway once the loading is over?
How long might the loading take?
What do you mean by if we download and then parse it?
Also, to confirm, the download should download a file to your computer I think, for this iteration, as in actually download a file. It shouldn't replace the text with the merged version, or do that and then re-validate it again.
@jyavorska@nadia_sotnikova@f_caplette When the user clicks the "Validate" button (with or without the simulation checkbox) it sends the content of the editor to the CI Lint which merges the various includes and validates the merged content. When the Lint returns the result back to the user we could directly return the merged yaml as part of the result.
The result today includes any errors, warnings and, if the content is valid, a list of jobs. We would need to enhance the Lint to also cache the merged yaml and return it as an additional field in the result.
This means that we don't need to go to the Lint again if the user wants to view the merged YAML. The merged YAML would be already passed to the view to be displayed. An idea could be to allow the user to switch between "initial yaml" and "merged yaml" view in the editor itself. On the frontend we would swap the content in the editor with the merged yaml so that the user can even play with the merged yaml. The user can switch back to the yaml initially provided as input, and again back and forth with the merged yaml.
What do you think about that?
Alternatively we could still take the content of the merged YAML (already available in the frontend) and rather than displaying it somewhere in the page we open the raw content on a new page when the user clicks "download file".
@jyavorska makes sense! The main correction I wanted to make was that we don't need another roundtrip to the Lint in order to get the merged YAML as we would make it available with the first pass of the Lint.
@nadia_sotnikova Jason answered my question, I somehow misunderstood the proposal.
What do you mean by if we download and then parse it?
I think I am just confused Currently the proposition says:
Add a button that downloads the combined YAML to your computer. This design is using the default small button component with a download left aligned icon.
So from my understanding, clicking download button will fill your text area with a combination of ALL your gitlab-ci.yml in a one file format and... that's it. If that's not the case, could we update the proposition with what we want?
I am confused as to why we are talking about how this button will also lint it. Or @fabiopitino comment's
we don't need another roundtrip to the Lint in order to get the merged YAML as we would make it available with the first pass of the Lint
But we would get the merged gitlab-ci by clicking on the download button right?
The download button should just download a file to your computer @f_caplette. It shouldn't replace the text with the merged version, or do that and then re-validate it again.
Maybe some day in the future but we will figure that out later.
@f_caplette Once #29568 (closed) is done we would also have the CI Lint result (from the controller) to return the merged YAML to the UI. At that point the merged yaml is already there, together with the rest of the result as well as the input content. All we need to do is to get the content and present it to the user in the form of a downloadable file.
@f_caplette is it possible for frontend to present the content as downloadable file without going through the controller again?
The download button should just download a file to your computer. It shouldn't replace the text with the merged version, or do that and then re-validate it again.
Given we want to provide the following MVC experience, it seems like we don't need any special loading state for the download button. @f_caplette What do you think?
The result today includes any errors, warnings and, if the content is valid, a list of jobs. We would need to enhance the Lint to also cache the merged yaml and return it as an additional field in the result.
This means that we don't need to go to the Lint again if the user wants to view the merged YAML. The merged YAML would be already passed to the view to be displayed. An idea could be to allow the user to switch between "initial yaml" and "merged yaml" view in the editor itself. On the frontend we would swap the content in the editor with the merged yaml so that the user can even play with the merged yaml. The user can switch back to the yaml initially provided as input, and again back and forth with the merged yaml.
@fabiopitino Thank you for the explanation and great ideas! In #244344 (closed) we'll automatically load and validate .gitlab-ci.yml when user navigates to the linter. It means that once you open the Lint page, you'll already have access to your yaml with merged includes, so we could show the UI with two tabs right away, for example. Or there could be some other way to switch back and forth between the two views. Is this what you envision?
Can you elaborate a bit on the use case for having the two views side by side? Feel free to comment on this issue. #249577 (closed)
@nadia_sotnikova Sorry about the slow response here. Correct, we wouldn't need any special loading state.
@fabiopitino I am not sure I understand what you mean Do you mean that if the user fills out the lint field with their data, and then press the lint button, then we technically could avoid another API call if the user clicks download? I thought that the download merge yml button would just respond with a blob that we can download on your machine. From my understand, filling the UI with the resulwould be a following item.
@nadia_sotnikova@f_caplette Perhaps in this issue we jumped immediately to a solution and I think we are planning it around that. In #29568 (closed) we will be making the merged yaml an attribute of the Ci::Lint::Result object. Currently this object is being sent to the view via the controller which contains data such as jobs, errors and warnings. We will be adding a new attributemerged_yaml which you can call as @result.merged_yaml in the HAML partials.
This means that the data is already there in the view and we don't need another roundtrip to the controller to return the data we already have.
At this point my initial question is whether we are restricting ourselves to the idea of downloading the merged yaml Is there another simple MVC we could have to show this data to the user directly in the UI? - cc @jyavorska
Also note that #29568 (closed) is currently blocking this issue from backend perspective as the merged yaml will only be available to frontend afterwads. I've mentioned it in #244838 (comment 411393561). I've now marked this issue as blocked in the related issues section.
Is there another simple MVC we could have to show this data to the user directly in the UI?
@fabiopitino There are definitely better ways to do it, but with the time and information that I had, I don't feel very confident to implement other solutions without any validation. A download seems like a safe MVC option to learn from.
But if you have any suggestions for taking this a step closer to the optimal experience, here's my current thinking. It seems like the ideal situation would be to have the main CI config file and the includes all accessible from one place where you can validate and edit them.
A step in this direction could be to have CI Lint validate and show CI with merged includes by default, and we could indicate which files are included in the linter, or just say that the CI Lint validates .gitlab-ci.yml with all includes. Afaik, we already validate the merged includes, so basically we need to reflect the current behavior in the UI.
cc @dhershkovitch since we talked about shifting our attention on improving the single editor experience vs. investing more into CI Lint. From that point of view, we may want to go with the simplest MVC here and move on to the editor improvements.
Jason Yavorskachanged title from Ability to render a processed (merged) gitlab-ci.yml file via UI to Ability to dowload a processed (merged) gitlab-ci.yml file via UI
changed title from Ability to render a processed (merged) gitlab-ci.yml file via UI to Ability to dowload a processed (merged) gitlab-ci.yml file via UI
Jason Yavorskachanged title from Ability to dowload a processed (merged) gitlab-ci.yml file via UI to Ability to download a processed (merged) gitlab-ci.yml file via UI
changed title from Ability to dowload a processed (merged) gitlab-ci.yml file via UI to Ability to download a processed (merged) gitlab-ci.yml file via UI
@dhershkovitch I think that the work here will be easily portable to the new section and the tests as well. It's going to slip to %13.7 anyway so if we decide to do it only in the new section, that's okay too.