Show the merged version of .gitlab-ci.yml (with include and extends)

added DEPRECATE_pipeline authoring UX devopsverify sectionops typefeature + 1 deleted label

This issue is a follow-up from #244838 (closed)#.

cc @jyavorska @dhershkovitch FYI @v_mishra

Setting label(s) grouppipeline authoring based on ~"Category:Pipeline Authoring" ~"ci::authoring".

added grouppipeline authoring label

added 1 deleted label

mentioned in issue #249055 (closed)

changed milestone to %Backlog

added workflowdesign label

@nadia_sotnikova I was wondering if it makes sense to work on a design for the Editor Lite instead of the CI Linter

@dhershkovitch Would you like to stop the work on the CI Lint and shift focus to the Editor Lite completely? That makes sense to me.

We have several issues open for the improvements of the CI Linter. Should we look at them together and see if they are still relevant for the Editor Lite?

For example, this particular issue is to make it more obvious in the CI Lint that it not only validates the .gitlab-ci.yml but also all of the includes. From the UI it's not clear since the Linter window says "Contents of .gitlab-ci.yml" and the includes are not visible anywhere.

I'm not sure how this experience translates to Editor Lite. A few questions I have about the Editor Lite:

Is this Editor Lite?
Does it have the same capabilities as CI Lint right now? Looks like it has some live linting (actually, not sure if it's called linting), but it's different.
Doesn't look like it validates the includes. It's just the main CI file.

mentioned in issue #258789 (closed)

@dhershkovitch Here's my stab at a proposal for this.

We already validate the includes for the file that's being validated. In #244838 (closed) that is scheduled for this milestone we'll allow users to download the yaml with merged results. It's an MVC solution.

Once we add the button, this is what a validated file with includes will look like. You see the includes are there, but it's not possible to view them in the context of this file unless you download the whole flat file and view it locally.

Experience that will be available as of 13.5

For the next iteration to make the includes more accessible, we could make it possible to expand them in the original file. By default they'll be collapsed. It will serve as an affordance that the includes are also being validated.

We already have this capability in the linter, and just need to apply it to the individual include lines.

Here's a quick sketch of what it could look like.

In the future we should look into whether the behavior of automatically validating includes is desired, but this can be done when we work on the linting/ validating experience for the separate CI editor.

@fabiopitino Originally it was your suggestion to expand/collapse the includes, so wanted to loop you in to see if the designs I created make sense/ are feasible.

@nadia_sotnikova designs looks good moving this to workflowplanning breakdown

Added the proposal to the issue description.

assigned to @nadia_sotnikova

marked this issue as related to #249577 (closed)

added workflowplanning breakdown label and removed workflowdesign label

@cheryl.li @samdbeckham this issue is ready to be weighted, should i label this as frontend?

I'll pop that label on now @dhershkovitch and get @f_caplette to refine and weight it.

@f_caplette could you refine and weight this issue for the frontend please?

@samdbeckham @dhershkovitch Could we make it this issue blocked by the issue to add the download yaml button (as from my understanding, it will be blocked)

Also since this will come after the work of the other issue, it means we just want to add the collapsible section to the includes correct?

@f_caplette

Could we make it this issue blocked by the issue to add the download yaml button (as from my understanding, it will be blocked)

We can but is there any technical reason that one is blocking the other?

Also since this will come after the work of the other issue, it means we just want to add the collapsible section to the includes correct?

Do you mean only to add the expanded version of .gitlab-ci.yml when the validation is successful? If this will make this issue easier to implement then i am ok with that, however i do see value of seen the expanded version of the ,gitlab-ci.yml file even if the includes are not correct

@dhershkovitch Sorry about the late response. I took some time to investigate this and make sure I understand this properly. The problem with this issue is that what we want to do, is to basically have access to the yaml of each includes file so that we can add their content under the "header" as a collapsible section. Right now, the FE doesn't have access to that data, so I had assumed that the work would come after the download merged yaml file, because then the API that gives the FE the fully merged yaml would be available, but maybe that's not what we want here?

If we want that after clicking "validate" the content is updated with the entire yaml, including the content of the merge files, then there would definitely be some BE work cc @cheryl.li

We need to establish how we want to proceed with these 2 issues:

Do we add in the API response of the validate the yaml of each included file? Are there any performance pitfalls?
Does the API endpoint that will return the fully merged yaml necessary before we can do the work here?

Maybe @furkanayhan would know

@f_caplette thank you for the ping. TBH, I am still trying to understand the proposal.

Right now, when you open the CI Lint page, we see a blank editor at first:

Firstly, users need to past their CI config files to there, right?

How will be the user-flow exactly?

User pastes the CI config
User clicks the "Validate" button with "Simulate a pipeline created for the default branch"
User sees the expanded version of the "include"s.

Where do we show it? Like in the proposal of this issue:
If so, we make the CI config invalid now, right?.

I probably misunderstand the proposal here. So it would be great if you could explain the user-flow, please?

cc @nadia_sotnikova @dhershkovitch

@furkanayhan In #244344 (closed) we're changing the current experience to automatically load the .gitlab-ci.yaml and validate it as you navigate to the Linter page.

The proposal is to make it possible to expand the includes within the original file, but by default they'll be collapsed.

@f_caplette

Does the API endpoint that will return the fully merged yaml necessary before we can do the work here?

Thanks for the explanation this makes sense now, I am marking #244344 (closed) as a blocker.

I also think it makes sense to wait for the backend API. But I would like to get the backend team approval.

/cc @cheryl.li @furkanayhan

@furkanayhan Is it clear what backend is to answer on this issue?

@f_caplette @dhershkovitch From what I understand in:

Does the API endpoint that will return the fully merged yaml necessary before we can do the work here?

Are you suggesting that the backend API needs to return the full yaml (with all included content)? I assume this would be needed if we are to refine this as a ~"candidate::13.7" issue.

@cheryl.li yes, in #244344 (closed) we will add the gitlab-ci.yml content automatically. And in this issue we would like to add the includes yml content in the linter, so it looks like FE needs to receive the includes from BE.

@cheryl.li My concerns about this issue were stated in the comment thread below. So, I would prefer to make a comment about backend side of this after seeing a clear picture of the issue.

added direction label

changed the description

added frontend label

assigned to @f_caplette

unassigned @nadia_sotnikova

mentioned in issue #249577 (closed)

mentioned in epic &4534

marked this issue as related to #244344 (closed)

added 1 deleted label and removed 1 deleted label

@nadia_sotnikova @f_caplette, After reading this issue carefully I still have some concerns:

What would be the user experience in case the user will expand the includes in the linter and then try to edit its content?
- Do we allow them to update the expanded includes?
- In case we do will we validate the updated version, (I assume not which means that updating is possible but unnecessary, this seams problematic from the user experience POV)
The user flow the way I see, users can validate their .gitlab-ci.yml file, update it when needed until the syntax is correct, then copy-paste the .gitlab-ci.yml from the linter to the file, if we will add the includes in the linter the user will unable to complete this flow.

@dhershkovitch That's a very good point actually and I think that's why it's been so hard to wrap my head around this issue

I am not sure there is an easy workaround for this. If the user resubmit the content of the newly generated content of the file with the includes lines and their content, will it passes? Probably not since includes local fetch the files in the repository, but we are adding content under it as well. And even if the BE ignores the content under the includes, then if the user modify that, and re-validates it, then it wouldn't evaluate their change, just the file of the include.

There might be other options to explore that might interesting. One very simple solution would be to simply show a banner/alert that says that "Your CI file has been validated with all of its includes" which is probably the easiest thing to communicate it to the user without getting into complex linting problems.

Another option would that instead of rendering the content under the include, we get the API endpoint that returns the fully merged yaml and that's what we render in the editor, that way the includes are "swapped" with the content and we avoid this in-between state that only exist in the UI and cannot be validated is gone. The user that edits that content will also be able to re-validate the content so this is the ideal solution imo.

@furkanayhan Just to confirm, do you know if we send a .gitlab-ci.yml content to the lint endpoint like this:

my_job:
  script: echo hello
  stage: build

includes:
  - local: my/file/path
      my_other_job:
        script: echo bye
        stage: deploy

What will happen to the jobs under the includes given that the local file exist? Would the linter ignore the job under the includes? Would it try to parse it and fail?

@f_caplette

There might be other options to explore that might interesting. One very simple solution would be to simply show a banner/alert that says that "Your CI file has been validated with all of its includes"

I guess this is a safe and easy bet for now, I think that we need to conduct some problem validation about the flow and the desired outcome, we should probably think beyond what is available today, but it needs to start with the user POV, i will take the followup on adding this to our workflowvalidation backlog

Follow up issue #273254 (closed)

@f_caplette I agree with your statements here.

If the user resubmit the content of the newly generated content of the file with the includes lines and their content, will it passes? Probably not

Correct!

One very simple solution would be to simply show a banner/alert that says that

Agree!

What will happen to the jobs under the includes given that the local file exist?

It is not even a valid YAML syntax, so it will fail.

Thank you for your comments here. Those what I also did not understand when I first looked at this issue.

Thanks for raising these questions!

Given the concerns and no easy way to address them at this time, this seems like a great MVC:

There might be other options to explore that might interesting. One very simple solution would be to simply show a banner/alert that says that "Your CI file has been validated with all of its includes" which is probably the easiest thing to communicate it to the user without getting into complex linting problems.

Thanks everyone for validating, I have created a separate issue for the MVC that I can easily work on without any BE involved in %13.6. Here is the issue: #273434 (closed)

added workflowvalidation backlog label and removed workflowplanning breakdown label

added 1 deleted label and removed 1 deleted label

mentioned in issue #273254 (closed)

mentioned in issue #273434 (closed)

mentioned in issue #280844 (closed)

@nadia_sotnikova @f_caplette

I am rethinking this issue, we know that our users would like to see a merged .gitlab-ci.yml (with the includes and extends),

Another option would that instead of rendering the content under the include, we get the API endpoint that returns the fully merged yaml and that's what we render in the editor, that way the includes are "swapped" with the content and we avoid this in-between state that only exist in the UI and cannot be validated is gone. The user that edits that content will also be able to re-validate the content so this is the ideal solution imo.

Maybe we can add a button in the linter where users could toggle between a merged/unmerge version?

We can enhance this experience by coloring the "swapped" lines (in the merge version) to indicate to our users that those are separated files. (maybe also adding a comment with the file name and the location)

WDYT?

I like the idea! @dhershkovitch If the Linter will show the full config with includes accessible right there in the file, would it make sense to allow to edit the whole merged config file all in one go in the Pipeline Editor? I wonder how unruly such merged config file can get.

I guess my question is, if we know that developers want to see a full config with merged includes and validate it, do they also want to edit that file and commit it all in one place rather than have to make changes in individual files?

When someone validates the config in the "Validate" tab which will have the linter in it, they will then want to save the changes as they troubleshoot. Makes no sense to have them copy the changes they made to the merged includes and go to the specific include file to save those changes. Wdyt?

We can enhance this experience by coloring the "swapped" lines (in the merge version) to indicate to our users that those are separated files. (maybe also adding a comment with the file name and the location)

Also cc @mrincon because he's been working on the linter and seems like we're now at a point where the linter will validate the config continuously as you type (correct me if I'm misinterpreting).

I wonder how unruly such merged config file can get.

Very! A gitlab project pipeline may have hundreds of jobs, and it make also add some templates that are not part of the project. Also, I think that the merge operation operations is not reversible, and users might not be able to work with a merged version of the yml directly.

When someone validates the config in the "Validate" tab which will have the linter in it, they will then want to save the changes as they troubleshoot.

@fabiopitino raised a very interesting question today about the lint layout: "why show it in a table?"

This tab becomes more interesting when we show the user the entire resulting file, merged with all the templates and includes.

I think this tab should not have editable text, but it would allow the user identify what the resulting config is and it might be easier to use than a Downloaded file.

I guess my question is, if we know that developers want to see a full config with merged includes and validate it, do they also want to edit that file and commit it all in one place rather than have to make changes in individual files?

@nadia_sotnikova Keeping configs in separate files has the advantage of making them reusable and/or easier to manage. Note that the include keyword allows also to include content from other projects or even from a remote URL. This means that the merged result cannot be committed in one place.

It may be ok to allow users to "play" with the merged YAML by changing and validating it but not committing it.

@nadia_sotnikova

I guess my question is, if we know that developers want to see a full config with merged includes and validate it, do they also want to edit that file and commit it all in one place rather than have to make changes in individual files?

This is an interesting question which requires thorough validation, I can only assume that the answer is it depends on the use case, I also see technical limitation which prevents us from implementing such behavior, I think that for now when should allow users to view the merged .gitlab-ci.yml file in a view-only mode, and have conducted some problem validation prior from taking the route of allowing our users to edit the include: files

@nadia_sotnikova I do want us to move forward with an MVC to allow users to see the merged .gitlab-ci.yml file in a view-only mode.

The way i envision this, is that we should have a button on the linter that would allow our users to view the merged version of the .gitlab-ci.yml file.

@f_caplette @fabiopitino any limitation regarding the permissions to view includes in private project? (do we allow to include files from private repos)

@dhershkovitch There was an issue to allow users to download the merged YAML. It would still be the easiest option I suppose, but not ideal experience. If we can show it in the UI, here are my thoughts on this:

The way i envision this, is that we should have a button on the linter that would allow our users to view the merged version of the .gitlab-ci.yml file.

As far as I remember from the demos, the Linter in the Pipeline Editor only shows the results of the linting and not the file itself. In that case, I think viewing the merged yaml would make much more sense in the actual editor tab. Wdyt?

We could use this toggle component to toggle between two views in the editor tab. It'd be a nice intuitive interaction, easy to switch from one to the other. Would it be feasible?

@nadia_sotnikova

There was an issue to allow users to download the merged YAML. It would still be the easiest option I suppose

#244838 (closed)

We can still do it, regardless of my proposal.

I think viewing the merged yaml would make much more sense in the actual editor tab.

As long as we'll be able to present to our users the merged version I am fine with that.

Maybe we should implement the download merged yaml at the first iteration and view the merged yaml in the 2nd one, although for a user it makes more sense to see the merged file first and only then allow it to be downloaded

@dhershkovitch @nadia_sotnikova I think that displaying the merged YAML in the UI would be easier to do than downloading it because the frontend already has the data coming from the Lint controller/GraphQL, it's just not displayed anywhere.

If instead we want to make it downloadable we have to go again to the controller, lint the content again and prompt the merged YAML for download.

Thanks @fabiopitino i believe it also provides an improved customer value then downloading a merged yaml file.

@nadia_sotnikova let's update the designs in the issue description

Technically it is possible to create a download in the browser with window.URL.createObjectURL(data);.

That said, a tab showing the contents it better UX IMO as well

As far as I remember from the demos, the Linter in the Pipeline Editor only shows the results of the linting and not the file itself....

Didnt we agree to move the linter as is since we didn't discuss pipeline simulation?

/cc @mrincon @nadia_sotnikova

added SUS Survey label

mentioned in issue #290656 (closed)

marked this issue as related to #290656 (closed)

mentioned in issue #290682

added workflowdesign label and removed workflowvalidation backlog label

added candidate13.9 label and removed 1 deleted label

added Low SUS Score label

changed title from Make it more obvious that CI Linter validates the includes as well as the .gitlab-ci.yml to Show the merged version of .gitlab-ci.yml (with include and extends)

@nadia_sotnikova I've update the issue description

@dhershkovitch I added a mock-up based on the latest editor demo with the buttons added in an area directly above the editor. Does the wording on the buttons make sense?

In case your CI config files is called something other than ".gitlab-ci.yml", we should either use a more generic copy for that view or change it based on the config file name.

My two cents are that hierarchically "viewing" the final YAML with merge includes should a tab be at the same level as the "visualization" tab or the "lint" tab.

In those 2 cases the user gets to see a holistic preview of their pipeline, same as the "YAML with merged includes".

@nadia_sotnikova @mrincon

Does the wording on the buttons make sense?

I would go with something more generic view merged file or view merged yaml

In case your CI config files is called something other than ".gitlab-ci.yml", we should either use a more generic copy for that view or change it based on the config file name

Miguel idea solved that problem

Anyways I do believe that the pipeline editor is likely to change in a future iteration

Great point, design in the description updated. @dhershkovitch

@mrincon As for the content of the "View merged YAML" tab, can we show it as a snippet to make it clear it's not possible to edit it?

@nadia_sotnikova I think we may be able to use the blob viewer to display it, if that's not possible, the we could use a read-only editor.

@mrincon That's fine by me, thanks!

changed the description

@samdbeckham, the final design is still not settled but I think we can weight this issue

The final design is in place moving this issue to workflowready for development

@f_caplette is there enough in here to weigh this now?

@samdbeckham Since we already have the content of the expanded yaml, adding a tab that show this as readonly shouldn't be more than a 2.

@fabiopitino is the merged result of the yaml file includes also the extends keyword

@dhershkovitch The merged result should contain the content after expanding the extends keyword. So the extends keywords should not be there and their content inlined instead.

@dhershkovitch @fabiopitino Is there any update on this thread? It seems it was never implemented.

The merged yaml API still does not bring in the extends reference and merge it into what would be the "final" job yaml.

So today the "merged yaml" is incomplete.

@flynneva the purpose of this issue was to view the merged CI yaml file in the UI, this is possible today by navigating into the pipeline editor, if there is a need to output the merged yaml through an API then you'll need to open a separated issue.

the purpose of this issue was to view the merged CI yaml file in the UI

@dhershkovitch I get that. My question was related to this thread that discusses the implementation and whether or not to expand the extends keyword in the merged yaml.

It appears in both the editor UI and via the API the merged yaml still includes the extends keywords...when @fabiopitino suggests it should not.

Do you or anyone else have any status on this?

Since this feature of expanding the extends in the merged yaml file is in the description of this issue...I think it relates to this issue.

It seems to me this issue was closed prematurely.

@dhershkovitch ah nevermind. I realize now that the extends are merged in there but the extends keyword is left in there to show when a job was extended. Nevermind

mentioned in issue #280802 (closed)

added workflowplanning breakdown label and removed workflowdesign label

Show the merged version of .gitlab-ci.yml (with include and extends)

Release notes

Problem to solve

Intended users

User experience goal

Proposal

What is the type of buyer?

Designs

Child items ...

Activity

Show the merged version of .gitlab-ci.yml (with include and extends)

Release notes

Problem to solve

Intended users

User experience goal

Proposal

What is the type of buyer?

Is blocked by

Relates to

Activity