Currently it is not possible to show upstream or downstream pipeline information in context of the current pipeline in order to be able to backtrace what triggered which pipeline and/or what other pipelines did it cause to trigger.
Proposal
When projects are connected via triggers or dependencies, we should display a complete pipeline graph of the entire flow, across all projects.
For any given project, show all upstream and downstream projects in the same pipeline graph, clearly indicating which jobs belong to the pipeline in focus. e.g. put external jobs
Note, this is not about showing artifact, image, submodule or other dependencies. It's about what pipelines cause other pipelines to run.
Upstream/downstream pipelines are expandable+collapsable showing their pipeline graph inline
Inline upstream/downstream pipeline graphs are positioned along the Y-axis according to the position of their project position in the upstream/downstream lists from which they are expanded from.
Only one downstream can be expanded at a time (if you expand another one, the initial one will be collapsed/replaced)
Expanded upstream pipelines do not show downstreams of their own which are non related to the current pipeline
Upstream/downstream loading states temporarily replace the pipeline status icon with a spinner
Scrolling stays unaffected, except for positioning expanded/collapsed content into view.
FE notes:
Fetch the main endpoint pipelines/{id}.json
Look for triggered_by for the upstream
Look for triggered for the downstream pipelines
Look for triggered_by.path and triggers[idx].path it only misses the .json
Frontend will start polling on click for each upstream/downstream
it’s not possible to have any upstream/downstream expanded by default bc we won’t have the data. the user has to click on it, we are all ok with this
We will poll the endpoints for both upstream & downstream pipelines
Once the user clicks to collapse, we will stop polling. if the user clicks again we fetch again and show a loading again
Larger projects, especially those adopting a micro-services architecture, often have a set of interdependent components that form the complete product. When working with these projects, it is important to be able to build, test, and deploy both the individual parts as well as the whole. For example when developers are working on a low level service, it is important to ensure downstream components are not negatively impacted by changes. With Cross Project Pipelines, GitLab will have first class support for managing and running CI/CD pipelines across projects. Users will be able to easily link a projects CI/CD stages into the full pipeline, as well as view current status.
Specific documentation blurb
In a multi-project scenario, it is very important for developers to access some basic details of related projects when viewing upstream and downstream links. Clicking on the project you're interested in, it will be expanded to a more comprehensive view that shows stages and jobs, as well as further dependencies. You can iterate on it in order to get the full view of the complete pipeline.
What does success look like, and how can we measure that?
The user is able to see which pipelines caused the current pipeline to run (all the way upstream to the original pipeline that set everything in motion).
The user is able to see which downstream pipelines are caused by the current pipeline, up until the last generation of every downstream.
@markpundsack nice! I am thinking in terms of nested pipeline views in which certain builds/jobs or actionable builds/jobs are promoted to be visible in the "nested" pipeline element.
I get this idea from sidefx houdini's node system:
Nesting could work, but sounds more like gitlab-ce#22550.
In this issue I was thinking of showing one big pipeline. Maybe by expanding and collapsing "nested" pipelines, we'd knock off both issues. I honestly don't know which is more valuable in practice, but I imagine many people coming from Jenkins will expect to see a single view of their entire cross-project pipeline.
ha nice, that's both crazy and nice at the same time. I think if we'd make the pipeline view draggable, it could support such big pipeline graphs (if you look up a big pipeline graph on your mobile it gives you a sense of what I mean)
We should however think of what we want to solve exactly... meeting expectations of users from a different product with a different context... or what does the user need in that view.
say we have multiple upstream projects of which we use data... (I assume we always pick the last stable/passed pipeline data)..don't we want a visual of those pipelines if we don't use the latest HEAD?
@dimitrieh That's a good question. If we support having a pipeline depend on multiple pipelines, it would trigger when any of the upstream projects change, and pull the latest from the unchanged ones. I still wonder if we can ignore that in this simpler issue, and leave that for https://gitlab.com/gitlab-org/gitlab-ce/issues/22558.
As I understand it, a dependency indicates that a pipeline cannot start until another succeeds. But that doesn't necessarily mean that the dependent pipeline will automatically start when the other one succeeds, so it's a different behavior from triggers. Shouldn't we represent them differently, then?
@cperessini My assumption is that there will be only one type of dependency that you declare in pipelines, and that will trigger downstream pipelines to run. But I can see the case for wanting an easy way to declare the dependency, but not trigger on it. Again, better suited for gitlab-ce#22558, so moving conversation there.
Thanks @dimitrieh, this looks great. I hadn't thought about displaying the current pipeline in a compressed view, but it makes sense. There's going to be some interesting graph drawing when you have multiple upstream/downstream flows. e.g. what would it look like if Prometheus also had two upstream projects, or if Gitaly had multiple downstream projects? Or even worse, what if some of those projects were the same? e.g. Prometheus depends on GitLab Shell, but not GitLab Build?
If we stick to only graphing the trigger pipeline, then there's always a 1:1 or 1:many relationship, but never a many:many relationship. That keeps the graph drawing so much simpler. And also has a nice property that all the pipelines shown were actually run together. i.e. in https://gitlab.com/gitlab-org/gitlab-ce/issues/22558#note_25260311, GitLab Build's pipeline could have been run several days earlier and you're just grabbing the latest result.
Just like we only show jobs that actually run in the current pipeline view; we hide jobs that are defined but not "run". There's an interesting parallel there.
Some more edge considerations. What would it look if Gitlab Shell had 12 stages to its pipeline? Are users going to be confused by having the same icon used to represent the overall project pipeline status and in the minigraph for that project pipeline? Can we drop the overall status and just show the mini-graph directly? How about going further and just drawing the mini graph directly, without a bounding oval around it? (I know the bounding oval presents a nice place to put the project name and other meta information.) I'd like to see an exploration of an expanded graph directly connecting the mini graphs. I wonder if users would want an even more expanded view of full pipeline graphs, not just the mini graphs. Probably gets really crazy at some point, and might not be the default view, but someone debugging or just trying to understand the full pipeline might want to see the whole thing in all its glory.
If we stick to only graphing the trigger pipeline, then there's always a 1:1 or 1:many relationship, but never a many:many relationship.
I think this actually makes the most sense.. with:
graph drawing is simpler
all the pipelines shown are actually run together
But also because the pipeline flow (graphically displayed with lines and nodes) is only important for showing things that are blocking other things.. this gives us a sense of what goes wrong where!
When a certain pipeline has a dependency on artifacts from another project, these are gathered from the last passed pipeline.. this will always be good code.. or at least it always will have passed the pipeline.. so not that interesting. Not to mention that it can be a dependency for multiple upstream and downstream projects at the same time.. complicating the drawing unnecessarily... However! We could show them differently, as it's still worth knowing which projects are imported to make a certain pipeline work and those will always be upstream projects.
bonus, we again don't need the trigger thingy..as the first project.. is the project pipeline that triggered everything.
Can we drop the overall status and just show the mini-graph directly?
Yes i think this makes sense here!
This would result in:
Can we drop the overall status and just show the mini-graph directly? How about going further and just drawing the mini graph directly, without a bounding oval around it?
but someone debugging or just trying to understand the full pipeline might want to see the whole thing in all its glory.
just a sneak.. but do you mean something like this?
What about a list for showing cross project pipelines? showing projects as stages instead of .. stages?
@dimitrieh First mockup is great. I'm still not sure how it would look if there were more than 5 stages in the minigraph. The +1 stuff seems totally unnecessary and is still subject to all the problems of the full many:many graph. It was really not obvious what they might mean, and even after clicking on them, is still a bit confusing.
For that last mockup, I was thinking more of the "full" pipeline, ovals with job names, not just status circles. One potential advantage of showing the fully expanded pipelines is that it would obviate the need for a separate tab. The Pipelines tab could show the entire cross-project pipeline.
What about a list for showing cross project pipelines? showing projects as stages instead of .. stages?
@dimitrieh Do you mean at the group or top level? Just like I want a global list of pipelines (gitlab-ce#18054), a global list of cross-project pipelines is probably useful. But something about having two separate tabs for project pipelines and cross-project pipelines seems wrong. It's a fine stepping stone, but I wonder what the end-game is.
I imagine https://gitlab.com/gitlab-org/gitlab-ce/issues/22550 would just augment the existing Pipelines tab. Is there a way to make the consolidate view an augmentation as well? Like clicking on an upstream or downstream project expands the pipeline inline to make the graph fuller, and adds any further upstream/downstream pipelines to the view. Maybe there's a button to expand them all at once. Or maybe having them always be expanded makes sense. I think the recent mockups in this issue are great for seeing relationships between pipelines, but is that the most common use-case, or do people want to see the actual jobs of the entire pipeline? Given that everything to the left of the current pipeline must necessarily be successful, is there value in showing the details? Or is the existence of optional manual jobs in the pipeline a reason to show the details? But even then, the mini-graph is still there, so you can get to individual jobs.
Now that I think of it, the mini graph is all I use most of the time now; I don't bother clicking on the full pipeline graph unless there's a problem rendering the dropdown list (which happens far too often). So I wonder if I'm thinking of this all wrong, and the focus is really on the mini graph in the MR view!? Should we think about how to show these cross-project dependencies in the MR view? It might be OK if it's a single line, but the branched trees we have will be really difficult to include. Well, unless we do the same collapsed trick we do for jobs, and show a single icon for downstream pipelines, click on it to see the list of downstream pipelines... I could imagine it even going further to then render next-level downstream pipelines dynamically as you click on them. Like see the minigraph with upstream and downstream, click on the downstream, see 2 downstream pipelines expanded to show their minigraphs, including their further downstream pipelines, click on them to see them expand to mini pipeline graphs... Crazy, I know. But as I use the mini graphs more, I really like them; both for the location (in the MR where I want them) and the compactness. I haven't really lamented the missing detail of the full pipeline graph.
At any rate, we still need the full pipeline view, so maybe we just also need a mini graph view of cross-project pipelines.
I see a failure in a pipeline and want to track down its root cause, which may be in an upstream pipeline.
I'm working in an MR, push a commit and want to watch the downstream pipeline to make sure it all succeeds.
I'm new to a project and want to understand the overall pipeline.
For 1 and 2, I'd want to work from the main MR view's mini pipeline graph, or at least the Pipelines tab.
For 3, perhaps it's OK to click on a different tab to see a more complete view. Then again, maybe 3 is better solved by gitlab-ce#20785 or gitlab-ce#21485.
Side note: We haven't defined what a failure in a cross-project pipeline means. If a downstream pipeline fails, should it cause the upstream pipeline to fail even though individual jobs/stages in that pipeline succeeded? Unfortunately, there are cases for both.
a) Consider open source project X with an unrelated project Y that declares a dependency on X so that Y is rebuilt every time X changes. There should be no way that the result of Y's pipeline affects X's.
b) Consider Omnibus which packages GitLab CE and GitLab EE. If a change in GitLab CE causes Omnibus to break, shouldn't the pipeline of GitLab CE reflect that it was the cause and thus notify the author of the change?
Perhaps that's the difference between a backward reference and a forward reference. In (a), Y would declare X as a dependency, but maybe in (b), GitLab CE would trigger Omnibus. Whoever declares the dependency claims failure status. That may not actually make sense, and even if it does, it may be too subtle to be intuitive. But one way to think about it is that if I declare the dependency in GitLab CE, then the dependency is part of CE's pipeline, and thus failure of downstream would fail CE's pipeline. Unfortunately, Omnibus would likely still need to declare what artifacts it needs from CE, so it still has to declare the dependency, and thus make things more ambiguous.
Further side note, back to (a), when looking at project X, would I really want to see anything about project Y? I think it would clutter my view if I saw all the project depended on mine, that I have no concern for. But when viewing project Y, I may need to see the pipeline for project X. Although honestly even in that case, I might not care about the details, just that the dependency exists and to know that that's why this particular pipeline was triggered. Sometimes project Y will have standalone pipeline because I pushed new code to it, sometimes it will have triggered pipelines because of an upstream push. Consumers of Ruby gems, for example, don't likely care about the pipeline for the gem. By virtue of being open source, I'd have permission to see the pipeline, but not the desire to do so (most of the time).
Can we tell the difference? Would we look at project membership and treat upstream/downstream project differently if I'm just a guess on them?
I took a spin at a view for showing expanded pipelines across projects. This view is supposed to be inside the Pipeline tab in a Pipeline page, but I'm showing it without context because it gets pretty big. If we were to include this, we would need to add @dimitrieh's idea for an area that you can pan around and zoom in/out of.
Overview
This is not part of the design. Just showing it for reference so you can get an idea of the structure of the pipeline.
Current project
Since we are in the Pipeline tab for a particular pipeline, we show that pipeline expanded by default. You can see its direct upstream and downstream pipelines.
Expanding the first level
When you click on a pipeline node, it expands to show you all stages and jobs. This pipeline has an upstream pipeline in the GitLab Shell project. Let's expand that one
Expanding the second level
When we expand the GitLab Shell pipeline we find out it doesn't have any upstream pipelines. We already knew it had a downstream pipeline (Gitaly), but we now see it has another one in GitLab EEP. I'm not sure if this is valuable information, since it doesn't affect the current project's pipeline, but it gives you a better sense of how all your projects communicate.
Expanding the third level
Finally, we can expand that second downstream pipeline of GitLab Shell and we see it doesn't have any more dependencies.
I forgot to include a way to collapse the pipelines back into nodes, but we should add that. We could also have a button to expand all nodes into pipelines like @markpundsack suggested, but I think it should only act on the nodes that are currently visible, and not on the ones that are about to be loaded.
Hi @ChadMalchow - we're really excited about this enhancement since it helps validate the use of GitLab CI for multi-stage pipelines. We are still making the internal business case for using EE over CE and keeping this feature at the EE (not EEP) level would help there.
Based on your website, EE is "Ideal for co-located teams who need additional security and workflow controls for their professional projects.". This feels like a professional project feature to me.
EEP is "ideal for distributed teams who need advanced workflow controls, High Availability, and Premium Support.". This is not an advanced workflow control, distributed or HA capability.
Having great dashboard makes a lot of sense, but I would focus now on extending only mini-pipeline graph on Merge Request Widget with additional pipelines.
This would give us a very nice way to show dependent pipelines, without spending a lot of time now on implementing full-fledged dashboard as described here. Most people will probably look for feedback on mini-graph anyway.
@dimitrieh Possibly. If you're just trying to see what ultimately triggered the entire thing, you couldn't care about any parallel projects that happen to be triggered by the same thing. On the other hand, if you want to see the big picture of everything running, then you need to add all upstream and downstream items upon expansion. I think the latter is likely to be important.
@filipa I think this design needs a slight iteration and can be toned down a notch. Both to avoid confusion and to make it more of an mvp. It will also solve your question of
how do we show a mini-graph with 10 stages?
Creating new design, as we speak.. check back within the hour
Got rid of the mini-pipelines (those will create confusion and make it harder to implement. Overall pipeline status should be enough.. and the pipelines will be inspectable)
Overall pipeline status is now also visible when an upstream or downstream pipeline is expanded
Upstream and downstream column denominators are now "italic" as suggested by @bikebilly
@filipa Experiment: Figma design link, can be used to see most up to date design (as we literally work in it and can be inspected for measurements and attribute values (needs free account)
for this first iteration we will not vanish any expanded upstream pipelines ( or upstream pipelines that have become visible due to expanding of upstream-downstream pipelines)
see if we can make original pipeline nodes more visible when there are expanded pipelines.. or multiple upstream pipeline columns
Expanded pipelines no longer will have column title.
Expansions should be able to go both directions upstream & downstream.. with only 1 upstream possible at every time (its the pipeline that triggered the downstream)
I can see a problem with polling that we already have in environments folder rows:
We poll the first endpoint - everything works great
User clicks to open an upstream/downstream pipeline - we start polling this new data
We are polling the 1st endpoint which doesn't have info about the open upstream/downstream.
Problem: There will be a timeframe where we will show the upstream/downstream that was opened as close.
Solution
When we poll, before we replace the new data, we check for the old ones that were open and keep them opened. Problem: Polling for the main endpoint will not be synchronised with polling for upstream/downstream content. Making this happened without the user noticing and guaranteeing we are always showing the correct data will be harder than we've been doing until now.
@iamphill@ayufan any ideas how we can make this without creating a bad UX?
@dimitrieh Can we drop the expansion triangle? Is it really necessary? It's kinda ugly, and I think clicking on the pipeline and having it expand will be perfectly fine without the additional indicator.
Can we drop the expansion triangle? Is it really necessary? It's kinda ugly, and I think clicking on the pipeline and having it expand will be perfectly fine without the additional indicator.
@markpundsack there are a few problems with that as far as I can see: The additional indicator makes the nodes stand out more from the regular nodes, also it indicates imo that this UI element will expand inline of this view. Also it will give a click area both at the top as well as at the bottom, in order to once again collapse the expanded upstream or downstream pipeline. Please see my following prototype:
The additional indicator makes the nodes stand out more from the regular nodes,
@dimitrieh Sure, but they already stand out enough.
also it indicates imo that this UI element will expand inline of this view.
This is the crux of it. Is that really necessary? I'm already used to all these elements being clickable so sure, it won't be immediately obvious that clicking on it will expand, but once you do it once, you'll know and remember. And if you're coming from 9.3 where clicking on it takes you to the related pipeline, this will be a pleasant surprise that it just shows it inline.
In the meantime, the additional element for interaction degrades the visuals for presentation. That's always a tough balance, but in this case, personally, I don't think it's worth it. It's also adding a new element that we've never made before so it'll likely take longer to implement. We can implement the first iteration without it, and if necessary, add it in a subsequent iteration.
Also it will give a click area both at the top as well as at the bottom, in order to once again collapse the expanded upstream or downstream pipeline.
Yes, but we're cutting the ability to collapse, and if we never ship that, the additional click area is unnecessary. I'm not sure collapsing will ever be required. It might, of course, but it might not. Maybe people only expand until they find the job they're interested in, and then interact with it. Maybe no one will never need to collapse again. Or maybe people will be fine refreshing if they really want to reset the image. This way we at least defer that work that might never happen.
Thanks @dimitrieh for the updated design. I'm wondering why the blue background is from top to bottom of the screen and not just around the project, I'm not sure that the handling of the downstream of the first upstream (last image) is the correct approach, why won't it just go under the current project (that can keep the blue background around it)?
I'm wondering why the blue background is from top to bottom of the screen and not just around the project,
@bikebilly This is as the view can become pretty difficult/complicated. You may not even see it anymore.. as you have scrolled down so much due to downstream pipelines of upstream pipelines. That's part of the reason as well why I don't want downstream projects be displayed right under the current pipeline nodes. The other reason is that I don't want people to confuse downstream pipelines of upstream pipelines to easy with nodes of the current pipeline.
Sure, but they already stand out enough.
As we are cutting scope for collapsing (@filipa), let's get rid of the collapse UI on the nodes. It will then look like this (as they already do):
@filipa Do you need an adjusted version of the prototype for this?
What does this means? nothing will happen when we click on them?
Regarding the arrow, I think we should keep it. Pipelines are usually complex enough on their one, if we remove visual helpers they will be impossible to interact with. We need to tell the user that there is something to be clicked on.
@markpundsack after looking at this again (in the morning) and after talking with @filipa I disagree with removing the arrow and little bottom bar of the nodes.
This is indeed the crux of it. However I do not feel "I'm already used to all these elements being clickable so sure, it won't be immediately obvious that clicking on it will expand, but once you do it once, you'll know and remember" is strong enough of an argument (plus its tailored to an experienced user).
We can perhaps iterate on it visually in a next release to improve "the degrading of the visuals for presentation" as you call it.
I feel for now showing the "caret" is more inline with the visual style that gitlab uses (see dropdowns or other UI elements that open up things inline throughout the interface). Until we decide to change up that behaviour throughout the application, i'd rather go for consistency where people can predict what happens because of other experiences throughout the application, than an incoherent experience based upon only prior usage of exactly that one UI element.
@dimitrieh let's not do anything of that in the first iteration, either lines or background. Let's keep this as simple as possible or I am afraid it will be too hard to ship everything :)
On a more personal note, it makes it more confuse for me as a user to understand why we have a blue background. Starting to wonder if we should do it at all
@filipa the problem we're trying to address is that if you've a quite complex pipeline, the original project could be hard to find after a few expansions... and that could be fine, since you're focusing on other projects and each of them might have the same relevance.
The fact is that with the current design the actual project is even less than all the others, as the others have the box around then while the current one has nothing. So, if not making it more, at least let it be the same as the others, not less.
@dimitrieh what do you think to make it just as any other project in the pipeline, but "already expanded"? It could be consistent, and probably simpler for frontend because it is the same design applied to other elements.
what do you think to make it just as any other project in the pipeline, but "already expanded"? It could be consistent, and probably simpler for frontend because it is the same design applied to other elements.
@bikebilly that may be a solution. However this would change it as well for CE.. resulting in unneeded extra UI overhead.. Also it could imply that it can be collapsed etc.
@dimitrieh@filipa as discussed on slack, let's keep it as simple as possible and just remove the blue background. We can improve the design of the original project in another iteration.
So, this iteration will just focus on having the upstream and downstream expansion, with no collapse.
Pipelines are usually complex enough on their one, if we remove visual helpers they will be impossible to interact with. We need to tell the user that there is something to be clicked on.
I agree with @dimitrieh and @filipa on this one. For the sake of consistency and affordance, the arrow should remain as long as the action is possible. We can explore it more as we iterate.
@filipa@dimitrieh since this is a very big frontend job and we really want this to be shipped in %9.4, maybe we can implement the expansion without the visual helper (in reality we already should have the hover highlight that at least is something, right?), and create an issue with all the improvements proposed here. If we've enough time we can consider to pull some of them in %9.4, but at least we can be sure to have the MVP delivered (even if very basic). If people will complain about the interactions to be not easy to spot, we can always improve them later.
@filipa the other open topic on this is the way to refresh the additional pipelines while opened. Did you figure out a simple way to address this problem? Otherwise we can consider to cut out also this feature from the scope of the first iteration in order to guarantee the deliverable.
The visual helper is just an icon and not a problem of scope. It's definitely not a problem to do it.
Regarding polling, even if we remove polling from this part of the feature we still have a tech problem to address. The problem is:
We receive the first endpoint and start poll. This one only has data regarding what the user sees at the moment.
User expands a pipeline and we make another request with more data that will render more things.
In the meanwhile we are polling 1. that does not have data or info from 2. Which means that FE needs to update the data a lot more carefully in order not to show an abrupt change.
What we have now is: we fetch 1. we receive the data, we update the rendered output. We won't be able to do it just like this. or the user will notice.
Adding step 4. which is to poll the expanded pipelines is the least of all this problem :)
@filipa thanks for the analysis but I'm not sure to understand what the user will see that is so bad. Could you please give an example of that? Some less-than-perfect behavior could be acceptable if it solves complexity.
@bikebilly i think @filipa is talking about the fact that updating the expanded pipeline's can give visual artifacts.. like re-expanding it (as if it wasn't expanded already). Having spoken with @filipa i know for a fact we have this same problem updating environments with cluster info. This is not ideal.. we expect it in this feature to be far more visible than with environments. I leave this up to @filipa to figure out
The visual helper is just an icon and not a problem of scope.
agreed here! Cutting scope is rather in how things function, rather than look, as those require far more time and effort
As discussed with @ayufan we will cut scope and have only one upstream first.
The idea is to prepare a data structure that will allow us to have more than one upstream and polling, but for now, we will only work on showing 1 upstream.
We leave everything out, but can still expand upstreams and downstreams
Similar as the other proposal, but replacing the overall pipeline status with a mini pipeline graph (so individual jobs are inspectable in context of the current pipeline). The nodes will scale horizontally to accomodate the stages of the mini pipeline graph)
@filipa yes, by the end of today let's try to figure out if it's possible to have upstreams of upstreams and downstreams of downstreams, this is a blocking factor for this iteration to be valuable.
@dimitrieh I'm not really sure about the arrows, could we have something similar to what we have for stage expansion?
Exactly 1 column for the upstream with one or more nodes inside it
Exactly 1 column for the downstream with one or more nodes inside it.
When we clean in each node we are redirected to another page
Logic
This logic is currently being treated in the view layer, it is being handled in the component.
What we need
What needs to be rendered:
Render 1 or more columns for the upstreams, each can have
Render 1 or more columns for the downstreams, each can have one or more node inside of it.
If an upstream has an upstream we need to render an arrow in right side of the node
If a downstream has an downstream we need to render an arrow in the left side of the node.
When one of this arrows is clicked we need to show a new column, either to the right or left of the clicked node, depending if it is an upstream or a downstream, respectively.
Logic
This logic needs to be moved to the store.
Store needs to save:
Upstreams
An array with each column of upstreams to be rendered. Server sends this as an object.
We get the first upstream an object and we pushed to the the array of upstreams.
Upstreams in the array will be rendered as "expanded", meaning:
When the user clicks the arrow, we get the next upstream and push it to the beginning of the array.
Problems to solve in here:
We need to prevent that the same pipeline can't be clicked twice. Meaning we need to add {expanded: false} to each upstream when we push it to the array of upstreams.
When the user clicks in the arrow we need to toggle this prop.
The arrow can only be rendered if the upstream has an upstream.
Upstream is an object, and if it has an upstream it will be an object inside an object, and so one in a circular structure - this allows us to have just one endpoint.
Downstreams
An array with each columns of downstreams to be rendered. Server sends this as an array. We can have more than one upstream in each column.
When the user clicks in the arrow we need to get the new downstream, that is an array inside the object clicked, creating a new column. We need to push it to the end of the downstreams array.
Question: when we have 2 downstreams inside of a column, and we click both arrows, which column is rendered first?
Realtime
Because we only have one endpoint, and we will only render an upstream of an upstream after the user clicks it an we render is as expanded, (same happens for the downstream).
When we receive the polling result we won't have the pipelines expanded. So if the user had previously clicked and the user is seeing an upstream of an upstream we need to guarantee the user still sees that after polling.
To do this we need to check the old state and see which ones were expanded and iterate through the new response and transform the new response into the old structure.
I am not sure of how performant/non performant this may be. Note that we currently do a similar approach for opened rows in environments table and it does not work that well.
Conclusion:
This design does not seems doable with realtime. Meaning: keeping the quality we have now will be hard. We might have inconsistency of data causing bad UX and bugs.
Because when the user starts interacting with it we need to keep 2 states. The one we receive from the server and the one the user is creating while clicking.
This means that everytime we receive the data from the server we need to transform the received data to match the state the user created. This would not be a problem with one upstream of an upstream but it might be a problem with this feature since we have upstreams of upstreams of upstream and so on. And the same happens for the downstream.
Managing 2 states at a time is a problem. My suggestion is to have a design that allows us to keep one state. Meaning that we render exactly what the server sends us.
This can be done in 2 ways:
When we poll we send parameters in the endpoint with the expanded pipelines and the API would sent the data structure needed to render everything
The user does not interact with it and all our problems are solved
?
@iamphill@jschatz1 Do you fully understand the problem with the 2 states and 2 different data structure to manage with polling? Do you have a better suggestion?
As discussed with the team, it seems that we need more time to have another MVP for expanding pipelines, so I'm postponing this issue until we are confident how to deal with all the data.
@dimitrieh we're probably not going to iterate further on multi-projects pipelines for a while. We want to see which is the feedback from our users and then we can schedule some other improvements as well.
As soon as we're back on this topic we can schedule this issue too.
Sorry if it wasn't clear... it was a (very late) reply to your previous post where you said you are waiting for feedback before coming back to the general multi-project pipelines topic:
We want to see which is the feedback from our users and then we can schedule some other improvements as well.
@jlenny @erushton this doesn't feel like it belongs in 11.5. This potentially needs a rigorous re-look at the proposal that currently stands (which is 1 year old). It needs a look from both UX, FE, and BE.
Please don't remove from 11.5 because it potentially needs to be looked at - let's confirm that's actually the case. I spoke with @sarrahvesselov who confirmed that ~"UX ready" items should be good to go, and an hour before the kickoff meeting is really too late to be looking at this for the first time and/or go back to the drawing board. I understand where you're coming from, but let's focus on how we can deliver vs. pulling it out just in case.
@jlenny just want to reiterate that this issue apparently didn't get any discussion (UX,FE, or BE) at all before the milestone started
Discussing this item only now might potentially introduce time being a problem towards the end of the cycle should we decided to implement this issue this release.
It was discussed in the team planning meetings, but with everyone coming back from holidays and trying to finish 11.4 it was truly a rush to get everything done and I could definitely imagine that everyone would have liked to have more time for deeper looks.
Going ahead and removing this from %11.5 after further discussion with @dimitrieh. We will try to get this item back to ~"UX ready" for a future release.
This apparently got considered by FE and BE in the last planning meeting. (This is good, but this didn't get documented. @erushton can we make sure engineering discussions get documented in their respective issues?)
Thanks @jlenny for removing ~"UX ready". Let's re-evaluate this issue with all parties included.
@dimitrieh can you please drive the effort to get this back to ~"UX ready"? If we're somehow able to do that quickly we can potentially bring this important feature back in as a Stretch item, since we already have FE and BE people allocated to the engineering work. Anything I can do to support, I'm here!
@filipa can you take a look at the current proposal in the description from a FE feasibility perspective, see if we have the available data etc?
It currently tackles the defined problem in just the pipeline detail view pipeline graph
It expands the pipelines inline to be able to show its nodes. (I wonder if just showing nodes that represent a pipeline without expanding would be enough here... thoughts @jlenny? see problem definition)
It will show any prior pipelines that triggered the current pipeline or any of its upstream parents (so multiple levels will be shown in order to be able to see its true source)
Same goes for downstream (note that there can be multiple downstream pipelines triggered)
From my POV the original ~"UX ready" design is good and does solve the issue at hand. As a general principle, if we do modify I'd prefer we simplify the MVC rather than make it more complex.
Also adding ~"devops:release" label since this will help with ~"release orchestration" by enabling visualization of cross-project release pipelines. @nolith please take a look to see if this is doable by your team in %11.6 since we have descoped https://gitlab.com/gitlab-org/gitlab-ce/issues/47118.
json serialization links upstream pipeline with triggered_by and downstream pipelines with triggered (an array)
Those extra pipelines do not have stages and so it will be not possible to expand them.
But we have a path item that can be used to fetch the details from the backed.
So in order to achieve the 3 following 3 points:
this iteration will just focus on having the upstream and downstream expansion, with no collapse
this iteration will not have the blue background for current project, or other ways to highlight it
this iteration will show only the direct upstream for the original project
Frontend can fetch the pipeline json as usual, then search for triggered_by.path and fetch that json representation again expading it into the right place.
Once we implement expand/collpase we can fetch also upstream-upstream and downstream pipelines.
Reason why I think its a better idea to have this logic in frontend:
It reduce the number of queries we have to perform to serialize the whole parent tree.
Once expande/collapse is available, we will not waste resources fetching useless information from database, but fetch them only on demand
it will allow for a deeper nesteting without impacting performances in the single call.
Great feature! Could this work if the downstream was the same projet?
My use case is a multi project mono repository. I use the push trigger to determine which folder needs the CI, and use trigger to run x-times the CI on different folders (projet).
@dimitrieh I like it! My only concern is the user not realizing that there is more to see and scroll horizontally. Can these jobs be expanded on by page load or are they always expanded by user interaction? Maybe scrolling to show the expanded part on click? Maybe having a coded “scroll bar” that shows your current position regarding the whole?
@pedroms Thanks! A scrollbar should indeed be part of this and actually already is, as we don't change any behavior regarding that from what currently already is.
The expand parts should be on click as we want to give focus to the current pipeline and there can be multiple downstreams ;).
Stages and their jobs are rendered as lists with just 1 connecting line connecting them
Upstream/downstream pipelines are rendered differently from jobs
Pipeline graph space will now be full screen width
Dropdown inner action button styling adjusted (now similar to web ide)
Upstream/downstream pipelines are expandable+collapsable showing their pipeline graph inline
Inline upstream/downstream pipeline graphs are positioned along the Y-axis according to the position of their project position in the upstream/downstream lists from which they are expanded from.
Only one downstream cane be expanded at a time (if you expand another one, the initial one will be collapsed/replaced)
Expanded upstream pipelines do not show downstreams of their own which are non related to the current pipeline
@dimitrieh this looks amazing! I'm super happy about this change, it will mean a lot less CSS
@jhampton @jlenny Looking at the mockups, these are a lot of changes, I don't think this is shippable for %11.6
The main UI changes need to happen in CE. @dimitrieh This new view should be for both CE and EE correct? Only upstream/downstrem are EE, right?
We reuse some of the components we have currently in the pipeline graph in other places, so I can't just change them all, I need to check exactly which ones can be changed and which ones we'll have to create new.
However, we can start the work:
Identify which components can be adapted;
Identify which components need to created from scratch;
Since this adding upstream/downstream pipelines, it would be easier to do it in Vuex, this is currently using our old store pattern. Moving into Vuex is not hard.
My proposal is we do the same as we did for the job page refactor: While backend does the API work we do the backstage work for %11.6:
CE
Create Vuex store and merge it to master without being used (with proper testing)
Create the new components we need and merge them into master without being used (with proper testing). @dimitrieh I will need the sketch specs with all the measurements, mobile, hover/active/etc states, all the clicks, all the behaviors.
Use the Vuex store + the new UI in a new MR and merge it into master
EE
Add support for the upstream pipeline in one MR and merge it
Add support for the downstream pipelines in another MR
If we break this down this way I think frontend would be able to ship this for %11.7, would that be ok?
EDIT: Forgot about one thing, we are polling the graph, however the user can click and close/open the upstream/downstream pipelines - we need to take this in consideration when polling and not hijack the user behavior
@jhampton Also, I think we need to do the UI refactor behind a feature flag to make sure we do not break the pipeline view and only notice it once it's deployed into production.
Looking at the mockups, these are a lot of changes, I don't think this is shippable for %11.6
Awesome you are as excited as I am! Sadly that would mean this wouldn't ship the value we are after in 11.6...
While backend does the API work we do the backstage work for %11.6
Would this mean even if we have something that is in scope, we would not be able to ship as the BE has not made their changes yet? Or are those doable in the same release as FE?
For a simplified concept reusing just the UI that already exists while doing no cleanup at all. Would the following be shippable in 11.6?
Upstream/downstream pipelines are expandable+collapsable showing their pipeline graph inline
Inline upstream/downstream pipeline graphs are positioned along the Y-axis according to the position of their project position in the upstream/downstream lists from which they are expanded from.
Only one downstream can be expanded at a time (if you expand another one, the initial one will be collapsed/replaced)
Expanded upstream pipelines do not show downstreams of their own which are non related to the current pipeline
Upstream/downstream loading states temporarily replace the pipeline status icon with a spinner
Scrolling stays unaffected, except for positioning expanded/collapsed content into view.
Agreed, the design looks amazing @dimitrieh! I definitely understand @filipa's concerns (good call on the feature flag).
@jlenny / @filipa what do you think about @dimitrieh's suggestion above? If that is doable in 11.6 (from an engineering perspective) and won't require too much re-work in 11.7 it could be a good first step.
I also think @filipa's suggestion about completing the backstage work in 11.6 is worth consideration. Those changes would not necessarily be visible in 11.6, but would set us up nicely for 11.7. Just thinking out loud.
Upstream/downstream loading states temporarily replace the pipeline status icon with a spinner what do you mean?
Instead of skeleton loading a spinner shows up. this is included in the mockups.
Scrolling stays unaffected, except for positioning expanded/collapsed content into view. what do you mean?
It means the expanded content from either upstream or downstream, scrolls into view. Also, when collapsed, the content once again scrolls into view without user interaction. See the prototype for more info.
Design
Upstream
Loading:
Expanded:
Downstream
Loading:
Multiple + one is expanded
Single + expanded:
Notes
Upstream/downstream pipelines are expandable+collapsable showing their pipeline graph inline
Inline upstream/downstream pipeline graphs are positioned along the Y-axis according to the position of their project position in the upstream/downstream lists from which they are expanded from.
Only one downstream can be expanded at a time (if you expand another one, the initial one will be collapsed/replaced)
Expanded upstream pipelines do not show downstreams of their own which are non related to the current pipeline
Upstream/downstream loading states temporarily replace the pipeline status icon with a spinner
Scrolling stays unaffected, except for positioning expanded/collapsed content into view.
@nolith how do I set this up on GDK? Could you give me a mocked response either way?
Does the pipeline/{id}.json include all the data for upstream and downstream pipelines?
@filipa specs are not created for this new design as it reuses only existing ui elements.
The only new color added is #f6faff (global color used for highlighting elements) for expanded upstreams or downstreams. Can you point me towards specific information you are missing?
maybe I did not explain myself properly. @pedroms you usually provide a link to a specs file where we can inspect each element and measurements. I think @annabeldunstone also showed it in the frontend team call recently.
@dimitrieh Could we make this the default and add it to every issue that requires mockups? it's a bit hard for me to guess the margins and colors without it.
If you could please add it for this issue I'd highly appreciate it.
@dimitrieh thought we'd be rendering a downstream of a downstream. However, given that this was not clear in the issue and there were no mockups for this, I did not implement this.
There was a slight discussion on slack, where I commented this would not happen for 11.6 but it was missed.
thought we'd be rendering a downstream of a downstream. However, given that this was not clear in the issue and there were no mockups for this, I did not implement this.
There was a slight discussion on slack, where I commented this would not happen for 11.6 but it was missed.
I agree this is a case of miss-communication and time shortage. The "recursiveness" of this feature should have been detailed out extremely explicitly, which it was not. It was written down, more as a given. There is now not enough time left in order to fix it.
@filipa I have to wonder if we could have done it, regarding time, if it was known from the beginning (How much more work is it? )
Didn't this statement in the description make it clear?
In a multi-project scenario, it is very important for developers to access some basic details of related projects when viewing upstream and downstream links. Clicking on the project you're interested in, it will be expanded to a more comprehensive view that shows stages and jobs, as well as further dependencies. You can iterate on it in order to get the full view of the complete pipeline.
@nolith I agree it looks a little weird - the other option was to merge the "scoped off" part back into this issue and reopen it. Both options seemed strange in their own way. I think this is OK, it will ensure we communicate the right thing to users of this feature.
@filipa we missed the problem to solve in this issue because it wasn't clear so we can't mark it as successfully delivered. The expected functionality was listed in multiple places but it didn't quite "catch" for whatever reason. We have a retrospective item to understand what happened there and correct going forward.
I have a general question about this feature / which version it is going to be in:
It appears that this is going to be a GitLab Starter feature according to both this issue, gitlab-org/gitlab-ee#8688 and the direction page. How would this feature be used/useful without the cross-projecttrigger functionality, which is currently only available in GitLab Premium or higher? Is there a plan to move cross-project triggers into GitLab Starter?
Good catch @matthias.vandemeent. This won't be useful without actually having/doing cross-project triggers. We're always looking for features to bring back to core and ones like these are high on the list, but we don't have plans at the moment to move this one.