[Feedback] Stacked Diffs via the GitLab CLI

added Category:GitLab CLI cli devopscreate groupcode review labels

added to epic &12025

added sectiondev label

mentioned in issue #7440 (closed)

added Stacked Diffs label

set weight to 1

changed milestone to %17.3

changed milestone to %17.4

assigned to @garyh

changed milestone to %17.1

added typeignore label

added twtriaged label

added Deliverable label

mentioned in issue create-stage#13203 (closed)

Setting health status to on track as the milestone has just begun.

Issue participants are welcome to override this by setting the health status to another value.

changed health status to on track

@csb-eq @thcipriani Here's the latest demo of the stacked diff functionality we're building into the CLI. I would love any thoughts you have on seeing this now and if you think it would work in your workflows.

This issue is scheduled for completion in this milestone but is not yet in development. Changing health status to 'needs attention'.

Issue participants are welcome to override this by setting the health status to another value.

changed health status to needs attention

Hi @phikai and @garyh, i am very excited to see this in the works!

here is my experience with this flow so far.

i learned about stacked diffs a few months ago from a video Theo Brown made on Graphite and it took me a while to get it but once it clicked i adopted the process and i now appreciate its implications on diff size and review quality.

i tried it out manually in GL by chaining MRs but its a lot of work that requirs a good understanding of git and GL to avoid mistakes.

since then i did some research and discovered a few prominent open source alternatives:

Git Patch Stack (my favorite so far)
Stacked Git (runner up)
Sapling (mixed feelings)
GitButler (a different approach)

unfortunetly none of the tools above integrate with GL at any capacity.

so here is what i think:

Graphite is gaining traction and people that want to use it with GL dont have that option so they are pushed to use github instead. perhaps it can be beneficial for GL to collaborate with the Graphite team and lead the way on this front.
the stacked diffs flow is awesome, improving both the authoring and review experiences, and its becoming more and more popular. supporting it within the GL ecosystem will ensure GL keeps up with the times and will draw in that crowd that feels not having it is a deal breaker.
there are already pretty good open source tools out there. perhaps its enough to just make integrations or wrap them with the GL CLI rather than building from scratch.
GL already supports changing targets of merged MRs. whats missing is an option to auto rebase upstream MRs from a changed MR, which i believe can be achieved via the GL API without changes to GL, but i did not check this so i might be mistaken.

i would love to hear from you, keep up the great work!

Thanks for the great feedback and insights @rtzoor! We're super excited to get a first version of this available for users, so stay tuned.

The manual is incomplete without references to:

Mercurial Queue - https://hgbook.red-bean.com/read/managing-change-with-mercurial-queues.html
Chromium CL (changelists) - https://commondatastorage.googleapis.com/chrome-infra-docs/flat/depot_tools/docs/html/depot_tools_tutorial.html#_managing_dependent_cls

This issue is scheduled for completion in this milestone but is in an early development stage. Changing health status to 'at risk'.

Issue participants are welcome to override this by setting the health status to another value.

changed health status to at risk

removed milestone %17.1

removed health status at risk

@garyh question With the multiple branches part of a stack is there a way to view a similar relationship of the branches that make up the change in the GitLab UI as there is in the CLI?

I am looking at this from a reviewer perspective where I could be reviewing an individual MR but also see the relation of this one change in context of the entire stack so that I wouldn't need to launch CLI to see the list of MRs.

@garyh praise I like that the rebasing is done for you on the related branches

I can't wait to get my hands on this, thanks @garyh!

One question, does glab stack sync support flags like --assignee, --label and --reviewer? In my team it is often the case that all branches in a stack are reviewed by the same person/people, so assigning these through the CLI would meaningfully streamline the process. If it is not yet possible, I would propose it as a possible future improvement

@cfratta Very good idea! I'll add that to the list #7571

mentioned in issue #7571

changed title from Feedback issue for: Stacked Diffs via the GitLab CLI to [Feedback] Stacked Diffs via the GitLab CLI

changed the description

mentioned in issue gitlab#24528

It seems to create corrupt MRs when working with a fork. I created a repository ( https://gitlab.softwareheritage.org/swh/devel/vlorentz-test/), then forked it ( https://gitlab.softwareheritage.org/vlorentz/vlorentz-test/), then cloned the fork, and here is what I did in the CLI:

 dev@desktop5  ~/vlorentz-test   main ~/glab stack create
? New stack title? test stack
! warning: invalid characters have been replaced with dashes: test-stack
New stack created with title "test-stack".

 dev@desktop5  ~/vlorentz-test   main  echo foo > test-file
 dev@desktop5  ~/vlorentz-test   main  git add test-file
 dev@desktop5  ~/vlorentz-test   main ✚  ~/glab stack save
? How would you describe this change? add the test file
• test-stack: Saved with message: "add the test file".

 dev@desktop5  ~/vlorentz-test   dev-test-stack-5b337685  echo bar > test-file
 dev@desktop5  ~/vlorentz-test   dev-test-stack-5b337685 ●  ~/glab stack save   
? How would you describe this change? change the test file
• test-stack: Saved with message: "change the test file".

 dev@desktop5  ~/vlorentz-test   dev-test-stack-74f87791  ~/glab stack sync
| Syncingcould not run sync: error determining base repo: none of the git remotes configured for this repository point to a known GitLab host. Please use `glab auth login` to authenticate and configure a new host for glab.

Configured remotes: gitlab.softwareheritage.org. are you in a git repository?

 dev@desktop5  ~/vlorentz-test   dev-test-stack-74f87791  ~/glab auth login                                                        
? What GitLab instance do you want to log into? GitLab Self-hosted Instance
? GitLab hostname: gitlab.softwareheritage.org
? API hostname: gitlab.softwareheritage.org
- Logging into gitlab.softwareheritage.org
? How would you like to sign in? Token

Tip: you can generate a Personal Access Token here https://gitlab.softwareheritage.org/-/profile/personal_access_tokens?scopes=api,write_repository
The minimum required scopes are 'api' and 'write_repository'.
? Paste your authentication token: **************************
? Choose default git protocol SSH
? Choose host API protocol HTTPS
- glab config set -h gitlab.softwareheritage.org git_protocol ssh
✓ Configured git protocol
- glab config set -h gitlab.softwareheritage.org api_protocol https
✓ Configured API protocol
✓ Logged in as vlorentz
 dev@desktop5  ~/vlorentz-test   dev-test-stack-74f87791  ~/glab stack sync                                                        
? Which should be the base repository (used for e.g. querying issues) for this directory? swh/devel/vlorentz-test

• dev-test-stack-5b337685 needs a merge request. Creating it now.

/ Syncing
• Merge request created!

!1 add the test file (dev-test-stack-5b337685)
 https://gitlab.softwareheritage.org/swh/devel/vlorentz-test/-/merge_requests/1


• dev-test-stack-74f87791 needs a merge request. Creating it now.

/ Syncing
• Merge request created!

!2 change the test file (dev-test-stack-74f87791)
 https://gitlab.softwareheritage.org/swh/devel/vlorentz-test/-/merge_requests/2


• Sync finished!

This creates the MRs on the upstream repository (https://gitlab.softwareheritage.org/swh/devel/vlorentz-test/), but pushes the branches on the fork ( https://gitlab.softwareheritage.org/vlorentz/vlorentz-test/) Opening the first MR shows a grey "Merge request contains no changes" error (because the MR's branch is assumed to be in the upstream, but it's not):

opening the second MR shows a red "Can't show this merge request because the source branch dev-test-stack-74f87791 is missing from project swh/devel/vlorentz-test. Close this merge request or update the source branch." error (because neither the MR's branch nor the base branch are in the upstream). I also can't close the second MR:

Excellent feature @garyh!

I wish it could support creating stacks from existing branches in the future: glab stack infer "My new awesome stack from the working branch". That way I don't have to strictly follow the 'stacking process' and just work as usual in one branch. After I've finished the work I can split it using stacks and enjoy the awesome features it brings!

Also, for future iterations I think it makes sense to group stack branches using a following pattern: stack-name/stack-iteration-hash. If we ever decide to support stacking in our UI this can potentially help with picking up all the necessary branches.

@slashmanov Great ideas! @phikai WDYT about inferring?

Also, for future iterations I think it makes sense to group stack branches using a following pattern: stack-name/stack-iteration-hash. If we ever decide to support stacking in our UI this can potentially help with picking up all the necessary branches.

I like this too. I'll just have to add a note that the next version will break your current stacks if you're in the middle of one

@garyh I think it's an interesting idea, something that could be worth looking at.

What I don't know enough about is if most people who commonly use stacked diffs intuitively work that way and so it's not about breaking down something that's done it's about building that way from the start. Breaking down a large branch seems like it could be something... but I'd probably want to make sure we have the other primitives right first.

i could not find references for this use case. however, i can imagine how it can be handy to be able to infer stacked diffs from a branch. the main advantage i see is it will enable authors to apply the paradigm on existing work, allowing them to benefit from it earlier.

I opened #7576 to discuss the branch naming piece.

Just another idea: if we ever decide to support stacked diffs in our UI we can infer directly from the UI. Or even more simply: allow to create multiple linked MRs without even going into the stacked diffs territory.

A use case I could see for something like glab stack infer would be to allow the team to mostly still use standard git commits in their day-to-day, but use glab stack to handle cases were I want to build off (stack off) of someone else's work. It is a slightly different use case, but also quite common on teams for someone to need just the first 1 or 2 of someone's diff.

gitGraph
  commit
  branch "shared-needed-change"
  commit
  branch "person-a-change"
  checkout "shared-needed-change"
  branch "person-b-change"
  commit
  checkout "person-a-change"
  commit

I know stacks diverging is it's own lengthy conversation, but I'd like it if glab stack infer or something else could just take on and accept that shared-needed-change is the base of a new stack that I want to build on, despite that diff being owned by someone else. Sync could still rebase my work on top of their's no problem, I just would be treating the root/base diff in my stack as read only.

Thinking about it more, it may be possible that glab stack create/glab stack new (one in the description above, other in the video) could be run on top of an existing branch and understand that it is someone else's diff and know to build on top of it separately. No sure how it is implemented based on what I've read so far.

@nathan.cain It will currently default to main as the target branch, but I think it should default to whatever branch you left off in (which could lead to nesting stacks and maybe needing a stack switch command...) and basing it from that.

I think that would be be a nice implementation. Enabling stacking diffs across individuals is a feature I've hand multiple teams coming to me (DevOps Engineer) asking for a quality solution for. The idea of being able to stack on top of any diff would handle this, but would need a good UX for nav or switch as it would allow for branching of stacks.

This was my first question when attempting to use it. I have an existing branch that I would like to specify as the first diff in a new stack.

This seems like a normal use case. Maybe I don't always want to use stacked diffs, and would prefer to use normal git usually, but only switch to a stack when I actually have multiple branches to rebase on top of each other. However I don't always know in advance when I'm going to need it.

This was also my first misstep. Without re-watching the video, I made a commit, and tried glab stack save and got no changes to save; however, there was a diff in git log @{u}..HEAD.

I note that this is different from the behavior of, e.g., sapling which requires sl add and sl commit.

Question: I created a stack with:

  main  ~/glab stack create
? New stack title? second test stack
! warning: invalid characters have been replaced with dashes: second-test-stack
New stack created with title "second-test-stack".
  main  echo foo > test-file
  main  git add test-file
  main ✚  ~/glab stack save

? How would you describe this change? add the test file
• second-test-stack: Saved with message: "add the test file".
  dev-second-test-stack-c74f60a1  echo bar > test-file

  dev-second-test-stack-c74f60a1 ●  ~/glab stack save  
? How would you describe this change? change the test file
• second-test-stack: Saved with message: "change the test file".

  dev-second-test-stack-196e4ef7  ~/glab stack sync
\ Syncing
• dev-second-test-stack-c74f60a1 needs a merge request. Creating it now.

\ Syncing
• Merge request created!

!3 add the test file (dev-second-test-stack-c74f60a1)
 https://gitlab.softwareheritage.org/swh/devel/vlorentz-test/-/merge_requests/3


• dev-second-test-stack-196e4ef7 needs a merge request. Creating it now.

\ Syncing
• Merge request created!

!4 change the test file (dev-second-test-stack-196e4ef7)
 https://gitlab.softwareheritage.org/swh/devel/vlorentz-test/-/merge_requests/4


• Sync finished!

Then on the web UI, made a edit suggestion on the first MR, to change the file, and accepted it. Now, the second MR is (as expected) in conflict. glab stack sync does not seem to be aware changes were made on the web UI, so how do I get out of this situation? manually git pull from the right branch?

mentioned in issue gitlab#455339

Quick feedback on naming:

move is used by at least one other git alternative client to mean cherry-picking stack of commits between branches, it may cause needless confusion (especially since both glab and git-branchless may be good companions in the future?) https://github.com/arxanas/git-branchless/wiki/Command:-git-move

@plib That's a good point- I wonder if maybe just navigate (or nav for alias) is a better choice here.

I like nav - more clear that you are moving through the stack rather than moving the stack itself.

mentioned in issue #7574 (closed)

mentioned in issue #7575 (closed)

@garyh I love this! It makes for creating chains of MRs so much easier, while ensuring you can keep MRs small.

Some questions:

What happens if the first MR in the stack is merged? Or how should these MRs merge? Do you just need to run a sync to update the target of the next one in the stack?
1. In that vein, what happens if one of the other MRs in the stack is merged first, instead?
2. Would it make sense to add MR dependencies to the 'child' MRs of the stack?

What happens if the first MR in the stack is merged? Or how should these MRs merge? Do you just need to run a sync to update the target of the next one in the stack?

It should automatically re-target correctly! So if you have stack diff main <- A <- B <- C and A gets merged, the new order will be main <- B <- C

In that vein, what happens if one of the other MRs in the stack is merged first, instead?

Unfortunately, that's where it starts to become the wild west Right now, if you have main <- A <- B <- C and B merges... you'd have main <- A+B <- C. That would suddenly add a lot more code for the reviewer of A. It's easy enough to say "just ignore the merge commit from B" to the reviewer but that's probably not an ideal solution

But I think this is a limitation of stacked diffs implemented through the git model, unfortunately. I haven't really come across a great solution to this in any of the other tools (without something like GitButler that takes over the entire branch workflow)

Would it make sense to add MR dependencies to the 'child' MRs of the stack?

Absolutely! I have this planned and I think that would mitigate it to an extent, but that needs to wait for actual API to be made

@garyh explained it well, just adding my 2c.

TLDR: i think its a specific use case of the stack splitting story.

if diff B does not depend on other diffs down the stack and no other diff up the stack depends on diff B that means diff B is independent and it can be merged directly to the stack target.

i think that will have to be up to the author to tell if its possible, which can possibly break stuff. its hard to detect if code from one diff depends on code from another diff. that is contextual information git is not aware of.

suppose you still want to do this, it could be done by creating a new stack and moving the diff from the original stack to the new one, this will require the upstream diffs to be rebased onto the previous target of the extracted diff, which is expected in this flow and should be handled by the cli.

from the perspective of the gitlab application, the developer could choose to change the merge target of that diff to the stack target, effectively extracting the diff from the stack, which will require similar maneuvers.

@rtzoor That's a good point! Maybe there can be some options in the save command to set the branch to the original target, if that's the case?

I've also considered having an option for an intermediate branch that everything gets merged to (and then that gets merged in a giant MR, but it's already been reviewed), but then there's the problem of displaying just the diffs of that branch...

There are some workflows at my company which would benefit from an intermediate integration branch due to some projects having requirements that only fully complete features/fixes are merged in. As long as the stack can be based on other branches/diffs, this should be something that would work without extra effort.

I think this bolsters my conclusion that this workflow will never work well without server-side support (or modifications of the UI to prevent user error).

There is a big risk of confusion from this workflow. Consider main <- A <- B:

Someone could merge B into A, negating the purpose of stacking.
Someone could merge A, something could go wrong with retargeting B then when someone clicks merge on B it shows as "merged" but the work is somewhere in the void.

Example 2 happened to us via the following:

hit merge on A
hit merge on B 8 seconds later

Both A and B showed as merged in the UI, but B was not present in the mainline branch. These were from a stack on our internal gitlab instance.

See the event log:

(in the output below A is mr91 and B is mr92)

irb(main):009:0> mr91.events
=> [...]
  created_at: Wed, 29 May 2024 22:58:08.411732000 UTC +00:00,
  updated_at: Wed, 29 May 2024 22:58:08.411732000 UTC +00:00,
  action: "merged",
  target_type: "MergeRequest",
  target_id: 10833>]
irb(main):010:0> mr92.events
=> [...]
  created_at: Wed, 29 May 2024 22:58:16.844530000 UTC +00:00,
  updated_at: Wed, 29 May 2024 22:58:16.844530000 UTC +00:00,
  action: "merged",
  target_type: "MergeRequest",
  target_id: 10835>

A merged, then B merged into A instead of mainline. I think this is a result of protected branch rules. That is, we used a wild-card that prevented A's branch from deletion following the merge, so B did not retarget main.

I could also believe that this has to do with the async nature of retargetting.

I recreated this situation on gitlab.com:

To clarify, you say the following in the description:

Move around to different stacks:

glab stack prev

glab stack next

glab stack first

glab stack last

glab stack move (uses a fuzzy finder)

Is this actually moving to different stacks or to different diffs within a single stack?

Good catch! It's moving to different diffs- I've updated the description

Regarding the glab stack save command - it seems odd to me that "saving" my current diff would result in a new diff and mr, the semantics of it could get confusing (and already are to me). I should need to be explicit about wanting to create a new diff on top of the stack (see Graphite gt create). Simply "saving" my current stack should imo result in an update to the current diff, like glab stack amend does.

I'd like to see either reusing the create command as Graphite does or using a new keyword that is clearly explicit about creating a new change (maybe glab stack layer?) - I favor the latter as it avoids the context overload on the same keyword. Stacked diffs are great, but it wouldn't be uncommon for me to make a few changes before wanting to label something as a diff and push it up.

Side note: I'd want either a flag like glab stack sync --pull or new keyword like glab stack pull to only pull and rebase changes as need, but not create a new mr for my currently under edit diff. This goes back to wanting to be explicit about my actions, in this case being explicit about when I submit work for review.

changed the description

mentioned in issue #7581

mentioned in issue #7582

Talked with some of the teams I support and they came up with a few more notes:

Stack Retargeting would be a very nice feature in a multi-dev environment where one dev realizes that they need another dev's work. This means that if dev one's stack could be retargeted to point at the specific diff of the other dev's stack that they need and for the tool to rebase each diff in dev one's stack accordingly. Whenever the shared diff is merged, it goes back to being two unrelated stacks.

stateDiagram-v2
  main_1: main
  main_2: main
  main_3: main
  main_4: main

  A_1: A
  A_2: A

  B_1: B
  B_2: B
  B_3: B
  B_4: B

  C_1: C
  C_2: C
  C_3: C

  D_1: D
  D_2: D
  D_3: D
  D_4: D

  E_1: E
  E_2: E
  E_3: E
  E_4: E

  Stack1_1: Dev-2-Stack
  Stack1_2: Dev-2-Stack
  Stack1_3: Dev-2-Stack
  Stack1_4: Dev-2-Stack

  Stack2_1: Dev-1-Stack
  Stack2_2: Dev-1-Stack
  Stack2_3: Dev-1-Stack
  Stack2_4: Dev-1-Stack
  
  state Initial {

    state Stack2_1 {
      B_1 --> main_1
      E_1 --> B_1
    }
    state Stack1_1 {
      A_1 --> main_1
      C_1 --> A_1
      D_1 --> C_1
    }
    main_1

  }

  state Retarget-B-To-C {
  
    state Stack2_2 {
      B_2 --> C_2
      E_2 --> B_2
    }
    state Stack1_2 {
      A_2 --> main_2
      C_2 --> A_2
      D_2 --> C_2
    }
    main_2

  }

  state Merge-A {
  
    state Stack2_3 {
      B_3 --> C_3
      E_3 --> B_3
    }
    state Stack1_3 {
      C_3 --> main_3
      D_3 --> C_3
    }
    main_3

  }

  state Merge-C {

    state Stack2_4 {
      B_4 --> main_4
      E_4 --> B_4
    }
    state Stack1_4 {
      D_4 --> main_4
    }
    main_4

  }

Allow for glab stack cast to cast a branch with multiple commits into a stack with multiple diffs and MRs.
```
glab stack cast <dev-branch>...<target-branch>
```
This will turn each commit in the dev branch above the target branch (will fail if dev is not based on the top of target) into a stacked diff with merge requests created for each one.

A way to delete or poison a stack - or the later parts of a stack.

For a variety of reasons, a team might make need to invalidate a set of work. If there are large/many/nested/etc stacks of diffs, that is a very manual process. A glab stack poison command could be used to delete branches and MRs for all later work based on the current diff.

two developers collaborating on stacks is a powerful feature of other stacking systems, but it seems impossible with this system, currently.

There seems to be a lot of local state saved in .git/stacked/* but none of this is propagated to the remote. I note the glab stack does create .git/refs/stacked/*; however, those folders are empty.

I think this has two consequences:

you lose your stack if you lose your local copy
developers are unable to collaborate

Please correct me if I've missed this.

I think you're both right about a future where stack collaboration makes sense. One of the things we've done is give us that option by building in some things locally, but not committing to anything on the GitLab instance side at this time. The purpose was really so we could be more confident in the workflow and base features/functionality before we incur the overhead of needing to model things on the main GitLab application where it's much harder to break and move things.

When testing this, I immediately started looking for a stack delete command.

I had created a test commit diff on my stack, then created another.

Now I have these two commits, but no way to get rid of one or both of them.

I tried to re-checkout my original branch, and then do stack new again, to start over with a new "stack" of diffs, but then stack move still saw my two original commits.

I guess I'm trying to treat this like an interactive rebase, where I can delete or move around individual commits/diffs as needed. But maybe that's the wrong way to think of it?

The main functionality I would like to get out of this tool (or one like it) is automation of rebase --onto behavior.

I like using standard git, rebase, and interactive rebase, because I'm familiar with them, and my IDE integrates well with this workflow (i.e. I can use GUI conflict resolution).

The only painful part is when I have stacked MRs, and I have to do a manual git rebase <previous upstream SHA> --onto <upstream branch> for each MR in the stack. Even that isn't bad, as long as the MR is kept squashed to a single commit, I can substitute head^ for <previous upstream SHA>, but it can still get tedious/error-prone if there's 3+ MRs in the stack.

[Feedback] Stacked Diffs via the GitLab CLI

Demo

Usage

Child items ...

Activity