In the last NPS survey, we got a significant amount of negative feedback on Merge Request performance. We don't get that same feedback internally, and we'd like to know why.
What hypotheses and/or assumptions do you have?
Our hypothesis is that our customers use GitLab MRs differently than we do, and that can impact the performance they experience. For example, internally, our goal is to submit the smallest possible MR. Also, we release every month, which limits the amount of comments that are generally associated with an MR.
What questions are you trying to answer?
Do self-managed customers have a different experience with MR performance than those on GitLab.com?
For companies with poor MR performance, how do they structure their work? Continuous Deployment? Agile (frequent deploys)? Waterfall? (deploys after months of work)?
For companies with poor MR performance, how many comments on average do their MRs have?
How much of the perceived poor MR performance is due to usability problems like interaction patterns?
Core questions
Additional questions
Relevant comments from survey
"Its pretty good overall. A bit slow sometimes.
Everything is excellent but the website's response time - if I navigate through files or larger MRs waiting times become a problem
...Also, we've been having some rather serious performance issues, uptime issues, and problems with Gitlab rendering large merge requests, which I never saw on Github.
Bad performance. Especially when dealing with large merge requests or files
GitLab is a great product. The only thing that really bothers me on a daily basis is performance. At least once a day I experience issues with the platform where MRs pages show errors, pipelines pages take a while to update, etc.
"Performance. Otherwise a great tool. Just slower than i would like given the complexity.
show MR page is slow to load, searching through MRs requires a you to click, and then press enter - that's slow, searching through MRs is slow in general, would be nice to be able to click the Pipeline status notifications and get directly into that notification's associated show MR page
One thing that is very very annoying is if the git diff is wrong. This happens especially for long-running MRs where master is merged in a couple of times. But if we cant rely on the git diff, what's the point of an MR?"
nope, at times the "changes" in MR is taking too much time to load in the UI (when the changes are more than 60). Just a feed back.
What persona, persona segment, or customer type experiences the problem most acutely?
Developers
What business decisions will be made based on this information?
We'll work with PM to prioritize MR performance improvements based on the results.
@ebrinkman - Please add any questions that you'd like this research to cover. Can you also help prioritize this appropriately based on the known requests for Create research?
Thanks for the issue @clenneville - to clarify are you seeking to understand why there's a discrepancy between external and internal feedback or more generalized research on MR performance from those customers who indicated poor performance? The latter seems necessary, the former valuable but not as important.
@ebrinkman - Good question. Both but with an emphasis on:
more generalized research on MR performance from those customers who indicated poor performance
I think part of what @clefelhocz1 was trying to understand is if part of the problem is a gap between how we (GitLab) view the world and how our customers actually work.
@clenneville FYI, progressive diffs only shipped in %12.8 (gitlab#38106 (closed)), and the other major improvement shipped in %12.9 (gitlab#33183 (closed)). Since many customers follow 1 or 2 releases behind, it is quite likely that many customers have yet to experienced the improvements. Since the NPS survey was done about a month ago, it is almost guaranteed that most respondents were not yet running 12.9 (unless they use GitLab.com)
Internally the issue feels like it was fixed a while back likely because we were toggling the feature flags on and off on the gitlab-org group for months finding and fixing bugs, before setting it default on. Because the merge request is critical to customer productivity we didn't want ship numerous regressions that 99% of users would then experience.
I don't want to discourage testing, but that might help shape your hypothesis, and influence optimal timing of research.
@clefelhocz1 - What do you think appropriate timing for this would be? Do you want to wait and see if we get the same feedback in our next NPS? (@fseifoddini for awareness)
@ebrinkman I think the generalized research is what we are more interested in. It should yield whether our hypothesis is correct as an aside. @jramsay it's great to hear that we have some improvements on the way. I'm not convinced it's the only problem in this area. I don't think we have the use-cases nailed as well as we think we do. If the NPS survey is imminent (May) then I'm okay waiting. That being said, any later and we can't realistically make progress on this as an OKR initiative. We need to tighten the cycle of these feedback loops and get to causal information.
Do we have sufficient usage data to establish typical "bounds" for the things you mention in the description like commit size, MR open duration, number of commits, etc? It'd be great to establish 25th/75th percentile values for these things to help understand what "typical" usage looks like so we don't spend too much time focusing on outlier usage. This is something the Data team could likely help with.
@clenneville working on it! I need to finish up the first version of the JTBD page(s), which should happen today and put in a MR for it. Then I can start to dive into this.
At first glance, I suspect that this MR research will be several different efforts done over the course of this quarter.
Gitlab has been pretty great so far, but the code review tooling is so poor I think I need to look elsewhere. The pull request pages take forever to load, they can't handle pull requests with over 800 lines well at all. It will collapse files with over 100-200 lines of changes for seemingly no reason (hiding the comments within them). You can't ignore whitespace in diffs, and the comments disappear from the diff if they push new commits, you need to look at them in the overview again. https://news.ycombinator.com/item?id=23164269
OH MY GOD this so much. This one specific anti-feature has no doubt caused countless bugs to be missed during code review. I can quote the issue thread by memory at this point i've revisited it so many times the last year(s?). For goodness sake, if the trade off is between 'seeing all of the changes during the review' vs 'having the page load quickly', the only people who choose 'quickly' are those who haven't been burned by human error missing a collapsed diff. Gitlab made the wrong trade off here in good/fast/cheap triangle. https://news.ycombinator.com/item?id=23167728
Yeah, this is definitely one of GitLab's weakest points at the moment and seriously impacts day-to-day work here as well. Simple features like ticking off files you've already reviewed are missing and even mid-sized MRs are a pain to review even on a beefy machine. https://news.ycombinator.com/item?id=23164649
I can relate. We use GitLab at our company. I still think the merge request UI is great. Its main problems are its slowness and the fact that files with a long diff are collapsed. As the full merge request is quite slow, I will usually open the diff of each commit one by one, then put review comments as a "per commit" basis. https://news.ycombinator.com/item?id=23164564
Performance: As @jramsay pointed out in #836 (comment 329534803), we shipped big performance improvements quite recently, in %12.8 and %12.9, so there's a chance that these users haven't experienced them.
@jramsay I'd love to chat with you more about the MR improvements that have been made and the use cases/user stories that were used to focus those improvements. :)
@mvanremmerden is there a place where I can see a list of the issues/UI improvements the UX team is working on? I want to make sure I focus on things we haven't discovered related to MRs when I conduct my interviews. :)
Also if there's anything that the designers need more info on, let me know! I can dig into that with my interviews as well.
@danielgruesso I saw here that they are moving to a monorepo approach for the www-gitlab-com repo. Do you think this will have any impact on how we, internally, manage MRs? Do you think they will continue to make small changes or will our changes become larger due to the move to the monorepo? or might it not affect MR usage at all?
@loriewhitaker seems the above link is broken, but, overall, I'd say the website is somewhat already monorepo. Don't think it will have impact in how we manage MRs currently.
Recruiting update: Thanks to @rdouglas-gitlab's wonderful work, we've got five people scheduled! I'll be adding them to the UX Research calendar - so please attend if you'd like to. Sessions will be recorded.
@loriewhitaker as chatted, here's a list of references where you can look into the comments of users that are frustrated by how GitLab handles large MRs. Some people mention the slowness, others mention the visibility behavior.
@pedroms thank you sooo much! I just posted a link to a shorter screener on those (except for the twitter one since i don't have a twitter account that's public).
We are having some difficulties with recruiting people who dislike the MR experience and no-shows. So far both no shows are people who disliked the MR experience! I've spoken with those who are happy with the experience, but now I really need to focus on those who are unsatisfied. I've also posted in the recruiting issue.
@clenneville I think they just picked the wrong time. Not sure why they did, as Calendly should show their calendar in the 24 hr clock.
We have yet to send this out to First Look - I've asked @rdouglas-gitlab and @evhoffmann if they could, but I wasn't sure where they were with the Qualtrics issues with sending emails out (it's been severely reducing our email output and I know they've been working to resolve it).
I've asked @rdouglas-gitlab and @evhoffmann if they could, but I wasn't sure where they were with the Qualtrics issues with sending emails out
Hello! I will indeed get an email out today, but we're also asking the TAMs and social team for help. People in First Look tend to be super fans, so we are unlikely to get enough people through there. FYI the email issues have been resolved as well
@evhoffmann I have also just pinged 5 people on Twitter that had previously expressed dissatisfaction with merge requests and that I then had contact with. Let's hope at least one or two of them sign up
Ok, TAMs have identified at least one customer where they've had issues with MRs, and are sending the screener. I've also asked community advocates to send the screener in response to any relatively recent negative mentions about MRs.
@loriewhitaker FYI in case it may help you for the analysis of this research, today I invested some time in reviewing all of the major issues around reviewing (not merging) large MRs and compiled a prioritized list of problems-possible solutions: &1417 (comment 384403074)
Also FYI for @mvanremmerden in case you stumble upon some less-than-ideal MR performance issues
@pedroms I linked all of the actionable insights to this research issue. The informative insights, along with the actionable ones, can all be found in the Dovetail project.
@danielgruesso@pedroms@mvanremmerden
TL;DR - the majority, if not all, of the insights I uncovered by interviewing 21 customers are the ones we've been aware of. There was no lurking large issue we hadn't uncovered. We just need to address the items we know about to increase satisfaction and efficiency of MRs.
I'm happy to discuss what I learned with anyone, but like I said, I don't think there's anything there that no one had thought of. I'm going to close this issue.
@loriewhitaker this is great, thank you for doing this urgent and valuable research! I really appreciate it and know how challenging it was (for many different reasons)
I'll look into your findings this week, as we'll have our Source Code group prioritization session on Thursday. I'll keep you posted