CI/CD & Security: Make features easier to test in GDK

It's very hard to test almost any feature in CI/CD in GDK. This leads to bugs and poor UX

It's also very hard to configure certain features locally without having to bother a member of CI/CD team. Often I end up mocking data on the frontend to make something work.

Here's a list of features that are not possible to test locally with GDK without extra set up:

1. Running a pipeline - pipelines are always stuck
2. Adding an existing GKE cluster - part of the data is not accessible (token)
3. Adding a new GKE cluster without an hack in the config file
4. Registry table
5. Security reports - all of them, both in Pipelines page and in MR widget
6. Linked pipelines
7. Deploy boards
8. A pipeline with all the empty states for job details page

(Some of the above features, I've done FE code without ever having them run locally. I always have to mock everything on the FE, making it impossible to test an entire flow. This is true for numbers: 4, 5, 6, 7 and 8)

@markpundsack @ayufan Fixing this requires dev time, can we schedule time to fix this in order to make development & review process easier?

I feel that if we had this a lot of bugs could be avoided and a lot more people could start working on CI/CD faster

---

EDIT:

Create a project in GDK with:

• a registry table
• deploy boards - EE
• linked pipelines - EE (multi project pipelines)
• all security reports - EE
• a running pipeline
• a merge request with a running pipeline
• a merge request with a successfull pipeline
• a merge request with a failed pipeline
• a project with all empty states of a job - one mocked pipeline with a job in each state: :success, :failed, :canceled, :skipped, :running, :manual, :created, :pending
• a project with a GKE cluster set up
• a working runner

ccing people that have had some trouble with this in the last weeks too: @annabeldunstone @andr3 @leipert @kushalpandya

feel free to chime in on this :)

@filipa Thanks a lot for opening this up!

Proposal

I have this simpler idea that I believe we can implement without touching how GDK works;

• Set up an independent ExpressJS server running on a certain port; let's call it CiMan.
• Have a development-only config on GDK instance that would redirect API calls to a certain pattern URL to corresponding URL in CiMan.
• http://127.0.0.1:3000/foo/bar/merge_requests/123.json -> http://127.0.0.1:9000/mock_mr/123/serilized.json
• Testing such mocking should then only involve creating named directories containing JSON files with mock data within CiMan on paths that loosely match with API paths (so that developers are spared of doing any configuration).

Example

I did something similar (without using express) in this MR where I had to add License Report information on MR widget (which acts similar to codeclimate) https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/5521, but obviously it could be simplified if we have a dedicated server accepting mock calls.

also ccing @gonzoyumo and @plafoucriere regarding the security reports feature

Thanks Filipa! We've definitely felt the same pain. I've often struggled to setup the local env in ways that allowed pipelines to actually run properly. Wether because the runner is no longer registered or because the I registered it wrong (or in another GDK).

I agree this can be hard to setup :( e.g. for security products when having to deal with the container registry (which we do a lot), docker doesn't play nice without having SSL configured. Trying to deal with it made me loose a lot of time and I ended up using fake reports with json files in the repo

This also has the bad side effect to make people relying more on gitlab.com's CI/CD, increasing its load... and it's sometime really slow making it painfull on a day to day workflow.

The mocking approach can be a good solution for prototyping Frontend but it could also prevent you to catch "real situation" issues. Improving the documentation and maybe teaching people could help having a working gdk with real use cases.

Thanks for taking the time to raise this @filipa I'm definitely in favor of allowing dev time on this

I'd love to see this fixed! Testing even small CI features can require a lot of setup, and after struggling with it for a while, I usually have to ask for help.

@gonzoyumo I also ran into issues setting up the container registry locally but now we have gitlab-development-kit!306 (merged) (merged in October) to make things a simple config change -> gdk reconfigure.

---

I created Mock CI to simulate an external CI service but those behave differently to GitLab CI in a few places. cc @kushalpandya as it is a similar concept to what you mentioned for GitLab CI or more general FE mocking.

https://gitlab.com/gitlab-org/apilab is also a thing but it hasn't proved very useful. Perhaps it could be repurposed(perhaps project name only) for the API middleware @kushalpandya mentioned.

---

For pipelines, we seed some data see /gitlab-org/gitlab-ci/pipelines but these aren't very varied to cover all states that we need to test. We could also add some copy-pasta Rails console commands to the docs in order to adjust existing or create pipelines (status and output).

@annabeldunstone @filipa @MadLittleMods Can you cover what do you need? We will allocate the backend resources to help with that :)

@gonzoyumo Can someone help from ~"Security Products" too?

changed milestone to %11.1

added Deliverable label

@ayufan @gonzoyumo we need, on GDK:

CI/CD

1. Pipelines that run, or clear instructions how to have pipelines running locally
2. At least one mocked pipeline with a job in each state: :success, :failed, :canceled, :skipped, :running, :manual, :created, :pending
3. A project with a registry table
4. A project with a cluster
5. Mock data to add an existant GKE cluster
6. A project with environments & deploy boards - EE
7. A project with linked pipelines - EE

Security reports

1. A project with all the security reports in both the MR widget and the pipeline view

I think @filipa covered everything! Thanks @ayufan

I think one thing is missing from @filipa's List:

1. Multi-Project Pipeline Graph: This could be mocked as well.

@filipa @ayufan @gonzoyumo you can find a simple way to create all the security reports in https://gitlab.com/gitlab-examples/security/security-reports/. Feel free to improve or to take a different direction if needed.

Mocking the security report is fine when working downstream (and I use it a lot, thanks for sharing this tip) but when we want to test tools integration, environment or job configuration we need a fully working setup. @MadLittleMods configuring the registry is not the hard part, this is made really easy in gdk (thanks to those who worked on that!) but the issue is that a lot of SP tools need Docker In Docker which makes things a lot harder to run locally without having SSL setup. Though this issue may be limited to the SP team ^^

@gonzoyumo @plafoucriere Can you consider allocating the backend developer and creating separate issue for ~"Security Products"?

assigned to @matteeyah

@matteeyah This is small issue that you could work on :)

changed weight to 4

added backend label

@ayufan Unfortunately, we don't have the capacity to handle that on our side in this iteration. We can keep the ~"Security Products" for later, but I can't allocate any resource on this in the near future. Thanks for your comprehension.

@filipa Can you edit the issue description so it lists the concrete points this issue aims to address? (I think they're contained in https://gitlab.com/gitlab-org/gitlab-ce/issues/46520#note_74168732 but I'm not 100% on it)

I'm currently running into issues updating the MR widget design. It would be great if there was a project with MRs in various states as well, so it would be possible to easily update the widget

changed the description

@matteeyah updated!

@annabeldunstone also added that to the list!

mentioned in issue #49201 (closed)

changed milestone to %Backlog

Should we move this issue to the GDK repo?

I also think it's a bit to wide in scope. We should piece it out into smaller, more focused issues. Possibly (in what I think the order of importance is):

1. Adding a runner to the GDK
2. A merge request with a

• running pipeline
• successful pipeline
• failed pipeline

3. A project with a cluster set-up
4. A project with all empty states of a job
5. A running pipeline (is this possible without a runner?)

This'll make the improvements bite-sized and much easier (more likely) to get picked up even without being formally scheduled.

Then we could look into figuring out the EE specific ones:

• Deploy boards - EE
• Linked pipelines - EE (multi project pipelines)
• All security reports - EE

WDYT @jlenny @bikebilly @ayufan @erushton?

unassigned @matteeyah

Agreed it should be broken into smaller issues. It does make sense to manage the runner through the GDK but it's also not very difficult to get it working following the documentation. From a product perspective I'm not really interested in prioritizing this right now.

From a product perspective I'm not really interested in prioritizing this right now.

@jlenny The amount of time it takes to set up all the CI features so UX and Frontend can test or work on new components is far too high. It's an unbelievable productivity killer, and as @filipa mentioned in the description, various states get lost or overlooked, leading to bugs and poor UX.

I'm all for splitting this into separate issues, but please prioritize this. It will make things so much easier for developers and designers, and GitLab CI will be much more polished as a result.

@jlenny This is blocking development. I understand that from a product perspective this may mean nothing, but if we don't have it, we can't do new features without bugs.

Think of this as an investment. We could ship a lot more new features if this was done.

@erushton normally I would expect that ongoing work to manage/improve the development workflow would be an accounted-for part of engineering capacity, rather than something we need to specially label as a "product capability" in order to consider doing, though I admit I'm new here and maybe missing something. Is this really something that should flow through product management?

added backstage [DEPRECATED] label

@jlenny I don't think this should flow through product management, but similarly that's coming from opinion and not necessarily established ways of working here. That said, seeing as we're both in agreement let's run with it.

@matteeyah I think you're right that moving this issue to the GDK project makes sense, can you go ahead and do that and then break it down into the more actionable, bite-size, issues you described. As for actually prioritizing those smaller issues, let's see what the actual stake-holders think:

@filipa and @annabeldunstone, does Matija's list make sense, or is there something that you'd like to see ahead of another piece?

I will work on getting some prioritization around our "backstage" work. However, given the challenges we've had recently with setting up our front-end developers for success I think this, or some of it's sub-issues once Matija breaks them down, should be near the top of the priority list.

added Quality Qualitytest-infrastructure labels

Thanks for raising this @filipa

I understand that from a product perspective this may mean nothing, but if we don't have it, we can't do new features without bugs. Think of this as an investment. We could ship a lot more new features if this was done.

I think the term we are looking for here is Testability :)

In addition of having such testdata only in the GDK, I think its also useful to have the same test datamodel on staging and review apps as well. So instead of seeding the DB like we always to can we use public APIs to setup such configurations? This way the provisioning can be automated and is repeatable in many environments.

can you go ahead and do that and then break it down into the more actionable, bite-size, issues you described

I agree with this. This sounds like a big task/want list. Can we break this down further into multiple issues or even epics? I can work on involving an engineer from the Quality team to help with this effort.

added devopsverify label

@erushton Yes, @matteeyah's list makes sense, I believe it will cover everything on CI/CD

@meks thanks, yes, having this on staging and review apps would be perfect.

Thanks @filipa

So instead of seeding the DB like we always to can we use public APIs to setup such configurations? This way the provisioning can be automated and is repeatable in many environments.

Also including @ddavison and @rymai in the conversation since this converges on setting up test data and planning QA data model.

marked this issue as related to gitlab-development-kit#381

marked this issue as related to gitlab-development-kit#379 (moved)

marked this issue as related to gitlab-development-kit#378

marked this issue as related to gitlab-development-kit#377 (moved)

marked this issue as related to gitlab-development-kit#380 (closed)

Here are the issues I created in the GDK repo:

• gitlab-development-kit#381
• gitlab-development-kit#379 (moved)
• gitlab-development-kit#378
• gitlab-development-kit#377 (moved)
• gitlab-development-kit#380 (closed)

---

I think having a runner a part of GDK might be redundant since we already have documentation on setting up a runner with GDK - https://gitlab.com/gitlab-org/gitlab-development-kit/blob/master/doc/howto/runner.md

---

In my opinion the order of priority for these issues is:

1. A project with a cluster set-up
2. A merge request with all pipeline states
3. A project with all empty states of a job

Optional:

1. Adding a runner to the GDK
2. A running pipeline (is this possible without a runner?)

Along with the issues, I'd like to point out the elephant in the room: seeding GitLab with data is molasses slow! A lot of developers (including me) opt not to seed the database at all and just create mock data as necessary. It takes ~2h to fully seed the DB on my laptop (MBP 15" 2015).

mentioned in issue #49809 (moved)

@filipa, @matteeyah and all, Configuration team is also dealing with the same problem (https://gitlab.com/gitlab-org/gitlab-ce/issues/49809), so we can help with Seed a project with a cluster setup.

marked this issue as related to #49809 (moved)

@mayra-cabrera I updated the issue and added these two points to gitlab-development-kit#378:

• Install applications in a GKE cluster
• Protect an Environment

Feel free to add anything I missed.

assigned to @meks

@matteeyah Fixtures are part of gitlab-ce repository not GDK. GDK I assume that describe the scripts to run gitlab easily in development environment, but it does not rule out that you use raw gitlab-ce with fixtures for testing :)

removed the relation with gitlab-development-kit#377 (moved)

removed the relation with gitlab-development-kit#379 (moved)

marked this issue as related to #49959 (moved)

marked this issue as related to #49960 (moved)

@ayufan You're right. I moved the two issues that are related solely to adding more data in the fixtures to the CE project. Thanks.

marked this issue as related to #49946 (moved)

I share my latest experience about what kind of toolsets we provided to FE developers to quickly reproduce specific scenarios. The context was JUnit XML Test Summary In MR widget which shipped in %11.2 .

First of all, we tried to extend a fixture-seeder of CI Pipelines i.e. db/fixtures/development/14_pipelines.rb. We added some fixture to let FE developers somehow to reproduce a debuggable state, however, by some reasons, it turned out this approch is unextendable and slow. So we avoided this way.

Secondly, we hacked the fixture class as controllable module. We added three methods to reproduce and permutate multiple states. Also, we made each task finishs within a second. Afterall, that was enough for debugging all considerable scenarios in our context. Hence, we were able to quickly iterate the development cycle.

For exmaple, here is the instruction I prepared for FE devs https://gitlab.com/snippets/1731728#note_91233415, and this is a real catchball between FE and BE

/cc @godfat

mentioned in merge request !20973 (closed)

mentioned in issue #51601 (closed)

added devopssecure label

added priority2 label

@erushton and @grzesiek can you please also take a look at this one? It got the ~P2 label which means it must be included in 11.5.

changed milestone to %11.5

@jlenny This was raised in my 1:1 with me earlier with @erushton that Quality will also be helping out. Hence assigned to me.

Our team will start looking into some of these for 11.5 that is why I added the ~P2 from my end.

@meks should it then have the ~"CI/CD" label? I'm unclear if that team is doing any work.

removed 1 deleted label

removed 1 deleted label

I removed them for now since ~"CI/CD" is not currently working on this.

added [deprecated] Accepting merge requests label

This issue has passed the feature freeze date and considered a missed-deliverable. Adding missed:11.5.

/cc @godfat

added missed-deliverable missed:11.5 labels

changed milestone to %11.6

We got our first missed-deliverable for Quality

Looks like @rymai has already moved https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/20973 to %11.6 hence I will do the same here.

We will re-evaluate some of this as part of review apps implementation, it might makes sense to enable testability here instead since its essentially our GDK in the cloud.

changed weight to 8

removed [deprecated] Accepting merge requests label

Pinging @jhampton and @shampton for visibility, most likely you'll bump into the same problems.

Thanks for the ping @filipa

changed milestone to %12.0

I just ran into this, setting up pipelines with GDK for an issue. I ended up using this video to set things up, and it worked well, but it took about an hour, and luckily I did not need a pipeline in a particular state — just an MR widget with a pipeline.

I am definitely not an expert at this project by any means, but I wonder if a script to run and seed only the local database with a particular record configuration would be a smaller way to get this into a testable state, rather than an ever-growing list? I saw links above that this approach died out before, but it seems like this is not being worked on now either? (It is not totally clear to me. )

@sarahghp in order to have pipeline's running we only need to have gitlab-runner locally.

@filipa Yes, I know — I set it up and added the link to the video for anyone else who encounters this issue. Are you saying the video adds things that are unnecessary?

I also expect at some point I will need a pipeline or other data in a specific state and from reading this, it is not clear that all the items on this list have been addressed. Is that incorrect?

If you only need to have specific data in the rails then I'd rather suggest to play with the DB seeds. Though depending on the kind of resource this might be complex. E.g. To have security vulnerabilities displayed in the Group Security Dashboard, we have some projects already seeded with some vulnerabilities, so there is no need to run pipelines.

added internal customer label

marked this issue as related to gitlab-design#310 (closed)

mentioned in issue #60326 (closed)

Milestone %11.9 has expired, this will be rescheduled to the next milestone in active development %11.11. If you think this is incorrect, please feel free to update the milestone to a more appropriate one.

We're also adding missed:11.9

changed milestone to %11.11

added missed:11.9 label

mentioned in issue gitlab-design#394 (closed)

/cc @jhampton

Removing the state labels here since this isn't about changing the product itself.

removed devopssecure devopsverify labels

mentioned in issue #63065 (closed)

Milestone %11.11 has expired, this will be rescheduled to the next milestone in active development %12.1. If you think this is incorrect, please feel free to update the milestone to a more appropriate one.

We're also adding missed:11.11

changed milestone to %12.1

added missed:11.11 label

mentioned in issue #65094 (closed)

changed milestone to %12.2

added missed:12.1 label

mentioned in issue #66572 (closed)

mentioned in issue #66594 (closed)

mentioned in issue #66618 (closed)

changed milestone to %12.3

added missed:12.2 label

Hey!

GitLab is moving all development for both GitLab Community Edition and Enterprise Edition into a single codebase. The current gitlab-ce repository will become a read-only mirror, without any proprietary code. All development is moved to the current gitlab-ee repository, which we will rename to just gitlab in the coming weeks. As part of this migration, issues will be moved to the current gitlab-ee project.

If you have any questions about all of this, please ask them in our dedicated FAQ issue.

https://gitlab.com/gitlab-org/gitlab-ee/issues/13855

You can also find more information here:

• https://gitlab.com/gitlab-org/gitlab-ee/issues/13304
• https://about.gitlab.com/2019/08/23/a-single-codebase-for-gitlab-community-and-enterprise-edition/

Frequently Asked Questions

For an up to date list of questions and answers, please take a look at https://gitlab.com/gitlab-org/gitlab-ee/issues/13855

What will we do with the repository names?

https://gitlab.com/gitlab-org/gitlab-ee/ will become https://gitlab.com/gitlab-org/gitlab, and https://gitlab.com/gitlab-org/gitlab-ce will become https://gitlab.com/gitlab-org/gitlab-foss.

Why rename gitlab-ce to gitlab-foss?

Using "gitlab" and "gitlab-ce" would be confusing, so we decided to rename gitlab-ce to gitlab-foss to make the purpose of this FOSS repository more clear

I created a merge requests for CE, and this got closed. What do I need to do?

You will need to create a merge request at https://gitlab.com/gitlab-org/gitlab-ee/. This link will eventually redirect to https://gitlab.com/gitlab-org/gitlab.

How does the licensing work in this new setup?

Everything in the ee/ directory is proprietary. Everything else is free and open source software. If your merge request does not change anything in the ee/ directory, the process of contributing changes is the same as when using the gitlab-ce repository.

Will you accept merge requests on the gitlab-ce/gitlab-foss project after it has been renamed?

No. Merge requests submitted to this project will be closed automatically.

Will I still be able to view old issues and merge requests in gitlab-ce/gitlab-foss?

Yes.

How will this affect users of GitLab CE using Omnibus?

No changes will be necessary, as the packages built remain the same.

How will this affect users of GitLab CE that build from source?

Once the project has been renamed, you will need to change your Git remotes to use this new URL. GitLab will take care of redirecting Git operations so there is no hard deadline, but we recommend doing this as soon as the projects have been renamed.

Where can I see a timeline of the remaining steps?

https://gitlab.com/gitlab-org/gitlab-ee/issues/13304

Will community contributions submitted to the new "gitlab" repository be available in the new gitlab-foss repository?

Yes, these changes will be synced to gitlab-foss automatically, several times per day.

locked this issue