Plugins

This issue evolved, latest state is in https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/17003

---

Companies want to add their own features to GitLab without sharing the code with others by contributing it to the main repository (for security reasons, to iterate quickly, to have something specific to them, or to do something that doesn't meet our definition of done). Right now they have to run on a fork, that makes GitLab hard to keep up to date and we see that they tend not to upgrade.

The proposal is to:

• Rename project services to plugins (to make it clear that we have plugins and because project services never was a good name)
• Allow local plugins by loading them automatically if they are in a directory that is not under version control in /var/opt/gitlab/gitlab-rails/plugins

1. Isn't this the same thing we bash against Jenkins for?
2. Can all integrations be relegated to single files? I would imagine that would limit the kinds of things that can be done with this plugin architecture. e.g. modifying code diff displays to show inline static analysis results. Maybe we don't have any plugins like that today, but people will want them.
3. There's value in third-party contributions. There's value in curating them and protecting users from the downside of unmanaged third-party contributions. Yet there's also value in allowing companies do have private integrations. Perhaps we could make it such that local plugins are only used for private company-specific integrations while public third-party integrations use a different (curated) mechanism.

we discussed this, do you agree with above?

@sytses yes we did. I agree with above.

For security reasons we can't have the plugins inside of /etc/gitlab directory. This directory contains secrets, individual files and directory permissions are security tightened. We could find a better place, maybe /var/opt/gitlab/gitlab-rails/plugins. The holding directory is already accessible to unicorn and has the necessary permissions.

I wonder whether we can avoid doing it this way tough. If we depend on user adding the file, bunch of things can go wrong (think of permissions, ownership, syntax errors and so on) and we have no way of verifying that those things are correct. It is even possible that the app could crash (in some cases).

Is it possible if we can add a plugin uploader from within the application itself? This way we can ensure the correctness of the file and inform the user. We also remove the dependency on the filesystem manual edit and with move with gitaly, this is becoming more important.

@markpundsack

1. Of our plugins most organizations will use only plugins that are part of our codebase and tested every release.
2. I'm not sure, but this seems like the simplest step.
3. I agree with that goal, how do we achieve that?

@marin /var/opt/gitlab/gitlab-rails/plugins works, I've changed the description. I'm not against adding the ability to upload, in that case we can put the files in /var/opt/gitlab/gitlab-rails/plugins right?

changed the description

@JobV what do you think?

I'm not against adding the ability to upload, in that case we can put the files in /var/opt/gitlab/gitlab-rails/plugins right?

Correct.

@marin /var/opt/gitlab/gitlab-rails/plugins is same as having plugins in root of GitLab CE repo, right? I want to experiment with this feature today and it seems like having it inside Rails directory is the best way to start.

@sytses it's a good idea.

Why not have all plugins (the ones we curate / test / maintain) and the potential customer plugins in the same directory / using the same mechanism? That would make things easier for both of us, as we only have to maintain a single method of working with these, and the contribution / creation of new plugins could easily be done on basis of our examples.

In general, we should prefer and recommend companies to use our regular APIs though. That's a much easier to maintain approach (for them) and doesn't force us to be more careful with these integration points.

cc @mydigitalself @jramsay

I made a working concept at https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/3687.

Why not have all plugins (the ones we curate / test / maintain) and the potential customer plugins in the same directory / using the same mechanism?

@JobV I don't recommend same directory because of next reasons:

• hard to determine where are customer plugins vs ours
• deep into structure (app/models/project_services/ vs plugins/)
• its easier to ask customer to do ls plugins in case they experience issues

However, the way plugins work will be same as existing project services. See https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/3687 and this screenshots:

→ ll plugins/
total 8
drwxr-xr-x   4 dzaporozhets  staff   128B Dec  8 14:50 ./
drwxr-xr-x  76 dzaporozhets  staff   2.4K Dec  8 14:47 ../
-rw-r--r--   1 dzaporozhets  staff     0B Dec  8 14:47 .gitkeep
-rw-r--r--   1 dzaporozhets  staff   652B Dec  8 14:50 example_service.rb

is same as having plugins in root of GitLab CE repo

@dzaporozhets It is not the same as Rails root, but we have a way of creating that directory during the build. Ideally, the path should be configurable but we can discuss that once we have an idea of how this could look.

@sytses @JobV

re @markpundsack's points

1. I'm very nervous about this. If I understand the architecture correctly, these plugins will execute within GitLab? This could mean that performance could be out of our hands, similar to FireFox, Chrome and - yes - Jenkins. Don't you achieve the same capability by adding a webhook to your own 3rd party app, and then calling back into GitLab via the API if needed. We could provide 'stub apps' in a few languages (Ruby, Node) that accept webhooks and act on them. People can them host them or, better still, easily spin them up in k8s etc...
2. If you have an app that utilises the API and hooks, then you can achieve richer functionality, although nothing that adds UI unless we provide APIs to do that somehow.
3. Slack & GitHub are both good examples of this. You wrap API & webhooks into an "App" and then you "install" the app into GitLab either at a server-level, group level or project level. They leverage the community by having a Marketplace or App Directory. This seems to be the defacto model. The model we are proposing seems at odds with this trend, and is also limited to rails, rather than being API-driven.

1. These will execute as GitLab and can cause performance degradations. But this is the only way to change things like the UI I think. We already have webhooks and custom git hooks https://docs.gitlab.com/ee/administration/custom_hooks.html
2. Exactly, is there an example of a UI API? Sounds complex.
3. That is a great example of an alternative approach. Maybe we should work with Fable to find out if this addresses their need.

Additional comment from DZ: We can validate the plugin on load, we send fake data and see if it returns the right result. Same rules as the project service already has in tests. If it doesn't we don't load it and write to a log file.

How will we prevent plugins (in general) from opening security holes?