SourceForge used to have a mailinglist. Right now everyone wanting a mailinglist for their project is required to use Google Groups.
It would be nice if this could be done with GitLab. We already have the ability to read email for the reply-to-email feature. And we already have service desk https://docs.gitlab.com/ee/user/project/service_desk.html that implements similar functionality.
Active for every project at mailinglist+group/project@example.com (no need to be able to turn off and on, nobody is subscribed by default and people can just unsubscribe
Subscribe by sending 'subscribe' as the first word in the body.
Unsubscribe by sending 'Unsubscribe' as the first word in the body.
Optional feature: People posting a single issue don't have to subscribe to a potentially high volume list: you will be Cc'd in replies. Please keep the Cc list intact when replying (use "Reply to all"). See https://git-scm.com/community
Optional feature: Store the emails. The first iteration will not store any emails and there will be no archive.
If we have email actions for issues (like !create), one could just mention everyone in the project and create an ad-hoc mailing list like that. We could even make it work with !mailinglist, which would do that in one command.
@JobV To make this feature easier to understand I propose we just have people email gitlab-ee/gitlab.org@gitlab.com to email everyone subscribed to the mailinglist and don't work with actions.
Hi everyone, I was the one who tweeted @gitlab about this. Thanks for making the issue @sytses.
My personal experience with gitlab is fairly small. I run it on a server to back up private code (a much older version) and just made my account on gitlab.com this week. I'm not a major stakeholder so perhaps my suggestions don't fit inline with gitlab's values. Given this information, here are some of my suggestions
Online View
Making sure the email list can be accessed through an online message board. This services 3 purposes. 1. Making it simple for someone to post to the board. 2, Discoverability. If the mailing list is indexable, it'll make it easy search for questions, bugs or feature requests from the past. 3, Linkability, being able to share past threads using a URL is very, very useful.
Basic Moderation Tools
Spam and harassment are fairly common on several mailing lists I frequent. Most of the issues are regarding promotion or recruitment. It would be nice to block people from emailing the list.
Subscription Levels
One of the best features of google group is the ability choose your subscription level. You can follow just threads you've posted, get a daily/weekly summary or get nothing at all.
I've had other ideas but I think I'll leave it to Gitlab's PMs.
I think it makes sense to have a basic visualization https://gitlab.com/gitlab-org/gitlab-ce/mailinglist/ in the first edition. Spam and moderation can wait because of the projects you subscribed to ask @JobV mentioned. Subscription frequency is needed but can also wait.
@toon when someone intends to contribute this we can consider the exact details of implementation. Signalling that we're interested in this and that we're willing to accept contributions is a good start, rather than doing work upfront for something that might not end up being implemented.
It makes sense to have it be driven by issues. We don't need to support deeper threading in a first iteration. Nor do we have to consider how closed/open is represented (it simply isn't).
The project I use the most is postgresql jdbc. Historically all issues,
discussion, patches came through a mailing list.
It was all in one place.
Now all the issues are on github. The mailing list sees almost no traffic.
The challenge for me is that it is very difficult to find the history of a
conversation about a particular problem because it is now split into
commits and issues.
Ideally I'd like to aggregate everything back into a single searchable
list. The mailing list just seemed to be the simplest.
It also allows people without git accounts to monitor and or provide
feedback.
Hopefully my situation may help drive the requirements.
If not, where will we store the emails (database?)? Some visual mock ups would be advised. => We won't for the first iteration, I updated the description to reflect this.
Are we sure we want the use the same address to subscribe?
Do we want a 1:1 relation with the instance users', or any email can subscribe?
Is this active only for public projects? If not, how we can protect private information in private projects? Do we want to do so?
There will some email that will be automatically sent by Gitlab to the ML? Like every issue thread? (This can be a question for a second iteration)
Technical questions:
We have basically a relation between a project and n email addresses. Where do we save them? I think there is a project table in the db? (I think I can figure it out, but I still haven't look at the code)
Are we sure we want the use the same address to subscribe?
Is this active only for public projects? If not, how we can protect private information in private projects? Do we want to do so?
To tackle these two downs, maybe we do subscribe and unsubscribe on another email, something like mailinglist+group/project+subscribe+randomGibberish@example.com?
Also, I think we should also think about how to remove someone from a ML (like, someone removed from a private project).
Or maybe in the first interaction we go only for public projects, and then we add the feature to internal/private project after having implemented a proper authorization / control system?
Are we sure we want the use the same address to subscribe?
I like your proposal with +subscribe. Although I don't really understand why we need the +randomGibberish?
Do we want a 1:1 relation with the instance users', or any email can subscribe?
I don't think that's needed in a first iteration, but it would be nice if active GitLab users can easily subscribe with a push of a button. Anyhow, it should also be possible for non-GitLab users to subscribe, because that's the point of having a mailing list and not using issues instead.
Is this active only for public projects? If not, how we can protect private information in private projects? Do we want to do so?
Good question. But like you proposed, maybe just start off with public projects.
There will some email that will be automatically sent by Gitlab to the ML? Like every issue thread? (This can be a question for a second iteration)
Let's iterate on that later.
We have basically a relation between a project and an email addresses. I think there is a project table in the db?
The projects table exists, but try to avoid adding columns to that table, because it's already too big. You can create a new table mailing_lists which has a foreign key project_id. We can store other settings about the mailing list there too.
Where do we save them?
At first, everything for the mailing list would be mail-driven. So subscribe and unsubscribe are done by sending emails (because then anyone can subscribe). We would keep these subscriptions in a new table (e.g. mailing_list_subscriptions). It would contain an email address and a foreign key mailing_list_id.
Besides that I have a few other questions/remarks:
Where would we store the mails themselves? They can become quite large, so storing them in the database won't be a good idea.
Hi @toon ,
thanks for your kind reply, let me expand a bit on a couple of points:
@rpadovani Thank you for taking initiative in this proposal!
Are we sure we want the use the same address to subscribe?
I like your proposal with +subscribe. Although I don't really understand why we need the +randomGibberish?
I think about it for privacy / security reasons: if a project is private, I expect the mailing list to be private as well. If we do not put some random chars in the subscribe email, I can subscribe to any private project ML just knowing the name of the project. While the name itself should be private, I think it doesn't require too many tries to guess the email to subscribe.
If we limit the scope of the first interaction to public projects, we can think about this later (maybe for private projects we will implement the 1:1 relation with users and disallow subscription for non team members)?
Anyhow, it should also be possible for non-GitLab users to subscribe, because that's the point of having a mailing list and not using issues instead.
Good point
The projects table exists, but try to avoid adding columns to that table, because it's already too big. You can create a new table mailing_lists which has a foreign key project_id. We can store other settings about the mailing list there too.
Thanks for the explanation
Where would we store the mails themselves? They can become quite large, so storing them in the database won't be a good idea.
Optional feature: Store the emails. The first iteration will not store any emails and there will be no archive.
In the first iteration we should only forward them as soon as we receive them, as far as I understand?
Overmorrow my semester break starts, so I will start coding.
Let me recap so we are all on the same line.
I start from a Minimum Viable Product with these characteristics:
Active/Available only for public projects
Mail mailinglist+group/project@example.com to send email. For the first implementation you need to be subscribed to send / receive
Mail mailinglist+group/project+subscribe@example.com to subscribe/unsubscribe to a ML.
No archive, only forwarding.
Possible future iteration (we will maybe open related issues when/if I will be able to propose the first implementation):
Archive
Some way of linking with the user, implementation in the UI
Available for internal/private projects
No need to subscribe to send email
Maybe other options (digest format / weekly recap).
Just a last question:
are we sure it is more convenient to start from scratch, instead of just including in the omnibus something like Mailman and calling its APIs to configure a project?
@rpadovani thanks for looking at this. Before we extend the scope of reply-by-email even further, I'd like to do https://gitlab.com/gitlab-org/gitlab-ce/issues/37403 - I know that you aren't responsible for this, but I think it's important that we can guarantee that these don't conflict with each other in a better way than 'some issue says so' 🙂
Mail mailinglist+group/project@example.com to send email. For the first implementation you need to be subscribed to send / receive => I would suggest one email address mailinglist+group/project@example.com and looking at the first word in the body to subscribe / unsubscribe instead of using a different email address. You should be able to unsubscribe by replying unsubscribe to an email
Mail mailinglist+group/project+subscribe@example.com to subscribe/unsubscribe to a ML. => same comment as above
can make it available for all projects, allow to turn on/off like everything else
Then we should check if who subscribe to the list is a team member, because as end user I totally expect that a ML for a private project is private itself. I'll do :)
In the first iteration we should only forward them as soon as we receive them, as far as I understand?
OK, fair enough.
if a project is private, I expect the mailing list to be private as well.
I agree!
If we do not put some random chars in the subscribe email, I can subscribe to any private project ML just knowing the name of the project.
This is just security by obscurity, this doesn't really help a lot. People still can pass around the email address to circumvent this.
I'm also concerned by people who are already subscribed to the list, but then later their access to the private project is revoked. In that case we have to make sure they are then also removed from the mailing list (or at least not receive the emails).
I made a small mockup to illustrate @sytses's proposal:
The possible values are:
"Only project members" => Everyone who is member by being a direct member or member of one of the groups who has access
"Everyone with access" => This depends on the "Project visibility" level:
"Public": Anyone can subscribe, even non-GitLab users.
"Internal": The subscribers need to have a GitLab user account.
"Private": Basically the same result as "Only project members"
So only if the project is "Public" and "Everyone with access", the mailing list can be used with email addresses that are not known by the GitLab instance. In all the other cases we have to be careful.
I think it's doable. Because an email address is unique within a GitLab instance. So when a mail arrives at the mailing list dispatcher, we can iterate through all the subscribers (list of email addresses). For each email address we search for the user linked to that email (if the email address is not linked to any user, we use nil for the user) and then check if that user has access to the project mailing list:
Ability.allowed?(user,:read_mailing_list,project)# `user` might be `nil`
Okay @toon, thanks for your explanations and time on this! I will start to work on it soon and create a WIP MR, so I can have feedbacks - my first attempt to do something big in Ruby, we'll see how it goes :-)
thanks for pointing it out. Do you think to work on it soon? Can I help with that?
That depends on what you mean by soon 🙂 We're currently working on GitLab 10.6, and we weren't planning to work on it in GitLab 10.7, so the earliest would be GitLab 10.8, which will be released on 22 May 2018.
If you'd like to help, that would be great! Please comment on the issue and we can discuss there!
That depends on what you mean by soon 🙂 We're currently working on GitLab 10.6, and we weren't planning to work on it in GitLab 10.7, so the earliest would be GitLab 10.8, which will be released on 22 May 2018.
I'd like starting working on the ML feature this weekend, since in March I have a bit more free time than usual. While I can do some preparatory work before, I will need also to manage email handler at some point (duh!).
If you'd like to help, that would be great! Please comment on the issue and we can discuss there!
Super, I am going to do so, and help tackling it down ;-)
@toon I started working on it: I opened the MR so I created a list of todos in the ML description, feel free to comment it.
Also, I started writing down some "code" (a lot of comments actually, to structure the code) and the DB tables. How do you want to proceed? I ping you when I do something interesting in the MR, so you can leave feedbacks while I am developing? Or do you prefer to keep an eye by your own, and I ping you when I am done (or blocked)?
@rpadovani It is mostly up to you. Normally I wouldn't actively keep an eye on it until you ask me something. So whenever you want, you can ping me, and I'll have a look.
This features would be highly appreciated on our side, and likely not only there.
In OSS communities that are hosting code on gitlab or github, we are frequently running into the questions where to host the related mailing list and - even worse - how to integrate both UI-based and ML-based workflows smoothly. Folks tracking projects via mailing list should not be decoupled from those talking to the project via tickets and PRs, e.g. Often the only answer is to block the UI-based input channels (as far as the platform allows this) to avoid a community split, which likely means losing at least some drive-by reporters/contributors. So I could imagine that this feature becomes the foundation of another valuable differentiating factor for gitlab over github.
Question: Are you planning to write your own ML backend service or rather integrate an existing one that comes with all the bells and whistles one would expect (e.g. an archiving service)? I would be concerned that reimplementing things from scratch could take quite a while until things work smoothly.
This features would be highly appreciated on our side, and likely not only
there.
In OSS communities that are hosting code on gitlab or github, we are
frequently running into the questions where to host the related mailing
list and - even worse - how to integrate both UI-based and ML-based
workflows smoothly. Folks tracking projects via mailing list should not be
decoupled from those talking to the project via tickets and PRs, e.g. Often
the only answer is to block the UI-based input channels (as far as the
platform allows this) to avoid a community split, which likely means losing
at least some drive-by reporters/contributors. So I could imagine that this
feature becomes the foundation of another valuable differentiating factor
for gitlab over github.
for me and the postgresql jdbc driver this would tip the scales. Assuming
it integrates with an existing list.
Real life took precedence in last months, but now I have more time to contribute again. I'll try to work on it a bit more in next days (atm rebasing on master).
Anyhow, if someone wants to help / take over / give feedback, just ping me :-)
this MR will allow user to enable/disable mailing list in project settings:
add feature flag for mailinglist (TBD)
add model and migration for representing mailing list (addressed by db/migrate/20180930120000_create_mailing_list.rb and app/models/mailing_list.rb, app/models/project.rb)
add basic UI for mailing list settings (addressed by app/views/admin/application_settings/_mailing_list_settings.html.haml), not sure though if we can allow all these options - I would expect that both IMAP/SMTP from global gitlab.yml should be used
this MR will allow users to subscribe/unsubscribe from the ML
add model and migration for keeping subscribed users - addressed by db/migrate/20180930120000_create_mailing_list.rb and partially by app/models/mailing_list_pending_subscriptions.rb
add email handler which processes subscription/unsubscription (including confirmation email) - addressed by lib/gitlab/email/handler/mailing_list_handler.rb
this MR will allow users to send emails to the ML
add email handler which processes an email sent to the ML and sends it to all subscribers
add UI which allows project owners to manage ML subscribers
list/delete subscribed emails
add API for managing mailing lists (TBD)
allows project admins to manage ML subscribers (same as with UI)
Store emails (TBD)
allows to store emails sent to the ML and display them in web UI and API
@jprovaznik thanks for breaking that down! Just so I'm clear, at which point would this feature become usable? Only the last step? Is there a way we could make it usable-but-minimal earlier?
@smcgivern I reordered them and added numbering, I suppose that in the list above MVP could consist of MRs 1-3. (4 and 5 could follow-up in a next iteration, but 4 for managing users would be still high priority).
@jprovaznik thanks! Just so I understand, does step 3 involve archiving the emails, or not? I think if we ever want to add it, it would probably be good to be clear about when that should be.
As for step 4 - prior to that, is there any way to manage subscribers, or do we just assume that keeping the mailing list address private is sufficient?
@jprovaznik thanks! Just so I understand, does step 3 involve archiving the emails, or not? I think if we ever want to add it, it would probably be good to be clear about when that should be.
@smcgivern thanks for pointing this out. Storing emails is not part of step 3 - added step 6 to track this.
As for step 4 - prior to that, is there any way to manage subscribers, or do we just assume that keeping the mailing list address private is sufficient?
@smcgivern originally I proposed steps in https://gitlab.com/gitlab-org/gitlab-ce/issues/4272#note_115526454 based on @rpadovani's existing MR. But also a second option was proposed in this issue - integrate an existing ML - this has both pros and cons. I'm not proposing we should change the direction now, but I'm curious what was the reasoning for not choosing this option?
@jprovaznik I think the problem with integrating an existing ML might be more work, especially if it's written in a different language, has a different release schedule, etc. However, writing our own has its own set of problems, too!
I think that (1) is 90% done, it needs only a feature flag.
(2) misses the unsubscribe (but could be easily implemented), but I am not sure clicking on the confirmation link on the email sent by Gitlab works. The others are missing almost completely.
I think 1-2-3 are the MVP, 4, 5, and 6 are good additions (and a lot more can come after, e.g digest, rss feeds).
All of them need also API and tests.
If you want, I can start working on 1, but it's also up to the timeline you want to follow
I am available to work on this a bit, but it's up to you choose if you prefer having someone that doesn't need so much help developing it. I know that can be burdensome having external contributors that are not high skilled working on some major feature.
Therefore I am available to give it a try, at least to the first 2 part, if it can work for you - otherwise I will continue contributing doing smaller patches in other areas :-)
These should together give us MVP for mailing list. All of them will be done behind feature flag so although tasks are not standalone and independent (I don't know how to split this better) each of them should be reasonably small and should address logical piece of functionality.
@rpadovani if you don't mind I assigned https://gitlab.com/gitlab-org/gitlab-ce/issues/57140 to you as you already addressed this part in your current MR 🙂 . I left other issues unassigned but please feel free to start on any of them if it attracts you and someone from ~Plan team can cooperate on it (I assigned %11.9 milestone to them.
GitLab is moving all development for both GitLab Community Edition
and Enterprise Edition into a single codebase. The current
gitlab-ce repository will become a read-only mirror, without any
proprietary code. All development is moved to the current
gitlab-ee repository, which we will rename to just gitlab in the
coming weeks.
As part of this migration, issues will be moved to the current
gitlab-ee project. We are starting with moving issues that have
been created at least a year ago, and are not part of the current
milestone.
If you have any questions about all of this, please ask them in our
dedicated FAQ issue.
You can also find more information here:
Using "gitlab" and "gitlab-ce" would be confusing, so we decided to
rename gitlab-ce to gitlab-foss to make the purpose of this FOSS
repository more clear
I created a merge requests for CE, and this got closed. What do I
need to do?
Everything in the ee/ directory is proprietary. Everything else is
free and open source software. If your merge request does not change
anything in the ee/ directory, the process of contributing changes
is the same as when using the gitlab-ce repository.
Will you accept merge requests on the gitlab-ce/gitlab-foss project
after it has been renamed?
No. Merge requests submitted to this project will be closed automatically.
Will I still be able to view old issues and merge requests in
gitlab-ce/gitlab-foss?
Yes.
How will this affect users of GitLab CE using Omnibus?
No changes will be necessary, as the packages built remain the same.
How will this affect users of GitLab CE that build from source?
Once the project has been renamed, you will need to change your Git
remotes to use this new URL. GitLab will take care of redirecting Git
operations so there is no hard deadline, but we recommend doing this
as soon as the projects have been renamed.
Where can I see a timeline of the remaining steps?