Skip to content

ADD sending release activities through ActivityPub

kik requested to merge gitlab-community/gitlab:activitypub-releases-sendout into master

What does this MR do and why?

⚠️ This is the last part of the merging of the overarching MR, you may read its description if you want to see a bigger picture. Feel free to ask as many questions as you want if you want to understand the protocol!

In ActivityPub, getting notified of new activities (activities here being similar to entries in a RSS feed), is done in two steps:

  1. the user interested in the resource subscribe to it. This was already implemented in previous work.
  2. when an activity happens (a new release for a project, in our current case), we push a JSON payload to the servers of each subscriber to let them know.

This commit is about this second part. We already have the list of subscribers and the URLs where to reach them (called the user inbox) in our database, we now need to fire the requests when the release event happens.

Worth noting: ActivityPub is well designed enough to make sure this process is optimized. It is done through each server having a shared inbox so that when 1000 users are on the same server, we just fire a single request for the 1000 of them. ActivityPub being a federated protocol, with a client/server architecture and many users aggregating on a few servers, this ensures the load does not get out of control. Most (all?) implementations support shared inbox, so it shouldn't be an issue. The logic is implemented so that subscriptions having a shared inbox are prioritized, and there will be at most 100 requests fired per run of the service, requeuing the job if we're not done yet (given the amount of servers in the Fediverse, it's unlikely for now that there will be subscribers to a project on more than 100 different servers, but who know where the Fediverse is going). In case of several batches, to decide if a subscription has been processed, we check its updated_at time.

If a server is unresponsive, we ignore it this time, so that we don't have to block the queue until it responds properly (which would happen if we let the job fail and retry later). We'll probably want later to bake something in to delete subscriptions of repeating offenders.

Note: we now have two places in the codebase that has similar logic to upload payloads to the user's inbox (here, and where we confirm subscription, implemented in a previous commit), so it's been abstracted away in a concern. This will be reused abundantly.

Note 2: We've had a previous discussion about the disabled Rubucop's rule.

How to set up and validate locally

A Sinatra app has been created to test communication with the application, as this is a server to server feature:

  1. prepare your GitLab local installation by making the flightjs/Fligthjs project public, enabling the activity_pub and activity_pub_project feature flags and allowing requests to the local network from webhooks and integrations. This can be done in a rails console with the following:
Feature.enable(:activity_pub)
Feature.enable(:activity_pub_project)
ApplicationSetting.first.update(allow_local_requests_from_web_hooks_and_services: true)
flight = Project.find_by_name("Flight")
flight.update(visibility: "public")
  1. Install and run the sinatra app:
git clone https://gitlab.com/oelmekki/activitypub-review
cd activitypub-review
bundle
ruby server.rb
  1. Visit the app URL and click the "subscribe" button
  2. You should see the success reply from GitLab (it may takes a while on first request in dev env)
  3. You have created a subscription. To verify that, you can go check the Sinatra logs, and you should see:
  • a request to /subscriber/inbox, this is the GitLab server sending the Accept activity
  • a dump of the request object, you can verify it correctly sets the Accept http header to application/ld+json; profile="https://www.w3.org/ns/activitystreams"
  • a dump of the activity posted, as a JSON object, with type "Accept" and containing the initial Follow activity as object.

⚠️ Note: if you want to run the test several times, you need to delete the subscription between each time: ActivityPub::ReleasesSubscription.destroy_all

Now let's test emitting activities, which is the purpose of this MR:

  1. Create a new release for the Flight project in GitLab:
flight = Project.find_by_name("Flight")
user = User.first
FactoryBot.create(:release, project: flight, author: user, released_at: Time.now)
  1. In rails console, invoke the cron task:
Releases::PublishEventWorker.new.perform

⚠️ If someone knows a better way to trigger cron tasks in development environment, please let me know!

  1. Then go check the Sinatra's app log. You should see:
  • a request to /subscriber/inbox, this is the GitLab server sending us the activity
  • a dump of the request object, you can verify it correctly sets the Accept http header to application/ld+json; profile="https://www.w3.org/ns/activitystreams"
  • a dump of the activity posted, as a JSON object, with type "Create" and information about the release as object.

Related to &11247

Edited by kik

Merge request reports