SeatLink, when multiple active licenses present, may overwrite License.current

Everyone can contribute. Help move this issue forward while earning points, leveling up and collecting rewards.

Summary

valid license definition: License is started ✔️ not expired ✔️

active subscription definition: not cancelled ✔️

Generally, there should be only 1 valid license for 1 active subscription at a time on a Self Managed server. In some rare cases, it may be possible for multiple valid licenses for multiple separate, (yet not necessarily active) subscriptions to be present simultaneously. Example: cancelled / debooked subscriptions

In this case, if SyncSeatLinkWorker is invoked for SeatLinkData containing both licenses consecutively, the result of SyncSeatLinkRequestWorker will cause GitLab to replace the current license with that of the license for the subscription that was last seat linked, leading to undefined, and likely, undesired behavior.

Steps to reproduce

  1. Have two separate cloud activated license subscriptions
    • optionally having different attributes (start/end/user_count) for better observability
  2. Activate both and observe which license is currently loaded
  3. Execute a SyncSeatLinkRequestWorker using SeatLinkData for the license key not currently loaded (ie not License.current)
  4. That license is now License.current
    • more precisely, a duplicate of that license is now License.current

After activating both of the licenses, notice that the most recent (2nd of 2) subscription you activated is currently loaded.

irb(main):026:0> License.all
=> 
[#<License:0x00007f4ea0e6e150
  id: 10,
  data:
   "<snip>",
  created_at: Tue, 06 Feb 2024 19:31:24.843937000 UTC +00:00,
  updated_at: Tue, 06 Feb 2024 19:31:24.843937000 UTC +00:00,
  cloud: true,
  last_synced_at: Tue, 06 Feb 2024 19:31:24.839314000 UTC +00:00>,
 #<License:0x00007f4ea0e6e088
  id: 11,
  data:
   "<snip>",
  created_at: Tue, 06 Feb 2024 19:32:49.758480000 UTC +00:00,
  updated_at: Tue, 06 Feb 2024 19:32:52.429429000 UTC +00:00,
  cloud: true,
  last_synced_at: Tue, 06 Feb 2024 19:32:52.429429000 UTC +00:00>]

irb(main):025:0> License.current.id
=> 11

Send a seat link for the unloaded license

irb(main):027:0> seat_link_data = Gitlab::SeatLinkData.new(key: License.find(10).data)
=> 
#<Gitlab::SeatLinkData:0x00007f4ea08f9ee0
...

irb(main):029:1* SyncSeatLinkRequestWorker.perform_async(
irb(main):030:1*       seat_link_data.timestamp.iso8601,
irb(main):031:1*       seat_link_data.key,
irb(main):032:1*       seat_link_data.max_users,
irb(main):033:1*       seat_link_data.billable_users_count,
irb(main):034:1*       seat_link_data.refresh_token
irb(main):035:0> )
=> "215b2a253199141eeda00862"

A 3rd license is present now, it's the exact key from the previously unloaded license, and because a new record is created, it becomes License.current

irb(main):036:0> License.all.count
=> 3

irb(main):037:0> License.current.data == seat_link_data.key
=> true

irb(main):038:0> License.all
=> 
[#<License:0x00007f4ebe5f5f28
  id: 10,
  data:
   "<snip>",
  created_at: Tue, 06 Feb 2024 19:31:24.843937000 UTC +00:00,
  updated_at: Tue, 06 Feb 2024 19:31:24.843937000 UTC +00:00,
  cloud: true,
  last_synced_at: Tue, 06 Feb 2024 19:31:24.839314000 UTC +00:00>,
 #<License:0x00007f4ebe5f5de8
  id: 11,
  data:
   "<snip>",
  created_at: Tue, 06 Feb 2024 19:32:49.758480000 UTC +00:00,
  updated_at: Tue, 06 Feb 2024 19:32:52.429429000 UTC +00:00,
  cloud: true,
  last_synced_at: Tue, 06 Feb 2024 19:32:52.429429000 UTC +00:00>,
 #<License:0x00007f4ebe5f5780
  id: 12,
  data:
   "<snip>",
  created_at: Tue, 06 Feb 2024 19:39:07.858762000 UTC +00:00,
  updated_at: Tue, 06 Feb 2024 19:39:07.858762000 UTC +00:00,
  cloud: true,
  last_synced_at: Tue, 06 Feb 2024 19:39:07.854421000 UTC +00:00>]

irb(main):039:0> License.current.id
=> 12

This happens because, after a successful seatlink response from the API server, the reset_license! method takes the license key that API sent back (a copy of the one we just sent), and checks if it is the currently loaded cloud license, which it is not, causing it to create a new License record with that key.

A new License record means an incremented License.id, meaning this license now becomes License.current.

Output of checks

Reproduced in docker gitlab/gitlab-ee:16.8.1-ee.0 (latest at time of writing)

Results of GitLab environment info

Expand for output related to GitLab environment info 
System information
System:		
Proxy:		no
Current User:	git
Using RVM:	no
Ruby Version:	3.1.4p223
Gem Version:	3.4.22
Bundler Version:2.5.4
Rake Version:	13.0.6
Redis Version:	7.0.15
Sidekiq Version:7.1.6
Go Version:	unknown

GitLab information
Version:	16.8.1-ee
Revision:	1242b447720
Directory:	/opt/gitlab/embedded/service/gitlab-rails
DB Adapter:	PostgreSQL
DB Version:	14.9
URL:		http://gitlaptop
HTTP Clone URL:	http://gitlaptop/some-group/some-project.git
SSH Clone URL:	git@gitlaptop:some-group/some-project.git
Elasticsearch:	no
Geo:		no
Using LDAP:	no
Using Omniauth:	yes
Omniauth Providers: 

GitLab Shell
Version:	14.33.0
Repository storages:
- default: 	unix:/var/opt/gitlab/gitaly/gitaly.socket
GitLab Shell path:		/opt/gitlab/embedded/service/gitlab-shell

Gitaly
- default Address: 	unix:/var/opt/gitlab/gitaly/gitaly.socket
- default Version: 	16.8.1
- default Git Version: 	2.42.0

Possible fixes

GitLab should probably only be sending SeatLink data for one license at a time.

Workarounds

Remove all licenses tied to the undesired subscription

Reported examples

  1. https://gitlab.zendesk.com/agent/tickets/498064
    • In the above example case, internal notes on the ticket show that both relevant subscriptions were sending SeatLink data each day, leading to this bug. It wasn't immediately clear how or why SyncSeatLinkWorker was being invoked for both license subscriptions. It seems that Gitlab::SeatLinkData.new() should always fetch License.current by default, but clearly something was causing that to not always be true. It did not matter that one of the subscriptions is cancelled in zuora.
  2. https://gitlab.zendesk.com/agent/tickets/674239

Support Priority Score: (0, 0, 0, 3, -, -, 3, -, -, -, -) => 6

Edited by Keven Hughes