Implement Geo support of Supply Chain Attestations behind FF

Aaron Huntsmanrequested to merge
580365-implement-geo-support-of-supply-chain-attestations-behind-ff into master

What does this MR do and why?

Adds implementation for Geo replication of SupplyChain::Attestation records, using blob replication strategy for the file field. Also implements selective sync by organization.

Note: The predicate_file field was added to the model but is not used and intended to be removed (discussion here).

Adds feature flags for safe rollout:

  • geo_supply_chain_attestation_replication - controls replication functionality
  • geo_supply_chain_attestation_checksumming - controls verification on primary

How to set up and validate locally

Prerequisites

Click to expand 1. **Set up Geo with GDK** - Follow the GDK Geo setup guide to configure a primary and secondary Geo instance - Enable object storage for Merge Request Diffs - Ensure both instances are running properly 2. **Enable Geo replication for attestations + organization features** - Run these Rails commands on your primary GDK instance in Rails console: ```ruby Feature.enable(:geo_supply_chain_attestation_replication) Feature.enable(:geo_supply_chain_attestation_force_primary_checksumming) Feature.enable_percentage_of_time(:allow_organization_creation, 100) Feature.enable_percentage_of_time(:organization_switching, 100) Feature.enable_percentage_of_time(:ui_for_organizations, 100) ``` 3. **Create test organizations** - Run these Rails commands to create test organizations with projects: ```ruby # Create first organization with owner org1 = Organizations::Organization.create!(name: 'Test Org 1', path: 'test-org-1', visibility_level: Organizations::Organization::PUBLIC) Organizations::OrganizationUser.create_organization_record_for(User.first.id, org1.id) 
  # Create second organization with owner
  org2 = Organizations::Organization.create!(name: 'Test Org 2', path: 'test-org-2', visibility_level: Organizations::Organization::PUBLIC)
  Organizations::OrganizationUser.create_organization_record_for(User.first.id, org2.id)
  ```
  1. Assign projects to new organizations
  • Find two projects that have pipelines with job artifacts: 
    Ci::JobArtifact.pluck(:project_id).uniq!
  • Reassign these projects to the newly created organizations, and create sample attestation files for their job artifacts: 
    # Replace with actual project, organization ids
new_project_orgs = [
    {project_id: 1, org_id: 1000},
    {project_id: 6, org_id: 1001}
]

new_project_orgs.each do |v|
  Project.find(v[:project_id]).namespace.update!(organization: v[:org_id])
  Project.find(v[:project_id]).update!(organization: v[:org_id])
end

new_project_orgs.map { |v| v[:project_id] }.each do |project_id|
Project.find(project_id).job_artifacts.each do |artifact|
    attestation = SupplyChain::Attestation.new(
      project_id: project_id,
      build_id: artifact.job_id,
      predicate_kind: :provenance,
      predicate_type: 'https://slsa.dev/provenance/v1',
      subject_digest: Digest::SHA256.hexdigest(artifact.id.to_s)
    )

    file = Tempfile.new(['attestation', '.json'])
    begin
      file.write({sample: (0...1024).map { (65 + rand(26)).chr }.join}.to_json)

      attestation.file = file
      attestation.save!
    ensure
      file.close
      file.unlink
    end
end
  1. Create a personal access token

Verify Replication

Click to expand

On primary node:

  1. Verify the files were uploaded to object storage in the uploads bucket.

  2. Verify the attestation was created and verification started:

    attestation.reload
attestation.supply_chain_attestation_state
# Should show verification_state as 'pending' or 'verification_succeeded'

On the Secondary Node:

  1. Check that the registry entry was created:

    Geo::SupplyChainAttestationRegistry.count
# Should be > 0

registry = Geo::SupplyChainAttestationRegistry.last
registry.supply_chain_attestation_id
# Should match the attestation ID from primary

  2. Trigger replication:

    Geo::SupplyChainAttestationReplicator.new(model_record_id: attestation.id).execute

  3. Verify the attestation file was replicated:

    registry.reload
registry.state
# Should be 'synced'

registry.verification_state
# Should be 'verification_succeeded'

  4. Verify the file was replicated in object storage on the secondary.

Primary Site Selective Checksumming by Organizations - Testing Steps

Click to expand

  1. In the primary GDK site: gdk switch 580365-implement-geo-support-of-supply-chain-attestations-behind-ff

  2. In the secondary GDK site: gdk switch 580365-implement-geo-support-of-supply-chain-attestations-behind-ff

  3. In the primary GDK site, open Rails console: bin/rails c

  4. Enable the FF: Feature.enable(:org_mover_extend_selective_sync_to_primary_checksumming)

  5. Enable the FF: Feature.enable(:geo_selective_sync_by_organizations)

  6. Get your current configuration:

    # Get your personal access token
export PRIVATE_TOKEN="your_personal_access_token"

# List all Geo sites to get the site ID
curl --header "PRIVATE-TOKEN: $PRIVATE_TOKEN" "http://localhost:3000/api/v4/geo_sites" | jq

# Store the site ID of the primary node
export SITE_ID=1  # Replace with your primary site ID

# Output organization objects for their IDs
bin/rails runner "pp Organizations::Organization.all"

# Store an organization ID for testing
export ORG_ID=1000  # Replace with your organization ID

  7. Enable selective checksumming by organization:

    # Enable selective checksumming by organization and select the specific organization
curl --request PUT \
  --header "PRIVATE-TOKEN: $PRIVATE_TOKEN" \
  --header "Content-Type: application/json" \
  --data '{
    "selective_sync_type": "organizations",
    "selective_sync_organization_ids": ['$ORG_ID']
  }' \
  "http://localhost:3000/api/v4/geo_sites/$SITE_ID"

  8. Verify the configuration:

    # Get the updated Geo site configuration and confirm that selective_sync_type is "organizations" and 
# organization_ids contains your organization ID
curl --header "PRIVATE-TOKEN: $PRIVATE_TOKEN" "http://localhost:3000/api/v4/geo_sites/$SITE_ID" | jq

  9. Wait a few minutes and verify the secondary site status

    # Get the updated site status and confirm that supply_chain_attestations_checksummed_count 
# matches the number of supply chain attestations that belong to your organization ID
curl --header "PRIVATE-TOKEN: $PRIVATE_TOKEN" "http://localhost:3000/api/v4/geo_sites/$SITE_ID/status" | jq

  10. Test with multiple organizations:

    export ORG_ID2=1001  # Replace with another organization ID

# Update to include multiple organizations
curl --request PUT \
  --header "PRIVATE-TOKEN: $PRIVATE_TOKEN" \
  --header "Content-Type: application/json" \
  --data '{
    "selective_sync_type": "organizations",
    "selective_sync_organization_ids": ['$ORG_ID','$ORG_ID2']
  }' \
  "http://localhost:3000/api/v4/geo_sites/$SITE_ID"

# Verify the update
curl --header "PRIVATE-TOKEN: $PRIVATE_TOKEN" "http://localhost:3000/api/v4/geo_sites/$SITE_ID" | jq

# Wait a few minutes and verify the Geo site status
curl --header "PRIVATE-TOKEN: $PRIVATE_TOKEN" "http://localhost:3000/api/v4/geo_sites/$SITE_ID/status" | jq

  11. Disable selective sync:

    # Reset back to no selective sync
curl --request PUT \
  --header "PRIVATE-TOKEN: $PRIVATE_TOKEN" \
  --header "Content-Type: application/json" \
  --data '{
    "selective_sync_type": ""
  }' \
  "http://localhost:3000/api/v4/geo_sites/$SITE_ID"

# Verify the update
curl --header "PRIVATE-TOKEN: $PRIVATE_TOKEN" "http://localhost:3000/api/v4/geo_sites/$SITE_ID"

# Wait a few minutes and verify the Geo site status
curl --header "PRIVATE-TOKEN: $PRIVATE_TOKEN" "http://localhost:3000/api/v4/geo_sites/$SITE_ID/status" | jq

# In the primary GDK site, disable the FF: 
bin/rails runner "pp Feature.disable(:geo_selective_sync_by_organizations)"

Secondary Site Selective Sync by Organizations - Testing Steps

Click to expand

  1. In the primary GDK site: gdk switch 580365-implement-geo-support-of-supply-chain-attestations-behind-ff

  2. In the secondary GDK site: gdk switch 580365-implement-geo-support-of-supply-chain-attestations-behind-ff

  3. In the primary GDK site, open Rails console: bin/rails c

  4. Enable the FF: Feature.enable(:geo_selective_sync_by_organizations)

  5. Get your current configuration:

    # Get your personal access token
export PRIVATE_TOKEN="your_personal_access_token"

# List all Geo sites to get the site ID
curl --header "PRIVATE-TOKEN: $PRIVATE_TOKEN" "http://localhost:3000/api/v4/geo_sites" | jq

# Store the site ID of the secondary node
export SITE_ID=2  # Replace with your secondary site ID

# Output organization objects for their IDs
bin/rails runner "pp Organizations::Organization.all"

# Store an organization ID for testing
export ORG_ID=1000  # Replace with your organization ID

  6. Enable selective sync by organization:

    # Enable selective sync by organization and select the specific organization
curl --request PUT \
  --header "PRIVATE-TOKEN: $PRIVATE_TOKEN" \
  --header "Content-Type: application/json" \
  --data '{
    "selective_sync_type": "organizations",
    "selective_sync_organization_ids": ['$ORG_ID']
  }' \
  "http://localhost:3000/api/v4/geo_sites/$SITE_ID"

  7. Verify the configuration:

    # Get the updated Geo site configuration and confirm that selective_sync_type is "organizations" and 
# organization_ids contains your organization ID
curl --header "PRIVATE-TOKEN: $PRIVATE_TOKEN" "http://localhost:3000/api/v4/geo_sites/$SITE_ID" | jq

  8. Wait a few minutes and verify the secondary site status

    # Get the updated site status and confirm that supply_chain_attestations_checksummed_count 
# matches the number of supply chain attestations that belong to your organization ID
curl --header "PRIVATE-TOKEN: $PRIVATE_TOKEN" "http://localhost:3000/api/v4/geo_sites/$SITE_ID/status" | jq

  9. Test with multiple organizations:

    export ORG_ID2=1001  # Replace with another organization ID

# Update to include multiple organizations
curl --request PUT \
  --header "PRIVATE-TOKEN: $PRIVATE_TOKEN" \
  --header "Content-Type: application/json" \
  --data '{
    "selective_sync_type": "organizations",
    "selective_sync_organization_ids": ['$ORG_ID','$ORG_ID2']
  }' \
  "http://localhost:3000/api/v4/geo_sites/$SITE_ID"

# Verify the update
curl --header "PRIVATE-TOKEN: $PRIVATE_TOKEN" "http://localhost:3000/api/v4/geo_sites/$SITE_ID" | jq

# Wait a few minutes and verify the Geo site status
curl --header "PRIVATE-TOKEN: $PRIVATE_TOKEN" "http://localhost:3000/api/v4/geo_sites/$SITE_ID/status" | jq

  10. Disable selective sync:

    # Reset back to no selective sync
curl --request PUT \
  --header "PRIVATE-TOKEN: $PRIVATE_TOKEN" \
  --header "Content-Type: application/json" \
  --data '{
    "selective_sync_type": ""
  }' \
  "http://localhost:3000/api/v4/geo_sites/$SITE_ID"

# Verify the update
curl --header "PRIVATE-TOKEN: $PRIVATE_TOKEN" "http://localhost:3000/api/v4/geo_sites/$SITE_ID"

# Wait a few minutes and verify the Geo site status
curl --header "PRIVATE-TOKEN: $PRIVATE_TOKEN" "http://localhost:3000/api/v4/geo_sites/$SITE_ID/status" | jq

# In the primary GDK site, disable the FF: 
bin/rails runner "pp Feature.disable(:org_mover_extend_selective_sync_to_primary_checksumming)"
bin/rails runner "pp Feature.disable(:geo_selective_sync_by_organizations)"

Database Queries

  • SupplyChain::Attestation.replicables_for_current_secondary(1..10000)

    • Raw SQL

      Click to expand 
      SELECT
    "slsa_attestations".*
FROM
    "slsa_attestations"
WHERE
    "slsa_attestations"."id" BETWEEN 1
    AND 10000
    AND "slsa_attestations"."project_id" IN (
        SELECT
            "projects"."id"
        FROM
            "projects"
        WHERE
            "projects"."namespace_id" IN (
                SELECT
                    "namespaces"."id"
                FROM
                    "namespaces"
                WHERE
                    "namespaces"."organization_id" IN (
                        SELECT
                            "geo_node_organization_links"."organization_id"
                        FROM
                            "geo_node_organization_links"
                        WHERE
                            "geo_node_organization_links"."geo_node_id" = 1
                    )
            )
    )

    • Query Plan: https://console.postgres.ai/gitlab/gitlab-production-main/sessions/47455/commands/143483

  • SupplyChain::Attestation.pluck_verifiable_ids_in_range(1..10000)

    • Raw SQL

      Click to expand 
      
SELECT
    "slsa_attestations"."id"
FROM
    "slsa_attestations"
WHERE
    "slsa_attestations"."id" BETWEEN 1
    AND 10000
    AND "slsa_attestations"."project_id" IN (
        SELECT
            "projects"."id"
        FROM
            "projects"
        WHERE
            "projects"."namespace_id" IN (
                SELECT
                    "namespaces"."id"
                FROM
                    "namespaces"
                WHERE
                    "namespaces"."organization_id" IN (
                        SELECT
                            "geo_node_organization_links"."organization_id"
                        FROM
                            "geo_node_organization_links"
                        WHERE
                            "geo_node_organization_links"."geo_node_id" = 1
                    )
            )
    )

    • Query Plan: https://console.postgres.ai/gitlab/gitlab-production-main/sessions/47455/commands/143484

Related to #580365 (closed)

Edited by Aaron Huntsman

Merge request reports