PoC: Support Flipper API in GitLab Feature Flag and connect to it via Flipper HTTP adapater (Hybrid)

TODO:

• Check the difference between Flipper API schema and Unleash API schema
• Connect to Flipper API in GitLab Feature Flag

Key points

• If nothing is configured, Feature class behaves exactly the same with the current behavior.
• If http adapter is configured, you can optionally control the flag state with GitLab Feature Flag.
• If a flag data is persisted in both database and http adapater, http adapter takes precedence over AR adapter.
• No migration required from AR adapter to HTTP adapter. We can gradually switch the feature flag workflow to GitLab feature flag.
• No immediate workflow change required in daily developments. Engineers still can use Feature class as they do today.
• Chatops (Feature) updates flag states in database. HTTP adapter is read-only mode and doesn't accept CRUD via chatops yet.
• On-premises/local development instance still uses database as persistent store. We don't need documentation change for alpha features that hidden by a flag.
• We need to keep maintaining schema mapping between unleash and flipper.
• If there is an unsupported gate in GitLab Feature Flag (e.g. percentage_of_time), engineers should use Flipper with ActiveRecord adapter instead GitLab Feature Flag (HTTP adapter).
• TODO: Cache isn't invalided in Feature if engineers updated a flag state in GitLab Feature Flag. GitLab Feature Flag must request to Feature to invalidate the cache on a specific feature key. (Maybe we can create a new endpoint in API::Features?)

Cache invalidation

Feature Flag Server:
  - GET api/v4/flipper/features

Workhorse:
  CacheInvalidation:
    - Polling flag data from Feature Flag Server (every 15sec)
    - If a feature state is changed, it request rails to invalidate the L1/L2 cache.
    - New endpoints: `api/v4/features/clear_cache` or `api/v4/features/clear_cache/:name`

GitLab-Rails
  - L1 Process Memory
  - L2 Redis
  - HTTP adapter (which connects to GitLab Feature Flag)
  - Active Record

Classes

• Flipper::Adapters::MultiPersistentLayer ... The adapter to combine multiple persistent layers and return flag data from the highest precedence.
• Flipper::Adapters::ReadonlyHttp ... Readonly adapter for HTTP

Terminologies

• GitLab feature flag ... Feature Flag product provided by GitLab. It mainly uses Unleash as feature flag engine.
• Feature library ... Internal library in GitLab-Rails to control FFs for our development. It mainly uses Flipper as feature flag engine.

GET /api/:version/feature_flags/unleash/:project_id(.:format) - 
GET /api/:version/feature_flags/unleash/:project_id/features(.:format) - Get a list of features (deprecated, v2 client support)
GET /api/:version/feature_flags/unleash/:project_id/client/features(.:format) - Get a list of features
POST /api/:version/feature_flags/unleash/:project_id/client/register(.:format) - 
POST /api/:version/feature_flags/unleash/:project_id/client/metrics(.:format) - 
GET /api/:version/feature_flags/flipper/:project_id/features(.:format) - 
GET /api/:version/feature_flags/flipper/:project_id/features/:feature_name(.:format) - 

• curl "http://local.gitlab.test:8181/api/v4/feature_flags/unleash/91/features?instance_id=jQErix5G9tpcwnKN3zE4&app_name=local.gitlab.test" | jq
• curl "http://local.gitlab.test:8181/api/v4/feature_flags/flipper/91/features?instance_id=jQErix5G9tpcwnKN3zE4&app_name=local.gitlab.test" | jq

boolean: true
actors: Project:9927571
percentage_of_actors: 22
percentage_of_time: 100

QA

Case: When a flag key exists in HTTP and doesn't exist in database

Expectation: The flag can be controlled in GitLab Feature Flag Result:

Case: When a flag key doesn't exist in HTTP but exists in database

Expectation: The flag can be controlled via Feature class (i.e. chatops) Result:

Case: When a flag key doesn't exist in both HTTP and database stores

Expectation: The flag can be controlled in GitLab Feature Flag Result:

Schema difference

Flipper

GET /features

features:
  - key:
    state: 
    gates:
      - key: (boolean/groups/actors/percentage_of_actors/percentage_of_time)
        name:
        value:

GET /features/{feature_name}

key:
state: 
gates:
  - key:
    name:
    value:

Unleash

GET features

version:
features:
  - name:
    description:
    enabled:
    strategies:
      - name:
        parameters:

(Unleash doesn't have an single get API)