Skip to content

[Discussion] How to handle required work item types to exist in DB before app run

During the initial stages of the work item types implementation we had a discussion around how to solve the the problem of having some missing data in the DB that we require for the app to work in !84836 (comment 910369861)

If you are getting this error on your new instance

In this issue we have discussed whether we should be able to rely on data that is seeded into the database as part of the DB setup steps. The conclusion is that we should be able to do so, as expecting that some instances do load seeds, would force us to write defensive code in many places of the application.

So if you are getting this error, all you need to do is make sure you have properly seeded your database as described in https://docs.gitlab.com/omnibus/settings/database.html#seed-the-database-fresh-installs-only. If you have additional questions, please feel free to ask in this issue.

If you are getting this error on an existing instance

This should be rare if you have followed the steps described in https://gitlab-com.gitlab.io/support/toolbox/upgrade-path. But if you run into this, please post a comment in this issue and we will help you with the resolution. The steps required for the fix might depend on the state of your data.

Problem

For some work item features to work, we need some records to exist in the work_item_types table. At the moment we have 2 mechanisms for this to happen.

  1. Production fixtures seed the db with the required work item type and widget definition information in https://gitlab.com/gitlab-org/gitlab/-/blob/130fc0c94587ecb5e708fffceb7e89a3c4eed24d/db/fixtures/production/003_create_base_work_item_types.rb
  2. Issue model code that will upsert the required data in the event a required type is missing from the DB in https://gitlab.com/gitlab-org/gitlab/-/blob/130fc0c94587ecb5e708fffceb7e89a3c4eed24d/app/models/work_items/type.rb#L74

Production fixtures (1) would be ideal as a single place to create this data only once for new instances, but there are a few reasons why this is not 100% reliable as described in #353552 (comment 901589414).

Issue model code (2) solved this issue for the most part in legacy code, since upserting the type information would happen the first time an issue of a given type was created. From this moment forward on existing instances we use migration to add new types or widget definitions.

With the existing implementation there's a still a small chance of failure in places like https://gitlab.com/gitlab-org/gitlab/-/blob/130fc0c94587ecb5e708fffceb7e89a3c4eed24d/app/graphql/resolvers/work_items/types_resolver.rb#L16 where we list work item types in order to create work items. If the DB is not seeded, and no issue is created before trying to create a work item through the work item GraphQL API, this won't work since the type ID is a required argument and they might not exist until created through the legacy issue creation interface.

Another caveat comes in migrations as described in !128307 (comment 1527736002). We currently write very defensive migrations designed not to fail if for whatever reason the type data in the DB is not as expected, but this could come with it's own problems since work item type data might be inconsistent for some instances and might be hard to debug errors. So we should probably fail in migrations if for whatever reason we cannot migrate as expected? Since we upsert this data, the fix could be as simple as running a single line of code in the rails console

Required data insert volume

At the time of writing we have 9 work item types defined in our DB. That means 9 rows are upserted on the work_item_types table. In addition to those, we create widget definitions for each of these 9 types. I think the best place to visualize this information is in https://gitlab.com/gitlab-org/gitlab/-/blob/b4ede883ab6101d8bdf7b0de5e1ca81d96b6ce1f/lib/gitlab/database_importers/work_items/base_type_importer.rb#L28.

From the base_importer we can see that we upsert 9 rows in the work_item_types table and 99 rows in the work_item_widget_definitions table. We don't expect this number to grow too fast, but it definitively can if we introduce new features based on the work_items architecture.

work_item_types table

Each row in this table is upserted with values in 5 columns

  1. name: text 255 char max
  2. icon_name: text 255 char max
  3. created_at: timestamp with time zone
  4. updated_at: timestamp with time zone
  5. base_type: smallint

work_item_widget_definitions table

Each row in this table is upserted with values in 3 columns

  1. work_item_type_id: bigint
  2. name: text
  3. widget_type: smallint

Possible solutions

  1. Can there be an initialization step that checks for the presence of the required data? Rails initializers might not be possible as described in !84836 (comment 912330965)
  2. Check for types presence where needed as described in !84836 (comment 920396233). This might require this check to be done many many times in instances like .com for example every time a new issue is created (as we need to list the work item types before creation). Might not be a big deal if we use some sort of redis caching mechanism?

Thought: Would it be something good that we can assume that the types exist in the DB everywhere in the application? Doing existence checks before creating a feature might not be ideal as it requires dev's knowledge on this particularity?

Edited by Mario Celi