POC - Override due dates for sample data
What does this MR do?
In order to precalculate and override due_dates for https://gitlab.com/gitlab-com/www-gitlab-com/-/issues as described in gitlab-com/www-gitlab-com#7397 (comment 416758822),
we can do the precalculation of median date in pre-import work, where we could read both issues.ndjson
and milestones.ndjson
. (This will work only for ndjson exports).
We are precalculating the dynamic date (median date from all issues and milestones), and then we continue calculating new values on the fly while importing each relation.
This is explained in 1st approach here
Pros
- It's simple. The JSON is not complex and nested, we only need to read issues.ndjson and milestones.ndjson, which contains a really simple structure. It's easy and simple to extract all
due_dates
and calculate the median value. - It's performant. We update/override actual due_dates values as we import them. So actual objects are persisted with precalculated due dates. No additional reads/writes to the database.
Cons
- NDJSON reader was designed to consume relations. Once a relation is consumed, you cannot consume it again. The whole idea around NDJSON was to consume whole import by reading it in one go. Now we need to read all issues and milestones in advance, in order to calculate the
median date
, and then we must read each relation (issue/milestone) again when we import/restore them.
Screenshots
Does this MR meet the acceptance criteria?
Conformity
-
Changelog entry -
Documentation (if required) -
Code review guidelines -
Merge request performance guidelines -
Style guides -
Database guides -
Separation of EE specific content
Availability and Testing
-
Review and add/update tests for this feature/bug. Consider all test levels. See the Test Planning Process. -
Tested in all supported browsers -
Informed Infrastructure department of a default or new setting change, if applicable per definition of done
Security
If this MR contains changes to processing or storing of credentials or tokens, authorization and authentication methods and other items described in the security review guidelines:
-
Label as security and @ mention @gitlab-com/gl-security/appsec
-
The MR includes necessary changes to maintain consistency between UI, API, email, or other methods -
Security reports checked/validated by a reviewer from the AppSec team
Edited by Grant Young