Skip to content

Changes

Page history
2024-06-12 entry authored by Marco Villegas's avatar Marco Villegas
Show whitespace changes
Inline Side-by-side
DevBlog/2024-06-12.md 0 → 100644
View page @ ecdd155d
# 2024-06-12
## tl;dr
Things are getting into the final stages 🎉
A new round of improvements was added in multiple codebases since the last entry
here.
On the data migration side, a better high water mark was incorporated, to react
better on the "catch-up" migration run.
A new drush command for drupal.org was implemented, to do data integrity
verification over the whole set of migrated users, comparing Drupal to Keycloak.
An extra change was also proposed for `openid_connect` module to allow
synchronizing the username and email from the identity provider.
The sub-site `authmap` table data population script was split in two, using a
CSV export, and the other part receiving that export, for easier infrastructure
requirements.
A new full test data migration has been initiated, and it is still in progress.
If it finishes correctly, without errors, the data migration could be scheduled.
The hope is that there are no more critical items to focus on.
## Better high water mark
The ad-hoc implemented highi water functionality for the user data migration has
been improved a bit.
It now supports Drupal 7 `tfa` and `tfa_basic` modules time stamp metadata around
configuration, seeds, and recovery codes.
More details at [#56][sso-i-56].
[sso-i-56]: https://gitlab.com/drupal-infrastructure/sso/drupalorg-sso/-/issues/56
## Full set integrity check
There was already individual integrity checks for a given drupal user.
That relied on per-user data retrieval.
Hence, doing it over the whole set was prohibitively long.
A new bulk data integrity checking drush command, intended to be run from the
drupal.org site has been added to be able to verify the data migration after it
completes.
It uses KC REST API to retrieve user data, queries the drupal database, and
compares all relevant information.
More details available at [#65][sso-i-65].
[sso-i-65]: https://gitlab.com/drupal-infrastructure/sso/drupalorg-sso/-/issues/65
## Post-migration data synchronization with KC
As some data will be needed to be kept in sync, depending on which bit it is; at
[#67][sso-i-67], the current behavior and possible next steps where documented.
At the end, a related change was proposed into `openid_connect` module, so it is
possible to get the changes over usernames and emails from KC.
More details at the upstream issue, [#3452530: Add optional username and email synchronization from identity provider][do-i-3452530], and [related MR !111][openid-connect-mr-111].
In summary, `openid_connect` user data synchronization on login will be handling
the KC to drupal changes, and there is already some `drupalorg` user hooks that
update KC for a few data points.
For the initial expected workflow, that is likely enough.
[do-i-3452530]: https://www.drupal.org/project/openid_connect/issues/3452530
[openid-connect-mr-111]: https://git.drupalcode.org/project/openid_connect/-/merge_requests/111
[sso-i-67]: https://gitlab.com/drupal-infrastructure/sso/drupalorg-sso/-/issues/67
## Sub-site `authmap` table data population changes
To avoid the need to setup special SQL access on different drupal sites via an
extra read-only connection to drupal.org, the `authmap` table data population
script has been split into two scripts.
First, after the user data migration finishes, the data will be exported from
drupal.org into a CSV file.
Then, the second script will be executed on every sub-site, for now
localize.d.o, to populate the `authmap` table from the exported CSV.
More details at [related comment at #64][sso-i-64-1936165882].
[sso-i-64-1936165882]: https://gitlab.com/drupal-infrastructure/sso/drupalorg-sso/-/issues/64#note_1936165882
## A second full set test migration run
With most of the pieces already in place, and to verify that the previous foudn
problems are fixed, a new full set test migration run was scheduled.
After re-initializing `drupal` KC instance to match `dev-drupal` configuration
changes, and remove all users, and a minimal test run, the new migration was
started.
There was a glitch, related to KC REST API timing out in one of the loads, but
that served as a good opportunity to test an interruption over the run; after
investigating a bit, the migration was re-started from the point it arrived to.
The data migration is still in progress, and expected to finish relatively soon
🤞.
More details available at [#68][sso-i-68].
[sso-i-68]: https://gitlab.com/drupal-infrastructure/sso/drupalorg-sso/-/issues/68
## Other changes
A gitlab environment was tested together with the `sso-drupal` devwww site, and
the `dev-drupal` KC instance, and it worked OK as expected.
Basically, if `bakery` keeps running, the existing drupal to gitlab
authentication keeps working OK, after logging in via KC into the drupal site.
A bit more details at [#62][sso-i-62]
KC Terms and conditions were disabled, details at [#63][sso-i-63].
Improved a bit the KC local run on the drupal-theme codebase, by [verifying a
bit downloaded providers][drupal-theme-commit-ef1bd2d].
Also, a few releases were created to better identify the codebase points that the
migration is relying on right now.
Namely [drupalorg-user-migration `1.0.0` git
tag][drupalorg-user-migration-1.0.0] and [drupal-theme `1.0.1` release][drupal-theme-1.0.1-release].
[drupal-theme-1.0.1-release]: https://gitlab.com/drupal-infrastructure/sso/drupal-theme/-/releases/1.0.1
[drupal-theme-commit-ef1bd2d]: https://gitlab.com/drupal-infrastructure/sso/drupal-theme/-/commit/ef1bd2d1f0c0d4caaaae20ce6f1c5ecfd868a685
[drupalorg-user-migration-1.0.0]: https://gitlab.com/drupal-infrastructure/sso/drupalorg-user-migration/-/tags/1.0.0
[sso-i-62]: https://gitlab.com/drupal-infrastructure/sso/drupalorg-sso/-/issues/62
[sso-i-63]: https://gitlab.com/drupal-infrastructure/sso/drupalorg-sso/-/issues/63
## Changes
New commits on [keycloak-realm-configuration][keycloak-realm-configuration]'s
`next` branch, now also over `main`.
- 50c0028 #51 Use the new d7-username validator on username
- 2520622 #17 Use d.o session timeout in KC
- 17ee33a #63 Disable terms and conditions
- 79431bc #67 Enable update email reequired action
- 88803e2 #68 Enable admin events logging
- 3a3df8c #68 Temporarily disable username validation
New commits on [drupal-theme][drupal-theme]'s `next` branch, now also on `main`,
and released as `1.0.1`.
- cb8f041 Bump to a first version
- ef1bd2d Use drupal-7-extension 1.0.0 and do a minimal check on retrieve providers
- 95ae548 Follow same local install pattern for drupal-theme provider
- b44be5d Use 1:1 port mapping for adminer helper container
- e26926c #68 Bump version
New commits on [drupalorg-user-migration][drupalorg-user-migration]'s `next`
branch, now also over `main`, and released as `1.0.0`.
- e0efb2d #56 Validate a bit more required source environment variables
- bd43d17 #56 Detect TFA seed and configuration changes on highwater mark
- 526bc6e #56 Detect TFA recovery code changes on highwater mark
- bb89231 #64 Split sub-site authmap populate into two steps
- 5f4cbb5 Add export database helper for lando
No new commits on [drupal-7-extension-helper][drupal-7-extension-helper]'s
`next` branch.
New changes on [drupalorg][drupalorg] module, all related to [#3448936: Add keycloak user integrity check command][do-i-3448936], pending to be reviewed and merged upstream from `3448936-kc-integrity-check` branch, as submitted to [drupalorg's MR !260][drupalorg-mr-260].
- 7311c097 Add Keycloack helper to get users
- 0c4a0496 Start drupalorg-keycloak-integrity-check drush command
- 31a35786 Skip intermediary variable on drupal user data retrieval
- b920a73d Unify the diff data into one place
- e0949441 Write the differences found to a CSV file passes as argument
- c199e75f Make username comparison case insensitive
- f8ed6568 Check email integrity
- e8ae3c26 Implement exit status on the integrity command
- 2834404a Check status integrity
- 144f6621 Check created timestamp integrity
- 19f4fb32 Write differences incrementally instead storing them and then writing
- 5b8dcd4e Check timezone integrity
- b07d6a9a Exclude anonymous user
- 4070682a Add wait and retry on Keycloak users retrieval helper
- 68976b1c Check both first and last name integrity
- 55f4e955 Release a bit of memory on KC user retrieval for integrity check
- 630937c2 Check email verified flag integrity
- 1f9d3316 Explain better time limit reached error on KC get users helper
- 13fb31ab Add KC helper to get users in a group
- f31b912c Check confirmed role integrity
- dabc19d2 Check multi-factor enabled status integrity
- 1c7122c5 Check user picture integrity
- 85ca28ff Change image style to add to KC on user update to match data migration code
- cad512d2 Fix typo
- 4915f44e Compare strings for time zone
- 59a1f16a Improve picture comparison
[do-i-3448936]: https://www.drupal.org/project/drupalorg/issues/3448936
[drupal-7-extension-helper]: https://gitlab.com/drupal-infrastructure/sso/drupal-7-extension-helper
[drupal-theme]: https://gitlab.com/drupal-infrastructure/sso/drupal-theme
[drupalorg-mr-260]: https://git.drupalcode.org/project/drupalorg/-/merge_requests/260
[drupalorg-user-migration]: https://gitlab.com/drupal-infrastructure/sso/drupalorg-user-migration
[drupalorg]: https://www.drupal.org/project/drupalorg
[keycloak-realm-configuration]: https://gitlab.com/drupal-infrastructure/sso/keycloak-realm-configuration
## Next steps
Wait for the current full set test migration run at [#68][sso-i-68] to finish.
If all goes well, we may want to start planning for doing this for production
instances.
There is still one development issue to implement, [#66][sso-i-66].
That may not be a blocker, since it can always be done manually, and user
deletion is not a extremely common.
[sso-i-66]: https://gitlab.com/drupal-infrastructure/sso/drupalorg-sso/-/issues/66
[sso-i-68]: https://gitlab.com/drupal-infrastructure/sso/drupalorg-sso/-/issues/68
[sso-kanban]: https://gitlab.com/drupal-infrastructure/sso/drupalorg-sso/-/boards/7446582