Read every page of system hooks in the gitlab_system_hooks data source

What does this MR do and why?

data "gitlab_system_hooks" sends a single GET /hooks and keeps what comes back. GitLab paginates that endpoint (present paginate(SystemHook.all) in lib/api/system_hooks.rb) and answers 20 hooks per page when no per_page is given, so on an instance with more than 20 system hooks the data source returns the first 20, and nothing in the result says the list was cut short.

This MR reads the list with gitlab.ScanAndCollect, the way gitlab_project_hooks and gitlab_group_hooks already read theirs, so the data source follows the pages until GitLab says there are none left. The schema does not change.

The basic acceptance test counted the hooks already on the instance with the same single request, so its offset stopped at 20 as well: once the instance holds more than 20 system hooks, that test expects the wrong count whether or not the data source pages. Both tests now count the existing hooks over every page through a small helper. The new TestAccDataSourceGitlabSystemHooks_pagination makes sure the instance holds at least 21 system hooks, one more than a page holds, creating only the ones that are missing, so it creates none when 21 or more are already there. It checks that the data source returns every hook on the instance, and the URL of the last hook it created when it created any. It runs non-parallel like the basic test, since system hooks belong to the whole instance.

How to verify

Against a local make testacc-up instance:

unset GITLAB_TOKEN && make testacc RUN=TestAccDataSourceGitlabSystemHooks_

With the data source as it is on main, the new test fails with Attribute 'hooks.#' expected "21", got "20". With this change both tests pass.

What I ran (GitLab CE 19.4.1, Go 1.26.5)

New tests with the data source as it is on main, on an instance with no system hooks:

=== RUN   TestAccDataSourceGitlabSystemHooks_basic
--- PASS: TestAccDataSourceGitlabSystemHooks_basic (3.14s)
=== RUN   TestAccDataSourceGitlabSystemHooks_pagination
    datasource_gitlab_system_hooks_test.go:74: Step 1/1 error: Check failed: Check 1/2 error: data.gitlab_system_hooks.this: Attribute 'hooks.#' expected "21", got "20"
--- FAIL: TestAccDataSourceGitlabSystemHooks_pagination (8.73s)

This branch, make testacc RUN=SystemHook:

--- PASS: TestAccDataSourceGitlabSystemHook_basic (4.49s)
--- PASS: TestAccDataSourceGitlabSystemHook_signingToken (3.07s)
--- PASS: TestAccDataSourceGitlabSystemHooks_basic (3.85s)
--- PASS: TestAccDataSourceGitlabSystemHooks_pagination (9.86s)
--- PASS: TestAccGitlabSystemHook_noToken (3.78s)
--- PASS: TestAccGitlabSystemHook_basic (4.29s)
--- PASS: TestAccGitlabSystemHook_signingToken (4.30s)
--- PASS: TestAccGitlabSystemHook_tokenPreservedOnUpdate (4.96s)
--- PASS: TestAccGitlabSystemHook_upgradeFromSDKToFramework (9.13s)

Both data source tests with system hooks already on the instance, counting the hooks the pagination test created from the instance's own API log:

Hooks already there Hooks the pagination test created This branch Same tests, data source as on main
0 21 both pass hooks.# expected "21", got "20"
5 16 both pass not run
20 1 both pass hooks.# expected "21", got "20"
21 0 both pass hooks.# expected "21", got "20"
25 0 both pass hooks.# expected "25", got "20"

The last column is the pagination test. In the rows with 20 or more hooks already there, the basic test fails on main's data source too, since it now expects every hook on the instance.

The basic test as it is on main, with 25 system hooks created on the instance beforehand:

main as is:
    Check 1/4 error: data.gitlab_system_hooks.this: Attribute 'hooks.#' expected "22", got "20"
main's test with the paging data source:
    Check 1/4 error: data.gitlab_system_hooks.this: Attribute 'hooks.#' expected "22", got "27"

Against a stand-in server answering GET /api/v4/hooks with 25 hooks, 20 per page and the X-* pagination headers, the data source made one request and stored 20 hooks before the change. After it, each read made two requests (/api/v4/hooks and /api/v4/hooks?page=2) and stored all 25.

make reviewable finishes cleanly and changes no tracked file, and make lint-golangci (0 issues), lint-tfprovider, lint-examples-tf, lint-examples-sh and lint-custom pass.

Related to !3203 (merged), which fixed the same first-page-only read in gitlab_group_provisioned_users, and !2904 (merged), which moved the data sources that paged by hand to ScanAndCollect.

Edited by José M. Requena Plens

Merge request reports

Loading
Loading