Fix Okta pagination results to include first 200 results
During the usage of the API, when pulling 2,284 users, only 2,148 users were returned from the paginated /users API endpoint.
After triage, it was discovered that the get method checks for pagination, and if true, passes the paginated URL to start getting paginated results with the next cursor, however the original 200 results before the cursor is not included in the final return results.
This has the I/O cost of getting the first 200 results twice, however this is a reasonable "cost" relative to adding a lot more code.
Before
$paginated_url = $this->generateNextPaginatedResultUrl($response->headers);
$paginated_results = $this->getPaginatedResults($paginated_url);
After
$paginated_results = $this->getPaginatedResults($this->base_url . $uri);
Reference
public function get(string $uri, array $request_data = []): object|string
{
try {
// Utilize HTTP to run a GET request against the base URL with the
// URI supplied from the parameter appended to the end.
$request = Http::withHeaders($this->request_headers)
->get($this->base_url . $uri, $request_data);
// Parse API Response and convert to returnable object with expected format
$response = $this->parseApiResponse($request, false);
$this->logResponse('get', $this->base_url . $uri, $response);
// If the response is a paginated response
if ($this->checkForPagination($response->headers) == true) {
// Get paginated URL and send the request to the getPaginatedResults
// helper function which loops through all paginated requests
$paginated_url = $this->generateNextPaginatedResultUrl($response->headers);
$paginated_results = $this->getPaginatedResults($paginated_url);
// The $paginated_results will be returned as an object of objects
// which needs to be converted to a flat object for standardizing
// the response returned. This needs to be a separate function
// instead of casting to an object due to return body complexities
// with nested array and object mixed notation.
$request->paginated_results = $this->convertPaginatedResponseToObject($paginated_results);
// Unset property for body and json
unset($request->body);
unset($request->json);
// Parse API Response and convert to returnable object with expected format
// The checkForPagination method will return a boolean that is passed.
$response = $this->parseApiResponse($request, true);
}
return $response;
} catch (\Illuminate\Http\Client\RequestException $exception) {
return $this->handleException($exception, get_class(), $uri);
}
}
public function getPaginatedResults(string $paginated_url): array
{
// Define empty array for adding API results to
$records = [];
// Perform API calls while $api_url is not null
do {
// Get the record
$request = Http::withHeaders($this->request_headers)
->get($paginated_url);
$response = $this->parseApiResponse($request);
$this->logResponse('get', $paginated_url, $response);
// Loop through each object from the response and add it to
// the $records array
foreach ($response->object as $api_record) {
$records[] = $api_record;
}
// Get next page of results by parsing link and updating URL
if ($this->checkForPagination($response->headers)) {
$paginated_url = $this->generateNextPaginatedResultUrl($response->headers);
} else {
$paginated_url = null;
}
} while ($paginated_url != null);
return $records;
}
Edited by Jeff Martin