Retry failed internal API requests

What does this MR do and why?

Currently if an internal AI request fails because of status code != 200, we fail CI job immediately. This MR changes this logic:

• checks if status code is retryable (typically 429 code), if yes it retries the request with a delay
• whole CI job is failed no matter what error occurs (e.g. if parsing of JSON response fails)

Related to #25 (closed)

How to test

• make sure internal API returns retryable error, e.g. with the following change in Rails:

diff --git a/ee/lib/api/internal/ai/x_ray/scan.rb b/ee/lib/api/internal/ai/x_ray/scan.rb
index 4e168dd33f55..8a8229c61ea0 100644
--- a/ee/lib/api/internal/ai/x_ray/scan.rb
+++ b/ee/lib/api/internal/ai/x_ray/scan.rb
@@ -13,6 +13,7 @@ class Scan < ::API::Base
           helpers ::API::Helpers::CloudConnector
 
           before do
+            too_many_requests!('Please retry later.', retry_after: 1.minute)
             authenticate_job!
             not_found! unless x_ray_enabled_on_instance?
             unauthorized! unless x_ray_available?

• run scanner e.g. in x-ray repo directory:

honza@fedora /opt/gitlab-development-kit/repository-x-ray (jp-retry-fetch)$ CI_JOB_TOKEN=0 CI_JOB_ID=0 CI_API_V4_URL=http://192.168.1.8:3000/api/v4 bin/x-ray-scan
2024/04/22 09:15:50 INFO Scanning in /opt/gitlab-development-kit/repository-x-ray
2024/04/22 09:15:50 INFO Found go.mod at /opt/gitlab-development-kit/repository-x-ray
2024/04/22 09:15:50 INFO Found go.mod at /opt/gitlab-development-kit/repository-x-ray/internal/testdata/dep_files
2024/04/22 09:15:50 INFO Found package.json at /opt/gitlab-development-kit/repository-x-ray/internal/testdata/dep_files
2024/04/22 09:15:50 INFO Found Gemfile.lock at /opt/gitlab-development-kit/repository-x-ray/internal/testdata/dep_files
2024/04/22 09:15:50 INFO Found pyproject.toml at /opt/gitlab-development-kit/repository-x-ray/internal/testdata/dep_files
2024/04/22 09:15:50 INFO Found requirements.txt at /opt/gitlab-development-kit/repository-x-ray/internal/testdata/dep_files
2024/04/22 09:15:50 INFO Found environment.yml at /opt/gitlab-development-kit/repository-x-ray/internal/testdata/dep_files
2024/04/22 09:15:50 INFO Found composer.json at /opt/gitlab-development-kit/repository-x-ray/internal/testdata/dep_files
2024/04/22 09:15:50 INFO Found pom.xml at /opt/gitlab-development-kit/repository-x-ray/internal/testdata/dep_files
2024/04/22 09:15:50 INFO Found build.gradle at /opt/gitlab-development-kit/repository-x-ray/internal/testdata/dep_files
2024/04/22 09:15:50 INFO Found build.gradle.kts at /opt/gitlab-development-kit/repository-x-ray/internal/testdata/dep_files
2024/04/22 09:15:50 INFO Found ConsoleApp1.csproj at /opt/gitlab-development-kit/repository-x-ray/internal/testdata/dep_files/ConsoleApp1
2024/04/22 09:15:50 INFO Found conanfile.txt at /opt/gitlab-development-kit/repository-x-ray/internal/testdata/dep_files
2024/04/22 09:15:50 INFO Found conanfile.py at /opt/gitlab-development-kit/repository-x-ray/internal/testdata/dep_files
2024/04/22 09:15:50 INFO Found vcpkg.json at /opt/gitlab-development-kit/repository-x-ray/internal/testdata/dep_files
2024/04/22 09:15:50 INFO Scanning go.mod
2024/04/22 09:15:50 INFO Found 4 Go packages
2024/04/22 09:15:50 INFO Scanning internal/testdata/dep_files/go.mod
2024/04/22 09:15:50 INFO Found 2 Go packages
2024/04/22 09:15:50 INFO Getting Go packages descriptions. This can take a while...
2024/04/22 09:15:50 INFO Processing batch... (1/1)
2024/04/22 09:15:50 WARN Request failed (code 429), will retry in 3s
2024/04/22 09:15:53 WARN Request failed (code 429), will retry in 9s
2024/04/22 09:16:03 WARN Request failed (code 429), will retry in 27s
2024/04/22 09:16:30 ERROR Failed to get completions. Reached maximum retries (3), giving up