feat: Add unit primitive for generate_embeddings_codebase

What does this MR do and why?

Adds a unit primitive for generate_embeddings_codebase. The unit primitive will be used in implementing the Codebase as Chat Context project. See blueprint.

References

Please include cross links to any resources that are relevant to this MR. This will give reviewers and future readers helpful context to give an efficient review of the changes introduced.

• gitlab-org/gitlab#538567 (closed)

Diff

main...538567-add-generate-embeddings-codebase-unit-primitive

Checklist

• If changes were made in /config or src/ruby folder
  • Change the VERSION constant to a minor version in src/ruby/lib/gitlab/cloud_connector/version.rb (according to SemVer):
    • bump a major version if the change is breaking the compatibility (changing the interface, removing Unit Primitives that are used by the backends, etc). Make sure all backends that use the library are prepared for the update.
    • bump a minor version if the change is compatible but adding a new feature (Unit Primitives added, the interface is expanded with an optional argument, etc)
    • bump a patch version if it's a bugfix, adding docs, tests, or a similar change that it is safe to auto-apply by the backends who use the library.
  • Run bundle lock in src/ruby and commit the change to Gemfile.lock.
• If changes were made in /config or src/python folder
  • Change the version: in src/python/pyproject.toml file (according to SemVer).
    • bump a major version if the change is breaking the compatibility (changing the interface, removing Unit Primitives that are used by the backends, etc). Make sure all backends that use the library are prepared for the update.
    • bump a minor version if the change is compatible but adding a new feature (Unit Primitives added, the interface is expanded with an optional argument, etc)
    • bump a patch version if it's a bugfix, adding docs, tests, or a similar change that it is safe to auto-apply by the backends who use the library.
  • Run poetry lock --no-update --verbose --no-cache and commit the change to poetry.lock
• Ensure the diff link above is up-to-date.
• Create an MR on depending Gitlab projects to bump the version.