Replace ruby parser by python parser
Problem to solve
In build: add script for ingestion to vertex ai se... (!757 - merged), we reused the existing ruby parser as-is for keeping the original behavior:
- https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/lib/gitlab/llm/embeddings/utils/docs_content_parser.rb.
- https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/lib/gitlab/llm/embeddings/utils/base_content_parser.rb
- https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/app/workers/llm/embedding/gitlab_documentation/create_embeddings_records_worker.rb
However, AI Gateway is python project so it's more natural to introduce the parser in python.
Proposal
Replace ruby parser by python parser