POC create sitemap for gitlab-org group and subgroup including public projects
The idea behind this issue is to analyze how useful a sitemap including all public projects in GitLab.com would be to increase the search crawling engines.
For this, we're going to start generating a sitemap for the gitlab-org group and subgroups including their public projects. This sitemap doesn't need an index because the number of links generated is below the high limit established for sitemaps (50.000 links).
Per each group we're going to render the following links:
- group overview
- epics (in case the group has them)
- issues
- merge requests
- packages and registries
Per each public project we're going to generate:
- project overview
- issues
- merge requests
- packages and registries
- snippets
For the purpose of this POC, before generating the sitemap, we need to retrieve first how many of the URLs generated are already indexed. Then, comparing the previous indexed files and the new ones we can measure how effective the sitemap could be for all public project and groups in .com.