Generate sitemap dynamically
What does this MR do?
In !45645 (merged) we created a POC that generates a sitemap.xml file for .com and the gitlab-org group. It basically creates the file in the /public folder.
Nevertheless, it seems we don't have enough access rights to write on /public and when the CM gitlab-com/gl-infra/production#2939 (closed) was requested, we realize that the server where the console runs is different from the web servers and it would be a hard task to achieve.
Therefore, to circumvent this and since the number of groups and projects is finite in the POC, we're going to generate it dynamically through a controller action. This way we will be able to evaluate the POC without too many troubles for infra.
The question is should we implement some sort of caching?
Well, the first thing we have to notice is that, if we cache this, if the visibility of one of the project or subgroups changes we have to invalidate the cache in order to return an accurate response. This means that we would need to retrieve all the projects and subgroups, build a cache key with them, and then store the file. This means that we have to include the database time in this tradeoff.
The file size generated is 578KB, which will need to go into the cache.
The database time involved is around 4 ms and the total time of the endpoint is 1 second.
I think there is no need to cache this sitemap. First, because the total operation time is acceptable and secondly because we are storing quite some data in the cache (and several might be added there until the record expires).
Refs #270051 (closed)