`cache_pages_domain_api` feature flag cause failures in staging

Summary

During the rollout of cache_pages_domain_api feature flag, the gitlab pages qa tests on staging started to fail.

After some investigation with @vshushlin, we found out that the application settings are different between rails nodes (where the cache is created) and sidekiq nodes (where the cache is invalidated). Since we us a hash for the application settings in the cache key, the sidekiq job never finds the cache key to invalide it.

Related:

• Failure in browser_ui/3_create/pages/new_static... (gitlab#376332 - closed)
• [Feature flag] Rollout of `cache_pages_domain_api` (gitlab#364127 - closed)

Possible fixes

{
    "items" : [
{
"option": "🅰️",
"description": "Instead of using the application settings hash in the cache key we could insert the hash in the cached value and validate it when reading the cache. This way the app settings is not part of the cache key, so it should get invalidated in the sidekiq nodes.",
"cons": "validating the cache manually based on the cached value."
},
{
"option": "🅱️",
"description": "Instead of using the application settings hash in the cache key we could have a two layers cache. The main cache key would save the app settings hash, which would be the key to the payload itself.",
"cons": "two reads on redis to find the cached value."
}
]
}