Containers don't see secret changes that happen after their pod is started due to initContainers
This was seen today during the cloud native demo, and took a little bit of further investigation to find the cause.
The demo can be seen here: https://www.youtube.com/watch?v=veDzVX-R58E
Essentially what happened was our unicorn pod initialised before our operator had a chance to update the secrets with recent changes.
This meant that the the secrets that the initContainers had were old, they then copied those old secrets into a memory volume, which is what the unicorn container used to try and start... and fail.
Unlike what I say during the demo, if you watch the terminal output, the probes are actually failing. And the unicorn container IS actually restarting. But because the initContainer are not run again, the new secret is never copied into the memory volume.
The reasons we use initContainers for secrets is the following:
- We did it to allow us to keep secrets out of ENV variables when working with upstream charts
- This method would allow our initContainers to grab secrets from elsewhere, like Vault, if needed.
- Because we were already doing it for 1, and 2 seemed like a good usecase, so we made it consistent