Generated plugin config files for different pipelines are stored in same plugin-specific location
This means that meltano elt pipelines running in parallel with a shared filesystem could overwrite each other's config files, resulting in plugins running with the wrong configuration.
We should store these config files in .meltano/run/elt/<job_id>/<run_id>, not .meltano/run/<plugin_name>, as intended by this line which misses a return: https://gitlab.com/meltano/meltano/-/blob/master/src/meltano/core/elt_context.py#L61
This will cause issues for people who are currently storing their tap.properties.json at .meltano/run/<tap>/tap.properties.json, as I have suggested a few times in Slack, so this would be a breaking change we need to explicitly call out in the CHANGELOG and blog post. These users should use .meltano/extractors/<tap>/tap.properties.json, or the new --catalog option (#2291 (closed)) instead.
The downside of storing the files in .meltano/run/elt/<job_id>/<run_id> is that we also store logs there: https://meltano.com/docs/project.html#meltano-directory, which we recommend exfiltrating from Docker containers: https://meltano.com/docs/production.html#storing-logs. It would be unexpected if config files with credentials and everything would also be exfiltrated. It'd be better to just keep them safe in the container.
We could consider automatically deleting these files at the end of a pipeline, so that they don't persist on the shared volume for too long, but that would make debugging more difficult, since the --config, --catalog, and --state paths printed by meltano --log-level=debug elt (debug mode) wouldn't exist anymore after the job fails. We could consider deleting only the config file, since it could contain sensitive values, and since it can be generated easily using meltano config anyway: #2343 (closed)
The other option would be to store logs elsewhere, but that would be another breaking change with regard to a recommendation we've made in the documentation and on Slack: #2344 (closed)