Make `Formats::Text.marshal_multiprocess` to be Ruby MRI GVL friendly
Currently the .marshal_multiprocess
can lock the Ruby thread for a long time when rendering metrics.
This happens, as it accepts a list of files, and processes all files without giving a thread yields to Ruby MRI.
We should rewrite this function: to be Ruby MRI friendly and improve concurrency of this method when running other Ruby workload:
def marshal_multiprocess(path = Prometheus::Client.configuration.multiprocess_files_dir)
file_list = Dir.glob(File.join(path, '*.db')).sort
.map {|f| Helper::PlainFile.new(f) }
.map {|f| [f.filepath, f.multiprocess_mode.to_sym, f.type.to_sym, f.pid] }
FastMmapedFile.to_metrics(file_list.to_a)
end
Proposal
Process a file list in batches of, like 10-100 and append the result to string. It seems that this might be a bad idea.
Likely better is to allow thread switching in some cases.
And in general debug how performant it is that if we have 10_000-100_000 metrics to render.
Edited by Kamil Trzciński