Add support for inlining blobs in the database
Before raising this MR, consider whether the following are required, and complete if so:
-
Unit tests -
Metrics -
Documentation update(s)
If not required, please explain in brief why not.
Description
This change adds support for storing blobs directly to the SQL index. It is inspired by !447 (closed) , but that MR is too old to be reworked into the current iteration of the index.
Changes proposed in this merge request:
- Add a new
inline-blob-maxsize
parameter to control storage of blobs directly in index - Add a new column to the SQL schema for inline blobs
- Rewrite SQL index to read from/write to the inline blob column when applicable
- Update the
_save_digests_to_index
helper function to take digest-blob pairs and a Session - Add a SQLite delegate for efficient bulk inlining of blobs when using SQLite
Validation
You can run the stress testing dockerfiles to validate that this works. The command I used was (from the tests/stress-testing-dockerfiles
directory):
STRESS_TESTING_DISABLE_PLATFORM_PROPERTIES=yes \
STRESS_TESTING_BGD_CONFIG=index-s3-inlined.yml \
STRESS_TESTING_CLIENT_REQUESTS=20 \
docker-compose -f compose-files/pges-recc-bbrht.yml up \
--scale buildgrid=1 --scale bots=2 \
--scale clients=1 --scale database=1
Issues addressed
N/A
Edited by Rohit Kothur