Skip to content

Add support for inlining blobs in the database

Rohit Kothur requested to merge rkothur/inline-blobs into master

Before raising this MR, consider whether the following are required, and complete if so:

  • Unit tests
  • Metrics
  • Documentation update(s)

If not required, please explain in brief why not.

Description

This change adds support for storing blobs directly to the SQL index. It is inspired by !447 (closed) , but that MR is too old to be reworked into the current iteration of the index.

Changes proposed in this merge request:

  • Add a new inline-blob-maxsize parameter to control storage of blobs directly in index
  • Add a new column to the SQL schema for inline blobs
  • Rewrite SQL index to read from/write to the inline blob column when applicable
  • Update the _save_digests_to_index helper function to take digest-blob pairs and a Session
  • Add a SQLite delegate for efficient bulk inlining of blobs when using SQLite

Validation

You can run the stress testing dockerfiles to validate that this works. The command I used was (from the tests/stress-testing-dockerfiles directory):

STRESS_TESTING_DISABLE_PLATFORM_PROPERTIES=yes \
STRESS_TESTING_BGD_CONFIG=index-s3-inlined.yml \
STRESS_TESTING_CLIENT_REQUESTS=20 \
docker-compose -f compose-files/pges-recc-bbrht.yml up \
  --scale buildgrid=1 --scale bots=2 \
  --scale clients=1 --scale database=1

Issues addressed

N/A

Edited by Rohit Kothur

Merge request reports