Add support for inlining blobs in the index

Previously all blobs managed by the index would be stored in a separate backing
store. For example, one might setup a Postgres index on top of S3. The index
tracks the access and size of the backing store.

Therefore, in order to retrieve a blob from the backing store, the index is
first checked and updated as necessary. In particular, for small blobs, the
additional overhead of reaching out to S3 to retrieve the blob can be reduced.

This commit adds support in BuildGrid for allowing users to specify that blobs
below a certain size should be stored within the index itself. This allows
flexibility around how much space is occupied by these 'inlined-blobs' and
allows the user to speed up blob access by leveraging the database.

The implementation of this feature is deliberately relatively forgiving in the
sense that if a blob that *should* be inlined in the index is not found there,
it continues to fallback to the underlying storage. This is to faciliate the
enablement of this feature without an expensive "backfill" of all blobs meeting
the size requirements. Additionally, it facilitates the ability to, say, reduce
the size threshold to lower and higher values after having populated the index
without ill effects. It is worth noting however, that in the case that a blob
is missing from the inline storage, BuildGrid does *not* go back and add
it. Only new blobs are added to the inlining.

Signed-off-by: Ed Baunton <ebaunton1@bloomberg.net>
7 jobs for edbaunton/features/inline-index-blobs in 25 minutes and 8 seconds (queued for 1 second)
latest
Status Job ID Name Coverage
  Test
passed #676463043
cleanup-compose-test

00:07:45

passed #676463045
documentation

00:10:45

passed #676463040
dummy-tests

00:10:45

passed #676463042
e2e-tests

00:14:22

passed #676463039
type-check

00:11:05

passed #676463037
unit-tests

00:23:08

 
  Post
passed #676463047
coverage

00:02:00

83.67%