Skip to content

ldb: Increase the DEFAULT_INDEX_CACHE_SIZE

Andréas Lerouxrequested to merge
samba-team/devel/samba:aleroux/ldb-default-index-cache-size into master

Description

Context

Running the following command on a large domain (>200k entries) takes about 3 days:

samba-tool domain schemaupgrade --schema=2019

Issue

With some debugging we (Tranquil IT team) found out the bottleneck was in the reindex, required because of the schema changes. And more precisely in the calls to tdb_find because the internal ldb was growing up for each object and iterating through the entire bucket matching the index hash.

Buckets were getting bigger and bigger during the reindex, up to ~4000 object/bucket. Meaning lot of iterations in tdb_find. The issue was not about the distribution because every bucket had approximately the same amount of records.

Suggestion

Increasing the number of bucket used when doing reindex allows to speed up the time, especially for larger domains. With the suggested value (8089, prime number) it took ~1h instead of 3 days.

I'm not aware of all side effects of changing the default value and I think there is a better way to fix this.

Maybe add this as a parameter to the samba-tool, or have a default value proportional to the number of entries in the domain ?

The MR #3605 was made in the same goal of speeding up the schema upgrade

EDIT

After discussing changing DEFAULT_INDEX_CACHE_SIZE which would cause memory overhead for all operations except some costly ones, such as schema upgrade, we decided to only increase the transaction_index_cache_size for the schema upgrade operation, as it is done for the join operation.

Checklist

  • Commits have Signed-off-by: with name/author being identical to the commit author
  • (optional) This MR is just one part towards a larger feature.
  • (optional, if backport required) Bugzilla bug filed and BUG: tag added
  • Test suite updated with functionality tests
  • Test suite updated with negative tests
  • Documentation updated
  • CI timeout is 3h or higher (see Settings/CICD/General pipelines/ Timeout)

Reviewer's checklist:

  • There is a test suite reasonably covering new functionality or modifications
  • Function naming, parameters, return values, types, etc., are consistent and according to README.Coding.md
  • This feature/change has adequate documentation added
  • No obvious mistakes in the code
Edited by Andréas Leroux

Merge request reports