SMB3 Persistent Handles

Ralph Böhmerequested to merge
samba-team/devel/samba:slow/ph-tests into master

Persistent Handles Design

Design Notes

Persistent Handles can be granted on a CREATE requesting a caching level of none or lease, but not an oplock level. Testing against Windows had revealed that while Windows does grant persistency with oplocks, overall behaviour with contenting access an oplock breaks seemed broken.

Options

  • Global persistent handles, sets SMB2_CAP_PERSISTENT_HANDLES in SMB2-NEGPROT response and controls whether smbXsrv_open_global.tdb, locking.tdb and brlock.tdb are opened with new db_open() flag DBWRAP_FLAG_PER_REC_PERSISTENT.

  • Per share continuous availability, sets SMB2_SHARE_CAP_CONTINUOUS_AVAILABILITY in SMB2-TCON response.

dbwrap and ctdb

New db_open() flag DBWRAP_FLAG_PER_REC_PERSISTENT: opens a secondary pesistent database NAME_persistent.tdb. When opening a database all existing records from NAME_persistent.tdb are copied to NAME.tdb depending on the presence or absence of a marker record in NAME.tdb.

New ctdb control CTDB_CONTROL_PUSH_RECORD that stores a record on all nodes in NAME.tdb.

New dbwrap_record_store() flags DBWRAP_STORE_PERSISTENT.

New ctdb_ltdb_header flags CTDB_REC_FLAG_PERSISTENT and CTDB_REC_FLAG_PERSISTENT_IN_PROGRESS.

New record header in the dbwrap_tdb.c struct dbwrap_tdb_header which stores and maintains DBWRAP_TDB_FLAG_PERSISTENT.

When fetching records, only NAME.tdb is used. When storing record data, a backup of the record data is stored in NAME_persistent.tdb.

dbwrap_ctdb store semantics:

  • DBWRAP_STORE_PERSISTENT:
    • store in volatile tdb with flag CTDB_REC_FLAG_PERSISTENT_IN_PROGRESS and additional header with server_id
    • push-record to all volatilve tbds (without CTDB_REC_FLAG_PERSISTENT_IN_PROGRESS stuff). Note that the local store done in the previous step has an incremented ctdb ltdb header rsn.
    • drop chainlock
    • store with txn on persistent backup tdb
    • fetch_locked_migrate loop
    • store again locally without CTDB_REC_FLAG_PERSISTENT_IN_PROGRESS

dbwrap_ctdb fetch-locked and do-locked semantics:

  • in the fetch-locked loop additionally check for CTDB_REC_FLAG_PERSISTENT_IN_PROGRESS
  • if set, check for stale a server_id and either use the record after removing the server_id header, or continue the fetched-locked loop
  • if CTDB_REC_FLAG_PERSISTENT is set for a record when calling dbwrap_store_record() without DBWRAP_STORE_PERSISTENT, the persistent backup record is deleted.

Processing

SMB

If a SMB2-CREATE DH2Q context has the SMB2_DHANDLE_FLAG_PERSISTENT flag set, call SMB_VFS_CREATE_FILE() with private_flags=NTCREATEX_FLAG_PERSISTENT_OPEN if the share capabilities includes SMB2_SHARE_CAP_CONTINUOUS_AVAILABILITY and the request requests a caching level of none or lease, but not for an oplock level.

FSA

In open_file_ntcreate() and open_directory() set fsp->op->global->persistent based on NTCREATEX_FLAG_PERSISTENT_OPEN.

When creating, modifying or deleting a record in smbXsrv_open_global.tdb, locking.tdb or brlock.tdb, DBWRAP_STORE_PERSISTENT is used based on a database specific logic:

  • smbXsrv_open_global.tdb: fsp->op->global->persistent,

  • locking.tdb: counter of persistent handles per share_mode_data (num_persistent),

  • brlock.tdb: loop over all locks per record counting the number of persistent locks

When processing any open and checking for incompatible opens and potential lease breaks, an existing open that is a disconnected Persistent Handle either

  • if the existing open has a RWH lease, trigger a deferred lease break and defer the new open,

  • otherwise let the new open fail with NT_STATUS_FILE_NOT_AVAILABLE.

VFS

The role of the VFS cookie returned from SMB_VFS_DURABLE_COOKIE() and stored in the smbXsrv_open_global.tdb record is turned from a "protective" cookie, to an "initial handle state" cookie as we can't rely on SMB_VFS_DURABLE_DISCONNECT() being called to update the cookie with filesystem metadata:

  • SMB_VFS_DURABLE_COOKIE(): for a Persisten Handle SMB_VFS_DURABLE_COOKIE() is only used to store some handle metadata like initial-delete-on-close,

  • SMB_VFS_DURABLE_DISCONNECT(): becomes basically a no-op for for a Persisten Handle,

  • SMB_VFS_DURABLE_RECONNECT(): skips the cookie filesystem validation, triggers deferred lease breaks.

Cleanup of stale Persistent Handles

cleanupd scans smbXsrv_open_global_persistent.tdb and signals the scavenger for every disconnected Persistent Handle. The scanning is done on two occasions:

  • after acquiring a cluster wide lock (glock),

  • when an smbd exits uncleanly.

Edited by Ralph Böhme

Merge request reports