feat: rfc-0008 Compliance, Versioning, and Corrections

Revant Nandgaonkarrequested to merge
rfc-0008 into main

TODO: RFC-0008 — Compliance, Versioning, and Corrections

RFC: docs/rfcs/rfc-0008-compliance-versioning-corrections.md Approach: Test-Driven Development (TDD). Write the failing test first, implement to make it pass. Coverage target: ≥ 95% (no regression from current baseline)

Context & Key Decisions

Read the full RFC before starting. Key architectural constraints:

  1. AuditMixin — lives in libs/framework-m-core/src/framework_m_core/domain/mixins.py. The ActivityLog DocType and AuditLogProtocol already exist. The mixin signals BaseController to auto-diff and emit audit entries on save.
  2. VersionedMixin — same file. Shadow table provisioning is gated by [compliance] versioning_enabled = true in framework_config.toml read via load_config() in libs/framework-m-core/src/framework_m_core/config.py. ORM checks this at schema-sync time.
  3. TemporalQueryNotSupportedError — new exception in libs/framework-m-core/src/framework_m_core/exceptions.py. Must raise (not silently degrade) when as_of is used without versioning enabled.
  4. CorrectionService — lives in libs/framework-m/src/framework_m/core/services/correction.py (alongside existing user_manager.py). Uses repositories and the controller's _validate_submitted_changes which already enforces immutability via ImmutableDocumentError.
  5. DatabaseAuditAdapter at libs/framework-m-standard/src/framework_m_standard/adapters/audit/database_audit.py currently uses in-memory stub storage. The TODO comment inside to inject a real RepositoryProtocol[ActivityLog] must be completed as part of Phase 1.
  6. ActivityLog field rename — doctype field has been renamed to doctype_name in libs/framework-m-core/src/framework_m_core/doctypes/activity_log.py (already in-progress per git diff). The AuditEntry model in libs/framework-m-core/src/framework_m_core/interfaces/audit.py still uses doctype. These must end up consistent. Chosen direction: keep doctype_name on ActivityLog (storage) and keep doctype on AuditEntry (protocol abstraction); the DatabaseAuditAdapter is responsible for mapping between the two.
  7. Test patterns: Mirror libs/framework-m-standard/tests/adapters/test_internal_workflow.py for async + mock patterns. Use pytest-asyncio, AsyncMock, and sqlite+aiosqlite:///:memory: from libs/framework-m/tests/conftest.py.

Phase 1: AuditMixin + Controller Auto-Diff + DatabaseAuditAdapter Wire-Up

Goal: Every DocType with AuditMixin automatically gets a JSON diff written to ActivityLog on save.

Step 1.0 — ActivityLog Field Rename (doctype → doctype_name) (in-progress)

This change is already partially applied per git diff. Complete and test it before any other Phase 1 work.

  • Complete the rename in libs/framework-m-core/src/framework_m_core/doctypes/activity_log.py:
    • Field doctype → doctype_name on the ActivityLog class.
    • Update the docstring example (doctype_name="Invoice", diff={...} instead of changes={...}).
    • Confirm diff field (not changes) is the canonical name — the existing class already uses diff; the git diff also shows changes being removed from the docstring example. Reconcile.
  • Update AuditEntry in libs/framework-m-core/src/framework_m_core/interfaces/audit.py:
    • Keep doctype: str on AuditEntry (protocol-level abstraction is unchanged).
    • Add a note in AuditEntry docstring: "Maps to ActivityLog.doctype_name in the database adapter."
  • Update DatabaseAuditAdapter in libs/framework-m-standard/src/framework_m_standard/adapters/audit/database_audit.py:
    • In log(): pass doctype_name=doctype (not doctype=doctype) when constructing ActivityLog(...).
    • In query(): when converting ActivityLog → AuditEntry, map activity_log.doctype_name → audit_entry.doctype.
    • In query() filter handling: map filters["doctype"] → FilterSpec(field="doctype_name", ...).
  • Write/update tests in libs/framework-m-core/tests/core/test_activity_log.py (file already exists):
    • Test ActivityLog(user_id=..., action=..., doctype_name="Invoice", document_id=...) constructs without error.
    • Test that constructing with doctype="Invoice" (old field name) raises ValidationError (Pydantic extra="forbid").
    • Test ActivityLog has no field named doctype (assert "doctype" not in ActivityLog.model_fields).
    • Test ActivityLog has no field named changes (assert "changes" not in ActivityLog.model_fields).
    • Test the diff field accepts dict[str, Any] | None.
  • Run pytest libs/framework-m-core/tests/core/test_activity_log.py -v — all pass.
  • Run pytest libs/framework-m-standard/tests/adapters/audit/ -v — all pass.

Step 1.1 — Define AuditMixin (TDD)

  • Write test libs/framework-m-core/tests/core/test_mixins.py (new file):

    • Test AuditMixin can be imported from framework_m_core.domain.mixins.
    • Test a class inheriting both BaseDocType and AuditMixin has _audit_enabled: ClassVar[bool] = True.
    • Test a plain BaseDocType subclass does not have _audit_enabled = True.
    • Test isinstance(doc, AuditMixin) returns True for a mixed-in DocType.

  • Implement AuditMixin in libs/framework-m-core/src/framework_m_core/domain/mixins.py:

    class AuditMixin:
    """Marker mixin: signals BaseController to auto-diff + log to ActivityLog."""
    _audit_enabled: ClassVar[bool] = True

    Add AuditMixin to __all__ in that file.

  • Run pytest libs/framework-m-core/tests/core/test_mixins.py -v

Step 1.2 — JSON Diff Domain Utility (TDD)

  • Write test in libs/framework-m-core/tests/unit/core/domain/test_diff.py:
    • compute_diff({}, {}) → {}.
    • compute_diff({"a": 1}, {"a": 2}) → {"a": {"old": 1, "new": 2}}.
    • compute_diff({"a": 1, "b": 2}, {"a": 1, "b": 3}) → only b in result.
    • Nested dict: top-level key diff treats nested value as opaque ({"old": {...}, "new": {...}}).
    • List fields: diffed by value equality.
    • id, creation, modified, modified_by are excluded from diffs by default (always-changing noise).
    • Error: Passing non-dict raises TypeError.
  • Implement compute_diff(old: dict, new: dict, exclude: set[str] | None = None) -> dict in libs/framework-m-core/src/framework_m_core/domain/diff.py.
    • Default exclude = {"id", "creation", "modified", "modified_by"}.
  • Run tests.

Step 1.3 — BaseController Auto-Audit Hook (TDD)

  • Write test libs/framework-m-core/tests/core/test_base_controller_audit.py (new file):
    • Fixture: MockAuditAdapter (implements AuditLogProtocol from libs/framework-m-core/src/framework_m_core/interfaces/audit.py — use InMemoryAuditAdapter for simplicity).
    • Fixture: AuditedDocType(BaseDocType, AuditMixin) stub.
    • Update happy path: before_save captures old state; after_save calls audit.log(action="update", changes=diff) — verify once called with correct args.
    • Insert happy path: after_insert calls audit.log(action="create") with no changes.
    • Non-AuditMixin DocType: audit.log is NOT called.
    • Empty diff: If old and new states are identical, audit.log is NOT called.
    • Audit failure propagates: If audit.log raises Exception, it propagates out of after_save.
    • job_id propagation: If context dict has "job_id" key, it is forwarded to audit.log metadata.
  • Implement in libs/framework-m-core/src/framework_m_core/domain/base_controller.py:
    • Add _pre_save_snapshot: dict | None = None instance variable.
    • Add _audit_adapter: AuditLogProtocol | None = None (injected at construction or via a setter).
    • before_save: If isinstance(self.doc, AuditMixin), capture self._pre_save_snapshot = self.doc.model_dump().
    • after_save / after_insert: If audit adapter present and doc is AuditMixin, compute diff via compute_diff, then await self._audit_adapter.log(...).
  • Run tests.

Step 1.4 — Complete DatabaseAuditAdapter Real Repository Wiring (TDD)

  • Write test libs/framework-m-standard/tests/adapters/audit/test_database_audit.py (new file):
    • Insert happy path: log(...) calls repository.save(ActivityLog(...)) with correct field values. Use AsyncMock for repository.
    • Query happy path: query(filters={"doctype": "Invoice"}) calls repository.list(...) with matching FilterSpec list.
    • Pagination: query(limit=10, offset=5) forwards limit/offset to repository.list.
    • Time filters: query(filters={"from_timestamp": some_dt}) generates FilterSpec(field="timestamp", operator=FilterOperator.GTE, value=some_dt).
    • Error propagation: DatabaseError raised by repository.save propagates (not swallowed).
    • Return type: query returns list[AuditEntry] with doctype field (mapped from ActivityLog.doctype_name).
  • Implement: Replace in-memory stub in libs/framework-m-standard/src/framework_m_standard/adapters/audit/database_audit.py. Inject repository: RepositoryProtocol[ActivityLog] in __init__. Wire log() → repository.save(ActivityLog(...)), query() → repository.list(...).
  • Run tests.

Phase 2: VersionedMixin + Configuration Gate

Goal: DocTypes opt into shadow table versioning via VersionedMixin. ORM respects [compliance] versioning_enabled config.

Step 2.1 — VersionedMixin Definition (TDD)

  • Write test in libs/framework-m-core/tests/unit/core/domain/test_mixins.py (append):

    • VersionedMixin importable from framework_m_core.domain.mixins.
    • DocType inheriting BaseDocType + VersionedMixin has _versioning_enabled: ClassVar[bool] = True.
    • isinstance(doc, VersionedMixin) returns True.
    • Plain BaseDocType does NOT have _versioning_enabled = True.
    • AuditMixin + VersionedMixin on same DocType has no MRO conflict.

  • Implement in libs/framework-m-core/src/framework_m_core/domain/mixins.py:

    class VersionedMixin:
    """Marker mixin: signals ORM to provision {table}_history shadow table."""
    _versioning_enabled: ClassVar[bool] = True

    Add to __all__.

  • Run tests.

Step 2.2 — Config Gate for Versioning (TDD)

  • Write test libs/framework-m-standard/tests/core/test_versioning_config.py (new file):
    • is_versioning_enabled({"compliance": {"versioning_enabled": True}}) → True.
    • is_versioning_enabled({}) → False.
    • is_versioning_enabled({"compliance": {"versioning_enabled": False}}) → False.
    • FRAMEWORK_M_STRONGSYNC_VERSIONING=true env var → True regardless of TOML. (monkeypatch os.environ).
    • FRAMEWORK_M_STRONGSYNC_VERSIONING=false env var overrides TOML True → False.
  • Implement is_versioning_enabled(config: dict | None = None) -> bool in libs/framework-m-standard/src/framework_m_standard/adapters/db/versioning_config.py (new file):
    • Reads config["compliance"]["versioning_enabled"] or calls load_config() if config is None.
    • Env var FRAMEWORK_M_STRONGSYNC_VERSIONING ("true"/"false") takes precedence over TOML.
  • Run tests.

Phase 3: Shadow Table Generation in SchemaMapper

Goal: SchemaMapper provisions {table}_history when DocType has VersionedMixin AND versioning is enabled.

Step 3.1 — SchemaMapper Versioning Extension (TDD)

  • Write test libs/framework-m-standard/tests/adapters/db/test_schema_mapper_versioning.py (new file):
    • Versioning enabled: Given class AuditedInvoice(BaseDocType, VersionedMixin) and config = {"compliance": {"versioning_enabled": True}}, mapper.create_versioned_tables(AuditedInvoice, metadata, config) returns 2 Table objects: main + auditedinvoice_history.
    • History table has all same columns as main PLUS _version_no INTEGER NOT NULL, _valid_from DATETIME, _valid_to DATETIME.
    • History table does NOT have a UNIQUE constraint on name.
    • History table has an index on _valid_from.
    • Versioning disabled: Returns only main table (no _history table).
    • Non-VersionedMixin DocType: Returns only main table regardless of config.
    • create_table unchanged: Existing create_table() remains unchanged and does not add history.
    • Schema cascade DDL: mapper.create_history_alteration_ddl("auditedinvoice_history", added_columns) generates ALTER TABLE ... ADD COLUMN ... DDL for each new column.
    • audit_db_url marker: When config["compliance"]["audit_db_url"] is set, history table's Table.info dict contains {"audit_db_url": "..."} so the repository layer can route writes.
  • Implement in libs/framework-m-standard/src/framework_m_standard/adapters/db/schema_mapper.py:
    • Add create_versioned_tables(model, metadata, config=None) -> list[Table].
    • Add private _create_history_table(main_table, metadata, config=None) -> Table.
    • Add create_history_alteration_ddl(history_table_name, added_columns) -> list[DDL].
    • Import is_versioning_enabled from libs/framework-m-standard/src/framework_m_standard/adapters/db/versioning_config.py.
  • Run pytest libs/framework-m-standard/tests/adapters/db/test_schema_mapper_versioning.py -v
  • Run pytest libs/framework-m-standard/tests/adapters/db/test_schema_mapper.py — must pass (no regressions).

Phase 4: TemporalQueryNotSupportedError Exception

Step 4.1 — New Exception (TDD)

  • Write test in libs/framework-m-core/tests/core/test_exceptions.py (append to existing file):

    • TemporalQueryNotSupportedError is a subclass of FrameworkError.
    • str(exception) mentions both the doctype name and that versioning is disabled.
    • Importable from framework_m_core.exceptions.
    • Present in __all__ of libs/framework-m-core/src/framework_m_core/exceptions.py.

  • Implement in libs/framework-m-core/src/framework_m_core/exceptions.py:

    class TemporalQueryNotSupportedError(FrameworkError):
    """Raised when as_of temporal query used but versioning_enabled is False."""
    def __init__(self, doctype_name: str) -> None:
        super().__init__(
            f"Temporal query (as_of) on '{doctype_name}' requires "
            "[compliance] versioning_enabled=true. "
            "Cannot silently return current state."
        )

    Add to __all__.

  • Run pytest libs/framework-m-core/tests/core/test_exceptions.py -v

Phase 5: Repository as_of Temporal Querying

Goal: RepositoryProtocol.get and list accept as_of: datetime | None. Raise TemporalQueryNotSupportedError when versioning is disabled and as_of is passed.

Step 5.1 — Update RepositoryProtocol Interface (TDD)

  • Write test libs/framework-m-core/tests/core/interfaces/test_repository_temporal.py (new file):
    • RepositoryProtocol.get signature accepts as_of: datetime | None = None (verify via inspect.signature).
    • RepositoryProtocol.list signature accepts as_of: datetime | None = None.
  • Implement in libs/framework-m-core/src/framework_m_core/interfaces/repository.py:
    • Add as_of: datetime | None = None parameter to get(id, as_of=None) and list(..., as_of=None).
  • Run pytest libs/framework-m-core/tests/core/interfaces/ -v

Step 5.2 — SQLAlchemy Temporal Queries in generic_repository.py (TDD)

  • Write test libs/framework-m-standard/tests/adapters/db/test_repo_temporal.py (new file):
    • Setup: Use db_session + clean_tables fixtures from libs/framework-m/tests/conftest.py. Define class VersionedItem(BaseDocType, VersionedMixin). Provision both main and history tables via create_versioned_tables.
    • get happy path: Insert → wait → update → repo.get(id, as_of=t_between_saves) returns first state.
    • list happy path: Multiple versioned docs; repo.list(as_of=t1) returns correct historical state for all.
    • as_of=None: Normal query hits main table only. No history table touched.
    • Versioning disabled + as_of passed: Raises TemporalQueryNotSupportedError.
    • as_of before any version exists: Returns None (no version at that time).
    • as_of as future datetime: Returns the current latest state (not an error).
  • Implement in libs/framework-m-standard/src/framework_m_standard/adapters/db/generic_repository.py:
    • get(id, as_of=None): If as_of is set and model lacks VersionedMixin, raise TemporalQueryNotSupportedError. If VersionedMixin and versioning enabled, query {table}_history for MAX(_version_no) WHERE modified <= as_of.
    • list(..., as_of=None): Same pattern.
  • Run pytest libs/framework-m-standard/tests/adapters/db/test_repo_temporal.py -v
  • Run pytest libs/framework-m-standard/tests/adapters/db/test_generic_repository.py — no regressions.

Phase 6: Repository Versioning Write Logic

Goal: Every repo.save() on a VersionedMixin DocType (when versioning_enabled) also writes a row to {table}_history in the same transaction.

Step 6.1 — History Row Write on Save (TDD)

  • Write test libs/framework-m-standard/tests/adapters/db/test_repo_versioning.py (new file):
    • Setup: db_session + clean_tables. class Invoice(BaseDocType, VersionedMixin, SubmittableMixin). Provision via create_versioned_tables.
    • Insert: repo.save(invoice) → 1 row in main, 1 row (_version_no=1) in invoice_history.
    • Update: repo.save(invoice) again → history has 2 rows (_version_no 1 and 2).
    • Increment: After 3 saves, history has 3 rows with _version_no 1, 2, 3.
    • _valid_from / _valid_to: Each history row's _valid_from = save time. Previous row's _valid_to = new row's _valid_from.
    • Non-VersionedMixin: repo.save(todo) writes only to main table.
    • Versioning disabled: Even VersionedMixin DocType writes only to main table.
    • Atomicity: Simulated DB failure mid-transaction — neither main nor history row committed.
    • audit_db_url routing: When Table.info["audit_db_url"] is set, history write uses secondary connection.
  • Implement in libs/framework-m-standard/src/framework_m_standard/adapters/db/generic_repository.py:
    • After successful insert/update on a VersionedMixin model (and is_versioning_enabled()), insert into {table}_history with same data + incremented _version_no and _valid_from = now(). Update previous row's _valid_to.
    • Run history insert within same session/transaction.
  • Run pytest libs/framework-m-standard/tests/adapters/db/test_repo_versioning.py -v

Phase 7: DocTypeMeta — correction_strategy Support

Goal: BaseDocType.Meta supports a correction_strategy attribute. CorrectionService reads this to select the correction flow.

Step 7.1 — correction_strategy on BaseDocType.Meta (TDD)

  • Write test in libs/framework-m-core/tests/unit/test_base_doctype.py (append to existing BaseDocType unit tests):

    • BaseDocType.get_correction_strategy() returns None by default.
    • Returns "amend_only" when Meta.correction_strategy = "amend_only".
    • Returns "in_place" when Meta.correction_strategy = "in_place".

  • Implement in libs/framework-m-core/src/framework_m_core/domain/base_doctype.py:

    @classmethod
def get_correction_strategy(cls) -> str | None:
    meta = getattr(cls, "Meta", None)
    return getattr(meta, "correction_strategy", None)

  • Run pytest libs/framework-m-core/tests/core/doctypes/ -v

Phase 8: CorrectionService — Amend, Restore, In-Place Edit

Goal: Implement the three correction flows from RFC Section 4 (State Machine).

Step 8.1 — Define and Test CorrectionService (TDD)

  • Write test libs/framework-m/tests/core/test_correction_service.py (new file):

    Fixtures:

    • mock_repository: AsyncMock of RepositoryProtocol (from libs/framework-m-core/src/framework_m_core/interfaces/repository.py).
    • mock_audit_adapter: AsyncMock of AuditLogProtocol (from libs/framework-m-core/src/framework_m_core/interfaces/audit.py).
    • submitted_invoice: Invoice(BaseDocType, SubmittableMixin) with docstatus=SUBMITTED, name="INV-001".
    • versioned_submitted_invoice: Invoice(BaseDocType, SubmittableMixin, VersionedMixin) with docstatus=SUBMITTED.

    Amend:

    • test_amend_cancels_original_and_creates_new_draft: Verify: original docstatus = CANCELLED saved, new doc with new id, name="INV-001-1", docstatus=DRAFT saved; audit.log(action="amend", metadata={"reason": "..."}) called.
    • test_amend_name_increments_correctly: INV-001 → INV-001-1, INV-001-1 → INV-001-2.
    • test_amend_raises_if_not_submitted: docstatus != SUBMITTED → ValueError.
    • test_amend_audit_failure_propagates: audit.log raises → exception is NOT swallowed.

    Restore:

    • test_restore_cancels_original_and_creates_snapshot_doc: Original CANCELLED; new doc from snapshot_data with incremented name, docstatus=SUBMITTED; audit.log(action="restore") called.
    • test_restore_raises_if_not_submitted: ValueError for non-submitted doc.
    • test_restore_raises_if_snapshot_empty: ValueError("snapshot_data cannot be empty").

    In-Place Edit (requires VersionedMixin):

    • test_in_place_edit_copies_to_history_and_resets_to_draft: History snapshot saved; main row's docstatus set to DRAFT; same id/name retained; audit.log(action="in_place_edit") called.
    • test_in_place_edit_raises_if_not_versioned: ValueError("in_place_edit requires VersionedMixin").
    • test_in_place_edit_raises_if_not_submitted: ValueError for non-submitted doc.

  • Implement libs/framework-m/src/framework_m/core/services/correction.py (new file):

    class CorrectionService:
    def __init__(self, repository: RepositoryProtocol, audit: AuditLogProtocol) -> None: ...
    async def amend(self, doc, reason: str, user: UserContext) -> BaseDocType: ...
    async def restore(self, doc, snapshot_data: dict, reason: str, user: UserContext) -> BaseDocType: ...
    async def in_place_edit(self, doc, reason: str, user: UserContext) -> BaseDocType: ...
    def _next_amended_name(self, name: str) -> str: ...  # "INV-001" → "INV-001-1"

  • Run pytest libs/framework-m/tests/core/test_correction_service.py -v

Phase 9: SiblingMixin — Metadata Overlay

Goal: Formalize RFC Section 3 — a sibling DocType declares itself as a metadata overlay of a parent.

Step 9.1 — SiblingMixin Definition (TDD)

  • Write test in libs/framework-m-core/tests/unit/core/domain/test_mixins.py (append):

    • SiblingMixin importable from framework_m_core.domain.mixins.
    • class LogisticsMeta(BaseDocType, SiblingMixin) with Meta.extends = "Invoice" → get_parent_doctype() returns "Invoice".
    • get_parent_doctype() raises ValueError when Meta.extends not set.
    • is_metadata_overlay() returns True when Meta.is_metadata_overlay = True.
    • is_metadata_overlay() returns False by default.

  • Implement in libs/framework-m-core/src/framework_m_core/domain/mixins.py:

    class SiblingMixin:
    """Marker mixin for Sibling/Metadata-Overlay DocTypes.
    Requires Meta.extends = "ParentDocTypeName".
    """
    @classmethod
    def get_parent_doctype(cls) -> str: ...
    @classmethod
    def is_metadata_overlay(cls) -> bool: ...

    Add to __all__.

  • Run pytest libs/framework-m-core/tests/unit/core/domain/test_mixins.py -v

Phase 10: ActivityLog Immutability Enforcement (TDD)

Goal: No code path can update or delete an ActivityLog record.

Step 10.1 — Guard ActivityLog from Mutations (TDD)

  • Write test in libs/framework-m-core/tests/core/test_activity_log.py (file already exists at libs/framework-m-core/tests/core/test_activity_log.py; append):
    • ActivityLog.Meta.permissions["write"] == [].
    • ActivityLog.Meta.permissions["delete"] == [].
    • DatabaseAuditAdapter.log(...) calls only repository.save() for insert (no update path).
    • Calling repository.save(existing_activity_log) (entity already has id) raises ImmutableDocumentError or PermissionDeniedError. Use AsyncMock wired to raise.
  • Implement guard in libs/framework-m-standard/src/framework_m_standard/adapters/audit/database_audit.py: Reject update calls — log() must always construct a fresh ActivityLog (no id passed in), so repository.save will insert, never update.
  • Run pytest libs/framework-m-core/tests/core/test_activity_log.py -v

Phase 11: Public API Exports

Goal: All new public symbols are importable from their package's top-level __init__.py.

Step 11.1 — Update Exports (TDD)

  • Write import-assertion tests (can be in test_mixins.py or a new libs/framework-m-core/tests/core/test_exports.py):
    • from framework_m_core.domain.mixins import AuditMixin, VersionedMixin, SiblingMixin
    • from framework_m_core.exceptions import TemporalQueryNotSupportedError
    • from framework_m.core.services.correction import CorrectionService
    • from framework_m_standard.adapters.db.versioning_config import is_versioning_enabled
  • Update __all__ in each relevant file where missing.
  • Run import tests.

Phase 12: Integration Tests

Goal: End-to-end flow with real in-memory SQLite.

Step 12.1 — Audit Integration (TDD)

  • Write test libs/framework-m/tests/integration/test_rfc_0008_audit.py (new file):
    • DocType with AuditMixin + SubmittableMixin. Insert → assert one ActivityLog row action="create".
    • Update a field → assert ActivityLog row action="update" with correct diff.
    • Submit → assert ActivityLog row action="submit".
    • Direct edit on submitted doc (no correction flow) → assert ImmutableDocumentError AND no action="update" log entry created.

Step 12.2 — Versioning Integration (TDD)

  • Write test libs/framework-m/tests/integration/test_rfc_0008_versioning.py (new file):
    • VersionedMixin DocType; versioning config enabled.
    • Insert → 1 history row. Update → 2 history rows with _version_no 1/2.
    • repo.get(id, as_of=t_between) → returns state at version 1.
    • Disable versioning config → repo.get(id, as_of=t) raises TemporalQueryNotSupportedError.

Step 12.3 — Correction Flow Integration (TDD)

  • Write test libs/framework-m/tests/integration/test_rfc_0008_correction.py (new file):
    • VersionedMixin + SubmittableMixin DocType; real DB.
    • Insert → Submit → correction_service.amend(doc, reason, user): original CANCELLED in DB; new draft with incremented name in DB; ActivityLog entry action="amend".
    • Insert → Submit → correction_service.in_place_edit(doc, reason, user): history snapshot row exists; main row is DRAFT with same id; ActivityLog entry action="in_place_edit".

Phase 13: Final Verification

  • Run full test suite: pytest --cov=libs/ --cov-report=term-missing
  • Confirm coverage ≥ 95%.
  • ruff check libs/ — no new lint errors.
  • mypy --strict libs/framework-m-core/src/ libs/framework-m-standard/src/ libs/framework-m/src/ — no new type errors.
  • Update docs/rfcs/rfc-0008-compliance-versioning-corrections.md Status from Proposed → Accepted.

New Files to Create

Path Purpose
libs/framework-m-core/src/framework_m_core/utils/__init__.py New utils package
libs/framework-m-core/src/framework_m_core/utils/diff.py compute_diff()
libs/framework-m-standard/src/framework_m_standard/adapters/db/versioning_config.py is_versioning_enabled()
libs/framework-m/src/framework_m/core/services/correction.py CorrectionService
libs/framework-m-core/tests/core/test_mixins.py Tests for all new mixins
libs/framework-m-core/tests/core/test_base_controller_audit.py Controller auto-audit tests
libs/framework-m-core/tests/core/interfaces/test_repository_temporal.py Protocol signature tests
libs/framework-m-standard/tests/core/test_versioning_config.py Config gate tests
libs/framework-m-standard/tests/adapters/audit/test_database_audit.py DatabaseAuditAdapter tests
libs/framework-m-standard/tests/adapters/db/test_schema_mapper_versioning.py Shadow table tests
libs/framework-m-standard/tests/adapters/db/test_repo_temporal.py Temporal query tests
libs/framework-m-standard/tests/adapters/db/test_repo_versioning.py History write tests
libs/framework-m/tests/core/test_correction_service.py CorrectionService unit tests
libs/framework-m/tests/integration/test_rfc_0008_audit.py Audit integration
libs/framework-m/tests/integration/test_rfc_0008_versioning.py Versioning integration
libs/framework-m/tests/integration/test_rfc_0008_correction.py Correction flow integration

Files to Modify

Path Change
libs/framework-m-core/src/framework_m_core/domain/mixins.py Add AuditMixin, VersionedMixin, SiblingMixin
libs/framework-m-core/src/framework_m_core/domain/base_doctype.py Add get_correction_strategy()
libs/framework-m-core/src/framework_m_core/domain/base_controller.py Add audit hook in before_save/after_save/after_insert
libs/framework-m-core/src/framework_m_core/interfaces/repository.py Add as_of: datetime | None = None to get and list
libs/framework-m-core/src/framework_m_core/exceptions.py Add TemporalQueryNotSupportedError
libs/framework-m-standard/src/framework_m_standard/adapters/audit/database_audit.py Wire real RepositoryProtocol[ActivityLog] injection
libs/framework-m-standard/src/framework_m_standard/adapters/db/schema_mapper.py Add create_versioned_tables(), _create_history_table(), create_history_alteration_ddl()
libs/framework-m-standard/src/framework_m_standard/adapters/db/generic_repository.py Add as_of temporal query + history row write on save
libs/framework-m-core/tests/core/test_activity_log.py Append immutability guard tests
libs/framework-m-core/tests/core/test_exceptions.py Append TemporalQueryNotSupportedError tests

Quick Reference: Correction Strategies (RFC §4)

Meta.correction_strategy Behaviour
"amend_only" (default) Cancel original → New doc with new id, incremented name
"restore" Cancel original → New doc from snapshot, auto-submitted
"in_place" Copy main row to history → Set main row docstatus=DRAFT (same id/name)

TODO: RFC-0008 UI — Correction Wizard, Audit Timeline, Sibling Overlay

RFC: docs/rfcs/rfc-0008-compliance-versioning-corrections.md
Architecture: docs/adr/0008-frontend-plugin-architecture.md · docs/adr/0010-multi-package-ui-composition.md
Approach: TDD — write the failing test first, then implement. All UI work is delivered as an extension into @framework-m/desk via the existing <Slot> system — not baked into desk core.
Coverage target: No regression from current baseline. Every new component and hook must have tests for all happy paths and all error/edge cases. Run vitest run --coverage after each phase to confirm.

Context & Key Decisions

  1. Delivery model: All RFC-0008 UI is a plugin contribution via @framework-m/plugin-sdk. Code lives in libs/framework-m-desk/src/ (desk's own plugin contributions) and is exported from @framework-m/desk. It plugs into FormView.tsx's existing <Slot name="form-actions" />, <Slot name="form-header" />, and <Slot name="form-footer" /> hooks.

  2. Refine.dev data layer: FormView uses useOne, useCreate, useUpdate from @refinedev/core. The new Correction endpoints (POST /api/v1/{doctype}/{id}/amend, /restore, /in_place_edit) are not standard CRUD — they must be called via direct fetch or a custom Refine resource action, not useUpdate. Use a new useCorrectionAction hook that wraps fetch.

  3. Locked-form UX: FormView currently renders an editable <AutoForm> regardless of docstatus. When doc.docstatus === 1 (SUBMITTED), the entire form must be read-only and a "Submitted — document is locked" banner shown. This is a change to FormView.tsx itself (not a plugin).

  4. History tab: Rendered in the <Slot name="form-footer" /> area. Fetches GET /api/v1/ActivityLog?document_id={id}&doctype_name={doctype} using Refine's useList hook against the ActivityLog resource.

  5. Sibling Overlay: For DocTypes with Meta.extends on companion records, an editable side-pane rendered even when parent docstatus=1. Uses a separate Refine resource for the sibling DocType.

  6. WorkflowActions.tsx — handles workflow transitions only. Do NOT modify it for correction flows. The "Correct" button is separate, rendered via <Slot name="form-actions" />.

  7. Test location: Vitest tests in libs/framework-m-desk/src/__tests__/. Mirror patterns from existing tests there.

Phase UI-1: Backend Correction API Endpoints

Status: Complete

These are Python endpoints that the UI calls — without them the UI cannot be tested end-to-end.

Step UI-1.1 — Correction REST Endpoints (TDD)

  • Write test libs/framework-m-standard/tests/adapters/web/test_correction_endpoints.py (new file):
    • POST /api/v1/{doctype}/{id}/amend with body {"reason": "Typo fix"}:
      • Returns 200 with new doc (id, name, docstatus=0).
      • Returns 422 if doc not submitted.
      • Returns 404 if doc not found.
    • POST /api/v1/{doctype}/{id}/restore with body {"reason": "...", "version_no": 1}:
      • Returns 200 with restored doc.
      • Returns 400 if version_no doesn't exist.
      • Returns 400 if versioning disabled.
    • POST /api/v1/{doctype}/{id}/in_place_edit with body {"reason": "Ops fix"}:
      • Returns 200 with same doc, docstatus=0.
      • Returns 400 if DocType does not have VersionedMixin.
    • All endpoints require auth — unauthenticated returns 401.
  • Implement in libs/framework-m-standard/src/framework_m_standard/adapters/web/correction_router.py (new file):
    • Three Litestar routes: POST /{doctype}/{id}/amend, /restore, /in_place_edit.
    • Inject CorrectionService from DI container.
    • Register router in libs/framework-m-standard/src/framework_m_standard/adapters/web/__init__.py.
  • Run pytest libs/framework-m-standard/tests/adapters/web/test_correction_endpoints.py -v

Step UI-1.2 — ActivityLog List API

  • Confirm GET /api/v1/ActivityLog with ?document_id=...&doctype_name=... filter works via the existing generic list endpoint. If not, add a dedicated read-only route.
  • Write test: Querying ActivityLog by document_id returns ordered list (newest first) with diff, action, user_id, timestamp.

Phase UI-2: Locked-Form UX in FormView

Status: Complete

Goal: Submitted documents show a read-only banner; the form fields are locked; Save button is hidden.

Step UI-2.1 — Read-Only Form State (TDD)

  • Write test libs/framework-m-desk/src/__tests__/FormView.submitted.test.tsx (new file):
    • When doc.docstatus === 1: <AutoForm> renders with readonly={true} (or uiSchema setting all fields to ui:readonly).
    • "Save" button is NOT rendered.
    • "Cancel" (navigate away) button is still rendered.
    • A <SubmittedBanner> element is visible (test by getByRole("alert")).
    • When doc.docstatus === 0 (draft): form is editable, Save button present, no banner.
    • When doc.docstatus === 2 (cancelled): form is read-only, banner says "Cancelled".
  • Implement in libs/framework-m-desk/src/pages/FormView.tsx:
    • Derive isLocked = formData.docstatus === 1 || formData.docstatus === 2 from form data.
    • Pass uiSchema={{ "ui:readonly": true }} to <AutoForm> when isLocked.
    • Conditionally hide Save button when isLocked.
    • Render <SubmittedBanner docstatus={formData.docstatus} /> when isLocked.
  • Implement libs/framework-m-desk/src/components/feedback/SubmittedBanner.tsx (new file):
    • Props: docstatus: 0 | 1 | 2.
    • Renders an <div role="alert"> with appropriate message and colour (amber for submitted, grey for cancelled).
  • Run vitest run libs/framework-m-desk/src/__tests__/FormView.submitted.test.tsx

Phase UI-3: useCorrectionAction Hook

Status: Complete

Goal: Reusable hook for calling the three correction endpoints. Returns { execute, isLoading, error, result }.

Step UI-3.1 — Hook Implementation (TDD)

  • Write test libs/framework-m-desk/src/__tests__/useCorrectionAction.test.tsx (new file):

    • Mock fetch. Calling execute("amend", {reason: "fix"}) calls POST /api/v1/{doctype}/{id}/amend.
    • On success: result is set to the response body, isLoading returns to false.
    • On 422: error is set to "Only submitted documents can be amended".
    • On network error: error is set to the error message.
    • isLoading is true while request is in flight.
    • Calling execute("restore", {reason: "r", version_no: 1}) calls /restore with correct body.
    • Calling execute("in_place_edit", {reason: "r"}) calls /in_place_edit.

  • Implement libs/framework-m-desk/src/hooks/useCorrectionAction.ts (new file):

    type CorrectionType = "amend" | "restore" | "in_place_edit";

export function useCorrectionAction(doctype: string, id: string) {
  // Returns { execute(type, payload), isLoading, error, result }
}

    Export from libs/framework-m-desk/src/hooks/index.ts.

  • Run tests.

Phase UI-4: CorrectionWizard Component

Goal: A modal dialog triggered by "Correct" button on a submitted document. Presents strategy options (Amend / In-Place Edit if available), collects a mandatory reason, and calls the appropriate endpoint.

Step UI-4.1 — CorrectionWizard (TDD)

  • Write test libs/framework-m-desk/src/__tests__/CorrectionWizard.test.tsx (new file):
    • Renders closed: No modal visible when isOpen={false}.
    • Opens: When isOpen={true}, modal dialog appears with title "Correct Document".
    • Reason required: Clicking "Confirm" without entering reason shows validation error "Reason is required". No API call made.
    • Amend happy path: Select "Amend" strategy, enter reason "Typo", click "Confirm" → useCorrectionAction.execute("amend", {reason: "Typo"}) called. On success, onSuccess callback called with new doc.
    • In-Place Edit option hidden: When supportsInPlaceEdit={false}, that radio option is not rendered.
    • Loading state: While executing, Confirm button shows "Processing..." and is disabled.
    • Error display: If useCorrectionAction returns error, it is shown inside the modal.
    • Cancel: Clicking Cancel closes modal without API call.
  • Implement libs/framework-m-desk/src/components/correction/CorrectionWizard.tsx (new file):
    • Props: doctype, id, isOpen, onClose, onSuccess, supportsInPlaceEdit: boolean.
    • Uses useCorrectionAction internally.
    • Strategy radio group: "Amend (creates new document)" | "In-Place Edit (retains document ID)" (shown only if supportsInPlaceEdit).
    • Reason <textarea> with minimum 10 character validation.
  • Create libs/framework-m-desk/src/components/correction/index.ts exporting CorrectionWizard.
  • Run tests.

Step UI-4.2 — "Correct" Button via <Slot> (TDD)

  • Write test libs/framework-m-desk/src/__tests__/CorrectButton.test.tsx (new file):
    • When docstatus === 1: "Correct" button is visible.
    • When docstatus !== 1: button is NOT visible.
    • Clicking "Correct" opens <CorrectionWizard>.
    • onSuccess navigates user to the new document's form view (/app/{doctype}/detail/{newId}).
  • Implement libs/framework-m-desk/src/components/correction/CorrectButton.tsx (new file):
    • Props: doctype, id, docstatus, supportsInPlaceEdit.
    • Renders a button that opens <CorrectionWizard> on click.
    • On success: calls navigate(/app/{doctype}/detail/{result.id}).
  • Register <CorrectButton> into FormView.tsx's <Slot name="form-actions" /> by wiring it in the desk's built-in plugin contribution (libs/framework-m-desk/src/plugins/).
  • Run tests.

Phase UI-5: Audit History Timeline Tab

Goal: A "History" tab at the bottom of FormView showing the ActivityLog timeline for the current document.

Step UI-5.1 — useAuditTimeline Hook (TDD)

  • Write test libs/framework-m-desk/src/__tests__/useAuditTimeline.test.tsx (new file):
    • Calls Refine useList({ resource: "ActivityLog", filters: [{field: "document_id", ...}, {field: "doctype_name", ...}], sorters: [{field: "timestamp", order: "desc"}] }).
    • Returns { entries, isLoading, error }.
    • When id is undefined, does NOT fetch (query disabled).
  • Implement libs/framework-m-desk/src/hooks/useAuditTimeline.ts (new file):
    • Uses useList from @refinedev/core.
    • resource = "ActivityLog", filters on document_id and doctype_name, sort by timestamp DESC.
    • Export from libs/framework-m-desk/src/hooks/index.ts.
  • Run tests.

Step UI-5.2 — AuditTimeline Component (TDD)

  • Write test libs/framework-m-desk/src/__tests__/AuditTimeline.test.tsx (new file):
    • Loading: Shows skeleton/spinner while isLoading.
    • Empty: Shows "No history yet" when entries array is empty.
    • Renders entries: Given 3 ActivityLog entries, renders 3 timeline items each with action, user_id, timestamp.
    • Diff display: Entry with action="update" and non-null diff renders a diff summary (e.g. "price: 100 → 120").
    • Create/Submit labels: action="create" shows "Document created", action="submit" shows "Document submitted".
    • Amend label: action="amend" shows "Amended — {reason from metadata}".
  • Implement libs/framework-m-desk/src/components/audit/AuditTimeline.tsx (new file):
    • Props: doctype: string, id: string | undefined.
    • Uses useAuditTimeline hook.
    • Each entry: icon (by action), user name/email, relative timestamp ("2h ago"), collapsible diff viewer.
  • Implement libs/framework-m-desk/src/components/audit/DiffViewer.tsx (new file):
    • Props: diff: Record<string, {old: unknown, new: unknown}> | null.
    • Renders a compact <table> of field / old value / new value rows. Renders nothing when diff is null or empty.
  • Wire into FormView.tsx: Render <AuditTimeline doctype={doctype} id={id} /> inside <Slot name="form-footer" /> contribution from the desk plugin.
  • Run tests.

Phase UI-6: Sibling Overlay Panel

Goal: For documents where the backend has a SiblingMixin companion DocType, show an editable side-panel even when the parent is locked (docstatus=1).

Step UI-6.1 — useSiblingDocTypes Hook (TDD)

  • Write test libs/framework-m-desk/src/__tests__/useSiblingDocTypes.test.tsx (new file):
    • Calls GET /api/v1/meta/{doctype}/siblings (new backend endpoint — see UI-1 phase).
    • Returns { siblings: Array<{doctype, label, extends}>, isLoading }.
    • When no siblings registered, returns empty array.
  • Implement libs/framework-m-desk/src/hooks/useSiblingDocTypes.ts (new file).
  • Backend: Add GET /api/v1/meta/{doctype}/siblings endpoint in libs/framework-m-standard/src/framework_m_standard/adapters/web/ that lists DocTypes with Meta.extends == doctype registered in the MetaRegistry.

Step UI-6.2 — SiblingPanel Component (TDD)

  • Write test libs/framework-m-desk/src/__tests__/SiblingPanel.test.tsx (new file):
    • When siblings is empty: panel is NOT rendered.
    • When one sibling exists: renders a tab/section with the sibling's label.
    • Sibling panel uses a standard <AutoForm> for the sibling DocType — fully editable regardless of parent docstatus.
    • Saving sibling calls Refine useCreate or useUpdate on the sibling resource (not the parent).
    • is_metadata_overlay=true sibling is rendered in the form's right-side pane; false ones render as a footer tab.
  • Implement libs/framework-m-desk/src/components/sibling/SiblingPanel.tsx (new file).
  • Wire into FormView.tsx via <Slot name="form-footer" />.
  • Run tests.

Phase UI-7: Time-Travel UI (Optional — as_of query)

Goal: On ListView, allow filtering records "as of" a specific date/time via a date-picker, hitting GET /api/v1/{doctype}?as_of=....

Step UI-7.1 — AsOfDatePicker on ListView (TDD)

  • Write test libs/framework-m-desk/src/__tests__/AsOfDatePicker.test.tsx (new file):
    • Date picker is NOT visible by default.
    • When parent DocType Meta.versioning_enabled is true, a "View as of date" toggle appears.
    • Selecting a date appends as_of=2025-01-01T00:00:00Z to the Refine useList filters.
    • When versioning is disabled, selecting a date shows an error banner "Time travel not available for this DocType".
  • Implement: libs/framework-m-desk/src/components/list/AsOfDatePicker.tsx (new file).
  • Wire into ListView.tsx header area via <Slot name="list-filters" /> (add that slot if not present).
  • Run tests.

Phase UI-8: Exports and Package Wiring

  • Export all new components and hooks from libs/framework-m-desk/src/index.ts:
    • CorrectionWizard, CorrectButton
    • AuditTimeline, DiffViewer
    • SiblingPanel
    • SubmittedBanner
    • useCorrectionAction, useAuditTimeline, useSiblingDocTypes
  • Verify libs/framework-m-ui/src/ (if it re-exports desk components) also exports them.
  • Run vitest run libs/framework-m-desk/ — all tests pass.
  • Run tsc --noEmit in libs/framework-m-desk/ — no type errors.

New Files Summary

Path Purpose
libs/framework-m-standard/src/framework_m_standard/adapters/web/correction_router.py Litestar routes for amend/restore/in_place_edit
libs/framework-m-standard/tests/adapters/web/test_correction_endpoints.py API endpoint tests
libs/framework-m-desk/src/components/feedback/SubmittedBanner.tsx Locked-doc banner
libs/framework-m-desk/src/components/correction/CorrectionWizard.tsx Correction modal with reason + strategy
libs/framework-m-desk/src/components/correction/CorrectButton.tsx Trigger button (plugs into form-actions Slot)
libs/framework-m-desk/src/components/correction/index.ts Barrel export
libs/framework-m-desk/src/components/audit/AuditTimeline.tsx History tab component
libs/framework-m-desk/src/components/audit/DiffViewer.tsx Field diff table
libs/framework-m-desk/src/components/sibling/SiblingPanel.tsx Editable sibling overlay
libs/framework-m-desk/src/components/list/AsOfDatePicker.tsx Time-travel date picker
libs/framework-m-desk/src/hooks/useCorrectionAction.ts Calls correction REST endpoints
libs/framework-m-desk/src/hooks/useAuditTimeline.ts Fetches ActivityLog via Refine useList
libs/framework-m-desk/src/hooks/useSiblingDocTypes.ts Fetches sibling DocType metadata
libs/framework-m-desk/src/__tests__/FormView.submitted.test.tsx Tests for locked form state
libs/framework-m-desk/src/__tests__/CorrectionWizard.test.tsx Modal wizard tests
libs/framework-m-desk/src/__tests__/CorrectButton.test.tsx Button + navigation tests
libs/framework-m-desk/src/__tests__/useAuditTimeline.test.tsx Hook tests
libs/framework-m-desk/src/__tests__/AuditTimeline.test.tsx Timeline rendering tests
libs/framework-m-desk/src/__tests__/SiblingPanel.test.tsx Sibling panel tests
libs/framework-m-desk/src/__tests__/AsOfDatePicker.test.tsx Time-travel UI tests

Files to Modify

Path Change
libs/framework-m-desk/src/pages/FormView.tsx Add locked-form logic (isLocked), <SubmittedBanner>, suppress Save when locked

Phase UI-9: Final Verification

  • Run full Vitest suite with coverage: 
    cd libs/framework-m-desk
npx vitest run --coverage
    Confirm no coverage regression vs. pre-RFC-0008 baseline. Every new file should have ≥ 90% line coverage.
  • If libs/framework-m-ui re-exports desk components, run its test suite too: 
    cd libs/framework-m-ui
npx vitest run --coverage
  • Type-check — no new errors: 
    cd libs/framework-m-desk && npx tsc --noEmit
cd libs/framework-m-ui   && npx tsc --noEmit
  • Lint — no new warnings: 
    cd libs/framework-m-desk && npx eslint src/
  • Confirm vitest.config.ts in libs/framework-m-desk/ has a coverage.thresholds block added for the new files: 
    coverage: {
  provider: "v8",
  include: ["src/**"],
  thresholds: { lines: 90, functions: 90, branches: 85 },
}
    Add this block to libs/framework-m-desk/vitest.config.ts so CI enforces it going forward. | libs/framework-m-desk/src/hooks/index.ts | Export useCorrectionAction, useAuditTimeline, useSiblingDocTypes | | libs/framework-m-desk/src/index.ts | Export all new components | | libs/framework-m-standard/src/framework_m_standard/adapters/web/__init__.py | Register correction_router |

UX Flow Summary (the user's journey)

User opens submitted Invoice INV-001
  → Form is READ-ONLY (amber "Submitted" banner, Save hidden)
  → Sees "History" tab below form showing audit trail
  → Clicks "Correct" button (in form-actions area)
    → CorrectionWizard modal opens
    → Selects strategy: "Amend" or "In-Place Edit"
    → Types mandatory reason (min 10 chars)
    → Clicks Confirm
      [Amend path]  → navigated to new draft INV-001-1 (editable)
      [In-Place]    → navigated back to INV-001 but now in DRAFT state
  → Edits, saves, re-submits normally via WorkflowActions "Submit" button

Common complaints this addresses:

  • "I can't edit a submitted document" → Clear locked banner + "Correct" button as the guided path.
  • "I don't know why a document changed" → History tab with diffs on every form.
  • "Amend creates a confusing new document number" → Strategy selector explains what each option does before committing.
  • "The transporter name changed but the invoice number can't change" → Sibling overlay panel stays editable.
Edited by Ansh Pansuriya

Merge request reports