Files
BuTzZ 6c3203d725
Rust development slice / linux (push) Failing after 1m10s
feat: snapshot Linux-first RMM MVP and pause handover
2026-11-12 22:06:25 +01:00

18 KiB
Raw Permalink Blame History

Reconciliation client handoff — isolated module, not yet production wire-up

Historical module-only handoff. Production wire-up has since been implemented in the agent dispatcher, helper control RPC and shared transport actor. Current contract and verification boundaries: deploy/linux/runtime-README.md. The earlier patch-position instructions below are retained as implementation history, not an assertion that the modules are still unregistered.

Only new files were added for this dispatch: crates/agent/src/reconcile.rs, crates/agent/tests/reconcile.rs, this document. No existing agent/helper/backend/domain/protocol file or Cargo manifest was changed. No VM, SSH, API approval, root effect or server restart was performed. The test module uses the actual public protocol/domain contracts, OpenSSL Ed25519 and real SQLite files, not substitute wire structs or an in-memory mock journal. Build-directory locks were allowed to resolve normally.

Public module API

  • ClientSync::new(Binding) — process-local mutation gate starts closed. Existing safety('connected') is never reset/deleted. The binding contains the already provisioned administrative key, not a key received with a response.
  • Channel { connection_generation, boot_id } — construct only after validating the current authenticated ServerHello. Use the native Linux boot UUID, never the agent instance/process UUID. Enrollment/assignment/tenant/asset/agent remain bound by Binding and by every decoded local offer.
  • SyncClock::now() and production NativeClock::default() — native synchronized clock plus wall/monotonic jump checks. A false clock closes mutation authority; it does not prevent independent passive telemetry.
  • initialize(&Connection) — creates only additive reconcile_* tables in the already locked, privately opened agent journal. Existing tables are required: counter, job_attempts, job_events. The second connection is the existing private consumption.sqlite with consumed/execution_evidence.
  • prepare(&mut journal, &consumption, Channel, Option<&HelperSnapshot>, &clock) returns PreparedSync { envelope, request }. It closes the old gate, checks integrity/checkpoint/generation, archives original results, records ambiguous requesting/running/verifying as unknown, reads real offers/consumption/boot/ outcomes/events and durably reserves the fresh request and sequence BEFORE sending. Resend that exact envelope on the same live channel. After lost response/reconnect, prepare a NEW request, using the durable prior request ID; it does not invent a checkpoint or execution grant.
  • apply(&mut journal, &SyncResponse, &clock) returns Applied { decision_id, acknowledged_message_ids, helper_instruction, reason }. Verify exact signed bytes before parsing, the separate reconciliation domain/profile, exact prost request hash, every current identity/channel/boot/journal/checkpoint/key/time binding, complete attempt coverage and exact receipt IDs. All agent changes, result archives, ACK deletions and decision chain commit in one transaction. Identical already-applied current responses are idempotent; stale/different responses cannot open a gate. Terminal local state is never downgraded. Negative server unknown decisions may carry an older server revision, but local revision is retained with max(local, server), never reset and never made startable.
  • apply_consumption(&mut consumption, &clock) applies only the agent process's own targeted signed unlocks in an independent SQLite transaction. Recompare grant/attempt/device and previous canonical local outcome, preserve consumed tombstones, persist exact decision evidence, then allow the process gate to open. Failure/crash between journal and consumption commits leaves mutations closed.
  • HelperInstruction is the ORIGINAL SyncResponse plus its signed decision ID, helper journal and checkpoint. Forward it unchanged to the authenticated helper. It is never a new grant, permit, shell command or unsigned unlock boolean.
  • confirm_helper(&HelperSnapshot) requires the independently authenticated root helper to report that exact decision/checkpoint as durably applied and no unsettled effects. can_start(&Channel, root: bool, &clock) stays false for root until then, and expires with the signed decision deadline. Select root from the verified immutable job identity, not from caller convenience.
  • close() on every disconnect, new Hello, transport fencing or revocation.
  • archive_events(&Connection) INSIDE the existing event-ACK transaction, before deleting an ACKed event. This preserves original result envelope ID/sequence/ boot/time/raw outputs for future sync. No historical result is rewrapped with a new sync envelope identity.
  • verify_response(...) is the pure independently testable signature/binding verifier; it does not apply or execute anything.

HelperSnapshot { journal_id, checkpoint, previous_decision_id, attempts } is a complete snapshot obtained from the root-owned helper journal via authenticated socket/peer credentials. It must NEVER be deserialized from a server response or an unauthenticated local socket. Canonical helper outcome bytes can be imported as a new durable agent JobResult with fresh event identity/revision (no new job/attempt/effect). This resolves a lost helper response while retaining both the older unknown result and the actual helper result. A consumed root attempt omitted by the helper, changed helper journal/checkpoint, missing authoritative server job/history, sequence rollback or generation regression remains blocked. Sticky recovery faults cannot be cleared merely by omitting the offending entry.

Minimal integration patch positions (after APT ownership release)

  1. main.rs module declarations: add mod reconcile;. Agent fields: retain one ClientSync and NativeClock, initialized from the pinned job Binding. A passive agent without a binding must not advertise sync/root readiness.
  2. Agent::session: advertise rmm.sync.v1; after validated ServerHello, before any permit request, obtain the authenticated helper snapshot if configured, call prepare, send its envelope. Both WSS and HTTPS singleton MessageBatch carry the SAME envelope. No transport-specific readiness shortcut.
  3. Inbound payload match: route current valid Payload::SyncResponse to apply; then apply_consumption; forward any helper instruction unchanged, fetch the helper's post-apply authenticated snapshot and call confirm_helper.
  4. Replace the first-session safe_session condition in the scheduler AND the immediately-before-effect dispatcher with can_start. Retain fresh independent ExecutionPermit verification/consumption, immutable plan binding, normal cancellation checks and provider resource locks. Sync never supplies start authority or creates an executable job. Resync before the <=60s gate lease expires, and call close() immediately when a channel fails/is fenced.
  5. In jobs::ack, call initialize once during open and archive_events in the same transaction as the concrete ACK's deletion. Without this hook, this module still archives pending results during prepare, but cannot recover an original event already erased by old code. Legacy final results can still reconcile canonical hashes against authoritative stored server results.
  6. Helper RPC needs typed authenticated snapshot/apply operations. Its new process gate must start closed. Independently verify the original decision signature, all helper identity/checkpoint/attempt/hash/action/boot/previous-outcome bindings against ROOT-owned state, apply targeted unlocks transactionally, keep consumed tombstones, then acknowledge its signed decision/checkpoint. An ordinary APT probe/capability ready flag is NOT sufficient. This dispatch deliberately did not modify the concurrently owned helper/transport files or claim this root RPC consumer already wired. Never call agent apply_consumption on a helper's root database as a substitute for independent helper verification.

The adapter refuses >256 attempts/receipts, >512 KiB canonical outcomes or a

1 MiB request, rather than dropping unknown attempts/results. There is no unsafe automatic compaction, journal deletion, new-grant retry or recovery-fault reset. Coordinated rollback of every independent ledger still requires operator recovery.

Targeted verification

Actually executed successfully during implementation:

Final result: 18 reconciliation integration tests passed; targeted Clippy with -D warnings and formatting check passed. No warnings were suppressed.

/home/Kevin/.cargo/bin/cargo test -p rmm-agent --test reconcile --locked
/home/Kevin/.cargo/bin/cargo clippy -p rmm-agent --test reconcile --locked -- -D warnings
/home/Kevin/.cargo/bin/rustfmt --edition 2024 --check crates/agent/src/reconcile.rs crates/agent/tests/reconcile.rs

Coverage includes real SQLite reopen, clean reconnect and accepted-unstarted jobs, exact signatures/all identity bindings/fake key, replay and response-loss recovery, original result retention and exact ACKs, canonical outcome rejection, unknown resource retention, sticky missing server/helper records and checkpoint regression, independent local outcome mismatch, actual helper-result import and original/new boot evidence. Helper/server responses in these local tests are fixtures; no real approval, helper root operation or backend/VM end-to-end acceptance is claimed.

Independent helper consumer (separate ownership increment)

New files: crates/privileged-helper/src/reconcile.rs and crates/privileged-helper/tests/reconcile.rs. Existing helper lib/main/APT and agent files were deliberately NOT changed. This is a complete SQLite/verifier core, not yet registered in the production RPC dispatcher.

Helper API and locking

  • ExecutorGuard::acquire(state_directory, owner_uid) opens the SAME existing executor.lock. Production uses owner UID 0. snapshot, apply and can_start verify the supplied connection is that directory's actual consumption.sqlite, not a memory/private copy. Replace, rather than nest, existing per-request lock_state acquisition. Hold this guard through grant consumption and effect.
  • HelperSync::new(independently_provisioned_Binding, &clock) always starts closed. The pin cannot be supplied in IPC or automatically replaced. The root-owned journal persists a binding hash; changed key/identity requires controlled recovery, not an in-band rebind.
  • snapshot(&mut db, &guard, generation, &clock, &boot) closes the previous lease, reads every actual consumed row plus saved Request, verifies the historical exact Ed25519 grant at its original validity window, checks current bound device/action/job/attempt/plan/authorization, and derives canonical outcomes, original native boot and durable per-row revisions. It atomically reserves a fresh monotone independent helper checkpoint. No agent-provided attempt list is used to construct this root snapshot.
  • apply(&mut db, &guard, &SignedResponse, original_request_protobuf, &clock, &boot) independently verifies the exact reconciliation signature/domain/profile/key, device and all current request/channel/journal/checkpoint/boot/time bindings. It hashes the ORIGINAL canonical SyncRequest::encode_to_vec() bytes, not a JSON rendering. All captured root attempts must appear in both request and signed decisions; root grant/action/plan/previous canonical outcome must match. The entire actual ledger fingerprint is rechecked while holding the same executor guard and SQLite immediate transaction. Mutation after capture, omissions, missing original evidence, unknown state, checkpoint/history rollback and independently detected agent-journal replacement fail closed; recovery faults are sticky and have no automatic reset API.
  • Only explicit, independently matched signed per-attempt unlock decisions update consumed.result and the saved outcome. The consumed grant/attempt stays permanently present. Definitive known outcomes may be unlocked while another unknown retains the global resource lock and readiness remains false. An unknown script cannot become successful from a server-ready/callback boolean or a new body; only the separately validated reboot reconciliation can supply a new definitive body. Reboot additionally compares the helper's OWN current boot source with the original persisted native boot and signed new-boot/inventory evidence. Process instance UUIDs are not a boot source.
  • Apply returns Applied { snapshot, unlocked_grants, ready }. Forward its snapshot as the authenticated checkpoint ACK. It has the applied decision ID and SAME checkpoint expected by ClientSync::confirm_helper. Do not call snapshot() again to make the ACK: that creates a new challenge/checkpoint.
  • can_start(&db, &guard, generation, &clock, &boot) rechecks process-local lease, deadline, actual boot, persisted decision/checkpoint, complete ledger fingerprint and absence of unresolved consumed resources immediately before a fresh grant. close() removes the process lease. Native implementations are NativeClock (synchronization/monotonic jump checks) and NativeBoot (Linux boot_id).

Gate admission is checked immediately before grant consumption while retaining the executor guard through save_start, the existing final fresh-grant/clock check and actual execution. After consumption the changed fingerprint intentionally blocks new starts; it does not retroactively revoke the already admitted locked operation. Do not retry admission after consuming the same grant or release/reacquire the guard between admission and execution. ExecutorGuard is movable but not shareable by reference across threads, preventing accidental concurrent use of one flock guard by separate request handlers.

No protocol dependency or manifest change was added: the helper currently has no direct prost/rmm-protocol dependency. Its private bounded parser reads the actual existing SyncRequest tags 1–12 and SyncAttempt tags 1–13. Unknown/duplicate fields, bad wire types, nonminimal/overflowing varints, truncation and size/count violations are denied. Nested JobResult/Envelope never serve as independent root proof. Production glue passes bytes from the real agent protocol type; shared-wire schema changes must update this fail-closed decoder explicitly, not silently.

Mapping and production patch positions

  1. Register mod reconcile; in the helper binary main.rs (the standalone module imports the package's existing library types). Construct one HelperSync from the root-owned config/key and NativeClock; do not restore an open gate from SQLite. Keep existing action policy, APT manifest/runtime validation, artifact checks, privilege boundary and peer ACL intact.
  2. Add typed snapshot/apply RPC variants after socket ACL/SO_PEERCRED validation. Apply accepts original decision_bytes, signature and original canonical SyncRequest bytes. It accepts NO external key or unlock/ready callback flag. General and APT listeners must share the same root DB and executor guard.
  3. Root Snapshot maps to agent HelperSnapshot: journal_id/checkpoint/previous_decision_id unchanged; each LocalAttempt maps UUIDs to strings and copies job_payload_hash, grant_id, consumption, state, revision, original_boot_id, outcome_sha256 and canonical outcome_bytes. Set optional result/result_envelope to None and result_message_id empty. The agent core already imports actual helper outcomes as new durable agent results when needed. Additional root-only plan/request hash fields remain available for audit and independent checks.
  4. Map agent HelperInstruction.response byte arrays directly to SignedResponse; pass the prepared SyncRequest's original prost bytes alongside it. Send the apply-returned authenticated snapshot to ClientSync::confirm_helper.
  5. Add HelperSync::can_start checks under ExecutorGuard before every general or APT grant consumption/effect path. Probe/capability flags do not replace that check. Shared checkpoint state alone never opens another process's gate. A fresh capture fences older process leases. Prefer one shared broker/gate for both profiles; if they are separate processes, reconcile the selected process before use rather than assuming both inherited readiness.

Replay safety bound without a new protocol nonce

Each new process requires a fresh in-memory capture and a newly committed helper checkpoint. Cached DB readiness/captures are not loaded as process authority. Additionally the signed issued_at must be >= process birth AND the private fresh capture time, and validity is at most 60 seconds. Thus an old cached response cannot initialize a restarted helper, even while its old TTL has not expired; ordinary same-process retries of the exact already-applied response are idempotent and do not extend the lease. A local clock ahead of the signer may cause a safe clock/reconciliation block: there is deliberately no skew grace that would admit a pre-process response. This is a strict liveness limitation of the current nonce-less contract, not an invented attestation or perpetual authorization. Coordinated restore of all independent histories remains operator recovery.

The helper integration tests use actual SQLite files, real Ed25519 and the shared executor lock, with fixture signatures and an injectable boot source for the simulated positive reboot change. No native reboot/service/APT effect or root deployment was performed. Targeted command:

/home/Kevin/.cargo/bin/cargo test -p rmm-privileged-helper --test reconcile --locked