18 KiB
Reconciliation client handoff — isolated module, not yet production wire-up
Historical module-only handoff. Production wire-up has since been implemented
in the agent dispatcher, helper control RPC and shared transport actor. Current
contract and verification boundaries: deploy/linux/runtime-README.md. The
earlier patch-position instructions below are retained as implementation history,
not an assertion that the modules are still unregistered.
Only new files were added for this dispatch:
crates/agent/src/reconcile.rs, crates/agent/tests/reconcile.rs, this document.
No existing agent/helper/backend/domain/protocol file or Cargo manifest was
changed. No VM, SSH, API approval, root effect or server restart was performed.
The test module uses the actual public protocol/domain contracts, OpenSSL
Ed25519 and real SQLite files, not substitute wire structs or an in-memory mock
journal. Build-directory locks were allowed to resolve normally.
Public module API
ClientSync::new(Binding)— process-local mutation gate starts closed. Existingsafety('connected')is never reset/deleted. The binding contains the already provisioned administrative key, not a key received with a response.Channel { connection_generation, boot_id }— construct only after validating the current authenticated ServerHello. Use the native Linux boot UUID, never the agent instance/process UUID. Enrollment/assignment/tenant/asset/agent remain bound byBindingand by every decoded local offer.SyncClock::now()and productionNativeClock::default()— native synchronized clock plus wall/monotonic jump checks. A false clock closes mutation authority; it does not prevent independent passive telemetry.initialize(&Connection)— creates only additivereconcile_*tables in the already locked, privately opened agent journal. Existing tables are required:counter,job_attempts,job_events. The second connection is the existing privateconsumption.sqlitewithconsumed/execution_evidence.prepare(&mut journal, &consumption, Channel, Option<&HelperSnapshot>, &clock)returnsPreparedSync { envelope, request }. It closes the old gate, checks integrity/checkpoint/generation, archives original results, records ambiguous requesting/running/verifying as unknown, reads real offers/consumption/boot/ outcomes/events and durably reserves the fresh request and sequence BEFORE sending. Resend that exact envelope on the same live channel. After lost response/reconnect, prepare a NEW request, using the durable prior request ID; it does not invent a checkpoint or execution grant.apply(&mut journal, &SyncResponse, &clock)returnsApplied { decision_id, acknowledged_message_ids, helper_instruction, reason }. Verify exact signed bytes before parsing, the separate reconciliation domain/profile, exact prost request hash, every current identity/channel/boot/journal/checkpoint/key/time binding, complete attempt coverage and exact receipt IDs. All agent changes, result archives, ACK deletions and decision chain commit in one transaction. Identical already-applied current responses are idempotent; stale/different responses cannot open a gate. Terminal local state is never downgraded. Negative server unknown decisions may carry an older server revision, but local revision is retained with max(local, server), never reset and never made startable.apply_consumption(&mut consumption, &clock)applies only the agent process's own targeted signed unlocks in an independent SQLite transaction. Recompare grant/attempt/device and previous canonical local outcome, preserve consumed tombstones, persist exact decision evidence, then allow the process gate to open. Failure/crash between journal and consumption commits leaves mutations closed.HelperInstructionis the ORIGINALSyncResponseplus its signed decision ID, helper journal and checkpoint. Forward it unchanged to the authenticated helper. It is never a new grant, permit, shell command or unsigned unlock boolean.confirm_helper(&HelperSnapshot)requires the independently authenticated root helper to report that exact decision/checkpoint as durably applied and no unsettled effects.can_start(&Channel, root: bool, &clock)stays false for root until then, and expires with the signed decision deadline. Selectrootfrom the verified immutable job identity, not from caller convenience.close()on every disconnect, new Hello, transport fencing or revocation.archive_events(&Connection)INSIDE the existing event-ACK transaction, before deleting an ACKed event. This preserves original result envelope ID/sequence/ boot/time/raw outputs for future sync. No historical result is rewrapped with a new sync envelope identity.verify_response(...)is the pure independently testable signature/binding verifier; it does not apply or execute anything.
HelperSnapshot { journal_id, checkpoint, previous_decision_id, attempts } is a
complete snapshot obtained from the root-owned helper journal via authenticated
socket/peer credentials. It must NEVER be deserialized from a server response or
an unauthenticated local socket. Canonical helper outcome bytes can be imported
as a new durable agent JobResult with fresh event identity/revision (no new
job/attempt/effect). This resolves a lost helper response while retaining both
the older unknown result and the actual helper result. A consumed root attempt
omitted by the helper, changed helper journal/checkpoint, missing authoritative
server job/history, sequence rollback or generation regression remains blocked.
Sticky recovery faults cannot be cleared merely by omitting the offending entry.
Minimal integration patch positions (after APT ownership release)
main.rsmodule declarations: addmod reconcile;. Agent fields: retain oneClientSyncandNativeClock, initialized from the pinned job Binding. A passive agent without a binding must not advertise sync/root readiness.Agent::session: advertisermm.sync.v1; after validated ServerHello, before any permit request, obtain the authenticated helper snapshot if configured, callprepare, send its envelope. Both WSS and HTTPS singleton MessageBatch carry the SAME envelope. No transport-specific readiness shortcut.- Inbound payload match: route current valid
Payload::SyncResponsetoapply; thenapply_consumption; forward any helper instruction unchanged, fetch the helper's post-apply authenticated snapshot and callconfirm_helper. - Replace the first-session
safe_sessioncondition in the scheduler AND the immediately-before-effect dispatcher withcan_start. Retain fresh independent ExecutionPermit verification/consumption, immutable plan binding, normal cancellation checks and provider resource locks. Sync never supplies start authority or creates an executable job. Resync before the <=60s gate lease expires, and callclose()immediately when a channel fails/is fenced. - In
jobs::ack, callinitializeonce during open andarchive_eventsin the same transaction as the concrete ACK's deletion. Without this hook, this module still archives pending results during prepare, but cannot recover an original event already erased by old code. Legacy final results can still reconcile canonical hashes against authoritative stored server results. - Helper RPC needs typed authenticated snapshot/apply operations. Its new process
gate must start closed. Independently verify the original decision signature,
all helper identity/checkpoint/attempt/hash/action/boot/previous-outcome bindings
against ROOT-owned state, apply targeted unlocks transactionally, keep consumed
tombstones, then acknowledge its signed decision/checkpoint. An ordinary APT
probe/capability ready flag is NOT sufficient. This dispatch deliberately did
not modify the concurrently owned helper/transport files or claim this root
RPC consumer already wired. Never call agent
apply_consumptionon a helper's root database as a substitute for independent helper verification.
The adapter refuses >256 attempts/receipts, >512 KiB canonical outcomes or a
1 MiB request, rather than dropping unknown attempts/results. There is no unsafe automatic compaction, journal deletion, new-grant retry or recovery-fault reset. Coordinated rollback of every independent ledger still requires operator recovery.
Targeted verification
Actually executed successfully during implementation:
Final result: 18 reconciliation integration tests passed; targeted Clippy
with -D warnings and formatting check passed. No warnings were suppressed.
/home/Kevin/.cargo/bin/cargo test -p rmm-agent --test reconcile --locked
/home/Kevin/.cargo/bin/cargo clippy -p rmm-agent --test reconcile --locked -- -D warnings
/home/Kevin/.cargo/bin/rustfmt --edition 2024 --check crates/agent/src/reconcile.rs crates/agent/tests/reconcile.rs
Coverage includes real SQLite reopen, clean reconnect and accepted-unstarted jobs, exact signatures/all identity bindings/fake key, replay and response-loss recovery, original result retention and exact ACKs, canonical outcome rejection, unknown resource retention, sticky missing server/helper records and checkpoint regression, independent local outcome mismatch, actual helper-result import and original/new boot evidence. Helper/server responses in these local tests are fixtures; no real approval, helper root operation or backend/VM end-to-end acceptance is claimed.
Independent helper consumer (separate ownership increment)
New files: crates/privileged-helper/src/reconcile.rs and
crates/privileged-helper/tests/reconcile.rs. Existing helper lib/main/APT and
agent files were deliberately NOT changed. This is a complete SQLite/verifier
core, not yet registered in the production RPC dispatcher.
Helper API and locking
ExecutorGuard::acquire(state_directory, owner_uid)opens the SAME existingexecutor.lock. Production uses owner UID 0.snapshot,applyandcan_startverify the supplied connection is that directory's actualconsumption.sqlite, not a memory/private copy. Replace, rather than nest, existing per-requestlock_stateacquisition. Hold this guard through grant consumption and effect.HelperSync::new(independently_provisioned_Binding, &clock)always starts closed. The pin cannot be supplied in IPC or automatically replaced. The root-owned journal persists a binding hash; changed key/identity requires controlled recovery, not an in-band rebind.snapshot(&mut db, &guard, generation, &clock, &boot)closes the previous lease, reads every actual consumed row plus saved Request, verifies the historical exact Ed25519 grant at its original validity window, checks current bound device/action/job/attempt/plan/authorization, and derives canonical outcomes, original native boot and durable per-row revisions. It atomically reserves a fresh monotone independent helper checkpoint. No agent-provided attempt list is used to construct this root snapshot.apply(&mut db, &guard, &SignedResponse, original_request_protobuf, &clock, &boot)independently verifies the exact reconciliation signature/domain/profile/key, device and all current request/channel/journal/checkpoint/boot/time bindings. It hashes the ORIGINAL canonicalSyncRequest::encode_to_vec()bytes, not a JSON rendering. All captured root attempts must appear in both request and signed decisions; root grant/action/plan/previous canonical outcome must match. The entire actual ledger fingerprint is rechecked while holding the same executor guard and SQLite immediate transaction. Mutation after capture, omissions, missing original evidence, unknown state, checkpoint/history rollback and independently detected agent-journal replacement fail closed; recovery faults are sticky and have no automatic reset API.- Only explicit, independently matched signed per-attempt
unlockdecisions updateconsumed.resultand the saved outcome. The consumed grant/attempt stays permanently present. Definitive known outcomes may be unlocked while another unknown retains the global resource lock and readiness remains false. An unknown script cannot become successful from a server-ready/callback boolean or a new body; only the separately validated reboot reconciliation can supply a new definitive body. Reboot additionally compares the helper's OWN current boot source with the original persisted native boot and signed new-boot/inventory evidence. Process instance UUIDs are not a boot source. - Apply returns
Applied { snapshot, unlocked_grants, ready }. Forward itssnapshotas the authenticated checkpoint ACK. It has the applied decision ID and SAME checkpoint expected byClientSync::confirm_helper. Do not callsnapshot()again to make the ACK: that creates a new challenge/checkpoint. can_start(&db, &guard, generation, &clock, &boot)rechecks process-local lease, deadline, actual boot, persisted decision/checkpoint, complete ledger fingerprint and absence of unresolved consumed resources immediately before a fresh grant.close()removes the process lease. Native implementations areNativeClock(synchronization/monotonic jump checks) andNativeBoot(Linux boot_id).
Gate admission is checked immediately before grant consumption while retaining the executor guard through save_start, the existing final fresh-grant/clock check and actual execution. After consumption the changed fingerprint intentionally blocks new starts; it does not retroactively revoke the already admitted locked operation. Do not retry admission after consuming the same grant or release/reacquire the guard between admission and execution. ExecutorGuard is movable but not shareable by reference across threads, preventing accidental concurrent use of one flock guard by separate request handlers.
No protocol dependency or manifest change was added: the helper currently has no direct prost/rmm-protocol dependency. Its private bounded parser reads the actual existing SyncRequest tags 1–12 and SyncAttempt tags 1–13. Unknown/duplicate fields, bad wire types, nonminimal/overflowing varints, truncation and size/count violations are denied. Nested JobResult/Envelope never serve as independent root proof. Production glue passes bytes from the real agent protocol type; shared-wire schema changes must update this fail-closed decoder explicitly, not silently.
Mapping and production patch positions
- Register
mod reconcile;in the helper binarymain.rs(the standalone module imports the package's existing library types). Construct one HelperSync from the root-owned config/key and NativeClock; do not restore an open gate from SQLite. Keep existing action policy, APT manifest/runtime validation, artifact checks, privilege boundary and peer ACL intact. - Add typed snapshot/apply RPC variants after socket ACL/SO_PEERCRED validation.
Apply accepts original
decision_bytes,signatureand original canonical SyncRequest bytes. It accepts NO external key or unlock/ready callback flag. General and APT listeners must share the same root DB and executor guard. - Root
Snapshotmaps to agentHelperSnapshot: journal_id/checkpoint/previous_decision_id unchanged; each LocalAttempt maps UUIDs to strings and copies job_payload_hash, grant_id, consumption, state, revision, original_boot_id, outcome_sha256 and canonical outcome_bytes. Set optional result/result_envelope to None and result_message_id empty. The agent core already imports actual helper outcomes as new durable agent results when needed. Additional root-only plan/request hash fields remain available for audit and independent checks. - Map agent
HelperInstruction.responsebyte arrays directly to SignedResponse; pass the prepared SyncRequest's original prost bytes alongside it. Send the apply-returned authenticated snapshot toClientSync::confirm_helper. - Add
HelperSync::can_startchecks under ExecutorGuard before every general or APT grant consumption/effect path. Probe/capability flags do not replace that check. Shared checkpoint state alone never opens another process's gate. A fresh capture fences older process leases. Prefer one shared broker/gate for both profiles; if they are separate processes, reconcile the selected process before use rather than assuming both inherited readiness.
Replay safety bound without a new protocol nonce
Each new process requires a fresh in-memory capture and a newly committed helper checkpoint. Cached DB readiness/captures are not loaded as process authority. Additionally the signed issued_at must be >= process birth AND the private fresh capture time, and validity is at most 60 seconds. Thus an old cached response cannot initialize a restarted helper, even while its old TTL has not expired; ordinary same-process retries of the exact already-applied response are idempotent and do not extend the lease. A local clock ahead of the signer may cause a safe clock/reconciliation block: there is deliberately no skew grace that would admit a pre-process response. This is a strict liveness limitation of the current nonce-less contract, not an invented attestation or perpetual authorization. Coordinated restore of all independent histories remains operator recovery.
The helper integration tests use actual SQLite files, real Ed25519 and the shared executor lock, with fixture signatures and an injectable boot source for the simulated positive reboot change. No native reboot/service/APT effect or root deployment was performed. Targeted command:
/home/Kevin/.cargo/bin/cargo test -p rmm-privileged-helper --test reconcile --locked