265 lines
18 KiB
Markdown
265 lines
18 KiB
Markdown
# Reconciliation client handoff — isolated module, not yet production wire-up
|
||
|
||
**Historical module-only handoff.** Production wire-up has since been implemented
|
||
in the agent dispatcher, helper control RPC and shared transport actor. Current
|
||
contract and verification boundaries: `deploy/linux/runtime-README.md`. The
|
||
earlier patch-position instructions below are retained as implementation history,
|
||
not an assertion that the modules are still unregistered.
|
||
|
||
Only new files were added for this dispatch:
|
||
`crates/agent/src/reconcile.rs`, `crates/agent/tests/reconcile.rs`, this document.
|
||
No existing agent/helper/backend/domain/protocol file or Cargo manifest was
|
||
changed. No VM, SSH, API approval, root effect or server restart was performed.
|
||
The test module uses the actual public protocol/domain contracts, OpenSSL
|
||
Ed25519 and real SQLite files, not substitute wire structs or an in-memory mock
|
||
journal. Build-directory locks were allowed to resolve normally.
|
||
|
||
## Public module API
|
||
|
||
* `ClientSync::new(Binding)` — process-local mutation gate starts closed. Existing
|
||
`safety('connected')` is never reset/deleted. The binding contains the already
|
||
provisioned administrative key, not a key received with a response.
|
||
* `Channel { connection_generation, boot_id }` — construct only after validating
|
||
the current authenticated ServerHello. Use the native Linux boot UUID, never
|
||
the agent instance/process UUID. Enrollment/assignment/tenant/asset/agent remain
|
||
bound by `Binding` and by every decoded local offer.
|
||
* `SyncClock::now()` and production `NativeClock::default()` — native synchronized
|
||
clock plus wall/monotonic jump checks. A false clock closes mutation authority;
|
||
it does not prevent independent passive telemetry.
|
||
* `initialize(&Connection)` — creates only additive `reconcile_*` tables in the
|
||
already locked, privately opened agent journal. Existing tables are required:
|
||
`counter`, `job_attempts`, `job_events`. The second connection is the existing
|
||
private `consumption.sqlite` with `consumed`/`execution_evidence`.
|
||
* `prepare(&mut journal, &consumption, Channel, Option<&HelperSnapshot>, &clock)`
|
||
returns `PreparedSync { envelope, request }`. It closes the old gate, checks
|
||
integrity/checkpoint/generation, archives original results, records ambiguous
|
||
requesting/running/verifying as unknown, reads real offers/consumption/boot/
|
||
outcomes/events and durably reserves the fresh request and sequence BEFORE
|
||
sending. Resend that exact envelope on the same live channel. After lost
|
||
response/reconnect, prepare a NEW request, using the durable prior request ID;
|
||
it does not invent a checkpoint or execution grant.
|
||
* `apply(&mut journal, &SyncResponse, &clock)` returns `Applied { decision_id,
|
||
acknowledged_message_ids, helper_instruction, reason }`. Verify exact signed
|
||
bytes before parsing, the separate reconciliation domain/profile, exact prost
|
||
request hash, every current identity/channel/boot/journal/checkpoint/key/time
|
||
binding, complete attempt coverage and exact receipt IDs. All agent changes,
|
||
result archives, ACK deletions and decision chain commit in one transaction.
|
||
Identical already-applied current responses are idempotent; stale/different
|
||
responses cannot open a gate. Terminal local state is never downgraded. Negative
|
||
server unknown decisions may carry an older server revision, but local revision
|
||
is retained with max(local, server), never reset and never made startable.
|
||
* `apply_consumption(&mut consumption, &clock)` applies only the agent process's
|
||
own targeted signed unlocks in an independent SQLite transaction. Recompare
|
||
grant/attempt/device and previous canonical local outcome, preserve consumed
|
||
tombstones, persist exact decision evidence, then allow the process gate to open.
|
||
Failure/crash between journal and consumption commits leaves mutations closed.
|
||
* `HelperInstruction` is the ORIGINAL `SyncResponse` plus its signed decision ID,
|
||
helper journal and checkpoint. Forward it unchanged to the authenticated helper.
|
||
It is never a new grant, permit, shell command or unsigned unlock boolean.
|
||
* `confirm_helper(&HelperSnapshot)` requires the independently authenticated root
|
||
helper to report that exact decision/checkpoint as durably applied and no
|
||
unsettled effects. `can_start(&Channel, root: bool, &clock)` stays false for
|
||
root until then, and expires with the signed decision deadline. Select `root`
|
||
from the verified immutable job identity, not from caller convenience.
|
||
* `close()` on every disconnect, new Hello, transport fencing or revocation.
|
||
* `archive_events(&Connection)` INSIDE the existing event-ACK transaction, before
|
||
deleting an ACKed event. This preserves original result envelope ID/sequence/
|
||
boot/time/raw outputs for future sync. No historical result is rewrapped with
|
||
a new sync envelope identity.
|
||
* `verify_response(...)` is the pure independently testable signature/binding
|
||
verifier; it does not apply or execute anything.
|
||
|
||
`HelperSnapshot { journal_id, checkpoint, previous_decision_id, attempts }` is a
|
||
complete snapshot obtained from the root-owned helper journal via authenticated
|
||
socket/peer credentials. It must NEVER be deserialized from a server response or
|
||
an unauthenticated local socket. Canonical helper outcome bytes can be imported
|
||
as a **new** durable agent JobResult with fresh event identity/revision (no new
|
||
job/attempt/effect). This resolves a lost helper response while retaining both
|
||
the older unknown result and the actual helper result. A consumed root attempt
|
||
omitted by the helper, changed helper journal/checkpoint, missing authoritative
|
||
server job/history, sequence rollback or generation regression remains blocked.
|
||
Sticky recovery faults cannot be cleared merely by omitting the offending entry.
|
||
|
||
## Minimal integration patch positions (after APT ownership release)
|
||
|
||
1. `main.rs` module declarations: add `mod reconcile;`. Agent fields: retain one
|
||
`ClientSync` and `NativeClock`, initialized from the pinned job Binding. A
|
||
passive agent without a binding must not advertise sync/root readiness.
|
||
2. `Agent::session`: advertise `rmm.sync.v1`; after validated ServerHello, before
|
||
any permit request, obtain the authenticated helper snapshot if configured,
|
||
call `prepare`, send its envelope. Both WSS and HTTPS singleton MessageBatch
|
||
carry the SAME envelope. No transport-specific readiness shortcut.
|
||
3. Inbound payload match: route current valid `Payload::SyncResponse` to `apply`;
|
||
then `apply_consumption`; forward any helper instruction unchanged, fetch the
|
||
helper's post-apply authenticated snapshot and call `confirm_helper`.
|
||
4. Replace the first-session `safe_session` condition in the scheduler AND the
|
||
immediately-before-effect dispatcher with `can_start`. Retain fresh independent
|
||
ExecutionPermit verification/consumption, immutable plan binding, normal
|
||
cancellation checks and provider resource locks. Sync never supplies start
|
||
authority or creates an executable job. Resync before the <=60s gate lease
|
||
expires, and call `close()` immediately when a channel fails/is fenced.
|
||
5. In `jobs::ack`, call `initialize` once during open and `archive_events` in the
|
||
same transaction as the concrete ACK's deletion. Without this hook, this module
|
||
still archives pending results during prepare, but cannot recover an original
|
||
event already erased by old code. Legacy final results can still reconcile
|
||
canonical hashes against authoritative stored server results.
|
||
6. Helper RPC needs typed authenticated snapshot/apply operations. Its new process
|
||
gate must start closed. Independently verify the original decision signature,
|
||
all helper identity/checkpoint/attempt/hash/action/boot/previous-outcome bindings
|
||
against ROOT-owned state, apply targeted unlocks transactionally, keep consumed
|
||
tombstones, then acknowledge its signed decision/checkpoint. An ordinary APT
|
||
probe/capability ready flag is NOT sufficient. This dispatch deliberately did
|
||
not modify the concurrently owned helper/transport files or claim this root
|
||
RPC consumer already wired. Never call agent `apply_consumption` on a helper's
|
||
root database as a substitute for independent helper verification.
|
||
|
||
The adapter refuses >256 attempts/receipts, >512 KiB canonical outcomes or a
|
||
>1 MiB request, rather than dropping unknown attempts/results. There is no unsafe
|
||
automatic compaction, journal deletion, new-grant retry or recovery-fault reset.
|
||
Coordinated rollback of every independent ledger still requires operator recovery.
|
||
|
||
## Targeted verification
|
||
|
||
Actually executed successfully during implementation:
|
||
|
||
Final result: **18 reconciliation integration tests passed**; targeted Clippy
|
||
with `-D warnings` and formatting check passed. No warnings were suppressed.
|
||
|
||
```sh
|
||
/home/Kevin/.cargo/bin/cargo test -p rmm-agent --test reconcile --locked
|
||
/home/Kevin/.cargo/bin/cargo clippy -p rmm-agent --test reconcile --locked -- -D warnings
|
||
/home/Kevin/.cargo/bin/rustfmt --edition 2024 --check crates/agent/src/reconcile.rs crates/agent/tests/reconcile.rs
|
||
```
|
||
|
||
Coverage includes real SQLite reopen, clean reconnect and accepted-unstarted jobs,
|
||
exact signatures/all identity bindings/fake key, replay and response-loss recovery,
|
||
original result retention and exact ACKs, canonical outcome rejection, unknown
|
||
resource retention, sticky missing server/helper records and checkpoint regression,
|
||
independent local outcome mismatch, actual helper-result import and original/new
|
||
boot evidence. Helper/server responses in these local tests are fixtures; no real
|
||
approval, helper root operation or backend/VM end-to-end acceptance is claimed.
|
||
|
||
## Independent helper consumer (separate ownership increment)
|
||
|
||
New files: `crates/privileged-helper/src/reconcile.rs` and
|
||
`crates/privileged-helper/tests/reconcile.rs`. Existing helper lib/main/APT and
|
||
agent files were deliberately NOT changed. This is a complete SQLite/verifier
|
||
core, **not yet registered in the production RPC dispatcher**.
|
||
|
||
### Helper API and locking
|
||
|
||
* `ExecutorGuard::acquire(state_directory, owner_uid)` opens the SAME existing
|
||
`executor.lock`. Production uses owner UID 0. `snapshot`, `apply` and `can_start`
|
||
verify the supplied connection is that directory's actual `consumption.sqlite`,
|
||
not a memory/private copy. Replace, rather than nest, existing per-request
|
||
`lock_state` acquisition. Hold this guard through grant consumption and effect.
|
||
* `HelperSync::new(independently_provisioned_Binding, &clock)` always starts closed.
|
||
The pin cannot be supplied in IPC or automatically replaced. The root-owned
|
||
journal persists a binding hash; changed key/identity requires controlled
|
||
recovery, not an in-band rebind.
|
||
* `snapshot(&mut db, &guard, generation, &clock, &boot)` closes the previous lease,
|
||
reads **every** actual consumed row plus saved Request, verifies the historical
|
||
exact Ed25519 grant at its original validity window, checks current bound
|
||
device/action/job/attempt/plan/authorization, and derives canonical outcomes,
|
||
original native boot and durable per-row revisions. It atomically reserves a
|
||
fresh monotone independent helper checkpoint. No agent-provided attempt list is
|
||
used to construct this root snapshot.
|
||
* `apply(&mut db, &guard, &SignedResponse, original_request_protobuf, &clock, &boot)`
|
||
independently verifies the exact reconciliation signature/domain/profile/key,
|
||
device and all current request/channel/journal/checkpoint/boot/time bindings.
|
||
It hashes the ORIGINAL canonical `SyncRequest::encode_to_vec()` bytes, not a
|
||
JSON rendering. All captured root attempts must appear in both request and
|
||
signed decisions; root grant/action/plan/previous canonical outcome must match.
|
||
The entire actual ledger fingerprint is rechecked while holding the same
|
||
executor guard and SQLite immediate transaction. Mutation after capture,
|
||
omissions, missing original evidence, unknown state, checkpoint/history rollback
|
||
and independently detected agent-journal replacement fail closed; recovery
|
||
faults are sticky and have no automatic reset API.
|
||
* Only explicit, independently matched signed per-attempt `unlock` decisions
|
||
update `consumed.result` and the saved outcome. The consumed grant/attempt stays
|
||
permanently present. Definitive known outcomes may be unlocked while another
|
||
unknown retains the global resource lock and readiness remains false. An unknown
|
||
script cannot become successful from a server-ready/callback boolean or a new
|
||
body; only the separately validated reboot reconciliation can supply a new
|
||
definitive body. Reboot additionally compares the helper's OWN current boot
|
||
source with the original persisted native boot and signed new-boot/inventory
|
||
evidence. Process instance UUIDs are not a boot source.
|
||
* Apply returns `Applied { snapshot, unlocked_grants, ready }`. Forward its
|
||
`snapshot` as the authenticated checkpoint ACK. It has the applied decision ID
|
||
and SAME checkpoint expected by `ClientSync::confirm_helper`. Do not call
|
||
`snapshot()` again to make the ACK: that creates a new challenge/checkpoint.
|
||
* `can_start(&db, &guard, generation, &clock, &boot)` rechecks process-local lease,
|
||
deadline, actual boot, persisted decision/checkpoint, complete ledger fingerprint
|
||
and absence of unresolved consumed resources immediately before a fresh grant.
|
||
`close()` removes the process lease. Native implementations are `NativeClock`
|
||
(synchronization/monotonic jump checks) and `NativeBoot` (Linux boot_id).
|
||
|
||
Gate admission is checked immediately before grant consumption while retaining
|
||
the executor guard through save_start, the existing final fresh-grant/clock check
|
||
and actual execution. After consumption the changed fingerprint intentionally
|
||
blocks **new** starts; it does not retroactively revoke the already admitted
|
||
locked operation. Do not retry admission after consuming the same grant or
|
||
release/reacquire the guard between admission and execution. ExecutorGuard is
|
||
movable but not shareable by reference across threads, preventing accidental
|
||
concurrent use of one flock guard by separate request handlers.
|
||
|
||
No protocol dependency or manifest change was added: the helper currently has no
|
||
direct prost/rmm-protocol dependency. Its private bounded parser reads the actual
|
||
existing SyncRequest tags 1–12 and SyncAttempt tags 1–13. Unknown/duplicate fields,
|
||
bad wire types, nonminimal/overflowing varints, truncation and size/count violations
|
||
are denied. Nested JobResult/Envelope never serve as independent root proof.
|
||
Production glue passes bytes from the real agent protocol type; shared-wire
|
||
schema changes must update this fail-closed decoder explicitly, not silently.
|
||
|
||
### Mapping and production patch positions
|
||
|
||
1. Register `mod reconcile;` in the helper **binary** `main.rs` (the standalone
|
||
module imports the package's existing library types). Construct one HelperSync
|
||
from the root-owned config/key and NativeClock; do not restore an open gate
|
||
from SQLite. Keep existing action policy, APT manifest/runtime validation,
|
||
artifact checks, privilege boundary and peer ACL intact.
|
||
2. Add typed snapshot/apply RPC variants after socket ACL/SO_PEERCRED validation.
|
||
Apply accepts original `decision_bytes`, `signature` and original canonical
|
||
SyncRequest bytes. It accepts NO external key or unlock/ready callback flag.
|
||
General and APT listeners must share the same root DB and executor guard.
|
||
3. Root `Snapshot` maps to agent `HelperSnapshot`:
|
||
journal_id/checkpoint/previous_decision_id unchanged; each LocalAttempt maps
|
||
UUIDs to strings and copies job_payload_hash, grant_id, consumption, state,
|
||
revision, original_boot_id, outcome_sha256 and canonical outcome_bytes.
|
||
Set optional result/result_envelope to None and result_message_id empty. The
|
||
agent core already imports actual helper outcomes as new durable agent results
|
||
when needed. Additional root-only plan/request hash fields remain available
|
||
for audit and independent checks.
|
||
4. Map agent `HelperInstruction.response` byte arrays directly to SignedResponse;
|
||
pass the prepared SyncRequest's original prost bytes alongside it. Send the
|
||
apply-returned authenticated snapshot to `ClientSync::confirm_helper`.
|
||
5. Add `HelperSync::can_start` checks under ExecutorGuard before **every** general
|
||
or APT grant consumption/effect path. Probe/capability flags do not replace
|
||
that check. Shared checkpoint state alone never opens another process's gate.
|
||
A fresh capture fences older process leases. Prefer one shared broker/gate for
|
||
both profiles; if they are separate processes, reconcile the selected process
|
||
before use rather than assuming both inherited readiness.
|
||
|
||
### Replay safety bound without a new protocol nonce
|
||
|
||
Each new process requires a fresh in-memory capture and a newly committed helper
|
||
checkpoint. Cached DB readiness/captures are not loaded as process authority.
|
||
Additionally the signed issued_at must be >= process birth AND the private fresh
|
||
capture time, and validity is at most 60 seconds. Thus an old cached response
|
||
cannot initialize a restarted helper, even while its old TTL has not expired;
|
||
ordinary same-process retries of the exact already-applied response are idempotent
|
||
and do not extend the lease. A local clock ahead of the signer may cause a safe
|
||
clock/reconciliation block: there is deliberately no skew grace that would admit
|
||
a pre-process response. This is a strict liveness limitation of the current
|
||
nonce-less contract, not an invented attestation or perpetual authorization.
|
||
Coordinated restore of all independent histories remains operator recovery.
|
||
|
||
The helper integration tests use actual SQLite files, real Ed25519 and the shared
|
||
executor lock, with fixture signatures and an injectable boot source for the
|
||
simulated positive reboot change. No native reboot/service/APT effect or root
|
||
deployment was performed. Targeted command:
|
||
|
||
```sh
|
||
/home/Kevin/.cargo/bin/cargo test -p rmm-privileged-helper --test reconcile --locked
|
||
```
|