Files
BuTzZ 6c3203d725
Rust development slice / linux (push) Failing after 1m10s
feat: snapshot Linux-first RMM MVP and pause handover
2026-11-12 22:06:25 +01:00

265 lines
18 KiB
Markdown
Raw Permalink Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# Reconciliation client handoff — isolated module, not yet production wire-up
**Historical module-only handoff.** Production wire-up has since been implemented
in the agent dispatcher, helper control RPC and shared transport actor. Current
contract and verification boundaries: `deploy/linux/runtime-README.md`. The
earlier patch-position instructions below are retained as implementation history,
not an assertion that the modules are still unregistered.
Only new files were added for this dispatch:
`crates/agent/src/reconcile.rs`, `crates/agent/tests/reconcile.rs`, this document.
No existing agent/helper/backend/domain/protocol file or Cargo manifest was
changed. No VM, SSH, API approval, root effect or server restart was performed.
The test module uses the actual public protocol/domain contracts, OpenSSL
Ed25519 and real SQLite files, not substitute wire structs or an in-memory mock
journal. Build-directory locks were allowed to resolve normally.
## Public module API
* `ClientSync::new(Binding)` — process-local mutation gate starts closed. Existing
`safety('connected')` is never reset/deleted. The binding contains the already
provisioned administrative key, not a key received with a response.
* `Channel { connection_generation, boot_id }` — construct only after validating
the current authenticated ServerHello. Use the native Linux boot UUID, never
the agent instance/process UUID. Enrollment/assignment/tenant/asset/agent remain
bound by `Binding` and by every decoded local offer.
* `SyncClock::now()` and production `NativeClock::default()` — native synchronized
clock plus wall/monotonic jump checks. A false clock closes mutation authority;
it does not prevent independent passive telemetry.
* `initialize(&Connection)` — creates only additive `reconcile_*` tables in the
already locked, privately opened agent journal. Existing tables are required:
`counter`, `job_attempts`, `job_events`. The second connection is the existing
private `consumption.sqlite` with `consumed`/`execution_evidence`.
* `prepare(&mut journal, &consumption, Channel, Option<&HelperSnapshot>, &clock)`
returns `PreparedSync { envelope, request }`. It closes the old gate, checks
integrity/checkpoint/generation, archives original results, records ambiguous
requesting/running/verifying as unknown, reads real offers/consumption/boot/
outcomes/events and durably reserves the fresh request and sequence BEFORE
sending. Resend that exact envelope on the same live channel. After lost
response/reconnect, prepare a NEW request, using the durable prior request ID;
it does not invent a checkpoint or execution grant.
* `apply(&mut journal, &SyncResponse, &clock)` returns `Applied { decision_id,
acknowledged_message_ids, helper_instruction, reason }`. Verify exact signed
bytes before parsing, the separate reconciliation domain/profile, exact prost
request hash, every current identity/channel/boot/journal/checkpoint/key/time
binding, complete attempt coverage and exact receipt IDs. All agent changes,
result archives, ACK deletions and decision chain commit in one transaction.
Identical already-applied current responses are idempotent; stale/different
responses cannot open a gate. Terminal local state is never downgraded. Negative
server unknown decisions may carry an older server revision, but local revision
is retained with max(local, server), never reset and never made startable.
* `apply_consumption(&mut consumption, &clock)` applies only the agent process's
own targeted signed unlocks in an independent SQLite transaction. Recompare
grant/attempt/device and previous canonical local outcome, preserve consumed
tombstones, persist exact decision evidence, then allow the process gate to open.
Failure/crash between journal and consumption commits leaves mutations closed.
* `HelperInstruction` is the ORIGINAL `SyncResponse` plus its signed decision ID,
helper journal and checkpoint. Forward it unchanged to the authenticated helper.
It is never a new grant, permit, shell command or unsigned unlock boolean.
* `confirm_helper(&HelperSnapshot)` requires the independently authenticated root
helper to report that exact decision/checkpoint as durably applied and no
unsettled effects. `can_start(&Channel, root: bool, &clock)` stays false for
root until then, and expires with the signed decision deadline. Select `root`
from the verified immutable job identity, not from caller convenience.
* `close()` on every disconnect, new Hello, transport fencing or revocation.
* `archive_events(&Connection)` INSIDE the existing event-ACK transaction, before
deleting an ACKed event. This preserves original result envelope ID/sequence/
boot/time/raw outputs for future sync. No historical result is rewrapped with
a new sync envelope identity.
* `verify_response(...)` is the pure independently testable signature/binding
verifier; it does not apply or execute anything.
`HelperSnapshot { journal_id, checkpoint, previous_decision_id, attempts }` is a
complete snapshot obtained from the root-owned helper journal via authenticated
socket/peer credentials. It must NEVER be deserialized from a server response or
an unauthenticated local socket. Canonical helper outcome bytes can be imported
as a **new** durable agent JobResult with fresh event identity/revision (no new
job/attempt/effect). This resolves a lost helper response while retaining both
the older unknown result and the actual helper result. A consumed root attempt
omitted by the helper, changed helper journal/checkpoint, missing authoritative
server job/history, sequence rollback or generation regression remains blocked.
Sticky recovery faults cannot be cleared merely by omitting the offending entry.
## Minimal integration patch positions (after APT ownership release)
1. `main.rs` module declarations: add `mod reconcile;`. Agent fields: retain one
`ClientSync` and `NativeClock`, initialized from the pinned job Binding. A
passive agent without a binding must not advertise sync/root readiness.
2. `Agent::session`: advertise `rmm.sync.v1`; after validated ServerHello, before
any permit request, obtain the authenticated helper snapshot if configured,
call `prepare`, send its envelope. Both WSS and HTTPS singleton MessageBatch
carry the SAME envelope. No transport-specific readiness shortcut.
3. Inbound payload match: route current valid `Payload::SyncResponse` to `apply`;
then `apply_consumption`; forward any helper instruction unchanged, fetch the
helper's post-apply authenticated snapshot and call `confirm_helper`.
4. Replace the first-session `safe_session` condition in the scheduler AND the
immediately-before-effect dispatcher with `can_start`. Retain fresh independent
ExecutionPermit verification/consumption, immutable plan binding, normal
cancellation checks and provider resource locks. Sync never supplies start
authority or creates an executable job. Resync before the <=60s gate lease
expires, and call `close()` immediately when a channel fails/is fenced.
5. In `jobs::ack`, call `initialize` once during open and `archive_events` in the
same transaction as the concrete ACK's deletion. Without this hook, this module
still archives pending results during prepare, but cannot recover an original
event already erased by old code. Legacy final results can still reconcile
canonical hashes against authoritative stored server results.
6. Helper RPC needs typed authenticated snapshot/apply operations. Its new process
gate must start closed. Independently verify the original decision signature,
all helper identity/checkpoint/attempt/hash/action/boot/previous-outcome bindings
against ROOT-owned state, apply targeted unlocks transactionally, keep consumed
tombstones, then acknowledge its signed decision/checkpoint. An ordinary APT
probe/capability ready flag is NOT sufficient. This dispatch deliberately did
not modify the concurrently owned helper/transport files or claim this root
RPC consumer already wired. Never call agent `apply_consumption` on a helper's
root database as a substitute for independent helper verification.
The adapter refuses >256 attempts/receipts, >512 KiB canonical outcomes or a
>1 MiB request, rather than dropping unknown attempts/results. There is no unsafe
automatic compaction, journal deletion, new-grant retry or recovery-fault reset.
Coordinated rollback of every independent ledger still requires operator recovery.
## Targeted verification
Actually executed successfully during implementation:
Final result: **18 reconciliation integration tests passed**; targeted Clippy
with `-D warnings` and formatting check passed. No warnings were suppressed.
```sh
/home/Kevin/.cargo/bin/cargo test -p rmm-agent --test reconcile --locked
/home/Kevin/.cargo/bin/cargo clippy -p rmm-agent --test reconcile --locked -- -D warnings
/home/Kevin/.cargo/bin/rustfmt --edition 2024 --check crates/agent/src/reconcile.rs crates/agent/tests/reconcile.rs
```
Coverage includes real SQLite reopen, clean reconnect and accepted-unstarted jobs,
exact signatures/all identity bindings/fake key, replay and response-loss recovery,
original result retention and exact ACKs, canonical outcome rejection, unknown
resource retention, sticky missing server/helper records and checkpoint regression,
independent local outcome mismatch, actual helper-result import and original/new
boot evidence. Helper/server responses in these local tests are fixtures; no real
approval, helper root operation or backend/VM end-to-end acceptance is claimed.
## Independent helper consumer (separate ownership increment)
New files: `crates/privileged-helper/src/reconcile.rs` and
`crates/privileged-helper/tests/reconcile.rs`. Existing helper lib/main/APT and
agent files were deliberately NOT changed. This is a complete SQLite/verifier
core, **not yet registered in the production RPC dispatcher**.
### Helper API and locking
* `ExecutorGuard::acquire(state_directory, owner_uid)` opens the SAME existing
`executor.lock`. Production uses owner UID 0. `snapshot`, `apply` and `can_start`
verify the supplied connection is that directory's actual `consumption.sqlite`,
not a memory/private copy. Replace, rather than nest, existing per-request
`lock_state` acquisition. Hold this guard through grant consumption and effect.
* `HelperSync::new(independently_provisioned_Binding, &clock)` always starts closed.
The pin cannot be supplied in IPC or automatically replaced. The root-owned
journal persists a binding hash; changed key/identity requires controlled
recovery, not an in-band rebind.
* `snapshot(&mut db, &guard, generation, &clock, &boot)` closes the previous lease,
reads **every** actual consumed row plus saved Request, verifies the historical
exact Ed25519 grant at its original validity window, checks current bound
device/action/job/attempt/plan/authorization, and derives canonical outcomes,
original native boot and durable per-row revisions. It atomically reserves a
fresh monotone independent helper checkpoint. No agent-provided attempt list is
used to construct this root snapshot.
* `apply(&mut db, &guard, &SignedResponse, original_request_protobuf, &clock, &boot)`
independently verifies the exact reconciliation signature/domain/profile/key,
device and all current request/channel/journal/checkpoint/boot/time bindings.
It hashes the ORIGINAL canonical `SyncRequest::encode_to_vec()` bytes, not a
JSON rendering. All captured root attempts must appear in both request and
signed decisions; root grant/action/plan/previous canonical outcome must match.
The entire actual ledger fingerprint is rechecked while holding the same
executor guard and SQLite immediate transaction. Mutation after capture,
omissions, missing original evidence, unknown state, checkpoint/history rollback
and independently detected agent-journal replacement fail closed; recovery
faults are sticky and have no automatic reset API.
* Only explicit, independently matched signed per-attempt `unlock` decisions
update `consumed.result` and the saved outcome. The consumed grant/attempt stays
permanently present. Definitive known outcomes may be unlocked while another
unknown retains the global resource lock and readiness remains false. An unknown
script cannot become successful from a server-ready/callback boolean or a new
body; only the separately validated reboot reconciliation can supply a new
definitive body. Reboot additionally compares the helper's OWN current boot
source with the original persisted native boot and signed new-boot/inventory
evidence. Process instance UUIDs are not a boot source.
* Apply returns `Applied { snapshot, unlocked_grants, ready }`. Forward its
`snapshot` as the authenticated checkpoint ACK. It has the applied decision ID
and SAME checkpoint expected by `ClientSync::confirm_helper`. Do not call
`snapshot()` again to make the ACK: that creates a new challenge/checkpoint.
* `can_start(&db, &guard, generation, &clock, &boot)` rechecks process-local lease,
deadline, actual boot, persisted decision/checkpoint, complete ledger fingerprint
and absence of unresolved consumed resources immediately before a fresh grant.
`close()` removes the process lease. Native implementations are `NativeClock`
(synchronization/monotonic jump checks) and `NativeBoot` (Linux boot_id).
Gate admission is checked immediately before grant consumption while retaining
the executor guard through save_start, the existing final fresh-grant/clock check
and actual execution. After consumption the changed fingerprint intentionally
blocks **new** starts; it does not retroactively revoke the already admitted
locked operation. Do not retry admission after consuming the same grant or
release/reacquire the guard between admission and execution. ExecutorGuard is
movable but not shareable by reference across threads, preventing accidental
concurrent use of one flock guard by separate request handlers.
No protocol dependency or manifest change was added: the helper currently has no
direct prost/rmm-protocol dependency. Its private bounded parser reads the actual
existing SyncRequest tags 1–12 and SyncAttempt tags 1–13. Unknown/duplicate fields,
bad wire types, nonminimal/overflowing varints, truncation and size/count violations
are denied. Nested JobResult/Envelope never serve as independent root proof.
Production glue passes bytes from the real agent protocol type; shared-wire
schema changes must update this fail-closed decoder explicitly, not silently.
### Mapping and production patch positions
1. Register `mod reconcile;` in the helper **binary** `main.rs` (the standalone
module imports the package's existing library types). Construct one HelperSync
from the root-owned config/key and NativeClock; do not restore an open gate
from SQLite. Keep existing action policy, APT manifest/runtime validation,
artifact checks, privilege boundary and peer ACL intact.
2. Add typed snapshot/apply RPC variants after socket ACL/SO_PEERCRED validation.
Apply accepts original `decision_bytes`, `signature` and original canonical
SyncRequest bytes. It accepts NO external key or unlock/ready callback flag.
General and APT listeners must share the same root DB and executor guard.
3. Root `Snapshot` maps to agent `HelperSnapshot`:
journal_id/checkpoint/previous_decision_id unchanged; each LocalAttempt maps
UUIDs to strings and copies job_payload_hash, grant_id, consumption, state,
revision, original_boot_id, outcome_sha256 and canonical outcome_bytes.
Set optional result/result_envelope to None and result_message_id empty. The
agent core already imports actual helper outcomes as new durable agent results
when needed. Additional root-only plan/request hash fields remain available
for audit and independent checks.
4. Map agent `HelperInstruction.response` byte arrays directly to SignedResponse;
pass the prepared SyncRequest's original prost bytes alongside it. Send the
apply-returned authenticated snapshot to `ClientSync::confirm_helper`.
5. Add `HelperSync::can_start` checks under ExecutorGuard before **every** general
or APT grant consumption/effect path. Probe/capability flags do not replace
that check. Shared checkpoint state alone never opens another process's gate.
A fresh capture fences older process leases. Prefer one shared broker/gate for
both profiles; if they are separate processes, reconcile the selected process
before use rather than assuming both inherited readiness.
### Replay safety bound without a new protocol nonce
Each new process requires a fresh in-memory capture and a newly committed helper
checkpoint. Cached DB readiness/captures are not loaded as process authority.
Additionally the signed issued_at must be >= process birth AND the private fresh
capture time, and validity is at most 60 seconds. Thus an old cached response
cannot initialize a restarted helper, even while its old TTL has not expired;
ordinary same-process retries of the exact already-applied response are idempotent
and do not extend the lease. A local clock ahead of the signer may cause a safe
clock/reconciliation block: there is deliberately no skew grace that would admit
a pre-process response. This is a strict liveness limitation of the current
nonce-less contract, not an invented attestation or perpetual authorization.
Coordinated restore of all independent histories remains operator recovery.
The helper integration tests use actual SQLite files, real Ed25519 and the shared
executor lock, with fixture signatures and an injectable boot source for the
simulated positive reboot change. No native reboot/service/APT effect or root
deployment was performed. Targeted command:
```sh
/home/Kevin/.cargo/bin/cargo test -p rmm-privileged-helper --test reconcile --locked
```