Files
rust-discovery/proto/BACKEND_CONTRACT.md
BuTzZ 6c3203d725
Rust development slice / linux (push) Failing after 1m10s
feat: snapshot Linux-first RMM MVP and pause handover
2026-11-12 22:06:25 +01:00

43 KiB
Raw Permalink Blame History

Backend integration contract — codec/protocol 1, development only

Root QueryLogs allowlist profile 1

Root query_logs now uses the same RootScope authorization path as root scripts, services and reboots at planning, submission recheck, human approval and final permit. It still requires explicit dev-root enablement, the selected tenant, asset, agent, enrollment/assignment, principal and helper-policy capability bindings; this change does not enable root or create/approve jobs.

Prefer the following optional action-specific object in each private root-policy scope (the surrounding policy remains schema_version 1):

"root_logs": {
  "schema_version": 1,
  "allowed_units": ["rmm-acceptance.service"],
  "sensitive_units": ["custom-audit.service"]
}

allowed_units is exact, case-sensitive, max 128 distinct .service names. An explicit empty list denies logs for that scope. There is no fallback to allowed_services: permission to mutate a service is not permission to read its logs, nor the reverse. Unknown log-profile versions, paths, wildcards, duplicates, option-like names and malformed lists fail closed. No SSH or shell fallback exists.

Backend actor environment variables, parsed only as bounded JSON string arrays:

  • RMM_ROOT_LOG_UNITS='["rmm-acceptance.service"]': explicit global log-unit source for selected scopes without a root_logs object. If a scope has root_logs, the environment acts only as an additional intersection/ceiling, never a way to widen its list. Unset plus no scoped list means empty/deny. Empty string or [] is explicit deny-all. A boolean, filename or shell fragment is not accepted.
  • RMM_ROOT_SENSITIVE_LOG_UNITS='["custom-audit.service"]': additional sensitive classifications. It never allows a unit or removes an existing classification.

Set these in the actual backend container/process if using environment-based configuration; a variable only in the VM harness/agent environment does not alter backend policy. Keep RMM_PROFILE=development, RMM_DEV_ROOT_ENABLED and the private RMM_DEV_ROOT_POLICY_FILE explicit; no test automatically sets them. The agent's RMM_LOG_UNITS_FILE and the helper's local unit policy remain independent limits. The current helper uses its own allowed_services list for journal-unit checks; that does not replace the backend's separate read allowlist or grant service mutation rights through this new backend log policy.

Every root-log authorization requires current root:execute and logs:read membership permissions. Sensitive sources additionally require current logs:read_sensitive, independently for requester and approver wherever the shared root authorization service is invoked. No viewer role, capability, allowlist, dev flag or root permission silently grants sensitive-log access. Sensitive classifications are unioned across all scopes for the same target, so choosing another principal/scope cannot declassify a source. ssh.service, sshd.service, sshd@*.service instances, auditd.service and systemd-logind.service are always sensitive, even if a configured sensitive list is empty. No membership is seeded or changed by this patch. Stored result-reader APIs are unchanged; this profile governs root execution authorization, not a new general data grant.

Root log jobs bind one exact unit and UTC interval, with nonnegative timestamps, positive interval at most exactly 24h (no rounded-hour widening), 1–1000 lines, 20–3600s runtime and 1–65536 output bytes. Standard capabilities remain required: query_logs, root:query_logs, rmm.sync.v1, helper.root.v1 and the exact helper.policy.sha256:. Full job/plan bytes and root identity are unchanged; helper verification, resource locks, current start permission and the human UI approval boundary still apply.

Policy tests use synthetic job shapes/permission requirements only, not fabricated DB memberships or approvals. A real selected-policy root journal read and sensitive member acceptance remain native/browser integration cases, not claimed by unit tests.

Public certificate evidence (migration 0012)

Device detail and list items now additionally expose:

  • certificate_sha256: lowercase SHA-256 of the current registered/most recently issued leaf DER certificate. Rotation updates this pin immediately; issuance alone is not proof that the new certificate was used in a TLS connection.
  • last_authenticated_certificate_sha256: nullable SHA-256 of the actual verified TLS peer certificate at the most recent successful registered-device binding lookup; last_authenticated_at is its database UTC RFC3339 observation time. Both are null until such evidence is observed after this migration. They are not backfilled from issuance or historical heartbeats.

The binding lookup atomically records evidence only for an unrevoked current pin or a still-valid previous-certificate overlap. It is called by WSS authentication and device mTLS HTTP paths, including key bootstrap, renewal and artifact access. Thus a lost renewal response/old-certificate overlap may leave the current pin new while the most recently authenticated peer remains old. A subsequent successful new-certificate Hello records the new peer without changing agent ID or enrollment generation. This is most recent authentication evidence, not an assertion that every concurrent channel uses the same certificate.

Fingerprints are derived by the TLS server, never from a client header/payload. Authentication time is separate from presence: these lookups do not update last_contact_at. Only an accepted fresh heartbeat updates presence; old telemetry, key/artifact GET and certificate renewal do not. Invalid/unregistered/revoked or expired-overlap certificates do not update authentication evidence. Current membership/tenant checks apply to both public DTOs; foreign assets remain 404. No private key, CSR, certificate PEM or raw credential is added to device DTOs.

This is the implemented backend increment, not Linux-MVP acceptance. No execution, human/browser approval, VM job, result replay, or helper acceptance is claimed by this document. tests/backend_contract.py deliberately stops at pending approval and denial of preapproval start requests.

Browser and HTTP

The compiled web/dist/index.html and /assets/* are served on the same TLS origin (RMM_WEB_DIR, Compose /web). Unknown API routes do not fall back to HTML. Development profile is required; production refuses to start.

Run python3 deploy/dev.py jobs-auth once for an existing initialized dev environment. It generates separate private browser and operator credentials in gitignored devstate/, never prints values, and does not authorize anything. It refuses credential rotation/overwrite. Existing viewer credentials remain read-only. devstate/browser-login.json is mode 0600; use this path for the browser harness rather than copying secrets into source or command arguments.

  • POST /auth/dev/login {username,password} requires exact Origin https://rmm-dev.internal:18443 (configurable RMM_UI_ORIGIN). A fixed simulated-testprincipal-a, tenant A only, gets a one-hour in-memory session. Cookie: __Host-rmm_session, HttpOnly, Secure, SameSite=Strict, Path=/.
  • GET /auth/session returns {principal:string,tenant_id,csrf_token,roles, testsimulation:true,approval_origin:"dev_test_browser_simulation"}.
  • Cookie-authenticated mutations require Origin and X-CSRF-Token. POST /auth/logout {} revokes the session. Server restart requires re-login; it does not erase durable plans, approvals, jobs or reservations.
  • An Authorization header always selects bearer authentication, even alongside a valid cookie. Bearer/MCP principals never get an interactive context.

Business prefix: /api/v1/tenants/{tenant}. Supported paths:

  • GET /devices, /devices/{asset}, /health, /operations, /operations/{operation}, /jobs/{job}, /operation-plans/{plan}.
  • POST /operation-plans, /operation-plans/{plan}/submit {idempotency_key}, /jobs/{job}/cancel {reason}.
  • POST /operation-plans/{plan}/authorizations {plan_hash,decision?}: decision defaults to approve; explicit reject supported. This handler requires the interactive cookie context, exact hash, persistent pending request, current scopes/capabilities/assignment, deadline and limiting policy. It is not exposed as an MCP service. Approval creates authorization, operation, fixed jobs and outbox atomically. Repeated approval returns the same operation.
  • POST /log-queries uses the LogQuery DTO in server/monitoring.rs. Stored results are bounded and explicitly partial. Live mode creates a fixed plan and pending request; it never invokes a collector directly.

Plan creation accepts the UI's spec-23.4 shape, including tenant_id, action string, targets, parameters, schedule/offline_policy, rollout and reason. Exactly one Linux UTF-8 script variant is supported; ambiguous variants fail closed. The internal typed PlanInput is also accepted and is used by shared services. agent_service remains the default. Selected root script/service/reboot plans require the explicit, expiring dev policy and independently enforcing helper profile described below; root is not enabled by default. Legacy package actions never gain root authority. Package actions bind architecture, recipe_version and installation_context; updates require reboot_policy:"never"; reboot binds dependency_group.

Plan detail/creation returns {plan_id,plan_hash,expires_at,plan_payload, plan_payload_bytes,resolved_targets:[{asset_id,agent_id}],excluded_targets:[], jobs:[{job_id,attempt_id,asset_id,agent_id,job_payload_hash,script_sha256,payload}], approval_state}. plan_payload_bytes is the actual stored byte array, not a JSON view re-hashed by the UI. Detail exposes a submitted pending request after reload. Job/attempt IDs in plans are reservations, not executable job rows.

Submit returns the stable operation including approval_state, state alias, planned_job_count, job_count, execution_state, counts and job summaries. Pending approval has zero jobs and null execution_state. Reusing an idempotency key for another plan conflicts; repeated submit of one plan returns its one operation. Operation/list summaries omit potentially large result outputs: fetch /jobs/{id} for the bounded result. Its stdout/stderr are raw JSON byte arrays.

Device, operation and package-recipe lists support ?limit=1..500&cursor=... (default 100). Signed cursors bind tenant, actual principal, resource/sort, page size and a fixed as_of, and expire after ten minutes. New rows after as_of are excluded; IDs use ascending UUID order. Additional dynamic filters are not implemented and unknown query fields are rejected on these paged routes. Inventory/health freshness and job status remain live observations, not a transactional historical snapshot. Health/log evidence remains explicitly bounded.

Shared Rust payloads and wire

Authoritative Rust contracts:

  • rmm_domain::jobs::{Action,JobPayload,PlanPayload,PlannedJob, ExecutionPermitPayload,JobState,RecipeReference,encode,hash}.
  • rmm_protocol::{Envelope,Payload,JobOffer,JobAccepted, ExecutionPermitRequest,ExecutionPermit,JobStateChanged,JobResult,CancelRequest}.
  • Schema mirror: proto/agent.proto; additive oneof tags 20–26, reserved 15–19. Existing passive tags and protocol version 1.0 remain unchanged. HTTP adds top-level MessageBatch and PollRequest. Reconciliation adds Envelope tags 27/28.

Codec 1: compact UTF-8 serde JSON of declared structs in declaration field order, explicit null optional fields, ordered arrays, UTC RFC3339 times, integer limits, no unordered parameter maps or floating-point values. The server stores exact plan/job bytes once. SHA-256 hashes those bytes. Script hashes cover exact UTF-8 bytes, including BOM/newlines; there is no postapproval normalization.

JobPayload: schema_version, tenant_id, asset_id, agent_id, enrollment_generation, assignment_version, job_id, attempt_id, typed Action, execution_identity, not_before, start_before, max_runtime_seconds, max_output_bytes. Action is a flattened tagged object (action:"run_script", interpreter, script_utf8, arguments, expected_stdout), inside JobPayload.action. Inspect the Rust enum for the other six complete typed action shapes.

Administrative signature profile: Ed25519 only, OpenSSL, 64-byte signature over the exact ExecutionPermitPayload bytes. Signed payload starts with fields domain:"rmm.execution-permit", schema_version:1. It binds grant_id, authorization_id, tenant_id, asset_id, agent_id, enrollment_generation, assignment_version, job_id, attempt_id, plan_hash, job_payload_hash, action, execution_identity, not_before, start_before (at most 60 seconds), max_runtime_seconds and key_id. Key ID is SHA-256 of raw 32-byte public key. The protected persistent key is devstate/jobs/ed25519.pem, separate from TLS CA and software-update keys. GET /api/v1/job-signing-key is authenticated and returns key_id, algorithm, schema_version, public_key_hex. Provision/pin that public key over the verified bootstrap channel. A key sent with a job is not a trust anchor.

Agent WSS sequence

  1. Connect with enrolled mTLS identity; ClientHello sequence >0 and connection generation 0. Advertise only action names that actually work on this device: run_script, refresh_inventory, query_logs, set_service_state, ensure_package, apply_updates, reboot. The server persists capabilities for planning checks.
  2. ServerHello selects the connection generation. Every subsequent fresh request uses it. Heartbeat/inventory handling stays compatible. Honest short bursts of journal/state traffic are allowed; limit is 20 inbound messages/second.
  3. JobOffer is sent only from the postapproval durable outbox, after ServerHello or the independent one-second WSS delivery timer/normal ACK traffic. It includes exact payload_bytes, hashes, authorization and fixed IDs. It grants no execution permission. Commit the offer to SQLite before sending JobAccepted with the same IDs/hash and a bounded nonempty journal_proof. Repeat unacknowledged durable envelopes unchanged.
  4. Server sends Ack only after durable acceptance/receipt commit. A fresh ExecutionPermitRequest immediately before effect includes job_id, attempt_id, job_payload_hash, assignment_version. Its response is ExecutionPermit, not Ack. Denial uses denial_code and empty payload/signature. Do not offline-buffer permit requests or treat any denial as authority to execute.
  5. On grant receipt, agent and helper verify Ed25519 against the pinned key, all signed bindings against the exact job bytes/current device identity, trusted time/deadline and execution identity, then durably consume both grant and (agent,generation,job,attempt) before effect. Hold local resource locks. The backend commits reservation and signed bytes before transmission. It never issues another grant for this attempt, even if the first response was lost.
  6. JobStateChanged carries revision >0, state, IDs and grant_id. Running, verifying, awaiting_reboot and unknown are nonterminal state events. JobResult carries revision, terminal/unknown/awaiting_reboot state, grant_id, raw bounded stdout/stderr, output_truncated, reboot_required, optional exit_code, evidence_json and error_code. Both are durable until concrete message-ID Ack. A valid late terminal result may reconstruct missing intermediate states. Lower-revision state evidence never resets a terminal/unknown ledger.
  7. Success evidence needs postconditions_passed:true. Scripts additionally require exit_code 0 and exact expected_stdout when bound. Reboot success needs evidence new_boot_id equal to the authenticated envelope boot_id, different from the boot recorded at grant reservation. Failed results need error_code. Cancellation needs not_started:true or abort_confirmed:true. A definite prestart failed/cancelled/expired result may omit grant_id only when no grant was ever reserved and evidence contains not_started:true.
  8. Grant issuance is not a start acknowledgement. After an unconfirmed reserved start window or running deadline the ledger becomes unknown, never expired/ cancelled by inference. A late definitive bound result can reconcile unknown, with an audit event. No retry grant is created. Unknown keeps resource/rollout reservations; a new action requires a newly approved plan, not redelivery.

Envelope message-ID identity includes observed_at/sequence/boot/payload; changing connection_generation on retransmission is the only hash-normalized field. Late durable job evidence may be ingested from a fenced channel, but it never refreshes presence or obtains new offers/grants. Telemetry aging quotas do not discard old job results. End-to-end restore synchronization remains an open gate.

MCP

Real rmcp 3.9.0, Streamable HTTP /mcp, tested protocol 2025-11-25 legacy initialize/notifications-initialized/session lifecycle. 2026-07-28 is not claimed. Use verified TLS and Authorization bearer on every HTTP request, plus Accept application/json, text/event-stream; carry returned Mcp-Session-Id. SDK validates Host and browser Origin; MCP cookies alone never authenticate.

Ten tools: tenants.list, inventory.list_devices/get_device, monitoring.get_health, logs.query, operations.plan/submit, jobs.get/cancel, sessions.request. No approval tool. Sessions explicitly return unsupported. Operations.plan arguments are {tenant_id,input:<same HTTP plan DTO>}; operations.submit {tenant_id,plan_id,idempotency_key}; jobs.get requires exactly one of operation_id/job_id. Tool schemas derive from the same Rust input types; adapters call shared job/inventory/monitoring services, not alternate policies.

Known acceptance and implementation gaps

  • Agent executor, journal consumption, independently verifying privileged helper, real browser approval, positive grant/result/replay tests, and all VM jobs are not implemented/proved by this backend dispatch. Do not advertise action capabilities merely because a provider type exists. Separate agent/helper workers own execution integration; this document does not claim their acceptance.
  • Production OIDC/JWKS/OAuth browser flow is not implemented. Production still refuses to start; this is not a production-ready auth implementation. Development now has persistent tenant-scoped memberships with distinct permissions and revocation checks, not a blanket viewer/operator approval mapping.
  • Backend reconciliation now supports clean/replayed/reconnected journals, durable result replay and targeted outcome decisions. The agent/helper consumer of the new protocol is a separate integration, not implemented by this dispatch. Missing authoritative jobs/history and regressed journals fail closed for manual recovery. Coordinated rollback of every independent ledger still requires an explicit operator restore procedure, not automatic readiness.
  • No standalone worker role, production artifact store, agent capability/forwarding integration for native APT acceptance, or separate append-only audit archive. Root scope configuration is deliberately not generated/enabled by this backend work.
  • tests/local_integration.py now retains the passive TLS/auth/RLS checks and checks durable authorization relationships instead of asserting job tables are absent. Its native fixture is selected by its own identity, never an arbitrary online VM.
  • No database/PKI reset or VM/credential access is part of this dispatch.

Backend follow-up: HTTPS, housekeeping, controls and catalog

HTTPS agent transport (mTLS, application/x-protobuf)

  • POST /agent/v1/messages: MessageBatch {protocol_major=1 [tag 1], repeated Envelope messages [tag 2]}. Maximum encoded body 1 MiB, 16 envelopes. Hello is a batch of exactly one ClientHello envelope, generation zero. Response is MessageBatch containing ServerHello, then possibly one already-approved offer. Fresh heartbeat/permit controls are singleton batches; multi-message batches contain durable inventory/job evidence only. Durable envelopes can be retried with their original message IDs. Failed partial batches may have already persisted earlier messages; replay is deduplicated and never re-executes work.
  • POST /agent/v1/poll: PollRequest {Envelope envelope [tag 1], uint32 wait_seconds [tag 2]}. Envelope is one fresh heartbeat with current generation. Wait is 0–25 seconds. Response is MessageBatch with the concrete heartbeat ACK and optionally one offer/cancel. No work is normal, not a job error. Do not interpret the heartbeat ACK as acceptance of a job or execution permit.
  • WSS and HTTPS call one durable ingest service. Switching transport requires a new Hello and increments generation. Old polls/streams cannot get new work or grants. Late bound durable job evidence does not refresh presence. Fresh replay, untrusted time, protocol/capability mismatch, and oversized input fail closed. Successful inbound traffic is rate-bounded per registered device in PostgreSQL.
  • WSS checks persistent revocation at one-second intervals, verifies the actual TLS peer certificate expiry during traffic, and forces reauthentication after at most one hour. Certificate identity/expiry come from TLS, never HTTP headers.

Certificate rotation and recovery

POST /agent/v1/certificates/renew is mTLS JSON {csr_pem} with a valid current P-256 CSR. Response {agent_id,certificate_pem,overlap_seconds:300}. The transaction stores new fingerprint/certificate/CSR hash and fences the connection generation. Agent/asset/enrollment generation are unchanged. The previous certificate binding has a five-minute overlap. Persist the exact pending CSR/key before requesting; repeating the same CSR bytes during overlap returns the same certificate after a lost response. Previous credentials cannot rotate again using a different CSR. Do not delete the old identity before atomically installing/verifying the response. Expired/revoked identities do not get a verification bypass or public recovery grant.

Background policy and interactive controls

A background task starts in gateway/all roles, visits the explicitly authorized development tenant registry in separate transaction-local RLS contexts, and runs every second with per-tenant advisory coordination. It expires pending requests and definitively unstarted jobs offline; reserved unconfirmed starts/running deadlines become unknown, never inferred cancelled/expired. Outbox redelivery uses the same durable lease and fixed attempt; WSS delivery no longer waits for inbound heartbeats. No scheduler path creates approvals, attempts or start permissions.

At start, current persistent requester/approver permissions, recipe references, identity/assignment, stops, deadlines, resource locks, plan rollout limits, failure thresholds and tenant cumulative limits are rechecked. Development tenant policy defaults are at most 100 reservations/hour and 10 concurrent reserved attempts; unknown retains its resource/rollout slot. Reboot dependency groups are serial; the development policy additionally permits only one active reboot per tenant, so changing a caller-supplied group label cannot bypass reboot serialization. Positive authorized quota/replay/effect acceptance remains a browser/agent test gate.

Cookie+CSRF+Origin+explicit permission are required for these JSON endpoints:

  • POST /api/v1/tenants/{tenant}/devices/{asset}/revocation {reason}
  • POST /api/v1/tenants/{tenant}/operation-plans/{plan}/revocation {reason}
  • POST /api/v1/tenants/{tenant}/execution-stop {stopped:boolean,reason}

Bearer credentials cannot invoke them, even with an additional browser cookie. Device revocation clears old-certificate overlap, fences active channels and requests cancellation. Plan revocation preserves execution evidence and blocks new starts; an already issued short-lived grant/running effect is not retroactively undone. Stop release never approves pending work. Audit records the real test principal/session, reason and explicit simulation origin. The matching UI controls and positive browser-driven administrative tests belong to the browser worker.

Immutable development catalog and artifacts

GET/POST /api/v1/tenants/{tenant}/package-recipes uses AptRecipe: {schema_version:1,provider:"apt",package,version,architecture:"amd64", recipe_version,installation_context:"system",reboot_policy:"never"}. Import is interactive catalog:write only. No shell command, download URL, installer option or implicit reboot can be supplied. Database versions are immutable and content hashed. Exact package/version/architecture identifies one immutable recipe; ensure_package additionally binds recipe_version. apply_updates requires each selection as name=exact-version, never a package-manager-wide upgrade.

PlanPayload adds recipe_references:[{package,version,architecture,recipe_version, sha256}], hashed inside the exact stored plan bytes and rechecked at approval/start. This is additive: empty references are omitted and old stored plan bytes round-trip unchanged (golden-tested). JobPayload and ExecutionPermit signatures/fields are unchanged. Legacy ensure_package/apply_updates root execution remains disabled; catalog presence does not enable privilege. The isolated APT-v2 contract below does not silently upgrade those old payloads.

POST /.../artifacts {bytes:[0..255],media_type} is interactive catalog:write only; maximum 256 KiB. GET /.../artifacts/{id} requires current tenant operations:read, returns attachment bytes and verifies the stored SHA-256/size on each read. A local ArtifactStore trait implementation stores private, immutable content-addressed files under the administrative key directory's artifacts/{tenant}/ tree, writes and syncs before publishing, and refuses production profile. This is not a general MCP upload tool, remote installer URL path, or production S3 adapter. Agent artifact downloads are now target/approval/reference authorized as described below; recipe-fed privileged package execution remains separate integration.

Sync/root/inventory/artifact integration profile (next agent/helper consumer)

Trust bootstrap and passive inventory

GET /api/v1/job-signing-key now also accepts the actual enrolled, unrevoked, unexpired device mTLS identity, using the same binding as WSS. There is no redirect and the response remains exactly {key_id,algorithm:"Ed25519", schema_version:1,public_key_hex} (raw 32-byte key, SHA-256 key ID). Explicit Authorization always selects bearer authentication and never falls back to mTLS/cookies after failure. The agent's existing no-redirect mTLS bootstrap works without putting any viewer/MCP credential on the device. Helper trust provisioning remains independent; an agent-supplied key is never helper authority.

The old strict rmm_domain::Inventory is unchanged. The gateway additionally validates inventory_document::validate over the existing linux.base JSON: optional observations array (<=32), collection_started_at and collection_completed_at (both required with observations). Each <=32 KiB observation has exactly scope,source_id,observed_at,quality,completeness,truncated, error_code,data; times are real RFC3339, scopes are unique linux names, quality and completeness are enumerated, and a device cannot assert received_at. Collection is bounded to one hour/72-hour telemetry age and the full JSON to 512 KiB. Existing base fields still validate strictly. Accepted additive data is stored, not stripped or relabelled valid/complete. RMM_EXTENDED_INVENTORY=1 is now backend-compatible and was exercised with the real local native collector; that is not VM acceptance. Root/control job payload limit is 65,536 bytes, matching the current independently verifying helper parser.

Exact public wire types (protocol major 1, minor 0)

rmm_protocol::Payload::{SyncRequest,SyncResponse} use new oneof tags 27,28. Earlier tags/reservations are unchanged. Both transports use the same service. Advertise rmm.sync.v1 in ClientHello. It closes the server start/offer gate on every Hello until a fresh valid reconciliation. Legacy clients remain passive/ first-session compatible, but their existing local reconnect block is not bypassed. Root always additionally needs helper readiness. SyncRequest is a singleton HTTPS MessageBatch or one WSS Envelope, with fresh current generation/boot/sequence/time.

SyncRequest fields/tags:

  1. request_id (UUID, durably reserve before sending)
  2. journal_id (stable agent SQLite journal UUID)
  3. checkpoint (monotone committed local checkpoint, u64 <= i64::MAX)
  4. previous_decision_id (last durably applied server decision, empty initially)
  5. assignment_version
  6. repeated SyncAttempt (<=256)
  7. repeated durable_message_ids (<=256 concrete unacknowledged IDs)
  8. journal_complete (true only for a complete unsettled-attempt summary)
  9. helper_journal_id (stable independent helper journal UUID, empty if absent)
  10. helper_checkpoint (its own monotone checkpoint)
  11. helper_previous_decision_id (its last durably applied signed checkpoint)
  12. previous_request_id (recover a lost agent response when decision ID is unknown)

SyncAttempt fields/tags:

  1. job_id; 2. attempt_id; 3. job_payload_hash; 4. grant_id (empty only if absent);
  2. consumption: unconsumed|consumed|uncertain;
  3. logical state: queued, accepted, requesting, running, verifying, awaiting_reboot, unknown, succeeded, failed, cancelled, expired;
  4. logical job-state revision; 8. original_boot_id;
  5. optional JobResult (may be omitted when supplied in field 13);
  6. result_message_id; 11. outcome_sha256;
  7. optional canonical outcome_bytes (<=512 KiB);
  8. optional original durable Envelope containing that JobResult.

For durable result replay, field 13 is required: preserve the original message ID, sequence, observation time, boot and payload. Rewrapping the same result ID with sync-envelope metadata is an identity conflict. Connection generation is the only receipt-hash-normalized envelope field. Optional field 9 must agree exactly with field 13 if both are supplied. Accepted-but-not-started is a logical summary, not an executor's raw SQLite status name; map a locally journalled offer to accepted.

SyncResponse has decision_bytes [1], signature [2]. It is returned instead of a generic Ack, only after the checkpoint/readiness/outcome transaction commits. The exact signed bytes are durably cached by request ID/content/current generation; same ID with different content or an old-generation replay is rejected. On lost response/reconnect, use a fresh request ID and previous_request_id pointing to the persisted old request, rather than inventing a checkpoint or replacing a grant.

Signed reconciliation / independent helper SyncAck

Authoritative Rust types are in rmm_domain::reconciliation: ReconcileDecisionPayload, AttemptDecision, ReconciledOutcome, canonical_outcome_bytes. Fixed Ed25519 over exact decision bytes, using the independently pinned administrative key. Domain is rmm.reconciliation-decision, schema_version 1, never rmm.execution-permit. Validity is at most 60 seconds. Verify signature before parsing/trusting fields; then current tenant, asset, agent, enrollment/assignment/connection generations, boot, journal IDs/checkpoints, request ID/hash, key ID and deadline. Request hash is SHA-256 of canonical prost SyncRequest::encode_to_vec(), not arbitrary reformatted JSON or Envelope bytes.

The payload additionally binds ready, helper_ready, reason, concrete acknowledged message IDs and per-attempt decisions. Every AttemptDecision binds fixed job and attempt, job-payload hash, grant ID, authorization/plan references, action, execution_identity, revision, exact outcome state/hash, previous local outcome hash, original boot and an explicit unlock decision. valid_profile checks only the generic profile/time bounds, not signature or caller-specific bindings.

Outcome codec 1 has the same field values/order as helper Outcome: state, stdout, stderr, exit_code, output_truncated, reboot_required, evidence, error_code. Use canonical_outcome_bytes, which recursively sorts evidence object keys independently of serde feature flags, preserves raw output bytes and explicit nulls, and returns the bytes to hash. If outcome_bytes is provided it must already be canonical. Hashing arbitrary local JSON spacing/key order is invalid.

The helper must independently compare its own journal/checkpoint, consumed grant, fixed job hash/identity, revision and previous_outcome_sha256 before applying a signed per-attempt unlock. For an unchanged terminal outcome it compares its own canonical outcome against outcome_sha256; no repeated outcome body is necessary. For verified reboot reconciliation the server supplies new canonical outcome_bytes and their hash. Keep consumed grant/attempt tombstones: unlocking a resource is not deleting consumption or permitting a second execution. Apply decisions idempotently and durably before reporting readiness. A decision is never a grant, even when ready=true. Continue to obtain a separate fresh ExecutionPermit.

Known definitive attempts can receive targeted unlock decisions while another attempt keeps global readiness false; do not turn such an unlock into global execution authority. Unknown free scripts with no bound postcondition remain manual needs-attention. Requesting/running/ambiguous grant consumption persists unknown with resource reservations, never retries automatically. A reboot can be reconciled only with the reserved original boot, authenticated different boot and persisted new-boot inventory; the exact definitive result is audited and signed. Missing authoritative jobs/hash/assignment mismatches create a sticky sync_recovery_required fault, not a new job. Omitting that attempt later cannot clear it. Sequence rollback and missing checkpoint history likewise fail closed. Manual controlled identity/ledger recovery is required for those faults.

Clean complete journals and accepted-unstarted jobs retain fixed IDs/revisions and regain readiness; Sync creates no operation, approval, outbox, attempt or permit. Only message IDs actually present in durable receipts/observations are ACKed. The backend tests prove real empty-journal/helper-checkpoint roundtrip, clean reconnect, response-loss recovery, signature/tamper checks and persistent restore denials. Accepted-unstarted classification is unit-tested. Positive consumed-job, independent helper apply/unlock and real reboot-result E2E cases still require the browser/agent/helper workers; no direct API/DB approval fixture was manufactured.

Explicit selected root profile (default OFF)

Require RMM_PROFILE=development, RMM_DEV_ROOT_ENABLED=true and RMM_DEV_ROOT_POLICY_FILE=/private/policy.json. The file must be regular, nonsymlink, <=64 KiB, mode 0600, owned by the administrative-key owner with a non-writable-by-others parent. Schema (configuration example, not an enablement or native proof performed by this dispatch):

{
  "schema_version": 1,
  "profile": "development",
  "expires_at": "REPLACE-WITH-EXPLICIT-RFC3339-DEADLINE",
  "scopes": [{
    "tenant_id": "SELECTED-TENANT-UUID",
    "asset_id": "SELECTED-ASSET-UUID",
    "agent_id": "SELECTED-AGENT-UUID",
    "enrollment_generation": 1,
    "assignment_version": 1,
    "principals": ["simulated-testprincipal-a"],
    "helper_policy_sha256": "EXACT-INDEPENDENTLY-PROVISIONED-HELPER-POLICY-HASH",
    "acceptance_evidence_id": "REAL-REVIEWED-NATIVE-HELPER-RECORD",
    "allowed_services": ["rmm-acceptance.service"],
    "allow_root_scripts": false,
    "allow_reboot": false
  }]
}

The evidence ID is trusted administrator deployment metadata, not remote attestation or proof manufactured from a capability string/unit test. Populate it only after independently reviewed helper provisioning/acceptance. The backend did not create such a file or enable root. Migration 0008 assigns root:execute only to the explicit dev test human, not viewers or all operators. Requester and approver need root:execute and selected scope; free scripts also scripts:execute.

Agent advertises root capabilities only after authenticating the actual root socket/peer and its independently provisioned helper policy: rmm.sync.v1, helper.root.v1, helper.policy.sha256:<64-lowercase-hex>, the base action, and root:<action>. A plain base capability is insufficient. Current profile permits explicitly allowed root scripts (<=300s), exact allowed running/stopped services (>=180s), and explicitly enabled reboot (>=60s), output <=64 KiB. Successful helper sync is required again at start. Current permissions, policy expiry/hash, assignment, approval, stops, deadlines and quotas are rechecked. Never relabel agent_service into root; the helper still verifies all signature, identity, payload, time, local-policy and one-time-consumption bindings itself. Legacy ensure_package/apply_updates without an exact apt_manifest are not root-enabled or submit/approval eligible. The optional schema-2 fixture profile below is a separate explicit opt-in; it is not enabled by this dispatch.

Agent artifacts and isolated APT-v2 contract

Plan creation optionally accepts top-level artifact_ids:[UUID...] (<=16 unique). The server reads and verifies actual immutable local objects, then hashes PlanPayload.artifact_references into the plan: each entry binds job_id, attempt_id and {artifact_id,sha256,size_bytes}. Empty references are omitted for old codec compatibility. Approval/start recheck those exact references and bytes.

GET /agent/v1/artifacts/{artifact_id}?job_id=...&attempt_id=...&offset=0&length=... requires device mTLS (bearer selection is rejected), current target assignment, an actual approved active job and a matching immutable plan reference. Reserved preapproval IDs confer no access. Cross-tenant/foreign/missing references return the same 404. Length is 1..512 KiB, default 512 KiB; the current local import adapter still limits each object to 256 KiB. Response is attachment bytes, 206/Content-Range for a partial chunk, and x-artifact-sha256 for the complete object hash. No free filesystem path or model-provided download URL exists. Positive authorized download through an actual UI-approved fixture remains a browser integration gate.

Authoritative helper-compatible APT schema 2 (latest parallel handoff)

Use the exact contract in tools/apt-transaction/HELPER-CONTRACT.md. Public domain types now match it: AptExecutionManifest, ResolvedAptTransaction, PackageArchive in rmm_domain::apt_transaction. All reject unknown/duplicate fields. This is schema 2 inside the existing schema-1 JobPayload Action, not a new unsigned IPC field or a changed ExecutionPermit profile.

Manifest fields, in declared serialization order: type:"rmm.apt.helper-action",schema_version:2,action,execution_profile:"dev-test", transaction_sha256,resolved_apt_transaction,transaction_bytes,provider_sha256, hook_sha256,apt_config_sha256,runtime_sha256,test_result_id. The exact transaction bytes must hash correctly and strictly decode to the complete typed nested object. The helper writes those original bytes, not a JSON view. Transaction fields: schema_version=1, recipe_id, recipe_version="1", platform="ubuntu-24.04-amd64", resolver_config_version="rmm-apt-local-v1", no_reboot=true, status_fingerprint, baseline_sha256, packages. PackageArchive fields: name, architecture, old_version, new_version, origin, sha256, size, archive_path.

The current native profile is only the two code-pinned, single-package rmm-mvp-test fixture transactions, architecture all, exact size 20672 and known SHA-256 pins shared with the helper contract:

  • rmm-apt-fixture-install-1.0: absent -> 1.0, 74036ae81223deab3aae994fd2a2a695af408741777e1ce17dcb4e32f1235867
  • rmm-apt-fixture-update-1.1: 1.0 -> 1.1, ef67c9fe0ba1443fcfe465768e1204763063c369a2a1da04af6027d8a9c00491

Both use rmm-test origin and fixed content-addressed root-store archive paths. No arbitrary catalog, dependency expansion, removal, URL, shell conversion or automatic update/reboot is accepted by this fixture enablement.

Domain Action changes preserve old bytes when options are None:

  • EnsurePackage adds recipe_id:Option<String> and apt_manifest:Option<AptExecutionManifest>, default None/omitted. Execution requires Some, outer recipe_id/version and package/version/architecture matching the exact nested apt-execute transaction.
  • ApplyUpdates adds the same optional apt_manifest. The executable fixture update uses provider apt, packages ["rmm-mvp-test"], reboot_policy never, and the exact nested update recipe/versions. The manifest, not an unpinned package-manager selection, binds the complete actual transaction.
  • Dedicated AptNativeAcceptance {recipe_id,recipe_version,apt_manifest} serializes action apt_native_acceptance, with manifest action apt-native-acceptance. Permit.action is exactly apt_native_acceptance. It is not run_script.

Action::validate() requires the full manifest for executable package actions. validate_plan_shape() only preserves immutable legacy catalog previews: their authorization_status is unsupported_manifest_required; submit/approval/start reject APT_MANIFEST_REQUIRED. Old missing-manifest jobs never gain execution rights. The HTTP normalizer accepts the complete typed apt_manifest and explicit recipe_id; its equality checks do not discard fields or accept parameter aliases. Transaction hash references are also part of the immutable plan. All target, identity, assignment, action, runtime, approval and grant bindings remain unchanged.

Native acceptance additionally requires current apt:native_acceptance permission (migration 0010 grants it only to the dev test human), root:execute, the selected device/root policy and successful agent/helper reconciliation. Root scope may add an optional, default-absent apt object: {allow_native_acceptance:false,allow_execute:false,provider_sha256,hook_sha256, apt_config_sha256,runtime_sha256}. Set pins only to reviewed installed bytes and runtime/config measurements. Capabilities must include apt.manifest.v2 and apt.native-acceptance.v1 for the dedicated test, or apt.native-accepted.v1 for ordinary fixture apt-execute, plus the base action/root:, helper profile and exact helper policy digest. Runtime is >=180s. Missing policy/pins/capabilities or local acceptance rejects; a capability is not native proof.

The existing independently enforcing helper bridge consumes the unchanged exact signed JobPayload bytes, verifies both manifest representations/code/runtime pins, root-private prefetch/sealing, current baseline, native locks and sandbox, and maintains its own consumption ledger. Backend work does not weaken its systemd sandbox, create a native acceptance record, advertise capabilities, install or run APT. Agent forwarding/capability advertisement, UI action creation and actual UI-approved native VM acceptance remain separate worker integrations.

Earlier generic artifact-reference utilities are retained as PinnedAptTransaction, AptArchiveReference and ResolvedAptJobPayloadV2, with AptExpectation and AptExpectedPackage aliases for data callers. Do not use the prototype V2 job envelope for the real helper bridge; the authoritative path above is the existing v1 JobPayload with a fully preserved schema-2 Action.apt_manifest.