Files
rust-discovery/deploy/linux/runtime-README.md
T
BuTzZ 6c3203d725
Rust development slice / linux (push) Failing after 1m10s
feat: snapshot Linux-first RMM MVP and pause handover
2026-11-12 22:06:25 +01:00

19 KiB

Agent/helper runtime — integrated source, VM acceptance pending

This is the current runtime contract. It supersedes the earlier module-only handoffs in README.md and tests/agent_sync/README.md. No installation, target job, service/reboot/package mutation, SSH tunnel or server restart was performed by this integration. Actual VM execution/acceptance belongs to the next worker. Nothing here approves an RMM job or substitutes for the WebUI approval boundary.

Binaries and complete CLI/environment

Both binaries support --help; unknown arguments fail closed. Normal agent operation has no positional arguments. Planned bootstrap invocation:

RMM_AGENT_STATE=/var/lib/rmm-agent
RMM_CA_FILE=/etc/rmm/ca.pem
RMM_CONNECT_ADDR=127.0.0.1:18443
RMM_TRANSPORT=https
/usr/local/libexec/rmm-agent

These are environment assignments, not an instruction to source a credential file. The trusted CA and TLS name rmm-dev.internal are always checked. TCP override changes neither DNS/SNI nor certificate verification. No proxy trust, invalid-certificate bypass or public Docker port is introduced.

Variable Meaning/default
RMM_AGENT_STATE Private 0700 owned agent state; default devstate/agent
RMM_CA_FILE Public trusted CA; default devstate/pki/ca.pem
RMM_CONNECT_ADDR Parsed TCP SocketAddr; default 127.0.0.1:18443
RMM_TRANSPORT auto (default), wss, or https; HTTPS can be explicitly selected for VM acceptance
RMM_ENROLLMENT_TOKEN_FILE First registration only; remove the bootstrap token after enrollment
RMM_JOB_BINDING_FILE Optional private agent-owned JSON Binding; absent means passive, no job capabilities
RMM_JOB_SIGNING_KEY_BOOTSTRAP=1 Authenticated device-mTLS GET, using the current rotation-store identity; pins without replacing existing trust
RMM_HELPER_SOCKET Configured general helper socket, e.g. /run/rmm-helper/control.sock
RMM_HELPER_POLICY_FILE Root-owned, agent-readable public general policy pin; required with general socket
RMM_APT_HELPER_SOCKET Dedicated fixed /run/rmm-apt-helper/helper.sock, never generic fallback
RMM_APT_HELPER_PROOF_FILE Root-owned public APT stack/profile/identity pin; see apt-helper-README.md
RMM_LOG_UNITS_FILE Private agent-owned JSON array of exact allowed service units for unprivileged live logs
RMM_EXTENDED_INVENTORY 1 by default with the now-compatible backend, 0 for legacy base-only JSON
RMM_AGENT_PROFILE development required only for the rotation-test interval override
RMM_ROTATION_TEST_AFTER_SECONDS Positive development-only certificate-age trigger; never bypasses chain/expiry/EKU/key/identity verification

Binding fields: tenant_id, asset_id, agent_id, enrollment_generation, assignment_version, public_key_hex. Device/asset must match enrollment. Empty key is allowed only with authenticated signing-key bootstrap; later startup must reauthenticate that bootstrap or use a fully provisioned binding. Key response is Ed25519/schema 1/raw 32-byte key with SHA-256 key ID, bounded and no redirects. The immutable agent cache is job-signing-key.json. Signing-pin replacement and identity/assignment recovery are controlled provisioning operations, not automatic trust in a key from a job. Helper trust stays independently root-provisioned.

The unit leaves the job-binding optional in /etc/rmm/agent.env, allowing a first passive enrollment. Set binding/key-bootstrap/helper variables only after the authenticated operator resolves the enrolled IDs and provisions their pins. Do not delete an old identity/journal to get a new first-session shortcut.

Integrated transport and sync

One bounded actor and one dispatcher serve WSS and HTTPS with identical envelopes, journal, result receipts, cancellation and signature checks. auto tries WSS first and falls back only after two typed HTTP upgrade-block classifications; 401/403, TLS/identity failures and ambiguous errors never select a bypass. WSS Ping/Pong are handled. HTTPS uses the existing MessageBatch/25-second poll contract. Poll cancellation cannot create a new attempt; fresh heartbeat IDs/ sequences are never reused or queued as offline telemetry. Actor queues are bounded.

Every authenticated Hello advertises rmm.sync.v1 and closes all local effect gates. Agent obtains actual root snapshots, durably reserves the fresh SyncRequest, checks exact signed SyncResponse bytes and all device/channel/boot/checkpoint/ request-hash/time bindings, commits targeted journal/receipt changes, applies its own consumption decisions, forwards the original signed response plus original request protobuf to every configured helper, then confirms actual applied root checkpoints. No generic ready boolean opens either process.

Both root profiles share actual DB/lock inodes. They join an unchanged pending root capture created after both process births, so ONE signed response/checkpoint can be independently applied by both. A new process never inherits readiness from the shared DB. Stale process captures are fenced; known outcomes may unlock specifically while an unknown retains the resource and global gates stay closed. Consumed grant/attempt tombstones are never removed.

Scheduler AND immediate worker effect admission check the current signed lease; the worker also rechecks fresh grant/time and the disconnect/rotation atomic fence. Helper checks its own process gate under the same executor guard before consumption. Unknown, missing authoritative history, clock failure and rollback remain closed. Clock/helper sync failure does not stop independent passive heartbeat/inventory traffic. Sync retries are bounded, happen on fresh inventory/result ACK and on a lease timer, and never retry a lost permit request as a new grant.

After reboot, old/new boot ID alone is not success. Fresh current-boot inventory is durably ACKed, a new signed decision must bind the persisted original boot and current authenticated boot plus inventory evidence, then each helper independently applies that specific canonical terminal outcome/unlock. Thus an initial negative sync does not deadlock waiting forever without the post-inventory resync.

Result envelopes are archived before concrete ACK deletion in one SQLite transaction. Historical message identity/output is retained for reconciliation. Inventory mapping preserves source/time/quality/completeness and unsupported states. Live journal permission warnings become permission_denied, not a fake empty success.

Helper bootstrap/policy inspection

Helper normal daemon: RMM_HELPER_CONFIG=/etc/rmm/helper.json (root-owned 0600). Both profiles use /var/lib/rmm-helper/consumption.sqlite and executor.lock. Helper config uses the existing binding, optional independent signing_key_file, agent_uid, socket_gid, socket, shared state, allowed_services, allow_reboot, allow_root_scripts; APT adds its existing sealed apt_provider configuration.

Public general pin, planned command, not executed on VM here:

RMM_HELPER_CONFIG=/etc/rmm/helper.json /usr/local/libexec/rmm-privileged-helper --inspect-general-profile

Install the exact JSON as root:rmm-agent 0640 in non-writable root-owned parents, then set RMM_HELPER_POLICY_FILE. It binds profile, device identity/key and exact policy hash. General policy probe uses root socket/peer credentials and a fresh challenge; capabilities reflect actual policy/interpreter availability. Root scripts remain default-off broad administration. Services are exact allowed running/stopped units with >=180s runtime, reboot separately enabled with >=60s. Root jobs are NEVER relabelled into agent_service.

APT install/probe/inspection modes and profile sandbox remain exactly documented in apt-helper-README.md. Both daemons need this same installed build and reviewed new proof hashes (binary bytes are part of the APT proof). Prefetched sealed archives, receipts predating the grant and exact baselines are still prerequisites; this integration introduces no unsafe generic APT download/install/simulation path. Root native-acceptance and accepted-recipe/manifest capabilities come only from authenticated real readiness. Successful native acceptance triggers a controlled reconnect to advertise newly accepted recipes; periodic safe capability refresh also reconnects rather than silently changing the Hello contract.

Passive APT planning and declarative no-change

The normal extended inventory worker now runs at startup and every 60 seconds (one inflight collector; not in the heartbeat handler). Unacknowledged durable traffic backpressures new collection without deleting old data; job results and control receipts retain priority. It includes the fixed linux.apt.planning scope from root-helper:apt_planning_v1. This currently refreshes the existing extended inventory scopes too; their actual individual source timestamps/quality remain intact. There is no arbitrary package query, requested live diagnosis or software effect hidden in this collector.

apt_planning_v1 uses the existing UID-authenticated dedicated Unix socket, fresh challenge, device/generation/signing-key identity, root peer and separately root-provisioned stack pin. Its response is the readiness envelope with a typed planning result and empty capability/acceptance fields. It does not depend on a prior native-acceptance record and cannot create one. Root validates the installed code, exact runtime/config policy pins and actual archive bytes, sizes, ownership/modes/link count and private prefetch receipts. No artifact is fetched or installed by this RPC. Administratively stage the known fixture archives via provision-fixture-store before planning/the final 60-second grant.

The fixed metadata child has a 4-second whole-process/output budget (32 KiB), the socket has 5-second IO limits, and root measurement timestamps are preserved. Dirty dpkg updates/triggers, foreign locks, unsafe/missing receipts, permission failure, timeout or missing helper yield an unknown observation with null data, not fake hashes or accepted capability. Runtime policy drift fails closed. Complete observations contain the suite's fixture_package, baseline_sha256, sealed_baseline_sha256, status_fingerprint, dpkg_updates_empty, pending_triggers_empty, runtime, runtime_sha256, provider_sha256, hook_sha256, apt_config_sha256 and prefetch. The full package baseline stays root-private at baselines/<sha256>.json; only hashes, fixture projection and minimal public runtime/artifact pins cross the wire. Stable sampling is checked before and after content-addressed exclusive write/file+directory fsync; existing identical seals are verified, never overwritten. This is passive metadata, not job authority. The ordinary durable inventory outbox/ACK path persists it.

Normal accepted fixture ensure_package/apply_updates may return POSTCONDITION_ALREADY_MET, changed=false, already_desired=true, postcondition_already_met=true, transaction_executed=false only when the entire selected set is independently installed at its exact target version and architecture, clean, matches the current approved baseline/fingerprint and has the expected fixture payload. Root signature, all original bindings, expiry/sync/local-policy/resource gates, all manifest/code/runtime/artifact pins, native accepted-recipe record and durable grant/attempt consumption remain mandatory. Only the historical old_version condition is skipped in this no-mutation branch. No APT/hook/native launch/effect marker is created. A foreign lock, dirty/unknown state or changed baseline cannot become success; native acceptance itself never uses this branch. Other versions still follow the unchanged no-reinstall/no-downgrade/no-remove mutation guards.

VM suite owner must update its absent/1.0-only adapter: after native update to 1.1, a normal exact desired-1.1 job must use the accepted update recipe with a fresh planning seal. A desired-1.0 job on 1.1 is still correctly denied; this implementation does not authorize removal/downgrade to make a stale test pass. No backend/domain/protobuf signatures or plan bytes have been changed.

Provider source changes change the embedded provider/helper/stack/runtime pins. Deploy both helpers from this build and provision reviewed new proof/policy pins before native acceptance; old native records cannot silently certify new runtime bytes. Manifest schema 2 / transaction schema 1 / fixture archives 1.0 and 1.1 (20,672 bytes, unchanged hashes) remain unchanged.

These are implemented source/local-fixture semantics, not VM or MVP acceptance.

Certificate rotation and recovery

RMM_AGENT_STATE/certificates/current.json is the authoritative atomic selector over private versioned device key/cert files. Existing enrolled files initialize the store exactly once; later connections/bootstrap use the selected identity. Rotation checks chain, current validity, exact agent CN, clientAuth purpose and public/private key match. It persists one pending key/CSR and retries those SAME CSR bytes after response loss. A verified swap closes the old actor/effect fence and requires a new Hello/Sync. Agent/enrollment IDs and independent job/helper signing pins do not change. Expired/revoked/invalid certs remain blocked and need authorized identity recovery; no auto-reenrollment or TLS downgrade.

Verification boundaries / concrete remaining policy issue

Local Rust source/unit/integration tests cover actual loopback mTLS dispatcher WSS/HTTPS clean reconnect, Ping, forged sync rejection, passive traffic with an untrusted effect clock, journal reopen/replay and independent helper checkpoint/ outcome/boot checks. Unsafe OS providers are mocked or denied; no local service, APT install or reboot was executed. This is not a real WebUI-approved VM result, native package acceptance, production auth acceptance or full Linux-MVP sign-off.

Concrete backend policy limitation, not bypassed here: RootScope::permits in crates/server/src/root_policy.rs does not match Action::QueryLogs and falls through to false. The typed, bounded root helper log handler exists, and the unprivileged allowlisted path is live, but a root live-log plan will be denied by that backend policy until its owner adds an explicit permitted scope. Do NOT work around it by relabelling identity, direct approval or a generic root shell.

Executed local verification

Using /home/Kevin/.cargo/bin/cargo, actually passed during integration:

cargo test -p rmm-agent -p rmm-privileged-helper -p rmm-agent-linux --all-targets --locked
cargo clippy -p rmm-agent -p rmm-privileged-helper -p rmm-agent-linux --all-targets --locked -- -D warnings
cargo build -p rmm-agent -p rmm-privileged-helper --locked
cargo fmt -p rmm-agent -p rmm-privileged-helper -p rmm-agent-linux -- --check
python3 tests/agent_jobs/test_install.py
target/debug/rmm-agent --help
target/debug/rmm-privileged-helper --help

111 executed Rust test cases passed (including reused transport tests), two create-only installer guard tests passed, both debug binaries built and both help commands returned successfully without loading secret configuration. No warnings/type errors were suppressed. A boxed tungstenite HTTP-response test constructor was corrected, not bypassed. No coverage percentage was measured.

Coverage boundaries: real loopback TLS/dispatcher WSS and HTTPS reconnect and forged sync rejection are exercised; helper cryptography/checkpoints/two-process shared signature and replay/unknown/reboot decisions use real SQLite/signatures with local fixtures. Native service/APT/reboot effects and positive root UID-0 daemon/VM RPC acceptance are NOT substituted by these tests. The built artifacts are target/debug/rmm-agent and target/debug/rmm-privileged-helper; a reviewed VM-compatible release artifact still needs the target worker's ELF/glibc/library preflight and actual authorized VM run.

Executed local verification — planning/no-change increment

After all source changes, the following complete targeted block passed with /home/Kevin/.cargo/bin on PATH (2026-11-12):

cargo test -p rmm-agent -p rmm-privileged-helper -p rmm-agent-linux --all-targets --locked
PYTHONDONTWRITEBYTECODE=1 python3 -m unittest discover -s tests/apt_transaction -p 'test_*.py' -v
cargo clippy -p rmm-agent -p rmm-privileged-helper -p rmm-agent-linux --all-targets --locked -- -D warnings
cargo fmt -p rmm-agent -p rmm-privileged-helper -p rmm-agent-linux -- --check
cargo build -p rmm-agent -p rmm-privileged-helper --locked
cargo build -p rmm-agent -p rmm-privileged-helper --release --locked
python3 tests/agent_jobs/test_install.py
target/debug/rmm-agent --help
target/debug/rmm-privileged-helper --help

Results: 115 Rust tests, 56 Python APT tests, 2 installer guard tests passed; warning-denying all-target Clippy and formatting passed; both Debug and Release binaries built; both help commands succeeded. Separate final AST parsing passed for 8 Python files. No full workspace/backend/web test suite was run.

New regressions cover shared Python/Rust canonical golden bytes/hash, receipt and timestamp freshness, private content-addressed seal reuse/change without full-baseline disclosure, sampling drift/dirty/foreign-lock/bad-artifact failure, normal exact already-desired install/update postconditions, no launch/effect markers, old-version mutation rejection, wrong architecture/newer version, audit/payload/expiry rejection, no-change outcome binding and durable duplicate grant/attempt rejection, honest missing-helper data and collection backpressure. Root ownership/kernel state and unsafe native package processes are mocked; existing harmless unprivileged seccomp tests really execute. No native host package mutation, root systemd/probe/VM action, approval or reboot was performed.

Earlier intermediate failures were corrected, not suppressed: the inherited two unformatted source edits, a mock reader missing the limit keyword argument, and a newly inserted function accidentally inheriting a test-only cfg attribute. The final complete block above passed after these corrections.

Final artifact SHA-256 (native developer-PC builds; not Bookworm/VM proof):

Artifact SHA-256
target/debug/rmm-agent 90a4eab064301740cd938a41a76a510ad3b4abc3e48a188ba12bfe22f123266c
target/debug/rmm-privileged-helper 54db2fe17841fef9af48bbeb446f45bda52f62e442a4900e0543e358095900b7
target/release/rmm-agent eac5434fd7a1e509c0971e4e05d4e1aedbde1bb90f7d7a8f9ec4d48ab4d34597
target/release/rmm-privileged-helper 581a033478b41365066df2dc9ac0d11e314160909b5fab612be0be97be1bbbdc
tools/apt-transaction/apt_transaction.py 57b695bf57ea2703393e39f92a1b6ef2314b4838f3dfb29e4e70da79687c9987

Remaining: suite-owner adapter update for exact desired 1.1 after native upgrade, reviewed deployment/new pins, actual dedicated-profile metadata RPC and native UI-approved fixture/no-change VM proof, target ELF/library compatibility, and VM resource/retention measurements for the new 60-second extended collection. No real current baseline change was induced on the host/VM: freshness tests use changed temporary/mock package state, not a disguised package installation. Linux-MVP and production acceptance remain open. Backend Root QueryLogs policy is owned by its separate worker and was neither changed nor reverified here.