Files
rust-discovery/tools/apt-transaction/README.md
T

433 lines
24 KiB
Markdown

# Root-helper APT transaction provider / explicit first native acceptance
**Latest 2026-11-13:** The separately WebUI-approved configure-only attempt
`33775478…` reached APT 0 and completely installed fixture 1.0, but old provider
`5ff260…` rejected acceptance on the actual host reboot flag. Its private worker
reported false; the historical job had no HOST-before sample. It remains
**unknown** (`FIXTURE_UNEXPECTED_REBOOT_REQUIRED`), with no acceptance record,
Root scope closed and APT endpoint stopped. No before measurement or successful
acceptance is inferred retrospectively.
New source implements an original-bound readonly failed-effect classification
and a parent-namespace host-before/after contract for future jobs, matching
Spec 1.13. Ambient true/true is reported honestly; changed host flags, private
worker flags or changed boot reject acceptance. All **81 Python tests passed**
in the readonly unprivileged offline container, including actual APT argument
regressions with guarded fake effects and unchanged real/copied status. Native
classification/new-runtime 1.1 acceptance remain pending. See
[`HELPER-CONTRACT.md`](HELPER-CONTRACT.md) for the exact versioned contract.
**Earlier 2026-11-13:** The Cloud fixture retry reached armed hook and APT exit 0,
leaving fixture 1.0 unpacked/unconfigured. An original-bound readonly proof and
normal signed RootSync now classified that original attempt as **failed**
(`APT_PARTIAL_UNCONFIGURED_VERIFIED`), preserving all histories/tombstones.
No native acceptance record or package repair exists. Root test policy remains
closed. Detailed real evidence: `../../docs/implementation-status.md`.
The configure-only recovery path, requiring its own new fixed WebUI approval, is implemented in source
(schema 4 manifest, transaction schema 2), with a dedicated passive recovery
projection and separate immutable recovery records. Its qualified runtime is now installed and its actual Cloud copied-database
sandbox readiness probe passed, preserving original state. **No configure-only
package effect or native package acceptance has yet run.** All **75 Python tests passed** in the offline readonly
unprivileged container, including actual APT 2.6.2 configure-only protocol and
guarded fake-dpkg arguments with unchanged real/copied status. See
[`HELPER-CONTRACT.md`](HELPER-CONTRACT.md) for the precise recovery boundary.
Source configuration now disables APT's `dpkg::selection::remove::approved`
rewrite, retaining the exact package's Configure operation while both pending
modes remain disabled and hook-pinned. Upstream explanation:
[APT 2.7.14 dpkgpm.cc](https://github.com/Debian/apt/blob/2.7.14/apt-pkg/deb/dpkgpm.cc#L1626-L1657).
Local actual APT 2.6.2 regression in an offline readonly unprivileged container
proved unpack + exact `--configure rmm-mvp-test:all`, no `--pending` or foreign
target; independent dpkg guard simulated effects, real/copied status unchanged.
67 Python tests passed with this opt-in regression. **This correction is now installed and sandbox-probed on the Cloud VM;
its native package effect remains unaccepted.** No automatic repair,
marker reset or implicit unknown-resolution is introduced. Older sections below
describe historical increments; current operational status is the linked log.
Reproduce that **local** configure regression using the existing development
image with APT 2.6.2 (no host APT command):
```sh
docker run --rm --read-only --network none --user 1000:1000 \
--tmpfs /tmp:rw,exec,nosuid,nodev,mode=1777 \
--tmpfs /tmp/opencode:rw,exec,nosuid,nodev,mode=1777 \
-v "$PWD/tools/apt-transaction:/src/tools/apt-transaction:ro" \
-v "$PWD/tests/apt_transaction:/src/tests/apt_transaction:ro" \
-v "$PWD/deploy/linux:/src/deploy/linux:ro" -w /src \
-e PYTHONDONTWRITEBYTECODE=1 -e RMM_APT_ARGUMENT_REGRESSION=1 \
rust-discovery-dev-build:local \
python3 -m unittest discover -s tests/apt_transaction -p 'test_*.py' -q
```
Current dedicated runner/probe/profile/agent integration and executed local
checks: [`../../deploy/linux/apt-helper-README.md`](../../deploy/linux/apt-helper-README.md).
The general helper unit remains unchanged; actual native VM measurements are
still required. Readiness alone advertises no accepted package capability.
**Helper integration contract update:**
[`HELPER-CONTRACT.md`](HELPER-CONTRACT.md) defines the implemented Rust helper
bridge, root installation/store layout, schema-2 embedded manifest and exact
remaining additive domain fields. It supersedes the older standalone signed
helper-action example below: signed-action.json must now contain the **full
original JobPayload bytes**, with `action.apt_manifest`, including exact
`transaction_bytes`. No detached/unsigned manifest is accepted. No agent
capability or native acceptance follows from this bridge build.
**Implemented, locally tested; NOT integrated or VM-tested.** No host package
installation, VM access, real native acceptance, production evidence, grant
issuance or WebUI approval has been performed here. The native process path is
real code (unshare/mount/capabilities/seccomp/APT/dpkg), not merely a parser;
its root/native execution is still unverified. Local root/installer execution
in tests is explicitly mocked. No real acceptance record was created.
Python >=3.9, standard library; OS dependencies: APT/dpkg, util-linux
unshare/mount, libseccomp.so.2, Linux x86-64 namespaces/seccomp/capabilities.
Only Ubuntu 24.04 amd64 is eligible; this is not a support/acceptance claim.
No Rust API, Cargo workspace changes, network endpoint, signature verifier,
grant service or automatic retry is included.
## The circular gate is replaced, not bypassed
There are now two distinct helper-only CLI operations:
```
/usr/bin/python3 -I /usr/lib/rmm-apt-transaction/apt_transaction.py native-acceptance
/usr/bin/python3 -I /usr/lib/rmm-apt-transaction/apt_transaction.py execute
```
* `native-acceptance`: dedicated **dev-test** action, exactly one code-pinned
fixture install OR upgrade, its own fixed job/attempt and actual WebUI approval.
It permits only absence of a *previous* native record for that fixture recipe.
It uses the same signed binding, archives, native sandbox, hooks, locks,
deadlines and independent verification as normal execution. No force/insecure/
skip-validation flag exists.
* `execute`: requires an existing matching record **and** a freshly verified,
separately UI-approved normal job/startgrant. An acceptance record never
substitutes for any signature or authorization.
**Scope matrix:** records produced here authorize only their exact fixture
recipe, package closure, runtime identity and **dev-test** profile. Production
is rejected (`PRODUCTION_ACCEPTANCE_REQUIRED`), even for a test record. General
catalog packages are rejected (`FIXTURE_RECIPE`); the bootstrap does not certify
arbitrary packages or widen provider support. Supporting those needs a separate
reviewed native-acceptance policy/provider revision, not relabelling this record.
The legacy unsigned administrator-created `native-acceptance.json` schema 1
is no longer used. Old helper contexts (schema 1) are rejected as well.
Each of these is a separate, one-time bootstrap recipe (recipe_version `1`):
| recipe_id | exact transition | architecture | origin |
|---|---|---|---|
| `rmm-apt-fixture-install-1.0` | absent -> 1.0 | all | rmm-test |
| `rmm-apt-fixture-update-1.1` | 1.0 -> 1.1 | all | rmm-test |
Exactly one package, `rmm-mvp-test`, no dependencies/scripts/triggers. One job
never installs both versions. Upgrade requires another fixed UI-approved job.
Pinned deterministic Debian ar/USTAR archives, **20,672 bytes each**:
```
1.0 74036ae81223deab3aae994fd2a2a695af408741777e1ce17dcb4e32f1235867
1.1 ef67c9fe0ba1443fcfe465768e1204763063c369a2a1da04af6027d8a9c00491
```
`build_fixture.py EXISTING_PRIVATE_OUTPUT_DIR` creates those exact bytes with
fixed metadata. It refuses overwrites and never installs/imports/publishes.
Real local dpkg-deb control/payload inspection confirms the only installed file
is `/usr/share/rmm-mvp-test/version`, with the selected version plus newline.
The builder now uses deterministic stdlib ar/USTAR construction, not the previous
timestamp-dependent `dpkg-deb --build` output. Older built archives do **not**
qualify for bootstrap; import these pinned bytes manually into the approved
catalog, verify and seal them before requesting the final 60-second grant.
## Mandatory signed-action and verified-helper contract
This module does **not** verify signatures. The root helper is the trust boundary.
Before writing any handoff files or invoking either mode, it MUST independently:
1. Verify the signed grant type/schema/algorithm/key, online validity and actual
human WebUI authorization (or explicitly labelled allowed dev-test UI
simulation), tenant/asset/agent identity and generation/assignment bindings.
2. Verify plan hash, job payload hash **of exact bytes**, fixed job/attempt,
action, execution profile, recipe/version, code/hook/config/runtime hashes,
exact complete resolved transaction and deadlines against that signed grant.
3. Enforce an explicit helper operation allowlist: `apt-native-acceptance` only
from the dedicated dev-test fixture-bootstrap action, never a generic
diagnostic/shell/API request. `apt-execute` must not invoke bootstrap mode.
4. Recheck trusted clocks/current rights and durably consume the grant and
attempt in its parent journal BEFORE spawning. Reject previous/uncertain
attempts. Unprivileged network/MCP callers cannot write these files or invoke
an unrestricted privileged subprocess API. Preserve peer-credential checks.
### Signed job payload: exact schema 1
The following object itself is the **signed/grant-hash-bound job payload**,
stored unchanged as `active/signed-action.json`; it is NOT an extra unsigned
root manifest or a caller's `verified: true` assertion. Fields are mandatory:
```json
{
"type": "rmm.apt.helper-action",
"schema_version": 1,
"action": "apt-native-acceptance",
"execution_profile": "dev-test",
"transaction_sha256": "<SHA256 of exact expectation.json bytes>",
"resolved_apt_transaction": "<complete OBJECT defined below, not a string/reference>",
"provider_sha256": "<installed apt_transaction.py SHA256>",
"hook_sha256": "<fixed installed hook SHA256>",
"apt_config_sha256": "<apt_config() bytes SHA256>",
"runtime_sha256": "<canonical runtime_identity() SHA256>",
"test_result_id": "<fixed acceptance evidence ID>"
}
```
The string placeholder for `resolved_apt_transaction` above MUST be replaced
with the complete nested object in actual payloads. Normal mode uses action
`apt-execute`. Production is a recognized profile but cannot use these test
records. The helper must verify the signed action, not merely trust ownership
of a separately supplied hash. The provider rechecks exact signed-payload hash,
type/schema/action/profile, nested transaction equality and code/config/runtime
bindings; it relies on the helper for cryptography and grant consumption.
### ResolvedAptTransaction schema 1
All fields mandatory; unknown fields/types and duplicate JSON keys are rejected:
```
schema_version: integer = 1
recipe_id: one exact fixture recipe above
recipe_version: "1"
platform: "ubuntu-24.04-amd64"
resolver_config_version: "rmm-apt-local-v1"
no_reboot: true
status_fingerprint: 64 lowercase hex characters
baseline_sha256: 64 lowercase hex characters
packages: [PackageArchive] # exactly one for this accepted scope
```
`PackageArchive` exact fields:
```
name: "rmm-mvp-test"
architecture: "all"
old_version: null for install, "1.0" for update
new_version: "1.0" for install, "1.1" for update
origin: "rmm-test"
sha256: corresponding code-pinned hash above
size: 20672
archive_path: "/var/lib/rmm-apt-transaction/archives/<same sha256>.deb"
```
The general parser still understands up to 128 package changes / 1 MiB, but that
does not authorize a broader execution scope. Frozen dataclasses/tuple closure;
strict Debian version ordering via read-only typed `dpkg --compare-versions`.
Status fingerprint hashes raw status/architecture/extended_states and trigger
database state. Dirty dpkg updates, pending triggers, repair, reinstall,
downgrade, removal, crossgrade and core/kernel/boot packages are unsupported.
`active/baseline.json` is the complete canonical all-package snapshot from
`installed_snapshot()` (`dpkg-query`), not just the changed package:
```json
{"package:architecture":{"version":"<version>","status":"install ok installed"}}
```
Its canonical byte SHA256 binds `baseline_sha256`. A new install requires actual
absence, not leftover config-files. Existing consistent noninstalled entries may
be present for other packages. Fingerprint/baseline are rechecked in the native
hook, not accepted from simulation as authorization.
### VerifiedHelperContext schema 2
Root helper writes `active/context.json` only after the checks/consumption above:
```json
{
"schema_version": 2,
"transaction_sha256": "<exact expectation.json bytes SHA256>",
"job_payload_sha256": "<exact signed-action.json bytes SHA256>",
"action": "apt-native-acceptance",
"execution_profile": "dev-test",
"effect_armed_ledger_id": "<stable parent journal row>",
"grant_id": "<verified grant ID>",
"job_id": "<fixed job ID>",
"attempt_id": "<fixed attempt ID>",
"authorization_id": "<UI authorization ID>",
"plan_sha256": "<verified plan hash>",
"not_before_ms": 1794484800000,
"start_before_ms": 1794484860000,
"boot_id": "<current boot UUID>",
"monotonic_start_before_ns": 123456789000
}
```
UTC integer milliseconds; start window <=60,000ms, exclusive upper deadline.
Helper translates that same verified deadline to same-boot CLOCK_MONOTONIC ns;
never grants another fresh window. Both clocks/boot are checked after expensive
verification, before native launch and at the final hook gate. An installation
validly started is not aborted merely because its start deadline later expires.
## Deployment / exact invocation
Planned, **not deployed here**. Root helper/operator installs reviewed files:
* `apt_transaction.py` -> `/usr/lib/rmm-apt-transaction/apt_transaction.py`, 0644
* fixed `hook` -> `/usr/lib/rmm-apt-transaction/hook`, 0755
* `/var/lib/rmm-apt-transaction` and `active`, 0700, root-owned
* sealed archives in `archives/`, root-owned; no nonroot writer
* active expectation/context/baseline/signed-action/apt.conf files, 0600
* empty root-owned active `empty`, `lists`, `cache` directories, 0700
All path components must be real root-owned nonsymlink/non-group/world-writable
paths. `apt.conf` must equal `apt_config()` bytes. Read-only `print-config` emits
them; root-only `inspect-runtime` emits canonical measured runtime identity for
the helper/planner's reviewed signed payload, not an acceptance claim.
For **either** operation, stdin is exactly:
```json
{"transaction_sha256":"<same approved hash>","effect_armed_ledger_id":"<same parent row>"}
```
Use sanitized environment and close inherited private fds. No secrets in argv.
An acceptance request must invoke `native-acceptance`; a normal signed action
must invoke `execute`. Mixing the two is rejected. Internal `hook` and
`sandbox-worker` modes are not exposed as privileged IPC operations.
## Native pipeline and evidence publication
Both modes run the same pipeline:
1. Root/signed action/recipe/runtime/preconditions/actual archive validation;
constant hook/config/code binding before APT. Native provider flock only
serializes this provider, never replaces APT's locks.
2. Bootstrap additionally writes/fsyncs O_EXCL
`bootstrap-attempt-<fixed recipe_id>.json` in BASE, independently of active/
and grant IDs. One-time per recipe; no automatic deletion/reset/retry, even
after a failed native precheck. An operator must reconcile failures and use
a separately reviewed lifecycle/revision, not remove tombstones to retry.
3. Separate native child via fixed `unshare --mount --pid --fork --mount-proc
--net`, private propagation, tmpfs `/run`/`tmp`/`var/tmp`/`dev`, protected
readonly/masked kernel/boot/control-plane/tool/archive paths.
4. Drop bounding/effective/permitted capability sets to exactly filesystem/UID
subset `{0,1,3,4,6,7,8,31}`; inherited/ambient empty. No SYS_BOOT/SYS_ADMIN/
SYS_PTRACE/KILL/RAWIO/NET_ADMIN/MKNOD. Set NoNewPrivileges, load inherited
seccomp denying reboot/kexec, module loading, mount/namespace changes,
ptrace/process writes and other kernel-control operations.
5. **Actual native prechecks inside that child:** different PID/mount/net ns,
expected namespace PID1, exact caps/NNP/Seccomp, readonly mount flags, tmpfs
identity, masked sysrq, no conventional systemd/dbus/helper sockets; invalid
reboot magic and invalid kexec flags/fds must return EPERM. These calls never
contain valid reboot magic/commands or valid kexec input. No PID1 signals.
6. Hold real existing `/var/lib/dpkg/lock-frontend` with fcntl; a separate native
`apt-get -o DPkg::Lock::Timeout=0 check` must report that frontend-lock conflict
(100 + expected lock diagnostic). `check` is read-only, not an installation
probe. Never delete locks, kill owners or use simulation to prove locking.
A busy foreign lock vetoes the bootstrap. Release only this process's fd.
7. Fsync native-prechecks record, then sandbox-ready marker. Native APT uses
exact local archives, `--no-remove --no-install-recommends` and
constant isolated config. No global APT hooks/sources/preferences; reviewed
dpkg global config is runtime-hash-bound in the signed action. Native foreign
lock wait remains bounded to 10 seconds.
**2026-11-13 native diagnosis:** `--no-download` prevented APT's local file
acquisition and emitted `**ERROR**` instead of the absolute archive filename
in protocol v3. The provider permits local acquisition of the presealed
archive; the AF_UNIX-only private worker still rejects actual IPv4/IPv6
socket creation. Both denials are now checked inside worker/hook/probe.
A permitted IP socket or unrelated resource error vetoes; this is not a
network-download allowance or acceptance/manifest/hash/path bypass.
8. Mandatory v3 pre-dpkg hook repeats signed/runtime/scope/sandbox checks,
validates the whole actual action set, actual hashes/control/payload,
status/baseline and final deadline; O_EXCL + file/directory fsync effect-armed
marker **before returning 0**. Second hook is vetoed. No fallback.
Immediately after successful file/directory fsync, UTC/monotonic deadline and
boot identity are checked again. Expiry vetoes the hook while retaining its
armed marker: conservative unknown/reconciliation, never marker reuse.
9. After native child exit 0 and valid bound markers/result, independently query
**all** installed versions/statuses, detect unexpected changes, require empty
`dpkg --audit` and exact installed fixture version-file bytes. Private /run
reboot flag is combined with host flag. Bootstrap requires no reboot flag.
10. Only then publish root-owned 0600 schema-2
`acceptance-<fixed recipe_id>.json`, staging/fsync complete bytes before
exclusive publication and directory fsync. No failed/stale-start/partial/
unknown execution is promoted. Parent must treat publication/IO uncertainty
as unknown and reconcile; a record is never an excuse to retry that job.
Record binds exact approved recipe/packages, full measured runtime (tool, hook,
APT config, resolved binaries including APT/dpkg/dpkg-deb/dpkg-query/Python/
unshare/mount/shell/libseccomp hashes and version strings, OS/kernel identity,
dpkg config fingerprint), transaction/job-payload/plan hashes, grant/job/attempt/
authorization/test-result IDs, precheck/hook/postcondition hashes and completion
time. It explicitly contains `test_acceptance: true`, `noProductionEvidence: true`.
Install and upgrade have separate records; one does not certify the other.
## Result / safety bounds
One JSON result on stdout: `effect_state` = `known_failed`, `verified`, or
`unknown`; stable `code`; structured `evidence`. Every structured bootstrap
result, including veto/unknown, carries `evidence.test_acceptance=true` and
`evidence.noProductionEvidence=true`. Verified bootstrap additionally reports
record hash/test-result ID. Exit 0 means structured result, not job success.
Malformed CLI/hook input exits 100 with generic diagnostics. Parent must journal
missing output/death/partial markers after possible launch as unknown.
No `awaitreboot` output in this version: no post-boot completion condition exists.
No reboot is invoked. Any reboot/follow-up/retry needs another fixed approved job.
No auto-configure-all, autoremove, dist-upgrade, repair, index refresh, installer
timeout killing, or automatic record/attempt clearing is provided.
APT retains its frontend lock and normally hands off the inner dpkg lock.
**No claim that both locks are held during the hook.** Direct foreign dpkg and
malicious root are outside the frontend-lock guarantee; native inner locking
still applies. Root scripts are intrinsically trusted host-filesystem code;
this is not general malicious-root containment or arbitrary deferred-side-effect
verification. Bootstrap avoids that expansion by accepting only pinned
script/trigger/dependency-free fixtures. Existing dpkg triggers/global config,
kernel, root administrators, trusted binaries/helper remain in the TCB.
## Primary sources and actual local verification
APT 2.7.14 sources read before parser implementation:
* <https://github.com/Debian/apt/blob/2.7.14/doc/apt.conf.5.xml>
* <https://github.com/Debian/apt/blob/2.7.14/apt-pkg/deb/dpkgpm.cc>
* <https://github.com/Debian/apt/blob/2.7.14/apt-pkg/deb/debsystem.cc>
Native read-only lock probe checked against `DoCheck` (`Cache.Open/CheckDeps`,
not dpkg/install) in:
<https://github.com/Debian/apt/blob/2.7.14/cmdline/apt-get.cc>.
Hook protocol **VERSION 3**, nine fields:
`name old-version old-arch old-multiarch direction new-version new-arch
new-multiarch action`. New install old triple `- - none`; `<` install/upgrade,
`>` downgrade/removal; action absolute archive path or `**CONFIGURE**`/
`**REMOVE**`. Config lines precede a blank separator and support percent escaping
and repeated unnamed lists. SHA/size/origin are NOT transmitted in v3: actual
sealed files are independently checked. Whole closure must match, including one
configure per unpack; extra/missing/repeated actions are vetoed.
`tests/apt_transaction/fixtures/install-v3.txt` is source-derived, **not a native
installation capture**.
Actually executed unprivileged, with **no host package mutation**:
```
PYTHONDONTWRITEBYTECODE=1 python3 -m unittest discover -s tests/apt_transaction -p 'test_*.py' -v
```
**39 tests passed** at the documentation update. Original 22 guard/archive/
parser/preconditions tests plus pinned fixture/control inspection, context/
signed manifest/recipe/profile/runtime binding, native-record matrix, fsync/
partial/negative evidence rejection, mocked full driver success/unknown/failure/
stale/redelivery, mocked native mount/capability/namespace/socket introspection,
and local unprivileged seccomp invalid-input EPERM probes.
The post-fsync deadline regression advances mocked UTC/monotonic clocks (both
and each independently), observes nonzero hook outcome/no modeled dpkg start,
and verifies that the durable marker survives and prevents a second handoff.
Native root/installer/VM parts are explicitly mocked under private temporary
BASE paths; mock-generated records are deleted with those dirs and are NOT
native evidence. Safe invalid-input probes and seccomp loading really ran in
short-lived unprivileged children. No valid reboot/kexec operation or host PID1
signal was attempted. Deterministic .deb bytes were really built/read/inspected,
never installed. Full root sandbox, real APT lock conflict, native fixture
install/update, helper signature integration and VM acceptance remain **unrun**.