ctf-ad-mirror-tool-rust/docs/backend-fixes-report.md

# Backend completion evidence — 2026-10-23

Scope: backend completion from local base `4b01fb4`; this is not a whole-project acceptance report. Changes are uncommitted. Root owns UI/web adaptation and the public API/journal documentation. No legacy sibling files, credentials or game hosts were used. Network fixtures use loopback; documentation-only addresses appear only in validation tests or injected probe futures, never outbound connections.

## Implemented fixes

### R1 — bounded shutdown and serialized control transactions

The original main task waited for Axum graceful completion before beginning TCP shutdown. A pending body extractor could therefore keep both the process and data listeners alive indefinitely.

Main now reacts to the signal independently of HTTP graceful completion, cancels background work and calls `Controller::shutdown`. The controller immediately marks itself stopping, rejects subsequent service/team mutations and deletion, obtains its existing operation mutex and shuts down the engine after the in-flight transaction has resolved. Automatic edits also check this state. The existing engine continues to own TCP sessions and its bounded drain; the data path gains no short idle timeout.

The asynchronous shutdown phase has one deadline of `network.drain_timeout_ms + 1000 ms`, beginning at signal handling. Runtime destruction adds at most another configured 100 ms wait for workers, via explicit `Runtime::shutdown_timeout`, rather than indefinitely joining an OS-blocked filesystem worker. The intended overall process shutdown budget is therefore **drain timeout + 1100 ms**, subject to OS scheduling. The 1000 ms allowance includes transaction/HTTP/background coordination; it is not added separately for each subsystem.

When the outer deadline expires the process logs the deadline and exits; requests may be interrupted. An already executing atomic filesystem replacement can have committed even if its HTTP response was interrupted. This is bounded shutdown, not a guarantee that forced shutdown can undo filesystem I/O or deliver every pending request. Ordinary in-flight transaction coordination and rollback complete before engine teardown when they fit the budget.

Regressions:

- `tests/shutdown.rs`: actual executable, live proxy/upstream session, unfinished unauthenticated JSON body; waits for real HTTP readiness, sends SIGTERM, proves the data listener stops accepting within one second and the process exits within three seconds without closing the partial-body peer first. Configuration uses a 100 ms drain timeout.
- `shutdown_serializes_with_in_flight_service_transaction`: starts a PUT transaction concurrently with controller shutdown, proves the committed topology matches disk and the resulting listener is closed.
- Custom service regression also proves service/team mutations are rejected after controller shutdown.

### R2 — probe observation time

Each probe captures its timestamp at completion. Batch publication copies that timestamp unchanged. The probe wrapper and full controller cycle admit deterministic future injection for tests; production still uses timed TCP connects.

`delayed_batch_does_not_refresh_early_candidate_observation` drives the actual controller publication and failover path: an early responsive replica is followed by a 450 ms delayed probe, while the configured health freshness window is 303 ms. Its published timestamp remains old and no promotion happens. The test performs no outbound traffic to the synthetic documentation addresses.

### R3 — wildcard loopback aliases and mapped IPv6

Topology validation canonicalizes IPv4-mapped IPv6 addresses for equality, duplicate-address checks and local-route detection. IPv4 loopback aliases are recognized with `is_loopback()`, independent of interface enumeration. A native IPv6 target is not treated as reachable through an IPv4-only wildcard listener. IPv6 wildcard detection is conservative about possible dual-stack acceptance. Bind conflict checks use the same family-aware equivalence.

`wildcard_loop_aliases_mapped_addresses_and_address_families` rejects self/cross-service routes through 127.0.0.1, 127.0.0.2 and mapped aliases while permitting a native ::1 target behind an IPv4-only wildcard on that port. The Linux-only runtime regression additionally connects 127.0.0.2 to an actual IPv4 wildcard listener, proving the platform behavior that requires the check.

### Direct-IP custom service adapter

`ServiceConfig` adds serde-defaulted `custom: bool` and `custom_ip: Option<IpAddr>`. `Team` adds serde-defaulted `hidden: bool`. Existing omitted fields deserialize to false/None. Standard PUT remains the sole operation.

A non-null custom IP is accepted only for a custom service. Inside the same serialized transaction it normalizes/reuses the address, or generates a hidden team with name equal to its IP and a stable collision-safe valid ID. It sets primary and removes duplicate primary references from replicas. Team plus service are validated and saved in one atomic topology replacement. Runtime rollback uses the original teams if persistence fails. A null custom IP leaves the submitted team selection authoritative. Automatic promotion clears custom_ip and keeps the former primary as a replica.

Regressions cover save/load, ordinary-team reuse without hiding it, generated-ID collisions, repeated-IP ID stability, restart, clearing direct-IP metadata, primary/replica deduplication and automatic promotion from a custom-IP service. A new actual service persistence failure test changes a live listener before inducing save failure, proves the old listener/route is restored and the changed listener closes, then proves a failed new direct-IP service leaves no listener or added team behind.

Additional requested metadata: serde-defaulted `Config.main_team_ip: Option<IpAddr>`, serialized `Snapshot.main_team_ip` and `Snapshot.health_max_age_ms`. The latter calls the same calculation used by automatic candidate selection: saturating `(interval_ms + timeout_ms) * 3`. No fake scoreboard team or scoreboard parser changes were introduced.

### Optional resilience

Implemented an absolute Engine.IO heartbeat deadline. Namespace setup, polling body reads and pong posting are bounded by the deadline; only receipt of a ping renews it. Continuous namespace/non-ping packets do not extend the session. The negotiated sum is bounded to 1..120000 ms, preserving subsecond negotiated intervals. The new wire-level fixture continuously returns non-ping frames and requires a fresh handshake.

Recoverable accept-error retry was deliberately left unchanged: the review did not reproduce that failure branch, and adding OS-specific resource-pressure classification without a reliable injection fixture would exceed this focused correction. The source-backed resilience concern remains open; normal accept, cancellation, connection-limit shedding and shutdown paths are covered by the existing forwarding suite.

## Verification

Native commands, all exit 0 on final source:

```
cargo fmt --check
cargo clippy --all-targets -- -D warnings
cargo test --all-targets
```

Native suite: 31 passing tests: 9 library, 8 control-plane, 2 ForcAD transport, 11 forwarding, 1 subprocess shutdown; main/example test targets also build.

Linux uses Orb's running `ubuntu` amd64 environment. Source is staged separately at `/home/ubuntu/mirror-tool-backend-validation`; only compiled dependency/artifact cache is shared with the previous validation checkout. Command:

```
orb -m ubuntu sh -lc 'cd /home/ubuntu/mirror-tool-backend-validation && CARGO_TARGET_DIR=/home/ubuntu/mirror-tool-validation/target cargo test --all-targets'
```

Final Linux result: **32 passed, 0 failed**, including the Linux-only alias runtime fixture; all main/example targets built. Orb's distribution Cargo lacks the `fmt` subcommand; formatter and Clippy gates are run natively, not represented as Linux passes.

Red checks were run in `/tmp/mirror-backend-regression-red`, restoring one original defect at a time. Original main reproduced the partial-body exit failure; publication-time timestamping failed the delayed-batch assertion; removing alias detection accepted 127.0.0.2; original scoreboard transport failed reconnect under continuous non-ping packets. Each targeted command exited 101 for the expected assertion. Logs are local `/tmp/mirror-red-{R1,R2,R3,heartbeat}.log`. The shared native project artifact cache was cleaned afterward and final source rebuilt, avoiding stale artifacts from the isolated negative runs.

One initial Linux shutdown fixture failed because it sent SIGTERM after TCP connect but before application readiness/handler registration. The fixture now waits for a real HTTP response, and the corrected Linux test passed. This was a fixture synchronization issue, not evidence of a fixed-source bounded-shutdown failure.

No low-file-descriptor accept-error fault injection, forced indefinitely blocked kernel filesystem operation or live production scoreboard/game-host testing was performed. UI restoration and whole-project acceptance remain root's responsibility.

## Modified paths owned by this task

- `src/config.rs`
- `src/control.rs`
- `src/main.rs`
- `src/scoreboard.rs`
- `tests/control_plane.rs`
- `tests/forcad_transport.rs`
- `tests/forwarding.rs` (new schema fields in existing literals only)
- `tests/shutdown.rs` (new)
- `examples/benchmark.rs` (new schema fields in existing literals only; explicitly coordinated with root)
- `docs/backend-fixes-report.md`

No edits to `src/engine.rs`, Cargo dependencies, root-owned UI files, scripts, other documentation or the legacy sibling were made by this task.

## Follow-up: accept supervision and early quiesce

This section supersedes the earlier accept-retry deferral and the earlier ordering description of controller shutdown.

Recoverable accept errors now keep the existing listener supervisor alive. The supervisor retries after 50 ms, doubles the delay up to a one-second cap and resets it after successful acceptance. The retry sleep listens for cancellation; it neither busy-loops nor delays shutdown until the timer expires. Errors classified as invalid input, unsupported operation or a disconnected listener remain terminal. Other errors, including OS resource-pressure/unknown errors such as descriptor or buffer exhaustion, are retried with the cap. Persistent resource exhaustion can keep retrying until shutdown, with a bounded attempt rate.

Each accept failure increments the new serialized `TrafficStats.accept_errors` counter, stores an `accept:` diagnostic in `last_error` and emits a warning including the retry delay. The counter is distinct from errors on accepted data sessions. A successful accept leaves historical diagnostics available; `running` remains true while retrying and false when the supervisor exits.

The injection seam is a private generic `Acceptor` used by the actual supervisor. Production uses `TcpListener`; unit fixtures substitute selected accept failures before returning to a real loopback listener. No public fault-injection interface or test-only production endpoint was introduced. New unit tests verify exact loopback payload delivery after injected resource errors, minimum retry spacing, reset after success, counters/diagnostics, a terminal error and cancellation during a scheduled 400 ms backoff. They do not depend on exhausting actual OS descriptors.

Controller shutdown now marks itself stopping and calls **Engine.quiesce before acquiring the controller transaction mutex**. Quiesce closes data acceptance and awaits listener acknowledgements, without forcing existing sessions or setting the engine's final stopped flag. The already executing controller transaction can still commit or roll back. Engine upsert in this permanently quiesced state can restore route/topology bookkeeping but cannot bind/reopen a listener. After the transaction mutex becomes available, ordinary engine shutdown performs the bounded session drain. This preserves the same outer process deadline and avoids accepting new clients while a filesystem operation delays transaction completion.

New regressions distinguish those stages:

- A controller unit fixture deliberately holds its transaction mutex; shutdown closes actual loopback acceptance while remaining blocked on that mutex, then finishes after release.
- A forwarding integration fixture verifies a preexisting stream continues byte-for-byte through quiesce, and a subsequent route upsert (the rollback path) cannot reopen the listener.
- The existing concurrent commit test and real persistence-failure tests remain green.

Follow-up native gates: `cargo fmt --check`, `cargo clippy --all-targets -- -D warnings`, `cargo test --all-targets`; **35 tests passed, 0 failed** (12 library, 8 control-plane, 2 transport, 12 forwarding, 1 shutdown). Orb amd64 all-target verification: **36 passed, 0 failed**, including the Linux alias fixture. Root will rerun final integrated release/benchmark gates after handoff. No real EMFILE or indefinitely blocked filesystem kernel call was induced.

Additional modified paths in this follow-up: `src/engine.rs`, `src/control.rs`, `tests/forwarding.rs`, and this appended report. The earlier statement that engine.rs was untouched applies only to the first handoff. Root now owns further benchmark example changes; this follow-up did not edit examples or UI/web files.

Follow-up negative controls were also exercised in the isolated Orb source staging: restoring fatal-on-every-accept-error made the recovery supervisor test fail; removing the early quiesce call made the held-transaction-lock test time out while acceptance remained open. Both exited 101 for the expected fault. Logs: `/tmp/mirror-red-accept-retry.log` and `/tmp/mirror-red-early-quiesce.log`. Correct sources were restored afterward and the full Orb suite rerun green. `last_error` intentionally remains historical after recovery so an accept success cannot erase a concurrent session error; root labels it “Последняя ошибка”.

## R6 follow-up: cancellation-safe topology transactions

Independent review reproduced cancellation after DELETE had saved the new disk topology but before engine drain and in-memory publication completed. The original `/tmp/mirror-review-cancel.py` reproduction was rerun before this fix and printed `disk services 0 live services 1 active after drain deadline 1`. Four deterministic regression tests were then added and run against the original caller-owned mutation bodies; all four failed because progress stopped after caller cancellation.

All existing topology mutation entrypoints now share an owned transaction wrapper: service PUT, service DELETE, team PUT and the automatic failover mutation stage. Admission first acquires the same serialized operation mutex and checks the stopping flag. Only then is one transaction worker spawned, owning both the controller Arc and an owned mutex guard. Dropping the HTTP/caller future drops its JoinHandle, which detaches this worker rather than aborting it. Apply, atomic save, publication, rollback and stream retirement therefore remain in the same owner until completion. A lost HTTP response does not imply that the mutation failed or was uncommitted; refresh the snapshot before deciding whether to retry. The public mutation signatures and normal result/error handling are unchanged. There is no team DELETE endpoint or controller method in this application, so none was invented.

The mutex is acquired before spawning, rather than inside a task per request. Consequently there is at most one executing/admitted transaction worker per controller, even if many clients disconnect. Callers still waiting for admission create no additional transaction worker and can be cancelled without any mutation. The normal HTTP/request layer retains its existing concurrency behavior; this change does not claim a new global HTTP connection limit.

The wrapper is also the admission stopping check: already admitted work can finish after shutdown begins, while queued/new work is rejected. Controller shutdown still quiesces the engine before waiting on the operation mutex, then drains only after the admitted worker finishes. In-flight commit or rollback cannot reopen listeners because engine quiescence remains permanent. The existing outer process deadline still bounds shutdown: process termination/runtime teardown can interrupt work once that deadline expires. This change guarantees independence from caller cancellation during normal operation, not transaction completion across forced process termination, kernel filesystem stalls, panic or machine failure.

Automatic TCP probe futures remain cancellable before mutation admission. Once its selection/apply/save stage is admitted it uses the same owned worker. Controller creation uses `Arc::new_cyclic` only to retain a Weak reference for worker ownership while preserving existing public `&self` mutation methods; there is no permanent strong reference cycle.

Deterministic regression fixtures use private `cfg(test)` persistence barriers before save and after successful save. They are absent from production. Coverage:

- Cancel DELETE after disk removal while an actual loopback proxy session remains in its drain. Disk, controller snapshot, engine registry and stream closure must converge after the caller drops.
- Cancel service PUT after disk save but before publication; the worker must publish the same topology and the listener must deliver exact real loopback bytes.
- Cancel service PUT before a deliberately failing save; runtime listener rollback and hidden-team rollback must still finish.
- Cancel team PUT after save; live and disk teams must converge through the same wrapper.
- Cancel automatic-selection caller after promotion save; live primary, replicas and cleared custom-IP metadata must still converge with disk.

The first four tests were observed red before the wrapper and green after it. The automatic-selection regression was added to cover the shared automatic path and passes. The actual disconnected-HTTP reproduction was rerun after `cargo build` and now prints `disk services 0 live services 0 active after drain deadline 0`.

Fresh native gates: `cargo fmt --check`, `cargo clippy --all-targets -- -D warnings`, `cargo test --all-targets`, plus `cargo build` for the executable reproduction. All pass; native suite now has **40 passed, 0 failed** (17 library, 8 control-plane, 2 transport, 12 forwarding, 1 shutdown). R6 modifies only `src/control.rs` and this report. Root owns public API documentation and final integrated release verification.

R6 Orb amd64 staging run completed: `CARGO_TARGET_DIR=/home/ubuntu/mirror-tool-validation/target cargo test --all-targets` in `/home/ubuntu/mirror-tool-backend-validation`: **41 passed, 0 failed** (the extra test is the Linux-only loopback alias fixture). Staging contains the final backend source; root's newest UI/benchmark changes still require root's integrated sync/release checks.