mirror of
https://github.com/mrvasil/ctf-ad-mirror-tool-rust.git
synced 2026-10-23 14:31:02 +00:00
Rewrite mirror manager in Rust with async forwarding and ForcAD routing
This commit is contained in:
commit
4b01fb47b1
7
.gitignore
vendored
Normal file
7
.gitignore
vendored
Normal file
@ -0,0 +1,7 @@
|
||||
/target/
|
||||
/data/
|
||||
/mirror.toml
|
||||
/.env
|
||||
/test-results/
|
||||
*.log
|
||||
.DS_Store
|
||||
7
AGENTS.md
Normal file
7
AGENTS.md
Normal file
@ -0,0 +1,7 @@
|
||||
# Repository rules
|
||||
|
||||
Read `agent.md` and `task_plan.md` on resume. Maintain the journal and evidence log for meaningful changes. Never modify the legacy sibling directory. Never copy its credentials or contact its game hosts.
|
||||
|
||||
Use `cargo fmt --check`, `cargo clippy --all-targets -- -D warnings`, `cargo test --all-targets` before claiming completion. Test Linux-specific behavior in Orb. Treat byte integrity, TCP half-close, backpressure, freshness and persistence as correctness requirements. Never describe request mirroring as database replication or guarantee zero latency/loss across failed networks.
|
||||
|
||||
Separate config, forwarding, control-plane, scoreboard and operator UI modules. Keep data-path synchronization short and never hold a lock across network I/O. Document public HTTP contracts in `docs/api.md`. Tests must exercise externally observable behavior, not mirror implementation details.
|
||||
2061
Cargo.lock
generated
Normal file
2061
Cargo.lock
generated
Normal file
File diff suppressed because it is too large
Load Diff
39
Cargo.toml
Normal file
39
Cargo.toml
Normal file
@ -0,0 +1,39 @@
|
||||
[package]
|
||||
name = "mirror-tool"
|
||||
version = "0.1.0"
|
||||
edition = "2024"
|
||||
rust-version = "1.85"
|
||||
description = "Async TCP traffic mirror and ForcAD-aware upstream router"
|
||||
license = "MIT"
|
||||
|
||||
[dependencies]
|
||||
anyhow = "1"
|
||||
axum = { version = "0.8", features = ["json", "macros"] }
|
||||
bytes = "1"
|
||||
clap = { version = "4", features = ["derive"] }
|
||||
futures-util = "0.3"
|
||||
if-addrs = "0.14"
|
||||
rand = "0.9"
|
||||
reqwest = { version = "0.12", default-features = false, features = ["json", "rustls-tls", "stream"] }
|
||||
serde = { version = "1", features = ["derive"] }
|
||||
serde_json = "1"
|
||||
sha2 = "0.10"
|
||||
socket2 = { version = "0.6", features = ["all"] }
|
||||
subtle = "2"
|
||||
tokio = { version = "1", features = ["full"] }
|
||||
tokio-util = { version = "0.7", features = ["rt"] }
|
||||
toml = "0.8"
|
||||
tower-http = { version = "0.6", features = ["limit", "trace", "set-header"] }
|
||||
tracing = "0.1"
|
||||
tracing-subscriber = { version = "0.3", features = ["env-filter"] }
|
||||
url = "2"
|
||||
|
||||
[dev-dependencies]
|
||||
tempfile = "3"
|
||||
tower = { version = "0.5", features = ["util"] }
|
||||
|
||||
[profile.release]
|
||||
lto = "thin"
|
||||
codegen-units = 1
|
||||
strip = true
|
||||
|
||||
92
README.md
Normal file
92
README.md
Normal file
@ -0,0 +1,92 @@
|
||||
# Mirror Tool
|
||||
|
||||
Асинхронный TCP mirror и панель управления маршрутами для A/D CTF. Один Rust-бинарник включает сетевое ядро, веб-интерфейс и клиент ForcAD. Python, Node и внешние CDN для работы не нужны.
|
||||
|
||||
Клиент общается с одним **основным upstream**. Входящий поток можно одновременно отправлять выбранным **репликам**; их ответы вычитываются и игнорируются. Смена маршрута применяется к новым соединениям. Открытая сессия сохраняет свой upstream до завершения.
|
||||
|
||||
## Запуск
|
||||
|
||||
Нужен актуальный stable Rust (проверенные версии записаны в `docs/verification.md`).
|
||||
|
||||
```sh
|
||||
cargo build --release --locked
|
||||
cp mirror.example.toml mirror.toml
|
||||
# Отредактируйте команды, адреса и порты. Пример использует тестовую сеть RFC 5737.
|
||||
read -s MIRROR_PASSWORD
|
||||
export MIRROR_PASSWORD
|
||||
./target/release/mirror-tool --config mirror.toml check
|
||||
./target/release/mirror-tool --config mirror.toml
|
||||
```
|
||||
|
||||
Пароль должен содержать минимум 12 символов. Панель по умолчанию доступна на `http://127.0.0.1:6700`. Для удалённого доступа используйте VPN/SSH-туннель либо HTTPS reverse proxy; при HTTPS установите `secure_cookie = true`. TLS для управляющей панели внутри бинарника не реализован. Переменные окружения с паролем и конфигурацию игры не коммитьте.
|
||||
|
||||
В панели: выберите основной маршрут, включите нужные реплики и запустите сервис. Радиокнопка меняет primary, флажок выбирает реплику. Настройки сохраняются после успешного ответа сервера; они переживают рестарт. Сервис можно добавить с произвольными портом слушателя и upstream-портом, команду — с IPv4/IPv6-адресом. Авто режим выбирает только из назначенных реплик.
|
||||
|
||||
## Доставка и перегрузка
|
||||
|
||||
| Режим | При медленной реплике | При недоступной/сломавшейся реплике |
|
||||
|---|---|---|
|
||||
| `isolate` (по умолчанию) | Ограниченная очередь; при заполнении реплика отключается для этой сессии. Primary продолжает работать. | Ошибка и отключение учитываются в статистике; primary независим. |
|
||||
| `strict` | Чтение клиента замедляется до скорости всех реплик (backpressure). | Сессия завершается с ошибкой; продолжать без обязательной реплики нельзя. |
|
||||
|
||||
Произвольные фрагменты потока не выбрасываются с продолжением передачи, как в старой версии. Но TCP mirror **не является репликацией БД**: серверы могут генерировать разные идентификаторы, challenge/nonce, ответы и побочные эффекты. TLS-поток нельзя содержательно зеркалировать в независимые TLS-сессии. Проверяйте совместимость конкретного протокола и checker перед игрой.
|
||||
|
||||
Абсолютные «0 потерь / 0 накладных расходов» недостижимы при сбоях сети и медленных репликах. Strict сохраняет порядок и полную передачу при здоровых участниках ценой скорости/доступности; isolate может доставить отключённой реплике только префикс. Счётчики байтов показывают успешные локальные socket writes, а не фиксацию данных приложением или БД. Незавершённый запрос никогда не переигрывается на другом upstream.
|
||||
|
||||
## ForcAD и автоматический выбор
|
||||
|
||||
```toml
|
||||
[scoreboard]
|
||||
kind = "forcad"
|
||||
url = "http://scoreboard.example:8080"
|
||||
poll_interval_ms = 3000
|
||||
stale_after_ms = 15000
|
||||
```
|
||||
|
||||
ForcAD: постоянная сессия Engine.IO v4 / Socket.IO polling, namespace `/game_events`, ping/pong и переподключение с ограниченным backoff. `init_scoreboard` задаёт метаданные и состояние; `update_scoreboard` обновляет состояние без потери метаданных. Поддерживается формат upstream [pomo-mondreganto/ForcAD](https://github.com/pomo-mondreganto/ForcAD), проверенный на `6ce3407`. События сопоставляются по точному имени сервиса и имени/IP команды. Настройте имена как на scoreboard. Произвольные несовместимые форки автоматически не угадываются.
|
||||
|
||||
Для adplatf укажите `kind = "adplatf"`: используется HTTP `/state`. URL не должен содержать пароль, query или fragment. Доступный endpoint должен отдавать публичную таблицу без отдельной аутентификации.
|
||||
|
||||
Автопереключение:
|
||||
|
||||
1. TCP-пробы выполняются параллельно с ограничением; отдельно считаются последовательные сбои и успешные проверки.
|
||||
2. Причина переключения — устойчивый TCP-сбой primary либо свежий checker CORRUPT/MUMBLE/DOWN. CHECK_FAILED и неизвестный статус не принимаются за неисправность сервиса.
|
||||
3. Кандидат должен быть назначенной репликой с недавней успешной TCP-проверкой и пройденным порогом восстановления. Свежий плохой checker исключает кандидата; UP получает приоритет. При равенстве используются `stolen`, затем ID.
|
||||
4. Работает cooldown, включая ручную смену primary. Самопроизвольного возврата на прежний сервер нет. Старый primary становится репликой.
|
||||
5. Устаревший или отключённый scoreboard не влияет на выбор; при отсутствии пригодного кандидата маршрут сохраняется. Активные сессии не мигрируют.
|
||||
|
||||
Встроенный proxy выбирает upstream; системные маршруты, nftables и iptables не меняются. Исходный IP клиента на upstream не сохраняется. UDP и PROXY protocol не реализованы.
|
||||
|
||||
## Ресурсы и остановка
|
||||
|
||||
Tokio multiplexes соединения без subprocess/thread на сессию. Общий `max_connections` ограничивает активные сессии всех сервисов; лишние подключения закрываются и учитываются как `rejected`. Очереди реплик ограничены `replica_queue_chunks`, общий бюджет их разделяемых payload — `max_buffered_bytes` (256 MiB по умолчанию). В этот бюджет не входят рабочие буферы чтения и TCP-буферы ОС. Рабочие буферы ограничены числом сессий/реплик; не выставляйте лимиты выше доступной памяти и fd.
|
||||
|
||||
Путь без реплик переиспользует буфер; зеркальный путь передаёт разделяемые `Bytes` без отдельной копии на реплику. Счётчики используют атомарные операции. TCP_NODELAY и TCP keepalive включены; connect timeout не превращается в короткий idle timeout.
|
||||
|
||||
Stop отключает приём новых соединений, сохраняя открытые. Удаление остановленного сервиса ждёт `drain_timeout_ms`, затем отменяет остаток. SIGINT/SIGTERM закрывает слушатели, даёт всем сессиям общий срок дренирования, затем отменяет и дожидается принадлежащих им задач. Isolate также ограничивает время отправки остатка репликам после завершения основного обмена. Strict в нормальном режиме может ожидать медленную реплику неограниченно; общая остановка остаётся ограниченной.
|
||||
|
||||
## Конфигурация и состояние
|
||||
|
||||
`mirror.toml` задаёт настройки процесса и начальную топологию. После первого изменения сохранённый `state_file` содержит авторитетный список команд/сервисов; TOML продолжает задавать сеть, мониторинг и scoreboard. Путь состояния отсчитывается от каталога TOML. Чтобы применить начальную топологию заново, остановите процесс и **переместите** state-файл в резервную копию. Файл не исполняется, версия и содержимое валидируются.
|
||||
|
||||
Запись состояния: временный файл на той же ФС → fsync → atomic rename; права `0600`. Сбой сохранения вызывает попытку отката runtime. Публикация topology сериализована с ручными и автоматическими изменениями. Runtime и ФС не образуют распределённую транзакцию: клиент, принятый в коротком окне до неудачного сохранения, может сохранить временно применённый маршрут. Необходимость отката выводится как ошибка.
|
||||
|
||||
Старый `config.py` не выполняется и не импортируется. Перенесите `TEAMS` в `[[teams]]`, `TASKS` в `[[services]]`, ForcAD URL в `[scoreboard]`. `PASSWORD` заменён переменной окружения, `SECRET_KEY` не нужен: сервер хранит случайные сессии в памяти. Старые cookie после перезапуска недействительны. API описан в [docs/api.md](docs/api.md).
|
||||
|
||||
Для Linux есть [systemd unit](deploy/mirror-tool.service): создайте отдельного пользователя `mirror-tool`, разместите бинарник в `/usr/local/bin/`, конфиг и закрытый `mirror.env` в `/etc/mirror-tool/`; в TOML задайте `state_file = "/var/lib/mirror-tool/state.json"`. Unit даёт только CAP_NET_BIND_SERVICE для портов ниже 1024; при ненужности удалите capability. При увеличении drain timeout увеличьте `TimeoutStopSec`. Unit поставляется как шаблон, автоматически не устанавливается.
|
||||
|
||||
## Разработка и проверка
|
||||
|
||||
```sh
|
||||
cargo fmt --check
|
||||
cargo clippy --all-targets --locked -- -D warnings
|
||||
cargo test --all-targets --locked
|
||||
node --check web/app.js
|
||||
cargo run --release --example benchmark -- 64
|
||||
```
|
||||
|
||||
Benchmark сравнивает direct loopback с proxy в том же процессе, проверяет каждый байт, выполняет три чередующихся прогона и 500 latency-проб. Это воспроизводимый локальный baseline, не прогноз скорости реальной сети. Результаты и практические ограничения: [docs/verification.md](docs/verification.md).
|
||||
|
||||
Для ручного теста UI на изолированном loopback: `cargo build`, затем `python3 scripts/ui_fixture.py target/debug/mirror-tool`. Это необязательная тестовая утилита; её публичный тестовый пароль указан в начале файла, конфиг/состояние временные. Не используйте fixture для реального доступа.
|
||||
|
||||
Структура: `config` — валидация, `engine` — TCP, `scoreboard` — протоколы таблиц, `control` — health/выбор/изменения, `store` — сохранение, `auth`/`web` — управление, `web/` — встроенный интерфейс. `agent.md` и журналы фиксируют ход работы.
|
||||
22
agent.md
Normal file
22
agent.md
Normal file
@ -0,0 +1,22 @@
|
||||
# Mirror Tool — working journal
|
||||
|
||||
## Objective
|
||||
Rewrite the legacy Python TCP mirror manager in Rust, in this repository. Preserve the operator workflow, repair networking/failover bugs, integrate ForcAD accurately, and verify performance rather than promise physically impossible zero overhead.
|
||||
|
||||
## Binding user decisions
|
||||
- 2026-10-23: automatic routing means upstream selection inside the TCP proxy; no nftables/iptables or host route changes.
|
||||
- Keep existing service/team selection, primary + replicas, custom services, statistics, availability and scoreboard views. No additional UI preferences yet.
|
||||
- Keep the legacy directories unchanged. Never copy credentials from their config into this repository.
|
||||
|
||||
## Current state
|
||||
Rust modules implemented: config, store, engine, scoreboard, control, auth, web and CLI. TCP engine implementer delivered 11 passing integration tests; UI implementer active on `web/*`. Parent owns integration, protocol fixtures, Linux verification, benchmark, final review and documentation. Read `task_plan.md`, `findings.md`, and `progress.md` before resuming.
|
||||
|
||||
## Verification policy
|
||||
No completion/performance claims without fresh output. Use loopback and Orb Linux fixtures; never contact configured legacy game hosts. Test exact byte delivery, half-close, idle connections, live target changes, failure selection, replica overload, ForcAD reconnects, auth and persistence. Record limits explicitly.
|
||||
|
||||
## Milestone 1
|
||||
- Async bounded engine, strict/isolate policies, replica response draining, hot target changes, RAII active accounting, acknowledged listener shutdown, bounded whole-engine drain.
|
||||
- Verified canonical ForcAD init/delta format; persistent Engine.IO 4 implementation with heartbeat and reconnect.
|
||||
- Typed config validation, same-filesystem fsync+rename state replacement, rollback on persistence failure, fresh-health deterministic selection with cooldown.
|
||||
- Cookie sessions (HttpOnly/SameSite), CSRF header, same-origin checks, login rate limiting, no secret persistence. HTTP binds locally by default; user supplies environment password.
|
||||
- Fresh engine evidence: 11 loopback tests passed (see docs/engine-report.md). Full integrated checks still pending.
|
||||
29
deploy/mirror-tool.service
Normal file
29
deploy/mirror-tool.service
Normal file
@ -0,0 +1,29 @@
|
||||
[Unit]
|
||||
Description=Mirror Tool TCP mirror and upstream router
|
||||
Wants=network-online.target
|
||||
After=network-online.target
|
||||
|
||||
[Service]
|
||||
Type=simple
|
||||
User=mirror-tool
|
||||
Group=mirror-tool
|
||||
WorkingDirectory=/var/lib/mirror-tool
|
||||
EnvironmentFile=/etc/mirror-tool/mirror.env
|
||||
ExecStart=/usr/local/bin/mirror-tool --config /etc/mirror-tool/mirror.toml
|
||||
Restart=on-failure
|
||||
RestartSec=3
|
||||
TimeoutStopSec=45
|
||||
StateDirectory=mirror-tool
|
||||
StateDirectoryMode=0700
|
||||
UMask=0077
|
||||
LimitNOFILE=131072
|
||||
NoNewPrivileges=true
|
||||
PrivateTmp=true
|
||||
ProtectSystem=strict
|
||||
ProtectHome=true
|
||||
RestrictAddressFamilies=AF_INET AF_INET6 AF_UNIX
|
||||
AmbientCapabilities=CAP_NET_BIND_SERVICE
|
||||
CapabilityBoundingSet=CAP_NET_BIND_SERVICE
|
||||
|
||||
[Install]
|
||||
WantedBy=multi-user.target
|
||||
43
docs/api.md
Normal file
43
docs/api.md
Normal file
@ -0,0 +1,43 @@
|
||||
# HTTP contract
|
||||
|
||||
JSON API, UTF-8. Authenticated sessions use an HttpOnly SameSite=Strict cookie. Mutating authenticated requests require `X-CSRF-Token` from login or session response. Browser requests must be same-origin. No secrets returned in snapshots. All failures return `{ "error": "human readable explanation" }` with non-2xx status. A snapshot is the source of truth; never assume a mutation succeeded before its 2xx response.
|
||||
|
||||
## Routes
|
||||
- `GET /` — embedded UI shell (login shown until session validated).
|
||||
- `GET /healthz` — process readiness `{status:"ok"}` (not upstream checker health).
|
||||
- `POST /api/login` JSON `{password:string}` → `{authenticated:true,csrf:string}`, Set-Cookie. 401 invalid, 429 limited.
|
||||
- `GET /api/session` → `{authenticated:true,csrf:string}`, 401 expired/missing.
|
||||
- `POST /api/logout` → `{ok:true}`, clears cookie.
|
||||
- `GET /api/snapshot` → full snapshot below.
|
||||
- `GET /api/events` — SSE, event name `snapshot`, full snapshot every second, heartbeat 15s. Session expiry closes stream. EventSource reconnects automatically; recheck session after errors.
|
||||
- `PUT /api/services/{id}` — full ServiceConfig JSON; id must match path. Atomically save and hot-apply. → `{ok:true}`. Validation/bind error: 400; persistence failure: 400 with explanation, runtime rollback attempted. Stopped listener drains established sessions. Changing primary, replicas, policy or auto mode does not restart a running listener. Existing sessions keep their previous route.
|
||||
- `DELETE /api/services/{id}` — requires `enabled:false` first; drains/removes then → `{ok:true}`.
|
||||
- `PUT /api/teams/{id}` — full Team JSON; new team or rename. Addresses of existing ids are immutable; add another id instead. → `{ok:true}`.
|
||||
|
||||
ServiceConfig:
|
||||
```json
|
||||
{"id":"notes","name":"Notes","listen":"0.0.0.0:18080","upstream_port":8080,"primary":"a","replicas":["b"],"enabled":false,"auto_switch":false,"replica_policy":"isolate"}
|
||||
```
|
||||
Team:
|
||||
```json
|
||||
{"id":"a","name":"Team A","address":"192.0.2.10"}
|
||||
```
|
||||
|
||||
## Snapshot
|
||||
```json
|
||||
{
|
||||
"now_ms": 1792700000000,
|
||||
"teams": [{"id":"a","name":"Team A","address":"192.0.2.10"}],
|
||||
"services": [{
|
||||
"id":"notes","name":"Notes","listen":"0.0.0.0:18080","upstream_port":8080,
|
||||
"primary":"a","replicas":[],"enabled":true,"auto_switch":false,"replica_policy":"isolate",
|
||||
"engine":{"running":true,"last_error":null,"stats":{"accepted":5,"active":1,"completed":4,"connect_errors":0,"io_errors":0,"rejected":0,"bytes_up":1234,"bytes_down":5678,"replica_disconnects":0,"replica_errors":0,"replica_bytes":0}},
|
||||
"health":{"a":{"reachable":true,"healthy":true,"consecutive_failures":0,"consecutive_successes":2,"checked_at_ms":1792700000000,"latency_ms":2}},
|
||||
"scoreboard":{"a":{"status":101,"stolen":3,"lost":0}}
|
||||
}],
|
||||
"scoreboard":{"enabled":true,"connected":true,"kind":"forcad","round":5,"updated_at_ms":1792700000000,"stale":false,"last_error":null,"teams":[],"tasks":[],"entries":{}},
|
||||
"events":[{"at_ms":1792700000000,"service":"notes","message":"Настройки сохранены"}]
|
||||
}
|
||||
```
|
||||
|
||||
Compute displayed bandwidth from positive byte-counter deltas / `now_ms` deltas, reset baseline when counters decrease. Show unknown health until checked. Scoreboard badges must indicate stale data; never imply stale checker state is current. Replica counters mean explicit disconnections/errors, not dropped arbitrary packets. `replica_policy`: `strict` uses backpressure, `isolate` disconnects a lagging replica per session.
|
||||
38
docs/engine-contract.md
Normal file
38
docs/engine-contract.md
Normal file
@ -0,0 +1,38 @@
|
||||
# Engine implementation contract
|
||||
|
||||
Implement `src/engine.rs` and `tests/forwarding.rs`. Root owns config.rs and other modules. Export from lib.rs by adding `pub mod engine;` only; do not edit config or Cargo dependencies without coordinating.
|
||||
|
||||
Public types:
|
||||
```rust
|
||||
pub struct Engine; // cloneable handle, internally shared
|
||||
impl Engine {
|
||||
pub fn new(settings: crate::config::NetworkConfig) -> Self;
|
||||
pub async fn upsert(&self, service: &crate::config::ServiceConfig, teams: &[crate::config::Team]) -> anyhow::Result<()>;
|
||||
pub async fn remove(&self, id: &str) -> anyhow::Result<()>;
|
||||
pub fn snapshots(&self) -> std::collections::BTreeMap<String, EngineStatus>;
|
||||
pub async fn shutdown(&self);
|
||||
}
|
||||
#[derive(Clone, Debug, serde::Serialize)]
|
||||
pub struct EngineStatus {
|
||||
pub running: bool,
|
||||
pub stats: TrafficStats,
|
||||
pub last_error: Option<String>,
|
||||
}
|
||||
#[derive(Clone, Debug, Default, serde::Serialize)]
|
||||
pub struct TrafficStats {
|
||||
pub accepted: u64, pub active: u64, pub completed: u64,
|
||||
pub connect_errors: u64, pub io_errors: u64, pub rejected: u64,
|
||||
pub bytes_up: u64, pub bytes_down: u64,
|
||||
pub replica_disconnects: u64, pub replica_errors: u64, pub replica_bytes: u64,
|
||||
}
|
||||
```
|
||||
|
||||
`upsert` binds before success, returns bind errors to caller, updates targets without restarting an unchanged listener, keeps existing sessions on their original route, supports enabled=false (stop accepting, let old sessions drain; bound drain/cancel on whole engine shutdown). Re-enable must be reliable. Explicit remove should retire metrics only after no leaked tasks; stop accepting and bound existing drain is acceptable, document semantics. Only forward new streams to the configured primary, no midstream retry/replay.
|
||||
|
||||
Tokio only, no subprocess/thread-per-connection. Process-wide connection semaphore before accept/spawn to bound tasks (or shed promptly with `rejected` counter). `TcpSocket`/TcpListener, TCP_NODELAY; connect timeout only, no short socket I/O timeout. Preserve both directions until EOF/half-close; shutdown opposite write half, do not abort reply on request EOF. Isolate requests and replica responses: always drain replica replies concurrently, ignore content. Memory bound by configured queue and chunk sizes, `Bytes` shared across sinks, no per-replica payload clones if feasible.
|
||||
|
||||
Replica policies:
|
||||
- `Isolate` (default): a bounded non-blocking queue per replica. A full queue or replica error explicitly disconnects that replica for THIS session and increments a counter. Never drop an arbitrary chunk and continue sending an invalid stream. Primary proceeds independently. No replica reconnect/replay midstream.
|
||||
- `Strict`: connect all required replicas before sending client bytes; bounded queues apply backpressure; replica failure ends the session explicitly (error metrics) instead of silently proceeding. Strict mode cannot promise availability/primary throughput when a replica is slower or absent.
|
||||
|
||||
Connection/task shutdown must be cancellation-safe, close owned sockets and release permits. Active metrics increment/decrement using RAII or equivalently reliable path. Increments on hot path relaxed atomics, no locked logging of payloads. No unsafe code needed. Include normal bulk bidirectional delivery, half-close, >3s idle, live retarget old/new streams, slow/failing replica behavior, bind conflict/error, connection limits, clean shutdown tests. Meaningful integration tests using loopback.
|
||||
55
docs/engine-report.md
Normal file
55
docs/engine-report.md
Normal file
@ -0,0 +1,55 @@
|
||||
# Async TCP engine implementation evidence
|
||||
|
||||
2026-10-23. Scope: `src/engine.rs`, `tests/forwarding.rs`, and the engine module export. All fixtures use local loopback sockets; no legacy credentials, game hosts, or payload logs were used.
|
||||
|
||||
## Implementation
|
||||
|
||||
- Tokio listener supervisors own a `JoinSet` of sessions. A semaphore shared by all services in one Engine bounds accepted active sessions, including upstream connection setup. Excess accepted sockets are immediately closed and counted as rejected, without creating a task.
|
||||
- Routes are copied at accept time under a short synchronous lock. Target-only changes preserve the listener and affect subsequent sessions. There is no primary retry/replay after partial delivery.
|
||||
- Bind errors propagate before replacing a working listener. Listener stop is acknowledged after the socket is dropped, making disable/re-enable deterministic. Disable stops accepting but allows established sessions to continue. Remove waits up to `drain_timeout_ms`, cancels remaining sessions, joins all session and replica tasks, and only then removes metrics. Whole-engine shutdown uses one drain deadline shared across services and retains snapshots with `running=false`, `active=0`.
|
||||
- Both TCP directions run concurrently until their respective EOFs, propagating write half-close without terminating the opposite direction. Only connection establishment has a timeout; idle primary streams have no I/O timeout. TCP_NODELAY is set on client and upstream sockets.
|
||||
- Each replica has a bounded chunk channel and concurrently drained response stream. `Bytes` references share one request chunk across sinks. Isolate uses nonblocking queue admission; saturation disconnects that replica for that session instead of skipping a chunk and continuing. Replica connection failure does not delay primary setup/traffic. Strict connects all configured replicas before reading client bytes, awaits queue capacity, and terminates the session on replica error.
|
||||
- Atomic counters use relaxed operations. Byte counters increment after each successful socket write, including partial chunks preceding an error. Active accounting and semaphore release occur after child workers have been aborted and joined on forced cancellation. `accepted` includes promptly rejected sockets; `completed` counts natural successful session completion, not forced cancellation. Replica failures can accompany a successful primary session in isolate mode.
|
||||
- Shared error/counter state survives a listener-address change while old sessions drain. No lock spans network I/O.
|
||||
|
||||
## Fresh verification
|
||||
|
||||
Native macOS command:
|
||||
|
||||
```text
|
||||
cargo test --test forwarding
|
||||
running 11 tests
|
||||
... all 11 tests ok ...
|
||||
test result: ok. 11 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 3.20s
|
||||
```
|
||||
|
||||
The tests cover:
|
||||
|
||||
1. Exact 2 MB binary delivery in both directions and byte counters.
|
||||
2. A reply generated only after request EOF, following 3.2 seconds idle.
|
||||
3. Live retargeting with old sessions retained on the original target, plus disable/re-enable.
|
||||
4. Bind conflict propagation with the prior listener still usable.
|
||||
5. A shared connection limit across two distinct service listeners, prompt rejection, and bounded active shutdown.
|
||||
6. Continued delivery after disabling a listener, followed by bounded remove and socket closure.
|
||||
7. Exact 1 MB strict replica request delivery while draining a 2 MB replica reply concurrently.
|
||||
8. Unavailable isolate replica preserving primary delivery and recording a single replica error/disconnect.
|
||||
9. Unavailable strict replica ending the session before any primary payload bytes.
|
||||
10. A stalled isolate replica explicitly disconnected during a 16 MB successful primary exchange.
|
||||
11. A stalled strict replica applying backpressure during a 32 MB upload, with shutdown joining its workers.
|
||||
|
||||
```text
|
||||
rustfmt --edition 2024 --check src/engine.rs tests/forwarding.rs
|
||||
exit 0, no output
|
||||
```
|
||||
|
||||
An earlier `cargo check --lib` passed. `cargo clippy --lib -- -D warnings` was attempted and reported `clippy::derivable_impls` in root-owned `src/config.rs`; that issue was reported to the root agent. The last integration-test run also emitted an unused `serde_json::json` import warning in concurrently developed `src/scoreboard.rs`, reported to the root agent. No clean whole-project clippy claim is made here.
|
||||
|
||||
The original replica fixture used 127.0.0.2, which macOS did not expose as a configured loopback address (`AddrNotAvailable`). The fixture now uses ::1 for replicas and 127.0.0.1 for the primary, preserving the common configured service port without altering host interfaces. Orb reports x86_64 with rustc/cargo 1.93.1; integrated Linux staging and test execution remain root-agent work.
|
||||
|
||||
## Limits and operator semantics
|
||||
|
||||
This is live TCP request mirroring, not database replication. It does not guarantee zero latency or loss during network/peer failures. Successful writes measure bytes accepted by the local socket, not application-level commit or remote durable receipt. A replica may have received only a prefix when it disconnects. No reconnect or replay is attempted within that session.
|
||||
|
||||
Strict mode can block indefinitely on a slow peer during normal operation and sacrifices primary throughput/availability to required replicas. Isolate mode can explicitly disconnect even a healthy replica when the configured queue cannot absorb a traffic burst. After both primary directions finish, isolated replica tail drain is bounded by `drain_timeout_ms`; unfinished tails are explicitly disconnected and counted. Primary socket write-half closure is propagated before that tail wait.
|
||||
|
||||
Payload memory is bounded by the configured session, replica, queue and buffer sizes, plus OS socket buffers; the maximum can still be large if the operator chooses large resource limits. Upsert expects validated topology/network settings from the configuration layer. Call and await `shutdown()` before disposing of the Engine; dropping a clone is not a shutdown operation. This report contains functional evidence, not a throughput/latency benchmark.
|
||||
21
docs/ui-report.md
Normal file
21
docs/ui-report.md
Normal file
@ -0,0 +1,21 @@
|
||||
# Operator UI implementation
|
||||
|
||||
The UI preserves the service-column workflow and compact per-team primary/replica controls. Russian labels match the previous operator surface. The visual direction is a navy instrument panel, warm route markers, lavender utility accents and restrained status colors; no animation, external fonts, CDN or frontend build is needed.
|
||||
|
||||
## Contracts and behavior
|
||||
|
||||
- `/` loads only `/app.css` and deferred `/app.js`; no inline handlers, inline styles or scripts. All dynamic names, addresses, events and errors are assigned as text nodes.
|
||||
- Bootstrap checks `/api/session`, then reads `/api/snapshot` and opens one `/api/events` EventSource. Login uses the documented password body. Authenticated mutations include the returned CSRF token.
|
||||
- Service start/stop, primary/replica selection, automatic selection and replica policy use PUT with the complete latest ServiceConfig. Successful mutations refresh the snapshot; failures expose the backend message and reconcile controls with the last confirmed snapshot.
|
||||
- Service dialog supports all configuration fields, optional primary while stopped and add/edit. Existing service ID is immutable in the editor; duplicate add IDs are rejected locally. Team add also permits a rename with the same ID/address; changing an existing address is rejected. Service deletion is disabled while enabled/running and requires an explicit confirmation with name, listener and route details.
|
||||
- Service and team DOM nodes are keyed and updated in place. SSE never rebuilds open modal fields or focused route controls. Write controls are locked during mutations, while snapshots continue to update telemetry.
|
||||
- Bandwidth uses byte-counter deltas divided by snapshot timestamp deltas, separately for upstream and downstream. Counter decreases and backwards timestamps reset the baseline; duplicate timestamps retain the previous rate. Absolute primary and replica bytes remain visible.
|
||||
- SSE interruption immediately marks the panel stale; lack of snapshots for eight seconds does the same. Session checks after disruption use a bounded 2–30 second backoff and stop while the stream is healthy. EventSource handles stream reconnects; 401 returns to login and closes the stream. Page lifecycle closes/reopens the stream without duplicating it.
|
||||
- TCP health is unknown until checked, with old checks marked stale after 15 seconds. TCP reachability is separate from checker health. ForcAD stale/disconnected state is explicit globally and on checker badges; stolen/lost counters are shown alongside status.
|
||||
- Native dialogs provide focus containment and Escape dismissal (disabled during save), controls have visible focus, table headers/captions and descriptive labels, notices use status/alert roles, layout fits mobile widths, and reduced-motion preferences are respected.
|
||||
|
||||
## Verification and remaining integration work
|
||||
|
||||
`node --check web/app.js` passes. Source inspection confirms one EventSource creation site, no interval polling, no `innerHTML`, no inline event/style attributes and no external asset URLs. The UI implementation did not modify backend, Cargo files or legacy templates and did not contact game hosts.
|
||||
|
||||
Real browser checks against the integrated backend, CSP response headers, authentication failures, modal focus under live updates, mutation errors, mobile layout, and stream/session recovery are delegated to the integrating parent task. Rust format/clippy/test gates and Orb checks belong to that integration; this report does not claim they have passed.
|
||||
134
examples/benchmark.rs
Normal file
134
examples/benchmark.rs
Normal file
@ -0,0 +1,134 @@
|
||||
//! Reproducible loopback comparison. Not a claim about production networks.
|
||||
use anyhow::Result;
|
||||
use mirror_tool::{
|
||||
config::{NetworkConfig, ReplicaPolicy, ServiceConfig, Team},
|
||||
engine::Engine,
|
||||
};
|
||||
use serde_json::json;
|
||||
use std::{net::SocketAddr, time::Instant};
|
||||
use tokio::{
|
||||
io::{AsyncReadExt, AsyncWriteExt},
|
||||
net::{TcpListener, TcpStream},
|
||||
};
|
||||
|
||||
async fn transfer(addr: SocketAddr, megabytes: usize) -> Result<f64> {
|
||||
let stream = TcpStream::connect(addr).await?;
|
||||
stream.set_nodelay(true)?;
|
||||
let (mut read, mut write) = stream.into_split();
|
||||
let start = Instant::now();
|
||||
let sender = async {
|
||||
let data = vec![0x5au8; 65536];
|
||||
for _ in 0..megabytes * 16 {
|
||||
write.write_all(&data).await?;
|
||||
}
|
||||
write.shutdown().await
|
||||
};
|
||||
let receiver = async {
|
||||
let mut data = vec![0; 65536];
|
||||
let mut total = 0;
|
||||
loop {
|
||||
let n = read.read(&mut data).await?;
|
||||
if n == 0 {
|
||||
break;
|
||||
}
|
||||
anyhow::ensure!(data[..n].iter().all(|b| *b == 0x5a), "byte corruption");
|
||||
total += n;
|
||||
}
|
||||
anyhow::ensure!(total == megabytes * 1048576, "truncated stream");
|
||||
Ok::<_, anyhow::Error>(())
|
||||
};
|
||||
tokio::try_join!(
|
||||
async {
|
||||
sender.await?;
|
||||
Ok::<_, anyhow::Error>(())
|
||||
},
|
||||
receiver
|
||||
)?;
|
||||
Ok((megabytes as f64 * 2.0) / start.elapsed().as_secs_f64())
|
||||
}
|
||||
async fn latency(addr: SocketAddr) -> Result<(f64, f64)> {
|
||||
let mut stream = TcpStream::connect(addr).await?;
|
||||
stream.set_nodelay(true)?;
|
||||
let mut times = vec![];
|
||||
let mut reply = [0u8; 32];
|
||||
for _ in 0..500 {
|
||||
let t = Instant::now();
|
||||
stream.write_all(&[42u8; 32]).await?;
|
||||
stream.read_exact(&mut reply).await?;
|
||||
anyhow::ensure!(reply == [42; 32], "latency reply corruption");
|
||||
times.push(t.elapsed().as_secs_f64() * 1e6);
|
||||
}
|
||||
stream.shutdown().await?;
|
||||
times.sort_by(f64::total_cmp);
|
||||
Ok((times[250], times[475]))
|
||||
}
|
||||
#[tokio::main]
|
||||
async fn main() -> Result<()> {
|
||||
let megabytes = std::env::args()
|
||||
.nth(1)
|
||||
.and_then(|v| v.parse().ok())
|
||||
.unwrap_or(64);
|
||||
let listener = TcpListener::bind("127.0.0.1:0").await?;
|
||||
let direct = listener.local_addr()?;
|
||||
let upstream = tokio::spawn(async move {
|
||||
loop {
|
||||
let (stream, _) = listener.accept().await.unwrap();
|
||||
stream.set_nodelay(true).unwrap();
|
||||
tokio::spawn(async move {
|
||||
let (mut read, mut write) = stream.into_split();
|
||||
let _ = tokio::io::copy(&mut read, &mut write).await;
|
||||
let _ = write.shutdown().await;
|
||||
});
|
||||
}
|
||||
});
|
||||
let reserve = TcpListener::bind("127.0.0.1:0").await?;
|
||||
let proxy = reserve.local_addr()?;
|
||||
drop(reserve);
|
||||
let engine = Engine::new(NetworkConfig::default());
|
||||
let service = ServiceConfig {
|
||||
id: "bench".into(),
|
||||
name: "Benchmark".into(),
|
||||
listen: proxy,
|
||||
upstream_port: direct.port(),
|
||||
primary: Some("local".into()),
|
||||
replicas: vec![],
|
||||
enabled: true,
|
||||
auto_switch: false,
|
||||
replica_policy: ReplicaPolicy::Isolate,
|
||||
};
|
||||
engine
|
||||
.upsert(
|
||||
&service,
|
||||
&[Team {
|
||||
id: "local".into(),
|
||||
name: "Loopback".into(),
|
||||
address: direct.ip(),
|
||||
}],
|
||||
)
|
||||
.await?;
|
||||
transfer(direct, 4).await?;
|
||||
transfer(proxy, 4).await?;
|
||||
let mut direct_runs = vec![];
|
||||
let mut proxy_runs = vec![];
|
||||
for round in 0..3 {
|
||||
if round % 2 == 0 {
|
||||
direct_runs.push(transfer(direct, megabytes).await?);
|
||||
proxy_runs.push(transfer(proxy, megabytes).await?);
|
||||
} else {
|
||||
proxy_runs.push(transfer(proxy, megabytes).await?);
|
||||
direct_runs.push(transfer(direct, megabytes).await?);
|
||||
}
|
||||
}
|
||||
let direct_latency = latency(direct).await?;
|
||||
let proxy_latency = latency(proxy).await?;
|
||||
engine.shutdown().await;
|
||||
upstream.abort();
|
||||
let _ = upstream.await;
|
||||
println!(
|
||||
"{}",
|
||||
serde_json::to_string_pretty(
|
||||
&json!({"platform":std::env::consts::OS,"arch":std::env::consts::ARCH,"payload_mib_per_run":megabytes,"throughput_unit":"MiB/s aggregate upload+echo download","direct":direct_runs,"proxy":proxy_runs,"direct_latency_us":{"p50":direct_latency.0,"p95":direct_latency.1},"proxy_latency_us":{"p50":proxy_latency.0,"p95":proxy_latency.1},"byte_integrity":"verified every byte", "stats":engine.snapshots()})
|
||||
)?
|
||||
);
|
||||
Ok(())
|
||||
}
|
||||
25
findings.md
Normal file
25
findings.md
Normal file
@ -0,0 +1,25 @@
|
||||
# Findings and decisions
|
||||
|
||||
## Legacy behavior (source verified)
|
||||
- Source: `../ad_scripts/mirror_tool`, commit `3011813` (2026-05-16); Downloads copy identical for 12 tracked files.
|
||||
- Flask panel :6700, one subprocess per proxied port, threads per connection/replica.
|
||||
- Primary returns responses; replica responses are never drained, so mirroring can stall.
|
||||
- Replica queue drops oldest chunks on overflow (corrupts TCP application stream).
|
||||
- Primary socket connect timeout remains a 3s read/write timeout.
|
||||
- Shared stop flag breaks TCP half-close; peer direction can stop prematurely.
|
||||
- `_pick_candidate` is undefined and its apparent body sits after an unconditional return.
|
||||
- UI stops/restarts forwarders for primary/replica edits despite live-retarget backend support.
|
||||
- Counters estimate speed from total bytes / time since last update, not interval deltas.
|
||||
- Global mutable process state, no route-state restart recovery, unacknowledged stdin controls, startup success before bind.
|
||||
- Canonical ForcAD `pomo-mondreganto/ForcAD`, inspected SHA `6ce3407be569b7f69f879501c5e2ca79231c1ff9`. Engine.IO 4 / Socket.IO namespace `/game_events`.
|
||||
- Crucial legacy bug: `init_scoreboard` envelope contains `{data:{teams,tasks,state}}`; `update_scoreboard` contains `{data:{round,round_start,team_tasks,...}}` (state only). Keep cached metadata and merge updates. Sources: backend/services/events/app.py, backend/lib/storage/game.py, front/src/store/index.js.
|
||||
- Use persistent Engine.IO polling (the upstream explicitly supports it), ping/pong, namespace ack, bounded HTTP bodies, bounded reconnect backoff and freshness based on received scoreboard data. No per-cycle Socket.IO reconnect.
|
||||
- adplatf uses GET `/state`; map UP/CORRUPT/MUMBLE/DOWN to 101/102/103/104.
|
||||
|
||||
## Architecture direction
|
||||
- Tokio tasks, bounded buffers, shared immutable route snapshots; no process/thread per connection.
|
||||
- Direct paths use Tokio bidirectional copy; mirrored paths share `Bytes` chunks across bounded channels. Always drain replica replies.
|
||||
- Explicit strict/isolate replica policy; no silent chunk dropping. Strict backpressure trades throughput for bounded reliable delivery.
|
||||
- Health and fresh scoreboard inform deterministic candidate selection with hysteresis/cooldown; no switch solely from stale/missing evidence.
|
||||
- Axum control plane, embedded static UI, single authenticated SSE stream instead of four polling loops.
|
||||
- No kernel routing work: user explicitly chose internal upstream routing.
|
||||
54
mirror.example.toml
Normal file
54
mirror.example.toml
Normal file
@ -0,0 +1,54 @@
|
||||
# Copy to mirror.toml. Set MIRROR_PASSWORD in your environment (12+ characters).
|
||||
# Bind the panel to a VPN interface or put it behind HTTPS when used remotely.
|
||||
[web]
|
||||
listen = "127.0.0.1:6700"
|
||||
password_env = "MIRROR_PASSWORD"
|
||||
state_file = "data/state.json"
|
||||
session_ttl_seconds = 43200
|
||||
secure_cookie = false # true behind HTTPS
|
||||
|
||||
[network]
|
||||
connect_timeout_ms = 2000
|
||||
buffer_size = 65536
|
||||
replica_queue_chunks = 16
|
||||
max_buffered_bytes = 268435456 # global shared payload budget for all replica queues
|
||||
max_connections = 2048
|
||||
drain_timeout_ms = 15000
|
||||
|
||||
[monitoring]
|
||||
interval_ms = 3000
|
||||
timeout_ms = 1000
|
||||
failure_threshold = 2
|
||||
recovery_threshold = 2
|
||||
cooldown_ms = 30000
|
||||
concurrency = 32
|
||||
|
||||
# Optional. ForcAD uses a persistent Engine.IO v4 connection; adplatf uses /state.
|
||||
# [scoreboard]
|
||||
# kind = "forcad"
|
||||
# url = "http://scoreboard.example:8080"
|
||||
# poll_interval_ms = 3000
|
||||
# stale_after_ms = 15000
|
||||
|
||||
[[teams]]
|
||||
id = "team-a"
|
||||
name = "Team A"
|
||||
address = "192.0.2.10"
|
||||
|
||||
[[teams]]
|
||||
id = "team-b"
|
||||
name = "Team B"
|
||||
address = "192.0.2.11"
|
||||
|
||||
[[services]]
|
||||
id = "notes"
|
||||
name = "Notes"
|
||||
listen = "0.0.0.0:18080"
|
||||
upstream_port = 8080
|
||||
primary = "team-a"
|
||||
replicas = ["team-b"]
|
||||
enabled = false
|
||||
auto_switch = false
|
||||
# isolate: primary stays fast; explicitly disconnect a slow/failed replica.
|
||||
# strict: backpressure to deliver all bytes to each healthy replica.
|
||||
replica_policy = "isolate"
|
||||
7
progress.md
Normal file
7
progress.md
Normal file
@ -0,0 +1,7 @@
|
||||
# Progress
|
||||
|
||||
2026-10-23: inspected both legacy copies and empty destination. Verified compiler availability on macOS; Orb Ubuntu amd64 has Rust and networking tools. Established scope and sent optional UI clarification (no preferences). No old files modified.
|
||||
|
||||
2026-10-23: created Cargo manifest/config validation and engine contract. TCP engine delegated to one implementer under subagent-driven-development. Canonical ForcAD source verified; found initial-vs-delta event mismatch in legacy client. Orb does not mount the macOS Desktop path directly; stage source under ~/Downloads for Linux runs.
|
||||
|
||||
2026-10-23: engine delivered and self-reviewed, 11 tests passed. Added zero-copy-reuse primary buffer path when no replicas are active (no per-read shared-Bytes allocation). Controller, health, state persistence/rollback, auth, API and CLI implemented. Added control-plane tests; integrated run waiting for UI app.js. UI implementation delegated after engine implementer finished. No live game endpoints contacted.
|
||||
78
scripts/ui_fixture.py
Normal file
78
scripts/ui_fixture.py
Normal file
@ -0,0 +1,78 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Disposable loopback-only UI fixture. Never reads legacy configuration.
|
||||
|
||||
Usage: python3 scripts/ui_fixture.py target/debug/mirror-tool
|
||||
Public test-only login: local-ui-fixture-only. Ctrl-C removes fixture state.
|
||||
"""
|
||||
import asyncio
|
||||
import os
|
||||
from pathlib import Path
|
||||
import signal
|
||||
import sys
|
||||
import tempfile
|
||||
|
||||
|
||||
async def echo(reader, writer):
|
||||
try:
|
||||
while data := await reader.read(65536):
|
||||
writer.write(data)
|
||||
await writer.drain()
|
||||
except (OSError, asyncio.CancelledError):
|
||||
pass
|
||||
finally:
|
||||
writer.close()
|
||||
await writer.wait_closed()
|
||||
|
||||
|
||||
async def main():
|
||||
executable = str(Path(sys.argv[1]).resolve())
|
||||
with tempfile.TemporaryDirectory(prefix="mirror-ui-") as folder:
|
||||
root = Path(folder)
|
||||
config = root / "mirror.toml"
|
||||
config.write_text('''[web]
|
||||
listen = "127.0.0.1:16700"
|
||||
state_file = "state.json"
|
||||
[network]
|
||||
drain_timeout_ms = 500
|
||||
[[teams]]
|
||||
id = "a"
|
||||
name = "Loopback A"
|
||||
address = "127.0.0.1"
|
||||
[[teams]]
|
||||
id = "b"
|
||||
name = "Loopback B"
|
||||
address = "::1"
|
||||
[[services]]
|
||||
id = "echo"
|
||||
name = "Echo"
|
||||
listen = "127.0.0.1:18081"
|
||||
upstream_port = 18082
|
||||
primary = "a"
|
||||
replicas = ["b"]
|
||||
enabled = false
|
||||
auto_switch = false
|
||||
replica_policy = "isolate"
|
||||
''')
|
||||
servers = [await asyncio.start_server(echo, address, 18082) for address in ("127.0.0.1", "::1")]
|
||||
env = os.environ.copy()
|
||||
env["MIRROR_PASSWORD"] = "local-ui-fixture-only"
|
||||
process = await asyncio.create_subprocess_exec(executable, "--config", str(config), env=env)
|
||||
stop = asyncio.Event()
|
||||
for sig in (signal.SIGINT, signal.SIGTERM):
|
||||
asyncio.get_running_loop().add_signal_handler(sig, stop.set)
|
||||
print("UI fixture: http://127.0.0.1:16700 (loopback only)", flush=True)
|
||||
waiter = asyncio.create_task(process.wait())
|
||||
stopper = asyncio.create_task(stop.wait())
|
||||
await asyncio.wait([waiter, stopper], return_when=asyncio.FIRST_COMPLETED)
|
||||
if process.returncode is None:
|
||||
process.terminate()
|
||||
await process.wait()
|
||||
stopper.cancel()
|
||||
await asyncio.gather(stopper, return_exceptions=True)
|
||||
for server in servers:
|
||||
server.close()
|
||||
await server.wait_closed()
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
asyncio.run(main())
|
||||
216
src/auth.rs
Normal file
216
src/auth.rs
Normal file
@ -0,0 +1,216 @@
|
||||
use crate::config::WebConfig;
|
||||
use anyhow::{Result, ensure};
|
||||
use axum::http::HeaderMap;
|
||||
use serde::Serialize;
|
||||
use sha2::{Digest, Sha256};
|
||||
use std::{
|
||||
collections::HashMap,
|
||||
net::IpAddr,
|
||||
sync::Mutex,
|
||||
time::{Duration, Instant},
|
||||
};
|
||||
use subtle::ConstantTimeEq;
|
||||
|
||||
#[derive(Clone)]
|
||||
pub struct Session {
|
||||
pub csrf: String,
|
||||
pub expires: Instant,
|
||||
}
|
||||
struct Attempts {
|
||||
count: u32,
|
||||
window: Instant,
|
||||
}
|
||||
pub struct Auth {
|
||||
password: [u8; 32],
|
||||
sessions: Mutex<HashMap<[u8; 32], Session>>,
|
||||
attempts: Mutex<HashMap<IpAddr, Attempts>>,
|
||||
settings: WebConfig,
|
||||
}
|
||||
#[derive(Serialize)]
|
||||
pub struct SessionInfo {
|
||||
pub authenticated: bool,
|
||||
pub csrf: String,
|
||||
}
|
||||
pub enum LoginError {
|
||||
Invalid,
|
||||
Limited,
|
||||
}
|
||||
fn digest(value: &str) -> [u8; 32] {
|
||||
Sha256::digest(value.as_bytes()).into()
|
||||
}
|
||||
fn random_token() -> String {
|
||||
rand::random::<[u8; 32]>()
|
||||
.iter()
|
||||
.map(|b| format!("{b:02x}"))
|
||||
.collect()
|
||||
}
|
||||
impl Auth {
|
||||
pub fn from_env(settings: &WebConfig) -> Result<Self> {
|
||||
let password = std::env::var(&settings.password_env).map_err(|_| {
|
||||
anyhow::anyhow!(
|
||||
"set {} to a password with at least 12 characters",
|
||||
settings.password_env
|
||||
)
|
||||
})?;
|
||||
ensure!(
|
||||
password.len() >= 12,
|
||||
"{} must contain at least 12 characters",
|
||||
settings.password_env
|
||||
);
|
||||
Ok(Self::new(settings.clone(), &password))
|
||||
}
|
||||
pub fn new(settings: WebConfig, password: &str) -> Self {
|
||||
Self {
|
||||
password: digest(password),
|
||||
settings,
|
||||
sessions: Mutex::new(HashMap::new()),
|
||||
attempts: Mutex::new(HashMap::new()),
|
||||
}
|
||||
}
|
||||
pub fn login(
|
||||
&self,
|
||||
password: &str,
|
||||
peer: IpAddr,
|
||||
) -> std::result::Result<(String, SessionInfo), LoginError> {
|
||||
let mut attempts = self.attempts.lock().unwrap();
|
||||
attempts.retain(|_, a| a.window.elapsed() < Duration::from_secs(30));
|
||||
if attempts.len() >= 4096 && !attempts.contains_key(&peer) {
|
||||
return Err(LoginError::Limited);
|
||||
}
|
||||
let attempt = attempts.entry(peer).or_insert(Attempts {
|
||||
count: 0,
|
||||
window: Instant::now(),
|
||||
});
|
||||
if attempt.count >= 5 {
|
||||
return Err(LoginError::Limited);
|
||||
}
|
||||
if !bool::from(self.password.ct_eq(&digest(password))) {
|
||||
attempt.count += 1;
|
||||
return Err(LoginError::Invalid);
|
||||
}
|
||||
attempts.remove(&peer);
|
||||
drop(attempts);
|
||||
let token = random_token();
|
||||
let csrf = random_token();
|
||||
let mut sessions = self.sessions.lock().unwrap();
|
||||
sessions.retain(|_, s| s.expires > Instant::now());
|
||||
if sessions.len() >= 1024 {
|
||||
return Err(LoginError::Limited);
|
||||
}
|
||||
sessions.insert(
|
||||
digest(&token),
|
||||
Session {
|
||||
csrf: csrf.clone(),
|
||||
expires: Instant::now() + Duration::from_secs(self.settings.session_ttl_seconds),
|
||||
},
|
||||
);
|
||||
let cookie = format!(
|
||||
"mirror_session={token}; HttpOnly; SameSite=Strict; Path=/; Max-Age={}{}",
|
||||
self.settings.session_ttl_seconds,
|
||||
if self.settings.secure_cookie {
|
||||
"; Secure"
|
||||
} else {
|
||||
""
|
||||
}
|
||||
);
|
||||
Ok((
|
||||
cookie,
|
||||
SessionInfo {
|
||||
authenticated: true,
|
||||
csrf,
|
||||
},
|
||||
))
|
||||
}
|
||||
pub fn session(&self, headers: &HeaderMap) -> Option<Session> {
|
||||
let key = cookie_key(headers)?;
|
||||
let mut sessions = self.sessions.lock().unwrap();
|
||||
sessions.retain(|_, s| s.expires > Instant::now());
|
||||
sessions.get(&key).cloned()
|
||||
}
|
||||
pub fn csrf_valid(&self, headers: &HeaderMap, session: &Session) -> bool {
|
||||
headers
|
||||
.get("x-csrf-token")
|
||||
.is_some_and(|v| bool::from(v.as_bytes().ct_eq(session.csrf.as_bytes())))
|
||||
}
|
||||
pub fn logout(&self, headers: &HeaderMap) -> String {
|
||||
if let Some(key) = cookie_key(headers) {
|
||||
self.sessions.lock().unwrap().remove(&key);
|
||||
}
|
||||
format!(
|
||||
"mirror_session=; HttpOnly; SameSite=Strict; Path=/; Max-Age=0{}",
|
||||
if self.settings.secure_cookie {
|
||||
"; Secure"
|
||||
} else {
|
||||
""
|
||||
}
|
||||
)
|
||||
}
|
||||
}
|
||||
fn cookie_key(headers: &HeaderMap) -> Option<[u8; 32]> {
|
||||
let token = headers
|
||||
.get("cookie")?
|
||||
.to_str()
|
||||
.ok()?
|
||||
.split(';')
|
||||
.find_map(|part| part.trim().strip_prefix("mirror_session="))?;
|
||||
if token.len() != 64 || !token.bytes().all(|b| b.is_ascii_hexdigit()) {
|
||||
return None;
|
||||
}
|
||||
Some(digest(token))
|
||||
}
|
||||
pub fn same_origin(headers: &HeaderMap) -> bool {
|
||||
let Some(origin) = headers.get("origin") else {
|
||||
return true;
|
||||
};
|
||||
let Some(host) = headers.get("host").and_then(|h| h.to_str().ok()) else {
|
||||
return false;
|
||||
};
|
||||
let Ok(origin) = origin.to_str().unwrap_or("").parse::<url::Url>() else {
|
||||
return false;
|
||||
};
|
||||
if !matches!(origin.scheme(), "http" | "https") {
|
||||
return false;
|
||||
}
|
||||
let Ok(expected) = format!("{}://{host}", origin.scheme()).parse::<url::Url>() else {
|
||||
return false;
|
||||
};
|
||||
origin.origin() == expected.origin()
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
#[test]
|
||||
fn session_csrf_and_revocation() {
|
||||
let a = Auth::new(WebConfig::default(), "test-passphrase");
|
||||
let (cookie, info) = a
|
||||
.login("test-passphrase", "127.0.0.1".parse().unwrap())
|
||||
.unwrap_or_else(|_| panic!("login failed"));
|
||||
let mut h = HeaderMap::new();
|
||||
h.insert("cookie", cookie.split(';').next().unwrap().parse().unwrap());
|
||||
let s = a.session(&h).unwrap();
|
||||
assert!(!a.csrf_valid(&h, &s));
|
||||
h.insert("x-csrf-token", info.csrf.parse().unwrap());
|
||||
assert!(a.csrf_valid(&h, &s));
|
||||
a.logout(&h);
|
||||
assert!(a.session(&h).is_none());
|
||||
}
|
||||
#[test]
|
||||
fn rejects_cross_origin_and_limits_attempts() {
|
||||
let mut h = HeaderMap::new();
|
||||
h.insert("host", "localhost:6700".parse().unwrap());
|
||||
h.insert("origin", "https://other.example".parse().unwrap());
|
||||
assert!(!same_origin(&h));
|
||||
let a = Auth::new(WebConfig::default(), "test-passphrase");
|
||||
for _ in 0..5 {
|
||||
assert!(matches!(
|
||||
a.login("bad", "127.0.0.1".parse().unwrap()),
|
||||
Err(LoginError::Invalid)
|
||||
));
|
||||
}
|
||||
assert!(matches!(
|
||||
a.login("test-passphrase", "127.0.0.1".parse().unwrap()),
|
||||
Err(LoginError::Limited)
|
||||
));
|
||||
}
|
||||
}
|
||||
366
src/config.rs
Normal file
366
src/config.rs
Normal file
@ -0,0 +1,366 @@
|
||||
use anyhow::{Context, Result, bail, ensure};
|
||||
use serde::{Deserialize, Serialize};
|
||||
use std::{
|
||||
collections::HashSet,
|
||||
net::{IpAddr, SocketAddr},
|
||||
path::{Path, PathBuf},
|
||||
};
|
||||
|
||||
#[derive(Clone, Debug, Default, Serialize, Deserialize)]
|
||||
#[serde(default, deny_unknown_fields)]
|
||||
pub struct Config {
|
||||
pub web: WebConfig,
|
||||
pub network: NetworkConfig,
|
||||
pub monitoring: MonitoringConfig,
|
||||
pub scoreboard: Option<ScoreboardConfig>,
|
||||
pub teams: Vec<Team>,
|
||||
pub services: Vec<ServiceConfig>,
|
||||
}
|
||||
|
||||
#[derive(Clone, Debug, Serialize, Deserialize)]
|
||||
#[serde(default, deny_unknown_fields)]
|
||||
pub struct WebConfig {
|
||||
pub listen: SocketAddr,
|
||||
pub password_env: String,
|
||||
pub state_file: PathBuf,
|
||||
pub session_ttl_seconds: u64,
|
||||
pub secure_cookie: bool,
|
||||
}
|
||||
|
||||
#[derive(Clone, Debug, Serialize, Deserialize)]
|
||||
#[serde(default, deny_unknown_fields)]
|
||||
pub struct NetworkConfig {
|
||||
pub connect_timeout_ms: u64,
|
||||
pub buffer_size: usize,
|
||||
pub replica_queue_chunks: usize,
|
||||
pub max_buffered_bytes: usize,
|
||||
pub max_connections: usize,
|
||||
pub drain_timeout_ms: u64,
|
||||
}
|
||||
|
||||
#[derive(Clone, Debug, Serialize, Deserialize)]
|
||||
#[serde(default, deny_unknown_fields)]
|
||||
pub struct MonitoringConfig {
|
||||
pub interval_ms: u64,
|
||||
pub timeout_ms: u64,
|
||||
pub failure_threshold: u32,
|
||||
pub recovery_threshold: u32,
|
||||
pub cooldown_ms: u64,
|
||||
pub concurrency: usize,
|
||||
}
|
||||
|
||||
#[derive(Clone, Debug, Serialize, Deserialize)]
|
||||
#[serde(deny_unknown_fields)]
|
||||
pub struct ScoreboardConfig {
|
||||
pub kind: ScoreboardKind,
|
||||
pub url: String,
|
||||
#[serde(default = "default_poll")]
|
||||
pub poll_interval_ms: u64,
|
||||
#[serde(default = "default_stale")]
|
||||
pub stale_after_ms: u64,
|
||||
}
|
||||
|
||||
#[derive(Clone, Debug, Serialize, Deserialize, PartialEq, Eq)]
|
||||
#[serde(rename_all = "snake_case")]
|
||||
pub enum ScoreboardKind {
|
||||
Forcad,
|
||||
Adplatf,
|
||||
}
|
||||
|
||||
#[derive(Clone, Debug, Serialize, Deserialize, PartialEq, Eq)]
|
||||
#[serde(deny_unknown_fields)]
|
||||
pub struct Team {
|
||||
pub id: String,
|
||||
pub name: String,
|
||||
pub address: IpAddr,
|
||||
}
|
||||
|
||||
#[derive(Clone, Debug, Serialize, Deserialize, PartialEq, Eq)]
|
||||
#[serde(deny_unknown_fields)]
|
||||
pub struct ServiceConfig {
|
||||
pub id: String,
|
||||
pub name: String,
|
||||
pub listen: SocketAddr,
|
||||
pub upstream_port: u16,
|
||||
#[serde(default)]
|
||||
pub primary: Option<String>,
|
||||
#[serde(default)]
|
||||
pub replicas: Vec<String>,
|
||||
#[serde(default)]
|
||||
pub enabled: bool,
|
||||
#[serde(default)]
|
||||
pub auto_switch: bool,
|
||||
#[serde(default)]
|
||||
pub replica_policy: ReplicaPolicy,
|
||||
}
|
||||
|
||||
#[derive(Clone, Copy, Debug, Default, Serialize, Deserialize, PartialEq, Eq)]
|
||||
#[serde(rename_all = "snake_case")]
|
||||
pub enum ReplicaPolicy {
|
||||
/// Bound memory, await every replica, never intentionally discard a byte.
|
||||
Strict,
|
||||
/// Protect primary latency; disconnect a lagging replica explicitly.
|
||||
#[default]
|
||||
Isolate,
|
||||
}
|
||||
|
||||
impl Default for WebConfig {
|
||||
fn default() -> Self {
|
||||
Self {
|
||||
listen: "127.0.0.1:6700".parse().unwrap(),
|
||||
password_env: "MIRROR_PASSWORD".into(),
|
||||
state_file: "data/state.json".into(),
|
||||
session_ttl_seconds: 43200,
|
||||
secure_cookie: false,
|
||||
}
|
||||
}
|
||||
}
|
||||
impl Default for NetworkConfig {
|
||||
fn default() -> Self {
|
||||
Self {
|
||||
connect_timeout_ms: 2000,
|
||||
buffer_size: 65536,
|
||||
replica_queue_chunks: 16,
|
||||
max_buffered_bytes: 268435456,
|
||||
max_connections: 2048,
|
||||
drain_timeout_ms: 15000,
|
||||
}
|
||||
}
|
||||
}
|
||||
impl Default for MonitoringConfig {
|
||||
fn default() -> Self {
|
||||
Self {
|
||||
interval_ms: 3000,
|
||||
timeout_ms: 1000,
|
||||
failure_threshold: 2,
|
||||
recovery_threshold: 2,
|
||||
cooldown_ms: 30000,
|
||||
concurrency: 32,
|
||||
}
|
||||
}
|
||||
}
|
||||
fn default_poll() -> u64 {
|
||||
3000
|
||||
}
|
||||
fn default_stale() -> u64 {
|
||||
15000
|
||||
}
|
||||
|
||||
pub fn valid_id(id: &str) -> bool {
|
||||
!id.is_empty()
|
||||
&& id.len() <= 64
|
||||
&& id
|
||||
.bytes()
|
||||
.all(|c| c.is_ascii_alphanumeric() || c == b'_' || c == b'-')
|
||||
}
|
||||
|
||||
impl Config {
|
||||
pub fn load(path: &Path) -> Result<Self> {
|
||||
let content = std::fs::read_to_string(path)
|
||||
.with_context(|| format!("read config {}", path.display()))?;
|
||||
let mut config: Self = toml::from_str(&content).context("parse TOML config")?;
|
||||
if config.web.state_file.is_relative() {
|
||||
config.web.state_file = path
|
||||
.parent()
|
||||
.unwrap_or(Path::new("."))
|
||||
.join(&config.web.state_file);
|
||||
}
|
||||
config.validate()?;
|
||||
Ok(config)
|
||||
}
|
||||
|
||||
pub fn validate(&self) -> Result<()> {
|
||||
ensure!(
|
||||
!self.web.password_env.is_empty(),
|
||||
"password_env must name an environment variable"
|
||||
);
|
||||
ensure!(
|
||||
(60..=604800).contains(&self.web.session_ttl_seconds),
|
||||
"session TTL must be 60..604800 seconds"
|
||||
);
|
||||
ensure!(
|
||||
(1024..=1048576).contains(&self.network.buffer_size),
|
||||
"buffer_size must be 1024..1048576"
|
||||
);
|
||||
ensure!(
|
||||
(1..=1024).contains(&self.network.replica_queue_chunks),
|
||||
"replica_queue_chunks must be 1..1024"
|
||||
);
|
||||
ensure!(
|
||||
(self.network.buffer_size..=2147483647).contains(&self.network.max_buffered_bytes),
|
||||
"max_buffered_bytes must cover one buffer and be <= 2 GiB - 1"
|
||||
);
|
||||
ensure!(
|
||||
(1..=1000000).contains(&self.network.max_connections),
|
||||
"max_connections must be positive and <= 1000000"
|
||||
);
|
||||
ensure!(
|
||||
(1..=60000).contains(&self.network.connect_timeout_ms),
|
||||
"connect_timeout_ms must be 1..60000"
|
||||
);
|
||||
ensure!(
|
||||
(1..=300000).contains(&self.network.drain_timeout_ms),
|
||||
"drain_timeout_ms must be 1..300000"
|
||||
);
|
||||
let m = &self.monitoring;
|
||||
ensure!(
|
||||
m.interval_ms >= 100
|
||||
&& m.timeout_ms > 0
|
||||
&& m.timeout_ms <= 60000
|
||||
&& m.concurrency > 0
|
||||
&& m.concurrency <= 1024
|
||||
&& m.failure_threshold > 0
|
||||
&& m.recovery_threshold > 0,
|
||||
"invalid monitoring limits"
|
||||
);
|
||||
if let Some(board) = &self.scoreboard {
|
||||
let url = url::Url::parse(&board.url).context("invalid scoreboard URL")?;
|
||||
ensure!(
|
||||
matches!(url.scheme(), "http" | "https")
|
||||
&& url.host_str().is_some()
|
||||
&& url.username().is_empty()
|
||||
&& url.password().is_none()
|
||||
&& url.query().is_none()
|
||||
&& url.fragment().is_none(),
|
||||
"scoreboard URL must be HTTP(S) without credentials, query or fragment"
|
||||
);
|
||||
ensure!(
|
||||
board.poll_interval_ms >= 100 && board.stale_after_ms >= board.poll_interval_ms,
|
||||
"invalid scoreboard intervals"
|
||||
);
|
||||
}
|
||||
validate_topology(&self.teams, &self.services)?;
|
||||
for service in &self.services {
|
||||
ensure!(
|
||||
!bind_conflicts(service.listen, self.web.listen),
|
||||
"service {} conflicts with web listener",
|
||||
service.id
|
||||
);
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
}
|
||||
|
||||
pub fn bind_conflicts(a: SocketAddr, b: SocketAddr) -> bool {
|
||||
a.port() == b.port() && (a.ip() == b.ip() || a.ip().is_unspecified() || b.ip().is_unspecified())
|
||||
}
|
||||
|
||||
pub fn validate_topology(teams: &[Team], services: &[ServiceConfig]) -> Result<()> {
|
||||
ensure!(
|
||||
teams.len() <= 1024 && services.len() <= 256,
|
||||
"topology limit: 1024 teams, 256 services"
|
||||
);
|
||||
let mut ids = HashSet::new();
|
||||
let mut names = HashSet::new();
|
||||
let mut addresses = HashSet::new();
|
||||
let mut local_addresses: HashSet<IpAddr> = if_addrs::get_if_addrs()
|
||||
.context("enumerate local interface addresses")?
|
||||
.into_iter()
|
||||
.map(|i| i.ip())
|
||||
.collect();
|
||||
local_addresses.insert("127.0.0.1".parse().unwrap());
|
||||
local_addresses.insert("::1".parse().unwrap());
|
||||
for team in teams {
|
||||
ensure!(
|
||||
valid_id(&team.id) && !team.name.trim().is_empty() && team.name.len() <= 128,
|
||||
"invalid team id/name"
|
||||
);
|
||||
ensure!(ids.insert(&team.id), "duplicate team id {}", team.id);
|
||||
ensure!(
|
||||
names.insert(&team.name),
|
||||
"duplicate team name {}",
|
||||
team.name
|
||||
);
|
||||
ensure!(addresses.insert(team.address), "duplicate team address");
|
||||
ensure!(
|
||||
!team.address.is_unspecified() && !team.address.is_multicast(),
|
||||
"team must be a unicast address"
|
||||
);
|
||||
}
|
||||
let mut service_ids = HashSet::new();
|
||||
for (index, service) in services.iter().enumerate() {
|
||||
ensure!(
|
||||
valid_id(&service.id) && !service.name.trim().is_empty() && service.name.len() <= 128,
|
||||
"invalid service id/name"
|
||||
);
|
||||
ensure!(
|
||||
service_ids.insert(&service.id),
|
||||
"duplicate service id {}",
|
||||
service.id
|
||||
);
|
||||
ensure!(
|
||||
service.listen.port() != 0 && service.upstream_port != 0,
|
||||
"service ports must be nonzero"
|
||||
);
|
||||
ensure!(
|
||||
!service.listen.ip().is_multicast(),
|
||||
"listener must be a local unicast or wildcard address"
|
||||
);
|
||||
ensure!(
|
||||
!service.enabled || service.primary.is_some(),
|
||||
"enabled service {} needs a primary",
|
||||
service.id
|
||||
);
|
||||
let mut targets = HashSet::new();
|
||||
for id in service.primary.iter().chain(service.replicas.iter()) {
|
||||
ensure!(
|
||||
targets.insert(id),
|
||||
"duplicate primary/replica on {}",
|
||||
service.id
|
||||
);
|
||||
let Some(team) = teams.iter().find(|t| &t.id == id) else {
|
||||
bail!("unknown team {} in {}", id, service.id)
|
||||
};
|
||||
let target = SocketAddr::new(team.address, service.upstream_port);
|
||||
for listener in services {
|
||||
let reaches_listener = target.port() == listener.listen.port()
|
||||
&& (target.ip() == listener.listen.ip()
|
||||
|| (listener.listen.ip().is_unspecified()
|
||||
&& local_addresses.contains(&target.ip())));
|
||||
ensure!(
|
||||
!reaches_listener,
|
||||
"{} routes to managed listener {} (proxy loop)",
|
||||
service.id,
|
||||
listener.id
|
||||
);
|
||||
}
|
||||
}
|
||||
ensure!(
|
||||
service.replicas.len() <= 32,
|
||||
"at most 32 replicas per service"
|
||||
);
|
||||
for other in &services[..index] {
|
||||
ensure!(
|
||||
!bind_conflicts(service.listen, other.listen),
|
||||
"service {} listener conflicts with {}",
|
||||
service.id,
|
||||
other.id
|
||||
);
|
||||
}
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
#[test]
|
||||
fn rejects_invalid_resource_bounds() {
|
||||
let mut c = Config::default();
|
||||
c.network.replica_queue_chunks = 0;
|
||||
assert!(c.validate().is_err());
|
||||
}
|
||||
#[test]
|
||||
fn rejects_credential_urls() {
|
||||
let c = Config {
|
||||
scoreboard: Some(ScoreboardConfig {
|
||||
kind: ScoreboardKind::Forcad,
|
||||
url: "http://user:secret@localhost/".into(),
|
||||
poll_interval_ms: 3000,
|
||||
stale_after_ms: 15000,
|
||||
}),
|
||||
..Config::default()
|
||||
};
|
||||
assert!(c.validate().is_err());
|
||||
}
|
||||
}
|
||||
473
src/control.rs
Normal file
473
src/control.rs
Normal file
@ -0,0 +1,473 @@
|
||||
use crate::{
|
||||
config::{Config, ServiceConfig, Team},
|
||||
engine::{Engine, EngineStatus},
|
||||
scoreboard::{BoardEntry, BoardSnapshot, Scoreboard, now_ms},
|
||||
store::{self, Topology},
|
||||
};
|
||||
use anyhow::{Context, Result, ensure};
|
||||
use futures_util::{StreamExt, stream};
|
||||
use serde::Serialize;
|
||||
use std::{
|
||||
collections::{BTreeMap, BTreeSet, VecDeque},
|
||||
net::SocketAddr,
|
||||
sync::{Arc, RwLock},
|
||||
time::{Duration, Instant},
|
||||
};
|
||||
use tokio::{net::TcpStream, sync::Mutex};
|
||||
use tokio_util::sync::CancellationToken;
|
||||
|
||||
#[derive(Clone, Debug, Default, Serialize)]
|
||||
pub struct Health {
|
||||
pub reachable: bool,
|
||||
pub healthy: bool,
|
||||
pub consecutive_failures: u32,
|
||||
pub consecutive_successes: u32,
|
||||
pub checked_at_ms: u64,
|
||||
pub latency_ms: u64,
|
||||
}
|
||||
#[derive(Clone, Debug, Serialize)]
|
||||
pub struct Event {
|
||||
pub at_ms: u64,
|
||||
pub service: Option<String>,
|
||||
pub message: String,
|
||||
}
|
||||
#[derive(Clone, Debug, Serialize)]
|
||||
pub struct ServiceView {
|
||||
#[serde(flatten)]
|
||||
pub config: ServiceConfig,
|
||||
pub engine: Option<EngineStatus>,
|
||||
pub health: BTreeMap<String, Health>,
|
||||
pub scoreboard: BTreeMap<String, BoardEntry>,
|
||||
}
|
||||
#[derive(Clone, Debug, Serialize)]
|
||||
pub struct Snapshot {
|
||||
pub now_ms: u64,
|
||||
pub teams: Vec<Team>,
|
||||
pub services: Vec<ServiceView>,
|
||||
pub scoreboard: BoardSnapshot,
|
||||
pub events: Vec<Event>,
|
||||
}
|
||||
|
||||
pub struct Controller {
|
||||
pub config: Config,
|
||||
pub engine: Engine,
|
||||
pub board: Scoreboard,
|
||||
topology: RwLock<Topology>,
|
||||
operations: Mutex<()>,
|
||||
health: RwLock<BTreeMap<SocketAddr, Health>>,
|
||||
switched: Mutex<BTreeMap<String, Instant>>,
|
||||
events: RwLock<VecDeque<Event>>,
|
||||
}
|
||||
impl Controller {
|
||||
pub async fn start(config: Config) -> Result<Arc<Self>> {
|
||||
let topology = Topology::load(&config)?;
|
||||
let this = Arc::new(Self {
|
||||
engine: Engine::new(config.network.clone()),
|
||||
board: Scoreboard::new(config.scoreboard.as_ref()),
|
||||
topology: RwLock::new(topology.clone()),
|
||||
operations: Mutex::new(()),
|
||||
health: RwLock::new(BTreeMap::new()),
|
||||
switched: Mutex::new(BTreeMap::new()),
|
||||
events: RwLock::new(VecDeque::new()),
|
||||
config,
|
||||
});
|
||||
for service in &topology.services {
|
||||
if let Err(e) = this.engine.upsert(service, &topology.teams).await {
|
||||
this.engine.shutdown().await;
|
||||
return Err(e).context("start saved service topology");
|
||||
}
|
||||
}
|
||||
this.event(None, "Панель запущена; конфигурация восстановлена".into());
|
||||
Ok(this)
|
||||
}
|
||||
pub fn topology(&self) -> Topology {
|
||||
self.topology.read().unwrap().clone()
|
||||
}
|
||||
fn event(&self, service: Option<String>, message: String) {
|
||||
let mut events = self.events.write().unwrap();
|
||||
if events.len() >= 100 {
|
||||
events.pop_back();
|
||||
}
|
||||
events.push_front(Event {
|
||||
at_ms: now_ms(),
|
||||
service,
|
||||
message,
|
||||
});
|
||||
}
|
||||
pub fn snapshot(&self) -> Snapshot {
|
||||
let topology = self.topology();
|
||||
let health = self.health.read().unwrap();
|
||||
let engines = self.engine.snapshots();
|
||||
let board = self.board.snapshot();
|
||||
let services = topology
|
||||
.services
|
||||
.iter()
|
||||
.map(|s| {
|
||||
let health = topology
|
||||
.teams
|
||||
.iter()
|
||||
.filter_map(|t| {
|
||||
health
|
||||
.get(&SocketAddr::new(t.address, s.upstream_port))
|
||||
.map(|h| (t.id.clone(), h.clone()))
|
||||
})
|
||||
.collect();
|
||||
let scores = topology
|
||||
.teams
|
||||
.iter()
|
||||
.filter_map(|t| board_entry(&board, s, t).map(|e| (t.id.clone(), e.clone())))
|
||||
.collect();
|
||||
ServiceView {
|
||||
config: s.clone(),
|
||||
engine: engines.get(&s.id).cloned(),
|
||||
health,
|
||||
scoreboard: scores,
|
||||
}
|
||||
})
|
||||
.collect();
|
||||
Snapshot {
|
||||
now_ms: now_ms(),
|
||||
teams: topology.teams,
|
||||
services,
|
||||
scoreboard: board,
|
||||
events: self.events.read().unwrap().iter().cloned().collect(),
|
||||
}
|
||||
}
|
||||
async fn persist(&self, state: Topology) -> Result<()> {
|
||||
let path = self.config.web.state_file.clone();
|
||||
tokio::task::spawn_blocking(move || store::save(&path, &state))
|
||||
.await
|
||||
.context("state writer task")??;
|
||||
Ok(())
|
||||
}
|
||||
pub async fn put_service(&self, service: ServiceConfig) -> Result<()> {
|
||||
let _lock = self.operations.lock().await;
|
||||
self.put_service_locked(service, false).await
|
||||
}
|
||||
async fn put_service_locked(&self, service: ServiceConfig, automatic: bool) -> Result<()> {
|
||||
let mut next = self.topology();
|
||||
let old = next.services.iter().find(|s| s.id == service.id).cloned();
|
||||
if let Some(current) = next.services.iter_mut().find(|s| s.id == service.id) {
|
||||
*current = service.clone();
|
||||
} else {
|
||||
next.services.push(service.clone());
|
||||
}
|
||||
next.validate(&self.config)?;
|
||||
self.engine.upsert(&service, &next.teams).await?;
|
||||
if let Err(e) = self.persist(next.clone()).await {
|
||||
let rollback = if let Some(old) = old.as_ref() {
|
||||
self.engine.upsert(old, &next.teams).await
|
||||
} else {
|
||||
self.engine.remove(&service.id).await
|
||||
};
|
||||
if let Err(rollback_error) = rollback {
|
||||
self.event(
|
||||
Some(service.id.clone()),
|
||||
format!("Ошибка сохранения и отката: {rollback_error}"),
|
||||
);
|
||||
return Err(e).context(format!("engine rollback failed: {rollback_error}"));
|
||||
}
|
||||
return Err(e);
|
||||
}
|
||||
*self.topology.write().unwrap() = next;
|
||||
if old.as_ref().and_then(|s| s.primary.as_ref()) != service.primary.as_ref() {
|
||||
self.switched
|
||||
.lock()
|
||||
.await
|
||||
.insert(service.id.clone(), Instant::now());
|
||||
}
|
||||
self.event(
|
||||
Some(service.id.clone()),
|
||||
if automatic {
|
||||
format!(
|
||||
"Автопереключение: {}",
|
||||
service.primary.as_deref().unwrap_or("—")
|
||||
)
|
||||
} else {
|
||||
"Настройки сохранены".into()
|
||||
},
|
||||
);
|
||||
Ok(())
|
||||
}
|
||||
pub async fn delete_service(&self, id: &str) -> Result<()> {
|
||||
let _lock = self.operations.lock().await;
|
||||
let mut next = self.topology();
|
||||
let old = next
|
||||
.services
|
||||
.iter()
|
||||
.find(|s| s.id == id)
|
||||
.context("service not found")?
|
||||
.clone();
|
||||
ensure!(!old.enabled, "stop service before deleting it");
|
||||
next.services.retain(|s| s.id != id);
|
||||
self.persist(next.clone()).await?;
|
||||
self.engine.remove(id).await?;
|
||||
*self.topology.write().unwrap() = next;
|
||||
self.switched.lock().await.remove(id);
|
||||
self.event(Some(id.into()), "Сервис удалён".into());
|
||||
Ok(())
|
||||
}
|
||||
pub async fn put_team(&self, team: Team) -> Result<()> {
|
||||
let _lock = self.operations.lock().await;
|
||||
let mut next = self.topology();
|
||||
if let Some(old) = next.teams.iter_mut().find(|t| t.id == team.id) {
|
||||
ensure!(
|
||||
old.address == team.address,
|
||||
"address changes require a new team id (existing sessions retain old routes)"
|
||||
);
|
||||
*old = team;
|
||||
} else {
|
||||
next.teams.push(team);
|
||||
}
|
||||
next.validate(&self.config)?;
|
||||
self.persist(next.clone()).await?;
|
||||
*self.topology.write().unwrap() = next;
|
||||
self.event(None, "Список команд сохранён".into());
|
||||
Ok(())
|
||||
}
|
||||
pub async fn monitor(self: Arc<Self>, cancel: CancellationToken) {
|
||||
let mut interval =
|
||||
tokio::time::interval(Duration::from_millis(self.config.monitoring.interval_ms));
|
||||
interval.set_missed_tick_behavior(tokio::time::MissedTickBehavior::Skip);
|
||||
loop {
|
||||
tokio::select! { _=cancel.cancelled()=>break,_=interval.tick()=>{} }
|
||||
self.check_cycle_cancellable(&cancel).await;
|
||||
}
|
||||
}
|
||||
pub async fn check_cycle(&self) {
|
||||
self.check_cycle_cancellable(&CancellationToken::new())
|
||||
.await;
|
||||
}
|
||||
async fn check_cycle_cancellable(&self, cancel: &CancellationToken) {
|
||||
let topology = self.topology();
|
||||
let endpoints: BTreeSet<_> = topology
|
||||
.services
|
||||
.iter()
|
||||
.flat_map(|s| {
|
||||
topology
|
||||
.teams
|
||||
.iter()
|
||||
.map(move |t| SocketAddr::new(t.address, s.upstream_port))
|
||||
})
|
||||
.collect();
|
||||
let timeout = Duration::from_millis(self.config.monitoring.timeout_ms);
|
||||
let checks = stream::iter(endpoints.iter().copied())
|
||||
.map(|address| async move {
|
||||
let start = Instant::now();
|
||||
let ok = matches!(
|
||||
tokio::time::timeout(timeout, TcpStream::connect(address)).await,
|
||||
Ok(Ok(_))
|
||||
);
|
||||
(address, ok, start.elapsed().as_millis() as u64)
|
||||
})
|
||||
.buffer_unordered(self.config.monitoring.concurrency)
|
||||
.collect::<Vec<_>>();
|
||||
let results = tokio::select! {_=cancel.cancelled()=>return,results=checks=>results};
|
||||
{
|
||||
let mut health = self.health.write().unwrap();
|
||||
health.retain(|key, _| endpoints.contains(key));
|
||||
for (address, ok, latency) in results {
|
||||
let h = health.entry(address).or_default();
|
||||
h.reachable = ok;
|
||||
h.checked_at_ms = now_ms();
|
||||
h.latency_ms = latency;
|
||||
if ok {
|
||||
h.consecutive_successes = h.consecutive_successes.saturating_add(1);
|
||||
h.consecutive_failures = 0;
|
||||
if h.consecutive_successes >= self.config.monitoring.recovery_threshold {
|
||||
h.healthy = true;
|
||||
}
|
||||
} else {
|
||||
h.consecutive_failures = h.consecutive_failures.saturating_add(1);
|
||||
h.consecutive_successes = 0;
|
||||
if h.consecutive_failures >= self.config.monitoring.failure_threshold {
|
||||
h.healthy = false;
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
// Serialize selection with manual edits, then evaluate the CURRENT topology.
|
||||
let _operation = self.operations.lock().await;
|
||||
if cancel.is_cancelled() {
|
||||
return;
|
||||
}
|
||||
let current = self.topology();
|
||||
let board = self.board.snapshot();
|
||||
for service in ¤t.services {
|
||||
if !service.enabled || !service.auto_switch {
|
||||
continue;
|
||||
}
|
||||
let cooldown = self
|
||||
.switched
|
||||
.lock()
|
||||
.await
|
||||
.get(&service.id)
|
||||
.is_some_and(|t| {
|
||||
t.elapsed() < Duration::from_millis(self.config.monitoring.cooldown_ms)
|
||||
});
|
||||
if cooldown {
|
||||
continue;
|
||||
}
|
||||
let candidate = {
|
||||
let health = self.health.read().unwrap();
|
||||
select_candidate(
|
||||
service,
|
||||
¤t.teams,
|
||||
&health,
|
||||
&board,
|
||||
self.config.monitoring.failure_threshold,
|
||||
self.config
|
||||
.monitoring
|
||||
.interval_ms
|
||||
.saturating_add(self.config.monitoring.timeout_ms)
|
||||
.saturating_mul(3),
|
||||
)
|
||||
};
|
||||
if let Some(candidate) = candidate {
|
||||
let mut new = service.clone();
|
||||
let old = new.primary.replace(candidate.clone());
|
||||
new.replicas.retain(|id| id != &candidate);
|
||||
if let Some(old) = old {
|
||||
new.replicas.push(old);
|
||||
}
|
||||
if let Err(e) = self.put_service_locked(new, true).await {
|
||||
self.event(
|
||||
Some(service.id.clone()),
|
||||
format!("Автопереключение не выполнено: {e}"),
|
||||
);
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
pub fn board_entry<'a>(
|
||||
board: &'a BoardSnapshot,
|
||||
service: &ServiceConfig,
|
||||
team: &Team,
|
||||
) -> Option<&'a BoardEntry> {
|
||||
// Names/IPs are authoritative mappings; an incidental numeric id match is unsafe.
|
||||
let task = board
|
||||
.tasks
|
||||
.iter()
|
||||
.find(|t| t.name == service.name || t.name == service.id)?;
|
||||
let board_team = board
|
||||
.teams
|
||||
.iter()
|
||||
.find(|t| t.name == team.name || t.address == Some(team.address))?;
|
||||
board.entries.get(&task.id)?.get(&board_team.id)
|
||||
}
|
||||
|
||||
pub fn select_candidate(
|
||||
service: &ServiceConfig,
|
||||
teams: &[Team],
|
||||
health: &BTreeMap<SocketAddr, Health>,
|
||||
board: &BoardSnapshot,
|
||||
failure_threshold: u32,
|
||||
max_health_age_ms: u64,
|
||||
) -> Option<String> {
|
||||
let primary = teams
|
||||
.iter()
|
||||
.find(|t| Some(&t.id) == service.primary.as_ref())?;
|
||||
let recent = |h: &&Health| now_ms().saturating_sub(h.checked_at_ms) <= max_health_age_ms;
|
||||
let ph = health
|
||||
.get(&SocketAddr::new(primary.address, service.upstream_port))
|
||||
.filter(recent);
|
||||
let tcp_failed = ph.is_some_and(|h| h.consecutive_failures >= failure_threshold);
|
||||
let checker_failed = !board.stale
|
||||
&& board_entry(board, service, primary).is_some_and(|e| matches!(e.status, 102..=104));
|
||||
if !tcp_failed && !checker_failed {
|
||||
return None;
|
||||
}
|
||||
let mut candidates = service
|
||||
.replicas
|
||||
.iter()
|
||||
.filter_map(|id| {
|
||||
let team = teams.iter().find(|t| &t.id == id)?;
|
||||
let h = health
|
||||
.get(&SocketAddr::new(team.address, service.upstream_port))
|
||||
.filter(recent)?;
|
||||
if !h.healthy || !h.reachable {
|
||||
return None;
|
||||
}
|
||||
let entry = (!board.stale)
|
||||
.then(|| board_entry(board, service, team))
|
||||
.flatten();
|
||||
if entry.is_some_and(|e| matches!(e.status, 102..=104)) {
|
||||
return None;
|
||||
}
|
||||
// Prefer checker UP; use lower stolen count as legacy tie-breaker, then id.
|
||||
Some((
|
||||
entry.is_none_or(|e| e.status != 101),
|
||||
entry.map_or(u64::MAX, |e| e.stolen),
|
||||
id.clone(),
|
||||
))
|
||||
})
|
||||
.collect::<Vec<_>>();
|
||||
candidates.sort();
|
||||
candidates.into_iter().next().map(|(_, _, id)| id)
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
use crate::config::ReplicaPolicy;
|
||||
fn fixtures() -> (ServiceConfig, Vec<Team>, BTreeMap<SocketAddr, Health>) {
|
||||
let teams = vec![
|
||||
Team {
|
||||
id: "a".into(),
|
||||
name: "A".into(),
|
||||
address: "192.0.2.1".parse().unwrap(),
|
||||
},
|
||||
Team {
|
||||
id: "b".into(),
|
||||
name: "B".into(),
|
||||
address: "192.0.2.2".parse().unwrap(),
|
||||
},
|
||||
];
|
||||
let service = ServiceConfig {
|
||||
id: "notes".into(),
|
||||
name: "Notes".into(),
|
||||
listen: "127.0.0.1:18080".parse().unwrap(),
|
||||
upstream_port: 8080,
|
||||
primary: Some("a".into()),
|
||||
replicas: vec!["b".into()],
|
||||
enabled: true,
|
||||
auto_switch: true,
|
||||
replica_policy: ReplicaPolicy::Isolate,
|
||||
};
|
||||
let health = teams
|
||||
.iter()
|
||||
.map(|t| {
|
||||
(
|
||||
SocketAddr::new(t.address, 8080),
|
||||
Health {
|
||||
reachable: t.id == "b",
|
||||
healthy: t.id == "b",
|
||||
consecutive_failures: if t.id == "a" { 2 } else { 0 },
|
||||
checked_at_ms: now_ms(),
|
||||
..Health::default()
|
||||
},
|
||||
)
|
||||
})
|
||||
.collect();
|
||||
(service, teams, health)
|
||||
}
|
||||
#[test]
|
||||
fn failover_requires_fresh_health_and_stable_failure() {
|
||||
let (s, t, mut h) = fixtures();
|
||||
let b = BoardSnapshot::default();
|
||||
assert_eq!(select_candidate(&s, &t, &h, &b, 2, 1000), Some("b".into()));
|
||||
h.get_mut(&SocketAddr::new(t[0].address, 8080))
|
||||
.unwrap()
|
||||
.consecutive_failures = 1;
|
||||
assert_eq!(select_candidate(&s, &t, &h, &b, 2, 1000), None);
|
||||
h.get_mut(&SocketAddr::new(t[0].address, 8080))
|
||||
.unwrap()
|
||||
.consecutive_failures = 2;
|
||||
h.get_mut(&SocketAddr::new(t[1].address, 8080))
|
||||
.unwrap()
|
||||
.checked_at_ms = 0;
|
||||
assert_eq!(select_candidate(&s, &t, &h, &b, 2, 1000), None);
|
||||
}
|
||||
}
|
||||
684
src/engine.rs
Normal file
684
src/engine.rs
Normal file
@ -0,0 +1,684 @@
|
||||
//! Bounded TCP request mirroring. Routes are captured once per accepted session.
|
||||
use std::{
|
||||
collections::BTreeMap,
|
||||
net::SocketAddr,
|
||||
sync::{
|
||||
Arc, Mutex, RwLock,
|
||||
atomic::{AtomicBool, AtomicU64, Ordering::Relaxed},
|
||||
},
|
||||
time::Duration,
|
||||
};
|
||||
|
||||
use crate::config::{NetworkConfig, ReplicaPolicy, ServiceConfig, Team};
|
||||
use anyhow::{Context, Result, anyhow, ensure};
|
||||
use bytes::Bytes;
|
||||
use serde::Serialize;
|
||||
use tokio::{
|
||||
io::{AsyncReadExt, AsyncWrite, AsyncWriteExt},
|
||||
net::{TcpListener, TcpSocket, TcpStream},
|
||||
sync::{OwnedSemaphorePermit, Semaphore, mpsc, oneshot},
|
||||
task::{JoinHandle, JoinSet},
|
||||
time::timeout,
|
||||
};
|
||||
use tokio_util::sync::CancellationToken;
|
||||
|
||||
#[derive(Clone, Debug, Default, Serialize)]
|
||||
pub struct TrafficStats {
|
||||
pub accepted: u64,
|
||||
pub active: u64,
|
||||
pub completed: u64,
|
||||
pub connect_errors: u64,
|
||||
pub io_errors: u64,
|
||||
pub rejected: u64,
|
||||
pub bytes_up: u64,
|
||||
pub bytes_down: u64,
|
||||
pub replica_disconnects: u64,
|
||||
pub replica_errors: u64,
|
||||
pub replica_bytes: u64,
|
||||
}
|
||||
#[derive(Clone, Debug, Serialize)]
|
||||
pub struct EngineStatus {
|
||||
pub running: bool,
|
||||
pub stats: TrafficStats,
|
||||
pub last_error: Option<String>,
|
||||
}
|
||||
#[derive(Default)]
|
||||
struct Counters {
|
||||
accepted: AtomicU64,
|
||||
active: AtomicU64,
|
||||
completed: AtomicU64,
|
||||
connect_errors: AtomicU64,
|
||||
io_errors: AtomicU64,
|
||||
rejected: AtomicU64,
|
||||
bytes_up: AtomicU64,
|
||||
bytes_down: AtomicU64,
|
||||
replica_disconnects: AtomicU64,
|
||||
replica_errors: AtomicU64,
|
||||
replica_bytes: AtomicU64,
|
||||
}
|
||||
impl Counters {
|
||||
fn snapshot(&self) -> TrafficStats {
|
||||
TrafficStats {
|
||||
accepted: self.accepted.load(Relaxed),
|
||||
active: self.active.load(Relaxed),
|
||||
completed: self.completed.load(Relaxed),
|
||||
connect_errors: self.connect_errors.load(Relaxed),
|
||||
io_errors: self.io_errors.load(Relaxed),
|
||||
rejected: self.rejected.load(Relaxed),
|
||||
bytes_up: self.bytes_up.load(Relaxed),
|
||||
bytes_down: self.bytes_down.load(Relaxed),
|
||||
replica_disconnects: self.replica_disconnects.load(Relaxed),
|
||||
replica_errors: self.replica_errors.load(Relaxed),
|
||||
replica_bytes: self.replica_bytes.load(Relaxed),
|
||||
}
|
||||
}
|
||||
}
|
||||
struct Active(Arc<Counters>);
|
||||
impl Drop for Active {
|
||||
fn drop(&mut self) {
|
||||
self.0.active.fetch_sub(1, Relaxed);
|
||||
}
|
||||
}
|
||||
#[derive(Clone)]
|
||||
struct Route {
|
||||
primary: SocketAddr,
|
||||
replicas: Vec<SocketAddr>,
|
||||
policy: ReplicaPolicy,
|
||||
}
|
||||
struct Generation {
|
||||
stop: CancellationToken,
|
||||
force: CancellationToken,
|
||||
stopped: Option<oneshot::Receiver<()>>,
|
||||
task: JoinHandle<()>,
|
||||
}
|
||||
struct Service {
|
||||
listen: SocketAddr,
|
||||
running: AtomicBool,
|
||||
route: Arc<RwLock<Option<Route>>>,
|
||||
counters: Arc<Counters>,
|
||||
error: Arc<Mutex<Option<String>>>,
|
||||
generations: Mutex<Vec<Generation>>,
|
||||
}
|
||||
struct Inner {
|
||||
settings: NetworkConfig,
|
||||
services: Mutex<BTreeMap<String, Arc<Service>>>,
|
||||
operations: tokio::sync::Mutex<()>,
|
||||
permits: Arc<Semaphore>,
|
||||
payload_budget: Arc<Semaphore>,
|
||||
stopped: AtomicBool,
|
||||
}
|
||||
#[derive(Clone)]
|
||||
pub struct Engine {
|
||||
inner: Arc<Inner>,
|
||||
}
|
||||
impl Engine {
|
||||
pub fn new(settings: NetworkConfig) -> Self {
|
||||
Self {
|
||||
inner: Arc::new(Inner {
|
||||
permits: Arc::new(Semaphore::new(settings.max_connections.max(1))),
|
||||
payload_budget: Arc::new(Semaphore::new(settings.max_buffered_bytes)),
|
||||
settings,
|
||||
services: Mutex::new(BTreeMap::new()),
|
||||
operations: tokio::sync::Mutex::new(()),
|
||||
stopped: AtomicBool::new(false),
|
||||
}),
|
||||
}
|
||||
}
|
||||
pub fn snapshots(&self) -> BTreeMap<String, EngineStatus> {
|
||||
self.inner
|
||||
.services
|
||||
.lock()
|
||||
.unwrap()
|
||||
.iter()
|
||||
.map(|(id, s)| {
|
||||
(
|
||||
id.clone(),
|
||||
EngineStatus {
|
||||
running: s.running.load(Relaxed),
|
||||
stats: s.counters.snapshot(),
|
||||
last_error: s.error.lock().unwrap().clone(),
|
||||
},
|
||||
)
|
||||
})
|
||||
.collect()
|
||||
}
|
||||
pub async fn upsert(&self, service: &ServiceConfig, teams: &[Team]) -> Result<()> {
|
||||
let _operation = self.inner.operations.lock().await;
|
||||
ensure!(!self.inner.stopped.load(Relaxed), "engine is shut down");
|
||||
let route = if service.enabled {
|
||||
let address = |id: &str| -> Result<SocketAddr> {
|
||||
Ok(SocketAddr::new(
|
||||
teams
|
||||
.iter()
|
||||
.find(|t| t.id == id)
|
||||
.ok_or_else(|| anyhow!("unknown team {id}"))?
|
||||
.address,
|
||||
service.upstream_port,
|
||||
))
|
||||
};
|
||||
Some(Route {
|
||||
primary: address(
|
||||
service
|
||||
.primary
|
||||
.as_deref()
|
||||
.context("enabled service requires primary")?,
|
||||
)?,
|
||||
replicas: service
|
||||
.replicas
|
||||
.iter()
|
||||
.map(|id| address(id))
|
||||
.collect::<Result<_>>()?,
|
||||
policy: service.replica_policy,
|
||||
})
|
||||
} else {
|
||||
None
|
||||
};
|
||||
let existing = self
|
||||
.inner
|
||||
.services
|
||||
.lock()
|
||||
.unwrap()
|
||||
.get(&service.id)
|
||||
.cloned();
|
||||
if let Some(s) = &existing {
|
||||
if s.listen == service.listen && s.running.load(Relaxed) && service.enabled {
|
||||
*s.route.write().unwrap() = route;
|
||||
return Ok(());
|
||||
}
|
||||
}
|
||||
// Bind before changing a working listener, so bind failure leaves it intact.
|
||||
let listener = if service.enabled {
|
||||
Some(
|
||||
TcpListener::bind(service.listen)
|
||||
.await
|
||||
.with_context(|| format!("bind {}", service.listen))?,
|
||||
)
|
||||
} else {
|
||||
None
|
||||
};
|
||||
if let Some(s) = &existing {
|
||||
stop_accepting(s).await;
|
||||
}
|
||||
let s = if let Some(old) = existing {
|
||||
if old.listen == service.listen {
|
||||
old
|
||||
} else {
|
||||
Arc::new(Service {
|
||||
listen: service.listen,
|
||||
running: AtomicBool::new(false),
|
||||
route: old.route.clone(),
|
||||
counters: old.counters.clone(),
|
||||
error: old.error.clone(),
|
||||
generations: Mutex::new(std::mem::take(&mut *old.generations.lock().unwrap())),
|
||||
})
|
||||
}
|
||||
} else {
|
||||
Arc::new(Service {
|
||||
listen: service.listen,
|
||||
running: AtomicBool::new(false),
|
||||
route: Arc::new(RwLock::new(None)),
|
||||
counters: Arc::new(Counters::default()),
|
||||
error: Arc::new(Mutex::new(None)),
|
||||
generations: Mutex::new(vec![]),
|
||||
})
|
||||
};
|
||||
*s.route.write().unwrap() = route;
|
||||
// Reap completed generations on every control-plane change.
|
||||
let finished = {
|
||||
let mut generations = s.generations.lock().unwrap();
|
||||
let mut finished = vec![];
|
||||
let mut i = 0;
|
||||
while i < generations.len() {
|
||||
if generations[i].task.is_finished() {
|
||||
finished.push(generations.swap_remove(i));
|
||||
} else {
|
||||
i += 1;
|
||||
}
|
||||
}
|
||||
finished
|
||||
};
|
||||
for generation in finished {
|
||||
let _ = generation.task.await;
|
||||
}
|
||||
if let Some(listener) = listener {
|
||||
s.running.store(true, Relaxed);
|
||||
*s.error.lock().unwrap() = None;
|
||||
let stop = CancellationToken::new();
|
||||
let force = CancellationToken::new();
|
||||
let (tx, rx) = oneshot::channel();
|
||||
let task = tokio::spawn(supervise(
|
||||
listener,
|
||||
s.clone(),
|
||||
self.inner.settings.clone(),
|
||||
(
|
||||
self.inner.permits.clone(),
|
||||
self.inner.payload_budget.clone(),
|
||||
),
|
||||
stop.clone(),
|
||||
force.clone(),
|
||||
tx,
|
||||
));
|
||||
s.generations.lock().unwrap().push(Generation {
|
||||
stop,
|
||||
force,
|
||||
stopped: Some(rx),
|
||||
task,
|
||||
});
|
||||
}
|
||||
self.inner
|
||||
.services
|
||||
.lock()
|
||||
.unwrap()
|
||||
.insert(service.id.clone(), s);
|
||||
Ok(())
|
||||
}
|
||||
/// Stop accepting, allow a bounded drain, then cancel and join every session.
|
||||
/// Metrics are retired only after all owned tasks have stopped.
|
||||
pub async fn remove(&self, id: &str) -> Result<()> {
|
||||
let _operation = self.inner.operations.lock().await;
|
||||
let service = self.inner.services.lock().unwrap().get(id).cloned();
|
||||
if let Some(s) = service {
|
||||
retire(&s, self.inner.settings.drain_timeout_ms).await;
|
||||
self.inner.services.lock().unwrap().remove(id);
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
pub async fn shutdown(&self) {
|
||||
let _operation = self.inner.operations.lock().await;
|
||||
self.inner.stopped.store(true, Relaxed);
|
||||
let services: Vec<_> = self
|
||||
.inner
|
||||
.services
|
||||
.lock()
|
||||
.unwrap()
|
||||
.values()
|
||||
.cloned()
|
||||
.collect();
|
||||
for s in &services {
|
||||
stop_accepting(s).await;
|
||||
}
|
||||
// One drain deadline for all services, rather than one deadline per service.
|
||||
let deadline = tokio::time::Instant::now()
|
||||
+ Duration::from_millis(self.inner.settings.drain_timeout_ms);
|
||||
loop {
|
||||
if services.iter().all(|s| {
|
||||
s.generations
|
||||
.lock()
|
||||
.unwrap()
|
||||
.iter()
|
||||
.all(|g| g.task.is_finished())
|
||||
}) {
|
||||
break;
|
||||
}
|
||||
if tokio::time::Instant::now() >= deadline {
|
||||
for s in &services {
|
||||
for g in s.generations.lock().unwrap().iter() {
|
||||
g.force.cancel();
|
||||
}
|
||||
}
|
||||
break;
|
||||
}
|
||||
tokio::time::sleep(Duration::from_millis(5)).await;
|
||||
}
|
||||
for s in &services {
|
||||
join_generations(s).await;
|
||||
}
|
||||
}
|
||||
}
|
||||
async fn stop_accepting(s: &Service) {
|
||||
s.running.store(false, Relaxed);
|
||||
let receivers = {
|
||||
let mut gs = s.generations.lock().unwrap();
|
||||
gs.iter_mut()
|
||||
.filter_map(|g| {
|
||||
g.stop.cancel();
|
||||
g.stopped.take()
|
||||
})
|
||||
.collect::<Vec<_>>()
|
||||
};
|
||||
for receiver in receivers {
|
||||
let _ = receiver.await;
|
||||
}
|
||||
}
|
||||
async fn join_generations(s: &Service) {
|
||||
// Called only after the drain deadline has been resolved; no timeout drops these handles.
|
||||
loop {
|
||||
let mut generation = s.generations.lock().unwrap().pop();
|
||||
let Some(g) = generation.as_mut() else { break };
|
||||
let result = (&mut g.task).await;
|
||||
let _ = result;
|
||||
}
|
||||
}
|
||||
async fn retire(s: &Service, drain_ms: u64) {
|
||||
stop_accepting(s).await;
|
||||
// Wait without moving handles out of their cancellation registry.
|
||||
let deadline = tokio::time::Instant::now() + Duration::from_millis(drain_ms);
|
||||
loop {
|
||||
if s.generations
|
||||
.lock()
|
||||
.unwrap()
|
||||
.iter()
|
||||
.all(|g| g.task.is_finished())
|
||||
{
|
||||
break;
|
||||
}
|
||||
if tokio::time::Instant::now() >= deadline {
|
||||
for g in s.generations.lock().unwrap().iter() {
|
||||
g.force.cancel();
|
||||
}
|
||||
break;
|
||||
}
|
||||
tokio::time::sleep(Duration::from_millis(5)).await;
|
||||
}
|
||||
join_generations(s).await;
|
||||
}
|
||||
async fn supervise(
|
||||
listener: TcpListener,
|
||||
service: Arc<Service>,
|
||||
settings: NetworkConfig,
|
||||
budgets: (Arc<Semaphore>, Arc<Semaphore>),
|
||||
stop: CancellationToken,
|
||||
force: CancellationToken,
|
||||
stopped: oneshot::Sender<()>,
|
||||
) {
|
||||
let (permits, payload_budget) = budgets;
|
||||
let mut sessions = JoinSet::new();
|
||||
loop {
|
||||
tokio::select! {
|
||||
biased;
|
||||
_=stop.cancelled()=>break,
|
||||
// Reap completed sessions before accepting more; a continuously busy
|
||||
// listener must not accumulate finished JoinSet entries indefinitely.
|
||||
_=sessions.join_next(), if !sessions.is_empty()=>{},
|
||||
result=listener.accept()=>match result {
|
||||
Ok((client,_))=> {
|
||||
service.counters.accepted.fetch_add(1,Relaxed);
|
||||
let Ok(permit)=permits.clone().try_acquire_owned() else { service.counters.rejected.fetch_add(1,Relaxed); drop(client); continue };
|
||||
let route=service.route.read().unwrap().clone();
|
||||
let Some(route)=route else { drop(client); continue };
|
||||
service.counters.active.fetch_add(1,Relaxed);
|
||||
let active=Active(service.counters.clone());
|
||||
let counters=service.counters.clone();
|
||||
let error_service=service.clone();
|
||||
let settings=settings.clone();
|
||||
let force=force.clone();
|
||||
let payload_budget=payload_budget.clone();
|
||||
sessions.spawn(async move {
|
||||
let _active=active;
|
||||
let _permit=permit;
|
||||
let mut workers=JoinSet::new();
|
||||
{
|
||||
tokio::select! {
|
||||
_=force.cancelled()=>{},
|
||||
result=forward(client,route,settings,counters.clone(),&mut workers,payload_budget)=>match result {
|
||||
Ok(())=>{ counters.completed.fetch_add(1,Relaxed); },
|
||||
Err(error)=>{ *error_service.error.lock().unwrap()=Some(error.to_string()); },
|
||||
}
|
||||
}
|
||||
}
|
||||
// Joining nested workers is part of session ownership, including force cancellation.
|
||||
workers.abort_all();
|
||||
while workers.join_next().await.is_some() {}
|
||||
});
|
||||
}
|
||||
Err(error)=> { *service.error.lock().unwrap()=Some(format!("accept: {error}")); service.running.store(false,Relaxed); break; }
|
||||
},
|
||||
}
|
||||
}
|
||||
drop(listener);
|
||||
let _ = stopped.send(());
|
||||
while sessions.join_next().await.is_some() {}
|
||||
}
|
||||
async fn connect(address: SocketAddr, ms: u64) -> Result<TcpStream> {
|
||||
let socket = if address.is_ipv4() {
|
||||
TcpSocket::new_v4()?
|
||||
} else {
|
||||
TcpSocket::new_v6()?
|
||||
};
|
||||
socket.set_nodelay(true)?;
|
||||
let stream = timeout(Duration::from_millis(ms), socket.connect(address))
|
||||
.await
|
||||
.context("connect timeout")??;
|
||||
configure_keepalive(&stream)?;
|
||||
Ok(stream)
|
||||
}
|
||||
fn configure_keepalive(stream: &TcpStream) -> std::io::Result<()> {
|
||||
let settings = socket2::TcpKeepalive::new()
|
||||
.with_time(Duration::from_secs(60))
|
||||
.with_interval(Duration::from_secs(10))
|
||||
.with_retries(3);
|
||||
socket2::SockRef::from(stream).set_tcp_keepalive(&settings)
|
||||
}
|
||||
struct Sink {
|
||||
tx: mpsc::Sender<Bytes>,
|
||||
cancel: CancellationToken,
|
||||
}
|
||||
struct BudgetedChunk {
|
||||
bytes: Vec<u8>,
|
||||
_permit: OwnedSemaphorePermit,
|
||||
}
|
||||
impl AsRef<[u8]> for BudgetedChunk {
|
||||
fn as_ref(&self) -> &[u8] {
|
||||
&self.bytes
|
||||
}
|
||||
}
|
||||
async fn forward(
|
||||
client: TcpStream,
|
||||
route: Route,
|
||||
settings: NetworkConfig,
|
||||
counters: Arc<Counters>,
|
||||
workers: &mut JoinSet<()>,
|
||||
payload_budget: Arc<Semaphore>,
|
||||
) -> Result<()> {
|
||||
client.set_nodelay(true)?;
|
||||
configure_keepalive(&client)?;
|
||||
let primary = match connect(route.primary, settings.connect_timeout_ms).await {
|
||||
Ok(s) => s,
|
||||
Err(e) => {
|
||||
counters.connect_errors.fetch_add(1, Relaxed);
|
||||
return Err(e.context("primary connect"));
|
||||
}
|
||||
};
|
||||
let strict_failure = CancellationToken::new();
|
||||
let mut sinks = Vec::new();
|
||||
for address in route.replicas {
|
||||
// Strict setup completes before reading any client bytes.
|
||||
let stream = if route.policy == ReplicaPolicy::Strict {
|
||||
match connect(address, settings.connect_timeout_ms).await {
|
||||
Ok(stream) => Some(stream),
|
||||
Err(error) => {
|
||||
counters.replica_errors.fetch_add(1, Relaxed);
|
||||
counters.replica_disconnects.fetch_add(1, Relaxed);
|
||||
counters.io_errors.fetch_add(1, Relaxed);
|
||||
return Err(error.context("strict replica connect"));
|
||||
}
|
||||
}
|
||||
} else {
|
||||
None
|
||||
};
|
||||
let (tx, rx) = mpsc::channel(settings.replica_queue_chunks.max(1));
|
||||
let cancel = CancellationToken::new();
|
||||
let worker_cancel = cancel.clone();
|
||||
let failure = strict_failure.clone();
|
||||
let stats = counters.clone();
|
||||
let config = settings.clone();
|
||||
let policy = route.policy;
|
||||
workers.spawn(async move {
|
||||
let result = tokio::select! {
|
||||
biased;
|
||||
_=worker_cancel.cancelled()=>return,
|
||||
result=replica(stream,address,rx,&config,&stats)=>result,
|
||||
};
|
||||
if result.is_err() {
|
||||
stats.replica_errors.fetch_add(1, Relaxed);
|
||||
stats.replica_disconnects.fetch_add(1, Relaxed);
|
||||
if policy == ReplicaPolicy::Strict {
|
||||
failure.cancel();
|
||||
}
|
||||
}
|
||||
});
|
||||
sinks.push(Sink { tx, cancel });
|
||||
}
|
||||
let (mut client_read, mut client_write) = client.into_split();
|
||||
let (mut primary_read, mut primary_write) = primary.into_split();
|
||||
let transfer = async {
|
||||
let up = async {
|
||||
let mut buffer = vec![0; settings.buffer_size.max(1)];
|
||||
loop {
|
||||
let n = client_read.read(&mut buffer).await?;
|
||||
if n == 0 {
|
||||
primary_write.shutdown().await?;
|
||||
break;
|
||||
}
|
||||
// The common unmirrored path reuses its buffer without a per-read
|
||||
// allocation/copy. Allocate shared chunks only while replicas exist.
|
||||
if sinks.is_empty() {
|
||||
write_counted(&mut primary_write, &buffer[..n], &counters.bytes_up).await?;
|
||||
continue;
|
||||
}
|
||||
let permit = if route.policy == ReplicaPolicy::Strict {
|
||||
Some(
|
||||
payload_budget
|
||||
.clone()
|
||||
.acquire_many_owned(n as u32)
|
||||
.await
|
||||
.map_err(std::io::Error::other)?,
|
||||
)
|
||||
} else {
|
||||
payload_budget.clone().try_acquire_many_owned(n as u32).ok()
|
||||
};
|
||||
let Some(permit) = permit else {
|
||||
// Global queue budget exhausted: isolate all mirrors for this
|
||||
// session explicitly instead of allocating beyond the budget.
|
||||
for sink in sinks.drain(..) {
|
||||
if !sink.tx.is_closed() {
|
||||
counters.replica_disconnects.fetch_add(1, Relaxed);
|
||||
}
|
||||
sink.cancel.cancel();
|
||||
}
|
||||
write_counted(&mut primary_write, &buffer[..n], &counters.bytes_up).await?;
|
||||
continue;
|
||||
};
|
||||
let bytes = Bytes::from_owner(BudgetedChunk {
|
||||
bytes: buffer[..n].to_vec(),
|
||||
_permit: permit,
|
||||
});
|
||||
let mut i = 0;
|
||||
while i < sinks.len() {
|
||||
let result = if route.policy == ReplicaPolicy::Strict {
|
||||
sinks[i].tx.send(bytes.clone()).await.map_err(|_| ())
|
||||
} else {
|
||||
sinks[i].tx.try_send(bytes.clone()).map_err(|_| ())
|
||||
};
|
||||
if result.is_err() {
|
||||
if route.policy == ReplicaPolicy::Strict {
|
||||
return Err(std::io::Error::other("strict replica unavailable"));
|
||||
}
|
||||
let sink = sinks.swap_remove(i);
|
||||
// A closed queue already recorded its worker error; a live full queue is an explicit disconnect.
|
||||
if !sink.tx.is_closed() {
|
||||
counters.replica_disconnects.fetch_add(1, Relaxed);
|
||||
}
|
||||
sink.cancel.cancel();
|
||||
} else {
|
||||
i += 1;
|
||||
}
|
||||
}
|
||||
write_counted(&mut primary_write, &bytes, &counters.bytes_up).await?;
|
||||
}
|
||||
sinks.clear();
|
||||
Ok::<_, std::io::Error>(())
|
||||
};
|
||||
let down = async {
|
||||
let mut buffer = vec![0; settings.buffer_size.max(1)];
|
||||
loop {
|
||||
let n = primary_read.read(&mut buffer).await?;
|
||||
if n == 0 {
|
||||
client_write.shutdown().await?;
|
||||
break;
|
||||
}
|
||||
write_counted(&mut client_write, &buffer[..n], &counters.bytes_down).await?;
|
||||
}
|
||||
Ok::<_, std::io::Error>(())
|
||||
};
|
||||
tokio::try_join!(up, down)?;
|
||||
Ok::<_, anyhow::Error>(())
|
||||
};
|
||||
let result = tokio::select! {
|
||||
biased;
|
||||
_=strict_failure.cancelled()=>Err(anyhow!("strict replica failed")),
|
||||
result=transfer=>result,
|
||||
};
|
||||
if result.is_err() {
|
||||
counters.io_errors.fetch_add(1, Relaxed);
|
||||
return result;
|
||||
}
|
||||
// No payload remains in either primary direction. Bound only the isolated replica tail.
|
||||
let drain = async { while workers.join_next().await.is_some() {} };
|
||||
if route.policy == ReplicaPolicy::Isolate {
|
||||
if timeout(Duration::from_millis(settings.drain_timeout_ms), drain)
|
||||
.await
|
||||
.is_err()
|
||||
{
|
||||
counters
|
||||
.replica_disconnects
|
||||
.fetch_add(workers.len() as u64, Relaxed);
|
||||
workers.abort_all();
|
||||
while workers.join_next().await.is_some() {}
|
||||
}
|
||||
} else {
|
||||
tokio::select! {
|
||||
biased;
|
||||
_=strict_failure.cancelled()=>{ counters.io_errors.fetch_add(1,Relaxed); return Err(anyhow!("strict replica failed while draining")); },
|
||||
_=drain=>{},
|
||||
}
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
async fn replica(
|
||||
stream: Option<TcpStream>,
|
||||
address: SocketAddr,
|
||||
mut rx: mpsc::Receiver<Bytes>,
|
||||
settings: &NetworkConfig,
|
||||
counters: &Counters,
|
||||
) -> Result<()> {
|
||||
let stream = match stream {
|
||||
Some(stream) => stream,
|
||||
None => connect(address, settings.connect_timeout_ms).await?,
|
||||
};
|
||||
let (mut reader, mut writer) = stream.into_split();
|
||||
let write = async {
|
||||
while let Some(bytes) = rx.recv().await {
|
||||
write_counted(&mut writer, &bytes, &counters.replica_bytes).await?;
|
||||
}
|
||||
writer.shutdown().await?;
|
||||
Ok::<_, std::io::Error>(())
|
||||
};
|
||||
let read = async {
|
||||
let mut buffer = vec![0; settings.buffer_size.max(1)];
|
||||
while reader.read(&mut buffer).await? != 0 {}
|
||||
Ok::<_, std::io::Error>(())
|
||||
};
|
||||
tokio::try_join!(write, read)?;
|
||||
Ok(())
|
||||
}
|
||||
|
||||
// Count every successful socket write, including a partial chunk before a later error.
|
||||
async fn write_counted<W: AsyncWrite + Unpin>(
|
||||
writer: &mut W,
|
||||
mut bytes: &[u8],
|
||||
counter: &AtomicU64,
|
||||
) -> std::io::Result<()> {
|
||||
while !bytes.is_empty() {
|
||||
let n = writer.write(bytes).await?;
|
||||
if n == 0 {
|
||||
return Err(std::io::Error::new(
|
||||
std::io::ErrorKind::WriteZero,
|
||||
"socket write returned zero",
|
||||
));
|
||||
}
|
||||
counter.fetch_add(n as u64, Relaxed);
|
||||
bytes = &bytes[n..];
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
7
src/lib.rs
Normal file
7
src/lib.rs
Normal file
@ -0,0 +1,7 @@
|
||||
pub mod auth;
|
||||
pub mod config;
|
||||
pub mod control;
|
||||
pub mod engine;
|
||||
pub mod scoreboard;
|
||||
pub mod store;
|
||||
pub mod web;
|
||||
94
src/main.rs
Normal file
94
src/main.rs
Normal file
@ -0,0 +1,94 @@
|
||||
use anyhow::{Context, Result};
|
||||
use clap::{Parser, Subcommand};
|
||||
use mirror_tool::{auth::Auth, config::Config, control::Controller, web};
|
||||
use std::path::PathBuf;
|
||||
use tokio_util::sync::CancellationToken;
|
||||
|
||||
#[derive(Parser)]
|
||||
#[command(version, about = "Async TCP mirror and ForcAD-aware upstream router")]
|
||||
struct Cli {
|
||||
#[arg(short, long, default_value = "mirror.toml", global = true)]
|
||||
config: PathBuf,
|
||||
#[command(subcommand)]
|
||||
command: Option<Command>,
|
||||
}
|
||||
#[derive(Subcommand)]
|
||||
enum Command {
|
||||
/// Validate TOML and any existing saved topology without binding or network I/O.
|
||||
Check,
|
||||
/// Print a portable sample configuration (no secrets).
|
||||
Example,
|
||||
}
|
||||
|
||||
#[tokio::main]
|
||||
async fn main() -> Result<()> {
|
||||
tracing_subscriber::fmt()
|
||||
.with_env_filter(
|
||||
tracing_subscriber::EnvFilter::try_from_default_env()
|
||||
.unwrap_or_else(|_| "mirror_tool=info".into()),
|
||||
)
|
||||
.init();
|
||||
let cli = Cli::parse();
|
||||
if matches!(cli.command, Some(Command::Example)) {
|
||||
print!("{}", include_str!("../mirror.example.toml"));
|
||||
return Ok(());
|
||||
}
|
||||
let config = Config::load(&cli.config)?;
|
||||
if matches!(cli.command, Some(Command::Check)) {
|
||||
let topology = mirror_tool::store::Topology::load(&config)?;
|
||||
println!(
|
||||
"Configuration valid: {} teams, {} services",
|
||||
topology.teams.len(),
|
||||
topology.services.len()
|
||||
);
|
||||
return Ok(());
|
||||
}
|
||||
let auth = Auth::from_env(&config.web)?;
|
||||
let listen = config.web.listen;
|
||||
// Reserve control listener before starting data listeners, so startup cannot half-succeed.
|
||||
let listener = tokio::net::TcpListener::bind(listen)
|
||||
.await
|
||||
.context("bind web listener")?;
|
||||
let controller = Controller::start(config).await?;
|
||||
let cancel = CancellationToken::new();
|
||||
let monitor = tokio::spawn(controller.clone().monitor(cancel.clone()));
|
||||
let board = controller
|
||||
.config
|
||||
.scoreboard
|
||||
.clone()
|
||||
.map(|config| tokio::spawn(controller.board.clone().run(config, cancel.clone())));
|
||||
let router = web::router(controller.clone(), auth, cancel.clone());
|
||||
tracing::info!(%listen,"Mirror Tool ready");
|
||||
let server = axum::serve(
|
||||
listener,
|
||||
router.into_make_service_with_connect_info::<std::net::SocketAddr>(),
|
||||
);
|
||||
let signal_cancel = cancel.clone();
|
||||
let result = server
|
||||
.with_graceful_shutdown(async move {
|
||||
shutdown_signal().await;
|
||||
signal_cancel.cancel();
|
||||
})
|
||||
.await;
|
||||
cancel.cancel();
|
||||
// Complete in-flight monitor transactions before tearing down the engine.
|
||||
let _ = monitor.await;
|
||||
if let Some(board) = board {
|
||||
let _ = board.await;
|
||||
}
|
||||
controller.engine.shutdown().await;
|
||||
result.context("HTTP server")?;
|
||||
Ok(())
|
||||
}
|
||||
async fn shutdown_signal() {
|
||||
#[cfg(unix)]
|
||||
{
|
||||
let mut term = tokio::signal::unix::signal(tokio::signal::unix::SignalKind::terminate())
|
||||
.expect("install SIGTERM handler");
|
||||
tokio::select! {_=tokio::signal::ctrl_c()=>{},_=term.recv()=>{}}
|
||||
}
|
||||
#[cfg(not(unix))]
|
||||
{
|
||||
let _ = tokio::signal::ctrl_c().await;
|
||||
}
|
||||
}
|
||||
457
src/scoreboard.rs
Normal file
457
src/scoreboard.rs
Normal file
@ -0,0 +1,457 @@
|
||||
//! ForcAD Engine.IO v4 polling transport and normalized scoreboard snapshots.
|
||||
//! Wire format checked against pomo-mondreganto/ForcAD at 6ce3407.
|
||||
use crate::config::{ScoreboardConfig, ScoreboardKind};
|
||||
use anyhow::{Context, Result, bail, ensure};
|
||||
use futures_util::StreamExt;
|
||||
use reqwest::Client;
|
||||
use serde::{Deserialize, Serialize};
|
||||
use serde_json::Value;
|
||||
use std::{
|
||||
collections::BTreeMap,
|
||||
net::IpAddr,
|
||||
sync::{Arc, RwLock},
|
||||
time::{Duration, SystemTime, UNIX_EPOCH},
|
||||
};
|
||||
use tokio_util::sync::CancellationToken;
|
||||
|
||||
const BODY_LIMIT: usize = 8 * 1024 * 1024;
|
||||
|
||||
pub fn now_ms() -> u64 {
|
||||
SystemTime::now()
|
||||
.duration_since(UNIX_EPOCH)
|
||||
.unwrap_or_default()
|
||||
.as_millis() as u64
|
||||
}
|
||||
|
||||
#[derive(Clone, Debug, Default, Serialize, Deserialize)]
|
||||
pub struct BoardTeam {
|
||||
pub id: String,
|
||||
pub name: String,
|
||||
pub address: Option<IpAddr>,
|
||||
}
|
||||
#[derive(Clone, Debug, Default, Serialize, Deserialize)]
|
||||
pub struct BoardTask {
|
||||
pub id: String,
|
||||
pub name: String,
|
||||
}
|
||||
#[derive(Clone, Debug, Default, Serialize, Deserialize)]
|
||||
pub struct BoardEntry {
|
||||
pub status: i64,
|
||||
pub stolen: u64,
|
||||
pub lost: u64,
|
||||
}
|
||||
#[derive(Clone, Debug, Serialize)]
|
||||
pub struct BoardSnapshot {
|
||||
pub enabled: bool,
|
||||
pub connected: bool,
|
||||
pub kind: Option<ScoreboardKind>,
|
||||
pub round: i64,
|
||||
pub updated_at_ms: Option<u64>,
|
||||
pub stale: bool,
|
||||
pub last_error: Option<String>,
|
||||
pub teams: Vec<BoardTeam>,
|
||||
pub tasks: Vec<BoardTask>,
|
||||
/// task id -> team id -> checker state
|
||||
pub entries: BTreeMap<String, BTreeMap<String, BoardEntry>>,
|
||||
}
|
||||
impl Default for BoardSnapshot {
|
||||
fn default() -> Self {
|
||||
Self {
|
||||
enabled: false,
|
||||
connected: false,
|
||||
kind: None,
|
||||
round: -1,
|
||||
updated_at_ms: None,
|
||||
stale: true,
|
||||
last_error: None,
|
||||
teams: vec![],
|
||||
tasks: vec![],
|
||||
entries: BTreeMap::new(),
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
#[derive(Clone)]
|
||||
pub struct Scoreboard {
|
||||
state: Arc<RwLock<BoardSnapshot>>,
|
||||
stale_after_ms: u64,
|
||||
}
|
||||
impl Scoreboard {
|
||||
pub fn new(config: Option<&ScoreboardConfig>) -> Self {
|
||||
Self {
|
||||
state: Arc::new(RwLock::new(BoardSnapshot {
|
||||
enabled: config.is_some(),
|
||||
kind: config.map(|c| c.kind.clone()),
|
||||
..BoardSnapshot::default()
|
||||
})),
|
||||
stale_after_ms: config.map_or(15000, |c| c.stale_after_ms),
|
||||
}
|
||||
}
|
||||
pub fn snapshot(&self) -> BoardSnapshot {
|
||||
let mut s = self.state.read().unwrap().clone();
|
||||
s.stale = !s.connected
|
||||
|| s.updated_at_ms
|
||||
.is_none_or(|t| now_ms().saturating_sub(t) > self.stale_after_ms);
|
||||
s
|
||||
}
|
||||
fn error(&self, message: &str) {
|
||||
let mut s = self.state.write().unwrap();
|
||||
s.connected = false;
|
||||
s.last_error = Some(message.to_owned());
|
||||
}
|
||||
fn connected(&self) {
|
||||
self.state.write().unwrap().connected = true;
|
||||
}
|
||||
pub fn apply_forcad(&self, event: &str, envelope: &Value) -> Result<()> {
|
||||
let data = envelope
|
||||
.get("data")
|
||||
.context("ForcAD event is missing data")?;
|
||||
let mut s = self.state.write().unwrap();
|
||||
if event == "init_scoreboard" {
|
||||
let teams = data
|
||||
.get("teams")
|
||||
.and_then(Value::as_array)
|
||||
.context("ForcAD init missing teams")?;
|
||||
let tasks = data
|
||||
.get("tasks")
|
||||
.and_then(Value::as_array)
|
||||
.context("ForcAD init missing tasks")?;
|
||||
let new_teams = teams
|
||||
.iter()
|
||||
.map(|t| {
|
||||
Ok(BoardTeam {
|
||||
id: id(&t["id"])?,
|
||||
name: t["name"].as_str().context("team name")?.into(),
|
||||
address: t["ip"].as_str().and_then(|x| x.parse().ok()),
|
||||
})
|
||||
})
|
||||
.collect::<Result<Vec<_>>>()?;
|
||||
let new_tasks = tasks
|
||||
.iter()
|
||||
.map(|t| {
|
||||
Ok(BoardTask {
|
||||
id: id(&t["id"])?,
|
||||
name: t["name"].as_str().context("task name")?.into(),
|
||||
})
|
||||
})
|
||||
.collect::<Result<Vec<_>>>()?;
|
||||
// Init may precede the game; retain empty state with known metadata.
|
||||
let (round, entries) = if data["state"].is_null() {
|
||||
(-1, BTreeMap::new())
|
||||
} else {
|
||||
parse_forcad_state(&data["state"])?
|
||||
};
|
||||
s.teams = new_teams;
|
||||
s.tasks = new_tasks;
|
||||
s.round = round;
|
||||
s.entries = entries;
|
||||
} else if event == "update_scoreboard" {
|
||||
ensure!(
|
||||
!s.tasks.is_empty() && !s.teams.is_empty(),
|
||||
"ForcAD update received before init"
|
||||
);
|
||||
let (round, entries) = parse_forcad_state(data)?;
|
||||
if round < s.round {
|
||||
return Ok(());
|
||||
}
|
||||
s.round = round;
|
||||
s.entries = entries;
|
||||
} else {
|
||||
return Ok(());
|
||||
}
|
||||
s.connected = true;
|
||||
s.updated_at_ms = Some(now_ms());
|
||||
s.last_error = None;
|
||||
Ok(())
|
||||
}
|
||||
pub fn apply_adplatf(&self, payload: &Value) -> Result<()> {
|
||||
let config = &payload["config"];
|
||||
let teams = config["teams"]
|
||||
.as_array()
|
||||
.context("adplatf missing config.teams")?;
|
||||
let tasks = config["services"]
|
||||
.as_array()
|
||||
.context("adplatf missing config.services")?;
|
||||
let round = payload["state"]["currentRound"]
|
||||
.as_i64()
|
||||
.context("adplatf missing currentRound")?;
|
||||
let mut entries: BTreeMap<String, BTreeMap<String, BoardEntry>> = BTreeMap::new();
|
||||
for (team, services) in payload["statuses"]
|
||||
.as_object()
|
||||
.context("adplatf missing statuses")?
|
||||
{
|
||||
for (task, status) in services.as_object().context("adplatf service statuses")? {
|
||||
let score = &payload["scores"][team][task];
|
||||
let status = match status["status"]
|
||||
.as_str()
|
||||
.unwrap_or("")
|
||||
.to_ascii_uppercase()
|
||||
.as_str()
|
||||
{
|
||||
"UP" => 101,
|
||||
"CORRUPT" => 102,
|
||||
"MUMBLE" => 103,
|
||||
"DOWN" => 104,
|
||||
_ => -1,
|
||||
};
|
||||
entries.entry(task.clone()).or_default().insert(
|
||||
team.clone(),
|
||||
BoardEntry {
|
||||
status,
|
||||
stolen: unsigned(&score["stolenFlags"]),
|
||||
lost: unsigned(&score["lostFlags"]),
|
||||
},
|
||||
);
|
||||
}
|
||||
}
|
||||
let mut s = self.state.write().unwrap();
|
||||
s.teams = teams
|
||||
.iter()
|
||||
.enumerate()
|
||||
.map(|(i, t)| BoardTeam {
|
||||
id: i.to_string(),
|
||||
name: t["name"].as_str().unwrap_or("").into(),
|
||||
address: t["ip"].as_str().and_then(|v| v.parse().ok()),
|
||||
})
|
||||
.collect();
|
||||
s.tasks = tasks
|
||||
.iter()
|
||||
.enumerate()
|
||||
.map(|(i, t)| BoardTask {
|
||||
id: i.to_string(),
|
||||
name: t.as_str().unwrap_or("").into(),
|
||||
})
|
||||
.collect();
|
||||
s.entries = entries;
|
||||
s.round = round;
|
||||
s.connected = true;
|
||||
s.updated_at_ms = Some(now_ms());
|
||||
s.last_error = None;
|
||||
Ok(())
|
||||
}
|
||||
pub async fn run(self, config: ScoreboardConfig, cancel: CancellationToken) {
|
||||
let client = match Client::builder()
|
||||
.connect_timeout(Duration::from_secs(5))
|
||||
.redirect(reqwest::redirect::Policy::none())
|
||||
.build()
|
||||
{
|
||||
Ok(c) => c,
|
||||
Err(_) => {
|
||||
self.error("cannot initialize scoreboard HTTP client");
|
||||
return;
|
||||
}
|
||||
};
|
||||
let mut backoff = 500;
|
||||
loop {
|
||||
let started = tokio::time::Instant::now();
|
||||
let result = tokio::select! {
|
||||
_=cancel.cancelled()=>break,
|
||||
r=async {
|
||||
match config.kind {
|
||||
ScoreboardKind::Forcad=>self.forcad_session(&client,&config).await,
|
||||
ScoreboardKind::Adplatf=>self.adplatf_loop(&client,&config).await,
|
||||
}
|
||||
}=>r
|
||||
};
|
||||
if let Err(e) = result {
|
||||
self.error(&e.to_string());
|
||||
}
|
||||
if started.elapsed() > Duration::from_secs(20) {
|
||||
backoff = 500;
|
||||
}
|
||||
tokio::select! { _=cancel.cancelled()=>break,_=tokio::time::sleep(Duration::from_millis(backoff))=>{} }
|
||||
backoff = (backoff * 2).min(30000);
|
||||
}
|
||||
self.state.write().unwrap().connected = false;
|
||||
}
|
||||
async fn adplatf_loop(&self, client: &Client, config: &ScoreboardConfig) -> Result<()> {
|
||||
let url = format!("{}/state", config.url.trim_end_matches('/'));
|
||||
loop {
|
||||
let response = client
|
||||
.get(&url)
|
||||
.timeout(Duration::from_secs(10))
|
||||
.send()
|
||||
.await
|
||||
.map_err(|_| anyhow::anyhow!("adplatf request failed"))?;
|
||||
let bytes = body(response).await?;
|
||||
self.apply_adplatf(&serde_json::from_slice(&bytes).context("invalid adplatf JSON")?)?;
|
||||
tokio::time::sleep(Duration::from_millis(config.poll_interval_ms)).await;
|
||||
}
|
||||
}
|
||||
async fn forcad_session(&self, client: &Client, config: &ScoreboardConfig) -> Result<()> {
|
||||
let mut base =
|
||||
url::Url::parse(&format!("{}/socket.io/", config.url.trim_end_matches('/')))?;
|
||||
base.query_pairs_mut()
|
||||
.append_pair("EIO", "4")
|
||||
.append_pair("transport", "polling");
|
||||
let response = client
|
||||
.get(base.clone())
|
||||
.timeout(Duration::from_secs(10))
|
||||
.send()
|
||||
.await
|
||||
.map_err(|_| anyhow::anyhow!("ForcAD handshake request failed"))?;
|
||||
let bytes = body(response).await?;
|
||||
let text = std::str::from_utf8(&bytes).context("ForcAD handshake encoding")?;
|
||||
let open = text
|
||||
.split('\u{1e}')
|
||||
.next()
|
||||
.unwrap_or("")
|
||||
.strip_prefix('0')
|
||||
.context("ForcAD requires Engine.IO v4 polling")?;
|
||||
let hello: Value = serde_json::from_str(open).context("ForcAD handshake JSON")?;
|
||||
let sid = hello["sid"].as_str().context("ForcAD missing session id")?;
|
||||
ensure!(
|
||||
!sid.is_empty()
|
||||
&& sid.len() <= 128
|
||||
&& sid
|
||||
.bytes()
|
||||
.all(|b| b.is_ascii_alphanumeric() || b == b'-' || b == b'_'),
|
||||
"invalid ForcAD session id"
|
||||
);
|
||||
base.query_pairs_mut().append_pair("sid", sid);
|
||||
let ping = hello["pingInterval"].as_u64().unwrap_or(25000);
|
||||
let timeout = hello["pingTimeout"].as_u64().unwrap_or(20000);
|
||||
let request_timeout =
|
||||
Duration::from_millis(ping.saturating_add(timeout).clamp(1000, 120000));
|
||||
post(client, &base, "40/game_events,").await?;
|
||||
let mut initialized = false;
|
||||
let connected_at = tokio::time::Instant::now();
|
||||
loop {
|
||||
let response = client
|
||||
.get(base.clone())
|
||||
.timeout(request_timeout)
|
||||
.send()
|
||||
.await
|
||||
.map_err(|_| anyhow::anyhow!("ForcAD poll failed or heartbeat timed out"))?;
|
||||
let bytes = body(response).await?;
|
||||
let text = std::str::from_utf8(&bytes).context("ForcAD poll encoding")?;
|
||||
ensure!(!text.is_empty(), "ForcAD empty polling response");
|
||||
for frame in text.split('\u{1e}') {
|
||||
if let Some(ping) = frame.strip_prefix('2') {
|
||||
post(client, &base, &format!("3{ping}")).await?;
|
||||
} else if frame.starts_with("40/game_events,") {
|
||||
self.connected();
|
||||
} else if let Some(event) = frame.strip_prefix("42/game_events,") {
|
||||
let payload: Vec<Value> =
|
||||
serde_json::from_str(event).context("ForcAD Socket.IO event JSON")?;
|
||||
if let (Some(name), Some(data)) =
|
||||
(payload.first().and_then(Value::as_str), payload.get(1))
|
||||
{
|
||||
if matches!(name, "init_scoreboard" | "update_scoreboard") {
|
||||
self.apply_forcad(name, data)?;
|
||||
initialized = true;
|
||||
}
|
||||
}
|
||||
} else if frame.starts_with('1')
|
||||
|| frame.starts_with("41/game_events")
|
||||
|| frame.starts_with("44/game_events")
|
||||
{
|
||||
bail!("ForcAD namespace disconnected or rejected");
|
||||
}
|
||||
}
|
||||
ensure!(
|
||||
initialized || connected_at.elapsed() < Duration::from_secs(60),
|
||||
"ForcAD did not send initial scoreboard"
|
||||
);
|
||||
}
|
||||
}
|
||||
}
|
||||
async fn post(client: &Client, url: &url::Url, payload: &str) -> Result<()> {
|
||||
let response = client
|
||||
.post(url.clone())
|
||||
.header("content-type", "text/plain;charset=UTF-8")
|
||||
.body(payload.to_owned())
|
||||
.timeout(Duration::from_secs(10))
|
||||
.send()
|
||||
.await
|
||||
.map_err(|_| anyhow::anyhow!("ForcAD control request failed"))?;
|
||||
let _ = body(response).await?;
|
||||
Ok(())
|
||||
}
|
||||
async fn body(response: reqwest::Response) -> Result<Vec<u8>> {
|
||||
ensure!(
|
||||
response.status().is_success(),
|
||||
"scoreboard HTTP status {}",
|
||||
response.status().as_u16()
|
||||
);
|
||||
ensure!(
|
||||
response.content_length().unwrap_or(0) <= BODY_LIMIT as u64,
|
||||
"scoreboard response exceeds 8 MiB"
|
||||
);
|
||||
let mut result = Vec::new();
|
||||
let mut stream = response.bytes_stream();
|
||||
while let Some(chunk) = stream.next().await {
|
||||
let chunk = chunk.map_err(|_| anyhow::anyhow!("scoreboard response read failed"))?;
|
||||
ensure!(
|
||||
result.len() + chunk.len() <= BODY_LIMIT,
|
||||
"scoreboard response exceeds 8 MiB"
|
||||
);
|
||||
result.extend_from_slice(&chunk);
|
||||
}
|
||||
Ok(result)
|
||||
}
|
||||
fn id(v: &Value) -> Result<String> {
|
||||
if let Some(s) = v.as_str() {
|
||||
ensure!(s.len() <= 128, "scoreboard id too long");
|
||||
Ok(s.to_owned())
|
||||
} else if let Some(n) = v.as_i64() {
|
||||
Ok(n.to_string())
|
||||
} else {
|
||||
bail!("scoreboard id is missing or invalid")
|
||||
}
|
||||
}
|
||||
fn unsigned(v: &Value) -> u64 {
|
||||
v.as_u64()
|
||||
.or_else(|| v.as_str().and_then(|s| s.parse().ok()))
|
||||
.unwrap_or(0)
|
||||
}
|
||||
type Entries = BTreeMap<String, BTreeMap<String, BoardEntry>>;
|
||||
fn parse_forcad_state(data: &Value) -> Result<(i64, Entries)> {
|
||||
let round = data["round"]
|
||||
.as_i64()
|
||||
.context("ForcAD state missing round")?;
|
||||
let mut entries: Entries = BTreeMap::new();
|
||||
for tt in data["team_tasks"]
|
||||
.as_array()
|
||||
.context("ForcAD state missing team_tasks")?
|
||||
{
|
||||
entries.entry(id(&tt["task_id"])?).or_default().insert(
|
||||
id(&tt["team_id"])?,
|
||||
BoardEntry {
|
||||
status: tt["status"].as_i64().unwrap_or(-1),
|
||||
stolen: unsigned(&tt["stolen"]),
|
||||
lost: unsigned(&tt["lost"]),
|
||||
},
|
||||
);
|
||||
}
|
||||
Ok((round, entries))
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
use serde_json::json;
|
||||
#[test]
|
||||
fn real_forcad_init_and_delta_retain_metadata() {
|
||||
let b = Scoreboard::new(None);
|
||||
b.apply_forcad("init_scoreboard",&json!({"data":{"teams":[{"id":1,"name":"A"}],"tasks":[{"id":2,"name":"Notes"}],"state":{"round":1,"team_tasks":[{"team_id":1,"task_id":2,"status":101}]}}})).unwrap();
|
||||
b.apply_forcad("update_scoreboard",&json!({"data":{"round":2,"team_tasks":[{"team_id":1,"task_id":2,"status":104,"stolen":4}]}})).unwrap();
|
||||
let s = b.snapshot();
|
||||
assert_eq!(s.round, 2);
|
||||
assert_eq!(s.teams[0].name, "A");
|
||||
assert_eq!(s.entries["2"]["1"].status, 104);
|
||||
assert_eq!(s.entries["2"]["1"].stolen, 4);
|
||||
b.apply_forcad(
|
||||
"update_scoreboard",
|
||||
&json!({"data":{"round":1,"team_tasks":[]}}),
|
||||
)
|
||||
.unwrap();
|
||||
assert_eq!(b.snapshot().round, 2);
|
||||
}
|
||||
#[test]
|
||||
fn adplatf_mapping() {
|
||||
let b = Scoreboard::new(None);
|
||||
b.apply_adplatf(&json!({"config":{"teams":[{"name":"A","ip":"192.0.2.1"}],"services":["Notes"]},"state":{"currentRound":3},"statuses":{"0":{"0":{"status":"UP"}}},"scores":{"0":{"0":{"stolenFlags":9}}}})).unwrap();
|
||||
assert_eq!(b.snapshot().entries["0"]["0"].stolen, 9);
|
||||
}
|
||||
}
|
||||
99
src/store.rs
Normal file
99
src/store.rs
Normal file
@ -0,0 +1,99 @@
|
||||
use crate::config::{Config, ServiceConfig, Team};
|
||||
use anyhow::{Context, Result, ensure};
|
||||
use serde::{Deserialize, Serialize};
|
||||
use std::{io::Write, path::Path};
|
||||
|
||||
#[derive(Clone, Debug, Serialize, Deserialize)]
|
||||
#[serde(deny_unknown_fields)]
|
||||
pub struct Topology {
|
||||
pub version: u32,
|
||||
pub teams: Vec<Team>,
|
||||
pub services: Vec<ServiceConfig>,
|
||||
}
|
||||
|
||||
impl Topology {
|
||||
pub fn from_config(config: &Config) -> Self {
|
||||
Self {
|
||||
version: 1,
|
||||
teams: config.teams.clone(),
|
||||
services: config.services.clone(),
|
||||
}
|
||||
}
|
||||
pub fn validate(&self, config: &Config) -> Result<()> {
|
||||
ensure!(self.version == 1, "unsupported state version");
|
||||
let mut effective = config.clone();
|
||||
effective.teams = self.teams.clone();
|
||||
effective.services = self.services.clone();
|
||||
effective.validate()
|
||||
}
|
||||
pub fn load(config: &Config) -> Result<Self> {
|
||||
let path = &config.web.state_file;
|
||||
if !path.exists() {
|
||||
return Ok(Self::from_config(config));
|
||||
}
|
||||
let meta = std::fs::metadata(path).context("read state metadata")?;
|
||||
ensure!(meta.len() <= 4 * 1024 * 1024, "state exceeds 4 MiB");
|
||||
let state: Self = serde_json::from_slice(&std::fs::read(path).context("read state")?)
|
||||
.context("parse saved state (move it aside to use config topology)")?;
|
||||
state.validate(config)?;
|
||||
Ok(state)
|
||||
}
|
||||
}
|
||||
|
||||
/// Atomic replacement on the same filesystem. State contains no credentials.
|
||||
pub fn save(path: &Path, state: &Topology) -> Result<()> {
|
||||
let parent = path
|
||||
.parent()
|
||||
.filter(|p| !p.as_os_str().is_empty())
|
||||
.unwrap_or(Path::new("."));
|
||||
std::fs::create_dir_all(parent).context("create state directory")?;
|
||||
let temp = parent.join(format!(
|
||||
".mirror-state-{}-{:016x}.tmp",
|
||||
std::process::id(),
|
||||
rand::random::<u64>()
|
||||
));
|
||||
let result = (|| {
|
||||
let mut options = std::fs::OpenOptions::new();
|
||||
options.create_new(true).write(true);
|
||||
#[cfg(unix)]
|
||||
{
|
||||
use std::os::unix::fs::OpenOptionsExt;
|
||||
options.mode(0o600);
|
||||
}
|
||||
let mut file = options.open(&temp).context("create state temporary file")?;
|
||||
let bytes = serde_json::to_vec_pretty(state)?;
|
||||
file.write_all(&bytes).context("write state")?;
|
||||
file.sync_all().context("sync state")?;
|
||||
std::fs::rename(&temp, path).context("commit state")?;
|
||||
// Directory sync is a durability enhancement; rename is already committed.
|
||||
// Do not report rollback-worthy failure after replacement is visible.
|
||||
if let Ok(dir) = std::fs::File::open(parent) {
|
||||
let _ = dir.sync_all();
|
||||
}
|
||||
Ok(())
|
||||
})();
|
||||
if result.is_err() {
|
||||
let _ = std::fs::remove_file(temp);
|
||||
}
|
||||
result
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
#[test]
|
||||
fn atomic_roundtrip_and_invalid_state_rejected() {
|
||||
let temp = tempfile::tempdir().unwrap();
|
||||
let mut cfg = Config::default();
|
||||
cfg.web.state_file = temp.path().join("state.json");
|
||||
let state = Topology::from_config(&cfg);
|
||||
save(&cfg.web.state_file, &state).unwrap();
|
||||
assert_eq!(Topology::load(&cfg).unwrap().version, 1);
|
||||
std::fs::write(
|
||||
&cfg.web.state_file,
|
||||
b"{\"version\":2,\"teams\":[],\"services\":[]}",
|
||||
)
|
||||
.unwrap();
|
||||
assert!(Topology::load(&cfg).is_err());
|
||||
}
|
||||
}
|
||||
220
src/web.rs
Normal file
220
src/web.rs
Normal file
@ -0,0 +1,220 @@
|
||||
use crate::{
|
||||
auth::{Auth, LoginError, SessionInfo, same_origin},
|
||||
config::{ServiceConfig, Team},
|
||||
control::Controller,
|
||||
};
|
||||
use axum::{
|
||||
Extension, Json, Router,
|
||||
extract::{ConnectInfo, DefaultBodyLimit, Path, Request, State, rejection::JsonRejection},
|
||||
http::{HeaderMap, HeaderValue, Method, StatusCode, header},
|
||||
middleware::{self, Next},
|
||||
response::{
|
||||
Html, IntoResponse, Response, Sse,
|
||||
sse::{Event, KeepAlive},
|
||||
},
|
||||
routing::{get, post, put},
|
||||
};
|
||||
use futures_util::stream;
|
||||
use serde::Deserialize;
|
||||
use serde_json::json;
|
||||
use std::{
|
||||
convert::Infallible,
|
||||
net::{IpAddr, Ipv4Addr, SocketAddr},
|
||||
sync::Arc,
|
||||
time::Duration,
|
||||
};
|
||||
use tokio_util::sync::CancellationToken;
|
||||
|
||||
#[derive(Clone)]
|
||||
struct WebState {
|
||||
controller: Arc<Controller>,
|
||||
auth: Arc<Auth>,
|
||||
shutdown: CancellationToken,
|
||||
}
|
||||
|
||||
pub fn router(controller: Arc<Controller>, auth: Auth, shutdown: CancellationToken) -> Router {
|
||||
let state = WebState {
|
||||
controller,
|
||||
auth: Arc::new(auth),
|
||||
shutdown,
|
||||
};
|
||||
let private = Router::new()
|
||||
.route("/api/session", get(session))
|
||||
.route("/api/logout", post(logout))
|
||||
.route("/api/snapshot", get(snapshot))
|
||||
.route("/api/events", get(events))
|
||||
.route(
|
||||
"/api/services/{id}",
|
||||
put(put_service).delete(delete_service),
|
||||
)
|
||||
.route("/api/teams/{id}", put(put_team))
|
||||
.route_layer(middleware::from_fn_with_state(state.clone(), guard));
|
||||
Router::new()
|
||||
.route("/", get(index))
|
||||
.route("/app.js", get(js))
|
||||
.route("/app.css", get(css))
|
||||
.route("/healthz", get(|| async { Json(json!({"status":"ok"})) }))
|
||||
.route("/api/login", post(login))
|
||||
.merge(private)
|
||||
.fallback(|| async { error(StatusCode::NOT_FOUND, "not found") })
|
||||
.layer(DefaultBodyLimit::max(64 * 1024))
|
||||
.layer(middleware::from_fn(security_headers))
|
||||
.with_state(state)
|
||||
}
|
||||
async fn index() -> Html<&'static str> {
|
||||
Html(include_str!("../web/index.html"))
|
||||
}
|
||||
async fn js() -> impl IntoResponse {
|
||||
(
|
||||
[(header::CONTENT_TYPE, "text/javascript; charset=utf-8")],
|
||||
include_str!("../web/app.js"),
|
||||
)
|
||||
}
|
||||
async fn css() -> impl IntoResponse {
|
||||
(
|
||||
[(header::CONTENT_TYPE, "text/css; charset=utf-8")],
|
||||
include_str!("../web/app.css"),
|
||||
)
|
||||
}
|
||||
fn error(status: StatusCode, message: impl ToString) -> Response {
|
||||
(status, Json(json!({"error":message.to_string()}))).into_response()
|
||||
}
|
||||
async fn security_headers(request: Request, next: Next) -> Response {
|
||||
let mut response = next.run(request).await;
|
||||
let h = response.headers_mut();
|
||||
h.insert(header::CACHE_CONTROL, HeaderValue::from_static("no-store"));
|
||||
h.insert(
|
||||
"x-content-type-options",
|
||||
HeaderValue::from_static("nosniff"),
|
||||
);
|
||||
h.insert("referrer-policy", HeaderValue::from_static("no-referrer"));
|
||||
h.insert("content-security-policy",HeaderValue::from_static("default-src 'self'; script-src 'self'; style-src 'self'; connect-src 'self'; img-src 'self' data:; base-uri 'none'; frame-ancestors 'none'; form-action 'self'"));
|
||||
response
|
||||
}
|
||||
async fn guard(State(state): State<WebState>, request: Request, next: Next) -> Response {
|
||||
let Some(session) = state.auth.session(request.headers()) else {
|
||||
return error(StatusCode::UNAUTHORIZED, "session expired; sign in again");
|
||||
};
|
||||
if !matches!(
|
||||
*request.method(),
|
||||
Method::GET | Method::HEAD | Method::OPTIONS
|
||||
) {
|
||||
if !same_origin(request.headers()) {
|
||||
return error(StatusCode::FORBIDDEN, "cross-origin request rejected");
|
||||
}
|
||||
if !state.auth.csrf_valid(request.headers(), &session) {
|
||||
return error(StatusCode::FORBIDDEN, "missing or invalid CSRF token");
|
||||
}
|
||||
}
|
||||
next.run(request).await
|
||||
}
|
||||
#[derive(Deserialize)]
|
||||
#[serde(deny_unknown_fields)]
|
||||
struct Login {
|
||||
password: String,
|
||||
}
|
||||
async fn login(
|
||||
State(state): State<WebState>,
|
||||
headers: HeaderMap,
|
||||
peer: Option<Extension<ConnectInfo<SocketAddr>>>,
|
||||
payload: Result<Json<Login>, JsonRejection>,
|
||||
) -> Response {
|
||||
if !same_origin(&headers) {
|
||||
return error(StatusCode::FORBIDDEN, "cross-origin request rejected");
|
||||
}
|
||||
let Ok(Json(login)) = payload else {
|
||||
return error(StatusCode::BAD_REQUEST, "invalid login JSON");
|
||||
};
|
||||
let address = peer.map_or(IpAddr::V4(Ipv4Addr::LOCALHOST), |p| p.0.0.ip());
|
||||
match state.auth.login(&login.password, address) {
|
||||
Ok((cookie, info)) => {
|
||||
// A successful re-login rotates and revokes the previous browser session.
|
||||
state.auth.logout(&headers);
|
||||
([(header::SET_COOKIE, cookie)], Json(info)).into_response()
|
||||
}
|
||||
Err(LoginError::Invalid) => error(StatusCode::UNAUTHORIZED, "Неверный пароль"),
|
||||
Err(LoginError::Limited) => error(
|
||||
StatusCode::TOO_MANY_REQUESTS,
|
||||
"Слишком много попыток; повторите через 30 секунд",
|
||||
),
|
||||
}
|
||||
}
|
||||
async fn session(State(state): State<WebState>, headers: HeaderMap) -> Response {
|
||||
match state.auth.session(&headers) {
|
||||
Some(s) => Json(SessionInfo {
|
||||
authenticated: true,
|
||||
csrf: s.csrf,
|
||||
})
|
||||
.into_response(),
|
||||
None => error(StatusCode::UNAUTHORIZED, "session expired"),
|
||||
}
|
||||
}
|
||||
async fn logout(State(state): State<WebState>, headers: HeaderMap) -> Response {
|
||||
(
|
||||
[(header::SET_COOKIE, state.auth.logout(&headers))],
|
||||
Json(json!({"ok":true})),
|
||||
)
|
||||
.into_response()
|
||||
}
|
||||
async fn snapshot(State(state): State<WebState>) -> Json<crate::control::Snapshot> {
|
||||
Json(state.controller.snapshot())
|
||||
}
|
||||
async fn put_service(
|
||||
State(state): State<WebState>,
|
||||
Path(id): Path<String>,
|
||||
payload: Result<Json<ServiceConfig>, JsonRejection>,
|
||||
) -> Response {
|
||||
let Ok(Json(service)) = payload else {
|
||||
return error(StatusCode::BAD_REQUEST, "invalid service JSON");
|
||||
};
|
||||
if id != service.id {
|
||||
return error(StatusCode::BAD_REQUEST, "path id does not match service id");
|
||||
}
|
||||
match state.controller.put_service(service).await {
|
||||
Ok(()) => Json(json!({"ok":true})).into_response(),
|
||||
Err(e) => error(StatusCode::BAD_REQUEST, format!("{e:#}")),
|
||||
}
|
||||
}
|
||||
async fn delete_service(State(state): State<WebState>, Path(id): Path<String>) -> Response {
|
||||
match state.controller.delete_service(&id).await {
|
||||
Ok(()) => Json(json!({"ok":true})).into_response(),
|
||||
Err(e) => error(StatusCode::BAD_REQUEST, format!("{e:#}")),
|
||||
}
|
||||
}
|
||||
async fn put_team(
|
||||
State(state): State<WebState>,
|
||||
Path(id): Path<String>,
|
||||
payload: Result<Json<Team>, JsonRejection>,
|
||||
) -> Response {
|
||||
let Ok(Json(team)) = payload else {
|
||||
return error(StatusCode::BAD_REQUEST, "invalid team JSON");
|
||||
};
|
||||
if id != team.id {
|
||||
return error(StatusCode::BAD_REQUEST, "path id does not match team id");
|
||||
}
|
||||
match state.controller.put_team(team).await {
|
||||
Ok(()) => Json(json!({"ok":true})).into_response(),
|
||||
Err(e) => error(StatusCode::BAD_REQUEST, format!("{e:#}")),
|
||||
}
|
||||
}
|
||||
async fn events(State(state): State<WebState>, headers: HeaderMap) -> impl IntoResponse {
|
||||
let mut interval = tokio::time::interval(Duration::from_secs(1));
|
||||
interval.set_missed_tick_behavior(tokio::time::MissedTickBehavior::Skip);
|
||||
let stream = stream::unfold(
|
||||
(state, headers, interval),
|
||||
|(state, headers, mut interval)| async move {
|
||||
tokio::select! {_=state.shutdown.cancelled()=>return None,_=interval.tick()=>{}}
|
||||
state.auth.session(&headers)?;
|
||||
let event = Event::default()
|
||||
.event("snapshot")
|
||||
.json_data(state.controller.snapshot())
|
||||
.ok()?;
|
||||
Some((Ok::<_, Infallible>(event), (state, headers, interval)))
|
||||
},
|
||||
);
|
||||
Sse::new(stream).keep_alive(
|
||||
KeepAlive::new()
|
||||
.interval(Duration::from_secs(15))
|
||||
.text("keepalive"),
|
||||
)
|
||||
}
|
||||
26
task_plan.md
Normal file
26
task_plan.md
Normal file
@ -0,0 +1,26 @@
|
||||
# Rust rewrite plan
|
||||
|
||||
## Acceptance
|
||||
One native executable with an async TCP engine and embedded web UI. No Python/runtime frontend build required for operation. Typed TOML configuration, atomic durable operator state, authenticated same-origin controls, reusable scoreboard connection, bounded networking resources, graceful shutdown, explicit replica delivery policies. Local functional/integration checks and Linux tests pass; benchmark results are measured and reproducible.
|
||||
|
||||
## Phases
|
||||
1. [complete] Capture legacy behavior and verify ForcAD wire format; establish module contracts.
|
||||
2. [complete] Implement configuration, async forwarding and lifecycle/metrics (11 engine tests passed).
|
||||
3. [in progress] Implement scoreboard clients, health monitoring and deterministic failover; wire-level fixtures pending.
|
||||
4. [in progress] Implement HTTP API, authentication, persistence and embedded UI (UI implementer active).
|
||||
5. [pending] Integration and failure tests; Linux/amd64 verification and throughput/latency benchmark.
|
||||
6. [pending] Independent review, fixes, deployment docs and evidence-backed handoff.
|
||||
|
||||
## Global constraints
|
||||
- User means upstream routing, not host firewall interception.
|
||||
- Never drop chunks from a live replica silently. Strict mode uses bounded backpressure; isolation mode disconnects a lagging replica explicitly and counts/reports it. Availability and lossless mirroring cannot both be guaranteed for an indefinitely slower/unavailable replica.
|
||||
- Existing sessions keep their selected target. Do not replay partially sent requests on another primary.
|
||||
- Preserve TCP half-close; no inherited 3-second idle timeout.
|
||||
- All fixtures use loopback/test addresses, not old configured hosts.
|
||||
- Do not store passwords, keys, tokens, session values or flags in source or logs.
|
||||
- No mandatory external UI CDNs. Single embedded frontend, accessible keyboard controls.
|
||||
|
||||
## Errors/research corrections
|
||||
- Initial CTF skill path under `.agents` was wrong; correct installed path is `.codex/skills/ctf-attack-defense`.
|
||||
- Guessed C4T-BuT-S4D ForcAD repository does not exist; discover canonical upstream before implementing its protocol.
|
||||
- Integrated build briefly blocked by not-yet-created UI assets while UI implementer was working; not a Rust source failure.
|
||||
271
tests/control_plane.rs
Normal file
271
tests/control_plane.rs
Normal file
@ -0,0 +1,271 @@
|
||||
use mirror_tool::{
|
||||
auth::Auth,
|
||||
config::{Config, ReplicaPolicy, ServiceConfig, Team},
|
||||
control::Controller,
|
||||
web,
|
||||
};
|
||||
use serde_json::{Value, json};
|
||||
use std::{net::SocketAddr, sync::Arc};
|
||||
use tokio::{
|
||||
io::{AsyncReadExt, AsyncWriteExt},
|
||||
net::TcpListener,
|
||||
};
|
||||
use tokio_util::sync::CancellationToken;
|
||||
|
||||
async fn fixture() -> (
|
||||
Arc<Controller>,
|
||||
String,
|
||||
CancellationToken,
|
||||
tempfile::TempDir,
|
||||
tokio::task::JoinHandle<()>,
|
||||
) {
|
||||
let temp = tempfile::tempdir().unwrap();
|
||||
let mut config = Config::default();
|
||||
config.web.state_file = temp.path().join("state.json");
|
||||
let controller = Controller::start(config.clone()).await.unwrap();
|
||||
let listener = TcpListener::bind("127.0.0.1:0").await.unwrap();
|
||||
let addr = listener.local_addr().unwrap();
|
||||
let cancel = CancellationToken::new();
|
||||
let router = web::router(
|
||||
controller.clone(),
|
||||
Auth::new(config.web, "fixture-passphrase"),
|
||||
cancel.clone(),
|
||||
);
|
||||
let stop = cancel.clone();
|
||||
let task = tokio::spawn(async move {
|
||||
axum::serve(
|
||||
listener,
|
||||
router.into_make_service_with_connect_info::<SocketAddr>(),
|
||||
)
|
||||
.with_graceful_shutdown(stop.cancelled_owned())
|
||||
.await
|
||||
.unwrap();
|
||||
});
|
||||
(controller, format!("http://{addr}"), cancel, temp, task)
|
||||
}
|
||||
async fn login(client: &reqwest::Client, url: &str) -> (String, String) {
|
||||
let response = client
|
||||
.post(format!("{url}/api/login"))
|
||||
.json(&json!({"password":"fixture-passphrase"}))
|
||||
.send()
|
||||
.await
|
||||
.unwrap();
|
||||
assert_eq!(response.status(), 200);
|
||||
let cookie = response.headers()["set-cookie"]
|
||||
.to_str()
|
||||
.unwrap()
|
||||
.split(';')
|
||||
.next()
|
||||
.unwrap()
|
||||
.to_owned();
|
||||
let body: Value = response.json().await.unwrap();
|
||||
(cookie, body["csrf"].as_str().unwrap().into())
|
||||
}
|
||||
#[tokio::test]
|
||||
async fn authenticated_api_csrf_persistence_and_sse() {
|
||||
let (controller, url, cancel, temp, task) = fixture().await;
|
||||
let client = reqwest::Client::new();
|
||||
assert_eq!(
|
||||
client
|
||||
.get(format!("{url}/api/snapshot"))
|
||||
.send()
|
||||
.await
|
||||
.unwrap()
|
||||
.status(),
|
||||
401
|
||||
);
|
||||
assert_eq!(
|
||||
client
|
||||
.post(format!("{url}/api/login"))
|
||||
.header("origin", "http://other.example")
|
||||
.json(&json!({"password":"fixture-passphrase"}))
|
||||
.send()
|
||||
.await
|
||||
.unwrap()
|
||||
.status(),
|
||||
403
|
||||
);
|
||||
let (cookie, csrf) = login(&client, &url).await;
|
||||
let team = json!({"id":"local","name":"Local","address":"127.0.0.1"});
|
||||
assert_eq!(
|
||||
client
|
||||
.put(format!("{url}/api/teams/local"))
|
||||
.header("cookie", &cookie)
|
||||
.json(&team)
|
||||
.send()
|
||||
.await
|
||||
.unwrap()
|
||||
.status(),
|
||||
403
|
||||
);
|
||||
assert_eq!(
|
||||
client
|
||||
.put(format!("{url}/api/teams/local"))
|
||||
.header("cookie", &cookie)
|
||||
.header("x-csrf-token", &csrf)
|
||||
.json(&team)
|
||||
.send()
|
||||
.await
|
||||
.unwrap()
|
||||
.status(),
|
||||
200
|
||||
);
|
||||
let service = json!({"id":"notes","name":"Notes","listen":"127.0.0.1:18081","upstream_port":18082,"primary":"local","replicas":[],"enabled":false,"auto_switch":true,"replica_policy":"isolate"});
|
||||
assert_eq!(
|
||||
client
|
||||
.put(format!("{url}/api/services/notes"))
|
||||
.header("cookie", &cookie)
|
||||
.header("x-csrf-token", &csrf)
|
||||
.json(&service)
|
||||
.send()
|
||||
.await
|
||||
.unwrap()
|
||||
.status(),
|
||||
200
|
||||
);
|
||||
let saved = std::fs::read_to_string(temp.path().join("state.json")).unwrap();
|
||||
assert!(saved.contains("notes"));
|
||||
assert!(!saved.contains("fixture-passphrase"));
|
||||
assert!(!saved.contains(&csrf));
|
||||
let snapshot: Value = client
|
||||
.get(format!("{url}/api/snapshot"))
|
||||
.header("cookie", &cookie)
|
||||
.send()
|
||||
.await
|
||||
.unwrap()
|
||||
.json()
|
||||
.await
|
||||
.unwrap();
|
||||
assert_eq!(snapshot["services"][0]["id"], "notes");
|
||||
let mut events = client
|
||||
.get(format!("{url}/api/events"))
|
||||
.header("cookie", &cookie)
|
||||
.send()
|
||||
.await
|
||||
.unwrap();
|
||||
let first = events.chunk().await.unwrap().unwrap();
|
||||
assert!(
|
||||
std::str::from_utf8(&first)
|
||||
.unwrap()
|
||||
.contains("event: snapshot")
|
||||
);
|
||||
drop(events);
|
||||
assert_eq!(
|
||||
client
|
||||
.post(format!("{url}/api/logout"))
|
||||
.header("cookie", &cookie)
|
||||
.header("x-csrf-token", &csrf)
|
||||
.send()
|
||||
.await
|
||||
.unwrap()
|
||||
.status(),
|
||||
200
|
||||
);
|
||||
assert_eq!(
|
||||
client
|
||||
.get(format!("{url}/api/session"))
|
||||
.header("cookie", &cookie)
|
||||
.send()
|
||||
.await
|
||||
.unwrap()
|
||||
.status(),
|
||||
401
|
||||
);
|
||||
cancel.cancel();
|
||||
task.await.unwrap();
|
||||
controller.engine.shutdown().await;
|
||||
let restarted = Controller::start(controller.config.clone()).await.unwrap();
|
||||
assert_eq!(restarted.topology().services[0].id, "notes");
|
||||
restarted.engine.shutdown().await;
|
||||
}
|
||||
#[tokio::test]
|
||||
async fn state_failure_rolls_back_listener_and_topology() {
|
||||
let temp = tempfile::tempdir().unwrap();
|
||||
let bad_parent = temp.path().join("not-directory");
|
||||
std::fs::write(&bad_parent, "x").unwrap();
|
||||
let mut config = Config::default();
|
||||
config.web.state_file = bad_parent.join("state.json");
|
||||
let controller = Controller::start(config).await.unwrap();
|
||||
let team = Team {
|
||||
id: "a".into(),
|
||||
name: "A".into(),
|
||||
address: "127.0.0.1".parse().unwrap(),
|
||||
};
|
||||
assert!(controller.put_team(team).await.is_err());
|
||||
assert!(controller.topology().teams.is_empty());
|
||||
controller.engine.shutdown().await;
|
||||
}
|
||||
#[tokio::test]
|
||||
async fn automatic_failover_is_durable_and_primary_requests_keep_working() {
|
||||
let temp = tempfile::tempdir().unwrap();
|
||||
let primary = TcpListener::bind("127.0.0.1:0").await.unwrap();
|
||||
let port = primary.local_addr().unwrap().port();
|
||||
drop(primary);
|
||||
let replica = TcpListener::bind((std::net::Ipv6Addr::LOCALHOST, port))
|
||||
.await
|
||||
.unwrap();
|
||||
let upstream = tokio::spawn(async move {
|
||||
loop {
|
||||
let (mut s, _) = replica.accept().await.unwrap();
|
||||
tokio::spawn(async move {
|
||||
let mut data = vec![];
|
||||
s.read_to_end(&mut data).await.unwrap();
|
||||
s.write_all(&data).await.unwrap();
|
||||
});
|
||||
}
|
||||
});
|
||||
let reserve = TcpListener::bind("127.0.0.1:0").await.unwrap();
|
||||
let listen = reserve.local_addr().unwrap();
|
||||
drop(reserve);
|
||||
let mut config = Config::default();
|
||||
config.web.state_file = temp.path().join("state.json");
|
||||
config.monitoring.cooldown_ms = 0;
|
||||
config.monitoring.timeout_ms = 100;
|
||||
config.teams = vec![
|
||||
Team {
|
||||
id: "a".into(),
|
||||
name: "A".into(),
|
||||
address: "127.0.0.1".parse().unwrap(),
|
||||
},
|
||||
Team {
|
||||
id: "b".into(),
|
||||
name: "B".into(),
|
||||
address: "::1".parse().unwrap(),
|
||||
},
|
||||
];
|
||||
config.services = vec![ServiceConfig {
|
||||
id: "notes".into(),
|
||||
name: "Notes".into(),
|
||||
listen,
|
||||
upstream_port: port,
|
||||
primary: Some("a".into()),
|
||||
replicas: vec!["b".into()],
|
||||
enabled: true,
|
||||
auto_switch: true,
|
||||
replica_policy: ReplicaPolicy::Isolate,
|
||||
}];
|
||||
let controller = Controller::start(config.clone()).await.unwrap();
|
||||
controller.check_cycle().await;
|
||||
controller.check_cycle().await;
|
||||
assert_eq!(
|
||||
controller.topology().services[0].primary.as_deref(),
|
||||
Some("b")
|
||||
);
|
||||
let mut client = tokio::net::TcpStream::connect(listen).await.unwrap();
|
||||
client.write_all(b"hello failover").await.unwrap();
|
||||
client.shutdown().await.unwrap();
|
||||
let mut bytes = vec![];
|
||||
client.read_to_end(&mut bytes).await.unwrap();
|
||||
assert_eq!(bytes, b"hello failover");
|
||||
assert_eq!(
|
||||
mirror_tool::store::Topology::load(&config)
|
||||
.unwrap()
|
||||
.services[0]
|
||||
.primary
|
||||
.as_deref(),
|
||||
Some("b")
|
||||
);
|
||||
controller.engine.shutdown().await;
|
||||
upstream.abort();
|
||||
let _ = upstream.await;
|
||||
}
|
||||
122
tests/forcad_transport.rs
Normal file
122
tests/forcad_transport.rs
Normal file
@ -0,0 +1,122 @@
|
||||
use axum::{
|
||||
Router,
|
||||
extract::{Query, State},
|
||||
routing::get,
|
||||
};
|
||||
use mirror_tool::{
|
||||
config::{ScoreboardConfig, ScoreboardKind},
|
||||
scoreboard::Scoreboard,
|
||||
};
|
||||
use serde_json::json;
|
||||
use std::{
|
||||
collections::HashMap,
|
||||
sync::{
|
||||
Arc,
|
||||
atomic::{AtomicUsize, Ordering::SeqCst},
|
||||
},
|
||||
time::Duration,
|
||||
};
|
||||
use tokio::sync::Mutex;
|
||||
use tokio_util::sync::CancellationToken;
|
||||
|
||||
#[derive(Default)]
|
||||
struct Mock {
|
||||
handshakes: AtomicUsize,
|
||||
polls: AtomicUsize,
|
||||
posts: Mutex<Vec<String>>,
|
||||
}
|
||||
async fn poll(
|
||||
State(mock): State<Arc<Mock>>,
|
||||
Query(query): Query<HashMap<String, String>>,
|
||||
) -> String {
|
||||
assert_eq!(query.get("EIO").map(String::as_str), Some("4"));
|
||||
assert_eq!(query.get("transport").map(String::as_str), Some("polling"));
|
||||
if !query.contains_key("sid") {
|
||||
let number = mock.handshakes.fetch_add(1, SeqCst) + 1;
|
||||
return format!(
|
||||
"0{}",
|
||||
json!({"sid":format!("test_{number}"),"pingInterval":1000,"pingTimeout":1000,"upgrades":[],"maxPayload":1000000})
|
||||
);
|
||||
}
|
||||
let call = mock.polls.fetch_add(1, SeqCst);
|
||||
if call == 0 {
|
||||
format!(
|
||||
"40/game_events,{{\"sid\":\"namespace-test\"}}\u{1e}42/game_events,{}",
|
||||
json!(["init_scoreboard",{"data":{"teams":[{"id":1,"name":"A"}],"tasks":[{"id":2,"name":"Notes"}],"state":{"round":1,"team_tasks":[{"team_id":1,"task_id":2,"status":101}]}}}])
|
||||
)
|
||||
} else if call == 1 {
|
||||
"2heartbeat".into()
|
||||
} else if call == 2 {
|
||||
format!(
|
||||
"42/game_events,{}",
|
||||
json!(["update_scoreboard",{"data":{"round":2,"team_tasks":[{"team_id":1,"task_id":2,"status":104,"stolen":7}]}}])
|
||||
)
|
||||
} else if call == 3 {
|
||||
tokio::time::sleep(Duration::from_millis(200)).await;
|
||||
"1".into()
|
||||
} else if query.get("sid").is_some_and(|s| s == "test_2") {
|
||||
tokio::time::sleep(Duration::from_millis(50)).await;
|
||||
format!(
|
||||
"40/game_events,{{}}\u{1e}42/game_events,{}",
|
||||
json!(["init_scoreboard",{"data":{"teams":[{"id":1,"name":"A"}],"tasks":[{"id":2,"name":"Notes"}],"state":{"round":3,"team_tasks":[{"team_id":1,"task_id":2,"status":101}]}}}])
|
||||
)
|
||||
} else {
|
||||
tokio::time::sleep(Duration::from_millis(50)).await;
|
||||
"2".into()
|
||||
}
|
||||
}
|
||||
async fn post(State(mock): State<Arc<Mock>>, body: String) -> &'static str {
|
||||
mock.posts.lock().await.push(body);
|
||||
"ok"
|
||||
}
|
||||
#[tokio::test]
|
||||
async fn actual_polling_handshake_ping_delta_and_reconnect() {
|
||||
let mock = Arc::new(Mock::default());
|
||||
let router = Router::new()
|
||||
.route("/socket.io/", get(poll).post(post))
|
||||
.with_state(mock.clone());
|
||||
let listener = tokio::net::TcpListener::bind("127.0.0.1:0").await.unwrap();
|
||||
let addr = listener.local_addr().unwrap();
|
||||
let server = tokio::spawn(async move {
|
||||
axum::serve(listener, router).await.unwrap();
|
||||
});
|
||||
let cfg = ScoreboardConfig {
|
||||
kind: ScoreboardKind::Forcad,
|
||||
url: format!("http://{addr}"),
|
||||
poll_interval_ms: 100,
|
||||
stale_after_ms: 2000,
|
||||
};
|
||||
let board = Scoreboard::new(Some(&cfg));
|
||||
let cancel = CancellationToken::new();
|
||||
let worker = tokio::spawn(board.clone().run(cfg, cancel.clone()));
|
||||
tokio::time::timeout(Duration::from_secs(3), async {
|
||||
loop {
|
||||
if board.snapshot().round == 2 {
|
||||
break;
|
||||
}
|
||||
tokio::time::sleep(Duration::from_millis(5)).await;
|
||||
}
|
||||
})
|
||||
.await
|
||||
.unwrap();
|
||||
let delta = board.snapshot();
|
||||
assert_eq!(delta.teams[0].name, "A");
|
||||
assert_eq!(delta.entries["2"]["1"].stolen, 7);
|
||||
assert!(mock.posts.lock().await.iter().any(|s| s == "3heartbeat"));
|
||||
tokio::time::timeout(Duration::from_secs(4), async {
|
||||
loop {
|
||||
if board.snapshot().round == 3 {
|
||||
break;
|
||||
}
|
||||
tokio::time::sleep(Duration::from_millis(10)).await;
|
||||
}
|
||||
})
|
||||
.await
|
||||
.unwrap();
|
||||
assert_eq!(mock.handshakes.load(SeqCst), 2);
|
||||
cancel.cancel();
|
||||
worker.await.unwrap();
|
||||
assert!(!board.snapshot().connected);
|
||||
server.abort();
|
||||
let _ = server.await;
|
||||
}
|
||||
423
tests/forwarding.rs
Normal file
423
tests/forwarding.rs
Normal file
@ -0,0 +1,423 @@
|
||||
use mirror_tool::{
|
||||
config::{NetworkConfig, ReplicaPolicy, ServiceConfig, Team},
|
||||
engine::Engine,
|
||||
};
|
||||
use std::{
|
||||
net::{IpAddr, Ipv4Addr, SocketAddr},
|
||||
time::Duration,
|
||||
};
|
||||
use tokio::{
|
||||
io::{AsyncReadExt, AsyncWriteExt},
|
||||
net::{TcpListener, TcpStream},
|
||||
task::JoinHandle,
|
||||
time::{sleep, timeout},
|
||||
};
|
||||
|
||||
fn settings() -> NetworkConfig {
|
||||
NetworkConfig {
|
||||
max_buffered_bytes: 65536,
|
||||
connect_timeout_ms: 200,
|
||||
buffer_size: 4096,
|
||||
replica_queue_chunks: 4,
|
||||
max_connections: 32,
|
||||
drain_timeout_ms: 100,
|
||||
}
|
||||
}
|
||||
async fn free_address() -> SocketAddr {
|
||||
TcpListener::bind("127.0.0.1:0")
|
||||
.await
|
||||
.unwrap()
|
||||
.local_addr()
|
||||
.unwrap()
|
||||
}
|
||||
fn service(listen: SocketAddr, upstream: SocketAddr) -> ServiceConfig {
|
||||
ServiceConfig {
|
||||
id: "service".into(),
|
||||
name: "Test".into(),
|
||||
listen,
|
||||
upstream_port: upstream.port(),
|
||||
primary: Some("primary".into()),
|
||||
replicas: vec![],
|
||||
enabled: true,
|
||||
auto_switch: false,
|
||||
replica_policy: ReplicaPolicy::Isolate,
|
||||
}
|
||||
}
|
||||
fn teams() -> Vec<Team> {
|
||||
vec![Team {
|
||||
id: "primary".into(),
|
||||
name: "Primary".into(),
|
||||
address: IpAddr::V4(Ipv4Addr::LOCALHOST),
|
||||
}]
|
||||
}
|
||||
async fn echo() -> (SocketAddr, JoinHandle<()>) {
|
||||
let listener = TcpListener::bind("127.0.0.1:0").await.unwrap();
|
||||
let address = listener.local_addr().unwrap();
|
||||
let task = tokio::spawn(async move {
|
||||
loop {
|
||||
let (mut stream, _) = listener.accept().await.unwrap();
|
||||
tokio::spawn(async move {
|
||||
let mut buffer = [0; 4096];
|
||||
loop {
|
||||
let n = stream.read(&mut buffer).await.unwrap_or(0);
|
||||
if n == 0 {
|
||||
let _ = stream.shutdown().await;
|
||||
break;
|
||||
}
|
||||
if stream.write_all(&buffer[..n]).await.is_err() {
|
||||
break;
|
||||
}
|
||||
}
|
||||
});
|
||||
}
|
||||
});
|
||||
(address, task)
|
||||
}
|
||||
async fn exchange(address: SocketAddr, payload: &[u8]) -> Vec<u8> {
|
||||
let stream = TcpStream::connect(address).await.unwrap();
|
||||
let (mut reader, mut writer) = stream.into_split();
|
||||
let upload = async {
|
||||
writer.write_all(payload).await.unwrap();
|
||||
writer.shutdown().await.unwrap();
|
||||
};
|
||||
let download = async {
|
||||
let mut response = vec![];
|
||||
reader.read_to_end(&mut response).await.unwrap();
|
||||
response
|
||||
};
|
||||
let (_, response) = timeout(Duration::from_secs(5), async {
|
||||
tokio::join!(upload, download)
|
||||
})
|
||||
.await
|
||||
.unwrap();
|
||||
response
|
||||
}
|
||||
async fn settled(engine: &Engine) {
|
||||
timeout(Duration::from_secs(2), async {
|
||||
loop {
|
||||
if engine.snapshots()["service"].stats.active == 0 {
|
||||
break;
|
||||
}
|
||||
sleep(Duration::from_millis(5)).await;
|
||||
}
|
||||
})
|
||||
.await
|
||||
.unwrap();
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn bulk_bidirectional_bytes_and_half_close() {
|
||||
let (upstream, server) = echo().await;
|
||||
let listen = free_address().await;
|
||||
let engine = Engine::new(settings());
|
||||
engine
|
||||
.upsert(&service(listen, upstream), &teams())
|
||||
.await
|
||||
.unwrap();
|
||||
let payload: Vec<u8> = (0..2_000_000).map(|n| (n % 251) as u8).collect();
|
||||
assert_eq!(exchange(listen, &payload).await, payload);
|
||||
settled(&engine).await;
|
||||
let stats = engine.snapshots()["service"].stats.clone();
|
||||
assert_eq!(stats.bytes_up, payload.len() as u64);
|
||||
assert_eq!(stats.bytes_down, payload.len() as u64);
|
||||
assert_eq!(stats.completed, 1);
|
||||
engine.shutdown().await;
|
||||
server.abort();
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn response_after_request_eof_and_idle_over_three_seconds() {
|
||||
let listener = TcpListener::bind("127.0.0.1:0").await.unwrap();
|
||||
let upstream = listener.local_addr().unwrap();
|
||||
let server = tokio::spawn(async move {
|
||||
let (mut stream, _) = listener.accept().await.unwrap();
|
||||
let mut request = vec![];
|
||||
stream.read_to_end(&mut request).await.unwrap();
|
||||
assert_eq!(request, b"after idle");
|
||||
stream.write_all(b"after eof").await.unwrap();
|
||||
});
|
||||
let listen = free_address().await;
|
||||
let engine = Engine::new(settings());
|
||||
engine
|
||||
.upsert(&service(listen, upstream), &teams())
|
||||
.await
|
||||
.unwrap();
|
||||
let mut stream = TcpStream::connect(listen).await.unwrap();
|
||||
sleep(Duration::from_millis(3200)).await;
|
||||
stream.write_all(b"after idle").await.unwrap();
|
||||
stream.shutdown().await.unwrap();
|
||||
let mut reply = vec![];
|
||||
timeout(Duration::from_secs(2), stream.read_to_end(&mut reply))
|
||||
.await
|
||||
.unwrap()
|
||||
.unwrap();
|
||||
assert_eq!(reply, b"after eof");
|
||||
server.await.unwrap();
|
||||
engine.shutdown().await;
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn retarget_only_new_sessions_and_reenable_without_port_leak() {
|
||||
let (a, server_a) = echo().await;
|
||||
let listener = TcpListener::bind("127.0.0.1:0").await.unwrap();
|
||||
let b = listener.local_addr().unwrap();
|
||||
let server_b = tokio::spawn(async move {
|
||||
let (mut stream, _) = listener.accept().await.unwrap();
|
||||
let mut request = vec![];
|
||||
stream.read_to_end(&mut request).await.unwrap();
|
||||
stream.write_all(b"new target").await.unwrap();
|
||||
});
|
||||
let listen = free_address().await;
|
||||
let engine = Engine::new(settings());
|
||||
let mut config = service(listen, a);
|
||||
engine.upsert(&config, &teams()).await.unwrap();
|
||||
let mut old = TcpStream::connect(listen).await.unwrap();
|
||||
old.write_all(b"before").await.unwrap();
|
||||
let mut before = [0; 6];
|
||||
old.read_exact(&mut before).await.unwrap();
|
||||
config.upstream_port = b.port();
|
||||
engine.upsert(&config, &teams()).await.unwrap();
|
||||
old.write_all(b"old target").await.unwrap();
|
||||
old.shutdown().await.unwrap();
|
||||
let mut reply = vec![];
|
||||
old.read_to_end(&mut reply).await.unwrap();
|
||||
assert_eq!(reply, b"old target");
|
||||
assert_eq!(exchange(listen, b"request").await, b"new target");
|
||||
config.enabled = false;
|
||||
engine.upsert(&config, &teams()).await.unwrap();
|
||||
assert!(!engine.snapshots()["service"].running);
|
||||
assert!(TcpStream::connect(listen).await.is_err());
|
||||
config.enabled = true;
|
||||
config.upstream_port = a.port();
|
||||
engine.upsert(&config, &teams()).await.unwrap();
|
||||
assert_eq!(exchange(listen, b"enabled").await, b"enabled");
|
||||
engine.shutdown().await;
|
||||
server_a.abort();
|
||||
server_b.await.unwrap();
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn bind_error_is_reported_and_working_listener_preserved() {
|
||||
let (upstream, server) = echo().await;
|
||||
let listen = free_address().await;
|
||||
let occupied = TcpListener::bind("127.0.0.1:0").await.unwrap();
|
||||
let engine = Engine::new(settings());
|
||||
let mut config = service(listen, upstream);
|
||||
engine.upsert(&config, &teams()).await.unwrap();
|
||||
config.listen = occupied.local_addr().unwrap();
|
||||
assert!(engine.upsert(&config, &teams()).await.is_err());
|
||||
assert_eq!(exchange(listen, b"still running").await, b"still running");
|
||||
engine.shutdown().await;
|
||||
server.abort();
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn process_connection_limit_sheds_and_shutdown_cancels_active_sessions() {
|
||||
let (upstream, server) = echo().await;
|
||||
let listen = free_address().await;
|
||||
let engine = Engine::new(NetworkConfig {
|
||||
max_connections: 1,
|
||||
..settings()
|
||||
});
|
||||
engine
|
||||
.upsert(&service(listen, upstream), &teams())
|
||||
.await
|
||||
.unwrap();
|
||||
let mut first = TcpStream::connect(listen).await.unwrap();
|
||||
first.write_all(b"a").await.unwrap();
|
||||
let mut byte = [0];
|
||||
first.read_exact(&mut byte).await.unwrap();
|
||||
let second_listen = free_address().await;
|
||||
let mut second_config = service(second_listen, upstream);
|
||||
second_config.id = "second".into();
|
||||
engine.upsert(&second_config, &teams()).await.unwrap();
|
||||
let mut rejected = TcpStream::connect(second_listen).await.unwrap();
|
||||
assert_eq!(
|
||||
timeout(Duration::from_secs(1), rejected.read(&mut byte))
|
||||
.await
|
||||
.unwrap()
|
||||
.unwrap(),
|
||||
0
|
||||
);
|
||||
assert_eq!(engine.snapshots()["second"].stats.rejected, 1);
|
||||
timeout(Duration::from_secs(1), engine.shutdown())
|
||||
.await
|
||||
.unwrap();
|
||||
assert_eq!(engine.snapshots()["service"].stats.active, 0);
|
||||
assert!(!engine.snapshots()["service"].running);
|
||||
assert_eq!(
|
||||
timeout(Duration::from_secs(1), first.read(&mut byte))
|
||||
.await
|
||||
.unwrap()
|
||||
.unwrap(),
|
||||
0
|
||||
);
|
||||
server.abort();
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn disable_keeps_existing_connection_and_remove_joins_it() {
|
||||
let (upstream, server) = echo().await;
|
||||
let listen = free_address().await;
|
||||
let engine = Engine::new(settings());
|
||||
let mut config = service(listen, upstream);
|
||||
engine.upsert(&config, &teams()).await.unwrap();
|
||||
let mut stream = TcpStream::connect(listen).await.unwrap();
|
||||
stream.write_all(b"a").await.unwrap();
|
||||
let mut byte = [0];
|
||||
stream.read_exact(&mut byte).await.unwrap();
|
||||
config.enabled = false;
|
||||
engine.upsert(&config, &teams()).await.unwrap();
|
||||
stream.write_all(b"b").await.unwrap();
|
||||
stream.read_exact(&mut byte).await.unwrap();
|
||||
assert_eq!(&byte, b"b");
|
||||
timeout(Duration::from_secs(1), engine.remove("service"))
|
||||
.await
|
||||
.unwrap()
|
||||
.unwrap();
|
||||
assert!(engine.snapshots().is_empty());
|
||||
assert_eq!(stream.read(&mut byte).await.unwrap(), 0);
|
||||
engine.shutdown().await;
|
||||
server.abort();
|
||||
}
|
||||
|
||||
// Distinct loopback addresses let primary and replica share the configured upstream port.
|
||||
async fn replica_fixture(
|
||||
policy: ReplicaPolicy,
|
||||
) -> (Engine, SocketAddr, TcpListener, JoinHandle<()>) {
|
||||
let (upstream, server) = echo().await;
|
||||
let replica = TcpListener::bind(SocketAddr::new("::1".parse().unwrap(), upstream.port()))
|
||||
.await
|
||||
.unwrap();
|
||||
let listen = free_address().await;
|
||||
let mut config = service(listen, upstream);
|
||||
config.replica_policy = policy;
|
||||
config.replicas = vec!["replica".into()];
|
||||
let mut teams = teams();
|
||||
teams.push(Team {
|
||||
id: "replica".into(),
|
||||
name: "Replica".into(),
|
||||
address: "::1".parse().unwrap(),
|
||||
});
|
||||
let engine = Engine::new(settings());
|
||||
engine.upsert(&config, &teams).await.unwrap();
|
||||
(engine, listen, replica, server)
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn strict_mirror_exact_bytes_and_drains_large_replica_replies() {
|
||||
let (engine, listen, replica, server) = replica_fixture(ReplicaPolicy::Strict).await;
|
||||
let payload: Vec<u8> = (0..1_000_000).map(|n| (n % 251) as u8).collect();
|
||||
let expected = payload.clone();
|
||||
let worker = tokio::spawn(async move {
|
||||
let (stream, _) = replica.accept().await.unwrap();
|
||||
let (mut reader, mut writer) = stream.into_split();
|
||||
let read = async {
|
||||
let mut bytes = vec![];
|
||||
reader.read_to_end(&mut bytes).await.unwrap();
|
||||
assert_eq!(bytes, expected);
|
||||
};
|
||||
let write = async {
|
||||
writer.write_all(&vec![17; 2_000_000]).await.unwrap();
|
||||
writer.shutdown().await.unwrap();
|
||||
};
|
||||
tokio::join!(read, write);
|
||||
});
|
||||
assert_eq!(exchange(listen, &payload).await, payload);
|
||||
timeout(Duration::from_secs(3), worker)
|
||||
.await
|
||||
.unwrap()
|
||||
.unwrap();
|
||||
settled(&engine).await;
|
||||
assert_eq!(
|
||||
engine.snapshots()["service"].stats.replica_bytes,
|
||||
payload.len() as u64
|
||||
);
|
||||
assert_eq!(engine.snapshots()["service"].stats.replica_disconnects, 0);
|
||||
engine.shutdown().await;
|
||||
server.abort();
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn isolate_unavailable_replica_preserves_primary_and_counts_failure() {
|
||||
let (engine, listen, replica, server) = replica_fixture(ReplicaPolicy::Isolate).await;
|
||||
drop(replica);
|
||||
// Give the worker time to fail before EOF rather than overflowing its bounded queue.
|
||||
let mut client = TcpStream::connect(listen).await.unwrap();
|
||||
client.write_all(b"one").await.unwrap();
|
||||
let mut bytes = [0; 3];
|
||||
client.read_exact(&mut bytes).await.unwrap();
|
||||
sleep(Duration::from_millis(50)).await;
|
||||
client.write_all(b"two").await.unwrap();
|
||||
client.shutdown().await.unwrap();
|
||||
let mut reply = vec![];
|
||||
client.read_to_end(&mut reply).await.unwrap();
|
||||
assert_eq!(reply, b"two");
|
||||
settled(&engine).await;
|
||||
let stats = engine.snapshots()["service"].stats.clone();
|
||||
assert_eq!(stats.replica_errors, 1);
|
||||
assert_eq!(stats.replica_disconnects, 1);
|
||||
assert_eq!(stats.io_errors, 0);
|
||||
engine.shutdown().await;
|
||||
server.abort();
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn strict_unavailable_replica_ends_session_before_primary_receives_payload() {
|
||||
let (engine, listen, replica, server) = replica_fixture(ReplicaPolicy::Strict).await;
|
||||
drop(replica);
|
||||
let mut client = TcpStream::connect(listen).await.unwrap();
|
||||
let _ = client.write_all(b"must not forward").await;
|
||||
let mut bytes = vec![];
|
||||
let _ = timeout(Duration::from_secs(1), client.read_to_end(&mut bytes))
|
||||
.await
|
||||
.unwrap();
|
||||
settled(&engine).await;
|
||||
let stats = engine.snapshots()["service"].stats.clone();
|
||||
assert_eq!(stats.bytes_up, 0);
|
||||
assert_eq!(stats.io_errors, 1);
|
||||
assert_eq!(stats.replica_errors, 1);
|
||||
engine.shutdown().await;
|
||||
server.abort();
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn isolate_slow_replica_disconnects_instead_of_corrupting_primary_stream() {
|
||||
let (engine, listen, replica, server) = replica_fixture(ReplicaPolicy::Isolate).await;
|
||||
let slow = tokio::spawn(async move {
|
||||
let (_stream, _) = replica.accept().await.unwrap();
|
||||
sleep(Duration::from_secs(20)).await;
|
||||
});
|
||||
let payload = vec![93; 16_000_000];
|
||||
assert_eq!(exchange(listen, &payload).await, payload);
|
||||
settled(&engine).await;
|
||||
assert!(engine.snapshots()["service"].stats.replica_disconnects >= 1);
|
||||
engine.shutdown().await;
|
||||
server.abort();
|
||||
slow.abort();
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn strict_slow_replica_backpressures_and_shutdown_joins_workers() {
|
||||
let (engine, listen, replica, server) = replica_fixture(ReplicaPolicy::Strict).await;
|
||||
let slow = tokio::spawn(async move {
|
||||
let (_stream, _) = replica.accept().await.unwrap();
|
||||
sleep(Duration::from_secs(20)).await;
|
||||
});
|
||||
let mut client = TcpStream::connect(listen).await.unwrap();
|
||||
let payload = vec![31; 32_000_000];
|
||||
assert!(
|
||||
timeout(Duration::from_millis(200), client.write_all(&payload))
|
||||
.await
|
||||
.is_err()
|
||||
);
|
||||
let stats = engine.snapshots()["service"].stats.clone();
|
||||
assert!(stats.bytes_up < payload.len() as u64);
|
||||
assert_eq!(stats.replica_disconnects, 0);
|
||||
timeout(Duration::from_secs(1), engine.shutdown())
|
||||
.await
|
||||
.unwrap();
|
||||
assert_eq!(engine.snapshots()["service"].stats.active, 0);
|
||||
server.abort();
|
||||
slow.abort();
|
||||
}
|
||||
4
web/app.css
Normal file
4
web/app.css
Normal file
File diff suppressed because one or more lines are too long
339
web/app.js
Normal file
339
web/app.js
Normal file
@ -0,0 +1,339 @@
|
||||
"use strict";
|
||||
|
||||
(() => {
|
||||
const $ = (id) => document.getElementById(id);
|
||||
const state = { csrf: "", snapshot: null, stream: null, busy: false, cards: new Map(), baselines: new Map(), rates: new Map(), stale: true, sessionTimer: null, staleTimer: null, sessionCheck: false, retryDelay: 2000, editId: null, deleteId: null, generation: 0 };
|
||||
const configKeys = ["id", "name", "listen", "upstream_port", "primary", "replicas", "enabled", "auto_switch", "replica_policy"];
|
||||
const statusNames = { 101: "UP", 102: "CORRUPT", 103: "MUMBLE", 104: "DOWN", 110: "CHECK FAIL" };
|
||||
function el(tag, className, text) {
|
||||
const node = document.createElement(tag);
|
||||
if (className) node.className = className;
|
||||
if (text !== undefined) node.textContent = text;
|
||||
return node;
|
||||
}
|
||||
function message(id, text) { $(id).textContent = text || ""; $(id).hidden = !text; }
|
||||
function config(service) { return Object.fromEntries(configKeys.map((key) => [key, key === "replicas" ? [...service[key]] : service[key]])); }
|
||||
function serviceById(id) { return state.snapshot?.services.find((service) => service.id === id); }
|
||||
function field(form, name) { return form.elements.namedItem(name); }
|
||||
function time(ms) { return ms ? new Date(ms).toLocaleTimeString("ru-RU") : "—"; }
|
||||
function bytes(value) {
|
||||
const count = Math.max(0, Number(value) || 0);
|
||||
const units = ["B", "KiB", "MiB", "GiB", "TiB"];
|
||||
const unit = count ? Math.min(Math.floor(Math.log(count) / Math.log(1024)), units.length - 1) : 0;
|
||||
return `${(count / 1024 ** unit).toLocaleString("ru-RU", { maximumFractionDigits: unit ? 1 : 0 })} ${units[unit]}`;
|
||||
}
|
||||
function setBusy(busy) {
|
||||
state.busy = busy;
|
||||
document.querySelectorAll("[data-write]").forEach((node) => { node.disabled = busy; });
|
||||
document.querySelectorAll("dialog form input, dialog form select, dialog [data-close]").forEach((node) => { node.disabled = busy || (node.name === "id" && Boolean(state.editId) && node.closest("#service-form") !== null); });
|
||||
if (state.snapshot) render(state.snapshot, false, !busy);
|
||||
}
|
||||
async function api(path, options = {}) {
|
||||
const headers = { ...options.headers };
|
||||
if (options.body !== undefined) headers["Content-Type"] = "application/json";
|
||||
if (options.method && options.method !== "GET" && state.csrf) headers["X-CSRF-Token"] = state.csrf;
|
||||
let response;
|
||||
try { response = await fetch(path, { ...options, headers, credentials: "same-origin", cache: "no-store" }); }
|
||||
catch (error) { throw new Error(`Ошибка сети: ${error.message}`); }
|
||||
const body = await response.json().catch(() => ({}));
|
||||
if (!response.ok) {
|
||||
const error = new Error(body.error || `Ошибка HTTP ${response.status}`);
|
||||
error.status = response.status;
|
||||
if (response.status === 401 && path !== "/api/login" && path !== "/api/session") showLogin("Сессия завершена. Войдите снова.");
|
||||
throw error;
|
||||
}
|
||||
return body;
|
||||
}
|
||||
function closeStream() {
|
||||
if (state.stream) state.stream.close();
|
||||
state.stream = null;
|
||||
clearTimeout(state.sessionTimer); clearTimeout(state.staleTimer);
|
||||
state.sessionTimer = null;
|
||||
}
|
||||
function showLogin(reason = "Введите пароль для входа.") {
|
||||
state.generation += 1;
|
||||
closeStream(); state.csrf = ""; state.stale = true;
|
||||
document.querySelectorAll("dialog[open]").forEach((dialog) => dialog.close());
|
||||
$("dashboard").hidden = true; $("login-view").hidden = false;
|
||||
$("login-status").textContent = reason;
|
||||
$("password").value = "";
|
||||
}
|
||||
function markStale(detail) {
|
||||
state.stale = true;
|
||||
$("stream-state").textContent = detail || "Нет свежих данных";
|
||||
$("stream-state").className = "warn";
|
||||
if (state.snapshot) render(state.snapshot, false);
|
||||
}
|
||||
function scheduleSessionCheck() {
|
||||
if (state.sessionTimer || state.sessionCheck || !state.csrf) return;
|
||||
state.sessionTimer = setTimeout(async () => {
|
||||
state.sessionTimer = null; state.sessionCheck = true;
|
||||
const generation = state.generation;
|
||||
try {
|
||||
const session = await api("/api/session");
|
||||
if (generation !== state.generation) return;
|
||||
state.csrf = session.csrf;
|
||||
state.retryDelay = Math.min(state.retryDelay * 2, 30000);
|
||||
} catch (error) {
|
||||
if (generation !== state.generation) return;
|
||||
if (error.status === 401) showLogin("Сессия завершена. Войдите снова.");
|
||||
else { markStale("Нет связи с сервером"); message("global-error", error.message); state.retryDelay = Math.min(state.retryDelay * 2, 30000); }
|
||||
} finally {
|
||||
state.sessionCheck = false;
|
||||
if (state.stale && state.csrf) scheduleSessionCheck();
|
||||
}
|
||||
}, state.retryDelay);
|
||||
}
|
||||
function watchFreshness() {
|
||||
clearTimeout(state.staleTimer);
|
||||
state.staleTimer = setTimeout(() => { markStale("Снимок устарел · > 8 с"); scheduleSessionCheck(); }, 8000);
|
||||
}
|
||||
function startStream() {
|
||||
closeStream();
|
||||
const stream = new EventSource("/api/events");
|
||||
state.stream = stream;
|
||||
stream.addEventListener("snapshot", (event) => {
|
||||
if (stream !== state.stream) return;
|
||||
try {
|
||||
const snapshot = JSON.parse(event.data);
|
||||
state.stale = false; state.retryDelay = 2000;
|
||||
clearTimeout(state.sessionTimer); state.sessionTimer = null;
|
||||
$("stream-state").textContent = "В реальном времени"; $("stream-state").className = "good";
|
||||
render(snapshot); watchFreshness();
|
||||
} catch { markStale("Некорректный снимок"); scheduleSessionCheck(); }
|
||||
});
|
||||
stream.onerror = () => { if (stream === state.stream) { markStale("Связь прервана · переподключение"); scheduleSessionCheck(); } };
|
||||
watchFreshness();
|
||||
}
|
||||
async function refresh() {
|
||||
const generation = state.generation;
|
||||
const snapshot = await api("/api/snapshot");
|
||||
if (generation === state.generation && state.csrf) render(snapshot);
|
||||
}
|
||||
async function enter(session) {
|
||||
state.generation += 1; state.csrf = session.csrf; state.stale = true;
|
||||
state.snapshot = null; state.baselines.clear(); state.rates.clear();
|
||||
state.cards.forEach((card) => card.root.remove()); state.cards.clear();
|
||||
message("login-error", ""); message("global-error", ""); $("password").value = "";
|
||||
$("login-view").hidden = true; $("dashboard").hidden = false;
|
||||
markStale("Подключение…");
|
||||
try { await refresh(); if (state.csrf) startStream(); }
|
||||
catch (error) { message("global-error", error.message); if (state.csrf) startStream(); }
|
||||
}
|
||||
async function mutate(path, method, body, errorId = "global-error") {
|
||||
if (state.busy) return false;
|
||||
setBusy(true); message(errorId, ""); $("action-status").textContent = "Сохраняем…";
|
||||
try {
|
||||
await api(path, { method, ...(body === undefined ? {} : { body: JSON.stringify(body) }) });
|
||||
await refresh();
|
||||
$("action-status").textContent = "Сохранено";
|
||||
return true;
|
||||
} catch (error) { message(errorId, error.message); $("action-status").textContent = "Изменение не подтверждено. Проверьте состояние и повторите."; return false; }
|
||||
finally { setBusy(false); }
|
||||
}
|
||||
async function updateService(id, change) {
|
||||
const service = serviceById(id);
|
||||
if (!service || state.busy) return;
|
||||
const next = { ...config(service), ...change };
|
||||
await mutate(`/api/services/${encodeURIComponent(id)}`, "PUT", next);
|
||||
}
|
||||
function control(type, label, onChange) {
|
||||
const input = el("input"); input.type = type; input.setAttribute("aria-label", label); input.dataset.write = "";
|
||||
input.addEventListener("change", onChange); return input;
|
||||
}
|
||||
function button(text, className, onClick) {
|
||||
const node = el("button", className, text); node.type = "button"; node.dataset.write = "";
|
||||
node.addEventListener("click", onClick); return node;
|
||||
}
|
||||
function createCard(id) {
|
||||
const root = el("article", "service-card");
|
||||
const header = el("header", "service-header"); const heading = el("div", "service-heading");
|
||||
const title = el("h2", "service-name"); const running = el("span", "service-state"); heading.append(title, running);
|
||||
const endpoint = el("p", "endpoint"); const route = el("p", "route-summary");
|
||||
const actions = el("div", "card-actions");
|
||||
const toggle = button("Запустить", "small accent", () => { const service = serviceById(id); if (service) updateService(id, { enabled: !service.enabled }); });
|
||||
const autoLabel = el("label", "auto-label");
|
||||
const auto = control("checkbox", "Автопереключение", () => updateService(id, { auto_switch: auto.checked })); autoLabel.append(auto, "Авто");
|
||||
const policyLabel = el("label", "policy-label"); const policy = el("select"); policy.dataset.write = ""; policy.setAttribute("aria-label", "Политика реплик");
|
||||
for (const [value, text] of [["isolate", "Isolate · отключать"], ["strict", "Strict · ждать"]]) { const option = el("option", "", text); option.value = value; policy.append(option); }
|
||||
policy.addEventListener("change", () => updateService(id, { replica_policy: policy.value })); policyLabel.append(policy);
|
||||
const edit = button("Настроить", "small", () => openService(id));
|
||||
actions.append(toggle, autoLabel, policyLabel, edit); header.append(heading, endpoint, route, actions);
|
||||
const stats = el("dl", "stats-grid"); const values = {};
|
||||
for (const [key, text] of [["active", "Активные"], ["accepted", "Принято"], ["completed", "Завершено"], ["errors", "Ошибки"], ["up", "↑ Основной / с"], ["down", "↓ Ответ / с"], ["replica_disconnects", "Откл. реплик"], ["replica_errors", "Ошибки реплик"]]) {
|
||||
const item = el("div", "stat"); values[key] = el("dd", "", "—"); item.append(el("dt", "", text), values[key]); stats.append(item);
|
||||
}
|
||||
const totals = el("div", "bytes-line"); const table = el("table", "team-table");
|
||||
const caption = el("caption", "sr-only", "Маршруты команд");
|
||||
const head = el("thead"); const headRow = el("tr");
|
||||
for (const [text, className] of [["Осн.", "control-col"], ["Реп.", "control-col"], ["Команда", ""], ["TCP", "health-col"], ["ForcAD", "score-col"]]) { const th = el("th", className, text); th.scope = "col"; headRow.append(th); }
|
||||
head.append(headRow); const tbody = el("tbody"); table.append(caption, head, tbody);
|
||||
const error = el("p", "service-error error-text"); error.hidden = true;
|
||||
const footer = el("footer", "card-footer"); const policyNote = el("span");
|
||||
const remove = button("Удалить", "text-button", () => openDelete(id)); footer.append(policyNote, remove);
|
||||
root.append(header, stats, totals, table, error, footer); $("service-board").append(root);
|
||||
return { root, title, running, endpoint, route, toggle, auto, policy, edit, values, totals, tbody, error, remove, policyNote, rows: new Map() };
|
||||
}
|
||||
function createRow(card, serviceId, team) {
|
||||
const root = el("tr"); const mainCell = el("td", "control-cell"); const replicaCell = el("td", "control-cell");
|
||||
const primary = control("radio", "", () => { const service = serviceById(serviceId); if (service) updateService(serviceId, { primary: team.id, replicas: service.replicas.filter((id) => id !== team.id) }); });
|
||||
primary.name = `primary-${serviceId}`;
|
||||
const replica = control("checkbox", "", () => { const service = serviceById(serviceId); if (service) updateService(serviceId, { replicas: replica.checked ? [...new Set([...service.replicas, team.id])] : service.replicas.filter((id) => id !== team.id) }); });
|
||||
mainCell.append(primary); replicaCell.append(replica);
|
||||
const meta = el("td"); const name = el("span", "team-name"); const address = el("span", "team-address"); meta.append(name, address);
|
||||
const healthCell = el("td"); const health = el("span", "health-badge"); healthCell.append(health);
|
||||
const scoreCell = el("td"); const score = el("span", "score-status"); const flags = el("span", "score-flags"); scoreCell.append(score, flags);
|
||||
root.append(mainCell, replicaCell, meta, healthCell, scoreCell); card.tbody.append(root);
|
||||
return { root, primary, replica, name, address, health, score, flags };
|
||||
}
|
||||
function trackRates(snapshot) {
|
||||
for (const service of snapshot.services) {
|
||||
const stats = service.engine?.stats || {}; const previous = state.baselines.get(service.id);
|
||||
const dt = previous ? snapshot.now_ms - previous.at : 0;
|
||||
if (previous && dt > 0) {
|
||||
const reset = stats.bytes_up < previous.up || stats.bytes_down < previous.down;
|
||||
state.rates.set(service.id, reset ? null : { up: (stats.bytes_up - previous.up) * 1000 / dt, down: (stats.bytes_down - previous.down) * 1000 / dt });
|
||||
} else if (!previous || dt < 0) state.rates.set(service.id, null);
|
||||
if (!previous || dt !== 0) state.baselines.set(service.id, { at: snapshot.now_ms, up: stats.bytes_up || 0, down: stats.bytes_down || 0 });
|
||||
}
|
||||
}
|
||||
function render(snapshot, fresh = true, reconcile = false) {
|
||||
if (fresh) {
|
||||
if (!Array.isArray(snapshot.services) || !Array.isArray(snapshot.teams) || !Number.isFinite(snapshot.now_ms)) throw new Error("Некорректный снимок сервера");
|
||||
if (state.snapshot && snapshot.now_ms < state.snapshot.now_ms) return;
|
||||
trackRates(snapshot); state.snapshot = snapshot;
|
||||
}
|
||||
const scoreboard = snapshot.scoreboard || {};
|
||||
const scoreStale = state.stale || scoreboard.stale || !scoreboard.connected;
|
||||
$("snapshot-time").textContent = `${time(snapshot.now_ms)}${state.stale ? " · устарел" : ""}`;
|
||||
$("service-count").textContent = `${snapshot.services.filter((service) => service.engine?.running).length} / ${snapshot.services.length} работают`;
|
||||
$("scoreboard-state").textContent = !scoreboard.enabled ? "Выключен" : scoreStale ? "Устаревшие данные" : `Раунд ${scoreboard.round ?? "—"}`;
|
||||
$("scoreboard-state").className = !scoreboard.enabled ? "muted" : scoreStale ? "warn" : "good";
|
||||
$("scoreboard-detail").textContent = scoreboard.enabled ? `${scoreboard.connected ? "Подключён" : "Нет подключения"} · обновлён ${time(scoreboard.updated_at_ms)}${scoreboard.last_error ? ` · ${scoreboard.last_error}` : ""}` : "ForcAD выключен. TCP-доступность проверяется отдельно от состояния чекера.";
|
||||
$("empty-state").hidden = snapshot.services.length > 0;
|
||||
const ids = new Set(snapshot.services.map((service) => service.id));
|
||||
for (const [id, card] of state.cards) if (!ids.has(id)) { card.root.remove(); state.cards.delete(id); state.baselines.delete(id); state.rates.delete(id); }
|
||||
for (const service of snapshot.services) {
|
||||
let card = state.cards.get(service.id);
|
||||
if (!card) { card = createCard(service.id); state.cards.set(service.id, card); }
|
||||
const engine = service.engine || {}; const stats = engine.stats || {}; const rate = state.rates.get(service.id);
|
||||
card.root.classList.toggle("running", Boolean(engine.running)); card.title.textContent = service.name;
|
||||
card.running.textContent = engine.running ? "● РАБОТАЕТ" : service.enabled ? "● ОШИБКА" : "○ ОСТАНОВЛЕН";
|
||||
card.running.className = `service-state ${engine.running ? "good" : service.enabled ? "bad" : "muted"}`;
|
||||
card.endpoint.textContent = `${service.listen} → :${service.upstream_port}`;
|
||||
const primary = snapshot.teams.find((team) => team.id === service.primary);
|
||||
card.route.textContent = `${primary ? primary.name : "Основной маршрут не выбран"} · реплик: ${service.replicas.length}`;
|
||||
card.toggle.textContent = service.enabled ? "Остановить" : "Запустить";
|
||||
card.toggle.disabled = state.busy || (!service.enabled && !service.primary);
|
||||
card.auto.disabled = state.busy; card.policy.disabled = state.busy; card.edit.disabled = state.busy;
|
||||
card.auto.setAttribute("aria-label", `Автопереключение: ${service.name}`); card.policy.setAttribute("aria-label", `Политика реплик: ${service.name}`);
|
||||
if (reconcile || document.activeElement !== card.auto) card.auto.checked = service.auto_switch;
|
||||
if (reconcile || document.activeElement !== card.policy) card.policy.value = service.replica_policy;
|
||||
card.remove.disabled = state.busy || service.enabled || engine.running;
|
||||
card.remove.title = service.enabled || engine.running ? "Сначала остановите сервис" : "Удалить настройки сервиса";
|
||||
card.policyNote.textContent = service.replica_policy === "strict" ? "Strict: общий поток ждёт реплики" : "Isolate: отстающая реплика отключается";
|
||||
for (const key of ["active", "accepted", "completed", "replica_disconnects", "replica_errors"]) card.values[key].textContent = (stats[key] || 0).toLocaleString("ru-RU");
|
||||
card.values.errors.textContent = ((stats.connect_errors || 0) + (stats.io_errors || 0) + (stats.rejected || 0)).toLocaleString("ru-RU");
|
||||
card.values.up.textContent = !state.stale && rate ? `${bytes(rate.up)}/с` : "—";
|
||||
card.values.down.textContent = !state.stale && rate ? `${bytes(rate.down)}/с` : "—";
|
||||
card.totals.textContent = `Всего: ↑ ${bytes(stats.bytes_up)} · ↓ ${bytes(stats.bytes_down)} · реплики ${bytes(stats.replica_bytes)}`;
|
||||
card.error.textContent = engine.last_error || ""; card.error.hidden = !engine.last_error;
|
||||
const teamIds = new Set(snapshot.teams.map((team) => team.id));
|
||||
for (const [id, row] of card.rows) if (!teamIds.has(id)) { row.root.remove(); card.rows.delete(id); }
|
||||
for (const team of snapshot.teams) {
|
||||
let row = card.rows.get(team.id);
|
||||
if (!row) { row = createRow(card, service.id, team); card.rows.set(team.id, row); }
|
||||
row.name.textContent = team.name; row.address.textContent = team.address;
|
||||
row.primary.setAttribute("aria-label", `Основной маршрут ${team.name}: ${service.name}`);
|
||||
row.replica.setAttribute("aria-label", `Реплика ${team.name}: ${service.name}`);
|
||||
if (reconcile || document.activeElement !== row.primary) row.primary.checked = service.primary === team.id;
|
||||
if (reconcile || document.activeElement !== row.replica) row.replica.checked = service.replicas.includes(team.id);
|
||||
row.primary.disabled = state.busy; row.replica.disabled = state.busy || service.primary === team.id;
|
||||
row.root.classList.toggle("is-primary", service.primary === team.id); row.root.classList.toggle("is-replica", service.replicas.includes(team.id));
|
||||
const health = service.health?.[team.id];
|
||||
const unknown = !health?.checked_at_ms;
|
||||
const healthStale = state.stale || (!unknown && snapshot.now_ms - health.checked_at_ms > 15000);
|
||||
row.health.textContent = unknown ? "?" : healthStale ? "Устар." : health.reachable ? "UP" : "DOWN";
|
||||
row.health.className = `health-badge ${unknown ? "muted" : healthStale ? "warn" : health.reachable ? "good" : "bad"}`;
|
||||
row.health.title = unknown ? "Ещё не проверено" : `TCP: ${health.reachable ? "доступен" : "недоступен"}; стабильно здоров: ${health.healthy ? "да" : "нет"}; ${health.latency_ms ?? "—"} мс; ${time(health.checked_at_ms)}`;
|
||||
const entry = service.scoreboard?.[team.id];
|
||||
row.score.textContent = entry ? `${statusNames[entry.status] || entry.status}${scoreStale ? " *" : ""}` : "—";
|
||||
row.score.className = `score-status ${scoreStale ? "stale-data" : entry?.status === 101 ? "good" : entry ? "bad" : "muted"}`;
|
||||
row.score.title = scoreStale ? "Состояние чекера устарело" : "Состояние чекера ForcAD";
|
||||
row.flags.textContent = entry ? `↑${entry.stolen ?? 0} ↓${entry.lost ?? 0}` : "↑— ↓—";
|
||||
row.flags.title = "Украдено / потеряно";
|
||||
}
|
||||
}
|
||||
if (fresh) {
|
||||
const events = [...(snapshot.events || [])].slice(-30).reverse();
|
||||
$("event-log").replaceChildren(...events.map((event) => { const li = el("li"); const stamp = el("time", "", time(event.at_ms)); li.append(stamp, document.createTextNode(`${event.service ? `${event.service}: ` : ""}${event.message}`)); return li; }));
|
||||
}
|
||||
}
|
||||
function openService(id = null) {
|
||||
const service = id ? serviceById(id) : null;
|
||||
if (state.busy || (id && !service)) return;
|
||||
state.editId = id;
|
||||
const form = $("service-form"); form.reset(); message("service-form-error", "");
|
||||
$("service-dialog-title").textContent = service ? "Настроить сервис" : "Добавить сервис";
|
||||
for (const key of ["id", "name", "listen", "upstream_port", "replica_policy"]) field(form, key).value = service?.[key] ?? ({ listen: "0.0.0.0:18080", upstream_port: 8080, replica_policy: "isolate" }[key] || "");
|
||||
field(form, "id").disabled = Boolean(id);
|
||||
field(form, "enabled").checked = service?.enabled || false; field(form, "auto_switch").checked = service?.auto_switch || false;
|
||||
const primary = field(form, "primary"); primary.replaceChildren();
|
||||
const none = el("option", "", "Не выбран (сервис остановлен)"); none.value = ""; primary.append(none);
|
||||
$("dialog-replicas").replaceChildren();
|
||||
for (const team of state.snapshot.teams) {
|
||||
const option = el("option", "", `${team.name} · ${team.address}`); option.value = team.id; primary.append(option);
|
||||
const label = el("label"); const input = el("input"); input.type = "checkbox"; input.name = "replicas"; input.value = team.id; input.checked = service?.replicas.includes(team.id) || false;
|
||||
label.append(input, document.createTextNode(team.name)); $("dialog-replicas").append(label);
|
||||
}
|
||||
primary.value = service?.primary || "";
|
||||
syncDialogReplicas(); $("service-dialog").showModal();
|
||||
}
|
||||
function syncDialogReplicas() {
|
||||
const primary = field($("service-form"), "primary").value;
|
||||
$("dialog-replicas").querySelectorAll("input").forEach((input) => { input.disabled = input.value === primary; if (input.disabled) input.checked = false; });
|
||||
}
|
||||
function openDelete(id) {
|
||||
const service = serviceById(id); if (!service || service.enabled || state.busy) return;
|
||||
state.deleteId = id; message("delete-error", "");
|
||||
$("delete-detail").textContent = `«${service.name}» (${service.listen}), основной маршрут ${service.primary || "не выбран"}, реплик: ${service.replicas.length}.`;
|
||||
$("delete-dialog").showModal();
|
||||
}
|
||||
$("login-form").addEventListener("submit", async (event) => {
|
||||
event.preventDefault(); $("login-button").disabled = true; message("login-error", "");
|
||||
try { await enter(await api("/api/login", { method: "POST", body: JSON.stringify({ password: $("password").value }) })); }
|
||||
catch (error) { message("login-error", error.message); }
|
||||
finally { $("login-button").disabled = false; }
|
||||
});
|
||||
$("logout").addEventListener("click", async () => { if (state.busy) return; setBusy(true); try { await api("/api/logout", { method: "POST" }); showLogin(); } catch (error) { message("global-error", error.message); } finally { setBusy(false); } });
|
||||
$("refresh").addEventListener("click", async () => { $("refresh").disabled = true; try { await refresh(); message("global-error", ""); } catch (error) { message("global-error", error.message); } finally { $("refresh").disabled = false; } });
|
||||
$("add-service").addEventListener("click", () => openService());
|
||||
$("add-team").addEventListener("click", () => { $("team-form").reset(); message("team-form-error", ""); $("team-dialog").showModal(); });
|
||||
document.querySelectorAll("[data-close]").forEach((node) => node.addEventListener("click", () => { if (!state.busy) $(node.dataset.close).close(); }));
|
||||
document.querySelectorAll("dialog").forEach((dialog) => dialog.addEventListener("cancel", (event) => { if (state.busy) event.preventDefault(); }));
|
||||
field($("service-form"), "primary").addEventListener("change", syncDialogReplicas);
|
||||
$("service-form").addEventListener("submit", async (event) => {
|
||||
event.preventDefault(); const form = event.currentTarget;
|
||||
const id = field(form, "id").value.trim();
|
||||
if (!state.editId && serviceById(id)) { message("service-form-error", "Сервис с таким ID уже существует. Откройте его настройки."); return; }
|
||||
const body = { id, name: field(form, "name").value.trim(), listen: field(form, "listen").value.trim(), upstream_port: Number(field(form, "upstream_port").value), primary: field(form, "primary").value || null, replicas: [...form.querySelectorAll("input[name=replicas]:checked")].map((input) => input.value), enabled: field(form, "enabled").checked, auto_switch: field(form, "auto_switch").checked, replica_policy: field(form, "replica_policy").value };
|
||||
if (body.enabled && !body.primary) { message("service-form-error", "Выберите основной маршрут перед запуском."); return; }
|
||||
if (await mutate(`/api/services/${encodeURIComponent(id)}`, "PUT", body, "service-form-error")) $("service-dialog").close();
|
||||
syncDialogReplicas();
|
||||
});
|
||||
$("team-form").addEventListener("submit", async (event) => {
|
||||
event.preventDefault(); const form = event.currentTarget;
|
||||
const body = Object.fromEntries(["id", "name", "address"].map((key) => [key, field(form, key).value.trim()]));
|
||||
const existing = state.snapshot.teams.find((team) => team.id === body.id);
|
||||
if (existing && existing.address !== body.address) { message("team-form-error", "У этого ID уже другой адрес. Выберите новый ID."); return; }
|
||||
if (await mutate(`/api/teams/${encodeURIComponent(body.id)}`, "PUT", body, "team-form-error")) $("team-dialog").close();
|
||||
});
|
||||
$("delete-form").addEventListener("submit", async (event) => {
|
||||
event.preventDefault(); const service = serviceById(state.deleteId);
|
||||
if (!service || service.enabled) { message("delete-error", "Сервис отсутствует или уже запущен. Закройте окно и проверьте состояние."); return; }
|
||||
if (await mutate(`/api/services/${encodeURIComponent(state.deleteId)}`, "DELETE", undefined, "delete-error")) $("delete-dialog").close();
|
||||
});
|
||||
window.addEventListener("pagehide", closeStream);
|
||||
window.addEventListener("pageshow", (event) => { if (event.persisted && state.csrf) { markStale("Переподключение…"); startStream(); } });
|
||||
(async () => { try { await enter(await api("/api/session")); } catch (error) { showLogin(error.status === 401 ? "Введите пароль для входа." : "Не удалось проверить сессию. Можно повторить вход."); if (error.status !== 401) message("login-error", error.message); } })();
|
||||
})();
|
||||
88
web/index.html
Normal file
88
web/index.html
Normal file
@ -0,0 +1,88 @@
|
||||
<!doctype html>
|
||||
<html lang="ru">
|
||||
<head>
|
||||
<meta charset="utf-8">
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1">
|
||||
<meta name="color-scheme" content="dark">
|
||||
<title>Mirror Tool — операторская</title>
|
||||
<link rel="stylesheet" href="/app.css">
|
||||
<script src="/app.js" defer></script>
|
||||
</head>
|
||||
<body>
|
||||
<div id="login-view" class="login-wrap">
|
||||
<main class="login-panel">
|
||||
<p class="eyebrow">TCP / ОПЕРАТОРСКАЯ</p>
|
||||
<h1>Mirror Tool<span class="brand-mark" aria-hidden="true">↗↗</span></h1>
|
||||
<p class="muted">Маршруты, реплики и состояние сервисов.</p>
|
||||
<form id="login-form">
|
||||
<label for="password">Пароль оператора</label>
|
||||
<input id="password" type="password" autocomplete="current-password" required>
|
||||
<button class="accent" id="login-button" type="submit">Войти</button>
|
||||
</form>
|
||||
<p id="login-status" class="muted" role="status">Проверяем сессию…</p>
|
||||
<p id="login-error" class="error-text" role="alert" hidden></p>
|
||||
</main>
|
||||
</div>
|
||||
<main id="dashboard" hidden>
|
||||
<header class="masthead">
|
||||
<div><p class="eyebrow">TCP / ОПЕРАТОРСКАЯ</p><h1>Mirror Tool<span class="brand-mark" aria-hidden="true">↗↗</span></h1></div>
|
||||
<nav class="actions" aria-label="Управление">
|
||||
<button id="add-team" data-write>+ Команда</button>
|
||||
<button id="add-service" class="accent" data-write>+ Сервис</button>
|
||||
<button id="refresh">Обновить</button>
|
||||
<button id="logout" data-write>Выйти</button>
|
||||
</nav>
|
||||
</header>
|
||||
<section class="telemetry" aria-label="Состояние панели">
|
||||
<div><span class="label">СВЯЗЬ</span><strong id="stream-state" class="warn">Подключение…</strong></div>
|
||||
<div><span class="label">СЕРВИСЫ</span><strong id="service-count">—</strong></div>
|
||||
<div><span class="label">FORCAD</span><strong id="scoreboard-state">—</strong></div>
|
||||
<div><span class="label">СНИМОК</span><strong id="snapshot-time">—</strong></div>
|
||||
</section>
|
||||
<p id="scoreboard-detail" class="status-detail muted"></p>
|
||||
<div id="global-error" class="notice error-text" role="alert" hidden></div>
|
||||
<p id="action-status" class="action-status" role="status" aria-live="polite"></p>
|
||||
<section id="service-board" class="service-board" aria-label="Сервисы"></section>
|
||||
<section id="empty-state" class="empty-state" hidden><h2>Добавьте первый сервис</h2><p>Создайте команду с адресом, затем задайте порт слушателя и основной маршрут.</p></section>
|
||||
<footer class="dashboard-footer"><p>◉ Основной маршрут □ Реплика</p><p>Изменения сохраняются сразу. Новые маршруты применяются к новым соединениям.</p></footer>
|
||||
<details class="journal"><summary>Журнал событий</summary><ol id="event-log"></ol></details>
|
||||
</main>
|
||||
<dialog id="service-dialog" aria-labelledby="service-dialog-title">
|
||||
<form id="service-form">
|
||||
<header class="dialog-header"><h2 id="service-dialog-title">Добавить сервис</h2><button type="button" data-close="service-dialog" aria-label="Закрыть окно сервиса">×</button></header>
|
||||
<div class="form-grid">
|
||||
<label>ID<input name="id" required pattern="[A-Za-z0-9_-]+" maxlength="80" placeholder="notes"><small>Латинские буквы, цифры, _ и -</small></label>
|
||||
<label>Название<input name="name" required maxlength="120" placeholder="Notes"></label>
|
||||
<label>Адрес слушателя<input name="listen" required placeholder="0.0.0.0:18080"></label>
|
||||
<label>Порт сервиса команды<input name="upstream_port" type="number" min="1" max="65535" required value="8080"></label>
|
||||
<label>Основной маршрут<select name="primary"></select></label>
|
||||
<label>Политика реплик<select name="replica_policy"><option value="isolate">Isolate — отключать отстающую</option><option value="strict">Strict — ждать все реплики</option></select></label>
|
||||
</div>
|
||||
<fieldset><legend>Реплики</legend><div id="dialog-replicas" class="replica-options"></div></fieldset>
|
||||
<div class="form-checks"><label><input name="enabled" type="checkbox"> Запустить слушатель</label><label><input name="auto_switch" type="checkbox"> Автопереключение</label></div>
|
||||
<p class="muted hint">Основной маршрут можно не выбирать у остановленного сервиса. Strict замедляет поток до скорости реплик; Isolate явно отключает отстающую реплику.</p>
|
||||
<p id="service-form-error" class="error-text" role="alert" hidden></p>
|
||||
<footer class="dialog-actions"><button type="button" data-close="service-dialog">Отмена</button><button type="submit" class="accent" data-write>Сохранить сервис</button></footer>
|
||||
</form>
|
||||
</dialog>
|
||||
<dialog id="team-dialog" aria-labelledby="team-dialog-title">
|
||||
<form id="team-form">
|
||||
<header class="dialog-header"><h2 id="team-dialog-title">Добавить команду</h2><button type="button" data-close="team-dialog" aria-label="Закрыть окно команды">×</button></header>
|
||||
<label>ID<input name="id" required pattern="[A-Za-z0-9_-]+" maxlength="80" placeholder="team-a"></label>
|
||||
<label>Название<input name="name" required maxlength="120" placeholder="Команда A"></label>
|
||||
<label>Адрес<input name="address" required placeholder="192.0.2.10"></label>
|
||||
<p class="muted hint">Адрес существующей команды неизменяем. Для другого адреса добавьте команду с новым ID.</p>
|
||||
<p id="team-form-error" class="error-text" role="alert" hidden></p>
|
||||
<footer class="dialog-actions"><button type="button" data-close="team-dialog">Отмена</button><button type="submit" class="accent" data-write>Сохранить команду</button></footer>
|
||||
</form>
|
||||
</dialog>
|
||||
<dialog id="delete-dialog" aria-labelledby="delete-title">
|
||||
<form id="delete-form">
|
||||
<h2 id="delete-title">Удалить сервис?</h2><p id="delete-detail"></p>
|
||||
<p class="muted">Настройки сервиса будут удалены. Установленные соединения завершатся в режиме дренирования.</p>
|
||||
<p id="delete-error" class="error-text" role="alert" hidden></p>
|
||||
<footer class="dialog-actions"><button type="button" data-close="delete-dialog" autofocus>Отмена</button><button type="submit" class="danger" data-write>Удалить сервис</button></footer>
|
||||
</form>
|
||||
</dialog>
|
||||
</body>
|
||||
</html>
|
||||
Loading…
Reference in New Issue
Block a user