mirror of
https://github.com/mrvasil/ctf-ad-mirror-tool-rust.git
synced 2026-10-23 14:31:02 +00:00
Restore legacy UI and finish Rust proxy correctness fixes
This commit is contained in:
parent
4b01fb47b1
commit
dbbbb8b255
2
.gitignore
vendored
2
.gitignore
vendored
@ -3,5 +3,7 @@
|
|||||||
/mirror.toml
|
/mirror.toml
|
||||||
/.env
|
/.env
|
||||||
/test-results/
|
/test-results/
|
||||||
|
/.superpowers/
|
||||||
*.log
|
*.log
|
||||||
.DS_Store
|
.DS_Store
|
||||||
|
/dist/
|
||||||
|
|||||||
20
README.md
20
README.md
@ -20,7 +20,11 @@ export MIRROR_PASSWORD
|
|||||||
|
|
||||||
Пароль должен содержать минимум 12 символов. Панель по умолчанию доступна на `http://127.0.0.1:6700`. Для удалённого доступа используйте VPN/SSH-туннель либо HTTPS reverse proxy; при HTTPS установите `secure_cookie = true`. TLS для управляющей панели внутри бинарника не реализован. Переменные окружения с паролем и конфигурацию игры не коммитьте.
|
Пароль должен содержать минимум 12 символов. Панель по умолчанию доступна на `http://127.0.0.1:6700`. Для удалённого доступа используйте VPN/SSH-туннель либо HTTPS reverse proxy; при HTTPS установите `secure_cookie = true`. TLS для управляющей панели внутри бинарника не реализован. Переменные окружения с паролем и конфигурацию игры не коммитьте.
|
||||||
|
|
||||||
В панели: выберите основной маршрут, включите нужные реплики и запустите сервис. Радиокнопка меняет primary, флажок выбирает реплику. Настройки сохраняются после успешного ответа сервера; они переживают рестарт. Сервис можно добавить с произвольными портом слушателя и upstream-портом, команду — с IPv4/IPv6-адресом. Авто режим выбирает только из назначенных реплик.
|
Панель сохраняет оформление старого `mirror_tool`: горизонтальные колонки сервисов, список команд, кружок primary, выбор реплики нажатием на строку, робот авто режима, открытие сервиса и переключатель запуска. При выборе нового primary прежний становится репликой. Кнопка «+» открывает прежнюю форму: название, порт и необязательный IPv4/IPv6. Предварительно создавать команду для произвольного IP не нужно. У custom-прокси остаются поле IP и удаление в самой колонке.
|
||||||
|
|
||||||
|
Настройки сохраняются после успешного ответа сервера и переживают рестарт. Дополнительные настройки открываются нажатием на название сервиса: имя, адрес слушателя, upstream-порт и поведение при медленной реплике. Основная панель сохраняет прежние четыре счётчика соединений; байты и диагностика реплик доступны в подсказке над ними и API. Авто режим выбирает только из назначенных реплик.
|
||||||
|
|
||||||
|
Исходный CSS перенесён с нормализацией пробелов в концах строк; Bootstrap 5.1.3 и Font Awesome Free 6.0.0 той же версии включены в бинарник. Для работы панели Интернет не нужен. Небольшие изменения: клавиатурное управление, индикаторы устаревших данных, сохранение формы во время обновлений и адаптация узкого экрана. Лицензии встроенных зависимостей доступны на `/licenses.txt` и в `web/vendor/`.
|
||||||
|
|
||||||
## Доставка и перегрузка
|
## Доставка и перегрузка
|
||||||
|
|
||||||
@ -47,6 +51,8 @@ ForcAD: постоянная сессия Engine.IO v4 / Socket.IO polling, name
|
|||||||
|
|
||||||
Для adplatf укажите `kind = "adplatf"`: используется HTTP `/state`. URL не должен содержать пароль, query или fragment. Доступный endpoint должен отдавать публичную таблицу без отдельной аутентификации.
|
Для adplatf укажите `kind = "adplatf"`: используется HTTP `/state`. URL не должен содержать пароль, query или fragment. Доступный endpoint должен отдавать публичную таблицу без отдельной аутентификации.
|
||||||
|
|
||||||
|
Для прежней цветной полосы состояния **своей команды** задайте `main_team_ip = "192.0.2.10"` в начале TOML, до секций. Это аналог `MAIN_TEAM_IP`; полоса независима от выбранного primary. Без этого параметра она остаётся нейтральной. Имена команд окрашиваются по checker, чипы показывают потерянные флаги; сортировка сохраняет прежний порядок status → lost → имя.
|
||||||
|
|
||||||
Автопереключение:
|
Автопереключение:
|
||||||
|
|
||||||
1. TCP-пробы выполняются параллельно с ограничением; отдельно считаются последовательные сбои и успешные проверки.
|
1. TCP-пробы выполняются параллельно с ограничением; отдельно считаются последовательные сбои и успешные проверки.
|
||||||
@ -63,15 +69,15 @@ Tokio multiplexes соединения без subprocess/thread на сесси
|
|||||||
|
|
||||||
Путь без реплик переиспользует буфер; зеркальный путь передаёт разделяемые `Bytes` без отдельной копии на реплику. Счётчики используют атомарные операции. TCP_NODELAY и TCP keepalive включены; connect timeout не превращается в короткий idle timeout.
|
Путь без реплик переиспользует буфер; зеркальный путь передаёт разделяемые `Bytes` без отдельной копии на реплику. Счётчики используют атомарные операции. TCP_NODELAY и TCP keepalive включены; connect timeout не превращается в короткий idle timeout.
|
||||||
|
|
||||||
Stop отключает приём новых соединений, сохраняя открытые. Удаление остановленного сервиса ждёт `drain_timeout_ms`, затем отменяет остаток. SIGINT/SIGTERM закрывает слушатели, даёт всем сессиям общий срок дренирования, затем отменяет и дожидается принадлежащих им задач. Isolate также ограничивает время отправки остатка репликам после завершения основного обмена. Strict в нормальном режиме может ожидать медленную реплику неограниченно; общая остановка остаётся ограниченной.
|
Stop отключает приём новых соединений, сохраняя открытые. Удаление остановленного сервиса ждёт `drain_timeout_ms`, затем отменяет остаток. SIGINT/SIGTERM запрещает новые управляющие изменения, закрывает слушатели и ограничивает завершение процесса общим бюджетом `drain_timeout_ms + 1100 ms` с учётом планирования ОС. Недочитанный HTTP-запрос не задерживает процесс бесконечно. При исчерпании бюджета остаток прерывается; сохранение могло завершиться до обрыва HTTP-ответа. Isolate также ограничивает время отправки остатка репликам после завершения основного обмена. Strict в нормальном режиме может ожидать медленную реплику неограниченно; общая остановка остаётся ограниченной.
|
||||||
|
|
||||||
## Конфигурация и состояние
|
## Конфигурация и состояние
|
||||||
|
|
||||||
`mirror.toml` задаёт настройки процесса и начальную топологию. После первого изменения сохранённый `state_file` содержит авторитетный список команд/сервисов; TOML продолжает задавать сеть, мониторинг и scoreboard. Путь состояния отсчитывается от каталога TOML. Чтобы применить начальную топологию заново, остановите процесс и **переместите** state-файл в резервную копию. Файл не исполняется, версия и содержимое валидируются.
|
`mirror.toml` задаёт настройки процесса и начальную топологию. После первого изменения сохранённый `state_file` содержит авторитетный список команд/сервисов; TOML продолжает задавать сеть, мониторинг и scoreboard. Путь состояния отсчитывается от каталога TOML. Чтобы применить начальную топологию заново, остановите процесс и **переместите** state-файл в резервную копию. Файл не исполняется, версия и содержимое валидируются.
|
||||||
|
|
||||||
Запись состояния: временный файл на той же ФС → fsync → atomic rename; права `0600`. Сбой сохранения вызывает попытку отката runtime. Публикация topology сериализована с ручными и автоматическими изменениями. Runtime и ФС не образуют распределённую транзакцию: клиент, принятый в коротком окне до неудачного сохранения, может сохранить временно применённый маршрут. Необходимость отката выводится как ошибка.
|
Запись состояния: временный файл на той же ФС → fsync → atomic rename; права `0600`. Сбой сохранения вызывает попытку отката runtime. Публикация topology сериализована с ручными и автоматическими изменениями. Runtime и ФС не образуют распределённую транзакцию: клиент, принятый в коротком окне до неудачного сохранения, может сохранить временно применённый маршрут. Необходимость отката выводится как ошибка. После допуска к транзакции она завершается независимо от HTTP-соединения: закрытие вкладки не оставляет сохранение/очистку наполовину. Если ответ потерян, обновите snapshot перед повтором — изменение могло сохраниться.
|
||||||
|
|
||||||
Старый `config.py` не выполняется и не импортируется. Перенесите `TEAMS` в `[[teams]]`, `TASKS` в `[[services]]`, ForcAD URL в `[scoreboard]`. `PASSWORD` заменён переменной окружения, `SECRET_KEY` не нужен: сервер хранит случайные сессии в памяти. Старые cookie после перезапуска недействительны. API описан в [docs/api.md](docs/api.md).
|
Старый `config.py` не выполняется и не импортируется. Перенесите `TEAMS` в `[[teams]]`, `TASKS` в `[[services]]`, ForcAD URL в `[scoreboard]`, `MAIN_TEAM_IP` в верхнеуровневый `main_team_ip`. `PASSWORD` заменён переменной окружения, `SECRET_KEY` не нужен: сервер хранит случайные сессии в памяти. Старые cookie после перезапуска недействительны. API описан в [docs/api.md](docs/api.md).
|
||||||
|
|
||||||
Для Linux есть [systemd unit](deploy/mirror-tool.service): создайте отдельного пользователя `mirror-tool`, разместите бинарник в `/usr/local/bin/`, конфиг и закрытый `mirror.env` в `/etc/mirror-tool/`; в TOML задайте `state_file = "/var/lib/mirror-tool/state.json"`. Unit даёт только CAP_NET_BIND_SERVICE для портов ниже 1024; при ненужности удалите capability. При увеличении drain timeout увеличьте `TimeoutStopSec`. Unit поставляется как шаблон, автоматически не устанавливается.
|
Для Linux есть [systemd unit](deploy/mirror-tool.service): создайте отдельного пользователя `mirror-tool`, разместите бинарник в `/usr/local/bin/`, конфиг и закрытый `mirror.env` в `/etc/mirror-tool/`; в TOML задайте `state_file = "/var/lib/mirror-tool/state.json"`. Unit даёт только CAP_NET_BIND_SERVICE для портов ниже 1024; при ненужности удалите capability. При увеличении drain timeout увеличьте `TimeoutStopSec`. Unit поставляется как шаблон, автоматически не устанавливается.
|
||||||
|
|
||||||
@ -82,11 +88,15 @@ cargo fmt --check
|
|||||||
cargo clippy --all-targets --locked -- -D warnings
|
cargo clippy --all-targets --locked -- -D warnings
|
||||||
cargo test --all-targets --locked
|
cargo test --all-targets --locked
|
||||||
node --check web/app.js
|
node --check web/app.js
|
||||||
cargo run --release --example benchmark -- 64
|
cargo run --release --example benchmark -- 64 none
|
||||||
|
cargo run --release --example benchmark -- 64 strict
|
||||||
|
cargo run --release --example benchmark -- 64 isolate
|
||||||
```
|
```
|
||||||
|
|
||||||
Benchmark сравнивает direct loopback с proxy в том же процессе, проверяет каждый байт, выполняет три чередующихся прогона и 500 latency-проб. Это воспроизводимый локальный baseline, не прогноз скорости реальной сети. Результаты и практические ограничения: [docs/verification.md](docs/verification.md).
|
Benchmark сравнивает direct loopback с proxy в том же процессе, проверяет каждый байт, выполняет три чередующихся прогона и 500 latency-проб. Это воспроизводимый локальный baseline, не прогноз скорости реальной сети. Результаты и практические ограничения: [docs/verification.md](docs/verification.md).
|
||||||
|
|
||||||
Для ручного теста UI на изолированном loopback: `cargo build`, затем `python3 scripts/ui_fixture.py target/debug/mirror-tool`. Это необязательная тестовая утилита; её публичный тестовый пароль указан в начале файла, конфиг/состояние временные. Не используйте fixture для реального доступа.
|
Для ручного теста UI на изолированном loopback: `cargo build`, затем `python3 scripts/ui_fixture.py target/debug/mirror-tool`. Это необязательная тестовая утилита; её публичный тестовый пароль указан в начале файла, конфиг/состояние временные. Не используйте fixture для реального доступа.
|
||||||
|
|
||||||
|
Сравнение с прежними шаблонами: установите `jinja2` в отдельный dev-venv и запустите `python scripts/legacy_ui_reference.py ../ad_scripts/mirror_tool`. Утилита проверяет сохранность исходных CSS-правил и рендерит старые страницы на синтетических loopback-данных, не читая `config.py`. Для дополнительной ForcAD-проверки есть `scripts/verify_forcad_library.py`, использующий настоящие версии Socket.IO/Engine.IO из upstream.
|
||||||
|
|
||||||
Структура: `config` — валидация, `engine` — TCP, `scoreboard` — протоколы таблиц, `control` — health/выбор/изменения, `store` — сохранение, `auth`/`web` — управление, `web/` — встроенный интерфейс. `agent.md` и журналы фиксируют ход работы.
|
Структура: `config` — валидация, `engine` — TCP, `scoreboard` — протоколы таблиц, `control` — health/выбор/изменения, `store` — сохранение, `auth`/`web` — управление, `web/` — встроенный интерфейс. `agent.md` и журналы фиксируют ход работы.
|
||||||
|
|||||||
15
agent.md
15
agent.md
@ -5,11 +5,16 @@ Rewrite the legacy Python TCP mirror manager in Rust, in this repository. Preser
|
|||||||
|
|
||||||
## Binding user decisions
|
## Binding user decisions
|
||||||
- 2026-10-23: automatic routing means upstream selection inside the TCP proxy; no nftables/iptables or host route changes.
|
- 2026-10-23: automatic routing means upstream selection inside the TCP proxy; no nftables/iptables or host route changes.
|
||||||
- Keep existing service/team selection, primary + replicas, custom services, statistics, availability and scoreboard views. No additional UI preferences yet.
|
- 2026-10-23 correction: preserve the original UI essentially literally: original CSS, layout, service columns, primary dots, replica row selection, robot switch, open-service icon, login and custom proxy name/port/optional direct-IP workflow. Only small improvements are authorized. The initial redesign was rejected; `web/*` now restores the original CSS/layout and interactions.
|
||||||
|
- Preserve manual promotion semantics: the previous primary becomes a replica. Direct-IP entry must not require the operator to create a team first.
|
||||||
- Keep the legacy directories unchanged. Never copy credentials from their config into this repository.
|
- Keep the legacy directories unchanged. Never copy credentials from their config into this repository.
|
||||||
|
|
||||||
## Current state
|
## Current state
|
||||||
Rust modules implemented: config, store, engine, scoreboard, control, auth, web and CLI. TCP engine implementer delivered 11 passing integration tests; UI implementer active on `web/*`. Parent owns integration, protocol fixtures, Linux verification, benchmark, final review and documentation. Read `task_plan.md`, `findings.md`, and `progress.md` before resuming.
|
Rust modules implemented: config, store, engine, scoreboard, control, auth, web and CLI. Initial commit `4b01fb4`, branch `rewrite/rust`; no remote or deployment. The original UI is restored and assets are embedded locally. `docs/ui-report.md` describes preserved controls and small improvements; source-CSS parity and real browser checks confirm the restoration.
|
||||||
|
|
||||||
|
Initial review R1-R5 are addressed: bounded shutdown with early quiesce, probe completion timestamps, Linux/mapped-address loop prevention, removal of rejected journal and restored previous-primary-as-replica. Transient accept recovery and absolute Engine.IO heartbeat are implemented. Final independent review found R6: request cancellation can abandon control transactions. Owned transactions now complete despite caller cancellation; five regressions and the original HTTP reproduction pass, and independent final review is approved.
|
||||||
|
|
||||||
|
Fresh root verification after R6: 40 tests on macOS arm64, 41 on Orb Linux amd64; native fmt/Clippy and release binary/examples on both pass. Real Socket.IO-library ForcAD interoperability passes. Browser covers start, route/replica promotion, robot, custom IP, settings, logout, actual390px viewport, watchdog stale/recovery and byte-exact1MiB half-close. Loopback benchmark results are checked in under `docs/benchmarks/`; isolate saturated its bounded replica queue and explicitly disconnected replicas as designed (3 on macOS,4 on Linux across5 sessions). `docs/verification.md` is the acceptance record.
|
||||||
|
|
||||||
## Verification policy
|
## Verification policy
|
||||||
No completion/performance claims without fresh output. Use loopback and Orb Linux fixtures; never contact configured legacy game hosts. Test exact byte delivery, half-close, idle connections, live target changes, failure selection, replica overload, ForcAD reconnects, auth and persistence. Record limits explicitly.
|
No completion/performance claims without fresh output. Use loopback and Orb Linux fixtures; never contact configured legacy game hosts. Test exact byte delivery, half-close, idle connections, live target changes, failure selection, replica overload, ForcAD reconnects, auth and persistence. Record limits explicitly.
|
||||||
@ -20,3 +25,9 @@ No completion/performance claims without fresh output. Use loopback and Orb Linu
|
|||||||
- Typed config validation, same-filesystem fsync+rename state replacement, rollback on persistence failure, fresh-health deterministic selection with cooldown.
|
- Typed config validation, same-filesystem fsync+rename state replacement, rollback on persistence failure, fresh-health deterministic selection with cooldown.
|
||||||
- Cookie sessions (HttpOnly/SameSite), CSRF header, same-origin checks, login rate limiting, no secret persistence. HTTP binds locally by default; user supplies environment password.
|
- Cookie sessions (HttpOnly/SameSite), CSRF header, same-origin checks, login rate limiting, no secret persistence. HTTP binds locally by default; user supplies environment password.
|
||||||
- Fresh engine evidence: 11 loopback tests passed (see docs/engine-report.md). Full integrated checks still pending.
|
- Fresh engine evidence: 11 loopback tests passed (see docs/engine-report.md). Full integrated checks still pending.
|
||||||
|
|
||||||
|
## Final handoff
|
||||||
|
- `docs/final-review.md`: independent approval; R1-R6 addressed, no remaining Critical/Important findings.
|
||||||
|
- `docs/verification.md`: final gates, browser/ForcAD evidence and measured performance; raw benchmark JSON in `docs/benchmarks/`.
|
||||||
|
- Generated, git-ignored `dist/` contains macOS arm64 and Linux amd64 binaries plus SHA256SUMS. Linux binary is glibc-linked from Orb Ubuntu; not an all-distribution static build.
|
||||||
|
- Legacy tracked files remain unchanged and all12 match Downloads. Local-only rewrite; configure real teams/ports/scoreboard manually before deployment. No service installed.
|
||||||
|
|||||||
13
docs/api.md
13
docs/api.md
@ -1,9 +1,10 @@
|
|||||||
# HTTP contract
|
# HTTP contract
|
||||||
|
|
||||||
JSON API, UTF-8. Authenticated sessions use an HttpOnly SameSite=Strict cookie. Mutating authenticated requests require `X-CSRF-Token` from login or session response. Browser requests must be same-origin. No secrets returned in snapshots. All failures return `{ "error": "human readable explanation" }` with non-2xx status. A snapshot is the source of truth; never assume a mutation succeeded before its 2xx response.
|
JSON API, UTF-8. Authenticated sessions use an HttpOnly SameSite=Strict cookie. Mutating authenticated requests require `X-CSRF-Token` from login or session response. Browser requests must be same-origin. No secrets returned in snapshots. Application-handled failures return `{ "error": "human readable explanation" }` with non-2xx status. Framework routing errors (for example 405) may have a plain-text body; clients must tolerate this. A snapshot is the source of truth; a 2xx response confirms completion. After admission under the serialized operation lock, topology mutations run independently of the HTTP connection through commit/rollback and cleanup. A lost response does not imply an uncommitted change: refresh the snapshot before retrying. Requests cancelled while waiting for admission make no change. Shutdown rejects new admissions and waits for admitted work within the documented process deadline.
|
||||||
|
|
||||||
## Routes
|
## Routes
|
||||||
- `GET /` — embedded UI shell (login shown until session validated).
|
- `GET /` — embedded UI shell (login shown until session validated).
|
||||||
|
- `GET /licenses.txt` — licenses and attribution for the bundled Bootstrap/Font Awesome assets.
|
||||||
- `GET /healthz` — process readiness `{status:"ok"}` (not upstream checker health).
|
- `GET /healthz` — process readiness `{status:"ok"}` (not upstream checker health).
|
||||||
- `POST /api/login` JSON `{password:string}` → `{authenticated:true,csrf:string}`, Set-Cookie. 401 invalid, 429 limited.
|
- `POST /api/login` JSON `{password:string}` → `{authenticated:true,csrf:string}`, Set-Cookie. 401 invalid, 429 limited.
|
||||||
- `GET /api/session` → `{authenticated:true,csrf:string}`, 401 expired/missing.
|
- `GET /api/session` → `{authenticated:true,csrf:string}`, 401 expired/missing.
|
||||||
@ -23,15 +24,21 @@ Team:
|
|||||||
{"id":"a","name":"Team A","address":"192.0.2.10"}
|
{"id":"a","name":"Team A","address":"192.0.2.10"}
|
||||||
```
|
```
|
||||||
|
|
||||||
|
Backward-compatible optional fields: ServiceConfig has `custom:false` and `custom_ip:null`; Team has `hidden:false`. The legacy add dialog sends `custom:true`, equal listen/upstream ports and an optional `custom_ip`. A non-null IP resolves/reuses its team, sets primary, and atomically saves team plus service. New anonymous IP targets have `hidden:true`; the UI shows them only in services that reference them. No separate operator team-creation step is required. Clearing `custom_ip` leaves the submitted primary/replicas authoritative. Manual quick promotion retains the previous primary as replica; automatic promotion does the same and clears the direct-IP override.
|
||||||
|
|
||||||
|
Configured/saved topology containing a non-null `custom_ip` must also contain its matching primary team. The PUT adapter creates that pair; raw TOML/state files are validated rather than silently repaired.
|
||||||
|
|
||||||
## Snapshot
|
## Snapshot
|
||||||
```json
|
```json
|
||||||
{
|
{
|
||||||
"now_ms": 1792700000000,
|
"now_ms": 1792700000000,
|
||||||
|
"main_team_ip": "192.0.2.10",
|
||||||
|
"health_max_age_ms": 12000,
|
||||||
"teams": [{"id":"a","name":"Team A","address":"192.0.2.10"}],
|
"teams": [{"id":"a","name":"Team A","address":"192.0.2.10"}],
|
||||||
"services": [{
|
"services": [{
|
||||||
"id":"notes","name":"Notes","listen":"0.0.0.0:18080","upstream_port":8080,
|
"id":"notes","name":"Notes","listen":"0.0.0.0:18080","upstream_port":8080,
|
||||||
"primary":"a","replicas":[],"enabled":true,"auto_switch":false,"replica_policy":"isolate",
|
"primary":"a","replicas":[],"enabled":true,"auto_switch":false,"replica_policy":"isolate",
|
||||||
"engine":{"running":true,"last_error":null,"stats":{"accepted":5,"active":1,"completed":4,"connect_errors":0,"io_errors":0,"rejected":0,"bytes_up":1234,"bytes_down":5678,"replica_disconnects":0,"replica_errors":0,"replica_bytes":0}},
|
"engine":{"running":true,"last_error":null,"stats":{"accepted":5,"active":1,"completed":4,"connect_errors":0,"io_errors":0,"rejected":0,"accept_errors":0,"bytes_up":1234,"bytes_down":5678,"replica_disconnects":0,"replica_errors":0,"replica_bytes":0}},
|
||||||
"health":{"a":{"reachable":true,"healthy":true,"consecutive_failures":0,"consecutive_successes":2,"checked_at_ms":1792700000000,"latency_ms":2}},
|
"health":{"a":{"reachable":true,"healthy":true,"consecutive_failures":0,"consecutive_successes":2,"checked_at_ms":1792700000000,"latency_ms":2}},
|
||||||
"scoreboard":{"a":{"status":101,"stolen":3,"lost":0}}
|
"scoreboard":{"a":{"status":101,"stolen":3,"lost":0}}
|
||||||
}],
|
}],
|
||||||
@ -41,3 +48,5 @@ Team:
|
|||||||
```
|
```
|
||||||
|
|
||||||
Compute displayed bandwidth from positive byte-counter deltas / `now_ms` deltas, reset baseline when counters decrease. Show unknown health until checked. Scoreboard badges must indicate stale data; never imply stale checker state is current. Replica counters mean explicit disconnections/errors, not dropped arbitrary packets. `replica_policy`: `strict` uses backpressure, `isolate` disconnects a lagging replica per session.
|
Compute displayed bandwidth from positive byte-counter deltas / `now_ms` deltas, reset baseline when counters decrease. Show unknown health until checked. Scoreboard badges must indicate stale data; never imply stale checker state is current. Replica counters mean explicit disconnections/errors, not dropped arbitrary packets. `replica_policy`: `strict` uses backpressure, `isolate` disconnects a lagging replica per session.
|
||||||
|
|
||||||
|
The restored panel retains the legacy four connection counters; byte totals and replica counters are available in their tooltip and the snapshot API. TCP freshness uses `health_max_age_ms`; the UI additionally marks the whole snapshot stale and disables mutations after five seconds without updates. The top service strip shows the configured `main_team_ip` checker status, independently of the selected upstream, as in the legacy panel. Unset main IP gives an uncolored strip. Checker names and lost-flag chips retain the original color/sort behavior.
|
||||||
|
|||||||
140
docs/backend-fixes-report.md
Normal file
140
docs/backend-fixes-report.md
Normal file
@ -0,0 +1,140 @@
|
|||||||
|
# Backend completion evidence — 2026-10-23
|
||||||
|
|
||||||
|
Scope: backend completion from local base `4b01fb4`; this is not a whole-project acceptance report. Changes are uncommitted. Root owns UI/web adaptation and the public API/journal documentation. No legacy sibling files, credentials or game hosts were used. Network fixtures use loopback; documentation-only addresses appear only in validation tests or injected probe futures, never outbound connections.
|
||||||
|
|
||||||
|
## Implemented fixes
|
||||||
|
|
||||||
|
### R1 — bounded shutdown and serialized control transactions
|
||||||
|
|
||||||
|
The original main task waited for Axum graceful completion before beginning TCP shutdown. A pending body extractor could therefore keep both the process and data listeners alive indefinitely.
|
||||||
|
|
||||||
|
Main now reacts to the signal independently of HTTP graceful completion, cancels background work and calls `Controller::shutdown`. The controller immediately marks itself stopping, rejects subsequent service/team mutations and deletion, obtains its existing operation mutex and shuts down the engine after the in-flight transaction has resolved. Automatic edits also check this state. The existing engine continues to own TCP sessions and its bounded drain; the data path gains no short idle timeout.
|
||||||
|
|
||||||
|
The asynchronous shutdown phase has one deadline of `network.drain_timeout_ms + 1000 ms`, beginning at signal handling. Runtime destruction adds at most another configured 100 ms wait for workers, via explicit `Runtime::shutdown_timeout`, rather than indefinitely joining an OS-blocked filesystem worker. The intended overall process shutdown budget is therefore **drain timeout + 1100 ms**, subject to OS scheduling. The 1000 ms allowance includes transaction/HTTP/background coordination; it is not added separately for each subsystem.
|
||||||
|
|
||||||
|
When the outer deadline expires the process logs the deadline and exits; requests may be interrupted. An already executing atomic filesystem replacement can have committed even if its HTTP response was interrupted. This is bounded shutdown, not a guarantee that forced shutdown can undo filesystem I/O or deliver every pending request. Ordinary in-flight transaction coordination and rollback complete before engine teardown when they fit the budget.
|
||||||
|
|
||||||
|
Regressions:
|
||||||
|
|
||||||
|
- `tests/shutdown.rs`: actual executable, live proxy/upstream session, unfinished unauthenticated JSON body; waits for real HTTP readiness, sends SIGTERM, proves the data listener stops accepting within one second and the process exits within three seconds without closing the partial-body peer first. Configuration uses a 100 ms drain timeout.
|
||||||
|
- `shutdown_serializes_with_in_flight_service_transaction`: starts a PUT transaction concurrently with controller shutdown, proves the committed topology matches disk and the resulting listener is closed.
|
||||||
|
- Custom service regression also proves service/team mutations are rejected after controller shutdown.
|
||||||
|
|
||||||
|
### R2 — probe observation time
|
||||||
|
|
||||||
|
Each probe captures its timestamp at completion. Batch publication copies that timestamp unchanged. The probe wrapper and full controller cycle admit deterministic future injection for tests; production still uses timed TCP connects.
|
||||||
|
|
||||||
|
`delayed_batch_does_not_refresh_early_candidate_observation` drives the actual controller publication and failover path: an early responsive replica is followed by a 450 ms delayed probe, while the configured health freshness window is 303 ms. Its published timestamp remains old and no promotion happens. The test performs no outbound traffic to the synthetic documentation addresses.
|
||||||
|
|
||||||
|
### R3 — wildcard loopback aliases and mapped IPv6
|
||||||
|
|
||||||
|
Topology validation canonicalizes IPv4-mapped IPv6 addresses for equality, duplicate-address checks and local-route detection. IPv4 loopback aliases are recognized with `is_loopback()`, independent of interface enumeration. A native IPv6 target is not treated as reachable through an IPv4-only wildcard listener. IPv6 wildcard detection is conservative about possible dual-stack acceptance. Bind conflict checks use the same family-aware equivalence.
|
||||||
|
|
||||||
|
`wildcard_loop_aliases_mapped_addresses_and_address_families` rejects self/cross-service routes through 127.0.0.1, 127.0.0.2 and mapped aliases while permitting a native ::1 target behind an IPv4-only wildcard on that port. The Linux-only runtime regression additionally connects 127.0.0.2 to an actual IPv4 wildcard listener, proving the platform behavior that requires the check.
|
||||||
|
|
||||||
|
### Direct-IP custom service adapter
|
||||||
|
|
||||||
|
`ServiceConfig` adds serde-defaulted `custom: bool` and `custom_ip: Option<IpAddr>`. `Team` adds serde-defaulted `hidden: bool`. Existing omitted fields deserialize to false/None. Standard PUT remains the sole operation.
|
||||||
|
|
||||||
|
A non-null custom IP is accepted only for a custom service. Inside the same serialized transaction it normalizes/reuses the address, or generates a hidden team with name equal to its IP and a stable collision-safe valid ID. It sets primary and removes duplicate primary references from replicas. Team plus service are validated and saved in one atomic topology replacement. Runtime rollback uses the original teams if persistence fails. A null custom IP leaves the submitted team selection authoritative. Automatic promotion clears custom_ip and keeps the former primary as a replica.
|
||||||
|
|
||||||
|
Regressions cover save/load, ordinary-team reuse without hiding it, generated-ID collisions, repeated-IP ID stability, restart, clearing direct-IP metadata, primary/replica deduplication and automatic promotion from a custom-IP service. A new actual service persistence failure test changes a live listener before inducing save failure, proves the old listener/route is restored and the changed listener closes, then proves a failed new direct-IP service leaves no listener or added team behind.
|
||||||
|
|
||||||
|
Additional requested metadata: serde-defaulted `Config.main_team_ip: Option<IpAddr>`, serialized `Snapshot.main_team_ip` and `Snapshot.health_max_age_ms`. The latter calls the same calculation used by automatic candidate selection: saturating `(interval_ms + timeout_ms) * 3`. No fake scoreboard team or scoreboard parser changes were introduced.
|
||||||
|
|
||||||
|
### Optional resilience
|
||||||
|
|
||||||
|
Implemented an absolute Engine.IO heartbeat deadline. Namespace setup, polling body reads and pong posting are bounded by the deadline; only receipt of a ping renews it. Continuous namespace/non-ping packets do not extend the session. The negotiated sum is bounded to 1..120000 ms, preserving subsecond negotiated intervals. The new wire-level fixture continuously returns non-ping frames and requires a fresh handshake.
|
||||||
|
|
||||||
|
Recoverable accept-error retry was deliberately left unchanged: the review did not reproduce that failure branch, and adding OS-specific resource-pressure classification without a reliable injection fixture would exceed this focused correction. The source-backed resilience concern remains open; normal accept, cancellation, connection-limit shedding and shutdown paths are covered by the existing forwarding suite.
|
||||||
|
|
||||||
|
## Verification
|
||||||
|
|
||||||
|
Native commands, all exit 0 on final source:
|
||||||
|
|
||||||
|
```
|
||||||
|
cargo fmt --check
|
||||||
|
cargo clippy --all-targets -- -D warnings
|
||||||
|
cargo test --all-targets
|
||||||
|
```
|
||||||
|
|
||||||
|
Native suite: 31 passing tests: 9 library, 8 control-plane, 2 ForcAD transport, 11 forwarding, 1 subprocess shutdown; main/example test targets also build.
|
||||||
|
|
||||||
|
Linux uses Orb's running `ubuntu` amd64 environment. Source is staged separately at `/home/ubuntu/mirror-tool-backend-validation`; only compiled dependency/artifact cache is shared with the previous validation checkout. Command:
|
||||||
|
|
||||||
|
```
|
||||||
|
orb -m ubuntu sh -lc 'cd /home/ubuntu/mirror-tool-backend-validation && CARGO_TARGET_DIR=/home/ubuntu/mirror-tool-validation/target cargo test --all-targets'
|
||||||
|
```
|
||||||
|
|
||||||
|
Final Linux result: **32 passed, 0 failed**, including the Linux-only alias runtime fixture; all main/example targets built. Orb's distribution Cargo lacks the `fmt` subcommand; formatter and Clippy gates are run natively, not represented as Linux passes.
|
||||||
|
|
||||||
|
Red checks were run in `/tmp/mirror-backend-regression-red`, restoring one original defect at a time. Original main reproduced the partial-body exit failure; publication-time timestamping failed the delayed-batch assertion; removing alias detection accepted 127.0.0.2; original scoreboard transport failed reconnect under continuous non-ping packets. Each targeted command exited 101 for the expected assertion. Logs are local `/tmp/mirror-red-{R1,R2,R3,heartbeat}.log`. The shared native project artifact cache was cleaned afterward and final source rebuilt, avoiding stale artifacts from the isolated negative runs.
|
||||||
|
|
||||||
|
One initial Linux shutdown fixture failed because it sent SIGTERM after TCP connect but before application readiness/handler registration. The fixture now waits for a real HTTP response, and the corrected Linux test passed. This was a fixture synchronization issue, not evidence of a fixed-source bounded-shutdown failure.
|
||||||
|
|
||||||
|
No low-file-descriptor accept-error fault injection, forced indefinitely blocked kernel filesystem operation or live production scoreboard/game-host testing was performed. UI restoration and whole-project acceptance remain root's responsibility.
|
||||||
|
|
||||||
|
## Modified paths owned by this task
|
||||||
|
|
||||||
|
- `src/config.rs`
|
||||||
|
- `src/control.rs`
|
||||||
|
- `src/main.rs`
|
||||||
|
- `src/scoreboard.rs`
|
||||||
|
- `tests/control_plane.rs`
|
||||||
|
- `tests/forcad_transport.rs`
|
||||||
|
- `tests/forwarding.rs` (new schema fields in existing literals only)
|
||||||
|
- `tests/shutdown.rs` (new)
|
||||||
|
- `examples/benchmark.rs` (new schema fields in existing literals only; explicitly coordinated with root)
|
||||||
|
- `docs/backend-fixes-report.md`
|
||||||
|
|
||||||
|
No edits to `src/engine.rs`, Cargo dependencies, root-owned UI files, scripts, other documentation or the legacy sibling were made by this task.
|
||||||
|
|
||||||
|
## Follow-up: accept supervision and early quiesce
|
||||||
|
|
||||||
|
This section supersedes the earlier accept-retry deferral and the earlier ordering description of controller shutdown.
|
||||||
|
|
||||||
|
Recoverable accept errors now keep the existing listener supervisor alive. The supervisor retries after 50 ms, doubles the delay up to a one-second cap and resets it after successful acceptance. The retry sleep listens for cancellation; it neither busy-loops nor delays shutdown until the timer expires. Errors classified as invalid input, unsupported operation or a disconnected listener remain terminal. Other errors, including OS resource-pressure/unknown errors such as descriptor or buffer exhaustion, are retried with the cap. Persistent resource exhaustion can keep retrying until shutdown, with a bounded attempt rate.
|
||||||
|
|
||||||
|
Each accept failure increments the new serialized `TrafficStats.accept_errors` counter, stores an `accept:` diagnostic in `last_error` and emits a warning including the retry delay. The counter is distinct from errors on accepted data sessions. A successful accept leaves historical diagnostics available; `running` remains true while retrying and false when the supervisor exits.
|
||||||
|
|
||||||
|
The injection seam is a private generic `Acceptor` used by the actual supervisor. Production uses `TcpListener`; unit fixtures substitute selected accept failures before returning to a real loopback listener. No public fault-injection interface or test-only production endpoint was introduced. New unit tests verify exact loopback payload delivery after injected resource errors, minimum retry spacing, reset after success, counters/diagnostics, a terminal error and cancellation during a scheduled 400 ms backoff. They do not depend on exhausting actual OS descriptors.
|
||||||
|
|
||||||
|
Controller shutdown now marks itself stopping and calls **Engine.quiesce before acquiring the controller transaction mutex**. Quiesce closes data acceptance and awaits listener acknowledgements, without forcing existing sessions or setting the engine's final stopped flag. The already executing controller transaction can still commit or roll back. Engine upsert in this permanently quiesced state can restore route/topology bookkeeping but cannot bind/reopen a listener. After the transaction mutex becomes available, ordinary engine shutdown performs the bounded session drain. This preserves the same outer process deadline and avoids accepting new clients while a filesystem operation delays transaction completion.
|
||||||
|
|
||||||
|
New regressions distinguish those stages:
|
||||||
|
|
||||||
|
- A controller unit fixture deliberately holds its transaction mutex; shutdown closes actual loopback acceptance while remaining blocked on that mutex, then finishes after release.
|
||||||
|
- A forwarding integration fixture verifies a preexisting stream continues byte-for-byte through quiesce, and a subsequent route upsert (the rollback path) cannot reopen the listener.
|
||||||
|
- The existing concurrent commit test and real persistence-failure tests remain green.
|
||||||
|
|
||||||
|
Follow-up native gates: `cargo fmt --check`, `cargo clippy --all-targets -- -D warnings`, `cargo test --all-targets`; **35 tests passed, 0 failed** (12 library, 8 control-plane, 2 transport, 12 forwarding, 1 shutdown). Orb amd64 all-target verification: **36 passed, 0 failed**, including the Linux alias fixture. Root will rerun final integrated release/benchmark gates after handoff. No real EMFILE or indefinitely blocked filesystem kernel call was induced.
|
||||||
|
|
||||||
|
Additional modified paths in this follow-up: `src/engine.rs`, `src/control.rs`, `tests/forwarding.rs`, and this appended report. The earlier statement that engine.rs was untouched applies only to the first handoff. Root now owns further benchmark example changes; this follow-up did not edit examples or UI/web files.
|
||||||
|
|
||||||
|
Follow-up negative controls were also exercised in the isolated Orb source staging: restoring fatal-on-every-accept-error made the recovery supervisor test fail; removing the early quiesce call made the held-transaction-lock test time out while acceptance remained open. Both exited 101 for the expected fault. Logs: `/tmp/mirror-red-accept-retry.log` and `/tmp/mirror-red-early-quiesce.log`. Correct sources were restored afterward and the full Orb suite rerun green. `last_error` intentionally remains historical after recovery so an accept success cannot erase a concurrent session error; root labels it “Последняя ошибка”.
|
||||||
|
|
||||||
|
## R6 follow-up: cancellation-safe topology transactions
|
||||||
|
|
||||||
|
Independent review reproduced cancellation after DELETE had saved the new disk topology but before engine drain and in-memory publication completed. The original `/tmp/mirror-review-cancel.py` reproduction was rerun before this fix and printed `disk services 0 live services 1 active after drain deadline 1`. Four deterministic regression tests were then added and run against the original caller-owned mutation bodies; all four failed because progress stopped after caller cancellation.
|
||||||
|
|
||||||
|
All existing topology mutation entrypoints now share an owned transaction wrapper: service PUT, service DELETE, team PUT and the automatic failover mutation stage. Admission first acquires the same serialized operation mutex and checks the stopping flag. Only then is one transaction worker spawned, owning both the controller Arc and an owned mutex guard. Dropping the HTTP/caller future drops its JoinHandle, which detaches this worker rather than aborting it. Apply, atomic save, publication, rollback and stream retirement therefore remain in the same owner until completion. A lost HTTP response does not imply that the mutation failed or was uncommitted; refresh the snapshot before deciding whether to retry. The public mutation signatures and normal result/error handling are unchanged. There is no team DELETE endpoint or controller method in this application, so none was invented.
|
||||||
|
|
||||||
|
The mutex is acquired before spawning, rather than inside a task per request. Consequently there is at most one executing/admitted transaction worker per controller, even if many clients disconnect. Callers still waiting for admission create no additional transaction worker and can be cancelled without any mutation. The normal HTTP/request layer retains its existing concurrency behavior; this change does not claim a new global HTTP connection limit.
|
||||||
|
|
||||||
|
The wrapper is also the admission stopping check: already admitted work can finish after shutdown begins, while queued/new work is rejected. Controller shutdown still quiesces the engine before waiting on the operation mutex, then drains only after the admitted worker finishes. In-flight commit or rollback cannot reopen listeners because engine quiescence remains permanent. The existing outer process deadline still bounds shutdown: process termination/runtime teardown can interrupt work once that deadline expires. This change guarantees independence from caller cancellation during normal operation, not transaction completion across forced process termination, kernel filesystem stalls, panic or machine failure.
|
||||||
|
|
||||||
|
Automatic TCP probe futures remain cancellable before mutation admission. Once its selection/apply/save stage is admitted it uses the same owned worker. Controller creation uses `Arc::new_cyclic` only to retain a Weak reference for worker ownership while preserving existing public `&self` mutation methods; there is no permanent strong reference cycle.
|
||||||
|
|
||||||
|
Deterministic regression fixtures use private `cfg(test)` persistence barriers before save and after successful save. They are absent from production. Coverage:
|
||||||
|
|
||||||
|
- Cancel DELETE after disk removal while an actual loopback proxy session remains in its drain. Disk, controller snapshot, engine registry and stream closure must converge after the caller drops.
|
||||||
|
- Cancel service PUT after disk save but before publication; the worker must publish the same topology and the listener must deliver exact real loopback bytes.
|
||||||
|
- Cancel service PUT before a deliberately failing save; runtime listener rollback and hidden-team rollback must still finish.
|
||||||
|
- Cancel team PUT after save; live and disk teams must converge through the same wrapper.
|
||||||
|
- Cancel automatic-selection caller after promotion save; live primary, replicas and cleared custom-IP metadata must still converge with disk.
|
||||||
|
|
||||||
|
The first four tests were observed red before the wrapper and green after it. The automatic-selection regression was added to cover the shared automatic path and passes. The actual disconnected-HTTP reproduction was rerun after `cargo build` and now prints `disk services 0 live services 0 active after drain deadline 0`.
|
||||||
|
|
||||||
|
Fresh native gates: `cargo fmt --check`, `cargo clippy --all-targets -- -D warnings`, `cargo test --all-targets`, plus `cargo build` for the executable reproduction. All pass; native suite now has **40 passed, 0 failed** (17 library, 8 control-plane, 2 transport, 12 forwarding, 1 shutdown). R6 modifies only `src/control.rs` and this report. Root owns public API documentation and final integrated release verification.
|
||||||
|
|
||||||
|
R6 Orb amd64 staging run completed: `CARGO_TARGET_DIR=/home/ubuntu/mirror-tool-validation/target cargo test --all-targets` in `/home/ubuntu/mirror-tool-backend-validation`: **41 passed, 0 failed** (the extra test is the Linux-only loopback alias fixture). Staging contains the final backend source; root's newest UI/benchmark changes still require root's integrated sync/release checks.
|
||||||
46
docs/benchmarks/linux-isolate.json
Normal file
46
docs/benchmarks/linux-isolate.json
Normal file
@ -0,0 +1,46 @@
|
|||||||
|
{
|
||||||
|
"arch": "x86_64",
|
||||||
|
"byte_integrity": "verified every primary reply byte",
|
||||||
|
"direct": [
|
||||||
|
2363.3479009113216,
|
||||||
|
2526.3205097041105,
|
||||||
|
2622.6125134063136
|
||||||
|
],
|
||||||
|
"direct_latency_us": {
|
||||||
|
"p50": 10.416,
|
||||||
|
"p95": 17.875
|
||||||
|
},
|
||||||
|
"mode": "isolate",
|
||||||
|
"payload_mib_per_run": 64,
|
||||||
|
"platform": "linux",
|
||||||
|
"proxy": [
|
||||||
|
2268.084746490409,
|
||||||
|
2306.6311464692035,
|
||||||
|
2221.4361963347656
|
||||||
|
],
|
||||||
|
"proxy_latency_us": {
|
||||||
|
"p50": 34.041999999999994,
|
||||||
|
"p95": 65.335
|
||||||
|
},
|
||||||
|
"stats": {
|
||||||
|
"bench": {
|
||||||
|
"last_error": null,
|
||||||
|
"running": false,
|
||||||
|
"stats": {
|
||||||
|
"accept_errors": 0,
|
||||||
|
"accepted": 5,
|
||||||
|
"active": 0,
|
||||||
|
"bytes_down": 205536896,
|
||||||
|
"bytes_up": 205536896,
|
||||||
|
"completed": 5,
|
||||||
|
"connect_errors": 0,
|
||||||
|
"io_errors": 0,
|
||||||
|
"rejected": 0,
|
||||||
|
"replica_bytes": 22470461,
|
||||||
|
"replica_disconnects": 4,
|
||||||
|
"replica_errors": 0
|
||||||
|
}
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"throughput_unit": "MiB/s aggregate upload+echo download (replica bytes excluded)"
|
||||||
|
}
|
||||||
46
docs/benchmarks/linux-none.json
Normal file
46
docs/benchmarks/linux-none.json
Normal file
@ -0,0 +1,46 @@
|
|||||||
|
{
|
||||||
|
"arch": "x86_64",
|
||||||
|
"byte_integrity": "verified every primary reply byte",
|
||||||
|
"direct": [
|
||||||
|
2520.9095662294544,
|
||||||
|
2656.0910527933584,
|
||||||
|
2377.891509108597
|
||||||
|
],
|
||||||
|
"direct_latency_us": {
|
||||||
|
"p50": 9.833,
|
||||||
|
"p95": 16.291999999999998
|
||||||
|
},
|
||||||
|
"mode": "none",
|
||||||
|
"payload_mib_per_run": 64,
|
||||||
|
"platform": "linux",
|
||||||
|
"proxy": [
|
||||||
|
2265.8313325423105,
|
||||||
|
2314.247469279992,
|
||||||
|
2084.750685561305
|
||||||
|
],
|
||||||
|
"proxy_latency_us": {
|
||||||
|
"p50": 20.542,
|
||||||
|
"p95": 28.959
|
||||||
|
},
|
||||||
|
"stats": {
|
||||||
|
"bench": {
|
||||||
|
"last_error": null,
|
||||||
|
"running": false,
|
||||||
|
"stats": {
|
||||||
|
"accept_errors": 0,
|
||||||
|
"accepted": 5,
|
||||||
|
"active": 0,
|
||||||
|
"bytes_down": 205536896,
|
||||||
|
"bytes_up": 205536896,
|
||||||
|
"completed": 5,
|
||||||
|
"connect_errors": 0,
|
||||||
|
"io_errors": 0,
|
||||||
|
"rejected": 0,
|
||||||
|
"replica_bytes": 0,
|
||||||
|
"replica_disconnects": 0,
|
||||||
|
"replica_errors": 0
|
||||||
|
}
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"throughput_unit": "MiB/s aggregate upload+echo download (replica bytes excluded)"
|
||||||
|
}
|
||||||
46
docs/benchmarks/linux-strict.json
Normal file
46
docs/benchmarks/linux-strict.json
Normal file
@ -0,0 +1,46 @@
|
|||||||
|
{
|
||||||
|
"arch": "x86_64",
|
||||||
|
"byte_integrity": "verified every primary reply byte",
|
||||||
|
"direct": [
|
||||||
|
2637.244957551585,
|
||||||
|
2315.9188718420974,
|
||||||
|
2342.568421299218
|
||||||
|
],
|
||||||
|
"direct_latency_us": {
|
||||||
|
"p50": 10.458,
|
||||||
|
"p95": 18.042
|
||||||
|
},
|
||||||
|
"mode": "strict",
|
||||||
|
"payload_mib_per_run": 64,
|
||||||
|
"platform": "linux",
|
||||||
|
"proxy": [
|
||||||
|
2230.0006658642615,
|
||||||
|
1984.2633653236892,
|
||||||
|
2317.5052852246995
|
||||||
|
],
|
||||||
|
"proxy_latency_us": {
|
||||||
|
"p50": 32.458,
|
||||||
|
"p95": 51.876000000000005
|
||||||
|
},
|
||||||
|
"stats": {
|
||||||
|
"bench": {
|
||||||
|
"last_error": null,
|
||||||
|
"running": false,
|
||||||
|
"stats": {
|
||||||
|
"accept_errors": 0,
|
||||||
|
"accepted": 5,
|
||||||
|
"active": 0,
|
||||||
|
"bytes_down": 205536896,
|
||||||
|
"bytes_up": 205536896,
|
||||||
|
"completed": 5,
|
||||||
|
"connect_errors": 0,
|
||||||
|
"io_errors": 0,
|
||||||
|
"rejected": 0,
|
||||||
|
"replica_bytes": 205536896,
|
||||||
|
"replica_disconnects": 0,
|
||||||
|
"replica_errors": 0
|
||||||
|
}
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"throughput_unit": "MiB/s aggregate upload+echo download (replica bytes excluded)"
|
||||||
|
}
|
||||||
46
docs/benchmarks/macos-isolate.json
Normal file
46
docs/benchmarks/macos-isolate.json
Normal file
@ -0,0 +1,46 @@
|
|||||||
|
{
|
||||||
|
"arch": "aarch64",
|
||||||
|
"byte_integrity": "verified every primary reply byte",
|
||||||
|
"direct": [
|
||||||
|
3891.277346141554,
|
||||||
|
3944.1502166586424,
|
||||||
|
4040.2180985483683
|
||||||
|
],
|
||||||
|
"direct_latency_us": {
|
||||||
|
"p50": 18.375,
|
||||||
|
"p95": 28.416
|
||||||
|
},
|
||||||
|
"mode": "isolate",
|
||||||
|
"payload_mib_per_run": 64,
|
||||||
|
"platform": "macos",
|
||||||
|
"proxy": [
|
||||||
|
2878.0375381536714,
|
||||||
|
2712.836717346342,
|
||||||
|
2771.545109332583
|
||||||
|
],
|
||||||
|
"proxy_latency_us": {
|
||||||
|
"p50": 46.25,
|
||||||
|
"p95": 80.792
|
||||||
|
},
|
||||||
|
"stats": {
|
||||||
|
"bench": {
|
||||||
|
"last_error": null,
|
||||||
|
"running": false,
|
||||||
|
"stats": {
|
||||||
|
"accept_errors": 0,
|
||||||
|
"accepted": 5,
|
||||||
|
"active": 0,
|
||||||
|
"bytes_down": 205536896,
|
||||||
|
"bytes_up": 205536896,
|
||||||
|
"completed": 5,
|
||||||
|
"connect_errors": 0,
|
||||||
|
"io_errors": 0,
|
||||||
|
"rejected": 0,
|
||||||
|
"replica_bytes": 18920268,
|
||||||
|
"replica_disconnects": 3,
|
||||||
|
"replica_errors": 0
|
||||||
|
}
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"throughput_unit": "MiB/s aggregate upload+echo download (replica bytes excluded)"
|
||||||
|
}
|
||||||
46
docs/benchmarks/macos-none.json
Normal file
46
docs/benchmarks/macos-none.json
Normal file
@ -0,0 +1,46 @@
|
|||||||
|
{
|
||||||
|
"arch": "aarch64",
|
||||||
|
"byte_integrity": "verified every primary reply byte",
|
||||||
|
"direct": [
|
||||||
|
2818.472719397018,
|
||||||
|
4080.629541672935,
|
||||||
|
4023.148821146676
|
||||||
|
],
|
||||||
|
"direct_latency_us": {
|
||||||
|
"p50": 18.5,
|
||||||
|
"p95": 33.708999999999996
|
||||||
|
},
|
||||||
|
"mode": "none",
|
||||||
|
"payload_mib_per_run": 64,
|
||||||
|
"platform": "macos",
|
||||||
|
"proxy": [
|
||||||
|
2747.282229591234,
|
||||||
|
3387.4607213343543,
|
||||||
|
2818.8037287664247
|
||||||
|
],
|
||||||
|
"proxy_latency_us": {
|
||||||
|
"p50": 33.875,
|
||||||
|
"p95": 47.916
|
||||||
|
},
|
||||||
|
"stats": {
|
||||||
|
"bench": {
|
||||||
|
"last_error": null,
|
||||||
|
"running": false,
|
||||||
|
"stats": {
|
||||||
|
"accept_errors": 0,
|
||||||
|
"accepted": 5,
|
||||||
|
"active": 0,
|
||||||
|
"bytes_down": 205536896,
|
||||||
|
"bytes_up": 205536896,
|
||||||
|
"completed": 5,
|
||||||
|
"connect_errors": 0,
|
||||||
|
"io_errors": 0,
|
||||||
|
"rejected": 0,
|
||||||
|
"replica_bytes": 0,
|
||||||
|
"replica_disconnects": 0,
|
||||||
|
"replica_errors": 0
|
||||||
|
}
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"throughput_unit": "MiB/s aggregate upload+echo download (replica bytes excluded)"
|
||||||
|
}
|
||||||
46
docs/benchmarks/macos-strict.json
Normal file
46
docs/benchmarks/macos-strict.json
Normal file
@ -0,0 +1,46 @@
|
|||||||
|
{
|
||||||
|
"arch": "aarch64",
|
||||||
|
"byte_integrity": "verified every primary reply byte",
|
||||||
|
"direct": [
|
||||||
|
4152.086382859668,
|
||||||
|
4042.222274855325,
|
||||||
|
4098.185611564874
|
||||||
|
],
|
||||||
|
"direct_latency_us": {
|
||||||
|
"p50": 19.083,
|
||||||
|
"p95": 34.958
|
||||||
|
},
|
||||||
|
"mode": "strict",
|
||||||
|
"payload_mib_per_run": 64,
|
||||||
|
"platform": "macos",
|
||||||
|
"proxy": [
|
||||||
|
1353.7697283381826,
|
||||||
|
1579.6862224827607,
|
||||||
|
1531.654695628502
|
||||||
|
],
|
||||||
|
"proxy_latency_us": {
|
||||||
|
"p50": 45.0,
|
||||||
|
"p95": 69.25
|
||||||
|
},
|
||||||
|
"stats": {
|
||||||
|
"bench": {
|
||||||
|
"last_error": null,
|
||||||
|
"running": false,
|
||||||
|
"stats": {
|
||||||
|
"accept_errors": 0,
|
||||||
|
"accepted": 5,
|
||||||
|
"active": 0,
|
||||||
|
"bytes_down": 205536896,
|
||||||
|
"bytes_up": 205536896,
|
||||||
|
"completed": 5,
|
||||||
|
"connect_errors": 0,
|
||||||
|
"io_errors": 0,
|
||||||
|
"rejected": 0,
|
||||||
|
"replica_bytes": 205536896,
|
||||||
|
"replica_disconnects": 0,
|
||||||
|
"replica_errors": 0
|
||||||
|
}
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"throughput_unit": "MiB/s aggregate upload+echo download (replica bytes excluded)"
|
||||||
|
}
|
||||||
@ -21,7 +21,7 @@ pub struct EngineStatus {
|
|||||||
#[derive(Clone, Debug, Default, serde::Serialize)]
|
#[derive(Clone, Debug, Default, serde::Serialize)]
|
||||||
pub struct TrafficStats {
|
pub struct TrafficStats {
|
||||||
pub accepted: u64, pub active: u64, pub completed: u64,
|
pub accepted: u64, pub active: u64, pub completed: u64,
|
||||||
pub connect_errors: u64, pub io_errors: u64, pub rejected: u64,
|
pub connect_errors: u64, pub io_errors: u64, pub rejected: u64, pub accept_errors: u64,
|
||||||
pub bytes_up: u64, pub bytes_down: u64,
|
pub bytes_up: u64, pub bytes_down: u64,
|
||||||
pub replica_disconnects: u64, pub replica_errors: u64, pub replica_bytes: u64,
|
pub replica_disconnects: u64, pub replica_errors: u64, pub replica_bytes: u64,
|
||||||
}
|
}
|
||||||
|
|||||||
@ -1,5 +1,7 @@
|
|||||||
# Async TCP engine implementation evidence
|
# Async TCP engine implementation evidence
|
||||||
|
|
||||||
|
Historical implementation-stage evidence. Final integrated results, accept supervision and quiesce follow-ups are recorded in `docs/verification.md` and `docs/backend-fixes-report.md`; the11-test count below predates those additions.
|
||||||
|
|
||||||
2026-10-23. Scope: `src/engine.rs`, `tests/forwarding.rs`, and the engine module export. All fixtures use local loopback sockets; no legacy credentials, game hosts, or payload logs were used.
|
2026-10-23. Scope: `src/engine.rs`, `tests/forwarding.rs`, and the engine module export. All fixtures use local loopback sockets; no legacy credentials, game hosts, or payload logs were used.
|
||||||
|
|
||||||
## Implementation
|
## Implementation
|
||||||
|
|||||||
83
docs/final-review.md
Normal file
83
docs/final-review.md
Normal file
@ -0,0 +1,83 @@
|
|||||||
|
# Final independent rewrite review — 2026-10-23
|
||||||
|
|
||||||
|
## Verdict
|
||||||
|
|
||||||
|
**Approved on source and targeted-regression review: no remaining Critical or Important findings.** R1–R6 are addressed in the reviewed source. The restored UI uses the legacy foundation, and the forwarding design meets the requested internal upstream-selection model. R6 was independently reproduced before repair and independently verified after repair. The parent owns the final integrated formatting/Clippy/native/Orb/release gates and completion of the project evidence log.
|
||||||
|
|
||||||
|
This report reviews local base `4b01fb47b1e8afcf2f985d0bb76d6c245d8637b4` on `rewrite/rust` plus the dirty implementation described by `/tmp/mirror-final-review.patch` and `/tmp/mirror-final-review-untracked.txt`. The historical R6 finding below identifies source before repair; the resolution section reviews the current cancellation-safe implementation in `/tmp/mirror-r6-review.patch`. There is no remote deployment or production-target validation in this review.
|
||||||
|
|
||||||
|
## Resolved Important finding
|
||||||
|
|
||||||
|
### R6 — HTTP disconnect cancelled a partially committed control transaction
|
||||||
|
|
||||||
|
- **Locations:** `src/web.rs:201` and `src/web.rs:207` await controller changes directly in the HTTP handler; `src/control.rs:267` persists deletion, `src/control.rs:268` awaits engine retirement, and `src/control.rs:269` publishes the new in-memory topology. Related mutation boundaries are `src/control.rs:216`, `src/control.rs:217`, `src/control.rs:232`, and `src/control.rs:291`.
|
||||||
|
- **Reproduction:** Start a loopback proxy with `drain_timeout_ms=2000`; hold an established client/upstream stream open; disable the service; send authenticated `DELETE /api/services/echo`; wait until the state file records zero services; close the DELETE HTTP socket before its response; wait 2.4 seconds; fetch `/api/snapshot`.
|
||||||
|
- **Pre-fix result, reproduced twice against the then-current debug executable:** `After disconnected DELETE: disk services 0 live services 1 active after drain deadline 1`. The second run explicitly checked the surviving active-session counter. The diagnostic script is `/tmp/mirror-review-cancel.py`; it uses a temporary directory, ephemeral loopback ports and a public fixture password, and cleans up the subprocess. No real credentials or game hosts are involved.
|
||||||
|
- **Cause:** Hyper drops the handler future when the peer disconnects. The transaction mutex guard is owned by that future, so it is dropped before the transaction completes. In the proven DELETE case the disk commit has happened, but engine retirement and topology publication are abandoned. The active session remains beyond the configured removal deadline because its retirement future no longer runs. PUT service/team changes have the same cancellation-sensitive structure around the detached blocking state writer; those specific timing windows were source-reviewed, not independently induced.
|
||||||
|
- **Impact/blast radius:** All three authenticated mutation endpoints can lose transactional ownership on client/network failure. A subsequent ordinary mutation can persist the stale topology again. This violates authoritative saved state and bounded retirement, despite correct rollback when persistence returns an ordinary error. It does not require an unauthenticated attacker: closing a browser tab or losing the operator connection is sufficient.
|
||||||
|
- **History and coverage:** The direct-handler transaction structure exists in initial commit `4b01fb4`; the shutdown serialization changes do not remove it. Existing tests cover storage failure, concurrent shutdown and body-extractor stalls, but do not disconnect a client after a mutation starts.
|
||||||
|
- **Repair acceptance criterion:** Keep admitted transactions owned independently of the HTTP request until commit/rollback and cleanup finish. Preserve serialization and shutdown admission ordering: shutdown must reject new work and coordinate every admitted transaction before normal teardown, subject to the existing overall process deadline. Merely moving persistence to another blocking task does not protect the rest of the transaction.
|
||||||
|
- **Regression acceptance criterion:** A real HTTP disconnect after DELETE's disk commit must still result in matching disk/snapshot topology and zero active sessions within the drain allowance. Add cancellation coverage for PUT persistence/rollback as appropriate to the ownership boundary. Re-run the existing shutdown/quiesce/storage regressions after changing transaction ownership.
|
||||||
|
|
||||||
|
### R6 resolution and independent acceptance
|
||||||
|
|
||||||
|
**Resolved.** `Controller::transaction` (`src/control.rs:191`) acquires the shared owned operation mutex and checks stopping before it spawns a transaction worker. There is no await/cancellation point between admission and transfer of the guard/controller Arc into that worker. Dropping the HTTP handler's JoinHandle detaches the transaction; apply, save, publication, rollback and stream retirement continue under one guard. Callers waiting on admission remain cancellable and do not allocate a detached worker. The controller retains only a Weak self-reference outside an active worker, avoiding a permanent strong cycle.
|
||||||
|
|
||||||
|
Manual service PUT/DELETE and team PUT use this wrapper; automatic promotion also owns its admitted selection/mutation phase. Shutdown first sets stopping and quiesces acceptance, then waits for the same transaction mutex before draining. Engine quiescence prevents an in-flight commit/rollback from reopening listeners. Cancellation before admission cannot mutate state. The existing overall process deadline still takes precedence over a stalled transaction; the repair does not promise completion through forced termination, panic or machine failure.
|
||||||
|
|
||||||
|
Fresh independent post-fix checks:
|
||||||
|
|
||||||
|
- Re-ran the original actual HTTP/socket-disconnect script against the rebuilt debug executable: `After disconnected DELETE: disk services 0 live services 0 active after drain deadline 0`.
|
||||||
|
- Ran `cargo test --lib cancellation_tests --locked`: **5 passed, 0 failed**. These cover DELETE cleanup with a real held TCP stream, service PUT after save, service PUT rollback after a failing save, team PUT after save and automatic promotion after save. The PUT success fixture also verifies exact loopback payload delivery; DELETE verifies stream closure and removal from the engine registry.
|
||||||
|
- Inspected the shutdown/quiesce tests and wrapper integration. The parent's final suite supplies the fresh whole-suite shutdown/storage checks; this reviewer did not rerun the full suite.
|
||||||
|
|
||||||
|
The five regressions included in the repository test suite cancel the controller caller task at deterministic transaction boundaries. The actual HTTP-disconnect scenario was independently executed as a local script and is **not included as an automated HTTP regression test**. The combined source inspection, deterministic cancellation tests and actual HTTP red/green evidence are sufficient for this finding's acceptance; this distinction is retained explicitly. README and `docs/api.md` correctly tell operators that a lost response can follow a committed mutation and to refresh the snapshot before retrying.
|
||||||
|
|
||||||
|
## Critical and Minor findings
|
||||||
|
|
||||||
|
No Critical findings were identified. No additional Minor finding is raised as a release condition. This is a bounded source and targeted-runtime review, not proof that every operating-system or malicious-input failure mode has been exhausted.
|
||||||
|
|
||||||
|
## Resolution assessment of R1–R5
|
||||||
|
|
||||||
|
| Finding | Assessment | Source and evidence |
|
||||||
|
|---|---|---|
|
||||||
|
| R1: incomplete HTTP body blocks process shutdown | Addressed | `src/main.rs:83` starts shutdown independently of HTTP graceful completion; the shared async deadline is drain + 1000 ms and explicit runtime teardown adds a 100 ms wait. `src/control.rs:102` marks stopping, quiesces before acquiring the controller transaction mutex, then drains. `tests/shutdown.rs` exercises the actual process with an incomplete body and live TCP peer. The distinct transaction-ownership defect R6 is resolved in the section above. |
|
||||||
|
| R2: publication-time health timestamps | Addressed | `observe_probe` captures `now_ms()` when each future finishes; publication copies `observed_at` unchanged. `delayed_batch_does_not_refresh_early_candidate_observation` traverses actual publication and candidate selection with a deliberately aged early result. |
|
||||||
|
| R3: wildcard Linux loopback/mapped IPv6 route loops | Addressed | `normalized_ip`, family-aware wildcard checks and `is_loopback()` reject aliases without relying solely on interface enumeration. Tests cover self/cross-service routes, mapped IPv6 and a native IPv6 target behind an IPv4-only listener. Linux has an actual 127.0.0.2-to-wildcard fixture. |
|
||||||
|
| R4: reversed event-journal window | Resolved by authorized UI restoration | The rejected journal is removed from the operator panel. The snapshot API continues to return the bounded newest-first backend events. There is no remaining `slice(-30).reverse()` rendering path. |
|
||||||
|
| R5: manual promotion loses former primary | Addressed | `web/app.js:87–92` removes the new primary from replicas and adds the former primary once, clearing the direct-IP override. This matches the legacy `selectPrimary` behavior. Replica rows still allow explicit removal. |
|
||||||
|
|
||||||
|
## Whole-rewrite assessment
|
||||||
|
|
||||||
|
| Binding requirement | Assessment |
|
||||||
|
|---|---|
|
||||||
|
| Native executable and module separation | Config, engine, control, store, auth/web and scoreboard are separate. Frontend assets are embedded; no runtime Python, Node or external CDN is required. |
|
||||||
|
| Internal upstream auto-selection only | No system route/firewall mutation is present. Auto mode selects only eligible configured replicas with cooldown, failure/recovery thresholds and fresh observations. |
|
||||||
|
| Existing streams and no replay | The supervisor captures an immutable route per accepted session. Retargeting updates future sessions; there is no reconnect/replay of partially sent application data. |
|
||||||
|
| Byte integrity, TCP half-close and idle | Primary directions are awaited together; EOF shuts down the matching write half without prematurely cancelling the opposite direction. Replica replies are drained concurrently. No short application idle timer was introduced. |
|
||||||
|
| Strict policy | Bounded channel/payload acquisition backpressures the client. Required-replica setup failure prevents payload forwarding; runtime replica errors cancel the entire primary session. Strict waiting on a slow live replica is explicitly documented. |
|
||||||
|
| Isolate policy | Queue/budget exhaustion cancels the complete affected replica instead of silently skipping a chunk and continuing its stream. Disconnect/error counters and bounded tail drain are present. Partial delivery to a disconnected replica is accurately documented. |
|
||||||
|
| Resource ownership | Global data-session permits, bounded replica queues, shared payload budget and owned nested worker joins are present. Working buffers and OS socket buffers are explicitly outside the payload budget. R6 now retains transaction/retirement ownership after caller cancellation. |
|
||||||
|
| Graceful shutdown/quiesce | Acceptance stops before waiting for a filesystem/control transaction; a quiesced upsert can restore bookkeeping but cannot reopen a listener. The process has a single outer shutdown deadline. The R6 ownership wrapper preserves these properties. |
|
||||||
|
| Persistence and ordinary rollback | Same-directory create-new temporary file, file sync, atomic rename, and attempted runtime rollback on precommit failure are implemented. Failed new direct-IP service creation removes its listener and does not publish its hidden team. The R6 wrapper now preserves ordinary transaction completion across HTTP cancellation. |
|
||||||
|
| ForcAD transport | Engine.IO 4 polling, namespace initialization, ping/pong, reconnect and distinct init/state updates are implemented. The absolute heartbeat deadline is renewed only by a ping; a continuously non-ping server cannot keep the connection alive indefinitely. Incoming bodies are bounded. |
|
||||||
|
| Transient accepts | Recoverable errors retry at 50 ms through a one-second cap, reset after success, and remain cancellation-aware. Counters/diagnostics distinguish accept failures. Injection tests use the production supervisor and real loopback transfer; actual OS descriptor exhaustion was not induced here. |
|
||||||
|
| Authentication and API | Cookie sessions, constant-time digest comparison, CSRF, browser origin checks, login throttling, restrictive response headers and body limits remain intact. Framework-error envelope differences are now documented. |
|
||||||
|
| Legacy UI foundation | The full original CSS block remains present after trailing-whitespace normalization; 24,037 original characters were checked independently. The original login/card/columns/controls/team selectors/custom dialog structure is retained with local dependency versions. Small accessibility, stale-state and layout additions are consistent with the user's scope. |
|
||||||
|
| Direct-IP workflow | Standard service PUT resolves/reuses the address atomically and creates a hidden target when needed. No manual team-creation step is required. Custom input/create, manual promotion and server validation are wired together. Parent's browser verification confirmed actual keyboard editing and persisted snapshots. |
|
||||||
|
| Claims and documentation | Documentation distinguishes request mirroring from database replication, local writes from application commitment, strict backpressure from isolate availability, and loopback benchmarks from production guarantees. Final journal/plan/verification completion remains the parent's responsibility. |
|
||||||
|
|
||||||
|
## Evidence and limits
|
||||||
|
|
||||||
|
The reviewer read the full production Rust modules, restored HTML/JS/CSS, changed-file patch inventory, relevant integration tests, API/deployment/README contracts, historical review, backend-fix report, UI report and task plan. Legacy `templates/base.html`, `index.html` and `login.html` were read only; the sibling configuration and credentials were not read or imported. The only repository file this reviewer writes is this report.
|
||||||
|
|
||||||
|
Fresh reviewer commands/evidence:
|
||||||
|
|
||||||
|
- Two pre-fix loopback subprocess reproductions of R6 and one post-fix reproduction confirming convergence, plus all five post-fix cancellation tests passing independently.
|
||||||
|
- Python normalized containment check: `Legacy CSS normalized containment: PASS 24037 characters`.
|
||||||
|
- `node --check web/app.js`: exit 0.
|
||||||
|
- `cargo metadata --format-version 1 --locked`: exit 0; no resolved package advertises an MSRV above the manifest's 1.85.
|
||||||
|
|
||||||
|
Parent-provided, separately executed pre-R6 integrated evidence: native format/Clippy, 35 native tests, 36 Orb amd64 tests including the Linux alias fixture, release builds on both platforms, real upstream-version Socket.IO/Engine.IO interoperability, three-mode byte-checked benchmarks and browser checks. The backend implementer reports post-R6 native formatting/Clippy/40 tests and Orb amd64/41 tests; the parent is running final integrated gates separately. These suites were deliberately not rerun wholesale by this reviewer. The parent reports the final browser direct-IP edit passed with actual keyboard input, 390px viewport had no document overflow, a seven-second SSE freeze disabled mutations, and resumed snapshots restored controls. These browser results are attributed to the parent, not represented as independently executed here.
|
||||||
|
|
||||||
|
No production scoreboard/game host, real low-fd exhaustion, indefinitely blocked kernel filesystem call, multi-process shared-state operation, or broad benchmark workload was tested by this reviewer. R6 targeted re-review is complete and approved; no new Critical/Important finding was found. Parent-integrated final build/test gates remain separately attributed.
|
||||||
84
docs/review.md
Normal file
84
docs/review.md
Normal file
@ -0,0 +1,84 @@
|
|||||||
|
# Independent review — 2026-10-23
|
||||||
|
|
||||||
|
Reviewed initial commit `4b01fb47b1e8afcf2f985d0bb76d6c245d8637b4`, the complete patch inventory in `/tmp/mirror-tool-review.patch`, repository requirements, implementation modules, integration tests, embedded UI, API/deployment documentation and benchmark. Line references below refer to that commit. No production targets were contacted; review reproductions used temporary configuration and loopback. This reviewer changed only this document.
|
||||||
|
|
||||||
|
## Verdict
|
||||||
|
|
||||||
|
**Changes required: five Important findings, no Critical finding.** The async architecture and the explicit strict/isolate delivery distinction match the requested rewrite, but the initial commit does not yet satisfy bounded process shutdown, trustworthy health freshness, complete proxy-loop prevention and two preserved operator workflows. Resolve R1–R5 and review their regression evidence before approval.
|
||||||
|
|
||||||
|
The parent reports 23 passing tests on native macOS and Orb amd64, plus clean Clippy. Those are parent-provided results, not reruns by this reviewer. They do not cover the defects below. The reviewer independently reproduced R1 with the built executable, verified R4 with a deterministic JavaScript example, and checked the remaining findings directly against source. UI, benchmark and Linux verification continue under the parent.
|
||||||
|
|
||||||
|
## Important findings
|
||||||
|
|
||||||
|
### R1 — An unfinished HTTP body prevents SIGTERM shutdown indefinitely
|
||||||
|
|
||||||
|
- **Location:** `src/main.rs:67–79`; `src/web.rs:123–130`.
|
||||||
|
- **Trigger:** Connect to the web listener, send `POST /api/login` with `Content-Type: application/json`, `Content-Length: 1000` and only `{` as its body; leave the socket open and send SIGTERM.
|
||||||
|
- **Evidence:** With `network.drain_timeout_ms=100`, the process remained alive after two seconds. Closing the incomplete HTTP socket immediately allowed exit status 0. No password or authenticated session is required for this reproduction.
|
||||||
|
- **Cause/impact:** The main task awaits Axum's unbounded graceful completion before calling `engine.shutdown()`. The body extractor is still waiting, so the engine's drain deadline never starts. Existing data listeners also continue accepting during this period. This contradicts the README's whole-process bounded shutdown guarantee and can force systemd to kill the process without completing the intended drain.
|
||||||
|
- **Fix:** On the signal, promptly stop data acceptance and establish one overall shutdown deadline. Bound HTTP graceful shutdown, stop/reject further control mutations, and coordinate already executing mutations before final engine teardown. Merely setting a short normal TCP idle timeout would violate the data-path requirements.
|
||||||
|
- **Regression:** A subprocess test holding an incomplete HTTP body and a live proxy session must prove listener closure and process termination within the documented shutdown bound, without closing the HTTP peer first.
|
||||||
|
|
||||||
|
### R2 — Health results receive the end-of-batch time, making old probes falsely fresh
|
||||||
|
|
||||||
|
- **Location:** `src/control.rs:254–272`, candidate freshness filter at `src/control.rs:387–389`.
|
||||||
|
- **Trigger:** With bounded monitoring concurrency, place a responsive candidate early in a probe cycle and enough slow endpoints after it that the whole cycle lasts longer than the allowed health age. Let that early candidate go down while the remaining probes finish.
|
||||||
|
- **Evidence:** Each future returns only `(address, ok, latency)`; all results are collected before `checked_at_ms = now_ms()` is assigned. Thus the early success is timestamped at publication rather than observation. This follows deterministically from the source; a network-timing reproduction was not run here.
|
||||||
|
- **Impact:** Automatic failover can promote a candidate whose last successful observation is already outside the freshness window. UI health also claims that the old result was just checked. For example, 640 unreachable endpoints at concurrency 32 and a one-second timeout can make early results roughly 20 seconds old, yet selection sees age zero against the default 12-second maximum.
|
||||||
|
- **Fix:** Capture the observation timestamp when each probe finishes and retain that timestamp when publishing. Consider streaming completed observations or probing relevant route targets first, but never refresh timestamps to compensate for a slow cycle.
|
||||||
|
- **Regression:** Exercise a fast candidate followed by delayed probes with a controlled clock/test seam; the candidate must be rejected once its actual observation ages out.
|
||||||
|
|
||||||
|
### R3 — Wildcard listener loop prevention misses Linux loopback aliases
|
||||||
|
|
||||||
|
- **Location:** `src/config.rs:247–255`, `src/config.rs:314–319`.
|
||||||
|
- **Trigger:** Configure a team at `127.0.0.2`, a service listener at `0.0.0.0:P`, and that service's upstream port at `P`; select that team as primary. On Linux the full `127.0.0.0/8` is loopback, even when interface enumeration lists only `127.0.0.1`.
|
||||||
|
- **Evidence:** The wildcard test requires membership in the enumerated address set. No `is_loopback()` check covers `127.0.0.2`. The parent has been asked to supply the Orb reproduction; this reviewer did not run Linux commands.
|
||||||
|
- **Impact:** The proxy recursively connects back to its own listener, consuming the global connection budget and disrupting other services. Monitoring can initiate the recursion too. A finite connection limit bounds the damage but does not make this valid topology.
|
||||||
|
- **Fix:** Account for the loopback range and IPv4-mapped IPv6 equivalence when checking routes against managed listeners, while respecting socket address-family behavior. Reject the loop before binding/starting.
|
||||||
|
- **Regression:** In Orb, validate both ordinary `127.0.0.1` and non-enumerated `127.0.0.2` wildcard self-routes and cross-service loops. Use loopback fixtures only.
|
||||||
|
|
||||||
|
### R4 — The operator journal hides the newest events after the thirtieth entry
|
||||||
|
|
||||||
|
- **Location:** `web/app.js:268`; producer order is `src/control.rs:99–110`.
|
||||||
|
- **Trigger:** Produce more than 30 events, then perform a manual change or automatic failover.
|
||||||
|
- **Evidence:** The controller inserts newest events at the front, but the UI uses `slice(-30).reverse()`. For descending event IDs `[100,99,...,1]`, the rendered selection contains IDs 1–30 and omits ID 100 (independently checked with Node).
|
||||||
|
- **Impact:** The operator cannot see recent route changes or errors. Once the server retains 100 events, the displayed window trails by 70 entries.
|
||||||
|
- **Fix:** Select the first 30 entries and choose the intended display order explicitly.
|
||||||
|
- **Regression:** Supply a snapshot with at least 35 distinguishable events; the most recent event must be visible and older entries outside the selected window absent.
|
||||||
|
|
||||||
|
### R5 — Quick primary promotion silently removes the old primary from the mirrored set
|
||||||
|
|
||||||
|
- **Location:** `web/app.js:179`; compare automatic promotion in `src/control.rs:327–333`.
|
||||||
|
- **Trigger:** Start with primary A and replica B, then click B's primary radio.
|
||||||
|
- **Evidence:** The UI sends primary B and replicas `[]`: it removes B from the existing replica list but never adds A. The legacy `selectPrimary` function in `../ad_scripts/mirror_tool/templates/index.html:259–270` explicitly adds the previous primary. The controller's automatic promotion also adds it. Legacy source was read only.
|
||||||
|
- **Impact:** A routine manual promotion stops mirroring to A and leaves no automatic failover candidate in the common two-server setup. This is an unintended workflow regression from the requested UI transfer, rather than merely cosmetic behavior.
|
||||||
|
- **Fix:** Preserve the previous primary as a replica in the quick promotion action, excluding the new primary and deduplicating. Keep the full edit form's explicitly selected replica list authoritative so an operator can intentionally remove a target.
|
||||||
|
- **Regression:** The quick action must send primary B, replicas `[A]`; subsequent snapshots must confirm the saved selection. An explicit full-form edit must still be able to choose zero replicas.
|
||||||
|
|
||||||
|
## Non-blocking observations
|
||||||
|
|
||||||
|
- **Transient accept errors:** `src/engine.rs:424` permanently ends a listener for every accept error, including resource-pressure/transient OS errors. Consider bounded cancellation-aware retry for recoverable errors. A low-fd loopback stress attempt here did **not** reproduce this branch: upstream connect failures occurred first and the listener recovered. This is source-backed resilience advice, not a claimed runtime failure.
|
||||||
|
- **Heartbeat deadline:** `src/scoreboard.rs:327–378` responds to Engine.IO pings and bounds each polling request, but does not track an absolute deadline since the last ping. A broken server returning non-ping frames continuously can keep the session connected beyond the negotiated heartbeat window. Freshness of scoreboard data remains separately enforced. An absolute heartbeat timer would improve protocol fidelity; no production failure was demonstrated.
|
||||||
|
- **API error envelope:** `docs/api.md` says all failures are JSON `{error:...}`, but Axum's default method-not-allowed/path-extraction responses are not normalized by the JSON error helper. Clarify the contract or normalize these framework responses. Normal documented mutations already translate JSON/body rejection into a JSON error.
|
||||||
|
|
||||||
|
## Specification and quality assessment
|
||||||
|
|
||||||
|
| Requirement | Initial review assessment |
|
||||||
|
|---|---|
|
||||||
|
| Native async Rust executable with embedded UI | Implemented; clean module boundaries and no runtime CDN/build requirement |
|
||||||
|
| Internal upstream selection, no kernel routing | Implemented; no host firewall/route mutations found |
|
||||||
|
| Existing sessions retain target; no partial replay | Route is cloned at acceptance; no reconnect/replay path found |
|
||||||
|
| TCP byte order, half-close and long idle | Both directions complete independently; EOF shuts down the opposite write half; no inherited short idle timeout found |
|
||||||
|
| Strict bounded backpressure | Bounded channels and shared payload semaphore; strict replica failure cancels the session |
|
||||||
|
| Isolate without chunk-drop continuation | Queue/budget pressure cancels the entire replica and increments counters; no continuation after dropping an arbitrary chunk found |
|
||||||
|
| Replica response draining and task ownership | Concurrent response drain and nested worker joining are present |
|
||||||
|
| Bounded shutdown | Engine-level drain is bounded; process-level behavior fails R1 |
|
||||||
|
| Fresh health and stable deterministic failover | Thresholds, cooldown, fresh checker gating and deterministic ordering are present; observation timestamps fail R2 |
|
||||||
|
| Validated topology/loop prevention | Validation is extensive; Linux wildcard self-route gap remains R3 |
|
||||||
|
| ForcAD init/state updates and reconnect | Metadata/state shapes are distinct, persistent polling/pong/reconnect implemented; out-of-order rounds rejected |
|
||||||
|
| Durable topology and rollback | Same-directory temporary write/fsync/rename; operation serialization and runtime rollback on pre-commit storage failure are present; short rollback window is accurately documented |
|
||||||
|
| Authenticated controls and operator safety | Random HttpOnly/SameSite sessions, CSRF, origin check, login limits, textContent rendering and restrictive CSP are present; no embedded production secrets found in reviewed material |
|
||||||
|
| Preserved UI workflow and operational evidence | Core service/team/route/statistics views present; latest-event display and quick primary promotion fail R4/R5 |
|
||||||
|
| Performance claims | Benchmark checks bytes and compares loopback paths; production and zero-overhead guarantees are correctly avoided. Timing results must come from the parent's fresh runs |
|
||||||
|
|
||||||
|
Overall, this is a coherent rewrite with substantive fixes for the legacy stream corruption, timeout and replica-response problems. Approval is withheld specifically for R1–R5, not for the optional observations. Re-review should inspect the final patch and fresh targeted regressions, then the required formatting, Clippy and all-target tests recorded by the parent.
|
||||||
@ -1,21 +1,23 @@
|
|||||||
# Operator UI implementation
|
# Legacy UI restoration
|
||||||
|
|
||||||
The UI preserves the service-column workflow and compact per-team primary/replica controls. Russian labels match the previous operator surface. The visual direction is a navy instrument panel, warm route markers, lavender utility accents and restrained status colors; no animation, external fonts, CDN or frontend build is needed.
|
The initial redesign was rejected. The implemented panel now uses the original `templates/base.html` CSS rules (only trailing whitespace normalized), the original Bootstrap 5.1.3 and Font Awesome Free 6.0.0 dependencies, and the original page structure. All assets, including the icon font and their licenses, are embedded in the executable. The reference templates and legacy directories are not modified.
|
||||||
|
|
||||||
## Contracts and behavior
|
Preserved interactions:
|
||||||
|
|
||||||
- `/` loads only `/app.css` and deferred `/app.js`; no inline handlers, inline styles or scripts. All dynamic names, addresses, events and errors are assigned as text nodes.
|
- Original login card and password input.
|
||||||
- Bootstrap checks `/api/session`, then reads `/api/snapshot` and opens one `/api/events` EventSource. Login uses the documented password body. Authenticated mutations include the returned CSRF token.
|
- Horizontal 360px service columns, header controls, selected-team line and scrollable team list.
|
||||||
- Service start/stop, primary/replica selection, automatic selection and replica policy use PUT with the complete latest ServiceConfig. Successful mutations refresh the snapshot; failures expose the backend message and reconcile controls with the last confirmed snapshot.
|
- Primary circle selects upstream; clicking the rest of a team row toggles its replica state. Promoting B from primary A/replica B yields primary B/replica A.
|
||||||
- Service dialog supports all configuration fields, optional primary while stopped and add/edit. Existing service ID is immutable in the editor; duplicate add IDs are rejected locally. Team add also permits a rename with the same ID/address; changing an existing address is rejected. Service deletion is disabled while enabled/running and requires an explicit confirmation with name, listener and route details.
|
- Robot toggles automatic switching; open icon/header opens the service port; original switch starts/stops forwarding.
|
||||||
- Service and team DOM nodes are keyed and updated in place. SSE never rebuilds open modal fields or focused route controls. Write controls are locked during mutations, while snapshots continue to update telemetry.
|
- Four original connection counters; original checker colors and negative lost-flag chips; status/lost/name sorting.
|
||||||
- Bandwidth uses byte-counter deltas divided by snapshot timestamp deltas, separately for upstream and downstream. Counter decreases and backwards timestamps reset the baseline; duplicate timestamps retain the previous rate. Absolute primary and replica bytes remain visible.
|
- Top strip represents the configured own-team `main_team_ip`, independently of primary selection.
|
||||||
- SSE interruption immediately marks the panel stale; lack of snapshots for eight seconds does the same. Session checks after disruption use a bounded 2–30 second backoff and stop while the stream is healthy. EventSource handles stream reconnects; 401 returns to login and closes the stream. Page lifecycle closes/reopens the stream without duplicating it.
|
- Plus opens name/port/optional-IP dialog. Custom columns retain editable direct-IP input and delete control. The backend atomically resolves the IP, with no manual team-creation step.
|
||||||
- TCP health is unknown until checked, with old checks marked stale after 15 seconds. TCP reachability is separate from checker health. ForcAD stale/disconnected state is explicit globally and on checker badges; stolen/lost counters are shown alongside status.
|
|
||||||
- Native dialogs provide focus containment and Escape dismissal (disabled during save), controls have visible focus, table headers/captions and descriptive labels, notices use status/alert roles, layout fits mobile widths, and reduced-motion preferences are respected.
|
|
||||||
|
|
||||||
## Verification and remaining integration work
|
Small additions: keyboard-operable selectors, readable control labels, focus indicators, reduced-motion support, narrower-screen sizing, stale-data indication and a five-second stream watchdog. Clicking the service name opens advanced settings without replacing the main layout. Forms retain drafts during snapshot updates. Mutations use the latest route selection, wait for a server result, and refresh the snapshot. Snapshot/CSRF/session contracts remain in `docs/api.md`.
|
||||||
|
|
||||||
`node --check web/app.js` passes. Source inspection confirms one EventSource creation site, no interval polling, no `innerHTML`, no inline event/style attributes and no external asset URLs. The UI implementation did not modify backend, Cargo files or legacy templates and did not contact game hosts.
|
The rejected diagnostic journal is removed from the panel, so initial review R4's incorrect journal slicing is no longer present. Backend events remain available through the snapshot API. R5 is addressed by the restored primary-promotion interaction.
|
||||||
|
|
||||||
Real browser checks against the integrated backend, CSP response headers, authentication failures, modal focus under live updates, mutation errors, mobile layout, and stream/session recovery are delegated to the integrating parent task. Rust format/clippy/test gates and Orb checks belong to that integration; this report does not claim they have passed.
|
## Verification pointers
|
||||||
|
|
||||||
|
`scripts/legacy_ui_reference.py` renders the unmodified old templates with public loopback data and checks that their normalized CSS rules are still present. It never imports old `config.py`. Browser comparison uses the same viewport and bundled dependency versions. Computed styles matched for column width, typography, backgrounds, radii, header/stat padding and gaps. The intentional height change removes the old 8px overflow; a small mobile rule allows a column to fit a narrow viewport.
|
||||||
|
|
||||||
|
Fresh browser/byte checks and final screenshots are recorded in `docs/verification.md`; `test-results/` holds local disposable artifacts and is git-ignored. UI controls were exercised against the real Rust server with isolated temporary state and loopback upstreams. No legacy game hosts were contacted.
|
||||||
|
|||||||
85
docs/verification.md
Normal file
85
docs/verification.md
Normal file
@ -0,0 +1,85 @@
|
|||||||
|
# Verification — 2026-10-23
|
||||||
|
|
||||||
|
All network checks used synthetic loopback endpoints. The legacy sibling and Downloads copy were references only; no credentials were copied, no old game endpoints contacted, no host firewall/routing changed and nothing deployed over the legacy service.
|
||||||
|
|
||||||
|
## Platforms and gates
|
||||||
|
|
||||||
|
| Platform | Toolchain | Final fresh checks |
|
||||||
|
|---|---|---|
|
||||||
|
| macOS arm64 | Rust/Cargo1.98.1 | fmt, Clippy all targets with warnings denied,40 tests, release binary and examples |
|
||||||
|
| Orb Ubuntu Linux amd64 | Rust/Cargo1.93.1 |41 tests, release binary and examples; extra test proves Linux127/8 wildcard behavior |
|
||||||
|
|
||||||
|
The declared Rust1.85 minimum was not independently built; use a current stable toolchain. Orb Cargo does not provide rustfmt, so formatting and Clippy are checked natively. Final R6 fix is included in these fresh root runs; independent re-review is recorded in `docs/final-review.md`.
|
||||||
|
|
||||||
|
Reproduce:
|
||||||
|
|
||||||
|
```sh
|
||||||
|
cargo fmt --check
|
||||||
|
cargo clippy --all-targets --locked -- -D warnings
|
||||||
|
cargo test --all-targets --locked
|
||||||
|
cargo build --release --bin mirror-tool --examples --locked
|
||||||
|
node --check web/app.js
|
||||||
|
```
|
||||||
|
|
||||||
|
Forwarding integration tests exercise byte-exact binary streams, request EOF with delayed response, idle beyond3s, target changes affecting new sessions, disable/re-enable and port reuse, strict mirror request integrity with large replica responses drained, failed/slow strict and isolate replicas, global connection limits, quiesce and bounded shutdown. Control tests cover authenticated same-origin/CSRF controls, durable restart, direct-IP target resolution, automatic selection, state-write rollback and health freshness. `tests/shutdown.rs` runs the actual process with an unfinished HTTP body and a live TCP session.
|
||||||
|
|
||||||
|
Fault-injection and negative controls for shutdown, observation timestamps, loop aliases, missing heartbeat, accept retry and early quiesce are detailed in `docs/backend-fixes-report.md`. Recoverable accept errors were injected into the actual private listener supervisor; OS descriptor exhaustion itself was not induced. Indefinitely blocked kernel filesystem calls were not induced.
|
||||||
|
|
||||||
|
## Legacy UI
|
||||||
|
|
||||||
|
`scripts/legacy_ui_reference.py` renders the unmodified original Jinja templates with synthetic data, without executing/importing legacy configuration. It confirms all24037 original CSS characters remain, modulo trailing whitespace normalization. Bootstrap5.1.3/Font Awesome6.0.0 and icon font are bundled locally with licenses. Computed styles matched original columns, font, colors, border radius, header/stat padding and gaps. The intentional small changes are listed in `docs/ui-report.md`.
|
||||||
|
|
||||||
|
Chrome against the release executable and temporary loopback fixture verified:
|
||||||
|
|
||||||
|
- Login/logout; service start; robot toggle; choosing primaryB retains primaryA as replica.
|
||||||
|
- A1MiB byte-exact primary echo and correct half-close through the UI-started proxy. Replica counter also reached1MiB, zero errors/disconnections.
|
||||||
|
- Original plus dialog creates a custom proxy directly from IPv6; advanced settings save listen/upstream ports and strict policy; editable direct-IP field updates primary without creating a team manually.
|
||||||
|
- Invalid self-loop custom address is rejected and add-dialog input remains (earlier restoration pass). Settings drafts survive SSE updates.
|
||||||
|
- Pausing the fixture process for seven seconds produces the five-second stale warning, disables mutations and marks TCP health unknown. Resuming fresh SSE clears the warning and restores controls.
|
||||||
|
- Actual390x844 viewport: innerWidth390, document width390, column width360; horizontal service-board scrolling retained. Browser-wide viewport setting was ineffective, so tab-scoped CDP metrics were used and cleared afterward.
|
||||||
|
|
||||||
|
Local disposable screenshots: `test-results/legacy-ui.jpg`, `test-results/restored-ui-final.jpg`, `test-results/restored-ui-mobile.jpg`. No app-origin JavaScript error was observed; browser-extension messages were excluded. Browser Use programmatic fill did not set native dirty-field state for change-on-blur; direct-IP saving was verified with real keyboard input and Tab.
|
||||||
|
|
||||||
|
## ForcAD interoperability
|
||||||
|
|
||||||
|
Canonical upstream inspected: `pomo-mondreganto/ForcAD` at `6ce3407be569b7f69f879501c5e2ca79231c1ff9`. Beyond wire fixtures, the final release was checked with actual python-socketio5.10.0/python-engineio4.8.0, matching upstream versions:
|
||||||
|
|
||||||
|
```sh
|
||||||
|
python scripts/verify_forcad_library.py target/release/mirror-tool
|
||||||
|
```
|
||||||
|
|
||||||
|
The optional dev environment needs `aiohttp`, `python-socketio==5.10.0`, `python-engineio==4.8.0`. Passed: initial metadata/state, delta updates retaining metadata, heartbeat with reused session, stale data, namespace disconnect and reconnect (two handshakes). This is interoperability with the inspected protocol, not a live tournament or arbitrary fork certification. Adplatf parsing is covered by synthetic fixtures, not a live deployment.
|
||||||
|
|
||||||
|
## Measured loopback performance
|
||||||
|
|
||||||
|
Three alternating direct/proxy transfers of64MiB per mode,500 sequential32-byte echo latency samples per path. Every primary response byte checked. Mirror modes use one real IPv6 replica plus IPv4 primary. Default engine limits, same-process runtime, warmup; aggregate upload+echo-download throughput excludes replica bytes. Numbers are local baselines, not one-way link capacity or a claim of improvement over the Python implementation. Both platforms ran on the same host; scheduling and virtualization affect results.
|
||||||
|
|
||||||
|
| Platform | Mode | Direct median MiB/s | Proxy median MiB/s | Direct p50 µs | Proxy p50 µs | Replica disconnects |
|
||||||
|
|---|---|---:|---:|---:|---:|---:|
|
||||||
|
| macOS arm64 | none |4023|2819|18.5|33.9|0|
|
||||||
|
| macOS arm64 | strict |4098|1532|19.1|45.0|0|
|
||||||
|
| macOS arm64 | isolate |3944|2772|18.4|46.2|3|
|
||||||
|
| Orb Linux amd64 | none |2521|2266|9.8|20.5|0|
|
||||||
|
| Orb Linux amd64 | strict |2343|2230|10.5|32.5|0|
|
||||||
|
| Orb Linux amd64 | isolate |2526|2268|10.4|34.0|4|
|
||||||
|
|
||||||
|
All primary byte totals matched; accept/connect/I/O/rejection/replica-error counters were zero. Strict replica local-write byte totals also matched all205536896 primary upload bytes (warmup, bulk and latency combined). Integration tests separately verify bytes actually read by the replica. In isolate mode the maximum-rate bursts saturated the default bounded queue and explicitly disconnected the replica: three disconnections on macOS and four on Linux across five proxy sessions per mode, with only a prefix mirrored for affected sessions. These are observed overload outcomes, not full-replica-delivery passes. Increase queue resources for expected bursts or select strict when every healthy replica must keep up; an indefinitely slower replica cannot maintain primary speed with finite memory and full delivery.
|
||||||
|
|
||||||
|
Raw records are checked in at `docs/benchmarks/{macos,linux}-{none,strict,isolate}.json`, including each throughput run, p50/p95 and all counters.
|
||||||
|
|
||||||
|
```sh
|
||||||
|
cargo build --release --example benchmark --locked
|
||||||
|
./target/release/examples/benchmark 64 none
|
||||||
|
./target/release/examples/benchmark 64 strict
|
||||||
|
./target/release/examples/benchmark 64 isolate
|
||||||
|
```
|
||||||
|
|
||||||
|
No zero-overhead/zero-loss claim is supported. Socket-write counters do not prove application commit; TCP mirroring is not database replication. Real-network load, multi-hour soak, live checker compatibility and systemd deployment remain environment-specific validation, not performed here.
|
||||||
|
|
||||||
|
## Final cancellation and release checks
|
||||||
|
|
||||||
|
The final controller admits at most one owned transaction worker after acquiring its operation lock. Five new automated regressions cancel callers during service save/publication, failed-save rollback, DELETE drain, team publication and automatic promotion. Native40 and Linux41 test counts include them. The original real-HTTP-disconnect DELETE reproduction independently changed from disk/live/active0/1/1 to0/0/0. The real HTTP reproduction is review evidence; automated committed regressions cancel at the transaction caller boundary rather than reproducing HTTP transport cancellation. Early quiesce, concurrent shutdown, rollback and process partial-body tests remain green.
|
||||||
|
|
||||||
|
Final root pipelines ran all format/Clippy/test/release gates successfully on the fixed source, then reran all six benchmarks, real ForcAD-library interoperability, sample configuration validation and JS syntax. The final release UI was restarted and rechecked for login, start, primary/replica promotion, robot, custom-IP creation and exact1MiB half-close. No game configuration was installed.
|
||||||
|
|
||||||
|
Generated local release artifacts are in git-ignored `dist/`, with `SHA256SUMS`. macOS artifact is arm64 Mach-O; Linux artifact is x86-64 glibc-linked ELF built in Orb Ubuntu, not a static binary certified for all Linux distributions. No systemd unit was installed. The12 tracked legacy reference files still match the Downloads copy; an existing untracked legacy `.gitignore` was not modified or removed.
|
||||||
@ -62,15 +62,8 @@ async fn latency(addr: SocketAddr) -> Result<(f64, f64)> {
|
|||||||
times.sort_by(f64::total_cmp);
|
times.sort_by(f64::total_cmp);
|
||||||
Ok((times[250], times[475]))
|
Ok((times[250], times[475]))
|
||||||
}
|
}
|
||||||
#[tokio::main]
|
fn echo(listener: TcpListener) -> tokio::task::JoinHandle<()> {
|
||||||
async fn main() -> Result<()> {
|
tokio::spawn(async move {
|
||||||
let megabytes = std::env::args()
|
|
||||||
.nth(1)
|
|
||||||
.and_then(|v| v.parse().ok())
|
|
||||||
.unwrap_or(64);
|
|
||||||
let listener = TcpListener::bind("127.0.0.1:0").await?;
|
|
||||||
let direct = listener.local_addr()?;
|
|
||||||
let upstream = tokio::spawn(async move {
|
|
||||||
loop {
|
loop {
|
||||||
let (stream, _) = listener.accept().await.unwrap();
|
let (stream, _) = listener.accept().await.unwrap();
|
||||||
stream.set_nodelay(true).unwrap();
|
stream.set_nodelay(true).unwrap();
|
||||||
@ -80,32 +73,73 @@ async fn main() -> Result<()> {
|
|||||||
let _ = write.shutdown().await;
|
let _ = write.shutdown().await;
|
||||||
});
|
});
|
||||||
}
|
}
|
||||||
});
|
})
|
||||||
|
}
|
||||||
|
#[tokio::main]
|
||||||
|
async fn main() -> Result<()> {
|
||||||
|
let megabytes = std::env::args()
|
||||||
|
.nth(1)
|
||||||
|
.and_then(|v| v.parse().ok())
|
||||||
|
.unwrap_or(64);
|
||||||
|
anyhow::ensure!(
|
||||||
|
(1..=4096).contains(&megabytes),
|
||||||
|
"payload must be 1..4096 MiB"
|
||||||
|
);
|
||||||
|
let mode = std::env::args().nth(2).unwrap_or_else(|| "none".into());
|
||||||
|
anyhow::ensure!(
|
||||||
|
matches!(mode.as_str(), "none" | "strict" | "isolate"),
|
||||||
|
"mode: none, strict or isolate"
|
||||||
|
);
|
||||||
|
let listener = TcpListener::bind("127.0.0.1:0").await?;
|
||||||
|
let direct = listener.local_addr()?;
|
||||||
|
let upstream = echo(listener);
|
||||||
|
let replica = if mode != "none" {
|
||||||
|
Some(echo(
|
||||||
|
TcpListener::bind((std::net::Ipv6Addr::LOCALHOST, direct.port())).await?,
|
||||||
|
))
|
||||||
|
} else {
|
||||||
|
None
|
||||||
|
};
|
||||||
|
let mut teams = vec![Team {
|
||||||
|
hidden: false,
|
||||||
|
id: "local".into(),
|
||||||
|
name: "Loopback".into(),
|
||||||
|
address: direct.ip(),
|
||||||
|
}];
|
||||||
|
if replica.is_some() {
|
||||||
|
teams.push(Team {
|
||||||
|
hidden: false,
|
||||||
|
id: "replica".into(),
|
||||||
|
name: "Replica".into(),
|
||||||
|
address: std::net::Ipv6Addr::LOCALHOST.into(),
|
||||||
|
});
|
||||||
|
}
|
||||||
let reserve = TcpListener::bind("127.0.0.1:0").await?;
|
let reserve = TcpListener::bind("127.0.0.1:0").await?;
|
||||||
let proxy = reserve.local_addr()?;
|
let proxy = reserve.local_addr()?;
|
||||||
drop(reserve);
|
drop(reserve);
|
||||||
let engine = Engine::new(NetworkConfig::default());
|
let engine = Engine::new(NetworkConfig::default());
|
||||||
let service = ServiceConfig {
|
let service = ServiceConfig {
|
||||||
|
custom: false,
|
||||||
|
custom_ip: None,
|
||||||
id: "bench".into(),
|
id: "bench".into(),
|
||||||
name: "Benchmark".into(),
|
name: "Benchmark".into(),
|
||||||
listen: proxy,
|
listen: proxy,
|
||||||
upstream_port: direct.port(),
|
upstream_port: direct.port(),
|
||||||
primary: Some("local".into()),
|
primary: Some("local".into()),
|
||||||
replicas: vec![],
|
replicas: if replica.is_some() {
|
||||||
|
vec!["replica".into()]
|
||||||
|
} else {
|
||||||
|
vec![]
|
||||||
|
},
|
||||||
enabled: true,
|
enabled: true,
|
||||||
auto_switch: false,
|
auto_switch: false,
|
||||||
replica_policy: ReplicaPolicy::Isolate,
|
replica_policy: if mode == "strict" {
|
||||||
|
ReplicaPolicy::Strict
|
||||||
|
} else {
|
||||||
|
ReplicaPolicy::Isolate
|
||||||
|
},
|
||||||
};
|
};
|
||||||
engine
|
engine.upsert(&service, &teams).await?;
|
||||||
.upsert(
|
|
||||||
&service,
|
|
||||||
&[Team {
|
|
||||||
id: "local".into(),
|
|
||||||
name: "Loopback".into(),
|
|
||||||
address: direct.ip(),
|
|
||||||
}],
|
|
||||||
)
|
|
||||||
.await?;
|
|
||||||
transfer(direct, 4).await?;
|
transfer(direct, 4).await?;
|
||||||
transfer(proxy, 4).await?;
|
transfer(proxy, 4).await?;
|
||||||
let mut direct_runs = vec![];
|
let mut direct_runs = vec![];
|
||||||
@ -124,10 +158,14 @@ async fn main() -> Result<()> {
|
|||||||
engine.shutdown().await;
|
engine.shutdown().await;
|
||||||
upstream.abort();
|
upstream.abort();
|
||||||
let _ = upstream.await;
|
let _ = upstream.await;
|
||||||
|
if let Some(replica) = replica {
|
||||||
|
replica.abort();
|
||||||
|
let _ = replica.await;
|
||||||
|
}
|
||||||
println!(
|
println!(
|
||||||
"{}",
|
"{}",
|
||||||
serde_json::to_string_pretty(
|
serde_json::to_string_pretty(
|
||||||
&json!({"platform":std::env::consts::OS,"arch":std::env::consts::ARCH,"payload_mib_per_run":megabytes,"throughput_unit":"MiB/s aggregate upload+echo download","direct":direct_runs,"proxy":proxy_runs,"direct_latency_us":{"p50":direct_latency.0,"p95":direct_latency.1},"proxy_latency_us":{"p50":proxy_latency.0,"p95":proxy_latency.1},"byte_integrity":"verified every byte", "stats":engine.snapshots()})
|
&json!({"platform":std::env::consts::OS,"arch":std::env::consts::ARCH,"mode":mode,"payload_mib_per_run":megabytes,"throughput_unit":"MiB/s aggregate upload+echo download (replica bytes excluded)","direct":direct_runs,"proxy":proxy_runs,"direct_latency_us":{"p50":direct_latency.0,"p95":direct_latency.1},"proxy_latency_us":{"p50":proxy_latency.0,"p95":proxy_latency.1},"byte_integrity":"verified every primary reply byte", "stats":engine.snapshots()})
|
||||||
)?
|
)?
|
||||||
);
|
);
|
||||||
Ok(())
|
Ok(())
|
||||||
|
|||||||
11
findings.md
11
findings.md
@ -18,8 +18,17 @@
|
|||||||
|
|
||||||
## Architecture direction
|
## Architecture direction
|
||||||
- Tokio tasks, bounded buffers, shared immutable route snapshots; no process/thread per connection.
|
- Tokio tasks, bounded buffers, shared immutable route snapshots; no process/thread per connection.
|
||||||
- Direct paths use Tokio bidirectional copy; mirrored paths share `Bytes` chunks across bounded channels. Always drain replica replies.
|
- Direct paths reuse read buffers; mirrored paths share `Bytes` chunks across bounded channels. Always drain replica replies.
|
||||||
- Explicit strict/isolate replica policy; no silent chunk dropping. Strict backpressure trades throughput for bounded reliable delivery.
|
- Explicit strict/isolate replica policy; no silent chunk dropping. Strict backpressure trades throughput for bounded reliable delivery.
|
||||||
- Health and fresh scoreboard inform deterministic candidate selection with hysteresis/cooldown; no switch solely from stale/missing evidence.
|
- Health and fresh scoreboard inform deterministic candidate selection with hysteresis/cooldown; no switch solely from stale/missing evidence.
|
||||||
- Axum control plane, embedded static UI, single authenticated SSE stream instead of four polling loops.
|
- Axum control plane, embedded static UI, single authenticated SSE stream instead of four polling loops.
|
||||||
- No kernel routing work: user explicitly chose internal upstream routing.
|
- No kernel routing work: user explicitly chose internal upstream routing.
|
||||||
|
|
||||||
|
## Final restoration and review
|
||||||
|
- Original base CSS retained (24037 characters modulo trailing whitespace); matching Bootstrap5.1.3/Font Awesome6.0.0 bundled locally. Main panel keeps original columns/rows/controls and four counters.
|
||||||
|
- Top status strip represents own-team main_team_ip independently of primary. Direct custom IP resolves its target atomically through standard service PUT.
|
||||||
|
- R1-R3 corrected with negative-control regressions; R4 redesigned journal removed; R5 old primary retained as replica. Absolute heartbeat and recoverable accept supervision also implemented.
|
||||||
|
- Final review discovered R6: cancellation of an HTTP request can cancel the controller future between disk/runtime/publication steps. Fix transaction ownership rather than merely trying to keep HTTP sockets alive.
|
||||||
|
- Browser Use fill does not trigger a native dirty-field change on blur in this environment; direct-IP behavior was verified with keyboard typing and Tab instead. Browser-wide viewport override was ineffective; tab-scoped CDP metrics produced actual390x844 and was cleared after verification.
|
||||||
|
|
||||||
|
- R6 resolved by one owned admitted transaction; cancelled waiters allocate no worker. Five cancellation regressions plus independent actual HTTP red/green prove cleanup/publication/rollback convergence. Final review approved, no remaining Critical/Important.
|
||||||
|
|||||||
@ -1,5 +1,8 @@
|
|||||||
# Copy to mirror.toml. Set MIRROR_PASSWORD in your environment (12+ characters).
|
# Copy to mirror.toml. Set MIRROR_PASSWORD in your environment (12+ characters).
|
||||||
# Bind the panel to a VPN interface or put it behind HTTPS when used remotely.
|
# Bind the panel to a VPN interface or put it behind HTTPS when used remotely.
|
||||||
|
# Optional: own team's checker status in the top strip, as in the legacy UI.
|
||||||
|
# main_team_ip = "192.0.2.10"
|
||||||
|
|
||||||
[web]
|
[web]
|
||||||
listen = "127.0.0.1:6700"
|
listen = "127.0.0.1:6700"
|
||||||
password_env = "MIRROR_PASSWORD"
|
password_env = "MIRROR_PASSWORD"
|
||||||
|
|||||||
10
progress.md
10
progress.md
@ -5,3 +5,13 @@
|
|||||||
2026-10-23: created Cargo manifest/config validation and engine contract. TCP engine delegated to one implementer under subagent-driven-development. Canonical ForcAD source verified; found initial-vs-delta event mismatch in legacy client. Orb does not mount the macOS Desktop path directly; stage source under ~/Downloads for Linux runs.
|
2026-10-23: created Cargo manifest/config validation and engine contract. TCP engine delegated to one implementer under subagent-driven-development. Canonical ForcAD source verified; found initial-vs-delta event mismatch in legacy client. Orb does not mount the macOS Desktop path directly; stage source under ~/Downloads for Linux runs.
|
||||||
|
|
||||||
2026-10-23: engine delivered and self-reviewed, 11 tests passed. Added zero-copy-reuse primary buffer path when no replicas are active (no per-read shared-Bytes allocation). Controller, health, state persistence/rollback, auth, API and CLI implemented. Added control-plane tests; integrated run waiting for UI app.js. UI implementation delegated after engine implementer finished. No live game endpoints contacted.
|
2026-10-23: engine delivered and self-reviewed, 11 tests passed. Added zero-copy-reuse primary buffer path when no replicas are active (no per-read shared-Bytes allocation). Controller, health, state persistence/rollback, auth, API and CLI implemented. Added control-plane tests; integrated run waiting for UI app.js. UI implementation delegated after engine implementer finished. No live game endpoints contacted.
|
||||||
|
|
||||||
|
2026-10-23: initial rewrite committed locally as `4b01fb4`; prior checks passed 23 tests on macOS and Orb amd64, Clippy and JS syntax. A real upstream-version Socket.IO fixture passed; a macOS byte-verified loopback benchmark showed measurable proxy overhead. Independent review found five Important issues (see `docs/review.md`); they remain open.
|
||||||
|
|
||||||
|
2026-10-23: user rejected the UI redesign and required the original appearance and functionality layout with only small changes. Source comparison confirms two unintended workflow changes: direct custom IP requires creating a team, and quick primary promotion discards the old primary from replicas. Explained these regressions and the unfinished backend review findings. Recorded the correction as binding; no UI restoration implemented yet. Original directories remain references, and the rewrite has not been deployed over them.
|
||||||
|
|
||||||
|
2026-10-23: restored literal legacy UI, bundled matching CSS/icon dependencies with licenses, kept former-primary replica and direct custom-IP flows. Added own-team strip, stale snapshot/health indicators, keyboard support and draft preservation. Source CSS parity and matching computed styles checked; browser verified primary/start/robot/custom IP/settings/logout,1MiB exact half-close,390px layout and SSE watchdog/recovery.
|
||||||
|
|
||||||
|
2026-10-23: fixed initial R1-R5 plus accept retry, early engine quiesce and absolute heartbeat deadline. Root fresh native35/Orb36 tests passed, native fmt/Clippy and both release builds passed. Real ForcAD library interop passed. Six benchmark runs saved under docs/benchmarks; strict replica byte totals complete, isolate saturation recorded3 explicit replica disconnections on each platform. Final reviewer found R6 transaction cancellation; fix/review in progress.
|
||||||
|
|
||||||
|
2026-10-23: R6 resolved with admitted owned transaction worker shared by service/team/automatic changes. Five cancellation regressions, original HTTP red/green and independent re-review accepted. Root fresh final gates: native40/Orb41 tests, native fmt/Clippy, both release builds, ForcAD library interoperability and sample config/JS checks all passed. Final release UI start/primary/replica/robot/custom creation and exact1MiB half-close rechecked. Six final benchmark records committed; isolate disconnections3macOS/4Linux documented explicitly. Prepared git-ignored dist binaries/checksums; no deployment.
|
||||||
|
|||||||
45
scripts/legacy_ui_reference.py
Normal file
45
scripts/legacy_ui_reference.py
Normal file
@ -0,0 +1,45 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
"""Render legacy templates using public loopback data; never import legacy config.
|
||||||
|
|
||||||
|
Development-only: pip install jinja2
|
||||||
|
Usage: python scripts/legacy_ui_reference.py ../ad_scripts/mirror_tool
|
||||||
|
Then serve test-results/legacy-reference on loopback for visual comparison.
|
||||||
|
"""
|
||||||
|
import argparse
|
||||||
|
from pathlib import Path
|
||||||
|
import re
|
||||||
|
import shutil
|
||||||
|
from jinja2 import Environment, FileSystemLoader, select_autoescape
|
||||||
|
|
||||||
|
|
||||||
|
def main():
|
||||||
|
parser = argparse.ArgumentParser(description=__doc__)
|
||||||
|
parser.add_argument('legacy', type=Path)
|
||||||
|
parser.add_argument('--output', type=Path, default=Path('test-results/legacy-reference'))
|
||||||
|
args = parser.parse_args()
|
||||||
|
template_dir = args.legacy / 'templates'
|
||||||
|
base = (template_dir / 'base.html').read_text()
|
||||||
|
original_css = base.split('<style>', 1)[1].split('</style>', 1)[0]
|
||||||
|
current_css = Path('web/app.css').read_text()
|
||||||
|
normalized_css = '\n'.join(line.rstrip() for line in original_css.splitlines())
|
||||||
|
assert normalized_css in current_css, 'Original CSS rules were changed or removed'
|
||||||
|
env = Environment(loader=FileSystemLoader(template_dir), autoescape=select_autoescape(['html']))
|
||||||
|
env.globals.update(get_flashed_messages=lambda **_: [], url_for=lambda _: '#')
|
||||||
|
data = dict(session={'authenticated': True}, tasks={'Echo': 18081},
|
||||||
|
teams={'Loopback A': '127.0.0.1', 'Loopback B': '::1'},
|
||||||
|
custom_proxies={}, port_monitoring_enabled=True, main_team_ip='127.0.0.1')
|
||||||
|
args.output.mkdir(parents=True, exist_ok=True)
|
||||||
|
for template in ('index.html', 'login.html'):
|
||||||
|
html = env.get_template(template).render(**data)
|
||||||
|
html = re.sub(r'<script\b[^>]*>.*?</script>', '', html, flags=re.S)
|
||||||
|
html = html.replace('https://cdn.jsdelivr.net/npm/bootstrap@5.1.3/dist/css/bootstrap.min.css', '/vendor/bootstrap.min.css')
|
||||||
|
html = html.replace('https://cdnjs.cloudflare.com/ajax/libs/font-awesome/6.0.0/css/all.min.css', '/vendor/fontawesome.min.css')
|
||||||
|
(args.output / template).write_text(html)
|
||||||
|
shutil.copytree('web/vendor', args.output / 'vendor', dirs_exist_ok=True)
|
||||||
|
(args.output / 'webfonts').mkdir(exist_ok=True)
|
||||||
|
shutil.copyfile('web/vendor/fa-solid-900.woff2', args.output / 'webfonts/fa-solid-900.woff2')
|
||||||
|
print(f'Original CSS preserved (trailing whitespace normalized) ({len(original_css)} characters); reference at {args.output}')
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == '__main__':
|
||||||
|
main()
|
||||||
108
scripts/verify_forcad_library.py
Normal file
108
scripts/verify_forcad_library.py
Normal file
@ -0,0 +1,108 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
"""Optional interoperability check against ForcAD's real Socket.IO libraries.
|
||||||
|
|
||||||
|
Install python-socketio==5.10.0 python-engineio==4.8.0 aiohttp in a disposable
|
||||||
|
venv, then run this script with the built mirror-tool binary as its argument.
|
||||||
|
No external hosts, legacy config or real credentials are used.
|
||||||
|
"""
|
||||||
|
import asyncio
|
||||||
|
import json
|
||||||
|
import os
|
||||||
|
from pathlib import Path
|
||||||
|
import sys
|
||||||
|
import tempfile
|
||||||
|
|
||||||
|
from aiohttp import web, ClientSession
|
||||||
|
import socketio
|
||||||
|
|
||||||
|
|
||||||
|
async def main():
|
||||||
|
connections = 0
|
||||||
|
active = set()
|
||||||
|
sio = socketio.AsyncServer(async_mode="aiohttp", ping_interval=0.2, ping_timeout=1)
|
||||||
|
app = web.Application()
|
||||||
|
sio.attach(app)
|
||||||
|
|
||||||
|
@sio.on("connect", namespace="/game_events")
|
||||||
|
async def connected(sid, environ):
|
||||||
|
nonlocal connections
|
||||||
|
connections += 1
|
||||||
|
active.add(sid)
|
||||||
|
await sio.emit("init_scoreboard", {"data": {
|
||||||
|
"teams": [{"id": 1, "name": "Fixture A"}],
|
||||||
|
"tasks": [{"id": 2, "name": "Notes"}],
|
||||||
|
"state": {"round": 1, "team_tasks": [{"team_id": 1, "task_id": 2, "status": 101}]},
|
||||||
|
}}, to=sid, namespace="/game_events")
|
||||||
|
|
||||||
|
@sio.on("disconnect", namespace="/game_events")
|
||||||
|
async def disconnected(sid):
|
||||||
|
active.discard(sid)
|
||||||
|
|
||||||
|
runner = web.AppRunner(app)
|
||||||
|
await runner.setup()
|
||||||
|
await web.TCPSite(runner, "127.0.0.1", 18333).start()
|
||||||
|
with tempfile.TemporaryDirectory(prefix="mirror-forcad-interop-") as temp:
|
||||||
|
config = Path(temp) / "mirror.toml"
|
||||||
|
config.write_text('''[web]
|
||||||
|
listen="127.0.0.1:16701"
|
||||||
|
state_file="state.json"
|
||||||
|
[scoreboard]
|
||||||
|
kind="forcad"
|
||||||
|
url="http://127.0.0.1:18333"
|
||||||
|
poll_interval_ms=100
|
||||||
|
stale_after_ms=1000
|
||||||
|
''')
|
||||||
|
env = os.environ.copy()
|
||||||
|
env["MIRROR_PASSWORD"] = "local-interop-fixture-only"
|
||||||
|
process = await asyncio.create_subprocess_exec(str(Path(sys.argv[1]).resolve()), "--config", str(config), env=env, stdout=asyncio.subprocess.DEVNULL)
|
||||||
|
try:
|
||||||
|
async with ClientSession() as client:
|
||||||
|
for _ in range(100):
|
||||||
|
try:
|
||||||
|
response = await client.get("http://127.0.0.1:16701/healthz")
|
||||||
|
await response.read()
|
||||||
|
break
|
||||||
|
except OSError:
|
||||||
|
await asyncio.sleep(0.03)
|
||||||
|
response = await client.post("http://127.0.0.1:16701/api/login", json={"password": env["MIRROR_PASSWORD"]})
|
||||||
|
assert response.status == 200
|
||||||
|
cookie = response.headers["Set-Cookie"].split(";", 1)[0]
|
||||||
|
await response.read()
|
||||||
|
|
||||||
|
async def snapshot():
|
||||||
|
async with client.get("http://127.0.0.1:16701/api/snapshot", headers={"Cookie": cookie}) as response:
|
||||||
|
assert response.status == 200
|
||||||
|
return (await response.json())["scoreboard"]
|
||||||
|
|
||||||
|
async def wait_for(predicate, timeout=4):
|
||||||
|
async with asyncio.timeout(timeout):
|
||||||
|
while True:
|
||||||
|
state = await snapshot()
|
||||||
|
if predicate(state):
|
||||||
|
return state
|
||||||
|
await asyncio.sleep(0.03)
|
||||||
|
|
||||||
|
await wait_for(lambda s: s["round"] == 1)
|
||||||
|
await sio.emit("update_scoreboard", {"data": {"round": 2, "team_tasks": [{"team_id": 1, "task_id": 2, "status": 104, "stolen": 17}]}}, namespace="/game_events")
|
||||||
|
delta = await wait_for(lambda s: s["round"] == 2)
|
||||||
|
assert delta["entries"]["2"]["1"]["stolen"] == 17
|
||||||
|
assert delta["teams"][0]["name"] == "Fixture A"
|
||||||
|
# Several real Engine.IO heartbeat exchanges must preserve one session.
|
||||||
|
await asyncio.sleep(0.7)
|
||||||
|
assert (await snapshot())["connected"]
|
||||||
|
assert connections == 1
|
||||||
|
await wait_for(lambda s: s["stale"])
|
||||||
|
for sid in list(active):
|
||||||
|
await sio.disconnect(sid, namespace="/game_events")
|
||||||
|
await wait_for(lambda s: s["round"] == 1 and s["connected"] and not s["stale"])
|
||||||
|
assert connections == 2
|
||||||
|
print(json.dumps({"library": "python-socketio 5.10.0 / python-engineio 4.8.0", "initial_and_delta": "passed", "metadata_retention": "passed", "heartbeat_session_reuse": "passed", "stale_data": "passed", "namespace_disconnect_reconnect": "passed", "connections": connections}, indent=2))
|
||||||
|
finally:
|
||||||
|
process.terminate()
|
||||||
|
await asyncio.wait_for(process.wait(), timeout=10)
|
||||||
|
await sio.shutdown()
|
||||||
|
await runner.cleanup()
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
asyncio.run(main())
|
||||||
@ -9,6 +9,7 @@ use std::{
|
|||||||
#[derive(Clone, Debug, Default, Serialize, Deserialize)]
|
#[derive(Clone, Debug, Default, Serialize, Deserialize)]
|
||||||
#[serde(default, deny_unknown_fields)]
|
#[serde(default, deny_unknown_fields)]
|
||||||
pub struct Config {
|
pub struct Config {
|
||||||
|
pub main_team_ip: Option<IpAddr>,
|
||||||
pub web: WebConfig,
|
pub web: WebConfig,
|
||||||
pub network: NetworkConfig,
|
pub network: NetworkConfig,
|
||||||
pub monitoring: MonitoringConfig,
|
pub monitoring: MonitoringConfig,
|
||||||
@ -70,6 +71,8 @@ pub enum ScoreboardKind {
|
|||||||
#[derive(Clone, Debug, Serialize, Deserialize, PartialEq, Eq)]
|
#[derive(Clone, Debug, Serialize, Deserialize, PartialEq, Eq)]
|
||||||
#[serde(deny_unknown_fields)]
|
#[serde(deny_unknown_fields)]
|
||||||
pub struct Team {
|
pub struct Team {
|
||||||
|
#[serde(default)]
|
||||||
|
pub hidden: bool,
|
||||||
pub id: String,
|
pub id: String,
|
||||||
pub name: String,
|
pub name: String,
|
||||||
pub address: IpAddr,
|
pub address: IpAddr,
|
||||||
@ -78,6 +81,10 @@ pub struct Team {
|
|||||||
#[derive(Clone, Debug, Serialize, Deserialize, PartialEq, Eq)]
|
#[derive(Clone, Debug, Serialize, Deserialize, PartialEq, Eq)]
|
||||||
#[serde(deny_unknown_fields)]
|
#[serde(deny_unknown_fields)]
|
||||||
pub struct ServiceConfig {
|
pub struct ServiceConfig {
|
||||||
|
#[serde(default)]
|
||||||
|
pub custom: bool,
|
||||||
|
#[serde(default)]
|
||||||
|
pub custom_ip: Option<IpAddr>,
|
||||||
pub id: String,
|
pub id: String,
|
||||||
pub name: String,
|
pub name: String,
|
||||||
pub listen: SocketAddr,
|
pub listen: SocketAddr,
|
||||||
@ -241,8 +248,18 @@ impl Config {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
pub fn normalized_ip(ip: IpAddr) -> IpAddr {
|
||||||
|
match ip {
|
||||||
|
IpAddr::V6(v6) => v6.to_ipv4_mapped().map(IpAddr::V4).unwrap_or(ip),
|
||||||
|
_ => ip,
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
pub fn bind_conflicts(a: SocketAddr, b: SocketAddr) -> bool {
|
pub fn bind_conflicts(a: SocketAddr, b: SocketAddr) -> bool {
|
||||||
a.port() == b.port() && (a.ip() == b.ip() || a.ip().is_unspecified() || b.ip().is_unspecified())
|
a.port() == b.port()
|
||||||
|
&& (normalized_ip(a.ip()) == normalized_ip(b.ip())
|
||||||
|
|| (a.ip().is_unspecified() && (a.is_ipv6() || normalized_ip(b.ip()).is_ipv4()))
|
||||||
|
|| (b.ip().is_unspecified() && (b.is_ipv6() || normalized_ip(a.ip()).is_ipv4())))
|
||||||
}
|
}
|
||||||
|
|
||||||
pub fn validate_topology(teams: &[Team], services: &[ServiceConfig]) -> Result<()> {
|
pub fn validate_topology(teams: &[Team], services: &[ServiceConfig]) -> Result<()> {
|
||||||
@ -271,12 +288,17 @@ pub fn validate_topology(teams: &[Team], services: &[ServiceConfig]) -> Result<(
|
|||||||
"duplicate team name {}",
|
"duplicate team name {}",
|
||||||
team.name
|
team.name
|
||||||
);
|
);
|
||||||
ensure!(addresses.insert(team.address), "duplicate team address");
|
|
||||||
ensure!(
|
ensure!(
|
||||||
!team.address.is_unspecified() && !team.address.is_multicast(),
|
addresses.insert(normalized_ip(team.address)),
|
||||||
|
"duplicate team address"
|
||||||
|
);
|
||||||
|
ensure!(
|
||||||
|
!normalized_ip(team.address).is_unspecified()
|
||||||
|
&& !normalized_ip(team.address).is_multicast(),
|
||||||
"team must be a unicast address"
|
"team must be a unicast address"
|
||||||
);
|
);
|
||||||
}
|
}
|
||||||
|
local_addresses = local_addresses.into_iter().map(normalized_ip).collect();
|
||||||
let mut service_ids = HashSet::new();
|
let mut service_ids = HashSet::new();
|
||||||
for (index, service) in services.iter().enumerate() {
|
for (index, service) in services.iter().enumerate() {
|
||||||
ensure!(
|
ensure!(
|
||||||
@ -301,6 +323,17 @@ pub fn validate_topology(teams: &[Team], services: &[ServiceConfig]) -> Result<(
|
|||||||
"enabled service {} needs a primary",
|
"enabled service {} needs a primary",
|
||||||
service.id
|
service.id
|
||||||
);
|
);
|
||||||
|
ensure!(
|
||||||
|
service.custom || service.custom_ip.is_none(),
|
||||||
|
"only custom services may set custom_ip"
|
||||||
|
);
|
||||||
|
if let Some(ip) = service.custom_ip {
|
||||||
|
ensure!(
|
||||||
|
teams.iter().any(|t| Some(&t.id) == service.primary.as_ref()
|
||||||
|
&& normalized_ip(t.address) == normalized_ip(ip)),
|
||||||
|
"custom_ip must match primary"
|
||||||
|
);
|
||||||
|
}
|
||||||
let mut targets = HashSet::new();
|
let mut targets = HashSet::new();
|
||||||
for id in service.primary.iter().chain(service.replicas.iter()) {
|
for id in service.primary.iter().chain(service.replicas.iter()) {
|
||||||
ensure!(
|
ensure!(
|
||||||
@ -311,12 +344,14 @@ pub fn validate_topology(teams: &[Team], services: &[ServiceConfig]) -> Result<(
|
|||||||
let Some(team) = teams.iter().find(|t| &t.id == id) else {
|
let Some(team) = teams.iter().find(|t| &t.id == id) else {
|
||||||
bail!("unknown team {} in {}", id, service.id)
|
bail!("unknown team {} in {}", id, service.id)
|
||||||
};
|
};
|
||||||
let target = SocketAddr::new(team.address, service.upstream_port);
|
let target = SocketAddr::new(normalized_ip(team.address), service.upstream_port);
|
||||||
for listener in services {
|
for listener in services {
|
||||||
let reaches_listener = target.port() == listener.listen.port()
|
let reaches_listener = target.port() == listener.listen.port()
|
||||||
&& (target.ip() == listener.listen.ip()
|
&& (target.ip() == normalized_ip(listener.listen.ip())
|
||||||
|| (listener.listen.ip().is_unspecified()
|
|| (listener.listen.ip().is_unspecified()
|
||||||
&& local_addresses.contains(&target.ip())));
|
&& (listener.listen.is_ipv6() || target.is_ipv4())
|
||||||
|
&& (target.ip().is_loopback()
|
||||||
|
|| local_addresses.contains(&target.ip()))));
|
||||||
ensure!(
|
ensure!(
|
||||||
!reaches_listener,
|
!reaches_listener,
|
||||||
"{} routes to managed listener {} (proxy loop)",
|
"{} routes to managed listener {} (proxy loop)",
|
||||||
|
|||||||
520
src/control.rs
520
src/control.rs
@ -10,7 +10,10 @@ use serde::Serialize;
|
|||||||
use std::{
|
use std::{
|
||||||
collections::{BTreeMap, BTreeSet, VecDeque},
|
collections::{BTreeMap, BTreeSet, VecDeque},
|
||||||
net::SocketAddr,
|
net::SocketAddr,
|
||||||
sync::{Arc, RwLock},
|
sync::{
|
||||||
|
Arc, RwLock, Weak,
|
||||||
|
atomic::{AtomicBool, Ordering},
|
||||||
|
},
|
||||||
time::{Duration, Instant},
|
time::{Duration, Instant},
|
||||||
};
|
};
|
||||||
use tokio::{net::TcpStream, sync::Mutex};
|
use tokio::{net::TcpStream, sync::Mutex};
|
||||||
@ -41,6 +44,8 @@ pub struct ServiceView {
|
|||||||
}
|
}
|
||||||
#[derive(Clone, Debug, Serialize)]
|
#[derive(Clone, Debug, Serialize)]
|
||||||
pub struct Snapshot {
|
pub struct Snapshot {
|
||||||
|
pub main_team_ip: Option<std::net::IpAddr>,
|
||||||
|
pub health_max_age_ms: u64,
|
||||||
pub now_ms: u64,
|
pub now_ms: u64,
|
||||||
pub teams: Vec<Team>,
|
pub teams: Vec<Team>,
|
||||||
pub services: Vec<ServiceView>,
|
pub services: Vec<ServiceView>,
|
||||||
@ -48,27 +53,41 @@ pub struct Snapshot {
|
|||||||
pub events: Vec<Event>,
|
pub events: Vec<Event>,
|
||||||
}
|
}
|
||||||
|
|
||||||
|
#[cfg(test)]
|
||||||
|
struct PersistenceGate {
|
||||||
|
after_save: bool,
|
||||||
|
reached: tokio::sync::Notify,
|
||||||
|
resume: tokio::sync::Notify,
|
||||||
|
}
|
||||||
pub struct Controller {
|
pub struct Controller {
|
||||||
pub config: Config,
|
pub config: Config,
|
||||||
pub engine: Engine,
|
pub engine: Engine,
|
||||||
pub board: Scoreboard,
|
pub board: Scoreboard,
|
||||||
topology: RwLock<Topology>,
|
topology: RwLock<Topology>,
|
||||||
operations: Mutex<()>,
|
operations: Arc<Mutex<()>>,
|
||||||
|
self_ref: Weak<Self>,
|
||||||
|
stopping: AtomicBool,
|
||||||
health: RwLock<BTreeMap<SocketAddr, Health>>,
|
health: RwLock<BTreeMap<SocketAddr, Health>>,
|
||||||
switched: Mutex<BTreeMap<String, Instant>>,
|
switched: Mutex<BTreeMap<String, Instant>>,
|
||||||
events: RwLock<VecDeque<Event>>,
|
events: RwLock<VecDeque<Event>>,
|
||||||
|
#[cfg(test)]
|
||||||
|
persistence_gate: std::sync::Mutex<Option<Arc<PersistenceGate>>>,
|
||||||
}
|
}
|
||||||
impl Controller {
|
impl Controller {
|
||||||
pub async fn start(config: Config) -> Result<Arc<Self>> {
|
pub async fn start(config: Config) -> Result<Arc<Self>> {
|
||||||
let topology = Topology::load(&config)?;
|
let topology = Topology::load(&config)?;
|
||||||
let this = Arc::new(Self {
|
let this = Arc::new_cyclic(|weak| Self {
|
||||||
engine: Engine::new(config.network.clone()),
|
engine: Engine::new(config.network.clone()),
|
||||||
board: Scoreboard::new(config.scoreboard.as_ref()),
|
board: Scoreboard::new(config.scoreboard.as_ref()),
|
||||||
topology: RwLock::new(topology.clone()),
|
topology: RwLock::new(topology.clone()),
|
||||||
operations: Mutex::new(()),
|
operations: Arc::new(Mutex::new(())),
|
||||||
|
self_ref: weak.clone(),
|
||||||
|
stopping: AtomicBool::new(false),
|
||||||
health: RwLock::new(BTreeMap::new()),
|
health: RwLock::new(BTreeMap::new()),
|
||||||
switched: Mutex::new(BTreeMap::new()),
|
switched: Mutex::new(BTreeMap::new()),
|
||||||
events: RwLock::new(VecDeque::new()),
|
events: RwLock::new(VecDeque::new()),
|
||||||
|
#[cfg(test)]
|
||||||
|
persistence_gate: std::sync::Mutex::new(None),
|
||||||
config,
|
config,
|
||||||
});
|
});
|
||||||
for service in &topology.services {
|
for service in &topology.services {
|
||||||
@ -80,6 +99,19 @@ impl Controller {
|
|||||||
this.event(None, "Панель запущена; конфигурация восстановлена".into());
|
this.event(None, "Панель запущена; конфигурация восстановлена".into());
|
||||||
Ok(this)
|
Ok(this)
|
||||||
}
|
}
|
||||||
|
pub async fn shutdown(&self) {
|
||||||
|
self.stopping.store(true, Ordering::SeqCst);
|
||||||
|
self.engine.quiesce().await;
|
||||||
|
let _operation = self.operations.lock().await;
|
||||||
|
self.engine.shutdown().await;
|
||||||
|
}
|
||||||
|
fn health_max_age_ms(&self) -> u64 {
|
||||||
|
self.config
|
||||||
|
.monitoring
|
||||||
|
.interval_ms
|
||||||
|
.saturating_add(self.config.monitoring.timeout_ms)
|
||||||
|
.saturating_mul(3)
|
||||||
|
}
|
||||||
pub fn topology(&self) -> Topology {
|
pub fn topology(&self) -> Topology {
|
||||||
self.topology.read().unwrap().clone()
|
self.topology.read().unwrap().clone()
|
||||||
}
|
}
|
||||||
@ -126,6 +158,8 @@ impl Controller {
|
|||||||
})
|
})
|
||||||
.collect();
|
.collect();
|
||||||
Snapshot {
|
Snapshot {
|
||||||
|
main_team_ip: self.config.main_team_ip,
|
||||||
|
health_max_age_ms: self.health_max_age_ms(),
|
||||||
now_ms: now_ms(),
|
now_ms: now_ms(),
|
||||||
teams: topology.teams,
|
teams: topology.teams,
|
||||||
services,
|
services,
|
||||||
@ -133,19 +167,86 @@ impl Controller {
|
|||||||
events: self.events.read().unwrap().iter().cloned().collect(),
|
events: self.events.read().unwrap().iter().cloned().collect(),
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
#[cfg(test)]
|
||||||
|
async fn persistence_barrier(&self, after_save: bool) {
|
||||||
|
let gate = self.persistence_gate.lock().unwrap().clone();
|
||||||
|
if let Some(gate) = gate.filter(|gate| gate.after_save == after_save) {
|
||||||
|
gate.reached.notify_one();
|
||||||
|
gate.resume.notified().await;
|
||||||
|
}
|
||||||
|
}
|
||||||
async fn persist(&self, state: Topology) -> Result<()> {
|
async fn persist(&self, state: Topology) -> Result<()> {
|
||||||
|
#[cfg(test)]
|
||||||
|
self.persistence_barrier(false).await;
|
||||||
let path = self.config.web.state_file.clone();
|
let path = self.config.web.state_file.clone();
|
||||||
tokio::task::spawn_blocking(move || store::save(&path, &state))
|
tokio::task::spawn_blocking(move || store::save(&path, &state))
|
||||||
.await
|
.await
|
||||||
.context("state writer task")??;
|
.context("state writer task")??;
|
||||||
|
#[cfg(test)]
|
||||||
|
self.persistence_barrier(true).await;
|
||||||
Ok(())
|
Ok(())
|
||||||
}
|
}
|
||||||
pub async fn put_service(&self, service: ServiceConfig) -> Result<()> {
|
/// Acquire admission before spawning: at most one owned transaction executes,
|
||||||
let _lock = self.operations.lock().await;
|
/// while cancelled waiters never allocate transaction tasks or mutate state.
|
||||||
self.put_service_locked(service, false).await
|
async fn transaction<F, Fut>(&self, operation: F) -> Result<()>
|
||||||
|
where
|
||||||
|
F: FnOnce(Arc<Self>) -> Fut + Send + 'static,
|
||||||
|
Fut: std::future::Future<Output = Result<()>> + Send + 'static,
|
||||||
|
{
|
||||||
|
let guard = self.operations.clone().lock_owned().await;
|
||||||
|
ensure!(
|
||||||
|
!self.stopping.load(Ordering::SeqCst),
|
||||||
|
"controller is shutting down"
|
||||||
|
);
|
||||||
|
let controller = self
|
||||||
|
.self_ref
|
||||||
|
.upgrade()
|
||||||
|
.context("controller is unavailable")?;
|
||||||
|
tokio::spawn(async move {
|
||||||
|
let _guard = guard;
|
||||||
|
operation(controller).await
|
||||||
|
})
|
||||||
|
.await
|
||||||
|
.context("control transaction task failed")?
|
||||||
}
|
}
|
||||||
async fn put_service_locked(&self, service: ServiceConfig, automatic: bool) -> Result<()> {
|
pub async fn put_service(&self, service: ServiceConfig) -> Result<()> {
|
||||||
|
self.transaction(move |controller| async move {
|
||||||
|
controller.put_service_locked(service, false).await
|
||||||
|
})
|
||||||
|
.await
|
||||||
|
}
|
||||||
|
async fn put_service_locked(&self, mut service: ServiceConfig, automatic: bool) -> Result<()> {
|
||||||
let mut next = self.topology();
|
let mut next = self.topology();
|
||||||
|
let old_teams = next.teams.clone();
|
||||||
|
if let Some(ip) = service.custom_ip {
|
||||||
|
ensure!(service.custom, "only custom services may set custom_ip");
|
||||||
|
let ip = crate::config::normalized_ip(ip);
|
||||||
|
service.custom_ip = Some(ip);
|
||||||
|
let id = if let Some(team) = next
|
||||||
|
.teams
|
||||||
|
.iter()
|
||||||
|
.find(|t| crate::config::normalized_ip(t.address) == ip)
|
||||||
|
{
|
||||||
|
team.id.clone()
|
||||||
|
} else {
|
||||||
|
let stem = format!("direct_{}", ip.to_string().replace(['.', ':'], "_"));
|
||||||
|
let mut id = stem.clone();
|
||||||
|
let mut suffix = 0;
|
||||||
|
while next.teams.iter().any(|t| t.id == id) {
|
||||||
|
suffix += 1;
|
||||||
|
id = format!("{stem}_{suffix}");
|
||||||
|
}
|
||||||
|
next.teams.push(Team {
|
||||||
|
id: id.clone(),
|
||||||
|
name: ip.to_string(),
|
||||||
|
address: ip,
|
||||||
|
hidden: true,
|
||||||
|
});
|
||||||
|
id
|
||||||
|
};
|
||||||
|
service.primary = Some(id.clone());
|
||||||
|
service.replicas.retain(|target| target != &id);
|
||||||
|
}
|
||||||
let old = next.services.iter().find(|s| s.id == service.id).cloned();
|
let old = next.services.iter().find(|s| s.id == service.id).cloned();
|
||||||
if let Some(current) = next.services.iter_mut().find(|s| s.id == service.id) {
|
if let Some(current) = next.services.iter_mut().find(|s| s.id == service.id) {
|
||||||
*current = service.clone();
|
*current = service.clone();
|
||||||
@ -156,7 +257,7 @@ impl Controller {
|
|||||||
self.engine.upsert(&service, &next.teams).await?;
|
self.engine.upsert(&service, &next.teams).await?;
|
||||||
if let Err(e) = self.persist(next.clone()).await {
|
if let Err(e) = self.persist(next.clone()).await {
|
||||||
let rollback = if let Some(old) = old.as_ref() {
|
let rollback = if let Some(old) = old.as_ref() {
|
||||||
self.engine.upsert(old, &next.teams).await
|
self.engine.upsert(old, &old_teams).await
|
||||||
} else {
|
} else {
|
||||||
self.engine.remove(&service.id).await
|
self.engine.remove(&service.id).await
|
||||||
};
|
};
|
||||||
@ -190,7 +291,13 @@ impl Controller {
|
|||||||
Ok(())
|
Ok(())
|
||||||
}
|
}
|
||||||
pub async fn delete_service(&self, id: &str) -> Result<()> {
|
pub async fn delete_service(&self, id: &str) -> Result<()> {
|
||||||
let _lock = self.operations.lock().await;
|
let id = id.to_owned();
|
||||||
|
self.transaction(
|
||||||
|
move |controller| async move { controller.delete_service_locked(&id).await },
|
||||||
|
)
|
||||||
|
.await
|
||||||
|
}
|
||||||
|
async fn delete_service_locked(&self, id: &str) -> Result<()> {
|
||||||
let mut next = self.topology();
|
let mut next = self.topology();
|
||||||
let old = next
|
let old = next
|
||||||
.services
|
.services
|
||||||
@ -208,7 +315,10 @@ impl Controller {
|
|||||||
Ok(())
|
Ok(())
|
||||||
}
|
}
|
||||||
pub async fn put_team(&self, team: Team) -> Result<()> {
|
pub async fn put_team(&self, team: Team) -> Result<()> {
|
||||||
let _lock = self.operations.lock().await;
|
self.transaction(move |controller| async move { controller.put_team_locked(team).await })
|
||||||
|
.await
|
||||||
|
}
|
||||||
|
async fn put_team_locked(&self, team: Team) -> Result<()> {
|
||||||
let mut next = self.topology();
|
let mut next = self.topology();
|
||||||
if let Some(old) = next.teams.iter_mut().find(|t| t.id == team.id) {
|
if let Some(old) = next.teams.iter_mut().find(|t| t.id == team.id) {
|
||||||
ensure!(
|
ensure!(
|
||||||
@ -239,6 +349,20 @@ impl Controller {
|
|||||||
.await;
|
.await;
|
||||||
}
|
}
|
||||||
async fn check_cycle_cancellable(&self, cancel: &CancellationToken) {
|
async fn check_cycle_cancellable(&self, cancel: &CancellationToken) {
|
||||||
|
let timeout = Duration::from_millis(self.config.monitoring.timeout_ms);
|
||||||
|
self.check_cycle_with_probe(cancel, |address| async move {
|
||||||
|
matches!(
|
||||||
|
tokio::time::timeout(timeout, TcpStream::connect(address)).await,
|
||||||
|
Ok(Ok(_))
|
||||||
|
)
|
||||||
|
})
|
||||||
|
.await;
|
||||||
|
}
|
||||||
|
async fn check_cycle_with_probe<F, Fut>(&self, cancel: &CancellationToken, probe: F)
|
||||||
|
where
|
||||||
|
F: Fn(SocketAddr) -> Fut,
|
||||||
|
Fut: std::future::Future<Output = bool>,
|
||||||
|
{
|
||||||
let topology = self.topology();
|
let topology = self.topology();
|
||||||
let endpoints: BTreeSet<_> = topology
|
let endpoints: BTreeSet<_> = topology
|
||||||
.services
|
.services
|
||||||
@ -250,26 +374,18 @@ impl Controller {
|
|||||||
.map(move |t| SocketAddr::new(t.address, s.upstream_port))
|
.map(move |t| SocketAddr::new(t.address, s.upstream_port))
|
||||||
})
|
})
|
||||||
.collect();
|
.collect();
|
||||||
let timeout = Duration::from_millis(self.config.monitoring.timeout_ms);
|
|
||||||
let checks = stream::iter(endpoints.iter().copied())
|
let checks = stream::iter(endpoints.iter().copied())
|
||||||
.map(|address| async move {
|
.map(|address| observe_probe(address, probe(address)))
|
||||||
let start = Instant::now();
|
|
||||||
let ok = matches!(
|
|
||||||
tokio::time::timeout(timeout, TcpStream::connect(address)).await,
|
|
||||||
Ok(Ok(_))
|
|
||||||
);
|
|
||||||
(address, ok, start.elapsed().as_millis() as u64)
|
|
||||||
})
|
|
||||||
.buffer_unordered(self.config.monitoring.concurrency)
|
.buffer_unordered(self.config.monitoring.concurrency)
|
||||||
.collect::<Vec<_>>();
|
.collect::<Vec<_>>();
|
||||||
let results = tokio::select! {_=cancel.cancelled()=>return,results=checks=>results};
|
let results = tokio::select! {_=cancel.cancelled()=>return,results=checks=>results};
|
||||||
{
|
{
|
||||||
let mut health = self.health.write().unwrap();
|
let mut health = self.health.write().unwrap();
|
||||||
health.retain(|key, _| endpoints.contains(key));
|
health.retain(|key, _| endpoints.contains(key));
|
||||||
for (address, ok, latency) in results {
|
for (address, ok, latency, observed_at) in results {
|
||||||
let h = health.entry(address).or_default();
|
let h = health.entry(address).or_default();
|
||||||
h.reachable = ok;
|
h.reachable = ok;
|
||||||
h.checked_at_ms = now_ms();
|
h.checked_at_ms = observed_at;
|
||||||
h.latency_ms = latency;
|
h.latency_ms = latency;
|
||||||
if ok {
|
if ok {
|
||||||
h.consecutive_successes = h.consecutive_successes.saturating_add(1);
|
h.consecutive_successes = h.consecutive_successes.saturating_add(1);
|
||||||
@ -286,9 +402,18 @@ impl Controller {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
// Serialize selection with manual edits, then evaluate the CURRENT topology.
|
// Only probe futures remain caller-owned. Once selection is admitted,
|
||||||
let _operation = self.operations.lock().await;
|
// its apply/save/rollback sequence has the same ownership as manual edits.
|
||||||
if cancel.is_cancelled() {
|
let cancel = cancel.clone();
|
||||||
|
let _ = self
|
||||||
|
.transaction(move |controller| async move {
|
||||||
|
controller.apply_automatic_locked(&cancel).await;
|
||||||
|
Ok(())
|
||||||
|
})
|
||||||
|
.await;
|
||||||
|
}
|
||||||
|
async fn apply_automatic_locked(&self, cancel: &CancellationToken) {
|
||||||
|
if cancel.is_cancelled() || self.stopping.load(Ordering::SeqCst) {
|
||||||
return;
|
return;
|
||||||
}
|
}
|
||||||
let current = self.topology();
|
let current = self.topology();
|
||||||
@ -316,15 +441,12 @@ impl Controller {
|
|||||||
&health,
|
&health,
|
||||||
&board,
|
&board,
|
||||||
self.config.monitoring.failure_threshold,
|
self.config.monitoring.failure_threshold,
|
||||||
self.config
|
self.health_max_age_ms(),
|
||||||
.monitoring
|
|
||||||
.interval_ms
|
|
||||||
.saturating_add(self.config.monitoring.timeout_ms)
|
|
||||||
.saturating_mul(3),
|
|
||||||
)
|
)
|
||||||
};
|
};
|
||||||
if let Some(candidate) = candidate {
|
if let Some(candidate) = candidate {
|
||||||
let mut new = service.clone();
|
let mut new = service.clone();
|
||||||
|
new.custom_ip = None;
|
||||||
let old = new.primary.replace(candidate.clone());
|
let old = new.primary.replace(candidate.clone());
|
||||||
new.replicas.retain(|id| id != &candidate);
|
new.replicas.retain(|id| id != &candidate);
|
||||||
if let Some(old) = old {
|
if let Some(old) = old {
|
||||||
@ -341,6 +463,15 @@ impl Controller {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
async fn observe_probe(
|
||||||
|
address: SocketAddr,
|
||||||
|
probe: impl std::future::Future<Output = bool>,
|
||||||
|
) -> (SocketAddr, bool, u64, u64) {
|
||||||
|
let start = Instant::now();
|
||||||
|
let ok = probe.await;
|
||||||
|
(address, ok, start.elapsed().as_millis() as u64, now_ms())
|
||||||
|
}
|
||||||
|
|
||||||
pub fn board_entry<'a>(
|
pub fn board_entry<'a>(
|
||||||
board: &'a BoardSnapshot,
|
board: &'a BoardSnapshot,
|
||||||
service: &ServiceConfig,
|
service: &ServiceConfig,
|
||||||
@ -415,17 +546,21 @@ mod tests {
|
|||||||
fn fixtures() -> (ServiceConfig, Vec<Team>, BTreeMap<SocketAddr, Health>) {
|
fn fixtures() -> (ServiceConfig, Vec<Team>, BTreeMap<SocketAddr, Health>) {
|
||||||
let teams = vec![
|
let teams = vec![
|
||||||
Team {
|
Team {
|
||||||
|
hidden: false,
|
||||||
id: "a".into(),
|
id: "a".into(),
|
||||||
name: "A".into(),
|
name: "A".into(),
|
||||||
address: "192.0.2.1".parse().unwrap(),
|
address: "192.0.2.1".parse().unwrap(),
|
||||||
},
|
},
|
||||||
Team {
|
Team {
|
||||||
|
hidden: false,
|
||||||
id: "b".into(),
|
id: "b".into(),
|
||||||
name: "B".into(),
|
name: "B".into(),
|
||||||
address: "192.0.2.2".parse().unwrap(),
|
address: "192.0.2.2".parse().unwrap(),
|
||||||
},
|
},
|
||||||
];
|
];
|
||||||
let service = ServiceConfig {
|
let service = ServiceConfig {
|
||||||
|
custom: false,
|
||||||
|
custom_ip: None,
|
||||||
id: "notes".into(),
|
id: "notes".into(),
|
||||||
name: "Notes".into(),
|
name: "Notes".into(),
|
||||||
listen: "127.0.0.1:18080".parse().unwrap(),
|
listen: "127.0.0.1:18080".parse().unwrap(),
|
||||||
@ -470,4 +605,329 @@ mod tests {
|
|||||||
.checked_at_ms = 0;
|
.checked_at_ms = 0;
|
||||||
assert_eq!(select_candidate(&s, &t, &h, &b, 2, 1000), None);
|
assert_eq!(select_candidate(&s, &t, &h, &b, 2, 1000), None);
|
||||||
}
|
}
|
||||||
|
#[tokio::test]
|
||||||
|
async fn delayed_batch_does_not_refresh_early_candidate_observation() {
|
||||||
|
let temp = tempfile::tempdir().unwrap();
|
||||||
|
let (mut service, mut teams, _) = fixtures();
|
||||||
|
let listener = tokio::net::TcpListener::bind("127.0.0.1:0").await.unwrap();
|
||||||
|
service.listen = listener.local_addr().unwrap();
|
||||||
|
drop(listener);
|
||||||
|
teams.push(Team {
|
||||||
|
id: "slow".into(),
|
||||||
|
name: "Slow".into(),
|
||||||
|
address: "192.0.2.3".parse().unwrap(),
|
||||||
|
hidden: false,
|
||||||
|
});
|
||||||
|
let mut config = Config::default();
|
||||||
|
config.web.state_file = temp.path().join("state.json");
|
||||||
|
config.monitoring.interval_ms = 100;
|
||||||
|
config.monitoring.timeout_ms = 1;
|
||||||
|
config.monitoring.concurrency = 1;
|
||||||
|
config.monitoring.cooldown_ms = 0;
|
||||||
|
config.monitoring.failure_threshold = 1;
|
||||||
|
config.monitoring.recovery_threshold = 1;
|
||||||
|
config.teams = teams;
|
||||||
|
config.services = vec![service];
|
||||||
|
let controller = Controller::start(config).await.unwrap();
|
||||||
|
controller
|
||||||
|
.check_cycle_with_probe(&CancellationToken::new(), |address| async move {
|
||||||
|
if address.ip() == "192.0.2.3".parse::<std::net::IpAddr>().unwrap() {
|
||||||
|
tokio::time::sleep(Duration::from_millis(450)).await;
|
||||||
|
}
|
||||||
|
address.ip() != "192.0.2.1".parse::<std::net::IpAddr>().unwrap()
|
||||||
|
})
|
||||||
|
.await;
|
||||||
|
let snapshot = controller.snapshot();
|
||||||
|
assert!(
|
||||||
|
snapshot
|
||||||
|
.now_ms
|
||||||
|
.saturating_sub(snapshot.services[0].health["b"].checked_at_ms)
|
||||||
|
>= 400
|
||||||
|
);
|
||||||
|
assert_eq!(snapshot.services[0].config.primary.as_deref(), Some("a"));
|
||||||
|
controller.shutdown().await;
|
||||||
|
}
|
||||||
|
#[tokio::test]
|
||||||
|
async fn shutdown_quiesces_before_waiting_for_control_transaction_lock() {
|
||||||
|
let temp = tempfile::tempdir().unwrap();
|
||||||
|
let (mut service, mut teams, _) = fixtures();
|
||||||
|
teams[0].address = "127.0.0.1".parse().unwrap();
|
||||||
|
teams[1].address = "::1".parse().unwrap();
|
||||||
|
let listener = tokio::net::TcpListener::bind("127.0.0.1:0").await.unwrap();
|
||||||
|
service.listen = listener.local_addr().unwrap();
|
||||||
|
drop(listener);
|
||||||
|
let listen = service.listen;
|
||||||
|
let mut config = Config::default();
|
||||||
|
config.web.state_file = temp.path().join("state.json");
|
||||||
|
config.teams = teams;
|
||||||
|
config.services = vec![service];
|
||||||
|
let controller = Controller::start(config).await.unwrap();
|
||||||
|
let transaction = controller.operations.lock().await;
|
||||||
|
let shutting_down = tokio::spawn({
|
||||||
|
let controller = controller.clone();
|
||||||
|
async move {
|
||||||
|
controller.shutdown().await;
|
||||||
|
}
|
||||||
|
});
|
||||||
|
tokio::time::timeout(Duration::from_millis(200), async {
|
||||||
|
while TcpStream::connect(listen).await.is_ok() {
|
||||||
|
tokio::task::yield_now().await;
|
||||||
|
}
|
||||||
|
})
|
||||||
|
.await
|
||||||
|
.unwrap();
|
||||||
|
assert!(
|
||||||
|
!shutting_down.is_finished(),
|
||||||
|
"shutdown did not wait for in-flight transaction"
|
||||||
|
);
|
||||||
|
assert!(TcpStream::connect(listen).await.is_err());
|
||||||
|
drop(transaction);
|
||||||
|
tokio::time::timeout(Duration::from_millis(200), shutting_down)
|
||||||
|
.await
|
||||||
|
.unwrap()
|
||||||
|
.unwrap();
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
#[cfg(test)]
|
||||||
|
mod cancellation_tests {
|
||||||
|
use super::*;
|
||||||
|
use crate::config::ReplicaPolicy;
|
||||||
|
use tokio::{
|
||||||
|
io::{AsyncReadExt, AsyncWriteExt},
|
||||||
|
net::TcpListener,
|
||||||
|
time::timeout,
|
||||||
|
};
|
||||||
|
async fn fixture(
|
||||||
|
bad_storage: bool,
|
||||||
|
) -> (
|
||||||
|
Arc<Controller>,
|
||||||
|
ServiceConfig,
|
||||||
|
TcpListener,
|
||||||
|
tempfile::TempDir,
|
||||||
|
) {
|
||||||
|
let temp = tempfile::tempdir().unwrap();
|
||||||
|
let mut config = Config::default();
|
||||||
|
config.network.drain_timeout_ms = 100;
|
||||||
|
config.monitoring.failure_threshold = 1;
|
||||||
|
config.monitoring.recovery_threshold = 1;
|
||||||
|
config.monitoring.cooldown_ms = 0;
|
||||||
|
config.web.state_file = temp.path().join("state.json");
|
||||||
|
if bad_storage {
|
||||||
|
let parent = temp.path().join("blocked");
|
||||||
|
std::fs::write(&parent, "x").unwrap();
|
||||||
|
config.web.state_file = parent.join("state.json");
|
||||||
|
}
|
||||||
|
let upstream = TcpListener::bind("127.0.0.1:0").await.unwrap();
|
||||||
|
let reserve = TcpListener::bind("127.0.0.1:0").await.unwrap();
|
||||||
|
let listen = reserve.local_addr().unwrap();
|
||||||
|
drop(reserve);
|
||||||
|
let service = ServiceConfig {
|
||||||
|
id: "cancel".into(),
|
||||||
|
name: "Cancel fixture".into(),
|
||||||
|
listen,
|
||||||
|
upstream_port: upstream.local_addr().unwrap().port(),
|
||||||
|
primary: None,
|
||||||
|
replicas: vec![],
|
||||||
|
enabled: true,
|
||||||
|
auto_switch: false,
|
||||||
|
replica_policy: ReplicaPolicy::Isolate,
|
||||||
|
custom: true,
|
||||||
|
custom_ip: Some("127.0.0.1".parse().unwrap()),
|
||||||
|
};
|
||||||
|
(
|
||||||
|
Controller::start(config).await.unwrap(),
|
||||||
|
service,
|
||||||
|
upstream,
|
||||||
|
temp,
|
||||||
|
)
|
||||||
|
}
|
||||||
|
fn gate(controller: &Controller, after_save: bool) -> Arc<PersistenceGate> {
|
||||||
|
let gate = Arc::new(PersistenceGate {
|
||||||
|
after_save,
|
||||||
|
reached: tokio::sync::Notify::new(),
|
||||||
|
resume: tokio::sync::Notify::new(),
|
||||||
|
});
|
||||||
|
*controller.persistence_gate.lock().unwrap() = Some(gate.clone());
|
||||||
|
gate
|
||||||
|
}
|
||||||
|
async fn until(mut condition: impl FnMut() -> bool) {
|
||||||
|
timeout(Duration::from_secs(1), async {
|
||||||
|
while !condition() {
|
||||||
|
tokio::task::yield_now().await;
|
||||||
|
}
|
||||||
|
})
|
||||||
|
.await
|
||||||
|
.unwrap();
|
||||||
|
}
|
||||||
|
#[tokio::test]
|
||||||
|
async fn dropped_delete_caller_does_not_abandon_saved_topology_or_stream_drain() {
|
||||||
|
let (controller, mut service, upstream, _temp) = fixture(false).await;
|
||||||
|
controller.put_service(service.clone()).await.unwrap();
|
||||||
|
let mut client = TcpStream::connect(service.listen).await.unwrap();
|
||||||
|
let (_peer, _) = timeout(Duration::from_secs(1), upstream.accept())
|
||||||
|
.await
|
||||||
|
.unwrap()
|
||||||
|
.unwrap();
|
||||||
|
service.enabled = false;
|
||||||
|
controller.put_service(service).await.unwrap();
|
||||||
|
let caller = tokio::spawn({
|
||||||
|
let controller = controller.clone();
|
||||||
|
async move { controller.delete_service("cancel").await }
|
||||||
|
});
|
||||||
|
until(|| {
|
||||||
|
store::Topology::load(&controller.config)
|
||||||
|
.unwrap()
|
||||||
|
.services
|
||||||
|
.is_empty()
|
||||||
|
})
|
||||||
|
.await;
|
||||||
|
caller.abort();
|
||||||
|
let _ = caller.await;
|
||||||
|
until(|| controller.topology().services.is_empty()).await;
|
||||||
|
assert!(controller.engine.snapshots().is_empty());
|
||||||
|
let mut byte = [0];
|
||||||
|
let result = timeout(Duration::from_millis(200), client.read(&mut byte))
|
||||||
|
.await
|
||||||
|
.unwrap();
|
||||||
|
assert!(matches!(result, Ok(0) | Err(_)));
|
||||||
|
assert!(
|
||||||
|
store::Topology::load(&controller.config)
|
||||||
|
.unwrap()
|
||||||
|
.services
|
||||||
|
.is_empty()
|
||||||
|
);
|
||||||
|
controller.shutdown().await;
|
||||||
|
}
|
||||||
|
#[tokio::test]
|
||||||
|
async fn dropped_put_caller_after_save_still_publishes_runtime_and_topology() {
|
||||||
|
let (controller, service, upstream, _temp) = fixture(false).await;
|
||||||
|
let barrier = gate(&controller, true);
|
||||||
|
let listen = service.listen;
|
||||||
|
let caller = tokio::spawn({
|
||||||
|
let controller = controller.clone();
|
||||||
|
async move { controller.put_service(service).await }
|
||||||
|
});
|
||||||
|
timeout(Duration::from_secs(1), barrier.reached.notified())
|
||||||
|
.await
|
||||||
|
.unwrap();
|
||||||
|
assert_eq!(
|
||||||
|
store::Topology::load(&controller.config)
|
||||||
|
.unwrap()
|
||||||
|
.services
|
||||||
|
.len(),
|
||||||
|
1
|
||||||
|
);
|
||||||
|
assert!(controller.topology().services.is_empty());
|
||||||
|
caller.abort();
|
||||||
|
let _ = caller.await;
|
||||||
|
barrier.resume.notify_one();
|
||||||
|
until(|| controller.topology().services.len() == 1).await;
|
||||||
|
assert_eq!(
|
||||||
|
store::Topology::load(&controller.config).unwrap().services,
|
||||||
|
controller.topology().services
|
||||||
|
);
|
||||||
|
let mut client = TcpStream::connect(listen).await.unwrap();
|
||||||
|
client.write_all(b"still routes").await.unwrap();
|
||||||
|
let (mut peer, _) = timeout(Duration::from_secs(1), upstream.accept())
|
||||||
|
.await
|
||||||
|
.unwrap()
|
||||||
|
.unwrap();
|
||||||
|
let mut bytes = [0; 12];
|
||||||
|
peer.read_exact(&mut bytes).await.unwrap();
|
||||||
|
assert_eq!(&bytes, b"still routes");
|
||||||
|
controller.shutdown().await;
|
||||||
|
}
|
||||||
|
#[tokio::test]
|
||||||
|
async fn dropped_put_caller_before_failed_save_still_rolls_back_listener_and_hidden_team() {
|
||||||
|
let (controller, service, _upstream, _temp) = fixture(true).await;
|
||||||
|
let barrier = gate(&controller, false);
|
||||||
|
let listen = service.listen;
|
||||||
|
let caller = tokio::spawn({
|
||||||
|
let controller = controller.clone();
|
||||||
|
async move { controller.put_service(service).await }
|
||||||
|
});
|
||||||
|
timeout(Duration::from_secs(1), barrier.reached.notified())
|
||||||
|
.await
|
||||||
|
.unwrap();
|
||||||
|
assert!(controller.engine.snapshots()["cancel"].running);
|
||||||
|
caller.abort();
|
||||||
|
let _ = caller.await;
|
||||||
|
barrier.resume.notify_one();
|
||||||
|
until(|| controller.engine.snapshots().is_empty()).await;
|
||||||
|
assert!(controller.topology().services.is_empty());
|
||||||
|
assert!(controller.topology().teams.is_empty());
|
||||||
|
assert!(!controller.config.web.state_file.exists());
|
||||||
|
assert!(TcpStream::connect(listen).await.is_err());
|
||||||
|
controller.shutdown().await;
|
||||||
|
}
|
||||||
|
#[tokio::test]
|
||||||
|
async fn dropped_team_caller_after_save_still_publishes_team() {
|
||||||
|
let (controller, _service, _upstream, _temp) = fixture(false).await;
|
||||||
|
let barrier = gate(&controller, true);
|
||||||
|
let team = Team {
|
||||||
|
id: "operator".into(),
|
||||||
|
name: "Operator".into(),
|
||||||
|
address: "127.0.0.1".parse().unwrap(),
|
||||||
|
hidden: false,
|
||||||
|
};
|
||||||
|
let caller = tokio::spawn({
|
||||||
|
let controller = controller.clone();
|
||||||
|
async move { controller.put_team(team).await }
|
||||||
|
});
|
||||||
|
timeout(Duration::from_secs(1), barrier.reached.notified())
|
||||||
|
.await
|
||||||
|
.unwrap();
|
||||||
|
caller.abort();
|
||||||
|
let _ = caller.await;
|
||||||
|
barrier.resume.notify_one();
|
||||||
|
until(|| controller.topology().teams.len() == 1).await;
|
||||||
|
assert_eq!(
|
||||||
|
store::Topology::load(&controller.config).unwrap().teams,
|
||||||
|
controller.topology().teams
|
||||||
|
);
|
||||||
|
controller.shutdown().await;
|
||||||
|
}
|
||||||
|
#[tokio::test]
|
||||||
|
async fn dropped_auto_selection_caller_after_save_still_publishes_promotion() {
|
||||||
|
let (controller, mut service, _upstream, _temp) = fixture(false).await;
|
||||||
|
controller
|
||||||
|
.put_team(Team {
|
||||||
|
id: "replica".into(),
|
||||||
|
name: "Replica".into(),
|
||||||
|
address: "::1".parse().unwrap(),
|
||||||
|
hidden: false,
|
||||||
|
})
|
||||||
|
.await
|
||||||
|
.unwrap();
|
||||||
|
service.auto_switch = true;
|
||||||
|
service.replicas = vec!["replica".into()];
|
||||||
|
controller.put_service(service).await.unwrap();
|
||||||
|
let barrier = gate(&controller, true);
|
||||||
|
let caller = tokio::spawn({
|
||||||
|
let controller = controller.clone();
|
||||||
|
async move {
|
||||||
|
controller
|
||||||
|
.check_cycle_with_probe(&CancellationToken::new(), |address| async move {
|
||||||
|
address.is_ipv6()
|
||||||
|
})
|
||||||
|
.await;
|
||||||
|
}
|
||||||
|
});
|
||||||
|
timeout(Duration::from_secs(1), barrier.reached.notified())
|
||||||
|
.await
|
||||||
|
.unwrap();
|
||||||
|
caller.abort();
|
||||||
|
let _ = caller.await;
|
||||||
|
barrier.resume.notify_one();
|
||||||
|
until(|| controller.topology().services[0].primary.as_deref() == Some("replica")).await;
|
||||||
|
let live = controller.topology();
|
||||||
|
assert_eq!(
|
||||||
|
store::Topology::load(&controller.config).unwrap().services,
|
||||||
|
live.services
|
||||||
|
);
|
||||||
|
assert!(live.services[0].custom_ip.is_none());
|
||||||
|
assert_eq!(live.services[0].replicas.len(), 1);
|
||||||
|
controller.shutdown().await;
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
246
src/engine.rs
246
src/engine.rs
@ -24,6 +24,7 @@ use tokio_util::sync::CancellationToken;
|
|||||||
|
|
||||||
#[derive(Clone, Debug, Default, Serialize)]
|
#[derive(Clone, Debug, Default, Serialize)]
|
||||||
pub struct TrafficStats {
|
pub struct TrafficStats {
|
||||||
|
pub accept_errors: u64,
|
||||||
pub accepted: u64,
|
pub accepted: u64,
|
||||||
pub active: u64,
|
pub active: u64,
|
||||||
pub completed: u64,
|
pub completed: u64,
|
||||||
@ -44,6 +45,7 @@ pub struct EngineStatus {
|
|||||||
}
|
}
|
||||||
#[derive(Default)]
|
#[derive(Default)]
|
||||||
struct Counters {
|
struct Counters {
|
||||||
|
accept_errors: AtomicU64,
|
||||||
accepted: AtomicU64,
|
accepted: AtomicU64,
|
||||||
active: AtomicU64,
|
active: AtomicU64,
|
||||||
completed: AtomicU64,
|
completed: AtomicU64,
|
||||||
@ -59,6 +61,7 @@ struct Counters {
|
|||||||
impl Counters {
|
impl Counters {
|
||||||
fn snapshot(&self) -> TrafficStats {
|
fn snapshot(&self) -> TrafficStats {
|
||||||
TrafficStats {
|
TrafficStats {
|
||||||
|
accept_errors: self.accept_errors.load(Relaxed),
|
||||||
accepted: self.accepted.load(Relaxed),
|
accepted: self.accepted.load(Relaxed),
|
||||||
active: self.active.load(Relaxed),
|
active: self.active.load(Relaxed),
|
||||||
completed: self.completed.load(Relaxed),
|
completed: self.completed.load(Relaxed),
|
||||||
@ -106,6 +109,7 @@ struct Inner {
|
|||||||
permits: Arc<Semaphore>,
|
permits: Arc<Semaphore>,
|
||||||
payload_budget: Arc<Semaphore>,
|
payload_budget: Arc<Semaphore>,
|
||||||
stopped: AtomicBool,
|
stopped: AtomicBool,
|
||||||
|
quiesced: AtomicBool,
|
||||||
}
|
}
|
||||||
#[derive(Clone)]
|
#[derive(Clone)]
|
||||||
pub struct Engine {
|
pub struct Engine {
|
||||||
@ -121,6 +125,7 @@ impl Engine {
|
|||||||
services: Mutex::new(BTreeMap::new()),
|
services: Mutex::new(BTreeMap::new()),
|
||||||
operations: tokio::sync::Mutex::new(()),
|
operations: tokio::sync::Mutex::new(()),
|
||||||
stopped: AtomicBool::new(false),
|
stopped: AtomicBool::new(false),
|
||||||
|
quiesced: AtomicBool::new(false),
|
||||||
}),
|
}),
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@ -187,7 +192,7 @@ impl Engine {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
// Bind before changing a working listener, so bind failure leaves it intact.
|
// Bind before changing a working listener, so bind failure leaves it intact.
|
||||||
let listener = if service.enabled {
|
let listener = if service.enabled && !self.inner.quiesced.load(Relaxed) {
|
||||||
Some(
|
Some(
|
||||||
TcpListener::bind(service.listen)
|
TcpListener::bind(service.listen)
|
||||||
.await
|
.await
|
||||||
@ -283,6 +288,23 @@ impl Engine {
|
|||||||
}
|
}
|
||||||
Ok(())
|
Ok(())
|
||||||
}
|
}
|
||||||
|
/// Permanently stop new acceptance without interrupting sessions or preventing
|
||||||
|
/// an in-flight controller transaction from committing or rolling back its route.
|
||||||
|
pub async fn quiesce(&self) {
|
||||||
|
self.inner.quiesced.store(true, Relaxed);
|
||||||
|
let _operation = self.inner.operations.lock().await;
|
||||||
|
let services: Vec<_> = self
|
||||||
|
.inner
|
||||||
|
.services
|
||||||
|
.lock()
|
||||||
|
.unwrap()
|
||||||
|
.values()
|
||||||
|
.cloned()
|
||||||
|
.collect();
|
||||||
|
for service in services {
|
||||||
|
stop_accepting(&service).await;
|
||||||
|
}
|
||||||
|
}
|
||||||
pub async fn shutdown(&self) {
|
pub async fn shutdown(&self) {
|
||||||
let _operation = self.inner.operations.lock().await;
|
let _operation = self.inner.operations.lock().await;
|
||||||
self.inner.stopped.store(true, Relaxed);
|
self.inner.stopped.store(true, Relaxed);
|
||||||
@ -372,8 +394,28 @@ async fn retire(s: &Service, drain_ms: u64) {
|
|||||||
}
|
}
|
||||||
join_generations(s).await;
|
join_generations(s).await;
|
||||||
}
|
}
|
||||||
async fn supervise(
|
trait Acceptor {
|
||||||
listener: TcpListener,
|
fn accept(
|
||||||
|
&self,
|
||||||
|
) -> impl std::future::Future<Output = std::io::Result<(TcpStream, SocketAddr)>> + Send;
|
||||||
|
}
|
||||||
|
impl Acceptor for TcpListener {
|
||||||
|
async fn accept(&self) -> std::io::Result<(TcpStream, SocketAddr)> {
|
||||||
|
TcpListener::accept(self).await
|
||||||
|
}
|
||||||
|
}
|
||||||
|
fn recoverable_accept_error(error: &std::io::Error) -> bool {
|
||||||
|
// Unknown/resource-pressure OS errors are retried too: EMFILE/ENFILE and
|
||||||
|
// buffer exhaustion do not imply that the listening socket is unusable.
|
||||||
|
!matches!(
|
||||||
|
error.kind(),
|
||||||
|
std::io::ErrorKind::InvalidInput
|
||||||
|
| std::io::ErrorKind::Unsupported
|
||||||
|
| std::io::ErrorKind::NotConnected
|
||||||
|
)
|
||||||
|
}
|
||||||
|
async fn supervise<A: Acceptor>(
|
||||||
|
listener: A,
|
||||||
service: Arc<Service>,
|
service: Arc<Service>,
|
||||||
settings: NetworkConfig,
|
settings: NetworkConfig,
|
||||||
budgets: (Arc<Semaphore>, Arc<Semaphore>),
|
budgets: (Arc<Semaphore>, Arc<Semaphore>),
|
||||||
@ -383,6 +425,7 @@ async fn supervise(
|
|||||||
) {
|
) {
|
||||||
let (permits, payload_budget) = budgets;
|
let (permits, payload_budget) = budgets;
|
||||||
let mut sessions = JoinSet::new();
|
let mut sessions = JoinSet::new();
|
||||||
|
let mut retry_delay = Duration::from_millis(50);
|
||||||
loop {
|
loop {
|
||||||
tokio::select! {
|
tokio::select! {
|
||||||
biased;
|
biased;
|
||||||
@ -392,6 +435,7 @@ async fn supervise(
|
|||||||
_=sessions.join_next(), if !sessions.is_empty()=>{},
|
_=sessions.join_next(), if !sessions.is_empty()=>{},
|
||||||
result=listener.accept()=>match result {
|
result=listener.accept()=>match result {
|
||||||
Ok((client,_))=> {
|
Ok((client,_))=> {
|
||||||
|
retry_delay = Duration::from_millis(50);
|
||||||
service.counters.accepted.fetch_add(1,Relaxed);
|
service.counters.accepted.fetch_add(1,Relaxed);
|
||||||
let Ok(permit)=permits.clone().try_acquire_owned() else { service.counters.rejected.fetch_add(1,Relaxed); drop(client); continue };
|
let Ok(permit)=permits.clone().try_acquire_owned() else { service.counters.rejected.fetch_add(1,Relaxed); drop(client); continue };
|
||||||
let route=service.route.read().unwrap().clone();
|
let route=service.route.read().unwrap().clone();
|
||||||
@ -421,11 +465,19 @@ async fn supervise(
|
|||||||
while workers.join_next().await.is_some() {}
|
while workers.join_next().await.is_some() {}
|
||||||
});
|
});
|
||||||
}
|
}
|
||||||
Err(error)=> { *service.error.lock().unwrap()=Some(format!("accept: {error}")); service.running.store(false,Relaxed); break; }
|
Err(error)=> {
|
||||||
|
service.counters.accept_errors.fetch_add(1,Relaxed);
|
||||||
|
*service.error.lock().unwrap()=Some(format!("accept: {error}"));
|
||||||
|
if !recoverable_accept_error(&error) { service.running.store(false,Relaxed); break; }
|
||||||
|
tracing::warn!(listen=%service.listen, %error, retry_ms=retry_delay.as_millis(), "listener accept failed; retrying");
|
||||||
|
tokio::select! { biased; _=stop.cancelled()=>break, _=tokio::time::sleep(retry_delay)=>{} }
|
||||||
|
retry_delay = (retry_delay * 2).min(Duration::from_secs(1));
|
||||||
|
}
|
||||||
},
|
},
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
drop(listener);
|
drop(listener);
|
||||||
|
service.running.store(false, Relaxed);
|
||||||
let _ = stopped.send(());
|
let _ = stopped.send(());
|
||||||
while sessions.join_next().await.is_some() {}
|
while sessions.join_next().await.is_some() {}
|
||||||
}
|
}
|
||||||
@ -682,3 +734,189 @@ async fn write_counted<W: AsyncWrite + Unpin>(
|
|||||||
}
|
}
|
||||||
Ok(())
|
Ok(())
|
||||||
}
|
}
|
||||||
|
|
||||||
|
#[cfg(test)]
|
||||||
|
mod accept_tests {
|
||||||
|
use super::*;
|
||||||
|
use std::{
|
||||||
|
collections::VecDeque,
|
||||||
|
io::{Error, ErrorKind},
|
||||||
|
};
|
||||||
|
struct FaultListener {
|
||||||
|
socket: TcpListener,
|
||||||
|
steps: Mutex<VecDeque<Option<ErrorKind>>>,
|
||||||
|
faults: mpsc::UnboundedSender<tokio::time::Instant>,
|
||||||
|
}
|
||||||
|
impl Acceptor for FaultListener {
|
||||||
|
async fn accept(&self) -> std::io::Result<(TcpStream, SocketAddr)> {
|
||||||
|
let fault = self.steps.lock().unwrap().pop_front().flatten();
|
||||||
|
if let Some(kind) = fault {
|
||||||
|
let _ = self.faults.send(tokio::time::Instant::now());
|
||||||
|
return Err(Error::from(kind));
|
||||||
|
}
|
||||||
|
self.socket.accept().await
|
||||||
|
}
|
||||||
|
}
|
||||||
|
async fn fixture(
|
||||||
|
steps: Vec<Option<ErrorKind>>,
|
||||||
|
upstream: SocketAddr,
|
||||||
|
) -> (
|
||||||
|
SocketAddr,
|
||||||
|
Arc<Service>,
|
||||||
|
CancellationToken,
|
||||||
|
oneshot::Receiver<()>,
|
||||||
|
JoinHandle<()>,
|
||||||
|
mpsc::UnboundedReceiver<tokio::time::Instant>,
|
||||||
|
) {
|
||||||
|
let socket = TcpListener::bind("127.0.0.1:0").await.unwrap();
|
||||||
|
let listen = socket.local_addr().unwrap();
|
||||||
|
let service = Arc::new(Service {
|
||||||
|
listen,
|
||||||
|
running: AtomicBool::new(true),
|
||||||
|
route: Arc::new(RwLock::new(Some(Route {
|
||||||
|
primary: upstream,
|
||||||
|
replicas: vec![],
|
||||||
|
policy: ReplicaPolicy::Isolate,
|
||||||
|
}))),
|
||||||
|
counters: Arc::new(Counters::default()),
|
||||||
|
error: Arc::new(Mutex::new(None)),
|
||||||
|
generations: Mutex::new(vec![]),
|
||||||
|
});
|
||||||
|
let stop = CancellationToken::new();
|
||||||
|
let (tx, rx) = oneshot::channel();
|
||||||
|
let (fault_tx, faults) = mpsc::unbounded_channel();
|
||||||
|
let settings = NetworkConfig::default();
|
||||||
|
let worker = tokio::spawn(supervise(
|
||||||
|
FaultListener {
|
||||||
|
socket,
|
||||||
|
steps: Mutex::new(steps.into()),
|
||||||
|
faults: fault_tx,
|
||||||
|
},
|
||||||
|
service.clone(),
|
||||||
|
settings.clone(),
|
||||||
|
(
|
||||||
|
Arc::new(Semaphore::new(4)),
|
||||||
|
Arc::new(Semaphore::new(settings.max_buffered_bytes)),
|
||||||
|
),
|
||||||
|
stop.clone(),
|
||||||
|
CancellationToken::new(),
|
||||||
|
tx,
|
||||||
|
));
|
||||||
|
(listen, service, stop, rx, worker, faults)
|
||||||
|
}
|
||||||
|
#[tokio::test]
|
||||||
|
async fn recoverable_accept_failure_retries_then_delivers_real_bytes_and_resets_backoff() {
|
||||||
|
let upstream = TcpListener::bind("127.0.0.1:0").await.unwrap();
|
||||||
|
let upstream_address = upstream.local_addr().unwrap();
|
||||||
|
let echo = tokio::spawn(async move {
|
||||||
|
let (mut stream, _) = upstream.accept().await.unwrap();
|
||||||
|
let mut bytes = vec![];
|
||||||
|
stream.read_to_end(&mut bytes).await.unwrap();
|
||||||
|
stream.write_all(&bytes).await.unwrap();
|
||||||
|
});
|
||||||
|
let fault = Some(ErrorKind::OutOfMemory);
|
||||||
|
let (listen, service, stop, stopped, worker, mut faults) = fixture(
|
||||||
|
vec![fault, fault, fault, None, fault, fault],
|
||||||
|
upstream_address,
|
||||||
|
)
|
||||||
|
.await;
|
||||||
|
let mut failures = vec![];
|
||||||
|
for _ in 0..3 {
|
||||||
|
failures.push(
|
||||||
|
timeout(Duration::from_secs(1), faults.recv())
|
||||||
|
.await
|
||||||
|
.unwrap()
|
||||||
|
.unwrap(),
|
||||||
|
);
|
||||||
|
}
|
||||||
|
assert!(
|
||||||
|
failures
|
||||||
|
.windows(2)
|
||||||
|
.all(|pair| pair[1].duration_since(pair[0]) >= Duration::from_millis(45)),
|
||||||
|
"accept errors busy-looped"
|
||||||
|
);
|
||||||
|
let mut client = TcpStream::connect(listen).await.unwrap();
|
||||||
|
client
|
||||||
|
.write_all(b"accept retry preserves payload")
|
||||||
|
.await
|
||||||
|
.unwrap();
|
||||||
|
client.shutdown().await.unwrap();
|
||||||
|
let mut bytes = vec![];
|
||||||
|
timeout(Duration::from_secs(2), client.read_to_end(&mut bytes))
|
||||||
|
.await
|
||||||
|
.unwrap()
|
||||||
|
.unwrap();
|
||||||
|
assert_eq!(bytes, b"accept retry preserves payload");
|
||||||
|
let first = timeout(Duration::from_secs(1), faults.recv())
|
||||||
|
.await
|
||||||
|
.unwrap()
|
||||||
|
.unwrap();
|
||||||
|
let second = timeout(Duration::from_secs(1), faults.recv())
|
||||||
|
.await
|
||||||
|
.unwrap()
|
||||||
|
.unwrap();
|
||||||
|
assert!(
|
||||||
|
second.duration_since(first) < Duration::from_millis(250),
|
||||||
|
"success did not reset accept backoff"
|
||||||
|
);
|
||||||
|
assert_eq!(service.counters.accept_errors.load(Relaxed), 5);
|
||||||
|
assert!(service.running.load(Relaxed));
|
||||||
|
stop.cancel();
|
||||||
|
timeout(Duration::from_millis(100), stopped)
|
||||||
|
.await
|
||||||
|
.unwrap()
|
||||||
|
.unwrap();
|
||||||
|
timeout(Duration::from_millis(100), worker)
|
||||||
|
.await
|
||||||
|
.unwrap()
|
||||||
|
.unwrap();
|
||||||
|
echo.await.unwrap();
|
||||||
|
assert!(TcpStream::connect(listen).await.is_err());
|
||||||
|
}
|
||||||
|
#[tokio::test]
|
||||||
|
async fn shutdown_interrupts_accept_backoff_and_terminal_failure_is_reported() {
|
||||||
|
let (listen, service, stop, stopped, worker, mut faults) = fixture(
|
||||||
|
vec![Some(ErrorKind::OutOfMemory); 8],
|
||||||
|
"127.0.0.1:9".parse().unwrap(),
|
||||||
|
)
|
||||||
|
.await;
|
||||||
|
// Four failures schedule a 400 ms backoff. Cancellation must not wait for it.
|
||||||
|
for _ in 0..4 {
|
||||||
|
timeout(Duration::from_secs(1), faults.recv())
|
||||||
|
.await
|
||||||
|
.unwrap()
|
||||||
|
.unwrap();
|
||||||
|
}
|
||||||
|
assert!(
|
||||||
|
service
|
||||||
|
.error
|
||||||
|
.lock()
|
||||||
|
.unwrap()
|
||||||
|
.as_ref()
|
||||||
|
.unwrap()
|
||||||
|
.starts_with("accept:")
|
||||||
|
);
|
||||||
|
stop.cancel();
|
||||||
|
timeout(Duration::from_millis(100), stopped)
|
||||||
|
.await
|
||||||
|
.unwrap()
|
||||||
|
.unwrap();
|
||||||
|
timeout(Duration::from_millis(100), worker)
|
||||||
|
.await
|
||||||
|
.unwrap()
|
||||||
|
.unwrap();
|
||||||
|
assert!(TcpStream::connect(listen).await.is_err());
|
||||||
|
let (_, service, _, stopped, worker, _) = fixture(
|
||||||
|
vec![Some(ErrorKind::InvalidInput)],
|
||||||
|
"127.0.0.1:9".parse().unwrap(),
|
||||||
|
)
|
||||||
|
.await;
|
||||||
|
timeout(Duration::from_millis(100), stopped)
|
||||||
|
.await
|
||||||
|
.unwrap()
|
||||||
|
.unwrap();
|
||||||
|
worker.await.unwrap();
|
||||||
|
assert!(!service.running.load(Relaxed));
|
||||||
|
assert_eq!(service.counters.accept_errors.load(Relaxed), 1);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|||||||
51
src/main.rs
51
src/main.rs
@ -1,6 +1,7 @@
|
|||||||
use anyhow::{Context, Result};
|
use anyhow::{Context, Result};
|
||||||
use clap::{Parser, Subcommand};
|
use clap::{Parser, Subcommand};
|
||||||
use mirror_tool::{auth::Auth, config::Config, control::Controller, web};
|
use mirror_tool::{auth::Auth, config::Config, control::Controller, web};
|
||||||
|
use std::future::IntoFuture;
|
||||||
use std::path::PathBuf;
|
use std::path::PathBuf;
|
||||||
use tokio_util::sync::CancellationToken;
|
use tokio_util::sync::CancellationToken;
|
||||||
|
|
||||||
@ -20,8 +21,17 @@ enum Command {
|
|||||||
Example,
|
Example,
|
||||||
}
|
}
|
||||||
|
|
||||||
#[tokio::main]
|
fn main() -> Result<()> {
|
||||||
async fn main() -> Result<()> {
|
let runtime = tokio::runtime::Builder::new_multi_thread()
|
||||||
|
.enable_all()
|
||||||
|
.build()?;
|
||||||
|
let result = runtime.block_on(run());
|
||||||
|
// A filesystem worker can remain blocked in the OS after the async deadline.
|
||||||
|
// Do not make runtime destruction an unbounded second shutdown phase.
|
||||||
|
runtime.shutdown_timeout(std::time::Duration::from_millis(100));
|
||||||
|
result
|
||||||
|
}
|
||||||
|
async fn run() -> Result<()> {
|
||||||
tracing_subscriber::fmt()
|
tracing_subscriber::fmt()
|
||||||
.with_env_filter(
|
.with_env_filter(
|
||||||
tracing_subscriber::EnvFilter::try_from_default_env()
|
tracing_subscriber::EnvFilter::try_from_default_env()
|
||||||
@ -64,19 +74,32 @@ async fn main() -> Result<()> {
|
|||||||
router.into_make_service_with_connect_info::<std::net::SocketAddr>(),
|
router.into_make_service_with_connect_info::<std::net::SocketAddr>(),
|
||||||
);
|
);
|
||||||
let signal_cancel = cancel.clone();
|
let signal_cancel = cancel.clone();
|
||||||
let result = server
|
let server = server
|
||||||
.with_graceful_shutdown(async move {
|
.with_graceful_shutdown(signal_cancel.cancelled_owned())
|
||||||
shutdown_signal().await;
|
.into_future();
|
||||||
signal_cancel.cancel();
|
tokio::pin!(server);
|
||||||
})
|
let result = tokio::select! {
|
||||||
.await;
|
result = &mut server => result,
|
||||||
|
_ = shutdown_signal() => {
|
||||||
|
cancel.cancel();
|
||||||
|
// HTTP body extraction has no idle deadline during normal operation.
|
||||||
|
// Shutdown alone bounds it, while serialized control transactions finish first.
|
||||||
|
let shutdown = async {
|
||||||
|
controller.shutdown().await;
|
||||||
|
let _ = monitor.await;
|
||||||
|
if let Some(board) = board { let _ = board.await; }
|
||||||
|
(&mut server).await
|
||||||
|
};
|
||||||
|
match tokio::time::timeout(
|
||||||
|
std::time::Duration::from_millis(controller.config.network.drain_timeout_ms)
|
||||||
|
+ std::time::Duration::from_secs(1), shutdown).await {
|
||||||
|
Ok(result) => result,
|
||||||
|
Err(_) => { tracing::warn!("shutdown deadline reached"); return Ok(()); }
|
||||||
|
}
|
||||||
|
}
|
||||||
|
};
|
||||||
cancel.cancel();
|
cancel.cancel();
|
||||||
// Complete in-flight monitor transactions before tearing down the engine.
|
controller.shutdown().await;
|
||||||
let _ = monitor.await;
|
|
||||||
if let Some(board) = board {
|
|
||||||
let _ = board.await;
|
|
||||||
}
|
|
||||||
controller.engine.shutdown().await;
|
|
||||||
result.context("HTTP server")?;
|
result.context("HTTP server")?;
|
||||||
Ok(())
|
Ok(())
|
||||||
}
|
}
|
||||||
|
|||||||
@ -311,24 +311,38 @@ impl Scoreboard {
|
|||||||
base.query_pairs_mut().append_pair("sid", sid);
|
base.query_pairs_mut().append_pair("sid", sid);
|
||||||
let ping = hello["pingInterval"].as_u64().unwrap_or(25000);
|
let ping = hello["pingInterval"].as_u64().unwrap_or(25000);
|
||||||
let timeout = hello["pingTimeout"].as_u64().unwrap_or(20000);
|
let timeout = hello["pingTimeout"].as_u64().unwrap_or(20000);
|
||||||
let request_timeout =
|
let request_timeout = Duration::from_millis(ping.saturating_add(timeout).clamp(1, 120000));
|
||||||
Duration::from_millis(ping.saturating_add(timeout).clamp(1000, 120000));
|
let mut heartbeat_deadline = tokio::time::Instant::now() + request_timeout;
|
||||||
post(client, &base, "40/game_events,").await?;
|
tokio::time::timeout_at(heartbeat_deadline, post(client, &base, "40/game_events,"))
|
||||||
|
.await
|
||||||
|
.context("ForcAD namespace deadline")??;
|
||||||
let mut initialized = false;
|
let mut initialized = false;
|
||||||
let connected_at = tokio::time::Instant::now();
|
let connected_at = tokio::time::Instant::now();
|
||||||
loop {
|
loop {
|
||||||
let response = client
|
let response = client
|
||||||
.get(base.clone())
|
.get(base.clone())
|
||||||
.timeout(request_timeout)
|
.timeout(heartbeat_deadline.saturating_duration_since(tokio::time::Instant::now()))
|
||||||
.send()
|
.send()
|
||||||
.await
|
.await
|
||||||
.map_err(|_| anyhow::anyhow!("ForcAD poll failed or heartbeat timed out"))?;
|
.map_err(|_| anyhow::anyhow!("ForcAD poll failed or heartbeat timed out"))?;
|
||||||
let bytes = body(response).await?;
|
let bytes = tokio::time::timeout_at(heartbeat_deadline, body(response))
|
||||||
|
.await
|
||||||
|
.context("ForcAD heartbeat deadline")??;
|
||||||
|
ensure!(
|
||||||
|
tokio::time::Instant::now() < heartbeat_deadline,
|
||||||
|
"ForcAD heartbeat deadline"
|
||||||
|
);
|
||||||
let text = std::str::from_utf8(&bytes).context("ForcAD poll encoding")?;
|
let text = std::str::from_utf8(&bytes).context("ForcAD poll encoding")?;
|
||||||
ensure!(!text.is_empty(), "ForcAD empty polling response");
|
ensure!(!text.is_empty(), "ForcAD empty polling response");
|
||||||
for frame in text.split('\u{1e}') {
|
for frame in text.split('\u{1e}') {
|
||||||
if let Some(ping) = frame.strip_prefix('2') {
|
if let Some(ping) = frame.strip_prefix('2') {
|
||||||
post(client, &base, &format!("3{ping}")).await?;
|
heartbeat_deadline = tokio::time::Instant::now() + request_timeout;
|
||||||
|
tokio::time::timeout_at(
|
||||||
|
heartbeat_deadline,
|
||||||
|
post(client, &base, &format!("3{ping}")),
|
||||||
|
)
|
||||||
|
.await
|
||||||
|
.context("ForcAD pong deadline")??;
|
||||||
} else if frame.starts_with("40/game_events,") {
|
} else if frame.starts_with("40/game_events,") {
|
||||||
self.connected();
|
self.connected();
|
||||||
} else if let Some(event) = frame.strip_prefix("42/game_events,") {
|
} else if let Some(event) = frame.strip_prefix("42/game_events,") {
|
||||||
|
|||||||
28
src/web.rs
28
src/web.rs
@ -53,6 +53,10 @@ pub fn router(controller: Arc<Controller>, auth: Auth, shutdown: CancellationTok
|
|||||||
.route("/", get(index))
|
.route("/", get(index))
|
||||||
.route("/app.js", get(js))
|
.route("/app.js", get(js))
|
||||||
.route("/app.css", get(css))
|
.route("/app.css", get(css))
|
||||||
|
.route("/vendor/bootstrap.min.css", get(bootstrap_css))
|
||||||
|
.route("/vendor/fontawesome.min.css", get(fontawesome_css))
|
||||||
|
.route("/webfonts/fa-solid-900.woff2", get(icon_font))
|
||||||
|
.route("/licenses.txt", get(licenses))
|
||||||
.route("/healthz", get(|| async { Json(json!({"status":"ok"})) }))
|
.route("/healthz", get(|| async { Json(json!({"status":"ok"})) }))
|
||||||
.route("/api/login", post(login))
|
.route("/api/login", post(login))
|
||||||
.merge(private)
|
.merge(private)
|
||||||
@ -76,6 +80,30 @@ async fn css() -> impl IntoResponse {
|
|||||||
include_str!("../web/app.css"),
|
include_str!("../web/app.css"),
|
||||||
)
|
)
|
||||||
}
|
}
|
||||||
|
async fn bootstrap_css() -> impl IntoResponse {
|
||||||
|
(
|
||||||
|
[(header::CONTENT_TYPE, "text/css; charset=utf-8")],
|
||||||
|
include_str!("../web/vendor/bootstrap.min.css"),
|
||||||
|
)
|
||||||
|
}
|
||||||
|
async fn fontawesome_css() -> impl IntoResponse {
|
||||||
|
(
|
||||||
|
[(header::CONTENT_TYPE, "text/css; charset=utf-8")],
|
||||||
|
include_str!("../web/vendor/fontawesome.min.css"),
|
||||||
|
)
|
||||||
|
}
|
||||||
|
async fn icon_font() -> impl IntoResponse {
|
||||||
|
(
|
||||||
|
[(header::CONTENT_TYPE, "font/woff2")],
|
||||||
|
include_bytes!("../web/vendor/fa-solid-900.woff2").as_slice(),
|
||||||
|
)
|
||||||
|
}
|
||||||
|
async fn licenses() -> impl IntoResponse {
|
||||||
|
(
|
||||||
|
[(header::CONTENT_TYPE, "text/plain; charset=utf-8")],
|
||||||
|
include_str!("../web/vendor/NOTICES.txt"),
|
||||||
|
)
|
||||||
|
}
|
||||||
fn error(status: StatusCode, message: impl ToString) -> Response {
|
fn error(status: StatusCode, message: impl ToString) -> Response {
|
||||||
(status, Json(json!({"error":message.to_string()}))).into_response()
|
(status, Json(json!({"error":message.to_string()}))).into_response()
|
||||||
}
|
}
|
||||||
|
|||||||
15
task_plan.md
15
task_plan.md
@ -5,11 +5,16 @@ One native executable with an async TCP engine and embedded web UI. No Python/ru
|
|||||||
|
|
||||||
## Phases
|
## Phases
|
||||||
1. [complete] Capture legacy behavior and verify ForcAD wire format; establish module contracts.
|
1. [complete] Capture legacy behavior and verify ForcAD wire format; establish module contracts.
|
||||||
2. [complete] Implement configuration, async forwarding and lifecycle/metrics (11 engine tests passed).
|
2. [complete] Async forwarding, lifecycle/metrics, backpressure and recoverable accept supervision.
|
||||||
3. [in progress] Implement scoreboard clients, health monitoring and deterministic failover; wire-level fixtures pending.
|
3. [complete] Scoreboard clients, observed-health freshness and deterministic failover.
|
||||||
4. [in progress] Implement HTTP API, authentication, persistence and embedded UI (UI implementer active).
|
4. [complete] Authenticated API/persistence and literal legacy UI restoration with original interactions.
|
||||||
5. [pending] Integration and failure tests; Linux/amd64 verification and throughput/latency benchmark.
|
5. [complete] Native and Orb integration/failure tests, ForcAD library interoperability, browser checks and measured benchmarks. Final fixed-source gates passed.
|
||||||
6. [pending] Independent review, fixes, deployment docs and evidence-backed handoff.
|
6. [complete] R6 cancellation safety fixed and independently accepted; final validation, documentation and local release artifacts complete.
|
||||||
|
|
||||||
|
## Latest correction and review gate
|
||||||
|
- Original appearance/workflows restored; source CSS parity and browser checks recorded in `docs/verification.md`.
|
||||||
|
- Initial `docs/review.md` is historical. Final findings and resolution belong in `docs/final-review.md`; R6 is independently verified and final review is approved.
|
||||||
|
- New code must preserve original columns, primary/replica interaction, direct-IP custom proxies, robot, own-team status strip and counters.
|
||||||
|
|
||||||
## Global constraints
|
## Global constraints
|
||||||
- User means upstream routing, not host firewall interception.
|
- User means upstream routing, not host firewall interception.
|
||||||
|
|||||||
@ -187,6 +187,7 @@ async fn state_failure_rolls_back_listener_and_topology() {
|
|||||||
config.web.state_file = bad_parent.join("state.json");
|
config.web.state_file = bad_parent.join("state.json");
|
||||||
let controller = Controller::start(config).await.unwrap();
|
let controller = Controller::start(config).await.unwrap();
|
||||||
let team = Team {
|
let team = Team {
|
||||||
|
hidden: false,
|
||||||
id: "a".into(),
|
id: "a".into(),
|
||||||
name: "A".into(),
|
name: "A".into(),
|
||||||
address: "127.0.0.1".parse().unwrap(),
|
address: "127.0.0.1".parse().unwrap(),
|
||||||
@ -223,17 +224,21 @@ async fn automatic_failover_is_durable_and_primary_requests_keep_working() {
|
|||||||
config.monitoring.timeout_ms = 100;
|
config.monitoring.timeout_ms = 100;
|
||||||
config.teams = vec![
|
config.teams = vec![
|
||||||
Team {
|
Team {
|
||||||
|
hidden: false,
|
||||||
id: "a".into(),
|
id: "a".into(),
|
||||||
name: "A".into(),
|
name: "A".into(),
|
||||||
address: "127.0.0.1".parse().unwrap(),
|
address: "127.0.0.1".parse().unwrap(),
|
||||||
},
|
},
|
||||||
Team {
|
Team {
|
||||||
|
hidden: false,
|
||||||
id: "b".into(),
|
id: "b".into(),
|
||||||
name: "B".into(),
|
name: "B".into(),
|
||||||
address: "::1".parse().unwrap(),
|
address: "::1".parse().unwrap(),
|
||||||
},
|
},
|
||||||
];
|
];
|
||||||
config.services = vec![ServiceConfig {
|
config.services = vec![ServiceConfig {
|
||||||
|
custom: true,
|
||||||
|
custom_ip: Some("127.0.0.1".parse().unwrap()),
|
||||||
id: "notes".into(),
|
id: "notes".into(),
|
||||||
name: "Notes".into(),
|
name: "Notes".into(),
|
||||||
listen,
|
listen,
|
||||||
@ -251,6 +256,8 @@ async fn automatic_failover_is_durable_and_primary_requests_keep_working() {
|
|||||||
controller.topology().services[0].primary.as_deref(),
|
controller.topology().services[0].primary.as_deref(),
|
||||||
Some("b")
|
Some("b")
|
||||||
);
|
);
|
||||||
|
assert!(controller.topology().services[0].custom_ip.is_none());
|
||||||
|
assert_eq!(controller.topology().services[0].replicas, vec!["a"]);
|
||||||
let mut client = tokio::net::TcpStream::connect(listen).await.unwrap();
|
let mut client = tokio::net::TcpStream::connect(listen).await.unwrap();
|
||||||
client.write_all(b"hello failover").await.unwrap();
|
client.write_all(b"hello failover").await.unwrap();
|
||||||
client.shutdown().await.unwrap();
|
client.shutdown().await.unwrap();
|
||||||
@ -269,3 +276,219 @@ async fn automatic_failover_is_durable_and_primary_requests_keep_working() {
|
|||||||
upstream.abort();
|
upstream.abort();
|
||||||
let _ = upstream.await;
|
let _ = upstream.await;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
fn direct_service(listen: SocketAddr, port: u16) -> ServiceConfig {
|
||||||
|
ServiceConfig {
|
||||||
|
id: "custom_notes".into(),
|
||||||
|
name: "Custom notes".into(),
|
||||||
|
listen,
|
||||||
|
upstream_port: port,
|
||||||
|
primary: None,
|
||||||
|
replicas: vec![],
|
||||||
|
enabled: true,
|
||||||
|
auto_switch: false,
|
||||||
|
replica_policy: ReplicaPolicy::Isolate,
|
||||||
|
custom: true,
|
||||||
|
custom_ip: Some("127.0.0.1".parse().unwrap()),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
#[tokio::test]
|
||||||
|
async fn custom_ip_is_atomic_reused_persisted_and_clearable() {
|
||||||
|
let temp = tempfile::tempdir().unwrap();
|
||||||
|
let mut cfg = Config::default();
|
||||||
|
cfg.web.state_file = temp.path().join("state.json");
|
||||||
|
let upstream = TcpListener::bind("127.0.0.1:0").await.unwrap();
|
||||||
|
let reserve = TcpListener::bind("127.0.0.1:0").await.unwrap();
|
||||||
|
let listen = reserve.local_addr().unwrap();
|
||||||
|
drop(reserve);
|
||||||
|
let controller = Controller::start(cfg.clone()).await.unwrap();
|
||||||
|
let mut service = direct_service(listen, upstream.local_addr().unwrap().port());
|
||||||
|
controller.put_service(service.clone()).await.unwrap();
|
||||||
|
let state = controller.topology();
|
||||||
|
assert_eq!(state.teams.len(), 1);
|
||||||
|
assert!(state.teams[0].hidden);
|
||||||
|
assert_eq!(state.teams[0].name, "127.0.0.1");
|
||||||
|
service.replicas = vec![state.teams[0].id.clone()];
|
||||||
|
controller.put_service(service.clone()).await.unwrap();
|
||||||
|
assert_eq!(controller.topology().teams.len(), 1);
|
||||||
|
assert!(controller.topology().services[0].replicas.is_empty());
|
||||||
|
let saved = mirror_tool::store::Topology::load(&cfg).unwrap();
|
||||||
|
assert_eq!(saved.services, controller.topology().services);
|
||||||
|
assert_eq!(saved.teams, controller.topology().teams);
|
||||||
|
service.primary = Some(state.teams[0].id.clone());
|
||||||
|
service.custom_ip = None;
|
||||||
|
service.replicas.clear();
|
||||||
|
controller.put_service(service).await.unwrap();
|
||||||
|
assert!(controller.topology().services[0].custom_ip.is_none());
|
||||||
|
controller.shutdown().await;
|
||||||
|
let mut stopped = controller.topology().services[0].clone();
|
||||||
|
stopped.enabled = false;
|
||||||
|
assert!(controller.put_service(stopped).await.is_err());
|
||||||
|
assert!(
|
||||||
|
controller
|
||||||
|
.put_team(Team {
|
||||||
|
id: "after".into(),
|
||||||
|
name: "after".into(),
|
||||||
|
address: "::1".parse().unwrap(),
|
||||||
|
hidden: false
|
||||||
|
})
|
||||||
|
.await
|
||||||
|
.is_err()
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[tokio::test]
|
||||||
|
async fn failed_service_persistence_closes_new_listener_and_restores_existing_route() {
|
||||||
|
let temp = tempfile::tempdir().unwrap();
|
||||||
|
let parent = temp.path().join("state");
|
||||||
|
std::fs::create_dir(&parent).unwrap();
|
||||||
|
let mut cfg = Config::default();
|
||||||
|
cfg.network.drain_timeout_ms = 100;
|
||||||
|
cfg.web.state_file = parent.join("state.json");
|
||||||
|
let upstream = TcpListener::bind("127.0.0.1:0").await.unwrap();
|
||||||
|
let reserve = TcpListener::bind("127.0.0.1:0").await.unwrap();
|
||||||
|
let listen = reserve.local_addr().unwrap();
|
||||||
|
drop(reserve);
|
||||||
|
let controller = Controller::start(cfg).await.unwrap();
|
||||||
|
let service = direct_service(listen, upstream.local_addr().unwrap().port());
|
||||||
|
controller.put_service(service.clone()).await.unwrap();
|
||||||
|
let before = controller.topology();
|
||||||
|
std::fs::remove_file(parent.join("state.json")).unwrap();
|
||||||
|
std::fs::remove_dir(&parent).unwrap();
|
||||||
|
std::fs::write(&parent, "blocked").unwrap();
|
||||||
|
let reserve = TcpListener::bind("127.0.0.1:0").await.unwrap();
|
||||||
|
let changed_listen = reserve.local_addr().unwrap();
|
||||||
|
drop(reserve);
|
||||||
|
let mut changed = service.clone();
|
||||||
|
changed.listen = changed_listen;
|
||||||
|
assert!(controller.put_service(changed).await.is_err());
|
||||||
|
assert_eq!(controller.topology().services, before.services);
|
||||||
|
assert_eq!(controller.topology().teams, before.teams);
|
||||||
|
assert!(
|
||||||
|
tokio::net::TcpStream::connect(changed_listen)
|
||||||
|
.await
|
||||||
|
.is_err()
|
||||||
|
);
|
||||||
|
let _client = tokio::net::TcpStream::connect(listen).await.unwrap();
|
||||||
|
tokio::time::timeout(std::time::Duration::from_secs(1), upstream.accept())
|
||||||
|
.await
|
||||||
|
.unwrap()
|
||||||
|
.unwrap();
|
||||||
|
let reserve = TcpListener::bind("127.0.0.1:0").await.unwrap();
|
||||||
|
let new_listen = reserve.local_addr().unwrap();
|
||||||
|
drop(reserve);
|
||||||
|
let mut new = service;
|
||||||
|
new.id = "new_service".into();
|
||||||
|
new.listen = new_listen;
|
||||||
|
new.custom_ip = Some("::1".parse().unwrap());
|
||||||
|
assert!(controller.put_service(new).await.is_err());
|
||||||
|
assert!(tokio::net::TcpStream::connect(new_listen).await.is_err());
|
||||||
|
assert_eq!(controller.topology().teams, before.teams);
|
||||||
|
controller.shutdown().await;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn wildcard_loop_aliases_mapped_addresses_and_address_families() {
|
||||||
|
let team = |ip: &str| Team {
|
||||||
|
id: "a".into(),
|
||||||
|
name: "A".into(),
|
||||||
|
address: ip.parse().unwrap(),
|
||||||
|
hidden: false,
|
||||||
|
};
|
||||||
|
let mut service = direct_service("0.0.0.0:38080".parse().unwrap(), 38080);
|
||||||
|
service.custom_ip = None;
|
||||||
|
service.primary = Some("a".into());
|
||||||
|
for ip in ["127.0.0.1", "127.0.0.2", "::ffff:127.0.0.2"] {
|
||||||
|
assert!(
|
||||||
|
mirror_tool::config::validate_topology(&[team(ip)], &[service.clone()]).is_err(),
|
||||||
|
"accepted self-loop {ip}"
|
||||||
|
);
|
||||||
|
}
|
||||||
|
// An IPv4-only listener cannot receive native IPv6 loopback traffic.
|
||||||
|
assert!(mirror_tool::config::validate_topology(&[team("::1")], &[service.clone()]).is_ok());
|
||||||
|
let mut other = service.clone();
|
||||||
|
other.id = "other".into();
|
||||||
|
other.listen = "0.0.0.0:38081".parse().unwrap();
|
||||||
|
other.upstream_port = 38082;
|
||||||
|
service.upstream_port = 38081;
|
||||||
|
assert!(
|
||||||
|
mirror_tool::config::validate_topology(&[team("127.0.0.2")], &[service, other]).is_err()
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[cfg(target_os = "linux")]
|
||||||
|
#[tokio::test]
|
||||||
|
async fn linux_unenumerated_loopback_alias_reaches_ipv4_wildcard() {
|
||||||
|
let wildcard = TcpListener::bind("0.0.0.0:0").await.unwrap();
|
||||||
|
let port = wildcard.local_addr().unwrap().port();
|
||||||
|
let _client = tokio::net::TcpStream::connect((std::net::Ipv4Addr::new(127, 0, 0, 2), port))
|
||||||
|
.await
|
||||||
|
.unwrap();
|
||||||
|
tokio::time::timeout(std::time::Duration::from_secs(1), wildcard.accept())
|
||||||
|
.await
|
||||||
|
.unwrap()
|
||||||
|
.unwrap();
|
||||||
|
}
|
||||||
|
|
||||||
|
#[tokio::test]
|
||||||
|
async fn custom_ip_ids_avoid_collisions_and_reuse_operator_teams() {
|
||||||
|
let temp = tempfile::tempdir().unwrap();
|
||||||
|
let mut cfg = Config::default();
|
||||||
|
cfg.web.state_file = temp.path().join("state.json");
|
||||||
|
cfg.teams = vec![Team {
|
||||||
|
id: "direct_127_0_0_1".into(),
|
||||||
|
name: "Operator team".into(),
|
||||||
|
address: "::1".parse().unwrap(),
|
||||||
|
hidden: false,
|
||||||
|
}];
|
||||||
|
let controller = Controller::start(cfg.clone()).await.unwrap();
|
||||||
|
let mut service = direct_service("127.0.0.1:38090".parse().unwrap(), 38091);
|
||||||
|
service.enabled = false;
|
||||||
|
controller.put_service(service.clone()).await.unwrap();
|
||||||
|
let generated = controller.topology().services[0].primary.clone().unwrap();
|
||||||
|
assert_eq!(generated, "direct_127_0_0_1_1");
|
||||||
|
service.custom_ip = Some("::1".parse().unwrap());
|
||||||
|
controller.put_service(service.clone()).await.unwrap();
|
||||||
|
assert_eq!(
|
||||||
|
controller.topology().services[0].primary.as_deref(),
|
||||||
|
Some("direct_127_0_0_1")
|
||||||
|
);
|
||||||
|
assert!(!controller.topology().teams[0].hidden);
|
||||||
|
service.custom_ip = Some("127.0.0.1".parse().unwrap());
|
||||||
|
controller.put_service(service).await.unwrap();
|
||||||
|
assert_eq!(
|
||||||
|
controller.topology().services[0].primary.as_deref(),
|
||||||
|
Some(generated.as_str())
|
||||||
|
);
|
||||||
|
controller.shutdown().await;
|
||||||
|
let restored = Controller::start(cfg).await.unwrap();
|
||||||
|
assert_eq!(restored.topology().teams.len(), 2);
|
||||||
|
assert_eq!(
|
||||||
|
restored.topology().services[0].primary.as_deref(),
|
||||||
|
Some(generated.as_str())
|
||||||
|
);
|
||||||
|
assert!(restored.topology().teams[1].hidden);
|
||||||
|
restored.shutdown().await;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[tokio::test]
|
||||||
|
async fn shutdown_serializes_with_in_flight_service_transaction() {
|
||||||
|
let temp = tempfile::tempdir().unwrap();
|
||||||
|
let mut cfg = Config::default();
|
||||||
|
cfg.web.state_file = temp.path().join("state.json");
|
||||||
|
cfg.network.drain_timeout_ms = 100;
|
||||||
|
let reserve = TcpListener::bind("127.0.0.1:0").await.unwrap();
|
||||||
|
let listen = reserve.local_addr().unwrap();
|
||||||
|
drop(reserve);
|
||||||
|
let controller = Controller::start(cfg.clone()).await.unwrap();
|
||||||
|
let service = direct_service(listen, 39001);
|
||||||
|
let (saved, ()) = tokio::join!(biased; controller.put_service(service), controller.shutdown());
|
||||||
|
saved.unwrap();
|
||||||
|
assert_eq!(
|
||||||
|
mirror_tool::store::Topology::load(&cfg).unwrap().services,
|
||||||
|
controller.topology().services
|
||||||
|
);
|
||||||
|
assert!(tokio::net::TcpStream::connect(listen).await.is_err());
|
||||||
|
assert!(!controller.engine.snapshots()["custom_notes"].running);
|
||||||
|
}
|
||||||
|
|||||||
@ -120,3 +120,63 @@ async fn actual_polling_handshake_ping_delta_and_reconnect() {
|
|||||||
server.abort();
|
server.abort();
|
||||||
let _ = server.await;
|
let _ = server.await;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
#[tokio::test]
|
||||||
|
async fn continuous_non_ping_packets_cannot_extend_negotiated_heartbeat() {
|
||||||
|
let handshakes = Arc::new(AtomicUsize::new(0));
|
||||||
|
let polls = Arc::new(AtomicUsize::new(0));
|
||||||
|
let state = (handshakes.clone(), polls.clone());
|
||||||
|
let router = Router::new()
|
||||||
|
.route(
|
||||||
|
"/socket.io/",
|
||||||
|
get(
|
||||||
|
|State((handshakes, polls)): State<(Arc<AtomicUsize>, Arc<AtomicUsize>)>,
|
||||||
|
Query(query): Query<HashMap<String, String>>| async move {
|
||||||
|
if !query.contains_key("sid") {
|
||||||
|
handshakes.fetch_add(1, SeqCst);
|
||||||
|
return format!(
|
||||||
|
"0{}",
|
||||||
|
json!({"sid":"no_ping", "pingInterval":500, "pingTimeout":500})
|
||||||
|
);
|
||||||
|
}
|
||||||
|
polls.fetch_add(1, SeqCst);
|
||||||
|
tokio::time::sleep(Duration::from_millis(10)).await;
|
||||||
|
"40/game_events,{}".into()
|
||||||
|
},
|
||||||
|
)
|
||||||
|
.post(|| async { "ok" }),
|
||||||
|
)
|
||||||
|
.with_state(state);
|
||||||
|
let listener = tokio::net::TcpListener::bind("127.0.0.1:0").await.unwrap();
|
||||||
|
let addr = listener.local_addr().unwrap();
|
||||||
|
let server = tokio::spawn(async move {
|
||||||
|
axum::serve(listener, router).await.unwrap();
|
||||||
|
});
|
||||||
|
let cfg = ScoreboardConfig {
|
||||||
|
kind: ScoreboardKind::Forcad,
|
||||||
|
url: format!("http://{addr}"),
|
||||||
|
poll_interval_ms: 100,
|
||||||
|
stale_after_ms: 2000,
|
||||||
|
};
|
||||||
|
let board = Scoreboard::new(Some(&cfg));
|
||||||
|
let cancel = CancellationToken::new();
|
||||||
|
let worker = tokio::spawn(board.run(cfg, cancel.clone()));
|
||||||
|
let reconnected = tokio::time::timeout(Duration::from_secs(4), async {
|
||||||
|
while handshakes.load(SeqCst) < 2 {
|
||||||
|
tokio::time::sleep(Duration::from_millis(10)).await;
|
||||||
|
}
|
||||||
|
})
|
||||||
|
.await;
|
||||||
|
cancel.cancel();
|
||||||
|
worker.await.unwrap();
|
||||||
|
server.abort();
|
||||||
|
let _ = server.await;
|
||||||
|
assert!(
|
||||||
|
reconnected.is_ok(),
|
||||||
|
"non-ping frames kept old session alive"
|
||||||
|
);
|
||||||
|
assert!(
|
||||||
|
polls.load(SeqCst) > 10,
|
||||||
|
"fixture did not continuously return frames"
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|||||||
@ -32,6 +32,8 @@ async fn free_address() -> SocketAddr {
|
|||||||
}
|
}
|
||||||
fn service(listen: SocketAddr, upstream: SocketAddr) -> ServiceConfig {
|
fn service(listen: SocketAddr, upstream: SocketAddr) -> ServiceConfig {
|
||||||
ServiceConfig {
|
ServiceConfig {
|
||||||
|
custom: false,
|
||||||
|
custom_ip: None,
|
||||||
id: "service".into(),
|
id: "service".into(),
|
||||||
name: "Test".into(),
|
name: "Test".into(),
|
||||||
listen,
|
listen,
|
||||||
@ -45,6 +47,7 @@ fn service(listen: SocketAddr, upstream: SocketAddr) -> ServiceConfig {
|
|||||||
}
|
}
|
||||||
fn teams() -> Vec<Team> {
|
fn teams() -> Vec<Team> {
|
||||||
vec![Team {
|
vec![Team {
|
||||||
|
hidden: false,
|
||||||
id: "primary".into(),
|
id: "primary".into(),
|
||||||
name: "Primary".into(),
|
name: "Primary".into(),
|
||||||
address: IpAddr::V4(Ipv4Addr::LOCALHOST),
|
address: IpAddr::V4(Ipv4Addr::LOCALHOST),
|
||||||
@ -295,6 +298,7 @@ async fn replica_fixture(
|
|||||||
config.replicas = vec!["replica".into()];
|
config.replicas = vec!["replica".into()];
|
||||||
let mut teams = teams();
|
let mut teams = teams();
|
||||||
teams.push(Team {
|
teams.push(Team {
|
||||||
|
hidden: false,
|
||||||
id: "replica".into(),
|
id: "replica".into(),
|
||||||
name: "Replica".into(),
|
name: "Replica".into(),
|
||||||
address: "::1".parse().unwrap(),
|
address: "::1".parse().unwrap(),
|
||||||
@ -421,3 +425,45 @@ async fn strict_slow_replica_backpressures_and_shutdown_joins_workers() {
|
|||||||
server.abort();
|
server.abort();
|
||||||
slow.abort();
|
slow.abort();
|
||||||
}
|
}
|
||||||
|
|
||||||
|
#[tokio::test]
|
||||||
|
async fn quiesce_keeps_existing_stream_but_route_rollback_cannot_reopen_listener() {
|
||||||
|
let upstream = TcpListener::bind("127.0.0.1:0").await.unwrap();
|
||||||
|
let address = upstream.local_addr().unwrap();
|
||||||
|
let task = tokio::spawn(async move {
|
||||||
|
let (mut stream, _) = upstream.accept().await.unwrap();
|
||||||
|
let mut bytes = vec![];
|
||||||
|
stream.read_to_end(&mut bytes).await.unwrap();
|
||||||
|
stream.write_all(&bytes).await.unwrap();
|
||||||
|
});
|
||||||
|
let engine = Engine::new(settings());
|
||||||
|
let listen = free_address().await;
|
||||||
|
let original = service(listen, address);
|
||||||
|
engine.upsert(&original, &teams()).await.unwrap();
|
||||||
|
let mut client = TcpStream::connect(listen).await.unwrap();
|
||||||
|
timeout(Duration::from_secs(1), async {
|
||||||
|
while engine.snapshots()["service"].stats.active == 0 {
|
||||||
|
tokio::task::yield_now().await;
|
||||||
|
}
|
||||||
|
})
|
||||||
|
.await
|
||||||
|
.unwrap();
|
||||||
|
engine.quiesce().await;
|
||||||
|
assert!(TcpStream::connect(listen).await.is_err());
|
||||||
|
engine.upsert(&original, &teams()).await.unwrap();
|
||||||
|
assert!(TcpStream::connect(listen).await.is_err());
|
||||||
|
assert!(!engine.snapshots()["service"].running);
|
||||||
|
client
|
||||||
|
.write_all(b"existing connection survives quiesce")
|
||||||
|
.await
|
||||||
|
.unwrap();
|
||||||
|
client.shutdown().await.unwrap();
|
||||||
|
let mut bytes = vec![];
|
||||||
|
timeout(Duration::from_secs(1), client.read_to_end(&mut bytes))
|
||||||
|
.await
|
||||||
|
.unwrap()
|
||||||
|
.unwrap();
|
||||||
|
assert_eq!(bytes, b"existing connection survives quiesce");
|
||||||
|
task.await.unwrap();
|
||||||
|
engine.shutdown().await;
|
||||||
|
}
|
||||||
|
|||||||
126
tests/shutdown.rs
Normal file
126
tests/shutdown.rs
Normal file
@ -0,0 +1,126 @@
|
|||||||
|
#![cfg(unix)]
|
||||||
|
use mirror_tool::config::{Config, ReplicaPolicy, ServiceConfig, Team};
|
||||||
|
use std::{
|
||||||
|
process::{Command, Stdio},
|
||||||
|
time::Duration,
|
||||||
|
};
|
||||||
|
use tokio::{
|
||||||
|
io::{AsyncReadExt, AsyncWriteExt},
|
||||||
|
net::{TcpListener, TcpStream},
|
||||||
|
time::{sleep, timeout},
|
||||||
|
};
|
||||||
|
|
||||||
|
#[tokio::test]
|
||||||
|
async fn partial_http_body_cannot_keep_process_or_proxy_listener_alive() {
|
||||||
|
let temp = tempfile::tempdir().unwrap();
|
||||||
|
let web = TcpListener::bind("127.0.0.1:0").await.unwrap();
|
||||||
|
let web_addr = web.local_addr().unwrap();
|
||||||
|
drop(web);
|
||||||
|
let data = TcpListener::bind("127.0.0.1:0").await.unwrap();
|
||||||
|
let data_addr = data.local_addr().unwrap();
|
||||||
|
drop(data);
|
||||||
|
let upstream = TcpListener::bind("127.0.0.1:0").await.unwrap();
|
||||||
|
let mut cfg = Config::default();
|
||||||
|
cfg.web.listen = web_addr;
|
||||||
|
cfg.web.state_file = temp.path().join("state.json");
|
||||||
|
cfg.network.drain_timeout_ms = 100;
|
||||||
|
cfg.teams = vec![Team {
|
||||||
|
id: "a".into(),
|
||||||
|
name: "A".into(),
|
||||||
|
address: "127.0.0.1".parse().unwrap(),
|
||||||
|
hidden: false,
|
||||||
|
}];
|
||||||
|
cfg.services = vec![ServiceConfig {
|
||||||
|
id: "notes".into(),
|
||||||
|
name: "Notes".into(),
|
||||||
|
listen: data_addr,
|
||||||
|
upstream_port: upstream.local_addr().unwrap().port(),
|
||||||
|
primary: Some("a".into()),
|
||||||
|
replicas: vec![],
|
||||||
|
enabled: true,
|
||||||
|
auto_switch: false,
|
||||||
|
replica_policy: ReplicaPolicy::Isolate,
|
||||||
|
custom: false,
|
||||||
|
custom_ip: None,
|
||||||
|
}];
|
||||||
|
let path = temp.path().join("mirror.toml");
|
||||||
|
std::fs::write(&path, toml::to_string(&cfg).unwrap()).unwrap();
|
||||||
|
let mut child = Command::new(env!("CARGO_BIN_EXE_mirror-tool"))
|
||||||
|
.arg("--config")
|
||||||
|
.arg(path)
|
||||||
|
.env("MIRROR_PASSWORD", "loopback-fixture-passphrase")
|
||||||
|
.stdout(Stdio::null())
|
||||||
|
.stderr(Stdio::null())
|
||||||
|
.spawn()
|
||||||
|
.unwrap();
|
||||||
|
let ready = timeout(Duration::from_secs(5), async {
|
||||||
|
loop {
|
||||||
|
if let Ok(s) = TcpStream::connect(web_addr).await {
|
||||||
|
break s;
|
||||||
|
}
|
||||||
|
sleep(Duration::from_millis(10)).await;
|
||||||
|
}
|
||||||
|
})
|
||||||
|
.await;
|
||||||
|
if ready.is_err() {
|
||||||
|
let _ = child.kill();
|
||||||
|
panic!("server did not start");
|
||||||
|
}
|
||||||
|
let mut ready_http = ready.unwrap();
|
||||||
|
ready_http
|
||||||
|
.write_all(b"GET /api/state HTTP/1.1\r\nHost: localhost\r\nConnection: close\r\n\r\n")
|
||||||
|
.await
|
||||||
|
.unwrap();
|
||||||
|
let mut reply = [0; 1024];
|
||||||
|
assert!(
|
||||||
|
timeout(Duration::from_secs(1), ready_http.read(&mut reply))
|
||||||
|
.await
|
||||||
|
.unwrap()
|
||||||
|
.unwrap()
|
||||||
|
> 0
|
||||||
|
);
|
||||||
|
let mut http = TcpStream::connect(web_addr).await.unwrap();
|
||||||
|
http.write_all(b"POST /api/login HTTP/1.1\r\nHost: localhost\r\nContent-Type: application/json\r\nContent-Length: 1000\r\n\r\n{").await.unwrap();
|
||||||
|
let _proxy = TcpStream::connect(data_addr).await.unwrap();
|
||||||
|
let _upstream_peer = timeout(Duration::from_secs(1), upstream.accept())
|
||||||
|
.await
|
||||||
|
.unwrap()
|
||||||
|
.unwrap();
|
||||||
|
assert!(
|
||||||
|
Command::new("kill")
|
||||||
|
.args(["-TERM", &child.id().to_string()])
|
||||||
|
.status()
|
||||||
|
.unwrap()
|
||||||
|
.success()
|
||||||
|
);
|
||||||
|
let closed = timeout(Duration::from_secs(1), async {
|
||||||
|
while TcpStream::connect(data_addr).await.is_ok() {
|
||||||
|
sleep(Duration::from_millis(5)).await;
|
||||||
|
}
|
||||||
|
})
|
||||||
|
.await;
|
||||||
|
if closed.is_err() {
|
||||||
|
let _ = child.kill();
|
||||||
|
panic!("proxy kept accepting during HTTP shutdown");
|
||||||
|
}
|
||||||
|
let result = timeout(Duration::from_secs(3), async {
|
||||||
|
loop {
|
||||||
|
if let Some(status) = child.try_wait().unwrap() {
|
||||||
|
break status;
|
||||||
|
}
|
||||||
|
sleep(Duration::from_millis(10)).await;
|
||||||
|
}
|
||||||
|
})
|
||||||
|
.await;
|
||||||
|
if result.is_err() {
|
||||||
|
let _ = child.kill();
|
||||||
|
panic!("partial HTTP body prevented bounded process exit");
|
||||||
|
}
|
||||||
|
assert!(
|
||||||
|
result.as_ref().unwrap().success(),
|
||||||
|
"unexpected process status: {:?}",
|
||||||
|
result
|
||||||
|
);
|
||||||
|
assert!(TcpStream::connect(data_addr).await.is_err());
|
||||||
|
drop(http);
|
||||||
|
}
|
||||||
918
web/app.css
918
web/app.css
File diff suppressed because one or more lines are too long
514
web/app.js
514
web/app.js
@ -1,339 +1,251 @@
|
|||||||
"use strict";
|
"use strict";
|
||||||
|
|
||||||
(() => {
|
(() => {
|
||||||
const $ = (id) => document.getElementById(id);
|
const $ = (id) => document.getElementById(id);
|
||||||
const state = { csrf: "", snapshot: null, stream: null, busy: false, cards: new Map(), baselines: new Map(), rates: new Map(), stale: true, sessionTimer: null, staleTimer: null, sessionCheck: false, retryDelay: 2000, editId: null, deleteId: null, generation: 0 };
|
const state = { snapshot: null, csrf: null, source: null, busy: false, stale: true, cards: new Map(), editId: null, watchdog: null };
|
||||||
const configKeys = ["id", "name", "listen", "upstream_port", "primary", "replicas", "enabled", "auto_switch", "replica_policy"];
|
const statusClass = { 101: "up", 102: "corrupt", 103: "mumble", 104: "down", 110: "fail" };
|
||||||
const statusNames = { 101: "UP", 102: "CORRUPT", 103: "MUMBLE", 104: "DOWN", 110: "CHECK FAIL" };
|
const statusOrder = { 101: 0, 102: 1, 103: 2, 104: 3, 110: 4 };
|
||||||
function el(tag, className, text) {
|
const statusName = { 101: "UP", 102: "CORRUPT", 103: "MUMBLE", 104: "DOWN", 110: "CHECK_FAILED" };
|
||||||
const node = document.createElement(tag);
|
const el = (tag, cls = "", text = "") => { const node = document.createElement(tag); node.className = cls; node.textContent = text; return node; };
|
||||||
if (className) node.className = className;
|
const serviceById = (id) => state.snapshot?.services.find((service) => service.id === id);
|
||||||
if (text !== undefined) node.textContent = text;
|
const teamById = (id) => state.snapshot?.teams.find((team) => team.id === id);
|
||||||
return node;
|
const portOf = (service) => Number(service.listen.slice(service.listen.lastIndexOf(":") + 1));
|
||||||
|
function button(cls, title, icon, action) {
|
||||||
|
const node = el("button", cls); node.type = "button"; node.title = title; node.setAttribute("aria-label", title);
|
||||||
|
if (icon) { const i = el("i", `fas fa-${icon}`); i.setAttribute("aria-hidden", "true"); node.append(i); }
|
||||||
|
node.addEventListener("click", (event) => { event.stopPropagation(); action(event); }); return node;
|
||||||
}
|
}
|
||||||
function message(id, text) { $(id).textContent = text || ""; $(id).hidden = !text; }
|
function alert(message) { $("alert-message").textContent = message; $("alert").hidden = !message; }
|
||||||
function config(service) { return Object.fromEntries(configKeys.map((key) => [key, key === "replicas" ? [...service[key]] : service[key]])); }
|
function closeStream() { state.source?.close(); state.source = null; clearTimeout(state.watchdog); }
|
||||||
function serviceById(id) { return state.snapshot?.services.find((service) => service.id === id); }
|
function watchUpdates() {
|
||||||
function field(form, name) { return form.elements.namedItem(name); }
|
clearTimeout(state.watchdog);
|
||||||
function time(ms) { return ms ? new Date(ms).toLocaleTimeString("ru-RU") : "—"; }
|
state.watchdog = setTimeout(() => { if (state.csrf) { state.stale = true; render(); alert("Нет свежих обновлений от сервера. Обновите подключение."); } }, 5000);
|
||||||
function bytes(value) {
|
|
||||||
const count = Math.max(0, Number(value) || 0);
|
|
||||||
const units = ["B", "KiB", "MiB", "GiB", "TiB"];
|
|
||||||
const unit = count ? Math.min(Math.floor(Math.log(count) / Math.log(1024)), units.length - 1) : 0;
|
|
||||||
return `${(count / 1024 ** unit).toLocaleString("ru-RU", { maximumFractionDigits: unit ? 1 : 0 })} ${units[unit]}`;
|
|
||||||
}
|
}
|
||||||
function setBusy(busy) {
|
function showLogin() {
|
||||||
state.busy = busy;
|
closeStream(); state.csrf = null; state.snapshot = null; state.stale = true;
|
||||||
document.querySelectorAll("[data-write]").forEach((node) => { node.disabled = busy; });
|
for (const dialog of document.querySelectorAll("dialog[open]")) dialog.close();
|
||||||
document.querySelectorAll("dialog form input, dialog form select, dialog [data-close]").forEach((node) => { node.disabled = busy || (node.name === "id" && Boolean(state.editId) && node.closest("#service-form") !== null); });
|
$("header").hidden = true; $("dashboard").hidden = true; $("login").hidden = false; document.title = "Вход";
|
||||||
if (state.snapshot) render(state.snapshot, false, !busy);
|
|
||||||
}
|
}
|
||||||
async function api(path, options = {}) {
|
async function api(path, options = {}) {
|
||||||
const headers = { ...options.headers };
|
const headers = { ...options.headers };
|
||||||
if (options.body !== undefined) headers["Content-Type"] = "application/json";
|
if (options.body !== undefined) headers["Content-Type"] = "application/json";
|
||||||
if (options.method && options.method !== "GET" && state.csrf) headers["X-CSRF-Token"] = state.csrf;
|
if (state.csrf && options.method && options.method !== "GET") headers["X-CSRF-Token"] = state.csrf;
|
||||||
let response;
|
let response;
|
||||||
try { response = await fetch(path, { ...options, headers, credentials: "same-origin", cache: "no-store" }); }
|
try { response = await fetch(path, { ...options, headers, credentials: "same-origin" }); }
|
||||||
catch (error) { throw new Error(`Ошибка сети: ${error.message}`); }
|
catch { throw new Error("Нет связи с сервером. Проверьте подключение и обновите страницу."); }
|
||||||
const body = await response.json().catch(() => ({}));
|
const data = await response.json().catch(() => ({}));
|
||||||
if (!response.ok) {
|
if (!response.ok) {
|
||||||
const error = new Error(body.error || `Ошибка HTTP ${response.status}`);
|
if (response.status === 401 && path !== "/api/login") showLogin();
|
||||||
error.status = response.status;
|
const error = new Error(data.error || `Ошибка сервера (${response.status})`); error.status = response.status; throw error;
|
||||||
if (response.status === 401 && path !== "/api/login" && path !== "/api/session") showLogin("Сессия завершена. Войдите снова.");
|
|
||||||
throw error;
|
|
||||||
}
|
}
|
||||||
return body;
|
return data;
|
||||||
}
|
}
|
||||||
function closeStream() {
|
function config(service) {
|
||||||
if (state.stream) state.stream.close();
|
return Object.fromEntries(["id", "name", "listen", "upstream_port", "primary", "replicas", "enabled", "auto_switch", "replica_policy", "custom", "custom_ip"].map((key) => [key, service[key] ?? ({ custom: false, custom_ip: null }[key])]));
|
||||||
state.stream = null;
|
|
||||||
clearTimeout(state.sessionTimer); clearTimeout(state.staleTimer);
|
|
||||||
state.sessionTimer = null;
|
|
||||||
}
|
}
|
||||||
function showLogin(reason = "Введите пароль для входа.") {
|
async function refresh() { render(await api("/api/snapshot")); watchUpdates(); }
|
||||||
state.generation += 1;
|
async function mutate(path, method, body, errorId) {
|
||||||
closeStream(); state.csrf = ""; state.stale = true;
|
if (state.busy || state.stale) return false;
|
||||||
document.querySelectorAll("dialog[open]").forEach((dialog) => dialog.close());
|
state.busy = true; render(); if (errorId) $(errorId).textContent = "";
|
||||||
$("dashboard").hidden = true; $("login-view").hidden = false;
|
|
||||||
$("login-status").textContent = reason;
|
|
||||||
$("password").value = "";
|
|
||||||
}
|
|
||||||
function markStale(detail) {
|
|
||||||
state.stale = true;
|
|
||||||
$("stream-state").textContent = detail || "Нет свежих данных";
|
|
||||||
$("stream-state").className = "warn";
|
|
||||||
if (state.snapshot) render(state.snapshot, false);
|
|
||||||
}
|
|
||||||
function scheduleSessionCheck() {
|
|
||||||
if (state.sessionTimer || state.sessionCheck || !state.csrf) return;
|
|
||||||
state.sessionTimer = setTimeout(async () => {
|
|
||||||
state.sessionTimer = null; state.sessionCheck = true;
|
|
||||||
const generation = state.generation;
|
|
||||||
try {
|
|
||||||
const session = await api("/api/session");
|
|
||||||
if (generation !== state.generation) return;
|
|
||||||
state.csrf = session.csrf;
|
|
||||||
state.retryDelay = Math.min(state.retryDelay * 2, 30000);
|
|
||||||
} catch (error) {
|
|
||||||
if (generation !== state.generation) return;
|
|
||||||
if (error.status === 401) showLogin("Сессия завершена. Войдите снова.");
|
|
||||||
else { markStale("Нет связи с сервером"); message("global-error", error.message); state.retryDelay = Math.min(state.retryDelay * 2, 30000); }
|
|
||||||
} finally {
|
|
||||||
state.sessionCheck = false;
|
|
||||||
if (state.stale && state.csrf) scheduleSessionCheck();
|
|
||||||
}
|
|
||||||
}, state.retryDelay);
|
|
||||||
}
|
|
||||||
function watchFreshness() {
|
|
||||||
clearTimeout(state.staleTimer);
|
|
||||||
state.staleTimer = setTimeout(() => { markStale("Снимок устарел · > 8 с"); scheduleSessionCheck(); }, 8000);
|
|
||||||
}
|
|
||||||
function startStream() {
|
|
||||||
closeStream();
|
|
||||||
const stream = new EventSource("/api/events");
|
|
||||||
state.stream = stream;
|
|
||||||
stream.addEventListener("snapshot", (event) => {
|
|
||||||
if (stream !== state.stream) return;
|
|
||||||
try {
|
|
||||||
const snapshot = JSON.parse(event.data);
|
|
||||||
state.stale = false; state.retryDelay = 2000;
|
|
||||||
clearTimeout(state.sessionTimer); state.sessionTimer = null;
|
|
||||||
$("stream-state").textContent = "В реальном времени"; $("stream-state").className = "good";
|
|
||||||
render(snapshot); watchFreshness();
|
|
||||||
} catch { markStale("Некорректный снимок"); scheduleSessionCheck(); }
|
|
||||||
});
|
|
||||||
stream.onerror = () => { if (stream === state.stream) { markStale("Связь прервана · переподключение"); scheduleSessionCheck(); } };
|
|
||||||
watchFreshness();
|
|
||||||
}
|
|
||||||
async function refresh() {
|
|
||||||
const generation = state.generation;
|
|
||||||
const snapshot = await api("/api/snapshot");
|
|
||||||
if (generation === state.generation && state.csrf) render(snapshot);
|
|
||||||
}
|
|
||||||
async function enter(session) {
|
|
||||||
state.generation += 1; state.csrf = session.csrf; state.stale = true;
|
|
||||||
state.snapshot = null; state.baselines.clear(); state.rates.clear();
|
|
||||||
state.cards.forEach((card) => card.root.remove()); state.cards.clear();
|
|
||||||
message("login-error", ""); message("global-error", ""); $("password").value = "";
|
|
||||||
$("login-view").hidden = true; $("dashboard").hidden = false;
|
|
||||||
markStale("Подключение…");
|
|
||||||
try { await refresh(); if (state.csrf) startStream(); }
|
|
||||||
catch (error) { message("global-error", error.message); if (state.csrf) startStream(); }
|
|
||||||
}
|
|
||||||
async function mutate(path, method, body, errorId = "global-error") {
|
|
||||||
if (state.busy) return false;
|
|
||||||
setBusy(true); message(errorId, ""); $("action-status").textContent = "Сохраняем…";
|
|
||||||
try {
|
try {
|
||||||
await api(path, { method, ...(body === undefined ? {} : { body: JSON.stringify(body) }) });
|
await api(path, { method, ...(body === undefined ? {} : { body: JSON.stringify(body) }) });
|
||||||
await refresh();
|
await refresh(); alert(""); return true;
|
||||||
$("action-status").textContent = "Сохранено";
|
} catch (error) {
|
||||||
return true;
|
if (errorId) $(errorId).textContent = error.message; else alert(error.message);
|
||||||
} catch (error) { message(errorId, error.message); $("action-status").textContent = "Изменение не подтверждено. Проверьте состояние и повторите."; return false; }
|
try { if (state.csrf) await refresh(); } catch { /* Preserve the actual mutation error. */ }
|
||||||
finally { setBusy(false); }
|
return false;
|
||||||
|
} finally { state.busy = false; render(); }
|
||||||
}
|
}
|
||||||
async function updateService(id, change) {
|
function updateService(id, patch, errorId) {
|
||||||
const service = serviceById(id);
|
const current = serviceById(id); if (!current) return Promise.resolve(false);
|
||||||
if (!service || state.busy) return;
|
return mutate(`/api/services/${encodeURIComponent(id)}`, "PUT", { ...config(current), ...patch }, errorId);
|
||||||
const next = { ...config(service), ...change };
|
|
||||||
await mutate(`/api/services/${encodeURIComponent(id)}`, "PUT", next);
|
|
||||||
}
|
}
|
||||||
function control(type, label, onChange) {
|
function startStream() {
|
||||||
const input = el("input"); input.type = type; input.setAttribute("aria-label", label); input.dataset.write = "";
|
closeStream(); const source = new EventSource("/api/events"); state.source = source;
|
||||||
input.addEventListener("change", onChange); return input;
|
source.addEventListener("snapshot", (event) => {
|
||||||
|
if (state.source !== source) return;
|
||||||
|
try { const wasStale = state.stale; render(JSON.parse(event.data)); watchUpdates(); if (wasStale) alert(""); }
|
||||||
|
catch { state.stale = true; render(); alert("Получены некорректные данные. Обновите страницу."); }
|
||||||
|
});
|
||||||
|
source.onerror = async () => {
|
||||||
|
if (state.source !== source) return;
|
||||||
|
state.stale = true; render(); alert("Нет обновлений от сервера. Переподключение…");
|
||||||
|
try { await api("/api/session"); } catch (error) { if (error.status !== 401) return; }
|
||||||
|
};
|
||||||
|
watchUpdates();
|
||||||
}
|
}
|
||||||
function button(text, className, onClick) {
|
async function enter(session) {
|
||||||
const node = el("button", className, text); node.type = "button"; node.dataset.write = "";
|
state.csrf = session.csrf; $("password").value = ""; $("login-error").textContent = "";
|
||||||
node.addEventListener("click", onClick); return node;
|
await refresh(); $("login").hidden = true; $("header").hidden = false; $("dashboard").hidden = false;
|
||||||
|
document.title = "Proxy Manager"; startStream();
|
||||||
}
|
}
|
||||||
function createCard(id) {
|
function openService(id) {
|
||||||
const root = el("article", "service-card");
|
const service = serviceById(id); if (!service) return;
|
||||||
const header = el("header", "service-header"); const heading = el("div", "service-heading");
|
const hostname = window.location.hostname.replace(/^\[|\]$/g, "");
|
||||||
const title = el("h2", "service-name"); const running = el("span", "service-state"); heading.append(title, running);
|
const host = hostname.includes(":") ? `[${hostname}]` : hostname;
|
||||||
const endpoint = el("p", "endpoint"); const route = el("p", "route-summary");
|
window.open(`http://${host}:${portOf(service)}`, "_blank", "noopener,noreferrer");
|
||||||
const actions = el("div", "card-actions");
|
}
|
||||||
const toggle = button("Запустить", "small accent", () => { const service = serviceById(id); if (service) updateService(id, { enabled: !service.enabled }); });
|
function selectPrimary(id, teamId) {
|
||||||
const autoLabel = el("label", "auto-label");
|
const service = serviceById(id); if (!service || service.primary === teamId) return;
|
||||||
const auto = control("checkbox", "Автопереключение", () => updateService(id, { auto_switch: auto.checked })); autoLabel.append(auto, "Авто");
|
const replicas = service.replicas.filter((value) => value !== teamId);
|
||||||
const policyLabel = el("label", "policy-label"); const policy = el("select"); policy.dataset.write = ""; policy.setAttribute("aria-label", "Политика реплик");
|
if (service.primary && !replicas.includes(service.primary)) replicas.push(service.primary);
|
||||||
for (const [value, text] of [["isolate", "Isolate · отключать"], ["strict", "Strict · ждать"]]) { const option = el("option", "", text); option.value = value; policy.append(option); }
|
updateService(id, { primary: teamId, replicas, custom_ip: null });
|
||||||
policy.addEventListener("change", () => updateService(id, { replica_policy: policy.value })); policyLabel.append(policy);
|
}
|
||||||
const edit = button("Настроить", "small", () => openService(id));
|
function toggleReplica(id, teamId) {
|
||||||
actions.append(toggle, autoLabel, policyLabel, edit); header.append(heading, endpoint, route, actions);
|
const service = serviceById(id); if (!service || service.primary === teamId) return;
|
||||||
const stats = el("dl", "stats-grid"); const values = {};
|
const replicas = service.replicas.includes(teamId) ? service.replicas.filter((value) => value !== teamId) : [...service.replicas, teamId];
|
||||||
for (const [key, text] of [["active", "Активные"], ["accepted", "Принято"], ["completed", "Завершено"], ["errors", "Ошибки"], ["up", "↑ Основной / с"], ["down", "↓ Ответ / с"], ["replica_disconnects", "Откл. реплик"], ["replica_errors", "Ошибки реплик"]]) {
|
updateService(id, { replicas });
|
||||||
const item = el("div", "stat"); values[key] = el("dd", "", "—"); item.append(el("dt", "", text), values[key]); stats.append(item);
|
}
|
||||||
|
function openSettings(id) {
|
||||||
|
const service = serviceById(id); if (!service || state.busy || state.stale) return;
|
||||||
|
state.editId = id; $("settings-error").textContent = "";
|
||||||
|
$("service-name").value = service.name; $("service-listen").value = service.listen;
|
||||||
|
$("service-port").value = service.upstream_port; $("service-policy").value = service.replica_policy;
|
||||||
|
$("settingsModal").showModal();
|
||||||
|
}
|
||||||
|
async function removeCustom(id) {
|
||||||
|
const service = serviceById(id); if (!service?.custom || state.busy || state.stale) return;
|
||||||
|
if (!window.confirm(`Удалить прокси «${service.name}»?`)) return;
|
||||||
|
if (service.enabled && !await updateService(id, { enabled: false })) return;
|
||||||
|
await mutate(`/api/services/${encodeURIComponent(id)}`, "DELETE");
|
||||||
|
}
|
||||||
|
function createCard(service) {
|
||||||
|
const id = service.id; const root = el("div", "task-column"); root.dataset.serviceId = id;
|
||||||
|
const strip = el("div", "strip sb-off");
|
||||||
|
const header = el("div", "task-header"); header.addEventListener("click", () => openService(id));
|
||||||
|
const wrap = el("div", "task-title-wrap"); const titleRow = el("div", "task-title-row");
|
||||||
|
const title = button("task-title task-title-button", "Настройки сервиса", null, () => openSettings(id));
|
||||||
|
const port = el("div", "task-port"); titleRow.append(title, port);
|
||||||
|
const selected = el("div", "selected-team", "Команда не выбрана"); wrap.append(titleRow, selected);
|
||||||
|
const controls = el("div", "task-controls");
|
||||||
|
const auto = button("auto-switch-button", "Авто-переключение", "robot", () => { const s = serviceById(id); if (s) updateService(id, { auto_switch: !s.auto_switch }); });
|
||||||
|
const open = button("open-button", "Открыть сервис", "up-right-from-square", () => openService(id));
|
||||||
|
const toggle = button("toggle-button", "Запустить/Остановить", null, () => {
|
||||||
|
const s = serviceById(id); if (!s) return;
|
||||||
|
if (!s.enabled && !s.primary) { alert("Выберите команду или введите IP"); return; }
|
||||||
|
updateService(id, { enabled: !s.enabled });
|
||||||
|
});
|
||||||
|
controls.append(auto, open, toggle); header.append(wrap, controls);
|
||||||
|
const customRow = el("div", "custom-ip-row"); const customIp = el("input", "input-modern");
|
||||||
|
customIp.placeholder = "IP"; customIp.setAttribute("aria-label", `IP: ${service.name}`); customIp.autocomplete = "off";
|
||||||
|
customIp.addEventListener("change", () => updateService(id, { custom_ip: customIp.value.trim() || null, replicas: [] }));
|
||||||
|
const remove = button("btn-delete", "Удалить", "trash", () => removeCustom(id)); customRow.append(customIp, remove);
|
||||||
|
const stats = el("div", "task-stats"); const values = {};
|
||||||
|
for (const [key, label, cls] of [["accepted", "Всего:", ""], ["active", "Активные:", "stat-active"], ["completed", "Закрыто:", ""], ["errors", "Ошибки:", "stat-error"]]) {
|
||||||
|
const item = el("div", "stat-item"); const value = el("span", `stat-value ${cls}`, "0"); values[key] = value;
|
||||||
|
item.append(el("span", "stat-label", label), value); stats.append(item);
|
||||||
}
|
}
|
||||||
const totals = el("div", "bytes-line"); const table = el("table", "team-table");
|
const error = el("p", "service-error"); error.setAttribute("role", "status");
|
||||||
const caption = el("caption", "sr-only", "Маршруты команд");
|
const panel = el("div", "command-panel"); const list = el("div", "command-list"); panel.append(el("div", "panel-title", "Команды"), list);
|
||||||
const head = el("thead"); const headRow = el("tr");
|
root.append(strip, header, customRow, stats, error, panel); $("service-board").append(root);
|
||||||
for (const [text, className] of [["Осн.", "control-col"], ["Реп.", "control-col"], ["Команда", ""], ["TCP", "health-col"], ["ForcAD", "score-col"]]) { const th = el("th", className, text); th.scope = "col"; headRow.append(th); }
|
return { root, strip, title, port, selected, auto, toggle, customRow, customIp, remove, stats, values, error, list, rows: new Map() };
|
||||||
head.append(headRow); const tbody = el("tbody"); table.append(caption, head, tbody);
|
|
||||||
const error = el("p", "service-error error-text"); error.hidden = true;
|
|
||||||
const footer = el("footer", "card-footer"); const policyNote = el("span");
|
|
||||||
const remove = button("Удалить", "text-button", () => openDelete(id)); footer.append(policyNote, remove);
|
|
||||||
root.append(header, stats, totals, table, error, footer); $("service-board").append(root);
|
|
||||||
return { root, title, running, endpoint, route, toggle, auto, policy, edit, values, totals, tbody, error, remove, policyNote, rows: new Map() };
|
|
||||||
}
|
}
|
||||||
function createRow(card, serviceId, team) {
|
function createRow(card, id, team) {
|
||||||
const root = el("tr"); const mainCell = el("td", "control-cell"); const replicaCell = el("td", "control-cell");
|
const root = el("div", "command-item"); root.dataset.teamId = team.id; root.dataset.value = team.address;
|
||||||
const primary = control("radio", "", () => { const service = serviceById(serviceId); if (service) updateService(serviceId, { primary: team.id, replicas: service.replicas.filter((id) => id !== team.id) }); });
|
root.setAttribute("role", "button"); root.tabIndex = 0;
|
||||||
primary.name = `primary-${serviceId}`;
|
root.addEventListener("click", () => toggleReplica(id, team.id));
|
||||||
const replica = control("checkbox", "", () => { const service = serviceById(serviceId); if (service) updateService(serviceId, { replicas: replica.checked ? [...new Set([...service.replicas, team.id])] : service.replicas.filter((id) => id !== team.id) }); });
|
root.addEventListener("keydown", (event) => { if (event.target === root && ["Enter", " "].includes(event.key)) { event.preventDefault(); toggleReplica(id, team.id); } });
|
||||||
mainCell.append(primary); replicaCell.append(replica);
|
const left = el("div", "command-left");
|
||||||
const meta = el("td"); const name = el("span", "team-name"); const address = el("span", "team-address"); meta.append(name, address);
|
const primary = button("primary-area", `Основной маршрут ${team.name}`, null, () => selectPrimary(id, team.id));
|
||||||
const healthCell = el("td"); const health = el("span", "health-badge"); healthCell.append(health);
|
const circle = el("span", "primary-toggle"); circle.append(el("span", "primary-dot")); primary.append(circle); primary.setAttribute("aria-pressed", "false");
|
||||||
const scoreCell = el("td"); const score = el("span", "score-status"); const flags = el("span", "score-flags"); scoreCell.append(score, flags);
|
const health = el("span", "option-status unknown"); left.append(primary, health);
|
||||||
root.append(mainCell, replicaCell, meta, healthCell, scoreCell); card.tbody.append(root);
|
const meta = el("div", "command-meta"); const name = el("span", "command-name"); const address = el("span", "command-ip"); const flags = el("span", "flag-chip", "-"); meta.append(name, address, flags);
|
||||||
return { root, primary, replica, name, address, health, score, flags };
|
root.append(left, meta); card.list.append(root); return { root, primary, health, name, address, flags };
|
||||||
}
|
}
|
||||||
function trackRates(snapshot) {
|
function render(snapshot) {
|
||||||
for (const service of snapshot.services) {
|
if (snapshot) {
|
||||||
const stats = service.engine?.stats || {}; const previous = state.baselines.get(service.id);
|
if (!Array.isArray(snapshot.services) || !Array.isArray(snapshot.teams) || !Number.isFinite(snapshot.now_ms)) throw new Error("Некорректный снимок");
|
||||||
const dt = previous ? snapshot.now_ms - previous.at : 0;
|
|
||||||
if (previous && dt > 0) {
|
|
||||||
const reset = stats.bytes_up < previous.up || stats.bytes_down < previous.down;
|
|
||||||
state.rates.set(service.id, reset ? null : { up: (stats.bytes_up - previous.up) * 1000 / dt, down: (stats.bytes_down - previous.down) * 1000 / dt });
|
|
||||||
} else if (!previous || dt < 0) state.rates.set(service.id, null);
|
|
||||||
if (!previous || dt !== 0) state.baselines.set(service.id, { at: snapshot.now_ms, up: stats.bytes_up || 0, down: stats.bytes_down || 0 });
|
|
||||||
}
|
|
||||||
}
|
|
||||||
function render(snapshot, fresh = true, reconcile = false) {
|
|
||||||
if (fresh) {
|
|
||||||
if (!Array.isArray(snapshot.services) || !Array.isArray(snapshot.teams) || !Number.isFinite(snapshot.now_ms)) throw new Error("Некорректный снимок сервера");
|
|
||||||
if (state.snapshot && snapshot.now_ms < state.snapshot.now_ms) return;
|
if (state.snapshot && snapshot.now_ms < state.snapshot.now_ms) return;
|
||||||
trackRates(snapshot); state.snapshot = snapshot;
|
state.snapshot = snapshot; state.stale = false;
|
||||||
}
|
}
|
||||||
const scoreboard = snapshot.scoreboard || {};
|
if (!state.snapshot) return;
|
||||||
const scoreStale = state.stale || scoreboard.stale || !scoreboard.connected;
|
const current = state.snapshot; const board = current.scoreboard || {}; const scoreFresh = !state.stale && board.connected && !board.stale;
|
||||||
$("snapshot-time").textContent = `${time(snapshot.now_ms)}${state.stale ? " · устарел" : ""}`;
|
const locked = state.busy || state.stale;
|
||||||
$("service-count").textContent = `${snapshot.services.filter((service) => service.engine?.running).length} / ${snapshot.services.length} работают`;
|
$("add-service").disabled = locked; $("logout").disabled = state.busy;
|
||||||
$("scoreboard-state").textContent = !scoreboard.enabled ? "Выключен" : scoreStale ? "Устаревшие данные" : `Раунд ${scoreboard.round ?? "—"}`;
|
for (const form of [$("customProxyForm"), $("settings-form")]) for (const node of form.querySelectorAll("input, select, button")) node.disabled = locked;
|
||||||
$("scoreboard-state").className = !scoreboard.enabled ? "muted" : scoreStale ? "warn" : "good";
|
for (const node of document.querySelectorAll("[data-close]")) node.disabled = state.busy;
|
||||||
$("scoreboard-detail").textContent = scoreboard.enabled ? `${scoreboard.connected ? "Подключён" : "Нет подключения"} · обновлён ${time(scoreboard.updated_at_ms)}${scoreboard.last_error ? ` · ${scoreboard.last_error}` : ""}` : "ForcAD выключен. TCP-доступность проверяется отдельно от состояния чекера.";
|
const ids = new Set(current.services.map((service) => service.id));
|
||||||
$("empty-state").hidden = snapshot.services.length > 0;
|
for (const [id, card] of state.cards) if (!ids.has(id)) { card.root.remove(); state.cards.delete(id); }
|
||||||
const ids = new Set(snapshot.services.map((service) => service.id));
|
for (const service of current.services) {
|
||||||
for (const [id, card] of state.cards) if (!ids.has(id)) { card.root.remove(); state.cards.delete(id); state.baselines.delete(id); state.rates.delete(id); }
|
|
||||||
for (const service of snapshot.services) {
|
|
||||||
let card = state.cards.get(service.id);
|
let card = state.cards.get(service.id);
|
||||||
if (!card) { card = createCard(service.id); state.cards.set(service.id, card); }
|
if (!card) { card = createCard(service); state.cards.set(service.id, card); }
|
||||||
const engine = service.engine || {}; const stats = engine.stats || {}; const rate = state.rates.get(service.id);
|
card.title.textContent = service.name; card.port.textContent = `:${portOf(service)}`;
|
||||||
card.root.classList.toggle("running", Boolean(engine.running)); card.title.textContent = service.name;
|
card.title.setAttribute("aria-label", `Настройки сервиса ${service.name}`); card.title.disabled = locked;
|
||||||
card.running.textContent = engine.running ? "● РАБОТАЕТ" : service.enabled ? "● ОШИБКА" : "○ ОСТАНОВЛЕН";
|
card.auto.classList.toggle("active", service.auto_switch); card.auto.setAttribute("aria-pressed", String(service.auto_switch));
|
||||||
card.running.className = `service-state ${engine.running ? "good" : service.enabled ? "bad" : "muted"}`;
|
card.auto.title = service.auto_switch ? "Авто-переключение включено" : "Авто-переключение"; card.auto.disabled = locked;
|
||||||
card.endpoint.textContent = `${service.listen} → :${service.upstream_port}`;
|
card.toggle.classList.toggle("active", Boolean(service.engine?.running)); card.toggle.disabled = locked;
|
||||||
const primary = snapshot.teams.find((team) => team.id === service.primary);
|
card.toggle.setAttribute("aria-label", `${service.enabled ? "Остановить" : "Запустить"} ${service.name}`); card.toggle.setAttribute("aria-pressed", String(Boolean(service.engine?.running)));
|
||||||
card.route.textContent = `${primary ? primary.name : "Основной маршрут не выбран"} · реплик: ${service.replicas.length}`;
|
card.customRow.hidden = !service.custom; card.customIp.disabled = locked; card.remove.disabled = locked;
|
||||||
card.toggle.textContent = service.enabled ? "Остановить" : "Запустить";
|
if (document.activeElement !== card.customIp) card.customIp.value = service.custom_ip || "";
|
||||||
card.toggle.disabled = state.busy || (!service.enabled && !service.primary);
|
const primary = teamById(service.primary);
|
||||||
card.auto.disabled = state.busy; card.policy.disabled = state.busy; card.edit.disabled = state.busy;
|
card.selected.replaceChildren();
|
||||||
card.auto.setAttribute("aria-label", `Автопереключение: ${service.name}`); card.policy.setAttribute("aria-label", `Политика реплик: ${service.name}`);
|
if (primary) {
|
||||||
if (reconcile || document.activeElement !== card.auto) card.auto.checked = service.auto_switch;
|
card.selected.append(el("span", "selected-team-name", primary.name));
|
||||||
if (reconcile || document.activeElement !== card.policy) card.policy.value = service.replica_policy;
|
if (primary.name !== primary.address) card.selected.append(el("span", "selected-team-ip", primary.address));
|
||||||
card.remove.disabled = state.busy || service.enabled || engine.running;
|
if (service.replicas.length) card.selected.append(el("span", "selected-team-tail", ` · дубли: ${service.replicas.length}`));
|
||||||
card.remove.title = service.enabled || engine.running ? "Сначала остановите сервис" : "Удалить настройки сервиса";
|
} else card.selected.textContent = "Команда не выбрана";
|
||||||
card.policyNote.textContent = service.replica_policy === "strict" ? "Strict: общий поток ждёт реплики" : "Isolate: отстающая реплика отключается";
|
const stats = service.engine?.stats || {};
|
||||||
for (const key of ["active", "accepted", "completed", "replica_disconnects", "replica_errors"]) card.values[key].textContent = (stats[key] || 0).toLocaleString("ru-RU");
|
card.stats.style.visibility = service.engine?.running || stats.active ? "visible" : "hidden";
|
||||||
card.values.errors.textContent = ((stats.connect_errors || 0) + (stats.io_errors || 0) + (stats.rejected || 0)).toLocaleString("ru-RU");
|
for (const key of ["accepted", "active", "completed"]) card.values[key].textContent = stats[key] || 0;
|
||||||
card.values.up.textContent = !state.stale && rate ? `${bytes(rate.up)}/с` : "—";
|
const errors = (stats.connect_errors || 0) + (stats.io_errors || 0) + (stats.rejected || 0) + (stats.accept_errors || 0);
|
||||||
card.values.down.textContent = !state.stale && rate ? `${bytes(rate.down)}/с` : "—";
|
card.values.errors.textContent = errors; card.values.errors.classList.toggle("text-danger", errors > 0);
|
||||||
card.totals.textContent = `Всего: ↑ ${bytes(stats.bytes_up)} · ↓ ${bytes(stats.bytes_down)} · реплики ${bytes(stats.replica_bytes)}`;
|
card.stats.title = `Байты: ↑ ${stats.bytes_up || 0}, ↓ ${stats.bytes_down || 0}; реплики: ${stats.replica_bytes || 0}; отключено реплик: ${stats.replica_disconnects || 0}; ошибок реплик: ${stats.replica_errors || 0}`;
|
||||||
card.error.textContent = engine.last_error || ""; card.error.hidden = !engine.last_error;
|
card.error.textContent = service.engine?.last_error ? `Последняя ошибка: ${service.engine.last_error}` : "";
|
||||||
const teamIds = new Set(snapshot.teams.map((team) => team.id));
|
const localTeam = current.teams.find((team) => team.address === current.main_team_ip);
|
||||||
|
const boardTeam = board.teams?.find((team) => team.address === current.main_team_ip || (localTeam && team.name === localTeam.name));
|
||||||
|
const boardTask = board.tasks?.find((task) => task.name === service.name || task.name === service.id);
|
||||||
|
const entry = scoreFresh && current.main_team_ip ? board.entries?.[boardTask?.id]?.[boardTeam?.id] : null;
|
||||||
|
card.strip.className = `strip sb-${statusClass[entry?.status] || "off"}`;
|
||||||
|
card.strip.title = !board.enabled ? "Scoreboard выключен" : !current.main_team_ip ? "Для статуса своей команды задайте main_team_ip в конфигурации" : scoreFresh ? `Своя команда: ${statusName[entry?.status] || "нет данных"}` : "Данные scoreboard устарели или соединение отсутствует";
|
||||||
|
const teams = current.teams.filter((team) => !team.hidden || service.primary === team.id || service.replicas.includes(team.id));
|
||||||
|
const teamIds = new Set(teams.map((team) => team.id));
|
||||||
for (const [id, row] of card.rows) if (!teamIds.has(id)) { row.root.remove(); card.rows.delete(id); }
|
for (const [id, row] of card.rows) if (!teamIds.has(id)) { row.root.remove(); card.rows.delete(id); }
|
||||||
for (const team of snapshot.teams) {
|
for (const team of teams) {
|
||||||
let row = card.rows.get(team.id);
|
let row = card.rows.get(team.id); if (!row) { row = createRow(card, service.id, team); card.rows.set(team.id, row); }
|
||||||
if (!row) { row = createRow(card, service.id, team); card.rows.set(team.id, row); }
|
row.name.textContent = team.name; row.address.textContent = team.address; row.address.hidden = team.name === team.address;
|
||||||
row.name.textContent = team.name; row.address.textContent = team.address;
|
row.root.classList.toggle("primary-active", service.primary === team.id); row.root.classList.toggle("replica-active", service.replicas.includes(team.id));
|
||||||
row.primary.setAttribute("aria-label", `Основной маршрут ${team.name}: ${service.name}`);
|
row.root.setAttribute("aria-label", `Реплика ${team.name}: ${service.name}`); row.root.setAttribute("aria-disabled", String(locked || service.primary === team.id));
|
||||||
row.replica.setAttribute("aria-label", `Реплика ${team.name}: ${service.name}`);
|
row.root.setAttribute("aria-pressed", String(service.replicas.includes(team.id))); row.primary.disabled = locked;
|
||||||
if (reconcile || document.activeElement !== row.primary) row.primary.checked = service.primary === team.id;
|
row.primary.setAttribute("aria-pressed", String(service.primary === team.id));
|
||||||
if (reconcile || document.activeElement !== row.replica) row.replica.checked = service.replicas.includes(team.id);
|
const h = service.health?.[team.id]; const recent = !state.stale && h?.checked_at_ms && current.now_ms - h.checked_at_ms <= (current.health_max_age_ms || 12000);
|
||||||
row.primary.disabled = state.busy; row.replica.disabled = state.busy || service.primary === team.id;
|
row.health.className = `option-status ${recent ? h.reachable ? "available" : "unavailable" : "unknown"}`;
|
||||||
row.root.classList.toggle("is-primary", service.primary === team.id); row.root.classList.toggle("is-replica", service.replicas.includes(team.id));
|
row.health.title = recent ? `TCP: ${h.reachable ? "доступен" : "недоступен"}, ${h.latency_ms} мс` : "TCP: нет свежих данных";
|
||||||
const health = service.health?.[team.id];
|
const score = service.scoreboard?.[team.id]; const cls = scoreFresh ? statusClass[score?.status] || "off" : "off";
|
||||||
const unknown = !health?.checked_at_ms;
|
row.name.className = `command-name sb-${cls}`; row.flags.classList.toggle("stale-data", !scoreFresh);
|
||||||
const healthStale = state.stale || (!unknown && snapshot.now_ms - health.checked_at_ms > 15000);
|
row.flags.textContent = score ? `-${score.lost ?? 0}` : "-";
|
||||||
row.health.textContent = unknown ? "?" : healthStale ? "Устар." : health.reachable ? "UP" : "DOWN";
|
row.flags.title = score ? `${statusName[score.status] || score.status}; потеряно: ${score.lost || 0}; украдено: ${score.stolen || 0}${scoreFresh ? "" : "; устаревшие данные"}` : "Нет данных scoreboard";
|
||||||
row.health.className = `health-badge ${unknown ? "muted" : healthStale ? "warn" : health.reachable ? "good" : "bad"}`;
|
}
|
||||||
row.health.title = unknown ? "Ещё не проверено" : `TCP: ${health.reachable ? "доступен" : "недоступен"}; стабильно здоров: ${health.healthy ? "да" : "нет"}; ${health.latency_ms ?? "—"} мс; ${time(health.checked_at_ms)}`;
|
if (scoreFresh && !card.list.contains(document.activeElement)) {
|
||||||
const entry = service.scoreboard?.[team.id];
|
const ordered = [...teams].sort((a, b) => {
|
||||||
row.score.textContent = entry ? `${statusNames[entry.status] || entry.status}${scoreStale ? " *" : ""}` : "—";
|
const sa = service.scoreboard?.[a.id]; const sb = service.scoreboard?.[b.id];
|
||||||
row.score.className = `score-status ${scoreStale ? "stale-data" : entry?.status === 101 ? "good" : entry ? "bad" : "muted"}`;
|
return (statusOrder[sa?.status] ?? 5) - (statusOrder[sb?.status] ?? 5) || (sa?.lost ?? Number.MAX_SAFE_INTEGER) - (sb?.lost ?? Number.MAX_SAFE_INTEGER) || a.name.localeCompare(b.name);
|
||||||
row.score.title = scoreStale ? "Состояние чекера устарело" : "Состояние чекера ForcAD";
|
});
|
||||||
row.flags.textContent = entry ? `↑${entry.stolen ?? 0} ↓${entry.lost ?? 0}` : "↑— ↓—";
|
ordered.forEach((team, index) => { const node = card.rows.get(team.id).root; if (card.list.children[index] !== node) card.list.insertBefore(node, card.list.children[index] || null); });
|
||||||
row.flags.title = "Украдено / потеряно";
|
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
if (fresh) {
|
|
||||||
const events = [...(snapshot.events || [])].slice(-30).reverse();
|
|
||||||
$("event-log").replaceChildren(...events.map((event) => { const li = el("li"); const stamp = el("time", "", time(event.at_ms)); li.append(stamp, document.createTextNode(`${event.service ? `${event.service}: ` : ""}${event.message}`)); return li; }));
|
|
||||||
}
|
|
||||||
}
|
|
||||||
function openService(id = null) {
|
|
||||||
const service = id ? serviceById(id) : null;
|
|
||||||
if (state.busy || (id && !service)) return;
|
|
||||||
state.editId = id;
|
|
||||||
const form = $("service-form"); form.reset(); message("service-form-error", "");
|
|
||||||
$("service-dialog-title").textContent = service ? "Настроить сервис" : "Добавить сервис";
|
|
||||||
for (const key of ["id", "name", "listen", "upstream_port", "replica_policy"]) field(form, key).value = service?.[key] ?? ({ listen: "0.0.0.0:18080", upstream_port: 8080, replica_policy: "isolate" }[key] || "");
|
|
||||||
field(form, "id").disabled = Boolean(id);
|
|
||||||
field(form, "enabled").checked = service?.enabled || false; field(form, "auto_switch").checked = service?.auto_switch || false;
|
|
||||||
const primary = field(form, "primary"); primary.replaceChildren();
|
|
||||||
const none = el("option", "", "Не выбран (сервис остановлен)"); none.value = ""; primary.append(none);
|
|
||||||
$("dialog-replicas").replaceChildren();
|
|
||||||
for (const team of state.snapshot.teams) {
|
|
||||||
const option = el("option", "", `${team.name} · ${team.address}`); option.value = team.id; primary.append(option);
|
|
||||||
const label = el("label"); const input = el("input"); input.type = "checkbox"; input.name = "replicas"; input.value = team.id; input.checked = service?.replicas.includes(team.id) || false;
|
|
||||||
label.append(input, document.createTextNode(team.name)); $("dialog-replicas").append(label);
|
|
||||||
}
|
|
||||||
primary.value = service?.primary || "";
|
|
||||||
syncDialogReplicas(); $("service-dialog").showModal();
|
|
||||||
}
|
|
||||||
function syncDialogReplicas() {
|
|
||||||
const primary = field($("service-form"), "primary").value;
|
|
||||||
$("dialog-replicas").querySelectorAll("input").forEach((input) => { input.disabled = input.value === primary; if (input.disabled) input.checked = false; });
|
|
||||||
}
|
|
||||||
function openDelete(id) {
|
|
||||||
const service = serviceById(id); if (!service || service.enabled || state.busy) return;
|
|
||||||
state.deleteId = id; message("delete-error", "");
|
|
||||||
$("delete-detail").textContent = `«${service.name}» (${service.listen}), основной маршрут ${service.primary || "не выбран"}, реплик: ${service.replicas.length}.`;
|
|
||||||
$("delete-dialog").showModal();
|
|
||||||
}
|
}
|
||||||
|
$("dismiss-alert").addEventListener("click", () => alert(""));
|
||||||
$("login-form").addEventListener("submit", async (event) => {
|
$("login-form").addEventListener("submit", async (event) => {
|
||||||
event.preventDefault(); $("login-button").disabled = true; message("login-error", "");
|
event.preventDefault(); $("login-button").disabled = true; $("login-error").textContent = "";
|
||||||
try { await enter(await api("/api/login", { method: "POST", body: JSON.stringify({ password: $("password").value }) })); }
|
try { await enter(await api("/api/login", { method: "POST", body: JSON.stringify({ password: $("password").value }) })); }
|
||||||
catch (error) { message("login-error", error.message); }
|
catch (error) { $("login-error").textContent = error.message; }
|
||||||
finally { $("login-button").disabled = false; }
|
finally { $("login-button").disabled = false; }
|
||||||
});
|
});
|
||||||
$("logout").addEventListener("click", async () => { if (state.busy) return; setBusy(true); try { await api("/api/logout", { method: "POST" }); showLogin(); } catch (error) { message("global-error", error.message); } finally { setBusy(false); } });
|
$("logout").addEventListener("click", async () => { if (state.busy) return; try { await api("/api/logout", { method: "POST" }); showLogin(); alert(""); } catch (error) { alert(error.message); } });
|
||||||
$("refresh").addEventListener("click", async () => { $("refresh").disabled = true; try { await refresh(); message("global-error", ""); } catch (error) { message("global-error", error.message); } finally { $("refresh").disabled = false; } });
|
$("refresh").addEventListener("click", async () => { try { await refresh(); alert(""); if (!state.source) startStream(); } catch (error) { alert(error.message); } });
|
||||||
$("add-service").addEventListener("click", () => openService());
|
$("add-service").addEventListener("click", () => { $("customProxyForm").reset(); $("custom-error").textContent = ""; $("addCustomModal").showModal(); });
|
||||||
$("add-team").addEventListener("click", () => { $("team-form").reset(); message("team-form-error", ""); $("team-dialog").showModal(); });
|
for (const node of document.querySelectorAll("[data-close]")) node.addEventListener("click", () => { if (!state.busy) $(node.dataset.close).close(); });
|
||||||
document.querySelectorAll("[data-close]").forEach((node) => node.addEventListener("click", () => { if (!state.busy) $(node.dataset.close).close(); }));
|
for (const dialog of document.querySelectorAll("dialog")) dialog.addEventListener("cancel", (event) => { if (state.busy) event.preventDefault(); });
|
||||||
document.querySelectorAll("dialog").forEach((dialog) => dialog.addEventListener("cancel", (event) => { if (state.busy) event.preventDefault(); }));
|
$("customProxyForm").addEventListener("submit", async (event) => {
|
||||||
field($("service-form"), "primary").addEventListener("change", syncDialogReplicas);
|
event.preventDefault(); const port = Number($("customPort").value);
|
||||||
$("service-form").addEventListener("submit", async (event) => {
|
const id = `custom-${Date.now().toString(36)}-${Math.random().toString(36).slice(2, 8)}`;
|
||||||
event.preventDefault(); const form = event.currentTarget;
|
const body = { id, name: $("customName").value.trim(), listen: `0.0.0.0:${port}`, upstream_port: port, primary: null, replicas: [], enabled: false, auto_switch: false, replica_policy: "isolate", custom: true, custom_ip: $("customIp").value.trim() || null };
|
||||||
const id = field(form, "id").value.trim();
|
if (await mutate(`/api/services/${encodeURIComponent(id)}`, "PUT", body, "custom-error")) $("addCustomModal").close();
|
||||||
if (!state.editId && serviceById(id)) { message("service-form-error", "Сервис с таким ID уже существует. Откройте его настройки."); return; }
|
|
||||||
const body = { id, name: field(form, "name").value.trim(), listen: field(form, "listen").value.trim(), upstream_port: Number(field(form, "upstream_port").value), primary: field(form, "primary").value || null, replicas: [...form.querySelectorAll("input[name=replicas]:checked")].map((input) => input.value), enabled: field(form, "enabled").checked, auto_switch: field(form, "auto_switch").checked, replica_policy: field(form, "replica_policy").value };
|
|
||||||
if (body.enabled && !body.primary) { message("service-form-error", "Выберите основной маршрут перед запуском."); return; }
|
|
||||||
if (await mutate(`/api/services/${encodeURIComponent(id)}`, "PUT", body, "service-form-error")) $("service-dialog").close();
|
|
||||||
syncDialogReplicas();
|
|
||||||
});
|
});
|
||||||
$("team-form").addEventListener("submit", async (event) => {
|
$("settings-form").addEventListener("submit", async (event) => {
|
||||||
event.preventDefault(); const form = event.currentTarget;
|
event.preventDefault();
|
||||||
const body = Object.fromEntries(["id", "name", "address"].map((key) => [key, field(form, key).value.trim()]));
|
if (await updateService(state.editId, { name: $("service-name").value.trim(), listen: $("service-listen").value.trim(), upstream_port: Number($("service-port").value), replica_policy: $("service-policy").value }, "settings-error")) $("settingsModal").close();
|
||||||
const existing = state.snapshot.teams.find((team) => team.id === body.id);
|
|
||||||
if (existing && existing.address !== body.address) { message("team-form-error", "У этого ID уже другой адрес. Выберите новый ID."); return; }
|
|
||||||
if (await mutate(`/api/teams/${encodeURIComponent(body.id)}`, "PUT", body, "team-form-error")) $("team-dialog").close();
|
|
||||||
});
|
|
||||||
$("delete-form").addEventListener("submit", async (event) => {
|
|
||||||
event.preventDefault(); const service = serviceById(state.deleteId);
|
|
||||||
if (!service || service.enabled) { message("delete-error", "Сервис отсутствует или уже запущен. Закройте окно и проверьте состояние."); return; }
|
|
||||||
if (await mutate(`/api/services/${encodeURIComponent(state.deleteId)}`, "DELETE", undefined, "delete-error")) $("delete-dialog").close();
|
|
||||||
});
|
});
|
||||||
window.addEventListener("pagehide", closeStream);
|
window.addEventListener("pagehide", closeStream);
|
||||||
window.addEventListener("pageshow", (event) => { if (event.persisted && state.csrf) { markStale("Переподключение…"); startStream(); } });
|
window.addEventListener("pageshow", (event) => { if (event.persisted && state.csrf) startStream(); });
|
||||||
(async () => { try { await enter(await api("/api/session")); } catch (error) { showLogin(error.status === 401 ? "Введите пароль для входа." : "Не удалось проверить сессию. Можно повторить вход."); if (error.status !== 401) message("login-error", error.message); } })();
|
(async () => { try { await enter(await api("/api/session")); } catch (error) { showLogin(); if (error.status !== 401) $("login-error").textContent = error.message; } })();
|
||||||
})();
|
})();
|
||||||
|
|||||||
132
web/index.html
132
web/index.html
@ -1,88 +1,64 @@
|
|||||||
<!doctype html>
|
<!DOCTYPE html>
|
||||||
<html lang="ru">
|
<html lang="ru">
|
||||||
<head>
|
<head>
|
||||||
<meta charset="utf-8">
|
<meta charset="UTF-8">
|
||||||
<meta name="viewport" content="width=device-width, initial-scale=1">
|
<meta name="viewport" content="width=device-width, initial-scale=1.0">
|
||||||
<meta name="color-scheme" content="dark">
|
<title>Proxy Manager</title>
|
||||||
<title>Mirror Tool — операторская</title>
|
<link href="/vendor/bootstrap.min.css" rel="stylesheet">
|
||||||
<link rel="stylesheet" href="/app.css">
|
<link href="/vendor/fontawesome.min.css" rel="stylesheet">
|
||||||
|
<link href="/app.css" rel="stylesheet">
|
||||||
<script src="/app.js" defer></script>
|
<script src="/app.js" defer></script>
|
||||||
</head>
|
</head>
|
||||||
<body>
|
<body>
|
||||||
<div id="login-view" class="login-wrap">
|
<div class="container mt-2">
|
||||||
<main class="login-panel">
|
<div id="alert" class="alert alert-danger alert-dismissible fade show" role="alert" hidden><span id="alert-message"></span><button id="dismiss-alert" type="button" class="btn-close btn-close-white" aria-label="Закрыть сообщение"></button></div>
|
||||||
<p class="eyebrow">TCP / ОПЕРАТОРСКАЯ</p>
|
<div id="login" class="row justify-content-center align-items-center login-row" hidden>
|
||||||
<h1>Mirror Tool<span class="brand-mark" aria-hidden="true">↗↗</span></h1>
|
<div class="col-md-6 col-lg-4"><div class="service-card text-center login-card">
|
||||||
<p class="muted">Маршруты, реплики и состояние сервисов.</p>
|
<div class="mb-4"><h2 class="login-heading">Proxy</h2><p class="login-caption">Введите пароль</p></div>
|
||||||
<form id="login-form">
|
<form id="login-form">
|
||||||
<label for="password">Пароль оператора</label>
|
<div class="mb-4"><input type="password" class="input-modern w-100 login-password" id="password" name="password" placeholder="Пароль" aria-label="Пароль" autocomplete="current-password" required></div>
|
||||||
<input id="password" type="password" autocomplete="current-password" required>
|
<p id="login-error" class="form-error" role="alert"></p><button id="login-button" type="submit" class="add-button w-100">Войти</button>
|
||||||
<button class="accent" id="login-button" type="submit">Войти</button>
|
</form>
|
||||||
</form>
|
</div></div>
|
||||||
<p id="login-status" class="muted" role="status">Проверяем сессию…</p>
|
</div>
|
||||||
<p id="login-error" class="error-text" role="alert" hidden></p>
|
<div id="header" class="d-flex justify-content-between align-items-center header-row" hidden>
|
||||||
</main>
|
<h1 class="section-title">Mirror Tool</h1>
|
||||||
</div>
|
<div class="header-actions">
|
||||||
<main id="dashboard" hidden>
|
<button id="refresh" class="refresh-button" title="Обновить" aria-label="Обновить"><i class="fas fa-sync-alt" aria-hidden="true"></i></button>
|
||||||
<header class="masthead">
|
<button id="logout" class="logout-button" title="Выход" aria-label="Выход"><i class="fas fa-sign-out-alt" aria-hidden="true"></i></button>
|
||||||
<div><p class="eyebrow">TCP / ОПЕРАТОРСКАЯ</p><h1>Mirror Tool<span class="brand-mark" aria-hidden="true">↗↗</span></h1></div>
|
<button id="add-service" class="add-button" title="Добавить прокси" aria-label="Добавить прокси"><i class="fas fa-plus" aria-hidden="true"></i></button>
|
||||||
<nav class="actions" aria-label="Управление">
|
|
||||||
<button id="add-team" data-write>+ Команда</button>
|
|
||||||
<button id="add-service" class="accent" data-write>+ Сервис</button>
|
|
||||||
<button id="refresh">Обновить</button>
|
|
||||||
<button id="logout" data-write>Выйти</button>
|
|
||||||
</nav>
|
|
||||||
</header>
|
|
||||||
<section class="telemetry" aria-label="Состояние панели">
|
|
||||||
<div><span class="label">СВЯЗЬ</span><strong id="stream-state" class="warn">Подключение…</strong></div>
|
|
||||||
<div><span class="label">СЕРВИСЫ</span><strong id="service-count">—</strong></div>
|
|
||||||
<div><span class="label">FORCAD</span><strong id="scoreboard-state">—</strong></div>
|
|
||||||
<div><span class="label">СНИМОК</span><strong id="snapshot-time">—</strong></div>
|
|
||||||
</section>
|
|
||||||
<p id="scoreboard-detail" class="status-detail muted"></p>
|
|
||||||
<div id="global-error" class="notice error-text" role="alert" hidden></div>
|
|
||||||
<p id="action-status" class="action-status" role="status" aria-live="polite"></p>
|
|
||||||
<section id="service-board" class="service-board" aria-label="Сервисы"></section>
|
|
||||||
<section id="empty-state" class="empty-state" hidden><h2>Добавьте первый сервис</h2><p>Создайте команду с адресом, затем задайте порт слушателя и основной маршрут.</p></section>
|
|
||||||
<footer class="dashboard-footer"><p>◉ Основной маршрут □ Реплика</p><p>Изменения сохраняются сразу. Новые маршруты применяются к новым соединениям.</p></footer>
|
|
||||||
<details class="journal"><summary>Журнал событий</summary><ol id="event-log"></ol></details>
|
|
||||||
</main>
|
|
||||||
<dialog id="service-dialog" aria-labelledby="service-dialog-title">
|
|
||||||
<form id="service-form">
|
|
||||||
<header class="dialog-header"><h2 id="service-dialog-title">Добавить сервис</h2><button type="button" data-close="service-dialog" aria-label="Закрыть окно сервиса">×</button></header>
|
|
||||||
<div class="form-grid">
|
|
||||||
<label>ID<input name="id" required pattern="[A-Za-z0-9_-]+" maxlength="80" placeholder="notes"><small>Латинские буквы, цифры, _ и -</small></label>
|
|
||||||
<label>Название<input name="name" required maxlength="120" placeholder="Notes"></label>
|
|
||||||
<label>Адрес слушателя<input name="listen" required placeholder="0.0.0.0:18080"></label>
|
|
||||||
<label>Порт сервиса команды<input name="upstream_port" type="number" min="1" max="65535" required value="8080"></label>
|
|
||||||
<label>Основной маршрут<select name="primary"></select></label>
|
|
||||||
<label>Политика реплик<select name="replica_policy"><option value="isolate">Isolate — отключать отстающую</option><option value="strict">Strict — ждать все реплики</option></select></label>
|
|
||||||
</div>
|
</div>
|
||||||
<fieldset><legend>Реплики</legend><div id="dialog-replicas" class="replica-options"></div></fieldset>
|
</div>
|
||||||
<div class="form-checks"><label><input name="enabled" type="checkbox"> Запустить слушатель</label><label><input name="auto_switch" type="checkbox"> Автопереключение</label></div>
|
<div id="dashboard" class="main-dashboard" hidden><div id="service-board" class="task-board board-scroll"></div></div>
|
||||||
<p class="muted hint">Основной маршрут можно не выбирать у остановленного сервиса. Strict замедляет поток до скорости реплик; Isolate явно отключает отстающую реплику.</p>
|
</div>
|
||||||
<p id="service-form-error" class="error-text" role="alert" hidden></p>
|
<dialog id="addCustomModal" class="legacy-modal" aria-labelledby="add-title">
|
||||||
<footer class="dialog-actions"><button type="button" data-close="service-dialog">Отмена</button><button type="submit" class="accent" data-write>Сохранить сервис</button></footer>
|
<div class="modal-dialog"><div class="modal-content">
|
||||||
</form>
|
<div class="modal-header"><h5 class="modal-title" id="add-title">Добавить прокси</h5><button type="button" class="btn-close btn-close-white" data-close="addCustomModal" aria-label="Закрыть"></button></div>
|
||||||
|
<form id="customProxyForm">
|
||||||
|
<div class="modal-body">
|
||||||
|
<div class="mb-3"><label for="customName" class="form-label">Название</label><input type="text" class="input-modern w-100" id="customName" placeholder="Название сервиса" maxlength="128" required></div>
|
||||||
|
<div class="mb-3"><label for="customPort" class="form-label">Порт</label><input type="number" class="input-modern w-100" id="customPort" min="1" max="65535" placeholder="8080" required></div>
|
||||||
|
<div class="mb-3"><label for="customIp" class="form-label">IP (опционально)</label><input type="text" class="input-modern w-100" id="customIp" placeholder="192.168.1.100" autocomplete="off"></div>
|
||||||
|
<p id="custom-error" class="form-error" role="alert"></p>
|
||||||
|
</div>
|
||||||
|
<div class="modal-footer"><button type="button" class="add-button cancel-button" data-close="addCustomModal">Отмена</button><button type="submit" class="add-button">Добавить</button></div>
|
||||||
|
</form>
|
||||||
|
</div></div>
|
||||||
</dialog>
|
</dialog>
|
||||||
<dialog id="team-dialog" aria-labelledby="team-dialog-title">
|
<dialog id="settingsModal" class="legacy-modal" aria-labelledby="settings-title">
|
||||||
<form id="team-form">
|
<div class="modal-dialog"><div class="modal-content">
|
||||||
<header class="dialog-header"><h2 id="team-dialog-title">Добавить команду</h2><button type="button" data-close="team-dialog" aria-label="Закрыть окно команды">×</button></header>
|
<div class="modal-header"><h5 class="modal-title" id="settings-title">Настройки сервиса</h5><button type="button" class="btn-close btn-close-white" data-close="settingsModal" aria-label="Закрыть"></button></div>
|
||||||
<label>ID<input name="id" required pattern="[A-Za-z0-9_-]+" maxlength="80" placeholder="team-a"></label>
|
<form id="settings-form">
|
||||||
<label>Название<input name="name" required maxlength="120" placeholder="Команда A"></label>
|
<div class="modal-body">
|
||||||
<label>Адрес<input name="address" required placeholder="192.0.2.10"></label>
|
<div class="mb-3"><label for="service-name" class="form-label">Название</label><input id="service-name" class="input-modern w-100" maxlength="128" required></div>
|
||||||
<p class="muted hint">Адрес существующей команды неизменяем. Для другого адреса добавьте команду с новым ID.</p>
|
<div class="mb-3"><label for="service-listen" class="form-label">Адрес и порт прокси</label><input id="service-listen" class="input-modern w-100" required></div>
|
||||||
<p id="team-form-error" class="error-text" role="alert" hidden></p>
|
<div class="mb-3"><label for="service-port" class="form-label">Порт сервиса команды</label><input id="service-port" type="number" min="1" max="65535" class="input-modern w-100" required></div>
|
||||||
<footer class="dialog-actions"><button type="button" data-close="team-dialog">Отмена</button><button type="submit" class="accent" data-write>Сохранить команду</button></footer>
|
<div class="mb-3"><label for="service-policy" class="form-label">При медленной реплике</label><select id="service-policy" class="input-modern w-100"><option value="isolate">Отключить реплику, сохранить скорость primary</option><option value="strict">Ждать реплику, замедлить общий поток</option></select></div>
|
||||||
</form>
|
<p id="settings-error" class="form-error" role="alert"></p>
|
||||||
</dialog>
|
</div>
|
||||||
<dialog id="delete-dialog" aria-labelledby="delete-title">
|
<div class="modal-footer"><button type="button" class="add-button cancel-button" data-close="settingsModal">Отмена</button><button type="submit" class="add-button">Сохранить</button></div>
|
||||||
<form id="delete-form">
|
</form>
|
||||||
<h2 id="delete-title">Удалить сервис?</h2><p id="delete-detail"></p>
|
</div></div>
|
||||||
<p class="muted">Настройки сервиса будут удалены. Установленные соединения завершатся в режиме дренирования.</p>
|
|
||||||
<p id="delete-error" class="error-text" role="alert" hidden></p>
|
|
||||||
<footer class="dialog-actions"><button type="button" data-close="delete-dialog" autofocus>Отмена</button><button type="submit" class="danger" data-write>Удалить сервис</button></footer>
|
|
||||||
</form>
|
|
||||||
</dialog>
|
</dialog>
|
||||||
</body>
|
</body>
|
||||||
</html>
|
</html>
|
||||||
|
|||||||
66
web/vendor/NOTICES.txt
vendored
Normal file
66
web/vendor/NOTICES.txt
vendored
Normal file
@ -0,0 +1,66 @@
|
|||||||
|
Bundled frontend dependencies
|
||||||
|
|
||||||
|
Bootstrap 5.1.3
|
||||||
|
https://github.com/twbs/bootstrap/tree/v5.1.3
|
||||||
|
|
||||||
|
The MIT License (MIT)
|
||||||
|
|
||||||
|
Copyright (c) 2011-2021 Twitter, Inc.
|
||||||
|
Copyright (c) 2011-2021 The Bootstrap Authors
|
||||||
|
|
||||||
|
Permission is hereby granted, free of charge, to any person obtaining a copy
|
||||||
|
of this software and associated documentation files (the "Software"), to deal
|
||||||
|
in the Software without restriction, including without limitation the rights
|
||||||
|
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
||||||
|
copies of the Software, and to permit persons to whom the Software is
|
||||||
|
furnished to do so, subject to the following conditions:
|
||||||
|
|
||||||
|
The above copyright notice and this permission notice shall be included in
|
||||||
|
all copies or substantial portions of the Software.
|
||||||
|
|
||||||
|
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
||||||
|
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
||||||
|
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
||||||
|
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
||||||
|
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
||||||
|
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
|
||||||
|
THE SOFTWARE.
|
||||||
|
|
||||||
|
|
||||||
|
Font Awesome Free 6.0.0
|
||||||
|
https://github.com/FortAwesome/Font-Awesome/tree/6.0.0
|
||||||
|
|
||||||
|
Font Awesome Free License
|
||||||
|
-------------------------
|
||||||
|
|
||||||
|
Font Awesome Free is free, open source, and GPL friendly. You can use it for
|
||||||
|
commercial projects, open source projects, or really almost whatever you want.
|
||||||
|
Full Font Awesome Free license: https://fontawesome.com/license/free.
|
||||||
|
|
||||||
|
# Icons: CC BY 4.0 License (https://creativecommons.org/licenses/by/4.0/)
|
||||||
|
In the Font Awesome Free download, the CC BY 4.0 license applies to all icons
|
||||||
|
packaged as SVG and JS file types.
|
||||||
|
|
||||||
|
# Fonts: SIL OFL 1.1 License (https://scripts.sil.org/OFL)
|
||||||
|
In the Font Awesome Free download, the SIL OFL license applies to all icons
|
||||||
|
packaged as web and desktop font files.
|
||||||
|
|
||||||
|
# Code: MIT License (https://opensource.org/licenses/MIT)
|
||||||
|
In the Font Awesome Free download, the MIT license applies to all non-font and
|
||||||
|
non-icon files.
|
||||||
|
|
||||||
|
# Attribution
|
||||||
|
Attribution is required by MIT, SIL OFL, and CC BY licenses. Downloaded Font
|
||||||
|
Awesome Free files already contain embedded comments with sufficient
|
||||||
|
attribution, so you shouldn't need to do anything additional when using these
|
||||||
|
files normally.
|
||||||
|
|
||||||
|
We've kept attribution comments terse, so we ask that you do not actively work
|
||||||
|
to remove them from files, especially code. They're a great way for folks to
|
||||||
|
learn about Font Awesome.
|
||||||
|
|
||||||
|
# Brand Icons
|
||||||
|
All brand icons are trademarks of their respective owners. The use of these
|
||||||
|
trademarks does not indicate endorsement of the trademark holder by Font
|
||||||
|
Awesome, nor vice versa. **Please do not use brand logos for any purpose except
|
||||||
|
to represent the company, product, or service to which they refer.**
|
||||||
22
web/vendor/bootstrap-LICENSE.txt
vendored
Normal file
22
web/vendor/bootstrap-LICENSE.txt
vendored
Normal file
@ -0,0 +1,22 @@
|
|||||||
|
The MIT License (MIT)
|
||||||
|
|
||||||
|
Copyright (c) 2011-2021 Twitter, Inc.
|
||||||
|
Copyright (c) 2011-2021 The Bootstrap Authors
|
||||||
|
|
||||||
|
Permission is hereby granted, free of charge, to any person obtaining a copy
|
||||||
|
of this software and associated documentation files (the "Software"), to deal
|
||||||
|
in the Software without restriction, including without limitation the rights
|
||||||
|
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
||||||
|
copies of the Software, and to permit persons to whom the Software is
|
||||||
|
furnished to do so, subject to the following conditions:
|
||||||
|
|
||||||
|
The above copyright notice and this permission notice shall be included in
|
||||||
|
all copies or substantial portions of the Software.
|
||||||
|
|
||||||
|
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
||||||
|
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
||||||
|
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
||||||
|
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
||||||
|
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
||||||
|
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
|
||||||
|
THE SOFTWARE.
|
||||||
7
web/vendor/bootstrap.min.css
vendored
Normal file
7
web/vendor/bootstrap.min.css
vendored
Normal file
File diff suppressed because one or more lines are too long
BIN
web/vendor/fa-solid-900.woff2
vendored
Normal file
BIN
web/vendor/fa-solid-900.woff2
vendored
Normal file
Binary file not shown.
34
web/vendor/fontawesome-LICENSE.txt
vendored
Normal file
34
web/vendor/fontawesome-LICENSE.txt
vendored
Normal file
@ -0,0 +1,34 @@
|
|||||||
|
Font Awesome Free License
|
||||||
|
-------------------------
|
||||||
|
|
||||||
|
Font Awesome Free is free, open source, and GPL friendly. You can use it for
|
||||||
|
commercial projects, open source projects, or really almost whatever you want.
|
||||||
|
Full Font Awesome Free license: https://fontawesome.com/license/free.
|
||||||
|
|
||||||
|
# Icons: CC BY 4.0 License (https://creativecommons.org/licenses/by/4.0/)
|
||||||
|
In the Font Awesome Free download, the CC BY 4.0 license applies to all icons
|
||||||
|
packaged as SVG and JS file types.
|
||||||
|
|
||||||
|
# Fonts: SIL OFL 1.1 License (https://scripts.sil.org/OFL)
|
||||||
|
In the Font Awesome Free download, the SIL OFL license applies to all icons
|
||||||
|
packaged as web and desktop font files.
|
||||||
|
|
||||||
|
# Code: MIT License (https://opensource.org/licenses/MIT)
|
||||||
|
In the Font Awesome Free download, the MIT license applies to all non-font and
|
||||||
|
non-icon files.
|
||||||
|
|
||||||
|
# Attribution
|
||||||
|
Attribution is required by MIT, SIL OFL, and CC BY licenses. Downloaded Font
|
||||||
|
Awesome Free files already contain embedded comments with sufficient
|
||||||
|
attribution, so you shouldn't need to do anything additional when using these
|
||||||
|
files normally.
|
||||||
|
|
||||||
|
We've kept attribution comments terse, so we ask that you do not actively work
|
||||||
|
to remove them from files, especially code. They're a great way for folks to
|
||||||
|
learn about Font Awesome.
|
||||||
|
|
||||||
|
# Brand Icons
|
||||||
|
All brand icons are trademarks of their respective owners. The use of these
|
||||||
|
trademarks does not indicate endorsement of the trademark holder by Font
|
||||||
|
Awesome, nor vice versa. **Please do not use brand logos for any purpose except
|
||||||
|
to represent the company, product, or service to which they refer.**
|
||||||
6
web/vendor/fontawesome.min.css
vendored
Normal file
6
web/vendor/fontawesome.min.css
vendored
Normal file
File diff suppressed because one or more lines are too long
Loading…
Reference in New Issue
Block a user