ad/firegex
1
0
Fork 0
mirror of https://github.com/Av0idD4rk/firegex.git synced 2026-10-23 16:31:00 +00:00
Code Issues Packages Projects Releases Wiki Activity

Merge pull request #43 from Pwnzer0tt1/dev

Browse Source 
Releasing Firegex 5.0.0 with new UI interface for filters with unified services model, proxy tunnels, QUIC, HTTP/2 HTTP/3 support
This commit is contained in:
Domingo Dirutigliano 2026-10-16 18:07:27 +02:00 committed by GitHub
commit 9e74ad2f61
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
328 changed files with 68065 additions and 13137 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

6
.dockerignore
View File

@ -20,7 +20,11 @@ Dockerfile
/frontend/build/**
/frontend/node_modules/
/backend/modules/cppregex
/backend/modules/proxy
/backend/modules/cpproxy
/backend/modules/fgex-proxy
# Built inside the compiler stage; a host build tree would only be copied in to be
# overwritten, and it is larger than the rest of the context put together.
/backend/proxysrc/target/
compose.yml
.firegex-compose.yml

2
.github/workflows/demo-pages.yml vendored
View File

@ -41,7 +41,7 @@ jobs:
working-directory: frontend/dist
run: |
# GitHub Pages serves 404.html for any path it has no file for. Handing it the
# SPA entrypoint is what makes deep links like /nfproxy/<id> resolve.
# SPA entrypoint is what makes deep links like /services/<id> resolve.
cp index.html 404.html
echo 'demo.firegex.pwnzer0tt1.it' > CNAME

167
.github/workflows/release.yml vendored
View File

@ -24,11 +24,42 @@ jobs:
- name: Build and run firegex
run: python3 run.py start -P testpassword
# No apt-get: the tests' dependencies are pure python. `iperf3`, matplotlib and
# numpy belong to the benchmarks, which CI does not run — a measurement has no
# pass or fail to gate a release on.
#
# As root, because what needs root **skips** otherwise — and a case that always
# skips is not a case this gate is checking. Through the script rather than `sudo
# pytest`, because the script installs the dependencies into the environment it is
# running in; a root process does not read the invoking user's site-packages.
- name: Run tests
run: sudo apt-get install -y iperf3 && cd tests && ./run_tests.sh
run: cd tests && sudo ./run_tests.sh
# The proxy engine's own suite: HTTP/2, HTTP/3 and QUIC termination, the Python worker's
# framing and deadlines, the UDP relay — much of it reachable from no integration test,
# and none of it was run before a release. In the image stage built for it, which is
# the compiler plus the filter library the worker tests start. `backend/` is mounted
# rather than baked in because the worker tests launch `../modules/services/pyworker.py`.
engine_tests:
runs-on: ${{ matrix.os }}
strategy:
matrix:
os: [ubuntu-latest, ubuntu-24.04-arm]
steps:
- name: Checkout repository
uses: actions/checkout@v4
- name: Build the engine's test image
run: docker build --target enginetest -t firegex-enginetest .
- name: Run the engine's tests
run: >-
docker run --rm -v "$PWD/backend:/execute" -w /execute/proxysrc
firegex-enginetest cargo test
docker_build:
needs: run_tests
needs: [run_tests, engine_tests]
runs-on: ${{ matrix.os }}
strategy:
matrix:
@ -217,8 +248,121 @@ jobs:
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
# The README's picture of the interface, retaken for every release so it never shows a
# version that no longer exists. From the demo build served locally rather than from the
# published demo: that deploys on its own schedule, and could still be the old one.
update_readme_screenshot:
needs: [run_tests, engine_tests]
runs-on: ubuntu-latest
permissions:
contents: write
steps:
- name: Checkout repository
uses: actions/checkout@v4
with:
ref: main
- name: Set up bun
uses: oven-sh/setup-bun@v2
with:
bun-version: latest
- name: Set up Python
uses: actions/setup-python@v5
with:
python-version: '3.12'
- name: Install Playwright
run: |
python -m pip install --upgrade pip
pip install playwright
playwright install --with-deps chromium
- name: Build the demo
working-directory: frontend
run: bun install --frozen-lockfile && VITE_VERSION="${{ github.ref_name }}" bun run build:demo
- name: Screenshot it
working-directory: frontend
run: |
bunx vite preview --port 4173 --strictPort &
timeout 60 bash -c 'until curl -sf http://localhost:4173/ > /dev/null; do sleep 0.5; done'
python3 ../docs/screenshot_demo.py http://localhost:4173/
- name: Commit the new screenshot
uses: stefanzweifel/git-auto-commit-action@v5
with:
commit_message: "Update README screenshot for ${{ github.ref_name }}"
file_pattern: 'docs/Firegex_Screenshot.png'
branch: main
# `firegex` carries a C extension — the llhttp binding in `fgex-lib/llhttp/` — so it
# is no longer one universal wheel. A wheel is built per platform and per interpreter
# here, and the sdist goes up beside them: where no wheel matches, pip compiles from
# that. Compiling is a thing pip knows how to do; assuming a system library is already
# installed is not, which is why the parser ships as source rather than as a `ctypes`
# binding against the distribution's `libllhttp`.
build_firegex_wheels:
needs: [run_tests, engine_tests]
runs-on: ${{ matrix.os }}
strategy:
fail-fast: false
matrix:
include:
- os: ubuntu-latest # manylinux + musllinux, x86_64
- os: ubuntu-24.04-arm # and the same for aarch64
# arm64 and x86_64 both, from one runner: the Intel macOS image this used to ask
# for (`macos-13`) has been retired, and a matrix entry naming a runner that no
# longer exists fails — which, with the publish job needing this one, blocked
# every PyPI release. The x86_64 wheel is cross-compiled; see CIBW_TEST_SKIP.
- os: macos-latest
- os: windows-latest
steps:
- uses: actions/checkout@v4
- name: Extract tag name
id: tag
shell: bash
run: echo TAG_NAME=$(echo $GITHUB_REF | cut -d / -f 3) >> $GITHUB_OUTPUT
- name: Update version
shell: bash
run: >-
sed -i.bak "s/{{VERSION_PLACEHOLDER}}/${{ steps.tag.outputs.TAG_NAME }}/g" fgex-lib/setup.py;
sed -i.bak "s/{{VERSION_PLACEHOLDER}}/${{ steps.tag.outputs.TAG_NAME }}/g" fgex-lib/firegex/__init__.py;
- name: Build wheels
# 3.x, because 2.x predates CPython 3.14 and quietly built nothing for it: the
# selector below matched no identifier it knew, which is not an error.
uses: pypa/cibuildwheel@v3.2.0
with:
package-dir: fgex-lib
output-dir: wheelhouse
env:
# The library declares `python_requires>=3.10`, so build what it claims to
# support and nothing else. PyPy is excluded: the datapath embeds CPython.
# 3.10 and 3.11 build because the binding declares its 3.12-only module slot
# only where it exists (`llhttp_module.c`) — unguarded, it failed to compile
# there, and the whole job with it.
CIBW_BUILD: "cp310-* cp311-* cp312-* cp313-* cp314-*"
CIBW_SKIP: "*-musllinux_i686 *_i686 pp*"
CIBW_ARCHS_MACOS: "arm64 x86_64"
# An x86_64 wheel built on an arm64 runner cannot be imported there without
# Rosetta, which the runner is not promised to have; the arm64 wheels, built
# from the same sources in the same job, are what the smoke test checks.
CIBW_TEST_SKIP: "*-macosx_x86_64"
# A wheel that imports is the only kind worth publishing, and the extension is
# the whole reason this job exists. The checks live in a file rather than in a
# `-c` string because the one that matters needs a `try`: the parser has to
# raise its *own* exception on malformed traffic, and this clean environment —
# no `pyllhttp`, no developer's leftovers — is the only place that is tested.
CIBW_TEST_COMMAND: python {package}/wheel_smoke.py
- uses: actions/upload-artifact@v4
with:
name: wheels-${{ matrix.os }}
path: wheelhouse/*.whl
pypi_publish_firegex:
needs: run_tests
needs: build_firegex_wheels
runs-on: ubuntu-latest
permissions:
contents: read
@ -240,16 +384,27 @@ jobs:
run: >-
sed -i "s/{{VERSION_PLACEHOLDER}}/${{ steps.tag.outputs.TAG_NAME }}/g" fgex-lib/setup.py;
sed -i "s/{{VERSION_PLACEHOLDER}}/${{ steps.tag.outputs.TAG_NAME }}/g" fgex-lib/firegex/__init__.py;
- name: Build package
run: cd fgex-lib && python -m build && mv ./dist ../
# The sdist is what pip falls back to where no wheel matches, so it has to carry
# the parser's C sources — `MANIFEST.in` is what puts them in.
- name: Build the sdist
run: cd fgex-lib && python -m build --sdist && mv ./dist ../
- name: Collect the wheels built for every platform
uses: actions/download-artifact@v4
with:
pattern: wheels-*
merge-multiple: true
path: dist
- name: Publish package
uses: pypa/gh-action-pypi-publish@27b31702a0e7fc50959f5ad993c78deac1bdfc29
with:
user: __token__
password: ${{ secrets.PYPI_API_TOKEN }}
# After `firegex`, never beside it: this package is nothing but `firegex==<tag>`, so
# published while a wheel job failed it is an alias for a version that does not exist,
# and `pip install fgex` fails for everybody until somebody notices.
pypi_publish_fgex_alias:
needs: run_tests
needs: pypi_publish_firegex
runs-on: ubuntu-latest
permissions:
contents: read

22
.gitignore vendored
View File

@ -26,7 +26,7 @@
/backend/binsrc/cppregex
/backend/modules/cpproxy
/backend/binsrc/cpproxy
/backend/modules/proxy
/backend/modules/fgex-proxy
/docker-compose.yml
/firegex-compose.yml
/.firegex-compose.yml
@ -38,8 +38,13 @@
/firegexfs
/firegex.py
/tests/benchmark.csv
/tests/comparemark.csv
# A benchmark run's own output. `bench/results/` holds the ones that were kept on
# purpose, so only the default filename a run drops beside the script is ignored —
# the three paths here before named scripts that moved into bench/ or were deleted.
/tests/bench/benchmark.csv
/tests/bench/*.csv
!/tests/bench/results/*.csv
# misc
**/.DS_Store
**/.env.local
@ -51,3 +56,14 @@
**/yarn-debug.log*
**/yarn-error.log*
.claude
# Rust proxy engine build artifacts
/backend/proxysrc/target/
# pytest
**/.pytest_cache/
# The vendored HTTP parser, built in place by an editable install.
/fgex-lib/firegex/_llhttp*.so
/fgex-lib/firegex/_llhttp*.pyd
/fgex-lib/build/

469
AGENTS.md Normal file
View File

File diff suppressed because one or more lines are too long

103
CLAUDE.md
View File

@ -1,103 +0,0 @@
# CLAUDE.md
This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
## What this is
Firegex is a firewall built for CTF Attack-Defense competitions: it sits in front of a vulnerable service and blocks/mangles malicious traffic before it reaches it. It ships several independent filtering modules (regex-based, Python-filter-based, port hijacking, plain nftables rules, TLS decrypt-and-reinspect) behind a single FastAPI backend and React frontend.
## Commands
### Running the whole app (Docker or standalone)
```bash
python3 run.py # build+start (Docker if available, else standalone chroot on Linux)
python3 run.py --standalone # force standalone mode (no Docker)
python3 run.py stop [--clear] # stop (optionally wipe the DB volume/rootfs)
python3 run.py restart # recreate the container WITHOUT rebuilding (stale image if source changed)
python3 run.py status
python3 run.py config --password [newpass] # change the running instance's password; omit the value to be prompted interactively
python3 run.py config --show # print the persisted settings (port, host, socket dir, auth)
python3 run.py start --unsafe-disable-auth # no authentication at all — see "Authentication" below
```
`run.py` has zero external dependencies (stdlib only) by design. `restart` runs `docker compose up -d --force-recreate`, so it re-reads the generated compose file and re-applies env vars (`ALLOWED_IPS`, `PROXY_IP_HEADER`, `UNSAFE_DISABLE_AUTH`) — a plain `docker compose restart` would keep the old container environment and silently ignore those. It still does **not** rebuild the image; after a source change use `stop` then the bare start command to force `docker compose up -d --build`.
Most `start`/`restart` flags are persisted to `.firegex-conf.json` and become the default for later runs, so a bare `python3 run.py start` inherits whatever was last passed.
### Frontend (`frontend/`)
```bash
bun install # bun.lock is canonical
bun run dev # vite dev server
bun run build # tsc && vite build
npx tsc --noEmit -p . # typecheck only (no separate lint script is configured)
```
### Backend (`backend/`)
Needs Linux with `NET_ADMIN` (nftables/nfqueue) and root, so it's normally only run via `run.py` (Docker or standalone). Direct iteration on a Linux box/VM:
```bash
cd backend && DEBUG=1 python3 app.py DEBUG
```
### Tests (`tests/`)
No pytest — plain scripts driven against a **live, already-running** firegex instance (`http://127.0.0.1:4444` by default), root/Linux required.
```bash
cd tests
./run_tests.sh [password] # full suite (defaults to "testpassword")
python3 nfregex_test.py -p <pw> -m tcp [--ipv6] [--tls]
python3 nfproxy_test.py -p <pw> [--ipv6] [--tls]
python3 ph_test.py -p <pw> -m tcp|udp [--ipv6]
python3 tls_test.py -p <pw> [--ipv6]
python3 api_test.py -p <pw>
python3 ipfilter_test.py # access control (--allowed-ips/--proxy-ip-header/--unsafe-disable-auth)
```
`ipfilter_test.py` is the odd one out: those settings are only read at process startup, so it drives `run.py stop`/`start` itself for each scenario and restores an unrestricted instance at the end. It is **not** part of `run_tests.sh` — run it on its own, and expect it to bounce the instance.
CI (`.github/workflows/release.yml`) runs **only on `release: published`**, not on pushes or pull requests — nothing validates a PR automatically. Its `run_tests` job (`ubuntu-latest` + `ubuntu-24.04-arm`) does `python3 run.py start -P testpassword` then `cd tests && ./run_tests.sh`, and gates every publish job below it.
## Architecture
### Three tiers
React (Vite + Mantine + react-router + TanStack Query + socket.io-client) → FastAPI backend → per-module C++ binaries that do the actual packet interception via nftables/NFQUEUE. The backend mostly orchestrates: SQLite-backed config, spawning/talking to the C++ processes over unix sockets, and deploying the built React app.
### Backend router auto-discovery (`backend/utils/loader.py`)
Every `.py` file in `backend/routers/` is auto-imported and mounted at `/api/<filename>` if it exposes a module-level `app` (a `fastapi.APIRouter`). Optional module-level `startup()`, `shutdown()`, `reset(params)` async functions are collected and called by the app's lifespan/`/api/reset` handlers. **Adding a new module = adding a new file here** — nothing else needs to register it.
### Per-module pattern
Each filtering module (`nfproxy`, `nfregex`, `porthijack`, `firewall`, `tls`) follows the same shape:
- `backend/routers/<mod>.py` — FastAPI endpoints + its own SQLite DB, via `utils.sqlite.SQLite`.
- `backend/modules/<mod>/models.py` — plain `Service`/filter model, `from_dict`.
- `backend/modules/<mod>/firewall.py` — `FirewallManager`/`ServiceManager` pair: `FirewallManager` tracks one `ServiceManager` per DB row (in-memory), `ServiceManager.next(status)` starts/stops the interceptor/nft rules for that one service.
- `backend/modules/<mod>/nftables.py` — builds/tears down the module's nftables rules; the manager class extends `utils.NFTableManager` (a `Singleton`), so there's exactly one nft-rule-set owner per module.
### nfproxy vs nfregex
Both intercept via NFQUEUE, but differently:
- **nfregex** (`backend/binsrc/nfregex.cpp` → `cppregex`): pure C++, matches PCRE2/hyperscan regexes against the stream entirely in the binary. Faster, but rules are just regex+mode.
- **nfproxy** (`backend/binsrc/nfproxy.cpp`, `binsrc/pyproxy/*`, embeds libpython → `cpproxy`): user-supplied Python filter code (via the `firegex` pip package, `fgex-lib/`) runs inside the C++ process through the embedded interpreter, talking back to the backend over a unix socket for stats/exceptions. Much more flexible, slower than nfregex.
### TLS Decrypt bridge (`backend/modules/tls/`)
A `tls_streams` DB row describes one public `ip:port` to decrypt. `ssl_port`/`clear_port` are two loopback ports **deterministically derived from a hash of `ip:port`** (`get_tls_ports()`), not stored arbitrarily. nginx (config generated by `modules/tls/nginx.py`) does the actual TLS work per active stream:
1. `listen loopback:ssl_port ssl` → terminates the public TLS connection, forwards **plaintext** to `loopback:clear_port`.
2. `listen loopback:clear_port` → re-encrypts (`proxy_ssl on`) and forwards to the real `ip:port`. The real backend is expected to also speak TLS (mirrors a CTF challenge service that natively does TLS) — this is not a generic "terminate TLS in front of a plaintext service" reverse proxy.
The only unencrypted hop is the loopback leg between `ssl_port` and `clear_port` — an nfproxy/nfregex service must attach there (`target_type="tls"`, `tls_stream_id=...`) to see decrypted content. Both modules' `nftables.py` has a `resolve_target(srv)` helper that swaps in `(loopback_ip, clear_port)` for `target_type="tls"` services instead of the service's own `ip_int`/`port` (which mirror the stream's real destination for display only). Stopping a TLS stream cascades to stop dependent filter services (and vice versa on start); deleting a stream still referenced by a service is rejected. This cross-module logic lives in `routers/tls.py`, which imports `routers.nfproxy`/`routers.nfregex` directly (one-directional — those modules only import `modules.tls.*`, never `routers.tls`).
### Authentication (`backend/app.py`)
Every module router is mounted onto `api = APIRouter(prefix="/api", dependencies=[Depends(is_loggined)])`, so a new file in `routers/` is authenticated by construction. The whole auth surface is four places: that dependency, the Socket.IO `connect` handler (reads the token out of `auth`), and the three routes declared on `app` directly rather than on `api` — `/api/status`, `/api/login`, `/api/set-password`. Adding an unauthenticated route means declaring it on `app`, which should stay a deliberate, rare act.
`--unsafe-disable-auth` (env `UNSAFE_DISABLE_AUTH`) hands access control to a reverse proxy: `check_login` returns `True`, the socket.io handler skips its check, and `APP_STATUS()` reports `run` even with no password stored. The password endpoints (`/api/login`, `/api/set-password`, `/api/change-password`) answer `403` in this mode on purpose — otherwise an anonymous caller could plant a credential that keeps working once authentication is turned back on. `run.py config --password` still works, writing the hash straight into the DB from the host, and an explicit `-P` at first start is kept so that re-enabling authentication lands on a known password instead of the open initial-setup state.
Independently of that, `IPFilterMiddleware` restricts access by CIDR (`--allowed-ips`, optionally reading the client IP from `--proxy-ip-header`). It **fails closed**: a missing, unparseable, or non-matching IP is denied. Note the header variant trusts a client-supplied value, so it only holds if the proxy overwrites it.
### SQLite wrapper (`backend/utils/sqlite.py`)
Each DB file's schema is versioned by hashing the Python schema dict; on mismatch the **entire file is deleted and recreated** (not migrated) — any schema dict change wipes that DB file on next startup, including `db/firegex.db`'s `keys_values` table (password/JWT secret). `dump()`/`load()` back the export/import backup feature; `dump()` deliberately excludes `password`/`secret` from exported `keys_values` rows, so `/api/import` explicitly re-preserves the current password/secret around the import.
### Real-time frontend updates
One Socket.IO event, `"update"`, whose payload is a list of strings treated as a react-query `queryKey` prefix (e.g. `["nfproxy"]`, `["tls_streams"]`). `App.tsx` does `queryClient.invalidateQueries({ queryKey: data })` — prefix matching invalidates any query whose key starts with that array. Convention: each module's mutating endpoints emit the same tag string that module's frontend queries use as the first `queryKey` element. `staleTime: Infinity` is set globally, so this event is the **only** automatic refresh mechanism — a missing/mismatched tag means silent staleness for other clients/tabs.
### Frontend structure
`pages/<Module>/` (routes) + `components/<Module>/` (ServiceRow, AddEditService, `utils.ts` with query key + API wrapper) per module, mirroring the backend split. Detail drill-down lives at `pages/<Module>/ServiceDetails.tsx` under a nested `:srv` route (see `App.tsx`'s `PageRouting`). Theme tokens are CSS variables in `src/index.css`; Mantine `Card` defaults are set globally in `index.tsx`. Prefer Mantine `Group`/`Stack` with `gap` over the legacy `Box className="center-flex"` utility classes still present in older parts of the codebase.
### `fgex-lib/`
Separate, independently-published PyPI package (`pip install fgex`/`firegex`) — the library users write nfproxy filters against (`@pyfilter`, `RawPacket`/`HttpRequest`/etc.), plus the `fgex` CLI and a local `proxysim` for testing filters without a real firegex instance. Not part of the backend/frontend build. `fgex-lib/README.md` is a short index (PyPI-facing) linking out to `docs/*.md`; it isn't rendered anywhere itself.
### Module documentation (`docs/*.md`)
Each module's user-facing docs (`docs/nfproxy.md`, `docs/nfregex.md`, `docs/porthijack.md`, `docs/firewall.md`, `docs/tls.md`) are single markdown sources, each imported directly with Vite's `?raw` suffix by a matching `<Module>Docs.tsx` component (e.g. `components/NFProxy/NFProxyDocs.tsx`) and rendered through the shared `components/MarkdownDocs.tsx` (React Markdown + `remark-gfm`, code fences routed through `CodeHighlight`, relative `.md` cross-links rewritten to the equivalent GitHub blob URL since they'd otherwise resolve against the app's own origin). `DocsButton.tsx`'s `DocType` enum wires each module to its component. Editing a module's docs means editing its one `docs/*.md` file — never the `*Docs.tsx` component directly. Reading these files across the `frontend/`↔repo-root boundary requires `vite.config.ts`'s `server.fs.allow: ['..']` (dev) and the `Dockerfile` frontend build stage's `COPY ./docs/*.md /docs/` (prod/standalone rootfs) to stay in sync with any path changes.

35
Dockerfile
View File

@ -15,9 +15,15 @@ RUN bun run build
# Base fedora container
FROM --platform=$TARGETARCH quay.io/fedora/fedora:44 AS base
# iproute: the proxy engine needs `ip rule`/`ip route` to bring the return traffic
# home when a service preserves the client's source address, and `ip link` is what puts
# the `firegex0` capture interface there. Nothing else is needed for the capture: the
# engine writes the decrypted traffic onto it itself, through a packet socket
# (`proxysrc/src/capture.rs`). `tc` used to be installed for mirroring it there, and
# nothing has used it since.
RUN dnf -y update && dnf install -y python3.14 libnetfilter_queue \
libnfnetlink libmnl libcap-ng-utils nftables \
vectorscan libtins python3-nftables libpcap nginx nginx-mod-stream && dnf clean all
libnfnetlink libmnl libcap-ng-utils nftables iproute \
vectorscan libtins python3-nftables libpcap && dnf clean all
RUN mkdir -p /execute/modules
WORKDIR /execute
@ -26,25 +32,42 @@ FROM --platform=$TARGETARCH base AS compiler
RUN dnf -y update && dnf install -y python3.14-devel @development-tools gcc-c++ \
libnetfilter_queue-devel libnfnetlink-devel libmnl-devel \
vectorscan-devel libtins-devel libpcap-devel boost-devel
vectorscan-devel libtins-devel libpcap-devel boost-devel cargo
COPY ./backend/binsrc /execute/binsrc
RUN g++ binsrc/nfregex.cpp -o cppregex -std=c++23 -O3 -Wall -lnetfilter_queue -pthread -lnfnetlink $(pkg-config --cflags --libs libtins libhs libmnl)
RUN g++ binsrc/nfproxy.cpp -o cpproxy -std=c++23 -O3 -Wall -lnetfilter_queue -lpython3.14 -pthread -lnfnetlink $(pkg-config --cflags --libs libtins libmnl python3)
RUN g++ binsrc/pyfilter.cpp -o cpproxy -std=c++23 -O3 -Wall -lnetfilter_queue -lpython3.14 -pthread -lnfnetlink $(pkg-config --cflags --libs libtins libmnl python3)
# The proxy datapath engine, built here rather than in a rust image on purpose: it
# links the same libhs the C++ binaries do, and a binary built against one distro's
# vectorscan and run against another's is a class of failure nobody would enjoy
# debugging at a competition.
COPY ./backend/proxysrc /execute/proxysrc
RUN cd /execute/proxysrc && cargo build --release
# Where the engine's own suite runs (`cargo test` in proxysrc). It starts real Python
# workers through the filter library, so it needs the compiler stage *and* that library
# with what it imports — without them the pyworker tests fail with a ModuleNotFoundError
# buried in captured output. Nothing below copies from it, and it sits before `final` so a
# build with no target (what compose does) is not handed this as the image.
FROM --platform=$TARGETARCH compiler AS enginetest
COPY ./fgex-lib /fgex-lib
RUN python3.14 -m pip install --no-cache-dir --break-system-packages /fgex-lib
#Building main conteiner
FROM --platform=$TARGETARCH base AS final
COPY ./backend/requirements.txt /execute/requirements.txt
COPY ./backend/pyproject.toml /execute/pyproject.toml
COPY ./fgex-lib /execute/fgex-lib
RUN dnf -y update && dnf install -y gcc-c++ python3.14-devel uv git &&\
uv pip install --no-cache --system ./fgex-lib &&\
uv pip install --no-cache --system -r /execute/requirements.txt &&\
uv pip install --no-cache --system -r /execute/pyproject.toml &&\
uv cache clean && dnf remove -y gcc-c++ python3.14-devel uv git && dnf clean all
COPY ./backend/ /execute/
COPY --from=compiler /execute/cppregex /execute/cpproxy /execute/modules/
COPY --from=compiler /execute/proxysrc/target/release/fgex-proxy /execute/modules/
COPY --from=frontend /app/dist/ ./frontend/
CMD ["/bin/sh", "/execute/docker-entrypoint.sh"]

39
README.md
View File

@ -84,11 +84,24 @@ This is especially useful if you lost the current password: it's applied directl
## Functionalities
- **[Netfilter Regex](docs/nfregex.md)**: filtering using [NFQUEUE](https://netfilter.org/projects/libnetfilter_queue/) with [nftables](https://netfilter.org/projects/nftables/) uses a c++ file that handle the regexes and the requests, blocking the malicius requests. PCRE2 regexes are used. The requests are intercepted kernel side, so this filter works immediatly (IPv4/6 and TCP/UDP supported)
- **[Services](docs/services.md)**: the main module. A service is one protected endpoint described in two independent parts — a **network layer** that says how its traffic is intercepted, and an ordered chain of **filters** that says what happens to it.
- A service protects a **list of addresses or network interfaces**, mixing IPv4, IPv6 and network interfaces (`eth0`, `wg0`, etc.) freely: intercept traffic on specific IPs or directly across an entire interface. Adding an address or interface to a running service drops no connections.
- Network layer: **NFQUEUE** (packets lifted to userspace with [nfqueue](https://netfilter.org/projects/libnetfilter_queue/) and [nftables](https://netfilter.org/projects/nftables/), nothing terminated, the kernel keeps forwarding if a filter dies) or **proxy** (the connection is terminated and reopened, which buys kernel reassembly, an unlimited ordered chain of filters, TLS and QUIC termination and real backpressure — while still dialling your service from the client's own address, so it never stops seeing who is talking to it).
- Filters: **regex**, matched by [hyperscan](https://github.com/VectorCamp/vectorscan) — the same engine on either layer, so a pattern means the same thing wherever it runs — or **Python**, your own code written against the [`firegex`](https://pypi.org/project/firegex/) library, with built-in models for HTTP and TCP streams. You never declare which protocol a Python filter speaks: asking for an `HttpRequest` is what makes it an HTTP filter, and a filter reads the addresses and ports as metadata while the payload is the only thing it can change.
- A regex **blocks** the connection, and that is deliberately all it does: rewriting had to scan one chunk at a time, so a pattern split across two TCP segments went through unredacted with nothing to say it had — a rule you trust and that quietly does not apply is worse than no rule. To change bytes rather than refuse them, use a Python filter, which sees the reassembled stream.
- **TLS** is one of the protocols a service can speak, not a separate thing to create: the proxy engine terminates the connection itself, the filters inspect the plaintext, and it is re-encrypted towards your service — with the client's ALPN mirrored rather than invented, and the decrypted stream written to a `firegex0` interface you can point a capture at. A TLS service occupies no extra port.
- Or **your own proxy** as the network layer: firegex rewrites the destination on the way in and changes it back on the way out, so a proxy you wrote yourself sits in the path invisibly. The escape hatch for a protocol nothing built in understands.
- **QUIC**, terminated: it is the fourth protocol a service can speak, and the only one with no alternative — past its first packet QUIC encrypts the frames and the stream boundaries as well as the payload, so nothing that merely forwards can be shown to a filter. Each stream inside reaches the chain as its own connection, and **HTTP/3 is rendered as the HTTP/1.1 it would have been**, so an `HttpRequest` filter or a hyperscan pattern written for a TCP service means exactly the same thing on QUIC without being rewritten.
- **HTTP/2 is rendered the same way**, and with it **gRPC**. HTTP/2 puts the method, the path and the headers in an HPACK-compressed frame, so a firewall that only forwarded it showed a filter a compression format: patterns matched nothing and an `HttpRequest` filter was never called, silently. The engine terminates it now — under TLS, or in the clear from the connection preface — so one pattern and one Python file cover HTTP/1.1, HTTP/2 and HTTP/3.
- **HTTP** is a protocol a service can speak in its own right, meaning *every* version at once: its TCP addresses carry HTTP/1.1 and HTTP/2, encrypted or not as each client chooses, and its UDP addresses carry HTTP/3 — one service, one chain, one certificate, instead of two or three services with the same filters copied between them by hand.
- And a Python filter can read **gRPC messages** rather than the framing they arrive in: one message at a time, as each completes, so a streaming or bidirectional call is filtered while it is still open rather than logged after it ends.
- **TCP or UDP**, on every layer — with full source IP transparency on both. On the proxy layer, UDP datagrams are relayed with one dedicated socket per address using transparent IP spoofing (`IP_TRANSPARENT`), with per-flow filter state and zero-downtime dynamic address additions. NFQUEUE filters packets in place with kernel-level fail-open.
- **The two inspecting layers are opposite trades, [compared side by side in the interface itself](docs/services.md#network-layers) while you choose**: the proxy owns both halves of a connection, so the kernel reassembles for it and a Python filter can rewrite a payload exactly, at the cost of rebuilding fail-open by hand; NFQUEUE is per packet and keeps forwarding by kernel guarantee if a filter dies, at the cost of userspace reassembly and a process per filter.
- **Charts of what each filter has refused** — per filter, per pattern and per `@pyfilter` function, each with **its share of the blocking**. Pick the window (last 15 minutes, hour, 6h, 24h, everything kept, or two exact instants) and the shape (bars, lines, stacked area, or share); **everything on the page counts the same window**, so the chart and the table beside it can never tell different stories. Alongside it, how much traffic arrived, in the unit each layer can honestly report.
- **The Python editor knows the library**: completion and hover for every model and its members — marking which of them you may write to — and the file is **checked as you type by the process that will run it**, so a mistake is flagged on its own line with the reason instead of surfacing later as "worker exited". The hints are introspected from the installed library, so they cannot drift from it.
- A **live log** per service shows what it is doing as it does it: what refused each connection (by name and pattern, not an opaque id), whatever your Python prints, and what the datapath says about its own health.
- A **pattern tester** is built in, and it runs the very engine that will enforce the answer — so a pattern it accepts is one you can save, and one it rejects tells you why in the engine's own words.
- **[Firewall Rules](docs/firewall.md)**: create basic firewall rules to allow and deny specific traffic, like ufw or iptables but using firegex graphic interface (by using [nftable](https://netfilter.org/projects/nftables/))
- **[Hijack Port to Proxy](docs/porthijack.md)**: redirect the traffic on a specific port to another port. Thanks to this you can start your own proxy, connecting to the real service using the loopback interface. Firegex will be resposable about the routing of the packets using internally [nftables](https://netfilter.org/projects/nftables/)
- **[Netfilter Proxy](docs/nfproxy.md)**: uses [nfqueue](https://netfilter.org/projects/libnetfilter_queue/) to simulate a python proxy, you can write your own filter in python and use it to filter the traffic. There are built-in some data handler to parse protocols like HTTP, and before apply the filter you can test it with fgex command (you need to install firegex lib from pypi).
- **[TLS Decryption](docs/tls.md)**: a decrypt-and-reinspect bridge for services that speak TLS natively. It transparently terminates the public TLS connection, exposes the decrypted traffic on a loopback port so a Netfilter Regex/Proxy service can inspect it, then re-encrypts before forwarding to the real backend.
Firegex can also be restricted to accept connections only from a set of trusted CIDR ranges (`--allowed-ips`, optionally combined with `--proxy-ip-header` when running behind a reverse proxy) — see `python3 run.py start -h`.
@ -102,11 +115,17 @@ python3 run.py start --host 127.0.0.1 --unsafe-disable-auth -P 'a-password-for-l
This disables Firegex's password, JWT, and Socket.IO authentication: **every request that reaches Firegex is a full administrator**, so bind it to loopback, a Unix socket, or otherwise make sure nothing but the proxy can reach it. Requests forwarded by the proxy, including headers such as `X-Forwarded-For`, reach Firegex normally. Use `--no-unsafe-disable-auth` to turn the built-in authentication back on.
It can also be switched **while Firegex is running**, from either end:
- **From the interface**, under the menu's *Firewall Access* section: *Turn Authentication Off* takes effect on the next request. The browser that did it keeps its session, so the same menu offers *Turn Authentication On* to undo it. A browser that arrives afterwards does not get that option — while authentication is off, anyone who can reach Firegex could otherwise set their own password and keep you out for good.
- **From the host**, with `python3 run.py config --unsafe-disable-auth` / `--no-unsafe-disable-auth`. This is the one that sticks: it writes to the running instance *and* persists the choice for the next start. A change made from the interface lasts only until Firegex restarts, since the start-up flag is what it comes up with. `config --password` re-enables it on its own — see below.
A few things worth knowing before using it:
- **The setting is persisted** in `.firegex-conf.json`, like `--port` or `--allowed-ips`: a later plain `python3 run.py start` keeps authentication disabled until you pass `--no-unsafe-disable-auth`. Every start prints a banner while it is active, and `python3 run.py config --show` reports it.
- **Set a password anyway** (`-P`, or later `python3 run.py config --password`). It is unused while authentication is disabled, but it is what you fall back on if you re-enable it. Without one, `--no-unsafe-disable-auth` brings Firegex up in its initial-setup state, where anyone who can reach it chooses the password.
- **The password cannot be changed over the API** while authentication is disabled (`/api/login`, `/api/set-password` and `/api/change-password` answer `403`) — otherwise an anonymous caller could plant a credential that keeps working once authentication is back on. `python3 run.py config --password` still works, since it writes to the database from the host.
- **The setting is persisted** in `.firegex-conf.json`, like `--port` or `--allowed-ips`: a later plain `python3 run.py start` keeps authentication disabled until you pass `--no-unsafe-disable-auth`. Every start prints a banner while it is active, and `python3 run.py config --show` reports it — including when the running instance currently disagrees with it.
- **Set a password anyway** (`-P` at first start). Without one, `--no-unsafe-disable-auth` brings Firegex up in its initial-setup state, where anyone who can reach it chooses the password.
- **`config --password` turns authentication back on**, if it was off — setting a password is asking for one to be asked for, and a password that is stored but never checked is worse than no password because it reads like one. It applies to the running instance and to the next start, both. Pass `--keep-auth-disabled` for the one case where that is not what you want: an instance held open behind a proxy that authenticates, with a password kept ready for the day it is not.
- **The password cannot be changed over the API** while authentication is disabled (`/api/login`, `/api/set-password` and `/api/change-password` answer `403`) — otherwise an anonymous caller could plant a credential that keeps working once authentication is back on. `python3 run.py config --password` still works, since it writes to the database from the host, and it takes effect on the next request rather than on the next restart.
- **`--allowed-ips` is not a substitute for the proxy's access control.** Combined with `--proxy-ip-header` it trusts a client-supplied header, so it only holds up if the proxy overwrites that header and nothing else can reach the port directly.
## Documentation
@ -116,10 +135,10 @@ Each module above has its own markdown guide under [`docs/`](docs/), covering ho
Heres a brief description about the firegex structure:
- [Frontend (React)](frontend/README.md)
- [Backend (FastAPI + C++)](backend/README.md)
- [Netfilter Proxy Python library (`firegex`/`fgex` pip package)](fgex-lib/README.md)
- [Backend (FastAPI + C++ and Rust datapaths)](backend/README.md)
- [Python filter library (`firegex`/`fgex` pip package)](fgex-lib/README.md)
More specific information about how Firegex works, and in particular about the nfproxy module, are available here (in italian only): [https://github.com/domysh/engineering-thesis](https://github.com/domysh/engineering-thesis) (PDF in the release attachments)
More specific information about how Firegex works, and in particular about the Python filter engine (called `nfproxy` at the time), are available here (in italian only): [https://github.com/domysh/engineering-thesis](https://github.com/domysh/engineering-thesis) (PDF in the release attachments)
![Firegex Working Scheme](docs/FiregexInternals.png)

259
backend/app.py
View File

@ -9,16 +9,15 @@ from fastapi import FastAPI, HTTPException, Depends, APIRouter
from fastapi.security import OAuth2PasswordBearer, OAuth2PasswordRequestForm
from jose import jwt
from utils.sqlite import SQLite
from utils import API_VERSION, FIREGEX_PORT, FIREGEX_HOST, FIREGEX_SOCKET, JWT_ALGORITHM, get_interfaces, socketio_emit, DEBUG, SysctlManager, NORELOAD, safe_join
from utils import boot_auth_mode, SAFE_DB_NAME, SAFE_PY_NAME, API_VERSION, FIREGEX_PORT, FIREGEX_HOST, FIREGEX_SOCKET, JWT_ALGORITHM, get_interfaces, socketio_emit, DEBUG, SysctlManager, NORELOAD, safe_join
from utils.loader import frontend_deploy, load_routers
from utils.models import ChangePasswordModel, IpInterface, PasswordChangeForm, PasswordForm, ResetRequest, StatusModel, StatusMessageModel
from utils.models import AuthModeForm, ChangePasswordModel, IpInterface, PasswordChangeForm, PasswordForm, ResetRequest, StatusModel, StatusMessageModel
from contextlib import asynccontextmanager
from fastapi.middleware.cors import CORSMiddleware
import socketio
from socketio.exceptions import ConnectionRefusedError
import hashlib
import hmac
import re
from ipaddress import ip_network, ip_address
# DB init
db = SQLite('db/firegex.db')
@ -39,12 +38,47 @@ async def lifespan(app):
ALLOWED_NETWORKS = [ip_network(ip.strip(), strict=False) for ip in os.getenv("ALLOWED_IPS", "").split(",") if ip.strip()]
PROXY_IP_HEADER = os.getenv("PROXY_IP_HEADER", "")
UNSAFE_DISABLE_AUTH = os.getenv("UNSAFE_DISABLE_AUTH", "").strip().lower() in {"1", "true", "yes", "on"}
# With UNSAFE_DISABLE_AUTH every request is already a full administrator, so the password
# endpoints would let an anonymous caller plant a credential that keeps working once
# authentication is turned back on. The password can only be set out of band, from the
# host, with "python3 run.py config --password".
AUTH_DISABLED_DETAIL = "Firegex authentication is disabled: the password can only be changed from the host with 'python3 run.py config --password'"
#: What the environment asked for at boot. Only the seed: the answer lives in the
#: database from here on, so that turning authentication off — or handing the running
#: instance a password — takes effect on the next request instead of on the next restart.
#: A flag that could only be read at startup meant `run.py config --password` printing
#: "it will take effect immediately" while the process it was talking to had already
#: decided that every caller was an administrator.
UNSAFE_DISABLE_AUTH_ENV = os.getenv("UNSAFE_DISABLE_AUTH", "").strip().lower() in {"1", "true", "yes", "on"}
#: Whether this boot's environment was written just now, from the host's configuration:
#: set by `docker-entrypoint.sh` on the first boot of a container, and by `run.py` for
#: every standalone start. Absent means the container is being started **again** by
#: Docker — after a reboot, a restart of the daemon, a crash under `restart:
#: unless-stopped` — with the environment it was created with, which is older than
#: anything `run.py config` has said since.
FRESH_BOOT = os.getenv("FIREGEX_FRESH_BOOT", "") == "1"
#: What the host last decided about authentication with `run.py config`, kept where a
#: restart can read it. See `seed_auth_mode`.
AUTH_HOST_KEY = "auth_disabled_host"
def auth_disabled() -> bool:
"""Is authentication off right now?
Read per request rather than cached. It is one indexed lookup in a SQLite file the
process already holds open, on a path that is about to do a PBKDF2 or a JWT decode,
and the alternative is a copy in memory that can disagree with the one on disk —
which is the whole bug this replaced.
"""
stored = db.get("auth_disabled")
if stored is None:
return UNSAFE_DISABLE_AUTH_ENV
return stored == "1"
# With authentication off every request is already a full administrator, so the password
# endpoints would let an anonymous caller plant a credential that keeps working once it
# is turned back on. Both are set out of band, from the host, with "python3 run.py
# config --password" and "--no-unsafe-disable-auth".
AUTH_DISABLED_DETAIL = ("Firegex authentication is disabled: the password can only be changed from "
"the host with 'python3 run.py config --password'")
class IPFilterMiddleware:
def __init__(self, app):
@ -122,7 +156,7 @@ utils.socketio = socketio.AsyncServer(
sio_app = socketio.ASGIApp(utils.socketio, socketio_path="/sock/socket.io", other_asgi_app=app)
app.mount("/sock", sio_app)
def APP_STATUS(): return "run" if UNSAFE_DISABLE_AUTH or db.get("password") is not None else "init"
def APP_STATUS(): return "run" if auth_disabled() or db.get("password") is not None else "init"
def JWT_SECRET(): return db.get("secret")
def _hash_psw_sync(psw: str) -> str:
@ -155,21 +189,32 @@ def create_access_token(data: dict):
async def refresh_frontend(additional:list[str]=[]):
await socketio_emit([]+additional)
async def check_login(token: str = Depends(oauth2_scheme)):
if UNSAFE_DISABLE_AUTH:
return True
def token_is_valid(token: str | None) -> bool:
"""Was this token issued by this instance to somebody who logged in?
Separate from `check_login` because one caller needs the question without the
"authentication is off, so yes" that `check_login` answers first: turning
authentication back on has to be asked by somebody who was an administrator *before*
it was turned off, and while it is off nothing new is signed — `/api/login` and
`/api/set-password` both refuse — so a valid token is exactly that proof.
"""
if not token:
return False
try:
payload = jwt.decode(token, JWT_SECRET(), algorithms=[JWT_ALGORITHM])
logged_in: bool = payload.get("logged_in")
except Exception:
return False
return logged_in
return bool(payload.get("logged_in"))
async def check_login(token: str = Depends(oauth2_scheme)):
if auth_disabled():
return True
return token_is_valid(token)
@utils.socketio.on("connect")
async def sio_connect(sid, environ, auth):
if not UNSAFE_DISABLE_AUTH and (not auth or not await check_login(auth.get("token"))):
if not auth_disabled() and (not auth or not await check_login(auth.get("token"))):
raise ConnectionRefusedError("Unauthorized")
utils.sid_list.add(sid)
@ -207,13 +252,13 @@ async def get_app_status(auth: bool = Depends(check_login)):
"status": APP_STATUS(),
"loggined": auth,
"version": API_VERSION,
"auth_disabled": UNSAFE_DISABLE_AUTH
"auth_disabled": auth_disabled()
}
@app.post("/api/login")
async def login_api(form: OAuth2PasswordRequestForm = Depends()):
"""Get a login token to use the firegex api"""
if UNSAFE_DISABLE_AUTH:
if auth_disabled():
raise HTTPException(status_code=403, detail=AUTH_DISABLED_DETAIL)
if APP_STATUS() != "run":
raise HTTPException(status_code=400)
@ -228,7 +273,7 @@ async def login_api(form: OAuth2PasswordRequestForm = Depends()):
@app.post('/api/set-password', response_model=ChangePasswordModel)
async def set_password(form: PasswordForm):
"""Set the password of firegex"""
if UNSAFE_DISABLE_AUTH:
if auth_disabled():
raise HTTPException(status_code=403, detail=AUTH_DISABLED_DETAIL)
if APP_STATUS() != "init":
raise HTTPException(status_code=400)
@ -241,7 +286,7 @@ async def set_password(form: PasswordForm):
@api.post('/change-password', response_model=ChangePasswordModel)
async def change_password(form: PasswordChangeForm):
"""Change the password of firegex"""
if UNSAFE_DISABLE_AUTH:
if auth_disabled():
raise HTTPException(status_code=403, detail=AUTH_DISABLED_DETAIL)
if APP_STATUS() != "run":
raise HTTPException(status_code=400)
@ -257,6 +302,51 @@ async def change_password(form: PasswordChangeForm):
return {"status":"ok", "access_token": create_access_token({"logged_in": True})}
@api.post('/auth-mode', response_model=StatusMessageModel)
async def set_auth_mode(form: AuthModeForm, token: str = Depends(oauth2_scheme)):
"""Turn authentication off on a running instance, and back on.
**Off** is a thing an administrator can decide: it hands access control to whatever
sits in front of firegex, which is a deployment choice rather than an escalation, and
the caller has just proved they are the administrator.
**On** is asked for differently, because while authentication is off every request
reaching firegex is already a full administrator — an anonymous caller could turn it
back on with a password of their own and keep the real operator out for good, which
is a lasting foothold and not merely the vandalism the mode already allows. So it
takes a token this instance signed *before* it was turned off. Nothing new is signed
while it is off (`/api/login` and `/api/set-password` both refuse), so holding one is
exactly the proof that is wanted, and the operator who turned it off still has theirs.
Whoever does not can do it from the host, where being able to ask is its own proof:
`run.py config --no-unsafe-disable-auth`.
Either way it lasts as long as the process, because the environment is what firegex
comes up with. `run.py config --[no-]unsafe-disable-auth` writes here *and* persists
the choice, which is how it survives a restart.
"""
if form.disabled == auth_disabled():
return {"status": "ok"}
if not form.disabled and not token_is_valid(token):
raise HTTPException(
status_code=403,
detail="Turning authentication back on takes a session from before it was "
"turned off, or the host: 'python3 run.py config --no-unsafe-disable-auth'",
)
if not form.disabled and db.get("password") is None:
# Otherwise the instance comes back up asking any passer-by to choose the password.
raise HTTPException(
status_code=400,
detail="There is no password to ask for. Set one first, from the host, with "
"'python3 run.py config --password'",
)
db.put("auth_disabled", "1" if form.disabled else "0")
# The sockets were authorised under the rule that has just changed. Dropping them is
# what makes every browser ask again, which is the only way they find out.
await disconnect_all()
await refresh_frontend()
return {"status": "ok"}
@api.get('/interfaces', response_model=list[IpInterface])
async def get_ip_interfaces():
"""Get a list of ip and ip6 interfaces"""
@ -265,8 +355,38 @@ async def get_ip_interfaces():
#Routers Loader
reset, startup, shutdown = load_routers(api)
async def startup_main():
def seed_auth_mode():
"""Decide at boot whether authentication is on.
Three writers, and a boot has to take the right one. `run.py start/restart` says it
in the environment of a container it has just created. `run.py config` says it on the
running instance and in `.firegex-conf.json` — but a container's environment is fixed
when it is created, so a container Docker started again by itself (a reboot, a daemon
restart, a crash under `restart: unless-stopped`) came back up with the value it was
created with. Authentication re-enabled with `run.py config --password` was off again
after the next reboot, while `config --show` went on saying it was on.
So `run.py config` also writes what it decided under `AUTH_HOST_KEY`, and a boot that
is not fresh takes that over its stale environment. A fresh one — a container `run.py`
has just created from the current configuration — takes the environment, which already
says the same thing, and clears the key. What the interface sets (`/api/auth-mode`)
writes neither, so it lasts as long as the process, as it says it does.
"""
disabled, keep_held = boot_auth_mode(db.get(AUTH_HOST_KEY), FRESH_BOOT,
UNSAFE_DISABLE_AUTH_ENV)
db.put("auth_disabled", "1" if disabled else "0")
if not keep_held:
db.query("DELETE FROM keys_values WHERE key = ?;", AUTH_HOST_KEY)
async def startup_main(seed_auth: bool = True):
db.init()
# Only when a process is starting. An import restarts the application state without
# anything about the deployment having changed, and re-deciding here opened a window —
# the length of every service's start — in which a container booted without
# authentication answered everybody as an administrator, whatever it had been set to.
if seed_auth:
seed_auth_mode()
if os.getenv("PSW_HASH_SET"):
db.put("password", os.getenv("PSW_HASH_SET"))
try:
@ -309,20 +429,17 @@ async def export_db():
temp_db = SQLite(os.path.join('db', f))
dbs[f] = temp_db.dump()
# Export nfproxy filters
if os.path.exists('db/nfproxy_filters'):
dbs['nfproxy_filters'] = {}
for f in os.listdir('db/nfproxy_filters'):
# Export the user's own filter code
if os.path.exists('db/service_filters'):
dbs['service_filters'] = {}
for f in os.listdir('db/service_filters'):
if f.endswith('.py'):
with open(os.path.join('db/nfproxy_filters', f), 'rb') as script_file:
dbs['nfproxy_filters'][f] = base64.b64encode(script_file.read()).decode('utf-8')
with open(os.path.join('db/service_filters', f), 'rb') as script_file:
dbs['service_filters'][f] = base64.b64encode(script_file.read()).decode('utf-8')
return dbs
# Backup entries are limited to plain basenames of the two shapes export_db()
# produces: "<name>.db" databases and "<id>.py" nfproxy filter files. The charset
# forbids path separators and "..", so a crafted key can't escape the target dir.
_SAFE_DB_NAME = re.compile(r'^[A-Za-z0-9_-]+\.db$')
_SAFE_PY_NAME = re.compile(r'^[A-Za-z0-9_-]+\.py$')
# The charset a backup entry's name has to satisfy lives in `utils`, beside `safe_join`:
# the two are one defence in two halves — what a name may contain, and where it may land.
@api.post('/import', response_model=StatusMessageModel)
async def import_db(data: dict):
@ -335,17 +452,17 @@ async def import_db(data: dict):
for key, value in data.items():
if not isinstance(key, str):
raise HTTPException(status_code=400, detail="Invalid backup: keys must be strings")
if _SAFE_DB_NAME.match(key):
if SAFE_DB_NAME.fullmatch(key):
if not isinstance(value, dict):
raise HTTPException(status_code=400, detail=f"Invalid backup: '{key}' must be an object")
# safe_join is defense-in-depth on top of the regex: it rejects any
# path that would resolve outside the db/ directory.
db_imports.append((safe_join('db', key), value))
elif key == 'nfproxy_filters':
elif key == 'service_filters':
if not isinstance(value, dict):
raise HTTPException(status_code=400, detail="Invalid backup: 'nfproxy_filters' must be an object")
raise HTTPException(status_code=400, detail="Invalid backup: 'service_filters' must be an object")
for fname, script_content in value.items():
if not isinstance(fname, str) or not _SAFE_PY_NAME.match(fname):
if not isinstance(fname, str) or not SAFE_PY_NAME.fullmatch(fname):
raise HTTPException(status_code=400, detail=f"Invalid backup: illegal filter filename '{fname}'")
if not isinstance(script_content, str):
raise HTTPException(status_code=400, detail=f"Invalid backup: filter '{fname}' must be a base64 string")
@ -353,38 +470,62 @@ async def import_db(data: dict):
decoded = base64.b64decode(script_content, validate=True)
except ValueError:
raise HTTPException(status_code=400, detail=f"Invalid backup: filter '{fname}' is not valid base64")
filter_imports.append((safe_join('db/nfproxy_filters', fname), decoded))
filter_imports.append((safe_join('db/service_filters', fname), decoded))
else:
raise HTTPException(status_code=400, detail=f"Invalid backup: unexpected entry '{key}'")
if not os.path.exists('db'):
os.makedirs('db')
# Backups never contain the password/secret (export_db strips them), so preserve
# the current session's own values instead of losing them on import.
current_password = db.get("password")
current_secret = db.get("secret")
# Backups never contain the password, the secret or the auth mode (export_db strips
# them), so preserve this instance's own values instead of losing them on import.
kept = {key: db.get(key) for key in ("password", "secret", "auth_disabled", AUTH_HOST_KEY)}
for db_path, db_data in db_imports:
temp_db = SQLite(str(db_path))
# Load into the existing schema; SQLite.load only writes to tables/columns
# that actually exist, so unknown fields in the backup are ignored.
temp_db.load(db_data)
# Every database is loaded once for nothing first, and only if all of them would load
# is any of them loaded for real. They used to go one after another: a refusal from
# one left the ones before it replaced and the rest of the import never ran — so a
# backup listing `firegex.db` first wiped its `keys_values` and never put this
# instance's password and secret back, leaving it asking the next visitor to choose a
# password. Load into the existing schema: SQLite.load only writes to tables and
# columns that actually exist, so unknown fields in the backup are ignored.
try:
for dry_run in (True, False):
for db_path, db_data in db_imports:
temp_db = SQLite(str(db_path))
try:
temp_db.load(db_data, dry_run=dry_run)
except Exception as e:
if dry_run:
raise HTTPException(
status_code=400,
detail=f"Invalid backup: '{os.path.basename(db_path)}' cannot be "
f"loaded, so nothing was imported: {e}",
)
raise
finally:
temp_db.disconnect()
if filter_imports:
os.makedirs('db/nfproxy_filters', exist_ok=True)
for filter_path, decoded in filter_imports:
with open(filter_path, 'wb') as script_file:
script_file.write(decoded)
if filter_imports:
os.makedirs('db/service_filters', exist_ok=True)
for filter_path, decoded in filter_imports:
with open(filter_path, 'wb') as script_file:
script_file.write(decoded)
finally:
# Whatever happened above: these are this instance's, and an import that stopped
# half way must not be the end of them.
for key, value in kept.items():
if value is not None and db.get(key) != value:
db.put(key, value)
if current_password is not None:
db.put("password", current_password)
if current_secret is not None:
db.put("secret", current_secret)
# Restart the application state
# Restart the application state, without re-deciding authentication: nothing about the
# deployment changed, and deciding again from the environment is what a boot does.
await shutdown_main()
await startup_main()
await startup_main(seed_auth=False)
# And once more after it, in case the restart recreated the database under an older
# schema and took the values with it.
for key, value in kept.items():
if value is not None and db.get(key) != value:
db.put(key, value)
return {'status': 'ok'}
@ -403,6 +544,4 @@ if __name__ == '__main__':
reload=DEBUG and not NORELOAD,
access_log=True,
workers=1, # Firewall module can't be replicated in multiple workers
# Later the firewall module will be moved to a separate process
# The webserver will communicate using redis (redis is also needed for websockets)
)

254
backend/binsrc/classes/netfilter.cpp
View File

@ -1,6 +1,5 @@
#include <vector>
#include <thread>
#include <chrono>
#include <iostream>
#include <type_traits>
#include "../utils.cpp"
@ -9,154 +8,123 @@
#ifndef NETFILTER_CLASS_CPP
#define NETFILTER_CLASS_CPP
namespace Firegex
{
namespace NfQueue
{
namespace Firegex {
namespace NfQueue {
template <typename Derived>
class ThreadNfQueue
{
public:
ThreadNfQueue() = default;
virtual ~ThreadNfQueue() = default;
template <typename Derived>
class ThreadNfQueue {
public:
ThreadNfQueue() = default;
virtual ~ThreadNfQueue() = default;
std::thread thr;
BlockingQueue<PktRequest<Derived> *> queue;
std::thread thr;
BlockingQueue<PktRequest<Derived>*> queue;
virtual void before_loop() {}
virtual void handle_next_packet(PktRequest<Derived> *pkt) {}
virtual void before_loop() {}
virtual void handle_next_packet(PktRequest<Derived>* pkt){}
void loop()
{
static_cast<Derived *>(this)->before_loop();
PktRequest<Derived> *pkt;
for (;;)
{
queue.take(pkt);
try
{
static_cast<Derived *>(this)->handle_next_packet(pkt);
}
catch (const std::exception &e)
{
std::cerr << "[error] Exception handling packet: " << e.what() << std::endl;
if (pkt->get_action() == FilterAction::NOACTION)
{
try
{
pkt->accept();
}
catch (...)
{
}
}
}
catch (...)
{
std::cerr << "[error] Unknown exception handling packet" << std::endl;
if (pkt->get_action() == FilterAction::NOACTION)
{
try
{
pkt->accept();
}
catch (...)
{
}
}
}
delete pkt;
void loop() {
static_cast<Derived*>(this)->before_loop();
PktRequest<Derived>* pkt;
for(;;) {
queue.take(pkt);
try {
static_cast<Derived*>(this)->handle_next_packet(pkt);
} catch (const std::exception& e) {
std::cerr << "[error] Exception handling packet: " << e.what() << std::endl;
if (pkt->get_action() == FilterAction::NOACTION) {
try {
pkt->accept();
} catch (...) {}
}
} catch (...) {
std::cerr << "[error] Unknown exception handling packet" << std::endl;
if (pkt->get_action() == FilterAction::NOACTION) {
try {
pkt->accept();
} catch (...) {}
}
}
void run_thread_loop()
{
thr = std::thread([this]()
{ this->loop(); });
}
};
template <typename Worker, typename = is_base_of<ThreadNfQueue<Worker>, Worker>>
void __real_handler(PktRequest<std::vector<Worker>> *pkt)
{
const size_t idx = hash_stream_id(pkt->sid) % pkt->ctx->size();
auto *converted_pkt = reinterpret_cast<PktRequest<Worker> *>(pkt);
converted_pkt->ctx = &((*pkt->ctx)[idx]);
converted_pkt->ctx->queue.put(converted_pkt);
delete pkt;
}
template <typename Worker, typename = is_base_of<ThreadNfQueue<Worker>, Worker>>
class MultiThreadQueue
{
static_assert(std::is_base_of_v<ThreadNfQueue<Worker>, Worker>,
"Worker must inherit from ThreadNfQueue<Worker>");
private:
std::vector<Worker> workers;
NfQueue<std::vector<Worker>, __real_handler<Worker>> *nfq;
uint16_t queue_num_;
public:
const size_t n_threads;
static constexpr int QUEUE_BASE_NUM = 1000;
// Listed in declaration order (that is the order they are really built in).
explicit MultiThreadQueue(size_t n_threads)
: workers(n_threads), n_threads(n_threads)
{
if (n_threads == 0)
throw std::invalid_argument("At least 1 thread required");
for (uint16_t qnum = QUEUE_BASE_NUM;; qnum++)
{
try
{
nfq = new NfQueue<std::vector<Worker>, __real_handler<Worker>>(qnum);
queue_num_ = qnum;
break;
}
catch (const std::invalid_argument &)
{
if (qnum == std::numeric_limits<uint16_t>::max())
throw std::runtime_error("No available queue numbers");
}
}
}
~MultiThreadQueue()
{
delete nfq;
}
void start()
{
for (auto &worker : workers)
{
worker.run_thread_loop();
}
for (;;)
{
try
{
nfq->handle_next_packet(&workers);
}
catch (const std::exception &e)
{
std::cerr << "[error] [MultiThreadQueue.start] " << e.what() << std::endl;
}
catch (...)
{
std::cerr << "[error] [MultiThreadQueue.start] unknown error while handling a packet" << std::endl;
}
}
}
uint16_t queue_num() const { return queue_num_; }
};
}
} // namespace Firegex::NfQueue
void run_thread_loop() {
thr = std::thread([this]() { this->loop(); });
}
};
template <typename Worker, typename = is_base_of<ThreadNfQueue<Worker>, Worker>>
void __real_handler(PktRequest<std::vector<Worker>>* pkt) {
const size_t idx = hash_stream_id(pkt->sid) % pkt->ctx->size();
auto* converted_pkt = reinterpret_cast<PktRequest<Worker>*>(pkt);
converted_pkt->ctx = &((*pkt->ctx)[idx]);
converted_pkt->ctx->queue.put(converted_pkt);
}
template <typename Worker, typename = is_base_of<ThreadNfQueue<Worker>, Worker>>
class MultiThreadQueue {
static_assert(std::is_base_of_v<ThreadNfQueue<Worker>, Worker>,
"Worker must inherit from ThreadNfQueue<Worker>");
private:
std::vector<Worker> workers;
NfQueue<std::vector<Worker>, __real_handler<Worker>> * nfq;
uint16_t queue_num_;
public:
const size_t n_threads;
static constexpr int QUEUE_BASE_NUM = 1000;
explicit MultiThreadQueue(size_t n_threads)
// Listed in declaration order (that is the order they are really built in).
: workers(n_threads), n_threads(n_threads)
{
if(n_threads == 0) throw std::invalid_argument("At least 1 thread required");
for(uint16_t qnum = QUEUE_BASE_NUM; ; qnum++) {
try {
nfq = new NfQueue<std::vector<Worker>, __real_handler<Worker>>(qnum);
queue_num_ = qnum;
break;
}
catch(const std::invalid_argument&) {
if(qnum == std::numeric_limits<uint16_t>::max())
throw std::runtime_error("No available queue numbers");
}
}
}
~MultiThreadQueue() {
delete nfq;
}
void start() {
for(auto& worker : workers) {
worker.run_thread_loop();
}
/* One unexpected message must not end the interception. Whatever
`handle_next_packet` throws — a netlink read that failed, a packet
libtins refuses — the nftables rules stay installed either way, so
leaving this loop means a service that still claims to be filtering
and is not. */
for (;;){
try {
nfq->handle_next_packet(&workers);
} catch (const std::exception& e) {
std::cerr << "[error] [MultiThreadQueue.start] " << e.what() << std::endl;
} catch (...) {
std::cerr << "[error] [MultiThreadQueue.start] unknown error while handling a packet" << std::endl;
}
}
}
uint16_t queue_num() const { return queue_num_; }
};
}} // namespace Firegex::NfQueue
#endif // NETFILTER_CLASS_CPP

112
backend/binsrc/classes/nfqueue.cpp
View File

@ -19,14 +19,10 @@ using namespace std;
namespace Firegex{
namespace NfQueue{
/* Largest packet we are willing to hand back to the kernel in a verdict.
NFQUEUE payloads are carried in a netlink attribute whose length field is
16 bit wide, so anything above this can never be re-injected anyway; the
cap exists to stop a python filter from making us allocate an arbitrary
amount of memory through mangle_custom_pkt(). */
const size_t MAX_VERDICT_PACKET_SIZE = 0xffff;
/* Reads the fail-open policy once, tolerating a missing variable.
`strcmp(getenv(...), "1")` was the previous spelling, which dereferences a
null pointer the moment the binary is run without the variable set — by
hand, or by anything that is not the backend.
Note that this only drives NFQA_CFG_F_FAIL_OPEN (what the kernel does when
the queue is full) and the fallback verdict used when a packet cannot be
parsed; the "what happens when this process is not running at all" half of
@ -174,6 +170,35 @@ class PktRequest {
return packet.data()+_header_size;
}
// The metadata a filter is allowed to know about the layers below it. Read-only by
// construction: these are copies pulled out of the headers, and there is no way
// back from them to the bytes they came from.
string src_ip(){
if (is_ipv6){
return ipv6 ? ipv6->src_addr().to_string() : string();
}
return ipv4 ? ipv4->src_addr().to_string() : string();
}
string dst_ip(){
if (is_ipv6){
return ipv6 ? ipv6->dst_addr().to_string() : string();
}
return ipv4 ? ipv4->dst_addr().to_string() : string();
}
uint16_t src_port(){
if (tcp) return tcp->sport();
if (udp) return udp->sport();
return 0;
}
uint16_t dst_port(){
if (tcp) return tcp->dport();
if (udp) return udp->dport();
return 0;
}
size_t data_size(){
return packet.size()-_header_size;
}
@ -209,58 +234,6 @@ class PktRequest {
return nullptr;
}
void set_packet(const char* data, size_t data_size){
// Parsing only the header with libtins
Tins::PDU *data_pdu = nullptr;
size_t total_size;
// The ipv6 branch has to patch the payload length before handing the
// buffer to libtins, and for a mangled packet `data` points straight at
// the bytes object owned by the python filter: work on a private copy
// instead of mutating the caller's memory.
string raw_packet(data, data_size);
char* raw = raw_packet.data();
if (is_ipv6){
delete ipv6;
ipv6 = nullptr;
if (data_size >= 40){ // 40 == fixed size of ipv6 header
// Resetting payload length before parsing to libtins
uint16_t payload_len = htons(data_size-40);
memcpy(((uint8_t *)raw)+4, &payload_len, 2);
}
ipv6 = new Tins::IPv6((uint8_t*)raw, data_size);
if (tcp){
tcp = ipv6->find_pdu<Tins::TCP>();
data_pdu = tcp;
}else if (udp){
udp = ipv6->find_pdu<Tins::UDP>();
data_pdu = udp;
}else{
data_pdu = ipv6;
}
total_size = ipv6->size();
}else{
delete ipv4;
ipv4 = nullptr;
ipv4 = new Tins::IP((uint8_t*)raw, data_size);
if (tcp){
tcp = ipv4->find_pdu<Tins::TCP>();
data_pdu = tcp;
}else if(udp){
udp = ipv4->find_pdu<Tins::UDP>();
data_pdu = udp;
}else{
data_pdu = ipv4;
}
total_size = ipv4->size();
}
_header_size = total_size - inner_data_size(data_pdu);
if (_header_size > data_size){
throw invalid_argument("Parsed header is larger than the packet itself");
}
// Libtins can skip data if the lenght is changed to a bigger len (due to ip header total lenght), so we need to specify the data section manually
set_data(raw+_header_size, data_size-_header_size);
}
void fix_tcp_ack(){
need_tcp_fixing = need_tcp_fix();
if(!need_tcp_fixing){
@ -343,27 +316,6 @@ class PktRequest {
}
}
void mangle_custom_pkt(const char* raw_pkt, size_t raw_pkt_size){
if (action == FilterAction::NOACTION){
try{
if (raw_pkt_size > MAX_VERDICT_PACKET_SIZE){
throw invalid_argument("Mangled packet is too big to be re-injected");
}
set_packet(raw_pkt, raw_pkt_size);
reserialize();
action = FilterAction::MANGLE;
}catch(const std::exception& e){
#ifdef DEBUG
cerr << "[DEBUG] [PktRequest.mangle_custom_pkt] " << e.what() << endl;
#endif
action = FilterAction::DROP;
}
perform_action(false);
}else{
throw invalid_argument("Cannot mangle a packet that has already been accepted or dropped");
}
}
FilterAction get_action(){
return action;
}

7
backend/binsrc/nfregex.cpp
View File

@ -38,8 +38,9 @@ void config_updater (){
}
}
try{
regex_config.reset(new RegexRules(raw_rules, regex_config->stream_mode()));
cerr << "[info] [updater] Config update done to ver "<< regex_config->ver() << endl;
auto next = make_shared<RegexRules>(raw_rules, regex_config.load()->stream_mode());
regex_config.store(next);
cerr << "[info] [updater] Config update done to ver "<< next->ver() << endl;
osyncstream(cout) << "ACK OK" << endl;
}catch(const std::exception& e){
cerr << "[error] [updater] Failed to build new configuration!" << endl;
@ -81,7 +82,7 @@ int main(int argc, char *argv[]){
bool fail_open = Firegex::NfQueue::nfqueue_fail_open();
regex_config.reset(new RegexRules(stream_mode));
regex_config.store(make_shared<RegexRules>(stream_mode));
MultiThreadQueue<RegexNfQueue> queue_manager(n_of_threads);
osyncstream(cout) << "QUEUE " << queue_manager.queue_num() << endl;

39
backend/binsrc/nfproxy.cpp → backend/binsrc/pyfilter.cpp
View File

@ -1,4 +1,3 @@
#define DEBUG
#define PY_SSIZE_T_CLEAN
#include <Python.h>
@ -22,7 +21,7 @@ How python code is handles:
User code example:
```python
from firegex.nfproxy import DROP, ACCEPT, pyfilter
from firegex.pyfilters import DROP, ACCEPT, pyfilter
@pyfilter
def invalid_curl_agent(http):
@ -36,9 +35,8 @@ The code is now edited adding an intestation and a end statement:
```python
<user_code>
__firegex_pyfilter_enabled = ["invalid_curl_agent", "func3"] # This list is dynamically generated by firegex backend
__firegex_proto = "http"
import firegex.nfproxy.internals
firegex.nfproxy.internals.compile(globals(), locals()) # This function can save other global variables, to use by the packet handler and is used generally to check and optimize the code
import firegex.pyfilters.internals
firegex.pyfilters.internals.compile(globals(), locals()) # This function can save other global variables, to use by the packet handler and is used generally to check and optimize the code
````
(First lines are the same to keep line of code consistent on exceptions messages)
@ -48,39 +46,49 @@ The globals and locals generated here are copied for each connection, and are us
Using C API will be injected in global context the following informations:
__firegex_packet_info = {
"data" = b"raw data found on L4",
"raw_packet" = b"raw packet",
"data" = b"the application payload",
"is_input" = True, # If the packet is incoming from a client
"is_ipv6" = False, # If the packet is ipv6
"is_tcp" = True, # If the packet is tcp
"src_ip" = "10.0.0.7", "dst_ip" = "10.0.0.1",
"src_port" = 51234, "dst_port" = 80,
}
Nothing below the application layer crosses this boundary as bytes: the addresses and
ports are metadata a filter reads, and nothing comes back but a verdict. The proxy
datapath terminates the connection and writes its own headers, so a filter that could
rewrite one here and not there would mean two different things depending on which
network layer it happened to be attached to.
The protocol is not passed in either: the library reads it off the filters' parameter
annotations, so a file that asks for an HttpRequest is an HTTP filter and one that asks
only for a RawPacket runs on anything.
As result the packet handler is responsible to return a dictionary in the global context with the following dictionary:
__firegex_pyfilter_result = {
"action": REJECT, # One of PyFilterResponse
"matched_by": "invalid_curl_agent", # The function that matched the packet (used if action = DROP or REJECT or MANGLE)
"mangled_packet": b"new packet" # The new packet to send to the kernel (used if action = MANGLE)
"matched_by": "invalid_curl_agent", # The function that matched the packet (used if action = DROP or REJECT)
}
PyFilterResponse {
ACCEPT = 0,
DROP = 1,
REJECT = 2,
MANGLE = 3,
// 3 was MANGLE, a payload rewrite; retired, and refused as INVALID
EXCEPTION = 4,
INVALID = 5
};
Every time a packet is received, the packet handler will execute the following code:
```python
firegex.nfproxy.internals.handle_packet(globals())
firegex.pyfilters.internals.handle_packet(globals())
````
The TCP stream is sorted by libtins using c++ code, but the c++ code is not responsabile di buffer the stream, but only to sort those
So firegex handle_packet has to implement a way to limit memory usage, this dipends on what methods you choose to use to filter packets
firegex lib will give you all the needed possibilities to do this is many ways
Final note: is not raccomanded to use variables that starts with __firegex_ in your code, because they may break the nfproxy
Final note: is not raccomanded to use variables that starts with __firegex_ in your code, because they may break the pyfilter
*/
@ -110,13 +118,10 @@ void config_updater (){
cerr << "[fatal] [updater] Control socket error: " << e.what() << endl;
exit(EXIT_FAILURE);
}
#ifdef DEBUG
cerr << "[DEBUG] [updater] Received code: " << code << endl;
#endif
cerr << "[info] [updater] Updating configuration" << endl;
PyEval_AcquireThread(state); //Restore GIL before executing python code
try{
config.reset(new PyCodeConfig(code));
config.store(make_shared<PyCodeConfig>(code));
cerr << "[info] [updater] Config update done" << endl;
control_socket << "ACK OK" << endl;
}catch(const std::exception& e){
@ -145,7 +150,7 @@ int main(int argc, char *argv[]) {
if (n_threads_str != nullptr) n_of_threads = ::atoi(n_threads_str);
if(n_of_threads <= 0) n_of_threads = 1;
config.reset(new PyCodeConfig());
config.store(make_shared<PyCodeConfig>());
MultiThreadQueue<PyProxyQueue> queue(n_of_threads);

116
backend/binsrc/pyproxy/pyproxy.cpp
View File

@ -30,7 +30,22 @@ namespace PyProxy {
class PyProxyQueue: public NfQueue::ThreadNfQueue<PyProxyQueue> {
private:
u_int16_t latest_config_ver = 0;
// The configuration this thread's contexts were built from.
uint32_t latest_config_ver = 0;
// A new configuration reaches the connections already open, the way it does on the
// other two engines: `cppregex` resets its matchers on a new version and the proxy
// layer hands the next chunk to the new chain. This was declared and never read, so
// here a stream kept the code it had started with for as long as it stayed open — a
// function switched off, or a check added against the attack in progress, went on not
// reaching the one connection it was meant for. Its state starts again, as it does
// there.
void follow_config(){
const uint32_t current = config.load()->version;
if (current == latest_config_ver) return;
sctx.clean_filters();
latest_config_ver = current;
}
public:
stream_ctx sctx;
StreamFollower follower;
@ -71,6 +86,14 @@ class PyProxyQueue: public NfQueue::ThreadNfQueue<PyProxyQueue> {
}
handle_packet_code = unmarshal_code(py_handle_packet_code);
// Streams whose start this process did not see are followed from the packet it
// does see. They are otherwise not followed at all, and a packet of one was simply
// accepted: every connection open before the service started, or before its chain
// was rebuilt — which adding a filter does — and every one quiet for longer than
// the follower keeps a stream, carried whatever it said afterwards. A persistent
// connection was a way past any filter added after it opened. Recovery mode is
// switched on for them in `on_new_stream`.
follower.follow_partial_streams(true);
// Setting callbacks for the stream follower
follower.new_stream_callback(bind(on_new_stream, placeholders::_1, this));
follower.stream_termination_callback(bind(on_stream_close, placeholders::_1, this));
@ -80,10 +103,6 @@ class PyProxyQueue: public NfQueue::ThreadNfQueue<PyProxyQueue> {
control_socket << "BLOCKED " << func_name << endl;
}
inline void print_mangle_reason(const string& func_name){
control_socket << "MANGLED " << func_name << endl;
}
inline void print_exception_reason(){
control_socket << "EXCEPTION" << endl;
}
@ -97,11 +116,60 @@ class PyProxyQueue: public NfQueue::ThreadNfQueue<PyProxyQueue> {
pyq->pkt->drop();// This is needed because the callback has to take the updated pkt pointer!
}
// One datagram, judged on its own.
//
// No stream to follow and no sequence numbers to fix: a datagram is complete in
// itself. Each flow still gets its own module globals, keyed the same way a
// connection is.
void filter_action_udp(NfQueue::PktRequest<PyProxyQueue>* pkt, const string& data){
// Idle flows first, so a limit is never met by flows that have already gone.
sctx.udp_expire(UDP_IDLE_SECONDS);
auto stream_search = sctx.streams_ctx.find(pkt->sid);
pyfilter_ctx* stream_match;
if (stream_search == sctx.streams_ctx.end()){
shared_ptr<PyCodeConfig> conf = config.load();
PyObject* compiled_code = conf->compiled_code();
if (compiled_code == nullptr){
return pkt->accept(); // no filter configured; nothing to ask
}
try{
stream_match = new pyfilter_ctx(compiled_code, handle_packet_code);
}catch(invalid_argument& e){
cerr << "[error] [filter_action_udp] Failed to create the filter context" << endl;
print_exception_reason();
return pkt->accept();
}
sctx.udp_make_room(max_udp_flows());
sctx.streams_ctx.insert_or_assign(pkt->sid, stream_match);
}else{
stream_match = stream_search->second;
}
sctx.udp_touch(pkt->sid);
auto result = stream_match->handle_packet(pkt, data, pkt->is_input);
switch(result.action){
case PyFilterResponse::ACCEPT:
return pkt->accept();
// There is no connection to close, so refusing means this datagram is not
// delivered. The next one from the same flow is judged afresh, which is the
// only thing "refuse" can mean without a connection to refuse.
case PyFilterResponse::DROP:
case PyFilterResponse::REJECT:
print_blocked_reason(*result.filter_match_by);
return pkt->drop();
case PyFilterResponse::EXCEPTION:
case PyFilterResponse::INVALID:
print_exception_reason();
sctx.clean_stream_by_id(pkt->sid);
return pkt->accept();
}
}
void filter_action(NfQueue::PktRequest<PyProxyQueue>* pkt, Stream& stream, const string& data, bool is_client){
auto stream_search = sctx.streams_ctx.find(pkt->sid);
pyfilter_ctx* stream_match;
if (stream_search == sctx.streams_ctx.end()){
shared_ptr<PyCodeConfig> conf = config;
shared_ptr<PyCodeConfig> conf = config.load();
//If config is not set, ignore the stream
PyObject* compiled_code = conf->compiled_code();
if (compiled_code == nullptr){
@ -123,6 +191,13 @@ class PyProxyQueue: public NfQueue::ThreadNfQueue<PyProxyQueue> {
stream.ignore_server_data();
return pkt->accept();
}
// This context starts after its connection did: the process met the
// stream halfway, or new code replaced the context it had. What arrives
// first can be the middle of a message, and the library must not judge
// a message by its second half — see `_met_mid_message` there.
if (stream.is_partial_stream() || sctx.taken_over.erase(pkt->sid) > 0){
stream_match->set_item_to_glob("__firegex_joined_late", PyBool_FromLong(1));
}
sctx.streams_ctx.insert_or_assign(pkt->sid, stream_match);
}
}else{
@ -145,16 +220,6 @@ class PyProxyQueue: public NfQueue::ThreadNfQueue<PyProxyQueue> {
stream.client_data_callback(bind(keep_fin_packet, this));
stream.server_data_callback(bind(keep_fin_packet, this));
return pkt->reject();
case PyFilterResponse::MANGLE:
pkt->mangle_custom_pkt(result.mangled_packet->c_str(), result.mangled_packet->size());
if (pkt->get_action() == NfQueue::FilterAction::DROP){
cerr << "[ERROR] [filter_action] Failed to mangle: Malformed Packet... the packet was dropped" << endl;
print_blocked_reason(*result.filter_match_by);
print_exception_reason();
}else{
print_mangle_reason(*result.filter_match_by);
}
return;
case PyFilterResponse::EXCEPTION:
case PyFilterResponse::INVALID:
print_exception_reason();
@ -169,7 +234,12 @@ class PyProxyQueue: public NfQueue::ThreadNfQueue<PyProxyQueue> {
}
static void on_data_recv(Stream& stream, PyProxyQueue* pyq, const string& data, bool is_client) {
// The direction is the packet's own, from the mark the rules put on it, rather than
// which side of the stream the follower calls the client: on a stream followed from
// the middle that is whoever it happened to see first, and the service answering
// before the client speaks again made the filter read the answer as the request.
static void on_data_recv(Stream& stream, PyProxyQueue* pyq, const string& data, bool) {
const bool is_client = pyq->pkt->is_input;
pyq->pkt->fix_data_payload();
pyq->filter_action(pyq->pkt, stream, data, is_client); //Only here the rebuilt_tcp_data is set
}
@ -220,6 +290,18 @@ class PyProxyQueue: public NfQueue::ThreadNfQueue<PyProxyQueue> {
void handle_next_packet(NfQueue::PktRequest<PyProxyQueue>* _pkt) override{
pkt = _pkt; // Setting packet context
follow_config();
if (pkt->l4_proto == NfQueue::L4Proto::UDP){
// Straight to the filter: the stream follower is TCP's, and so is every
// piece of machinery above it. This used to fall into the check below and
// throw — with a message claiming UDP was supported.
filter_action_udp(pkt, string(pkt->data(), pkt->data_size()));
if (pkt->get_action() == NfQueue::FilterAction::NOACTION){
return pkt->accept();
}
return;
}
if (pkt->l4_proto != NfQueue::L4Proto::TCP){
throw invalid_argument("Only TCP and UDP are supported");

14
backend/binsrc/pyproxy/settings.cpp
View File

@ -5,6 +5,7 @@
#include <marshal.h>
#include <vector>
#include <memory>
#include <atomic>
#include <iostream>
#include "../utils.cpp"
@ -15,7 +16,9 @@ namespace PyProxy {
class PyCodeConfig;
shared_ptr<PyCodeConfig> config;
// Atomic for the reason `regex_config` is: the updater swaps it while the queue threads
// copy it for every new stream.
std::atomic<shared_ptr<PyCodeConfig>> config;
UnixClientConnection control_socket;
PyObject* unmarshal_code(string encoded_code){
@ -24,8 +27,14 @@ PyObject* unmarshal_code(string encoded_code){
}
class PyCodeConfig{
private:
// Only the updater thread builds configurations, so this needs no lock.
static inline uint32_t glob_seq = 0;
public:
string encoded_code;
// Which configuration this is, so a queue thread can tell that the contexts it
// holds were built from code no longer in force. 0 is the empty one it starts on.
uint32_t version = 0;
PyCodeConfig(const string& pycode){
PyObject* compiled_code = Py_CompileStringExFlags(pycode.c_str(), "<pyfilter>", Py_file_input, NULL, 2);
@ -58,6 +67,7 @@ class PyCodeConfig{
}
encoded_code = string(PyBytes_AsString(code_dump), PyBytes_Size(code_dump));
Py_DECREF(code_dump);
version = ++glob_seq;
}
PyObject* compiled_code(){
@ -78,7 +88,7 @@ string py_handle_packet_code;
void init_handle_packet_code(){
PyObject* compiled_code = Py_CompileStringExFlags(
"firegex.nfproxy.internals.handle_packet(globals())\n", "<pyfilter>",
"firegex.pyfilters.internals.handle_packet(globals())\n", "<pyfilter>",
Py_file_input, NULL, 2);
PyObject* code_dump = PyMarshal_WriteObjectToString(compiled_code, 4);
Py_DECREF(compiled_code);

219
backend/binsrc/pyproxy/stream_ctx.cpp
View File

@ -5,6 +5,11 @@
#include <iostream>
#include <tins/tcp_ip/stream_identifier.h>
#include <map>
#include <list>
#include <set>
#include <cstdlib>
#include <atomic>
#include <chrono>
#include <Python.h>
#include "../classes/netfilter.cpp"
#include "../classes/nfqueue.cpp"
@ -20,38 +25,72 @@ namespace PyProxy {
class PyCodeConfig;
class PyProxyQueue;
// The library's `Action` values, which are the wire format between it and this binary.
// 3 was `MANGLE`, a rewrite of the payload; it is gone from the library, and the number
// stays unused so that nothing old can ever mean something new. Anything outside the
// valid set is `INVALID`, which fails open and is reported like an exception.
enum PyFilterResponse {
ACCEPT = 0,
DROP = 1,
REJECT = 2,
MANGLE = 3,
EXCEPTION = 4,
INVALID = 5
};
const PyFilterResponse VALID_PYTHON_RESPONSE[4] = {
const PyFilterResponse VALID_PYTHON_RESPONSE[3] = {
PyFilterResponse::ACCEPT,
PyFilterResponse::DROP,
PyFilterResponse::REJECT,
PyFilterResponse::MANGLE
};
struct py_filter_response {
PyFilterResponse action;
string* filter_match_by = nullptr;
string* mangled_packet = nullptr;
py_filter_response(PyFilterResponse action, string* filter_match_by = nullptr, string* mangled_packet = nullptr):
action(action), filter_match_by(filter_match_by), mangled_packet(mangled_packet){}
py_filter_response(PyFilterResponse action, string* filter_match_by = nullptr):
action(action), filter_match_by(filter_match_by){}
~py_filter_response(){
delete mangled_packet;
delete filter_match_by;
}
};
typedef Tins::TCPIP::StreamIdentifier stream_id;
// How long between two tracebacks out of this process, in seconds.
//
// Filter code that throws throws on every packet — a typo does not fire once — and a
// traceback is a dozen lines. Unthrottled, one broken filter writes thousands of lines a
// minute into a log that is a bounded ring, so the flood does not merely repeat itself:
// it pushes out the first traceback, which was the one worth reading, along with
// everything that was in the log before the filter broke.
//
// The `EXCEPTION` sent to the backend is deliberately *not* throttled with it. That is
// one token, it costs nothing, and it is what the backend counts to say "still raising,
// N more since" — so the operator still learns how often this is happening while being
// shown the traceback once.
constexpr int64_t TRACEBACK_QUIET_SECONDS = 30;
// Process-wide rather than per stream: the flood is one broken filter, not one broken
// connection, and throttling per connection would let a hundred connections print a
// hundred copies of the same fault. Relaxed ordering is enough — two threads racing here
// print one traceback each, which is not a problem worth a lock.
static std::atomic<int64_t> last_traceback_at{-TRACEBACK_QUIET_SECONDS * 2};
inline bool traceback_is_due() {
const int64_t now = std::chrono::duration_cast<std::chrono::seconds>(
std::chrono::steady_clock::now().time_since_epoch()).count();
int64_t previous = last_traceback_at.load(std::memory_order_relaxed);
if (now - previous < TRACEBACK_QUIET_SECONDS) return false;
return last_traceback_at.compare_exchange_strong(
previous, now, std::memory_order_relaxed);
}
// How many dropped filter contexts may wait for a full collection, and how many have on
// this queue thread — each thread runs an interpreter of its own, so each counts its own.
constexpr unsigned COLLECT_EVERY = 64;
static thread_local unsigned dropped_since_collect = 0;
struct pyfilter_ctx {
PyObject * glob = nullptr;
@ -75,7 +114,18 @@ struct pyfilter_ctx {
~pyfilter_ctx(){
Py_DECREF(glob);
Py_DECREF(py_handle_packet);
PyGC_Collect();
// The module's functions hold these globals and the globals hold the functions,
// so a context is a cycle only the collector frees. Left to the interpreter's
// own schedule, one that lived long enough to be promoted waits for a full pass,
// which is triggered by object counts rather than by bytes — measured, a filter
// building a table at module level held ~200 MB more across a stream of short
// connections. A full pass per context held memory flat and cost ~1 ms each, a
// ceiling of about two hundred connections a second per queue thread. One pass
// every `COLLECT_EVERY` contexts keeps both.
if (++dropped_since_collect >= COLLECT_EVERY){
dropped_since_collect = 0;
PyGC_Collect();
}
}
inline void set_item_to_glob(const char* key, PyObject* value){
@ -111,22 +161,38 @@ struct pyfilter_ctx {
PyObject * packet_info = PyDict_New();
pkt->reserialize();
// The application payload, and metadata about everything under it. No header
// bytes cross this boundary in either direction: a filter reads where the
// traffic came from and edits what it carries, which is the one contract both
// network layers can honestly offer.
string src_ip = pkt->src_ip(), dst_ip = pkt->dst_ip();
set_item_to_dict(packet_info, "data", PyBytes_FromStringAndSize(data.c_str(), data.size()));
set_item_to_dict(packet_info, "l4_size", PyLong_FromLong(pkt->data_size()));
set_item_to_dict(packet_info, "raw_packet", PyBytes_FromStringAndSize(pkt->packet.c_str(), pkt->packet.size()));
set_item_to_dict(packet_info, "is_input", PyBool_FromLong(is_client));
set_item_to_dict(packet_info, "is_ipv6", PyBool_FromLong(pkt->is_ipv6));
set_item_to_dict(packet_info, "is_tcp", PyBool_FromLong(pkt->l4_proto == NfQueue::L4Proto::TCP));
set_item_to_dict(packet_info, "src_ip", PyUnicode_FromStringAndSize(src_ip.c_str(), src_ip.size()));
set_item_to_dict(packet_info, "dst_ip", PyUnicode_FromStringAndSize(dst_ip.c_str(), dst_ip.size()));
set_item_to_dict(packet_info, "src_port", PyLong_FromLong(pkt->src_port()));
set_item_to_dict(packet_info, "dst_port", PyLong_FromLong(pkt->dst_port()));
// Set packet info to the global context
set_item_to_glob("__firegex_packet_info", packet_info);
// No collection here. The interpreter's own collector is on (`before_loop`
// makes sure), and a full pass per packet cost about a millisecond with the
// library loaded — a ceiling of a thousand packets a second per queue thread,
// spent finding nothing.
PyObject * result = PyEval_EvalCode(py_handle_packet, glob, glob);
PyGC_Collect();
del_item_from_glob("__firegex_packet_info");
if (PyErr_Occurred()){
cerr << "[error] [handle_packet] Failed to execute the code " << result << endl;
PyErr_Print();
// Shown at most once every TRACEBACK_QUIET_SECONDS; the EXCEPTION below goes
// every time, and is what carries the count.
if (traceback_is_due()){
cerr << "[error] [handle_packet] Failed to execute the code " << result << endl;
PyErr_Print();
} else {
PyErr_Clear();
}
#ifdef DEBUG
cerr << "[DEBUG] [handle_packet] Exception raised" << endl;
#endif
@ -210,28 +276,8 @@ struct pyfilter_ctx {
del_item_from_glob("__firegex_pyfilter_result");
return py_filter_response(action_enum, func_name);
}
if (action_enum == PyFilterResponse::MANGLE){
PyObject* mangled_packet = PyDict_GetItemString(result, "mangled_packet");
if (mangled_packet == nullptr){
del_item_from_glob("__firegex_pyfilter_result");
#ifdef DEBUG
cerr << "[DEBUG] [handle_packet] No result mangled_packet found" << endl;
#endif
return py_filter_response(PyFilterResponse::INVALID);
}
if (!PyBytes_Check(mangled_packet)){
#ifdef DEBUG
cerr << "[DEBUG] [handle_packet] mangled_packet is not a bytes" << endl;
#endif
del_item_from_glob("__firegex_pyfilter_result");
return py_filter_response(PyFilterResponse::INVALID);
}
string* pkt_str = new string(PyBytes_AsString(mangled_packet), PyBytes_Size(mangled_packet));
del_item_from_glob("__firegex_pyfilter_result");
return py_filter_response(PyFilterResponse::MANGLE, func_name, pkt_str);
}
//Should never reach this point, but just in case of new action not managed...
delete func_name;
del_item_from_glob("__firegex_pyfilter_result");
return py_filter_response(PyFilterResponse::INVALID);
}
@ -241,19 +287,118 @@ struct pyfilter_ctx {
typedef map<stream_id, pyfilter_ctx*> matching_map;
// How long a UDP flow keeps its filter's state with nothing arriving. A datagram has no
// close to observe, so this is the only thing that ends a flow — the same minute the
// proxy layer's relay gives one.
constexpr int64_t UDP_IDLE_SECONDS = 60;
inline long long env_number(const char* name){
const char* env = getenv(name);
if (env == nullptr) return 0;
char* end = nullptr;
long long parsed = strtoll(env, &end, 10);
return end != env ? parsed : 0;
}
// How many UDP flows one queue thread may hold a filter's state for at once: the
// service's own "most connections at once", shared out between the threads — each holds
// the contexts of the flows hashed to it, so a limit applied per thread would be the
// operator's number times the thread count. 0, missing or unreadable means no limit, and
// idle flows still go after `UDP_IDLE_SECONDS`, which is what keeps that from growing
// without end.
inline size_t max_udp_flows(){
static const size_t value = [](){
const long long limit = env_number("FIREGEX_MAX_FLOWS");
if (limit <= 0) return (size_t)0;
const long long threads = env_number("NTHREADS") > 0 ? env_number("NTHREADS") : 1;
return (size_t)((limit + threads - 1) / threads);
}();
return value;
}
inline int64_t steady_seconds(){
return std::chrono::duration_cast<std::chrono::seconds>(
std::chrono::steady_clock::now().time_since_epoch()).count();
}
struct stream_ctx {
matching_map streams_ctx;
NfQueue::tcp_ack_map tcp_ack_ctx;
// The UDP flows holding a context, most recently used first, with when each was
// last seen — so the one to let go of, whether for being idle or to make room, is
// always at the back. TCP contexts are not in here and are never evicted: a TCP
// stream is released when libtins sees it close, and dropping its state while it
// is still open would have its next packet parsed from a clean slate mid-message.
list<pair<stream_id, int64_t>> udp_recent;
map<stream_id, list<pair<stream_id, int64_t>>::iterator> udp_where;
// The TCP streams whose filter context was thrown away while they were open — new
// code arrived — so the context built for their next packet knows it is taking over
// halfway, possibly in the middle of a message.
set<stream_id> taken_over;
void clean_stream_by_id(stream_id sid){
taken_over.erase(sid);
auto stream_search = streams_ctx.find(sid);
if (stream_search != streams_ctx.end()){
auto stream_match = stream_search->second;
delete stream_match;
streams_ctx.erase(stream_search->first);
}
auto flow = udp_where.find(sid);
if (flow != udp_where.end()){
udp_recent.erase(flow->second);
udp_where.erase(flow);
}
}
// This UDP flow was just used.
void udp_touch(const stream_id& sid){
auto flow = udp_where.find(sid);
if (flow != udp_where.end()){
udp_recent.erase(flow->second);
}
udp_recent.emplace_front(sid, steady_seconds());
udp_where[sid] = udp_recent.begin();
}
// Let go of the UDP flows nothing has arrived on for `idle` seconds.
void udp_expire(int64_t idle){
const int64_t now = steady_seconds();
while (!udp_recent.empty() && now - udp_recent.back().second >= idle){
stream_id sid = udp_recent.back().first;
clean_stream_by_id(sid);
}
}
// Make room for one more UDP flow under `limit`, letting the least recently used go;
// a limit of zero is none. Only UDP flows are counted and only UDP flows go: the
// ceiling used to be applied to every context, so a burst of datagrams could take the
// state of a TCP connection still in the middle of a request.
void udp_make_room(size_t limit){
while (limit > 0 && udp_where.size() >= limit && !udp_recent.empty()){
stream_id sid = udp_recent.back().first;
clean_stream_by_id(sid);
}
}
// Every filter context, TCP and UDP alike, for when the code they were built from is
// no longer the code in force. The sequence bookkeeping stays: it belongs to the
// connection, not to the filter, and a stream whose payload was already cut would
// have its acknowledgements go wrong without it.
void clean_filters(){
for (auto ele: streams_ctx){
if (udp_where.find(ele.first) == udp_where.end()){
taken_over.insert(ele.first);
}
delete ele.second;
}
streams_ctx.clear();
udp_recent.clear();
udp_where.clear();
}
void clean_tcp_ack_by_id(stream_id sid){
@ -266,14 +411,12 @@ struct stream_ctx {
}
void clean(){
for (auto ele: streams_ctx){
delete ele.second;
}
clean_filters();
taken_over.clear();
for (auto ele: tcp_ack_ctx){
delete ele.second;
}
tcp_ack_ctx.clear();
streams_ctx.clear();
}
};

7
backend/binsrc/regex/regex_rules.cpp
View File

@ -8,6 +8,7 @@
#include <vector>
#include <hs.h>
#include <memory>
#include <atomic>
using namespace std;
@ -194,7 +195,11 @@ class RegexRules{
}
};
shared_ptr<RegexRules> regex_config;
// Atomic, because it is swapped by the updater thread while every queue thread copies
// it per packet. A plain shared_ptr read and reset from two threads at once is a data
// race: the copy can take a reference on a control block the reset has just freed, and
// an operator editing a pattern under load could crash the binary with it.
std::atomic<shared_ptr<RegexRules>> regex_config;
void inline scratch_setup(regex_ruleset &conf, hs_scratch_t* & scratch){
if (scratch == nullptr && conf.hs_db != nullptr){

10
backend/binsrc/regex/regexfilter.cpp
View File

@ -39,7 +39,7 @@ public:
NfQueue::PktRequest<RegexNfQueue>* pkt;
bool filter_action(NfQueue::PktRequest<RegexNfQueue>* pkt, const string& data){
shared_ptr<RegexRules> conf = regex_config;
shared_ptr<RegexRules> conf = regex_config.load();
auto current_version = conf->ver();
if (current_version != latest_config_ver){
@ -181,6 +181,14 @@ public:
}
void before_loop() override{
// Streams whose start this process did not see are followed from the packet it
// does see. They are otherwise not followed at all, and a packet of one was simply
// accepted: every connection open before the service started, or before its chain
// was rebuilt — which adding a filter does — and every one quiet for longer than
// the follower keeps a stream, carried whatever it said afterwards. A persistent
// connection was a way past any filter added after it opened. Recovery mode is
// switched on for them in `on_new_stream`.
follower.follow_partial_streams(true);
follower.new_stream_callback(bind(on_new_stream, placeholders::_1, this));
follower.stream_termination_callback(bind(on_stream_close, placeholders::_1, this));
}

51
backend/docker-entrypoint.sh
View File

@ -2,6 +2,17 @@
chown nobody -R /execute/
# Whether this is the first boot of this container, told to the backend. A container's
# environment is fixed when it is created, and Docker starting it again by itself — after
# a reboot, a restart of the daemon, a crash under `restart: unless-stopped` — brings back
# that environment, which may be older than what `run.py config` has said since. The
# marker lives in the container's own layer: it survives a restart and is gone from a
# container `run.py` creates anew.
if [ ! -e /execute/.firegex-booted ]; then
touch /execute/.firegex-booted
export FIREGEX_FRESH_BOOT=1
fi
# Create socket directory if SOCKET_DIR is set
if [ -n "$SOCKET_DIR" ]; then
mkdir -p "$SOCKET_DIR"
@ -20,18 +31,34 @@ fi
echo "[*] Attempting to start with capabilities..."
if capsh --caps="cap_net_admin,cap_setpcap,cap_setuid,cap_setgid,cap_sys_nice+eip" \
--keep=1 \
--user=nobody \
--addamb=cap_net_admin,cap_sys_nice \
-- -c "exit 0"
then
exec capsh --caps="cap_net_admin,cap_setpcap,cap_setuid,cap_setgid,cap_sys_nice+eip" \
--keep=1 \
--user=nobody \
--addamb=cap_net_admin,cap_sys_nice \
--shell=/usr/bin/python3 \
-- /execute/app.py DOCKER
# Two sets, tried in order, because they are not equally important.
#
# `cap_net_raw` is what lets the proxy engine open the packet socket it writes decrypted
# traffic to, for an operator capturing `firegex0`. It is a capture aid: a host that will
# not grant it should lose the capture, not the firewall. So the set without it is tried
# next, and only if *that* fails does anything run as root.
try_caps() {
caps="$1"
ambient="$2"
capsh --caps="$caps" --keep=1 --user=nobody --addamb="$ambient" -- -c "exit 0" 2>/dev/null
}
run_caps() {
exec capsh --caps="$1" --keep=1 --user=nobody --addamb="$2" \
--shell=/usr/bin/python3 -- /execute/app.py DOCKER
}
FULL="cap_net_admin,cap_net_raw,cap_setpcap,cap_setuid,cap_setgid,cap_sys_nice+eip"
FULL_AMB="cap_net_admin,cap_net_raw,cap_sys_nice"
BASE="cap_net_admin,cap_setpcap,cap_setuid,cap_setgid,cap_sys_nice+eip"
BASE_AMB="cap_net_admin,cap_sys_nice"
if try_caps "$FULL" "$FULL_AMB"; then
run_caps "$FULL" "$FULL_AMB"
elif try_caps "$BASE" "$BASE_AMB"; then
echo "[!] no cap_net_raw: TLS services will run, but their decrypted traffic"
echo " cannot be written to the capture interface"
run_caps "$BASE" "$BASE_AMB"
else
echo "[!] capsh failed, running with root user"
exec python3 /execute/app.py DOCKER

39
backend/modules/firewall/firewall.py
View File

@ -1,4 +1,5 @@
import asyncio
import traceback
from modules.firewall.nftables import FiregexTables
from modules.firewall.models import Rule, FirewallSettings
from utils.sqlite import SQLite
@ -6,18 +7,44 @@ from modules.firewall.models import Action
nft = FiregexTables()
#: How often the kernel is asked whether the firewall is still there. The services
#: module asks the same of its own table, for the same reason: `nft flush ruleset` —
#: the first line of a Debian `/etc/nftables.conf` — takes this with it, and a
#: default-deny firewall silently became no firewall at all.
TABLE_WATCH = 3.0
class FirewallManager:
def __init__(self, db:SQLite):
self.db = db
self.lock = asyncio.Lock()
self._watch: asyncio.Task | None = None
async def close(self):
self.stop_watching()
async with self.lock:
nft.reset()
async def init(self):
nft.init()
await self.reload()
if self._watch is None or self._watch.done():
self._watch = asyncio.create_task(self._watch_table())
async def _watch_table(self):
while True:
await asyncio.sleep(TABLE_WATCH)
try:
if self.enabled and not nft.intact():
print("[error] [firewall] the firewall's nftables tables were removed "
"by something else; putting them back", flush=True)
await self.reload()
except Exception:
traceback.print_exc()
def stop_watching(self) -> None:
if self._watch is not None:
self._watch.cancel()
self._watch = None
async def reload(self):
async with self.lock:
@ -40,7 +67,8 @@ class FirewallManager:
multicast_dns=self.multicast_dns,
allow_upnp=self.allow_upnp,
drop_invalid=self.drop_invalid,
allow_dhcp=self.allow_dhcp
allow_dhcp=self.allow_dhcp,
allow_dnat=self.allow_dnat,
)
@settings.setter
@ -53,6 +81,7 @@ class FirewallManager:
self.allow_upnp=value.allow_upnp
self.drop_invalid=value.drop_invalid
self.allow_dhcp=value.allow_dhcp
self.allow_dnat=value.allow_dnat
@property
def policy(self):
@ -134,3 +163,11 @@ class FirewallManager:
def allow_dhcp(self, value):
self.db.set("allow_dhcp", "1" if value else "0")
@property
def allow_dnat(self):
return self.db.get("allow_dnat", "1") == "1"
@allow_dnat.setter
def allow_dnat(self, value):
self.db.set("allow_dnat", "1" if value else "0")

5
backend/modules/firewall/models.py
View File

@ -77,3 +77,8 @@ class FirewallSettings(BaseModel):
allow_upnp: bool
drop_invalid: bool
allow_dhcp: bool
#: Forwarded traffic whose destination another NAT rewrote — a port Docker or podman
#: published, a router's port forward — is left to the rules that published it when no
#: firegex rule matched it, instead of meeting the forward policy. Defaulted, so a
#: client or a backup that predates it still validates.
allow_dnat: bool = True

300
backend/modules/firewall/nftables.py
View File

@ -1,47 +1,121 @@
"""The plain-nftables firewall, in tables of firegex's own.
**Everything here is named `fgex_`, and that is not only cosmetic.** This module used to
write into the tables called `filter` and `mangle` — the ones `iptables-nft` claims —
putting base chains named INPUT and FORWARD beside iptables' own and setting their
policy. Two things went wrong with that, and both were reported from real hosts:
* **iptables stops working.** `iptables-nft` refuses a whole table as soon as it holds
one rule it cannot express in its own format, and the first thing this module writes
is a native `ct state` match — iptables stores that as an `xt` match blob instead. From
then on `iptables -L`, `iptables-save` and everything built on them answer
``table `filter' is incompatible, use 'nft' tool``, on a box where Docker, ufw,
fail2ban or the organisers' own scripts are the ones asking. It took nothing exotic to
trigger: `allow_established` and `drop_invalid` are the two settings that write `ct
state`, and both are on in any sane configuration.
* **Stopping firegex opened the host.** `reset()` set the policy of `filter INPUT` back
to `accept`, and that chain belonged to the *administrator*: a host carrying a
default-deny policy had it quietly removed the moment the firegex firewall was
disabled.
Owning the tables fixes both by construction: a base chain carries its own policy
wherever it lives, so default-deny is expressed exactly as before, and a `drop` is
terminal no matter which table's chain reached it. What changes is that firegex now
*adds* a layer instead of taking over somebody else's — an `accept` here means "firegex
does not block this", never "nothing else will", which is what it already meant in
practice. Everything installed is visible under one prefix and removed by deleting two
tables per family.
One thing did change, and `dnat_rules` is what puts it back: sharing iptables' own
`FORWARD` chain, traffic Docker or podman accepted never met firegex's policy, while a
chain of our own is evaluated on its own — so a drop policy dropped every connection to a
published container port, every connection between two containers and every one a
container made to the outside. `allow_dnat` (on by default) leaves that traffic, when no
rule of ours matched it, to the container runtime's rules as it always was.
"""
from modules.firewall.models import FirewallSettings, Action, Rule, Protocol, Mode, Table
from utils import nftables_int_to_json, ip_family, NFTableManager, is_ip_parse
import copy
#: The bridges a container runtime forwards through: Docker's default one and its user
#: networks (`br-<id>`), and podman's. Traffic arriving from one of them is a container
#: talking — to another container, or out — and iptables' `FORWARD`, where Docker and
#: podman accept it, is where firegex's rules used to sit too.
CONTAINER_BRIDGES = ("docker0", "br-*", "docker_gwbridge", "podman*", "cni-podman*")
#: Where a rule's `table` — which is the operator's word, stored in the database and
#: shown in the interface — actually lands. The two are deliberately not the same string
#: any more: "filter" and "mangle" describe what a rule *does*, and used to double as the
#: name of somebody else's table.
NFT_TABLES = {
Table.FILTER: "fgex_filter",
Table.MANGLE: "fgex_mangle",
}
class FiregexTables(NFTableManager):
rules_chain_in = "firegex_firewall_rules_in"
rules_chain_out = "firegex_firewall_rules_out"
rules_chain_fwd = "firegex_firewall_rules_fwd"
filter_table = "filter"
mangle_table = "mangle"
rules_chain_in = "fgex_rules_in"
rules_chain_out = "fgex_rules_out"
rules_chain_fwd = "fgex_rules_fwd"
filter_table = NFT_TABLES[Table.FILTER]
mangle_table = NFT_TABLES[Table.MANGLE]
#: The base chains, and the one rules chain each of them hands over to. Hook and
#: priority are the same ones the rules used to be evaluated at, so what reaches a
#: rule is unchanged; only the table around it is firegex's.
base_chains = [
# (table, chain, hook, priority, follows the configured policy)
("filter", "fgex_input", "input", 0, True, rules_chain_in),
("filter", "fgex_forward", "forward", 0, True, rules_chain_fwd),
("filter", "fgex_output", "output", 0, False, rules_chain_out),
("mangle", "fgex_prerouting", "prerouting", -150, False, rules_chain_in),
("mangle", "fgex_postrouting", "postrouting", -150, False, rules_chain_out),
]
#: The rules chains, and which table each lives in. `fgex_rules_fwd` is filter-only:
#: mangle has no forward hook in this layout, which is why a mangle rule in forward
#: mode is refused by the router before it gets here.
rules_chains = [
("filter", rules_chain_in), ("filter", rules_chain_out), ("filter", rules_chain_fwd),
("mangle", rules_chain_in), ("mangle", rules_chain_out),
]
def _table(self, which: str) -> str:
return self.filter_table if which == "filter" else self.mangle_table
def _skeleton(self, policy: str):
"""Everything that has to exist before a single rule can be added.
The base chains are flushed and re-pointed rather than inspected for a jump that
may already be there. Each holds exactly one rule and nothing else, so rebuilding
is shorter than checking and cannot leave a duplicate behind; the version that
checked had to, because the chain belonged to iptables and emptying it was not
firegex's to do.
"""
for family in ("ip", "ip6"):
for table in (self.filter_table, self.mangle_table):
yield {"add": {"table": {"name": table, "family": family}}}
for which, chain in self.rules_chains:
yield {"add": {"chain": {
"family": family, "table": self._table(which), "name": chain,
}}}
for which, chain, hook, prio, follows_policy, target in self.base_chains:
table = self._table(which)
yield {"add": {"chain": {
"family": family, "table": table, "name": chain,
"type": "filter", "hook": hook, "prio": prio,
"policy": policy if follows_policy else Action.ACCEPT,
}}}
yield {"flush": {"chain": {"family": family, "table": table, "name": chain}}}
yield {"add": {"rule": {
"family": family, "table": table, "chain": chain,
"expr": [{"jump": {"target": target}}],
}}}
def init_comands(self, policy:str=Action.ACCEPT, opt:
FirewallSettings|None = None):
rules = [
{"add":{"table":{"name":self.filter_table,"family":"ip"}}},
{"add":{"table":{"name":self.filter_table,"family":"ip6"}}},
{"add":{"table":{"name":self.mangle_table,"family":"ip"}}},
{"add":{"table":{"name":self.mangle_table,"family":"ip6"}}},
{"add":{"chain":{"family":"ip","table":self.filter_table, "name":"INPUT","type":"filter","hook":"input","prio":0,"policy":policy}}},
{"add":{"chain":{"family":"ip6","table":self.filter_table,"name":"INPUT","type":"filter","hook":"input","prio":0,"policy":policy}}},
{"add":{"chain":{"family":"ip","table":self.filter_table,"name":"FORWARD","type":"filter","hook":"forward","prio":0,"policy":policy}}},
{"add":{"chain":{"family":"ip6","table":self.filter_table,"name":"FORWARD","type":"filter","hook":"forward","prio":0,"policy":policy}}},
{"add":{"chain":{"family":"ip","table":self.filter_table,"name":"OUTPUT","type":"filter","hook":"output","prio":0,"policy":Action.ACCEPT}}},
{"add":{"chain":{"family":"ip6","table":self.filter_table,"name":"OUTPUT","type":"filter","hook":"output","prio":0,"policy":Action.ACCEPT}}},
{"add":{"chain":{"family":"ip","table":self.mangle_table, "name":"PREROUTING","type":"filter","hook":"prerouting","prio":-150,"policy":Action.ACCEPT}}},
{"add":{"chain":{"family":"ip6","table":self.mangle_table,"name":"PREROUTING","type":"filter","hook":"prerouting","prio":-150,"policy":Action.ACCEPT}}},
{"add":{"chain":{"family":"ip","table":self.mangle_table, "name":"POSTROUTING","type":"filter","hook":"postrouting","prio":-150,"policy":Action.ACCEPT}}},
{"add":{"chain":{"family":"ip6","table":self.mangle_table,"name":"POSTROUTING","type":"filter","hook":"postrouting","prio":-150,"policy":Action.ACCEPT}}},
{"add":{"chain":{"family":"ip","table":self.filter_table,"name":self.rules_chain_in}}},
{"add":{"chain":{"family":"ip6","table":self.filter_table,"name":self.rules_chain_in}}},
{"add":{"chain":{"family":"ip","table":self.filter_table,"name":self.rules_chain_out}}},
{"add":{"chain":{"family":"ip6","table":self.filter_table,"name":self.rules_chain_out}}},
{"add":{"chain":{"family":"ip","table":self.filter_table,"name":self.rules_chain_fwd}}},
{"add":{"chain":{"family":"ip6","table":self.filter_table,"name":self.rules_chain_fwd}}},
{"add":{"chain":{"family":"ip","table":self.mangle_table,"name":self.rules_chain_in}}},
{"add":{"chain":{"family":"ip6","table":self.mangle_table,"name":self.rules_chain_in}}},
{"add":{"chain":{"family":"ip","table":self.mangle_table,"name":self.rules_chain_out}}},
{"add":{"chain":{"family":"ip6","table":self.mangle_table,"name":self.rules_chain_out}}},
]
rules = list(self._skeleton(policy))
if opt is None:
return rules
@ -173,66 +247,46 @@ class FiregexTables(NFTableManager):
return rules
def __init__(self):
super().__init__(self.init_comands(),[
#Needed to reset to ALLOW when fireall is disabled (DO NOT REMOVE)
{"add":{"chain":{"family":"ip","table":self.filter_table, "name":"INPUT","type":"filter","hook":"input","prio":0,"policy":Action.ACCEPT}}},
{"add":{"chain":{"family":"ip6","table":self.filter_table,"name":"INPUT","type":"filter","hook":"input","prio":0,"policy":Action.ACCEPT}}},
{"add":{"chain":{"family":"ip","table":self.filter_table,"name":"FORWARD","type":"filter","hook":"forward","prio":0,"policy":Action.ACCEPT}}},
{"add":{"chain":{"family":"ip6","table":self.filter_table,"name":"FORWARD","type":"filter","hook":"forward","prio":0,"policy":Action.ACCEPT}}},
# Taking the firewall down is now deleting what firegex owns, nothing more.
# This used to have to put things *back*: the base chains were the host's, so
# stopping meant re-adding `filter INPUT` with an accept policy — which silently
# undid an administrator's default-deny — and flushing the rules chains one by
# one because deleting the table around them was not an option. `add` before
# `delete` is there because a batch is atomic and deleting a table that is not
# there fails the whole thing; adding one that already exists does nothing.
self._teardown = [
command
for family in ("ip", "ip6")
for table in (self.filter_table, self.mangle_table)
for command in (
{"add": {"table": {"name": table, "family": family}}},
{"delete": {"table": {"name": table, "family": family}}},
)
]
super().__init__(self.init_comands(), self._teardown)
{"flush":{"chain":{"table":self.filter_table,"family":"ip", "name":self.rules_chain_in}}},
{"flush":{"chain":{"table":self.filter_table,"family":"ip", "name":self.rules_chain_out}}},
{"flush":{"chain":{"table":self.filter_table,"family":"ip", "name":self.rules_chain_fwd}}},
{"flush":{"chain":{"table":self.filter_table,"family":"ip6", "name":self.rules_chain_in}}},
{"flush":{"chain":{"table":self.filter_table,"family":"ip6", "name":self.rules_chain_out}}},
{"flush":{"chain":{"table":self.filter_table,"family":"ip6", "name":self.rules_chain_fwd}}},
def intact(self) -> bool:
"""Whether the firewall firegex installed is still in the kernel.
{"flush":{"chain":{"table":self.mangle_table,"family":"ip", "name":self.rules_chain_in}}},
{"flush":{"chain":{"table":self.mangle_table,"family":"ip", "name":self.rules_chain_out}}},
{"flush":{"chain":{"table":self.mangle_table,"family":"ip6", "name":self.rules_chain_in}}},
{"flush":{"chain":{"table":self.mangle_table,"family":"ip6", "name":self.rules_chain_out}}}
])
def chain_to_firegex(self, chain:str, table:str):
if table == self.filter_table:
match chain:
case "INPUT":
return self.rules_chain_in
case "OUTPUT":
return self.rules_chain_out
case "FORWARD":
return self.rules_chain_fwd
elif table == self.mangle_table:
match chain:
case "PREROUTING":
return self.rules_chain_in
case "POSTROUTING":
return self.rules_chain_out
return None
def insert_firegex_chains(self):
rules:list[dict] = list(self.list_rules(tables=[self.filter_table, self.mangle_table], chains=["INPUT", "OUTPUT", "FORWARD", "PREROUTING", "POSTROUTING"]))
for table in [self.filter_table, self.mangle_table]:
for family in ["ip", "ip6"]:
for chain in ["INPUT", "OUTPUT", "FORWARD"] if table == self.filter_table else ["PREROUTING", "POSTROUTING"]:
found = False
rule_to_add = [{ "jump": { "target": self.chain_to_firegex(chain, table) }}]
for r in rules:
if r.get("family") == family and r.get("table") == table and r.get("chain") == chain and r.get("expr") == rule_to_add:
found = True
break
if found:
continue
yield { "add":{ "rule": {
"family": family,
"table": table,
"chain": chain,
"expr": rule_to_add
}}}
The input chain of the IPv4 filter table always holds its jump, so it is the one
place a deleted table and an emptied one both show.
"""
code, listed, _ = self.raw_cmd({"list": {"chain": {
"family": "ip", "table": self.filter_table, "name": "fgex_input",
}}})
return code == 0 and any("rule" in item for item in listed["nftables"])
def set(self, srvs:list[Rule], policy:str=Action.ACCEPT, opt:FirewallSettings = None):
"""Replace the whole firewall with this one, or leave the one in force alone.
**One batch, teardown included.** nft applies a batch whole or not at all, and the
teardown used to be a batch of its own sent first: when nft then refused a rule —
an interface name longer than the kernel allows was enough — the old tables were
already gone and the new ones never arrived. A default-deny firewall became no
firewall at all, and the watcher putting it back failed the same way every few
seconds. Sent together, a refusal leaves the previous firewall exactly as it was.
"""
srvs = list(srvs)
self.reset()
if policy == Action.REJECT:
policy = Action.DROP
srvs.append(Rule(
@ -248,23 +302,70 @@ class FiregexTables(NFTableManager):
table=Table.FILTER
))
rules = self.init_comands(policy, opt) + list(self.insert_firegex_chains()) + self.get_rules(*srvs)
self.cmd(*rules)
# No hooking step any more: the base chains are firegex's own and `_skeleton`
# already pointed each of them at its rules chain.
rules = self.init_comands(policy, opt) + self.get_rules(*srvs) + self.dnat_rules(opt)
self.cmd(*self._teardown, *rules)
def dnat_rules(self, opt: FirewallSettings | None) -> list[dict]:
"""Leave port-forwarded traffic to the rules that forwarded it, when no rule of
ours has matched it.
A consequence of owning the base chains that the move to them did not keep. In
iptables' own `FORWARD` chain, which is where firegex's rules used to be, Docker's
and podman's accept for a published port came first or came after firegex's jump
in the same chain — either way, traffic to a container's published port was
accepted without ever meeting firegex's policy. A base chain of our own is
evaluated on its own, so with the policy at drop every connection to a published
container port was dropped, on the CTF box where that is how every service is
reached. The rules chain still runs first, so a forward rule the operator wrote
applies to that traffic exactly as before; this only decides what the *policy*
does with what no rule matched, and the runtime's own chain still decides on its
own whether it accepts it.
"""
if opt is None or not opt.allow_dnat:
return []
matches = [
{"match": {"op": "in", "left": {"ct": {"key": "status"}}, "right": "dnat"}},
# What the containers send, to each other and out. Leaving only the published
# ports to the runtime was the first version, and with the policy at drop it
# cut a web container off from its database on the same bridge, and every
# container off from the network — neither of which met firegex's policy
# while its rules lived in iptables' `FORWARD` beside Docker's own accepts.
*({"match": {"op": "==", "left": {"meta": {"key": "iifname"}}, "right": bridge}}
for bridge in CONTAINER_BRIDGES),
]
return [
{"add": {"rule": {
"family": family, "table": self.filter_table, "chain": "fgex_forward",
"expr": [match, {"accept": None}],
}}}
for family in ("ip", "ip6")
for match in matches
]
def get_rules(self,*srvs:Rule):
rules = []
final_srvs:list[Rule] = []
for ele in srvs:
if ele.proto == Protocol.BOTH:
# Copies both ways: the rule handed in is the caller's, and rewriting its
# protocol in place turned a stored "both" into "tcp" for whoever read it
# next.
udp_rule = copy.deepcopy(ele)
udp_rule.proto = Protocol.UDP.value
ele.proto = Protocol.TCP.value
final_srvs.append(udp_rule)
tcp_rule = copy.deepcopy(ele)
tcp_rule.proto = Protocol.TCP.value
final_srvs.extend((udp_rule, tcp_rule))
continue
final_srvs.append(ele)
families = ["ip", "ip6"]
for srv in final_srvs:
# Per rule. It was set once, before the loop, and narrowed by the first rule
# naming an address — so every rule after that one was installed for that
# family only: an "accept port 80" written below an IPv4 rule did not exist
# for IPv6, and with the policy at drop every IPv6 client was refused.
families = ["ip", "ip6"]
ip_filters = []
if srv.src != "":
@ -297,7 +398,10 @@ class FiregexTables(NFTableManager):
for fam in families:
rules.append({ "add":{ "rule": {
"family": fam,
"table": srv.table,
# `srv.table` is the operator's word for what the rule does, kept in
# the database and in the API; the table it lands in is firegex's.
# The two were one string until that string was iptables' table.
"table": NFT_TABLES.get(srv.table, self.filter_table),
"chain": self.rules_chain_out if srv.output_mode else self.rules_chain_in if srv.input_mode else self.rules_chain_fwd,
"expr": ip_filters + port_filters + end_rules
}}})

248
backend/modules/nfproxy/firegex.py
View File

@ -1,248 +0,0 @@
from modules.nfproxy.nftables import FiregexTables
from modules.nfproxy.models import Service, PyFilter
import os
import asyncio
import traceback
from fastapi import HTTPException
import time
from utils import run_func
from utils import nicenessify
nft = FiregexTables()
MODULE = "nfproxy"
OUTSTREAM_BUFFER_SIZE = 1024*10
class FiregexInterceptor:
def __init__(self):
self.srv:Service
self.filter_map_lock:asyncio.Lock
self.filter_map: dict[str, PyFilter]
self.update_config_lock:asyncio.Lock
self.process:asyncio.subprocess.Process
self.update_task: asyncio.Task
self.server_task: asyncio.Task
self.sock_path: str
self.unix_sock: asyncio.Server
self.ack_arrived = False
self.ack_status = None
self.ack_fail_what = "Queue response timed-out"
self.ack_lock = asyncio.Lock()
self.sock_reader:asyncio.StreamReader = None
self.sock_writer:asyncio.StreamWriter = None
self.sock_conn_lock:asyncio.Lock
self.last_time_exception = 0
self.outstrem_function = None
self.expection_function = None
self.outstrem_task: asyncio.Task
self.outstrem_buffer = ""
self.watchdog_task: asyncio.Task = None
self.exit_callback = None
self.stopped = False
@classmethod
async def start(cls, srv: Service, outstream_func=None, exception_func=None, on_exit=None):
self = cls()
self.srv = srv
self.exit_callback = on_exit
self.filter_map_lock = asyncio.Lock()
self.update_config_lock = asyncio.Lock()
self.sock_conn_lock = asyncio.Lock()
self.outstrem_function = outstream_func
self.expection_function = exception_func
if not self.sock_conn_lock.locked():
await self.sock_conn_lock.acquire()
self.sock_path = f"/tmp/firegex_nfproxy_{srv.id}.sock"
if os.path.exists(self.sock_path):
os.remove(self.sock_path)
self.unix_sock = await asyncio.start_unix_server(self._server_listener,path=self.sock_path)
self.server_task = asyncio.create_task(self.unix_sock.serve_forever())
queue_range = await self._start_binary()
self.update_task = asyncio.create_task(self.update_stats())
self.watchdog_task = asyncio.create_task(self._watch_process())
nft.add(self.srv, queue_range)
if not self.ack_lock.locked():
await self.ack_lock.acquire()
return self
async def _stream_handler(self):
while True:
try:
out_data = (await self.process.stdout.read(1024*10)).decode(errors="ignore")
except asyncio.exceptions.LimitOverrunError:
self.outstrem_buffer = ""
continue
except Exception as e:
self.ack_arrived = False
self.ack_status = False
self.ack_fail_what = "Can't read from nfq client"
self.ack_lock.release()
await self.stop(expected=False)
traceback.print_exc() # Python can't print it alone? nope it's python... wasted 1 day :)
raise HTTPException(status_code=500, detail="Can't read from nfq client") from e
self.outstrem_buffer+=out_data
if len(self.outstrem_buffer) > OUTSTREAM_BUFFER_SIZE:
self.outstrem_buffer = self.outstrem_buffer[-OUTSTREAM_BUFFER_SIZE:]+"\n"
if self.outstrem_function:
await run_func(self.outstrem_function, self.srv.id, out_data)
async def _start_binary(self):
proxy_binary_path = os.path.abspath(os.path.join(os.path.dirname(__file__), "../cpproxy"))
self.process = await asyncio.create_subprocess_exec(
proxy_binary_path, stdin=asyncio.subprocess.DEVNULL,
stdout=asyncio.subprocess.PIPE,
stderr=asyncio.subprocess.STDOUT,
env=dict(os.environ, **{
"NTHREADS": os.getenv("NTHREADS","1"),
"FIREGEX_NFQUEUE_FAIL_OPEN": "1" if self.srv.fail_open else "0",
"FIREGEX_NFPROXY_SOCK": self.sock_path
}),
)
nicenessify(-10, self.process.pid)
self.outstrem_task = asyncio.create_task(self._stream_handler())
try:
async with asyncio.timeout(3):
await self.sock_conn_lock.acquire()
line_fut = await self.sock_reader.readuntil()
except asyncio.TimeoutError:
self.process.kill()
raise Exception("Binary don't returned queue number until timeout")
line = line_fut.decode()
if line.startswith("QUEUE "):
params = line.split()
return (int(params[1]), int(params[1]))
else:
self.process.kill()
raise Exception("Invalid binary output")
async def _server_listener(self, reader:asyncio.StreamReader, writer:asyncio.StreamWriter):
if self.sock_reader or self.sock_writer:
writer.write_eof() # Technically never reached
writer.close()
reader.feed_eof()
return
self.sock_reader = reader
self.sock_writer = writer
self.sock_conn_lock.release()
async def update_stats(self):
try:
while True:
try:
line = (await self.sock_reader.readuntil()).decode()
except Exception as e:
self.ack_arrived = False
self.ack_status = False
self.ack_fail_what = "Can't read from nfq client"
self.ack_lock.release()
await self.stop(expected=False)
raise HTTPException(status_code=500, detail="Can't read from nfq client") from e
if line.startswith("BLOCKED "):
filter_name = line.split()[1]
async with self.filter_map_lock:
if filter_name in self.filter_map:
self.filter_map[filter_name].blocked_packets+=1
await self.filter_map[filter_name].update()
if line.startswith("MANGLED "):
filter_name = line.split()[1]
async with self.filter_map_lock:
if filter_name in self.filter_map:
self.filter_map[filter_name].edited_packets+=1
await self.filter_map[filter_name].update()
if line.startswith("EXCEPTION"):
self.last_time_exception = int(time.time()*1000) #ms timestamp
if self.expection_function:
await run_func(self.expection_function, self.srv.id, self.last_time_exception)
if line.startswith("ACK "):
self.ack_arrived = True
self.ack_status = line.split()[1].upper() == "OK"
if not self.ack_status:
self.ack_fail_what = " ".join(line.split()[2:])
self.ack_lock.release()
except asyncio.CancelledError:
pass
except asyncio.IncompleteReadError:
pass
except Exception:
traceback.print_exc()
async def _watch_process(self):
"""Detects an unexpected death of the interceptor binary.
The nftables rules are installed independently of this process and
survive it, so a crashed interceptor leaves the service completely
unfiltered (silently accepted when the service is fail_open, dropped
otherwise) while the UI keeps reporting it as active. Nothing used to
notice: the stdout reader just saw EOF and returned.
"""
try:
returncode = await self.process.wait()
except asyncio.CancelledError:
return
if self.stopped:
return # We killed it ourselves
print(f"[error] [{MODULE}] The interceptor of service {self.srv.id} exited unexpectedly with code {returncode}")
if self.exit_callback:
try:
await run_func(self.exit_callback, returncode)
except asyncio.CancelledError:
raise
except Exception:
traceback.print_exc()
async def stop(self, expected: bool = True):
"""Tears the interceptor down.
`expected` tells the watchdog whether this shutdown was asked for: an
internal failure passes False so that the crash is still reported and
the service can be recovered, instead of looking like a clean stop.
"""
self.stopped = expected
# The watchdog itself calls stop() through the exit callback: cancelling
# the running task from inside would abort the recovery.
if self.watchdog_task and self.watchdog_task is not asyncio.current_task():
self.watchdog_task.cancel()
self.server_task.cancel()
self.update_task.cancel()
self.unix_sock.close()
self.outstrem_task.cancel()
if os.path.exists(self.sock_path):
os.remove(self.sock_path)
if self.process and self.process.returncode is None:
self.process.kill()
async def _update_config(self, code):
async with self.update_config_lock:
if self.sock_writer:
self.sock_writer.write(len(code).to_bytes(4, byteorder='big')+code.encode())
await self.sock_writer.drain()
try:
async with asyncio.timeout(3):
await self.ack_lock.acquire()
except TimeoutError:
self.ack_fail_what = "Queue response timed-out"
if not self.ack_arrived or not self.ack_status:
await self.stop()
raise HTTPException(status_code=500, detail=f"NFQ error: {self.ack_fail_what}")
else:
raise HTTPException(status_code=400, detail="Socket not ready")
async def reload(self, filters:list[PyFilter]):
async with self.filter_map_lock:
if os.path.exists(f"db/nfproxy_filters/{self.srv.id}.py"):
with open(f"db/nfproxy_filters/{self.srv.id}.py") as f:
filter_file = f.read()
else:
filter_file = ""
self.filter_map = {ele.name: ele for ele in filters}
await self._update_config(
filter_file + "\n\n" +
"__firegex_pyfilter_enabled = [" + ", ".join([repr(f.name) for f in filters]) + "]\n" +
"__firegex_proto = " + repr(self.srv.proto) + "\n" +
"import firegex.nfproxy.internals\n" +
"firegex.nfproxy.internals.compile(globals())\n"
)

185
backend/modules/nfproxy/firewall.py
View File

@ -1,185 +0,0 @@
import asyncio
import time
import traceback
from modules.nfproxy.firegex import FiregexInterceptor
from modules.nfproxy.nftables import FiregexTables, FiregexFilter
from modules.nfproxy.models import Service, PyFilter
from utils.sqlite import SQLite
from utils import run_func, socketio_emit
class STATUS:
STOP = "stop"
ACTIVE = "active"
nft = FiregexTables()
MODULE = "nfproxy"
# How many times a service may be brought back up before it is declared broken,
# and after how long without a crash the counter goes back to zero.
MAX_RESTART_ATTEMPTS = 5
CRASH_COUNTER_RESET_SECONDS = 60
class ServiceManager:
def __init__(self, srv: Service, db, outstream_func=None, exception_func=None):
self.srv = srv
self.db = db
self.status = STATUS.STOP
self.filters: dict[str, FiregexFilter] = {}
self.lock = asyncio.Lock()
self.interceptor = None
self._crash_count = 0
self._last_crash_time = 0.0
self.outstream_function = outstream_func
self.last_exception_time = 0
async def excep_internal_handler(srv, exc_time):
self.last_exception_time = exc_time
if exception_func:
await run_func(exception_func, srv, exc_time)
self.exception_function = excep_internal_handler
async def _update_filters_from_db(self):
pyfilters = [
PyFilter.from_dict(ele, self.db) for ele in
self.db.query("SELECT * FROM pyfilter WHERE service_id = ? AND active=1;", self.srv.id)
]
#Filter check
old_filters = set(self.filters.keys())
new_filters = set([f.name for f in pyfilters])
#remove old filters
for f in old_filters:
if f not in new_filters:
del self.filters[f]
#add new filters
for f in new_filters:
if f not in old_filters:
self.filters[f] = [ele for ele in pyfilters if ele.name == f][0]
if self.interceptor:
await self.interceptor.reload(self.filters.values())
def __update_status_db(self, status):
self.db.query("UPDATE services SET status = ? WHERE service_id = ?;", status, self.srv.id)
async def next(self,to,persist:bool=True):
async with self.lock:
if to == STATUS.STOP:
await self.stop(persist=persist)
if to == STATUS.ACTIVE:
await self.restart()
def _set_status(self,status,persist:bool=True):
self.status = status
if persist:
self.__update_status_db(status)
def read_outstrem_buffer(self):
if self.interceptor:
return self.interceptor.outstrem_buffer
else:
return ""
async def _on_interceptor_exit(self, returncode: int):
"""Recovers from an unexpected death of the interceptor binary.
Restarting rebuilds both the process and its nftables rules; it is
attempted a bounded number of times so that a systematically crashing
interceptor does not turn into a restart loop. When the budget is
exhausted the service is stopped for good, which at least removes the
rules and makes the failure visible instead of leaving a service that
claims to be active while filtering nothing.
"""
async with self.lock:
if self.interceptor is None or self.status != STATUS.ACTIVE:
return # Already being stopped on purpose
# The process is already gone, but its sockets and reader tasks are
# not: release them before building a replacement.
await self.interceptor.stop()
self.interceptor = None
now = time.monotonic()
if now - self._last_crash_time > CRASH_COUNTER_RESET_SECONDS:
self._crash_count = 0
self._last_crash_time = now
self._crash_count += 1
if self._crash_count > MAX_RESTART_ATTEMPTS:
print(f"[error] [{MODULE}] Service {self.srv.id} crashed {self._crash_count} times (last exit code {returncode}), giving up and stopping it")
await self.stop()
else:
print(f"[warning] [{MODULE}] Restarting the interceptor of service {self.srv.id} after exit code {returncode} ({self._crash_count}/{MAX_RESTART_ATTEMPTS})")
try:
await self.start()
except Exception:
traceback.print_exc()
await self.stop()
await socketio_emit([MODULE])
async def start(self):
if not self.interceptor:
nft.delete(self.srv)
self.interceptor = await FiregexInterceptor.start(self.srv, outstream_func=self.outstream_function, exception_func=self.exception_function, on_exit=self._on_interceptor_exit)
await self._update_filters_from_db()
self._set_status(STATUS.ACTIVE)
async def stop(self,persist:bool=True):
nft.delete(self.srv)
if self.interceptor:
await self.interceptor.stop()
self.interceptor = None
self._set_status(STATUS.STOP,persist=persist)
async def restart(self):
await self.stop()
await self.start()
async def update_filters(self):
async with self.lock:
await self._update_filters_from_db()
class FirewallManager:
def __init__(self, db:SQLite, outstream_func=None, exception_func=None):
self.db = db
self.service_table: dict[str, ServiceManager] = {}
self.lock = asyncio.Lock()
self.outstream_function = outstream_func
self.exception_function = exception_func
async def close(self):
for key in list(self.service_table.keys()):
try:
await self.remove(key, persist=False)
except Exception:
# Don't let one broken service block shutdown of the others
self.service_table.pop(key, None)
async def remove(self,srv_id,persist:bool=True):
async with self.lock:
if srv_id in self.service_table:
await self.service_table[srv_id].next(STATUS.STOP,persist=persist)
del self.service_table[srv_id]
async def init(self):
nft.init()
await self.reload()
async def reload(self):
async with self.lock:
services = self.db.query('SELECT * FROM services;')
for srv in services:
if srv["service_id"] in self.service_table:
continue
srv_obj = Service.from_dict(srv)
self.service_table[srv_obj.id] = ServiceManager(srv_obj, self.db, outstream_func=self.outstream_function, exception_func=self.exception_function)
await self.service_table[srv_obj.id].next(srv_obj.status)
def get(self,srv_id) -> ServiceManager:
if srv_id in self.service_table:
return self.service_table[srv_id]
else:
raise ServiceNotFoundException()
class ServiceNotFoundException(Exception):
pass

66
backend/modules/nfproxy/models.py
View File

@ -1,66 +0,0 @@
class Service:
def __init__(
self,
service_id: str,
status: str,
port: int | None,
name: str,
proto: str,
ip_int: str | None,
fail_open: bool,
target_type: str = "flow",
tls_stream_id: str | None = None,
tls_cert: str | None = None,
tls_key: str | None = None,
**other,
):
self.id = service_id
self.status = status
self.port = port
self.name = name
self.proto = proto
self.ip_int = ip_int
self.fail_open = fail_open
self.target_type = target_type
self.tls_stream_id = tls_stream_id
self.tls_cert = tls_cert
self.tls_key = tls_key
@classmethod
def from_dict(cls, var: dict):
return cls(**var)
class PyFilter:
def __init__(
self,
name: str,
service_id: str,
blocked_packets: int,
edited_packets: int,
active: bool,
db,
**other,
):
self.name = name
self.blocked_packets = blocked_packets
self.edited_packets = edited_packets
self.active = active
self.__db = db
self.srv_id = service_id
async def update(self):
self.__db.query(
"UPDATE pyfilter SET blocked_packets = ?, edited_packets = ? WHERE name = ? AND service_id = ?;",
self.blocked_packets,
self.edited_packets,
self.name,
self.srv_id,
)
def __repr__(self):
return f"<PyFilter {self.name}>"
@classmethod
def from_dict(cls, var: dict, db):
return cls(**var, db=db)

169
backend/modules/nfproxy/nftables.py
View File

@ -1,169 +0,0 @@
from modules.nfproxy.models import Service
from utils import ip_parse, ip_family, NFTableManager, nftables_int_to_json
from modules.tls.manager import get_tls_ports, TLSManager
def convert_protocol_to_l4(proto:str):
if proto == "tcp":
return "tcp"
elif proto == "http":
return "tcp"
else:
raise Exception("Invalid protocol")
def resolve_target(srv: Service) -> tuple[str, int] | None:
"""Returns the (ip_int, port) a service's intercept rule should actually match.
For target_type=='tls' this is the loopback clear_port where nginx exposes the
decrypted plaintext, not the service's own (real backend) ip_int/port. Returns
None if the linked TLS stream can't be resolved (e.g. it was deleted)."""
if srv.target_type == "tls":
tls_stream = TLSManager().get_stream(srv.tls_stream_id)
if not tls_stream:
return None
_, clear_port = get_tls_ports(tls_stream["ip_int"], tls_stream["port"])
loopback_ip = "::1" if ip_family(tls_stream["ip_int"]) == "ip6" else "127.0.0.1"
return loopback_ip, clear_port
return srv.ip_int, srv.port
def build_queue_statement(srv: Service, init: int, end: int) -> dict:
"""Builds the nftables `queue` statement for a service.
The `bypass` flag tells the kernel to accept packets when no process is
bound to the queue, which is what happens after the interceptor binary
dies. Tying it to the service's fail_open setting keeps the two halves of
the policy consistent: without fail_open a dead interceptor must not
silently let the traffic through unfiltered.
"""
statement = {"num": str(init) if init == end else {"range": [init, end]}}
if srv.fail_open:
statement["flags"] = ["bypass"]
return {"queue": statement}
class FiregexFilter:
def __init__(self, proto:str, port:int, ip_int:str, target:str, id:int):
self.id = id
self.target = target
self.proto = proto
self.port = int(port)
self.ip_int = str(ip_int)
def __eq__(self, o: object) -> bool:
if isinstance(o, FiregexFilter):
return self.port == o.port and self.proto == o.proto and ip_parse(self.ip_int) == ip_parse(o.ip_int)
if isinstance(o, Service):
return self.matches(o)
return False
def matches(self, srv: Service) -> bool:
if self.proto != convert_protocol_to_l4(srv.proto):
return False
target = resolve_target(srv)
if target is None:
return False
target_ip, target_port = target
return self.port == target_port and ip_parse(self.ip_int) == ip_parse(target_ip)
class FiregexTables(NFTableManager):
input_chain = "nfproxy_input"
output_chain = "nfproxy_output"
nat_chain = "nfproxy_nat"
def __init__(self):
super().__init__([
{"add":{"chain":{ #Input chain attached before conntrack see it
"family":"inet",
"table":self.table_name,
"name":self.input_chain,
"type":"filter",
"hook":"prerouting",
"prio":-308,
"policy":"accept"
}}},
{"add":{"chain":{ #Output chain attached after conntrack saw it
"family":"inet",
"table":self.table_name,
"name":self.output_chain,
"type":"filter",
"hook":"postrouting",
"prio":108,
"policy":"accept"
}}}
],[
{"flush":{"chain":{"table":self.table_name,"family":"inet", "name":self.input_chain}}},
{"delete":{"chain":{"table":self.table_name,"family":"inet", "name":self.input_chain}}},
{"flush":{"chain":{"table":self.table_name,"family":"inet", "name":self.output_chain}}},
{"delete":{"chain":{"table":self.table_name,"family":"inet", "name":self.output_chain}}},
])
def add(self, srv:Service, queue_range):
for ele in self.get():
if ele.matches(srv):
return
target = resolve_target(srv)
if target is None:
return
target_ip, target_port = target
init, end = queue_range
if init > end:
init, end = end, init
self.cmd(
{ "insert":{ "rule": { # Send non-TLS outbound traffic to NFQUEUE (Client -> Server)
"family": "inet",
"table": self.table_name,
"chain": self.output_chain,
"expr": [
{'match': {'left': {'payload': {'protocol': ip_family(target_ip), 'field': 'saddr'}}, 'op': '==', 'right': nftables_int_to_json(target_ip)}},
{'match': {"left": { "payload": {"protocol": convert_protocol_to_l4(str(srv.proto)), "field": "dport"}}, "op": "==", "right": int(target_port)}},
{"mangle": {"key": {"meta": {"key": "mark"}},"value": 0x1338}},
build_queue_statement(srv, init, end)
]
}}},
{"insert":{"rule":{ # Send non-TLS inbound traffic to NFQUEUE (Server -> Client)
"family": "inet",
"table": self.table_name,
"chain": self.input_chain,
"expr": [
{'match': {'left': {'payload': {'protocol': ip_family(target_ip), 'field': 'saddr'}}, 'op': '==', 'right': nftables_int_to_json(target_ip)}},
{'match': {"left": { "payload": {"protocol": convert_protocol_to_l4(str(srv.proto)), "field": "sport"}}, "op": "==", "right": int(target_port)}},
{"mangle": {"key": {"meta": {"key": "mark"}},"value": 0x1337}},
build_queue_statement(srv, init, end)
]
}}}
)
def get(self) -> list[FiregexFilter]:
res = []
for filter in self.list_rules(tables=[self.table_name], chains=[self.input_chain,self.output_chain]):
try:
ip_int = None
if isinstance(filter["expr"][0]["match"]["right"],str):
ip_int = str(ip_parse(filter["expr"][0]["match"]["right"]))
else:
ip_int = f'{filter["expr"][0]["match"]["right"]["prefix"]["addr"]}/{filter["expr"][0]["match"]["right"]["prefix"]["len"]}'
res.append(FiregexFilter(
target=filter["chain"],
id=int(filter["handle"]),
proto=filter["expr"][1]["match"]["left"]["payload"]["protocol"],
port=filter["expr"][1]["match"]["right"],
ip_int=ip_int
))
except (KeyError, TypeError, IndexError):
pass # Rule has unexpected structure (e.g., intermediate NAT rule), skip
return res
def delete(self, srv:Service):
for filter in self.get():
if filter.matches(srv):
self.cmd({ "delete":{ "rule": {
"family": "inet",
"table": self.table_name,
"chain": filter.target,
"handle": filter.id
}}})

233
backend/modules/nfregex/firegex.py
View File

@ -1,233 +0,0 @@
from modules.nfregex.nftables import FiregexTables
from utils import run_func
from modules.nfregex.models import Service, Regex
import os
import asyncio
import traceback
from utils import DEBUG
from fastapi import HTTPException
from utils import nicenessify
nft = FiregexTables()
MODULE = "nfregex"
async def test_regex_validity(regex: str) -> bool:
proxy_binary_path = os.path.join(os.path.dirname(os.path.abspath(__file__)),"../cppregex")
process = await asyncio.create_subprocess_exec(
proxy_binary_path,
stdout=asyncio.subprocess.PIPE,
stdin=asyncio.subprocess.DEVNULL,
env={"FIREGEX_TEST_REGEX": regex},
)
await process.wait()
if process.returncode != 0:
message = (await process.stdout.read()).decode()
return False, message
return True, "ok"
class RegexFilter:
def __init__(
self, regex,
is_case_sensitive=True,
input_mode=False,
output_mode=False,
blocked_packets=0,
id=None,
update_func = None
):
self.regex = regex
self.is_case_sensitive = is_case_sensitive
if input_mode == output_mode:
input_mode = output_mode = True # (False, False) == (True, True)
self.input_mode = input_mode
self.output_mode = output_mode
self.blocked = blocked_packets
self.id = id
self.update_func = update_func
self.compiled_regex = self.compile()
@classmethod
def from_regex(cls, regex:Regex, update_func = None):
return cls(
id=regex.id, regex=regex.regex, is_case_sensitive=regex.is_case_sensitive,
blocked_packets=regex.blocked_packets,
input_mode = regex.mode in ["C","B"], output_mode=regex.mode in ["S","B"],
update_func = update_func
)
def compile(self):
if isinstance(self.regex, str):
self.regex = self.regex.encode()
if not isinstance(self.regex, bytes):
raise Exception("Invalid Regex Paramether")
case_sensitive = "1" if self.is_case_sensitive else "0"
if self.input_mode:
yield case_sensitive + "C" + self.regex.hex()
if self.output_mode:
yield case_sensitive + "S" + self.regex.hex()
async def update(self):
if self.update_func:
await run_func(self.update_func, self)
class FiregexInterceptor:
def __init__(self):
self.srv:Service
self.filter_map_lock:asyncio.Lock
self.filter_map: dict[str, RegexFilter]
self.regex_filters: set[RegexFilter]
self.update_config_lock:asyncio.Lock
self.process:asyncio.subprocess.Process
self.update_task: asyncio.Task
self.ack_arrived = False
self.ack_status = None
self.ack_fail_what = "Queue response timed-out"
self.ack_lock = asyncio.Lock()
self.watchdog_task: asyncio.Task = None
self.exit_callback = None
self.stopped = False
@classmethod
async def start(cls, srv: Service, on_exit=None):
self = cls()
self.srv = srv
self.exit_callback = on_exit
self.filter_map_lock = asyncio.Lock()
self.update_config_lock = asyncio.Lock()
queue_range = await self._start_binary()
self.update_task = asyncio.create_task(self.update_blocked())
self.watchdog_task = asyncio.create_task(self._watch_process())
nft.add(self.srv, queue_range)
if not self.ack_lock.locked():
await self.ack_lock.acquire()
return self
async def _start_binary(self):
proxy_binary_path = os.path.join(os.path.dirname(os.path.abspath(__file__)),"../cppregex")
self.process = await asyncio.create_subprocess_exec(
proxy_binary_path,
stdout=asyncio.subprocess.PIPE, stdin=asyncio.subprocess.PIPE,
env=dict(os.environ, **{
"MATCH_MODE": "stream" if self.srv.proto == "tcp" else "block",
"NTHREADS": os.getenv("NTHREADS","1"),
"FIREGEX_NFQUEUE_FAIL_OPEN": "1" if self.srv.fail_open else "0",
}),
)
nicenessify(-10, self.process.pid)
line_fut = self.process.stdout.readuntil()
try:
line_fut = await asyncio.wait_for(line_fut, timeout=3)
except asyncio.TimeoutError:
self.process.kill()
raise Exception("Invalid binary output")
line = line_fut.decode()
if line.startswith("QUEUE "):
params = line.split()
return (int(params[1]), int(params[1]))
else:
self.process.kill()
raise Exception("Invalid binary output")
async def update_blocked(self):
try:
while True:
line = (await self.process.stdout.readuntil()).decode()
if DEBUG:
print(line)
if line.startswith("BLOCKED "):
regex_id = line.split()[1]
async with self.filter_map_lock:
if regex_id in self.filter_map:
self.filter_map[regex_id].blocked+=1
await self.filter_map[regex_id].update()
if line.startswith("ACK "):
self.ack_arrived = True
self.ack_status = line.split()[1].upper() == "OK"
if not self.ack_status:
self.ack_fail_what = " ".join(line.split()[2:])
self.ack_lock.release()
except asyncio.CancelledError:
pass
except asyncio.IncompleteReadError:
pass
except Exception:
traceback.print_exc()
async def _watch_process(self):
"""Detects an unexpected death of the interceptor binary.
The nftables rules are installed independently of this process and
survive it, so a crashed interceptor leaves the service completely
unfiltered (silently accepted when the service is fail_open, dropped
otherwise) while the UI keeps reporting it as active. Nothing used to
notice: the stdout reader just saw EOF and returned.
"""
try:
returncode = await self.process.wait()
except asyncio.CancelledError:
return
if self.stopped:
return # We killed it ourselves
print(f"[error] [{MODULE}] The interceptor of service {self.srv.id} exited unexpectedly with code {returncode}")
if self.exit_callback:
try:
await run_func(self.exit_callback, returncode)
except asyncio.CancelledError:
raise
except Exception:
traceback.print_exc()
async def stop(self, expected: bool = True):
"""Tears the interceptor down.
`expected` tells the watchdog whether this shutdown was asked for: an
internal failure passes False so that the crash is still reported and
the service can be recovered, instead of looking like a clean stop.
"""
self.stopped = expected
# The watchdog itself calls stop() through the exit callback: cancelling
# the running task from inside would abort the recovery.
if self.watchdog_task and self.watchdog_task is not asyncio.current_task():
self.watchdog_task.cancel()
self.update_task.cancel()
if self.process and self.process.returncode is None:
self.process.kill()
async def _update_config(self, filters_codes):
async with self.update_config_lock:
self.process.stdin.write((" ".join(filters_codes)+"\n").encode())
await self.process.stdin.drain()
try:
async with asyncio.timeout(3):
await self.ack_lock.acquire()
except TimeoutError:
self.ack_fail_what = "Queue response timed-out"
if not self.ack_arrived or not self.ack_status:
await self.stop()
raise HTTPException(status_code=500, detail=f"NFQ error: {self.ack_fail_what}")
async def reload(self, filters:list[RegexFilter]):
async with self.filter_map_lock:
self.filter_map = self.compile_filters(filters)
filters_codes = self.get_filter_codes()
await self._update_config(filters_codes)
def get_filter_codes(self):
filters_codes = list(self.filter_map.keys())
filters_codes.sort(key=lambda a: self.filter_map[a].blocked, reverse=True)
return filters_codes
def compile_filters(self, filters:list[RegexFilter]):
res = {}
for filter_obj in filters:
try:
raw_filters = filter_obj.compile()
for filter in raw_filters:
res[filter] = filter_obj
except Exception:
pass
return res

173
backend/modules/nfregex/firewall.py
View File

@ -1,173 +0,0 @@
import asyncio
import time
import traceback
from modules.nfregex.firegex import FiregexInterceptor, RegexFilter
from modules.nfregex.nftables import FiregexTables, FiregexFilter
from modules.nfregex.models import Regex, Service
from utils.sqlite import SQLite
from utils import socketio_emit
class STATUS:
STOP = "stop"
ACTIVE = "active"
nft = FiregexTables()
MODULE = "nfregex"
# How many times a service may be brought back up before it is declared broken,
# and after how long without a crash the counter goes back to zero.
MAX_RESTART_ATTEMPTS = 5
CRASH_COUNTER_RESET_SECONDS = 60
class ServiceManager:
def __init__(self, srv: Service, db):
self.srv = srv
self.db = db
self.status = STATUS.STOP
self.filters: dict[int, FiregexFilter] = {}
self.lock = asyncio.Lock()
self.interceptor = None
self._crash_count = 0
self._last_crash_time = 0.0
async def _update_filters_from_db(self):
regexes = [
Regex.from_dict(ele) for ele in
self.db.query("SELECT * FROM regexes WHERE service_id = ? AND active=1;", self.srv.id)
]
#Filter check
old_filters = set(self.filters.keys())
new_filters = set([f.id for f in regexes])
#remove old filters
for f in old_filters:
if f not in new_filters:
del self.filters[f]
#add new filters
for f in new_filters:
if f not in old_filters:
filter = [ele for ele in regexes if ele.id == f][0]
self.filters[f] = RegexFilter.from_regex(filter, self._stats_updater)
if self.interceptor:
await self.interceptor.reload(self.filters.values())
def __update_status_db(self, status):
self.db.query("UPDATE services SET status = ? WHERE service_id = ?;", status, self.srv.id)
async def next(self,to,persist:bool=True):
async with self.lock:
if to == STATUS.STOP:
await self.stop(persist=persist)
if to == STATUS.ACTIVE:
await self.restart()
def _stats_updater(self,filter:RegexFilter):
self.db.query("UPDATE regexes SET blocked_packets = ? WHERE regex_id = ?;", filter.blocked, filter.id)
def _set_status(self,status,persist:bool=True):
self.status = status
if persist:
self.__update_status_db(status)
async def _on_interceptor_exit(self, returncode: int):
"""Recovers from an unexpected death of the interceptor binary.
Restarting rebuilds both the process and its nftables rules; it is
attempted a bounded number of times so that a systematically crashing
interceptor does not turn into a restart loop. When the budget is
exhausted the service is stopped for good, which at least removes the
rules and makes the failure visible instead of leaving a service that
claims to be active while filtering nothing.
"""
async with self.lock:
if self.interceptor is None or self.status != STATUS.ACTIVE:
return # Already being stopped on purpose
# The process is already gone, but its sockets and reader tasks are
# not: release them before building a replacement.
await self.interceptor.stop()
self.interceptor = None
now = time.monotonic()
if now - self._last_crash_time > CRASH_COUNTER_RESET_SECONDS:
self._crash_count = 0
self._last_crash_time = now
self._crash_count += 1
if self._crash_count > MAX_RESTART_ATTEMPTS:
print(f"[error] [{MODULE}] Service {self.srv.id} crashed {self._crash_count} times (last exit code {returncode}), giving up and stopping it")
await self.stop()
else:
print(f"[warning] [{MODULE}] Restarting the interceptor of service {self.srv.id} after exit code {returncode} ({self._crash_count}/{MAX_RESTART_ATTEMPTS})")
try:
await self.start()
except Exception:
traceback.print_exc()
await self.stop()
await socketio_emit([MODULE])
async def start(self):
if not self.interceptor:
nft.delete(self.srv)
self.interceptor = await FiregexInterceptor.start(self.srv, on_exit=self._on_interceptor_exit)
await self._update_filters_from_db()
self._set_status(STATUS.ACTIVE)
async def stop(self,persist:bool=True):
nft.delete(self.srv)
if self.interceptor:
await self.interceptor.stop()
self.interceptor = None
self._set_status(STATUS.STOP,persist=persist)
async def restart(self):
await self.stop()
await self.start()
async def update_filters(self):
async with self.lock:
await self._update_filters_from_db()
class FirewallManager:
def __init__(self, db:SQLite):
self.db = db
self.service_table: dict[str, ServiceManager] = {}
self.lock = asyncio.Lock()
async def close(self):
for key in list(self.service_table.keys()):
try:
await self.remove(key, persist=False)
except Exception:
# Don't let one broken service block shutdown of the others
self.service_table.pop(key, None)
async def remove(self,srv_id,persist:bool=True):
async with self.lock:
if srv_id in self.service_table:
await self.service_table[srv_id].next(STATUS.STOP,persist=persist)
del self.service_table[srv_id]
async def init(self):
nft.init()
await self.reload()
async def reload(self):
async with self.lock:
services = self.db.query('SELECT * FROM services;')
for srv in services:
if srv["service_id"] in self.service_table:
continue
srv_obj = Service.from_dict(srv)
self.service_table[srv_obj.id] = ServiceManager(srv_obj, self.db)
await self.service_table[srv_obj.id].next(srv_obj.status)
def get(self,srv_id) -> ServiceManager:
if srv_id in self.service_table:
return self.service_table[srv_id]
else:
raise ServiceNotFoundException()
class ServiceNotFoundException(Exception):
pass

35
backend/modules/nfregex/models.py
View File

@ -1,35 +0,0 @@
import base64
class Service:
def __init__(self, service_id: str, status: str, port: int | None, name: str, proto: str, ip_int: str | None, fail_open: bool, target_type: str = "flow", tls_stream_id: str | None = None, tls_cert: str | None = None, tls_key: str | None = None, **other):
self.id = service_id
self.status = status
self.port = port
self.name = name
self.proto = proto
self.ip_int = ip_int
self.fail_open = fail_open
self.target_type = target_type
self.tls_stream_id = tls_stream_id
self.tls_cert = tls_cert
self.tls_key = tls_key
@classmethod
def from_dict(cls, var: dict):
return cls(**var)
class Regex:
def __init__(self, regex_id: int, regex: bytes, mode: str, service_id: str, blocked_packets: int, is_case_sensitive: bool, active: bool, **other):
self.regex = regex
self.mode = mode
self.service_id = service_id
self.blocked_packets = blocked_packets
self.id = regex_id
self.is_case_sensitive = is_case_sensitive
self.active = active
@classmethod
def from_dict(cls, var: dict):
var['regex'] = base64.b64decode(var['regex'])
return cls(**var)

164
backend/modules/nfregex/nftables.py
View File

@ -1,164 +0,0 @@
from modules.nfregex.models import Service
from utils import ip_parse, ip_family, NFTableManager, nftables_int_to_json
from modules.tls.manager import get_tls_ports, TLSManager
def resolve_target(srv: Service) -> tuple[str, int] | None:
"""Returns the (ip_int, port) a service's intercept rule should actually match.
For target_type=='tls' this is the loopback clear_port where nginx exposes the
decrypted plaintext, not the service's own (real backend) ip_int/port. Returns
None if the linked TLS stream can't be resolved (e.g. it was deleted)."""
if srv.target_type == "tls":
tls_stream = TLSManager().get_stream(srv.tls_stream_id)
if not tls_stream:
return None
_, clear_port = get_tls_ports(tls_stream["ip_int"], tls_stream["port"])
loopback_ip = "::1" if ip_family(tls_stream["ip_int"]) == "ip6" else "127.0.0.1"
return loopback_ip, clear_port
return srv.ip_int, srv.port
def build_queue_statement(srv: Service, init: int, end: int) -> dict:
"""Builds the nftables `queue` statement for a service.
The `bypass` flag tells the kernel to accept packets when no process is
bound to the queue, which is what happens after the interceptor binary
dies. Tying it to the service's fail_open setting keeps the two halves of
the policy consistent: without fail_open a dead interceptor must not
silently let the traffic through unfiltered.
"""
statement = {"num": str(init) if init == end else {"range": [init, end]}}
if srv.fail_open:
statement["flags"] = ["bypass"]
return {"queue": statement}
class FiregexFilter:
def __init__(self, proto:str, port:int, ip_int:str, target:str, id:int):
self.id = id
self.target = target
self.proto = proto
self.port = int(port)
self.ip_int = str(ip_int)
def __eq__(self, o: object) -> bool:
if isinstance(o, FiregexFilter):
return self.port == o.port and self.proto == o.proto and ip_parse(self.ip_int) == ip_parse(o.ip_int)
if isinstance(o, Service):
return self.matches(o)
return False
def matches(self, srv: Service) -> bool:
if self.proto != str(srv.proto):
return False
target = resolve_target(srv)
if target is None:
return False
target_ip, target_port = target
return self.port == target_port and ip_parse(self.ip_int) == ip_parse(target_ip)
class FiregexTables(NFTableManager):
input_chain = "nfregex_input"
output_chain = "nfregex_output"
nat_chain = "nfregex_nat"
def __init__(self):
super().__init__([
{"add":{"chain":{
"family":"inet",
"table":self.table_name,
"name":self.input_chain,
"type":"filter",
"hook":"prerouting",
"prio":-307,
"policy":"accept"
}}},
{"add":{"chain":{
"family":"inet",
"table":self.table_name,
"name":self.output_chain,
"type":"filter",
"hook":"postrouting",
"prio":107,
"policy":"accept"
}}}
],[
{"flush":{"chain":{"table":self.table_name,"family":"inet", "name":self.input_chain}}},
{"delete":{"chain":{"table":self.table_name,"family":"inet", "name":self.input_chain}}},
{"flush":{"chain":{"table":self.table_name,"family":"inet", "name":self.output_chain}}},
{"delete":{"chain":{"table":self.table_name,"family":"inet", "name":self.output_chain}}},
])
def add(self, srv:Service, queue_range):
for ele in self.get():
if ele.matches(srv):
return
target = resolve_target(srv)
if target is None:
return
target_ip, target_port = target
init, end = queue_range
if init > end:
init, end = end, init
self.cmd(
{ "insert":{ "rule": {
"family": "inet",
"table": self.table_name,
"chain": self.output_chain,
"expr": [
{'match': {'left': {'payload': {'protocol': ip_family(target_ip), 'field': 'saddr'}}, 'op': '==', 'right': nftables_int_to_json(target_ip)}},
{'match': {"left": { "payload": {"protocol": str(srv.proto), "field": "sport"}}, "op": "==", "right": int(target_port)}},
{"mangle": {"key": {"meta": {"key": "mark"}},"value": 0x1338}},
build_queue_statement(srv, init, end)
]
}}},
{"insert":{"rule":{
"family": "inet",
"table": self.table_name,
"chain": self.input_chain,
"expr": [
{'match': {'left': {'payload': {'protocol': ip_family(target_ip), 'field': 'daddr'}}, 'op': '==', 'right': nftables_int_to_json(target_ip)}},
{'match': {"left": { "payload": {"protocol": str(srv.proto), "field": "dport"}}, "op": "==", "right": int(target_port)}},
{"mangle": {"key": {"meta": {"key": "mark"}},"value": 0x1337}},
build_queue_statement(srv, init, end)
]
}}}
)
def get(self) -> list[FiregexFilter]:
res = []
for filter in self.list_rules(tables=[self.table_name], chains=[self.input_chain,self.output_chain]):
try:
ip_int = None
if isinstance(filter["expr"][0]["match"]["right"],str):
ip_int = str(ip_parse(filter["expr"][0]["match"]["right"]))
else:
ip_int = f'{filter["expr"][0]["match"]["right"]["prefix"]["addr"]}/{filter["expr"][0]["match"]["right"]["prefix"]["len"]}'
res.append(FiregexFilter(
target=filter["chain"],
id=int(filter["handle"]),
proto=filter["expr"][1]["match"]["left"]["payload"]["protocol"],
port=filter["expr"][1]["match"]["right"],
ip_int=ip_int
))
except (KeyError, TypeError, IndexError):
pass # Rule has unexpected structure (e.g., intermediate NAT rule), skip
return res
def delete(self, srv:Service):
delete_cmds = []
for filter in self.get():
if filter.matches(srv):
delete_cmds.append({ "delete":{ "rule": {
"family": "inet",
"table": self.table_name,
"chain": filter.target,
"handle": filter.id
}}})
if delete_cmds:
self.cmd(*delete_cmds)

0
backend/modules/porthijack/__init__.py
View File

84
backend/modules/porthijack/firewall.py
View File

@ -1,84 +0,0 @@
import asyncio
from modules.porthijack.nftables import FiregexTables
from modules.porthijack.models import Service
from utils.sqlite import SQLite
nft = FiregexTables()
class ServiceNotFoundException(Exception):
pass
class ServiceManager:
def __init__(self, srv: Service, db):
self.srv = srv
self.db = db
self.active = False
self.lock = asyncio.Lock()
async def enable(self):
if not self.active:
async with self.lock:
nft.delete(self.srv)
nft.add(self.srv)
self._set_status(True)
async def disable(self,persist:bool=True):
if self.active:
async with self.lock:
nft.delete(self.srv)
self._set_status(False,persist=persist)
async def refresh(self, srv:Service):
self.srv = srv
if self.active:
await self.restart()
def _set_status(self,active,persist:bool=True):
self.active = active
if persist:
self.db.query("UPDATE services SET active = ? WHERE service_id = ?;", active, self.srv.service_id)
async def restart(self):
await self.disable()
await self.enable()
class FirewallManager:
def __init__(self, db:SQLite):
self.db = db
self.service_table: dict[str, ServiceManager] = {}
self.lock = asyncio.Lock()
async def close(self):
for key in list(self.service_table.keys()):
try:
await self.remove(key, persist=False)
except Exception:
# Don't let one broken service block shutdown of the others
self.service_table.pop(key, None)
async def remove(self,srv_id,persist:bool=True):
async with self.lock:
if srv_id in self.service_table:
await self.service_table[srv_id].disable(persist=persist)
del self.service_table[srv_id]
async def init(self):
nft.init()
await self.reload()
async def reload(self):
async with self.lock:
for srv in self.db.query('SELECT * FROM services;'):
srv = Service.from_dict(srv)
if srv.service_id in self.service_table:
continue
self.service_table[srv.service_id] = ServiceManager(srv, self.db)
if srv.active:
await self.service_table[srv.service_id].enable()
def get(self,srv_id) -> ServiceManager:
if srv_id in self.service_table:
return self.service_table[srv_id]
else:
raise ServiceNotFoundException()

14
backend/modules/porthijack/models.py
View File

@ -1,14 +0,0 @@
class Service:
def __init__(self, service_id: str, active: bool, public_port: int, proxy_port: int, name: str, proto: str, ip_src: str, ip_dst:str, **other):
self.service_id = service_id
self.active = active
self.public_port = public_port
self.proxy_port = proxy_port
self.name = name
self.proto = proto
self.ip_src = ip_src
self.ip_dst = ip_dst
@classmethod
def from_dict(cls, var: dict):
return cls(**var)

102
backend/modules/porthijack/nftables.py
View File

@ -1,102 +0,0 @@
from modules.porthijack.models import Service
from utils import addr_parse, ip_parse, ip_family, NFTableManager, nftables_json_to_int
class FiregexHijackRule():
def __init__(self, proto:str, public_port:int,proxy_port:int, ip_src:str, ip_dst:str, target:str, id:int):
self.id = id
self.target = target
self.proto = proto
self.public_port = public_port
self.proxy_port = proxy_port
self.ip_src = str(ip_src)
self.ip_dst = str(ip_dst)
def __eq__(self, o: object) -> bool:
if isinstance(o, FiregexHijackRule) or isinstance(o, Service):
return self.public_port == o.public_port and self.proto == o.proto and ip_parse(self.ip_src) == ip_parse(o.ip_src)
return False
class FiregexTables(NFTableManager):
prerouting_porthijack = "prerouting_porthijack"
postrouting_porthijack = "postrouting_porthijack"
def __init__(self):
super().__init__([
{"add":{"chain":{
"family":"inet",
"table":self.table_name,
"name":self.prerouting_porthijack,
"type":"filter",
"hook":"prerouting",
"prio":-310,
"policy":"accept"
}}},
{"add":{"chain":{
"family":"inet",
"table":self.table_name,
"name":self.postrouting_porthijack,
"type":"filter",
"hook":"postrouting",
"prio":110,
"policy":"accept"
}}}
],[
{"flush":{"chain":{"table":self.table_name,"family":"inet", "name":self.prerouting_porthijack}}},
{"delete":{"chain":{"table":self.table_name,"family":"inet", "name":self.prerouting_porthijack}}},
{"flush":{"chain":{"table":self.table_name,"family":"inet", "name":self.postrouting_porthijack}}},
{"delete":{"chain":{"table":self.table_name,"family":"inet", "name":self.postrouting_porthijack}}}
])
def add(self, srv:Service):
for ele in self.get():
if ele.__eq__(srv):
return
self.cmd({ "insert":{ "rule": {
"family": "inet",
"table": self.table_name,
"chain": self.prerouting_porthijack,
"expr": [
{'match': {'left': {'payload': {'protocol': ip_family(srv.ip_src), 'field': 'daddr'}}, 'op': '==', 'right': addr_parse(srv.ip_src)}},
{'match': {'left': {'payload': {'protocol': str(srv.proto), 'field': 'dport'}}, 'op': '==', 'right': int(srv.public_port)}},
{'mangle': {'key': {'payload': {'protocol': str(srv.proto), 'field': 'dport'}}, 'value': int(srv.proxy_port)}},
{'mangle': {'key': {'payload': {'protocol': ip_family(srv.ip_src), 'field': 'daddr'}}, 'value': addr_parse(srv.ip_dst)}}
]
}}})
self.cmd({ "insert":{ "rule": {
"family": "inet",
"table": self.table_name,
"chain": self.postrouting_porthijack,
"expr": [
{'match': {'left': {'payload': {'protocol': ip_family(srv.ip_dst), 'field': 'saddr'}}, 'op': '==', 'right': addr_parse(srv.ip_dst)}},
{'match': {'left': { "payload": {"protocol": str(srv.proto), "field": "sport"}}, "op": "==", "right": int(srv.proxy_port)}},
{'mangle': {'key': {'payload': {'protocol': str(srv.proto), 'field': 'sport'}}, 'value': int(srv.public_port)}},
{'mangle': {'key': {'payload': {'protocol': ip_family(srv.ip_dst), 'field': 'saddr'}}, 'value': addr_parse(srv.ip_src)}}
]
}}})
def get(self) -> list[FiregexHijackRule]:
res = []
for filter in self.list_rules(tables=[self.table_name], chains=[self.prerouting_porthijack,self.postrouting_porthijack]):
res.append(FiregexHijackRule(
target=filter["chain"],
id=int(filter["handle"]),
proto=filter["expr"][1]["match"]["left"]["payload"]["protocol"],
public_port=filter["expr"][1]["match"]["right"] if filter["chain"] == self.prerouting_porthijack else filter["expr"][2]["mangle"]["value"],
proxy_port=filter["expr"][1]["match"]["right"] if filter["chain"] == self.postrouting_porthijack else filter["expr"][2]["mangle"]["value"],
ip_src=nftables_json_to_int(filter["expr"][0]["match"]["right"]) if filter["chain"] == self.prerouting_porthijack else nftables_json_to_int(filter["expr"][3]["mangle"]["value"]),
ip_dst=nftables_json_to_int(filter["expr"][0]["match"]["right"]) if filter["chain"] == self.postrouting_porthijack else nftables_json_to_int(filter["expr"][3]["mangle"]["value"]),
))
return res
def delete(self, srv:Service):
for filter in self.get():
if filter.__eq__(srv):
self.cmd({ "delete":{ "rule": {
"family": "inet",
"table": self.table_name,
"chain": filter.target,
"handle": filter.id
}}})

0
backend/modules/nfproxy/__init__.py → backend/modules/services/__init__.py
View File

358
backend/modules/services/conntrack.py Normal file
View File

@ -0,0 +1,358 @@
"""Forgetting UDP flows, so the next datagram of each is steered by the rules in force now.
Conntrack translates a flow once, at its first packet, and the proxy layer's `redirect`
is a translation. TCP lives with that: a connection is one thing from its handshake to
its close, and one opened before the service started is simply carried to the end. UDP
has no close — a flow lasts for as long as datagrams keep arriving inside the timeout —
and that turned the same rule into three failures:
* **a flow older than the service was never filtered**, for as long as it kept talking:
nothing redirected it, and nothing ever would. A client could open one before the
protection went up and keep it open indefinitely;
* **a flow carried across a restart stayed with the engine being retired**, under the
filters it started with, and kept that engine alive until the drain gave up on it;
* **and once that engine was gone, the flow went nowhere.** Conntrack went on sending it
to a relay port nobody listened on, every datagram refreshing the entry that did so,
until the client happened to fall silent for two minutes.
A UDP flow can simply be forgotten: its next datagram is a new flow, judged by whatever
rules are there now, and neither end had any state that depended on the old entry. So
this deletes the entries, over ctnetlink — the kernel's own interface, the one the
`conntrack` tool speaks, with nothing to install and nothing to spawn.
TCP is never touched here: forgetting a connection breaks it.
"""
import ipaddress
import socket
import struct
from dataclasses import dataclass
from typing import Callable, Iterable
NETLINK_NETFILTER = 12
NLM_F_REQUEST = 0x1
NLM_F_ACK = 0x4
NLM_F_DUMP = 0x300
NLMSG_ERROR = 2
NLMSG_DONE = 3
NFNL_SUBSYS_CTNETLINK = 1
IPCTNL_MSG_CT_GET = 1
IPCTNL_MSG_CT_DELETE = 2
NLA_TYPE_MASK = 0x3FFF
CTA_TUPLE_ORIG = 1
CTA_TUPLE_REPLY = 2
CTA_STATUS = 3
CTA_MARK = 8
CTA_ZONE = 18
CTA_TUPLE_IP = 1
CTA_TUPLE_PROTO = 2
CTA_IP_V4_SRC = 1
CTA_IP_V4_DST = 2
CTA_IP_V6_SRC = 3
CTA_IP_V6_DST = 4
CTA_PROTO_NUM = 1
CTA_PROTO_SRC_PORT = 2
CTA_PROTO_DST_PORT = 3
#: `IPS_DST_NAT`: the flow's destination was rewritten — by a `redirect`, among others.
IPS_DST_NAT = 1 << 5
IPPROTO_UDP = 17
_HEADER = struct.Struct("=IHHII")
_NFGEN = struct.Struct("=BBH")
_ATTR = struct.Struct("=HH")
@dataclass
class Flow:
"""One conntrack entry, as much of it as deciding and deleting need."""
family: int
l4proto: int
orig_src: ipaddress._BaseAddress | None
orig_dst: ipaddress._BaseAddress | None
orig_sport: int | None
orig_dport: int | None
reply_src: ipaddress._BaseAddress | None
reply_sport: int | None
status: int
mark: int
#: The original tuple and the zone exactly as the kernel sent them, which is what a
#: delete names the entry by.
raw_orig: bytes
raw_zone: bytes = b""
@property
def redirected(self) -> bool:
return bool(self.status & IPS_DST_NAT)
# --- the wire -------------------------------------------------------------------
def _align(n: int) -> int:
return (n + 3) & ~3
def attr(kind: int, payload: bytes) -> bytes:
"""One netlink attribute, padded."""
raw = _ATTR.pack(_ATTR.size + len(payload), kind) + payload
return raw + b"\0" * (_align(len(raw)) - len(raw))
def attrs(data: bytes) -> Iterable[tuple[int, bytes, bytes]]:
"""`(type, payload, the whole attribute)` for each attribute in `data`."""
pos = 0
while pos + _ATTR.size <= len(data):
length, kind = _ATTR.unpack_from(data, pos)
if length < _ATTR.size or pos + length > len(data):
return
yield kind & NLA_TYPE_MASK, data[pos + _ATTR.size:pos + length], data[pos:pos + length]
pos += _align(length)
def message(kind: int, flags: int, seq: int, family: int, payload: bytes = b"") -> bytes:
body = _NFGEN.pack(family, 0, 0) + payload
return _HEADER.pack(_HEADER.size + len(body), (NFNL_SUBSYS_CTNETLINK << 8) | kind,
flags, seq, 0) + body
def messages(data: bytes) -> Iterable[tuple[int, bytes]]:
"""`(type, payload)` for each netlink message in one read."""
pos = 0
while pos + _HEADER.size <= len(data):
length, kind, _flags, _seq, _pid = _HEADER.unpack_from(data, pos)
if length < _HEADER.size or pos + length > len(data):
return
yield kind, data[pos + _HEADER.size:pos + length]
pos += _align(length)
def _tuple(data: bytes) -> tuple:
src = dst = sport = dport = None
proto = 0
for kind, payload, _ in attrs(data):
if kind == CTA_TUPLE_IP:
for ip_kind, value, _ in attrs(payload):
if ip_kind in (CTA_IP_V4_SRC, CTA_IP_V6_SRC):
src = ipaddress.ip_address(value)
elif ip_kind in (CTA_IP_V4_DST, CTA_IP_V6_DST):
dst = ipaddress.ip_address(value)
elif kind == CTA_TUPLE_PROTO:
for proto_kind, value, _ in attrs(payload):
if proto_kind == CTA_PROTO_NUM:
proto = value[0]
elif proto_kind == CTA_PROTO_SRC_PORT:
sport = struct.unpack("!H", value[:2])[0]
elif proto_kind == CTA_PROTO_DST_PORT:
dport = struct.unpack("!H", value[:2])[0]
return src, dst, sport, dport, proto
def parse_flow(payload: bytes) -> Flow | None:
"""A conntrack entry out of one `IPCTNL_MSG_CT_NEW` message's payload."""
if len(payload) < _NFGEN.size:
return None
family = payload[0]
orig = reply = None
raw_orig = raw_zone = b""
status = mark = 0
for kind, value, whole in attrs(payload[_NFGEN.size:]):
if kind == CTA_TUPLE_ORIG:
orig, raw_orig = _tuple(value), whole
elif kind == CTA_TUPLE_REPLY:
reply = _tuple(value)
elif kind == CTA_STATUS:
status = struct.unpack("!I", value[:4])[0]
elif kind == CTA_MARK:
mark = struct.unpack("!I", value[:4])[0]
elif kind == CTA_ZONE:
raw_zone = whole
if orig is None:
return None
reply_src, _, reply_sport, _, _ = reply if reply else (None, None, None, None, 0)
return Flow(
family=family, l4proto=orig[4],
orig_src=orig[0], orig_dst=orig[1], orig_sport=orig[2], orig_dport=orig[3],
reply_src=reply_src, reply_sport=reply_sport,
status=status, mark=mark, raw_orig=raw_orig, raw_zone=raw_zone,
)
def delete_request(flow: Flow, seq: int) -> bytes:
return message(IPCTNL_MSG_CT_DELETE, NLM_F_REQUEST | NLM_F_ACK, seq, flow.family,
flow.raw_orig + flow.raw_zone)
# --- talking to the kernel ------------------------------------------------------
def _read_until_done(sock: socket.socket) -> Iterable[bytes]:
"""The payload of every entry a dump answers with."""
while True:
data = sock.recv(1 << 18)
if not data:
return
for kind, payload in messages(data):
if kind == NLMSG_DONE:
return
if kind == NLMSG_ERROR:
error = struct.unpack_from("=i", payload)[0] if len(payload) >= 4 else 0
if error:
raise OSError(-error, "conntrack dump refused")
return
yield payload
def _acknowledged(sock: socket.socket) -> int:
"""The error a request was answered with, 0 for none."""
while True:
for kind, payload in messages(sock.recv(1 << 16)):
if kind == NLMSG_ERROR:
return -struct.unpack_from("=i", payload)[0] if len(payload) >= 4 else 0
def delete_where(chosen: Callable[[Flow], bool]) -> int:
"""Delete every UDP entry `chosen` picks. Returns how many went.
The table is read once and the deletes follow: a flow that ended in between is
answered `ENOENT`, which is the state being asked for.
"""
with socket.socket(socket.AF_NETLINK, socket.SOCK_RAW, NETLINK_NETFILTER) as sock:
sock.settimeout(5)
sock.bind((0, 0))
sock.send(message(IPCTNL_MSG_CT_GET, NLM_F_REQUEST | NLM_F_DUMP, 1, socket.AF_UNSPEC))
doomed = []
for payload in _read_until_done(sock):
flow = parse_flow(payload)
if flow is not None and flow.l4proto == IPPROTO_UDP and chosen(flow):
doomed.append(flow)
gone = 0
for seq, flow in enumerate(doomed, start=2):
sock.send(delete_request(flow, seq))
error = _acknowledged(sock)
if error == 0:
gone += 1
elif error != 2: # ENOENT: it ended by itself meanwhile
raise OSError(error, "conntrack refused a delete")
return gone
# --- what to forget -------------------------------------------------------------
Target = tuple[ipaddress.IPv4Network | ipaddress.IPv6Network, int]
def targets(pairs: Iterable[tuple[str, int]]) -> list[Target]:
"""`(address or network, port)` pairs as something a flow can be compared with."""
return [(ipaddress.ip_network(str(ip), strict=False), int(port)) for ip, port in pairs]
def headed_for(flow: Flow, wanted: list[Target]) -> bool:
"""Whether a flow was going to one of `wanted` before anything rewrote it."""
if flow.orig_dst is None or flow.orig_dport is None:
return False
return any(
flow.orig_dport == port and flow.orig_dst.version == network.version
and flow.orig_dst in network
for network, port in wanted
)
def chooser(*, moved: list[Target], unsteered: list[Target], spare_mark: int,
bound: set[int]) -> Callable[[Flow], bool]:
"""Which flows `forget_udp` lets go of. Split out so the decision can be tested
without a kernel."""
def chosen(flow: Flow) -> bool:
if flow.mark == spare_mark:
return False
if headed_for(flow, moved):
return True
if headed_for(flow, unsteered):
return not flow.redirected or flow.reply_sport not in bound
return False
return chosen
def forget_udp(*, moved: Iterable[tuple[str, int]] = (),
unsteered: Iterable[tuple[str, int]] = (), spare_mark: int) -> int:
"""Forget UDP flows headed for these addresses, so their next datagram is steered by
the rules in force now.
* `moved`: every flow headed there, including the ones a redirect already took to an
engine — a datagram flow can change engines between two datagrams and lose nothing
but the filter's own state for it, which is what a new configuration costs anyway.
* `unsteered`: only the ones nothing is carrying — never redirected at all, or
redirected to a port nobody listens on any more. What a QUIC edge needs: its
connections are terminated by an engine, and moved to another one mid-connection
they would end, while one still draining is carrying them.
`spare_mark` is the engine's own: its dials to the service are flows headed for the
same address, and they are its to keep.
"""
moved_to, unsteered_to = targets(moved), targets(unsteered)
if not moved_to and not unsteered_to:
return 0
return delete_where(chooser(
moved=moved_to, unsteered=unsteered_to, spare_mark=spare_mark,
bound=bound_udp_ports() if unsteered_to else set(),
))
def bound_udp_ports() -> set[int]:
"""Every UDP port some socket on this host is bound to, from `/proc/net/udp{,6}`."""
found = set()
for path in ("/proc/net/udp", "/proc/net/udp6"):
try:
with open(path) as table:
next(table, None)
for line in table:
fields = line.split()
if len(fields) > 1 and ":" in fields[1]:
found.add(int(fields[1].rsplit(":", 1)[1], 16))
except OSError:
continue
return found
def forget_udp_redirected_to(ports: Iterable[int], local: set) -> int:
"""Forget the UDP flows a redirect sent to one of these ports on this host.
What is left of an engine once it has gone: every such flow points at a relay nobody
listens on any more. Recognised by the reply coming from that port on one of this
host's own addresses — which is what a redirect writes into the reply tuple, and what
no flow forwarded to a container or another host can have.
"""
wanted = {int(port) for port in ports}
if not wanted:
return 0
return delete_where(
lambda flow: flow.redirected and flow.reply_sport in wanted
and flow.reply_src is not None and flow.reply_src in local
)
def local_addresses() -> set:
"""Every address this host holds right now."""
import psutil
found = set()
for entries in psutil.net_if_addrs().values():
for entry in entries:
if entry.family in (socket.AF_INET, socket.AF_INET6):
try:
found.add(ipaddress.ip_address(entry.address.split("%")[0]))
except ValueError:
pass
return found

1026
backend/modules/services/firewall.py Normal file
View File

File diff suppressed because it is too large Load Diff

207
backend/modules/services/logs.py Normal file
View File

@ -0,0 +1,207 @@
"""What a running service is actually doing, as it does it.
Everything interesting a datapath has to say — a connection refused and by which rule,
a filter that printed something, a filter that raised, a warning about falling back —
used to go to the backend's own stderr, where nobody looks during a competition. This
keeps a bounded tail of it per service and pushes new lines to the browser.
Two properties matter more than the feature itself:
**It cannot grow.** A service under attack blocks thousands of connections a minute and
a filter in a print loop produces output as fast as the traffic arrives. The ring is
capped and the oldest lines fall off; nothing here can turn logging into an outage.
**It cannot flood the browser.** Lines are coalesced and flushed on a timer rather than
emitted one by one, and a flush that would carry more than it should says how many it
dropped instead of carrying them.
The same flush is also what tells the browser that the **counters** moved. A block bumps
a filter's total, a pattern's, a function's and the bucketed history, and the interface
holds all of those with `staleTime: Infinity` — nothing refetches on its own, so without
a signal the numbers and the chart sit still while the log scrolls past underneath them.
Riding the flush that already exists beats a second timer, and it is exactly true: the
lines that carry a block are the events that change a counter. It is throttled harder
than the log, because invalidating every query of a service is a heavier thing to ask of
the browser than appending a line.
"""
import asyncio
import time
from collections import deque
import utils
#: Lines kept per service. Enough to see what just happened, small enough that a
#: hundred services cost a few megabytes between them.
HISTORY = 500
#: How long lines are allowed to pile up before the browser is told. Long enough that a
#: burst becomes one message, short enough to still read as live.
FLUSH_INTERVAL = 0.25
#: Most lines carried in a single flush. Beyond this the browser is told the count
#: rather than the content — a thousand identical block lines inform nobody.
FLUSH_LIMIT = 50
#: How often a service may tell the browser its counters are stale. Much slower than the
#: log's own flush: this one invalidates every query of the service — its filters, its
#: patterns, its functions, its statistics — and doing that four times a second under
#: attack would be its own kind of flood.
COUNTER_INTERVAL = 2.0
class LEVEL:
INFO = "info"
BLOCK = "block"
OUTPUT = "output"
WARN = "warn"
ERROR = "error"
class ServiceLog:
"""One service's rolling tail, and the pending batch on its way to the browser."""
def __init__(self, service_id: str):
self.service_id = service_id
self.history: deque[dict] = deque(maxlen=HISTORY)
self._pending: list[dict] = []
self._dropped = 0
self._flush_task: asyncio.Task | None = None
#: When the browser was last told that this service's counters had moved, and
#: the emit waiting for the current window to close.
self._counters_told = 0.0
self._counters_task: asyncio.Task | None = None
#: Counts every line ever added, so the frontend can tell "nothing new" from
#: "the tail happens to look the same".
self.seq = 0
def add(self, level: str, text: str) -> None:
for line in text.splitlines() or [""]:
line = line.rstrip()
if not line:
continue
self.seq += 1
entry = {"at": int(time.time() * 1000), "level": level, "text": line, "seq": self.seq}
self.history.append(entry)
if len(self._pending) < FLUSH_LIMIT:
self._pending.append(entry)
else:
self._dropped += 1
self._schedule()
def _schedule(self) -> None:
if self._flush_task and not self._flush_task.done():
return
try:
loop = asyncio.get_running_loop()
except RuntimeError:
return # no loop to emit on; the history still has the lines
self._flush_task = loop.create_task(self._flush_later())
async def _flush_later(self) -> None:
await asyncio.sleep(FLUSH_INTERVAL)
entries, dropped = self._pending, self._dropped
self._pending, self._dropped = [], 0
if not entries:
return
if dropped:
# A number of its own. It used to reuse the last line's, which the browser
# dedupes on: whichever of the two arrived second was thrown away, and it was
# usually this one — the only line saying how much had not been shown.
self.seq += 1
entries.append({
"at": int(time.time() * 1000),
"level": LEVEL.WARN,
"text": f"... and {dropped} more line(s) in the same instant",
"seq": self.seq,
})
# Reached through the module rather than imported by value: `utils.socketio` is
# assigned when the app starts, so a direct import would capture the `None` it
# holds at import time and silently never emit anything.
try:
if utils.socketio is not None:
await utils.socketio.emit(
"log", {"service_id": self.service_id, "entries": entries}
)
await self._tell_counters_moved(entries)
except Exception:
pass # a browser that is not listening is not a reason to disturb the datapath
async def _tell_counters_moved(self, entries: list[dict]) -> None:
"""Say that the totals and the chart are out of date, at most every so often.
Only for blocks: they are the lines that come with a counter having changed. A
filter's `print()` or a warning moves nothing, and invalidating every query of
the service for one would be work done for no reason.
Rate-limited with a **trailing** emit rather than by dropping what falls inside
the window. Dropping was the first version and it left the interface frozen on
stale numbers: a burst emitted once at its start, every flush after that was
discarded, and when the traffic stopped there was no further flush to carry the
final count — so the chart sat at whatever it happened to reach two seconds in.
The last block of a burst is precisely the one worth hearing about.
"""
# A warning or an error changes the service list too: it carries the latest one
# (`last_problem`).
if not any(entry["level"] in (LEVEL.BLOCK, LEVEL.WARN, LEVEL.ERROR)
for entry in entries):
return
waited = time.monotonic() - self._counters_told
if waited >= COUNTER_INTERVAL:
await self._emit_counters()
return
# Inside the window: one emit is scheduled for when it closes, and further
# blocks arriving before then fold into it.
if self._counters_task and not self._counters_task.done():
return
self._counters_task = asyncio.get_running_loop().create_task(
self._emit_counters_after(COUNTER_INTERVAL - waited)
)
async def _emit_counters_after(self, delay: float) -> None:
await asyncio.sleep(delay)
try:
await self._emit_counters()
except Exception:
pass # a browser that is not listening is not a reason to raise here
async def _emit_counters(self) -> None:
self._counters_told = time.monotonic()
if utils.socketio is not None:
await utils.socketio_emit(["services"])
def entries(self) -> list[dict]:
return list(self.history)
def last_problem(self) -> dict | None:
"""The newest warning or error still in the tail, for the service list to show.
The live log is on the service's own page, and a warning nobody has that page open
for is a warning nobody reads: a datapath that died and came back, a filter that
raised, a fallback that lost the client's address. The list carries the latest one
beside the service, and clearing the log is how it is acknowledged.
"""
for entry in reversed(self.history):
if entry["level"] in (LEVEL.WARN, LEVEL.ERROR):
return entry
return None
def clear(self) -> None:
self.history.clear()
self._pending.clear()
self._dropped = 0
_logs: dict[str, ServiceLog] = {}
def log_for(service_id: str) -> ServiceLog:
log = _logs.get(service_id)
if log is None:
log = _logs[service_id] = ServiceLog(service_id)
return log
def forget(service_id: str) -> None:
"""Drop a deleted service's tail, so nothing outlives what it was about."""
_logs.pop(service_id, None)

55
backend/modules/services/mirror.py Normal file
View File

@ -0,0 +1,55 @@
"""The interface the engine writes decrypted traffic to.
TLS is terminated inside the engine, which is what lets a protected service occupy no
port beyond the one it already answered on — and it means the plaintext is never a packet
on any interface, so there would be nothing for a capture tool to read. The engine writes
it out itself (`proxysrc/src/capture.rs`), framed as the TCP stream it was before
encryption, onto the device this module puts there.
All that is needed here is the device: a `dummy` interface, which is a sink that carries
whatever is sent to it and goes nowhere. One for the whole instance, so every TLS
service's plaintext arrives on the same interface and a single capture covers the lot.
It exists for as long as firegex does, put there by `FirewallManager.init` and taken
away by `close`. It used to come and go with the TLS services themselves, on the argument
that an interface present while nothing is decrypting is one somebody points a capture at
and watches stay empty. Watching it stay empty turned out to be the better failure: a
capture tool is attached once, at the start of a round, and an interface that disappears
underneath it takes the tool with it — Zeek and Suricata exit rather than wait, and the
restart of a single TLS service was enough to do it. So the device now outlives any one
service, and the thing an operator points at in the morning is still there in the
afternoon.
"""
import subprocess
#: What an operator types into a capture tool. Fixed, and the same name the engine uses.
DEVICE = "firegex0"
def _run(*args: str) -> subprocess.CompletedProcess:
return subprocess.run(args, capture_output=True, text=True)
def present() -> bool:
return _run("ip", "link", "show", DEVICE).returncode == 0
def ensure() -> bool:
"""Put the device there, if it is not already. Returns whether it is now.
A failure is not an error anybody hears about: a capture aid that could stop a
service from starting would be worse than no capture aid. The engine finds no
interface, writes nothing, and the traffic is filtered and forwarded exactly as it
would have been.
"""
if present():
return True
if _run("ip", "link", "add", DEVICE, "type", "dummy").returncode != 0:
return False
return _run("ip", "link", "set", DEVICE, "up").returncode == 0
def release() -> None:
"""Take it away again, at shutdown and nowhere else."""
_run("ip", "link", "del", DEVICE)

547
backend/modules/services/models.py Normal file
View File

@ -0,0 +1,547 @@
"""The one service model: a network layer, and an ordered chain of filters on it.
Firegex used to have a module per combination — `nfregex` was NFQUEUE plus regexes,
`nfproxy` was NFQUEUE plus Python, `porthijack` was a redirect, `tls` was its own module — and
each one owned its own services, its own page and its own database. Choosing how to
inspect traffic therefore also chose how to intercept it, and the two have nothing to
do with each other.
Here they are separate. A [`Service`] says only where the traffic is and how to get
hold of it; a [`Filter`] says what to do with it. Any filter runs on any transport
that can host it, and the service does not have to be recreated to change either.
"""
import base64
import uuid
from utils import is_ip_parse
class TRANSPORT:
"""How the traffic is intercepted."""
#: Packets are queued to userspace and a verdict is handed back to the kernel.
#: Nothing is terminated, so the service sees the original connection, and the
#: kernel keeps forwarding if the filter dies. It cannot rewrite a payload
#: without desynchronising the stream.
NFQUEUE = "nfqueue"
#: The connection is terminated and reopened towards the service. That buys exact
#: rewriting, kernel-side reassembly, backpressure and TLS; it costs the kernel's
#: fail-open backstop, which the engine rebuilds itself.
PROXY = "proxy"
#: The traffic is handed to a proxy the operator runs themselves, by rewriting the
#: destination on the way in and undoing it on the way out. Firegex inspects
#: nothing here — it only arranges for their proxy to be in the path — so no filter
#: can be attached. It is the escape hatch for a protocol nothing built in
#: understands.
EXTERNAL = "external"
ALL = (NFQUEUE, PROXY, EXTERNAL)
class L4:
"""What a service speaks on the wire. A property of the service.
`tls` sits beside `tcp` and `udp` rather than being a flag on top of one of them,
because that is the shape of the thing: a service speaks TLS or it does not, and
"TLS" is what an operator would answer if asked what is on the wire. It used to be a
boolean, which made three combinations representable that are not real — TLS on a
UDP service, on the nfqueue layer, and on a hand-off — and each of them had to be
caught by a check written by hand. Two of those are now unrepresentable instead.
What the *kernel* matches on is a different question with a smaller answer, which is
what `l4_of` gives: an address carries that, so `(ip, port, proto)` stays a key that
means what it says. Without it a TCP service and a TLS service could both claim one
`ip:port`, which on the wire is the same port twice.
"""
TCP = "tcp"
UDP = "udp"
#: TLS over TCP, terminated by the engine.
TLS = "tls"
#: QUIC, terminated by the engine. UDP on the wire, TLS 1.3 inside it, and streams
#: inside that — which is the only reason a filter has anything to look at: once past
#: the Initial packet, QUIC encrypts the frames and the stream boundaries along with
#: the payload, so a layer that forwards has nothing to show anybody.
QUIC = "quic"
#: Every version of HTTP, on whichever transport each address carries it.
#:
#: The odd one out, and deliberately so. The other four say what is on the wire;
#: this one says what the *service* is, and lets each address say how it is reached
#: — `:80` in the clear, `:443` under TLS, `:443/udp` under QUIC. One service, one
#: chain, one certificate, and HTTP/1.1, HTTP/2 and HTTP/3 all rendered to the
#: filters as the same HTTP/1.1.
#:
#: It exists because the alternative was two or three services with hand-copied
#: filter chains: HTTP/1.1 and HTTP/2 live on `tls` (TCP) and HTTP/3 on `quic`
#: (UDP), and a chain that has to be kept in step by hand is a chain that stops
#: protecting one of them silently — the same argument that put several addresses
#: under one service in the first place.
#:
#: Nothing about the *kernel* becomes ambiguous, because nothing here is asked of
#: the kernel: each address still carries its own `tcp` or `udp`, which is what the
#: rules match and what `(ip, port, proto)` is keyed on. What the connection turns
#: out to be — TLS or not, HTTP/2 or not — is decided by the engine per connection,
#: from what the client actually sent.
HTTP = "http"
ALL = (TCP, UDP, TLS, QUIC, HTTP)
@staticmethod
def l4_of(proto: str) -> str:
"""The transport a rule has to match to catch this service's traffic.
Two of the four are the other two wearing a hat: `tls` is TCP on the wire and
`quic` is UDP. That is what keeps `(ip, port, proto)` a key meaning what it says
— without it a TLS service and a TCP one, or a QUIC service and a UDP one, could
each claim one `ip:port` between them, which on the wire is the same port twice.
`http` answers `tcp` here, which is the right *default* for an address that does
not say otherwise and never the whole answer: an `http` service's addresses each
carry their own, because it is the one protocol whose addresses are not all on
the same transport. Ask `Service.carries` rather than the service's protocol
whenever the question is about the kernel.
"""
return L4.UDP if str(proto) in (L4.UDP, L4.QUIC) else L4.TCP
@staticmethod
def edges_of(proto: str) -> tuple[str, ...]:
"""Which transports an address of this service may be reached on.
One for every protocol but `http`, which is the point of `http`.
"""
if str(proto) == L4.HTTP:
return (L4.TCP, L4.UDP)
return (L4.l4_of(proto),)
class PROTO:
"""Which application protocol a Python filter is written against.
Never chosen: **read off the filter's own code**. The library decides when a filter
is called from what its parameters are annotated with, so asking for an
`HttpRequest` is what makes a file an HTTP filter and asking only for a `RawPacket`
is what makes one protocol-agnostic. Stored here so the interface can show it and
so a chain can be described without executing anything; the code remains the only
thing that decides it.
A file that asks for two *different* application protocols is refused when it is
saved, naming both functions — a connection is only ever one of them.
"""
TCP = "tcp"
HTTP = "http"
ALL = (TCP, HTTP)
class KIND:
"""What a filter is."""
#: Patterns matched by hyperscan. Both transports match with the same library, so
#: a pattern means the same thing wherever it runs.
REGEX = "regex"
#: The user's own Python, running out of process.
PYFILTER = "pyfilter"
ALL = (REGEX, PYFILTER)
class STATUS:
STOP = "stop"
ACTIVE = "active"
# A matching pattern closes the connection. There is no second action, and there is no
# `action` column any more.
#
# There used to be `rewrite`: replace the matched bytes and let the traffic continue. It
# was removed because it could not keep the promise it made. Rewriting scanned one chunk
# at a time — bytes already forwarded cannot be taken back, so a match straddling two
# chunks was never rewritable — and the result was measured: a pattern sent whole reached
# the service rewritten, and the same pattern split across two TCP segments reached it
# *intact*. Blocking, which scans in hyperscan's stream mode, caught the split version.
#
# So the failure was silent and the operator had no way to see it: no block, no log, no
# counter, just a rewrite rule that had quietly not applied. A filter that works on the
# sender's segmentation is worse than no filter, because it is trusted. Blocking is the
# verdict this can honestly offer on a stream, so it is the only one offered.
class MODE:
"""Which half of the traffic a pattern is matched against."""
CLIENT_TO_SERVER = "C"
SERVER_TO_CLIENT = "S"
BOTH = "B"
ALL = (CLIENT_TO_SERVER, SERVER_TO_CLIENT, BOTH)
class UPSTREAM:
"""What the service *behind one address* speaks.
The engine used to have one answer and never asked: whatever the client spoke, the
service was assumed to speak too — a TLS connection was re-encrypted on the way out,
a QUIC one re-encoded as QUIC. That is right whenever firegex is carrying somebody
else's encryption, and wrong in the case an operator most often has, which is a
service that speaks neither and never will.
So it is three answers now, and they are about the *service*, not about the client:
* `same` — it speaks what the client spoke. The default, and what every service did
before there was a choice;
* `tcp` — it answers in the clear. Firegex terminates the client's TLS or QUIC and
forwards the plaintext, which makes firegex the thing that *adds* the encryption
rather than the thing that carries it;
* `tls` — it speaks TLS, whatever the client used to get here. On a TLS service that
is `same` by another name; under QUIC it is the one that matters, because it is
how an HTTPS-only service is reached from an HTTP/3 client.
Under QUIC the last two are not a pass-through: HTTP/3 has no cleartext form, so what
reaches the service is the HTTP/1.1 the filters were already being shown. That is the
one place in the engine where what leaves is not the version that arrived, and it is
only ever the operator's explicit choice.
"""
SAME = "same"
TCP = "tcp"
TLS = "tls"
ALL = (SAME, TCP, TLS)
@staticmethod
def env(upstream: str) -> str:
"""What the engine is told. One variable, because it is one question."""
return {UPSTREAM.TCP: "plain", UPSTREAM.TLS: "tls"}.get(str(upstream), "same")
class Address:
"""One address a service is protected on.
A service is one thing to protect and one way of intercepting it; *where* it is
reachable is a list, because the same service routinely answers on more than one
address — a v4 and a v6 one, a public and an internal one, two ports of the same
daemon. Making the operator create one service per address meant keeping their
filter chains in step by hand, and a chain that drifted was a chain that stopped
protecting one of them silently.
The transport protocol is carried here as well as on the service. Denormalised on
purpose: it is what makes `(ip, port, proto)` a usable uniqueness key, so a TCP
service and a UDP one can share an address the way the kernel lets them. It is
derived — `L4.l4_of(edge)` — because `tls` is TCP on the wire and `quic` is UDP, and
what the kernel matches is a smaller question than what the address carries.
Two things an address says for itself, and both are why it says anything at all:
**what is spoken at it** (`edge`) and **where that traffic goes** (`target_port`).
Firegex was transparent and only transparent — the client dialled the service's own
address and the engine dialled it back as the client — which meant a service could
only be protected where it already listened. One service answering in the clear on
`:80`, reached over TLS on `:443` and over HTTP/3 on a UDP port, is three rows
pointing at one port.
"""
def __init__(
self,
address_id: str,
service_id: str,
ip_int: str,
port: int,
proto: str = L4.TCP,
edge: str | None = None,
target_port: int | None = None,
upstream: str = UPSTREAM.SAME,
proxy_ip: str | None = None,
proxy_port: int | None = None,
**other,
):
self.id = address_id
self.service_id = service_id
self.ip_int = ip_int
self.port = port
self.proto = proto
#: What clients speak *at this address*: one of `tcp`, `tls`, `udp`, `quic` —
#: the same words a service speaks, minus `http`, which is the one that means
#: "ask the addresses". For every protocol but that one it is the service's own
#: and the operator never sees it; on an `http` service it is the choice, and it
#: is what decides whether this address needs a certificate behind it and which
#: transport the kernel matches.
self.edge = str(edge or proto)
#: Where the traffic that arrives here is actually sent, when that is not the
#: port it arrived on. `None` is the transparent case and the default: firegex
#: dials the address the client dialled, as the client, and the service sees
#: exactly what it would have seen without any of this.
#:
#: Set, this address is a *publication*: one service answered on `:80` in the
#: clear and reached on `:443` under TLS, or on a QUIC port, without moving. The
#: address is where the world knocks; this is where the service is.
self.target_port = int(target_port) if target_port else None
#: What the service *behind this address* speaks. See [`UPSTREAM`]. It belongs to
#: the address rather than to the service because that is where the question has
#: an answer: one daemon is reached over TLS on one port and in the clear on
#: another, and what firegex does on the way out is a property of the way in.
self.upstream = str(upstream or UPSTREAM.SAME)
#: Where the operator's own proxy is listening for *this* address. Only for
#: `external`, and per address rather than per service: the return rule rewrites
#: the source port back to the original one, so two addresses handed to the same
#: proxy endpoint could not be told apart on the way out.
self.proxy_ip = proxy_ip
self.proxy_port = proxy_port
@classmethod
def from_dict(cls, var: dict):
return cls(**var)
def to_dict(self) -> dict:
return {
"address_id": self.id,
"service_id": self.service_id,
"ip_int": self.ip_int,
"port": self.port,
"proto": self.proto,
"edge": self.edge,
"target_port": self.target_port,
"upstream": self.upstream,
"proxy_ip": self.proxy_ip,
"proxy_port": self.proxy_port,
}
@property
def is_interface(self) -> bool:
return not is_ip_parse(self.ip_int)
@property
def is_ipv6(self) -> bool:
return ":" in str(self.ip_int) if not self.is_interface else False
def __repr__(self):
return f"<Address {self.ip_int}:{self.port}/{self.proto}>"
class Service:
"""One protected endpoint and the transport in front of it."""
def __init__(
self,
service_id: str,
name: str,
status: str,
proto: str,
transport: str,
fail_open: bool = True,
max_connections: int = 0,
over_limit_forwards: bool = False,
first_byte_timeout: int = 0,
tls_cert: str | None = None,
tls_key: str | None = None,
addresses: list | None = None,
**other,
):
self.id = service_id
self.name = name
self.status = status
self.proto = proto
self.transport = transport
self.fail_open = bool(fail_open)
#: How many connections or UDP flows may be carried at once; 0 means no limit.
self.max_connections = max(0, int(max_connections or 0))
#: Whether what does not fit is forwarded with no filter in front of it.
self.over_limit_forwards = bool(over_limit_forwards)
#: Seconds a connection may carry nothing at all before it is closed; 0 is never.
self.first_byte_timeout = max(0, int(first_byte_timeout or 0))
self.tls_cert = tls_cert
self.tls_key = tls_key
#: Every address this service is protected on. Filled by whoever read the
#: service out of the database; a service with none is configured but
#: unreachable, and starting it installs no rules.
self.addresses: list[Address] = list(addresses or [])
@classmethod
def from_dict(cls, var: dict):
return cls(**var)
@property
def decrypts(self) -> bool:
"""Whether the engine decrypts this service's traffic before filtering it.
Read off what the service speaks rather than held beside it. Only the proxy layer
can do it: decrypting means terminating the connection, and that is what that
layer does. It was once orthogonal to the layer, which took an nginx in front of
NFQUEUE to arrange — so the connection was terminated anyway, and the one
property that layer has over the other was already gone.
Named for what it does rather than for TLS, because two protocols do it now and
QUIC is the one with no alternative: TLS over TCP can at least be carried past
unopened, while a QUIC packet past the handshake has its frames encrypted too.
Both need the same thing from the operator, which is a certificate.
`http` is in the list because a certificate is the whole of what it adds: a
cleartext HTTP service is already `tcp`, and HTTP/2 in the clear is recognised
there from the connection preface whatever the service is called. What `http`
buys over that is the TLS and the QUIC edges, and neither exists without one.
"""
return str(self.proto) in (L4.TLS, L4.QUIC, L4.HTTP)
@property
def l4(self) -> str:
"""The transport the kernel sees, which is what a rule has to match.
Only meaningful where a service has one. An `http` service is reached on both,
and everything that used to ask this of such a service asks `carries` instead:
the answer here is its addresses' default, not a claim about all of them.
"""
return L4.l4_of(self.proto)
def carries(self, l4: str) -> bool:
"""Whether any of this service's addresses is reached over `l4`.
The question every rule, listener and relay actually has, and the one that
stopped having a single answer when `http` arrived. A service with no addresses
yet is answered from its protocol, so a check made at creation — before there is
an address to look at — still means something.
"""
if not self.addresses:
return l4 in L4.edges_of(self.proto)
return any(L4.l4_of(addr.proto or self.proto) == l4 for addr in self.addresses)
@property
def udp_addresses(self) -> list:
"""The addresses that need a relay of their own: datagrams, and QUIC."""
return [
addr
for addr in self.addresses
if L4.l4_of(addr.proto or self.proto) == L4.UDP
]
@property
def has_ipv6(self) -> bool:
"""Whether any of its addresses is IPv6.
Decides the family a proxy listener has to be opened in, which is why it is a
property of the service and not of one address: one listener serves them all.
"""
return any(addr.is_ipv6 for addr in self.addresses)
@property
def has_ipv6_tcp(self) -> bool:
"""Whether the *TCP* listener needs to be able to accept IPv6.
The narrower question, and the one worth asking before rebuilding a service: the
only listener that can have its family wrong is the TCP one. Everything relayed
per address — datagrams, and QUIC — binds a socket in the family of the address
when it arrives. On an `http` service the two sit side by side, so an IPv6 HTTP/3
address must not cost every TCP connection on the service a restart it did not
need.
"""
udp = {addr.id for addr in self.udp_addresses}
return any(addr.is_ipv6 for addr in self.addresses if addr.id not in udp)
def to_dict(self) -> dict:
# The certificate goes out, the key never does: it is write-only from the
# API's point of view, and a service listing is the easiest place to leak it.
return {
"service_id": self.id,
"name": self.name,
"status": self.status,
"proto": self.proto,
"transport": self.transport,
"fail_open": self.fail_open,
"addresses": [addr.to_dict() for addr in self.addresses],
}
class Filter:
"""One link in a service's chain."""
def __init__(
self,
filter_id: str,
service_id: str,
position: int,
kind: str,
name: str,
active: bool,
proto: str = PROTO.TCP,
blocked: int = 0,
**other,
):
self.id = filter_id
self.service_id = service_id
self.position = position
self.kind = kind
self.name = name
self.active = bool(active)
self.proto = proto
self.blocked = blocked
@classmethod
def from_dict(cls, var: dict):
return cls(**var)
def to_dict(self) -> dict:
return {
"filter_id": self.id,
"service_id": self.service_id,
"position": self.position,
"kind": self.kind,
"name": self.name,
"active": self.active,
"proto": self.proto,
"blocked": self.blocked,
}
class Regex:
"""One pattern inside a regex filter.
Stored base64-encoded because a pattern is bytes, not text: matching a raw byte
sequence is a normal thing to want, and JSON has no way to carry one.
"""
def __init__(
self,
regex_id: str,
filter_id: str,
regex: bytes,
mode: str,
case_sensitive: bool,
active: bool,
blocked: int = 0,
**other,
):
self.id = regex_id
self.filter_id = filter_id
self.regex = regex
self.mode = mode
self.case_sensitive = bool(case_sensitive)
self.active = bool(active)
self.blocked = blocked
@classmethod
def from_dict(cls, var: dict):
var = dict(var)
var["regex"] = base64.b64decode(var["regex"])
return cls(**var)
def to_dict(self) -> dict:
return {
"regex_id": self.id,
"filter_id": self.filter_id,
"regex": base64.b64encode(self.regex).decode(),
"mode": self.mode,
"case_sensitive": self.case_sensitive,
"active": self.active,
"blocked": self.blocked,
}
@property
def is_input(self) -> bool:
return self.mode in (MODE.CLIENT_TO_SERVER, MODE.BOTH)
@property
def is_output(self) -> bool:
return self.mode in (MODE.SERVER_TO_CLIENT, MODE.BOTH)
def new_id() -> str:
return str(uuid.uuid4())

1200
backend/modules/services/nftables.py Normal file
View File

File diff suppressed because it is too large Load Diff

101
backend/modules/services/pyapi.py Normal file
View File

@ -0,0 +1,101 @@
"""What the filter library offers, read out of the library itself.
The editor needs to know which models exist, what each one can be asked for, and which
of those can be written to. All of that is already stated once — in the classes the
library exposes and in `type_annotations_associations`, the table that decides when a
filter is called — so it is introspected rather than described a second time.
A hand-written list would have been quicker and would be wrong within a release: a
property added to `HttpRequest` would be missing from the hints, and one renamed would
be offered under a name that no longer resolves. Autocompletion that lies is worse than
none, for the same reason a regex tester that disagrees with the engine is.
"""
import inspect
#: Members every object has and nobody wants offered.
_SKIP = {"count", "index"}
def _clean(doc: str | None) -> str:
return inspect.cleandoc(doc or "").strip()
def _members(cls) -> list[dict]:
"""The public properties of a model, with what they are for and whether they change.
Writability is reported rather than assumed: the boundary the filter API rests on is
that a filter reads and answers with a verdict, and an editor that says so while you
type teaches it better than a paragraph in the documentation. Nothing is writable
today — the payload was, for `UNSTABLE_MANGLE`, and stopped being with it.
"""
out = []
for name, member in inspect.getmembers(cls, lambda m: isinstance(m, property)):
if name.startswith("_") or name in _SKIP:
continue
out.append({
"name": name,
"doc": _clean(member.__doc__),
"writable": member.fset is not None,
})
for name, member in inspect.getmembers(cls, inspect.isfunction):
if name.startswith("_") or name in _SKIP:
continue
try:
signature = f"{name}{inspect.signature(member)}"
except (TypeError, ValueError):
signature = f"{name}(...)"
out.append({"name": name, "doc": _clean(member.__doc__),
"writable": False, "signature": signature})
return sorted(out, key=lambda m: m["name"])
def describe() -> dict:
"""Everything the editor is allowed to suggest, built from the live library."""
from firegex.pyfilters import (
ACCEPT, DROP, REJECT, ExceptionAction, FullStreamAction,
)
from firegex.pyfilters.models import type_annotations_associations
models: dict[str, dict] = {}
for proto, types in type_annotations_associations.items():
for cls in types:
entry = models.setdefault(cls.__name__, {
"name": cls.__name__,
"doc": _clean(cls.__doc__),
"members": _members(cls),
"protocols": [],
})
entry["protocols"].append(proto)
for entry in models.values():
entry["protocols"].sort()
verdicts = [
{"name": "ACCEPT", "value": ACCEPT.value,
"doc": "Forward this chunk. Returning None means the same thing."},
{"name": "REJECT", "value": REJECT.value,
"doc": "Refuse the connection. Everything still in the stream goes with it."},
{"name": "DROP", "value": DROP.value,
"doc": "Stop the connection's traffic. On NFQUEUE this chunk and every one after "
"it are dropped without closing anything; on the proxy layer, where a "
"stream cannot skip bytes, the connection is closed as REJECT does."},
]
settings = [
{"name": "FGEX_STREAM_MAX_SIZE",
"doc": "Bytes of one stream a model may accumulate before "
"FGEX_FULL_STREAM_ACTION decides what to do."},
{"name": "FGEX_FULL_STREAM_ACTION",
"doc": "What happens when a stream reaches that size.",
"values": [f"FullStreamAction.{name}" for name in FullStreamAction.__members__]},
{"name": "FGEX_INVALID_ENCODING_ACTION",
"doc": "What happens when a parser cannot read the traffic — a reply that is "
"not valid HTTP, for instance.",
"values": [f"ExceptionAction.{name}" for name in ExceptionAction.__members__]},
]
return {
"models": sorted(models.values(), key=lambda m: m["name"]),
"verdicts": verdicts,
"settings": settings,
}

475
backend/modules/services/pyworker.py Normal file
View File

@ -0,0 +1,475 @@
#!/usr/bin/env python3
"""Runs a user's Python filters, one process away from the datapath.
Started by the engine, never by the backend: the engine is the one that has to kill it
when it stops answering, and something else owning its lifetime would defeat the point
of moving the code out of process at all.
The filters are run through the **real** `firegex.pyfilters` library — the same one the
NFQUEUE binary embeds, the same one `fgex pyfilters` simulates against, the same one the
documentation describes. An earlier version of this file defined its own look-alike
`@pyfilter` that took `(data, direction)`, and that was a mistake of exactly the kind a
regex engine written twice would be: a filter written from the docs loaded on one
transport and failed on the other, and the docs were right about neither.
Each connection gets its own set of module globals, which is what the API promises —
code at module level runs once per stream and the same globals are reused for every
packet of that stream. That is why every frame carries a connection id, and why the
engine sends a close frame: without one, a service would accumulate one set of globals
per connection for as long as it ran.
Frames are length-prefixed and binary:
in [u32 len][u8 kind][u64 connection][payload]
kind: 0 client->server, 1 server->client,
2 connection closed, 3 connection opened
out [u32 len][u8 verdict][u8 name len][name]
verdict: 0 accept, 1 reject
The name is the `@pyfilter` function that decided, so a block is attributed to one
function of a file rather than to the file as a whole — a file routinely holds several,
and "this filter blocked 900 connections" is not an answer when you are trying to find
out which of them is doing it.
Neither an open nor a close frame is answered. A close has no verdict to give about a
connection that is already over; an open carries that connection's metadata — the
addresses and ports, as JSON — which the filter reads and cannot change.
Nothing below the application layer crosses in the other direction. There used to be a
`FAKE:IP:TCP:HEADERS:` prefix here, handed to the library as `raw_packet` so that a
filter written against real headers would find something there. It found a lie: the
proxy terminates the connection and writes its own headers, so editing them changed
nothing, while the same filter on NFQUEUE edited a real packet. Metadata in, a verdict
out — that is the one contract both network layers can honestly keep. There is no
payload coming back either: rewriting went with `UNSTABLE_MANGLE`, for the reason the
library's `Action` gives.
"""
import json
import re
import struct
import sys
import time
import traceback
VERDICT_ACCEPT = 0
VERDICT_REJECT = 1
# Sent once, after the user's file has been compiled. It is what lets the engine refuse
# a ruleset whose code does not load, instead of discovering it later as traffic that
# quietly stopped being filtered.
READY = 0xFF
#: Written to stderr when the user's code raises, beside the traceback itself.
#:
#: The traceback says a filter broke and names its file and line, which is what an
#: operator needs to fix it. It does not say what happened to the *connection*, and that
#: is the part they have to know: it went through unfiltered. The engine cannot report it
#: because it never learns of it — the exception is caught here and answered with ACCEPT,
#: so nothing crosses the frame boundary to tell it apart from a filter that simply
#: agreed. So it is marked on the diagnostics channel instead, and `Transport._pump_stderr`
#: turns the marker back into the one sentence `_on_exception` writes for either network
#: layer. A wire token like `BLOCKED` or `[fatal]`, and spelled the same in both places.
EXCEPTION_MARK = "[exception] [filter]"
#: How long between two tracebacks out of the same worker.
#:
#: A filter that throws usually throws on every packet, and a traceback is a dozen lines.
#: Unthrottled, one broken filter writes thousands of lines a minute into a log that
#: holds a few hundred — so the flood does not merely repeat itself, it pushes out the
#: first traceback, which was the one worth reading, along with everything that was in
#: the log before the filter broke.
#:
#: The mark is *not* throttled with it: it is one short token the engine turns into a
#: single rate-limited sentence on the other side, and it is what carries the count. So
#: the operator still learns how often this is happening, and gets one traceback rather
#: than the same one several hundred times.
TRACEBACK_QUIET = 30
KIND_C2S = 0
KIND_S2C = 1
KIND_CLOSE = 2
KIND_OPEN = 3
def _build_filter_code(path: str, enabled: list[str] | None = None) -> str:
"""Turn the user's file into the module the library expects to execute.
Identical in shape to what `firegex.pyfilters.proxysim` builds, because it has to be:
the library decides which filters exist and when each one is callable from these
trailing lines, and a different preamble would mean a different set of filters.
`enabled` is the subset the operator left switched on; `None` means all of them. It
is intersected with what the file actually defines rather than trusted, so a stale
selection naming a function that has since been deleted cannot stop the file from
loading — the library would refuse the whole module over one missing name.
No protocol is written in. The file shows which one it speaks by what its filters
ask for, so there is nothing here that could disagree with the code — an earlier
version passed one in from the outside, and a filter asking for an `HttpRequest`
under a file declared `tcp` was refused with an error about its own annotation.
"""
from firegex.pyfilters.internals import get_filter_names
with open(path) as f:
source = f.read()
names = get_filter_names(source)
if enabled is not None:
wanted = set(enabled)
names = [name for name in names if name in wanted]
return (
source
+ "\n\n__firegex_pyfilter_enabled = ["
+ ", ".join(repr(name) for name in names)
+ "]\n"
"import firegex.pyfilters.internals\n"
"firegex.pyfilters.internals.compile(globals())\n"
)
def _where_in_source(source: str, exc: BaseException, filename: str) -> dict:
"""Point at the line of the operator's own file, when there is one to point at.
Three ways, in order of how directly they know:
* a `SyntaxError` carries its own position, and the code never ran;
* a traceback frame belonging to the file — the module body raised while loading.
`<string>` counts as theirs: the library execs the source to find its filters
before the module is built under its real name, and everything the library itself
runs from has a real path, so an anonymous frame is the user's;
* neither, which is the interesting case. The library refuses a filter by *name*
("Invalid type annotation X for function Y", "Parameter 'p' of Z has none"), from
frames that are all its own, so any identifier in the message is looked back up
among the file's own `def`s. Without this the operator is told which function is
wrong and left to go and find it.
"""
lines = source.splitlines()
def at(line: int) -> dict:
return {"line": line, "column": 0,
"text": lines[line - 1].rstrip() if 0 < line <= len(lines) else ""}
if isinstance(exc, SyntaxError) and exc.lineno:
return {"line": exc.lineno, "column": exc.offset or 0,
"text": (exc.text or "").rstrip()}
mine = {filename, "<string>"}
frames = [f for f in traceback.extract_tb(exc.__traceback__) if f.filename in mine]
if frames:
return at(frames[-1].lineno or 0)
# The *last* definition of a name, because that is the one Python is left holding:
# a file that defines `check` twice runs the second, and pointing at the first would
# send the operator to a function that is not the one being complained about.
defs = {}
for number, text in enumerate(lines, start=1):
found = re.match(r"\s*(?:async\s+)?def\s+(\w+)\s*\(", text)
if found:
defs[found.group(1)] = number
for word in re.findall(r"\w+", str(exc)):
if word in defs:
return at(defs[word])
return {"line": 0, "column": 0, "text": ""}
#: The rename this package went through, and the one message worth saying about it.
#:
#: `firegex.nfproxy` was the old spelling of `firegex.pyfilters`, aliased through 4.x and
#: removed in 5.0.0. `from firegex.nfproxy import pyfilter` is the first line of every
#: filter written before that, and the error Python raises for it — "No module named
#: 'firegex.nfproxy'" — is accurate and tells the operator nothing about what to write
#: instead. They meet it here, in the editor, with the line already marked; the
#: alternative is meeting it at the first start after an upgrade, which at a competition
#: is the start of a round.
RENAMED_IMPORT = "firegex.nfproxy"
RENAMED_TO = "firegex.pyfilters"
def _explain(exc: BaseException, message: str) -> str:
"""Add what to do about it, where there is something to say."""
if isinstance(exc, ImportError) and RENAMED_IMPORT in (str(exc) or ""):
return (f"{message}. `{RENAMED_IMPORT}` was renamed to `{RENAMED_TO}` in 5.0.0 "
f"and the old name no longer resolves: change the import at the top of "
f"this file, and the rest of the filter works unchanged")
return message
def check(path: str) -> dict:
"""Would this file load, and what does it define?
Run by the backend before a save is accepted, in a process of its own — which is
both why the answer can be trusted and why a filter with a `while True:` at module
level cannot take the interface down with it. It is the *same* build and the same
compile the datapath performs, so what is accepted here is what will run.
"""
from firegex.pyfilters.internals import get_filters_info
try:
with open(path) as f:
source = f.read()
except OSError as e:
return {"ok": False, "error": {"type": "OSError", "message": str(e),
"line": 0, "column": 0, "text": "",
"traceback": ""}}
try:
infos = get_filters_info(source) if source.strip() else []
# Not just "does it parse": the whole module, exactly as the worker builds it,
# so an error the library only raises at compile time is raised here too.
Filters(_build_filter_code(path), path)
except BaseException as e: # noqa: BLE001 - anything the user's code can raise
return {
"ok": False,
"error": {
"type": type(e).__name__,
# A SyntaxError's text carries "(<string>, line N)", which names a file
# the operator never wrote and repeats a position reported separately.
"message": _explain(e, re.sub(
r"\s*\((?:<[^>]*>|[^()]*\.py), line \d+\)$", "",
str(e) or type(e).__name__)),
**_where_in_source(source, e, path),
"traceback": traceback.format_exc(),
},
}
return {
"ok": True,
"proto": infos[0].proto if infos else "tcp",
"filters": [info.name for info in infos],
# Which models the file actually asks for. The protocol alone does not say: a
# `tcp` file wanting only a RawPacket runs on datagrams too, while one wanting a
# TCP stream cannot — and the difference decides whether attaching it to a UDP
# service gives you a filter or a filter that never fires.
"models": sorted({
annotation.__name__
for info in infos
for annotation in info.params
}),
}
class Filters:
"""The compiled filter module, plus one set of globals per live connection."""
def __init__(self, code: str, filename: str = "<filter>"):
# Compiled once, under the operator's own file name: every connection reuses the
# code object instead of re-parsing the source, and a traceback names the file
# they wrote rather than `<string>`, with line numbers that match what they see
# in the editor. The preamble is appended *after* their source precisely so
# those numbers line up.
self.code = compile(code, filename, "exec")
self.contexts: dict[int, dict] = {}
#: What the engine said about each live connection, kept so every chunk of it
#: carries the same addresses without them being repeated on the wire.
self.endpoints: dict[int, dict] = {}
# Run once here so a file that cannot load is a startup failure, which the
# engine turns into a refused ruleset rather than a service that silently stops
# filtering.
probe: dict = {}
exec(self.code, probe, probe) # noqa: S102 - running the user's filter is the feature
def context(self, connection: int) -> dict:
ctx = self.contexts.get(connection)
if ctx is None:
ctx = {}
exec(self.code, ctx, ctx) # noqa: S102
# No open frame came for it, so this worker is taking over a connection
# already under way — new code, or one restarted after the last died — and
# its first chunk can be the middle of a message. The library then carries
# what it cannot read instead of judging half a message; see
# `_met_mid_message` in the HTTP model.
if connection not in self.endpoints:
ctx["__firegex_joined_late"] = True
self.contexts[connection] = ctx
return ctx
def opened(self, connection: int, payload: bytes) -> None:
try:
meta = json.loads(payload.decode())
except (ValueError, UnicodeDecodeError):
return
if isinstance(meta, dict):
self.endpoints[connection] = meta
def close(self, connection: int) -> None:
self.contexts.pop(connection, None)
self.endpoints.pop(connection, None)
def run(self, connection: int, data: bytes, is_input: bool) -> tuple[int, str]:
"""One chunk through the chain. Returns the verdict, and which function decided."""
from firegex.pyfilters import ACCEPT, DROP, REJECT
ctx = self.context(connection)
# Missing endpoints are empty strings and zeros rather than an error: the engine
# sends them best-effort, and a filter losing the client's address is a worse
# outcome than a filter that sees "" for it, but neither is a reason to stop
# filtering the traffic.
meta = self.endpoints.get(connection, {})
client_ip = meta.get("client_ip", "")
client_port = int(meta.get("client_port", 0))
server_ip = meta.get("server_ip", "")
server_port = int(meta.get("server_port", 0))
ctx["__firegex_packet_info"] = {
"data": data,
"is_input": is_input,
"is_ipv6": bool(meta.get("is_ipv6", False)),
# Said by the engine rather than assumed: this worker serves TCP connections,
# UDP flows and QUIC streams alike. Two words rather than one, because they
# answer different questions — `is_tcp` is what is on the wire, `l4` is what
# the library's stream and HTTP models are built on, and QUIC answers those
# two differently. Absent, `l4` derives from the other: the NFQUEUE engines
# carry TCP and UDP and have nothing else to say.
"is_tcp": bool(meta.get("is_tcp", True)),
"l4": meta.get("l4"),
"src_ip": client_ip if is_input else server_ip,
"src_port": client_port if is_input else server_port,
"dst_ip": server_ip if is_input else client_ip,
"dst_port": server_port if is_input else client_port,
}
try:
exec( # noqa: S102
"firegex.pyfilters.internals.handle_packet(globals())", ctx, ctx
)
finally:
ctx.pop("__firegex_packet_info", None)
result = ctx.pop("__firegex_pyfilter_result", None)
if not isinstance(result, dict):
# The library did not answer. Saying nothing about a chunk is not a reason
# to stop carrying it.
print("[warn] [pyworker] no verdict from the filter chain", file=sys.stderr)
return VERDICT_ACCEPT, ""
action = result.get("action")
# Which function decided, so the block lands on it rather than on the whole file.
matched = result.get("matched_by")
matched = matched if isinstance(matched, str) else ""
if action == ACCEPT.value:
return VERDICT_ACCEPT, ""
if action == REJECT.value:
return VERDICT_REJECT, matched
if action == DROP.value:
# A proxy has no way to swallow one chunk and keep the stream coherent —
# the application on the other side would be reading a hole. Closing is the
# honest equivalent, and it is what the operator meant by dropping.
return VERDICT_REJECT, matched
print(f"[warn] [pyworker] unknown action {action!r}", file=sys.stderr)
return VERDICT_ACCEPT, ""
def _read_exactly(stream, count: int) -> bytes | None:
buf = b""
while len(buf) < count:
chunk = stream.read(count - len(buf))
if not chunk:
return None
buf += chunk
return buf
def _write(stream, verdict: int, matched: str) -> None:
# The name is length-prefixed with a single byte: a Python identifier fits, and a
# separator would have to be one that cannot appear in one.
name = matched.encode()[:255]
stream.write(struct.pack(">I", len(name) + 2) + bytes([verdict, len(name)]) + name)
stream.flush()
def main() -> int:
if len(sys.argv) < 2:
print("usage: pyworker.py <filter file> [* | =fn,fn] | --check <filter file>",
file=sys.stderr)
return 2
if sys.argv[1] == "--check":
# One JSON object on stdout and nothing else, so the caller never has to guess
# which part of the output was the answer.
if len(sys.argv) < 3:
print("usage: pyworker.py --check <filter file>", file=sys.stderr)
return 2
# Checking a file means *running* its module body, and `print()` is the most
# natural thing in the world to reach for while writing a filter. On stdout it
# would land in the middle of the answer: the backend parses this output as JSON,
# a stray line makes it unparseable, and the operator is told `the check produced
# no answer` about a file whose only sin was printing while it loaded.
#
# So the real stdout is put aside before the user's code can reach it, and
# anything the file prints goes to the diagnostics channel — the same swap the
# worker itself makes below, for the same reason.
answer = sys.stdout
sys.stdout = sys.stderr
try:
result = check(sys.argv[2])
finally:
sys.stdout = answer
answer.write(json.dumps(result))
answer.flush()
return 0
# `*` (or nothing at all) means every function the file defines, which is what a
# hand-run worker wants; `=a,b` means exactly those, and a bare `=` means none.
# A marker rather than a bare list because "all" and "none" are both real answers
# and an empty string cannot stand for both.
enabled = None
if len(sys.argv) > 2 and sys.argv[2].startswith("="):
enabled = [name.strip() for name in sys.argv[2][1:].split(",") if name.strip()]
try:
filters = Filters(_build_filter_code(sys.argv[1], enabled), sys.argv[1])
except Exception:
traceback.print_exc()
return 1
stdin = sys.stdin.buffer
stdout = sys.stdout.buffer
# The user's code shares this process, and `print()` is the most natural thing in
# the world to reach for while debugging a filter. On stdout it would land in the
# middle of a length-prefixed frame: the engine reads the text as a frame header,
# sees an absurd length, and kills the worker — so the filter would be restarted on
# every packet and the operator would be told nothing except that it stopped
# working. Their output belongs on the diagnostics channel, which the backend now
# captures into the service log, so it goes there and stdout stays the protocol.
sys.stdout = sys.stderr
stdout.write(struct.pack(">I", 2) + bytes([READY, 0]))
stdout.flush()
# Monotonic rather than wall clock: the machine's clock moving is not a reason to
# start printing tracebacks again, or to stop.
last_traceback = -TRACEBACK_QUIET
while True:
header = _read_exactly(stdin, 4)
if header is None:
return 0 # the engine went away; so do we
(length,) = struct.unpack(">I", header)
body = _read_exactly(stdin, length)
if body is None or len(body) < 9:
return 0
kind = body[0]
(connection,) = struct.unpack(">Q", body[1:9])
payload = body[9:]
if kind == KIND_OPEN:
filters.opened(connection, payload)
continue # deliberately unanswered
if kind == KIND_CLOSE:
filters.close(connection)
continue # deliberately unanswered
try:
verdict, matched = filters.run(connection, payload, kind == KIND_C2S)
except Exception:
# One connection's filter blowing up must not take the process down: the
# other connections it is serving would lose their filtering with it. The
# chunk is accepted, which is the fail-open half — and the mark is what makes
# that visible rather than silent.
now = time.monotonic()
if now - last_traceback >= TRACEBACK_QUIET:
last_traceback = now
traceback.print_exc()
print(EXCEPTION_MARK, file=sys.stderr, flush=True)
verdict, matched = VERDICT_ACCEPT, ""
_write(stdout, verdict, matched)
if __name__ == "__main__":
sys.exit(main())

254
backend/modules/services/stats.py Normal file
View File

@ -0,0 +1,254 @@
"""What each rule has actually refused, and when.
The service list carries a running total per filter, per pattern and per function, which
answers "is this rule doing anything at all". It does not answer the question an operator
asks during a competition — *when* did this start, and which rule is carrying it right
now — because a cumulative counter looks the same whether a filter blocked a thousand
connections an hour ago or is blocking them this minute.
So blocks are also counted into fixed time buckets, **per rule** rather than per filter.
Keeping the finer key is what lets every number on the page answer the same question: ask
for the last fifteen minutes and the chart, the totals and the shares all mean the last
fifteen minutes. Storing only per filter would have left the chart honouring the range
and the table beside it quietly reporting all of time.
Two properties keep that from becoming a liability of its own, and they are the same two
the live log has:
* **Bounded.** Only the last `HISTORY_BUCKETS` are kept, and older rows are pruned as new
ones are written. A service that has been refusing traffic for a week stores the same
amount as one that started an hour ago.
* **Coalesced.** A block bumps a counter in memory, and memory is written out at most
every `FLUSH_INTERVAL` seconds. A service refusing thousands of connections a minute
must not be able to take the backend down through its own bookkeeping — which is
exactly what one row per block would do.
There is deliberately no background task. Flushing happens on the next write once the
interval has passed, and always on a read, so what the interface shows is current without
anything having to be started, cancelled or waited for at shutdown.
"""
import math
import time
#: The finest resolution anything is stored at. Every range the interface offers is an
#: aggregate of these, so a wider window costs no more rows — only wider bars.
BUCKET_SECONDS = 60
#: How far back the history goes. Two days at a minute each: long enough to cover a
#: competition and the night before it, small enough to stay a few thousand rows.
HISTORY_BUCKETS = 48 * 60
#: How long counts may sit in memory before being written out.
FLUSH_INTERVAL = 5
_pending: dict[tuple[str, str, str, int], int] = {}
_last_flush = 0.0
def bucket_of(when: float | None = None) -> int:
return int((when if when is not None else time.time()) // BUCKET_SECONDS) * BUCKET_SECONDS
def retention_start() -> int:
"""The oldest instant anything is still known about."""
return bucket_of() - (HISTORY_BUCKETS - 1) * BUCKET_SECONDS
def record(service_id: str, filter_id: str, rule_id: str, when: float | None = None) -> None:
"""Count one refusal. Cheap on purpose: this runs on the block path.
`rule_id` is whatever the datapath reported — a pattern, or `<filter>/<function>`,
or the filter itself. `filter_id` is carried alongside it so the chart can group by
filter without a join back to a table the rule may have been deleted from.
"""
key = (service_id, filter_id, rule_id, bucket_of(when))
_pending[key] = _pending.get(key, 0) + 1
def flush(db, force: bool = False) -> None:
"""Write the counts out, and drop whatever has aged out of the window."""
global _last_flush
now = time.time()
if not force and now - _last_flush < FLUSH_INTERVAL:
return
_last_flush = now
if not _pending:
return
batch = list(_pending.items())
_pending.clear()
queries = [
(
"INSERT INTO block_history (service_id, filter_id, rule_id, bucket, blocked) "
"VALUES (?, ?, ?, ?, ?) ON CONFLICT(rule_id, bucket) "
"DO UPDATE SET blocked = blocked + excluded.blocked;",
service_id,
filter_id,
rule_id,
bucket,
count,
)
for (service_id, filter_id, rule_id, bucket), count in batch
]
# Pruning rides along with the write rather than being scheduled: there is no moment
# at which rows can accumulate without something also having written one.
queries.append(("DELETE FROM block_history WHERE bucket < ?;", retention_start()))
db.queries(queries)
def forget(db, service_id: str) -> None:
"""A deleted service's history has nothing left to be about."""
for key in [k for k in _pending if k[0] == service_id]:
del _pending[key]
db.query("DELETE FROM block_history WHERE service_id = ?;", service_id)
def forget_filter(db, filter_id: str) -> None:
for key in [k for k in _pending if k[1] == filter_id]:
del _pending[key]
db.query("DELETE FROM block_history WHERE filter_id = ?;", filter_id)
def disown_rule(db, filter_id: str, rule_id: str) -> None:
"""Take a rule's history away from it without taking it off the chart.
Editing a pattern keeps the row — its place in the filter, and whatever else was set
on it — but the counts belonged to the text that used to be there, so the new text
must not inherit them: a per-pattern table that carried them over would credit blocks
to a matcher that never made them.
What it must *not* do is delete them. Those connections were refused, by this filter,
in those minutes, and the chart is answering *when this service was under attack* —
a question a later typo fix has nothing to say about. Dropping the rows took a bite
out of the timeline every time a pattern was corrected, which on a filter holding one
pattern emptied it. So they are re-keyed to the filter itself, which is already the
token the datapath reports when nothing finer named itself: the chart and the
per-filter totals group by filter and do not change at all, while the per-pattern
table correctly credits the new text with nothing.
The pending batch moves with them, or the old pattern's last few hits would be
flushed under the new one.
"""
if filter_id == rule_id:
return # already the filter's own; there is nothing to take it away from
for key in [k for k in _pending if k[2] == rule_id]:
service_id, held_by, _, bucket = key
moved = (service_id, held_by, filter_id, bucket)
_pending[moved] = _pending.get(moved, 0) + _pending.pop(key)
db.queries([
(
"INSERT INTO block_history (service_id, filter_id, rule_id, bucket, blocked) "
"SELECT service_id, filter_id, ?, bucket, blocked FROM block_history "
"WHERE rule_id = ? ON CONFLICT(rule_id, bucket) "
"DO UPDATE SET blocked = blocked + excluded.blocked;",
filter_id,
rule_id,
),
("DELETE FROM block_history WHERE rule_id = ?;", rule_id),
])
#: Most bars any chart may hold, whatever step is asked for. Past this a plot stops
#: being a shape and becomes a texture, and the browser is drawing a rectangle per bar.
MAX_STEPS = 400
def bucket_width(start: int, end: int, target: int, want: int | None = None) -> int:
"""How wide one bar has to be for a range to stay readable.
Two ways to say it, and both end at the same place. Without `want`, the window is
shown in roughly `target` bars whatever it spans: twelve hours at one-minute
resolution is seven hundred bars in a few hundred pixels. With `want`, the operator
has picked the step themselves — five minutes is five minutes whether they are
looking at an hour or at a day, which is what makes two ranges comparable by eye.
Either way the answer is a whole number of stored buckets, so widening is addition
and never interpolation, and either way it is floored at `MAX_STEPS` bars: a step
fine enough to melt the chart is refused by widening it, not by refusing the request,
and the width actually used is reported back so the interface can say what it drew.
"""
span = max(BUCKET_SECONDS, end - start + BUCKET_SECONDS)
least = math.ceil(span / BUCKET_SECONDS / MAX_STEPS)
if want:
steps = max(1, round(want / BUCKET_SECONDS))
else:
steps = max(1, math.ceil(span / BUCKET_SECONDS / max(1, target)))
return BUCKET_SECONDS * max(1, steps, least)
def window(db, service_id: str, start: int, end: int, target: int = 60,
want: int | None = None) -> dict:
"""What was refused between two instants, per filter, in even steps.
Every step is present whether or not anything happened in it, because a chart with
the quiet minutes missing is a chart that lies about the shape of a burst.
"""
flush(db, force=True)
start, end = bucket_of(start), bucket_of(end)
if end < start:
start, end = end, start
width = bucket_width(start, end, target, want)
# Aligned to the step, so the same range always lands on the same edges and a
# refresh does not shuffle every bar sideways.
first = (start // width) * width
edges = list(range(first, end + width, width))
index = {edge: i for i, edge in enumerate(edges)}
series: dict[str, list[int]] = {}
rows = db.query(
"SELECT filter_id, bucket, SUM(blocked) blocked FROM block_history "
"WHERE service_id = ? AND bucket >= ? AND bucket <= ? "
"GROUP BY filter_id, bucket;",
service_id,
first,
end,
)
for row in rows:
slot = index.get((int(row["bucket"]) // width) * width)
if slot is None:
continue
counts = series.setdefault(row["filter_id"], [0] * len(edges))
counts[slot] += int(row["blocked"])
return {"edges": edges, "width": width, "series": series}
def totals_by_filter(db, service_id: str, start: int, end: int) -> dict[str, int]:
"""What each filter refused in the range, summed over the rules inside it.
Not a lookup of the filter's own id: that token only appears when a whole filter
refused something without naming a rule. A regex block is reported by its pattern
and a Python one by `<filter>/<function>`, so a filter's total is the sum of what
its rules did — which is also what makes the per-filter figures add up to the total.
"""
flush(db, force=True)
start, end = bucket_of(start), bucket_of(end)
if end < start:
start, end = end, start
return {
row["filter_id"]: int(row["blocked"])
for row in db.query(
"SELECT filter_id, SUM(blocked) blocked FROM block_history "
"WHERE service_id = ? AND bucket >= ? AND bucket <= ? GROUP BY filter_id;",
service_id,
start,
end,
)
}
def totals(db, service_id: str, start: int, end: int) -> dict[str, int]:
"""What each rule refused in the range, keyed by whatever the datapath reported."""
flush(db, force=True)
start, end = bucket_of(start), bucket_of(end)
if end < start:
start, end = end, start
return {
row["rule_id"]: int(row["blocked"])
for row in db.query(
"SELECT rule_id, SUM(blocked) blocked FROM block_history "
"WHERE service_id = ? AND bucket >= ? AND bucket <= ? GROUP BY rule_id;",
service_id,
start,
end,
)
}

1568
backend/modules/services/transports.py Normal file
View File

File diff suppressed because it is too large Load Diff

1
backend/modules/tls/__init__.py
View File

@ -1 +0,0 @@
# TLS Decrypt Module

92
backend/modules/tls/manager.py
View File

@ -1,92 +0,0 @@
import hashlib
from ipaddress import ip_interface
from utils.sqlite import SQLite
def ip_parse(ip: str) -> str:
try:
return str(ip_interface(ip).network)
except Exception:
return ip
def get_tls_ports(ip: str, port: int) -> tuple[int, int]:
ip_addr = ip.split("/")[0]
key = f"{ip_addr}:{port}"
h = int.from_bytes(hashlib.sha256(key.encode('utf-8')).digest()[:4], 'big')
ssl_port = 10000 + (h % 10000)
clear_port = 20000 + (h % 10000)
return ssl_port, clear_port
TLS_SCHEMA = {
'tls_streams': {
'id': 'VARCHAR(100) PRIMARY KEY',
'name': 'VARCHAR(100) NOT NULL',
'ip_int': 'VARCHAR(100) NOT NULL',
'port': 'INT NOT NULL',
'cert': 'TEXT NOT NULL',
'key': 'TEXT NOT NULL',
'status': 'VARCHAR(50) NOT NULL', # 'active' or 'stop'
'ssl_port': 'INT NOT NULL',
'clear_port': 'INT NOT NULL',
'UNIQUE': '(ip_int, port)'
}
}
class TLSManager:
def __init__(self, db_path='db/nft-tls.db'):
self.db = SQLite(db_path, schema=TLS_SCHEMA)
self.db.init()
def add_stream(self, stream_id: str, name: str, ip_int: str, port: int, cert: str, key: str, status: str = 'stop') -> None:
ip_int = ip_parse(ip_int)
ssl_port, clear_port = get_tls_ports(ip_int, port)
self.db.query(
"INSERT INTO tls_streams (id, name, ip_int, port, cert, key, status, ssl_port, clear_port) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?);",
stream_id, name, ip_int, port, cert, key, status, ssl_port, clear_port
)
def get_streams(self) -> list[dict]:
return self.db.query("SELECT * FROM tls_streams;")
def get_stream(self, stream_id: str) -> dict | None:
res = self.db.query("SELECT * FROM tls_streams WHERE id = ?;", stream_id)
return res[0] if res else None
def get_stream_by_ip_port(self, ip_int: str, port: int) -> dict | None:
ip_int = ip_parse(ip_int)
res = self.db.query("SELECT * FROM tls_streams WHERE ip_int = ? AND port = ?;", ip_int, port)
return res[0] if res else None
def update_status(self, stream_id: str, status: str) -> None:
self.db.query("UPDATE tls_streams SET status = ? WHERE id = ?;", status, stream_id)
def update_stream(self, stream_id: str, name: str | None = None, ip_int: str | None = None, port: int | None = None, cert: str | None = None, key: str | None = None) -> None:
stream = self.get_stream(stream_id)
if not stream:
return
fields: dict = {}
if name is not None:
fields["name"] = name
if cert is not None:
fields["cert"] = cert
if key is not None:
fields["key"] = key
new_ip = ip_parse(ip_int) if ip_int is not None else stream["ip_int"]
new_port = port if port is not None else stream["port"]
if ip_int is not None:
fields["ip_int"] = new_ip
if port is not None:
fields["port"] = new_port
if ip_int is not None or port is not None:
ssl_port, clear_port = get_tls_ports(new_ip, new_port)
fields["ssl_port"] = ssl_port
fields["clear_port"] = clear_port
if not fields:
return
set_clause = ", ".join(f"{k} = ?" for k in fields)
self.db.query(f"UPDATE tls_streams SET {set_clause} WHERE id = ?;", *fields.values(), stream_id)
def delete_stream(self, stream_id: str) -> None:
self.db.query("DELETE FROM tls_streams WHERE id = ?;", stream_id)
def get_active_streams(self) -> list[dict]:
return self.db.query("SELECT * FROM tls_streams WHERE status = 'active';")

43
backend/modules/tls/nftables.py
View File

@ -1,43 +0,0 @@
from utils import NFTableManager, nftables_int_to_json, ip_family
from modules.tls.manager import TLSManager
class TLSTables(NFTableManager):
nat_chain = "tls_nat"
def __init__(self):
super().__init__([
{"add":{"chain":{ #NAT chain for DNAT
"family":"inet",
"table":self.table_name,
"name":self.nat_chain,
"type":"nat",
"hook":"prerouting",
"prio":-100,
"policy":"accept"
}}}
],[
{"flush":{"chain":{"table":self.table_name,"family":"inet", "name":self.nat_chain}}},
])
def reload(self):
self.init()
manager = TLSManager()
streams = manager.get_active_streams()
cmds = []
for srv in streams:
loopback_ip = "::1" if ip_family(srv["ip_int"]) == "ip6" else "127.0.0.1"
cmds.append({"insert":{ "rule": { # Redirect TLS traffic to local Nginx SSL proxy
"family": "inet",
"table": self.table_name,
"chain": self.nat_chain,
"expr": [
{'match': {'left': {'payload': {'protocol': ip_family(srv["ip_int"]), 'field': 'daddr'}}, 'op': '==', 'right': nftables_int_to_json(srv["ip_int"])}},
{'match': {'left': {'payload': {'protocol': 'tcp', 'field': 'dport'}}, 'op': '==', 'right': int(srv["port"])}},
{'match': {'left': {'meta': {'key': 'iifname'}}, 'op': '!=', 'right': 'lo'}},
{'dnat': {'family': ip_family(srv["ip_int"]), 'addr': loopback_ip, 'port': int(srv["ssl_port"])}}
]
}}})
if cmds:
self.cmd(*cmds)
tls_firewall = TLSTables()

83
backend/modules/tls/nginx.py
View File

@ -1,83 +0,0 @@
import subprocess
import os
import glob
from modules.tls.manager import TLSManager
NGINX_CONF_PATH = "/tmp/firegex_nginx.conf"
NGINX_PID_PATH = "/tmp/firegex_nginx.pid"
NGINX_LOG_PATH = "/tmp/firegex_nginx_error.log"
def update_nginx() -> None:
manager = TLSManager()
streams = manager.get_active_streams()
if not streams:
subprocess.run(["nginx", "-c", NGINX_CONF_PATH, "-s", "stop", "-e", NGINX_LOG_PATH], stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)
# Clean up certificates, configuration, and pid
for pattern in ("/tmp/firegex_cert_*", "/tmp/firegex_key_*", NGINX_CONF_PATH, NGINX_PID_PATH, NGINX_LOG_PATH):
for path in (glob.glob(pattern) if "*" in pattern else [pattern]):
try:
os.remove(path)
except OSError:
pass
return
servers_config = []
for srv in streams:
srv_id = srv["id"]
cert_path = f"/tmp/firegex_cert_{srv_id}.crt"
key_path = f"/tmp/firegex_key_{srv_id}.key"
with open(cert_path, "w") as f:
f.write(srv.get("cert") or "")
with open(key_path, "w") as f:
f.write(srv.get("key") or "")
ssl_port = srv["ssl_port"]
clear_port = srv["clear_port"]
ip_addr = srv["ip_int"].split("/")[0]
is_ipv6 = ":" in ip_addr
loopback_ip = "[::1]" if is_ipv6 else "127.0.0.1"
dest_ip = f"[{ip_addr}]" if is_ipv6 else ip_addr
servers_config.append(
f" server {{\n"
f" listen {loopback_ip}:{ssl_port} ssl;\n"
f" proxy_pass {loopback_ip}:{clear_port};\n"
f" ssl_certificate {cert_path};\n"
f" ssl_certificate_key {key_path};\n"
f" ssl_protocols TLSv1.2 TLSv1.3;\n"
f" }}\n"
f" server {{\n"
f" listen {loopback_ip}:{clear_port};\n"
f" proxy_pass {dest_ip}:{srv['port']};\n"
f" proxy_ssl on;\n"
f" proxy_ssl_verify off;\n"
f" proxy_ssl_protocols TLSv1.2 TLSv1.3;\n"
f" }}"
)
config_content = ""
if os.path.isdir("/usr/share/nginx/modules"):
config_content += "include /usr/share/nginx/modules/*.conf;\n"
config_content += (
f"pid {NGINX_PID_PATH};\n"
f"error_log {NGINX_LOG_PATH} info;\n\n"
f"events {{\n"
f" worker_connections 1024;\n"
f"}}\n\n"
f"stream {{\n"
+ "\n".join(servers_config) + "\n"
f"}}\n"
)
with open(NGINX_CONF_PATH, "w") as f:
f.write(config_content)
# Try to reload. If Nginx is not running, start it.
res = subprocess.run(["nginx", "-c", NGINX_CONF_PATH, "-s", "reload", "-e", NGINX_LOG_PATH], stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)
if res.returncode != 0:
subprocess.run(["nginx", "-c", NGINX_CONF_PATH, "-e", NGINX_LOG_PATH], check=True)

22
backend/modules/tls/service.py
View File

@ -1,22 +0,0 @@
from modules.tls.manager import TLSManager
from modules.tls.nginx import update_nginx
from modules.tls.nftables import tls_firewall
from utils import socketio_emit
async def reload_tls():
update_nginx()
tls_firewall.reload()
await socketio_emit(["tls_streams"])
async def activate_stream(stream_id: str | None) -> bool:
"""Activate a TLS stream if it exists and isn't already active. Returns False if the stream doesn't exist."""
if not stream_id:
return False
manager = TLSManager()
stream = manager.get_stream(stream_id)
if not stream:
return False
if stream["status"] != "active":
manager.update_status(stream_id, "active")
await reload_tls()
return True

1061
backend/proxysrc/Cargo.lock generated Normal file
View File

File diff suppressed because it is too large Load Diff

78
backend/proxysrc/Cargo.toml Normal file
View File

@ -0,0 +1,78 @@
[package]
name = "fgex-proxy"
version = "0.1.0"
edition = "2021"
description = "Proxy datapath engine for firegex (fail-open spike)"
license = "GPL-3.0-or-later"
[dependencies]
tokio = { version = "1", features = ["rt-multi-thread", "net", "io-util", "io-std", "macros", "time", "sync"] }
# Raw setsockopt: IP_TRANSPARENT and SO_ORIGINAL_DST have no safe wrapper.
libc = "0.2"
socket2 = "0.6"
# Filter rules arrive as one JSON line on stdin
serde = { version = "1", features = ["derive"] }
serde_json = "1"
# TLS terminated by the engine, so the plaintext never leaves the process.
# `ring` rather than the default aws-lc-rs: it builds without cmake or nasm, which
# keeps the Docker stage a plain `cargo build`.
tokio-rustls = { version = "0.26", default-features = false, features = ["ring", "tls12", "logging"] }
rustls = { version = "0.23", default-features = false, features = ["ring", "tls12", "std", "logging"] }
rustls-pemfile = "2"
# QUIC, terminated by the engine for the same reason TLS is: after the Initial packet
# everything — frames, stream boundaries, even the packet number — is encrypted, so a
# layer that only forwards has nothing a filter could be shown. `ring` again, and the
# same rustls the TCP side already links.
quinn = { version = "0.11", default-features = false, features = ["runtime-tokio", "rustls-ring", "log"] }
# HTTP/3, terminated on top of QUIC. A binding rather than a second implementation: the
# frames and QPACK are somebody else's problem, and what this engine does with them —
# presenting each exchange to the chain as the HTTP/1.1 it would have been — is the part
# that belongs here.
h3 = "0.0.8"
h3-quinn = "0.0.10"
# HTTP/2, terminated on TCP for the reason HTTP/3 is terminated on QUIC: its method, path
# and headers are HPACK-compressed, so a layer that only forwards shows a filter a
# compression format. A binding again — HPACK, multiplexing and flow control are the
# crate's problem, and a second implementation of HTTP/2 in this tree is exactly what
# `AGENTS.md` forbids.
h2 = "0.4"
# HTTP/1.1 towards the service, for the one thing this engine could not do: speak an
# older version to the service than the client used. A client edge that is terminated —
# HTTP/3 over QUIC — is already rendered to the chain as HTTP/1.1, so putting those same
# bytes on a socket is not a translation invented here, it is the rendering being sent
# where the service can read it. The *response* is what needs the crate: HTTP/1.1 is the
# one version this engine never parsed, and inventing a parser for it beside hyper's is
# the thing `AGENTS.md` forbids. Client and http1 only — nothing here serves HTTP/1.1,
# and the h2 client above stays h2's.
hyper = { version = "1", default-features = false, features = ["client", "http1"] }
hyper-util = { version = "0.1", default-features = false, features = ["tokio"] }
http-body-util = "0.1"
http-body = "1"
bytes = "1"
http = "1"
[dev-dependencies]
# Test certificates are generated, never committed.
rcgen = "0.13"
# The QUIC tests stand a real client and a real service either side of the relay: the
# same crates the engine links, so what they prove is what will run.
quinn = { version = "0.11", default-features = false, features = ["runtime-tokio", "rustls-ring"] }
rustls = { version = "0.23", default-features = false, features = ["ring", "std"] }
# And a real HTTP/3 service and client either side of it, for the same reason.
h3 = "0.0.8"
h3-quinn = "0.0.10"
# The same, for HTTP/2: the tests stand a real h2 client and a real h2 service either
# side of the proxy, so what they prove is what will run.
h2 = "0.4"
tokio-rustls = { version = "0.26", default-features = false, features = ["ring", "tls12", "logging"] }
bytes = "1"
http = "1"
# Catching a panicking filter is what keeps the datapath alive, so the binary
# must unwind rather than abort. This is not a tuning knob, it is load-bearing.
[profile.release]
panic = "unwind"
opt-level = 3
[profile.dev]
panic = "unwind"

592
backend/proxysrc/README.md Normal file
View File

@ -0,0 +1,592 @@
# The proxy datapath engine
`fgex-proxy`: one of the two network layers a service can be put on. Instead of lifting
packets to userspace and handing a verdict back to the kernel — which is what
`backend/binsrc/` does — it terminates the connection and owns both halves.
It is what a service selects as its **proxy** layer in `modules/services/transports.py`,
and it is what makes TLS, QUIC, HTTP/2 and HTTP/3 filterable at all.
> This file is the engine's own documentation: what it does, and why it is shaped this
> way. The product-level view — which layer to choose and what each one costs — is
> [`docs/services.md`](../../docs/services.md), and the rules a change here has to keep
> are in [`AGENTS.md`](../../AGENTS.md). Where any of the three disagree, `AGENTS.md`
> wins; this file has been wrong before by being left behind while the engine moved.
## What it has to prove first
The NFQUEUE engine fails open in the kernel: the nft rule carries `flags: ["bypass"]` and
the binary asks for `NFQA_CFG_F_FAIL_OPEN`. If firegex dies, stalls, or falls behind,
packets flow anyway. In attack/defense that is the difference between "the filter didn't
fire" and "I lost the SLA".
A userspace proxy has no such backstop: if the process dies, the socket dies with it. So
the property is rebuilt by hand, and it has to hold against user code that is assumed to
be broken:
| Failure | Handled by | Outcome |
| --- | --- | --- |
| Filter panics | `catch_unwind` around every call | Chunk forwarded, filter disabled at once |
| Filter never returns | Deadline on the blocking task | Chunk forwarded, filter disabled after 3 in a row |
| Every filter disabled | `degraded` flag | Chain skipped entirely — no copy, no thread hop |
| Connection handler panics | One task per connection | That connection only; the listener lives |
| Python worker hangs or dies | Killed and respawned on the next chunk | That chunk forwarded |
| Backend goes away | stdin closing is not a shutdown | The current ruleset keeps being enforced |
The degraded path is deliberately the cheapest one. Failing open must not cost more than
working normally, or the failure mode becomes a second failure. `panic = "unwind"` in
`Cargo.toml` is load-bearing rather than tuning: an aborting binary cannot catch anything.
`tests/fail_open.rs` asserts each row over real sockets.
## What the proxy buys
- **Every version of HTTP, filtered the same way.** HTTP/2 and HTTP/3 compress their
method, path and headers (HPACK, QPACK), so a layer that only forwards shows a filter a
compression format. This one terminates them and renders each exchange as the HTTP/1.1
it would have been, so one pattern and one Python file cover all three.
- **TLS and QUIC.** The plaintext never leaves the process.
- **Reassembly from the kernel**, instead of a stream follower and the out-of-order caveat
that comes with it.
- **An ordered chain inside one process**, rather than one process and one reassembly pass
per filter.
- **Real connection metadata.** A filter is told the client's and the service's address
and port once per connection (`ConnectionMeta`, a `KIND_OPEN` frame), and nothing below
the application layer beyond that. That is more than a header could honestly say here:
this engine terminated the connection, so the headers on the wire are its own.
- **Backpressure for free.** A slow filter slows the sender through TCP flow control
rather than overflowing a fixed-size queue and dropping packets.
- **Clean closes.** A refused connection is shut down properly instead of leaving the peer
to infer something from missing packets.
What it costs is the kernel backstop above, and being in the path at all: the connection
the service sees is one this process opened.
## The limitation that decided the design
A blocking filter thread **cannot be cancelled**. When one blows the deadline the engine
walks away, but the thread lives until its own code decides to stop — it holds a
blocking-pool slot until the process restarts. The damage is bounded (a filter that times
out repeatedly is disabled and never called again) but bounded is not zero.
That is why user Python runs **out of process**: a hung worker can be killed, a hung
thread cannot. It also keeps `libpython` out of the datapath binary, which is where a good
share of the C++ engine's complexity lives.
Compiled patterns cannot hang, so they declare `may_block() == false` and run **inline**
against the relay's own buffers — no thread hop, no copy. That is not a micro-optimisation:
it took ten rules from ~1260 to ~2950 MB/s in the benchmark that found it. The deadline
machinery is still there and still the default; it is simply no longer charged to filters
that cannot use it.
## Rules
One JSON line on stdin carries the whole ruleset, one `ACK OK` / `ACK FAIL <why>` line
comes back — the same shape `cppregex` uses for its filter codes. Blocks are reported as
`BLOCKED <id>` so the backend can attribute them.
```json
[{"kind":"regex","id":"r1","filter":"f1","pattern":"FLAG\\{[a-z]+\\}","direction":"s2c","case_sensitive":true},
{"kind":"python","id":"f2","code_path":"/execute/db/service_filters/f2.py",
"enabled":["block_traversal"],"command":["python3","/execute/modules/services/pyworker.py"],
"timeout_ms":1000}]
```
Four decisions worth knowing:
- **A pattern blocks. There is no second action.** There used to be a `replace` rule, and
it was withdrawn because it could not keep its promise: rewriting scanned one chunk at a
time — bytes already forwarded cannot be taken back — so a match straddling two chunks
was never rewritten. Measured: a pattern sent whole arrived redacted, the same pattern
split across two TCP segments arrived intact, and blocking (which scans in hyperscan's
stream mode) caught both. The miss was not the problem; its silence was.
- **The order the backend sent is the order that runs.** `filter` groups the patterns of
one link so they share a compiled database, which is what makes fifty patterns cost
about what one costs — and a group is exactly a filter, so grouping never crosses a
boundary the operator can see. An earlier version grouped every regex and then every
Python rule, which turned the chain into a set. `tests/rules.rs` pins both.
- **A rejected ruleset changes nothing.** One bad pattern refuses the whole update and the
previous rules keep enforcing. Failing open covers a filter that breaks at runtime; it
is not a licence to quietly drop rules the operator asked for. Each pattern is compiled
alone first and thrown away, only so a failure can name *which* rule is wrong —
hyperscan reports the first one and stops.
- **A block closes the connection, not the direction.** Half-closing would let a service
that answers without waiting for the request still reach a client whose request was
refused. On QUIC and HTTP/2 the refusal closes the *connection* rather than the stream,
for the same reason: a block a client can simply retry on the next stream is not a block.
Matching is hyperscan (`src/hyperscan.rs`, a binding to `libhs`), the same library
`cppregex` uses on the other layer. There is no second regex engine in this tree, and
adding `regex` to `Cargo.toml` would undo that.
## Python filters
The API is the one a `firegex` user already knows, and it is the **same library** the
NFQUEUE binary embeds — `firegex.pyfilters`, with all of its models:
```python
from firegex.pyfilters import pyfilter, ACCEPT, REJECT, HttpRequest
@pyfilter
def block_path_traversal(req: HttpRequest):
return REJECT if "../" in req.url else ACCEPT
```
`modules/services/pyworker.py` is the child process that runs it. A filter file's protocol
is read off the annotations, never declared; what a filter may read and write is the same
on both layers (`RawPacket` metadata in, payload out, nothing below the application
layer). See [`docs/pyfilter.md`](../../docs/pyfilter.md).
Frames are length-prefixed binary on the worker's stdin/stdout:
`[u32 len][u8 kind][u64 connection][payload]`, big-endian, `len` covering everything after
itself.
| kind | meaning |
| --- | --- |
| `0` / `1` | a chunk, client→server / server→client |
| `2` | the connection is over — so the worker can drop its module globals |
| `3` | the connection's metadata, one JSON object, once |
The reply is `[u32 len][verdict][name len][name][payload]`. The name is whichever
`@pyfilter` function decided, so a block is attributed as `<filter>/<function>` — the same
token the NFQUEUE binaries report, which is what lets the backend have one way to
attribute one.
Three things are load-bearing:
- **The worker announces itself** (`READY`) once it has executed the user's file. Waiting
for it is what lets a ruleset carrying code that does not load be refused *when it is
applied*, rather than discovered later as traffic that quietly stopped being filtered.
- **The connection id is in every frame**, because the documented promise is that each
stream gets its own module globals. One client's state deciding another client's verdict
is both a false positive and a way to smuggle a pattern past a filter.
- **`stdout` is the protocol channel**, so the worker reassigns `sys.stdout = sys.stderr`
before running user code: a `print()` landing in the middle of a frame gets the worker
killed on every packet with nothing to explain it. `tests/pyworker.rs` has a regression
test for exactly that.
Tracebacks go to stderr, where the backend already reads them; the worker also marks them
so the backend can say *the traffic was forwarded unfiltered* alongside the traceback,
which is the sentence the NFQUEUE side produces from an `EXCEPTION`.
One worker per rule, one conversation at a time. Serialising is the honest simple thing
while a frame is a request and a response; a pool is the next step if Python filtering
ever becomes the bottleneck.
## TLS
The engine terminates it: it decrypts from the client, the chain sees plaintext that never
leaves the process, and it re-encrypts on the way out — `rustls` with the `ring` provider,
so there is no OpenSSL in the build and the Docker stage stays a plain `cargo build`.
What that replaced was **nginx**: one listener terminating the public connection and
forwarding plaintext to a second loopback port, which re-encrypted towards the real
service, with the filter engine attached to the leg in between. It cost a generated config,
**a pair of ports derived from hashing `ip:port`** — chosen rather than assigned, so they
could collide with something real — a `tls_streams` table, an nft chain of its own, and a
failure mode where one unloadable certificate stopped nginx and took every TLS service on
the instance down with it. A TLS service now occupies no port a plain one does not, and
the blast radius is structural: each service carries its certificate into its own process.
**ALPN is mirrored, and the ordering is the design.** The client's handshake is *started*
and held at the ClientHello (`tokio_rustls::LazyConfigAcceptor`), because that message
carries the protocols the client will speak; the upstream handshake goes next offering
exactly that list; and whatever the service picked is the single protocol the client is
then told. Answering something the service did not say is how a proxy breaks a protocol it
was only carrying — a client told `h2` against an HTTP/1.1 service sends frames nothing can
read. When the service picks nothing, the client is told nothing: no agreement is invented.
Before this, ALPN was dropped in both directions and every client silently fell back to
HTTP/1.1.
**The upstream certificate is not verified**, mirroring the `proxy_ssl_verify off` nginx
was configured with. That is the right call rather than laziness: the service on the other
side is the thing firegex is defending, it is behind loopback or a private link, and in a
competition it is invariably self-signed. Verifying it would only ever refuse to protect it.
Certificate and key reach the engine as **paths**, never as environment values: a private
key in an env dump is a private key in a log.
What cannot be carried, and is a limit rather than a missing setting: **TLS 1.0 and 1.1**,
which rustls does not implement; the cipher list, which is fixed and forward-secret only;
and **client certificates**, since the certificate a client would present is one only the
service can check. The key must be one `ring` will sign with — RSA ≥ 2048, ECDSA or
Ed25519; nginx could be told to take a 1024-bit key with `@SECLEVEL=1` and rustls has no
equivalent, so a smaller one is refused at startup with the reason on stderr.
## QUIC, and HTTP/3
**QUIC is terminated, and unlike TLS there was never a second option.** TLS over TCP can
be carried past unopened — the bytes are framed by a transport the kernel understands — so
filtering the ciphertext is a thing an operator could choose. Past its Initial packet QUIC
encrypts the frames, the stream boundaries and the packet number along with the payload,
so a packet queued to NFQUEUE is a datagram of noise.
To the kernel it is UDP, so the addresses, the rules and the relay map are the code that
was already there for datagrams; what changes is that the port is bound by something that
terminates (`quic::QuicManager`) instead of something that forwards (`udp::UdpManager`).
`relays::Relays` is the two-armed enum both the startup list and the control channel go
through.
- **The ALPN is mirrored, as on the TLS path.** quinn offers nothing to hold a ClientHello
at, but an Initial packet's keys come from the connection ID it carries in the clear, so
`quic_hello.rs` reads the client's list off the socket before the handshake is answered.
The service is dialled first offering exactly that list, and the client is advertised
the one protocol the service chose — nothing to configure, whatever the service speaks.
A hello that cannot be read falls back to offering `h3`, and says so.
- **One stream is one connection to the chain**, with its own sessions and its own module
globals, released when the stream ends. A filter's state follows a stream of bytes from
start to end, which is what a QUIC stream is and what a QUIC connection is not.
- **A refusal closes the connection**, not the stream.
- **Datagrams are carried, and one connection's are one flow.** `L4::QuicDatagram` is its
own value rather than `Quic` with a flag, because the two answer the stream question
differently: a datagram gets `RawPacket` and nothing else, which is the same answer the
plain UDP relay gives. The flow opens on the first datagram, not on the connection, so a
connection that negotiates the extension and never uses it costs no filter state. A
refused datagram is **dropped and the connection carries on** — a datagram is complete by
the time it is judged. `DATAGRAM_BUFFER` bounds what one connection may have waiting.
- **They are not carried on HTTP/3, and the handshake says so** rather than the peer
finding out: an HTTP/3 datagram names the stream it belongs to, and the request streams
this proxy opens towards the service are not the ones the client opened. So the `h3`
accept configuration is built with a transport that does not advertise the extension.
- **WebTransport is not carried**, and it is a bigger question than its datagrams: its
streams are bound to a session by a varint every one of them carries, which breaks both
"one stream is one connection to the chain" and the HTTP/1.1 rendering. It needs a
session model, not a switch.
- **0-RTT is refused** (`max_early_data_size = 0`): early data is replayable and a refused
request cannot be un-delivered. And **every unvalidated client gets a Retry**, so nothing
is opened towards the service on the word of a source address nobody has checked.
**HTTP/3 is rendered to the chain as the HTTP/1.1 it would have been** (`src/h3.rs`, on
hyperium's `h3` — a binding, not a second implementation). That is the point of the whole
exercise: QPACK would otherwise be what a filter is shown, and every pattern written for
the TCP service beside it would silently stop matching.
## HTTP/2, and one rendering for every version
**`src/http1.rs` is the rendering, and `h2.rs`/`h3.rs` are what is left of each protocol
once it is taken out.** The HTTP/1.1 view a filter is shown — the framing decision, the
head hold, `render_request`/`render_response`, `judge` — lives in one place, and both
protocols are adapters to somebody else's parser. Two copies would be two things a filter
can be shown, and the day they drift the symptom is a pattern that matches over one
version and not another with nothing to say so.
An HTTP/2 client can be put in front of a service that speaks HTTP/1.1: `h2.rs` takes its
upstream as an `Outbound`, so it is either another HTTP/2 connection or `h1up::H1Upstream`
where the address said the service is cleartext — exactly as on the QUIC edge.
Three things differ from h3 and each is a way to get this wrong:
- **Flow control is ours to run.** `h2` hands received bytes over and waits for them to be
released. Releasing on read means no backpressure and this process buying memory for
whichever side is faster; never releasing stalls the stream after the first window. So
capacity is released **after the piece has been forwarded**, and outbound writes reserve
capacity before writing.
- **The end of a message rides on its last frame.** A head that *is* the whole message has
to be sent with `end_of_stream` on the head; sending it open and closing with an empty
DATA frame turns a **trailers-only** answer into one with a body — which is exactly a
gRPC status-only reply, and gRPC clients refuse the mangled version. `head_is_the_message`
asks the sender rather than inferring.
- **One stream is one connection to the chain**, unlike HTTP/1.1 where a keep-alive
connection carries many requests through one set of filter state. HTTP/2 interleaves its
streams, and sharing state between two of them is a way to smuggle a pattern past a
filter by splitting it across them.
Also:
- **`h2c` — HTTP/2 in the clear — is recognised from the connection preface, not from a
setting.** Most gRPC is deployed without TLS behind a load balancer, so leaving it out
would have made "gRPC is filtered" half true. `sniff()` in `proxy.rs` **peeks** rather
than reads, so a connection that turns out to be something else is handed to the byte
pumps exactly as it arrived and there is no buffer to replay. It **races the service**,
because a protocol where the server speaks first (SMTP, SSH, most game protocols) would
otherwise be held waiting for a client that is correctly waiting for a banner — and it is
bounded (`SNIFF_WAIT`), so two silent peers cannot hold two descriptors. The same peek
answers the TLS question for an `http` service, which is why there is one `Opening` and
not two sniffs.
- **Server push is refused in the handshake**, not by dropping what arrives, and
**extended CONNECT is not advertised**; a plain `CONNECT` stream is reset with a log line
rather than rendered, because a tunnel is opaque bytes with no HTTP/1.1 message in it.
- **A refusal closes the connection with `Reason::CANCEL`**, deliberately not
`REFUSED_STREAM`, which tells a client the request was never acted on and may be retried.
- **A bypassed chain is not terminated.** Rendering for a chain with nothing to say is work
paid for no answer, so a service with no filters, one whose filters have all been
disabled, and a connection admitted past the limit with `over_limit_forwards` all take
the byte pump. The cost is that such a connection stays a byte pump for its whole life
even if a filter is pushed a moment later.
- **The HTTP/1.1 `Upgrade: h2c` handshake is not carried**, and that is a limit rather than
an unconfigured setting: the upgrade request is filtered as the HTTP/1.1 request it is,
but if the service answers `101` everything after it is HTTP/2 nobody rendered. RFC 9113
deprecated the mechanism and nginx dropped it in 1.25.1.
## UDP
The TCP path recovers where a connection was headed with `SO_ORIGINAL_DST`, which the
kernel implements **for TCP and SCTP only**. So UDP is relayed with **one socket per
protected address**, each with its upstream fixed — nothing has to be recovered because
nothing was lost. The engine prints one `UDP <upstream> <port>` line per relay after
`PORT`, and the backend points each address's rule at its own port.
Per-flow filter state is keyed by client address and released after `IDLE`: UDP has no
close to observe, so a timeout is the only thing that ends a flow. Both directions are
inspected. `Verdict::Reject` drops the datagram rather than closing anything. Replies leave
through the **listener** socket so conntrack rewrites them to appear from the address the
client dialled; sending from the upstream socket would reach a client not expecting that
source. The relay's own dial carries the self-mark, or the redirect rule would catch it and
loop.
New relays can be added to a running engine over the control channel, so an address added
to a running service costs nobody their connection.
## Holding up under load
**A service carries at most `FGEX_PROXY_MAX_CONNECTIONS` at once**, and the operator says
what happens to the rest. It exists because the alternative was measured: silent
connections cost two descriptors each — one from the client, one to the service, since the
upstream is dialled on accept — and ~505 of them from one host exhausted the container's
1024 and took **every** service down. UDP was cheaper still: 600 datagrams from 600 forged
sources in 0.03 s took 400 descriptors and held them a minute after the sender left.
Say what it buys, and no more. **A cap does not save the service being attacked** — it
cannot tell a connection that is silent because it is an attack from one that is silent
because the client is slow. What it buys is that one service's attacker stops being
everyone's: measured, 400 silent connections against a limited service left the service
beside it answering.
- One number for both halves, because TCP connections and UDP flows spend the same
descriptors. `0` is no limit, kept as a value rather than an absence.
- `Slot` is a guard with a `Drop`, not a decrement at the end of `handle_connection`: that
function has a dozen ways out, and a counter that leaks on any one of them is a limit
that tightens until it refuses everything.
- The accept loop **backs off** after a failed accept. Out of descriptors, the loop spun on
`accept` → `EMFILE` → print → `accept`, measured at ~40% of a core with four clients
knocking, writing a log line per attempt into the pipe the backend reads.
- On QUIC, `over_limit_forwards` is honoured but is not free: the connection is still
terminated, decrypted and re-encrypted, because there is no such thing as forwarding a
QUIC connection unopened.
**`FGEX_PROXY_FIRST_BYTE_TIMEOUT` is what the cap cannot be.** A cap contains a phantom
flood; a deadline ends it — measured, with the limit full and the attacker still holding
every socket, clients got back in once it passed. Two properties are load-bearing:
- **Either direction counts.** The flag is set by whichever pump moves a byte first, so a
service that greets its client satisfies it with its banner. Requiring the *client* to
speak would hang every server-speaks-first protocol.
- **Only until the first byte, never again.** A connection that has spoken and gone quiet
is a session, and sessions think. This is not an idle timeout.
The flag is set *before* the chain runs, because a chunk a filter goes on to refuse is
still a connection that said something. The watchdog fires through the same stop signal a
refusal uses, so both directions shut down cleanly — a dropped peer reads a truncation, and
this connection has done nothing to deserve one.
## Preserving the client address
The service must keep seeing the real client, or everything that logs, rate-limits or bans
by address starts seeing one address for the whole internet. Source preservation is
**unconditional**: the engine always dials the service from the client's address
(`IP_TRANSPARENT`, with a kernel-chosen port). `CAP_NET_ADMIN` is probed at startup,
because a missing capability would otherwise surface only as every service quietly seeing
the proxy's address. If a transparent dial fails at runtime the engine falls back to its
own address — traffic beats identity — but counts it and says so once, loudly.
The work is in getting the reply back. With the service on this same host, two flows look
almost identical:
| Flow | src | dst | must |
| --- | --- | --- | --- |
| Engine answering the client | `service:port` | `client:client_port` | leave the box |
| Service answering the engine | `service:port` | `client:<ephemeral>` | stay local |
No address tells them apart — only an ephemeral port chosen at connect time, which nothing
can be written against. **Conntrack can**: the first belongs to the intercepted, DNATed
connection, the second is a separate entry the engine opened itself. So the output chain
reads `ct status dnat accept` for the first and marks everything else from a protected
service home, and the marked packets are delivered locally by an `fwmark` rule plus a
`local` default route. `modules/services/nftables.py` installs all of it.
**Only the redirect intercept exists.** A `tproxy` mode was implemented and removed: the
intercept it creates makes no NAT entry, so there is no marker to separate those two flows,
and a divert rule broad enough to catch the service's reply also reroutes the engine's own
SYN-ACK away from the client — the connection then hangs rather than fails. That was
measured. Since the topology firegex actually runs in has the service on the same host or
behind a private link, the mode that cannot serve it was not worth keeping.
The engine stamps `SO_MARK` (`SELF_MARK`, `0x133A`) on every connection it opens, and the
intercept rules skip it — otherwise, where both ends are local, an intercept in `nat output`
would catch the engine's own dial and loop it back into itself.
## Where the plaintext goes
Terminating in-process means the decrypted traffic is never a packet, so there is nothing
for a capture tool to read. `src/capture.rs` writes it out: each connection's plaintext
framed as the TCP stream it was, sent over `AF_PACKET` to a dummy interface (`firegex0`)
that carries every decrypted service's traffic and nothing else.
They are **reconstructed** packets. The bytes are real — what the filters saw and what was
forwarded — but the framing is built here, because the framing on the wire was encrypted:
sequence numbers from zero, no retransmissions, segmentation from this engine's read sizes.
A SYN and a FIN are emitted so a capture tool has a stream to follow rather than orphan
segments. **Say so wherever it is offered: a capture from here is not evidence of what was
on the wire.**
On QUIC each *stream* is written as its own TCP conversation, with a synthetic client port
from a cycling counter — every stream of one connection shares the connection's four-tuple,
and writing them out as they are would interleave a hundred of them into one conversation
whose bytes decode as nothing. On HTTP/3 and HTTP/2 what is written is the **rendering**,
which is the only way this interface can keep the promise it makes everywhere else: what is
on it is what the filters saw. It is not what left the process, and the two cannot disagree
because they are the same bytes.
Every failure is silent and local. No interface, no socket (it needs `CAP_NET_RAW`), a send
that fails — the traffic goes on exactly as it would have. A capture aid must never be able
to interrupt the service it is watching. The interface itself belongs to the instance and
is created by `modules/services/mirror.py` before any engine starts.
## Talking to it
Two channels, both line-oriented, and the datapath never waits on either.
**stdout — what the engine says.** The handshake first, then events:
| line | meaning |
| --- | --- |
| `PORT <n>` | the port it bound. Pass `:0` and read this back; nothing keeps a registry |
| `UDP <upstream> <port>` | one per relay, in the order they were asked for |
| `BLOCKED <id>` | a rule refused something; `<filter>/<function>` for Python |
| `ACK OK` / `ACK FAIL <why>` | the answer to one control line |
| `STATS seen=… refused=… live=… over_limit=… no_first_byte=…` | every 2 s, first one immediately |
Everything descriptive — `[info]`, `[warn]`, `[fatal]` — goes to **stderr**, which the
backend captures and turns into the service's log. `_died_because` keeps the last `[fatal]`
line, so a service that will not start reports the engine's own reason rather than "the
proxy engine did not report a listening port".
**stdin — what it is told.** One command per line:
| line | meaning |
| --- | --- |
| `[{…}, {…}]` | the whole ruleset, replacing the previous one |
| `ADD_UDP <addr> [same\|plain\|tls]` | bind one more relay; answers `UDP …` then `ACK` |
| `PUBLISH <dialled> <edge> <onward> [<service>\|-]` | what is spoken at one address and where the service behind it is |
| `WITHDRAW <dialled>` | forget one published address |
stdin closing is **not** a shutdown: the backend being gone is exactly when the datapath
outliving its control channel matters.
## Environment
| variable | meaning |
| --- | --- |
| `FGEX_PROXY_LISTEN` | **required.** `[::]:0` for dual-stack, which is what the backend passes |
| `FGEX_PROXY_UPSTREAM` | **required.** `original` (recover it per connection) or a literal address |
| `FGEX_PROXY_SPOOF_SOURCE` | dial the service as the client. The backend always sets it |
| `FGEX_PROXY_TARGETS` | `<addr>[\|<edge>[\|<onward>]][=<service>]`, comma-separated; the addresses that are not simply the service |
| `FGEX_PROXY_UDP` | `<addr>[\|<onward>]`, comma-separated; one relay each |
| `FGEX_PROXY_TLS` / `_TLS_OPTIONAL` | terminate TLS on the TCP listener; `_OPTIONAL` means only for the connections that start a handshake |
| `FGEX_PROXY_TLS_CERT` / `_TLS_KEY` | **paths**, never the material |
| `FGEX_PROXY_QUIC` | bind the relays with a QUIC endpoint instead of a datagram socket |
| `FGEX_PROXY_MAX_CONNECTIONS` | 0 is no limit |
| `FGEX_PROXY_OVER_LIMIT_FORWARD` | forward what does not fit, unfiltered, instead of refusing it |
| `FGEX_PROXY_FIRST_BYTE_TIMEOUT` | seconds; 0 is off |
| `FGEX_PROXY_FILTER_TIMEOUT_MS` | how long a blocking filter gets. Default 2000 |
| `FGEX_PROXY_CONNECT_TIMEOUT_MS` | upstream dial and handshakes. Default 5000 |
| `FGEX_PROXY_SELF_MARK` | hex; default `133A`. Must match `PROXY_SELF_MARK` in the backend |
| `FGEX_PROXY_PYWORKER` | fallback path to the worker, for a Python rule that carries no `command` |
| `FGEX_PROXY_FILTERS` | `panic`, `hang`, `block:<needle>` — **test scaffolding only** |
| `NTHREADS` | tokio worker threads. What `run.py --threads` sets, on both layers |
`FGEX_PROXY_BINARY` is read by the *backend*, not by this, and points at the binary — which
is what a dev build overrides.
There is one more mode: **`--debug-regex`** reads one JSON request on stdin, answers on
stdout and exits, starting no listener and touching no rules. It is what
`POST /api/services/debug-regex` shells out to, so the in-app pattern tester is answered by
the engine that would enforce the answer rather than by an approximation that would bless
patterns hyperscan rejects. It reports `unscannable` for patterns that are valid, will run,
and cannot be highlighted — validity is judged in **stream** mode, which is the mode a
pattern actually runs in, while highlighting needs a block-mode scan that does not accept
quite the same language.
## Layout
| file | what is in it |
| --- | --- |
| `src/main.rs` | environment, startup handshake, the runtime, the stats ticker, `--debug-regex` |
| `src/filter.rs` | verdicts, the `Filter` trait, the chain and its isolation policy. `ChainHandle` swaps the chain under live connections |
| `src/rules.rs` | the filters the backend configures, and the JSON they arrive in |
| `src/hyperscan.rs` | the `libhs` binding: stream and block scanning, compilation, validation |
| `src/proxy.rs` | accept loop, the opening peek, TLS and ALPN, the limit, the byte pumps |
| `src/http1.rs` | **the** HTTP/1.1 rendering, and the traits both HTTP protocols implement |
| `src/h2.rs` | HTTP/2, on hyperium's `h2` |
| `src/h3.rs` | HTTP/3, on hyperium's `h3` |
| `src/h1up.rs` | HTTP/1.1 *towards the service*, on hyper's client |
| `src/quic.rs` | QUIC termination, streams, datagrams, the relay manager |
| `src/udp.rs` | datagram relays and per-flow state |
| `src/relays.rs` | the two-armed enum both relay kinds go through |
| `src/tls.rs` | termination, re-encryption, ALPN mirroring, the deliberately permissive upstream verifier |
| `src/transparent.rs` | `IP_TRANSPARENT`, `SO_ORIGINAL_DST`, the self-mark, the startup capability probe |
| `src/capture.rs` | the reconstructed packets written to `firegex0` |
| `src/pyworker.rs` | the child process running the user's Python, and its deadline |
| `src/control.rs` | the stdin channel, running alongside the datapath |
| `src/debug.rs` | `--debug-regex` |
| `src/spec.rs` | filters built from a text spec. Test scaffolding only |
Tests: `tests/fail_open.rs` (the guarantees above, over real sockets), `tests/rules.rs`
(matching, direction, grouping, order), `tests/pyworker.rs` (a hung worker killed, a
crashed one recovering), `tests/tls.rs`, `tests/quic.rs`, `tests/h2.rs`, `tests/h3.rs`, and
`tests/throughput.rs` (the matcher benchmark, `--ignored`).
## Building and testing
Built in the Dockerfile's `compiler` stage rather than in a `rust` image on purpose: it
links the same `libhs` the C++ binaries do, and a binary built against one distro's
vectorscan and run against another's is a bad afternoon.
```bash
docker build --target compiler -t firegex-compiler .
docker run --rm -v "$PWD/backend:/execute" -w /execute/proxysrc firegex-compiler cargo test
cd backend/proxysrc && cargo test # on a host with libhs + fgex installed
cargo test --release --test throughput -- --ignored --nocapture
```
Mount `backend/`, **not** `backend/proxysrc/`: `tests/pyworker.rs` launches
`../modules/services/pyworker.py`, which imports `firegex`, which imports `brotli`. Without
both on `PYTHONPATH` the pyworker tests fail with a `ModuleNotFoundError` buried in captured
output and read as real failures. The repo's own `fgex-lib` has to come *first* on the path,
or a stale image's `firegex` wins.
The product-level suite is `tests/`, driven by pytest against a running instance;
`tests/integration/test_http_versions.py` and `tests/integration/test_grpc.py` are the two
that say whether the rendering in here really works.
## Measured against NFQUEUE
**The proxy is not the slow one, and the docs used to say it was.** Moving bytes it is
about **2.2× NFQUEUE at one thread and 4.7× at eight**: NFQUEUE pays a userspace round trip
*per packet*, while this pays once per connection and then the kernel moves the bytes. On
short connections the two are indistinguishable.
The numbers, the method and the corrections that produced them live in
[`tests/bench/README.md`](../../tests/bench/README.md) and `tests/bench/results/`, and that
is deliberately the only place they are written down — a table copied into a second file is
a table that will still be quoting a figure nobody can reproduce a year from now.
The trade between the two layers is **fail-open and transparency, not speed**. Keep
`docs/services.md`, `LayerChoice.tsx` and `transportSummary()` saying that.
## Backend side
`backend/modules/services/` holds the other half:
- `transports.py` — `ProxyTransport` starts this binary, reads the handshake, keeps the
relay map and pushes the ruleset. `NfqueueTransport` and `ExternalTransport` are the
other two layers, behind the same interface.
- `nftables.py` — the redirect, the divert rule and the return path, all in `table inet
fgex`. `NAT_PRIORITY` is `-120`, below `dstnat`, so a container runtime's published port
cannot rewrite the destination before firegex's rule matches it.
- `firewall.py` — one `ServiceManager` per service: the engine is started first because it
is what picks the listening port, and the rules are written afterwards. Stopping goes the
other way round.
- `pyworker.py` — the child process this engine spawns for a Python rule, and
`--check`, which is what decides whether a filter can be saved at all.
- `mirror.py` — the `firegex0` device, created before any engine starts.

406
backend/proxysrc/src/capture.rs Normal file
View File

@ -0,0 +1,406 @@
//! The decrypted stream, written out as the packets it never was.
//!
//! TLS is terminated in this process, which is what lets a protected service occupy no
//! port beyond the one it already answered on — and it means the plaintext is never a
//! packet on any interface, so there is nothing for a capture tool to see. Everything an
//! operator wants to do with the decrypted traffic (read it in Wireshark, keep a pcap of
//! a round, grep a flow after the fact) would have gone with the ports.
//!
//! So the engine writes it out itself. Each connection's plaintext is framed as the TCP
//! stream it was before encryption and sent to a dummy interface, where tcpdump and
//! Wireshark read it like anything else. One interface for the whole instance: every TLS
//! service's plaintext arrives there and nothing else does.
//!
//! **These are reconstructed packets.** The bytes are real — they are exactly what the
//! filters saw and exactly what was forwarded — but the framing around them is made up
//! here, because the framing that actually crossed the wire was encrypted. Sequence
//! numbers start at zero, there are no retransmissions, and the segmentation is this
//! engine's read sizes rather than the peer's. What that buys is a stream Wireshark can
//! follow; what it costs is that this is not evidence of what was on the wire, and the
//! documentation says so.
//!
//! **A QUIC stream is reconstructed the same way, one TCP stream each.** A QUIC stream
//! is ordered, reliable and starts at zero, which is precisely what a [`Tap`] rebuilds —
//! but every stream of one connection shares the connection's four-tuple, so writing
//! them out as they are would interleave a hundred of them into a single conversation
//! that no tool could take apart. So each is given a **synthetic client port** by
//! [`Tap::open_stream`], and that is one more piece of invented framing on top of the
//! invented framing this whole file is: the port identifies a stream and was never a
//! port anybody bound. It has to be said wherever a capture from here is offered, next
//! to the sentence about sequence numbers, because an operator reading a port back out
//! of Wireshark will otherwise go looking for a socket that does not exist.
//!
//! Failure is always silent and always local. If the interface is not there, or the
//! socket cannot be opened, or a send fails, the traffic goes on exactly as it would
//! have: a capture aid must never be able to interrupt a service it is only watching.
use std::ffi::CString;
use std::net::{IpAddr, SocketAddr};
use std::sync::atomic::{AtomicBool, AtomicU16, AtomicU32, Ordering};
use std::sync::Arc;
/// The interface the decrypted traffic is written to. A fixed name on purpose: it is
/// something an operator types into a capture tool.
pub const DEVICE: &str = "firegex0";
/// How much payload goes in one segment. Under the dummy device's 1500-byte MTU with
/// room for the largest headers this writes (Ethernet + IPv6 + TCP).
const SEGMENT: usize = 1400;
const ETH_HEADER: usize = 14;
const IPV4_HEADER: usize = 20;
const IPV6_HEADER: usize = 40;
const TCP_HEADER: usize = 20;
const FIN: u8 = 0x01;
const SYN: u8 = 0x02;
const PSH: u8 = 0x08;
const ACK: u8 = 0x10;
/// A raw socket bound to the capture interface, shared by every connection.
pub struct Capture {
fd: libc::c_int,
ifindex: libc::c_int,
}
// The fd is used only for `sendto`, which is thread-safe on a packet socket.
unsafe impl Send for Capture {}
unsafe impl Sync for Capture {}
impl Drop for Capture {
fn drop(&mut self) {
unsafe { libc::close(self.fd) };
}
}
impl Capture {
/// Open the capture interface, or `None` if there is nothing to write to.
///
/// `None` is the ordinary case, not an error: the interface exists only while some
/// TLS service is running, and a host that cannot have one still runs services.
pub fn open() -> Option<Arc<Capture>> {
let name = CString::new(DEVICE).ok()?;
let ifindex = unsafe { libc::if_nametoindex(name.as_ptr()) };
if ifindex == 0 {
return None;
}
// `ETH_P_ALL` in the socket's protocol would also make it *receive* every frame
// on every interface, which is a firehose nobody reads. Zero sends only.
let fd = unsafe { libc::socket(libc::AF_PACKET, libc::SOCK_RAW | libc::SOCK_CLOEXEC, 0) };
if fd < 0 {
return None;
}
Some(Arc::new(Capture { fd, ifindex: ifindex as libc::c_int }))
}
fn send(&self, frame: &[u8]) {
let mut addr: libc::sockaddr_ll = unsafe { std::mem::zeroed() };
addr.sll_family = libc::AF_PACKET as u16;
addr.sll_ifindex = self.ifindex;
addr.sll_halen = 6;
unsafe {
// The return value is deliberately ignored: a capture that cannot keep up,
// or an interface that went away, must not become an error on the path of
// the traffic it is watching.
libc::sendto(
self.fd,
frame.as_ptr() as *const libc::c_void,
frame.len(),
libc::MSG_DONTWAIT,
&addr as *const libc::sockaddr_ll as *const libc::sockaddr,
std::mem::size_of::<libc::sockaddr_ll>() as libc::socklen_t,
);
}
}
}
/// One connection's reconstruction: the two ends, and where each direction has got to.
pub struct Tap {
capture: Arc<Capture>,
client: SocketAddr,
server: SocketAddr,
/// Next sequence number for client → service.
up: AtomicU32,
/// Next sequence number for service → client.
down: AtomicU32,
/// Whether the end of the stream has been written.
ended: AtomicBool,
}
/// A reconstruction always gets an end, whichever way its owner left.
///
/// The TCP path closes its tap where the connection is torn down, and could; the paths
/// that carry one stream of many have a dozen ways out — a refused request returns before
/// anything is joined — and a stream left without a FIN is one Wireshark holds open
/// waiting for bytes that are never coming. So the last holder emits it, and
/// [`Tap::closed`] stays callable where the end is known and worth saying explicitly.
impl Drop for Tap {
fn drop(&mut self) {
self.closed();
}
}
/// Hands out the ports that tell one multiplexed stream from another.
///
/// A counter and not a hash of the stream id: ids restart at zero on every QUIC
/// connection, so deriving the port from one would drop two clients' first streams into
/// the same conversation. Cycling the ephemeral range instead gives the same answer a
/// NAT gives, and reuse only becomes possible after 16384 live streams — at which point
/// a new SYN is what separates them, exactly as it does for a real port that came round
/// again.
static NEXT_STREAM_PORT: AtomicU16 = AtomicU16::new(0);
const EPHEMERAL_BASE: u16 = 49152;
const EPHEMERAL_SPAN: u16 = 65535 - EPHEMERAL_BASE + 1;
fn stream_port() -> u16 {
EPHEMERAL_BASE + NEXT_STREAM_PORT.fetch_add(1, Ordering::Relaxed) % EPHEMERAL_SPAN
}
impl Tap {
/// Start a reconstruction for one stream of a multiplexed connection.
///
/// The client's address keeps its IP and loses its port, because the port is the only
/// field left that can tell two streams of one connection apart: the service's end is
/// the service's and changing it would misattribute the traffic, and the client's IP
/// is what an operator filters on. What comes back is therefore a conversation
/// between a real host and a real service on a port that never existed.
pub fn open_stream(
capture: Option<Arc<Capture>>,
client: SocketAddr,
server: SocketAddr,
) -> Option<Arc<Tap>> {
let mut client = client;
client.set_port(stream_port());
Tap::open(capture, client, server)
}
/// Start a reconstruction, and emit the handshake that opens it.
///
/// The handshake is not decoration: without a SYN, Wireshark has no beginning for
/// the stream and cannot reassemble it, so the payloads show up as unrelated
/// segments of a conversation it never saw start.
pub fn open(capture: Option<Arc<Capture>>, client: SocketAddr, server: SocketAddr) -> Option<Arc<Tap>> {
// Mixed families would need a translation nobody asked for; in practice both
// ends of a connection the engine terminates are the same family.
if (client.is_ipv4()) != (server.is_ipv4()) {
return None;
}
let tap = Arc::new(Tap {
capture: capture?,
client,
server,
up: AtomicU32::new(0),
down: AtomicU32::new(0),
ended: AtomicBool::new(false),
});
// Interleaved with the increments, not emitted and then counted: each of these
// reads the two counters as they stand, so a SYN-ACK sent before the client's
// SYN has been counted acknowledges nothing and reads as a broken handshake in
// any tool that checks. SYN consumes a sequence number; so does SYN-ACK.
tap.emit(true, SYN, &[]);
tap.up.fetch_add(1, Ordering::Relaxed);
tap.emit(false, SYN | ACK, &[]);
tap.down.fetch_add(1, Ordering::Relaxed);
tap.emit(true, ACK, &[]);
Some(tap)
}
/// One chunk of plaintext, in the direction it was forwarded.
///
/// What is written here is what was *forwarded*, after any rewriting: a capture that
/// showed the bytes a filter replaced would disagree with what the two ends actually
/// received, which is the one thing it is for.
pub fn wrote(&self, from_client: bool, data: &[u8]) {
for chunk in data.chunks(SEGMENT) {
self.emit(from_client, PSH | ACK, chunk);
let counter = if from_client { &self.up } else { &self.down };
counter.fetch_add(chunk.len() as u32, Ordering::Relaxed);
}
}
/// Close the reconstruction, so the stream has an end as well as a beginning.
///
/// Idempotent: [`Drop`] calls it too, and a stream that ended twice would read as one
/// that was closed and then closed again.
pub fn closed(&self) {
if self.ended.swap(true, Ordering::Relaxed) {
return;
}
self.emit(true, FIN | ACK, &[]);
self.emit(false, FIN | ACK, &[]);
}
fn emit(&self, from_client: bool, flags: u8, payload: &[u8]) {
let (src, dst) = if from_client {
(self.client, self.server)
} else {
(self.server, self.client)
};
let (seq, ack) = if from_client {
(self.up.load(Ordering::Relaxed), self.down.load(Ordering::Relaxed))
} else {
(self.down.load(Ordering::Relaxed), self.up.load(Ordering::Relaxed))
};
let frame = match (src.ip(), dst.ip()) {
(IpAddr::V4(s), IpAddr::V4(d)) => {
build_v4(s, d, src.port(), dst.port(), seq, ack, flags, payload)
}
(IpAddr::V6(s), IpAddr::V6(d)) => {
build_v6(s, d, src.port(), dst.port(), seq, ack, flags, payload)
}
_ => return,
};
self.capture.send(&frame);
}
}
/// Ethernet header. The addresses are locally administered and made up: a dummy device
/// has no peer to address, and a capture tool only needs the ethertype to know what
/// follows.
fn ethernet(out: &mut Vec<u8>, ethertype: u16) {
out.extend_from_slice(&[0x02, 0, 0, 0, 0, 0x02]); // destination
out.extend_from_slice(&[0x02, 0, 0, 0, 0, 0x01]); // source
out.extend_from_slice(&ethertype.to_be_bytes());
}
fn ones_complement(sum: u32) -> u16 {
let mut sum = sum;
while sum >> 16 != 0 {
sum = (sum & 0xffff) + (sum >> 16);
}
!(sum as u16)
}
fn sum_bytes(bytes: &[u8]) -> u32 {
let mut sum = 0u32;
let mut i = 0;
while i + 1 < bytes.len() {
sum += u16::from_be_bytes([bytes[i], bytes[i + 1]]) as u32;
i += 2;
}
if i < bytes.len() {
sum += (bytes[i] as u32) << 8;
}
sum
}
#[allow(clippy::too_many_arguments)]
fn tcp_header(
src_port: u16,
dst_port: u16,
seq: u32,
ack: u32,
flags: u8,
payload: &[u8],
pseudo: u32,
) -> Vec<u8> {
let mut tcp = Vec::with_capacity(TCP_HEADER);
tcp.extend_from_slice(&src_port.to_be_bytes());
tcp.extend_from_slice(&dst_port.to_be_bytes());
tcp.extend_from_slice(&seq.to_be_bytes());
tcp.extend_from_slice(&ack.to_be_bytes());
tcp.push(5 << 4); // data offset: five 32-bit words, no options
tcp.push(flags);
tcp.extend_from_slice(&65535u16.to_be_bytes()); // window
tcp.extend_from_slice(&[0, 0]); // checksum, filled in below
tcp.extend_from_slice(&[0, 0]); // urgent pointer
let checksum = ones_complement(pseudo + sum_bytes(&tcp) + sum_bytes(payload));
tcp[16..18].copy_from_slice(&checksum.to_be_bytes());
tcp
}
#[allow(clippy::too_many_arguments)]
fn build_v4(
src: std::net::Ipv4Addr,
dst: std::net::Ipv4Addr,
src_port: u16,
dst_port: u16,
seq: u32,
ack: u32,
flags: u8,
payload: &[u8],
) -> Vec<u8> {
let tcp_len = TCP_HEADER + payload.len();
let pseudo = sum_bytes(&src.octets())
+ sum_bytes(&dst.octets())
+ libc::IPPROTO_TCP as u32
+ tcp_len as u32;
let tcp = tcp_header(src_port, dst_port, seq, ack, flags, payload, pseudo);
let mut out = Vec::with_capacity(ETH_HEADER + IPV4_HEADER + tcp_len);
ethernet(&mut out, 0x0800);
let start = out.len();
out.push(0x45); // version 4, five-word header
out.push(0); // dscp/ecn
out.extend_from_slice(&((IPV4_HEADER + tcp_len) as u16).to_be_bytes());
out.extend_from_slice(&[0, 0]); // identification
out.extend_from_slice(&0x4000u16.to_be_bytes()); // don't fragment
out.push(64); // ttl
out.push(libc::IPPROTO_TCP as u8);
out.extend_from_slice(&[0, 0]); // checksum, filled in below
out.extend_from_slice(&src.octets());
out.extend_from_slice(&dst.octets());
let checksum = ones_complement(sum_bytes(&out[start..]));
out[start + 10..start + 12].copy_from_slice(&checksum.to_be_bytes());
out.extend_from_slice(&tcp);
out.extend_from_slice(payload);
out
}
#[allow(clippy::too_many_arguments)]
fn build_v6(
src: std::net::Ipv6Addr,
dst: std::net::Ipv6Addr,
src_port: u16,
dst_port: u16,
seq: u32,
ack: u32,
flags: u8,
payload: &[u8],
) -> Vec<u8> {
let tcp_len = TCP_HEADER + payload.len();
let pseudo = sum_bytes(&src.octets())
+ sum_bytes(&dst.octets())
+ libc::IPPROTO_TCP as u32
+ tcp_len as u32;
let tcp = tcp_header(src_port, dst_port, seq, ack, flags, payload, pseudo);
let mut out = Vec::with_capacity(ETH_HEADER + IPV6_HEADER + tcp_len);
ethernet(&mut out, 0x86DD);
out.extend_from_slice(&0x6000_0000u32.to_be_bytes()); // version 6, no traffic class
out.extend_from_slice(&(tcp_len as u16).to_be_bytes()); // payload length
out.push(libc::IPPROTO_TCP as u8); // next header
out.push(64); // hop limit
out.extend_from_slice(&src.octets());
out.extend_from_slice(&dst.octets());
out.extend_from_slice(&tcp);
out.extend_from_slice(payload);
out
}
#[cfg(test)]
mod tests {
use super::*;
/// The one property that makes multiplexed streams separable in a capture.
///
/// Two streams of one connection differ in nothing a capture tool looks at — same
/// addresses, same service port — so if this stopped handing out distinct ports they
/// would arrive as one interleaved conversation, which is worse than not capturing
/// them at all: it reads as a single stream whose bytes make no sense.
#[test]
fn every_stream_gets_a_port_of_its_own() {
let mut seen = std::collections::HashSet::new();
for _ in 0..EPHEMERAL_SPAN as usize {
let port = stream_port();
assert!(port >= EPHEMERAL_BASE, "{port} is not an ephemeral port");
assert!(seen.insert(port), "{port} came round before the range was spent");
}
// And then it does come round, rather than running out: a long-lived engine has
// more streams than there are ports, and a new SYN is what separates the reuse —
// exactly as it does for a real port the kernel hands out again.
assert!(!seen.insert(stream_port()));
}
}

186
backend/proxysrc/src/control.rs Normal file
View File

@ -0,0 +1,186 @@
//! The control channel: one JSON ruleset per line on stdin, one ACK per line out.
//!
//! Same shape the NFQUEUE engines use, so the backend side is the code it already
//! has: write the whole ruleset, wait for `ACK OK` or `ACK FAIL <why>`.
use std::net::SocketAddr;
use std::time::Duration;
use tokio::io::{AsyncBufReadExt, BufReader};
use crate::filter::{ChainHandle, FilterChain};
use crate::proxy::{Edge, Onward, Published, Targets};
use crate::relays::Relays;
use crate::rules::parse_ruleset_reusing;
fn ack(ok: bool, detail: &str) {
crate::report::reply(if ok { "ACK OK".to_string() } else { format!("ACK FAIL {detail}") });
}
/// Apply rulesets and handle control commands until stdin closes.
///
/// A rejected update leaves the running chain in place: the engine keeps enforcing
/// what it already had rather than falling back to enforcing nothing, which is the
/// opposite of the fail-open rule and deliberately so. Failing open covers a filter
/// that breaks at runtime; it is not a licence to silently drop a ruleset the
/// operator asked for.
pub async fn serve_stdin(
chain: ChainHandle,
deadline: Duration,
relays: Relays,
targets: std::sync::Arc<Targets>,
) {
let mut lines = BufReader::new(tokio::io::stdin()).lines();
loop {
let line = match lines.next_line().await {
Ok(Some(line)) => line,
// stdin closed: the backend is gone. Keep serving traffic with the
// ruleset we have — the datapath outliving its control channel is the
// whole point of failing open.
Ok(None) => {
eprintln!("[info] [control] stdin closed, keeping the current ruleset");
return;
}
Err(e) => {
eprintln!("[warn] [control] cannot read stdin: {e}");
return;
}
};
let trimmed = line.trim();
if trimmed.is_empty() {
continue;
}
// `ADD_UDP` whatever is behind it: the command names what the *rules* match on,
// and a QUIC service is UDP to the kernel. The backend asks for a port to point
// an address at, and that question has one answer whichever relay gives it.
if let Some(rest) = trimmed.strip_prefix("ADD_UDP ") {
// `<address>` or `<address> <onward>` — what the service behind this relay
// speaks, which is the relay's own answer and not the process's.
let mut parts = rest.split_whitespace();
let target_str = parts.next().unwrap_or("").trim();
let onward = match parts.next() {
None => Onward::Same,
Some(word) => match Onward::from_word(word) {
Some(onward) => onward,
None => {
let err = format!("unknown upstream {word:?}");
eprintln!("[warn] [control] {err}");
ack(false, &err);
continue;
}
},
};
match target_str.parse::<SocketAddr>() {
Ok(upstream) => match relays.add_relay(upstream, onward).await {
Ok(port) => {
crate::report::reply(format!("UDP {upstream}|{} {port}", onward.word()));
ack(true, "");
}
Err(e) => {
eprintln!("[warn] [control] failed to add UDP relay for {upstream}: {e}");
ack(false, &e.to_string());
}
},
Err(e) => {
let err = format!("invalid target address {target_str}: {e}");
eprintln!("[warn] [control] {err}");
ack(false, &err);
}
}
continue;
}
// `PUBLISH <dialled> <target>` and `WITHDRAW <dialled>`: one address is being
// served on a port the service does not listen on. Sent while the engine runs
// because an address can be added to a running service, and the startup list
// (`FGEX_PROXY_TARGETS`) says the same thing for the ones it began with.
if let Some(rest) = trimmed.strip_prefix("PUBLISH ") {
// `PUBLISH <dialled> <edge> <onward> [<service>]`: what is spoken at one
// address, what is spoken to the service behind it, and — where it is not
// simply the service — where that service is. A dash for the last says the
// address *is* the service, which is every address until an operator says
// otherwise.
let mut parts = rest.split_whitespace();
let public = parts.next().map(str::parse::<SocketAddr>);
let edge = match parts.next() {
Some("tls") => Some(Edge::Tls),
Some("any") | Some("clear") | None => Some(Edge::Whatever),
Some(_) => None,
};
let onward = match parts.next() {
Some("plain") => Some(Onward::Plain),
Some("tls") => Some(Onward::Tls),
Some("same") | None => Some(Onward::Same),
Some(_) => None,
};
let target = match parts.next() {
None | Some("-") => Ok(None),
Some(addr) => addr.parse::<SocketAddr>().map(Some),
};
match (public, edge, onward, target) {
(Some(Ok(public)), Some(edge), Some(upstream), Ok(target)) => {
targets.publish(public, Published { target, edge, upstream });
match target {
Some(target) => {
eprintln!("[info] [control] {public} now fronts {target}")
}
None => eprintln!("[info] [control] {public} declared"),
}
ack(true, "");
}
_ => {
let err =
format!("PUBLISH wants `<address> <edge> <onward> [<service>]`, got {rest:?}");
eprintln!("[warn] [control] {err}");
ack(false, &err);
}
}
continue;
}
if let Some(rest) = trimmed.strip_prefix("WITHDRAW ") {
match rest.trim().parse::<SocketAddr>() {
Ok(public) => {
targets.withdraw(&public);
ack(true, "");
}
Err(e) => {
let err = format!("invalid address {rest:?}: {e}");
eprintln!("[warn] [control] {err}");
ack(false, &err);
}
}
continue;
}
// Everything that has to be started is started here, while the old chain is
// still the one enforcing — on a blocking thread, because starting a Python
// worker takes an interpreter's start-up and an import, and this task shares the
// runtime with every relay: with one worker thread, which is what a one-core
// host is given, the service carried nothing while a reload was loading a file.
let line = trimmed.to_string();
let previous = chain.current().filters();
let prepared = tokio::task::spawn_blocking(move || {
parse_ruleset_reusing(&line, &previous).and_then(|filters| {
for filter in &filters {
filter.prepare()?;
}
Ok(filters)
})
})
.await
.unwrap_or_else(|e| Err(format!("preparing the ruleset failed: {e}")));
match prepared {
Ok(filters) => {
let n = filters.len();
chain.replace(FilterChain::new(filters, deadline));
eprintln!("[info] [control] ruleset replaced, {n} rule(s) active");
ack(true, "");
}
Err(e) => {
eprintln!("[warn] [control] ruleset rejected, keeping the previous one: {e}");
ack(false, &e);
}
}
}
}

313
backend/proxysrc/src/debug.rs Normal file
View File

@ -0,0 +1,313 @@
//! Answering "what would this pattern actually do?" with the engine that would do it.
//!
//! An in-app regex tester is only worth having if it is the same matcher the datapath
//! runs. A tester built on Python's `re` would accept backreferences hyperscan rejects,
//! disagree about what `.` matches, and quietly bless a pattern that will be refused
//! the moment it is saved — which is worse than having no tester, because the operator
//! would trust it.
//!
//! So the backend asks this binary instead. One JSON request in, one JSON response
//! out, block mode with start-of-match offsets so the answer can be highlighted.
use serde::{Deserialize, Serialize};
use crate::hyperscan::{self, Mode, Pattern};
/// Enough matches to show, few enough to render. A pattern like `.*` over a sizeable
/// sample matches at almost every offset, and an operator learns nothing from the
/// ten thousandth one.
const MATCH_LIMIT: usize = 1000;
#[derive(Deserialize)]
pub struct DebugRequest {
pub patterns: Vec<DebugPattern>,
/// The sample to scan, base64-encoded: a pattern is matched against bytes, and
/// testing one against bytes that are not valid text is a normal thing to want.
pub sample: String,
}
#[derive(Deserialize)]
pub struct DebugPattern {
pub id: String,
pub expr: String,
#[serde(default = "yes")]
pub case_sensitive: bool,
}
fn yes() -> bool {
true
}
#[derive(Serialize)]
pub struct DebugResponse {
pub matches: Vec<DebugMatch>,
/// Per pattern, the compile error if there was one. A ruleset with one broken
/// pattern still reports the others, because the operator is usually editing one
/// pattern and should not lose the answers about the rest.
pub errors: Vec<DebugError>,
/// Set when the whole request failed rather than one pattern.
#[serde(skip_serializing_if = "Option::is_none")]
pub error: Option<String>,
/// Patterns that are valid and will run, but that this tester cannot show matches
/// for. Highlighting needs block-mode scanning, and block mode rejects a few
/// patterns streaming accepts. Saying so beats either refusing them or silently
/// showing no matches.
pub unscannable: Vec<DebugError>,
pub truncated: bool,
}
#[derive(Serialize)]
pub struct DebugMatch {
pub id: String,
pub from: u64,
pub to: u64,
}
#[derive(Serialize, Debug)]
pub struct DebugError {
pub id: String,
pub error: String,
}
fn b64_decode(input: &str) -> Result<Vec<u8>, String> {
// A dependency for this would be a dependency in the datapath, and the datapath's
// dependency list is a thing worth keeping short.
let mut out = Vec::with_capacity(input.len() * 3 / 4);
let mut acc: u32 = 0;
let mut bits = 0u32;
for ch in input.bytes() {
let value = match ch {
b'A'..=b'Z' => ch - b'A',
b'a'..=b'z' => ch - b'a' + 26,
b'0'..=b'9' => ch - b'0' + 52,
b'+' => 62,
b'/' => 63,
b'=' | b'\n' | b'\r' => continue,
_ => return Err("the sample is not valid base64".to_string()),
} as u32;
acc = (acc << 6) | value;
bits += 6;
if bits >= 8 {
bits -= 8;
out.push((acc >> bits) as u8);
}
}
Ok(out)
}
/// Scan one sample with one set of patterns.
pub fn run(request: &str) -> DebugResponse {
let req: DebugRequest = match serde_json::from_str(request) {
Ok(r) => r,
Err(e) => {
return DebugResponse {
matches: Vec::new(),
errors: Vec::new(),
error: Some(format!("malformed request: {e}")),
unscannable: Vec::new(),
truncated: false,
}
}
};
let sample = match b64_decode(&req.sample) {
Ok(s) => s,
Err(e) => {
return DebugResponse {
matches: Vec::new(),
errors: Vec::new(),
error: Some(e),
unscannable: Vec::new(),
truncated: false,
}
}
};
// Compiled one at a time rather than as a set: a set reports only the first
// pattern that failed, and the point of a tester is to say which of the operator's
// patterns is wrong and why.
let mut errors = Vec::new();
let mut unscannable = Vec::new();
let mut good = Vec::new();
for p in &req.patterns {
let pattern = Pattern {
expr: p.expr.clone(),
case_sensitive: p.case_sensitive,
};
// Validity is judged against the mode this pattern will actually run in, so the
// tester's verdict is the same one the datapath will reach.
let runtime_mode = Mode::Stream;
if let Err(e) = hyperscan::Database::compile(std::slice::from_ref(&pattern), runtime_mode)
{
errors.push(DebugError { id: p.id.clone(), error: e });
continue;
}
// Showing where it matches is a separate question: only block mode reports
// offsets, and it accepts slightly less.
match hyperscan::Database::compile(std::slice::from_ref(&pattern), Mode::Block) {
Ok(_) => good.push((p.id.clone(), pattern)),
Err(e) => unscannable.push(DebugError { id: p.id.clone(), error: e }),
}
}
if good.is_empty() {
return DebugResponse {
matches: Vec::new(),
errors,
error: None,
unscannable,
truncated: false,
};
}
let (ids, patterns): (Vec<String>, Vec<Pattern>) = good.into_iter().unzip();
let db = match hyperscan::Database::compile(&patterns, Mode::Block) {
Ok(db) => db,
Err(e) => {
return DebugResponse {
matches: Vec::new(),
errors,
error: Some(e),
unscannable,
truncated: false,
}
}
};
let hits = match hyperscan::scan_block(&db, &sample, MATCH_LIMIT) {
Ok(h) => h,
Err(e) => {
return DebugResponse {
matches: Vec::new(),
errors,
error: Some(e),
unscannable,
truncated: false,
}
}
};
let truncated = hits.len() >= MATCH_LIMIT;
DebugResponse {
matches: hits
.into_iter()
.map(|m| DebugMatch {
id: ids
.get(m.id as usize)
.cloned()
.unwrap_or_else(|| "unknown".to_string()),
from: m.from,
to: m.to,
})
.collect(),
errors,
error: None,
unscannable,
truncated,
}
}
#[cfg(test)]
mod tests {
use super::*;
fn ask(patterns: &str, sample: &[u8]) -> DebugResponse {
// base64 by hand, to keep the encoder and the decoder honest about each other
const ALPHABET: &[u8] = b"ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/";
let mut encoded = String::new();
for block in sample.chunks(3) {
let b = [
block[0],
*block.get(1).unwrap_or(&0),
*block.get(2).unwrap_or(&0),
];
let n = ((b[0] as u32) << 16) | ((b[1] as u32) << 8) | b[2] as u32;
for i in 0..4 {
if i <= block.len() {
encoded.push(ALPHABET[((n >> (18 - i * 6)) & 63) as usize] as char);
} else {
encoded.push('=');
}
}
}
run(&format!(r#"{{"patterns":{patterns},"sample":"{encoded}"}}"#))
}
#[test]
fn reports_where_a_pattern_matched() {
let res = ask(r#"[{"id":"a","expr":"FLAG\\{[a-z]+\\}"}]"#, b"xx FLAG{abc} yy");
assert!(res.error.is_none(), "{:?}", res.error);
assert_eq!(res.matches.len(), 1);
assert_eq!(res.matches[0].id, "a");
assert_eq!((res.matches[0].from, res.matches[0].to), (3, 12));
}
#[test]
fn reports_every_match_not_just_the_first() {
let res = ask(r#"[{"id":"a","expr":"ab"}]"#, b"ab_ab_ab");
assert_eq!(res.matches.len(), 3);
}
#[test]
fn names_the_pattern_that_failed_and_keeps_the_rest() {
let res = ask(
r#"[{"id":"bad","expr":"(unclosed"},{"id":"good","expr":"ok"}]"#,
b"this is ok",
);
assert_eq!(res.errors.len(), 1);
assert_eq!(res.errors[0].id, "bad");
assert!(!res.errors[0].error.is_empty());
assert_eq!(res.matches.len(), 1, "the valid pattern still reported");
assert_eq!(res.matches[0].id, "good");
}
#[test]
fn honours_case_sensitivity_per_pattern() {
let res = ask(
r#"[{"id":"s","expr":"ABC"},{"id":"i","expr":"ABC","case_sensitive":false}]"#,
b"abc",
);
let ids: Vec<&str> = res.matches.iter().map(|m| m.id.as_str()).collect();
assert_eq!(ids, vec!["i"]);
}
#[test]
fn matches_bytes_that_are_not_text() {
let res = ask(r#"[{"id":"a","expr":"\\xff\\xfe"}]"#, &[0x41, 0xff, 0xfe]);
assert!(res.error.is_none(), "{:?}", res.error);
assert_eq!(res.matches.len(), 1);
assert_eq!((res.matches[0].from, res.matches[0].to), (1, 3));
}
/// The two modes do not accept exactly the same patterns, and judging a pattern by
/// the wrong one refuses something that works. A bounded repeat compiles for
/// streaming — which is how a blocking pattern runs — but not for the block-mode
/// scanning the tester highlights with.
#[test]
fn a_pattern_is_judged_by_the_mode_it_will_run_in() {
let res = ask(r#"[{"id":"a","expr":"a{1,1000}b"}]"#, b"aab");
assert!(res.errors.is_empty(), "a valid blocking pattern was refused: {:?}", res.errors);
assert_eq!(
res.unscannable.len(),
1,
"it is valid but cannot be highlighted, and that has to be said"
);
}
#[test]
fn a_malformed_request_is_an_error_not_a_panic() {
let res = run("{not json");
assert!(res.error.is_some());
}
#[test]
fn an_empty_sample_matches_nothing_and_does_not_fail() {
let res = ask(r#"[{"id":"a","expr":"x"}]"#, b"");
assert!(res.error.is_none());
assert!(res.matches.is_empty());
}
}

618
backend/proxysrc/src/filter.rs Normal file
View File

@ -0,0 +1,618 @@
//! The filter layer, and the isolation that keeps it from taking the datapath down.
//!
//! Everything here exists to uphold one rule: **a filter may fail, the traffic may
//! not**. The NFQUEUE engine gets that property from the kernel (`NFQA_CFG_F_FAIL_OPEN`
//! plus the `bypass` flag on the nft rule): if userspace stops answering, packets flow
//! anyway. A userspace proxy has no such backstop, so it has to be built here.
use std::panic::{catch_unwind, AssertUnwindSafe};
use std::sync::atomic::{AtomicBool, AtomicU64, Ordering};
use std::sync::Arc;
use std::sync::OnceLock;
use std::time::{Duration, Instant};
/// Which half of the connection a chunk came from.
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
pub enum Direction {
/// Data travelling from the client towards the protected service.
ClientToServer,
/// Data travelling from the protected service back to the client.
ServerToClient,
}
impl Direction {
pub fn as_str(self) -> &'static str {
match self {
Direction::ClientToServer => "c->s",
Direction::ServerToClient => "s->c",
}
}
}
/// What a filter decided about a chunk of stream data.
#[derive(Clone, Debug, PartialEq, Eq)]
pub enum Verdict {
/// Forward the chunk unchanged.
Accept,
/// Close the connection. Carries the id of the rule that refused it, so a block
/// is attributed without a shared slot two connections could race over.
Reject(Option<String>),
}
/// What a filter is shown about one chunk of traffic. Borrowed, so a filter that
/// cannot block is handed the relay's own buffers with nothing copied.
///
/// Only the new bytes. There is deliberately no re-scan region: a filter that has to
/// find a pattern straddling two chunks keeps the state to do so in its own session,
/// which costs nothing per chunk. An overlap window was the previous answer and it
/// was both slower and bounded — it could only find patterns shorter than itself.
pub struct FilterCtx<'a> {
pub direction: Direction,
/// The bytes that just arrived.
pub chunk: &'a [u8],
/// Which connection these bytes belong to.
///
/// Both directions of one connection share it, because that is what a stream is
/// to a filter. A filter that keeps state — the user's Python keeps a whole set of
/// module globals — needs this to keep one client's state away from another's, and
/// nothing else in the datapath can tell it for them.
pub connection: ConnectionId,
}
/// What a filter is allowed to know about the layers below the application one.
///
/// Addresses and ports, and nothing else — no header bytes in either direction. The
/// two network layers cannot honestly offer the same thing below this line: the proxy
/// terminated the connection and writes its own headers, so a filter rewriting one
/// here would be rewriting nothing, while on NFQUEUE the same edit desynchronises the
/// stream. Metadata is the part that means the same thing on both, so metadata is what
/// crosses.
#[derive(Clone, Copy, Debug)]
pub struct ConnectionMeta {
/// The peer that opened the connection.
pub client: std::net::SocketAddr,
/// Where it was actually headed — the protected service, not this proxy.
pub server: std::net::SocketAddr,
/// What is underneath. A filter is shown it because the library's stream and HTTP
/// models need a stream to be built on, and a filter written against `RawPacket` may
/// legitimately want to know what it is looking at.
pub l4: L4,
}
/// What carries a connection.
///
/// A boolean until QUIC arrived, which broke the question in two: a QUIC stream is
/// ordered and reliable, so every model that needs a stream applies to it, and it is
/// carried by UDP, so every rule that matches the wire matches UDP. Answering either of
/// those with "is it TCP" gets the other one wrong — as a boolean this either hid the
/// stream models from QUIC or told a filter it was looking at TCP.
#[derive(Clone, Copy, PartialEq, Eq, Debug)]
pub enum L4 {
Tcp,
Udp,
/// A stream inside a QUIC connection.
Quic,
/// A QUIC DATAGRAM frame: inside a QUIC connection, and outside its streams.
///
/// Its own value rather than [`L4::Quic`] with a flag, because the two answer the
/// stream question differently and everything the library builds on a stream turns
/// on that answer. Telling a filter `udp` instead would answer both questions
/// correctly — unreliable, and UDP on the wire — and lose the only thing left worth
/// knowing, which is that this arrived inside a connection that is also carrying
/// streams.
QuicDatagram,
}
impl L4 {
/// Whether the bytes arrive as an ordered, reliable stream — which is what the
/// library's stream and HTTP models are built on.
pub fn is_stream(self) -> bool {
matches!(self, L4::Tcp | L4::Quic)
}
/// The name the filter library knows it by.
pub fn name(self) -> &'static str {
match self {
L4::Tcp => "tcp",
L4::Udp => "udp",
L4::Quic => "quic",
L4::QuicDatagram => "quic-datagram",
}
}
}
/// Identifies one connection for the lifetime of the process.
pub type ConnectionId = u64;
/// Hands out connection ids. Wrapping would take 585 years at a million per second.
static NEXT_CONNECTION: AtomicU64 = AtomicU64::new(1);
pub fn next_connection_id() -> ConnectionId {
NEXT_CONNECTION.fetch_add(1, Ordering::Relaxed)
}
/// A unit of inspection. Deliberately synchronous and blocking: filters are user
/// code (today C++-embedded Python, tomorrow whatever we choose), and the engine
/// must stay correct when that code misbehaves rather than assume it will not.
pub trait Filter: Send + Sync + 'static {
fn name(&self) -> &str;
fn inspect(&self, ctx: &FilterCtx<'_>) -> Verdict;
/// Whether this filter might not return.
///
/// User code has to be assumed capable of hanging, so it runs on a blocking
/// thread under a deadline — which costs a thread hop and a copy of every chunk.
/// A compiled regular expression cannot hang (the engine has no backtracking),
/// so it says so and is run inline instead. A benchmark put that overhead at
/// most of the difference between filtering and not filtering.
fn may_block(&self) -> bool {
true
}
/// Do whatever has to succeed before this filter can be trusted to run.
///
/// Called when a ruleset is applied, never on the datapath. A filter that cannot
/// get ready refuses the whole update, which is deliberately not fail-open: that
/// covers a rule breaking under traffic, not a rule the operator asked for and
/// never got.
fn prepare(&self) -> Result<(), String> {
Ok(())
}
/// Open this filter's state for one direction of one connection.
///
/// `None` means the filter is stateless for that direction and [`Filter::inspect`]
/// answers instead. Matching state must never be shared between connections —
/// one client's bytes deciding another client's verdict is both a false positive
/// and a way to smuggle a pattern past the filter by splitting it across two
/// connections.
///
/// A stateful filter has to be non-blocking. The blocking path abandons its
/// thread when a filter blows the deadline, and a thread that owns connection
/// state cannot be abandoned without losing it; [`FilterChain::with_policy`]
/// refuses the combination rather than leaving it to be discovered under load.
fn open_session(&self, _dir: Direction) -> Option<Box<dyn FilterSession>> {
None
}
/// A connection has started, and here is what is known about it.
///
/// Called once per connection before any of its chunks, so a filter that reports
/// metadata to something outside this process has it before the first verdict is
/// asked for. Most filters do not care and take the default.
fn connection_opened(&self, _connection: ConnectionId, _meta: &ConnectionMeta) {}
/// This connection is over; release whatever was being kept for it.
///
/// Sessions are dropped by the connection that owned them, so a filter whose state
/// is entirely in its session need not implement this. A filter holding state
/// somewhere else — the Python worker keeps a set of globals per connection, in a
/// process of its own — has no other way to learn that it can let go, and would
/// otherwise grow for as long as the service runs.
fn connection_closed(&self, _connection: ConnectionId) {}
/// What makes this filter the same filter in the next ruleset, if anything does.
///
/// A ruleset is rebuilt whole on every edit, and a filter that keeps per-connection
/// state outside its sessions loses all of it when it is rebuilt: the Python worker
/// is a process, and a new one meets every open connection halfway. A filter that
/// answers here, with a key that changes whenever its configuration does, is carried
/// over into the new chain instead when the key matches. `None` — the default — is
/// always rebuilt.
fn reuse_key(&self) -> Option<&str> {
None
}
}
/// One filter's state for one direction of one connection.
///
/// Split from [`Filter`] because the two have different lifetimes: the filter is the
/// compiled configuration, shared and swapped wholesale, while the session is the
/// per-connection position in the stream.
pub trait FilterSession: Send + 'static {
fn inspect(&mut self, ctx: &FilterCtx<'_>) -> Verdict;
}
/// Every filter's state for one direction of one connection.
///
/// Held by the relay and handed to [`FilterChain::run`], so the sessions live and die
/// with the connection they belong to. Rebuilt when the chain underneath is replaced:
/// the sessions belong to the old chain's filters, and a swap means the operator
/// changed the rules — matching a new ruleset against old partial state would report
/// matches for patterns that were never in force.
pub struct ChainSessions {
/// The connection these belong to, handed to every filter that is consulted.
connection: ConnectionId,
/// Which chain these sessions were opened against. Zero means "none yet"; live
/// chains number from one.
generation: u64,
slots: Vec<Option<Box<dyn FilterSession>>>,
}
impl ChainSessions {
pub fn new(connection: ConnectionId) -> Self {
Self {
connection,
generation: 0,
slots: Vec::new(),
}
}
}
/// Why a filter stopped being consulted.
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
pub enum DisableReason {
Panicked,
TimedOut,
}
/// Counters worth reporting back to the backend. Kept plain so the spike does not
/// grow a metrics dependency before it has earned one.
#[derive(Debug, Default)]
pub struct ChainStats {
pub inspected: AtomicU64,
pub accepted: AtomicU64,
pub rejected: AtomicU64,
pub panics: AtomicU64,
pub timeouts: AtomicU64,
pub filters_disabled: AtomicU64,
pub bypassed_chunks: AtomicU64,
}
/// Numbers chains so a connection can notice its sessions are stale. Starts at one
/// because zero means "no sessions opened yet".
static CHAIN_GENERATION: AtomicU64 = AtomicU64::new(1);
/// How long a filter that kept missing its deadline sits out before it is asked again.
///
/// Slow is not broken. A Python filter misses its deadline when exchanges queue behind one
/// another, which is what a flood does — the one moment its protection is wanted — and
/// disabled for the life of the chain it stayed off until somebody next edited a rule,
/// with one log line to say so. So it is tried again after a pause, and if it is still
/// slow it loses its say again after the same few misses: a filter that genuinely never
/// returns strands a few threads per pause rather than one per chunk. A panic is broken
/// code, and stays out until the chain is replaced.
pub const REARM_AFTER: Duration = Duration::from_secs(30);
/// Milliseconds on a monotonic clock, never zero, so zero can mean "not scheduled".
fn monotonic_ms() -> u64 {
static START: OnceLock<Instant> = OnceLock::new();
START.get_or_init(Instant::now).elapsed().as_millis() as u64 + 1
}
struct Slot {
filter: Arc<dyn Filter>,
/// Set once the filter has misbehaved enough to lose its say.
disabled: AtomicBool,
consecutive_timeouts: AtomicU64,
/// When a filter that timed out is asked again (`monotonic_ms`), or 0 for never.
rearm_at_ms: AtomicU64,
}
/// An immutable set of filters. Swapping the whole chain is how configuration
/// changes reach live connections, so a chain is never mutated in place.
pub struct FilterChain {
slots: Vec<Slot>,
/// Identifies this chain, so a connection can tell that the one it opened its
/// sessions against is no longer the live one.
generation: u64,
deadline: Duration,
max_consecutive_timeouts: u64,
/// How long a filter that timed out sits out. `REARM_AFTER` but for tests.
rearm_after: Duration,
/// The earliest `rearm_at_ms` of any slot, or 0: what lets the hot path skip the clock.
next_rearm_ms: AtomicU64,
/// Trips when no filter is left to consult, making the chain a pure relay.
degraded: AtomicBool,
pub stats: Arc<ChainStats>,
}
impl FilterChain {
pub fn new(filters: Vec<Arc<dyn Filter>>, deadline: Duration) -> Self {
Self::with_policy(filters, deadline, 3)
}
/// The filters this chain consults, in order — for the next ruleset to carry over
/// the ones it has not changed (`Filter::reuse_key`).
pub fn filters(&self) -> Vec<Arc<dyn Filter>> {
self.slots.iter().map(|slot| slot.filter.clone()).collect()
}
pub fn with_policy(
filters: Vec<Arc<dyn Filter>>,
deadline: Duration,
max_consecutive_timeouts: u64,
) -> Self {
let degraded = AtomicBool::new(filters.is_empty());
// A stateful filter on the blocking path would lose its state the first time
// it blew the deadline, silently and only under load. Refusing it here makes
// that a programming error instead of an incident.
debug_assert!(
filters
.iter()
.all(|f| !f.may_block()
|| f.open_session(Direction::ClientToServer).is_none()),
"a filter that may block cannot carry per-connection state"
);
Self {
slots: filters
.into_iter()
.map(|filter| Slot {
filter,
disabled: AtomicBool::new(false),
consecutive_timeouts: AtomicU64::new(0),
rearm_at_ms: AtomicU64::new(0),
})
.collect(),
generation: CHAIN_GENERATION.fetch_add(1, Ordering::Relaxed),
deadline,
max_consecutive_timeouts,
rearm_after: REARM_AFTER,
next_rearm_ms: AtomicU64::new(0),
degraded,
stats: Arc::new(ChainStats::default()),
}
}
/// The same chain, with filters that time out asked again after `after` rather than
/// `REARM_AFTER`.
pub fn rearming_after(mut self, after: Duration) -> Self {
self.rearm_after = after;
self
}
pub fn empty() -> Self {
Self::new(Vec::new(), Duration::from_millis(50))
}
/// True when the chain has nothing left to say and the relay can skip it
/// entirely — no copy, no thread hop. The degraded path has to be the cheap
/// one, otherwise failing open costs more than working normally.
pub fn is_bypassed(&self) -> bool {
self.degraded.load(Ordering::Relaxed)
}
pub fn disabled_filters(&self) -> Vec<&str> {
self.slots
.iter()
.filter(|s| s.disabled.load(Ordering::Relaxed))
.map(|s| s.filter.name())
.collect()
}
fn disable(&self, slot: &Slot, reason: DisableReason) {
if slot.disabled.swap(true, Ordering::Relaxed) {
return; // already out of the loop, do not double count
}
self.stats.filters_disabled.fetch_add(1, Ordering::Relaxed);
match reason {
DisableReason::TimedOut => {
let at = monotonic_ms() + self.rearm_after.as_millis() as u64;
slot.rearm_at_ms.store(at, Ordering::Relaxed);
let _ = self.next_rearm_ms.fetch_update(Ordering::Relaxed, Ordering::Relaxed,
|next| (next == 0 || at < next).then_some(at));
crate::diag!(
"[warn] [filter] '{}' kept missing its {:?} deadline: traffic keeps \
flowing without it, and it is asked again in {}s",
slot.filter.name(),
self.deadline,
self.rearm_after.as_secs(),
);
}
DisableReason::Panicked => crate::diag!(
"[warn] [filter] '{}' panicked: traffic keeps flowing without it until the \
filters are next applied — save a change to them, or restart the service",
slot.filter.name(),
),
}
if self
.slots
.iter()
.all(|s| s.disabled.load(Ordering::Relaxed))
{
self.degraded.store(true, Ordering::Relaxed);
crate::diag!("[warn] [filter] every filter is disabled, chain is now a plain relay");
}
}
/// Tell every filter that a connection has started, and what it is.
pub fn connection_opened(&self, connection: ConnectionId, meta: &ConnectionMeta) {
for slot in &self.slots {
slot.filter.connection_opened(connection, meta);
}
}
/// Tell every filter that a connection is over.
///
/// Called once per connection, not once per direction: a stream is one thing to a
/// filter, and telling it twice would free state the other direction still needs.
pub fn connection_closed(&self, connection: ConnectionId) {
for slot in &self.slots {
slot.filter.connection_closed(connection);
}
}
/// Run the chain over one chunk.
///
/// Put back the filters whose pause is over. See `REARM_AFTER`.
fn rearm(&self) {
let now = monotonic_ms();
let mut next = 0;
for slot in &self.slots {
let at = slot.rearm_at_ms.load(Ordering::Relaxed);
if at == 0 {
continue;
}
if now < at {
next = if next == 0 { at } else { next.min(at) };
continue;
}
// Whoever wins this puts the filter back; everybody else sees it done.
if slot
.rearm_at_ms
.compare_exchange(at, 0, Ordering::Relaxed, Ordering::Relaxed)
.is_ok()
{
slot.consecutive_timeouts.store(0, Ordering::Relaxed);
slot.disabled.store(false, Ordering::Relaxed);
self.degraded.store(false, Ordering::Relaxed);
crate::diag!(
"[info] [filter] '{}' is back in the chain after sitting out {}s",
slot.filter.name(),
self.rearm_after.as_secs(),
);
}
}
self.next_rearm_ms.store(next, Ordering::Relaxed);
}
/// Any way a filter can fail — panic, or simply never returning — resolves to
/// `Accept` for the chunk in flight. The chunk is never held hostage by the
/// filter's failure.
pub async fn run(
&self,
dir: Direction,
data: &[u8],
sessions: &mut ChainSessions,
) -> Verdict {
let due = self.next_rearm_ms.load(Ordering::Relaxed);
if due != 0 && monotonic_ms() >= due {
self.rearm();
}
if self.is_bypassed() {
self.stats.bypassed_chunks.fetch_add(1, Ordering::Relaxed);
return Verdict::Accept;
}
// First chunk of this connection, or the chain was swapped underneath it.
if sessions.generation != self.generation {
sessions.slots = self
.slots
.iter()
.map(|slot| slot.filter.open_session(dir))
.collect();
sessions.generation = self.generation;
}
self.stats.inspected.fetch_add(1, Ordering::Relaxed);
for (index, slot) in self.slots.iter().enumerate() {
if slot.disabled.load(Ordering::Relaxed) {
continue;
}
// A filter that cannot hang is run right here: no thread hop, no copy.
// The deadline exists for code that might never return, and paying for
// it on code that cannot is what the benchmark showed we were doing.
if !slot.filter.may_block() {
let ctx = FilterCtx {
direction: dir,
chunk: data,
connection: sessions.connection,
};
let outcome = match sessions.slots.get_mut(index).and_then(Option::as_mut) {
Some(session) => {
catch_unwind(AssertUnwindSafe(|| session.inspect(&ctx)))
}
None => catch_unwind(AssertUnwindSafe(|| slot.filter.inspect(&ctx))),
};
match outcome {
Ok(Verdict::Accept) => {}
Ok(Verdict::Reject(by)) => {
self.stats.rejected.fetch_add(1, Ordering::Relaxed);
let id = by.clone().unwrap_or_else(|| slot.filter.name().to_string());
crate::report::blocked(&id);
return Verdict::Reject(by);
}
Err(_) => {
self.stats.panics.fetch_add(1, Ordering::Relaxed);
self.disable(slot, DisableReason::Panicked);
}
}
continue;
}
// The filter needs owned data: it runs on a blocking thread that may
// outlive this call if it hangs, so it cannot borrow the relay buffer.
let owned_chunk = data.to_vec();
let filter = Arc::clone(&slot.filter);
let connection = sessions.connection;
let handle = tokio::task::spawn_blocking(move || {
let ctx = FilterCtx {
direction: dir,
chunk: &owned_chunk,
connection,
};
catch_unwind(AssertUnwindSafe(|| filter.inspect(&ctx)))
});
match tokio::time::timeout(self.deadline, handle).await {
Ok(Ok(Ok(verdict))) => {
slot.consecutive_timeouts.store(0, Ordering::Relaxed);
match verdict {
Verdict::Accept => {}
Verdict::Reject(by) => {
self.stats.rejected.fetch_add(1, Ordering::Relaxed);
// The backend attributes the block to a rule by this id,
// the same way cppregex reports `BLOCKED <id>`.
let id = by.clone().unwrap_or_else(|| slot.filter.name().to_string());
crate::report::blocked(&id);
return Verdict::Reject(by);
}
}
}
// The filter panicked: it is broken code, not a transient hiccup,
// so it loses its say immediately.
Ok(Ok(Err(_))) | Ok(Err(_)) => {
self.stats.panics.fetch_add(1, Ordering::Relaxed);
self.disable(slot, DisableReason::Panicked);
}
// The filter blew the deadline. One slow call can be load, so it
// takes a few in a row to lose the filter. The blocking thread is
// abandoned; it cannot be cancelled, and waiting for it is exactly
// the stall we refuse to have.
Err(_) => {
self.stats.timeouts.fetch_add(1, Ordering::Relaxed);
let seen = slot.consecutive_timeouts.fetch_add(1, Ordering::Relaxed) + 1;
if seen >= self.max_consecutive_timeouts {
self.disable(slot, DisableReason::TimedOut);
}
}
}
}
self.stats.accepted.fetch_add(1, Ordering::Relaxed);
Verdict::Accept
}
}
/// Hands the live chain to every connection, and lets it be swapped underneath
/// them. Reconfiguring must not cost established connections: dropping them on a
/// filter edit would be a regression against the NFQUEUE engine, which reloads
/// its rules in place.
#[derive(Clone)]
pub struct ChainHandle {
rx: tokio::sync::watch::Receiver<Arc<FilterChain>>,
tx: Arc<tokio::sync::watch::Sender<Arc<FilterChain>>>,
}
impl ChainHandle {
pub fn new(chain: FilterChain) -> Self {
let (tx, rx) = tokio::sync::watch::channel(Arc::new(chain));
Self {
rx,
tx: Arc::new(tx),
}
}
/// The chain to use for the next chunk. Cheap enough to call per chunk.
pub fn current(&self) -> Arc<FilterChain> {
self.rx.borrow().clone()
}
/// Install a new chain. Connections in flight pick it up on their next chunk.
pub fn replace(&self, chain: FilterChain) {
let _ = self.tx.send(Arc::new(chain));
}
}

335
backend/proxysrc/src/h1up.rs Normal file
View File

@ -0,0 +1,335 @@
//! Speaking HTTP/1.1 to the service, under an edge that is not HTTP/1.1.
//!
//! Everything else in this engine forwards to the service in the version the client
//! arrived in: the rendering is a view shown to the filters, never a translation the
//! service sees. This module is the one exception, and it is the operator's to ask for
//! (`FGEX_PROXY_UPSTREAM`): an ordinary web service that speaks HTTP/1.1 and nothing else
//! can be put behind an HTTP/3 edge, with firegex terminating QUIC in front of it.
//!
//! **Half of it is free and half of it needed a crate.** The request direction is already
//! done by the time it gets here — [`crate::http1`] renders every exchange as the HTTP/1.1
//! it would have been, because that is what the chain has to be shown, so putting those
//! same bytes on a socket invents nothing. The *answer* is what could not be done: HTTP/1.1
//! is the one version this engine has never parsed. An HTTP/1.1 connection elsewhere is a
//! byte pump — the filters look at bytes, and what parses them is the library on the Python
//! side — so there was no response parser to reach for, and writing one beside hyper's is
//! exactly what `AGENTS.md` forbids. So the answer comes back through hyper's client, and
//! this module is a binding: framing, chunked encoding and keep-alive are the crate's
//! problem.
//!
//! **One connection per exchange.** HTTP/1.1 has no multiplexing, and the edge above does:
//! a QUIC stream is one request, so it gets one upstream connection of its own, dialled
//! from the client's own address like every other dial this engine makes. Pooling would
//! put two clients' requests on one socket, and with it the thing `ChainSessions` exists
//! to prevent — one client's bytes deciding another's verdict — at a layer where nobody
//! would think to look for it.
use std::io;
use std::net::SocketAddr;
use std::pin::Pin;
use std::sync::Arc;
use std::task::{Context, Poll};
use std::time::Duration;
use bytes::Bytes;
use http_body_util::BodyExt;
use crate::http1::{Answer, Incoming, Outbound, OutboundBody};
/// The body of a request being written piece by piece into hyper.
///
/// hyper takes a body as one value and pulls from it; the chain produces pieces as it
/// accepts them. A channel is the join between the two, and its bound is what keeps a
/// fast client from buying memory in this process while a slow service reads.
struct Streamed {
rx: tokio::sync::mpsc::Receiver<http_body::Frame<Bytes>>,
}
impl http_body::Body for Streamed {
type Data = Bytes;
type Error = io::Error;
fn poll_frame(
self: Pin<&mut Self>,
cx: &mut Context<'_>,
) -> Poll<Option<Result<http_body::Frame<Bytes>, Self::Error>>> {
self.get_mut().rx.poll_recv(cx).map(|frame| frame.map(Ok))
}
}
/// How many pieces of one request may be waiting for the service at once.
const IN_FLIGHT: usize = 8;
/// How this engine reaches a service that speaks HTTP/1.1.
///
/// Dial parameters and nothing else: there is no connection here, because each exchange
/// opens its own. Cloned per exchange the way the HTTP/3 sender above it is.
#[derive(Clone)]
pub(crate) struct H1Upstream {
pub(crate) upstream: SocketAddr,
/// Dialled as this address, so the service sees the real client. The same
/// unconditional source preservation every other dial here makes.
pub(crate) client: SocketAddr,
/// Present when the service speaks TLS: the client half of the handshake.
pub(crate) tls: Option<Arc<rustls::ClientConfig>>,
/// The name the client asked for, presented to the service in turn. A service that
/// picks its certificate or its virtual host by SNI would otherwise be handed the bare
/// address — which TLS does not even send — and answer as its default host.
pub(crate) server_name: Option<String>,
pub(crate) connect_timeout: Duration,
pub(crate) self_mark: Option<u32>,
/// Dial as the client. Off is the fallback the TCP path also has, and it is here for
/// the same reason: a service seeing one source address for everyone is a regression
/// nobody would attribute to us.
pub(crate) spoof: bool,
}
impl H1Upstream {
async fn connect(&self) -> io::Result<tokio::net::TcpStream> {
if !self.spoof {
return tokio::time::timeout(
self.connect_timeout,
crate::transparent::connect_plain(self.upstream, self.self_mark),
)
.await
.map_err(|_| io::Error::new(io::ErrorKind::TimedOut, "upstream connect timed out"))?;
}
// Losing the client's address is bad, losing the connection is worse — the same
// trade, and just as loud about it, as the TCP and QUIC dials make.
match tokio::time::timeout(
self.connect_timeout,
crate::transparent::connect_as(self.client.ip(), self.upstream, self.self_mark),
)
.await
{
Ok(Ok(stream)) => Ok(stream),
failed => {
let why = match failed {
Ok(Err(e)) => e.to_string(),
_ => "timed out (is the return path diverted?)".to_string(),
};
crate::diag_throttled!("[warn] [h1] dials that could not keep the client's address", 1, 60,
"[warn] [h1] cannot reach {} as {}: {why}. Falling back to our own \
address — the service will not see real client IPs.",
self.upstream,
self.client.ip(),
);
tokio::time::timeout(
self.connect_timeout,
crate::transparent::connect_plain(self.upstream, self.self_mark),
)
.await
.map_err(|_| {
io::Error::new(io::ErrorKind::TimedOut, "upstream connect timed out")
})?
}
}
}
}
impl Outbound for H1Upstream {
type Error = io::Error;
type Body = H1Body;
type Answer = H1Answer;
async fn open(
&mut self,
request: http::Request<()>,
ends_it: bool,
) -> io::Result<(Self::Body, Self::Answer)> {
let socket = self.connect().await?;
let _ = socket.set_nodelay(true);
let (tx, rx) = tokio::sync::mpsc::channel(IN_FLIGHT);
let body = Streamed { rx };
// The head goes out as HTTP/1.1 whatever it arrived as, and that is more than
// restating the version. HTTP/2 and HTTP/3 carry the target split across
// `:scheme`, `:authority` and `:path`, which `http::Uri` holds as one absolute
// URI — and written to a socket unchanged that is the *proxy* form of a request
// line, `GET https://host/path`, which an origin server is entitled to refuse and
// which every service reading the path back gets wrong. So the target is reduced
// to its origin form and the authority moves to `host`, which is exactly the
// decision `http1::render_request` already makes for the chain: what the filters
// were shown and what the service receives are the same message, and the two
// deciding it differently is the drift this engine is arranged against.
let (mut parts, _) = request.into_parts();
parts.version = http::Version::HTTP_11;
if let Some(authority) = parts.uri.authority().map(|a| a.to_string()) {
if !parts.headers.contains_key(http::header::HOST) {
if let Ok(value) = http::HeaderValue::from_str(&authority) {
parts.headers.insert(http::header::HOST, value);
}
}
}
parts.uri = parts
.uri
.path_and_query()
.map(|target| target.as_str())
.unwrap_or("/")
.parse()
.map_err(|e| io::Error::other(format!("cannot render the request target: {e}")))?;
let request = http::Request::from_parts(parts, body);
let answer = match &self.tls {
Some(config) => {
let name = match self.server_name.as_deref().map(crate::tls::server_name) {
Some(Ok(name)) => name,
_ => crate::tls::server_name(&self.upstream.ip().to_string())?,
};
let stream = tokio::time::timeout(
self.connect_timeout,
crate::tls::connector(config.clone()).connect(name, socket),
)
.await
.map_err(|_| {
io::Error::new(io::ErrorKind::TimedOut, "upstream TLS handshake timed out")
})??;
Self::handshake(stream, request).await?
}
None => Self::handshake(socket, request).await?,
};
// A head that is the whole message gets a body that is already over: hyper then
// frames the request with no body at all, which is what HTTP/1.1 does for one.
// The flag exists for HTTP/2, where it has to ride on the head; here it only
// saves a channel nobody will write to.
Ok((H1Body { tx: (!ends_it).then_some(tx) }, answer))
}
}
impl H1Upstream {
/// The hyper side of opening one exchange, shared by the two kinds of socket.
async fn handshake<S>(io: S, request: http::Request<Streamed>) -> io::Result<H1Answer>
where
S: tokio::io::AsyncRead + tokio::io::AsyncWrite + Unpin + Send + 'static,
{
let (mut sender, connection) =
hyper::client::conn::http1::handshake(hyper_util::rt::TokioIo::new(io))
.await
.map_err(io::Error::other)?;
// hyper drives the socket from here; without this task nothing is ever written.
// Its end is this exchange's end, so it is dropped with the answer rather than
// detached — one connection per exchange is what makes that safe to say.
let driving = tokio::spawn(async move {
if let Err(e) = connection.await {
crate::diag_throttled!("[info] [h1] connections to the service that ended with an error", 20, 10, "[info] [h1] the connection to the service ended: {e}");
}
});
// Not awaited here: the service is allowed to answer before it has read the whole
// request — a refusal, a redirect, a `413` — and the body is still being written
// into the channel above by the other half of this exchange. Awaiting it now is a
// deadlock with a name.
let pending = Box::pin(sender.send_request(request));
Ok(H1Answer {
pending: Some(pending),
body: None,
trailers: None,
_driving: driving,
})
}
}
/// The request body's writing half.
pub(crate) struct H1Body {
/// Taken by `finish`, because dropping the sender is what ends the body: hyper reads
/// the close as the last frame and writes the terminating chunk. Waiting on the
/// channel instead — for the *receiver* to go — is waiting for the service to answer
/// a request this end has not finished sending, which is a deadlock with a name.
tx: Option<tokio::sync::mpsc::Sender<http_body::Frame<Bytes>>>,
}
impl H1Body {
async fn send(&mut self, frame: http_body::Frame<Bytes>) -> io::Result<()> {
let Some(tx) = self.tx.as_ref() else {
return Err(io::Error::other("this request body has already ended"));
};
tx.send(frame)
.await
.map_err(|_| io::Error::other("the service stopped reading the request"))
}
}
impl OutboundBody for H1Body {
type Error = io::Error;
async fn data(&mut self, chunk: Bytes) -> io::Result<()> {
self.send(http_body::Frame::data(chunk)).await
}
async fn trailers(&mut self, trailers: http::HeaderMap) -> io::Result<()> {
self.send(http_body::Frame::trailers(trailers)).await
}
async fn finish(&mut self) -> io::Result<()> {
self.tx.take();
Ok(())
}
}
type Pending = Pin<
Box<dyn std::future::Future<Output = hyper::Result<http::Response<hyper::body::Incoming>>> + Send>,
>;
/// The service's answer, read the way every other answer in this engine is.
pub(crate) struct H1Answer {
/// Taken by `response()`, which is the first thing the reading half does.
pending: Option<Pending>,
body: Option<hyper::body::Incoming>,
/// A trailer section arrives as a frame among the body's, so it is kept here until
/// the body is done and somebody asks for it — which is the order the rendering
/// wants it in.
trailers: Option<http::HeaderMap>,
/// Dropped with the answer, which cancels the connection task: one connection per
/// exchange means its life is exactly this one's.
_driving: tokio::task::JoinHandle<()>,
}
impl Answer for H1Answer {
async fn response(&mut self) -> io::Result<http::Response<()>> {
let pending = self
.pending
.take()
.ok_or_else(|| io::Error::other("the answer was already taken"))?;
let response = pending.await.map_err(io::Error::other)?;
let (mut parts, body) = response.into_parts();
// This answer goes on in HTTP/2 or HTTP/3, where what an HTTP/1.1 connection says
// about itself has no place — and the rendering the chain is shown settles its own
// framing from what is left.
crate::http1::drop_connection_headers(&mut parts.headers);
self.body = Some(body);
Ok(http::Response::from_parts(parts, ()))
}
}
impl Incoming for H1Answer {
type Error = io::Error;
fn ended(&self) -> bool {
// hyper knows: a response framed with no body at all — `204`, `304`, a declared
// length of zero — is over before anybody reads it.
use http_body::Body;
self.body.as_ref().is_none_or(|body| body.is_end_stream())
}
async fn data(&mut self) -> io::Result<Option<Bytes>> {
let Some(body) = self.body.as_mut() else {
return Ok(None);
};
while let Some(frame) = body.frame().await {
let frame = frame.map_err(io::Error::other)?;
match frame.into_data() {
Ok(data) => return Ok(Some(data)),
// Not data, so it is the trailer section, and it is the last thing the
// body will produce. Kept for whoever asks next rather than dropped.
Err(frame) => {
if let Ok(trailers) = frame.into_trailers() {
self.trailers = Some(trailers);
}
}
}
}
Ok(None)
}
async fn trailers(&mut self) -> io::Result<Option<http::HeaderMap>> {
Ok(self.trailers.take())
}
}

827
backend/proxysrc/src/h2.rs Normal file
View File

@ -0,0 +1,827 @@
//! HTTP/2, terminated by the engine — and shown to the filters as HTTP/1.1.
//!
//! The rendering lives in [`crate::http1`]; what is here is the adapter, and the reason it
//! exists is the same one that put [`crate::h3`] there. HTTP/2 puts the method, the path
//! and the headers in an **HPACK-compressed** HEADERS frame, so a connection that
//! negotiated `h2` and was only forwarded showed a filter a compression format: a pattern
//! written against a request line matched nothing, and a filter asking for an
//! `HttpRequest` was handed the `PRI * HTTP/2.0` preface — parsed as one fictitious
//! request — and then frames it could not read, so it was never called again. No block, no
//! log, no counter: the exact failure everything else in this engine is arranged against.
//! Only a body travelled in the clear. Since gRPC *is* HTTP/2, that also meant gRPC could
//! not be filtered at all.
//!
//! Unlike QUIC, terminating was never forced here — the bytes are framed by a transport
//! the kernel understands, and firegex could have gone on forwarding them. It is forced by
//! the promise instead: a filter written once has to work everywhere, and there is no way
//! to keep that promise over HPACK without decompressing it, which is what parsing HTTP/2
//! means.
//!
//! Three things differ from h3 and are worth stating, because each of them is a way to get
//! this wrong:
//!
//! * **Flow control is ours to run.** `h2` hands received bytes over and waits for them to
//! be released; a proxy that releases on read has no backpressure and buffers whatever
//! the faster side sends, and one that never releases stalls after the first window. So
//! capacity is released **after the piece has been forwarded**, which is what makes the
//! window follow the service rather than this process, and outbound writes reserve
//! capacity first rather than queueing in memory.
//! * **The end of a message rides on its last frame**, not on a separate close. A response
//! of head-and-nothing-else must be sent with `end_of_stream` **on the head**: send it
//! open and then close it with an empty DATA frame and a *trailers-only* answer stops
//! being one — which is precisely the shape of a gRPC status-only reply, and gRPC
//! clients refuse it. [`head_is_the_message`] is the one place that decision is made,
//! for both directions.
//! * **A stream is a connection to the chain**, which on HTTP/1.1 it is not. There a
//! keep-alive connection carries many requests through one set of filter state; here the
//! streams are concurrent and interleaved, and sharing state between them is exactly
//! "one client's bytes deciding another client's verdict" — the thing [`ChainSessions`]
//! exists to prevent. So an h2 stream gets its own, as an h3 request stream does.
//!
//! **Server push is not carried, and the service is told so in the handshake**
//! (`enable_push(false)`) rather than having its pushes dropped — the same courtesy the
//! QUIC edge pays about datagrams. **Extended CONNECT is not advertised**, and a plain
//! `CONNECT` stream is refused rather than rendered: a tunnel is opaque bytes, and
//! inventing an HTTP/1.1 request for one would be the fabrication this module exists to
//! avoid.
use std::io;
use std::net::SocketAddr;
use std::sync::atomic::{AtomicBool, Ordering};
use std::sync::Arc;
use std::time::Duration;
use bytes::Bytes;
use tokio::io::{AsyncRead, AsyncWrite};
use crate::capture::{Capture, Tap};
use crate::filter::{
next_connection_id, ChainHandle, ChainSessions, ConnectionId, ConnectionMeta, Direction, L4,
};
use crate::http1::{
carried, close_body, conflicting_authority, head_framing, judge, push_body, render_request,
render_response, Answer,
Framing, Incoming, Outbound, OutboundBody, Rendered,
};
use crate::proxy::ProxyStats;
/// How many streams one client connection may have open at once.
///
/// The h2 crate's default is none at all, and on this edge every stream is an exchange of
/// its own — towards an HTTP/1.1 service, a connection of its own to the service. So one
/// client connection, which `max_connections` counts once, could open as many connections
/// to the protected service as it liked, and a Python filter's state with each. A hundred
/// is what the QUIC edge already allows (quinn's default), so the two versions of one
/// service answer the same, and it is what RFC 9113 recommends as a floor.
const MAX_CONCURRENT_STREAMS: u32 = 100;
/// How long a refused connection is given to put its GOAWAY on the wire.
///
/// A refusal that simply dropped the socket would reach the client as a truncated TLS
/// stream, which is the peer having to guess — the same reason the byte pumps shut down
/// rather than abort. Bounded, because a peer that has stopped reading must not be able to
/// hold the task open.
const GOAWAY_GRACE: Duration = Duration::from_secs(2);
impl Incoming for h2::RecvStream {
type Error = h2::Error;
fn data(&mut self) -> impl std::future::Future<Output = Result<Option<Bytes>, h2::Error>> + Send {
// Named through the trait's own path rather than `self.data()`: the inherent
// method has the same name, and which one a bare call resolves to is not something
// this file should depend on.
async move { h2::RecvStream::data(self).await.transpose() }
}
fn trailers(
&mut self,
) -> impl std::future::Future<Output = Result<Option<http::HeaderMap>, h2::Error>> + Send {
h2::RecvStream::trailers(self)
}
fn release(&mut self, taken: usize) -> Result<(), h2::Error> {
self.flow_control().release_capacity(taken)
}
fn ended(&self) -> bool {
self.is_end_stream()
}
}
/// HTTP/2 towards the service, which is what an HTTP/2 client edge used to always get.
///
/// A handle to the connection `carry` opened, cloned per stream — which is what HTTP/2
/// multiplexing is, and why this is a clone rather than a dial.
#[derive(Clone)]
pub(crate) struct H2Upstream(h2::client::SendRequest<Bytes>);
impl Outbound for H2Upstream {
type Error = io::Error;
type Body = H2Send;
type Answer = H2Recv;
async fn open(
&mut self,
request: http::Request<()>,
ends_it: bool,
) -> io::Result<(Self::Body, Self::Answer)> {
// The parts travel as they arrived; only the version is restated, because what
// goes out is HTTP/2 again.
let (mut parts, _) = request.into_parts();
parts.version = http::Version::HTTP_2;
let (response, to_service) = self
.0
.send_request(http::Request::from_parts(parts, ()), ends_it)
.map_err(io::Error::other)?;
Ok((
H2Send { inner: to_service, done: ends_it },
H2Recv { pending: Some(response), body: None, trailers: None },
))
}
}
/// The request body on its way to the service, over HTTP/2.
pub(crate) struct H2Send {
inner: h2::SendStream<Bytes>,
/// Whether the end of the stream has already gone out. Writing past it is a user
/// error `h2` reports rather than ignores, and the end can leave on the head, on a
/// trailer section, or on an empty final frame — three ways to get here once.
done: bool,
}
impl OutboundBody for H2Send {
type Error = io::Error;
async fn data(&mut self, chunk: Bytes) -> io::Result<()> {
send_body(&mut self.inner, chunk, false).await.map_err(io::Error::other)
}
async fn trailers(&mut self, trailers: http::HeaderMap) -> io::Result<()> {
self.done = true;
self.inner.send_trailers(trailers).map_err(io::Error::other)
}
async fn finish(&mut self) -> io::Result<()> {
if self.done {
return Ok(());
}
self.done = true;
self.inner.send_data(Bytes::new(), true).map_err(io::Error::other)
}
}
/// The service's answer over HTTP/2, read the way every other answer in this engine is.
pub(crate) struct H2Recv {
pending: Option<h2::client::ResponseFuture>,
body: Option<h2::RecvStream>,
trailers: Option<http::HeaderMap>,
}
impl Answer for H2Recv {
async fn response(&mut self) -> io::Result<http::Response<()>> {
let pending = self
.pending
.take()
.ok_or_else(|| io::Error::other("the answer was already taken"))?;
let response = pending.await.map_err(io::Error::other)?;
let (parts, body) = response.into_parts();
self.body = Some(body);
Ok(http::Response::from_parts(parts, ()))
}
}
impl Incoming for H2Recv {
type Error = io::Error;
async fn data(&mut self) -> io::Result<Option<Bytes>> {
let Some(body) = self.body.as_mut() else {
return Ok(None);
};
Incoming::data(body).await.map_err(io::Error::other)
}
async fn trailers(&mut self) -> io::Result<Option<http::HeaderMap>> {
if let Some(trailers) = self.trailers.take() {
return Ok(Some(trailers));
}
let Some(body) = self.body.as_mut() else {
return Ok(None);
};
Incoming::trailers(body).await.map_err(io::Error::other)
}
fn release(&mut self, taken: usize) -> io::Result<()> {
match self.body.as_mut() {
Some(body) => body.release(taken).map_err(io::Error::other),
None => Ok(()),
}
}
fn ended(&self) -> bool {
self.body.as_ref().is_some_and(|body| body.is_end_stream())
}
}
/// Everything one terminated HTTP/2 connection is carried with.
///
/// Cloned per stream: every field is a handle already, so a clone is a few reference
/// counts and the streams need nothing from each other.
#[derive(Clone)]
pub struct Carriage {
pub client: SocketAddr,
pub upstream: SocketAddr,
pub chain: ChainHandle,
pub stats: Arc<ProxyStats>,
/// Where a reconstruction of each stream is written, when anything is listening.
pub capture: Option<Arc<Capture>>,
/// How long this connection may say nothing at all. The question is the
/// *connection's*, exactly as it is on QUIC: a peer that opens twenty streams and
/// speaks on none is the shape it is there to catch.
pub first_byte_timeout: Option<Duration>,
}
impl Carriage {
fn rendered(&self) -> Rendered<'_> {
Rendered {
chain: &self.chain,
stats: &self.stats,
client: self.client,
layer: "h2",
}
}
/// Tell the chain about a stream, in the words the rest of the engine uses.
///
/// `L4::Tcp`, with nothing to distinguish it from HTTP/1.1 on the same service — which
/// is the whole point. An h2 stream is a stream, and it is TCP on the wire, so both
/// questions the enum answers have the same answer they have one layer up.
fn open(&self) -> ConnectionId {
let connection = next_connection_id();
self.chain.current().connection_opened(
connection,
&ConnectionMeta {
client: self.client,
server: self.upstream,
l4: L4::Tcp,
},
);
connection
}
/// A reconstruction of one stream, for whoever is watching the capture interface.
///
/// Per stream and not per connection: a stream is what has a beginning, an order and
/// an end. What it has not got is a port of its own — every stream of one connection
/// shares the four-tuple — so [`Tap::open_stream`] invents one, and the documentation
/// says so wherever this interface is offered.
fn tap(&self) -> Option<Arc<Tap>> {
Tap::open_stream(self.capture.clone(), self.client, self.upstream)
}
}
/// Ends the whole connection because a rule refused something on one of its streams.
///
/// A refusal ends the connection, not the stream. Resetting one stream of many would leave
/// the client free to ask again on the next, which is not what a block means anywhere else
/// in firegex — and on HTTP/1.1 keep-alive a refused request already takes the connection
/// with it.
#[derive(Clone)]
struct Refusal(Arc<tokio::sync::watch::Sender<bool>>);
impl Refusal {
fn new() -> (Self, tokio::sync::watch::Receiver<bool>) {
let (tx, rx) = tokio::sync::watch::channel(false);
(Self(Arc::new(tx)), rx)
}
fn refuse(&self) {
let _ = self.0.send(true);
}
}
/// Carry one connection that both ends agreed to speak HTTP/2 on.
pub async fn carry<C, S>(client_io: C, service_io: S, ctx: Carriage) -> io::Result<()>
where
C: AsyncRead + AsyncWrite + Unpin + Send + 'static,
S: AsyncRead + AsyncWrite + Unpin + Send + 'static,
{
let (sender, driver) = h2::client::Builder::new()
// Said in the handshake rather than by dropping what arrives: a pushed response
// this proxy cannot render is one the service should not spend a stream on, and a
// peer that learns from SETTINGS is better off than one that watches pushes
// vanish. The same answer the QUIC edge gives about datagrams on h3.
.enable_push(false)
.handshake::<_, Bytes>(service_io)
.await
.map_err(io::Error::other)?;
carry_with(client_io, H2Upstream(sender), Some(tokio::spawn(driver)), ctx).await
}
/// Carry an HTTP/2 client in front of a service that speaks **HTTP/1.1**.
///
/// The edge the QUIC side has had all along, arriving here: what the filters are shown is
/// the same HTTP/1.1 rendering either way, so sending that rendering on to the service
/// invents nothing. It is what lets an ordinary cleartext web service be reached over
/// HTTP/2 with only firegex holding a certificate — the deployment that could already be
/// reached over HTTP/3 and, until this existed, not over HTTP/2, for no reason an operator
/// could see.
///
/// No connection driver, because there is no connection: `H1Upstream` dials once per
/// exchange, which is also what keeps two clients' requests off one socket.
pub(crate) async fn carry_to_h1<C>(
client_io: C,
upstream: crate::h1up::H1Upstream,
ctx: Carriage,
) -> io::Result<()>
where
C: AsyncRead + AsyncWrite + Unpin + Send + 'static,
{
carry_with(client_io, upstream, None, ctx).await
}
async fn carry_with<C, U>(
client_io: C,
upstream: U,
mut driving: Option<tokio::task::JoinHandle<Result<(), h2::Error>>>,
ctx: Carriage,
) -> io::Result<()>
where
C: AsyncRead + AsyncWrite + Unpin + Send + 'static,
U: Outbound + Clone + Send + 'static,
{
// Extended CONNECT is left unadvertised, which is the default: a service that wants
// WebTransport over HTTP/2 needs a session model this proxy has not got, and
// advertising it would invite streams whose rendering would be a fabrication. Push is
// turned off towards the service above, in the same spirit.
let mut server = h2::server::Builder::new()
.max_concurrent_streams(MAX_CONCURRENT_STREAMS)
.max_header_list_size(crate::http1::MAX_HEAD_BYTES)
.handshake::<_, Bytes>(client_io)
.await
.map_err(io::Error::other)?;
let (refusal, mut refused) = Refusal::new();
// Set by the first head that crosses in either direction. A service that answers
// before its client has spoken satisfies it, exactly as a request would — the
// deadline is about silence, not about who speaks first.
let spoken = Arc::new(AtomicBool::new(false));
let watchdog = ctx.first_byte_timeout.map(|deadline| {
let spoken = Arc::clone(&spoken);
let refusal = refusal.clone();
let stats = Arc::clone(&ctx.stats);
tokio::spawn(async move {
tokio::time::sleep(deadline).await;
if !spoken.load(Ordering::Relaxed) {
stats.no_first_byte.fetch_add(1, Ordering::Relaxed);
// Through the same channel a refusal uses, so the connection is wound
// down with a GOAWAY rather than dropped: a peer dropped mid-stream reads
// a truncation, and this connection has done nothing to deserve one.
refusal.refuse();
}
})
});
// The exchanges already in flight, kept rather than detached. A client winding the
// connection down leaves `accept()` answering `None` while answers are still owed, and
// walking away there would cut off the last request of every well-behaved client.
let mut inflight = tokio::task::JoinSet::new();
// A refusal and a first-byte deadline both end the connection here rather than
// letting it run out; the difference between them is in the log, not in what is
// done about it.
let mut ended_early = false;
loop {
tokio::select! {
accepted = server.accept() => match accepted {
Some(Ok((request, respond))) => {
let ctx = ctx.clone();
let refusal = refusal.clone();
let mut sender = upstream.clone();
let spoken = Arc::clone(&spoken);
inflight.spawn(async move {
let connection = ctx.open();
let outcome =
exchange(&ctx, &refusal, &spoken, connection, request, respond, &mut sender)
.await;
// Whatever happened, the filter's state for this stream goes — a
// refused request and a completed one both end it.
ctx.chain.current().connection_closed(connection);
if let Err(e) = outcome {
crate::diag_throttled!("[info] [h2] streams that ended with an error", 20, 10, "[info] [h2] stream ended: {e}");
}
});
}
Some(Err(e)) => {
crate::diag_throttled!("[info] [h2] client connections that ended with an error", 20, 10, "[info] [h2] the client's connection ended: {e}");
break;
}
// The client is done with the connection, which is the ordinary way out.
None => break,
},
// The service ended its side. Waiting for the client to notice would mean
// holding a connection that has nowhere to go. Only where there *is* a
// connection: an HTTP/1.1 upstream opens one per exchange, so there is no
// shared driver to outlive and nothing to watch here.
reason = async { driving.as_mut().unwrap().await }, if driving.is_some() => {
if let Ok(Err(e)) = reason {
crate::diag_throttled!("[info] [h2] connections the service ended", 20, 10, "[info] [h2] the service ended the connection: {e}");
}
break;
}
_ = refused.changed() => {
ended_early = true;
break;
}
}
}
if ended_early {
// CANCEL rather than REFUSED_STREAM: the latter tells a client the request was
// never acted on and may be retried, and a block a client retries past is not a
// block. The reason reaches the operator through the service log, which is where
// the filter that refused it is already named.
server.abrupt_shutdown(h2::Reason::CANCEL);
// Polled rather than dropped, so the GOAWAY is actually written before the socket
// goes. Bounded: a peer that has stopped reading must not hold this open.
let _ = tokio::time::timeout(
GOAWAY_GRACE,
std::future::poll_fn(|cx| server.poll_closed(cx)),
)
.await;
// Drained even here, and this is the part that is easy to get wrong: the
// exchanges still in flight have to *end* rather than be dropped, because ending
// is what releases the filter state each of them holds — for Python, a whole set
// of module globals, living in the worker under a connection id that only
// `connection_closed` frees. Aborting them by dropping the set leaked one of
// those per stream, on the path an attacker triggers on purpose. They finish
// immediately: the connection under them is already shut down.
let _ = tokio::time::timeout(GOAWAY_GRACE, drain(&mut inflight)).await;
} else {
// Nothing here is unbounded: every filter an exchange consults runs under the
// chain's own deadline, and the connection has the socket's. Unbounded on
// purpose, though — a client winding the connection down leaves answers still
// owed, and walking away would cut off its last exchange.
drain(&mut inflight).await;
}
drop(upstream);
if let Some(driving) = driving {
driving.abort();
}
if let Some(watchdog) = watchdog {
watchdog.abort();
}
Ok(())
}
/// Wait for every exchange still in flight to end.
///
/// Ending is what releases what a filter is holding for that stream, so this is not
/// tidiness: a set of exchanges dropped rather than joined leaves their state behind.
async fn drain(inflight: &mut tokio::task::JoinSet<()>) {
while inflight.join_next().await.is_some() {}
}
/// Whether a message's head is the whole of the message.
///
/// Its own function because getting it wrong is invisible on HTTP/1.1 and fatal on
/// HTTP/2: the end of a message rides on its last frame, so a head that *is* the entire
/// message has to be sent as one. Sending it open and closing with an empty DATA frame
/// turns a trailers-only answer — a gRPC status-only reply is exactly that — into a
/// message with a body, which gRPC clients refuse.
///
/// Three things have to hold together, and the sender is the authority on the third: no
/// first chunk was read, no trailer section is still owed, and the peer really did end its
/// half. The last one is why a `content-length: 0` request stays headers-only through this
/// proxy instead of growing an empty DATA frame it did not have.
fn head_is_the_message<I: Incoming>(
first: &Option<Bytes>,
early: &Option<Option<http::HeaderMap>>,
from: &I,
) -> bool {
first.is_none() && !matches!(early, Some(Some(_))) && from.ended()
}
/// One request and its answer.
///
/// The two directions are carried **at the same time**, not one after the other, which is
/// the same shape the byte pumps have and for the same reason: a service is allowed to
/// answer before it has read the whole request — a refusal, a redirect, a `413` — and a
/// proxy that insists on finishing the upload first turns that into a stall neither end
/// can explain.
#[allow(clippy::too_many_arguments)]
async fn exchange<O: Outbound>(
ctx: &Carriage,
refusal: &Refusal,
spoken: &AtomicBool,
connection: ConnectionId,
request: http::Request<h2::RecvStream>,
respond: h2::server::SendResponse<Bytes>,
sender: &mut O,
) -> io::Result<()> {
spoken.store(true, Ordering::Relaxed);
let (parts, mut from_client) = request.into_parts();
let head = http::Request::from_parts(parts, ());
// A tunnel is opaque bytes with no HTTP/1.1 message to render, so it is refused rather
// than shown to the chain as something it is not. Naming it beats a stream that hangs.
if head.method() == http::Method::CONNECT {
crate::diag_throttled!("[warn] [h2] CONNECT streams refused", 5, 30,
"[warn] [h2] {}: a CONNECT stream arrived. firegex terminates HTTP/2 to show \
the filters each exchange as HTTP/1.1, and a tunnel has no such exchange in \
it, so the stream is refused rather than carried unread.",
ctx.client
);
let mut respond = respond;
respond.send_reset(h2::Reason::REFUSED_STREAM);
return Ok(());
}
// One authority, or none: the chain is shown one `Host` and the service is handed the
// request as it came, so two that disagree would be a filter reading one host while
// the service answers for another. Malformed by RFC 9113 §8.3.1, and treated so.
if let Some(why) = conflicting_authority(&head) {
crate::diag_throttled!("[warn] [h2] requests with two authorities", 5, 30,
"[warn] [h2] {}: {why}, so the filters would be shown one host while the \
service routes on the other. The stream is reset as malformed.",
ctx.client
);
let mut respond = respond;
respond.send_reset(h2::Reason::PROTOCOL_ERROR);
return Ok(());
}
// A request that carries no body must not be shown `transfer-encoding: chunked`, which
// it never had — a filter looking for that header is usually looking for smuggling,
// and one invented by the proxy in front of it is the worst possible answer. So the
// framing is settled before the head is rendered, and settled from as little as will
// do it.
let (framing, first, early) = head_framing(&mut from_client, head.headers())
.await
.map_err(io::Error::other)?;
let head_ends_it = head_is_the_message(&first, &early, &from_client);
// One reconstruction for the exchange, not one per direction: a request and its
// response are one conversation, which is what they would have been on HTTP/1.1.
let tap = ctx.tap();
let rendered = ctx.rendered();
let mut sessions = ChainSessions::new(connection);
let mut view = render_request(&head, &framing);
if let Some(chunk) = &first {
push_body(&mut view, chunk, &framing);
}
// A message that ended with a trailer section and no body is rendered whole here,
// terminator and trailers included: there is no later chunk to hang them off, and a
// trailer section shown to nobody is a piece of the message that travelled unread.
if let Some(trailers) = early.as_ref() {
close_body(&mut view, &framing, trailers.as_ref());
}
if !judge(&rendered, Direction::ClientToServer, &view, &mut sessions, tap.as_ref()).await {
refusal.refuse();
return Ok(());
}
// Only now does the service hear about it, in whatever version it speaks — HTTP/2
// again where the service speaks it, HTTP/1.1 where the address said the service is a
// cleartext one. The rendering above is the same either way, which is what makes the
// second case a forward rather than a translation invented here.
let (parts, _) = head.into_parts();
let (mut to_service, from_service) = sender
.open(http::Request::from_parts(parts, ()), head_ends_it)
.await
.map_err(io::Error::other)?;
if let Some(chunk) = first {
let len = chunk.len();
to_service.data(chunk).await.map_err(io::Error::other)?;
// After the forward, never before: the window that reopens is the one the service
// has actually taken the bytes off, which is what backpressure means here.
from_client.release(len).map_err(io::Error::other)?;
}
let (asked, answered) = tokio::join!(
carry_request(
ctx, refusal, from_client, to_service, framing, sessions, tap.clone(), early,
head_ends_it,
),
// Its own sessions, the same id: both halves of one exchange are one connection to
// a filter that keeps state, and each direction keeps its own position in it.
carry_response(ctx, refusal, spoken, connection, from_service, respond, tap.clone()),
);
if let Some(tap) = tap {
tap.closed();
}
asked?;
answered
}
/// The rest of the request body, judged chunk by chunk and forwarded.
#[allow(clippy::too_many_arguments)]
async fn carry_request<B: OutboundBody>(
ctx: &Carriage,
refusal: &Refusal,
mut from_client: h2::RecvStream,
mut to_service: B,
framing: Framing,
mut sessions: ChainSessions,
tap: Option<Arc<Tap>>,
early: Option<Option<http::HeaderMap>>,
head_ended_it: bool,
) -> io::Result<()>
where
B::Error: std::error::Error + Send + Sync + 'static,
{
// The head carried the end of the stream, so this half is already finished. Touching
// it again would be writing past the end, which `h2` reports as a user error rather
// than ignoring.
if head_ended_it {
return Ok(());
}
let rendered = ctx.rendered();
let trailers = match early {
// The body was over before the head was rendered, so both were shown to the chain
// together. Reading either again would be reading past the end of the stream.
Some(trailers) => trailers,
None => {
// A pattern split across two DATA frames is still a pattern, so the session
// carries on from the head.
while let Some(chunk) = Incoming::data(&mut from_client)
.await
.map_err(io::Error::other)?
{
let mut view = Vec::new();
push_body(&mut view, &chunk, &framing);
if !judge(&rendered, Direction::ClientToServer, &view, &mut sessions, tap.as_ref())
.await
{
refusal.refuse();
return Ok(());
}
let len = chunk.len();
to_service.data(chunk).await.map_err(io::Error::other)?;
from_client.release(len).map_err(io::Error::other)?;
}
let trailers = Incoming::trailers(&mut from_client)
.await
.map_err(io::Error::other)?;
let mut tail = Vec::new();
close_body(&mut tail, &framing, trailers.as_ref());
if !tail.is_empty()
&& !judge(&rendered, Direction::ClientToServer, &tail, &mut sessions, tap.as_ref())
.await
{
refusal.refuse();
return Ok(());
}
carried(&rendered, framing, trailers)
}
};
if let Some(trailers) = trailers {
to_service.trailers(trailers).await.map_err(io::Error::other)?;
}
to_service.finish().await.map_err(io::Error::other)
}
/// The answer, rendered and judged the same way.
#[allow(clippy::too_many_arguments)]
async fn carry_response<A: Answer>(
ctx: &Carriage,
refusal: &Refusal,
spoken: &AtomicBool,
connection: ConnectionId,
mut from_service: A,
mut respond: h2::server::SendResponse<Bytes>,
tap: Option<Arc<Tap>>,
) -> io::Result<()>
where
<A as Incoming>::Error: std::error::Error + Send + Sync + 'static,
{
let rendered = ctx.rendered();
let head = from_service.response().await.map_err(io::Error::other)?;
spoken.store(true, Ordering::Relaxed);
let mut sessions = ChainSessions::new(connection);
// The answer's framing is settled the same way the request's was, and for the second
// half of the same reason: a service that sends its head and then waits — a streaming
// RPC, a long poll — must not have that head held here, or the client is waiting for
// something that is waiting for the client.
let (framing, first, early) = head_framing(&mut from_service, head.headers())
.await
.map_err(io::Error::other)?;
let head_ends_it = head_is_the_message(&first, &early, &from_service);
let mut view = render_response(&head, &framing);
if let Some(chunk) = &first {
push_body(&mut view, chunk, &framing);
}
if let Some(trailers) = early.as_ref() {
close_body(&mut view, &framing, trailers.as_ref());
}
if !judge(&rendered, Direction::ServerToClient, &view, &mut sessions, tap.as_ref()).await {
refusal.refuse();
return Ok(());
}
let (mut parts, _) = head.into_parts();
parts.version = http::Version::HTTP_2;
// `end_of_stream` on the head where the head is the message: a status-only gRPC answer
// is HEADERS and nothing else, and closing it with an empty DATA frame instead would
// make it a message with a body that gRPC clients then refuse.
let mut to_client = respond
.send_response(http::Response::from_parts(parts, ()), head_ends_it)
.map_err(io::Error::other)?;
if let Some(chunk) = first {
let len = chunk.len();
send_body(&mut to_client, chunk, false).await.map_err(io::Error::other)?;
from_service.release(len).map_err(io::Error::other)?;
}
if head_ends_it {
return Ok(());
}
let trailers = match early {
Some(trailers) => trailers,
None => {
while let Some(chunk) = Incoming::data(&mut from_service)
.await
.map_err(io::Error::other)?
{
let mut view = Vec::new();
push_body(&mut view, &chunk, &framing);
if !judge(&rendered, Direction::ServerToClient, &view, &mut sessions, tap.as_ref())
.await
{
refusal.refuse();
return Ok(());
}
let len = chunk.len();
send_body(&mut to_client, chunk, false).await.map_err(io::Error::other)?;
from_service.release(len).map_err(io::Error::other)?;
}
let trailers = Incoming::trailers(&mut from_service)
.await
.map_err(io::Error::other)?;
let mut tail = Vec::new();
close_body(&mut tail, &framing, trailers.as_ref());
if !tail.is_empty()
&& !judge(&rendered, Direction::ServerToClient, &tail, &mut sessions, tap.as_ref())
.await
{
refusal.refuse();
return Ok(());
}
carried(&rendered, framing, trailers)
}
};
finish(&mut to_client, trailers).map_err(io::Error::other)
}
/// End a half of an exchange, with the trailer section it is allowed to carry.
///
/// A message whose head already carried `end_of_stream` is finished, and touching the
/// stream again would be writing past the end of it; `h2` reports that as a user error
/// rather than ignoring it, so the emptiness is checked here rather than at each call.
fn finish(
to: &mut h2::SendStream<Bytes>,
trailers: Option<http::HeaderMap>,
) -> Result<(), h2::Error> {
match trailers {
Some(trailers) => to.send_trailers(trailers),
None => to.send_data(Bytes::new(), true),
}
}
/// Write one piece of a body, taking the window as it opens.
///
/// `send_data` on its own queues whatever it is given inside the connection, so a client
/// that uploads faster than the service reads would be buying memory in this process —
/// which is the resource the connection limit exists to defend. Reserving first and
/// writing what is granted is what turns the far end's window into backpressure on the
/// near one, and it is the other half of releasing capacity only after a forward.
async fn send_body(
to: &mut h2::SendStream<Bytes>,
mut data: Bytes,
end: bool,
) -> Result<(), h2::Error> {
if data.is_empty() {
return to.send_data(data, end);
}
while !data.is_empty() {
to.reserve_capacity(data.len());
let granted = match std::future::poll_fn(|cx| to.poll_capacity(cx)).await {
Some(granted) => granted?,
// The far end is gone. Saying so as an error lets the caller wind the
// exchange down the way every other failure here does.
None => return to.send_data(data, end),
};
if granted == 0 {
continue;
}
let piece = data.split_to(granted.min(data.len()));
to.send_data(piece, end && data.is_empty())?;
}
Ok(())
}

512
backend/proxysrc/src/h3.rs Normal file
View File

@ -0,0 +1,512 @@
//! HTTP/3, terminated on top of QUIC — and shown to the filters as HTTP/1.1.
//!
//! The rendering itself lives in [`crate::http1`], which is where the reasoning about it
//! belongs now that HTTP/2 is shown to the chain the same way. What is left here is the
//! adapter: hyperium's `h3` parses and re-encodes the frames, this module decides when to
//! ask the chain and what to do with the answer.
//!
//! Terminating is not a choice on this layer the way it is on TCP. Past its Initial packet
//! QUIC encrypts the frames, the stream boundaries and the packet number, so a layer that
//! only forwards has nothing a filter could be shown. What goes on to the service is
//! HTTP/3 again, re-encoded — the rendering is a view, never a translation the service
//! sees. Unless the operator says the service speaks HTTP/1.1 and nothing else, which is
//! the one place that sentence stops being true and is written down where it is chosen:
//! see [`crate::h1up`].
//!
//! One request is one connection to the chain, which is [`Carrier::open`]'s rule applied
//! to h3: a request stream carries exactly one exchange, so a filter's per-connection
//! state — for Python, a whole set of module globals — lives exactly as long as the
//! request it belongs to.
use std::future::Future;
use std::io;
use std::sync::atomic::Ordering;
use std::sync::Arc;
use bytes::{Buf, Bytes};
use h3::error::StreamError;
use crate::capture::Tap;
use crate::filter::{ChainSessions, ConnectionId, Direction};
use crate::http1::{
carried, close_body, conflicting_authority, connection_specific, head_framing, judge,
push_body, render_request,
render_response, Answer,
Framing, Incoming, Outbound, OutboundBody, Rendered,
};
use crate::quic::{Behind, Carrier};
impl<S, B> Incoming for h3::server::RequestStream<S, B>
where
S: h3::quic::RecvStream + Send,
B: Buf + Send,
{
type Error = StreamError;
fn data(&mut self) -> impl Future<Output = Result<Option<Bytes>, StreamError>> + Send {
async move { Ok(self.recv_data().await?.map(collect)) }
}
fn trailers(
&mut self,
) -> impl Future<Output = Result<Option<http::HeaderMap>, StreamError>> + Send {
self.recv_trailers()
}
}
impl<S, B> Incoming for h3::client::RequestStream<S, B>
where
S: h3::quic::RecvStream + Send,
B: Buf + Send,
{
type Error = StreamError;
fn data(&mut self) -> impl Future<Output = Result<Option<Bytes>, StreamError>> + Send {
async move { Ok(self.recv_data().await?.map(collect)) }
}
fn trailers(
&mut self,
) -> impl Future<Output = Result<Option<http::HeaderMap>, StreamError>> + Send {
self.recv_trailers()
}
}
/// How this connection's exchanges are described to the shared rendering.
fn rendered<'a>(carrier: &'a Carrier) -> Rendered<'a> {
Rendered {
chain: &carrier.chain,
stats: &carrier.stats,
client: carrier.client,
layer: "h3",
}
}
/// The two halves of one h3 request stream, as the shared rendering wants them.
type H3Send = h3::client::RequestStream<
<h3_quinn::BidiStream<Bytes> as h3::quic::BidiStream<Bytes>>::SendStream,
Bytes,
>;
type H3Recv = h3::client::RequestStream<
<h3_quinn::BidiStream<Bytes> as h3::quic::BidiStream<Bytes>>::RecvStream,
Bytes,
>;
/// HTTP/3 towards the service, which is what every HTTP/3 client edge used to get.
impl Outbound for h3::client::SendRequest<h3_quinn::OpenStreams, Bytes> {
type Error = StreamError;
type Body = H3Send;
type Answer = H3Recv;
async fn open(
&mut self,
request: http::Request<()>,
// HTTP/3 frames the end of a message on the stream, so nothing has to be said on
// the head: the body half is finished by the caller when there is none, exactly
// as it is when there is one. The flag is HTTP/2's problem.
_ends_it: bool,
) -> Result<(Self::Body, Self::Answer), StreamError> {
// The parts travel as they arrived; only the version is restated, because what
// goes out is HTTP/3 again.
let (mut parts, _) = request.into_parts();
parts.version = http::Version::HTTP_3;
let stream = self
.send_request(http::Request::from_parts(parts, ()))
.await?;
let (send, recv) = stream.split();
Ok((send, recv))
}
}
impl OutboundBody for H3Send {
type Error = StreamError;
async fn data(&mut self, chunk: Bytes) -> Result<(), StreamError> {
self.send_data(chunk).await
}
async fn trailers(&mut self, trailers: http::HeaderMap) -> Result<(), StreamError> {
self.send_trailers(trailers).await
}
async fn finish(&mut self) -> Result<(), StreamError> {
h3::client::RequestStream::finish(self).await
}
}
impl Answer for H3Recv {
async fn response(&mut self) -> Result<http::Response<()>, StreamError> {
self.recv_response().await
}
}
/// Carry one HTTP/3 connection: accept requests, and forward each one.
///
/// Where they are forwarded *to* is the operator's choice, and the only thing that
/// changes with it is which sender the exchanges are handed. Everything the chain is
/// shown is built by [`crate::http1`] either way — which is the whole reason the choice
/// could be offered at all.
pub(crate) async fn carry(carrier: &Carrier) {
// **No GREASE, in either direction.** The reserved frame and stream types exist so
// that implementations keep their handling of unknown extensions honest, which is a
// worthy thing for a browser to do and is not firegex's to do on somebody's service.
// Sending them means putting a frame on the wire that the client never sent, and the
// first real server this was pointed at — aioquic — stopped answering requests
// entirely because of it: the trailing grease frame left its end-of-stream
// unreported, so the request was received and never completed. A firewall that
// breaks the service it is protecting in order to exercise that service's protocol
// handling has its priorities backwards.
let mut server = match h3::server::builder()
.send_grease(false)
.max_field_section_size(u64::from(crate::http1::MAX_HEAD_BYTES))
.build(h3_quinn::Connection::new(carrier.peer.clone()))
.await
{
Ok(connection) => connection,
Err(e) => {
crate::diag_throttled!("[info] [h3] HTTP/3 connections that did not start", 20, 10, "[info] [h3] the client's HTTP/3 connection did not start: {e}");
return;
}
};
match carrier.service.clone() {
Behind::Quic(connection) => {
let (mut driver, sender) = match h3::client::builder()
.send_grease(false)
.build(h3_quinn::Connection::new(connection))
.await
{
Ok(pair) => pair,
Err(e) => {
crate::diag_throttled!("[warn] [h3] connections to a service that did not speak HTTP/3", 1, 30, "[warn] [h3] {} does not speak HTTP/3: {e}", carrier.upstream);
carrier.peer.close(0u32.into(), b"the service did not speak HTTP/3");
return;
}
};
// The client half of h3 needs something to run its connection: control
// streams, settings, and the QPACK streams underneath every header block.
// Without this task nothing upstream ever completes.
let driving = tokio::spawn(async move { driver.wait_idle().await.to_string() });
accept_loop(carrier, &mut server, sender, Some(driving)).await;
}
// A service that speaks HTTP/1.1 has no connection to run and nothing to keep
// alive between exchanges: each one dials its own, so there is no driver here
// and nothing for the loop to watch but the client.
Behind::Http1(upstream) => accept_loop(carrier, &mut server, upstream, None).await,
}
}
/// Take requests until the client is done, and let each one run on its own.
async fn accept_loop<O>(
carrier: &Carrier,
server: &mut h3::server::Connection<h3_quinn::Connection, Bytes>,
sender: O,
driving: Option<tokio::task::JoinHandle<String>>,
) where
O: Outbound + Clone + Send + 'static,
{
// The requests already in flight, kept rather than detached. A client winding the
// connection down — a GOAWAY, or simply no more requests — leaves `accept()`
// answering `None` while answers are still owed, and walking away there would cut
// off the last exchange of every well-behaved client.
let mut inflight = tokio::task::JoinSet::new();
let mut driving = driving;
loop {
let accepted = tokio::select! {
accepted = server.accept() => accepted,
// The service ended its side. Waiting for the client to notice would mean
// waiting out the idle timeout with a connection that has nowhere to go.
// With an HTTP/1.1 service there is no such side to watch — its connections
// live and die with one exchange — so this arm simply never fires.
reason = async {
match driving.as_mut() {
Some(driver) => driver.await.ok(),
None => std::future::pending().await,
}
} => {
if let Some(reason) = reason {
crate::diag_throttled!("[info] [h3] connections the service ended", 20, 10, "[info] [h3] the service ended the connection: {reason}");
}
carrier.peer.close(0u32.into(), b"the service closed the connection");
break;
}
};
match accepted {
Ok(Some(resolver)) => {
let carrier = carrier.clone();
let mut sender = sender.clone();
inflight.spawn(async move {
let connection = carrier.open();
let outcome = exchange(&carrier, connection, resolver, &mut sender).await;
// Whatever happened, the filter's state for this request goes — a
// refused request and a completed one both end it.
carrier.chain.current().connection_closed(connection);
if let Err(e) = outcome {
crate::diag_throttled!("[info] [h3] requests that ended with an error", 20, 10, "[info] [h3] request ended: {e}");
}
});
}
// The client is done with the connection, which is the ordinary way out.
Ok(None) => break,
Err(e) => {
crate::diag_throttled!("[info] [h3] client connections that ended with an error", 20, 10, "[info] [h3] the client's connection ended: {e}");
break;
}
}
}
// Nothing here is unbounded: each exchange rides on connections with an idle
// timeout, and every filter it consults runs under the chain's own deadline.
while inflight.join_next().await.is_some() {}
if let Some(driving) = driving {
driving.abort();
}
}
/// One request and its answer.
///
/// The two directions are carried **at the same time**, not one after the other, which
/// is the same shape the TCP pumps have and for the same reason: a service is allowed to
/// answer before it has read the whole request — a refusal, a redirect, a `413` — and a
/// proxy that insists on finishing the upload first turns that into a stall neither end
/// can explain.
///
/// Generic over where it is going, and over nothing else: what the chain is shown is the
/// same rendering whatever answers, which is the promise the whole module makes.
async fn exchange<O: Outbound>(
carrier: &Carrier,
connection: ConnectionId,
resolver: h3::server::RequestResolver<h3_quinn::Connection, Bytes>,
sender: &mut O,
) -> io::Result<()> {
let (request, mut from_client) = resolver.resolve_request().await.map_err(io::Error::other)?;
carrier.spoken.store(true, Ordering::Relaxed);
// One authority, or none: the chain is shown one `Host` and the service is handed the
// request as it came, so two that disagree would be a filter reading one host while
// the service answers for another. Malformed, and the stream is stopped as such.
if let Some(why) = conflicting_authority(&request) {
crate::diag_throttled!("[warn] [h3] requests with two authorities", 5, 30,
"[warn] [h3] {}: {why}, so the filters would be shown one host while the \
service routes on the other. The stream is stopped as malformed.",
carrier.client
);
from_client.stop_stream(h3::error::Code::H3_MESSAGE_ERROR);
return Ok(());
}
// Malformed as well (RFC 9114 §4.2), and not refused by the h3 crate: see
// `connection_specific`. HTTP/2 needs no such line, because h2 refuses them itself.
if let Some(why) = connection_specific(request.headers()) {
crate::diag_throttled!("[warn] [h3] requests carrying connection-specific headers", 5, 30,
"[warn] [h3] {}: {why}. The stream is stopped as malformed.",
carrier.client
);
from_client.stop_stream(h3::error::Code::H3_MESSAGE_ERROR);
return Ok(());
}
// A request that carries no body must not be shown `transfer-encoding: chunked`,
// which it never had — a filter looking for that header is usually looking for
// smuggling, and one invented by the proxy in front of it is the worst possible
// answer. So the framing is settled before the head is rendered, and settled from
// as little as will do it.
let (framing, first, early) = head_framing(&mut from_client, request.headers())
.await
.map_err(io::Error::other)?;
// One reconstruction for the exchange, not one per direction: a request and its
// response are one conversation, which is what they would have been on TCP.
let tap = carrier.tap();
let rendered = rendered(carrier);
// This direction's filter state, and the head as the chain will see it.
let mut sessions = ChainSessions::new(connection);
let mut view = render_request(&request, &framing);
if let Some(chunk) = &first {
push_body(&mut view, chunk, &framing);
}
// A message that ended with a trailer section and no body is rendered whole here,
// terminator and trailers included: there is no later chunk to hang them off, and a
// trailer section shown to nobody is a piece of the message that travelled unread.
if let Some(trailers) = early.as_ref() {
close_body(&mut view, &framing, trailers.as_ref());
}
if !judge(&rendered, Direction::ClientToServer, &view, &mut sessions, tap.as_ref()).await {
carrier.refuse();
return Ok(());
}
// Only now does the service hear about it, in whatever version it speaks.
let (parts, _) = request.into_parts();
// Whether this head is the whole message. On HTTP/3 the request stream says so the
// way HTTP/2 does, and the flag only matters to an upstream that has to state it on
// the head — but it is computed from the same three answers wherever it is asked.
let ends_it = first.is_none() && !matches!(early, Some(Some(_))) && from_client.ended();
let (mut to_service, from_service) = sender
.open(http::Request::from_parts(parts, ()), ends_it)
.await
.map_err(io::Error::other)?;
if let Some(chunk) = first {
to_service.data(chunk).await.map_err(io::Error::other)?;
}
let (to_client, from_client) = from_client.split();
let (asked, answered) = tokio::join!(
carry_request(carrier, from_client, to_service, framing, sessions, tap.clone(), early),
// Its own sessions, the same id: both halves of one exchange are one connection
// to a filter that keeps state, and each direction keeps its own position in it.
carry_response(carrier, connection, from_service, to_client, tap.clone()),
);
if let Some(tap) = tap {
tap.closed();
}
asked?;
answered
}
/// The rest of the request body, judged chunk by chunk and forwarded.
#[allow(clippy::too_many_arguments)]
async fn carry_request<B: OutboundBody>(
carrier: &Carrier,
mut from_client: h3::server::RequestStream<
<h3_quinn::BidiStream<Bytes> as h3::quic::BidiStream<Bytes>>::RecvStream,
Bytes,
>,
mut to_service: B,
framing: Framing,
mut sessions: ChainSessions,
tap: Option<Arc<Tap>>,
early: Option<Option<http::HeaderMap>>,
) -> io::Result<()>
where
B::Error: std::error::Error + Send + Sync + 'static,
{
let rendered = rendered(carrier);
let trailers = match early {
// The body was over before the head was rendered, so both were shown to the chain
// together. Reading either again would be reading past the end of the stream.
Some(trailers) => trailers,
None => {
// A pattern split across two DATA frames is still a pattern, so the session
// carries on from the head.
while let Some(chunk) = from_client.data().await.map_err(io::Error::other)? {
let mut view = Vec::new();
push_body(&mut view, &chunk, &framing);
if !judge(&rendered, Direction::ClientToServer, &view, &mut sessions, tap.as_ref())
.await
{
carrier.refuse();
return Ok(());
}
to_service.data(chunk).await.map_err(io::Error::other)?;
}
let trailers = from_client.trailers().await.map_err(io::Error::other)?;
let mut tail = Vec::new();
close_body(&mut tail, &framing, trailers.as_ref());
if !tail.is_empty()
&& !judge(&rendered, Direction::ClientToServer, &tail, &mut sessions, tap.as_ref())
.await
{
carrier.refuse();
return Ok(());
}
carried(&rendered, framing, trailers)
}
};
if let Some(trailers) = trailers {
to_service
.trailers(trailers)
.await
.map_err(io::Error::other)?;
}
to_service.finish().await.map_err(io::Error::other)
}
/// The answer, rendered and judged the same way.
async fn carry_response<A: Answer>(
carrier: &Carrier,
connection: ConnectionId,
mut from_service: A,
mut to_client: h3::server::RequestStream<
<h3_quinn::BidiStream<Bytes> as h3::quic::BidiStream<Bytes>>::SendStream,
Bytes,
>,
tap: Option<Arc<Tap>>,
) -> io::Result<()>
where
<A as Incoming>::Error: std::error::Error + Send + Sync + 'static,
{
let rendered = rendered(carrier);
let response = from_service.response().await.map_err(io::Error::other)?;
let mut sessions = ChainSessions::new(connection);
// The answer's framing is settled the same way the request's was, and for the second
// half of the same reason: a service that sends its head and then waits — a streaming
// RPC, a long poll — must not have that head held here, or the client is waiting for
// something that is waiting for the client.
let (framing, first, early) = head_framing(&mut from_service, response.headers())
.await
.map_err(io::Error::other)?;
let mut view = render_response(&response, &framing);
if let Some(chunk) = &first {
push_body(&mut view, chunk, &framing);
}
if let Some(trailers) = early.as_ref() {
close_body(&mut view, &framing, trailers.as_ref());
}
if !judge(&rendered, Direction::ServerToClient, &view, &mut sessions, tap.as_ref()).await {
carrier.refuse();
return Ok(());
}
let (mut parts, _) = response.into_parts();
parts.version = http::Version::HTTP_3;
to_client
.send_response(http::Response::from_parts(parts, ()))
.await
.map_err(io::Error::other)?;
if let Some(chunk) = first {
to_client.send_data(chunk).await.map_err(io::Error::other)?;
}
let trailers = match early {
Some(trailers) => trailers,
None => {
while let Some(chunk) = from_service.data().await.map_err(io::Error::other)? {
let mut view = Vec::new();
push_body(&mut view, &chunk, &framing);
if !judge(&rendered, Direction::ServerToClient, &view, &mut sessions, tap.as_ref())
.await
{
carrier.refuse();
return Ok(());
}
to_client.send_data(chunk).await.map_err(io::Error::other)?;
}
let trailers = from_service.trailers().await.map_err(io::Error::other)?;
let mut tail = Vec::new();
close_body(&mut tail, &framing, trailers.as_ref());
if !tail.is_empty()
&& !judge(&rendered, Direction::ServerToClient, &tail, &mut sessions, tap.as_ref())
.await
{
carrier.refuse();
return Ok(());
}
carried(&rendered, framing, trailers)
}
};
if let Some(trailers) = trailers {
to_client
.send_trailers(trailers)
.await
.map_err(io::Error::other)?;
}
to_client.finish().await.map_err(io::Error::other)
}
/// One DATA frame's bytes, however many pieces the parser had them in.
fn collect(mut buf: impl Buf) -> Bytes {
let len = buf.remaining();
buf.copy_to_bytes(len)
}

523
backend/proxysrc/src/http1.rs Normal file
View File

@ -0,0 +1,523 @@
//! The HTTP/1.1 view every terminated HTTP protocol is shown to the chain as.
//!
//! A filter written against `HttpRequest` is a filter written once. The same file is
//! supposed to work on every layer, and the library decides when to call it from what its
//! parameters are annotated with — not from what the service happens to speak. HTTP/2
//! puts the method, the path and the headers in an HPACK-compressed HEADERS frame and
//! HTTP/3 in a QPACK one, so a filter handed the bytes of either would be handed a
//! compression format, and every pattern written for the HTTP/1.1 service beside it would
//! silently stop matching.
//!
//! So each exchange is rendered as the HTTP/1.1 message it would have been, and that is
//! what the chain is shown. `HttpRequest`, `HttpResponse` and a hyperscan pattern all mean
//! the same thing on every version, and nothing had to learn a new protocol. The same
//! bargain the capture interface makes: **the bytes are real and the framing is
//! reconstructed**. Concretely, and this belongs wherever this is documented —
//!
//! * the request line says `HTTP/1.1`, because there is no other version an HTTP/1 parser
//! will read, and the version on the wire was not it;
//! * a body with no `content-length` is shown chunked, which is what that message is in
//! HTTP/1.1 — and the head is held until it is known whether a body is coming, so that a
//! request without one is not shown a framing header it never had. Held *briefly*: a
//! declared length settles the question outright and waits for nothing, a message with
//! no body has already ended its stream, and what is left is a sender holding the stream
//! open, which [`HEAD_HOLD`] ends rather than deadlocking an exchange the service is
//! supposed to speak first in;
//! * a **trailer section is always shown to the chain**, which is what decides the framing
//! of a message that has one and no body: in HTTP/1.1 a trailer section belongs to a
//! chunked message and nowhere else. The one message that cannot carry one in the
//! rendering is one that declared its length, and there the trailers are dropped rather
//! than forwarded past a filter that never saw them;
//! * `host` is rendered from the `:authority` pseudo-header, which is where both HTTP/2
//! and HTTP/3 put it;
//! * hop-by-hop headers do not appear: both versions forbid them, a request carrying one
//! is refused as malformed ([`connection_specific`]), and an answer from a service
//! reached over HTTP/1.1 has its own taken out before it goes on
//! ([`drop_connection_headers`]).
//!
//! A message whose declared `content-length` disagrees with the body it then sends is
//! malformed in HTTP/2 and HTTP/3 as well, and nothing here tries to repair it: the frames
//! are forwarded as they came, and the rendering carries the same disagreement — which an
//! HTTP/1 parser reads as a message that does not end where it said it would. That is what
//! the library's invalid-encoding action is for, and it refuses by default, which is the
//! same answer the same traffic gets over HTTP/1.1.
//!
//! What it is *not* is a translation the service ever sees: upstream, the message goes
//! back out in the version it arrived in, re-encoded by the same library that parsed it.
//!
//! **The capture interface is shown this same rendering**, one TCP conversation per
//! exchange, which is the only way `firegex0` can keep the promise it makes everywhere
//! else: what is written there is what the filters saw. It is not what left this process
//! towards the service, so the rendering is a view in the capture exactly as it is a view
//! to a filter, and the two cannot disagree because they are the same bytes.
//!
//! # Why this module exists at all
//!
//! It was inside `h3.rs`, and HTTP/2 wanted every line of it. Copying it would have been
//! two renderings that can come to disagree — the same argument the [`Incoming`] trait
//! already makes *within* one protocol, one level up. The day they diverge the symptom is
//! a pattern that matches over HTTP/3 and not over HTTP/2, silently, which is the failure
//! this whole engine is arranged against. So the rendering lives here once, and
//! [`crate::h2`] and [`crate::h3`] are what is left of each protocol after it is taken
//! out: an adapter to somebody else's parser.
use std::future::Future;
use std::net::SocketAddr;
use std::sync::atomic::Ordering;
use std::sync::Arc;
use std::time::Duration;
use bytes::Bytes;
use crate::capture::Tap;
use crate::filter::{ChainHandle, ChainSessions, Direction, Verdict};
use crate::proxy::ProxyStats;
/// How the body of one message is framed in the view the filters are shown.
#[derive(Clone, Copy, PartialEq, Eq)]
pub(crate) enum Framing {
/// The message said how long it is, so the body is shown as it arrives.
Sized,
/// It did not, so it is shown chunked — the HTTP/1.1 spelling of that.
Chunked,
/// There is no body at all, and nothing is added to say so.
Empty,
}
/// How long the head is held while it is still unknown whether a body follows.
///
/// Only ever paid by a message that declared no length *and* has not yet said whether it
/// has a body — which is to say, one whose sender is holding the stream open. A message
/// with a `content-length` is rendered from its head alone and waits for nothing; one
/// without a body has already finished its stream, so the answer is there before this
/// timer starts.
///
/// It exists because holding the head indefinitely deadlocks an exchange the service is
/// supposed to speak first in: a bidirectional RPC — gRPC's among them — sends a head and
/// then waits to be answered, and a proxy that will not forward the head until a body
/// arrives is waiting for something that is waiting for it. When it expires the message is
/// rendered chunked, which is what a message of undeclared length *is* in HTTP/1.1, so
/// nothing is invented that the sender did not already imply.
pub(crate) const HEAD_HOLD: Duration = Duration::from_millis(100);
/// Everything the rendering needs about who is being carried, and no more.
///
/// Neither protocol's connection type appears here on purpose: what this module does is
/// the same on both, and a parameter that differed would be the seam along which the two
/// renderings could start to drift again.
pub(crate) struct Rendered<'a> {
pub(crate) chain: &'a ChainHandle,
pub(crate) stats: &'a Arc<ProxyStats>,
pub(crate) client: SocketAddr,
/// `h2` or `h3`, for the one log line this module writes.
pub(crate) layer: &'static str,
}
/// The receiving half of one direction of an exchange.
///
/// Four types end up here — a server and a client request stream on each of two crates —
/// and nothing unifies them, while every question this module asks a message is the same
/// on all of them. One trait rather than the same framing decision written four times: the
/// copies are renderings that can come to disagree, and the whole point of the rendering
/// is that a filter sees one thing.
pub(crate) trait Incoming {
/// Whatever the underlying crate says went wrong. Never inspected here — the caller
/// owns the protocol and its errors, this module owns the framing.
type Error;
fn data(&mut self) -> impl Future<Output = Result<Option<Bytes>, Self::Error>> + Send;
fn trailers(
&mut self,
) -> impl Future<Output = Result<Option<http::HeaderMap>, Self::Error>> + Send;
/// Give back the window one piece took, once it has been forwarded and not before.
///
/// HTTP/2 is the only protocol here that asks: it hands received bytes over and waits
/// for them to be released, so releasing on *read* would mean no backpressure and
/// this process buying memory for whichever side is faster, while never releasing
/// stalls the stream after its first window. Everything else frames its own flow and
/// answers with the default.
fn release(&mut self, taken: usize) -> Result<(), Self::Error> {
let _ = taken;
Ok(())
}
/// Whether the sender has already said there is nothing more coming.
///
/// Only meaningful beside a head that produced no body and no trailer section: the
/// three together are what decide whether the head **is** the message, which HTTP/2
/// has to state on the head itself (see [`Outbound::open`]). A protocol whose sender
/// cannot say it in advance answers `false`, which is always safe — the message is
/// then framed the ordinary way and ends when its body does.
fn ended(&self) -> bool {
false
}
}
/// Where a rendered message is sent once the chain has agreed to it.
///
/// The mirror of [`Incoming`], and it exists for the same reason one level further out:
/// what leaves this engine towards the service used to be, always, the protocol the
/// client arrived in — so the sending half could be written against one crate's concrete
/// types. It is a choice now (`FGEX_PROXY_UPSTREAM`), because a service that speaks
/// HTTP/1.1 can be put behind an HTTP/3 edge, and the alternative to this trait was a
/// second copy of the orchestration in [`crate::h3`] — request in one direction, answer
/// in the other, framing decided on both — which is the drift this module exists to
/// prevent, one level down.
///
/// Note what is *not* here: nothing about rendering. The bytes a filter sees are built by
/// this module whatever the service turns out to speak, so an implementation of this
/// trait only has to know how to put a head, a body and a trailer section on the wire.
pub(crate) trait Outbound: Send {
type Error: std::error::Error + Send + Sync + 'static;
/// Where this exchange's body is written.
type Body: OutboundBody<Error = Self::Error> + Send;
/// Where its answer comes back.
type Answer: Answer<Error = Self::Error> + Send;
/// Send the head, and take the two halves the rest of the exchange runs on.
///
/// `ends_it` says the head **is** the whole message. It is a parameter rather than a
/// later `finish()` because not every protocol can say it afterwards: on HTTP/2 the
/// end of a message rides on its last frame, so a head sent open and then closed with
/// an empty DATA frame is a message *with a body* — which is exactly what a
/// status-only gRPC answer is not, and what gRPC clients refuse. HTTP/1.1 frames it
/// from the headers and has nothing to do with the flag but close the body early.
fn open(
&mut self,
request: http::Request<()>,
ends_it: bool,
) -> impl Future<Output = Result<(Self::Body, Self::Answer), Self::Error>> + Send;
}
/// The body of a message on its way to the service.
///
/// Imperative rather than a stream handed over whole, because that is the shape both ends
/// of this already have: the chain judges one piece at a time and forwards what it
/// accepts, so a piece is written when it is allowed and not before.
pub(crate) trait OutboundBody: Send {
type Error;
fn data(&mut self, chunk: Bytes) -> impl Future<Output = Result<(), Self::Error>> + Send;
fn trailers(
&mut self,
trailers: http::HeaderMap,
) -> impl Future<Output = Result<(), Self::Error>> + Send;
/// No more of it is coming. Separate from dropping, because on some protocols the
/// end of a message is a frame and not the absence of one.
fn finish(&mut self) -> impl Future<Output = Result<(), Self::Error>> + Send;
}
/// The service's answer: its head first, then the body every [`Incoming`] gives.
pub(crate) trait Answer: Incoming + Send {
fn response(
&mut self,
) -> impl Future<Output = Result<http::Response<()>, <Self as Incoming>::Error>> + Send;
}
/// What has to be known before a message's head can be rendered, and no more than that.
///
/// Three answers come back together because they are one decision: the framing, the first
/// body chunk if reading it is what settled the question, and — where the body was over
/// before it began — the trailer section, which is already in hand by then and which
/// decides the framing too. A message that ends with trailers and no body *is* a chunked
/// message in HTTP/1.1, and that is the only place a trailer section can be rendered, so
/// it has to be known here rather than discovered after the head has gone out.
///
/// `Some(trailers)` in the third slot means the body stream is finished and its trailer
/// section has already been taken; the caller must not read either again.
#[allow(clippy::type_complexity)]
pub(crate) async fn head_framing<B: Incoming + Send>(
body: &mut B,
headers: &http::HeaderMap,
) -> Result<(Framing, Option<Bytes>, Option<Option<http::HeaderMap>>), B::Error> {
// A declared length frames the body by itself, so there is nothing to wait for and
// nothing a later frame could change. This is also what keeps an upload that the
// service answers early — a `413`, a redirect, a refusal — moving in both directions.
if headers.contains_key(http::header::CONTENT_LENGTH) {
return Ok((Framing::Sized, None, None));
}
// Both crates keep a half-read frame's state in the stream rather than in the future,
// so letting the deadline drop it loses nothing.
match tokio::time::timeout(HEAD_HOLD, body.data()).await {
Ok(first) => match first? {
Some(chunk) => Ok((Framing::Chunked, Some(chunk), None)),
None => {
// The body is over. Whatever ended it — the end of the stream or a
// trailer section — has already been read into the stream, so this
// resolves without waiting for anything.
let trailers = body.trailers().await?;
let framing = if trailers.is_some() {
Framing::Chunked
} else {
Framing::Empty
};
Ok((framing, None, Some(trailers)))
}
},
Err(_) => Ok((Framing::Chunked, None, None)),
}
}
/// The trailer section this engine is willing to forward.
///
/// A trailer section the chain was never shown is a piece of the message that travelled
/// unfiltered, which is the one thing nothing else here does. Only one framing leaves
/// nowhere to render it: a message that declared its length, whose HTTP/1.1 view ends
/// where the header said it would — anything appended after that is not part of the
/// message any parser reading the rendering would see. So it is dropped, which is what an
/// intermediary is allowed to do with a trailer section it cannot carry, and said out loud
/// rather than done quietly.
pub(crate) fn carried(
rendered: &Rendered<'_>,
framing: Framing,
trailers: Option<http::HeaderMap>,
) -> Option<http::HeaderMap> {
if framing == Framing::Sized && trailers.is_some() {
crate::diag_throttled!("[warn] [http] trailer sections dropped from a message of declared length", 5, 30,
"[warn] [{}] {}: a trailer section arrived on a message that declared its \
length. The HTTP/1.1 the filters are shown has nowhere to put one, so it is \
dropped rather than forwarded unread.",
rendered.layer, rendered.client,
);
return None;
}
trailers
}
/// Ask the chain about one piece of the rendered message.
pub(crate) async fn judge(
rendered: &Rendered<'_>,
direction: Direction,
view: &[u8],
sessions: &mut ChainSessions,
tap: Option<&Arc<Tap>>,
) -> bool {
// Re-read every time, so a ruleset pushed mid-request takes effect without anybody
// losing their connection — the same rule the byte pumps follow.
match rendered
.chain
.current()
.run(direction, view, sessions)
.await
{
Verdict::Accept => {
// Written on the way past, and only once the chain has agreed: what a capture
// shows has to be what was let through, or it is a record of a decision that
// was not taken. A refused piece leaves the conversation ending where the
// traffic did, which is the truth about it.
if let Some(tap) = tap {
tap.wrote(direction == Direction::ClientToServer, view);
}
true
}
Verdict::Reject(_) => {
rendered
.stats
.closed_by_filter
.fetch_add(1, Ordering::Relaxed);
false
}
}
}
/// Why a request names its authority twice and disagrees with itself, or `None`.
///
/// HTTP/2 and HTTP/3 carry the authority in `:authority`, and a client may send a `host`
/// header beside it. The rendering shows the chain one `Host` line, while the request goes
/// on to the service with both — and a service reading HTTP/2 or HTTP/3 routes on
/// `:authority`, which RFC 9113 §8.3.1 says wins. So a request whose two disagree would
/// show a filter one host and deliver itself to another: `:authority: admin.internal` with
/// `host: public.example` walks past every rule written against the admin host. The RFC
/// calls such a request malformed, and so does this. A second `host` header is the same
/// question asked twice, and is answered the same way.
pub(crate) fn conflicting_authority(request: &http::Request<()>) -> Option<String> {
let mut hosts = request.headers().get_all(http::header::HOST).iter();
let host = hosts.next();
if hosts.next().is_some() {
return Some("the request carries more than one host header".to_string());
}
match (request.uri().authority(), host) {
(Some(authority), Some(host))
if !host.as_bytes().eq_ignore_ascii_case(authority.as_str().as_bytes()) =>
{
Some(format!(
":authority says {authority} and the host header says {}",
String::from_utf8_lossy(host.as_bytes())
))
}
_ => None,
}
}
/// The most a client's head may take — the header section of one request, decoded.
///
/// Neither crate bounds it usefully by default: h2 allows 16 MiB, and h3 sets no limit at
/// all, so an HTTP/3 request could keep one header block growing for as long as flow
/// control kept crediting it, a hundred streams to a connection. A mebibyte is the budget
/// the filter library holds a stream to by default (`FGEX_STREAM_MAX_SIZE`), well past
/// what any HTTP/1.1 server accepts in a head, and far enough from the usual range that
/// a large cookie or a heavy set of gRPC metadata is never what it refuses.
pub(crate) const MAX_HEAD_BYTES: u32 = 1024 * 1024;
/// Headers about one connection rather than about the message it carries.
///
/// HTTP/2 and HTTP/3 have no such thing (RFC 9113 §8.2.2, RFC 9114 §4.2): the framing and
/// the connection's lifetime are the protocol's own business, so a message carrying one
/// is malformed. `te` is the exception, and only as `trailers`.
const CONNECTION_SPECIFIC: [&str; 6] =
["connection", "keep-alive", "proxy-connection", "transfer-encoding", "upgrade", "te"];
/// Why a request that arrived over HTTP/2 or HTTP/3 carries a header only a connection
/// can, or `None`.
///
/// The h2 crate refuses these itself; h3 does not, so an HTTP/3 client could hand a
/// service reached over HTTP/1.1 a `connection` naming headers for it to drop, or a `te`
/// the upstream then extends into a transfer coding the service is told to undo — none of
/// which a filter shown the rendering would recognise as the request the service got.
pub(crate) fn connection_specific(headers: &http::HeaderMap) -> Option<String> {
for name in CONNECTION_SPECIFIC {
for value in headers.get_all(name) {
if name == "te" && value.as_bytes().eq_ignore_ascii_case(b"trailers") {
continue;
}
return Some(format!(
"the request carries `{name}`, which belongs to a connection rather than to \
a message"
));
}
}
None
}
/// Take out of an HTTP/1.1 answer what belongs to that connection, before it goes on in
/// a version that forbids it.
///
/// Every HTTP/1.1 service says something about its connection — a body of unknown length
/// comes `transfer-encoding: chunked`, a connection it will keep open says `keep-alive` —
/// and those were passed on as they came: HTTP/2 refused to send the response at all, and
/// an HTTP/3 client is required to reject one. What `connection` itself names goes too,
/// since that is what the header is for (RFC 9110 §7.6.1).
pub(crate) fn drop_connection_headers(headers: &mut http::HeaderMap) {
let named: Vec<http::HeaderName> = headers
.get_all(http::header::CONNECTION)
.iter()
.filter_map(|value| value.to_str().ok())
.flat_map(|value| value.split(','))
.filter_map(|name| http::HeaderName::from_bytes(name.trim().as_bytes()).ok())
.collect();
for name in named {
headers.remove(name);
}
for name in CONNECTION_SPECIFIC {
headers.remove(name);
}
}
/// The request, as the HTTP/1.1 it would have been.
///
/// Called only for a request that has passed [`conflicting_authority`], so a `host` header
/// and `:authority`, where both are present, say the same thing.
pub(crate) fn render_request(request: &http::Request<()>, framing: &Framing) -> Vec<u8> {
let target = request
.uri()
.path_and_query()
.map(|p| p.as_str().to_string())
.unwrap_or_else(|| "/".to_string());
let mut view = format!("{} {} HTTP/1.1\r\n", request.method(), target).into_bytes();
// Both versions carry it as `:authority`, and an HTTP/1.1 parser wants a `Host` line.
// A request that also sent `host` outright keeps its own, which is the same value.
if !request.headers().contains_key(http::header::HOST) {
if let Some(authority) = request.uri().authority() {
view.extend_from_slice(format!("host: {authority}\r\n").as_bytes());
}
}
render_headers(&mut view, request.headers(), framing);
view
}
/// And the response.
pub(crate) fn render_response(response: &http::Response<()>, framing: &Framing) -> Vec<u8> {
let status = response.status();
let mut view = format!(
"HTTP/1.1 {} {}\r\n",
status.as_u16(),
status.canonical_reason().unwrap_or("")
)
.into_bytes();
render_headers(&mut view, response.headers(), framing);
view
}
fn render_headers(view: &mut Vec<u8>, headers: &http::HeaderMap, framing: &Framing) {
for (name, value) in headers {
view.extend_from_slice(name.as_str().as_bytes());
view.extend_from_slice(b": ");
view.extend_from_slice(value.as_bytes());
view.extend_from_slice(b"\r\n");
}
if matches!(framing, Framing::Chunked) {
view.extend_from_slice(b"transfer-encoding: chunked\r\n");
}
view.extend_from_slice(b"\r\n");
}
/// One body frame, framed the way the head said it would be.
pub(crate) fn push_body(view: &mut Vec<u8>, chunk: &[u8], framing: &Framing) {
match framing {
Framing::Chunked => {
view.extend_from_slice(format!("{:x}\r\n", chunk.len()).as_bytes());
view.extend_from_slice(chunk);
view.extend_from_slice(b"\r\n");
}
_ => view.extend_from_slice(chunk),
}
}
/// The end of the body, where the framing has one to show.
pub(crate) fn close_body(view: &mut Vec<u8>, framing: &Framing, trailers: Option<&http::HeaderMap>) {
if !matches!(framing, Framing::Chunked) {
return;
}
view.extend_from_slice(b"0\r\n");
if let Some(trailers) = trailers {
for (name, value) in trailers {
view.extend_from_slice(name.as_str().as_bytes());
view.extend_from_slice(b": ");
view.extend_from_slice(value.as_bytes());
view.extend_from_slice(b"\r\n");
}
}
view.extend_from_slice(b"\r\n");
}
#[cfg(test)]
mod tests {
use super::conflicting_authority;
fn request(authority: Option<&str>, hosts: &[&str]) -> http::Request<()> {
let uri = match authority {
Some(authority) => format!("https://{authority}/path"),
None => "/path".to_string(),
};
let mut builder = http::Request::builder().uri(uri);
for host in hosts {
builder = builder.header(http::header::HOST, *host);
}
builder.body(()).unwrap()
}
#[test]
fn one_authority_said_once_or_twice_alike_is_fine() {
assert_eq!(conflicting_authority(&request(Some("example.com"), &[])), None);
assert_eq!(conflicting_authority(&request(None, &["example.com"])), None);
assert_eq!(conflicting_authority(&request(Some("example.com"), &["example.com"])), None);
assert_eq!(conflicting_authority(&request(Some("Example.com"), &["example.COM"])), None);
}
#[test]
fn two_authorities_that_disagree_are_malformed() {
assert!(conflicting_authority(&request(Some("admin.internal"), &["public.example"])).is_some());
assert!(conflicting_authority(&request(None, &["a.example", "b.example"])).is_some());
}
}

578
backend/proxysrc/src/hyperscan.rs Normal file
View File

@ -0,0 +1,578 @@
//! Bindings to the hyperscan/vectorscan C API.
//!
//! Firegex already matches with hyperscan on the NFQUEUE side (`binsrc/regex/`),
//! and the proxy transport has to match with the *same* engine — not a similar one.
//! Two engines would mean a pattern that blocks on one transport and not the other,
//! and an in-app regex debugger that tests neither. So this is a binding rather than
//! a second implementation, and the flags below deliberately mirror
//! `regex_rules.cpp`: `SINGLEMATCH | ALLOWEMPTY`, plus `CASELESS` when asked.
//!
//! Stream mode is the other reason. A proxy sees a byte stream in arbitrary chunks,
//! and a pattern may straddle two of them. Rescanning an overlap window bounds the
//! cost but also bounds what can match. `hs_scan_stream` carries the automaton state
//! across chunks instead, so a match is found wherever it falls, at no extra scanning.
use std::ffi::{c_char, c_int, c_uint, c_void, CStr, CString};
use std::ptr;
use std::sync::Arc;
#[allow(non_camel_case_types)]
type hs_database_t = c_void;
#[allow(non_camel_case_types)]
type hs_scratch_t = c_void;
#[allow(non_camel_case_types)]
type hs_stream_t = c_void;
#[repr(C)]
struct hs_compile_error_t {
message: *mut c_char,
expression: c_int,
}
type MatchEventHandler = extern "C" fn(
id: c_uint,
from: u64,
to: u64,
flags: c_uint,
context: *mut c_void,
) -> c_int;
const HS_SUCCESS: c_int = 0;
const HS_SCAN_TERMINATED: c_int = -3;
const HS_FLAG_CASELESS: c_uint = 1;
#[allow(dead_code)]
const HS_FLAG_SINGLEMATCH: c_uint = 8;
const HS_FLAG_ALLOWEMPTY: c_uint = 16;
const HS_FLAG_SOM_LEFTMOST: c_uint = 256;
const HS_MODE_BLOCK: c_uint = 1;
const HS_MODE_STREAM: c_uint = 2;
#[link(name = "hs")]
extern "C" {
fn hs_compile_multi(
expressions: *const *const c_char,
flags: *const c_uint,
ids: *const c_uint,
elements: c_uint,
mode: c_uint,
platform: *const c_void,
db: *mut *mut hs_database_t,
error: *mut *mut hs_compile_error_t,
) -> c_int;
fn hs_free_database(db: *mut hs_database_t) -> c_int;
fn hs_free_compile_error(error: *mut hs_compile_error_t) -> c_int;
fn hs_alloc_scratch(db: *const hs_database_t, scratch: *mut *mut hs_scratch_t) -> c_int;
fn hs_free_scratch(scratch: *mut hs_scratch_t) -> c_int;
fn hs_open_stream(
db: *const hs_database_t,
flags: c_uint,
stream: *mut *mut hs_stream_t,
) -> c_int;
fn hs_scan_stream(
id: *mut hs_stream_t,
data: *const c_char,
length: c_uint,
flags: c_uint,
scratch: *mut hs_scratch_t,
on_event: Option<MatchEventHandler>,
ctxt: *mut c_void,
) -> c_int;
fn hs_close_stream(
id: *mut hs_stream_t,
scratch: *mut hs_scratch_t,
on_event: Option<MatchEventHandler>,
ctxt: *mut c_void,
) -> c_int;
fn hs_reset_stream(
id: *mut hs_stream_t,
flags: c_uint,
scratch: *mut hs_scratch_t,
on_event: Option<MatchEventHandler>,
ctxt: *mut c_void,
) -> c_int;
fn hs_scan(
db: *const hs_database_t,
data: *const c_char,
length: c_uint,
flags: c_uint,
scratch: *mut hs_scratch_t,
on_event: Option<MatchEventHandler>,
ctxt: *mut c_void,
) -> c_int;
}
/// How a compiled database will be used. The mode is baked into the database, so a
/// database compiled for one cannot be scanned with the other.
#[derive(Clone, Copy, PartialEq, Eq, Debug)]
pub enum Mode {
/// Live traffic: state carries across chunks, first match per pattern wins.
Stream,
/// One self-contained buffer, every match reported with its start offset.
/// This is what the in-app debugger scans with.
Block,
}
/// One pattern as the operator wrote it.
pub struct Pattern {
pub expr: String,
pub case_sensitive: bool,
}
/// A compiled set of patterns.
///
/// Immutable once built, which is what lets every connection share one `Arc` of it
/// instead of recompiling per connection — and what makes the `Sync` below sound.
pub struct Database {
db: *mut hs_database_t,
mode: Mode,
count: usize,
}
// hyperscan states that a compiled database is read-only at scan time and may be
// shared by any number of concurrent scans, each with its own scratch. The scratch
// is the mutable part, and it is owned per-scanner below.
unsafe impl Send for Database {}
unsafe impl Sync for Database {}
impl Drop for Database {
fn drop(&mut self) {
if !self.db.is_null() {
unsafe { hs_free_database(self.db) };
}
}
}
fn take_compile_error(err: *mut hs_compile_error_t) -> String {
if err.is_null() {
return "hyperscan failed to compile the pattern".to_string();
}
let msg = unsafe {
if (*err).message.is_null() {
"hyperscan failed to compile the pattern".to_string()
} else {
CStr::from_ptr((*err).message).to_string_lossy().into_owned()
}
};
unsafe { hs_free_compile_error(err) };
msg
}
impl Database {
/// Compile the patterns into one database, ids being their index in `patterns`.
///
/// One database for many patterns is the whole point of hyperscan: the automata
/// are merged, so scanning for fifty patterns costs about what scanning for one
/// costs. Adding a pattern must therefore rebuild, never chain.
pub fn compile(patterns: &[Pattern], mode: Mode) -> Result<Database, String> {
if patterns.is_empty() {
return Err("no patterns to compile".to_string());
}
let owned: Vec<CString> = patterns
.iter()
.map(|p| {
CString::new(p.expr.as_bytes())
.map_err(|_| "a pattern contains a null byte".to_string())
})
.collect::<Result<_, _>>()?;
let ptrs: Vec<*const c_char> = owned.iter().map(|s| s.as_ptr()).collect();
let base = match mode {
Mode::Stream => HS_FLAG_ALLOWEMPTY,
// The debugger wants every match and where it started, so neither
// SINGLEMATCH (one report per pattern) nor the default no-start-offset.
Mode::Block => HS_FLAG_SOM_LEFTMOST | HS_FLAG_ALLOWEMPTY,
};
let flags: Vec<c_uint> = patterns
.iter()
.map(|p| if p.case_sensitive { base } else { base | HS_FLAG_CASELESS })
.collect();
let ids: Vec<c_uint> = (0..patterns.len() as c_uint).collect();
let hs_mode = match mode {
Mode::Stream => HS_MODE_STREAM,
// No SOM horizon flag: those are a streaming-mode bound on how far
// back a start offset may be remembered. Block mode scans one buffer and
// always reports the true start.
Mode::Block => HS_MODE_BLOCK,
};
let mut db: *mut hs_database_t = ptr::null_mut();
let mut err: *mut hs_compile_error_t = ptr::null_mut();
let rc = unsafe {
hs_compile_multi(
ptrs.as_ptr(),
flags.as_ptr(),
ids.as_ptr(),
patterns.len() as c_uint,
hs_mode,
ptr::null(),
&mut db,
&mut err,
)
};
if rc != HS_SUCCESS {
return Err(take_compile_error(err));
}
Ok(Database { db, mode, count: patterns.len() })
}
/// Compile one pattern on its own, purely to find out whether it is valid.
///
/// Worth the extra compile: `hs_compile_multi` reports the first bad pattern and
/// stops, so a ruleset with one typo would otherwise reject every pattern in it
/// with a message about a different one.
pub fn validate(expr: &str, case_sensitive: bool, mode: Mode) -> Result<(), String> {
Database::compile(
&[Pattern { expr: expr.to_string(), case_sensitive }],
mode,
)
.map(|_| ())
}
pub fn len(&self) -> usize {
self.count
}
pub fn mode(&self) -> Mode {
self.mode
}
}
struct Scratch {
scratch: *mut hs_scratch_t,
}
impl Drop for Scratch {
fn drop(&mut self) {
if !self.scratch.is_null() {
unsafe { hs_free_scratch(self.scratch) };
}
}
}
impl Scratch {
fn alloc(db: &Database) -> Result<Scratch, String> {
let mut scratch: *mut hs_scratch_t = ptr::null_mut();
let rc = unsafe { hs_alloc_scratch(db.db, &mut scratch) };
if rc != HS_SUCCESS {
return Err(format!("hyperscan could not allocate scratch space ({rc})"));
}
Ok(Scratch { scratch })
}
}
/// Context handed to the C callback. `id` of the first match, if any.
struct FirstMatch {
hit: Option<u32>,
}
extern "C" fn on_first_match(
id: c_uint,
_from: u64,
_to: u64,
_flags: c_uint,
context: *mut c_void,
) -> c_int {
// SAFETY: `context` is the `&mut FirstMatch` passed to hs_scan_stream, which
// outlives the scan and is not aliased — hyperscan calls this synchronously.
let res = unsafe { &mut *(context as *mut FirstMatch) };
res.hit = Some(id as u32);
-1 // stop scanning: one match is enough to decide a verdict
}
/// A scanner bound to one direction of one connection.
///
/// Owns its scratch rather than borrowing a shared one: scratch is the mutable
/// working memory of a scan and must not be touched by two scans at once. Tying its
/// lifetime to the stream's makes that unrepresentable instead of a rule to remember.
pub struct StreamScanner {
db: Arc<Database>,
stream: *mut hs_stream_t,
scratch: Scratch,
}
// Both pointers are owned exclusively by this struct and never shared; moving the
// whole scanner to another thread moves both together.
unsafe impl Send for StreamScanner {}
impl Drop for StreamScanner {
fn drop(&mut self) {
if !self.stream.is_null() {
unsafe { hs_close_stream(self.stream, self.scratch.scratch, None, ptr::null_mut()) };
}
}
}
impl StreamScanner {
pub fn open(db: Arc<Database>) -> Result<StreamScanner, String> {
if db.mode != Mode::Stream {
return Err("this database was not compiled for stream scanning".to_string());
}
let scratch = Scratch::alloc(&db)?;
let mut stream: *mut hs_stream_t = ptr::null_mut();
let rc = unsafe { hs_open_stream(db.db, 0, &mut stream) };
if rc != HS_SUCCESS {
return Err(format!("hyperscan could not open a stream ({rc})"));
}
Ok(StreamScanner { db, stream, scratch })
}
/// Feed the next chunk. Returns the id of the pattern that matched, if one did.
///
/// The stream keeps its own state, so a caller must never re-feed bytes it has
/// already fed — that would match patterns that never actually appeared.
pub fn scan(&mut self, data: &[u8]) -> Result<Option<u32>, String> {
if data.is_empty() {
return Ok(None);
}
let mut res = FirstMatch { hit: None };
// The C API takes a 32-bit length. Real chunks are far below that, but a
// caller is not obliged to know it, so feed in pieces rather than truncate.
for piece in data.chunks(u32::MAX as usize) {
let rc = unsafe {
hs_scan_stream(
self.stream,
piece.as_ptr() as *const c_char,
piece.len() as c_uint,
0,
self.scratch.scratch,
Some(on_first_match),
&mut res as *mut FirstMatch as *mut c_void,
)
};
if rc == HS_SCAN_TERMINATED {
break;
}
if rc != HS_SUCCESS {
return Err(format!("hyperscan stream scan failed ({rc})"));
}
}
Ok(res.hit)
}
/// Reset the stream to its initial state, clearing any previous match or terminated status.
pub fn reset(&mut self) -> Result<(), String> {
if self.stream.is_null() {
return Ok(());
}
let rc = unsafe {
hs_reset_stream(
self.stream,
0,
self.scratch.scratch,
None,
ptr::null_mut(),
)
};
if rc != HS_SUCCESS {
// If reset failed, close and reopen the stream cleanly.
unsafe { hs_close_stream(self.stream, self.scratch.scratch, None, ptr::null_mut()) };
self.stream = ptr::null_mut();
let reopen_rc = unsafe { hs_open_stream(self.db.db, 0, &mut self.stream) };
if reopen_rc != HS_SUCCESS {
return Err(format!("hyperscan reset stream failed ({rc}/{reopen_rc})"));
}
}
Ok(())
}
pub fn pattern_count(&self) -> usize {
self.db.count
}
}
/// One match found by a block-mode scan.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub struct Match {
pub id: u32,
pub from: u64,
pub to: u64,
}
struct AllMatches {
hits: Vec<Match>,
limit: usize,
}
extern "C" fn on_every_match(
id: c_uint,
from: u64,
to: u64,
_flags: c_uint,
context: *mut c_void,
) -> c_int {
// SAFETY: as above — synchronous, single-threaded, outlives the scan.
let res = unsafe { &mut *(context as *mut AllMatches) };
res.hits.push(Match { id: id as u32, from, to });
// A pathological pattern over a large sample can match at nearly every offset;
// the debugger wants a readable answer, not every one of them.
if res.hits.len() >= res.limit {
-1
} else {
0
}
}
/// Scan one self-contained buffer and report every match with its offsets.
///
/// This is what the debugger runs, so what it shows is what the datapath would see.
pub fn scan_block(db: &Database, data: &[u8], limit: usize) -> Result<Vec<Match>, String> {
if db.mode != Mode::Block {
return Err("this database was not compiled for block scanning".to_string());
}
let scratch = Scratch::alloc(db)?;
scan_block_with(db, scratch.scratch, data, limit)
}
fn scan_block_with(
db: &Database,
scratch: *mut hs_scratch_t,
data: &[u8],
limit: usize,
) -> Result<Vec<Match>, String> {
let mut res = AllMatches { hits: Vec::new(), limit };
let rc = unsafe {
hs_scan(
db.db,
data.as_ptr() as *const c_char,
data.len() as c_uint,
0,
scratch,
Some(on_every_match),
&mut res as *mut AllMatches as *mut c_void,
)
};
if rc != HS_SUCCESS && rc != HS_SCAN_TERMINATED {
return Err(format!("hyperscan block scan failed ({rc})"));
}
Ok(res.hits)
}
#[cfg(test)]
mod tests {
use super::*;
fn stream_db(exprs: &[(&str, bool)]) -> Arc<Database> {
let pats: Vec<Pattern> = exprs
.iter()
.map(|(e, cs)| Pattern { expr: e.to_string(), case_sensitive: *cs })
.collect();
Arc::new(Database::compile(&pats, Mode::Stream).expect("compile"))
}
#[test]
fn matches_a_pattern_in_one_chunk() {
let db = stream_db(&[("needle", true)]);
let mut sc = StreamScanner::open(db).unwrap();
assert_eq!(sc.scan(b"a haystack with a needle in it").unwrap(), Some(0));
}
#[test]
fn reports_which_pattern_matched() {
let db = stream_db(&[("alpha", true), ("bravo", true), ("charlie", true)]);
let mut sc = StreamScanner::open(db).unwrap();
assert_eq!(sc.scan(b"...bravo...").unwrap(), Some(1));
}
/// The reason stream mode exists: a match split across two writes.
#[test]
fn matches_across_a_chunk_boundary() {
let db = stream_db(&[("secret-token", true)]);
let mut sc = StreamScanner::open(db).unwrap();
assert_eq!(sc.scan(b"GET /?k=secr").unwrap(), None);
assert_eq!(sc.scan(b"et-token HTTP/1.1").unwrap(), Some(0));
}
/// ...and it holds however small the pieces are, which no overlap window does.
#[test]
fn matches_across_many_one_byte_chunks() {
let db = stream_db(&[("abcdefghij", true)]);
let mut sc = StreamScanner::open(db).unwrap();
let mut hit = None;
for b in b"xxxabcdefghijxxx" {
if let Some(id) = sc.scan(&[*b]).unwrap() {
hit = Some(id);
}
}
assert_eq!(hit, Some(0));
}
#[test]
fn honours_case_sensitivity() {
let sensitive = stream_db(&[("NeEdLe", true)]);
let mut sc = StreamScanner::open(sensitive).unwrap();
assert_eq!(sc.scan(b"needle").unwrap(), None);
let insensitive = stream_db(&[("NeEdLe", false)]);
let mut sc = StreamScanner::open(insensitive).unwrap();
assert_eq!(sc.scan(b"needle").unwrap(), Some(0));
}
#[test]
fn scans_arbitrary_bytes_not_just_utf8() {
let db = stream_db(&[(r"\x00\xff\xfe", true)]);
let mut sc = StreamScanner::open(db).unwrap();
assert_eq!(sc.scan(&[0x41, 0x00, 0xff, 0xfe, 0x42]).unwrap(), Some(0));
}
#[test]
fn two_scanners_of_one_database_are_independent() {
let db = stream_db(&[("split-me", true)]);
let mut a = StreamScanner::open(db.clone()).unwrap();
let mut b = StreamScanner::open(db).unwrap();
assert_eq!(a.scan(b"spl").unwrap(), None);
// b has seen nothing, so a's partial progress must not carry into it
assert_eq!(b.scan(b"it-me").unwrap(), None);
assert_eq!(a.scan(b"it-me").unwrap(), Some(0));
}
#[test]
fn a_bad_pattern_names_itself() {
let err = Database::validate("(unclosed", true, Mode::Stream).unwrap_err();
assert!(!err.is_empty(), "the compile error should carry a message");
assert!(Database::validate("(closed)", true, Mode::Stream).is_ok());
}
#[test]
fn block_mode_reports_every_match_with_offsets() {
let db = Database::compile(
&[Pattern { expr: "a.c".to_string(), case_sensitive: true }],
Mode::Block,
)
.unwrap();
let hits = scan_block(&db, b"abc___axc", 100).unwrap();
assert_eq!(hits.len(), 2);
assert_eq!((hits[0].from, hits[0].to), (0, 3));
assert_eq!((hits[1].from, hits[1].to), (6, 9));
}
#[test]
fn block_mode_stops_at_the_limit() {
let db = Database::compile(
&[Pattern { expr: "x".to_string(), case_sensitive: true }],
Mode::Block,
)
.unwrap();
let hits = scan_block(&db, &[b'x'; 1000], 10).unwrap();
assert_eq!(hits.len(), 10);
}
#[test]
fn a_database_refuses_the_wrong_scan_mode() {
let block = Database::compile(
&[Pattern { expr: "x".to_string(), case_sensitive: true }],
Mode::Block,
)
.unwrap();
assert!(StreamScanner::open(Arc::new(block)).is_err());
let stream = Database::compile(
&[Pattern { expr: "x".to_string(), case_sensitive: true }],
Mode::Stream,
)
.unwrap();
assert!(scan_block(&stream, b"x", 10).is_err());
}
}

32
backend/proxysrc/src/lib.rs Normal file
View File

@ -0,0 +1,32 @@
//! Proxy datapath engine for firegex.
//!
//! An alternative to the NFQUEUE engine: instead of lifting packets to userspace and
//! handing a verdict back to the kernel, terminate the connection and own both halves.
//! That buys exact rewriting, kernel-side reassembly and real backpressure — and costs
//! the kernel's fail-open backstop, which [`filter`] rebuilds by hand.
pub mod control;
pub mod debug;
pub mod filter;
pub mod h1up;
pub mod h2;
pub mod h3;
pub mod http1;
pub mod hyperscan;
pub mod proxy;
pub mod quic;
pub mod quic_hello;
pub mod pyworker;
pub mod rules;
pub mod spec;
pub mod capture;
pub mod tls;
pub mod transparent;
pub mod relays;
pub mod report;
pub mod udp;
pub use filter::{
ChainHandle, ChainSessions, Direction, Filter, FilterChain, FilterCtx, FilterSession, Verdict,
};
pub use proxy::{Proxy, ProxyConfig, Upstream};

497
backend/proxysrc/src/main.rs Normal file
View File

@ -0,0 +1,497 @@
use std::net::SocketAddr;
use std::process::exit;
use std::sync::Arc;
use std::time::Duration;
use fgex_proxy::control::serve_stdin;
use fgex_proxy::filter::{ChainHandle, FilterChain};
use fgex_proxy::proxy::{Proxy, ProxyConfig, TlsSetup, Upstream};
use fgex_proxy::quic::{QuicConfig, QuicManager, QuicSetup};
use fgex_proxy::relays::Relays;
use fgex_proxy::spec::parse_filters;
use fgex_proxy::transparent::probe_capability;
/// How often the engine reports its connection counters. One short line, so the cost is
/// nothing; the first is sent immediately so a freshly started service has a baseline
/// rather than an unknown for the first interval.
const STATS_INTERVAL: Duration = Duration::from_secs(2);
fn env_or_exit(key: &str) -> String {
match std::env::var(key) {
Ok(v) => v,
Err(_) => {
eprintln!("[fatal] [main] {key} is required");
exit(2);
}
}
}
fn env_flag(key: &str) -> bool {
matches!(
std::env::var(key).unwrap_or_default().trim(),
"1" | "true" | "yes"
)
}
fn parse_addr(key: &str, raw: &str) -> SocketAddr {
match raw.parse() {
Ok(a) => a,
Err(e) => {
eprintln!("[fatal] [main] {key}='{raw}' is not a valid address: {e}");
exit(2);
}
}
}
/// `FGEX_PROXY_TLS_CERT` / `_KEY` are paths, not the material itself: a certificate
/// on a command line or in an env dump is a certificate in a log somewhere.
fn env_usize(key: &str) -> Option<usize> {
std::env::var(key).ok()?.trim().parse().ok()
}
/// The certificate and key this service terminates with, read once.
///
/// Read rather than passed: a certificate on a command line or in an environment dump is
/// a certificate in a log somewhere. Both edges that need it — TLS over TCP and QUIC —
/// ask for the same pair, so it is read here and handed to whichever wants it.
fn read_material() -> Result<Option<(String, String)>, String> {
let cert_path = std::env::var("FGEX_PROXY_TLS_CERT").ok();
let key_path = std::env::var("FGEX_PROXY_TLS_KEY").ok();
match (cert_path, key_path) {
(Some(cert), Some(key)) => {
let certificate =
std::fs::read_to_string(&cert).map_err(|e| format!("cannot read {cert}: {e}"))?;
let key =
std::fs::read_to_string(&key).map_err(|e| format!("cannot read the key: {e}"))?;
Ok(Some((certificate, key)))
}
(None, None) => Ok(None),
_ => Err("FGEX_PROXY_TLS_CERT and FGEX_PROXY_TLS_KEY go together".to_string()),
}
}
fn build_tls(material: Option<&(String, String)>, optional: bool) -> Result<TlsSetup, String> {
let server = match material {
Some((cert, key)) => Some(fgex_proxy::tls::server_config(cert, key)?),
None => None,
};
// Built whenever this engine terminates anything, and *used* per address: whether a
// given connection is re-encrypted on the way out is the address's answer
// (`Onward`), not the process's. A plain TCP service terminates nothing, so it has
// no client configuration and never did — which is why none of this reaches it.
let upstream = match server {
Some(_) => Some(fgex_proxy::tls::client_config()?),
None => None,
};
Ok(TlsSetup {
server,
upstream,
optional,
})
}
/// Answer one regex-debug request and exit, instead of starting a datapath.
///
/// Its own mode rather than a long-running service: the backend asks rarely, an
/// operator's half-written pattern must not be able to affect anything that is
/// carrying traffic, and a process that exits cannot leak scratch space or state
/// between requests.
fn debug_regex() -> ! {
use std::io::Read;
let mut request = String::new();
if let Err(e) = std::io::stdin().read_to_string(&mut request) {
eprintln!("[fatal] [debug] cannot read the request: {e}");
exit(2);
}
let response = fgex_proxy::debug::run(&request);
match serde_json::to_string(&response) {
Ok(json) => println!("{json}"),
Err(e) => {
eprintln!("[fatal] [debug] cannot encode the response: {e}");
exit(2);
}
}
exit(0);
}
/// How many worker threads the runtime gets.
///
/// `NTHREADS` is what `run.py --threads` sets, and it used to reach only the NFQUEUE
/// binaries — this engine took every core whatever the operator asked for, so the same
/// flag meant one thing on one layer and nothing on the other. It means the same thing
/// on both now. Absent or unparseable is "as many as there are cores", which is tokio's
/// own default and the right answer when nobody has said otherwise.
fn worker_threads() -> usize {
std::env::var("NTHREADS")
.ok()
.and_then(|v| v.trim().parse::<usize>().ok())
.filter(|n| *n > 0)
.unwrap_or_else(|| std::thread::available_parallelism().map(|n| n.get()).unwrap_or(1))
}
fn main() {
// Built by hand rather than through `#[tokio::main]`, which offers no way to say how
// many workers it should have.
let runtime = match tokio::runtime::Builder::new_multi_thread()
.worker_threads(worker_threads())
.enable_all()
.build()
{
Ok(rt) => rt,
Err(e) => {
eprintln!("[fatal] [main] cannot start the runtime: {e}");
exit(1);
}
};
runtime.block_on(run());
}
async fn run() {
// Checked before anything else reads the environment: this mode starts no
// listener, opens no sockets and touches no rules.
if std::env::args().any(|a| a == "--debug-regex") {
debug_regex();
}
let listen_raw = env_or_exit("FGEX_PROXY_LISTEN");
let listen = parse_addr("FGEX_PROXY_LISTEN", &listen_raw);
// "original" puts one listener in front of every intercepted service; a literal
// address keeps the simpler one-listener-one-service shape.
let upstream_raw = env_or_exit("FGEX_PROXY_UPSTREAM");
let upstream = if matches!(upstream_raw.trim(), "original" | "auto") {
Upstream::Original
} else {
Upstream::Fixed(parse_addr("FGEX_PROXY_UPSTREAM", &upstream_raw))
};
// Only the outbound dial needs IP_TRANSPARENT now: the listener is a plain one
// that traffic is redirected into. Binding a foreign address was a tproxy
// requirement, and tproxy is gone.
let spoof_source = env_flag("FGEX_PROXY_SPOOF_SOURCE");
// Fail here rather than per connection: a missing CAP_NET_ADMIN would otherwise
// show up only as every service seeing the proxy's address instead of the client's.
if spoof_source {
if let Err(e) = probe_capability(listen.is_ipv6()) {
eprintln!(
"[fatal] [main] transparent mode needs CAP_NET_ADMIN, IP_TRANSPARENT failed: {e}"
);
exit(1);
}
}
let deadline = std::env::var("FGEX_PROXY_FILTER_TIMEOUT_MS")
.ok()
.and_then(|v| v.parse::<u64>().ok())
.unwrap_or(2000);
let connect_timeout = std::env::var("FGEX_PROXY_CONNECT_TIMEOUT_MS")
.ok()
.and_then(|v| v.parse::<u64>().ok())
.unwrap_or(5000);
let specs = std::env::var("FGEX_PROXY_FILTERS").unwrap_or_default();
let filters = match parse_filters(&specs) {
Ok(f) => f,
Err(e) => {
eprintln!("[fatal] [main] bad FGEX_PROXY_FILTERS: {e}");
exit(2);
}
};
// Read before either edge is built: TLS over TCP and QUIC terminate with the same
// certificate, and a service speaks one of them or the other.
let material = match read_material() {
Ok(material) => material,
Err(e) => {
eprintln!("[fatal] [main] {e}");
exit(2);
}
};
// QUIC is UDP on the wire, so it takes the place of the datagram relays rather than
// sitting beside them: the same map of one port per protected address, bound by
// something that terminates instead of something that forwards.
let quic = env_flag("FGEX_PROXY_QUIC");
// Whether the TCP listener terminates TLS, and whether it insists on it.
//
// `FGEX_PROXY_TLS_OPTIONAL` is what an `http` service sets: one certificate, one
// chain, and each connection carried as whatever the client opened it with — in the
// clear on one port and under TLS on another. It also means TLS on TCP and QUIC on
// UDP live in **one** engine process, which they could not before: the two used to be
// exclusive here, on the reasoning that a service speaks one or the other, and an
// HTTP service speaks both.
let tls_optional = env_flag("FGEX_PROXY_TLS_OPTIONAL");
// Absent, the old rule: a certificate means terminate, unless this is a QUIC service,
// where the certificate belongs to the QUIC edge and the TCP listener has nothing
// pointed at it. Said explicitly by anything that wants both.
let terminate_tls = env_flag("FGEX_PROXY_TLS") || tls_optional || !quic;
let tls = match build_tls(
if terminate_tls { material.as_ref() } else { None },
tls_optional,
) {
Ok(setup) => setup,
Err(e) => {
eprintln!("[fatal] [main] {e}");
exit(2);
}
};
// Stamped on every socket this engine opens towards a service, so the rules that
// intercept that service can tell our own dial from the traffic they are meant to
// catch. Read once: the UDP relays need the same mark as the TCP path.
let cfg_self_mark = std::env::var("FGEX_PROXY_SELF_MARK")
.ok()
.and_then(|v| u32::from_str_radix(v.trim_start_matches("0x"), 16).ok())
.or(Some(fgex_proxy::transparent::SELF_MARK));
// Read once and shared by both paths: TCP connections and UDP flows are the same
// resource being defended, and two numbers meaning "how much of it may be spent"
// would be two things to keep in step.
let max_connections = env_usize("FGEX_PROXY_MAX_CONNECTIONS").unwrap_or(0);
let over_limit_forwards = env_flag("FGEX_PROXY_OVER_LIMIT_FORWARD");
// In seconds, because that is the unit an operator thinks in for "how long may a
// connection say nothing". Zero and absent both mean off.
let first_byte_timeout = match env_usize("FGEX_PROXY_FIRST_BYTE_TIMEOUT").unwrap_or(0) {
0 => None,
secs => Some(Duration::from_secs(secs as u64)),
};
// Opened once for the process and shared by every path that reconstructs something:
// the interface belongs to the instance, not to a service or a connection.
let capture = fgex_proxy::capture::Capture::open();
// Where each published address sends its traffic, for the ones this engine starts
// with. `10.0.0.1:443=10.0.0.1:80` reads as "what arrives at the first is the
// service at the second" — an address published on a port the service does not
// listen on. Absent, and for every address not named, the answer is the address
// itself, which is what transparent means and what every service had before.
let targets = std::sync::Arc::new(fgex_proxy::proxy::Targets::default());
if let Ok(spec) = std::env::var("FGEX_PROXY_TARGETS") {
for entry in spec.split(',').map(str::trim).filter(|s| !s.is_empty()) {
// `10.0.0.1:443|tls=10.0.0.1:80`, and every part after the address optional:
// the edge says what is spoken there and the target where the service is,
// and an address that says neither is one this engine is simply in front of.
let (address, target) = match entry.split_once('=') {
Some((address, target)) => (
address,
Some(parse_addr("FGEX_PROXY_TARGETS", target.trim())),
),
None => (entry, None),
};
// `<address>|<edge>|<onward>`, both words optional and both defaulting to
// the answer every address gave before there was a question.
let mut words = address.split('|');
let address = words.next().unwrap_or(address);
let edge = match words.next() {
Some("tls") => fgex_proxy::proxy::Edge::Tls,
None | Some("clear") | Some("any") => fgex_proxy::proxy::Edge::Whatever,
Some(other) => {
eprintln!("[fatal] [main] FGEX_PROXY_TARGETS: unknown edge '{other}'");
exit(2);
}
};
let upstream = match words.next() {
Some("plain") => fgex_proxy::proxy::Onward::Plain,
Some("tls") => fgex_proxy::proxy::Onward::Tls,
None | Some("same") => fgex_proxy::proxy::Onward::Same,
Some(other) => {
eprintln!("[fatal] [main] FGEX_PROXY_TARGETS: unknown upstream '{other}'");
exit(2);
}
};
let public = parse_addr("FGEX_PROXY_TARGETS", address.trim());
// Said out loud, because everything else about an address is visible in the
// rules and this is not: it lives in this process and nowhere else, so an
// operator wondering why a port behaves the way it does has nothing to read.
eprintln!(
"[info] [main] {public} carries {} to {}",
match edge {
fgex_proxy::proxy::Edge::Tls => "TLS only",
_ => "whatever arrives",
},
match (target, upstream) {
(Some(target), fgex_proxy::proxy::Onward::Plain) => {
format!("{target} in the clear")
}
(Some(target), fgex_proxy::proxy::Onward::Tls) => format!("{target} on TLS"),
(Some(target), _) => format!("{target} as it arrived"),
(None, fgex_proxy::proxy::Onward::Plain) => "the service in the clear".into(),
(None, fgex_proxy::proxy::Onward::Tls) => "the service on TLS".into(),
(None, _) => "the service as it arrived".into(),
}
);
targets.publish(public, fgex_proxy::proxy::Published { target, edge, upstream });
}
}
let cfg = ProxyConfig {
listen,
upstream,
spoof_source,
connect_timeout: Duration::from_millis(connect_timeout),
tls,
// `None` simply means nothing is watching, which is the ordinary case.
capture: capture.clone(),
// Zero means no limit, which is what it was before there was one. The backend
// always sends a value; the default here is for anyone running the engine by
// hand, where a surprise limit would be worse than none.
max_connections,
over_limit_forwards,
first_byte_timeout,
self_mark: cfg_self_mark,
targets: std::sync::Arc::clone(&targets),
};
let chain = ChainHandle::new(FilterChain::new(filters, Duration::from_millis(deadline)));
let chain_handle = chain.clone();
let proxy = match Proxy::bind(cfg, chain).await {
Ok(p) => p,
Err(e) => {
eprintln!("[fatal] [main] cannot bind {listen}: {e}");
exit(1);
}
};
// Handshake, in the shape the other engines use (`QUEUE <n>` from cppregex):
// one machine-readable line on stdout, so the caller can pass port 0 and learn
// which port it actually got. Everything descriptive goes to stderr.
let bound = proxy.local_addr().unwrap();
// Taken before the UDP relays are built: they share it, so one service reports one
// pair of numbers whichever half of it the traffic arrived on.
let counters = proxy.stats();
fgex_proxy::report::reply(format!("PORT {}", bound.port()));
eprintln!(
"[info] [main] listening on {bound} -> {upstream_raw} \
(spoof_source={spoof_source})"
);
// The relays this service's addresses get, bound at startup and added to on demand.
// Which kind is not a per-address choice: a service speaks QUIC or it speaks
// datagrams, and the one it speaks is what every one of its addresses gets.
let relays = if quic {
let (cert, key) = match &material {
Some(pair) => pair,
None => {
eprintln!(
"[fatal] [main] a QUIC service needs a certificate: QUIC carries TLS 1.3 \
inside it, and terminating it is the only way a filter sees anything"
);
exit(2);
}
};
let setup = match QuicSetup::build(cert, key) {
Ok(setup) => Arc::new(setup),
Err(e) => {
eprintln!("[fatal] [main] {e}");
exit(2);
}
};
eprintln!(
"[info] [main] QUIC terminated here; each client's own protocols are offered \
to the service"
);
Relays::Quic(QuicManager::new(
chain_handle.clone(),
QuicConfig {
setup,
capture: capture.clone(),
self_mark: cfg_self_mark,
spoof_source,
max_connections,
over_limit_forwards,
first_byte_timeout,
connect_timeout: Duration::from_millis(connect_timeout),
// The default for a relay that was added without saying; the backend
// says, per address, on `FGEX_PROXY_UDP` and on `ADD_UDP`.
upstream: fgex_proxy::proxy::Onward::Same,
},
Arc::clone(&counters),
))
} else {
Relays::Datagram(fgex_proxy::udp::UdpManager::new(
chain_handle.clone(),
cfg_self_mark,
spoof_source,
max_connections,
over_limit_forwards,
Arc::clone(&counters),
))
};
// UDP, when the backend asks for it: a comma-separated list of protected
// addresses, one relay each. One socket per address rather than one for all of
// them, because `SO_ORIGINAL_DST` — which is how the TCP side learns where a
// connection was headed — is TCP and SCTP only. With the upstream fixed per socket
// there is nothing to recover.
if let Ok(spec) = std::env::var("FGEX_PROXY_UDP") {
for target in spec.split(',').filter(|s| !s.trim().is_empty()) {
// `<address>` or `<address>|<onward>`: what the service behind this one
// relay speaks. Per relay because a relay is one protected address, and
// two ports of one service can be reached differently.
let (target, onward) = match target.trim().split_once('|') {
Some((target, word)) => match fgex_proxy::proxy::Onward::from_word(word) {
Some(onward) => (target, onward),
None => {
eprintln!("[fatal] [main] FGEX_PROXY_UDP: unknown upstream '{word}'");
exit(2);
}
},
None => (target.trim(), fgex_proxy::proxy::Onward::Same),
};
let upstream = parse_addr("FGEX_PROXY_UDP", target.trim());
match relays.add_relay(upstream, onward).await {
Ok(port) => {
// With what it speaks onward, because that is half of what names a
// relay: two addresses sending to one service port can want different
// answers, and each gets a relay of its own.
fgex_proxy::report::reply(format!("UDP {upstream}|{} {port}", onward.word()));
}
Err(e) => {
eprintln!("[fatal] [main] cannot bind a UDP relay for {upstream}: {e}");
exit(1);
}
}
}
}
// Rulesets and control commands (e.g. ADD_UDP) arrive on stdin from here on.
// It runs alongside the datapath rather than gating it: traffic must flow
// before, during and after a rule change or relay addition.
let deadline_dur = Duration::from_millis(deadline);
let control_chain = chain_handle.clone();
let control_relays = relays.clone();
tokio::spawn(async move { serve_stdin(control_chain, deadline_dur, control_relays, targets).await });
// Both counters live on the proxy, not on the chain: a chain is replaced wholesale
// every time a ruleset is pushed, and counters that reset on a rule edit would make
// the ratio jump for a reason nobody could see.
tokio::spawn(async move {
let mut tick = tokio::time::interval(STATS_INTERVAL);
loop {
// The first tick fires immediately, which is the point: zero out of zero is
// a baseline, and "no answer yet" is not.
tick.tick().await;
// `live` and `over_limit` ride along with the two that were already here:
// the backend parses this line as key/value pairs, so a new number costs no
// protocol. `over_limit` is cumulative on purpose — a service that hit the
// wall once an hour ago still says so, which is the whole point of a trace.
// Dropped rather than waited for when the backend is behind: the next one
// carries the same cumulative numbers two seconds later.
fgex_proxy::report::event(format!(
"STATS seen={} refused={} live={} over_limit={} no_first_byte={}",
counters.accepted.load(std::sync::atomic::Ordering::Relaxed),
counters.closed_by_filter.load(std::sync::atomic::Ordering::Relaxed),
counters.live.load(std::sync::atomic::Ordering::Relaxed),
counters.over_limit.load(std::sync::atomic::Ordering::Relaxed),
counters.no_first_byte.load(std::sync::atomic::Ordering::Relaxed),
));
}
});
if let Err(e) = proxy.serve().await {
eprintln!("[fatal] [main] accept loop died: {e}");
exit(1);
}
}

1199
backend/proxysrc/src/proxy.rs Normal file
View File

File diff suppressed because it is too large Load Diff

470
backend/proxysrc/src/pyworker.rs Normal file
View File

@ -0,0 +1,470 @@
//! User Python code, kept out of the datapath process.
//!
//! The C++ engine embeds CPython, which is where a good share of its
//! complexity lives: sub-interpreters with their own GIL, `check_multi_interp_extensions`
//! locking out most of the ecosystem, and `allow_threads = 0` locking out the user's own.
//!
//! Here the code runs in a child process instead, for one reason above all: **a hung
//! worker can be killed, a hung thread cannot**. The engine's own filter deadline can
//! only walk away from a blocking thread and leave it holding a pool slot until the
//! process restarts — measured and asserted in `tests/fail_open.rs`. A child that
//! misses its deadline is killed outright and respawned on the next chunk, and the
//! chunk it was holding is forwarded.
//!
//! Frames are length-prefixed and binary, in both directions:
//!
//! ```text
//! engine -> worker [u32 len][u8 kind][u64 connection][payload]
//! kind: 0 client->server, 1 server->client,
//! 2 connection closed, 3 connection opened
//! worker -> engine [u32 len][u8 verdict][u8 name len][name][payload]
//! verdict: 0 accept, 1 reject
//! ```
//!
//! The name is the `@pyfilter` function that decided, which is what makes a block
//! attributable to one function of a file rather than to the file as a whole — a file
//! routinely holds several, and "this filter blocked 900 connections" is not an answer
//! when you are trying to find out which of them is doing it.
//!
//! Neither an open nor a close frame is answered. A close has no verdict to give about
//! a connection that is already over, and waiting for one would make tearing down a
//! connection depend on a process the engine is willing to kill; an open carries the
//! connection's metadata — the addresses and ports a filter is allowed to read — once,
//! rather than repeating it on every chunk.
use std::io::{Read, Write};
use std::os::fd::AsRawFd;
use std::process::{Child, Command, Stdio};
use std::sync::atomic::{AtomicU64, Ordering};
use std::sync::Mutex;
use std::time::{Duration, Instant};
use crate::filter::{ConnectionId, ConnectionMeta, Direction, Filter, FilterCtx, Verdict, L4};
const VERDICT_ACCEPT: u8 = 0;
const VERDICT_REJECT: u8 = 1;
/// Sent by the worker once the user's file has run. Waiting for it is what turns
/// "this code does not load" into a refused ruleset instead of a service that
/// silently stopped filtering.
const READY: u8 = 0xFF;
/// What a frame means. The worker keeps one set of the user's module globals per
/// connection, which is what the documented API promises, so every frame has to say
/// which connection it belongs to — and something has to say when one is over, or the
/// worker would grow for as long as the service runs.
const KIND_C2S: u8 = 0;
const KIND_S2C: u8 = 1;
const KIND_CLOSE: u8 = 2;
/// Carries the connection's metadata as one JSON object. Everything below the
/// application layer that a filter may see arrives here and nowhere else — there are no
/// header bytes in the data frames, and no way to ask for any.
const KIND_OPEN: u8 = 3;
/// `[u32 len][u8 kind][u64 connection][payload]`, big-endian, len covering everything
/// after itself.
fn build_frame(connection: ConnectionId, kind: u8, payload: &[u8]) -> Vec<u8> {
let mut frame = Vec::with_capacity(payload.len() + 13);
frame.extend_from_slice(&(payload.len() as u32 + 9).to_be_bytes());
frame.push(kind);
frame.extend_from_slice(&connection.to_be_bytes());
frame.extend_from_slice(payload);
frame
}
/// How long a worker gets to import and execute the user's file. Longer than a
/// per-chunk deadline on purpose: this one pays for interpreter start-up.
const STARTUP_GRACE: Duration = Duration::from_secs(10);
/// Cap on one frame, so a confused worker cannot ask us to allocate the machine away.
const MAX_FRAME: u32 = 64 * 1024 * 1024;
/// Cap on the open and close frames waiting for the next exchange. Reached only when
/// nothing has asked the worker anything for thousands of connections — a filter that
/// has lost its say is still told about every connection — and past it they are
/// dropped: a connection then reaches the worker without its addresses, or leaves state
/// behind in a worker that was not going to be asked anything anyway.
const MAX_PENDING: usize = 1024 * 1024;
#[derive(Debug, Default)]
pub struct WorkerStats {
/// Times the worker was killed for missing its deadline or dying on its own.
pub restarts: AtomicU64,
/// Chunks forwarded because the worker could not answer for them.
pub failed_open: AtomicU64,
}
/// A filter backed by a child process running the user's Python.
pub struct PyWorkerRule {
id: String,
command: Vec<String>,
code_path: String,
/// The subset of the file's `@pyfilter` functions to run; `None` means all of them.
enabled: Option<Vec<String>>,
deadline: Duration,
/// One worker, one conversation at a time. Serialising is the honest simple
/// thing: a frame is a request and a response, and interleaving them would need
/// request ids and a worker able to answer out of order. A pool comes later.
child: Mutex<Option<Child>>,
/// Open and close frames not yet written, in order. They need no answer, and they
/// are told from the relay itself — on the async runtime — where taking `child`
/// meant waiting out whatever exchange held it, up to its whole deadline, and where
/// a worker that had to be started first was started right there. A filter hanging
/// on one connection held every new connection of the service in that queue, on the
/// threads that carry all the others. The next exchange writes them ahead of its own
/// chunk, which runs where blocking is allowed and keeps an open ahead of the data.
pending: Mutex<Vec<u8>>,
pub stats: std::sync::Arc<WorkerStats>,
/// Everything that would make a new worker behave differently from this one; see
/// `Filter::reuse_key`. `None` when the file could not be read to say so.
reuse_key: Option<String>,
}
impl PyWorkerRule {
pub fn new(
id: String,
command: Vec<String>,
code_path: String,
enabled: Option<Vec<String>>,
deadline: Duration,
) -> Self {
Self {
id,
command,
code_path,
enabled,
deadline,
child: Mutex::new(None),
pending: Mutex::new(Vec::new()),
stats: std::sync::Arc::new(WorkerStats::default()),
reuse_key: None,
}
}
/// What this rule would be built from, as one string: the id, the command, which
/// functions are on, the deadline and the code itself. The same string means a new
/// worker would run exactly what this one runs, so it is kept instead — with the
/// state it holds for every open connection. The code is read rather than trusted to
/// its path, because saving a filter rewrites the same file.
pub fn key_for(
id: &str,
command: &[String],
code_path: &str,
enabled: &Option<Vec<String>>,
deadline: Duration,
) -> Option<String> {
use std::hash::{Hash, Hasher};
let code = std::fs::read(code_path).ok()?;
let mut hasher = std::collections::hash_map::DefaultHasher::new();
code.hash(&mut hasher);
Some(format!(
"{id}\0{command:?}\0{code_path}\0{enabled:?}\0{}\0{:016x}",
deadline.as_millis(),
hasher.finish()
))
}
/// Answer to `key` from `Filter::reuse_key`.
pub fn reusable_as(mut self, key: Option<String>) -> Self {
self.reuse_key = key;
self
}
fn spawn(&self) -> Result<Child, String> {
let (program, args) = self
.command
.split_first()
.ok_or_else(|| "no command for the worker".to_string())?;
let mut child = Command::new(program)
.args(args)
// The file, and which of its functions are switched on. Which protocol it
// speaks is the file's own business — the library reads it off what the
// filters ask for, so there is nothing here that could disagree with the
// code.
//
// The selection is a marker plus a comma-separated list, because "run
// everything" and "run nothing" are both real answers and an empty string
// cannot mean both: `*` is all of them, `=` followed by names is exactly
// those, and `=` alone is none.
.arg(&self.code_path)
.arg(match &self.enabled {
None => "*".to_string(),
Some(names) => format!("={}", names.join(",")),
})
.stdin(Stdio::piped())
.stdout(Stdio::piped())
// Tracebacks go to our stderr, where the backend already reads them.
.stderr(Stdio::inherit())
.spawn()
.map_err(|e| format!("cannot start the worker: {e}"))?;
// Wait for it to say it loaded. A file that raises on import never gets here.
let deadline_at = Instant::now() + STARTUP_GRACE;
let ready = (|| -> Result<(), String> {
let stdout = child.stdout.as_mut().ok_or("worker has no stdout")?;
let mut header = [0u8; 4];
read_before(stdout, &mut header, deadline_at)?;
// The same shape as a verdict — marker plus an empty name — so there is one
// frame layout on this channel rather than one for the handshake and
// another for everything after it.
if u32::from_be_bytes(header) != 2 {
return Err("worker did not announce itself".to_string());
}
let mut marker = [0u8; 2];
read_before(stdout, &mut marker, deadline_at)?;
if marker[0] != READY {
return Err("worker did not announce itself".to_string());
}
Ok(())
})();
if let Err(why) = ready {
let _ = child.kill();
let _ = child.wait();
return Err(format!("the filter code did not load: {why}"));
}
Ok(child)
}
/// Which `@pyfilter` functions of the file are switched on, or `None` for all.
///
/// Deciding here rather than in the file is the point: a function is turned off
/// without editing the code that defines it, and turned back on without the
/// operator having to remember what they deleted.
/// Hold a frame that needs no answer for the next exchange. See `pending`.
fn queue(&self, frame: Vec<u8>) {
let Ok(mut pending) = self.pending.lock() else { return };
if pending.len() + frame.len() > MAX_PENDING {
pending.clear();
}
pending.extend_from_slice(&frame);
}
pub fn enabled(&self) -> Option<&[String]> {
self.enabled.as_deref()
}
/// Start the worker now, so a ruleset carrying code that cannot load is refused
/// at the moment it is applied rather than discovered later as unfiltered traffic.
pub fn warm_up(&self) -> Result<(), String> {
let mut slot = self
.child
.lock()
.map_err(|_| "worker lock poisoned".to_string())?;
if slot.is_none() {
*slot = Some(self.spawn()?);
}
Ok(())
}
/// Kill and forget, so the next chunk starts a fresh one.
fn discard(&self, slot: &mut Option<Child>, why: &str) {
if let Some(mut child) = slot.take() {
let _ = child.kill();
let _ = child.wait();
self.stats.restarts.fetch_add(1, Ordering::Relaxed);
crate::diag!("[warn] [pyworker] '{}' killed: {why}", self.id);
}
}
fn exchange(
&self,
connection: ConnectionId,
dir: Direction,
chunk: &[u8],
) -> Result<Verdict, String> {
let mut slot = self
.child
.lock()
.map_err(|_| "worker lock poisoned".to_string())?;
if slot.is_none() {
*slot = Some(self.spawn()?);
}
let deadline_at = Instant::now() + self.deadline;
let result = (|| -> Result<Verdict, String> {
let child = slot.as_mut().expect("just spawned");
let stdin = child.stdin.as_mut().ok_or("worker has no stdin")?;
let frame = build_frame(
connection,
match dir {
Direction::ClientToServer => KIND_C2S,
Direction::ServerToClient => KIND_S2C,
},
chunk,
);
let queued = std::mem::take(
&mut *self.pending.lock().map_err(|_| "worker queue poisoned".to_string())?,
);
if !queued.is_empty() {
write_before(stdin, &queued, deadline_at)?;
}
write_before(stdin, &frame, deadline_at)?;
let stdout = child.stdout.as_mut().ok_or("worker has no stdout")?;
let mut header = [0u8; 4];
read_before(stdout, &mut header, deadline_at)?;
let len = u32::from_be_bytes(header);
if len == 0 || len > MAX_FRAME {
return Err(format!("worker sent a {len} byte frame"));
}
let mut body = vec![0u8; len as usize];
read_before(stdout, &mut body, deadline_at)?;
// `[verdict][name len][name][payload]`. The name is whichever function
// decided; it is what lets a block be counted against one function of the
// file instead of the whole file.
if body.len() < 2 {
return Err("worker sent a truncated verdict".to_string());
}
let name_len = body[1] as usize;
if body.len() < 2 + name_len {
return Err("worker sent a verdict whose name runs past the frame".to_string());
}
let name = String::from_utf8_lossy(&body[2..2 + name_len]).into_owned();
match body[0] {
VERDICT_ACCEPT => Ok(Verdict::Accept),
VERDICT_REJECT => Ok(Verdict::Reject(Some(if name.is_empty() {
self.id.clone()
} else {
// `<rule>/<function>`, the one token both network layers report, so
// the backend has a single way to attribute a block.
format!("{}/{}", self.id, name)
}))),
other => Err(format!("worker sent verdict {other}")),
}
})();
if let Err(ref why) = result {
// Anything that goes wrong here — a deadline, a crash, a protocol
// violation — costs the worker its life, not the traffic.
self.discard(&mut slot, why);
}
result
}
}
impl Filter for PyWorkerRule {
fn name(&self) -> &str {
&self.id
}
fn connection_opened(&self, connection: ConnectionId, meta: &ConnectionMeta) {
// Best effort and unanswered, like the close frame: the metadata is a
// convenience for the filter, and a worker that has died will be respawned by
// the first chunk anyway — which then arrives without endpoints rather than not
// at all. Losing the traffic over it would be the wrong trade.
let payload = serde_json::json!({
"client_ip": meta.client.ip().to_string(),
"client_port": meta.client.port(),
"server_ip": meta.server.ip().to_string(),
"server_port": meta.server.port(),
"is_ipv6": meta.server.is_ipv6(),
// Both, and they are not the same question. `is_tcp` is what the NFQUEUE
// side has always sent and what a filter reads to know the wire; `l4` is what
// decides whether a stream can be built, which is true of QUIC and not of a
// datagram. A library that only received the first would have to guess one
// of them.
"is_tcp": meta.l4 == L4::Tcp,
"l4": meta.l4.name(),
})
.to_string();
self.queue(build_frame(connection, KIND_OPEN, payload.as_bytes()));
}
fn connection_closed(&self, connection: ConnectionId) {
// Best effort, and deliberately not on the datapath's critical path: the
// connection is already over, so nothing is waiting on this. A worker that
// has died in the meantime simply has no state left to free.
self.queue(build_frame(connection, KIND_CLOSE, &[]));
}
fn prepare(&self) -> Result<(), String> {
self.warm_up()
}
fn reuse_key(&self) -> Option<&str> {
self.reuse_key.as_deref()
}
fn inspect(&self, ctx: &FilterCtx<'_>) -> Verdict {
match self.exchange(ctx.connection, ctx.direction, ctx.chunk) {
Ok(verdict) => verdict,
Err(_) => {
// The rule loses its say for this chunk; the chunk goes through.
self.stats.failed_open.fetch_add(1, Ordering::Relaxed);
Verdict::Accept
}
}
}
}
/// Wait until the fd is ready, or the deadline passes.
fn wait_ready(fd: i32, writable: bool, deadline_at: Instant) -> Result<(), String> {
let remaining = deadline_at.saturating_duration_since(Instant::now());
if remaining.is_zero() {
return Err("worker missed its deadline".to_string());
}
let mut pfd = libc::pollfd {
fd,
events: if writable {
libc::POLLOUT
} else {
libc::POLLIN
},
revents: 0,
};
let timeout_ms = remaining.as_millis().min(i32::MAX as u128) as libc::c_int;
let rc = unsafe { libc::poll(&mut pfd, 1, timeout_ms) };
match rc {
0 => Err("worker missed its deadline".to_string()),
n if n < 0 => Err(format!("poll failed: {}", std::io::Error::last_os_error())),
_ => {
if pfd.revents & (libc::POLLERR | libc::POLLNVAL) != 0 {
return Err("worker pipe broke".to_string());
}
Ok(())
}
}
}
/// Write it all, giving up when the deadline passes rather than blocking forever on
/// a worker that has stopped reading.
fn write_before<W: Write + AsRawFd>(
out: &mut W,
mut buf: &[u8],
deadline_at: Instant,
) -> Result<(), String> {
while !buf.is_empty() {
wait_ready(out.as_raw_fd(), true, deadline_at)?;
match out.write(buf) {
Ok(0) => return Err("worker closed its input".to_string()),
Ok(n) => buf = &buf[n..],
Err(ref e) if e.kind() == std::io::ErrorKind::Interrupted => {}
Err(e) => return Err(format!("cannot write to the worker: {e}")),
}
}
out.flush()
.map_err(|e| format!("cannot flush to the worker: {e}"))
}
fn read_before<R: Read + AsRawFd>(
input: &mut R,
buf: &mut [u8],
deadline_at: Instant,
) -> Result<(), String> {
let mut filled = 0;
while filled < buf.len() {
wait_ready(input.as_raw_fd(), false, deadline_at)?;
match input.read(&mut buf[filled..]) {
Ok(0) => return Err("worker exited".to_string()),
Ok(n) => filled += n,
Err(ref e) if e.kind() == std::io::ErrorKind::Interrupted => {}
Err(e) => return Err(format!("cannot read from the worker: {e}")),
}
}
Ok(())
}
impl Drop for PyWorkerRule {
fn drop(&mut self) {
if let Ok(mut slot) = self.child.lock() {
if let Some(mut child) = slot.take() {
let _ = child.kill();
let _ = child.wait();
}
}
}
}

1057
backend/proxysrc/src/quic.rs Normal file
View File

File diff suppressed because it is too large Load Diff

678
backend/proxysrc/src/quic_hello.rs Normal file
View File

@ -0,0 +1,678 @@
//! What a QUIC client offers, read off its first packets before anybody answers them.
//!
//! The TLS path mirrors ALPN by holding the ClientHello open, asking the service with the
//! client's own list and telling the client what came back. QUIC was thought to leave
//! nothing to hold: the ClientHello travels inside an Initial packet, encrypted. But the
//! Initial keys are derived from the Destination Connection ID the client put in the
//! clear (RFC 9001 §5.2) — deliberately, so a middlebox can read exactly this — and so
//! the list can be read before the handshake is answered. That is what this does.
//!
//! What it replaced was a list of candidates the operator had to write down per service,
//! `h3` unless told otherwise: a QUIC service speaking anything else was unreachable
//! through firegex until somebody found the setting. With the client's own list the
//! service chooses from what the client actually offered, exactly as over TLS, and there
//! is nothing to configure.
//!
//! It reads; it never answers. The socket the endpoint owns is wrapped
//! ([`SniffingSocket`]) so every datagram quinn is about to process passes by here first,
//! and only long-header Initial packets are looked at — one comparison for everything
//! else. A ClientHello too large for one packet (post-quantum key shares make that
//! ordinary) is reassembled from its CRYPTO frames across datagrams, in whatever order
//! they arrive.
use std::collections::{HashMap, VecDeque};
use std::fmt;
use std::io::{self, IoSliceMut};
use std::net::SocketAddr;
use std::pin::Pin;
use std::sync::{Arc, Mutex};
use std::task::{Context, Poll};
use std::time::{Duration, Instant};
use quinn::udp::{RecvMeta, Transmit};
use quinn::{AsyncUdpSocket, UdpPoller};
use rustls::quic::{Suite, Version};
use rustls::Side;
/// How many clients' hellos may be in progress at once. Past it the one that has been
/// waiting longest makes room — a hello is claimed within milliseconds of arriving, so the
/// oldest is the one least likely to still be wanted. Forged Initials cost the sender
/// nothing (the keys are public, so a valid one can be made from any address), and with
/// `MAX_HELLO` this bounds what a flood can make this process hold to 16 MiB.
///
/// It used to refuse newcomers instead, so a thousand forged Initials every `KEPT_FOR`
/// were enough to stop every real client's hello from being read.
const MAX_PENDING: usize = 1024;
/// How long an unfinished or unclaimed hello is kept.
const KEPT_FOR: Duration = Duration::from_secs(10);
/// How often the table is swept for hellos older than `KEPT_FOR`. Not on every packet: a
/// sweep walks the whole table, and every packet of a flood would pay for it.
const SWEEP_EVERY: Duration = Duration::from_secs(1);
/// Most CRYPTO bytes kept for one hello. A ClientHello is one or two kilobytes, a few with
/// post-quantum key shares and a resumption ticket; this is a cap on what a peer can make
/// this process hold, not a size anybody meets.
const MAX_HELLO: usize = 16 * 1024;
/// Most CRYPTO fragments kept for one hello, whatever their size. Real clients split a
/// hello into a handful; the cap is what stops a peer sending pieces nobody counted —
/// empty ones, which `MAX_HELLO` did not see at all, grew one hello without end and had
/// every packet sort all of them again, inside the endpoint's receive path.
const MAX_FRAGMENTS: usize = 64;
/// One client's hello as it arrives.
struct Pending {
/// The connection ID its Initials were protected with. A new one means a new attempt —
/// a Retry answered, or another connection from the same port.
dcid: Vec<u8>,
fragments: Vec<(u64, Vec<u8>)>,
held: usize,
alpn: Option<Vec<Vec<u8>>>,
since: Instant,
}
/// The hellos in progress, and the order they began in.
struct Table {
by_client: HashMap<SocketAddr, Pending>,
/// When each entry began, oldest first. An entry replaced or claimed leaves its line
/// here behind it; eviction skips a line whose time no longer matches the entry's.
order: VecDeque<(SocketAddr, Instant)>,
swept: Instant,
}
impl Table {
fn new() -> Self {
Self { by_client: HashMap::new(), order: VecDeque::new(), swept: Instant::now() }
}
fn sweep(&mut self, now: Instant) {
if now.duration_since(self.swept) < SWEEP_EVERY && self.by_client.len() < MAX_PENDING {
return;
}
self.swept = now;
self.by_client.retain(|_, p| now.duration_since(p.since) < KEPT_FOR);
let by_client = &self.by_client;
self.order.retain(|(from, since)| by_client.get(from).is_some_and(|p| p.since == *since));
}
/// Make room for one more by letting go of the entry that began longest ago.
fn evict_oldest(&mut self) {
while let Some((from, since)) = self.order.pop_front() {
if self.by_client.get(&from).is_some_and(|p| p.since == since) {
self.by_client.remove(&from);
return;
}
}
}
fn start(&mut self, from: SocketAddr, dcid: Vec<u8>, now: Instant) -> &mut Pending {
self.order.push_back((from, now));
self.by_client.insert(from, Pending {
dcid,
fragments: Vec::new(),
held: 0,
alpn: None,
since: now,
});
self.by_client.get_mut(&from).expect("just inserted")
}
}
/// The hellos being read, by the address they came from.
pub struct Hellos {
pending: Mutex<Table>,
arrived: tokio::sync::Notify,
suite: Suite,
}
impl fmt::Debug for Hellos {
fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
f.debug_struct("Hellos").finish_non_exhaustive()
}
}
impl Default for Hellos {
fn default() -> Self {
Self::new()
}
}
impl Hellos {
pub fn new() -> Self {
// Every version's Initial packets are protected with this suite, whatever the
// handshake goes on to choose: RFC 9001 §5.2.
let suite = rustls::crypto::ring::cipher_suite::TLS13_AES_128_GCM_SHA256
.tls13()
.and_then(|s| s.quic_suite())
.expect("TLS13_AES_128_GCM_SHA256 carries QUIC");
Self {
pending: Mutex::new(Table::new()),
arrived: tokio::sync::Notify::new(),
suite,
}
}
/// Look at one datagram received from `from`. Cheap for anything but an Initial.
pub fn observe(&self, from: SocketAddr, datagram: &[u8]) {
let mut rest = datagram;
// An Initial can have other packets coalesced behind it in one datagram; only
// the Initials matter, and they come first.
while let Some(packet) = decrypt_initial(&self.suite, rest) {
rest = &rest[packet.length..];
self.absorb(from, packet);
}
}
fn absorb(&self, from: SocketAddr, packet: Initial) {
let Ok(mut table) = self.pending.lock() else { return };
let now = Instant::now();
table.sweep(now);
// A new attempt — a Retry answered, another connection from the same port — or one
// kept past its time starts again; an address seen for the first time needs room.
let fresh = match table.by_client.get(&from) {
Some(entry) => entry.dcid != packet.dcid || now.duration_since(entry.since) >= KEPT_FOR,
None => {
if table.by_client.len() >= MAX_PENDING {
table.evict_oldest();
}
true
}
};
let entry = if fresh {
table.start(from, packet.dcid.clone(), now)
} else {
table.by_client.get_mut(&from).expect("looked up above")
};
if entry.alpn.is_some() {
return;
}
let mut grew = false;
for (offset, data) in packet.crypto {
// An empty piece says nothing, and one past where any hello could reach is not
// part of one; neither is worth keeping.
if data.is_empty() {
continue;
}
let reaches = offset.saturating_add(data.len() as u64);
if reaches > MAX_HELLO as u64
|| entry.fragments.len() >= MAX_FRAGMENTS
|| entry.held + data.len() > MAX_HELLO
{
break;
}
entry.held += data.len();
entry.fragments.push((offset, data));
grew = true;
}
if !grew {
return;
}
if let Some(alpn) = client_hello_alpn(&entry.fragments) {
entry.alpn = Some(alpn);
entry.fragments.clear();
self.arrived.notify_waiters();
}
}
fn take(&self, from: SocketAddr) -> Option<Vec<Vec<u8>>> {
let mut table = self.pending.lock().ok()?;
if table.by_client.get(&from)?.alpn.is_some() {
return table.by_client.remove(&from).and_then(|p| p.alpn);
}
None
}
/// The protocols the client at `from` offered, waiting up to `wait` for the rest of a
/// hello split across datagrams. `None` when it could not be read.
pub async fn offered(&self, from: SocketAddr, wait: Duration) -> Option<Vec<Vec<u8>>> {
let deadline = tokio::time::Instant::now() + wait;
loop {
// Registered before looking, so a hello completed in between is not missed.
let notified = self.arrived.notified();
tokio::pin!(notified);
notified.as_mut().enable();
if let Some(alpn) = self.take(from) {
return Some(alpn);
}
if tokio::time::timeout_at(deadline, notified).await.is_err() {
return self.take(from);
}
}
}
}
/// The socket a relay's endpoint receives through, reading hellos on the way past.
#[derive(Debug)]
pub struct SniffingSocket {
inner: Arc<dyn AsyncUdpSocket>,
hellos: Arc<Hellos>,
}
impl SniffingSocket {
pub fn new(inner: Arc<dyn AsyncUdpSocket>, hellos: Arc<Hellos>) -> Self {
Self { inner, hellos }
}
}
impl AsyncUdpSocket for SniffingSocket {
fn create_io_poller(self: Arc<Self>) -> Pin<Box<dyn UdpPoller>> {
Arc::clone(&self.inner).create_io_poller()
}
fn try_send(&self, transmit: &Transmit) -> io::Result<()> {
self.inner.try_send(transmit)
}
fn poll_recv(
&self,
cx: &mut Context,
bufs: &mut [IoSliceMut<'_>],
meta: &mut [RecvMeta],
) -> Poll<io::Result<usize>> {
let polled = self.inner.poll_recv(cx, bufs, meta);
if let Poll::Ready(Ok(count)) = &polled {
for (buf, meta) in bufs.iter().zip(meta.iter()).take(*count) {
let received = &buf[..meta.len.min(buf.len())];
// Several datagrams of `stride` bytes each where the kernel coalesced them.
let stride = if meta.stride == 0 { received.len() } else { meta.stride };
for datagram in received.chunks(stride.max(1)) {
if is_long_header(datagram) {
self.hellos.observe(meta.addr, datagram);
}
}
}
}
polled
}
fn local_addr(&self) -> io::Result<SocketAddr> {
self.inner.local_addr()
}
fn max_transmit_segments(&self) -> usize {
self.inner.max_transmit_segments()
}
fn max_receive_segments(&self) -> usize {
self.inner.max_receive_segments()
}
fn may_fragment(&self) -> bool {
self.inner.may_fragment()
}
}
fn is_long_header(datagram: &[u8]) -> bool {
datagram.first().is_some_and(|b| b & 0x80 != 0)
}
/// One Initial packet, decrypted: what it was protected with, and its CRYPTO frames.
struct Initial {
/// How many bytes of the datagram it took, so a packet coalesced behind it is found.
length: usize,
dcid: Vec<u8>,
crypto: Vec<(u64, Vec<u8>)>,
}
fn varint(buf: &[u8], pos: &mut usize) -> Option<u64> {
let first = *buf.get(*pos)?;
let len = 1usize << (first >> 6);
let bytes = buf.get(*pos..*pos + len)?;
let mut value = u64::from(first & 0x3f);
for byte in &bytes[1..] {
value = (value << 8) | u64::from(*byte);
}
*pos += len;
Some(value)
}
/// Decrypt the Initial packet at the start of `packet`, if that is what it is.
fn decrypt_initial(suite: &Suite, packet: &[u8]) -> Option<Initial> {
let first = *packet.first()?;
if first & 0x80 == 0 {
return None;
}
let version = u32::from_be_bytes(packet.get(1..5)?.try_into().ok()?);
let (version, initial_type) = match version {
0x0000_0001 => (Version::V1, 0),
0x6b33_43cf => (Version::V2, 1),
0xff00_001d..=0xff00_0020 => (Version::V1Draft, 0),
_ => return None,
};
if (first >> 4) & 0x03 != initial_type {
return None;
}
let mut pos = 5;
let dcid_len = usize::from(*packet.get(pos)?);
let dcid = packet.get(pos + 1..pos + 1 + dcid_len)?.to_vec();
pos += 1 + dcid_len;
let scid_len = usize::from(*packet.get(pos)?);
pos += 1 + scid_len;
let token_len = usize::try_from(varint(packet, &mut pos)?).ok()?;
pos = pos.checked_add(token_len)?;
let length = usize::try_from(varint(packet, &mut pos)?).ok()?;
let pn_offset = pos;
let end = pn_offset.checked_add(length)?;
if end > packet.len() {
return None;
}
let keys = suite.keys(&dcid, Side::Server, version);
let sample_len = keys.remote.header.sample_len();
let sample = packet.get(pn_offset + 4..pn_offset + 4 + sample_len)?.to_vec();
let mut buf = packet[..end].to_vec();
let (head, tail) = buf.split_at_mut(1);
let pn_field = tail.get_mut(pn_offset - 1..pn_offset + 3)?;
keys.remote
.header
.decrypt_in_place(&sample, &mut head[0], pn_field)
.ok()?;
let pn_len = usize::from(buf[0] & 0x03) + 1;
let mut number = 0u64;
for byte in &buf[pn_offset..pn_offset + pn_len] {
number = (number << 8) | u64::from(*byte);
}
let (header, payload) = buf.split_at_mut(pn_offset + pn_len);
let plain = keys
.remote
.packet
.decrypt_in_place(number, header, payload)
.ok()?;
Some(Initial {
length: end,
dcid,
crypto: crypto_frames(plain)?,
})
}
/// The CRYPTO frames of a decrypted Initial payload. `None` for a frame an Initial cannot
/// carry, which means this was not what it looked like.
fn crypto_frames(payload: &[u8]) -> Option<Vec<(u64, Vec<u8>)>> {
let mut frames = Vec::new();
let mut pos = 0;
while pos < payload.len() {
match varint(payload, &mut pos)? {
0x00 | 0x01 => {} // PADDING, PING
kind @ (0x02 | 0x03) => {
// ACK: largest, delay, range count, first range, the ranges, and ECN.
varint(payload, &mut pos)?;
varint(payload, &mut pos)?;
let ranges = varint(payload, &mut pos)?;
varint(payload, &mut pos)?;
for _ in 0..ranges {
varint(payload, &mut pos)?;
varint(payload, &mut pos)?;
}
if kind == 0x03 {
for _ in 0..3 {
varint(payload, &mut pos)?;
}
}
}
0x06 => {
let offset = varint(payload, &mut pos)?;
let len = usize::try_from(varint(payload, &mut pos)?).ok()?;
let data = payload.get(pos..pos.checked_add(len)?)?;
frames.push((offset, data.to_vec()));
pos += len;
}
// CONNECTION_CLOSE ends what there is to read.
0x1c => break,
_ => return None,
}
}
Some(frames)
}
/// The ALPN list of a ClientHello, once enough of it has arrived to have one.
///
/// `Some(vec![])` for a complete hello that offers none — which QUIC does not allow, and
/// which the handshake is left to refuse.
fn client_hello_alpn(fragments: &[(u64, Vec<u8>)]) -> Option<Vec<Vec<u8>>> {
// What has arrived contiguously from the start, in whatever order it came.
let mut sorted: Vec<&(u64, Vec<u8>)> = fragments.iter().collect();
sorted.sort_by_key(|(offset, _)| *offset);
let mut hello: Vec<u8> = Vec::new();
for (offset, data) in sorted {
let offset = usize::try_from(*offset).ok()?;
if offset > hello.len() {
break;
}
let overlap = hello.len() - offset;
if overlap < data.len() {
hello.extend_from_slice(&data[overlap..]);
}
}
if hello.len() < 4 || hello[0] != 0x01 {
return None;
}
let len = (usize::from(hello[1]) << 16) | (usize::from(hello[2]) << 8) | usize::from(hello[3]);
let body = hello.get(4..4 + len)?;
Some(alpn_of(body).unwrap_or_default())
}
fn alpn_of(body: &[u8]) -> Option<Vec<Vec<u8>>> {
let u16_at = |pos: usize| -> Option<usize> {
Some(usize::from(u16::from_be_bytes(body.get(pos..pos + 2)?.try_into().ok()?)))
};
// Legacy version and random, then the session id, the suites and the compressions.
let mut pos = 2 + 32;
pos += 1 + usize::from(*body.get(pos)?);
pos += 2 + u16_at(pos)?;
pos += 1 + usize::from(*body.get(pos)?);
let end = pos.checked_add(2 + u16_at(pos)?)?;
pos += 2;
while pos + 4 <= end {
let kind = u16_at(pos)?;
let len = u16_at(pos + 2)?;
let data = body.get(pos + 4..pos + 4 + len)?;
pos += 4 + len;
if kind != 16 {
continue;
}
let mut names = Vec::new();
let mut at = 2;
while at < data.len() {
let name_len = usize::from(data[at]);
names.push(data.get(at + 1..at + 1 + name_len)?.to_vec());
at += 1 + name_len;
}
return Some(names);
}
Some(Vec::new())
}
#[cfg(test)]
mod tests {
use super::*;
fn push_varint(out: &mut Vec<u8>, value: u64) {
if value < 64 {
out.push(value as u8);
} else if value < 16384 {
out.extend_from_slice(&(0x4000 | value as u16).to_be_bytes());
} else {
out.extend_from_slice(&(0x8000_0000 | value as u32).to_be_bytes());
}
}
/// A ClientHello offering `alpn`, and nothing else a parser would need.
fn hello_with(alpn: &[&[u8]]) -> Vec<u8> {
let mut names = Vec::new();
for name in alpn {
names.push(name.len() as u8);
names.extend_from_slice(name);
}
let mut ext = vec![0x00, 0x10];
ext.extend_from_slice(&((names.len() + 2) as u16).to_be_bytes());
ext.extend_from_slice(&(names.len() as u16).to_be_bytes());
ext.extend_from_slice(&names);
let mut body = vec![0x03, 0x03];
body.extend_from_slice(&[0u8; 32]);
body.push(0); // no session id
body.extend_from_slice(&[0x00, 0x02, 0x13, 0x01]); // one suite
body.extend_from_slice(&[0x01, 0x00]); // no compression
body.extend_from_slice(&(ext.len() as u16).to_be_bytes());
body.extend_from_slice(&ext);
let mut hello = vec![0x01];
hello.extend_from_slice(&(body.len() as u32).to_be_bytes()[1..]);
hello.extend_from_slice(&body);
hello
}
/// A client Initial carrying these CRYPTO fragments, protected the way a client
/// protects one — with rustls's own client-side keys, so what is read back is checked
/// against the other half of the library rather than against itself.
fn client_initial(dcid: &[u8], number: u64, crypto: &[(u64, &[u8])]) -> Vec<u8> {
let keys = Hellos::new().suite.keys(dcid, Side::Client, Version::V1);
let mut payload = Vec::new();
for (offset, data) in crypto {
payload.push(0x06);
push_varint(&mut payload, *offset);
push_varint(&mut payload, data.len() as u64);
payload.extend_from_slice(data);
}
payload.push(0x01); // a PING, as Chrome scatters among the fragments
payload.resize(payload.len().max(1100), 0); // PADDING, to the size a client sends
let mut packet = vec![0xc1, 0, 0, 0, 1, dcid.len() as u8];
packet.extend_from_slice(dcid);
packet.extend_from_slice(&[0, 0]); // no source id, no token
packet.extend_from_slice(&(0x4000 | (2 + payload.len() + 16) as u16).to_be_bytes());
let pn_offset = packet.len();
packet.extend_from_slice(&(number as u16).to_be_bytes());
let tag = keys.local.packet.encrypt_in_place(number, &packet, &mut payload).unwrap();
packet.extend_from_slice(&payload);
packet.extend_from_slice(tag.as_ref());
let sample = packet[pn_offset + 4..pn_offset + 20].to_vec();
let (first, rest) = packet.split_at_mut(1);
keys.local
.header
.encrypt_in_place(&sample, &mut first[0], &mut rest[pn_offset - 1..pn_offset + 1])
.unwrap();
packet
}
fn from() -> SocketAddr {
"192.0.2.1:4433".parse().unwrap()
}
#[test]
fn a_protected_initial_is_read_for_what_it_offers() {
let hellos = Hellos::new();
let hello = hello_with(&[b"h3", b"fgex-test"]);
hellos.observe(from(), &client_initial(b"\x83\x94\xc8\xf0\x3e\x51\x57\x08", 0, &[(0, &hello)]));
assert_eq!(hellos.take(from()), Some(vec![b"h3".to_vec(), b"fgex-test".to_vec()]));
}
/// Post-quantum key shares make a ClientHello bigger than one packet, and clients
/// scatter its pieces: two Initials, the second one first.
#[test]
fn a_hello_across_two_packets_arriving_backwards_is_reassembled() {
let hellos = Hellos::new();
let hello = hello_with(&[b"doq"]);
let (a, b) = hello.split_at(30);
let dcid = b"\x01\x02\x03\x04\x05\x06\x07\x08";
hellos.observe(from(), &client_initial(dcid, 1, &[(30, b)]));
assert_eq!(hellos.take(from()), None, "half a hello was read as a whole one");
hellos.observe(from(), &client_initial(dcid, 0, &[(0, a)]));
assert_eq!(hellos.take(from()), Some(vec![b"doq".to_vec()]));
}
/// A Retry answered is a new Initial under a new connection ID, and what was gathered
/// under the old one is not mixed into it.
#[test]
fn a_new_connection_id_starts_again() {
let hellos = Hellos::new();
let old = hello_with(&[b"old"]);
let new = hello_with(&[b"new"]);
hellos.observe(from(), &client_initial(b"\x11\x11\x11\x11\x11\x11\x11\x11", 0, &[(0, &old[..20])]));
hellos.observe(from(), &client_initial(b"\x22\x22\x22\x22\x22\x22\x22\x22", 0, &[(0, &new)]));
assert_eq!(hellos.take(from()), Some(vec![b"new".to_vec()]));
}
#[test]
fn anything_but_an_initial_is_passed_over() {
let hellos = Hellos::new();
hellos.observe(from(), &[0x40, 1, 2, 3, 4, 5, 6, 7]);
hellos.observe(from(), &[0xe0, 0, 0, 0, 1, 0, 0]);
let mut tampered = client_initial(b"\x09\x09\x09\x09\x09\x09\x09\x09", 0, &[(0, &hello_with(&[b"h3"]))]);
let last = tampered.len() - 1;
tampered[last] ^= 0xff;
hellos.observe(from(), &tampered);
assert_eq!(hellos.take(from()), None);
}
fn kept_for(hellos: &Hellos, client: SocketAddr) -> Option<usize> {
hellos.pending.lock().unwrap().by_client.get(&client).map(|p| p.fragments.len())
}
/// Empty CRYPTO frames cost nothing to send and were counted by nothing: an Initial
/// full of them grew one client's hello by hundreds of pieces per packet, without end,
/// and every packet sorted all of them again inside the endpoint's receive path.
#[test]
fn empty_pieces_are_not_kept() {
let hellos = Hellos::new();
let empties: Vec<(u64, &[u8])> = (0..300u64).map(|i| (i, &b""[..])).collect();
let dcid = b"\x31\x32\x33\x34\x35\x36\x37\x38";
for number in 0..50 {
hellos.observe(from(), &client_initial(dcid, number, &empties));
}
assert_eq!(kept_for(&hellos, from()), Some(0));
}
/// Pieces that never join up from the start are held, but only so many of them.
#[test]
fn a_hello_is_kept_in_a_bounded_number_of_pieces() {
let hellos = Hellos::new();
let dcid = b"\x41\x42\x43\x44\x45\x46\x47\x48";
for number in 0..200u64 {
hellos.observe(from(), &client_initial(dcid, number, &[(100 + number * 2, b"x")]));
}
let kept = kept_for(&hellos, from()).unwrap();
assert!(kept <= MAX_FRAGMENTS, "{kept} pieces kept for one hello");
}
#[test]
fn a_piece_beyond_any_hello_is_not_kept() {
let hellos = Hellos::new();
let dcid = b"\x51\x52\x53\x54\x55\x56\x57\x58";
hellos.observe(from(), &client_initial(dcid, 0, &[(MAX_HELLO as u64 + 1, b"far")]));
assert_eq!(kept_for(&hellos, from()), Some(0));
}
/// A table full of hellos nobody finished makes room rather than turning newcomers
/// away: refusing them meant a thousand forged Initials every few seconds stopped
/// every real client's hello from being read.
#[test]
fn a_full_table_makes_room_for_the_next_client() {
let hellos = Hellos::new();
let unfinished = hello_with(&[b"h3"]);
for n in 0..MAX_PENDING as u32 {
let forged = SocketAddr::from(([10, (n >> 16) as u8, (n >> 8) as u8, n as u8], 443));
let dcid = n.to_be_bytes().repeat(2);
hellos.observe(forged, &client_initial(&dcid, 0, &[(0, &unfinished[..10])]));
}
assert_eq!(hellos.pending.lock().unwrap().by_client.len(), MAX_PENDING);
let hello = hello_with(&[b"fgex-late"]);
hellos.observe(from(), &client_initial(b"\x61\x62\x63\x64\x65\x66\x67\x68", 0, &[(0, &hello)]));
assert_eq!(hellos.take(from()), Some(vec![b"fgex-late".to_vec()]));
assert!(hellos.pending.lock().unwrap().by_client.len() <= MAX_PENDING);
}
#[tokio::test]
async fn a_hello_still_arriving_is_waited_for() {
let hellos = Arc::new(Hellos::new());
let hello = hello_with(&[b"h3"]);
let packet = client_initial(b"\x0a\x0b\x0c\x0d\x0e\x0f\x10\x11", 0, &[(0, &hello)]);
let later = Arc::clone(&hellos);
tokio::spawn(async move {
tokio::time::sleep(Duration::from_millis(50)).await;
later.observe(from(), &packet);
});
assert_eq!(hellos.offered(from(), Duration::from_secs(2)).await, Some(vec![b"h3".to_vec()]));
assert_eq!(hellos.offered(from(), Duration::from_millis(50)).await, None);
}
}

39
backend/proxysrc/src/relays.rs Normal file
View File

@ -0,0 +1,39 @@
//! The per-address relays a UDP-family service is made of.
//!
//! Two kinds, one question. `SO_ORIGINAL_DST` is TCP and SCTP only, so neither a
//! datagram relay nor a QUIC endpoint can recover where traffic was headed: both are
//! bound one per protected address with the upstream already known, and both answer the
//! backend with the port it should point that address's rule at. Which of the two a
//! service gets is decided once, by what the service speaks.
//!
//! An enum rather than a trait: there are two, the set is not open, and the control
//! channel only ever asks them the one thing.
use std::io;
use std::net::SocketAddr;
use crate::quic::QuicManager;
use crate::udp::UdpManager;
#[derive(Clone)]
pub enum Relays {
/// Datagrams relayed as they are, each client's flow filtered on its own.
Datagram(UdpManager),
/// QUIC, terminated: the streams inside are what the filters see.
Quic(QuicManager),
}
impl Relays {
/// Bind a relay for one protected address, or report the one already bound.
pub async fn add_relay(
&self,
upstream: SocketAddr,
onward: crate::proxy::Onward,
) -> io::Result<u16> {
match self {
// A plain datagram relay forwards bytes; there is nothing for it to speak.
Relays::Datagram(manager) => manager.add_relay(upstream).await,
Relays::Quic(manager) => manager.add_relay(upstream, onward).await,
}
}
}

256
backend/proxysrc/src/report.rs Normal file
View File

@ -0,0 +1,256 @@
//! Every line this engine sends the backend, written by a thread of its own.
//!
//! stdout is a pipe the backend reads, and a pipe fills. Each refused connection used to
//! be a `println!("BLOCKED …")` from the thread that refused it — so when the backend fell
//! behind, which a flood of refused connections makes it do, the pipe filled and those
//! writes blocked: the runtime threads carrying every other connection sat waiting on it.
//! Measured, 5000 refused connections took 74 seconds to get through, and a legitimate
//! client meanwhile waited a median 4.5 seconds and half the time not at all — an attacker
//! slowing a service to a halt by sending it exactly what firegex was there to refuse.
//!
//! So nothing on the datapath writes to stdout any more. Lines go into a bounded queue and
//! a thread of their own writes them out; a report of what happened (`BLOCKED`, `STATS`)
//! is dropped and counted when the queue is full, and a reply the backend is waiting for
//! (`PORT`, `UDP`, `ACK`) waits for room, so it is never lost and never overtakes a line
//! sent before it.
use std::io::Write;
use std::sync::atomic::{AtomicU64, Ordering};
use std::sync::mpsc::{sync_channel, SyncSender, TrySendError};
use std::sync::OnceLock;
use std::time::{Duration, Instant};
/// How many lines may wait for the backend. At a few dozen bytes each this is a megabyte;
/// what it buys is that a burst of refusals is reported whole rather than cut off.
const QUEUE: usize = 32 * 1024;
/// How often a count of reports that had to be dropped is written out.
const DROPS_SAID_EVERY: Duration = Duration::from_secs(5);
static SENDER: OnceLock<SyncSender<String>> = OnceLock::new();
static DROPPED: AtomicU64 = AtomicU64::new(0);
fn sender() -> &'static SyncSender<String> {
SENDER.get_or_init(|| {
let (tx, rx) = sync_channel::<String>(QUEUE);
std::thread::Builder::new()
.name("fgex-report".to_string())
.spawn(move || {
let stdout = std::io::stdout();
let mut said = Instant::now();
while let Ok(first) = rx.recv() {
let mut out = stdout.lock();
let _ = writeln!(out, "{first}");
// Whatever else is already waiting goes in the same write.
while let Ok(line) = rx.try_recv() {
let _ = writeln!(out, "{line}");
}
let _ = out.flush();
drop(out);
if said.elapsed() >= DROPS_SAID_EVERY {
let dropped = DROPPED.swap(0, Ordering::Relaxed);
if dropped > 0 {
eprintln!(
"[warn] [report] {dropped} refusals were not reported to the \
backend: it was not reading fast enough. They were refused all \
the same; only the counters and the log are short of them."
);
}
said = Instant::now();
}
}
})
.expect("cannot start the thread that talks to the backend");
tx
})
}
/// A reply the backend is waiting for. Waits for room rather than being dropped.
pub fn reply(line: String) {
let _ = sender().send(line);
}
/// A report of something that happened. Never waits: dropped and counted when the backend
/// is too far behind, so the datapath is never held up by it.
pub fn event(line: String) {
if let Err(TrySendError::Full(_)) = sender().try_send(line) {
DROPPED.fetch_add(1, Ordering::Relaxed);
}
}
/// A rule refused a connection.
pub fn blocked(id: &str) {
event(format!("BLOCKED {id}"));
}
// --- what the datapath says about itself ------------------------------------------------
//
// stderr is a pipe the backend reads too, and it fills the same way stdout did. Worse, it
// is written straight from the threads carrying traffic: a line per connection that ended
// with an error, or per client that spoke the wrong thing at a TLS address, was a write
// every runtime thread queued up on — `eprintln!` takes one lock for the whole process —
// and once the backend fell behind, one that blocked them all. Several of those lines are
// the client's to trigger at will. So the datapath's diagnostics go through a queue and a
// thread of their own, exactly as its reports do, and the ones a client can cause as often
// as it likes are counted rather than written past a few.
static DIAG: OnceLock<SyncSender<String>> = OnceLock::new();
static DIAG_DROPPED: AtomicU64 = AtomicU64::new(0);
fn diag_sender() -> &'static SyncSender<String> {
DIAG.get_or_init(|| {
let (tx, rx) = sync_channel::<String>(QUEUE);
std::thread::Builder::new()
.name("fgex-diag".to_string())
.spawn(move || {
let stderr = std::io::stderr();
while let Ok(first) = rx.recv() {
let mut out = stderr.lock();
let _ = writeln!(out, "{first}");
while let Ok(line) = rx.try_recv() {
let _ = writeln!(out, "{line}");
}
let dropped = DIAG_DROPPED.swap(0, Ordering::Relaxed);
if dropped > 0 {
let _ = writeln!(
out,
"[warn] [report] {dropped} diagnostic line(s) were dropped: the \
backend was not reading them fast enough"
);
}
let _ = out.flush();
}
})
.expect("cannot start the thread that writes the engine's diagnostics");
tx
})
}
/// A line about the datapath's own health. Never waits: dropped and counted when the
/// backend is too far behind, so nothing carrying traffic is held up by it.
pub fn diag(line: String) {
if let Err(TrySendError::Full(_)) = diag_sender().try_send(line) {
DIAG_DROPPED.fetch_add(1, Ordering::Relaxed);
}
}
/// Milliseconds on a monotonic clock, never zero.
fn now_ms() -> u64 {
static START: OnceLock<Instant> = OnceLock::new();
START.get_or_init(Instant::now).elapsed().as_millis() as u64 + 1
}
/// At most `per_window` lines per window from one place in the code; the rest counted and
/// said as one line when the next window opens.
///
/// For the lines a client can make the engine write as often as it likes — a connection
/// ending badly, a cleartext client at a TLS address, a malformed request. The first few
/// of a window are what an operator needs to see what is happening; the count says how
/// much of it there is; the thousands in between would only push everything else out of
/// a log that holds a few hundred lines.
pub struct Throttle {
/// Says what was held back, tag included: `[warn] [proxy] cleartext clients at a TLS
/// address`, so the summary is classified like the lines it stands for.
what: &'static str,
per_window: u64,
window_ms: u64,
opened_ms: AtomicU64,
said: AtomicU64,
held: AtomicU64,
}
impl Throttle {
pub const fn new(what: &'static str, per_window: u64, window_secs: u64) -> Self {
Self {
what,
per_window,
window_ms: window_secs * 1000,
opened_ms: AtomicU64::new(0),
said: AtomicU64::new(0),
held: AtomicU64::new(0),
}
}
pub fn say(&self, line: impl FnOnce() -> String) {
self.say_at(now_ms(), line, &mut diag);
}
fn say_at(&self, now: u64, line: impl FnOnce() -> String, out: &mut dyn FnMut(String)) {
let opened = self.opened_ms.load(Ordering::Relaxed);
if now.saturating_sub(opened) >= self.window_ms
&& self
.opened_ms
.compare_exchange(opened, now, Ordering::Relaxed, Ordering::Relaxed)
.is_ok()
{
self.said.store(0, Ordering::Relaxed);
let held = self.held.swap(0, Ordering::Relaxed);
if held > 0 {
out(format!(
"{}: {held} more like this in the last {}s, not written",
self.what,
now.saturating_sub(opened) / 1000,
));
}
}
if self.said.fetch_add(1, Ordering::Relaxed) < self.per_window {
out(line());
} else {
self.held.fetch_add(1, Ordering::Relaxed);
}
}
}
/// `eprintln!` for the datapath: through the queue, never blocking. See [`diag`].
#[macro_export]
macro_rules! diag {
($($arg:tt)*) => {
$crate::report::diag(format!($($arg)*))
};
}
/// [`diag!`], at most `per` times every `secs` seconds from this call site. See [`Throttle`].
#[macro_export]
macro_rules! diag_throttled {
($what:expr, $per:expr, $secs:expr, $($arg:tt)*) => {{
static THROTTLE: $crate::report::Throttle =
$crate::report::Throttle::new($what, $per, $secs);
THROTTLE.say(|| format!($($arg)*));
}};
}
#[cfg(test)]
mod tests {
use super::*;
fn run(throttle: &Throttle, at: u64, n: usize) -> Vec<String> {
let mut said = Vec::new();
for i in 0..n {
throttle.say_at(at, || format!("line {i}"), &mut |line| said.push(line));
}
said
}
#[test]
fn a_window_says_its_first_lines_and_holds_the_rest() {
let throttle = Throttle::new("[info] [test] things", 3, 10);
assert_eq!(run(&throttle, 50_000, 100), vec!["line 0", "line 1", "line 2"]);
}
#[test]
fn what_was_held_is_counted_when_the_next_window_opens() {
let throttle = Throttle::new("[warn] [test] things", 2, 10);
run(&throttle, 50_000, 7);
let said = run(&throttle, 61_000, 1);
assert_eq!(said.len(), 2, "{said:?}");
assert!(said[0].starts_with("[warn] [test] things: 5 more"), "{said:?}");
assert_eq!(said[1], "line 0");
}
#[test]
fn a_quiet_window_says_nothing_about_holding_anything_back() {
let throttle = Throttle::new("[info] [test] things", 2, 10);
run(&throttle, 50_000, 1);
assert_eq!(run(&throttle, 70_000, 1), vec!["line 0"]);
}
}

344
backend/proxysrc/src/rules.rs Normal file
View File

@ -0,0 +1,344 @@
//! The filters the backend actually configures, and the wire format that carries them.
//!
//! One JSON line in, the whole ruleset at once — the same shape `cppregex` uses when
//! it takes its filter codes on stdin. Replacing the set wholesale means there is no
//! partial state to reason about, and [`crate::filter::ChainHandle`] swaps it under
//! live connections so a rule change costs nobody their connection.
use std::sync::Arc;
use std::time::Duration;
use serde::Deserialize;
use crate::filter::{Direction, Filter, FilterCtx, FilterSession, Verdict};
use crate::hyperscan;
use crate::pyworker::PyWorkerRule;
/// Which half of the traffic a rule looks at.
#[derive(Clone, Copy, Debug, Deserialize, PartialEq, Eq, Default)]
#[serde(rename_all = "lowercase")]
pub enum RuleDirection {
#[default]
Both,
/// Client to server: requests.
C2s,
/// Server to client: responses, where a leaked flag would show up.
S2c,
}
impl RuleDirection {
fn label(self) -> &'static str {
match self {
RuleDirection::Both => "both ways",
RuleDirection::C2s => "c->s",
RuleDirection::S2c => "s->c",
}
}
fn covers(self, dir: Direction) -> bool {
match self {
RuleDirection::Both => true,
RuleDirection::C2s => dir == Direction::ClientToServer,
RuleDirection::S2c => dir == Direction::ServerToClient,
}
}
}
// A matching pattern refuses the connection. There is no second action.
//
// could not keep its promise on a stream. Rewriting scanned one chunk at a time — bytes
// already forwarded cannot be taken back, so a match straddling two chunks was never
// rewritable — and the failure was silent: no block, no log, nothing to see. Measured,
// a pattern sent in one write reached the service rewritten and the same pattern split
// across two segments reached it untouched. Blocking scans in stream mode and caught
// both. Whether a rule fires should not depend on how the sender happened to segment.
/// One rule as the backend sends it.
#[derive(Clone, Debug, Deserialize)]
#[serde(tag = "kind", rename_all = "lowercase")]
pub enum RuleSpec {
/// A pattern, and what to do where it matches.
Regex {
id: String,
/// Which filter this pattern belongs to.
///
/// It decides what shares a compiled database, and therefore what the operator
/// ordered: patterns of one filter are one thing they placed, and two filters
/// are two. Grouping by "whatever arrived next" would silently merge two filters
/// the operator deliberately separated.
#[serde(default)]
filter: String,
pattern: String,
#[serde(default)]
direction: RuleDirection,
#[serde(default = "default_true")]
case_sensitive: bool,
},
/// Hand the traffic to the user's Python, in a process of its own.
Python {
id: String,
/// File the worker executes. The backend writes it; the engine only runs it.
code_path: String,
/// Which of the file's `@pyfilter` functions are switched on.
///
/// Absent means every one the file defines; a list means exactly those, and an
/// empty list means none. The three are deliberately distinct: "not configured"
/// and "the operator switched everything off" want opposite answers, and
/// collapsing them would either run filters that were turned off or stop
/// running filters nobody touched.
#[serde(default)]
enabled: Option<Vec<String>>,
/// How to start the worker. Configured rather than hardcoded so a standalone
/// install can point at its own interpreter.
#[serde(default)]
command: Vec<String>,
#[serde(default = "default_python_timeout_ms")]
timeout_ms: u64,
},
}
fn default_true() -> bool {
true
}
/// Generous compared to the chain's own deadline: this one has a process start-up
/// behind it the first time round.
fn default_python_timeout_ms() -> u64 {
1000
}
/// Every blocking pattern that applies to one direction, matched in a single pass.
///
/// This is hyperscan, the same library `cppregex` matches with on the NFQUEUE side —
/// not a second engine that happens to accept similar syntax. One engine is what lets
/// a pattern mean the same thing whichever transport carries it, and what lets the
/// in-app debugger show the operator what will actually happen.
///
/// One database per direction rather than one per rule: hyperscan merges the automata,
/// so scanning for fifty patterns costs about what scanning for one costs. That is the
/// property a per-rule loop gave away, and a benchmark against nfregex made the loss
/// impossible to miss.
pub struct HyperscanRule {
db: Arc<hyperscan::Database>,
/// Parallel to the database's pattern ids: which rule each index belongs to.
ids: Arc<Vec<String>>,
label: String,
direction: RuleDirection,
}
impl Filter for HyperscanRule {
fn name(&self) -> &str {
&self.label
}
fn may_block(&self) -> bool {
false
}
fn inspect(&self, _ctx: &FilterCtx<'_>) -> Verdict {
// Only reached for a direction this rule does not cover, or when opening the
// scanner failed. Both mean "this rule has nothing to say about this chunk",
// and a rule with nothing to say must never be the reason traffic stops.
Verdict::Accept
}
fn open_session(&self, dir: Direction) -> Option<Box<dyn FilterSession>> {
if !self.direction.covers(dir) {
return None;
}
match hyperscan::StreamScanner::open(Arc::clone(&self.db)) {
Ok(scanner) => Some(Box::new(HyperscanSession {
scanner,
ids: Arc::clone(&self.ids),
})),
// Fail open, loudly. The alternative is refusing connections because the
// engine could not allocate scratch space, which is the traffic paying
// for our problem.
Err(e) => {
crate::diag_throttled!("[warn] [filter] connections a scanner could not be opened for", 5, 10, "[warn] [filter] '{}' could not open a scanner ({e}): this connection goes unfiltered", self.label);
None
}
}
}
}
/// One connection's position in the stream, for one direction.
struct HyperscanSession {
scanner: hyperscan::StreamScanner,
ids: Arc<Vec<String>>,
}
impl FilterSession for HyperscanSession {
fn inspect(&mut self, ctx: &FilterCtx<'_>) -> Verdict {
match self.scanner.scan(ctx.chunk) {
Ok(Some(index)) => {
// The id travels with the verdict rather than being stashed on the
// rule, so two connections blocking at the same instant cannot be
// credited to each other's pattern.
let id = self
.ids
.get(index as usize)
.cloned()
.unwrap_or_else(|| "unknown".to_string());
// Reset the stream so that subsequent scans on this session (e.g. further UDP datagrams
// or connections) don't fail due to HS_SCAN_TERMINATED.
let _ = self.scanner.reset();
Verdict::Reject(Some(id))
}
Ok(None) => Verdict::Accept,
// A scan that errors has told us nothing about the chunk, so the chunk
// goes through. Reset the scanner so future chunks/datagrams can still be scanned.
Err(e) => {
let _ = self.scanner.reset();
crate::diag_throttled!("[warn] [filter] scans that failed", 5, 10, "[warn] [filter] scan failed ({e}): scanner reset, chunk forwarded");
Verdict::Accept
}
}
}
}
/// Parse a whole ruleset from one JSON line.
///
/// All or nothing: a single bad rule rejects the update and leaves the running chain
/// untouched, because half a ruleset is worse than the previous one.
pub fn parse_ruleset(line: &str) -> Result<Vec<Arc<dyn Filter>>, String> {
parse_ruleset_reusing(line, &[])
}
/// [`parse_ruleset`], carrying over any filter of `previous` that the new ruleset
/// describes exactly as it was (`Filter::reuse_key`).
///
/// A ruleset is sent whole on every edit, so without this a pattern changed in one
/// filter restarted the Python worker of another, and the new worker met every open
/// connection in the middle: its module globals were gone, its HTTP parser started on
/// the second half of whatever request was in flight, and the addresses it had been told
/// when each connection opened were never told again.
pub fn parse_ruleset_reusing(
line: &str,
previous: &[Arc<dyn Filter>],
) -> Result<Vec<Arc<dyn Filter>>, String> {
let specs: Vec<RuleSpec> =
serde_json::from_str(line).map_err(|e| format!("malformed ruleset: {e}"))?;
// The order the backend sent is the order the operator set, and it is kept. An
// Python — which quietly turned the chain into a set, so a filter placed after
// another could run before it. Nothing visible said so until a test asked whether
// a pattern could catch what the filter before it had written.
//
// Patterns of one filter still share a database, because that is what makes matching
// cost the same at fifty rules as at one. A group is exactly a filter, so grouping
// never crosses a boundary the operator can see.
let mut out: Vec<Arc<dyn Filter>> = Vec::new();
let mut pending: Vec<(String, RuleDirection, hyperscan::Pattern)> = Vec::new();
let mut pending_filter: Option<String> = None;
fn flush(
pending: &mut Vec<(String, RuleDirection, hyperscan::Pattern)>,
out: &mut Vec<Arc<dyn Filter>>,
) -> Result<(), String> {
if pending.is_empty() {
return Ok(());
}
for direction in [RuleDirection::C2s, RuleDirection::S2c] {
let selected: Vec<_> = pending
.iter()
.filter(|(_, d, _)| {
d.covers(match direction {
RuleDirection::C2s => Direction::ClientToServer,
_ => Direction::ServerToClient,
})
})
.collect();
if selected.is_empty() {
continue;
}
let patterns: Vec<hyperscan::Pattern> = selected
.iter()
.map(|(_, _, p)| hyperscan::Pattern {
expr: p.expr.clone(),
case_sensitive: p.case_sensitive,
})
.collect();
let ids: Vec<String> =
selected.iter().map(|(id, _, _)| id.clone()).collect();
let db = hyperscan::Database::compile(&patterns, hyperscan::Mode::Stream)
.map_err(|e| format!("cannot build the rule set: {e}"))?;
out.push(Arc::new(HyperscanRule {
label: format!("{} blocking rule(s) {}", ids.len(), direction.label()),
db: Arc::new(db),
ids: Arc::new(ids),
direction,
}));
}
pending.clear();
Ok(())
}
for spec in specs {
match spec {
RuleSpec::Regex {
id,
filter,
pattern,
direction,
case_sensitive,
} => {
// A different filter is a different thing the operator placed, so it
// ends the group before it and takes its own position.
if pending_filter.as_deref() != Some(filter.as_str()) {
flush(&mut pending, &mut out)?;
pending_filter = Some(filter.clone());
}
// Compiled alone first, only to be thrown away: hyperscan reports the
// first pattern that failed and stops, so a set that fails to build says
// nothing about which of the others were fine. The operator needs to know
// which rule to fix.
hyperscan::Database::validate(&pattern, case_sensitive, hyperscan::Mode::Stream)
.map_err(|e| format!("invalid regex '{pattern}': {e}"))?;
pending.push((
id,
direction,
hyperscan::Pattern { expr: pattern, case_sensitive },
));
}
RuleSpec::Python {
id,
code_path,
command,
enabled,
timeout_ms,
} => {
// A process is not a pattern, so it ends the run of patterns before it
// and takes its place in the order.
flush(&mut pending, &mut out)?;
pending_filter = None;
let command = if command.is_empty() {
// The engine has no idea where the backend installed the worker, so
// the backend says. The env var is the fallback for a hand-run
// engine; there is no sensible hardcoded path.
let worker = std::env::var("FGEX_PROXY_PYWORKER").map_err(|_| {
"a python rule needs `command`, or FGEX_PROXY_PYWORKER set".to_string()
})?;
vec!["python3".to_string(), worker]
} else {
command
};
let deadline = Duration::from_millis(timeout_ms);
let key = PyWorkerRule::key_for(&id, &command, &code_path, &enabled, deadline);
let kept = key.as_deref().and_then(|key| {
previous
.iter()
.find(|filter| filter.reuse_key() == Some(key))
.cloned()
});
out.push(match kept {
Some(filter) => filter,
None => Arc::new(
PyWorkerRule::new(id, command, code_path, enabled, deadline)
.reusable_as(key),
),
})
}
}
}
flush(&mut pending, &mut out)?;
Ok(out)
}

156
backend/proxysrc/src/spec.rs Normal file
View File

@ -0,0 +1,156 @@
//! Filters built from a text spec.
//!
//! Provisional: the real engine will take its filters from the backend. This exists
//! so the datapath's guarantees can be driven from outside the process — the Python
//! suite needs to be able to ask for a filter that panics, or one that never returns,
//! without linking against the crate.
use std::sync::atomic::{AtomicBool, Ordering};
use std::sync::Arc;
use std::time::Duration;
use crate::filter::{Direction, Filter, FilterCtx, FilterSession, Verdict};
/// Panics on every chunk. Stands in for user code with a bug in it.
pub struct PanicFilter {
name: String,
}
impl Filter for PanicFilter {
fn name(&self) -> &str {
&self.name
}
fn inspect(&self, _ctx: &FilterCtx<'_>) -> Verdict {
panic!("filter '{}' panicked on purpose", self.name);
}
}
/// Set only by [`release_hangs`], so a test can let abandoned filter threads exit.
/// The binary never sets it: for the Python suite the hang has to be genuine.
static HANG_RELEASE: AtomicBool = AtomicBool::new(false);
/// Let every [`HangFilter`] currently stuck in `inspect` return.
///
/// Needed because a blocking filter thread cannot be cancelled: the engine walks
/// away from it, but the thread lives until its own code decides to stop. Tests
/// would otherwise hang on runtime shutdown, which waits for the blocking pool —
/// the same reason a hung filter costs a pool thread until the process restarts.
pub fn release_hangs() {
HANG_RELEASE.store(true, Ordering::Relaxed);
}
/// Never returns on its own. Stands in for user code that deadlocks or waits on
/// something that will not happen — the failure mode `catch_unwind` cannot help with.
pub struct HangFilter {
name: String,
}
impl Filter for HangFilter {
fn name(&self) -> &str {
&self.name
}
fn inspect(&self, _ctx: &FilterCtx<'_>) -> Verdict {
while !HANG_RELEASE.load(Ordering::Relaxed) {
std::thread::sleep(Duration::from_millis(20));
}
Verdict::Accept
}
}
/// Closes the connection when the needle shows up.
///
/// Stateful for the same reason the real regex filter is: a needle that arrives in
/// two writes is still a needle, and a test filter that missed it would let the
/// integration suite pass on a datapath that had lost the property.
pub struct BlockFilter {
name: String,
needle: Vec<u8>,
}
impl Filter for BlockFilter {
fn name(&self) -> &str {
&self.name
}
fn may_block(&self) -> bool {
false
}
fn inspect(&self, _ctx: &FilterCtx<'_>) -> Verdict {
Verdict::Accept // never reached: this filter always opens a session
}
fn open_session(&self, _dir: Direction) -> Option<Box<dyn FilterSession>> {
Some(Box::new(BlockSession {
name: self.name.clone(),
needle: self.needle.clone(),
tail: Vec::new(),
}))
}
}
struct BlockSession {
name: String,
needle: Vec<u8>,
tail: Vec<u8>,
}
impl FilterSession for BlockSession {
fn inspect(&mut self, ctx: &FilterCtx<'_>) -> Verdict {
let mut window = std::mem::take(&mut self.tail);
window.extend_from_slice(ctx.chunk);
let hit = contains(&window, &self.needle);
// Keep just enough of the tail that a needle straddling the boundary is
// found once, and never twice.
let keep = window.len().min(self.needle.len().saturating_sub(1));
self.tail = window[window.len() - keep..].to_vec();
if hit {
Verdict::Reject(Some(self.name.clone()))
} else {
Verdict::Accept
}
}
}
fn contains(haystack: &[u8], needle: &[u8]) -> bool {
if needle.is_empty() || needle.len() > haystack.len() {
return false;
}
haystack.windows(needle.len()).any(|w| w == needle)
}
/// Parse one spec: `panic`, `hang` or `block:<needle>`.
///
/// There was a `replace:<from>:<to>` here, and it went with the rewriting verdict rather
/// than being dropped from the scaffolding on its own — there is nothing left for a test
/// filter to ask the chain to do but accept or refuse.
pub fn parse_filter(spec: &str) -> Result<Arc<dyn Filter>, String> {
let spec = spec.trim();
let (kind, rest) = match spec.split_once(':') {
Some((k, r)) => (k, Some(r)),
None => (spec, None),
};
match (kind, rest) {
("panic", _) => Ok(Arc::new(PanicFilter {
name: "panic".to_string(),
})),
("hang", _) => Ok(Arc::new(HangFilter {
name: "hang".to_string(),
})),
("block", Some(needle)) if !needle.is_empty() => Ok(Arc::new(BlockFilter {
name: format!("block({needle})"),
needle: needle.as_bytes().to_vec(),
})),
_ => Err(format!("unknown filter spec '{spec}'")),
}
}
/// Parse a comma-separated list. An empty string yields no filters.
pub fn parse_filters(specs: &str) -> Result<Vec<Arc<dyn Filter>>, String> {
specs
.split(',')
.map(str::trim)
.filter(|s| !s.is_empty())
.map(parse_filter)
.collect()
}

182
backend/proxysrc/src/tls.rs Normal file
View File

@ -0,0 +1,182 @@
//! TLS, terminated by the engine itself.
//!
//! The TLS module does this with nginx today: one listener terminates the public
//! connection and forwards plaintext to a second loopback port, which re-encrypts and
//! forwards to the real service. A filter engine attaches to the leg in between. It
//! works, and it costs an nginx config generated per stream, a pair of ports derived
//! from a hash, and one hop where the traffic is in the clear.
//!
//! Terminating here removes all three: the engine decrypts, the rules see plaintext
//! that never leaves the process, and it re-encrypts on the way out.
use std::io;
use std::sync::Arc;
use rustls::client::danger::{HandshakeSignatureValid, ServerCertVerified, ServerCertVerifier};
use rustls::pki_types::{CertificateDer, PrivateKeyDer, ServerName, UnixTime};
use rustls::{ClientConfig, DigitallySignedStruct, ServerConfig, SignatureScheme};
use tokio_rustls::{TlsAcceptor, TlsConnector};
/// Certificate and key for the connection the engine terminates.
pub fn server_config(cert_pem: &str, key_pem: &str) -> Result<Arc<ServerConfig>, String> {
let certs: Vec<CertificateDer<'static>> = rustls_pemfile::certs(&mut cert_pem.as_bytes())
.collect::<Result<_, _>>()
.map_err(|e| format!("cannot read the certificate: {e}"))?;
if certs.is_empty() {
return Err("the certificate is empty".to_string());
}
let key: PrivateKeyDer<'static> = rustls_pemfile::private_key(&mut key_pem.as_bytes())
.map_err(|e| format!("cannot read the private key: {e}"))?
.ok_or_else(|| "the private key is empty".to_string())?;
ServerConfig::builder()
.with_no_client_auth()
.with_single_cert(certs, key)
.map(Arc::new)
.map_err(|e| format!("certificate and key do not go together: {e}"))
}
/// Accepts whatever the protected service presents.
///
/// This mirrors nginx's `proxy_ssl_verify off`, which is what the TLS module already
/// configures, and it is the right call here rather than laziness: the service being
/// protected is on the other side of a loopback or a private link, it is the thing
/// firegex is defending, and in a competition it is invariably self-signed. Verifying
/// it would only ever refuse to protect it.
#[derive(Debug)]
struct AcceptAnyServer(Arc<rustls::crypto::CryptoProvider>);
impl ServerCertVerifier for AcceptAnyServer {
fn verify_server_cert(
&self,
_end_entity: &CertificateDer<'_>,
_intermediates: &[CertificateDer<'_>],
_server_name: &ServerName<'_>,
_ocsp: &[u8],
_now: UnixTime,
) -> Result<ServerCertVerified, rustls::Error> {
Ok(ServerCertVerified::assertion())
}
fn verify_tls12_signature(
&self,
message: &[u8],
cert: &CertificateDer<'_>,
dss: &DigitallySignedStruct,
) -> Result<HandshakeSignatureValid, rustls::Error> {
rustls::crypto::verify_tls12_signature(
message,
cert,
dss,
&self.0.signature_verification_algorithms,
)
}
fn verify_tls13_signature(
&self,
message: &[u8],
cert: &CertificateDer<'_>,
dss: &DigitallySignedStruct,
) -> Result<HandshakeSignatureValid, rustls::Error> {
rustls::crypto::verify_tls13_signature(
message,
cert,
dss,
&self.0.signature_verification_algorithms,
)
}
fn supported_verify_schemes(&self) -> Vec<SignatureScheme> {
self.0.signature_verification_algorithms.supported_schemes()
}
}
/// Re-encrypting towards the service.
pub fn client_config() -> Result<Arc<ClientConfig>, String> {
let provider = provider();
let config = ClientConfig::builder_with_provider(Arc::clone(&provider))
.with_safe_default_protocol_versions()
.map_err(|e| format!("cannot configure TLS: {e}"))?
.dangerous()
.with_custom_certificate_verifier(Arc::new(AcceptAnyServer(provider)))
.with_no_client_auth();
Ok(Arc::new(config))
}
/// The same thing for QUIC, where TLS 1.3 is not a preference.
///
/// QUIC carries its handshake in 1.3 and only 1.3, and a config still offering 1.2 is
/// refused outright by `QuicClientConfig` rather than quietly negotiating down — so the
/// version list is narrowed here, where the reason can be written next to it, instead of
/// at the call site as a conversion error nobody can act on.
pub fn quic_client_config() -> Result<ClientConfig, String> {
let provider = provider();
Ok(
ClientConfig::builder_with_provider(Arc::clone(&provider))
.with_protocol_versions(&[&rustls::version::TLS13])
.map_err(|e| format!("cannot configure QUIC: {e}"))?
.dangerous()
.with_custom_certificate_verifier(Arc::new(AcceptAnyServer(provider)))
.with_no_client_auth(),
)
}
/// The process-wide crypto provider, installed once.
fn provider() -> Arc<rustls::crypto::CryptoProvider> {
if let Some(installed) = rustls::crypto::CryptoProvider::get_default() {
return Arc::clone(installed);
}
let provider = rustls::crypto::ring::default_provider();
// Losing the race is fine: whoever won installed the same thing.
let _ = provider.clone().install_default();
rustls::crypto::CryptoProvider::get_default()
.cloned()
.unwrap_or_else(|| Arc::new(provider))
}
/// The upstream config, offering the protocols the client asked for.
///
/// Cloned per connection rather than kept in a table: the list comes from the client, so
/// there is no fixed set to precompute, and a config clone is nothing beside the
/// handshake it is about to do. An empty list means the client offered none, and then
/// neither do we — offering something on its behalf would be answering a question it did
/// not ask.
pub fn with_alpn(config: &Arc<ClientConfig>, wanted: &[Vec<u8>]) -> Arc<ClientConfig> {
if wanted.is_empty() {
return Arc::clone(config);
}
let mut copy = (**config).clone();
copy.alpn_protocols = wanted.to_vec();
Arc::new(copy)
}
/// The server config, advertising the one protocol the service agreed to.
///
/// One, not a list: the service has already chosen, and offering the client anything else
/// would let it pick something the other side of this proxy cannot speak. `None` means
/// the service chose nothing, so the client is told nothing and both ends fall back
/// together.
pub fn answering_with(config: &Arc<ServerConfig>, agreed: Option<&[u8]>) -> Arc<ServerConfig> {
match agreed {
Some(protocol) => {
let mut copy = (**config).clone();
copy.alpn_protocols = vec![protocol.to_vec()];
Arc::new(copy)
}
None => Arc::clone(config),
}
}
pub fn acceptor(config: Arc<ServerConfig>) -> TlsAcceptor {
TlsAcceptor::from(config)
}
pub fn connector(config: Arc<ClientConfig>) -> TlsConnector {
TlsConnector::from(config)
}
/// The name to present upstream. Nothing verifies it here, but rustls needs one.
pub fn server_name(host: &str) -> Result<ServerName<'static>, io::Error> {
ServerName::try_from(host.to_string())
.map_err(|e| io::Error::new(io::ErrorKind::InvalidInput, format!("bad server name: {e}")))
}

278
backend/proxysrc/src/transparent.rs Normal file
View File

@ -0,0 +1,278 @@
//! Source-IP transparency: making the protected service see the real client.
//!
//! The NFQUEUE engine gets this for free — it never terminates anything, so the
//! service sees the original packets. A proxy has to work for it, and if it does not,
//! every service that logs, rate-limits or bans by client address quietly starts
//! seeing one address for the whole internet. That is a correctness regression the
//! operator would never notice, so the capability is checked at startup and the
//! per-connection fallback is counted and logged rather than silent.
use std::io;
use std::net::{IpAddr, Ipv4Addr, Ipv6Addr, SocketAddr};
use std::os::fd::{AsRawFd, RawFd};
use socket2::{Domain, Protocol, Socket, Type};
use tokio::net::{TcpListener, TcpSocket, TcpStream, UdpSocket};
// Not in libc: netfilter's own option numbers.
const SO_ORIGINAL_DST: libc::c_int = 80;
const IP6T_SO_ORIGINAL_DST: libc::c_int = 80;
const IP_TRANSPARENT: libc::c_int = 19;
const IPV6_TRANSPARENT: libc::c_int = 75;
/// Stamped on every connection the engine opens itself.
///
/// Without it an intercept rule cannot tell the engine's own dial apart from the
/// traffic it is supposed to intercept, and on a path where both ends are local —
/// nginx's clear leg in the TLS module is exactly that — the engine redirects itself
/// in a loop. Deliberately not the mark policy routing keys off: that one means
/// "deliver this here", and stamping it on an outbound connection would strand it.
pub const SELF_MARK: u32 = 0x133A;
fn setsockopt_int(
fd: RawFd,
level: libc::c_int,
name: libc::c_int,
value: libc::c_int,
) -> io::Result<()> {
let rc = unsafe {
libc::setsockopt(
fd,
level,
name,
&value as *const libc::c_int as *const libc::c_void,
std::mem::size_of::<libc::c_int>() as libc::socklen_t,
)
};
if rc != 0 {
return Err(io::Error::last_os_error());
}
Ok(())
}
/// Allow the socket to bind and receive on addresses that are not ours.
///
/// Needs `CAP_NET_ADMIN`; without it this fails with `EPERM` and the caller has to
/// decide whether that is fatal.
pub fn set_transparent(fd: RawFd, ipv6: bool) -> io::Result<()> {
if ipv6 {
setsockopt_int(fd, libc::IPPROTO_IPV6, IPV6_TRANSPARENT, 1)
} else {
setsockopt_int(fd, libc::IPPROTO_IP, IP_TRANSPARENT, 1)
}
}
/// Turn a v4-mapped IPv6 address back into the IPv4 address it is.
///
/// A service answering on both stacks is served by one dual-stack listener, so an IPv4
/// connection arrives on it wearing a `::ffff:a.b.c.d` costume. Everything downstream —
/// the conntrack lookup, the source-preserving dial, the metadata a filter reads —
/// wants the address the packet actually had, and a mapped address matches none of the
/// v4 machinery it needs to reach.
pub fn unmap(addr: SocketAddr) -> SocketAddr {
match addr {
SocketAddr::V6(v6) => match v6.ip().to_ipv4_mapped() {
Some(v4) => SocketAddr::new(IpAddr::V4(v4), v6.port()),
None => addr,
},
v4 => v4,
}
}
/// Where the client was actually trying to reach.
///
/// Always from conntrack: the destination was rewritten before we saw it, and
/// conntrack is what still remembers the original. There used to be a second way
/// (tproxy, which preserves the addresses in place), but it forced the operator to
/// understand the difference and could not reach a service on this host at all. The
/// proxy should look like it is not there, and that is not a setting.
pub fn original_destination(stream: &TcpStream) -> io::Result<SocketAddr> {
original_dst_from_conntrack(stream)
}
fn original_dst_from_conntrack(stream: &TcpStream) -> io::Result<SocketAddr> {
let fd = stream.as_raw_fd();
// Unmapped first: on a dual-stack listener an IPv4 connection reports a v6 local
// address, and asking IPv6 conntrack about an IPv4 flow simply fails.
match unmap(stream.local_addr()?) {
SocketAddr::V4(_) => {
let mut addr: libc::sockaddr_in = unsafe { std::mem::zeroed() };
let mut len = std::mem::size_of::<libc::sockaddr_in>() as libc::socklen_t;
let rc = unsafe {
libc::getsockopt(
fd,
libc::IPPROTO_IP,
SO_ORIGINAL_DST,
&mut addr as *mut libc::sockaddr_in as *mut libc::c_void,
&mut len,
)
};
if rc != 0 {
return Err(io::Error::last_os_error());
}
Ok(SocketAddr::new(
IpAddr::V4(Ipv4Addr::from(u32::from_be(addr.sin_addr.s_addr))),
u16::from_be(addr.sin_port),
))
}
SocketAddr::V6(_) => {
let mut addr: libc::sockaddr_in6 = unsafe { std::mem::zeroed() };
let mut len = std::mem::size_of::<libc::sockaddr_in6>() as libc::socklen_t;
let rc = unsafe {
libc::getsockopt(
fd,
libc::IPPROTO_IPV6,
IP6T_SO_ORIGINAL_DST,
&mut addr as *mut libc::sockaddr_in6 as *mut libc::c_void,
&mut len,
)
};
if rc != 0 {
return Err(io::Error::last_os_error());
}
Ok(SocketAddr::new(
IpAddr::V6(Ipv6Addr::from(addr.sin6_addr.s6_addr)),
u16::from_be(addr.sin6_port),
))
}
}
}
pub fn bind_listener(addr: SocketAddr) -> io::Result<TcpListener> {
let socket = if addr.is_ipv6() {
let sock = TcpSocket::new_v6()?;
let _ = setsockopt_int(sock.as_raw_fd(), libc::IPPROTO_IPV6, libc::IPV6_V6ONLY, 0);
sock
} else {
TcpSocket::new_v4()?
};
socket.set_reuseaddr(true)?;
socket.bind(addr)?;
socket.listen(1024)
}
/// Dial the service as the client: same source address, kernel-chosen source port.
///
/// The reply comes back to an address that is not ours, so it only reaches this
/// socket if the ruleset diverts it — the `socket transparent` match in prerouting
/// plus a local route for marked packets. Without that the connection simply hangs,
/// which is why the integration test asserts a full round trip and not just the
/// address the service reports.
pub async fn connect_as(
client: IpAddr,
upstream: SocketAddr,
self_mark: Option<u32>,
) -> io::Result<TcpStream> {
if client.is_ipv6() != upstream.is_ipv6() {
return Err(io::Error::new(
io::ErrorKind::InvalidInput,
"cannot spoof an IPv4 client towards an IPv6 service, or the reverse",
));
}
let socket = if upstream.is_ipv6() {
TcpSocket::new_v6()?
} else {
TcpSocket::new_v4()?
};
set_transparent(socket.as_raw_fd(), upstream.is_ipv6())?;
if let Some(mark) = self_mark {
set_self_mark(socket.as_raw_fd(), mark)?;
}
socket.set_reuseaddr(true)?;
socket.bind(SocketAddr::new(client, 0))?;
socket.connect(upstream).await
}
/// Dial without impersonating anyone, but still marked as ours.
pub async fn connect_plain(upstream: SocketAddr, self_mark: Option<u32>) -> io::Result<TcpStream> {
let socket = if upstream.is_ipv6() {
TcpSocket::new_v6()?
} else {
TcpSocket::new_v4()?
};
if let Some(mark) = self_mark {
set_self_mark(socket.as_raw_fd(), mark)?;
}
socket.connect(upstream).await
}
/// A UDP socket wearing the client's address, with nothing dialled yet.
///
/// Split out from [`connect_as_udp`] for QUIC: quinn owns its socket and sends to an
/// address per packet, which a *connected* socket refuses (`EISCONN`). Everything that
/// makes the address someone else's — `IP_TRANSPARENT`, the mark the intercept rules
/// skip, and the bind to a foreign address — happens here; who it then talks to is the
/// caller's business.
pub fn bind_as_udp(
client: IpAddr,
upstream: SocketAddr,
self_mark: Option<u32>,
) -> io::Result<std::net::UdpSocket> {
if client.is_ipv6() != upstream.is_ipv6() {
return Err(io::Error::new(
io::ErrorKind::InvalidInput,
"cannot spoof an IPv4 client towards an IPv6 service, or the reverse",
));
}
let domain = if upstream.is_ipv6() {
Domain::IPV6
} else {
Domain::IPV4
};
let socket = Socket::new(domain, Type::DGRAM, Some(Protocol::UDP))?;
socket.set_nonblocking(true)?;
set_transparent(socket.as_raw_fd(), upstream.is_ipv6())?;
if let Some(mark) = self_mark {
set_self_mark(socket.as_raw_fd(), mark)?;
}
socket.set_reuse_address(true)?;
let bind_addr: socket2::SockAddr = SocketAddr::new(client, 0).into();
socket.bind(&bind_addr)?;
Ok(socket.into())
}
/// Dial the service as the client over UDP: same source address, kernel-chosen source port.
pub async fn connect_as_udp(
client: IpAddr,
upstream: SocketAddr,
self_mark: Option<u32>,
) -> io::Result<UdpSocket> {
let std_sock = bind_as_udp(client, upstream, self_mark)?;
let tokio_sock = UdpSocket::from_std(std_sock)?;
tokio_sock.connect(upstream).await?;
Ok(tokio_sock)
}
/// Dial UDP without impersonating anyone, but still marked as ours.
pub async fn connect_plain_udp(
upstream: SocketAddr,
self_mark: Option<u32>,
) -> io::Result<UdpSocket> {
let bind: SocketAddr = if upstream.is_ipv6() {
"[::]:0".parse().unwrap()
} else {
"0.0.0.0:0".parse().unwrap()
};
let socket = UdpSocket::bind(bind).await?;
if let Some(mark) = self_mark {
set_self_mark(socket.as_raw_fd(), mark)?;
}
socket.connect(upstream).await?;
Ok(socket)
}
/// Mark a socket as ours, so the intercept rules can skip it.
pub fn set_self_mark(fd: RawFd, mark: u32) -> io::Result<()> {
setsockopt_int(fd, libc::SOL_SOCKET, libc::SO_MARK, mark as libc::c_int)
}
/// Check at startup that we can actually do this, so a missing `CAP_NET_ADMIN` is a
/// loud failure instead of a silent loss of the client's identity.
pub fn probe_capability(ipv6: bool) -> io::Result<()> {
let socket = if ipv6 {
TcpSocket::new_v6()?
} else {
TcpSocket::new_v4()?
};
set_transparent(socket.as_raw_fd(), ipv6)
}

481
backend/proxysrc/src/udp.rs Normal file
View File

@ -0,0 +1,481 @@
//! The proxy layer for UDP: relays with transparent source IP preservation.
//!
//! The TCP side of this engine stays invisible in two ways at once: it recovers where
//! the client was originally headed with `SO_ORIGINAL_DST`, and it dials the service
//! *from the client's own address* via `IP_TRANSPARENT`.
//!
//! On UDP:
//! * **`SO_ORIGINAL_DST` is TCP and SCTP only.** The kernel answers `ENOPROTOOPT` for a
//! UDP socket, so a single listener cannot ask where a datagram was going. This relay
//! handles it by binding **one socket per protected address**, each with the
//! upstream already known.
//! * **The client's address is preserved.** The relay dials the upstream service from
//! the client's own address using `IP_TRANSPARENT` and mark-based policy routing,
//! keeping source IP transparency on both TCP and UDP.
//!
//! What is kept is everything the filter layer cares about: each client flow is a
//! connection with its own filter state and its own module globals, both directions are
//! inspected.
use std::collections::HashMap;
use std::io;
use std::net::SocketAddr;
use std::sync::atomic::{AtomicBool, Ordering};
use std::sync::Arc;
use std::time::{Duration, Instant};
use tokio::net::UdpSocket;
use crate::proxy::{ProxyStats, Slot};
use crate::filter::{
next_connection_id, ChainHandle, ChainSessions, ConnectionId, ConnectionMeta, Direction,
FilterChain, Verdict, L4,
};
/// The largest datagram this relay will carry. Comfortably past the practical MTU and
/// past anything a filter is going to be shown in one piece.
const MAX_DATAGRAM: usize = 65_535;
/// How long a client flow with no traffic is kept before its filter state is released.
///
/// UDP has no close to observe, so this is the only thing that ends a flow. Too short
/// and a filter keeping per-stream state forgets mid-conversation; too long and a
/// service under a spoofed-source flood accumulates one session per forged address.
const IDLE: Duration = Duration::from_secs(60);
/// How often idle flows are swept.
const SWEEP: Duration = Duration::from_secs(10);
/// How many of one flow's datagrams may wait for its verdicts. Past it they are dropped,
/// which is what the socket buffer would have done with them: a flow whose filter is slow
/// falls behind on its own, and the budget bounds what it can make this process hold.
const FLOW_QUEUE: usize = 32;
struct Flow {
/// This flow's datagrams, on their way to be judged and sent on — by a task of the
/// flow's own, so a slow verdict holds up this client and nobody else.
///
/// The receive loop used to judge every datagram itself, one after the other, for
/// every client of the address: a filter taking a second over one client's datagram
/// was a second in which no other client's was even read, and the socket buffer
/// overflowing behind it dropped theirs.
to_service: tokio::sync::mpsc::Sender<Vec<u8>>,
forward_task: tokio::task::JoinHandle<()>,
connection: ConnectionId,
/// What this flow is judged by: the service's chain, or no chain at all for a flow
/// admitted past the limit because the operator chose to forward what does not fit.
/// Decided once, when the flow opens — the same trade a TCP connection admitted past
/// the limit makes for its whole life.
chain: ChainHandle,
reply_task: tokio::task::JoinHandle<()>,
last_seen: Instant,
/// This flow's place in the service's count of what it is carrying, released when the
/// flow is. Held for the side effect of dropping it.
_slot: Slot,
}
/// One protected address, relayed.
pub struct UdpRelay {
listener: Arc<UdpSocket>,
/// Where this listener's traffic goes. Fixed, which is the whole trick: with the
/// upstream known up front there is nothing to recover per datagram, and the kernel
/// option that cannot answer for UDP is never asked.
upstream: SocketAddr,
chain: ChainHandle,
self_mark: Option<u32>,
pub spoof_source: bool,
/// How many connections and flows the service may carry at once, counted together
/// with its TCP connections in `stats.live`. `0` means no limit.
max_flows: usize,
/// Whether a flow from a new source past the limit is carried unfiltered rather than
/// dropped. The operator's choice, the same one the TCP side offers.
over_limit_forwards: bool,
/// Shared with the TCP side, so one service reports one pair of numbers and spends one
/// budget.
stats: Arc<ProxyStats>,
}
impl UdpRelay {
/// Bind a listener for one protected address. Port 0 lets the kernel choose, and
/// [`UdpRelay::local_addr`] reports what it chose.
#[allow(clippy::too_many_arguments)]
pub async fn bind(
listen: SocketAddr,
upstream: SocketAddr,
chain: ChainHandle,
self_mark: Option<u32>,
spoof_source: bool,
max_flows: usize,
over_limit_forwards: bool,
stats: Arc<ProxyStats>,
) -> io::Result<Self> {
let listener = UdpSocket::bind(listen).await?;
Ok(Self {
listener: Arc::new(listener),
upstream,
chain,
self_mark,
spoof_source,
max_flows,
over_limit_forwards,
stats,
})
}
pub fn local_addr(&self) -> io::Result<SocketAddr> {
self.listener.local_addr()
}
/// Relay forever.
///
/// Nothing one flow does may end this loop: a datagram that cannot be forwarded
/// costs that datagram, and a filter that refuses one costs that one too. The
/// listener outliving every flow is the same property the TCP side has.
pub async fn serve(self) -> io::Result<()> {
let mut flows: HashMap<SocketAddr, Flow> = HashMap::new();
let mut buf = vec![0u8; MAX_DATAGRAM];
let mut sweep = tokio::time::interval(SWEEP);
sweep.tick().await;
loop {
let (len, client) = tokio::select! {
received = self.listener.recv_from(&mut buf) => match received {
Ok(pair) => pair,
// A per-datagram error (an ICMP port-unreachable landing on the
// socket, most often) must not end the relay.
Err(e) => {
crate::diag_throttled!("[warn] [udp] failed receives", 5, 10, "[warn] [udp] receive failed: {e}");
continue;
}
},
_ = sweep.tick() => {
self.expire(&mut flows);
continue;
}
};
let flow = match flows.get_mut(&client) {
Some(flow) => flow,
None => {
// The cheapest attack there is, and it was unbounded: a flow is
// created per *source address*, a datagram's source is not verified
// by any handshake, and a flow holds a socket, a task and a set of
// filter sessions for `IDLE` after the last datagram. Six hundred
// datagrams from six hundred forged sources, sent in three
// hundredths of a second, took four hundred descriptors and held
// them for a minute after the sender had gone.
//
// Counted in the **same** number as the service's TCP connections,
// because they spend the same descriptors and the limit is one budget:
// it was a count of this relay's own flows, so a service with three UDP
// addresses could hold three times its limit in flows and every one of
// them beside a full complement of TCP connections.
let live = self.stats.live.fetch_add(1, Ordering::Relaxed) + 1;
let slot = Slot(Arc::clone(&self.stats));
let limit = self.max_flows;
let over = limit > 0 && live > limit as u64;
let chain = if over {
self.stats.over_limit.fetch_add(1, Ordering::Relaxed);
if !self.stats.warned_limit.swap(true, Ordering::Relaxed) {
crate::diag!(
"[warn] [udp] {limit} concurrent connections and flows \
reached; new flows are being {} until it clears",
if self.over_limit_forwards { "carried unfiltered" } else { "dropped" },
);
}
if !self.over_limit_forwards {
drop(slot);
continue;
}
// Carried as a flow of its own, with no chain in front of it. It
// used to be a datagram sent from a socket closed straight after,
// so the service's answer reached a port nobody was listening on:
// the request went through and the client never heard back, which
// is not what forwarding is for.
ChainHandle::new(FilterChain::empty())
} else {
if live * 2 <= limit as u64 {
// Armed again once there is real room, as on the TCP side.
self.stats.warned_limit.store(false, Ordering::Relaxed);
}
self.chain.clone()
};
match self.open(client, chain, slot).await {
Ok(flow) => {
self.stats.accepted.fetch_add(1, Ordering::Relaxed);
flows.entry(client).or_insert(flow)
}
Err(e) => {
crate::diag_throttled!("[warn] [udp] flows that could not reach the service", 5, 10, "[warn] [udp] cannot reach {} for {client}: {e}", self.upstream);
continue;
}
}
}
};
flow.last_seen = Instant::now();
match flow.to_service.try_send(buf[..len].to_vec()) {
Ok(()) => {}
Err(tokio::sync::mpsc::error::TrySendError::Full(_)) => {
crate::diag_throttled!("[warn] [udp] datagrams dropped behind a slow verdict", 5, 10, "[warn] [udp] {client} is sending faster than its datagrams are judged; one was dropped");
}
// Its task is gone — it only ends when the flow does — so the flow goes
// too, and the client's next datagram opens a new one.
Err(tokio::sync::mpsc::error::TrySendError::Closed(_)) => {
if let Some(flow) = flows.remove(&client) {
flow.close();
}
}
}
}
}
/// Start relaying one client's flow: a socket towards the service, a task judging and
/// sending on what the client sends, and one carrying the answers back, all by
/// `chain`.
async fn open(&self, client: SocketAddr, chain: ChainHandle, slot: Slot) -> io::Result<Flow> {
let upstream = if self.spoof_source {
match crate::transparent::connect_as_udp(client.ip(), self.upstream, self.self_mark).await {
Ok(s) => s,
Err(e) => {
self.stats.source_spoof_failures.fetch_add(1, Ordering::Relaxed);
if !self.stats.warned_spoof.swap(true, Ordering::Relaxed) {
crate::diag!(
"[warn] [udp] cannot reach {} as {}: {e}. \
Falling back to our own address — the service will not see real client IPs.",
self.upstream,
client.ip()
);
}
crate::transparent::connect_plain_udp(self.upstream, self.self_mark).await?
}
}
} else {
crate::transparent::connect_plain_udp(self.upstream, self.self_mark).await?
};
let upstream = Arc::new(upstream);
let connection = next_connection_id();
// Told once, before any datagram of this flow is judged. `client` is the real
// peer: a filter reads who is talking, and that stays true whatever address the
// relay dials from.
chain.current().connection_opened(
connection,
&ConnectionMeta {
client,
server: self.upstream,
l4: L4::Udp,
},
);
let refused = Arc::new(AtomicBool::new(false));
let (to_service, datagrams) = tokio::sync::mpsc::channel(FLOW_QUEUE);
let forward_task = tokio::spawn(forward(
datagrams,
Arc::clone(&upstream),
self.upstream,
chain.clone(),
connection,
Arc::clone(&refused),
Arc::clone(&self.stats),
));
let reply_task = tokio::spawn(replies(
Arc::clone(&upstream),
Arc::clone(&self.listener),
client,
chain.clone(),
connection,
Arc::clone(&refused),
Arc::clone(&self.stats),
));
Ok(Flow {
to_service,
forward_task,
connection,
chain,
reply_task,
last_seen: Instant::now(),
_slot: slot,
})
}
/// Release the flows nothing has been heard from.
///
/// The only thing that ends a UDP flow: there is no close to observe. Without it a
/// service would accumulate one set of filter state — and, for a Python filter, one
/// set of module globals — per client address for as long as it ran.
fn expire(&self, flows: &mut HashMap<SocketAddr, Flow>) {
let now = Instant::now();
let done: Vec<SocketAddr> = flows
.iter()
.filter(|(_, flow)| now.duration_since(flow.last_seen) > IDLE)
.map(|(addr, _)| *addr)
.collect();
for addr in done {
if let Some(flow) = flows.remove(&addr) {
flow.close();
}
}
}
}
impl Flow {
/// Let go of everything this flow holds, and tell the filters it is over.
fn close(self) {
self.forward_task.abort();
self.reply_task.abort();
self.chain.current().connection_closed(self.connection);
}
}
/// Judge one flow's datagrams in the order they arrived, and send on what is accepted.
async fn forward(
mut datagrams: tokio::sync::mpsc::Receiver<Vec<u8>>,
upstream: Arc<UdpSocket>,
service: SocketAddr,
chain: ChainHandle,
connection: ConnectionId,
refused: Arc<AtomicBool>,
stats: Arc<ProxyStats>,
) {
// This direction's own filter state, exactly as the TCP pumps keep theirs.
let mut sessions = ChainSessions::new(connection);
while let Some(datagram) = datagrams.recv().await {
// Re-read the handle every datagram, so a chain swapped in mid-flow takes effect
// without anyone losing their session.
let verdict = chain
.current()
.run(Direction::ClientToServer, &datagram, &mut sessions)
.await;
if let Verdict::Reject(_) = verdict {
// There is no connection to close, so refusing the datagram is the whole of
// what refusing can mean here.
if !refused.swap(true, Ordering::Relaxed) {
stats.closed_by_filter.fetch_add(1, Ordering::Relaxed);
}
continue;
}
if let Err(e) = upstream.send(&datagram).await {
crate::diag_throttled!("[warn] [udp] datagrams that could not be forwarded", 5, 10, "[warn] [udp] cannot forward to {service}: {e}");
}
}
}
/// Carry one flow's answers back, inspecting them on the way.
///
/// Replies leave through the *listener* socket, so conntrack rewrites them to appear
/// from the address the client dialled. Sending them from the upstream socket would
/// reach a client that is not expecting that source and would be dropped by it.
async fn replies(
upstream: Arc<UdpSocket>,
listener: Arc<UdpSocket>,
client: SocketAddr,
chain: ChainHandle,
connection: ConnectionId,
refused: Arc<AtomicBool>,
stats: Arc<ProxyStats>,
) {
// This direction's own filter state, exactly as the TCP pumps keep theirs.
let mut sessions = ChainSessions::new(connection);
let mut buf = vec![0u8; MAX_DATAGRAM];
loop {
let len = match upstream.recv(&mut buf).await {
Ok(len) => len,
// The service's port answered a datagram with ICMP "unreachable", which a
// connected socket reports once, on its next call: the service was down for a
// moment — being restarted, most often. Nothing about the flow is over. This
// used to end the task, which left the flow in the relay's table with nobody
// reading the service's answers, so a client that went on talking never heard
// back again while its datagrams kept the flow from ever going idle.
Err(e) if e.kind() == io::ErrorKind::ConnectionRefused => continue,
Err(e) => {
crate::diag_throttled!("[info] [udp] flows that ended with an error", 20, 10, "[info] [udp] flow for {client} ended: {e}");
return;
}
};
let verdict = chain
.current()
.run(Direction::ServerToClient, &buf[..len], &mut sessions)
.await;
let payload: &[u8] = match &verdict {
Verdict::Accept => &buf[..len],
// There is no connection to close, so refusing the datagram is the whole of
// what refusing can mean here. The client simply never receives it.
Verdict::Reject(_) => {
if !refused.swap(true, Ordering::Relaxed) {
stats.closed_by_filter.fetch_add(1, Ordering::Relaxed);
}
continue;
}
};
if let Err(e) = listener.send_to(payload, client).await {
crate::diag_throttled!("[warn] [udp] answers that could not be delivered", 5, 10, "[warn] [udp] cannot answer {client}: {e}");
}
}
}
/// Manages running UDP relays and allows adding new relays dynamically at runtime.
#[derive(Clone)]
pub struct UdpManager {
chain: ChainHandle,
self_mark: Option<u32>,
spoof_source: bool,
max_flows: usize,
over_limit_forwards: bool,
stats: Arc<ProxyStats>,
relays: Arc<tokio::sync::Mutex<HashMap<SocketAddr, u16>>>,
}
impl UdpManager {
pub fn new(
chain: ChainHandle,
self_mark: Option<u32>,
spoof_source: bool,
max_flows: usize,
over_limit_forwards: bool,
stats: Arc<ProxyStats>,
) -> Self {
Self {
chain,
self_mark,
spoof_source,
max_flows,
over_limit_forwards,
stats,
relays: Arc::new(tokio::sync::Mutex::new(HashMap::new())),
}
}
pub async fn add_relay(&self, upstream: SocketAddr) -> io::Result<u16> {
let mut map = self.relays.lock().await;
if let Some(&port) = map.get(&upstream) {
return Ok(port);
}
let bind: SocketAddr = if upstream.is_ipv6() {
"[::]:0".parse().unwrap()
} else {
"0.0.0.0:0".parse().unwrap()
};
let relay = UdpRelay::bind(
bind,
upstream,
self.chain.clone(),
self.self_mark,
self.spoof_source,
self.max_flows,
self.over_limit_forwards,
Arc::clone(&self.stats),
)
.await?;
let port = relay.local_addr()?.port();
map.insert(upstream, port);
tokio::spawn(async move {
if let Err(e) = relay.serve().await {
crate::diag!("[fatal] [udp] relay for {upstream} died: {e}");
}
});
Ok(port)
}
}

334
backend/proxysrc/tests/fail_open.rs Normal file
View File

@ -0,0 +1,334 @@
//! The claims the proxy engine has to earn before it deserves to exist.
//!
//! Every test drives real sockets: an echo server standing in for the protected
//! service, the proxy in front of it, and a client that must keep working no matter
//! what the filters do.
use std::sync::atomic::Ordering;
use std::sync::Arc;
use std::time::{Duration, Instant};
use fgex_proxy::filter::{ChainHandle, Filter, FilterChain, FilterCtx, Verdict};
use fgex_proxy::proxy::{Proxy, ProxyConfig};
use fgex_proxy::spec::parse_filters;
use tokio::io::{AsyncReadExt, AsyncWriteExt};
use tokio::net::{TcpListener, TcpStream};
/// Echo server: whatever it receives, it sends straight back.
async fn spawn_echo() -> std::net::SocketAddr {
let listener = TcpListener::bind("127.0.0.1:0").await.unwrap();
let addr = listener.local_addr().unwrap();
tokio::spawn(async move {
loop {
let Ok((mut sock, _)) = listener.accept().await else {
continue;
};
tokio::spawn(async move {
let mut buf = vec![0u8; 16 * 1024];
loop {
match sock.read(&mut buf).await {
Ok(0) | Err(_) => break,
Ok(n) => {
if sock.write_all(&buf[..n]).await.is_err() {
break;
}
}
}
}
});
}
});
addr
}
async fn spawn_proxy(filters: &str, deadline_ms: u64) -> (std::net::SocketAddr, ChainHandle) {
spawn_proxy_with(FilterChain::new(
parse_filters(filters).unwrap(),
Duration::from_millis(deadline_ms),
))
.await
}
async fn spawn_proxy_with(chain: FilterChain) -> (std::net::SocketAddr, ChainHandle) {
let upstream = spawn_echo().await;
let chain = ChainHandle::new(chain);
let proxy = Proxy::bind(
ProxyConfig::fixed("127.0.0.1:0".parse().unwrap(), upstream),
chain.clone(),
)
.await
.unwrap();
let addr = proxy.local_addr().unwrap();
tokio::spawn(proxy.serve());
(addr, chain)
}
/// Send one payload, read one reply.
async fn roundtrip(addr: std::net::SocketAddr, payload: &[u8]) -> std::io::Result<Vec<u8>> {
let mut sock = TcpStream::connect(addr).await?;
sock.write_all(payload).await?;
let mut buf = vec![0u8; 64 * 1024];
let n = tokio::time::timeout(Duration::from_secs(5), sock.read(&mut buf)).await??;
buf.truncate(n);
Ok(buf)
}
#[tokio::test]
async fn relays_traffic_with_no_filters() {
let (addr, _chain) = spawn_proxy("", 200).await;
let got = roundtrip(addr, b"hello firegex").await.unwrap();
assert_eq!(got, b"hello firegex");
}
/// The headline guarantee: user code that panics costs its own say, not the traffic.
#[tokio::test]
async fn panicking_filter_keeps_traffic_flowing() {
let (addr, chain) = spawn_proxy("panic", 200).await;
let got = roundtrip(addr, b"payload one").await.unwrap();
assert_eq!(
got, b"payload one",
"a panicking filter must not eat the data"
);
// And the next connection is served too: the failure is contained, not fatal.
let got = roundtrip(addr, b"payload two").await.unwrap();
assert_eq!(got, b"payload two");
let live = chain.current();
assert!(live.stats.panics.load(Ordering::Relaxed) >= 1);
assert_eq!(live.disabled_filters(), vec!["panic"]);
assert!(
live.is_bypassed(),
"with its only filter disabled the chain should degrade to a plain relay"
);
}
/// The failure `catch_unwind` cannot catch: a filter that simply never returns.
///
/// One test rather than three because releasing the stuck threads at the end is
/// process-wide, and the harness runs tests in parallel within one process.
#[tokio::test]
async fn hanging_filter_fails_open_then_loses_its_say() {
let rearm = Duration::from_millis(1500);
let (addr, chain) = spawn_proxy_with(
FilterChain::new(parse_filters("hang").unwrap(), Duration::from_millis(50))
.rearming_after(rearm),
)
.await;
// The deadline, not the filter, decides when the chunk moves.
let started = Instant::now();
let got = roundtrip(addr, b"still moving").await.unwrap();
assert_eq!(got, b"still moving");
assert!(
started.elapsed() < Duration::from_secs(3),
"the chunk waited on the filter instead of failing open (took {:?})",
started.elapsed()
);
assert!(chain.current().stats.timeouts.load(Ordering::Relaxed) >= 1);
// A single slow call could be load, so it takes a few in a row to lose it.
for _ in 0..4 {
assert_eq!(roundtrip(addr, b"tick").await.unwrap(), b"tick");
}
let live = chain.current();
assert_eq!(live.disabled_filters(), vec!["hang"]);
assert!(live.is_bypassed());
// Each timeout abandons a blocking thread that cannot be cancelled, so the
// damage has to be bounded: once disabled, the filter is never called again
// and the leak stops growing however much traffic follows.
let leaked = live.stats.timeouts.load(Ordering::Relaxed);
for _ in 0..10 {
assert_eq!(roundtrip(addr, b"more").await.unwrap(), b"more");
}
assert_eq!(
live.stats.timeouts.load(Ordering::Relaxed),
leaked,
"a disabled filter is still being called, so the thread leak is unbounded"
);
// Slow is not broken: after a pause it is asked again, rather than staying out until
// somebody next edits a rule. It used to stay out for the life of the chain — which
// for a Python filter meant a flood, the one moment it was wanted, switched it off.
tokio::time::sleep(rearm + Duration::from_millis(300)).await;
let answered = roundtrip(addr, b"again").await.unwrap();
let asked = live.stats.timeouts.load(Ordering::Relaxed) - leaked;
let back_in = live.disabled_filters().is_empty();
let bypassed = live.is_bypassed();
// Let the abandoned threads exit, otherwise dropping the runtime waits on them — and
// before asserting, so a failure below fails the test rather than hanging it.
fgex_proxy::spec::release_hangs();
assert_eq!(answered, b"again");
// Once per direction: the echo comes back through the chain as well.
assert_eq!(asked, 2, "the filter was not asked again after its pause");
assert!(back_in, "two misses after the pause took it out again");
assert!(!bypassed);
}
/// Failing open is only worth anything if the filters do something when healthy.
#[tokio::test]
async fn healthy_filter_still_blocks() {
let (addr, _chain) = spawn_proxy("block:FLAG{", 200).await;
let got = roundtrip(addr, b"harmless request").await.unwrap();
assert_eq!(got, b"harmless request");
// Rejected: the proxy closes cleanly, so the client reads EOF rather than
// hanging on a silently dropped packet.
let got = roundtrip(addr, b"give me FLAG{...}").await.unwrap();
assert!(got.is_empty(), "blocked payload came through: {got:?}");
}
/// A rule refuses a connection, not a direction: a service that answers without
/// waiting for the request must not still reach a client whose request was blocked.
#[tokio::test]
async fn a_reject_closes_the_whole_connection() {
// A server that greets on connect, before it has read anything.
let listener = TcpListener::bind("127.0.0.1:0").await.unwrap();
let upstream = listener.local_addr().unwrap();
tokio::spawn(async move {
loop {
let Ok((mut sock, _)) = listener.accept().await else {
continue;
};
tokio::spawn(async move {
tokio::time::sleep(Duration::from_millis(80)).await;
let _ = sock.write_all(b"UNSOLICITED").await;
});
}
});
let chain = ChainHandle::new(FilterChain::new(
parse_filters("block:FLAG{").unwrap(),
Duration::from_millis(200),
));
let proxy = Proxy::bind(
ProxyConfig::fixed("127.0.0.1:0".parse().unwrap(), upstream),
chain,
)
.await
.unwrap();
let addr = proxy.local_addr().unwrap();
tokio::spawn(proxy.serve());
let mut sock = TcpStream::connect(addr).await.unwrap();
sock.write_all(b"give me FLAG{x}").await.unwrap();
let mut buf = vec![0u8; 1024];
let n = tokio::time::timeout(Duration::from_secs(3), sock.read(&mut buf))
.await
.unwrap()
.unwrap();
assert_eq!(
n,
0,
"the server's greeting reached a client whose request was blocked: {:?}",
&buf[..n]
);
}
/// A broken filter must not take its healthy neighbours down with it.
#[tokio::test]
async fn one_broken_filter_does_not_disable_the_others() {
let (addr, chain) = spawn_proxy("panic,block:FLAG{", 200).await;
// Trip the panicking filter.
let got = roundtrip(addr, b"warmup").await.unwrap();
assert_eq!(got, b"warmup");
let live = chain.current();
assert_eq!(live.disabled_filters(), vec!["panic"]);
assert!(
!live.is_bypassed(),
"the surviving filter should keep the chain active"
);
// The healthy one is still enforcing.
let got = roundtrip(addr, b"take the FLAG{x}").await.unwrap();
assert!(got.is_empty(), "the surviving filter stopped enforcing");
}
/// Reconfiguration must not cost established connections — the NFQUEUE engine
/// reloads its rules in place, and this one may not be worse.
#[tokio::test]
async fn chain_swap_applies_to_live_connections() {
let (addr, chain) = spawn_proxy("", 200).await;
let mut sock = TcpStream::connect(addr).await.unwrap();
sock.write_all(b"before").await.unwrap();
let mut buf = vec![0u8; 1024];
let n = sock.read(&mut buf).await.unwrap();
assert_eq!(&buf[..n], b"before");
// Swap in a rewriting chain while the connection is open.
chain.replace(FilterChain::new(
parse_filters("block:after").unwrap(),
Duration::from_millis(200),
));
sock.write_all(b"after").await.unwrap();
let n = tokio::time::timeout(Duration::from_secs(5), sock.read(&mut buf))
.await
.unwrap()
.unwrap();
assert_eq!(
n,
0,
"the swapped-in chain did not close the connection"
);
}
/// A filter that panics only sometimes still loses its say on the first panic,
/// and the chunk that tripped it is forwarded rather than dropped.
struct FlakyFilter {
seen: std::sync::atomic::AtomicU64,
}
impl Filter for FlakyFilter {
fn name(&self) -> &str {
"flaky"
}
fn inspect(&self, _ctx: &FilterCtx<'_>) -> Verdict {
if self.seen.fetch_add(1, Ordering::Relaxed) == 1 {
panic!("second call blows up");
}
Verdict::Accept
}
}
#[tokio::test]
async fn chunk_that_trips_a_panic_is_still_delivered() {
let upstream = spawn_echo().await;
let chain = ChainHandle::new(FilterChain::new(
vec![Arc::new(FlakyFilter {
seen: std::sync::atomic::AtomicU64::new(0),
})],
Duration::from_millis(200),
));
let proxy = Proxy::bind(
ProxyConfig::fixed("127.0.0.1:0".parse().unwrap(), upstream),
chain.clone(),
)
.await
.unwrap();
let addr = proxy.local_addr().unwrap();
tokio::spawn(proxy.serve());
let mut sock = TcpStream::connect(addr).await.unwrap();
let mut buf = vec![0u8; 1024];
for expected in [&b"first"[..], &b"second"[..], &b"third"[..]] {
sock.write_all(expected).await.unwrap();
let n = tokio::time::timeout(Duration::from_secs(5), sock.read(&mut buf))
.await
.unwrap()
.unwrap();
assert_eq!(&buf[..n], expected, "a chunk was lost around the panic");
}
assert_eq!(chain.current().disabled_filters(), vec!["flaky"]);
}

1135
backend/proxysrc/tests/h2.rs Normal file
View File

File diff suppressed because it is too large Load Diff

885
backend/proxysrc/tests/h3.rs Normal file
View File

@ -0,0 +1,885 @@
//! HTTP/3 carried by the engine, and shown to the filters as HTTP/1.1.
//!
//! The point of the translation is that a filter written once works on both: a pattern
//! that blocks a path over TCP has to block the same path over QUIC, where on the wire
//! that path was a QPACK-compressed header block. These tests put a real HTTP/3 client and
//! a real HTTP/3 service either side of the relay and check what the chain was shown.
use std::net::SocketAddr;
use std::sync::Arc;
use std::time::Duration;
use bytes::{Buf, Bytes};
use fgex_proxy::filter::{ChainHandle, FilterChain};
use fgex_proxy::proxy::ProxyStats;
use fgex_proxy::proxy::Onward;
use fgex_proxy::quic::{QuicConfig, QuicRelay, QuicSetup};
use fgex_proxy::spec::parse_filters;
use fgex_proxy::tls;
use quinn::crypto::rustls::{QuicClientConfig, QuicServerConfig};
fn self_signed() -> (String, String) {
let cert =
rcgen::generate_simple_self_signed(vec!["localhost".to_string(), "127.0.0.1".to_string()])
.unwrap();
(cert.cert.pem(), cert.key_pair.serialize_pem())
}
/// An HTTP/3 service that answers with what it was asked, body and all.
async fn spawn_h3_echo(cert: &str, key: &str) -> SocketAddr {
let mut config = (*tls::server_config(cert, key).unwrap()).clone();
config.alpn_protocols = vec![b"h3".to_vec()];
let endpoint = quinn::Endpoint::server(
quinn::ServerConfig::with_crypto(Arc::new(QuicServerConfig::try_from(config).unwrap())),
"127.0.0.1:0".parse().unwrap(),
)
.unwrap();
let addr = endpoint.local_addr().unwrap();
tokio::spawn(async move {
while let Some(incoming) = endpoint.accept().await {
tokio::spawn(async move {
let Ok(connection) = incoming.await else {
return;
};
let Ok(mut h3) = h3::server::builder()
.build(h3_quinn::Connection::new(connection))
.await
else {
return;
};
while let Ok(Some(resolver)) = h3.accept().await {
tokio::spawn(async move {
let Ok((request, mut stream)) = resolver.resolve_request().await else {
return;
};
let mut body = Vec::new();
while let Ok(Some(mut chunk)) = stream.recv_data().await {
let len = chunk.remaining();
body.extend_from_slice(&chunk.copy_to_bytes(len));
}
// One path answers with something a rule is looking for, and
// says it in bytes that appear nowhere in the request: that is
// what makes a test of the *answer* a test of the answer.
let answer = if request.uri().path() == "/secret" {
"here is FLAG{only-in-the-answer}".to_string()
} else {
format!(
"you asked for {} and sent {} bytes: {}",
request.uri().path(),
body.len(),
String::from_utf8_lossy(&body)
)
};
let response = http::Response::builder()
.status(http::StatusCode::OK)
.header("content-length", answer.len().to_string())
.body(())
.unwrap();
let _ = stream.send_response(response).await;
let _ = stream.send_data(Bytes::from(answer)).await;
let _ = stream.finish().await;
});
}
});
}
});
addr
}
/// A service that speaks HTTP/1.1 and nothing else, which is what most of the web is.
///
/// Deliberately hand-written rather than a crate: what these tests are about is what
/// leaves this engine, so the far end has to be something that reads exactly what an
/// HTTP/1.1 service reads and nothing more forgiving.
async fn spawn_h1_echo() -> SocketAddr {
let listener = tokio::net::TcpListener::bind("127.0.0.1:0").await.unwrap();
let addr = listener.local_addr().unwrap();
tokio::spawn(async move {
loop {
let Ok((mut socket, _)) = listener.accept().await else {
return;
};
tokio::spawn(async move {
use tokio::io::{AsyncReadExt, AsyncWriteExt};
let mut buf = Vec::new();
let mut chunk = [0u8; 4096];
// The head, then whatever the framing says the body is.
let head_end = loop {
let n = match socket.read(&mut chunk).await {
Ok(0) | Err(_) => return,
Ok(n) => n,
};
buf.extend_from_slice(&chunk[..n]);
if let Some(at) = buf.windows(4).position(|w| w == b"\r\n\r\n") {
break at + 4;
}
};
let head = String::from_utf8_lossy(&buf[..head_end]).to_lowercase();
let path = head.split_whitespace().nth(1).unwrap_or("/").to_string();
let declared = head
.split("content-length:")
.nth(1)
.and_then(|rest| rest.split("\r\n").next())
.and_then(|v| v.trim().parse::<usize>().ok());
let mut body = buf[head_end..].to_vec();
match declared {
Some(want) => {
while body.len() < want {
let n = match socket.read(&mut chunk).await {
Ok(0) | Err(_) => break,
Ok(n) => n,
};
body.extend_from_slice(&chunk[..n]);
}
body.truncate(want.min(body.len()));
}
// Chunked, which is what an undeclared body is in HTTP/1.1 — and what
// the rendering shows a filter, so it is what has to arrive here.
None if head.contains("transfer-encoding: chunked") => {
let mut decoded = Vec::new();
loop {
while !body.windows(2).any(|w| w == b"\r\n") {
let n = match socket.read(&mut chunk).await {
Ok(0) | Err(_) => break,
Ok(n) => n,
};
body.extend_from_slice(&chunk[..n]);
}
let Some(at) = body.windows(2).position(|w| w == b"\r\n") else {
break;
};
let size = usize::from_str_radix(
String::from_utf8_lossy(&body[..at]).trim(),
16,
)
.unwrap_or(0);
if size == 0 {
break;
}
body.drain(..at + 2);
while body.len() < size + 2 {
let n = match socket.read(&mut chunk).await {
Ok(0) | Err(_) => break,
Ok(n) => n,
};
body.extend_from_slice(&chunk[..n]);
}
decoded.extend_from_slice(&body[..size.min(body.len())]);
body.drain(..(size + 2).min(body.len()));
}
body = decoded;
}
None => body.clear(),
}
let answer = format!(
"you asked for {path} and sent {} bytes: {}",
body.len(),
String::from_utf8_lossy(&body)
);
let _ = socket
.write_all(
format!(
"HTTP/1.1 200 OK\r\ncontent-length: {}\r\nconnection: close\r\n\r\n{answer}",
answer.len()
)
.as_bytes(),
)
.await;
let _ = socket.flush().await;
});
}
});
addr
}
async fn spawn_relay(upstream: SocketAddr, filters: &str) -> SocketAddr {
spawn_relay_to(upstream, filters, Onward::Same).await
}
async fn spawn_relay_to(
upstream: SocketAddr,
filters: &str,
protocol: Onward,
) -> SocketAddr {
let (cert, key) = self_signed();
let chain = ChainHandle::new(FilterChain::new(
parse_filters(filters).unwrap(),
Duration::from_millis(500),
));
let cfg = QuicConfig {
setup: Arc::new(QuicSetup::build(&cert, &key).unwrap()),
self_mark: None,
spoof_source: false,
max_connections: 0,
over_limit_forwards: false,
first_byte_timeout: None,
connect_timeout: Duration::from_secs(5),
upstream: protocol,
// No interface to reconstruct anything onto, which is every host without the
// capture device — what it would carry is pinned by the integration suite, which
// has a real one.
capture: None,
};
let relay = QuicRelay::bind(
"127.0.0.1:0".parse().unwrap(),
upstream,
chain,
cfg,
Arc::new(ProxyStats::default()),
)
.unwrap();
let addr = relay.local_addr().unwrap();
tokio::spawn(Arc::new(relay).serve());
addr
}
/// One request through the relay, and what came back.
async fn request(
addr: SocketAddr,
method: &str,
path: &str,
body: Option<&[u8]>,
declare_length: bool,
) -> Result<String, String> {
let mut config = tls::quic_client_config().unwrap();
config.alpn_protocols = vec![b"h3".to_vec()];
let mut endpoint = quinn::Endpoint::client("127.0.0.1:0".parse().unwrap()).unwrap();
endpoint.set_default_client_config(quinn::ClientConfig::new(Arc::new(
QuicClientConfig::try_from(config).unwrap(),
)));
let connection = endpoint
.connect(addr, "localhost")
.map_err(|e| e.to_string())?
.await
.map_err(|e| e.to_string())?;
let (mut driver, mut sender) = h3::client::new(h3_quinn::Connection::new(connection))
.await
.map_err(|e| e.to_string())?;
let driving = tokio::spawn(async move {
let _ = driver.wait_idle().await;
});
let answer = async {
let mut builder = http::Request::builder()
.method(method)
.uri(format!("https://localhost{path}"));
if declare_length {
builder = builder.header("content-length", body.map(|b| b.len()).unwrap_or(0).to_string());
}
let outgoing = builder.body(()).map_err(|e| e.to_string())?;
let mut stream = sender.send_request(outgoing).await.map_err(|e| e.to_string())?;
if let Some(body) = body {
stream
.send_data(Bytes::copy_from_slice(body))
.await
.map_err(|e| e.to_string())?;
}
stream.finish().await.map_err(|e| e.to_string())?;
stream.recv_response().await.map_err(|e| e.to_string())?;
let mut got = Vec::new();
while let Some(mut chunk) = stream.recv_data().await.map_err(|e| e.to_string())? {
let len = chunk.remaining();
got.extend_from_slice(&chunk.copy_to_bytes(len));
}
Ok(String::from_utf8_lossy(&got).into_owned())
}
.await;
driving.abort();
answer
}
#[tokio::test]
async fn an_exchange_goes_through() {
let (cert, key) = self_signed();
let upstream = spawn_h3_echo(&cert, &key).await;
let addr = spawn_relay(upstream, "").await;
let answer = request(addr, "GET", "/hello", None, false).await.unwrap();
assert_eq!(answer, "you asked for /hello and sent 0 bytes: ");
}
/// The whole reason for rendering the exchange: a pattern written against an HTTP
/// request matches one that arrived as a QPACK header block.
#[tokio::test]
async fn a_pattern_matches_the_request_line() {
let (cert, key) = self_signed();
let upstream = spawn_h3_echo(&cert, &key).await;
let addr = spawn_relay(upstream, "block:GET /admin").await;
let allowed = request(addr, "GET", "/public", None, false).await.unwrap();
assert!(allowed.contains("/public"), "an unrelated request was affected: {allowed}");
let refused = request(addr, "GET", "/admin", None, false).await;
assert!(
refused.is_err(),
"a request the chain refused was answered anyway: {refused:?}"
);
}
/// A header the request did not carry must not appear in what the filters are shown.
///
/// This is what the head is held back for. A bodiless request rendered as
/// `transfer-encoding: chunked` would be a lie in front of every filter looking for that
/// header — and a filter looking for it is usually looking for request smuggling, which
/// makes a fabricated one the worst possible answer.
#[tokio::test]
async fn a_request_with_no_body_is_shown_no_framing_header() {
let (cert, key) = self_signed();
let upstream = spawn_h3_echo(&cert, &key).await;
let addr = spawn_relay(upstream, "block:transfer-encoding").await;
let answer = request(addr, "GET", "/plain", None, false).await.unwrap();
assert!(answer.contains("/plain"), "a framing header was invented: {answer}");
}
/// And a body whose length was never declared is shown chunked, because that is what
/// that message is in HTTP/1.1 — so a parser reading the rendering finds a body.
#[tokio::test]
async fn a_body_of_unknown_length_is_shown_chunked() {
let (cert, key) = self_signed();
let upstream = spawn_h3_echo(&cert, &key).await;
let addr = spawn_relay(upstream, "block:transfer-encoding").await;
let refused = request(addr, "POST", "/upload", Some(b"anything"), false).await;
assert!(
refused.is_err(),
"a body with no declared length was shown unframed: {refused:?}"
);
}
/// A declared length is passed through as it was, and the body shown raw beneath it.
#[tokio::test]
async fn a_declared_length_is_left_alone() {
let (cert, key) = self_signed();
let upstream = spawn_h3_echo(&cert, &key).await;
let addr = spawn_relay(upstream, "block:transfer-encoding").await;
let answer = request(addr, "POST", "/upload", Some(b"declared"), true)
.await
.unwrap();
assert!(answer.contains("sent 8 bytes"), "the body did not arrive: {answer}");
}
/// Both directions are inspected, and a refused answer does not reach the client.
#[tokio::test]
async fn the_answer_is_inspected_too() {
let (cert, key) = self_signed();
let upstream = spawn_h3_echo(&cert, &key).await;
let addr = spawn_relay(upstream, "block:FLAG{").await;
let answer = request(addr, "GET", "/harmless", None, false).await;
assert!(answer.is_ok(), "an unrelated exchange was affected: {answer:?}");
// Nothing in this request carries the needle; everything that does is in the reply.
let refused = request(addr, "GET", "/secret", None, false).await;
match refused {
Err(_) => {}
Ok(body) => panic!("the answer reached the client anyway: {body}"),
}
}
/// A service that answers a fixed body and remembers the trailer section it was sent.
///
/// Separate from the echo above on purpose: what these tests are asking is what reached
/// the *service*, which an answer that reflects the request cannot tell them apart from
/// what reached the client.
async fn spawn_h3_recorder(
cert: &str,
key: &str,
) -> (SocketAddr, Arc<std::sync::Mutex<Vec<String>>>) {
let seen: Arc<std::sync::Mutex<Vec<String>>> = Arc::new(std::sync::Mutex::new(Vec::new()));
let mut config = (*tls::server_config(cert, key).unwrap()).clone();
config.alpn_protocols = vec![b"h3".to_vec()];
let endpoint = quinn::Endpoint::server(
quinn::ServerConfig::with_crypto(Arc::new(QuicServerConfig::try_from(config).unwrap())),
"127.0.0.1:0".parse().unwrap(),
)
.unwrap();
let addr = endpoint.local_addr().unwrap();
let recorded = Arc::clone(&seen);
tokio::spawn(async move {
while let Some(incoming) = endpoint.accept().await {
let recorded = Arc::clone(&recorded);
tokio::spawn(async move {
let Ok(connection) = incoming.await else { return };
let Ok(mut h3) = h3::server::builder()
.build(h3_quinn::Connection::new(connection))
.await
else {
return;
};
while let Ok(Some(resolver)) = h3.accept().await {
let recorded = Arc::clone(&recorded);
tokio::spawn(async move {
let Ok((_request, mut stream)) = resolver.resolve_request().await else {
return;
};
while let Ok(Some(_)) = stream.recv_data().await {}
if let Ok(Some(trailers)) = stream.recv_trailers().await {
for (name, value) in trailers.iter() {
recorded.lock().unwrap().push(format!(
"{name}={}",
value.to_str().unwrap_or("?")
));
}
}
let answer = "recorded";
let response = http::Response::builder()
.status(http::StatusCode::OK)
.header("content-length", answer.len().to_string())
.body(())
.unwrap();
let _ = stream.send_response(response).await;
let _ = stream.send_data(Bytes::from(answer)).await;
let _ = stream.finish().await;
});
}
});
}
});
(addr, seen)
}
/// A service that answers the moment the head arrives, without waiting for a body.
///
/// The shape of a bidirectional RPC and of every service that replies before an upload
/// finishes — a `413`, a redirect, a refusal.
async fn spawn_h3_eager(cert: &str, key: &str) -> SocketAddr {
let mut config = (*tls::server_config(cert, key).unwrap()).clone();
config.alpn_protocols = vec![b"h3".to_vec()];
let endpoint = quinn::Endpoint::server(
quinn::ServerConfig::with_crypto(Arc::new(QuicServerConfig::try_from(config).unwrap())),
"127.0.0.1:0".parse().unwrap(),
)
.unwrap();
let addr = endpoint.local_addr().unwrap();
tokio::spawn(async move {
while let Some(incoming) = endpoint.accept().await {
tokio::spawn(async move {
let Ok(connection) = incoming.await else { return };
let Ok(mut h3) = h3::server::builder()
.build(h3_quinn::Connection::new(connection))
.await
else {
return;
};
while let Ok(Some(resolver)) = h3.accept().await {
tokio::spawn(async move {
let Ok((_request, mut stream)) = resolver.resolve_request().await else {
return;
};
let answer = "the service spoke first";
let response = http::Response::builder()
.status(http::StatusCode::OK)
.header("content-length", answer.len().to_string())
.body(())
.unwrap();
let _ = stream.send_response(response).await;
let _ = stream.send_data(Bytes::from(answer)).await;
let _ = stream.finish().await;
});
}
});
}
});
addr
}
/// One request that ends with a trailer section, and what came back.
async fn request_with_trailer(
addr: SocketAddr,
body: &[u8],
declare_length: bool,
trailer: &str,
) -> Result<String, String> {
let mut config = tls::quic_client_config().unwrap();
config.alpn_protocols = vec![b"h3".to_vec()];
let mut endpoint = quinn::Endpoint::client("127.0.0.1:0".parse().unwrap()).unwrap();
endpoint.set_default_client_config(quinn::ClientConfig::new(Arc::new(
QuicClientConfig::try_from(config).unwrap(),
)));
let connection = endpoint
.connect(addr, "localhost")
.map_err(|e| e.to_string())?
.await
.map_err(|e| e.to_string())?;
let (mut driver, mut sender) = h3::client::new(h3_quinn::Connection::new(connection))
.await
.map_err(|e| e.to_string())?;
let driving = tokio::spawn(async move {
let _ = driver.wait_idle().await;
});
let answer = async {
let mut builder = http::Request::builder()
.method("POST")
.uri("https://localhost/up");
if declare_length {
builder = builder.header("content-length", body.len().to_string());
}
let outgoing = builder.body(()).map_err(|e| e.to_string())?;
let mut stream = sender.send_request(outgoing).await.map_err(|e| e.to_string())?;
if !body.is_empty() {
stream
.send_data(Bytes::copy_from_slice(body))
.await
.map_err(|e| e.to_string())?;
}
let mut trailers = http::HeaderMap::new();
trailers.insert("x-note", http::HeaderValue::from_str(trailer).unwrap());
stream.send_trailers(trailers).await.map_err(|e| e.to_string())?;
let _ = stream.finish().await;
stream.recv_response().await.map_err(|e| e.to_string())?;
let mut got = Vec::new();
while let Some(mut chunk) = stream.recv_data().await.map_err(|e| e.to_string())? {
let len = chunk.remaining();
got.extend_from_slice(&chunk.copy_to_bytes(len));
}
Ok(String::from_utf8_lossy(&got).into_owned())
}
.await;
driving.abort();
answer
}
/// A trailer section on a message whose body was never framed by a declared length is
/// shown to the chain, which is the only reason it may then be forwarded.
///
/// The bodiless case is the one that matters most and the one that used to slip past: a
/// message of HEADERS and TRAILERS and nothing in between — which is exactly a gRPC
/// trailers-only answer — had no chunked body to hang a trailer section off, so the
/// rendering ended at the head and the trailers travelled to the far side unread. They
/// decide the framing now: in HTTP/1.1 a message with trailers *is* a chunked one.
#[tokio::test]
async fn a_trailer_section_is_shown_to_the_chain() {
let (cert, key) = self_signed();
let upstream = spawn_h3_echo(&cert, &key).await;
let addr = spawn_relay(upstream, "block:SNEAKY").await;
for (body, declared, what) in [
(&b""[..], false, "no body at all"),
(&b"harmless"[..], false, "a body of undeclared length"),
] {
let refused = request_with_trailer(addr, body, declared, "SNEAKY").await;
assert!(
refused.is_err(),
"a needle in the trailers of a message with {what} was not seen: {refused:?}"
);
}
}
/// And where the rendering cannot carry one, it is dropped rather than forwarded unread.
///
/// A declared length means the HTTP/1.1 view of the message ends where the header said it
/// would, so there is nowhere to put a trailer section — and forwarding a piece of a
/// message no filter was shown is the one thing this engine does not do anywhere else.
/// Dropping is what an intermediary is allowed to do with a trailer section it cannot
/// carry; the exchange itself is untouched.
#[tokio::test]
async fn a_trailer_the_rendering_cannot_carry_is_not_forwarded() {
let (cert, key) = self_signed();
let (upstream, seen) = spawn_h3_recorder(&cert, &key).await;
let addr = spawn_relay(upstream, "").await;
let answer = request_with_trailer(addr, b"harmless", true, "SNEAKY").await;
assert_eq!(answer.as_deref(), Ok("recorded"), "the exchange was disturbed");
assert!(
seen.lock().unwrap().is_empty(),
"a trailer section no filter was shown reached the service: {:?}",
seen.lock().unwrap()
);
// Without the declared length the same trailers are renderable, so they are shown and
// then carried: the rule is what the chain saw, not a dislike of trailers.
let (upstream, seen) = spawn_h3_recorder(&cert, &key).await;
let addr = spawn_relay(upstream, "").await;
let answer = request_with_trailer(addr, b"harmless", false, "SNEAKY").await;
assert_eq!(answer.as_deref(), Ok("recorded"));
assert_eq!(
seen.lock().unwrap().as_slice(),
["x-note=SNEAKY"],
"a trailer section the chain accepted was dropped anyway"
);
}
/// An exchange the service is supposed to speak first in is not held waiting for a body.
///
/// The head is held only until the framing is known, and a client that sends a head and
/// then waits to be answered — a bidirectional RPC, gRPC's among them — never sends one.
/// Holding it there deadlocked the exchange: the proxy waiting for a body, the client
/// waiting for the answer that would tell it what to send.
#[tokio::test]
async fn a_service_that_speaks_first_is_not_waited_out() {
let (cert, key) = self_signed();
let upstream = spawn_h3_eager(&cert, &key).await;
let addr = spawn_relay(upstream, "").await;
let mut config = tls::quic_client_config().unwrap();
config.alpn_protocols = vec![b"h3".to_vec()];
let mut endpoint = quinn::Endpoint::client("127.0.0.1:0".parse().unwrap()).unwrap();
endpoint.set_default_client_config(quinn::ClientConfig::new(Arc::new(
QuicClientConfig::try_from(config).unwrap(),
)));
let connection = endpoint.connect(addr, "localhost").unwrap().await.unwrap();
let (mut driver, mut sender) = h3::client::new(h3_quinn::Connection::new(connection))
.await
.unwrap();
let driving = tokio::spawn(async move {
let _ = driver.wait_idle().await;
});
let outgoing = http::Request::builder()
.method("POST")
.uri("https://localhost/stream")
.body(())
.unwrap();
let mut stream = sender.send_request(outgoing).await.unwrap();
// Deliberately no body and no finish: this client is waiting to be spoken to.
let answered = tokio::time::timeout(Duration::from_secs(5), stream.recv_response()).await;
driving.abort();
assert!(
answered.is_ok(),
"the head never reached a service that answers without waiting for a body"
);
assert_eq!(answered.unwrap().unwrap().status(), http::StatusCode::OK);
}
// --- a service that speaks HTTP/1.1, behind an HTTP/3 edge ----------------------------
//
// The one place in this engine where what leaves is not the version that arrived. It is
// the operator's choice (`FGEX_PROXY_UPSTREAM`) and it exists because most of the web is
// an HTTP/1.1 service that will never speak QUIC: firegex terminates HTTP/3 in front of
// it and sends on the rendering the filters were already being shown.
#[tokio::test]
async fn an_http1_service_answers_an_http3_client() {
let upstream = spawn_h1_echo().await;
let addr = spawn_relay_to(upstream, "", Onward::Plain).await;
let answer = request(addr, "GET", "/hello", None, false).await.unwrap();
assert_eq!(answer, "you asked for /hello and sent 0 bytes: ");
}
/// A body, declared — which is the framing that travels as a `content-length` and is
/// read back by the far end as one.
#[tokio::test]
async fn a_declared_body_reaches_an_http1_service() {
let upstream = spawn_h1_echo().await;
let addr = spawn_relay_to(upstream, "", Onward::Plain).await;
let answer = request(addr, "POST", "/upload", Some(b"twelve bytes"), true)
.await
.unwrap();
assert_eq!(answer, "you asked for /upload and sent 12 bytes: twelve bytes");
}
/// And one that declared nothing, which is chunked in HTTP/1.1 — the framing the chain
/// is shown, so it had better be the framing the service is sent.
#[tokio::test]
async fn an_undeclared_body_reaches_an_http1_service_chunked() {
let upstream = spawn_h1_echo().await;
let addr = spawn_relay_to(upstream, "", Onward::Plain).await;
let answer = request(addr, "POST", "/upload", Some(b"twelve bytes"), false)
.await
.unwrap();
assert_eq!(answer, "you asked for /upload and sent 12 bytes: twelve bytes");
}
/// The promise that makes the whole thing worth having: one filter file, and it does not
/// care which version either end speaks. The same pattern that blocks this request
/// against an HTTP/3 service blocks it against an HTTP/1.1 one.
#[tokio::test]
async fn a_pattern_matches_with_an_http1_service_behind() {
let upstream = spawn_h1_echo().await;
let addr = spawn_relay_to(upstream, "block:GET /admin", Onward::Plain).await;
assert!(
request(addr, "GET", "/admin", None, false).await.is_err(),
"the request reached an HTTP/1.1 service that a pattern should have refused"
);
let answer = request(addr, "GET", "/public", None, false).await.unwrap();
assert_eq!(answer, "you asked for /public and sent 0 bytes: ");
}
/// A `host` header that disagrees with `:authority` is malformed, and never reaches the
/// service — the HTTP/3 half of the same case in `h2.rs`.
#[tokio::test]
async fn a_host_that_disagrees_with_the_authority_is_refused() {
let (cert, key) = self_signed();
let upstream = spawn_h3_echo(&cert, &key).await;
let addr = spawn_relay(upstream, "").await;
let ask = |host: &'static str| async move {
let mut config = tls::quic_client_config().unwrap();
config.alpn_protocols = vec![b"h3".to_vec()];
let mut endpoint = quinn::Endpoint::client("127.0.0.1:0".parse().unwrap()).unwrap();
endpoint.set_default_client_config(quinn::ClientConfig::new(Arc::new(
QuicClientConfig::try_from(config).unwrap(),
)));
let connection = endpoint
.connect(addr, "localhost")
.map_err(|e| e.to_string())?
.await
.map_err(|e| e.to_string())?;
let (mut driver, mut sender) = h3::client::new(h3_quinn::Connection::new(connection))
.await
.map_err(|e| e.to_string())?;
let driving = tokio::spawn(async move {
let _ = driver.wait_idle().await;
});
let answer = async {
let outgoing = http::Request::builder()
.method("GET")
.uri("https://localhost/who")
.header("host", host)
.body(())
.map_err(|e| e.to_string())?;
let mut stream = sender.send_request(outgoing).await.map_err(|e| e.to_string())?;
stream.finish().await.map_err(|e| e.to_string())?;
stream.recv_response().await.map_err(|e| e.to_string())?;
let mut got = Vec::new();
while let Some(mut chunk) = stream.recv_data().await.map_err(|e| e.to_string())? {
let len = chunk.remaining();
got.extend_from_slice(&chunk.copy_to_bytes(len));
}
Ok::<String, String>(String::from_utf8_lossy(&got).into_owned())
}
.await;
driving.abort();
answer
};
let agreeing = ask("localhost").await;
assert!(
agreeing.as_deref().is_ok_and(|a| a.contains("/who")),
"a host agreeing with :authority was not carried: {agreeing:?}"
);
let disagreeing = tokio::time::timeout(Duration::from_secs(5), ask("admin.internal"))
.await
.expect("a malformed request was left hanging");
assert!(disagreeing.is_err(), "a disagreeing host was carried: {disagreeing:?}");
}
/// One GET through the relay carrying `headers`, and the answer's status, head and body.
async fn ask_with(
addr: SocketAddr,
headers: &[(&'static str, &'static str)],
) -> Result<(u16, http::HeaderMap, String), String> {
let mut config = tls::quic_client_config().unwrap();
config.alpn_protocols = vec![b"h3".to_vec()];
let mut endpoint = quinn::Endpoint::client("127.0.0.1:0".parse().unwrap()).unwrap();
endpoint.set_default_client_config(quinn::ClientConfig::new(Arc::new(
QuicClientConfig::try_from(config).unwrap(),
)));
let connection = endpoint
.connect(addr, "localhost")
.map_err(|e| e.to_string())?
.await
.map_err(|e| e.to_string())?;
let (mut driver, mut sender) = h3::client::new(h3_quinn::Connection::new(connection))
.await
.map_err(|e| e.to_string())?;
let driving = tokio::spawn(async move {
let _ = driver.wait_idle().await;
});
let answer = async {
let mut builder = http::Request::builder().method("GET").uri("https://localhost/who");
for (name, value) in headers {
builder = builder.header(*name, *value);
}
let outgoing = builder.body(()).map_err(|e| e.to_string())?;
let mut stream = sender.send_request(outgoing).await.map_err(|e| e.to_string())?;
stream.finish().await.map_err(|e| e.to_string())?;
let response = stream.recv_response().await.map_err(|e| e.to_string())?;
let mut got = Vec::new();
while let Some(mut chunk) = stream.recv_data().await.map_err(|e| e.to_string())? {
let len = chunk.remaining();
got.extend_from_slice(&chunk.copy_to_bytes(len));
}
Ok::<_, String>((
response.status().as_u16(),
response.headers().clone(),
String::from_utf8_lossy(&got).into_owned(),
))
}
.await;
driving.abort();
answer
}
/// A header about a connection is malformed in HTTP/3, and the h3 crate does not refuse
/// it. Carried to a service reached over HTTP/1.1 it would be read there as a statement
/// about *that* connection — which headers to drop, which transfer coding to undo — and a
/// filter shown the rendering would not recognise the request the service got.
#[tokio::test]
async fn a_connection_header_is_refused_over_http3() {
let upstream = spawn_h1_echo().await;
let addr = spawn_relay_to(upstream, "", Onward::Plain).await;
let trailers = ask_with(addr, &[("te", "trailers")]).await;
assert!(trailers.is_ok(), "`te: trailers`, the one allowed, was refused: {trailers:?}");
for header in [("transfer-encoding", "chunked"), ("connection", "x-forwarded-for"),
("keep-alive", "timeout=5"), ("upgrade", "websocket"), ("te", "gzip")] {
let refused = tokio::time::timeout(Duration::from_secs(5), ask_with(addr, &[header]))
.await
.expect("a malformed request was left hanging");
assert!(refused.is_err(), "a request carrying {header:?} was carried: {refused:?}");
}
}
/// What an HTTP/1.1 service answers with says things about its own connection — a body
/// sent chunked, a connection offered for reuse — that an HTTP/3 client is required to
/// reject a response for. They are the service's, and they stay behind.
#[tokio::test]
async fn an_http1_answer_reaches_an_http3_client_without_its_connection_headers() {
let listener = tokio::net::TcpListener::bind("127.0.0.1:0").await.unwrap();
let upstream = listener.local_addr().unwrap();
tokio::spawn(async move {
while let Ok((mut socket, _)) = listener.accept().await {
tokio::spawn(async move {
use tokio::io::{AsyncReadExt, AsyncWriteExt};
let mut buf = [0u8; 4096];
let _ = socket.read(&mut buf).await;
let _ = socket
.write_all(
b"HTTP/1.1 200 OK\r\nconnection: keep-alive, x-internal\r\n\
keep-alive: timeout=5\r\nx-internal: secret\r\n\
content-type: text/plain\r\ntransfer-encoding: chunked\r\n\r\n\
5\r\nhello\r\n0\r\n\r\n",
)
.await;
});
}
});
let addr = spawn_relay_to(upstream, "", Onward::Plain).await;
let (_, headers, body) = ask_with(addr, &[]).await.unwrap();
assert_eq!(body, "hello");
for name in ["connection", "keep-alive", "transfer-encoding", "x-internal"] {
assert!(!headers.contains_key(name), "`{name}` reached the HTTP/3 client: {headers:?}");
}
assert_eq!(headers.get("content-type").map(|v| v.as_bytes()), Some(&b"text/plain"[..]));
}
/// A head is bounded, which the h3 crate does not do by default at all.
#[tokio::test]
async fn a_head_past_the_limit_is_refused_over_http3() {
let upstream = spawn_h1_echo().await;
let addr = spawn_relay_to(upstream, "", Onward::Plain).await;
let big: &'static str = Box::leak("a".repeat(2 * 1024 * 1024).into_boxed_str());
let refused = tokio::time::timeout(Duration::from_secs(10), ask_with(addr, &[("x-big", big)]))
.await
.expect("an oversized head was left hanging");
// The crate answers `431 Request Header Fields Too Large` itself, and nothing is sent on.
assert!(
!matches!(refused, Ok((200, _, _))),
"a 2 MiB head was carried: {:?}",
refused.map(|r| r.2)
);
let large: &'static str = Box::leak("a".repeat(256 * 1024).into_boxed_str());
assert!(
matches!(ask_with(addr, &[("x-large", large)]).await, Ok((200, _, _))),
"a 256 KiB head was refused"
);
}

609
backend/proxysrc/tests/pyworker.rs Normal file
View File

@ -0,0 +1,609 @@
//! User Python, and the promise that moving it out of process was worth it.
//!
//! The engine's own deadline can only walk away from a blocking thread; the thread
//! keeps its pool slot until the process restarts, which `fail_open.rs` asserts and
//! the README calls out. A child process is different: it can be killed. These tests
//! are the difference being real rather than claimed.
//!
//! Every filter here is written against the real `firegex.pyfilters` API — parameters
//! annotated with a model, verdicts from the library — because that is the only API
//! there is. These tests used to use a look-alike that took `(data, direction)`, which
//! meant they passed against something no user could have written from the docs.
use std::io::Write;
use std::time::{Duration, Instant};
use fgex_proxy::filter::{
next_connection_id, ChainSessions, ConnectionMeta, Direction, FilterChain, Verdict, L4,
};
use fgex_proxy::rules::{parse_ruleset, parse_ruleset_reusing};
/// Write a filter file and give back its path, kept for the process's lifetime.
fn filter_file(name: &str, body: &str) -> String {
let path = std::env::temp_dir().join(format!("fgex-pyworker-{name}.py"));
let mut f = std::fs::File::create(&path).unwrap();
f.write_all(body.as_bytes()).unwrap();
path.to_string_lossy().into_owned()
}
/// The worker the backend ships, found relative to this crate.
fn worker() -> String {
std::path::Path::new(env!("CARGO_MANIFEST_DIR"))
.join("../modules/services/pyworker.py")
.to_string_lossy()
.into_owned()
}
fn chain_for(path: &str, timeout_ms: u64) -> FilterChain {
chain_with_timeout(path, None, timeout_ms)
}
/// A chain over one python file, saying which of its functions are switched on.
///
/// `None` leaves the field out entirely, which means every function the file defines;
/// `Some(&[])` switches all of them off. The two are deliberately different answers.
fn chain_with(path: &str, enabled: Option<&[&str]>) -> FilterChain {
chain_with_timeout(path, Some(enabled), 2000)
}
fn chain_with_timeout(
path: &str,
enabled: Option<Option<&[&str]>>,
timeout_ms: u64,
) -> FilterChain {
let selection = match enabled {
None | Some(None) => String::new(),
Some(Some(names)) => format!(
r#""enabled":[{}],"#,
names
.iter()
.map(|n| format!("\"{n}\""))
.collect::<Vec<_>>()
.join(",")
),
};
let json = format!(
r#"[{{"kind":"python","id":"py1","code_path":"{path}","timeout_ms":{timeout_ms},
{selection}"command":["python3","{}"]}}]"#,
worker()
);
FilterChain::new(
parse_ruleset(&json).unwrap(),
// Well past the worker's own, so the worker's deadline is what fires and the
// chain is not the thing masking it.
Duration::from_millis(timeout_ms * 4 + 2000),
)
}
async fn feed(c: &FilterChain, chunks: &[&[u8]]) -> Vec<Verdict> {
feed_from(c, chunks, None).await
}
/// Feed chunks, optionally telling the chain who is talking to whom first.
async fn feed_from(
c: &FilterChain,
chunks: &[&[u8]],
meta: Option<ConnectionMeta>,
) -> Vec<Verdict> {
let connection = next_connection_id();
if let Some(meta) = meta {
c.connection_opened(connection, &meta);
}
let mut sessions = ChainSessions::new(connection);
let mut out = Vec::new();
for chunk in chunks {
out.push(c.run(Direction::ClientToServer, chunk, &mut sessions).await);
}
out
}
#[tokio::test]
async fn a_python_filter_blocks_and_passes() {
let path = filter_file(
"block",
r#"
from firegex.pyfilters import pyfilter, ACCEPT, REJECT
from firegex.pyfilters.models import RawPacket
@pyfilter
def block_traversal(packet: RawPacket):
return REJECT if b"../" in packet.data else ACCEPT
"#,
);
let c = chain_for(&path, 1000);
assert_eq!(feed(&c, &[b"GET /index"]).await, vec![Verdict::Accept]);
assert_eq!(
feed(&c, &[b"GET /../../etc/passwd"]).await,
vec![Verdict::Reject(Some("py1/block_traversal".to_string()))]
);
}
/// The headline. A filter that never returns is killed, the chunk it was holding
/// goes through, and the next chunk gets a fresh worker rather than a dead engine.
#[tokio::test]
async fn a_hanging_python_filter_is_killed_and_traffic_continues() {
let path = filter_file(
"hang",
r#"
import time
from firegex.pyfilters import pyfilter, ACCEPT
from firegex.pyfilters.models import RawPacket
# Module level, so it is per connection: the library gives each stream its own globals.
seen = 0
@pyfilter
def sometimes_hangs(packet: RawPacket):
global seen
seen += 1
if seen == 1:
time.sleep(3600)
return ACCEPT
"#,
);
let c = chain_for(&path, 300);
let started = Instant::now();
assert_eq!(feed(&c, &[b"first"]).await, vec![Verdict::Accept]);
assert!(
started.elapsed() < Duration::from_secs(3),
"the chunk waited on the worker instead of failing open (took {:?})",
started.elapsed()
);
// A fresh worker takes over, so filtering resumes instead of being lost for good.
assert_eq!(feed(&c, &[b"second"]).await, vec![Verdict::Accept]);
assert_eq!(feed(&c, &[b"third"]).await, vec![Verdict::Accept]);
// The chain never had to disable anything: the worker absorbed the failure.
assert!(!c.is_bypassed());
assert!(c.disabled_filters().is_empty());
}
/// Telling the worker about a connection is not allowed to wait on it.
///
/// The open and close frames are sent from the relay itself, on the async runtime. They
/// used to take the worker's lock there — held by whatever exchange was in progress, up
/// to its whole deadline — so one connection feeding a hanging filter held every new
/// connection of the service on the threads that carry all the others.
#[tokio::test]
async fn a_connection_opening_does_not_wait_on_a_hanging_filter() {
let path = filter_file(
"hang-open",
r#"
import time
from firegex.pyfilters import pyfilter, ACCEPT
from firegex.pyfilters.models import RawPacket
@pyfilter
def hangs_on_request(packet: RawPacket):
if b"HANG" in packet.data:
time.sleep(3600)
return ACCEPT
"#,
);
let c = std::sync::Arc::new(chain_for(&path, 1500));
// Started first, so the worker is up and the only thing held is the exchange.
assert_eq!(feed(&c, &[b"warm"]).await, vec![Verdict::Accept]);
let stuck = {
let c = std::sync::Arc::clone(&c);
tokio::spawn(async move { feed(&c, &[b"HANG"]).await })
};
tokio::time::sleep(Duration::from_millis(300)).await;
let meta = ConnectionMeta {
client: "10.0.0.9:51000".parse().unwrap(),
server: "10.0.0.1:8080".parse().unwrap(),
l4: L4::Tcp,
};
let started = Instant::now();
let other = next_connection_id();
c.connection_opened(other, &meta);
c.connection_closed(other);
assert!(
started.elapsed() < Duration::from_millis(200),
"a new connection waited {:?} on another connection's hanging filter",
started.elapsed()
);
assert_eq!(stuck.await.unwrap(), vec![Verdict::Accept]);
}
/// A filter that crashes on one chunk must not cost the ones after it.
#[tokio::test]
async fn a_crashing_python_filter_recovers() {
let path = filter_file(
"crash",
r#"
import os
from firegex.pyfilters import pyfilter, ACCEPT, REJECT
from firegex.pyfilters.models import RawPacket
@pyfilter
def dies_once(packet: RawPacket):
if b"boom" in packet.data:
os._exit(1)
return REJECT if b"deny" in packet.data else ACCEPT
"#,
);
let c = chain_for(&path, 500);
assert_eq!(feed(&c, &[b"hello"]).await, vec![Verdict::Accept]);
// The worker dies mid-answer: the chunk is forwarded rather than held.
assert_eq!(feed(&c, &[b"boom"]).await, vec![Verdict::Accept]);
// And the replacement is enforcing again.
assert_eq!(
feed(&c, &[b"deny me"]).await,
vec![Verdict::Reject(Some("py1/dies_once".to_string()))]
);
}
/// A filter file that does not even import must not take the traffic with it.
#[tokio::test]
async fn a_worker_that_cannot_start_fails_open() {
let path = filter_file("broken", "this is not python at all(((\n");
let c = chain_for(&path, 500);
assert_eq!(feed(&c, &[b"still moving"]).await, vec![Verdict::Accept]);
}
/// A filter that raises costs the chunk its filtering, and nothing else.
///
/// The library runs the chain in one pass and does not isolate a filter that raises:
/// the exception aborts the whole chain for that packet, which is also what happens on
/// NFQUEUE. So the honest promise is not "the other filters still get a say", it is
/// "the traffic keeps moving and the next chunk is filtered again". Asserting the
/// former would be asserting a behaviour no transport actually has.
#[tokio::test]
async fn a_raising_filter_fails_open_and_the_worker_survives() {
let path = filter_file(
"raises",
r#"
from firegex.pyfilters import pyfilter, ACCEPT, REJECT
from firegex.pyfilters.models import RawPacket
@pyfilter
def sometimes_raises(packet: RawPacket):
if b"boom" in packet.data:
raise ValueError("boom")
return REJECT if b"deny" in packet.data else ACCEPT
"#,
);
let c = chain_for(&path, 1000);
assert_eq!(feed(&c, &[b"fine"]).await, vec![Verdict::Accept]);
// The chunk that made it raise goes through rather than being held.
assert_eq!(feed(&c, &[b"boom"]).await, vec![Verdict::Accept]);
// And the worker is still there, still enforcing.
assert_eq!(
feed(&c, &[b"deny me"]).await,
vec![Verdict::Reject(Some("py1/sometimes_raises".to_string()))]
);
assert!(!c.is_bypassed());
assert!(c.disabled_filters().is_empty());
}
/// A filter that prints must not break the protocol.
///
/// `print()` is the first thing anybody reaches for while debugging a filter, and the
/// worker shares its process with the user's code. On stdout that text lands inside a
/// length-prefixed frame: the engine reads it as a frame header, sees an absurd length
/// and kills the worker — so the filter is restarted on every packet, filters nothing,
/// and says nothing about why. The worker sends the user's output to stderr instead.
#[tokio::test]
async fn a_filter_that_prints_keeps_working() {
let path = filter_file(
"chatty",
r#"
from firegex.pyfilters import pyfilter, ACCEPT, REJECT
from firegex.pyfilters.models import RawPacket
@pyfilter
def noisy(packet: RawPacket):
print("saw", len(packet.data), "bytes")
return REJECT if b"deny" in packet.data else ACCEPT
"#,
);
let c = chain_for(&path, 1000);
assert_eq!(feed(&c, &[b"hello"]).await, vec![Verdict::Accept]);
// Still enforcing after having printed, which it would not be if the output had
// gone down the protocol channel and cost the worker its life.
assert_eq!(
feed(&c, &[b"deny me"]).await,
vec![Verdict::Reject(Some("py1/noisy".to_string()))]
);
assert_eq!(feed(&c, &[b"and again"]).await, vec![Verdict::Accept]);
assert_eq!(
feed(&c, &[b"deny me too"]).await,
vec![Verdict::Reject(Some("py1/noisy".to_string()))]
);
}
/// The documented promise: each stream gets its own module globals.
///
/// This is why every frame carries a connection id. Sharing one set of globals across
/// connections would let one client's history decide another client's verdict — a
/// false positive that only shows up under concurrent traffic, and a way to smuggle a
/// pattern past a stateful filter by splitting it over two connections.
#[tokio::test]
async fn each_connection_gets_its_own_globals() {
let path = filter_file(
"percall",
r#"
from firegex.pyfilters import pyfilter, ACCEPT, REJECT
from firegex.pyfilters.models import RawPacket
seen = 0
@pyfilter
def second_chunk_is_refused(packet: RawPacket):
global seen
seen += 1
return REJECT if seen > 1 else ACCEPT
"#,
);
let c = chain_for(&path, 1000);
let first = next_connection_id();
let mut a = ChainSessions::new(first);
assert_eq!(
c.run(Direction::ClientToServer, b"one", &mut a).await,
Verdict::Accept
);
assert_eq!(
c.run(Direction::ClientToServer, b"two", &mut a).await,
Verdict::Reject(Some("py1/second_chunk_is_refused".to_string())),
"the counter did not survive between chunks of one connection"
);
// A different connection starts from zero, whatever the first one did.
let mut b = ChainSessions::new(next_connection_id());
assert_eq!(
c.run(Direction::ClientToServer, b"one", &mut b).await,
Verdict::Accept,
"one connection's globals leaked into another's"
);
// Closing releases them, so a long-lived service does not accumulate one set of
// globals per connection it has ever carried.
c.connection_closed(first);
let mut again = ChainSessions::new(first);
assert_eq!(
c.run(Direction::ClientToServer, b"one", &mut again).await,
Verdict::Accept,
"the closed connection's globals were still there"
);
}
/// Python and regex rules coexist: the compiled ones run inline, the worker does not.
#[tokio::test]
async fn python_and_regex_rules_run_together() {
let path = filter_file(
"coexist",
r#"
from firegex.pyfilters import pyfilter, ACCEPT, REJECT
from firegex.pyfilters.models import RawPacket
@pyfilter
def block_py(packet: RawPacket):
return REJECT if b"by-python" in packet.data else ACCEPT
"#,
);
let json = format!(
r#"[{{"kind":"regex","id":"re1","pattern":"by-regex"}},
{{"kind":"python","id":"py1","code_path":"{path}","timeout_ms":1000,
"command":["python3","{}"]}}]"#,
worker()
);
let c = FilterChain::new(parse_ruleset(&json).unwrap(), Duration::from_secs(5));
assert_eq!(feed(&c, &[b"harmless"]).await, vec![Verdict::Accept]);
assert_eq!(
feed(&c, &[b"blocked by-regex here"]).await,
vec![Verdict::Reject(Some("re1".to_string()))]
);
assert_eq!(
feed(&c, &[b"blocked by-python here"]).await,
vec![Verdict::Reject(Some("py1/block_py".to_string()))]
);
}
/// A filter reads the addresses, and cannot read anything below the application layer.
///
/// This is the boundary the whole model rests on: metadata in, payload out. It also has
/// to be *real* metadata — the proxy terminated the connection, so the headers on the
/// wire are the engine's own, and an earlier version handed the filter a literal
/// `FAKE:IP:TCP:HEADERS:` prefix instead. A filter written against that did one thing
/// here and a different thing on NFQUEUE.
#[tokio::test]
async fn a_filter_is_told_who_is_talking_to_whom() {
let path = filter_file(
"endpoints",
r#"
from firegex.pyfilters import pyfilter, ACCEPT, REJECT
from firegex.pyfilters.models import RawPacket
@pyfilter
def only_from_that_client(packet: RawPacket):
if packet.client_ip != "10.0.0.9" or packet.client_port != 51000:
return REJECT
if packet.server_ip != "10.0.0.1" or packet.server_port != 8080:
return REJECT
# There is nothing below the application layer to read, and asking for it fails
# rather than answering with something invented.
if hasattr(packet, "raw_packet"):
return REJECT
return ACCEPT
"#,
);
let c = chain_for(&path, 2000);
let meta = ConnectionMeta {
client: "10.0.0.9:51000".parse().unwrap(),
server: "10.0.0.1:8080".parse().unwrap(),
l4: L4::Tcp,
};
assert_eq!(
feed_from(&c, &[b"hello"], Some(meta)).await,
vec![Verdict::Accept],
"the filter did not see the addresses it was told about"
);
}
/// The protocol a filter file speaks is read off the file, not passed to it.
///
/// Nothing in the ruleset says `http`, and the worker is started with the path alone.
/// The file is an HTTP filter because it asks for an `HttpRequest`, and a `RawPacket`
/// filter sits beside it in the same file because `http` provides both.
#[tokio::test]
async fn the_protocol_is_read_off_the_code() {
let path = filter_file(
"detected",
r#"
from firegex.pyfilters import pyfilter, ACCEPT, REJECT
from firegex.pyfilters.models import HttpRequest, RawPacket
@pyfilter
def refuse_traversal(request: HttpRequest):
return REJECT if "../" in (request.url or "") else ACCEPT
@pyfilter
def look_at_the_bytes(packet: RawPacket):
return ACCEPT
"#,
);
let c = chain_for(&path, 2000);
assert_eq!(
feed(&c, &[b"GET /shop HTTP/1.1\r\nHost: x\r\n\r\n"]).await,
vec![Verdict::Accept]
);
assert_eq!(
feed(&c, &[b"GET /../../etc/passwd HTTP/1.1\r\nHost: x\r\n\r\n"]).await,
vec![Verdict::Reject(Some("py1/refuse_traversal".to_string()))]
);
}
/// A file holds several functions, and each one can be switched off on its own.
///
/// The code is untouched: only the list of names the library is given changes, and that
/// list is what decides whether a function is ever called. Deleting the code to stop
/// consulting a filter, and pasting it back to resume, is what this replaces.
#[tokio::test]
async fn one_function_of_a_file_can_be_switched_off() {
let path = filter_file(
"selectable",
r#"
from firegex.pyfilters import pyfilter, ACCEPT, REJECT
from firegex.pyfilters.models import RawPacket
@pyfilter
def refuse_a(packet: RawPacket):
return REJECT if b"AAA" in packet.data else ACCEPT
@pyfilter
def refuse_b(packet: RawPacket):
return REJECT if b"BBB" in packet.data else ACCEPT
"#,
);
// Nothing said: every function the file defines runs.
let all = chain_with(&path, None);
assert_eq!(
feed(&all, &[b"carrying AAA"]).await,
vec![Verdict::Reject(Some("py1/refuse_a".to_string()))],
"a block has to name the function that made it, not just the file"
);
assert_eq!(
feed(&all, &[b"carrying BBB"]).await,
vec![Verdict::Reject(Some("py1/refuse_b".to_string()))]
);
// One of the two: the other is still in the file, and no longer consulted.
let only_a = chain_with(&path, Some(&["refuse_a"]));
assert_eq!(
feed(&only_a, &[b"carrying AAA"]).await,
vec![Verdict::Reject(Some("py1/refuse_a".to_string()))]
);
assert_eq!(
feed(&only_a, &[b"carrying BBB"]).await,
vec![Verdict::Accept],
"a switched-off function must not decide anything"
);
// An empty list is not the same as saying nothing: the operator switched every one
// of them off, and the file inspects nothing until one is switched back on.
let none: [&str; 0] = [];
let off = chain_with(&path, Some(&none));
assert_eq!(feed(&off, &[b"carrying AAA"]).await, vec![Verdict::Accept]);
assert_eq!(feed(&off, &[b"carrying BBB"]).await, vec![Verdict::Accept]);
}
/// An edit elsewhere in the chain must not restart a Python filter that did not change.
///
/// Every edit sends the whole ruleset, and a new worker meets every open connection
/// halfway: its module globals gone, its HTTP parser starting on the second half of the
/// request in flight. The same filter, described the same way, is carried over with the
/// state it keeps for each connection; new code is a new worker, as it has to be.
#[tokio::test]
async fn an_unchanged_worker_is_carried_into_the_next_ruleset_with_its_state() {
const REMEMBERS: &str = r#"from firegex.pyfilters import pyfilter, ACCEPT, REJECT
from firegex.pyfilters.models import RawPacket
# Module level, so it is per connection: the library gives each stream its own globals.
said_first = False
@pyfilter
def second_after_first(packet: RawPacket):
global said_first
if b"FIRST" in packet.data:
said_first = True
return REJECT if said_first and b"SECOND" in packet.data else ACCEPT
"#;
let path = filter_file("carried", REMEMBERS);
let json = format!(
r#"[{{"kind":"python","id":"py1","code_path":"{path}","timeout_ms":2000,
"command":["python3","{}"]}}]"#,
worker()
);
let deadline = Duration::from_secs(10);
let connection = next_connection_id();
let before = FilterChain::new(parse_ruleset(&json).unwrap(), deadline);
let mut sessions = ChainSessions::new(connection);
assert_eq!(
before.run(Direction::ClientToServer, b"FIRST", &mut sessions).await,
Verdict::Accept
);
// The same ruleset again, as an edit to some other filter would send it.
let after = FilterChain::new(
parse_ruleset_reusing(&json, &before.filters()).unwrap(),
deadline,
);
assert!(
std::sync::Arc::ptr_eq(&before.filters()[0], &after.filters()[0]),
"an unchanged Python filter was rebuilt"
);
let mut sessions = ChainSessions::new(connection);
assert_eq!(
after.run(Direction::ClientToServer, b"SECOND", &mut sessions).await,
Verdict::Reject(Some("py1/second_after_first".to_string())),
"the connection's state did not survive an edit that did not touch this filter"
);
// New code is a new worker, and starts every connection afresh.
filter_file("carried", &format!("{REMEMBERS}\n# edited\n"));
let edited = FilterChain::new(
parse_ruleset_reusing(&json, &after.filters()).unwrap(),
deadline,
);
assert!(
!std::sync::Arc::ptr_eq(&after.filters()[0], &edited.filters()[0]),
"edited code was not given a worker of its own"
);
let mut sessions = ChainSessions::new(connection);
assert_eq!(
edited.run(Direction::ClientToServer, b"SECOND", &mut sessions).await,
Verdict::Accept
);
}

402
backend/proxysrc/tests/quic.rs Normal file
View File

@ -0,0 +1,402 @@
//! QUIC terminated by the engine, and the three things that has to get right.
//!
//! A client and a service, both speaking real QUIC, either side of the relay. What is
//! being proved is not that bytes arrive — it is that a rule can see them at all, which
//! on this transport is only true because the connection is terminated: on the wire the
//! stream boundaries and the frames are encrypted along with the payload.
//!
//! The protocol negotiated here is deliberately **not** `h3`. These are the tests for a
//! QUIC service that speaks something of its own, where a stream is a stream of bytes and
//! nothing reads them but the chain; HTTP/3 is carried by a module that understands it,
//! and is tested in `h3.rs`.
use std::net::SocketAddr;
use std::sync::Arc;
use std::time::Duration;
use fgex_proxy::filter::{ChainHandle, FilterChain};
use fgex_proxy::proxy::ProxyStats;
use fgex_proxy::proxy::Onward;
use fgex_proxy::quic::{QuicConfig, QuicRelay, QuicSetup};
use fgex_proxy::spec::parse_filters;
use fgex_proxy::tls;
use quinn::crypto::rustls::{QuicClientConfig, QuicServerConfig};
/// Generated per run: no key material belongs in the repository.
fn self_signed() -> (String, String) {
let cert =
rcgen::generate_simple_self_signed(vec!["localhost".to_string(), "127.0.0.1".to_string()])
.unwrap();
(cert.cert.pem(), cert.key_pair.serialize_pem())
}
fn alpn(list: &[&str]) -> Vec<Vec<u8>> {
list.iter().map(|p| p.as_bytes().to_vec()).collect()
}
/// A service that speaks QUIC and echoes every stream, standing in for the real one.
async fn spawn_quic_echo(cert: &str, key: &str, speaks: &[&str]) -> SocketAddr {
let mut config = (*tls::server_config(cert, key).unwrap()).clone();
config.alpn_protocols = alpn(speaks);
let crypto = QuicServerConfig::try_from(config).unwrap();
let endpoint = quinn::Endpoint::server(
quinn::ServerConfig::with_crypto(Arc::new(crypto)),
"127.0.0.1:0".parse().unwrap(),
)
.unwrap();
let addr = endpoint.local_addr().unwrap();
tokio::spawn(async move {
while let Some(incoming) = endpoint.accept().await {
tokio::spawn(async move {
let Ok(connection) = incoming.await else {
return;
};
// Datagrams, echoed the same way the streams are: their own task, so a
// service that is answering one is still accepting the other.
let datagrams = connection.clone();
tokio::spawn(async move {
while let Ok(datagram) = datagrams.read_datagram().await {
if datagrams.send_datagram(datagram).is_err() {
return;
}
}
});
while let Ok((mut send, mut recv)) = connection.accept_bi().await {
tokio::spawn(async move {
let mut buf = vec![0u8; 16 * 1024];
while let Ok(Some(n)) = recv.read(&mut buf).await {
if send.write_all(&buf[..n]).await.is_err() {
return;
}
}
let _ = send.finish();
});
}
});
}
});
addr
}
/// The relay, in front of that service.
async fn spawn_relay(upstream: SocketAddr, filters: &str) -> SocketAddr {
let (cert, key) = self_signed();
let chain = ChainHandle::new(FilterChain::new(
parse_filters(filters).unwrap(),
Duration::from_millis(500),
));
let cfg = QuicConfig {
setup: Arc::new(QuicSetup::build(&cert, &key).unwrap()),
// No mark and no impersonation: `SO_MARK` and `IP_TRANSPARENT` both need
// CAP_NET_ADMIN, which a test run has no business requiring. What they change is
// which address the service sees, and nothing here is asking about that.
self_mark: None,
spoof_source: false,
max_connections: 0,
over_limit_forwards: false,
first_byte_timeout: None,
connect_timeout: Duration::from_secs(5),
upstream: Onward::Same,
// Nothing to write a reconstruction to, which is the ordinary case on any host
// without the capture interface and the case every test here runs in.
capture: None,
};
let relay = QuicRelay::bind(
"127.0.0.1:0".parse().unwrap(),
upstream,
chain,
cfg,
Arc::new(ProxyStats::default()),
)
.unwrap();
let addr = relay.local_addr().unwrap();
tokio::spawn(Arc::new(relay).serve());
addr
}
/// One message, and every message underneath it.
///
/// quinn nests the reason: a read failing because the connection went away says only
/// "connection lost", and *why* it was lost — the application close, with the reason
/// firegex put in it — is one `source()` further down. A test that looked at the top
/// line alone could not tell a block from a crash.
fn because(error: &dyn std::error::Error) -> String {
let mut parts = vec![error.to_string()];
let mut current = error.source();
while let Some(inner) = current {
parts.push(inner.to_string());
current = inner.source();
}
parts.join(": ")
}
struct Exchange {
/// What the client ended up speaking — which is what the *service* chose.
protocol: Option<String>,
echoed: Result<Vec<u8>, String>,
}
/// One client, one stream, one answer.
async fn roundtrip(addr: SocketAddr, payload: &[u8], speaks: &[&str]) -> Exchange {
let mut config = tls::quic_client_config().unwrap();
config.alpn_protocols = alpn(speaks);
let mut endpoint = quinn::Endpoint::client("127.0.0.1:0".parse().unwrap()).unwrap();
endpoint.set_default_client_config(quinn::ClientConfig::new(Arc::new(
QuicClientConfig::try_from(config).unwrap(),
)));
let connecting = endpoint.connect(addr, "localhost").unwrap();
let connection = match tokio::time::timeout(Duration::from_secs(5), connecting).await {
Ok(Ok(connection)) => connection,
Ok(Err(e)) => {
return Exchange {
protocol: None,
echoed: Err(e.to_string()),
}
}
Err(_) => {
return Exchange {
protocol: None,
echoed: Err("the handshake timed out".to_string()),
}
}
};
let protocol = connection
.handshake_data()
.and_then(|d| d.downcast::<quinn::crypto::rustls::HandshakeData>().ok())
.and_then(|d| d.protocol)
.map(|p| String::from_utf8_lossy(&p).into_owned());
let echoed = async {
let (mut send, mut recv) = connection.open_bi().await.map_err(|e| because(&e))?;
send.write_all(payload).await.map_err(|e| because(&e))?;
send.finish().map_err(|e| because(&e))?;
recv.read_to_end(64 * 1024).await.map_err(|e| because(&e))
}
.await;
Exchange { protocol, echoed }
}
/// The whole point: a stream goes through, and it went through something that decrypted
/// it on the way.
///
/// It also exercises the Retry this relay answers a new client with — every first
/// connection from an address arrives unvalidated, so a malformed token would show up
/// here as a handshake that never completes rather than as a subtle weakness.
#[tokio::test]
async fn carries_a_stream_through() {
let (cert, key) = self_signed();
let upstream = spawn_quic_echo(&cert, &key, &["fgex-raw"]).await;
let addr = spawn_relay(upstream, "").await;
let exchange = roundtrip(addr, b"hello over quic", &["fgex-raw"]).await;
assert_eq!(exchange.echoed.unwrap(), b"hello over quic");
}
/// The reason terminating it is worth the trouble: a rule matches bytes that on the wire
/// were inside an encrypted packet, with the frame headers encrypted too.
#[tokio::test]
async fn rules_see_the_plaintext_inside_the_streams() {
let (cert, key) = self_signed();
let upstream = spawn_quic_echo(&cert, &key, &["fgex-raw"]).await;
let addr = spawn_relay(upstream, "block:FLAG{").await;
let harmless = roundtrip(addr, b"harmless request", &["fgex-raw"]).await;
assert_eq!(harmless.echoed.unwrap(), b"harmless request");
let blocked = roundtrip(addr, b"give me FLAG{secret}", &["fgex-raw"]).await;
let error = blocked
.echoed
.expect_err("a rule let the payload through, or the connection was left open");
// Closed on purpose, with a reason the client can read — not a stream reset and not
// a silence to time out on.
assert!(
error.contains("blocked by firegex"),
"the client was not told why: {error}"
);
}
/// A refusal ends the connection, not just the stream that carried it.
///
/// The alternative — resetting one stream — would leave the client free to ask again on
/// the next one, which is not what a block means anywhere else in firegex.
#[tokio::test]
async fn a_refusal_takes_the_whole_connection() {
let (cert, key) = self_signed();
let upstream = spawn_quic_echo(&cert, &key, &["fgex-raw"]).await;
let addr = spawn_relay(upstream, "block:FLAG{").await;
let mut config = tls::quic_client_config().unwrap();
config.alpn_protocols = alpn(&["fgex-raw"]);
let mut endpoint = quinn::Endpoint::client("127.0.0.1:0".parse().unwrap()).unwrap();
endpoint.set_default_client_config(quinn::ClientConfig::new(Arc::new(
QuicClientConfig::try_from(config).unwrap(),
)));
let connection = endpoint.connect(addr, "localhost").unwrap().await.unwrap();
// One stream says something forbidden...
let (mut send, mut recv) = connection.open_bi().await.unwrap();
send.write_all(b"here is a FLAG{x}").await.unwrap();
let _ = send.finish();
let _ = recv.read_to_end(4096).await;
// ...and a second one, on the same connection, has nowhere to go.
let opened = tokio::time::timeout(Duration::from_secs(2), connection.open_bi()).await;
assert!(
matches!(opened, Ok(Err(_))),
"the connection outlived the block"
);
}
/// The service picks from what the client offered, and the client is told that and
/// nothing else — the TLS path's mirroring, on QUIC.
///
/// The relay used to offer the service a list of its own, `h3` unless the operator named
/// others, because the client's hello was taken to be unreadable. It is read off the
/// Initial instead (`quic_hello`), so a service speaking something nobody configured is
/// carried as it is.
#[tokio::test]
async fn the_service_chooses_from_what_the_client_offered() {
let (cert, key) = self_signed();
let upstream = spawn_quic_echo(&cert, &key, &["fgex-test"]).await;
let addr = spawn_relay(upstream, "").await;
let exchange = roundtrip(addr, b"ping", &["h3", "fgex-test"]).await;
assert_eq!(exchange.protocol.as_deref(), Some("fgex-test"));
assert_eq!(exchange.echoed.unwrap(), b"ping");
}
/// A hello too big for one packet — which post-quantum key shares make ordinary — is read
/// across the packets it came in.
#[tokio::test]
async fn a_hello_spanning_several_packets_is_read_whole() {
let (cert, key) = self_signed();
let upstream = spawn_quic_echo(&cert, &key, &["fgex-last"]).await;
let addr = spawn_relay(upstream, "").await;
let filler: Vec<String> = (0..40).map(|i| format!("fgex-unspoken-{i:02}-{}", "x".repeat(40))).collect();
let mut offers: Vec<&str> = filler.iter().map(String::as_str).collect();
offers.push("fgex-last");
let exchange = roundtrip(addr, b"ping", &offers).await;
assert_eq!(exchange.protocol.as_deref(), Some("fgex-last"));
assert_eq!(exchange.echoed.unwrap(), b"ping");
}
/// A client that speaks nothing the service does is refused by the service's handshake,
/// the way it would be with no firegex in the way.
#[tokio::test]
async fn a_client_that_speaks_something_else_is_refused() {
let (cert, key) = self_signed();
let upstream = spawn_quic_echo(&cert, &key, &["fgex-test"]).await;
let addr = spawn_relay(upstream, "").await;
let exchange = roundtrip(addr, b"ping", &["h3"]).await;
assert!(
exchange.echoed.is_err(),
"a client was told a protocol the service never agreed to"
);
}
/// One client connection, for the tests that need more of it than `roundtrip` gives.
async fn connect(addr: SocketAddr, speaks: &[&str]) -> quinn::Connection {
let mut config = tls::quic_client_config().unwrap();
config.alpn_protocols = alpn(speaks);
let mut endpoint = quinn::Endpoint::client("127.0.0.1:0".parse().unwrap()).unwrap();
endpoint.set_default_client_config(quinn::ClientConfig::new(Arc::new(
QuicClientConfig::try_from(config).unwrap(),
)));
endpoint.connect(addr, "localhost").unwrap().await.unwrap()
}
/// A datagram goes through, and comes back.
#[tokio::test]
async fn datagrams_are_carried_both_ways() {
let (cert, key) = self_signed();
let upstream = spawn_quic_echo(&cert, &key, &["fgex-raw"]).await;
let addr = spawn_relay(upstream, "").await;
let connection = connect(addr, &["fgex-raw"]).await;
assert!(
connection.max_datagram_size().is_some(),
"the extension was not advertised to a client that could have used it"
);
connection
.send_datagram(bytes::Bytes::from_static(b"ping"))
.unwrap();
let back = tokio::time::timeout(Duration::from_secs(5), connection.read_datagram())
.await
.expect("the datagram never came back")
.unwrap();
assert_eq!(&back[..], b"ping");
}
/// A rule refusing a datagram drops that datagram, and nothing else.
///
/// The opposite of what a refusal does to a stream, on purpose: there is no conversation
/// to end, so ending the connection would take a hundred other streams with it over one
/// message that was already complete when it was judged.
#[tokio::test]
async fn a_refused_datagram_is_dropped_and_the_connection_lives() {
let (cert, key) = self_signed();
let upstream = spawn_quic_echo(&cert, &key, &["fgex-raw"]).await;
let addr = spawn_relay(upstream, "block:forbidden").await;
let connection = connect(addr, &["fgex-raw"]).await;
connection
.send_datagram(bytes::Bytes::from_static(b"forbidden"))
.unwrap();
// Nothing comes back, because nothing was forwarded.
assert!(
tokio::time::timeout(Duration::from_millis(500), connection.read_datagram())
.await
.is_err(),
"a refused datagram reached the service anyway"
);
// And the connection is still there to be used.
connection
.send_datagram(bytes::Bytes::from_static(b"allowed"))
.unwrap();
let back = tokio::time::timeout(Duration::from_secs(5), connection.read_datagram())
.await
.expect("the connection was closed by a datagram that should only have been dropped")
.unwrap();
assert_eq!(&back[..], b"allowed");
}
/// The datagrams of one connection are one flow, and a filter keeps its state across
/// them.
///
/// The same answer the plain UDP relay gives — its sessions live on the flow, not on the
/// datagram — and the reason is that the alternative is a way through: a pattern split
/// over two datagrams would otherwise be two halves that match nothing, sent a
/// millisecond apart. What a datagram does *not* get is the stream models built on top of
/// a session, which is `L4::QuicDatagram`'s job and not this one's.
#[tokio::test]
async fn a_datagram_flow_keeps_its_filter_state() {
let (cert, key) = self_signed();
let upstream = spawn_quic_echo(&cert, &key, &["fgex-raw"]).await;
let addr = spawn_relay(upstream, "block:secret").await;
let connection = connect(addr, &["fgex-raw"]).await;
connection
.send_datagram(bytes::Bytes::from_static(b"sec"))
.unwrap();
let back = tokio::time::timeout(Duration::from_secs(5), connection.read_datagram())
.await
.expect("half a needle is not a needle, and this one was blocked")
.unwrap();
assert_eq!(&back[..], b"sec");
// The half that completes it is refused, though it arrived in a datagram of its own.
connection
.send_datagram(bytes::Bytes::from_static(b"ret"))
.unwrap();
assert!(
tokio::time::timeout(Duration::from_millis(500), connection.read_datagram())
.await
.is_err(),
"a needle split across two datagrams went through"
);
}

253
backend/proxysrc/tests/rules.rs Normal file
View File

@ -0,0 +1,253 @@
//! The filters the backend actually configures, and the boundary case that decides
//! whether they are worth anything.
use std::sync::Arc;
use std::time::Duration;
use fgex_proxy::filter::{
next_connection_id, ChainHandle, ChainSessions, Direction, FilterChain, Verdict,
};
use fgex_proxy::rules::parse_ruleset;
fn chain(json: &str) -> FilterChain {
FilterChain::new(parse_ruleset(json).unwrap(), Duration::from_millis(500))
}
async fn feed(c: &FilterChain, dir: Direction, chunks: &[&[u8]]) -> Vec<Verdict> {
let mut sessions = ChainSessions::new(next_connection_id());
let mut out = Vec::new();
for chunk in chunks {
out.push(c.run(dir, chunk, &mut sessions).await);
}
out
}
/// A block carries the id of the pattern that fired, so the tests can name it.
fn reject(id: &str) -> Verdict {
Verdict::Reject(Some(id.to_string()))
}
#[tokio::test]
async fn regex_rule_blocks_on_match() {
let c = chain(r#"[{"kind":"regex","id":"r1","pattern":"FLAG\\{[a-z]+\\}"}]"#);
let v = feed(&c, Direction::ClientToServer, &[b"nothing here"]).await;
assert_eq!(v, vec![Verdict::Accept]);
let v = feed(&c, Direction::ClientToServer, &[b"give me FLAG{abc} now"]).await;
assert_eq!(v, vec![reject("r1")]);
}
/// The reason the chain keeps a per-direction sessions at all. Split the pattern
/// across two reads and a chunk-only matcher goes blind — which is exactly the
/// evasion nfregex has to reassemble the stream to prevent.
#[tokio::test]
async fn regex_rule_matches_across_a_segment_boundary() {
let c = chain(r#"[{"kind":"regex","id":"r1","pattern":"/etc/passwd"}]"#);
let v = feed(
&c,
Direction::ClientToServer,
&[b"GET /etc/pa", b"sswd HTTP/1.1"],
)
.await;
assert_eq!(
v,
vec![Verdict::Accept, reject("r1")],
"a pattern split over two reads was missed"
);
}
/// The property an overlap window could not have at any size.
///
/// The previous matcher kept the last 4096 bytes and rescanned them with each chunk,
/// so a match spanning more than that was invisible — and raising the number would
/// have cost every byte of every connection. hyperscan carries the automaton state
/// instead: the span is unbounded and costs nothing per chunk.
#[tokio::test]
async fn regex_rule_matches_across_more_than_the_old_window() {
let c = chain(r#"[{"kind":"regex","id":"r1","pattern":"BEGIN.{9000,}END"}]"#);
let filler = vec![b'.'; 3000];
let v = feed(
&c,
Direction::ClientToServer,
&[b"BEGIN", &filler, &filler, &filler, &filler, b"END"],
)
.await;
assert_eq!(
v.last(),
Some(&reject("r1")),
"a match spanning 12000 bytes was missed"
);
}
#[tokio::test]
async fn regex_rule_honours_direction() {
// Only the server's answers are inspected: a request mentioning it is fine.
let c = chain(r#"[{"kind":"regex","id":"r1","pattern":"FLAG","direction":"s2c"}]"#);
let v = feed(&c, Direction::ClientToServer, &[b"where is the FLAG"]).await;
assert_eq!(v, vec![Verdict::Accept]);
let v = feed(&c, Direction::ServerToClient, &[b"here: FLAG{x}"]).await;
assert_eq!(v, vec![reject("r1")]);
}
#[tokio::test]
async fn regex_rule_can_ignore_case() {
let c =
chain(r#"[{"kind":"regex","id":"r1","pattern":"union select","case_sensitive":false}]"#);
let v = feed(&c, Direction::ClientToServer, &[b"1 UNION SELECT 1"]).await;
assert_eq!(v, vec![reject("r1")]);
}
#[tokio::test]
async fn a_bad_rule_rejects_the_whole_ruleset() {
// Half a ruleset is worse than the one already running, so nothing is applied.
let bad =
r#"[{"kind":"regex","id":"ok","pattern":"a"},{"kind":"regex","id":"bad","pattern":"("}]"#;
let err = match parse_ruleset(bad) {
Err(e) => e,
Ok(rules) => panic!(
"a broken pattern was accepted ({} rules built)",
rules.len()
),
};
assert!(err.contains("invalid regex"), "unhelpful error: {err}");
assert!(parse_ruleset("not json").is_err());
assert!(parse_ruleset(r#"[{"kind":"nope","id":"x"}]"#).is_err());
assert!(parse_ruleset("[]").unwrap().is_empty());
}
/// Rules reach connections that are already open, which is what makes editing them
/// during a competition safe.
#[tokio::test]
async fn a_new_ruleset_reaches_an_open_connection() {
let handle = ChainHandle::new(chain("[]"));
let mut sessions = ChainSessions::new(next_connection_id());
let v = handle
.current()
.run(Direction::ClientToServer, b"FLAG{x}", &mut sessions)
.await;
assert_eq!(v, Verdict::Accept);
handle.replace(chain(r#"[{"kind":"regex","id":"r1","pattern":"FLAG"}]"#));
let v = handle
.current()
.run(Direction::ClientToServer, b"FLAG{x}", &mut sessions)
.await;
assert_eq!(v, reject("r1"));
}
/// The sessions is per connection: one client's traffic must never complete another
/// client's pattern.
#[tokio::test]
async fn windows_do_not_leak_between_connections() {
let c = Arc::new(chain(
r#"[{"kind":"regex","id":"r1","pattern":"/etc/passwd"}]"#,
));
let mut first = ChainSessions::new(next_connection_id());
assert_eq!(
c.run(Direction::ClientToServer, b"GET /etc/pa", &mut first)
.await,
Verdict::Accept
);
// A different connection sending the tail must not be blocked by the half a
// previous client left behind.
let mut second = ChainSessions::new(next_connection_id());
assert_eq!(
c.run(Direction::ClientToServer, b"sswd HTTP/1.1", &mut second)
.await,
Verdict::Accept
);
}
/// Blocking rules that share a direction and a case setting are compiled into one
/// automaton, so adding rules stops costing a scan each. Measured against nfregex,
/// the per-rule scan was the difference between 100 MB/s and 2400.
#[tokio::test]
async fn blocking_rules_are_matched_in_one_pass() {
let many: Vec<String> = (0..50)
// Suffixed so no pattern is a prefix of another: the set reports the first
// index that matched, and overlapping patterns would make that ambiguous.
.map(|i| format!(r#"{{"kind":"regex","id":"r{i}","pattern":"needle{i}z"}}"#))
.collect();
let filters = parse_ruleset(&format!("[{}]", many.join(","))).unwrap();
// Two matchers, not fifty: one hyperscan database per direction, each holding
// every pattern that applies that way. A chunk is scanned by exactly one of
// them, so the cost of matching stops depending on how many rules there are —
// the property a per-rule loop gave away and a benchmark caught.
assert_eq!(
filters.len(),
2,
"50 blocking rules should collapse into one matcher per direction, got {}",
filters.len()
);
let c = FilterChain::new(filters, Duration::from_millis(500));
let v = feed(&c, Direction::ClientToServer, &[b"harmless"]).await;
assert_eq!(v, vec![Verdict::Accept]);
// Still attributed to the individual rule that fired.
let v = feed(
&c,
Direction::ClientToServer,
&[b"here is needle37z for you"],
)
.await;
assert_eq!(v, vec![reject("r37")]);
}
/// Rules that cannot share an automaton stay apart, and still work.
#[tokio::test]
async fn only_direction_splits_the_matchers() {
let filters = parse_ruleset(
r#"[{"kind":"regex","id":"a","pattern":"aaa","direction":"c2s"},
{"kind":"regex","id":"b","pattern":"bbb","direction":"s2c"},
{"kind":"regex","id":"c","pattern":"ccc","case_sensitive":false}]"#,
)
.unwrap();
// Two: one blocking matcher per direction.
// Case-insensitivity does not split anything — hyperscan takes it as a per-pattern flag.
assert_eq!(
filters.len(),
2,
"one blocking matcher per direction"
);
let c = FilterChain::new(filters, Duration::from_millis(500));
assert_eq!(
feed(&c, Direction::ClientToServer, &[b"aaa"]).await,
vec![reject("a")]
);
assert_eq!(
feed(&c, Direction::ClientToServer, &[b"bbb"]).await,
vec![Verdict::Accept],
"an s2c rule fired on client traffic"
);
assert_eq!(
feed(&c, Direction::ServerToClient, &[b"CCC"]).await,
vec![reject("c")]
);
}
/// Does the scan actually traverse the whole buffer? Plant the needle at the very
/// last byte of a 64 KiB chunk: a matcher that short-circuits would miss it.
#[tokio::test]
async fn a_needle_at_the_end_of_a_chunk_is_found() {
let rules: Vec<String> = (0..50)
.map(|i| format!(r#"{{"kind":"regex","id":"r{i}","pattern":"needle{i}z"}}"#))
.collect();
let c = chain(&format!("[{}]", rules.join(",")));
let mut chunk = vec![b'x'; 64 * 1024];
let needle = b"needle37z";
let at = chunk.len() - needle.len();
chunk[at..].copy_from_slice(needle);
let mut s = ChainSessions::new(next_connection_id());
assert_eq!(
c.run(Direction::ClientToServer, &chunk, &mut s).await,
reject("r37")
);
}

63
backend/proxysrc/tests/throughput.rs Normal file
View File

@ -0,0 +1,63 @@
//! What the filter layer costs, in bytes per second.
//!
//! Ignored by default: it measures rather than asserts, and a number that depends on
//! the machine has no business failing a test run. Run it deliberately:
//!
//! ```text
//! cargo test --release --test throughput -- --ignored --nocapture
//! ```
//!
//! Read the slope, not the absolute number. The buffer is reused and stays in cache,
//! so the throughput here is an upper bound on the matcher alone — real traffic is
//! bounded by sockets long before it is bounded by this. What the benchmark is for is
//! the shape of the curve: matching should cost about the same at two hundred rules
//! as at one, because every pattern for a direction lives in one hyperscan database.
//! A per-rule loop made that curve steep enough to be the reason this file exists.
use std::time::{Duration, Instant};
use fgex_proxy::filter::{next_connection_id, ChainSessions, Direction, FilterChain, Verdict};
use fgex_proxy::rules::parse_ruleset;
const CHUNK: usize = 64 * 1024; // what the relay actually reads at a time
const TOTAL: usize = 512 * 1024 * 1024;
fn ruleset(n: usize) -> String {
let rules: Vec<String> = (0..n)
.map(|i| format!(r#"{{"kind":"regex","id":"r{i}","pattern":"needle{i}z"}}"#))
.collect();
format!("[{}]", rules.join(","))
}
async fn measure(rules: usize) -> f64 {
let chain = FilterChain::new(
parse_ruleset(&ruleset(rules)).unwrap(),
Duration::from_millis(500),
);
// Traffic that matches nothing, but is not one repeated byte either: a uniform
// buffer is not what a matcher meets in practice and can flatter it.
let chunk: Vec<u8> = (0..CHUNK).map(|i| b'a' + ((i * 7 + i / 31) % 26) as u8).collect();
let mut sessions = ChainSessions::new(next_connection_id());
let start = Instant::now();
let mut sent = 0usize;
while sent < TOTAL {
assert_eq!(
chain.run(Direction::ClientToServer, &chunk, &mut sessions).await,
Verdict::Accept
);
sent += CHUNK;
}
let secs = start.elapsed().as_secs_f64();
(sent as f64) / secs / (1024.0 * 1024.0)
}
#[tokio::test(flavor = "multi_thread")]
#[ignore = "a measurement, not an assertion"]
async fn filter_throughput_by_rule_count() {
println!("\n rules MB/s");
for n in [0, 1, 10, 50, 200] {
let mbs = measure(n).await;
println!(" {n:>5} {mbs:>9.0}");
}
}

342
backend/proxysrc/tests/tls.rs Normal file
View File

@ -0,0 +1,342 @@
//! TLS terminated by the engine, and the reason that is worth doing.
//!
//! The TLS module gets here with nginx: terminate on one loopback port, forward
//! plaintext to a second, re-encrypt, forward on — and attach a filter engine to the
//! leg in between, which is the one hop where the traffic is in the clear. Doing it
//! in the engine means the plaintext exists only inside this process, and the rules
//! that need to see it are already there.
use std::sync::Arc;
use std::time::Duration;
use fgex_proxy::filter::{ChainHandle, FilterChain};
use fgex_proxy::proxy::{Edge, Onward, Proxy, ProxyConfig, Published, TlsSetup};
use fgex_proxy::spec::parse_filters;
use fgex_proxy::tls;
use tokio::io::{AsyncReadExt, AsyncWriteExt};
use tokio::net::{TcpListener, TcpStream};
/// Generated per run: no key material belongs in the repository.
fn self_signed() -> (String, String) {
let cert =
rcgen::generate_simple_self_signed(vec!["localhost".to_string(), "127.0.0.1".to_string()])
.unwrap();
(cert.cert.pem(), cert.key_pair.serialize_pem())
}
/// Plain echo, standing in for a service that does not speak TLS itself.
async fn spawn_plain_echo() -> std::net::SocketAddr {
let listener = TcpListener::bind("127.0.0.1:0").await.unwrap();
let addr = listener.local_addr().unwrap();
tokio::spawn(async move {
loop {
let Ok((mut sock, _)) = listener.accept().await else {
continue;
};
tokio::spawn(async move {
let mut buf = vec![0u8; 16 * 1024];
loop {
match sock.read(&mut buf).await {
Ok(0) | Err(_) => break,
Ok(n) => {
if sock.write_all(&buf[..n]).await.is_err() {
break;
}
}
}
}
});
}
});
addr
}
/// An echo that speaks TLS, standing in for the CTF service that natively does.
async fn spawn_tls_echo(cert: &str, key: &str) -> std::net::SocketAddr {
let config = tls::server_config(cert, key).unwrap();
let listener = TcpListener::bind("127.0.0.1:0").await.unwrap();
let addr = listener.local_addr().unwrap();
tokio::spawn(async move {
let acceptor = tls::acceptor(config);
loop {
let Ok((sock, _)) = listener.accept().await else {
continue;
};
let acceptor = acceptor.clone();
tokio::spawn(async move {
let Ok(mut stream) = acceptor.accept(sock).await else {
return;
};
let mut buf = vec![0u8; 16 * 1024];
loop {
match stream.read(&mut buf).await {
Ok(0) | Err(_) => break,
Ok(n) => {
if stream.write_all(&buf[..n]).await.is_err() {
break;
}
}
}
}
});
}
});
addr
}
async fn spawn_proxy(
upstream: std::net::SocketAddr,
filters: &str,
tls_setup: TlsSetup,
) -> std::net::SocketAddr {
let chain = ChainHandle::new(FilterChain::new(
parse_filters(filters).unwrap(),
Duration::from_millis(500),
));
let mut cfg = ProxyConfig::fixed("127.0.0.1:0".parse().unwrap(), upstream);
cfg.tls = tls_setup;
let proxy = Proxy::bind(cfg, chain).await.unwrap();
let addr = proxy.local_addr().unwrap();
tokio::spawn(proxy.serve());
addr
}
/// A client that speaks TLS to the proxy, accepting whatever it presents.
async fn tls_roundtrip(addr: std::net::SocketAddr, payload: &[u8]) -> std::io::Result<Vec<u8>> {
let connector = tls::connector(tls::client_config().unwrap());
let sock = TcpStream::connect(addr).await?;
let name = tls::server_name("localhost").unwrap();
let mut stream = connector.connect(name, sock).await?;
stream.write_all(payload).await?;
let mut buf = vec![0u8; 64 * 1024];
let n = tokio::time::timeout(Duration::from_secs(5), stream.read(&mut buf)).await??;
buf.truncate(n);
Ok(buf)
}
#[tokio::test]
async fn terminates_tls_and_relays_to_a_plain_service() {
let (cert, key) = self_signed();
let upstream = spawn_plain_echo().await;
let addr = spawn_proxy(
upstream,
"",
TlsSetup {
server: Some(tls::server_config(&cert, &key).unwrap()),
upstream: None,
optional: false,
},
)
.await;
let got = tls_roundtrip(addr, b"hello over tls").await.unwrap();
assert_eq!(got, b"hello over tls");
}
/// The shape the TLS module is for: the service speaks TLS too, so the engine
/// decrypts, inspects, and re-encrypts. nginx needs two listeners and a loopback hop
/// in the clear to do this.
#[tokio::test]
async fn decrypts_inspects_and_re_encrypts() {
let (cert, key) = self_signed();
let upstream = spawn_tls_echo(&cert, &key).await;
let addr = spawn_proxy(
upstream,
"",
TlsSetup {
server: Some(tls::server_config(&cert, &key).unwrap()),
upstream: Some(tls::client_config().unwrap()),
optional: false,
},
)
.await;
let got = tls_roundtrip(addr, b"end to end").await.unwrap();
assert_eq!(got, b"end to end");
}
/// The point of decrypting at all: a rule that could never have matched the
/// ciphertext blocks the connection.
#[tokio::test]
async fn rules_see_the_plaintext() {
let (cert, key) = self_signed();
let upstream = spawn_tls_echo(&cert, &key).await;
let addr = spawn_proxy(
upstream,
"block:FLAG{",
TlsSetup {
server: Some(tls::server_config(&cert, &key).unwrap()),
upstream: Some(tls::client_config().unwrap()),
optional: false,
},
)
.await;
let got = tls_roundtrip(addr, b"harmless request").await.unwrap();
assert_eq!(got, b"harmless request");
// Blocked, and closed properly: over TLS an abrupt drop reaches the peer as a
// truncation error, which is exactly the "left to guess" this engine avoids.
let got = tls_roundtrip(addr, b"give me FLAG{secret}")
.await
.expect("a blocked TLS connection was torn down instead of closed");
assert!(
got.is_empty(),
"a rule missed the decrypted payload: {got:?}"
);
}
/// A certificate and key that do not go together is a configuration error, and has to
/// be one at startup rather than a listener that refuses every handshake.
#[tokio::test]
async fn a_mismatched_key_is_refused() {
let (cert, _) = self_signed();
let (_, other_key) = self_signed();
let err = tls::server_config(&cert, &other_key).unwrap_err();
assert!(err.contains("do not go together"), "unhelpful error: {err}");
assert!(tls::server_config("", "").is_err());
}
/// TLS is off by default, and off must cost nothing: the plain path is the one the
/// benchmark measured.
#[tokio::test]
async fn tls_off_leaves_the_plain_path_alone() {
let upstream = spawn_plain_echo().await;
let addr = spawn_proxy(upstream, "", TlsSetup::default()).await;
assert!(TlsSetup::default().is_off());
let mut sock = TcpStream::connect(addr).await.unwrap();
sock.write_all(b"plain as ever").await.unwrap();
let mut buf = vec![0u8; 1024];
let n = sock.read(&mut buf).await.unwrap();
assert_eq!(&buf[..n], b"plain as ever");
let _ = Arc::new(());
}
/// A TLS service that answers with the server name its client asked for, and nothing else.
async fn spawn_sni_reporter(cert: &str, key: &str) -> std::net::SocketAddr {
let config = tls::server_config(cert, key).unwrap();
let listener = TcpListener::bind("127.0.0.1:0").await.unwrap();
let addr = listener.local_addr().unwrap();
tokio::spawn(async move {
let acceptor = tls::acceptor(config);
loop {
let Ok((sock, _)) = listener.accept().await else {
continue;
};
let acceptor = acceptor.clone();
tokio::spawn(async move {
let Ok(mut stream) = acceptor.accept(sock).await else {
return;
};
let mut buf = vec![0u8; 1024];
let _ = stream.read(&mut buf).await;
let name = stream.get_ref().1.server_name().unwrap_or("<none>").to_string();
let _ = stream.write_all(name.as_bytes()).await;
let _ = stream.shutdown().await;
});
}
});
addr
}
/// The engine as an `http` service runs it: TLS terminated only for the connections that
/// start a handshake, with the service's address declared as the encrypted one and the
/// service behind it speaking `upstream_speaks`.
async fn spawn_http_edge(
upstream: std::net::SocketAddr,
filters: &str,
cert: &str,
key: &str,
upstream_speaks: Onward,
) -> std::net::SocketAddr {
let chain = ChainHandle::new(FilterChain::new(
parse_filters(filters).unwrap(),
Duration::from_millis(500),
));
let mut cfg = ProxyConfig::fixed("127.0.0.1:0".parse().unwrap(), upstream);
cfg.tls = TlsSetup {
server: Some(tls::server_config(cert, key).unwrap()),
upstream: Some(tls::client_config().unwrap()),
optional: true,
};
cfg.targets.publish(
upstream,
Published { target: None, edge: Edge::Tls, upstream: upstream_speaks },
);
let proxy = Proxy::bind(cfg, chain).await.unwrap();
let addr = proxy.local_addr().unwrap();
tokio::spawn(proxy.serve());
addr
}
/// An HTTPS address answers whether or not a filter is attached yet.
///
/// A chain with nothing to say used to skip the look at what the client opened with, so
/// every connection to an address declared as the encrypted one was refused as "not TLS":
/// a brand new `http` service answered nobody until its first filter, and one whose
/// filters were all switched off — or had all lost their say by panicking — went dark
/// the same way. The opposite of failing open.
#[tokio::test]
async fn a_tls_address_answers_with_no_filter_attached() {
let (cert, key) = self_signed();
let upstream = spawn_plain_echo().await;
for filters in ["", "block:ZZ-NO-SUCH-BYTES-ZZ"] {
let addr = spawn_http_edge(upstream, filters, &cert, &key, Onward::Plain).await;
let got = tls_roundtrip(addr, b"hello").await;
assert_eq!(
got.as_deref().ok(),
Some(&b"hello"[..]),
"the TLS address did not answer with the chain {filters:?}: {got:?}"
);
}
}
/// The same for a service that speaks TLS itself: re-encrypted, chain or no chain.
#[tokio::test]
async fn a_tls_address_re_encrypts_with_no_filter_attached() {
let (cert, key) = self_signed();
let upstream = spawn_tls_echo(&cert, &key).await;
let addr = spawn_http_edge(upstream, "", &cert, &key, Onward::Same).await;
let got = tls_roundtrip(addr, b"end to end").await.unwrap();
assert_eq!(got, b"end to end");
}
/// And the promise the declaration makes still holds without a filter: a client that
/// opens the encrypted address in the clear is refused rather than carried.
#[tokio::test]
async fn a_tls_address_refuses_the_clear_with_no_filter_attached() {
let (cert, key) = self_signed();
let upstream = spawn_plain_echo().await;
let addr = spawn_http_edge(upstream, "", &cert, &key, Onward::Plain).await;
let mut sock = TcpStream::connect(addr).await.unwrap();
sock.write_all(b"GET / HTTP/1.1\r\n\r\n").await.unwrap();
let mut buf = vec![0u8; 1024];
let n = tokio::time::timeout(Duration::from_secs(5), sock.read(&mut buf))
.await
.expect("the refused connection was left hanging")
.unwrap_or(0);
assert_eq!(n, 0, "a cleartext client was carried: {:?}", &buf[..n]);
}
/// The name the client asked for is the name the service is asked for.
///
/// A service choosing its certificate or its virtual host by SNI was handed the bare
/// address instead — which TLS does not even send — and answered as its default host.
#[tokio::test]
async fn the_client_s_server_name_reaches_the_service() {
let (cert, key) = self_signed();
let upstream = spawn_sni_reporter(&cert, &key).await;
let addr = spawn_proxy(
upstream,
"",
TlsSetup {
server: Some(tls::server_config(&cert, &key).unwrap()),
upstream: Some(tls::client_config().unwrap()),
optional: false,
},
)
.await;
let got = tls_roundtrip(addr, b"who am I talking to").await.unwrap();
assert_eq!(String::from_utf8_lossy(&got), "localhost");
}

319
backend/proxysrc/tests/udp.rs Normal file
View File

@ -0,0 +1,319 @@
//! The datagram relay: what the connection limit means for it, and what it counts.
//!
//! Everything here dials the service plainly (`spoof_source` off), because a test has no
//! policy route to bring a transparent reply home. What is being asked is independent of
//! it: whether a flow past the limit hears back, whether flows spend the same budget TCP
//! connections do, and whether the counters the backend divides are in the same unit.
use std::sync::atomic::Ordering;
use std::sync::Arc;
use std::time::Duration;
use fgex_proxy::filter::{ChainHandle, Direction, Filter, FilterChain, FilterCtx, Verdict};
use fgex_proxy::proxy::ProxyStats;
use fgex_proxy::spec::parse_filters;
use fgex_proxy::udp::UdpRelay;
use tokio::net::UdpSocket;
/// A UDP service answering every datagram with the same bytes.
async fn spawn_echo() -> std::net::SocketAddr {
let socket = UdpSocket::bind("127.0.0.1:0").await.unwrap();
let addr = socket.local_addr().unwrap();
tokio::spawn(async move {
let mut buf = vec![0u8; 2048];
loop {
let Ok((n, from)) = socket.recv_from(&mut buf).await else {
continue;
};
let _ = socket.send_to(&buf[..n], from).await;
}
});
addr
}
async fn spawn_relay(
upstream: std::net::SocketAddr,
filters: &str,
limit: usize,
over_limit_forwards: bool,
stats: Arc<ProxyStats>,
) -> std::net::SocketAddr {
let chain = ChainHandle::new(FilterChain::new(
parse_filters(filters).unwrap(),
Duration::from_millis(500),
));
let relay = UdpRelay::bind(
"127.0.0.1:0".parse().unwrap(),
upstream,
chain,
None,
false,
limit,
over_limit_forwards,
stats,
)
.await
.unwrap();
let addr = relay.local_addr().unwrap();
tokio::spawn(relay.serve());
addr
}
/// Send one datagram from a fresh source and wait a moment for an answer.
async fn ask(relay: std::net::SocketAddr, payload: &[u8]) -> Option<Vec<u8>> {
let client = UdpSocket::bind("127.0.0.1:0").await.unwrap();
client.send_to(payload, relay).await.unwrap();
let mut buf = vec![0u8; 2048];
match tokio::time::timeout(Duration::from_millis(700), client.recv_from(&mut buf)).await {
Ok(Ok((n, _))) => Some(buf[..n].to_vec()),
_ => None,
}
}
/// "Forward what does not fit" has to mean the client is answered.
///
/// A flow past the limit used to be one datagram sent from a socket closed straight after,
/// so the request reached the service and its answer went to a port nobody held any more:
/// forwarded, and useless to every request/response protocol UDP carries.
#[tokio::test]
async fn a_flow_past_the_limit_is_answered_when_forwarding() {
let upstream = spawn_echo().await;
let stats = Arc::new(ProxyStats::default());
let relay = spawn_relay(upstream, "", 1, true, Arc::clone(&stats)).await;
// The first source takes the one place and keeps it: flows are kept until idle.
let first = UdpSocket::bind("127.0.0.1:0").await.unwrap();
first.send_to(b"first", relay).await.unwrap();
let mut buf = vec![0u8; 64];
let n = tokio::time::timeout(Duration::from_secs(2), first.recv(&mut buf))
.await
.expect("the first flow was not answered")
.unwrap();
assert_eq!(&buf[..n], b"first");
assert_eq!(ask(relay, b"second").await.as_deref(), Some(&b"second"[..]));
assert_eq!(stats.over_limit.load(Ordering::Relaxed), 1);
}
/// Without forwarding, what does not fit is dropped, and the limit says so.
#[tokio::test]
async fn a_flow_past_the_limit_is_dropped_otherwise() {
let upstream = spawn_echo().await;
let stats = Arc::new(ProxyStats::default());
let relay = spawn_relay(upstream, "", 1, false, Arc::clone(&stats)).await;
let first = UdpSocket::bind("127.0.0.1:0").await.unwrap();
first.send_to(b"first", relay).await.unwrap();
let mut buf = vec![0u8; 64];
tokio::time::timeout(Duration::from_secs(2), first.recv(&mut buf))
.await
.expect("the first flow was not answered")
.unwrap();
assert_eq!(ask(relay, b"second").await, None);
assert_eq!(stats.over_limit.load(Ordering::Relaxed), 1);
}
/// One budget for TCP connections and UDP flows, because they spend the same descriptors.
///
/// Each relay used to count only its own flows, so a service with several UDP addresses
/// could hold several times its limit beside a full complement of TCP connections.
#[tokio::test]
async fn flows_spend_the_same_budget_as_connections() {
let upstream = spawn_echo().await;
let stats = Arc::new(ProxyStats::default());
// As if the TCP side of the service were already carrying one connection.
stats.live.store(1, Ordering::Relaxed);
let relay = spawn_relay(upstream, "", 1, false, Arc::clone(&stats)).await;
assert_eq!(ask(relay, b"no room").await, None, "a flow was admitted past the limit");
assert_eq!(stats.live.load(Ordering::Relaxed), 1, "a refused flow kept its place");
}
/// A service that goes away for a moment and comes back is answering again, for the
/// client that kept talking through it.
///
/// A datagram sent while it was down is answered with ICMP "port unreachable", which the
/// flow's connected socket reports as an error on its next call. That error ended the task
/// carrying the answers back while the flow stayed in the table — and a client that kept
/// sending kept it from ever going idle, so it never heard from the service again.
#[tokio::test]
async fn a_flow_outlives_its_service_restarting() {
let first = UdpSocket::bind("127.0.0.1:0").await.unwrap();
let upstream = first.local_addr().unwrap();
let echo = |socket: UdpSocket| {
tokio::spawn(async move {
let mut buf = vec![0u8; 2048];
loop {
let Ok((n, from)) = socket.recv_from(&mut buf).await else {
continue;
};
let _ = socket.send_to(&buf[..n], from).await;
}
})
};
let service = echo(first);
let relay = spawn_relay(upstream, "", 0, false, Arc::new(ProxyStats::default())).await;
// One source throughout: this is about a flow that already exists.
let client = UdpSocket::bind("127.0.0.1:0").await.unwrap();
assert_eq!(exchange(&client, relay, b"before").await.as_deref(), Some(&b"before"[..]));
service.abort();
let _ = service.await;
assert_eq!(exchange(&client, relay, b"while down").await, None);
echo(UdpSocket::bind(upstream).await.unwrap());
assert_eq!(
exchange(&client, relay, b"after").await.as_deref(),
Some(&b"after"[..]),
"the service came back and the flow that kept talking was not answered"
);
}
/// One datagram from `client`, and the answer if one comes.
async fn exchange(
client: &UdpSocket,
relay: std::net::SocketAddr,
payload: &[u8],
) -> Option<Vec<u8>> {
client.send_to(payload, relay).await.unwrap();
let mut buf = vec![0u8; 2048];
match tokio::time::timeout(Duration::from_millis(700), client.recv(&mut buf)).await {
Ok(Ok(n)) => Some(buf[..n].to_vec()),
_ => None,
}
}
/// Flows are counted in the unit the backend divides by: flows seen, flows refused.
///
/// UDP counted neither, so a service refusing datagrams all day reported no connections
/// and a refused share of zero.
#[tokio::test]
async fn flows_and_refusals_are_counted_once_each() {
let upstream = spawn_echo().await;
let stats = Arc::new(ProxyStats::default());
let relay = spawn_relay(upstream, "block:DENYME", 0, false, Arc::clone(&stats)).await;
let client = UdpSocket::bind("127.0.0.1:0").await.unwrap();
let mut buf = vec![0u8; 64];
client.send_to(b"hello", relay).await.unwrap();
let n = tokio::time::timeout(Duration::from_secs(2), client.recv(&mut buf))
.await
.expect("an innocent datagram was not answered")
.unwrap();
assert_eq!(&buf[..n], b"hello");
for _ in 0..3 {
client.send_to(b"DENYME please", relay).await.unwrap();
}
tokio::time::sleep(Duration::from_millis(300)).await;
assert_eq!(stats.accepted.load(Ordering::Relaxed), 1, "one flow, counted once");
assert_eq!(stats.live.load(Ordering::Relaxed), 1, "the flow holds one place");
assert_eq!(
stats.closed_by_filter.load(Ordering::Relaxed),
1,
"three refused datagrams of one flow are one refused flow"
);
}
/// Takes its time over a client's datagram carrying `slow`, and none over anything else —
/// the echo coming back included, or each exchange would pay for it twice.
struct SlowOn;
impl Filter for SlowOn {
fn name(&self) -> &str {
"slow-on"
}
fn inspect(&self, ctx: &FilterCtx<'_>) -> Verdict {
if ctx.direction == Direction::ClientToServer && ctx.chunk.windows(4).any(|w| w == b"slow") {
std::thread::sleep(Duration::from_millis(1500));
}
Verdict::Accept
}
}
/// One client's slow verdict holds up that client, and nobody else.
///
/// The receive loop used to judge every datagram of every client itself, one after the
/// other: a filter taking a second and a half over one client's datagram was a second and
/// a half in which no other client of the address was even read.
#[tokio::test]
async fn a_slow_verdict_for_one_client_does_not_hold_up_another() {
let upstream = spawn_echo().await;
let chain = ChainHandle::new(FilterChain::new(
vec![Arc::new(SlowOn) as Arc<dyn Filter>],
Duration::from_secs(5),
));
let relay = UdpRelay::bind(
"127.0.0.1:0".parse().unwrap(),
upstream,
chain,
None,
false,
0,
false,
Arc::new(ProxyStats::default()),
)
.await
.unwrap();
let relay_addr = relay.local_addr().unwrap();
tokio::spawn(relay.serve());
let slow = UdpSocket::bind("127.0.0.1:0").await.unwrap();
slow.send_to(b"slow please", relay_addr).await.unwrap();
tokio::time::sleep(Duration::from_millis(100)).await;
let started = std::time::Instant::now();
assert_eq!(ask(relay_addr, b"quick").await.as_deref(), Some(&b"quick"[..]),
"the other client was not answered while the first waited for its verdict");
assert!(started.elapsed() < Duration::from_millis(600),
"the other client waited {:?} behind the slow verdict", started.elapsed());
let mut buf = vec![0u8; 64];
let n = tokio::time::timeout(Duration::from_secs(3), slow.recv(&mut buf))
.await
.expect("the slow client was never answered")
.unwrap();
assert_eq!(&buf[..n], b"slow please");
}
/// Within one flow the order is kept: a datagram is not overtaken by the one after it
/// while its verdict is still being decided.
#[tokio::test]
async fn one_flows_datagrams_keep_their_order() {
let upstream = spawn_echo().await;
let chain = ChainHandle::new(FilterChain::new(
vec![Arc::new(SlowOn) as Arc<dyn Filter>],
Duration::from_secs(5),
));
let relay = UdpRelay::bind(
"127.0.0.1:0".parse().unwrap(),
upstream,
chain,
None,
false,
0,
false,
Arc::new(ProxyStats::default()),
)
.await
.unwrap();
let relay_addr = relay.local_addr().unwrap();
tokio::spawn(relay.serve());
let client = UdpSocket::bind("127.0.0.1:0").await.unwrap();
client.send_to(b"slow first", relay_addr).await.unwrap();
client.send_to(b"then second", relay_addr).await.unwrap();
let mut buf = vec![0u8; 64];
let mut got = Vec::new();
for _ in 0..2 {
let n = tokio::time::timeout(Duration::from_secs(3), client.recv(&mut buf))
.await
.expect("an answer never came")
.unwrap();
got.push(buf[..n].to_vec());
}
assert_eq!(got, vec![b"slow first".to_vec(), b"then second".to_vec()]);
}

15
backend/pyproject.toml Normal file
View File

@ -0,0 +1,15 @@
# Not a package: only read for its dependencies (`uv pip install -r pyproject.toml` in the
# Dockerfile). The version firegex reports is in utils/__init__.py.
[project]
name = "firegex-backend"
version = "0.0.0"
dependencies = [
"fastapi[all]",
"httpx",
"uvicorn[standard]",
"psutil",
"python-jose[cryptography]",
"python-socketio",
"brotli",
# nftables comes from the distribution (python3-nftables), not from PyPI.
]

8
backend/requirements.txt
View File

@ -1,8 +0,0 @@
fastapi[all]
httpx
uvicorn[standard]
psutil
python-jose[cryptography]
python-socketio
brotli
#git+https://salsa.debian.org/pkg-netfilter-team/pkg-nftables#egg=nftables&subdirectory=py

138
backend/routers/firewall.py
View File

@ -1,4 +1,6 @@
import re
import sqlite3
import traceback
from fastapi import APIRouter, HTTPException
from utils.sqlite import SQLite
from utils import ip_parse, ip_family, socketio_emit
@ -50,6 +52,7 @@ async def startup():
await firewall.init()
async def shutdown():
firewall.stop_watching()
if not firewall.keep_rules:
await firewall.close()
db.disconnect()
@ -57,8 +60,29 @@ async def shutdown():
async def refresh_frontend(additional:list[str]=[]):
await socketio_emit(["firewall"]+additional)
async def apply_changes():
await firewall.reload()
async def apply_changes(undo=None):
"""Put what the database now says into the kernel, or put the database back.
`undo` restores what was stored before the change. nft refusing the new firewall
leaves the old one in force (`FiregexTables.set` sends it as one batch), so the
database has to follow: kept, the refused configuration would be what the watcher
tries to put back every few seconds and what the next boot comes up with — failing
each time, with no firewall behind it.
"""
try:
await firewall.reload()
except Exception as e:
if undo is None:
raise
undo()
try:
await firewall.reload()
except Exception:
traceback.print_exc()
raise HTTPException(
status_code=400,
detail=f"nftables refused the new firewall, so nothing was changed: {e}",
)
await refresh_frontend()
return {'status': 'ok'}
@ -71,8 +95,13 @@ async def get_settings():
@app.put("/settings", response_model=StatusMessageModel)
async def set_settings(form: FirewallSettings):
"""Set the firewall settings"""
before = firewall.settings
firewall.settings = form
return await apply_changes()
def undo():
firewall.settings = before
return await apply_changes(undo)
@app.get('/rules', response_model=RuleInfo)
async def get_rule_list():
@ -86,8 +115,13 @@ async def get_rule_list():
@app.post('/enable', response_model=StatusMessageModel)
async def enable_firewall():
"""Request enabling the firewall"""
was = firewall.enabled
firewall.enabled = True
return await apply_changes()
def undo():
firewall.enabled = was
return await apply_changes(undo)
@app.post('/disable', response_model=StatusMessageModel)
async def disable_firewall():
@ -95,27 +129,40 @@ async def disable_firewall():
firewall.enabled = False
return await apply_changes()
#: What an address field may hold when it is not an address: an interface name, as the
#: kernel takes one, or one ending in `*` to match every interface starting that way
#: (`br-*`). Fifteen characters at most, which is what `IFNAMSIZ` leaves for the name.
#:
#: Anything else used to be stored on the strength of containing no `/`, and nft then
#: refused the whole firewall over it — a name one character too long, or with a quote in
#: it, was enough to take every rule down, the policy with them.
INTERFACE = re.compile(r"[A-Za-z0-9_.:-]{1,15}|[A-Za-z0-9_.:-]{0,14}\*")
def _address_or_interface(value: str, what: str) -> tuple[str, bool]:
"""`value` normalised, and whether it is an IP address rather than an interface."""
value = (value or "").strip()
if value == "":
return value, False
try:
return ip_parse(value), True
except ValueError:
pass
if not INTERFACE.fullmatch(value):
raise HTTPException(
status_code=400,
detail=f"Invalid {what} address {value!r}: it is neither an IP address nor an "
f"interface name (at most 15 letters, digits and '_.:-', optionally "
f"ending in '*')",
)
return value, False
def parse_and_check_rule(rule:RuleModel):
if rule.table == Table.MANGLE and rule.mode == Mode.FORWARD:
raise HTTPException(status_code=400, detail="Mangle table does not support forward mode")
is_src_ip = is_dst_ip = True
try:
rule.src = ip_parse(rule.src)
except ValueError:
is_src_ip = False
try:
rule.dst = ip_parse(rule.dst)
except ValueError:
is_dst_ip = False
if not is_src_ip and "/" in rule.src: # Slash is not allowed in ip interfaces names
raise HTTPException(status_code=400, detail="Invalid source address")
if not is_dst_ip and "/" in rule.dst:
raise HTTPException(status_code=400, detail="Invalid destination address")
rule.src, is_src_ip = _address_or_interface(rule.src, "source")
rule.dst, is_dst_ip = _address_or_interface(rule.dst, "destination")
if is_src_ip and is_dst_ip and ip_family(rule.dst) != ip_family(rule.src):
raise HTTPException(status_code=400, detail="Destination and source addresses must be of the same family")
@ -125,30 +172,43 @@ def parse_and_check_rule(rule:RuleModel):
return rule
RULE_COLUMNS = ("active", "name", "proto", "src", "dst", "port_src_from", "port_dst_from",
"port_src_to", "port_dst_to", "action", "mode", "`table`")
def _write_rules(rows: list[dict]) -> None:
"""Replace every stored rule with `rows`, in order, in one transaction."""
db.queries(["DELETE FROM rules"] + [
(
f"INSERT INTO rules (rule_id, {', '.join(RULE_COLUMNS)}) "
f"VALUES (?, {', '.join('?' for _ in RULE_COLUMNS)})",
rid, *(row[column.strip('`')] for column in RULE_COLUMNS),
)
for rid, row in enumerate(rows)
])
@app.post('/rules', response_model=StatusMessageModel)
async def add_new_service(form: RuleFormAdd):
"""Edit rule table"""
rules = [parse_and_check_rule(ele) for ele in form.rules]
before = db.query(f"SELECT {', '.join(RULE_COLUMNS)} FROM rules ORDER BY rule_id;")
policy_before = firewall.policy
try:
db.queries(["DELETE FROM rules"]+
[("""
INSERT INTO rules (
rule_id, active, name,
proto,
src, dst,
port_src_from, port_dst_from,
port_src_to, port_dst_to,
action, mode, `table`
) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ? ,?, ?, ?)""",
rid, ele.active, ele.name,
ele.proto,
ele.src, ele.dst,
ele.port_src_from, ele.port_dst_from,
ele.port_src_to, ele.port_dst_to,
ele.action, ele.mode, ele.table
) for rid, ele in enumerate(rules)]
)
_write_rules([
{
"active": ele.active, "name": ele.name, "proto": ele.proto,
"src": ele.src, "dst": ele.dst,
"port_src_from": ele.port_src_from, "port_dst_from": ele.port_dst_from,
"port_src_to": ele.port_src_to, "port_dst_to": ele.port_dst_to,
"action": ele.action, "mode": ele.mode, "table": ele.table,
}
for ele in rules
])
firewall.policy = form.policy.value
except sqlite3.IntegrityError:
raise HTTPException(status_code=400, detail="Error saving the rules: maybe there are duplicated rules")
return await apply_changes()
def undo():
_write_rules(before)
firewall.policy = policy_before
return await apply_changes(undo)

454
backend/routers/nfproxy.py
View File

@ -1,454 +0,0 @@
import secrets
import sqlite3
from fastapi import APIRouter, HTTPException
from pydantic import BaseModel
from modules.nfproxy.nftables import FiregexTables
from modules.nfproxy.firewall import STATUS, FirewallManager, ServiceNotFoundException
from utils.sqlite import SQLite
from utils import ip_parse, refactor_name, socketio_emit, PortType
from modules.tls.service import activate_stream
from utils.models import ResetRequest, StatusMessageModel
import os
from firegex.nfproxy.internals import get_filter_names
from fastapi.responses import PlainTextResponse
from modules.nfproxy.nftables import convert_protocol_to_l4
import asyncio
import traceback
from utils import DEBUG
import utils
class ServiceModel(BaseModel):
service_id: str
status: str
port: PortType|None = None
name: str
proto: str
ip_int: str|None = None
n_filters: int
edited_packets: int
blocked_packets: int
fail_open: bool
target_type: str
tls_stream_id: str|None = None
class RenameForm(BaseModel):
name:str
class SettingsForm(BaseModel):
port: PortType|None = None
proto: str|None = None
ip_int: str|None = None
fail_open: bool|None = None
target_type: str|None = None
tls_stream_id: str|None = None
class PyFilterModel(BaseModel):
name: str
service_id: str
blocked_packets: int
edited_packets: int
active: bool
class ServiceAddForm(BaseModel):
name: str
port: PortType|None = None
proto: str
ip_int: str|None = None
fail_open: bool = True
target_type: str = "flow"
tls_stream_id: str|None = None
class ServiceAddResponse(BaseModel):
status:str
service_id: str|None = None
class SetPyFilterForm(BaseModel):
code: str
app = APIRouter()
db = SQLite('db/nft-pyfilters.db', {
'services': {
'service_id': 'VARCHAR(100) PRIMARY KEY',
'status': 'VARCHAR(100) NOT NULL',
'target_type': 'VARCHAR(10) NOT NULL CHECK(target_type IN ("flow", "tls")) DEFAULT "flow"',
'tls_stream_id': 'VARCHAR(100)',
'port': 'INT CHECK(port > 0 and port < 65536)',
'name': 'VARCHAR(100) NOT NULL UNIQUE',
'proto': 'VARCHAR(3) NOT NULL CHECK (proto IN ("tcp", "http"))',
'l4_proto': 'VARCHAR(3) NOT NULL CHECK (l4_proto IN ("tcp", "udp"))',
'ip_int': 'VARCHAR(100)',
'fail_open': 'BOOLEAN NOT NULL CHECK (fail_open IN (0, 1)) DEFAULT 1',
},
'pyfilter': {
'name': 'VARCHAR(100) NOT NULL',
'service_id': 'VARCHAR(100) NOT NULL',
'blocked_packets': 'INTEGER UNSIGNED NOT NULL DEFAULT 0',
'edited_packets': 'INTEGER UNSIGNED NOT NULL DEFAULT 0',
'active' : 'BOOLEAN NOT NULL CHECK (active IN (0, 1)) DEFAULT 1',
'FOREIGN KEY (service_id)':'REFERENCES services (service_id)',
'PRIMARY KEY': '(name, service_id)'
},
'QUERY':[
"CREATE UNIQUE INDEX IF NOT EXISTS unique_services ON services (port, ip_int, l4_proto);",
"CREATE UNIQUE INDEX IF NOT EXISTS unique_pyfilter_service ON pyfilter (name, service_id);"
]
})
async def refresh_frontend(additional:list[str]=[]):
await socketio_emit(["nfproxy"]+additional)
async def reset(params: ResetRequest):
if not params.delete:
db.backup()
await firewall.close()
FiregexTables().reset()
if params.delete:
db.delete()
db.init()
else:
db.restore()
try:
await firewall.init()
except Exception as e:
raise HTTPException(status_code=500, detail=str(e))
async def startup():
db.init()
try:
await firewall.init()
except Exception as e:
print("WARNING cannot start firewall:", e)
utils.socketio.on("nfproxy-outstream-join", join_outstream)
utils.socketio.on("nfproxy-outstream-leave", leave_outstream)
utils.socketio.on("nfproxy-exception-join", join_exception)
utils.socketio.on("nfproxy-exception-leave", leave_exception)
async def shutdown():
await firewall.close()
db.disconnect()
def gen_service_id():
while True:
res = secrets.token_hex(8)
if len(db.query('SELECT 1 FROM services WHERE service_id = ?;', res)) == 0:
break
return res
async def outstream_func(service_id, data):
await utils.socketio.emit(f"nfproxy-outstream-{service_id}", data, room=f"nfproxy-outstream-{service_id}")
async def exception_func(service_id, timestamp):
await utils.socketio.emit(f"nfproxy-exception-{service_id}", timestamp, room=f"nfproxy-exception-{service_id}")
firewall = FirewallManager(db, outstream_func=outstream_func, exception_func=exception_func)
@app.get('/services', response_model=list[ServiceModel])
async def get_service_list():
"""Get the list of existent firegex services"""
res = db.query("""
SELECT
s.service_id service_id,
s.status status,
s.port port,
s.name name,
s.proto proto,
s.ip_int ip_int,
s.fail_open fail_open,
s.target_type target_type,
s.tls_stream_id tls_stream_id,
COUNT(f.name) n_filters,
COALESCE(SUM(f.blocked_packets),0) blocked_packets,
COALESCE(SUM(f.edited_packets),0) edited_packets
FROM services s LEFT JOIN pyfilter f ON s.service_id = f.service_id
GROUP BY s.service_id;
""")
return res
@app.get('/services/{service_id}', response_model=ServiceModel)
async def get_service_by_id(service_id: str):
"""Get info about a specific service using his id"""
res = db.query("""
SELECT
s.service_id service_id,
s.status status,
s.port port,
s.name name,
s.proto proto,
s.ip_int ip_int,
s.fail_open fail_open,
s.target_type target_type,
s.tls_stream_id tls_stream_id,
COUNT(f.name) n_filters,
COALESCE(SUM(f.blocked_packets),0) blocked_packets,
COALESCE(SUM(f.edited_packets),0) edited_packets
FROM services s LEFT JOIN pyfilter f ON s.service_id = f.service_id
WHERE s.service_id = ? GROUP BY s.service_id;
""", service_id)
if len(res) == 0:
raise HTTPException(status_code=400, detail="This service does not exists!")
return res[0]
@app.post('/services/{service_id}/stop', response_model=StatusMessageModel)
async def service_stop(service_id: str):
"""Request the stop of a specific service"""
await firewall.get(service_id).next(STATUS.STOP)
await refresh_frontend()
return {'status': 'ok'}
@app.post('/services/{service_id}/start', response_model=StatusMessageModel)
async def service_start(service_id: str):
"""Request the start of a specific service"""
srv = db.query("SELECT target_type, tls_stream_id FROM services WHERE service_id = ?;", service_id)
if srv and srv[0]["target_type"] == "tls":
if not await activate_stream(srv[0]["tls_stream_id"]):
raise HTTPException(status_code=400, detail="Linked TLS stream not found")
await firewall.get(service_id).next(STATUS.ACTIVE)
await refresh_frontend()
return {'status': 'ok'}
@app.delete('/services/{service_id}', response_model=StatusMessageModel)
async def service_delete(service_id: str):
"""Request the deletion of a specific service"""
db.query('DELETE FROM services WHERE service_id = ?;', service_id)
db.query('DELETE FROM pyfilter WHERE service_id = ?;', service_id)
if os.path.exists(f"db/nfproxy_filters/{service_id}.py"):
os.remove(f"db/nfproxy_filters/{service_id}.py")
await firewall.remove(service_id)
await refresh_frontend()
return {'status': 'ok'}
@app.put('/services/{service_id}/rename', response_model=StatusMessageModel)
async def service_rename(service_id: str, form: RenameForm):
"""Request to change the name of a specific service"""
form.name = refactor_name(form.name)
if not form.name:
raise HTTPException(status_code=400, detail="The name cannot be empty!")
try:
db.query('UPDATE services SET name=? WHERE service_id = ?;', form.name, service_id)
except sqlite3.IntegrityError:
raise HTTPException(status_code=400, detail="This name is already used")
await refresh_frontend()
return {'status': 'ok'}
@app.put('/services/{service_id}/settings', response_model=StatusMessageModel)
async def service_settings(service_id: str, form: SettingsForm):
"""Request to change the settings of a specific service (will cause a restart)"""
srv_check = db.query("SELECT ip_int, port, target_type, tls_stream_id FROM services WHERE service_id = ?;", service_id)
if len(srv_check) == 0:
raise HTTPException(status_code=404, detail="Service not found")
old_srv = srv_check[0]
if form.port is not None and (form.port < 1 or form.port > 65535):
raise HTTPException(status_code=400, detail="Invalid port")
if form.ip_int is not None:
try:
form.ip_int = ip_parse(form.ip_int)
except ValueError:
raise HTTPException(status_code=400, detail="Invalid address")
new_ip = form.ip_int if form.ip_int is not None else old_srv["ip_int"]
new_port = form.port if form.port is not None else old_srv["port"]
new_target_type = form.target_type if form.target_type is not None else old_srv["target_type"]
new_tls_stream_id = form.tls_stream_id if form.tls_stream_id is not None else old_srv["tls_stream_id"]
if new_target_type == "tls" and not new_tls_stream_id:
raise HTTPException(status_code=400, detail="TLS stream ID is required when target type is tls")
if new_target_type == "flow" and (new_ip is None or new_port is None):
raise HTTPException(status_code=400, detail="IP and Port are required when target type is flow")
keys = []
values = []
for key, value in form.model_dump(exclude_none=True).items():
keys.append(key)
values.append(value)
if key == "proto":
keys.append("l4_proto")
values.append(convert_protocol_to_l4(value))
if len(keys) == 0:
raise HTTPException(status_code=400, detail="No settings to change provided")
try:
db.query(f'UPDATE services SET {", ".join([f"{key}=?" for key in keys])} WHERE service_id = ?;', *values, service_id)
except sqlite3.IntegrityError:
raise HTTPException(status_code=400, detail="A service with these settings already exists")
old_status = firewall.get(service_id).status
await firewall.remove(service_id)
if old_status == STATUS.ACTIVE and new_target_type == "tls":
await activate_stream(new_tls_stream_id)
await firewall.reload()
await firewall.get(service_id).next(old_status)
await refresh_frontend()
return {'status': 'ok'}
@app.get('/services/{service_id}/pyfilters', response_model=list[PyFilterModel])
async def get_service_pyfilter_list(service_id: str):
"""Get the list of the pyfilters of a service"""
if not db.query("SELECT 1 FROM services s WHERE s.service_id = ?;", service_id):
raise HTTPException(status_code=400, detail="This service does not exists!")
return db.query("""
SELECT
name, blocked_packets, edited_packets, active, service_id
FROM pyfilter WHERE service_id = ?;
""", service_id)
@app.get('/services/{service_id}/pyfilters/{filter_name}', response_model=PyFilterModel)
async def get_pyfilter_by_id(service_id: str, filter_name: str):
"""Get pyfilter info using his id"""
res = db.query("""
SELECT
name, blocked_packets, edited_packets, active, service_id
FROM pyfilter WHERE name = ? AND service_id = ?;
""", filter_name, service_id)
if len(res) == 0:
raise HTTPException(status_code=400, detail="This filter does not exists!")
return res[0]
@app.post('/services/{service_id}/pyfilters/{filter_name}/enable', response_model=StatusMessageModel)
async def pyfilter_enable(service_id: str, filter_name: str):
"""Request the enabling of a pyfilter"""
res = db.query('SELECT * FROM pyfilter WHERE name = ? AND service_id = ?;', filter_name, service_id)
if len(res) != 0:
db.query('UPDATE pyfilter SET active=1 WHERE name = ? AND service_id = ?;', filter_name, service_id)
await firewall.get(res[0]["service_id"]).update_filters()
await refresh_frontend()
return {'status': 'ok'}
@app.post('/services/{service_id}/pyfilters/{filter_name}/disable', response_model=StatusMessageModel)
async def pyfilter_disable(service_id: str, filter_name: str):
"""Request the deactivation of a pyfilter"""
res = db.query('SELECT * FROM pyfilter WHERE name = ? AND service_id = ?;', filter_name, service_id)
if len(res) != 0:
db.query('UPDATE pyfilter SET active=0 WHERE name = ? AND service_id = ?;', filter_name, service_id)
await firewall.get(res[0]["service_id"]).update_filters()
await refresh_frontend()
return {'status': 'ok'}
@app.post('/services', response_model=ServiceAddResponse)
async def add_new_service(form: ServiceAddForm):
"""Add a new service"""
if form.target_type == "flow":
if form.ip_int is None or form.port is None:
raise HTTPException(status_code=400, detail="IP and Port are required when target type is flow")
try:
form.ip_int = ip_parse(form.ip_int)
except ValueError:
raise HTTPException(status_code=400, detail="Invalid address")
elif form.target_type == "tls":
if not form.tls_stream_id:
raise HTTPException(status_code=400, detail="TLS stream ID is required when target type is tls")
if form.proto not in ["tcp", "http"]:
raise HTTPException(status_code=400, detail="Invalid protocol")
srv_id = None
try:
srv_id = gen_service_id()
db.query("INSERT INTO services (service_id, name, port, status, proto, ip_int, fail_open, l4_proto, target_type, tls_stream_id) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?)",
srv_id, refactor_name(form.name), form.port, STATUS.STOP, form.proto, form.ip_int, form.fail_open, convert_protocol_to_l4(form.proto), form.target_type, form.tls_stream_id)
except sqlite3.IntegrityError:
raise HTTPException(status_code=400, detail="This type of service already exists")
await firewall.reload()
await refresh_frontend()
return {'status': 'ok', 'service_id': srv_id}
@app.put('/services/{service_id}/code', response_model=StatusMessageModel)
async def set_pyfilters_code(service_id: str, form: SetPyFilterForm):
"""Set the python filter for a service"""
service = db.query("SELECT service_id, proto FROM services WHERE service_id = ?;", service_id)
if len(service) == 0:
raise HTTPException(status_code=400, detail="This service does not exists!")
service = service[0]
service_id = service["service_id"]
srv_proto = service["proto"]
try:
async with asyncio.timeout(8):
try:
found_filters = get_filter_names(form.code, srv_proto)
except Exception as e:
if DEBUG:
traceback.print_exc()
raise HTTPException(status_code=400, detail="Compile error: "+str(e))
# Remove filters that are not in the new code
existing_filters = db.query("SELECT name FROM pyfilter WHERE service_id = ?;", service_id)
existing_filters = [ele["name"] for ele in existing_filters]
for filter in existing_filters:
if filter not in found_filters:
db.query("DELETE FROM pyfilter WHERE name = ?;", filter)
# Add filters that are in the new code but not in the database
for filter in found_filters:
if not db.query("SELECT 1 FROM pyfilter WHERE service_id = ? AND name = ?;", service_id, filter):
db.query("INSERT INTO pyfilter (name, service_id) VALUES (?, ?);", filter, service["service_id"])
# Eventually edited filters will be reloaded
os.makedirs("db/nfproxy_filters", exist_ok=True)
with open(f"db/nfproxy_filters/{service_id}.py", "w") as f:
f.write(form.code)
await firewall.get(service_id).update_filters()
await refresh_frontend()
except asyncio.TimeoutError:
if DEBUG:
traceback.print_exc()
raise HTTPException(status_code=400, detail="The operation took too long")
return {'status': 'ok'}
@app.get('/services/{service_id}/code', response_class=PlainTextResponse)
async def get_pyfilters_code(service_id: str):
"""Get the python filter for a service"""
if not db.query("SELECT 1 FROM services WHERE service_id = ?;", service_id):
raise HTTPException(status_code=400, detail="This service does not exists!")
try:
with open(f"db/nfproxy_filters/{service_id}.py") as f:
return f.read()
except FileNotFoundError:
return ""
#Socket io events
async def join_outstream(sid, data):
"""Client joins a room."""
srv = data.get("service")
if srv:
try:
srv_manager = firewall.get(srv)
room = f"nfproxy-outstream-{srv}"
await utils.socketio.enter_room(sid, room)
await utils.socketio.emit(room, srv_manager.read_outstrem_buffer(), room=sid)
except ServiceNotFoundException:
pass
async def leave_outstream(sid, data):
"""Client leaves a room."""
srv = data.get("service")
if srv:
await utils.socketio.leave_room(sid, f"nfproxy-outstream-{srv}")
async def join_exception(sid, data):
"""Client joins a room."""
srv = data.get("service")
if srv:
try:
srv_manager = firewall.get(srv)
room = f"nfproxy-exception-{srv}"
await utils.socketio.enter_room(sid, room)
await utils.socketio.emit(room, srv_manager.last_exception_time, room=sid)
except ServiceNotFoundException:
pass
async def leave_exception(sid, data):
"""Client leaves a room."""
srv = data.get("service")
if srv:
await utils.socketio.leave_room(sid, f"nfproxy-exception-{srv}")

456
backend/routers/nfregex.py
View File

@ -1,456 +0,0 @@
from base64 import b64decode
import secrets
import sqlite3
from fastapi import APIRouter, HTTPException
from fastapi.responses import PlainTextResponse
from pydantic import BaseModel
from modules.nfregex.nftables import FiregexTables
from modules.nfregex.firewall import STATUS, FirewallManager
from utils.sqlite import SQLite
from utils import ip_parse, refactor_name, socketio_emit, PortType
from modules.tls.service import activate_stream
from utils.models import ResetRequest, StatusMessageModel
from modules.nfregex.firegex import test_regex_validity
class ServiceModel(BaseModel):
status: str
service_id: str
port: PortType|None = None
name: str
proto: str
ip_int: str|None = None
n_regex: int
n_packets: int
fail_open: bool
target_type: str
tls_stream_id: str|None = None
class RenameForm(BaseModel):
name:str
class SettingsForm(BaseModel):
port: PortType|None = None
proto: str|None = None
ip_int: str|None = None
fail_open: bool|None = None
target_type: str|None = None
tls_stream_id: str|None = None
class RegexModel(BaseModel):
regex:str
mode:str
id:int
service_id:str
n_packets:int
is_case_sensitive:bool
active:bool
class RegexAddForm(BaseModel):
service_id: str
regex: str
mode: str
active: bool|None = None
is_case_sensitive: bool
class ServiceAddForm(BaseModel):
name: str
port: PortType|None = None
proto: str
ip_int: str|None = None
fail_open: bool = False
target_type: str = "flow"
tls_stream_id: str|None = None
class ServiceAddResponse(BaseModel):
status:str
service_id: str|None = None
app = APIRouter()
db = SQLite('db/nft-regex.db', {
'services': {
'service_id': 'VARCHAR(100) PRIMARY KEY',
'status': 'VARCHAR(100) NOT NULL',
'target_type': 'VARCHAR(10) NOT NULL CHECK(target_type IN ("flow", "tls")) DEFAULT "flow"',
'tls_stream_id': 'VARCHAR(100)',
'port': 'INT CHECK(port > 0 and port < 65536)',
'name': 'VARCHAR(100) NOT NULL UNIQUE',
'proto': 'VARCHAR(3) NOT NULL CHECK (proto IN ("tcp", "udp"))',
'ip_int': 'VARCHAR(100)',
'fail_open': 'BOOLEAN NOT NULL CHECK (fail_open IN (0, 1)) DEFAULT 1',
},
'regexes': {
'regex': 'TEXT NOT NULL',
'mode': 'VARCHAR(1) NOT NULL CHECK (mode IN ("C", "S", "B"))', # C = to the client, S = to the server, B = both
'service_id': 'VARCHAR(100) NOT NULL',
'blocked_packets': 'INTEGER UNSIGNED NOT NULL DEFAULT 0',
'regex_id': 'INTEGER PRIMARY KEY',
'is_case_sensitive' : 'BOOLEAN NOT NULL CHECK (is_case_sensitive IN (0, 1))',
'active' : 'BOOLEAN NOT NULL CHECK (active IN (0, 1)) DEFAULT 1',
'FOREIGN KEY (service_id)':'REFERENCES services (service_id)',
},
'QUERY':[
"CREATE UNIQUE INDEX IF NOT EXISTS unique_services ON services (port, ip_int, proto);",
"CREATE UNIQUE INDEX IF NOT EXISTS unique_regex_service ON regexes (regex,service_id,mode,is_case_sensitive);"
]
})
async def refresh_frontend(additional:list[str]=[]):
await socketio_emit(["nfregex"]+additional)
async def reset(params: ResetRequest):
if not params.delete:
db.backup()
await firewall.close()
FiregexTables().reset()
if params.delete:
db.delete()
db.init()
else:
db.restore()
try:
await firewall.init()
except Exception as e:
raise HTTPException(status_code=500, detail=str(e))
async def startup():
db.init()
try:
await firewall.init()
except Exception as e:
print("WARNING cannot start firewall:", e)
async def shutdown():
await firewall.close()
db.disconnect()
def gen_service_id():
while True:
res = secrets.token_hex(8)
if len(db.query('SELECT 1 FROM services WHERE service_id = ?;', res)) == 0:
break
return res
firewall = FirewallManager(db)
@app.get('/services', response_model=list[ServiceModel])
async def get_service_list():
"""Get the list of existent firegex services"""
res = db.query("""
SELECT
s.service_id service_id,
s.status status,
s.port port,
s.name name,
s.proto proto,
s.ip_int ip_int,
s.fail_open fail_open,
s.target_type target_type,
s.tls_stream_id tls_stream_id,
COUNT(r.regex_id) n_regex,
COALESCE(SUM(r.blocked_packets),0) n_packets
FROM services s LEFT JOIN regexes r ON s.service_id = r.service_id
GROUP BY s.service_id;
""")
return res
@app.get('/services/{service_id}', response_model=ServiceModel)
async def get_service_by_id(service_id: str):
"""Get info about a specific service using his id"""
res = db.query("""
SELECT
s.service_id service_id,
s.status status,
s.port port,
s.name name,
s.proto proto,
s.ip_int ip_int,
s.fail_open fail_open,
s.target_type target_type,
s.tls_stream_id tls_stream_id,
COUNT(r.regex_id) n_regex,
COALESCE(SUM(r.blocked_packets),0) n_packets
FROM services s LEFT JOIN regexes r ON s.service_id = r.service_id
WHERE s.service_id = ? GROUP BY s.service_id;
""", service_id)
if len(res) == 0:
raise HTTPException(status_code=400, detail="This service does not exists!")
return res[0]
@app.post('/services/{service_id}/stop', response_model=StatusMessageModel)
async def service_stop(service_id: str):
"""Request the stop of a specific service"""
await firewall.get(service_id).next(STATUS.STOP)
await refresh_frontend()
return {'status': 'ok'}
@app.post('/services/{service_id}/start', response_model=StatusMessageModel)
async def service_start(service_id: str):
"""Request the start of a specific service"""
srv = db.query("SELECT target_type, tls_stream_id FROM services WHERE service_id = ?;", service_id)
if srv and srv[0]["target_type"] == "tls":
if not await activate_stream(srv[0]["tls_stream_id"]):
raise HTTPException(status_code=400, detail="Linked TLS stream not found")
await firewall.get(service_id).next(STATUS.ACTIVE)
await refresh_frontend()
return {'status': 'ok'}
@app.delete('/services/{service_id}', response_model=StatusMessageModel)
async def service_delete(service_id: str):
"""Request the deletion of a specific service"""
db.query('DELETE FROM services WHERE service_id = ?;', service_id)
db.query('DELETE FROM regexes WHERE service_id = ?;', service_id)
await firewall.remove(service_id)
await refresh_frontend()
return {'status': 'ok'}
@app.put('/services/{service_id}/rename', response_model=StatusMessageModel)
async def service_rename(service_id: str, form: RenameForm):
"""Request to change the name of a specific service"""
form.name = refactor_name(form.name)
if not form.name:
raise HTTPException(status_code=400, detail="The name cannot be empty!")
try:
db.query('UPDATE services SET name=? WHERE service_id = ?;', form.name, service_id)
except sqlite3.IntegrityError:
raise HTTPException(status_code=400, detail="This name is already used")
await refresh_frontend()
return {'status': 'ok'}
@app.put('/services/{service_id}/settings', response_model=StatusMessageModel)
async def service_settings(service_id: str, form: SettingsForm):
"""Request to change the settings of a specific service (will cause a restart)"""
srv_check = db.query("SELECT proto, ip_int, port, target_type, tls_stream_id FROM services WHERE service_id = ?;", service_id)
if len(srv_check) == 0:
raise HTTPException(status_code=404, detail="Service not found")
old_srv = srv_check[0]
if form.proto is not None and form.proto not in ["tcp", "udp"]:
raise HTTPException(status_code=400, detail="Invalid protocol")
if form.port is not None and (form.port < 1 or form.port > 65535):
raise HTTPException(status_code=400, detail="Invalid port")
if form.ip_int is not None:
try:
form.ip_int = ip_parse(form.ip_int)
except ValueError:
raise HTTPException(status_code=400, detail="Invalid address")
new_ip = form.ip_int if form.ip_int is not None else old_srv["ip_int"]
new_port = form.port if form.port is not None else old_srv["port"]
new_proto = form.proto if form.proto is not None else old_srv["proto"]
new_target_type = form.target_type if form.target_type is not None else old_srv["target_type"]
new_tls_stream_id = form.tls_stream_id if form.tls_stream_id is not None else old_srv["tls_stream_id"]
if new_target_type == "tls":
if new_proto != "tcp":
raise HTTPException(status_code=400, detail="TLS is only supported for TCP protocol")
if not new_tls_stream_id:
raise HTTPException(status_code=400, detail="TLS stream ID is required when target type is tls")
if new_target_type == "flow" and (new_ip is None or new_port is None):
raise HTTPException(status_code=400, detail="IP and Port are required when target type is flow")
keys = []
values = []
for key, value in form.model_dump(exclude_none=True).items():
keys.append(key)
values.append(value)
if len(keys) == 0:
raise HTTPException(status_code=400, detail="No settings to change provided")
try:
db.query(f'UPDATE services SET {", ".join([f"{key}=?" for key in keys])} WHERE service_id = ?;', *values, service_id)
except sqlite3.IntegrityError:
raise HTTPException(status_code=400, detail="A service with these settings already exists")
old_status = firewall.get(service_id).status
await firewall.remove(service_id)
if old_status == STATUS.ACTIVE and new_target_type == "tls":
await activate_stream(new_tls_stream_id)
await firewall.reload()
await firewall.get(service_id).next(old_status)
await refresh_frontend()
return {'status': 'ok'}
@app.get('/services/{service_id}/regexes', response_model=list[RegexModel])
async def get_service_regexe_list(service_id: str):
"""Get the list of the regexes of a service"""
if not db.query("SELECT 1 FROM services s WHERE s.service_id = ?;", service_id):
raise HTTPException(status_code=400, detail="This service does not exists!")
return db.query("""
SELECT
regex, mode, regex_id `id`, service_id,
blocked_packets n_packets, is_case_sensitive, active
FROM regexes WHERE service_id = ?;
""", service_id)
@app.get('/regexes/{regex_id}', response_model=RegexModel)
async def get_regex_by_id(regex_id: int):
"""Get regex info using his id"""
res = db.query("""
SELECT
regex, mode, regex_id `id`, service_id,
blocked_packets n_packets, is_case_sensitive, active
FROM regexes WHERE `id` = ?;
""", regex_id)
if len(res) == 0:
raise HTTPException(status_code=400, detail="This regex does not exists!")
return res[0]
@app.delete('/regexes/{regex_id}', response_model=StatusMessageModel)
async def regex_delete(regex_id: int):
"""Delete a regex using his id"""
res = db.query('SELECT * FROM regexes WHERE regex_id = ?;', regex_id)
if len(res) != 0:
db.query('DELETE FROM regexes WHERE regex_id = ?;', regex_id)
await firewall.get(res[0]["service_id"]).update_filters()
await refresh_frontend()
return {'status': 'ok'}
@app.post('/regexes/{regex_id}/enable', response_model=StatusMessageModel)
async def regex_enable(regex_id: int):
"""Request the enabling of a regex"""
res = db.query('SELECT * FROM regexes WHERE regex_id = ?;', regex_id)
if len(res) != 0:
db.query('UPDATE regexes SET active=1 WHERE regex_id = ?;', regex_id)
await firewall.get(res[0]["service_id"]).update_filters()
await refresh_frontend()
return {'status': 'ok'}
@app.post('/regexes/{regex_id}/disable', response_model=StatusMessageModel)
async def regex_disable(regex_id: int):
"""Request the deactivation of a regex"""
res = db.query('SELECT * FROM regexes WHERE regex_id = ?;', regex_id)
if len(res) != 0:
db.query('UPDATE regexes SET active=0 WHERE regex_id = ?;', regex_id)
await firewall.get(res[0]["service_id"]).update_filters()
await refresh_frontend()
return {'status': 'ok'}
@app.post('/regexes', response_model=StatusMessageModel)
async def add_new_regex(form: RegexAddForm):
"""Add a new regex"""
regex_correct, message = await test_regex_validity(b64decode(form.regex))
if not regex_correct:
raise HTTPException(status_code=400, detail=f"Invalid regex: {message}")
try:
db.query("INSERT INTO regexes (service_id, regex, mode, is_case_sensitive, active ) VALUES (?, ?, ?, ?, ?);",
form.service_id, form.regex, form.mode, form.is_case_sensitive, True if form.active is None else form.active )
except sqlite3.IntegrityError:
raise HTTPException(status_code=400, detail="An identical regex already exists")
await firewall.get(form.service_id).update_filters()
await refresh_frontend()
return {'status': 'ok'}
@app.post('/services', response_model=ServiceAddResponse)
async def add_new_service(form: ServiceAddForm):
"""Add a new service"""
if form.target_type == "flow":
if form.ip_int is None or form.port is None:
raise HTTPException(status_code=400, detail="IP and Port are required when target type is flow")
try:
form.ip_int = ip_parse(form.ip_int)
except ValueError:
raise HTTPException(status_code=400, detail="Invalid address")
elif form.target_type == "tls":
if not form.tls_stream_id:
raise HTTPException(status_code=400, detail="TLS stream ID is required when target type is tls")
if form.proto not in ["tcp", "udp"]:
raise HTTPException(status_code=400, detail="Invalid protocol")
if form.target_type == "tls" and form.proto != "tcp":
raise HTTPException(status_code=400, detail="TLS is only supported for TCP protocol")
srv_id = None
try:
srv_id = gen_service_id()
db.query("INSERT INTO services (service_id ,name, port, status, proto, ip_int, fail_open, target_type, tls_stream_id) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?)",
srv_id, refactor_name(form.name), form.port, STATUS.STOP, form.proto, form.ip_int, form.fail_open, form.target_type, form.tls_stream_id)
except sqlite3.IntegrityError:
raise HTTPException(status_code=400, detail="This type of service already exists")
await firewall.reload()
await refresh_frontend()
return {'status': 'ok', 'service_id': srv_id}
@app.get('/metrics', response_class = PlainTextResponse)
async def metrics():
"""Aggregate metrics"""
stats = db.query("""
SELECT
s.name,
s.status,
r.regex,
r.mode,
r.is_case_sensitive,
r.blocked_packets,
r.active
FROM regexes r LEFT JOIN services s ON s.service_id = r.service_id;
""")
metrics = []
def sanitize(s):
return s.replace('\\', '\\\\').replace('"', '\\"').replace('\n', '\\n')
for stat in stats:
props = f'service_name="{sanitize(stat["name"])}",regex="{sanitize(b64decode(stat["regex"]).decode())}",mode="{stat["mode"]}",is_case_sensitive="{stat["is_case_sensitive"]}"'
metrics.append(f'firegex_blocked_packets{{{props}}} {stat["blocked_packets"]}')
metrics.append(f'firegex_active{{{props}}} {int(stat["active"] and stat["status"] == "active")}')
return "\n".join(metrics)
@app.get('/services/{service_id}/export')
async def export_service_regexes(service_id: str):
"""Export all regexes for a specific service as JSON"""
if len(db.query('SELECT 1 FROM services WHERE service_id = ?;', service_id)) == 0:
raise HTTPException(status_code=404, detail="Service not found")
regexes = db.query("SELECT regex, mode, is_case_sensitive, active FROM regexes WHERE service_id = ?;", service_id)
return {"regexes": regexes}
@app.post('/services/{service_id}/import', response_model=StatusMessageModel)
async def import_service_regexes(service_id: str, data: dict):
"""Import regexes for a specific service from JSON"""
if len(db.query('SELECT 1 FROM services WHERE service_id = ?;', service_id)) == 0:
raise HTTPException(status_code=404, detail="Service not found")
regexes = data.get("regexes", [])
if not isinstance(regexes, list):
raise HTTPException(status_code=400, detail="Invalid format")
for r in regexes:
regex = r.get("regex")
mode = r.get("mode")
is_case_sensitive = r.get("is_case_sensitive", False)
active = r.get("active", True)
if not regex or not mode:
continue
try:
db.query('INSERT INTO regexes (regex, mode, service_id, is_case_sensitive, active) VALUES (?, ?, ?, ?, ?);',
regex, mode, service_id, int(is_case_sensitive), int(active))
# Since we inserted it directly, we need to notify the firewall to update its rules if it's active
if active and firewall.get(service_id).status == STATUS.ACTIVE:
# Easiest way to sync is to restart the service rule engine
await firewall.get(service_id).next(STATUS.ACTIVE)
except sqlite3.IntegrityError:
# Skip duplicates
pass
# Force reload of all rules for this service
old_status = firewall.get(service_id).status
await firewall.remove(service_id)
await firewall.reload()
await firewall.get(service_id).next(old_status)
await refresh_frontend()
return {'status': 'ok'}

184
backend/routers/porthijack.py
View File

@ -1,184 +0,0 @@
import secrets
import sqlite3
from fastapi import APIRouter, HTTPException
from pydantic import BaseModel
from modules.porthijack.models import Service
from utils.sqlite import SQLite
from utils import addr_parse, ip_family, refactor_name, socketio_emit, PortType
from utils.models import ResetRequest, StatusMessageModel
from modules.porthijack.nftables import FiregexTables
from modules.porthijack.firewall import FirewallManager
class ServiceModel(BaseModel):
service_id: str
active: bool
public_port: PortType
proxy_port: PortType
name: str
proto: str
ip_src: str
ip_dst: str
class RenameForm(BaseModel):
name:str
class ServiceAddForm(BaseModel):
name: str
public_port: PortType
proxy_port: PortType
proto: str
ip_src: str
ip_dst: str
class ServiceAddResponse(BaseModel):
status:str
service_id: str|None = None
app = APIRouter()
db = SQLite('db/port-hijacking.db', {
'services': {
'service_id': 'VARCHAR(100) PRIMARY KEY',
'active' : 'BOOLEAN NOT NULL CHECK (active IN (0, 1))',
'public_port': 'INT NOT NULL CHECK(public_port > 0 and public_port < 65536)',
'proxy_port': 'INT NOT NULL CHECK(proxy_port > 0 and proxy_port < 65536)',
'name': 'VARCHAR(100) NOT NULL UNIQUE',
'proto': 'VARCHAR(3) NOT NULL CHECK (proto IN ("tcp", "udp"))',
'ip_src': 'VARCHAR(100) NOT NULL',
'ip_dst': 'VARCHAR(100) NOT NULL',
}
})
async def reset(params: ResetRequest):
if not params.delete:
db.backup()
await firewall.close()
FiregexTables().reset()
if params.delete:
db.delete()
db.init()
else:
db.restore()
await firewall.init()
async def startup():
db.init()
await firewall.init()
async def shutdown():
await firewall.close()
db.disconnect()
async def refresh_frontend(additional:list[str]=[]):
await socketio_emit(["porthijack"]+additional)
def gen_service_id():
while True:
res = secrets.token_hex(8)
if len(db.query('SELECT 1 FROM services WHERE service_id = ?;', res)) == 0:
break
return res
firewall = FirewallManager(db)
@app.get('/services', response_model=list[ServiceModel])
async def get_service_list():
"""Get the list of existent firegex services"""
return db.query("SELECT service_id, active, public_port, proxy_port, name, proto, ip_src, ip_dst FROM services;")
@app.get('/services/{service_id}', response_model=ServiceModel)
async def get_service_by_id(service_id: str):
"""Get info about a specific service using his id"""
res = db.query("SELECT service_id, active, public_port, proxy_port, name, proto, ip_src, ip_dst FROM services WHERE service_id = ?;", service_id)
if len(res) == 0:
raise HTTPException(status_code=400, detail="This service does not exists!")
return res[0]
@app.post('/services/{service_id}/stop', response_model=StatusMessageModel)
async def service_stop(service_id: str):
"""Request the stop of a specific service"""
await firewall.get(service_id).disable()
await refresh_frontend()
return {'status': 'ok'}
@app.post('/services/{service_id}/start', response_model=StatusMessageModel)
async def service_start(service_id: str):
"""Request the start of a specific service"""
await firewall.get(service_id).enable()
await refresh_frontend()
return {'status': 'ok'}
@app.delete('/services/{service_id}', response_model=StatusMessageModel)
async def service_delete(service_id: str):
"""Request the deletion of a specific service"""
db.query('DELETE FROM services WHERE service_id = ?;', service_id)
await firewall.remove(service_id)
await refresh_frontend()
return {'status': 'ok'}
@app.put('/services/{service_id}/rename', response_model=StatusMessageModel)
async def service_rename(service_id: str, form: RenameForm):
"""Request to change the name of a specific service"""
form.name = refactor_name(form.name)
if not form.name:
raise HTTPException(status_code=400, detail="The name cannot be empty!")
try:
db.query('UPDATE services SET name=? WHERE service_id = ?;', form.name, service_id)
except sqlite3.IntegrityError:
raise HTTPException(status_code=400, detail="This name is already used")
await refresh_frontend()
return {'status': 'ok'}
class ChangeDestination(BaseModel):
ip_dst: str
proxy_port: PortType
@app.put('/services/{service_id}/change-destination', response_model=StatusMessageModel)
async def service_change_destination(service_id: str, form: ChangeDestination):
"""Request to change the proxy destination of the service"""
try:
form.ip_dst = addr_parse(form.ip_dst)
except ValueError:
raise HTTPException(status_code=400, detail="Invalid address")
srv = Service.from_dict(db.query('SELECT * FROM services WHERE service_id = ?;', service_id)[0])
if ip_family(form.ip_dst) != ip_family(srv.ip_src):
raise HTTPException(status_code=400, detail="The destination ip is not of the same family as the source ip")
try:
db.query('UPDATE services SET proxy_port=?, ip_dst=? WHERE service_id = ?;', form.proxy_port, form.ip_dst, service_id)
except sqlite3.IntegrityError:
raise HTTPException(status_code=400, detail="Invalid proxy port or service")
srv.ip_dst = form.ip_dst
srv.proxy_port = form.proxy_port
await firewall.get(service_id).refresh(srv)
await refresh_frontend()
return {'status': 'ok'}
@app.post('/services', response_model=ServiceAddResponse)
async def add_new_service(form: ServiceAddForm):
"""Add a new service"""
try:
form.ip_src = addr_parse(form.ip_src)
form.ip_dst = addr_parse(form.ip_dst)
except ValueError:
raise HTTPException(status_code=400, detail="Invalid address")
if ip_family(form.ip_dst) != ip_family(form.ip_src):
raise HTTPException(status_code=400, detail="Destination and source addresses must be of the same family")
if form.proto not in ["tcp", "udp"]:
raise HTTPException(status_code=400, detail="Invalid protocol")
srv_id = None
try:
srv_id = gen_service_id()
db.query("INSERT INTO services (service_id, active, public_port, proxy_port, name, proto, ip_src, ip_dst) VALUES (?, ?, ?, ?, ?, ?, ?, ?)",
srv_id, False, form.public_port, form.proxy_port , form.name, form.proto, form.ip_src, form.ip_dst)
except sqlite3.IntegrityError:
raise HTTPException(status_code=400, detail="This type of service already exists")
await firewall.reload()
await refresh_frontend()
return {'status': 'ok', 'service_id': srv_id}

2569
backend/routers/services.py Normal file
View File

File diff suppressed because it is too large Load Diff

177
backend/routers/tls.py
View File

@ -1,177 +0,0 @@
from fastapi import APIRouter, HTTPException
import subprocess
from pydantic import BaseModel
from modules.tls.manager import TLSManager, ip_parse
from modules.tls.service import reload_tls
from utils.models import StatusMessageModel
import secrets
import routers.nfproxy as nfproxy_router
import routers.nfregex as nfregex_router
def gen_service_id():
while True:
res = secrets.token_hex(8)
if not manager.get_stream(res):
break
return res
class TLSStreamAddForm(BaseModel):
name: str
ip_int: str
port: int
cert: str
key: str
enabled: bool = True
class TLSStreamEditForm(BaseModel):
name: str | None = None
ip_int: str | None = None
port: int | None = None
cert: str | None = None
key: str | None = None
class TLSStreamResponse(BaseModel):
status: str
stream_id: str | None = None
app = APIRouter()
manager = TLSManager()
# (module, human name) pairs of the routers that can depend on a TLS stream
DEPENDENT_ROUTERS = (("nfproxy", nfproxy_router), ("nfregex", nfregex_router))
def get_dependent_services(stream_id: str) -> list[tuple[str, "object", str, str]]:
"""Returns (module_name, router_module, service_id, status) for every nfproxy/nfregex service linked to a TLS stream."""
deps = []
for module_name, router_mod in DEPENDENT_ROUTERS:
rows = router_mod.db.query(
"SELECT service_id, status FROM services WHERE target_type = 'tls' AND tls_stream_id = ?;", stream_id
)
for row in rows:
deps.append((module_name, router_mod, row["service_id"], row["status"]))
return deps
async def sync_dependent_services_address(stream_id: str, new_ip: str, new_port: int):
"""After a stream's ip/port changed, mirror it onto dependent services and rebind their live nft rules."""
touched_modules = set()
for module_name, router_mod, srv_id, status in get_dependent_services(stream_id):
router_mod.db.query("UPDATE services SET ip_int = ?, port = ? WHERE service_id = ?;", new_ip, new_port, srv_id)
await router_mod.firewall.remove(srv_id)
await router_mod.firewall.reload()
await router_mod.firewall.get(srv_id).next(status)
touched_modules.add(module_name)
for module_name, router_mod in DEPENDENT_ROUTERS:
if module_name in touched_modules:
await router_mod.refresh_frontend()
async def cascade_stop_dependents(stream_id: str):
"""When a TLS stream is stopped, stop every active service that depends on it."""
touched_modules = set()
for module_name, router_mod, srv_id, status in get_dependent_services(stream_id):
if status == "active":
await router_mod.firewall.get(srv_id).next("stop")
touched_modules.add(module_name)
for module_name, router_mod in DEPENDENT_ROUTERS:
if module_name in touched_modules:
await router_mod.refresh_frontend()
async def startup():
await reload_tls()
async def shutdown():
subprocess.run(["nginx", "-c", "/tmp/firegex_nginx.conf", "-s", "stop", "-e", "/tmp/firegex_nginx_error.log"], stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)
async def reset(form):
if form.delete:
manager.db.delete()
manager.db.init()
await reload_tls()
@app.get('/streams')
async def get_streams():
return manager.get_streams()
@app.post('/streams', response_model=TLSStreamResponse)
async def add_stream(form: TLSStreamAddForm):
try:
ip_int = ip_parse(form.ip_int)
except ValueError:
raise HTTPException(status_code=400, detail="Invalid address")
if manager.get_stream_by_ip_port(ip_int, form.port):
raise HTTPException(status_code=400, detail="A TLS stream for this IP and Port already exists")
stream_id = gen_service_id()
try:
manager.add_stream(stream_id, form.name, ip_int, form.port, form.cert, form.key, status="active" if form.enabled else "stop")
except Exception as e:
raise HTTPException(status_code=500, detail=str(e))
await reload_tls()
return {"status": "ok", "stream_id": stream_id}
@app.put('/streams/{stream_id}', response_model=StatusMessageModel)
async def edit_stream(stream_id: str, form: TLSStreamEditForm):
stream = manager.get_stream(stream_id)
if not stream:
raise HTTPException(status_code=404, detail="TLS stream not found")
new_ip_int = None
if form.ip_int is not None:
try:
new_ip_int = ip_parse(form.ip_int)
except ValueError:
raise HTTPException(status_code=400, detail="Invalid address")
if form.port is not None and (form.port < 1 or form.port > 65535):
raise HTTPException(status_code=400, detail="Invalid port")
check_ip = new_ip_int if new_ip_int is not None else stream["ip_int"]
check_port = form.port if form.port is not None else stream["port"]
address_changed = check_ip != stream["ip_int"] or check_port != stream["port"]
if address_changed:
existing = manager.get_stream_by_ip_port(check_ip, check_port)
if existing and existing["id"] != stream_id:
raise HTTPException(status_code=400, detail="A TLS stream for this IP and Port already exists")
try:
manager.update_stream(stream_id, name=form.name, ip_int=new_ip_int, port=form.port, cert=form.cert, key=form.key)
except Exception as e:
raise HTTPException(status_code=500, detail=str(e))
if address_changed:
await sync_dependent_services_address(stream_id, check_ip, check_port)
await reload_tls()
return {"status": "ok"}
@app.delete('/streams/{stream_id}', response_model=StatusMessageModel)
async def delete_stream(stream_id: str):
stream = manager.get_stream(stream_id)
if not stream:
raise HTTPException(status_code=404, detail="TLS stream not found")
if get_dependent_services(stream_id):
raise HTTPException(status_code=400, detail="Cannot delete this TLS stream because it is actively used by NFProxy or NFRegex services.")
manager.delete_stream(stream_id)
await reload_tls()
return {"status": "ok"}
@app.post('/streams/{stream_id}/start', response_model=StatusMessageModel)
async def start_stream(stream_id: str):
if not manager.get_stream(stream_id):
raise HTTPException(status_code=404, detail="TLS stream not found")
manager.update_status(stream_id, "active")
await reload_tls()
return {"status": "ok"}
@app.post('/streams/{stream_id}/stop', response_model=StatusMessageModel)
async def stop_stream(stream_id: str):
if not manager.get_stream(stream_id):
raise HTTPException(status_code=404, detail="TLS stream not found")
manager.update_status(stream_id, "stop")
await reload_tls()
await cascade_stop_dependents(stream_id)
return {"status": "ok"}

136
backend/utils/__init__.py
View File

@ -1,16 +1,18 @@
import asyncio
from ipaddress import ip_address, ip_interface
import os
import re
import socket
import psutil
import sys
import nftables
try:
import nftables
except ImportError:
nftables = None
from socketio import AsyncServer
from typing import Annotated, List, Union
from functools import wraps
from pydantic import BaseModel, ValidationError, Field
from pydantic import BaseModel, Field
import traceback
from utils.models import StatusMessageModel
from pathlib import Path
from fastapi import HTTPException, status
@ -103,6 +105,40 @@ API_VERSION = _get_version()
PortType = Annotated[int, Field(gt=0, lt=65536)]
#: What a backup entry may be called: a plain basename of one of the two shapes
#: `export_db()` produces — `<name>.db` for a database and `<id>.py` for a filter's code.
#: The charset forbids path separators and `..`, so a crafted key cannot escape the
#: directory it is joined onto.
#:
#: Here rather than in `app.py`, beside `safe_join`, because the two are one defence in
#: two halves: this decides what a name may contain and that one decides where it may
#: land. Keeping them apart meant the charset could only be exercised by importing the
#: whole application — a server, a JWT library and everything else `app.py` pulls in —
#: to check two regular expressions.
#:
#: Matched with `fullmatch`, never `match`: in Python `$` also matches *before a trailing
#: newline*, so `"services.db\n"` satisfied a pattern written to mean "this and nothing
#: else".
SAFE_DB_NAME = re.compile(r'[A-Za-z0-9_-]+\.db')
SAFE_PY_NAME = re.compile(r'[A-Za-z0-9_-]+\.py')
def boot_auth_mode(held: str | None, fresh_boot: bool, env_disabled: bool) -> tuple[bool, bool]:
"""Whether authentication starts off disabled, and whether the host's stored decision
is kept — `(disabled, keep_held)`.
`held` is what `run.py config` last stored for the host (`"1"`, `"0"` or `None`). A
fresh boot — a container or a standalone start `run.py` has just written the
environment for — takes the environment, which already says what the configuration
says, and the stored decision is dropped. Any other boot is Docker starting an existing
container again with the environment it was created with, which may be older than the
stored decision, and the stored decision wins.
"""
if fresh_boot or held is None:
return env_disabled, False
return held == "1", True
def safe_join(base_dir: Union[str, Path], *paths: str) -> Path:
"""
Safely join a base directory with one or more path components.
@ -187,6 +223,31 @@ def is_ip_parse(ip:str):
except Exception:
return False
def is_interface_name(name: str) -> bool:
if not isinstance(name, str):
return False
name = name.strip()
return bool(re.match(r'^[a-zA-Z0-9_.:-]+$', name)) and 0 < len(name) <= 15
def parse_ip_or_int(ip: str) -> str:
"""Normalise what an operator typed into what the rules and the key will use.
**Stripped once, before either question is asked.** `is_interface_name` strips and
`is_ip_parse` does not, so surrounding whitespace used to decide *which of the two an
address was*: `" 10.0.0.1 "` failed the IP test, passed the interface-name test —
every character in it is in that charset — and was stored verbatim. Nothing then
broke loudly, which is the problem: the rules still matched (both sides re-parse the
value), but `"10.0.0.1"` and `"10.0.0.1/32"` are different strings, so the
`(ip_int, port, proto)` uniqueness key no longer saw them as the same address and two
services could each believe they were protecting it.
"""
ip_str = str(ip).strip()
if is_ip_parse(ip_str):
return ip_parse(ip_str)
if is_interface_name(ip_str):
return ip_str
raise ValueError(f"'{ip}' is neither a valid IP address nor a valid interface name")
def addr_parse(ip:str):
return str(ip_address(ip))
@ -201,18 +262,21 @@ def get_interfaces():
yield {"name": int_name, "addr":interf.address}
return list(_get_interfaces())
def get_interface_ips(iface_name: str) -> list[str]:
ips = []
for int_name, interfs in psutil.net_if_addrs().items():
if int_name == iface_name:
for interf in interfs:
if interf.family in [socket.AF_INET, socket.AF_INET6]:
ips.append(interf.address)
return ips
def nftables_int_to_json(ip_int):
ip_int = ip_parse(ip_int)
ip_addr = str(ip_int).split("/")[0]
ip_addr_cidr = int(str(ip_int).split("/")[1])
return {"prefix": {"addr": ip_addr, "len": ip_addr_cidr}}
def nftables_json_to_int(ip_json_int):
if isinstance(ip_json_int,str):
return str(ip_parse(ip_json_int))
else:
return f'{ip_json_int["prefix"]["addr"]}/{ip_json_int["prefix"]["len"]}'
class Singleton(object):
__instance = None
def __new__(class_, *args, **kwargs):
@ -222,12 +286,12 @@ class Singleton(object):
class NFTableManager(Singleton):
table_name = "firegex"
table_name = "fgex"
def __init__(self, init_cmd, reset_cmd):
self.__init_cmds = init_cmd
self.__reset_cmds = reset_cmd
self.nft = nftables.Nftables()
self.nft = nftables.Nftables() if nftables is not None else None
def raw_cmd(self, *cmds):
return self.nft.json_cmd({"nftables": list(cmds)})
@ -247,15 +311,27 @@ class NFTableManager(Singleton):
def reset(self):
self.raw_cmd(*self.__reset_cmds)
def list_rules(self, tables = None, chains = None):
for filter in [ele["rule"] for ele in self.raw_list() if "rule" in ele ]:
def list_rules(self, tables = None, chains = None, family: str | None = None):
for filter in [ele["rule"] for ele in self.raw_list(tables, family) if "rule" in ele ]:
if tables and filter["table"] not in tables:
continue
if chains and filter["chain"] not in chains:
continue
yield filter
def raw_list(self):
def raw_list(self, tables = None, family: str | None = None):
# One table of ours when that is all that is asked for, rather than the host's
# whole ruleset: on a node running Docker or Kubernetes that is thousands of rules
# of somebody else's, serialised and parsed on every lookup — and a lookup runs
# inside the backend's one event loop, on every start, stop, address change and
# statistics request. A table that is not there is simply no rules.
if tables and family:
out = []
for table in tables:
code, listed, _ = self.raw_cmd({"list": {"table": {"family": family, "name": table}}})
if code == 0:
out.extend(listed["nftables"])
return out
return self.cmd({"list": {"ruleset": None}})["nftables"]
def _json_like(obj: BaseModel|List[BaseModel], unset=False, convert_keys:dict[str, str]=None, exclude:list[str]=None, mode:str="json"):
@ -275,36 +351,6 @@ def json_like(obj: BaseModel|List[BaseModel], unset=False, convert_keys:dict[str
return [_json_like(ele, unset=unset, convert_keys=convert_keys, exclude=exclude, mode=mode) for ele in obj]
return _json_like(obj, unset=unset, convert_keys=convert_keys, exclude=exclude, mode=mode)
def register_event(sio_server: AsyncServer, event_name: str, model: BaseModel, response_model: BaseModel|None = None):
def decorator(func):
@sio_server.on(event_name) # Automatically registers the event
@wraps(func)
async def wrapper(sid, data):
try:
# Parse and validate incoming data
parsed_data = model.model_validate(data)
except ValidationError:
return json_like(StatusMessageModel(status=f"Invalid {event_name} request"))
# Call the original function with the parsed data
result = await func(sid, parsed_data)
# If a response model is provided, validate the output
if response_model:
try:
parsed_result = response_model.model_validate(result)
except ValidationError:
traceback.print_exc()
return json_like(StatusMessageModel(status=f"SERVER ERROR: Invalid {event_name} response"))
else:
parsed_result = result
# Emit the validated result
if parsed_result:
if isinstance(parsed_result, BaseModel):
return json_like(parsed_result)
return parsed_result
return wrapper
return decorator
def nicenessify(priority:int, pid:int|None=None):
try:
pid = os.getpid() if pid is None else pid

8
backend/utils/models.py
View File

@ -16,6 +16,14 @@ class PasswordChangeForm(BaseModel):
password: str
expire: bool
class AuthModeForm(BaseModel):
#: True turns authentication off, False turns it back on. The two are **not**
#: symmetrical and the endpoint says so rather than this form: turning it off is an
#: administrator's deployment choice, while turning it on — asked while it is off,
#: when every caller is already an administrator — takes a token this instance signed
#: *before* it was turned off, and a password to ask for. See `set_auth_mode`.
disabled: bool
class ChangePasswordModel(BaseModel):
status: str
access_token: str|None = None

36
backend/utils/sqlite.py
View File

@ -43,9 +43,6 @@ class SQLite():
if were_active:
self.connect()
def delete_backup(self):
self.__backup = None
def disconnect(self) -> None:
if self.conn:
self.conn.close()
@ -133,12 +130,33 @@ class SQLite():
for table in tables:
tname = table["name"]
if tname == 'keys_values':
res[tname] = self.query(f"SELECT * FROM {tname} WHERE key NOT IN ('password', 'secret');")
# Four keys never travel in a backup. `password` and `secret` are this
# instance's credentials, and `auth_disabled` with `auth_disabled_host`
# are *whether it asks for them* — a property of where this firegex is
# deployed, not of the configuration being backed up. A backup that could
# carry them would be a backup that can turn a firewall's authentication
# off on restore.
res[tname] = self.query(
f"SELECT * FROM {tname} WHERE key NOT IN "
f"('password', 'secret', 'auth_disabled', 'auth_disabled_host');"
)
else:
res[tname] = self.query(f"SELECT * FROM {tname};")
return res
def load(self, data: dict):
def load(self, data: dict, dry_run: bool = False):
"""Replace the tables named in `data` with its rows, all in one transaction.
`dry_run` does every write and then rolls it back: what an import uses to find out
whether each of its databases *would* load before it changes any of them. Loaded
one after another for real, a refusal from the third left the first two replaced —
`keys_values` among them, which is where the password and the signing secret live.
Every identifier is quoted. Columns are taken from the table itself rather than
from the backup, so quoting cannot let anything in, and leaving them bare made a
column called `table` — the firewall's — a syntax error: no backup holding a
single firewall rule could be imported at all.
"""
self.connect()
cur = self.conn.cursor()
try:
@ -151,7 +169,7 @@ class SQLite():
# Ignore anything that isn't a proper list of row objects
if not isinstance(rows, list) or not all(isinstance(r, dict) for r in rows):
continue
cur.execute(f"DELETE FROM {table_name};")
cur.execute(f'DELETE FROM "{table_name}";')
if rows:
# Column names are identifiers and can't be parameterized, so only
# keep the ones that actually exist on the table - this stops a
@ -162,11 +180,11 @@ class SQLite():
if not cols:
continue
placeholders = ", ".join(["?"] * len(cols))
col_names = ", ".join(cols)
query = f"INSERT INTO {table_name} ({col_names}) VALUES ({placeholders})"
col_names = ", ".join(f'"{c}"' for c in cols)
query = f'INSERT INTO "{table_name}" ({col_names}) VALUES ({placeholders})'
for row in rows:
cur.execute(query, [row.get(c) for c in cols])
cur.execute("COMMIT")
cur.execute("ROLLBACK" if dry_run else "COMMIT")
except Exception as e:
cur.execute("ROLLBACK")
raise e

BIN
docs/Firegex_Screenshot.png
View File

Binary file not shown.
Side by Side Swipe Overlay

Before

Width:  |  Height:  |  Size: 298 KiB

After

Width:  |  Height:  |  Size: 246 KiB

36
docs/firewall.md
View File

@ -7,13 +7,13 @@ Firewall Rules is a plain [nftables](https://netfilter.org/) allow/drop/reject r
Each rule matches on:
- **Protocol**: `tcp`, `udp`, `both` (adds a matching TCP and UDP rule) or `any` (matches all protocols, ports are ignored).
- **Source / Destination**: an IP/CIDR, or an interface name (e.g. `eth0`) instead of an address.
- **Source / Destination**: an IP/CIDR, or an interface name (e.g. `eth0`) instead of an address — at most 15 characters of letters, digits and `_.:-`, as the kernel takes them, optionally ending in `*` to match every interface starting that way (`br-*`).
- **Source / Destination port range**: `from`/`to` for each side; leave as the full `1-65535` range to match any port.
- **Direction** (`mode`): `in` (incoming traffic — the nftables `INPUT`/`PREROUTING` hook), `out` (outgoing traffic — `OUTPUT`/`POSTROUTING`), or `forward` (traffic routed through this host, `FORWARD` — only meaningful with the `filter` table, see below).
- **Table**: `filter` — standard firewall rules evaluated at the normal input/output/forward hooks; `mangle` — rules evaluated earlier in the pipeline (`prerouting`/`postrouting`, before routing decisions), useful when a rule needs to run before other processing (e.g. before a [Hijack Port to Proxy](porthijack.md) or [TLS Decrypt](tls.md) rule takes effect on the same traffic).
- **Table**: `filter` — standard firewall rules evaluated at the normal input/output/forward hooks; `mangle` — rules evaluated earlier in the pipeline (`prerouting`/`postrouting`, before routing decisions), useful when a rule needs to run before other processing (e.g. before a an external-proxy [service](services.md) or TLS rule takes effect on the same traffic).
- **Action**: `accept`, `drop`, or `reject` (closes the connection with an ICMP/RST reply instead of silently dropping it). `reject` on outgoing (`out`) traffic isn't meaningful — Firegex silently treats it as `drop` in that direction.
Rules are evaluated in order; the first match wins. Traffic that matches no rule falls through to the global **policy** (`accept`/`drop`/`reject`), which applies to incoming and forwarded traffic — outgoing traffic is always allowed by default regardless of the policy, so Firegex itself is never at risk of losing its own outbound connectivity by misconfiguring rules.
Rules are evaluated in order; the first match wins. A change nftables refuses is refused as a whole: the firewall already in force stays exactly as it was, and so does the saved configuration. Traffic that matches no rule falls through to the global **policy** (`accept`/`drop`/`reject`), which applies to incoming and forwarded traffic — outgoing traffic is always allowed by default regardless of the policy, so Firegex itself is never at risk of losing its own outbound connectivity by misconfiguring rules.
### Global settings
@ -27,9 +27,35 @@ Beyond the rule list and policy, a few toggles affect the whole module:
- **allow_dhcp**: always accept DHCP traffic.
- **multicast_dns**: always accept mDNS (multicast DNS) traffic.
- **allow_upnp**: always accept UPnP traffic.
- **allow_dnat** (on by default, "Leave container traffic to the container runtime"): forwarded traffic whose destination another NAT rewrote — a port **Docker or podman published** for a container, a router's port forward — and traffic **the containers send**, to each other or out (anything arriving from `docker0`, `br-*`, `docker_gwbridge`, `podman*` or `cni-podman*`), is left to the container runtime's own rules when none of *yours* matched it, instead of meeting the forward policy. Without it, a drop policy cuts a web container off from its database on the same bridge. Your `forward` rules still apply to it first, so a rule that drops it drops it.
Each of these, when enabled, inserts a small accept rule ahead of your own rules — they're conveniences for common cases you'd otherwise have to write by hand.
Each of the others, when enabled, inserts a small accept rule ahead of your own rules — they're conveniences for common cases you'd otherwise have to write by hand. **allow_dnat** is the exception: it sits after your rules, because it only decides what the *policy* does with what they did not match.
Why it exists: a published container port is reached by *forwarding* — Docker rewrites the destination to the container's address, and the traffic is routed to it rather than delivered to this host — so an `in` rule for that port never matches it, and with the policy at `drop` it would be dropped at the forward hook. When Firegex's rules lived in the tables `iptables` uses, Docker's own accept in that same chain let such traffic through; in tables of Firegex's own it has to be said. Turn it off to have the forward policy apply to published ports too, and allow them with `forward` rules.
## How it works
Rules are compiled directly into nftables' JSON rule format and applied via the nftables JSON API — there's no packet interception/inspection involved (unlike [Netfilter Regex](nfregex.md) or [Netfilter Proxy](nfproxy.md), which sit in front of a service via nfqueue). Firegex maintains its own dedicated chains (jumped to from the base `INPUT`/`OUTPUT`/`FORWARD` filter hooks and `PREROUTING`/`POSTROUTING` mangle hooks) so its rules can be fully reset without touching anything else on the system.
Rules are compiled directly into nftables' JSON rule format and applied via the nftables JSON API — there's no packet interception/inspection involved (unlike [services](services.md), which sit in front of a service and inspect its traffic).
**Everything Firegex installs lives in tables of its own, and every object is named `fgex_`.** The filter rules go in `fgex_filter` and the mangle ones in `fgex_mangle` (one of each per address family), with base chains `fgex_input`, `fgex_forward`, `fgex_output`, `fgex_prerouting` and `fgex_postrouting` handing over to `fgex_rules_in`, `fgex_rules_out` and `fgex_rules_fwd`. You can see the whole of it with:
```bash
sudo nft list ruleset | grep -A100 fgex_
```
Two things follow from Firegex owning those tables rather than writing into the ones `iptables` uses:
- **`iptables` keeps working.** Earlier versions put their chains into the tables named `filter` and `mangle`, which is where `iptables-nft` — the default `iptables` on most current distributions — keeps its own. `iptables` rejects a whole table as soon as it contains a rule it cannot express in its own format, and the connection-tracking rules Firegex writes for **allow_established** and **drop_invalid** are exactly that. The result was `iptables -L` and `iptables-save` answering ``table `filter' is incompatible, use 'nft' tool`` on a host where Docker, ufw, fail2ban or the organisers' own scripts were the ones asking. That cannot happen now: `iptables` never looks at a table Firegex owns.
- **Turning the firewall off no longer touches the host's own.** The default policy is a property of a base chain, and it used to be *your* `INPUT` chain that Firegex set to `drop` — and set back to `accept` on shutdown, silently undoing a default-deny an administrator had configured elsewhere. Firegex now sets the policy on its own base chain, so a reset is just deleting two tables.
**If something else removes those tables, Firegex puts them back within a few seconds.** `nft -f /etc/nftables.conf` starts with `flush ruleset` on Debian, and so does `systemctl restart nftables`; either one used to switch the Firegex firewall off with nothing to say so. The same watch covers the services' table: each running service is steered at its filters again without restarting, and its log records the gap as an error, since traffic was not filtered while it lasted. The lasting fix is still to make whatever manages your nftables leave the `fgex` tables alone. With `flush ruleset`, for example, you can delete only your own tables instead.
If you are upgrading from a version that used the shared tables, whatever it left behind is still there — Firegex does not go looking in tables it no longer owns. Clear it by hand once, and check the policy while you are there, since an older Firegex was the one driving it:
```bash
sudo nft list table ip filter # and ip6, and the mangle pair
```
A rule's **table** field (`filter` or `mangle`) still means what it always did — it's the hook the rule is evaluated at, not the name of the nftables table it ends up in.
Note that "accept" here means *Firegex* does not block the packet, not that nothing else will: other base chains at the same hook still get their say, exactly as they do between any two nftables tables. A `drop`, on the other hand, is final wherever it comes from.

323
docs/nfproxy.md
View File

@ -1,323 +0,0 @@
# 🌐 Netfilter Proxy
Netfilter Proxy (`nfproxy`) is a filtering module that leverages [nfqueue](https://netfilter.org/projects/libnetfilter_queue/) to intercept network packets, then runs your own Python filter code against them inside the C++ core via an embedded interpreter. Unlike [Netfilter Regex](nfregex.md), which only matches raw byte patterns, nfproxy gives filters structured access to the traffic (parsed HTTP requests/responses, TCP streams, per-stream history, ...) and lets them make arbitrary decisions in Python.
## How to use it
1. Create a service: pick a protocol (`tcp` or `http`), the target `ip:port` (or attach it to a [TLS Decrypt](tls.md) stream, to see decrypted traffic), and a name.
2. Upload one or more Python filter files (see [writing a filter](#writing-a-filter) below). Each file can define multiple `@pyfilter` functions.
3. Start the service. From then on, every matching packet is routed through nfqueue to the embedded Python interpreter, which runs your filters and returns an action (accept/drop/reject/mangle) back to the C++ core.
4. The service page shows live counters (packets seen, blocked/mangled) and any Python exception raised by a filter, reported back over a unix socket — no restart needed to see them.
Filters can be edited and re-uploaded while the service is running: changes take effect on the next packet. `fail_open` (an advanced per-service option) controls what happens if the filter process itself crashes or can't be reached: enabled, traffic is allowed through unfiltered rather than blocked, trading protection for availability — useful if you'd rather risk letting an attack through than take the service down during a competition.
## Writing a filter
Install the library and CLI:
```bash
pip install -U firegex
```
`fgex` is an alias package for `firegex`: installing either one gives you the same `firegex` module and the `fgex`/`firegex` CLI commands.
### The `pyfilter` decorator
```python
from firegex.nfproxy import pyfilter
```
`pyfilter` marks a function as an active filter. A filter function:
- must be decorated with `@pyfilter`;
- must have every parameter type-annotated with one of the data structures listed below (parameters without a type annotation make the filter invalid, and default/keyword-argument values are ignored — the caller always builds the arguments itself from the live packet data);
- must `return` one of the [packet statements](#packet-statements) below (or `None`, which behaves like `ACCEPT`).
```python
from firegex.nfproxy import pyfilter, ACCEPT, REJECT
@pyfilter
def none_filter(): # A filter that does nothing
return ACCEPT
```
Filter names (the function name) must be unique within a filter file.
Each TCP stream (i.e. each connection) gets its own isolated set of global variables: code at module level runs once per stream, and the same globals are reused across every packet of that stream. Don't store state in another module's globals — that memory is shared across every stream handled by the same thread and will cause data to leak/interfere between unrelated connections. Global variable names starting with `__firegex` are reserved for internal use.
```python
from firegex.nfproxy import pyfilter, ACCEPT, REJECT
from firegex.nfproxy.models import HttpRequest
@pyfilter
def filter_with_args(http_request: HttpRequest):
if http_request.body and b"ILLEGAL" in http_request.body:
return REJECT
```
Here the filter is only called once the data required to build an `HttpRequest` is available (i.e. once the HTTP headers have been parsed, and again once the body is complete). If a filter needs multiple parameters, it's only called once every parameter can be built from the data received so far.
### Packet statements
A filter must return one of these values (importable from `firegex.nfproxy`):
| Statement | Effect |
|---|---|
| `ACCEPT` | The packet is accepted and forwarded to the destination. This is also the default behavior if `None` is returned. |
| `REJECT` | The connection is closed and every packet in the stream is dropped. |
| `DROP` | This packet, and every subsequent packet in the stream, is silently dropped (unlike `REJECT`, this doesn't simulate a connection closure). |
| `UNSTABLE_MANGLE` | The packet is modified and forwarded. Only available when filtering `RawPacket` — see below. This is an unstable feature, use it carefully. |
### Data structures
These are the types you can use as filter parameter annotations. Some are only available when the service's protocol is `http` (marked **HTTP only**); the rest work for both `tcp` and `http` services.
#### `RawPacket`
```python
from firegex.nfproxy.models import RawPacket
```
The raw packet as received from nfqueue — the lowest level of abstraction, and the **only** data structure that can be mutated (for `UNSTABLE_MANGLE`).
- `data: bytes` — the packet's data, reassembled/reordered as a TCP stream (read-only).
- `is_input: bool` — `True` for an incoming packet, `False` for outgoing.
- `is_ipv6: bool` — `True` for IPv6, `False` for IPv4.
- `is_tcp: bool` — `True` for TCP, `False` for UDP.
- `l4_size: int` — size of the layer-4 payload (read-only).
- `raw_packet_header_len: int` — size of the original packet's header (read-only).
- `raw_packet: bytes` — the full packet, including the IP/TCP headers. Writable: assigning to it mangles the packet if the filter returns `UNSTABLE_MANGLE`, and updates `l4_size`/`l4_data` accordingly.
- `l4_data: bytes` — the layer-4 payload only, taken directly from the raw packet. Also writable for mangling; updates `l4_size`/`raw_packet` accordingly.
Be careful: `l4_data`/`raw_packet` reflect a single physical packet as seen by nfqueue, while `data` is the TCP stream already reassembled/reordered by the C++ core — they can legitimately differ (e.g. out-of-order packets are accepted by default without invoking Python at all).
#### `TCPInputStream` (alias `TCPClientStream`)
```python
from firegex.nfproxy.models import TCPInputStream
```
The assembled TCP stream in the input (client → server) direction. A filter using this type is only called for incoming packets.
- `data: bytes` — the entire input-direction stream assembled so far (read-only).
- `total_stream_size: int` — size of that stream (read-only).
- `is_ipv6: bool`
#### `TCPOutputStream` (alias `TCPServerStream`)
```python
from firegex.nfproxy.models import TCPOutputStream
```
Same as `TCPInputStream`, but for the output (server → client) direction; only called for outgoing packets.
#### `HttpRequest` — HTTP only
```python
from firegex.nfproxy.models import HttpRequest
```
The current HTTP request. This handler is called up to twice: once when the headers are complete, and again once the body is complete (if the whole request arrives in a single TCP packet, it's called only once).
- `method: bytes` — the request method.
- `url: str | None` — the request URL.
- `headers: dict[str, str]` — request headers, keys/values exactly as received (case-sensitive); a repeated header becomes a list of values.
- `get_header(header: str, default=None) -> str` — looks up a header case-insensitively; if the header was repeated, its values are joined with a comma (this method never returns a list).
- `user_agent: str`
- `content_encoding: str`
- `content_length: int | None`
- `body: bytes` — `None` until the body has arrived.
- `body_decoded` — the body decoded according to `content_encoding` (`gzip`, `br`, `deflate` and `zstd` are supported). `False` if decoding failed and `body` isn't `None`.
- `http_version: str`
- `keep_alive: bool`
- `should_upgrade: bool`
- `upgrading_to_h2: bool`
- `upgrading_to_ws: bool`
- `ws_stream: list[websockets.frames.Frame]` — decoded WebSocket frames (permessage-deflate supported); see the [websockets docs](https://websockets.readthedocs.io/en/stable/).
- `stream: bytes` — buffer of the raw WebSocket traffic in this direction; only meaningful once `should_upgrade` is `True`.
- `headers_complete: bool`
- `message_complete: bool`
- `total_size: int` — size of the whole request seen so far.
- `history: HttpHistory` — previously completed requests/responses on this same stream, see [`HttpHistory`](#httphistory-alias-httpstreamhistory--http-only) below.
#### `HttpRequestHeader` — HTTP only
```python
from firegex.nfproxy.models import HttpRequestHeader
```
Same fields as `HttpRequest`, but only called once, when the headers are complete — the body is never buffered and `body` is always `None`.
#### `HttpFullRequest` — HTTP only
```python
from firegex.nfproxy.models import HttpFullRequest
```
Same fields as `HttpRequest`, but only called once the whole request (headers + body) is complete. Completed instances of this type are also what gets stored in `HttpHistory.requests`.
#### `HttpResponse` — HTTP only
```python
from firegex.nfproxy.models import HttpResponse
```
The current HTTP response — same shape and calling convention as `HttpRequest` (up to twice: headers complete, then body complete), plus:
- `status_code: int`
All the other fields listed for `HttpRequest` (`headers`, `get_header`, `body`, `body_decoded`, `content_encoding`, `content_length`, `http_version`, `keep_alive`, `should_upgrade`, `upgrading_to_h2`, `upgrading_to_ws`, `ws_stream`, `stream`, `headers_complete`, `message_complete`, `total_size`, `history`, `user_agent`) apply here too.
#### `HttpResponseHeader` — HTTP only
```python
from firegex.nfproxy.models import HttpResponseHeader
```
Same as `HttpResponse`, but only called once the headers are complete; `body` is always `None`.
#### `HttpFullResponse` — HTTP only
```python
from firegex.nfproxy.models import HttpFullResponse
```
Same as `HttpResponse`, but only called once the whole response is complete. Completed instances of this type are what gets stored in `HttpHistory.responses`.
#### `HttpHistory` (alias `HttpStreamHistory`) — HTTP only
```python
from firegex.nfproxy.models import HttpHistory
```
Gives a filter access to previously *completed* requests/responses on the same TCP stream — useful for correlating a response with the request(s) that came before it on a keep-alive connection, or for stateful logic that spans more than one exchange.
- `requests: list[HttpFullRequest]` — a snapshot copy of the requests completed so far on this stream (does not include the request currently being processed).
- `responses: list[HttpFullResponse]` — a snapshot copy of the completed responses so far.
You can use it in two ways:
```python
from firegex.nfproxy import pyfilter, REJECT
from firegex.nfproxy.models import HttpHistory, HttpResponse
# 1. as its own filter parameter
@pyfilter
def check_history(hist: HttpHistory):
if len(hist.requests) > 50:
return REJECT
# 2. via the .history property available on any Http* instance
@pyfilter
def check_previous_requests(resp: HttpResponse):
if any(b"admin" in req.url.encode() for req in resp.history.requests if req.url):
return REJECT
```
The number of entries kept per stream is capped by the `FGEX_MAX_HISTORY_SIZE` global (default `100`) — see [Other global options](#other-global-options) below; once the cap is reached, the oldest entry is dropped as a new one is added.
## Stream limiter
What happens if a single TCP stream carries a lot of data? Past a configurable size, a "full stream" action kicks in. First import the enum:
```python
from firegex.nfproxy import FullStreamAction
```
Then set these in the filter file's globals:
- `FGEX_STREAM_MAX_SIZE: int` — maximum size (in bytes) of a stream before `FGEX_FULL_STREAM_ACTION` is triggered. This limit applies per data structure independently: e.g. if `TCPInputStream` has reached the limit but `HttpResponse` hasn't, the action only affects `TCPInputStream`. Default: 1MB.
- `FGEX_FULL_STREAM_ACTION: FullStreamAction` — the action taken once the limit is exceeded. Default: `FullStreamAction.FLUSH`.
`FullStreamAction` values:
| Value | Effect |
|---|---|
| `FLUSH` | Flush the stream and keep acquiring new packets (default). |
| `DROP` | Drop the next stream packets, like a `DROP` filter statement. |
| `REJECT` | Reject the stream and close the connection, like a `REJECT` filter statement. |
| `ACCEPT` | Stop calling pyfilters and accept the rest of the traffic as-is. |
## Other global options
```python
from firegex.nfproxy import ExceptionAction
```
- `FGEX_INVALID_ENCODING_ACTION: ExceptionAction` — action taken when parsing hits an invalid/unsupported encoding (a parser-level failure). Default: `ExceptionAction.REJECT`. Values: `ACCEPT` (accept the packet that caused the error), `DROP` (drop the connection), `REJECT` (reject the connection), `NOACTION` (do nothing — the error is signaled and the stream is accepted without calling any more pyfilters on it).
- `FGEX_MAX_HISTORY_SIZE: int` — max number of requests/responses kept per stream by [`HttpHistory`](#httphistory-alias-httpstreamhistory--http-only). Default: `100`.
## Testing a filter locally
You don't need a running Firegex instance to try out a filter: the `firegex` pip package ships a local proxy simulator.
```text
➤ fgex nfproxy -h
Usage: fgex nfproxy [OPTIONS] FILTER_FILE ADDRESS PORT
Run an nfproxy simulation
╭─ Arguments ─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│ * filter_file TEXT The path to the filter file [default: None] [required] │
│ * address TEXT The address of the target to proxy [default: None] [required] │
│ * port INTEGER The port of the target to proxy [default: None] [required] │
╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
╭─ Options ───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│ --proto [tcp|http] The protocol to proxy [default: tcp] │
│ --from-address TEXT The address of the local server [default: None] │
│ --from-port INTEGER The port of the local server [default: 7474] │
│ -6 Use IPv6 for the connection │
│ --help -h Show this message and exit. │
╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
```
This runs a local proxy in front of `ADDRESS:PORT`, applying `FILTER_FILE` to the traffic — no Docker, no nftables/root privileges required. The filter file is reloaded automatically whenever it changes on disk, so you don't need to restart the simulator while iterating.
```bash
fgex nfproxy test_http.py 127.0.0.1 8080 --proto http
```
## Full example
```python
# Example filter file for an http-protocol service
from firegex.nfproxy.models import RawPacket, HttpRequest, HttpHistory
from firegex.nfproxy import pyfilter, ACCEPT, REJECT, DROP, UNSTABLE_MANGLE, FullStreamAction
# Lowest level of abstraction: only RawPacket can be mangled.
@pyfilter
def mangle_example(packet: RawPacket):
if b"TEST_MANGLING" in packet.l4_data:
packet.l4_data = packet.l4_data.replace(b"TEST", b"UNSTABLE")
return UNSTABLE_MANGLE
if b"BAD DATA" in packet.data:
return DROP
return ACCEPT
# Higher level of abstraction: parsed HTTP requests.
@pyfilter
def http_filter(http: HttpRequest):
if http.method == b"GET" and http.url and "test" in http.url:
return REJECT
# Using history to see previous requests on the same keep-alive stream.
@pyfilter
def repeated_probe_filter(hist: HttpHistory):
if len(hist.requests) > 20:
return REJECT
# Stream size limits (applies per data structure, e.g. only to RawPacket streams above)
FGEX_STREAM_MAX_SIZE = 4096
FGEX_FULL_STREAM_ACTION = FullStreamAction.REJECT
```
## How it works
The proxy is built on a multi-threaded architecture that embeds Python for dynamic filtering:
- **Packet interception**: the [nfqueue](https://netfilter.org/projects/libnetfilter_queue/) kernel module (part of [netfilter](https://netfilter.org/)) intercepts network packets; the rules attaching nfqueue to traffic are generated with the nftables JSON API by the Python manager.
- **Packet reading**: a dedicated thread reads packets from nfqueue.
- **Multi-threaded analysis**: the C++ binary launches multiple threads, each with its own Python interpreter — Python 3.12's [per-interpreter GIL](https://peps.python.org/pep-0684/) makes this real multithreading. Traffic is distributed across threads by hashing IP/port, so all packets of the same flow are handled by the same thread.
- **Python filter integration**: uploaded filters run inside these interpreters.
- **HTTP parsing**: [a Python wrapper for llhttp](https://github.com/domysh/pyllhttp) (forked/adapted to work across multiple interpreters) parses HTTP traffic.

35
docs/nfregex.md
View File

@ -1,35 +0,0 @@
# 📡 Netfilter Regex
Netfilter Regex (`nfregex`) filters network traffic by matching regular expressions directly against packet content, entirely inside a C++ binary (no embedded interpreter, no user-supplied code). It's the fastest filtering module Firegex offers, at the cost of flexibility: if you need custom logic (parsing structured data, stateful decisions, mangling packets), use [Netfilter Proxy](nfproxy.md) instead.
## How to use it
1. Create a service: pick a protocol (`tcp` or `udp`), the target `ip:port` (or attach it to a [TLS Decrypt](tls.md) stream, to see decrypted traffic, `tcp` only), and a name.
2. Add one or more regexes to the service.
3. Start the service. Matching traffic is dropped/rejected according to each regex's direction; everything else passes through unmodified — nfregex has no mangling capability (only [Netfilter Proxy](nfproxy.md) can mangle traffic).
Each regex has:
- **Pattern**: a PCRE2-compatible regular expression, matched as raw bytes (not text) against the traffic.
- **Direction** (`mode`): `S` — match only client → server traffic (to the server); `C` — match only server → client traffic (to the client); `B` — match both directions.
- **Case sensitivity**: matching can be case-sensitive or case-insensitive per regex.
- **Active/inactive**: a regex can be disabled without deleting it — inactive regexes are kept (and still shown with their stats) but never evaluated against traffic.
The service page shows, per regex, how many packets it has blocked so far, and whether it's currently active.
`fail_open` (an advanced per-service option, shared with nfproxy) controls what happens if the underlying nfqueue/binary can't be reached: if enabled, traffic is allowed through unfiltered rather than blocked.
### TCP vs UDP matching
- **TCP**: nfregex reassembles and reorders the stream before matching, so a regex can match a pattern that's split across multiple packets. Out-of-order packets are handled transparently.
- **UDP**: there's no stream to reassemble — each datagram is matched independently, and only the match context (not the full payload) is retained afterwards.
## How it works
The packet filtering process is implemented in C++ and involves several key steps:
- **Packet interception**: the [nfqueue](https://netfilter.org/projects/libnetfilter_queue/) kernel module intercepts network packets (a [netfilter](https://netfilter.org/) module) 🔍. The rules attaching nfqueue to the traffic are generated via the nftables JSON API by the Python manager.
- **Packet reading**: a dedicated thread reads packets from nfqueue. 🧵
- **Packet parsing**: intercepted packets are parsed by [libtins](https://libtins.github.io/), a C++ library that extracts the payload from each packet. 📄
- **Multi-threaded analysis**: multiple threads analyze packets concurrently. While the nfqueue module balances load based solely on IP addresses — resulting in a single thread handling all traffic in NAT environments like CTF networks — Firegex manages threads at the user level differently: traffic is routed based on IP addresses combined with port hashing, giving a more balanced workload while guaranteeing that a given flow is always analyzed by the same thread. ⚡️
- **TCP handling**: for TCP connections, libtins uses a TCP follower to reorder packets received from the kernel. 📈
- **Regex matching**: the extracted payload is matched using [vectorscan](https://github.com/VectorCamp/vectorscan) — a fork of [hyperscan](https://github.com/intel/hyperscan) that also runs on arm64. 🎯

31
docs/porthijack.md
View File

@ -1,31 +0,0 @@
# ⚡️ Hijack Port to Proxy
Hijack Port to Proxy (`porthijack`) redirects traffic arriving on a public `ip:port` to a proxy you run yourself on `localhost`, without touching the target service's own configuration. Your proxy can then reach the real service on loopback (`127.0.0.1`/`::1`) and do whatever it wants with the traffic — including things Firegex's other modules can't do, like mangling packets (something [Netfilter Proxy](nfproxy.md) can't fully guarantee, since altering TCP payload sizes can desync the stream, while a userspace proxy fully re-terminates the connection and can freely resize packets).
## How to use it
1. Write and run your own proxy, listening on some local port, forwarding to the real service on loopback.
2. Create a service in the UI: the public `ip:port` clients connect to, the protocol (`tcp` or `udp`), and the local port your proxy listens on.
3. Start the service. Public traffic to that `ip:port` is now transparently redirected to your proxy; your proxy's own traffic to the real service (over loopback) is left untouched.
To the original client, responses still appear to come from the original public `ip:port` — the client never sees your proxy's address. To the real service (if it's configured to only accept loopback connections, or logs source IPs), requests appear to come from your proxy, not from the original client.
### Using the nfproxy simulator as your proxy
You don't have to write a proxy from scratch: the `firegex` pip package's `fgex nfproxy` simulator (see the [Netfilter Proxy guide](nfproxy.md)) is a ready-made local proxy that applies a Python filter file to the traffic, and can double as the proxy this module redirects to — this is a convenient way to get nfproxy-style filtering (including mangling, since the simulator fully re-terminates the connection) for a service, driven entirely by port hijacking instead of nfqueue.
```bash
pip install -U firegex
fgex nfproxy test_http.py 127.0.0.1 8080 --proto http --from-port 13377
```
Then create a Hijack Port to Proxy service pointing at local port `13377`. Note: in the simulator, `RawPacket.raw_packet` always equals `RawPacket.l4_data` (there's no real IP/TCP header to mangle, since the simulator re-terminates the connection at the application layer) — see the [nfproxy guide](nfproxy.md) for more on `RawPacket`.
## How it works
This module works by rewriting packet fields directly with [nftables](https://netfilter.org/) rules — there's no connection tracking/NAT involved, just field mangling on matching packets:
- **Inbound** (prerouting): for packets destined to the public `ip:port`, the destination address and port are rewritten in place to loopback and your proxy's port.
- **Outbound** (postrouting): for packets sourced from your proxy's port (destined back to the original client), the source address and port are rewritten back to the public `ip:port`, so the client sees replies coming from where it expects.
This mangling only applies to packets actually matching the public `ip:port`/protocol you configured — traffic to loopback that isn't part of a hijacked flow is left alone.

Some files were not shown because too many files have changed in this diff Show More