ad/firegex
1
0
Fork 0
mirror of https://github.com/Av0idD4rk/firegex.git synced 2026-10-23 16:31:00 +00:00
Code Issues Packages Projects Releases Wiki Activity

improved addr/int selection UI + change llhttp managment

Browse Source
This commit is contained in:
Domingo Dirutigliano 2026-09-13 15:54:56 +02:00
parent 9d2a4dc4bf
commit e491a14899
No known key found for this signature in database
147 changed files with 18991 additions and 3332 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

70
.github/workflows/release.yml vendored
View File

@ -24,8 +24,11 @@ jobs:
- name: Build and run firegex
run: python3 run.py start -P testpassword
# No apt-get: the tests' dependencies are pure python. `iperf3`, matplotlib and
# numpy belong to the benchmarks, which CI does not run — a measurement has no
# pass or fail to gate a release on.
- name: Run tests
run: sudo apt-get install -y iperf3 && cd tests && ./run_tests.sh
run: cd tests && ./run_tests.sh
docker_build:
needs: run_tests
@ -217,8 +220,59 @@ jobs:
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
pypi_publish_firegex:
# `firegex` carries a C extension — the llhttp binding in `fgex-lib/llhttp/` — so it
# is no longer one universal wheel. A wheel is built per platform and per interpreter
# here, and the sdist goes up beside them: where no wheel matches, pip compiles from
# that. Compiling is a thing pip knows how to do; assuming a system library is already
# installed is not, which is why the parser ships as source rather than as a `ctypes`
# binding against the distribution's `libllhttp`.
build_firegex_wheels:
needs: run_tests
runs-on: ${{ matrix.os }}
strategy:
fail-fast: false
matrix:
include:
- os: ubuntu-latest # manylinux + musllinux, x86_64
- os: ubuntu-24.04-arm # and the same for aarch64
- os: macos-latest # arm64
- os: macos-13 # x86_64, while GitHub still offers it
- os: windows-latest
steps:
- uses: actions/checkout@v4
- name: Extract tag name
id: tag
shell: bash
run: echo TAG_NAME=$(echo $GITHUB_REF | cut -d / -f 3) >> $GITHUB_OUTPUT
- name: Update version
shell: bash
run: >-
sed -i.bak "s/{{VERSION_PLACEHOLDER}}/${{ steps.tag.outputs.TAG_NAME }}/g" fgex-lib/setup.py;
sed -i.bak "s/{{VERSION_PLACEHOLDER}}/${{ steps.tag.outputs.TAG_NAME }}/g" fgex-lib/firegex/__init__.py;
- name: Build wheels
uses: pypa/cibuildwheel@v2.21.3
with:
package-dir: fgex-lib
output-dir: wheelhouse
env:
# The library declares `python_requires>=3.10`, so build what it claims to
# support and nothing else. PyPy is excluded: the datapath embeds CPython.
CIBW_BUILD: "cp310-* cp311-* cp312-* cp313-* cp314-*"
CIBW_SKIP: "*-musllinux_i686 *_i686 pp*"
# A wheel that imports is the only kind worth publishing, and the extension is
# the whole reason this job exists. The checks live in a file rather than in a
# `-c` string because the one that matters needs a `try`: the parser has to
# raise its *own* exception on malformed traffic, and this clean environment —
# no `pyllhttp`, no developer's leftovers — is the only place that is tested.
CIBW_TEST_COMMAND: python {package}/wheel_smoke.py
- uses: actions/upload-artifact@v4
with:
name: wheels-${{ matrix.os }}
path: wheelhouse/*.whl
pypi_publish_firegex:
needs: build_firegex_wheels
runs-on: ubuntu-latest
permissions:
contents: read
@ -240,8 +294,16 @@ jobs:
run: >-
sed -i "s/{{VERSION_PLACEHOLDER}}/${{ steps.tag.outputs.TAG_NAME }}/g" fgex-lib/setup.py;
sed -i "s/{{VERSION_PLACEHOLDER}}/${{ steps.tag.outputs.TAG_NAME }}/g" fgex-lib/firegex/__init__.py;
- name: Build package
run: cd fgex-lib && python -m build && mv ./dist ../
# The sdist is what pip falls back to where no wheel matches, so it has to carry
# the parser's C sources — `MANIFEST.in` is what puts them in.
- name: Build the sdist
run: cd fgex-lib && python -m build --sdist && mv ./dist ../
- name: Collect the wheels built for every platform
uses: actions/download-artifact@v4
with:
pattern: wheels-*
merge-multiple: true
path: dist
- name: Publish package
uses: pypa/gh-action-pypi-publish@27b31702a0e7fc50959f5ad993c78deac1bdfc29
with:

9
.gitignore vendored
View File

@ -55,4 +55,11 @@
# Rust proxy engine build artifacts
/backend/proxysrc/target/
# Copied from the repository root by fgex-lib/setup.py at build time.
# pytest
**/.pytest_cache/
# The vendored HTTP parser, built in place by an editable install.
/fgex-lib/firegex/_llhttp*.so
/fgex-lib/firegex/_llhttp*.pyd
/fgex-lib/build/

74
CLAUDE.md
View File

@ -38,17 +38,39 @@ cd backend && DEBUG=1 python3 app.py DEBUG
```
### Tests (`tests/`)
No pytest — plain scripts driven against a **live, already-running** firegex instance (`http://127.0.0.1:4444` by default), root/Linux required.
**One suite, one runner: `pytest`.** Everything drives a **live, already-running** firegex (`http://127.0.0.1:4444` by default), root/Linux, except `unit/` which needs nothing running at all.
```bash
cd tests
./run_tests.sh [password] # full suite (defaults to "testpassword")
python3 regex_lib_test.py # firegex.regex — needs libhs, not an instance
python3 services_test.py -p <pw> -t proxy|nfqueue|external [--ipv6] [--tls]
python3 api_test.py -p <pw>
python3 benchmark.py -p <pw> -r <n regexes> [-t proxy|nfqueue]
python3 ipfilter_test.py # access control
./run_tests.sh # installs deps, waits for the instance, runs pytest
./run_tests.sh --set-pass # against one still in its initial-setup state
pytest unit # needs no instance
pytest -k udp # one subject
pytest --layer proxy --no-ipv6 -x # narrow, stop at the first failure
pytest standalone # restarts the instance; opt-in, see below
```
`services_test.py` is the important one: the same filters are exercised on both network layers, because a filter that only works on one of them is exactly the failure the unified model exists to prevent. It also covers the parts that are easy to get subtly wrong — a second address added to a *running* service, a filter whose protocol is detected from its code, the bucketed statistics, and UDP being refused on the proxy layer with a reason. Run it twice in a row when changing anything about addresses or TLS: state left behind by the first pass only shows up on the second.
| Directory | Needs an instance | What it holds |
|---|---|---|
| `unit/` | no | `firegex.regex` against libhs, the `firegex.pyfilters` models, the address→nftables translation |
| `integration/` | yes | the product, parametrised over the combinations |
| `standalone/` | yes, and restarts it | settings read once at process startup |
| `helpers/` | — | the API client, stand-in services, certificates, the traffic `Channel` |
| `bench/` | yes | measurement, not testing; `pytest` does not collect it |
**The combinations are parameters, not a shell loop.** `integration/conftest.py` holds `Layer(transport, ipv6, tls)` and the fixtures that fan out over it — `inspecting_layer`, `filtering_layer`, `proxy_layer`, `tls_layer`, `external_layer`, `any_layer`. Running both network layers with the same filters is the point: a filter that only works on one of them is exactly the failure the unified model exists to prevent. This was a shell script invoking one 1100-line program eight times, which meant the first failure ended the run and the other seven combinations were never reached, and a host without IPv6 on loopback failed rather than skipped.
Three things that have to stay true of this layout:
- **`helpers/`, never `utils/`.** The backend's own package is called `utils`, and this directory's used to be too — so whichever imported first won, and `unit/test_nftables_addresses.py` (which imports the backend) and everything importing `helpers.firegexapi` could not be collected in one pytest process. That name collision is why the suite *could not* be unified before.
- **The interception canary.** It is perfectly possible to reach the API of an instance whose datapath protects a different machine — a container without host networking, a VM with the port forwarded. Every call returns `ok` and not one packet goes near any of it, which without a check reads as dozens of "the filter did not block" failures rather than as a suite pointed somewhere it cannot reach. `datapath_in_our_path` asks once, plainly, and everything needing interception is skipped with the reason.
- **Skip with a reason rather than fail.** A laptop with no IPv6 loopback, a non-root run, an instance elsewhere — none of those are firegex bugs, and `-ra` is in `addopts` so the summary says which half of the suite the machine declined.
`Channel` (`helpers/traffic.py`) is the one place that knows whether TLS is in the way. The old suite branched on it at every traffic check, about twenty times.
Run the suite **twice in a row** when changing anything about addresses or TLS: state left behind by the first pass only shows up on the second.
`standalone/` is opt-in because it drives `run.py stop`/`start` itself — the IP allowlist and the boot-time auth mode are read once at startup, so there is no way to test them against a running instance. It restores an unrestricted one at the end, which every other module depends on.
The Rust datapath has its own suite, which needs no running instance but does need the production toolchain (vectorscan and Python 3.14 — the filter library imports `compression.zstd`):
```bash
@ -57,11 +79,9 @@ cd backend/proxysrc && cargo test # on a host with libhs +
cargo test --release --test throughput -- --ignored --nocapture # the matcher benchmark
```
The suite runs against an instance with authentication turned off as well: `login()` reads a 403 there as "there is nothing to log in to" rather than as a failure, since every request is already allowed. `api_test.py` is the exception and says so — half of it is about passwords, and those endpoints answer 403 in that mode by design. Note `BearerSession.post` copies the header dict rather than writing into it: the form `Content-Type` belongs to the login request alone, and it used to stick, which went unnoticed only because a successful login replaces the whole dict a moment later.
The suite runs against an instance with authentication turned off as well: `login()` reads a 403 there as "there is nothing to log in to" rather than as a failure, since every request is already allowed. `integration/test_auth.py` is the exception and skips itself — it is about passwords, and those endpoints answer 403 in that mode by design. Note `BearerSession.post` copies the header dict rather than writing into it: the form `Content-Type` belongs to the login request alone, and it used to stick, which went unnoticed only because a successful login replaces the whole dict a moment later.
`ipfilter_test.py` is the odd one out: those settings are only read at process startup, so it drives `run.py stop`/`start` itself for each scenario and restores an unrestricted instance at the end. It is **not** part of `run_tests.sh` — run it on its own, and expect it to bounce the instance.
CI (`.github/workflows/release.yml`) runs **only on `release: published`**, not on pushes or pull requests — nothing validates a PR automatically. Its `run_tests` job (`ubuntu-latest` + `ubuntu-24.04-arm`) does `python3 run.py start -P testpassword` then `cd tests && ./run_tests.sh`, and gates every publish job below it.
CI (`.github/workflows/release.yml`) runs **only on `release: published`**, not on pushes or pull requests — nothing validates a PR automatically. Its `run_tests` job (`ubuntu-latest` + `ubuntu-24.04-arm`) does `python3 run.py start -P testpassword` then `cd tests && ./run_tests.sh`, and gates every publish job below it. The tests' dependencies are pure python; `iperf3`, matplotlib and numpy belong to `bench/requirements.txt` and CI installs none of them.
## Architecture
@ -69,20 +89,22 @@ CI (`.github/workflows/release.yml`) runs **only on `release: published`**, not
React (Vite + Mantine + react-router + TanStack Query + socket.io-client) → FastAPI backend → per-transport datapath binaries that do the actual interception. The backend mostly orchestrates: SQLite-backed config, spawning/talking to those processes, and deploying the built React app.
### The layers are compared in the interface, not only in the docs
`components/Services/LayerChoice.tsx` puts the trade in front of the operator where the choice is made: a table of what each layer costs, plus the paragraph for the one selected. `transportSummary()` carries the same sentence onto every service row and detail page, and `losesClientAddress()` marks the one combination that gives something up. Keep them in step with `docs/services.md` — the point is that an operator never has to read documentation to find out that one layer rewrites exactly and the other has a kernel-guaranteed fail-open.
`components/Services/LayerChoice.tsx` puts the trade in front of the operator where the choice is made: a table of what each layer costs, plus the paragraph for the one selected. `transportSummary()` carries the same sentence onto every service row and detail page. Keep them in step with `docs/services.md` — the point is that an operator never has to read documentation to find out that one layer reassembles in the kernel and the other has a kernel-guaranteed fail-open. There used to be a `losesClientAddress()` beside them, marking the one combination that gave the client's address up; source transparency is unconditional on every layer now, including UDP, so it answered `false` for every input and the badge it gated could not render. Do not reintroduce it without a combination that actually loses something.
### The two layers (`backend/modules/services/`)
The central idea, and the thing to preserve when changing anything here: **how traffic is intercepted and what is done with it are independent choices.**
- **Network layer** (`transports.py`): `nfqueue` — packets queued to userspace, verdict handed back, nothing terminated, kernel fail-open via `NFQA_CFG_F_FAIL_OPEN` plus `bypass` on the nft rule. `proxy` — the connection is terminated and reopened, which buys exact rewriting, kernel reassembly, backpressure and an ordered chain, and costs the kernel backstop (rebuilt by hand in `proxysrc/src/filter.rs`).
**The proxy is not the slow one, and the docs used to say it was.** Measured: 2.2× NFQUEUE's bulk throughput at one thread and 4.7× at eight, because NFQUEUE pays a userspace round trip *per packet* while the proxy pays once per connection and then the kernel moves the bytes. `NTHREADS` reaches the Rust engine too since 5.0.0 — `main.rs` builds its tokio runtime with that many workers rather than `#[tokio::main]` taking every core, so `run.py --threads` means one thing on both layers instead of one thing on one and nothing on the other. On short connections the two are indistinguishable — that claim cost three contradictory measurements before `tests/connbench.py` started interleaving the layers within each pass instead of measuring one and then the other, which was handing the machine's drift to whichever went second. The trade between the layers is fail-open and transparency, not speed; keep `docs/services.md`, `LayerChoice.tsx` and `transportSummary()` saying that. `external` — a stateless destination rewrite that puts a proxy the operator wrote in the path (this is what `porthijack` used to be); no firegex process is involved, so `check()` refuses any filter, since a filter attached there would be one that never runs.
**The proxy is not the slow one, and the docs used to say it was.** Measured: 2.2× NFQUEUE's bulk throughput at one thread and 4.7× at eight, because NFQUEUE pays a userspace round trip *per packet* while the proxy pays once per connection and then the kernel moves the bytes. `NTHREADS` reaches the Rust engine too since 5.0.0 — `main.rs` builds its tokio runtime with that many workers rather than `#[tokio::main]` taking every core, so `run.py --threads` means one thing on both layers instead of one thing on one and nothing on the other. On short connections the two are indistinguishable — that claim cost three contradictory measurements before `tests/bench/connbench.py` started interleaving the layers within each pass instead of measuring one and then the other, which was handing the machine's drift to whichever went second. The trade between the layers is fail-open and transparency, not speed; keep `docs/services.md`, `LayerChoice.tsx` and `transportSummary()` saying that. `external` — a stateless destination rewrite that puts a proxy the operator wrote in the path (this is what `porthijack` used to be); no firegex process is involved, so `check()` refuses any filter, since a filter attached there would be one that never runs.
- **Filter layer**: `regex` (patterns compiled into one hyperscan database) or `pyfilter` (the user's Python). A service holds an ordered list of them.
`Transport.check()` is where an impossible combination is refused **before** anything starts: more than `MAX_CHAIN_POSITIONS` filters on `nfqueue`, any filter at all on `external`, and the `tls` protocol anywhere but the proxy layer. It is asked twice: at creation with an empty chain, for what the service alone makes impossible, and again whenever a chain is pushed. UDP was never removed from the datapath: `cppregex` runs in block mode for it and `cpproxy` handles it, and the only thing that had gone missing was the choice in the UI.
**UDP on the proxy layer (`proxysrc/src/udp.rs`).** The TCP path recovers where a connection was headed with `SO_ORIGINAL_DST`, which the kernel implements **for TCP and SCTP only** (`ENOPROTOOPT` otherwise). So UDP is relayed with **one socket per protected address**, each with its upstream fixed — nothing has to be recovered because nothing was lost. The engine prints one `UDP <upstream> <port>` line per relay after `PORT`, and `_add_proxy` points each address's rule at its own port.
**One spelling of an upstream, and one owner of the map.** `udp_relay_key()` in `nftables.py` is the only place the `host:port` / `[host]:port` token is built, and `udp_relay_host()` the only place an interface name is resolved to something a relay can bind — IPv4 first and link-local last, because an interface almost always carries an `fe80::` address it was never configured with and binding there puts the service where nobody dials. Both were written out by hand in four and three places respectively, and the engine's map, the rule that points at a relay and the manager adding an address to a running service all have to agree on the token exactly; when they stop, the symptom is a service whose UDP silently goes nowhere. For the same reason `ProxyTransport.udp_ports` is the *only* map of relays: `start()` returns that dict as `_steer["udp_ports"]`, so it is the same object the rules read, and `ServiceManager.address_added` asks `add_udp_target()` for a relay rather than keeping a second copy of the answer beside it.
Client address transparency is preserved via `transparent::connect_as_udp` using `IP_TRANSPARENT` with mark `0x1339` and policy routing `table 1339`. New UDP addresses can be added to running services dynamically via `ADD_UDP` over the control channel without restarting the proxy.
What UDP keeps: per-flow filter state keyed by client address and released after `IDLE` — UDP has no close to observe, so a timeout is the only thing that ends a flow — and both directions inspected. `Verdict::Reject` drops the datagram rather than closing anything, because there is nothing to close. Replies leave through the **listener** socket so conntrack rewrites them to appear from the address the client dialled; sending from the upstream socket would reach a client not expecting that source. The relay's own dial carries `SELF_MARK`, or the redirect rule would catch it and loop.
@ -98,7 +120,7 @@ Consequences worth keeping:
- **`proto` is denormalised onto each address** and rewritten when the service's changes. That is what makes `(ip, port, proto)` a usable uniqueness key, so a TCP service and a UDP one can share an address exactly as the kernel allows.
- **The `external` proxy endpoint belongs to the address, not the service**, and a partial unique index forbids two addresses sharing one: the return rule recognises the operator's proxy by address and port to put the original port back, so two addresses behind one endpoint could not be told apart on the way out.
- **The proxy listener is dual-stack whenever the host supports IPv6** (`[::]:0`), and `transparent::unmap` turns the v4-mapped peer and local addresses back into IPv4 before anything else in the engine touches them — conntrack, the source-preserving dial and the metadata a filter reads all want the address the packet actually had.
- **Addresses accept network interface names as well as IPs.** An entry in `service_addresses` can be an IP/CIDR or a network interface name (1-15 chars, e.g. `eth0`, `wg0`, `tun0`, `lo`), checked via `utils.is_interface_name` and marked by `Address.is_interface`. Nftables matches use `meta iifname <iface>` for inbound/prerouting and `meta oifname <iface>` for outbound/postrouting/route. On `proxy` (TCP/TLS), redirect rules match `iifname` and `route_chain` marks return packets with `0x1339` on `oifname`, skipping `nat_output_chain` to avoid hijacking unrelated outbound host traffic. On UDP `proxy`, the primary IP assigned to the interface is resolved dynamically via `psutil.net_if_addrs()` to bind the relay. On `nfqueue`, interface traffic is queued natively in prerouting/postrouting without needing an IP. On `external`, interfaces are disallowed because return rewrites require a concrete IP.
- **Addresses accept network interface names as well as IPs.** An entry in `service_addresses` can be an IP/CIDR or a network interface name (1-15 chars, e.g. `eth0`, `wg0`, `tun0`, `lo`), checked via `utils.is_interface_name` and marked by `Address.is_interface`. The interface is not a second spelling of an address, and the interface is what the operator is choosing when the address is somebody else's to hand out. The frontend says which of the two it understood *while it is being typed* — `addressKind` in `components/InterfaceInput.tsx` puts it on a badge in the field, because `eth-0` is a perfectly good interface name and a perfectly wrong address, and the two are matched by completely different rules. That rule is written once on the frontend too, as `isInterfaceName`/`isIpAddress` in `js/utils.tsx`, reading the same charset the backend does — it used to be pasted into `AddEditService.tsx` and `AddressList.tsx`, two copies of a rule the picker beside them did not consult at all, which is how the hand-off layer came to offer interface names in the list and refuse them on submit. `InterfaceInput` lists interfaces and addresses as two groups, each interface carrying what it holds right now (one question decides between them: "and what is on it"), and stays a combobox rather than a select because an address a service will answer on need not exist on this host yet. **Link-local is not offered** — every interface carries an `fe80::` it was never configured with, which doubled the list with addresses that would put a service where no client looks for it, the same reason `udp_relay_host()` takes them last; one typed by hand is still accepted, and an interface holding nothing else says "link-local only" rather than listing it. A row is **the badge as a column, and everything else in the one beside it** — value, then the note, wrapping inside that column and never under the badge. Neither of those two is given a fixed share, because which one is long changes with the group (the addresses a link carries, or the address itself): fixing it took the whole width from a full-length IPv6 and left the other wrapping a character per line, a row a screen tall beside an address cut off mid-digit; and three items in one wrapping row put that same address at the container's left edge, under the badge, three lines tall. The list's own width is **computed from the longest value in it**, between a floor and a ceiling, and three simpler rules were wrong first: inherited from the content it collapsed to the field's width, because content allowed to wrap asks for less than it needs to stay on one line; pinned to the widest row any host could produce, every list was that wide beside fields half the size; and a floor alone put the one long address back over three lines. A host answering on `10.60.3.1` gets the floor, one carrying a full-length IPv6 gets the room to show it with its note beside it, and monospace is what makes the arithmetic honest. **`AddressList` reads in the same words it was chosen in** — the same badge at the same `BADGE_WIDTH`, then the value, then `on eth0` for an address and `now 10.10.0.3` for an interface — because a row an operator comes back to check, described differently from the list it came out of, is a reason to wonder whether it is the same thing. What is *shown* is `bareAddress`: the backend stores what it parsed, so `127.0.0.1` comes back as `127.0.0.1/32`, a spelling nobody types that reads like a range beside real ones; a host-length prefix is dropped for display and a real one is kept, which is also what tells `addressKind` apart. A real range survives an edit for the same reason — the modal seeded itself by cutting at the first slash, so opening "move this address" on a `/16` and saving narrowed a service to one host of it. Where interfaces are not an answer — the `external` proxy endpoint, and the service address on that layer — they are not listed at all and a name typed there is badged red, rather than offered and then refused on submit. Nftables matches use `meta iifname <iface>` for inbound/prerouting and `meta oifname <iface>` for outbound/postrouting/route. On `proxy` (TCP/TLS), redirect rules match `iifname` and `route_chain` marks return packets with `0x1339` on `oifname`, skipping `nat_output_chain` to avoid hijacking unrelated outbound host traffic. On UDP `proxy`, the primary IP assigned to the interface is resolved dynamically via `psutil.net_if_addrs()` to bind the relay. On `nfqueue`, interface traffic is queued natively in prerouting/postrouting without needing an IP. On `external`, interfaces are disallowed because return rewrites require a concrete IP.
**An ordered chain works on every layer, and gets there differently.** The proxy walks a list inside one process. NFQUEUE cannot — its binaries fuse transport and filter — so there a chain is a chain of *processes*: one binary per filter, each with its own queue and its own pair of base chains, at `-307 + position` (prerouting) and `107 + position` (postrouting). A packet a filter accepts carries on to the next base chain in the same hook, so **filter order is base-chain priority order**. This was established by experiment, not assumed: two `cppregex` processes at adjacent priorities, where the second refuses traffic the first passed. The cost is a process and a reassembly pass per filter, which is why `MAX_CHAIN_POSITIONS` exists at all.
@ -107,8 +129,9 @@ Changing the *shape* of an NFQUEUE chain (adding, removing, reordering) cannot b
### One engine per concern, never two
Two implementations of the same concept is the mistake this module was restructured to remove, and it is worth restating because it keeps being tempting:
- **Regex matching is hyperscan, everywhere — there is no second regex engine in the tree.** `binsrc/regex/` on the NFQUEUE side, `proxysrc/src/hyperscan.rs` on the proxy side, a binding rather than a reimplementation. Rewriting uses it too: block-mode scanning gives start-of-match offsets, and the matched span is replaced with a literal. That is why the replacement has no capture-group references — hyperscan reports where a match is, not what its groups captured, and inventing them would mean a second engine deciding them. Adding `regex` back to `Cargo.toml` would undo this.
- **A pattern is validated against the mode it will run in.** Blocking compiles for streaming, rewriting for block scanning, and the two do not accept quite the same patterns (`a{1,1000}b` streams but does not block-scan). Judging by the wrong one refuses rules that work — the same class of lie as a tester that disagrees with the engine. `debug.rs` reports `unscannable` for patterns that are valid but cannot be highlighted.
- **Regex matching is hyperscan, everywhere — there is no second regex engine in the tree.** `binsrc/regex/` on the NFQUEUE side, `proxysrc/src/hyperscan.rs` on the proxy side, a binding rather than a reimplementation. Adding `regex` back to `Cargo.toml` would undo this.
- **A pattern blocks. There is no second action, and `rewrite` is not coming back.** It existed, and was withdrawn because it could not keep the promise it made: rewriting scanned one chunk at a time, since bytes already forwarded cannot be taken back, so a match straddling two chunks was never rewritten. Measured — a pattern sent whole arrived redacted, the same pattern split across two TCP segments arrived intact, and blocking (which scans in stream mode) caught both. The miss was not the problem; its silence was — no block, no log, no counter, just a rule the operator trusted that had quietly not applied. The removal has to be complete to mean anything, and it was half-done for a while: an `action` column that was gone from the datapath still had a `rewritten` field travelling from `debug.rs` through the backend model to a panel in `RegexDebugger.tsx` that could never render, a single-member `Action` enum in `firegex.regex`, and a `replace_with` on the frontend `Regex` type. If any of those spellings reappears, it is residue, not a feature.
- **A pattern is validated against the mode it will run in**, which is streaming, because that is how a pattern follows a connection across chunk boundaries. hyperscan does not accept quite the same patterns in every mode (`a{1,1000}b` streams but does not block-scan), and judging by the wrong one refuses rules that work — the same class of lie as a tester that disagrees with the engine. Block-mode scanning survives for one job only: reporting *where* a match starts, so the tester can highlight it. `debug.rs` reports `unscannable` for patterns that are valid, will run, and cannot be highlighted.
- **The Python API is `firegex.pyfilters`, everywhere.** `cpproxy` embeds it; `modules/services/pyworker.py` runs it out of process. An earlier version of that worker defined a look-alike `@pyfilter` taking `(data, direction)`, which meant a filter written from the docs loaded on one transport and failed on the other.
- **A filter file's protocol is read off the file, never declared.** The library already decides when to call a filter from what its parameters are annotated with, so `infer_proto` in `firegex/pyfilters/internals` reads the same table back the other way: asking for an `HttpRequest` makes a file an HTTP filter, asking only for a `RawPacket` makes one run on anything, and asking for two *different* application protocols is refused naming both functions. `type_annotations_associations` is the single place a protocol is defined; a second list of names would be one more thing to keep in step, and the day it fell behind the symptom would be a filter that silently never runs. The backend stores the detected value on the row so the interface can show it, and `PUT .../code` is what recomputes it. Passing one in from outside is exactly what used to make an HTTP filter on a TCP service a 500: the *service's* protocol was sent to the datapath, so the filter was refused for asking for an `HttpRequest`, which is the only thing an HTTP filter does.
- **A filter file's functions run in the order the file defines them**, and the first to answer anything but ACCEPT ends the packet. `@pyfilter` sets a mark on the function and hands it back; `collect_pyfilters` reads a module's globals, which are in definition order. What that replaced was a *set of names* on a module-global attribute of the decorator, so the order was whatever hashing produced — which decided which function got credited with a block and whether an earlier `UNSTABLE_MANGLE` survived, with nothing in the file able to tell you which. The `position` column on `pyfilters` carries the same order to the interface, which used to list them `ORDER BY name`. Three consequences of that registry are also gone with it: `get_filters_info` had to clear it before and after (and reached it by `exec`ing import strings), two compilations could race, and a function registered from somewhere the module globals could not reach was registered and then not found.
@ -120,6 +143,7 @@ Two implementations of the same concept is the mistake this module was restructu
- A filter is called directly. `context_call` used to write the arguments into the module globals, `eval` a call expression built from them, and delete the three keys again — per filter, per packet — which is exactly `func(*args)`, because the operator's code was `exec`ed into those globals and so the function's `__globals__` *is* that dict.
- The default for `FGEX_STREAM_MAX_SIZE` is 1 MiB, as documented. It read `1*8e20` under a comment saying "1MB", so `FGEX_FULL_STREAM_ACTION` never fired unless a size was set by hand, and the float it produced would have been refused by the setter beside it.
- **The editor's hints are introspected, never written down.** `modules/services/pyapi.py` reads the models out of `type_annotations_associations` and their members out of the classes — including `writable`, from `property.fset`, which is how the interface can say that the payload is the only thing a filter may assign to. `GET /api/services/pyfilter-api` serves it and `pyfilterHints.ts` feeds Monaco's completion and hover from it. A second, hand-written description would be wrong the first time a model gained a property, and autocompletion that lies is worse than none. The route is declared **before** `/{service_id}`, or that one matches it and looks for a service by that name.
- **A filter that raises says so the same way on both layers.** `_on_exception` writes one sentence — the traffic was forwarded unfiltered — and it has to be reachable from either. NFQUEUE gets there directly: the interpreter is embedded in the binary, which sends `EXCEPTION`. The proxy cannot, because `pyworker.py` catches the exception in a separate process and answers with an ordinary ACCEPT, so nothing crosses the frame boundary that the engine could tell apart from a filter agreeing. It marks stderr with `EXCEPTION_MARK` instead and `Transport._pump_stderr` turns that back into the same call. The traceback is printed *beside* it, not instead of it: the traceback says which filter and which line, and the sentence says what happened to the connection — for a while the proxy printed only the first, which tells an operator a filter broke and not that traffic went through unchecked. The marker is spelled in both files because the worker is launched as a script and imports nothing from the backend, which is what keeps the user's code out of this process's imports; `tests/integration/test_resilience.py` asks for the sentence on both layers and so keeps them in step.
- **A block names the function, not just the file.** Both layers report `<filter_id>/<function>`: the NFQUEUE binaries already reported the function name, and the proxy worker now returns it in the verdict frame (`[verdict][name len][name][payload]`). `ServiceManager._on_block` splits that one token in one place. A file with ten functions blocking 900 connections is not an answer to "which of them is doing it".
- **Nothing below the application layer crosses into a filter.** `RawPacket` exposes addresses, ports, family and direction as read-only metadata, and `data` — the payload — as the only writable thing; `mangled_data` comes back, never a packet. That boundary exists because the two layers cannot honestly offer the same thing underneath it: NFQUEUE has a real header whose rewriting desynchronises the stream, while the proxy terminated the connection and writes its own. The old model papered over that by handing the proxy a literal `FAKE:IP:TCP:HEADERS:` prefix as `raw_packet`, so the same filter did different things depending on where it was attached. On the proxy the metadata is *better* than a header would be — the engine knows both ends and sends them in a `KIND_OPEN` frame (`ConnectionMeta` → `Filter::connection_opened`), once per connection rather than on every chunk.
- **The order the operator set is the order that runs.** `parse_ruleset` groups patterns by their `filter` id and keeps the positions the backend sent. An earlier version grouped every block, then the Python — which turned the chain into a set, so a filter placed after another could run before it, and nothing said so. Within one filter the operator expressed no order, so blocking goes before rewriting there; across filters, never reorder. `tests/rules.rs` pins both.
@ -225,7 +249,7 @@ It belongs to the **proxy** layer: decrypting means terminating the connection,
What that replaced: nginx terminated on one derived loopback port and re-encrypted from a second, with the filters attached to the leg in between. Protecting one address cost a pair of ports produced by hashing `ip:port` — *chosen* ports, which could collide with something real — plus a generated config, a `tls_streams` table keyed by address id, an nft chain of its own, and a failure mode where one unloadable certificate stopped nginx and took **every** TLS service on the instance down with it. All of that is gone: `modules/tls/` no longer exists, and the blast radius is now structural, since each service carries its certificate into its own engine process.
**ALPN is mirrored, and the ordering that makes it possible is the design.** The client's handshake is *started* and held at the ClientHello (`tokio_rustls::LazyConfigAcceptor`), because that message carries the protocols the client will speak; the upstream handshake goes next offering exactly that list, and whatever the service picked is the single protocol the client is then told (`tls::with_alpn` / `tls::answering_with`, both cloning the config per connection — the list comes from the client, so there is no fixed set to precompute, and a config clone is nothing beside a handshake). Answering something the service did not say is how a proxy breaks a protocol it was only carrying: a client told `h2` against an HTTP/1.1 service sends frames nothing can read. When the service picks nothing the client is told nothing — no agreement is invented. `tests/services_test.py` pins all four cases including both mismatches. Before this, ALPN was dropped in both directions and every client silently fell back to HTTP/1.1.
**ALPN is mirrored, and the ordering that makes it possible is the design.** The client's handshake is *started* and held at the ClientHello (`tokio_rustls::LazyConfigAcceptor`), because that message carries the protocols the client will speak; the upstream handshake goes next offering exactly that list, and whatever the service picked is the single protocol the client is then told (`tls::with_alpn` / `tls::answering_with`, both cloning the config per connection — the list comes from the client, so there is no fixed set to precompute, and a config clone is nothing beside a handshake). Answering something the service did not say is how a proxy breaks a protocol it was only carrying: a client told `h2` against an HTTP/1.1 service sends frames nothing can read. When the service picks nothing the client is told nothing — no agreement is invented. `tests/integration/test_tls.py` pins all five cases, including both mismatches and a client that offers only one protocol. Before this, ALPN was dropped in both directions and every client silently fell back to HTTP/1.1.
What cannot be carried, and is a limit rather than a missing setting: **TLS 1.0 and 1.1**, which rustls does not implement at all; the cipher list, which is fixed and forward-secret only; and **client certificates**, since `with_no_client_auth()` is the only honest answer when the certificate a client would present is one only the service can check. Say these are impossible rather than unconfigured — an operator who needs TLS 1.0 needs the `external` layer, not a flag.
@ -233,7 +257,7 @@ The key must be one `ring` will sign with — RSA ≥ 2048 bits, ECDSA, or Ed255
**The decrypted traffic is written to an interface of its own, and those are reconstructed packets.** Terminating in-process means the plaintext is never a packet, so `proxysrc/src/capture.rs` frames it as the TCP stream it was and sends it to `firegex0` over `AF_PACKET`: one interface carrying every TLS service's plaintext and nothing else, which is what a capture tool wants pointed at it. The bytes are real — what the filters saw and what was forwarded — but the framing is invented, because the framing on the wire was encrypted. Sequence numbers start at zero, there are no retransmissions, and the segmentation is the engine's read sizes; a SYN and a FIN are emitted so Wireshark has a stream to follow rather than orphan segments. Say so wherever it is offered: a capture from here is not evidence of what was on the wire.
Three things keep it from being able to hurt anything. `modules/services/mirror.py` creates the device before the engine starts (a socket opened before the interface exists finds nothing) and `FirewallManager.release_capture_if_idle()` removes it once nothing is decrypting, so nobody points a capture at an interface that will stay empty. Every failure is silent and local — no interface, no socket, a failed send — and the traffic goes on exactly as it would have. And the `AF_PACKET` socket needs `CAP_NET_RAW`, which `docker-entrypoint.sh` asks for **and starts without**: it tries the full capability set, falls back to the set without `cap_net_raw` while saying what that costs, and only then runs as root. A host that will not grant it loses the capture, not the firewall.
Three things keep it from being able to hurt anything. `modules/services/mirror.py` creates the device in `FirewallManager.init()`, before any engine starts (a socket opened before the interface exists finds nothing), and `release_capture()` removes it from the process's `shutdown()` hook and nowhere else — deliberately not from `close()`, which `reset()` also calls, or the device would vanish and reappear underneath whatever was capturing from it. It is the **instance's** device, not each TLS service's: it used to be created and released per service, on the argument that an interface present while nothing is decrypting is one somebody points a capture at and watches stay empty — but a capture tool is attached once, for a round, and Zeek and Suricata exit rather than wait when the interface under them disappears, which restarting a single TLS service was enough to cause. Watching it stay empty is the better failure. `release_capture_if_idle()` is gone; do not add a per-service release back. Every failure is silent and local — no interface, no socket, a failed send — and the traffic goes on exactly as it would have. And the `AF_PACKET` socket needs `CAP_NET_RAW`, which `docker-entrypoint.sh` asks for **and starts without**: it tries the full capability set, falls back to the set without `cap_net_raw` while saying what that costs, and only then runs as root. A host that will not grant it loses the capture, not the firewall.
Note for tests: a client now dials the service's own address, because there is no second port to be pointed at. The old note about the DNAT skipping loopback belonged to nginx's public-port redirect and went with it.
@ -251,7 +275,7 @@ Frontend note: a 401 used to call `window.location.reload()`. A token left over
Independently of that, `IPFilterMiddleware` restricts access by CIDR (`--allowed-ips`, optionally reading the client IP from `--proxy-ip-header`). It **fails closed**: a missing, unparseable, or non-matching IP is denied. Note the header variant trusts a client-supplied value, so it only holds if the proxy overwrites it.
### SQLite wrapper (`backend/utils/sqlite.py`)
Each DB file's schema is versioned by hashing the Python schema dict; on mismatch the **entire file is deleted and recreated** (not migrated) — any schema dict change wipes that DB file on next startup, including `db/firegex.db`'s `keys_values` table (password/JWT secret). `dump()`/`load()` back the export/import backup feature; `dump()` deliberately excludes `password`, `secret` **and `auth_disabled`** from exported `keys_values` rows, so `/api/import` explicitly re-preserves all three around the import. The third one is the newest and the easiest to lose: it is not a credential, it is *whether credentials are asked for*, which belongs to the deployment and not to the configuration being restored — a backup able to carry it is a backup able to turn a firewall's authentication off. `auth_disabled` is also put back **after** `startup_main()`, not before like the other two, because `startup_main` seeds it from the environment: right when a process is starting, wrong during an import where nothing restarted and the environment did not change. `tests/api_test.py` turns authentication off *around* the import on purpose — asserting it is still on afterwards passes either way on a container whose environment agrees, and the bug only shows where the running state and the boot-time environment differ.
Each DB file's schema is versioned by hashing the Python schema dict; on mismatch the **entire file is deleted and recreated** (not migrated) — any schema dict change wipes that DB file on next startup, including `db/firegex.db`'s `keys_values` table (password/JWT secret). `dump()`/`load()` back the export/import backup feature; `dump()` deliberately excludes `password`, `secret` **and `auth_disabled`** from exported `keys_values` rows, so `/api/import` explicitly re-preserves all three around the import. The third one is the newest and the easiest to lose: it is not a credential, it is *whether credentials are asked for*, which belongs to the deployment and not to the configuration being restored — a backup able to carry it is a backup able to turn a firewall's authentication off. `auth_disabled` is also put back **after** `startup_main()`, not before like the other two, because `startup_main` seeds it from the environment: right when a process is starting, wrong during an import where nothing restarted and the environment did not change. `tests/integration/test_backup.py` turns authentication off *around* the import on purpose — asserting it is still on afterwards passes either way on a container whose environment agrees, and the bug only shows where the running state and the boot-time environment differ.
### Real-time frontend updates
One Socket.IO event, `"update"`, whose payload is a list of strings treated as a react-query `queryKey` prefix (e.g. `["services"]`, `["firewall"]`). `App.tsx` does `queryClient.invalidateQueries({ queryKey: data })` — prefix matching invalidates any query whose key starts with that array. Convention: each module's mutating endpoints emit the same tag string that module's frontend queries use as the first `queryKey` element. `staleTime: Infinity` is set globally, so this event is the **only** automatic refresh mechanism — a missing/mismatched tag means silent staleness for other clients/tabs.
@ -262,11 +286,17 @@ One Socket.IO event, `"update"`, whose payload is a list of strings treated as a
### `fgex-lib/`
Separate, independently-published PyPI package (`pip install fgex`/`firegex`) — the library users write Python filters against (`@pyfilter`, `RawPacket`/`HttpRequest`/etc.), plus the `fgex` CLI, a local `proxysim` for testing filters, and `firegex.regex` for testing *pattern* rulesets, without a real firegex instance.
The library module is **`firegex.pyfilters`**. It was `firegex.nfproxy` when that was one of several modules, and `firegex/nfproxy/__init__.py` still resolves — by aliasing the submodules into `sys.modules`, not by re-exporting them. That distinction is load-bearing: `type_annotations_associations` maps model **classes** to protocols, and both the library and the datapath decide when to call a filter by looking an annotation up in that table, so a shim that built second copies of `HttpRequest` would leave old filter files annotating a class the table has never heard of — importing cleanly and then never being called. One class per model, two spellings of one package. The CLI is `fgex pyfilters`, with `fgex nfproxy` kept as a hidden alias.
The library module is **`firegex.pyfilters`**, and the CLI command is `fgex pyfilters`. Both were `nfproxy` when that was one of several modules. A compatibility alias carried the old spellings through 4.x and was **removed in 5.0.0**, which is a breaking change and the reason the version is a major one: `from firegex.nfproxy import pyfilter` is the first line of every filter written before it, including the ones already sitting in a running firegex's database.
Two things about that removal are worth keeping. The directory has to go, not just its contents — a `firegex/nfproxy/` left behind with only a `__pycache__` in it is a *namespace package*, so the import succeeds and hands back something empty, and `from firegex.nfproxy import pyfilter` then fails with an `ImportError` about a name rather than a `ModuleNotFoundError` about a module. And the failure is made to explain itself where the operator meets it: `pyworker.py --check` recognises this one import and says what to write instead, because the alternative is a filter that stops loading at the first start after an upgrade, which at a competition is the start of a round.
Do not reintroduce the alias as a file of re-exports if it ever comes back. `type_annotations_associations` maps model **classes** to protocols, and both the library and the datapath decide when to call a filter by looking an annotation up in that table, so a shim building second copies of `HttpRequest` would leave a filter annotating a class the table has never heard of — importing cleanly and then never being called. The old one aliased submodules into `sys.modules` for exactly that reason: one class per model, two spellings of one package.
**Deploying is `run.py`, and only `run.py`.** The package used to carry a `fgex deploy` passthrough that exec'd a copy of it; it is gone, along with the build-time copy `setup.py` made. The package is for writing and testing filters — `firegex.pyfilters`, `firegex.regex`, the two simulators — and starting a firegex is a separate job done from a clone. Do not reintroduce a deployment command here: the reason the old one was a verbatim passthrough was that any second description of `run.py`'s argument surface drifts, and not shipping one at all is the same argument taken one step further.
`firegex/regex/` binds `libhs` through ctypes rather than depending on a wheel: no build step, nothing to go stale, and by construction the same library the product loads. It is a third binding to one engine, not a third engine — the distinction that matters is that there is still only one thing deciding what matches. Where libhs is absent the tools raise `HyperscanMissing` and stop; adding an `re` fallback would reintroduce exactly the lie the in-app tester was built to avoid. `tests/regex_lib_test.py` pins the agreement (validity per mode, direction, rewriting precedence, the shared file format). Not part of the backend/frontend build. `fgex-lib/README.md` is a short index (PyPI-facing) linking out to `docs/*.md`; it isn't rendered anywhere itself.
**The HTTP parser ships as source, and nobody upstream will bump it.** `fgex-lib/llhttp/` carries llhttp itself (`lib/`, verbatim from a `release/*` tag) and our binding beside it, because `pyllhttp` is archived, every binding with prebuilt wheels fails to load in the subinterpreters `cpproxy` runs filters in, and a ctypes binding against the system `libllhttp` would make `pip install firegex` assume a package installed beforehand. `PROVENANCE.md` there is what makes that maintainable: which tag, which checksums, what is ours and what is theirs, and how to move it. Each release changes what the parser *accepts* — going from 9.2.1 to 9.4.3, an empty `Transfer-Encoding` stopped being accepted and tabs around `Content-Length` started being — so `tests/unit/test_http_parsing.py` pins both through the public models: a change there is a decision about what filters are shown, never a test to fix. A parser more lenient than the service behind it is how a filter gets bypassed while still looking switched on. **The binding names itself and nothing else**: `pyllhttp` is where it was forked from, not something it uses, and the error path that imported it at runtime raised `SystemError: ... returned NULL without setting an exception` on every malformed request anywhere that archived package was not installed — which is everywhere except the machines it was written on. `wheel_smoke.py` is the check that could have caught it, and the reason it is a file rather than a `-c` string: `cibuildwheel` runs it against each wheel in an environment with nothing else in it, parsing something malformed as well as something valid.
`firegex/regex/` binds `libhs` through ctypes rather than depending on a wheel: no build step, nothing to go stale, and by construction the same library the product loads. It is a third binding to one engine, not a third engine — the distinction that matters is that there is still only one thing deciding what matches. Where libhs is absent the tools raise `HyperscanMissing` and stop; adding an `re` fallback would reintroduce exactly the lie the in-app tester was built to avoid. `tests/unit/test_regex_library.py` pins the agreement: validity in the mode a pattern will run in, direction, the shared file format, and that a ruleset asking to `rewrite` is refused rather than quietly treated as a block. Not part of the backend/frontend build. `fgex-lib/README.md` is a short index (PyPI-facing) linking out to `docs/*.md`; it isn't rendered anywhere itself.
### Module documentation (`docs/*.md`)
Each module's user-facing docs (`docs/services.md`, `docs/pyfilter.md`, `docs/firewall.md`) are single markdown sources, each imported directly with Vite's `?raw` suffix by a matching `<Module>Docs.tsx` component (e.g. `components/Services/ServicesDocs.tsx`) and rendered through the shared `components/MarkdownDocs.tsx` (React Markdown + `remark-gfm`, code fences routed through `CodeHighlight`, relative `.md` cross-links rewritten to the equivalent GitHub blob URL since they'd otherwise resolve against the app's own origin). `DocsButton.tsx`'s `DocType` enum wires each module to its component. Every language a fence uses has to be in `index.tsx`'s `createHighlighter({ langs })` — Shiki throws for one it was not given, and that throw is uncaught, so it takes the whole page down rather than the block it could not colour. Editing a module's docs means editing its one `docs/*.md` file — never the `*Docs.tsx` component directly. Reading these files across the `frontend/`↔repo-root boundary requires `vite.config.ts`'s `server.fs.allow: ['..']` (dev) and the `Dockerfile` frontend build stage's `COPY ./docs/*.md /docs/` (prod/standalone rootfs) to stay in sync with any path changes.

10
README.md
View File

@ -86,13 +86,13 @@ This is especially useful if you lost the current password: it's applied directl
- **[Services](docs/services.md)**: the main module. A service is one protected endpoint described in two independent parts — a **network layer** that says how its traffic is intercepted, and an ordered chain of **filters** that says what happens to it.
- A service protects a **list of addresses or network interfaces**, mixing IPv4, IPv6 and network interfaces (`eth0`, `wg0`, etc.) freely: intercept traffic on specific IPs or directly across an entire interface. Adding an address or interface to a running service drops no connections.
- Network layer: **NFQUEUE** (packets lifted to userspace with [nfqueue](https://netfilter.org/projects/libnetfilter_queue/) and [nftables](https://netfilter.org/projects/nftables/), nothing terminated, the kernel keeps forwarding if a filter dies) or **proxy** (the connection is terminated and reopened, which buys exact rewriting, an ordered chain of filters, and real backpressure — while still dialling your service from the client's own address, so it never stops seeing who is talking to it).
- Network layer: **NFQUEUE** (packets lifted to userspace with [nfqueue](https://netfilter.org/projects/libnetfilter_queue/) and [nftables](https://netfilter.org/projects/nftables/), nothing terminated, the kernel keeps forwarding if a filter dies) or **proxy** (the connection is terminated and reopened, which buys kernel reassembly, an unlimited ordered chain of filters, TLS termination and real backpressure — while still dialling your service from the client's own address, so it never stops seeing who is talking to it).
- Filters: **regex**, matched by [hyperscan](https://github.com/VectorCamp/vectorscan) — the same engine on either layer, so a pattern means the same thing wherever it runs — or **Python**, your own code written against the [`firegex`](https://pypi.org/project/firegex/) library, with built-in models for HTTP and TCP streams. You never declare which protocol a Python filter speaks: asking for an `HttpRequest` is what makes it an HTTP filter, and a filter reads the addresses and ports as metadata while the payload is the only thing it can change.
- A regex can **block** the connection or **rewrite** what it matched (redact a leaked flag, neuter a header). Rewriting is exact on the proxy layer, which owns both connections; the NFQUEUE layer refuses it rather than desynchronising the stream.
- **TLS** is a switch on the service, not a separate thing to create: nginx terminates the connection, the filters inspect the plaintext, and it is re-encrypted towards your service.
- A regex **blocks** the connection, and that is deliberately all it does: rewriting had to scan one chunk at a time, so a pattern split across two TCP segments went through unredacted with nothing to say it had — a rule you trust and that quietly does not apply is worse than no rule. To change bytes rather than refuse them, use a Python filter, which sees the reassembled stream.
- **TLS** is one of the protocols a service can speak, not a separate thing to create: the proxy engine terminates the connection itself, the filters inspect the plaintext, and it is re-encrypted towards your service — with the client's ALPN mirrored rather than invented, and the decrypted stream written to a `firegex0` interface you can point a capture at. A TLS service occupies no extra port.
- Or **your own proxy** as the network layer: firegex rewrites the destination on the way in and changes it back on the way out, so a proxy you wrote yourself sits in the path invisibly. The escape hatch for a protocol nothing built in understands.
- **TCP or UDP**, on every layer — with full source IP transparency on both TCP and UDP. On the proxy layer, UDP datagrams are relayed with dedicated sockets using transparent IP spoofing (`IP_TRANSPARENT`), with exact rewriting, per-flow filter state, and zero-downtime dynamic address additions. NFQUEUE filters packets in place with kernel-level fail-open.
- **The two inspecting layers are opposite trades, [compared side by side in the interface itself](docs/services.md#network-layers) while you choose**: the proxy owns both halves of a connection, so it rewrites exactly and lets the kernel reassemble, at the cost of weight and of rebuilding fail-open by hand; NFQUEUE is per packet, costs almost nothing and keeps forwarding by kernel guarantee if a filter dies, at the cost of userspace reassembly and no pattern rewriting.
- **TCP or UDP**, on every layer — with full source IP transparency on both. On the proxy layer, UDP datagrams are relayed with one dedicated socket per address using transparent IP spoofing (`IP_TRANSPARENT`), with per-flow filter state and zero-downtime dynamic address additions. NFQUEUE filters packets in place with kernel-level fail-open.
- **The two inspecting layers are opposite trades, [compared side by side in the interface itself](docs/services.md#network-layers) while you choose**: the proxy owns both halves of a connection, so the kernel reassembles for it and a Python filter can rewrite a payload exactly, at the cost of rebuilding fail-open by hand; NFQUEUE is per packet and keeps forwarding by kernel guarantee if a filter dies, at the cost of userspace reassembly and a process per filter.
- **Charts of what each filter has refused** — per filter, per pattern and per `@pyfilter` function, each with **its share of the blocking**. Pick the window (last 15 minutes, hour, 6h, 24h, everything kept, or two exact instants) and the shape (bars, lines, stacked area, or share); **everything on the page counts the same window**, so the chart and the table beside it can never tell different stories. Alongside it, how much traffic arrived, in the unit each layer can honestly report.
- **The Python editor knows the library**: completion and hover for every model and its members — marking which of them you may write to — and the file is **checked as you type by the process that will run it**, so a mistake is flagged on its own line with the reason instead of surfacing later as "worker exited". The hints are introspected from the installed library, so they cannot drift from it.
- A **live log** per service shows what it is doing as it does it: what refused each connection (by name and pattern, not an opaque id), whatever your Python prints, and what the datapath says about its own health.

37
backend/binsrc/pyproxy/stream_ctx.cpp
View File

@ -5,6 +5,8 @@
#include <iostream>
#include <tins/tcp_ip/stream_identifier.h>
#include <map>
#include <atomic>
#include <chrono>
#include <Python.h>
#include "../classes/netfilter.cpp"
#include "../classes/nfqueue.cpp"
@ -54,6 +56,35 @@ struct py_filter_response {
typedef Tins::TCPIP::StreamIdentifier stream_id;
// How long between two tracebacks out of this process, in seconds.
//
// Filter code that throws throws on every packet — a typo does not fire once — and a
// traceback is a dozen lines. Unthrottled, one broken filter writes thousands of lines a
// minute into a log that is a bounded ring, so the flood does not merely repeat itself:
// it pushes out the first traceback, which was the one worth reading, along with
// everything that was in the log before the filter broke.
//
// The `EXCEPTION` sent to the backend is deliberately *not* throttled with it. That is
// one token, it costs nothing, and it is what the backend counts to say "still raising,
// N more since" — so the operator still learns how often this is happening while being
// shown the traceback once.
constexpr int64_t TRACEBACK_QUIET_SECONDS = 30;
// Process-wide rather than per stream: the flood is one broken filter, not one broken
// connection, and throttling per connection would let a hundred connections print a
// hundred copies of the same fault. Relaxed ordering is enough — two threads racing here
// print one traceback each, which is not a problem worth a lock.
static std::atomic<int64_t> last_traceback_at{-TRACEBACK_QUIET_SECONDS * 2};
inline bool traceback_is_due() {
const int64_t now = std::chrono::duration_cast<std::chrono::seconds>(
std::chrono::steady_clock::now().time_since_epoch()).count();
int64_t previous = last_traceback_at.load(std::memory_order_relaxed);
if (now - previous < TRACEBACK_QUIET_SECONDS) return false;
return last_traceback_at.compare_exchange_strong(
previous, now, std::memory_order_relaxed);
}
struct pyfilter_ctx {
PyObject * glob = nullptr;
@ -134,8 +165,14 @@ struct pyfilter_ctx {
del_item_from_glob("__firegex_packet_info");
if (PyErr_Occurred()){
// Shown at most once every TRACEBACK_QUIET_SECONDS; the EXCEPTION below goes
// every time, and is what carries the count.
if (traceback_is_due()){
cerr << "[error] [handle_packet] Failed to execute the code " << result << endl;
PyErr_Print();
} else {
PyErr_Clear();
}
#ifdef DEBUG
cerr << "[DEBUG] [handle_packet] Exception raised" << endl;
#endif

114
backend/modules/services/firewall.py
View File

@ -25,7 +25,7 @@ from modules.services import transports
from modules.services import stats
from modules.services.logs import LEVEL, forget as forget_log, log_for
from modules.services.models import KIND, STATUS, Address, Filter, Regex, Service
from modules.services.nftables import FiregexTables
from modules.services.nftables import FiregexTables, NoRelayAddress, udp_relay_host
from utils.sqlite import SQLite
nft = FiregexTables()
@ -38,6 +38,14 @@ CODE_DIR = "db/service_filters"
#: arriving twice — once at the service and once in the operator's log.
OVER_LIMIT_QUIET = 30
#: How long between two reports that a filter is raising. Same reasoning as the line
#: above, for the same shape of problem: user code that throws usually throws on every
#: packet, so a line each is the fault arriving once per packet in a log that holds a
#: few hundred lines. What the operator needs is that it is happening, how often, and one
#: traceback — not the same traceback several hundred times, with everything else that
#: was in the log pushed out behind it.
RAISED_QUIET = 30
def code_path(filter_id: str) -> str:
return os.path.join(CODE_DIR, f"{filter_id}.py")
@ -79,6 +87,8 @@ class ServiceManager:
#: When the limit was last mentioned in the log, so a flood is reported rather
#: than reproduced.
self._over_limit_told = 0.0
self._raised_told = 0.0
self._raised_since = 0
#: What the running datapath needs the rules to point at — a queue number per
#: filter, or the port the proxy is listening on. Kept so an address added later
#: can be steered at the same datapath instead of restarting it.
@ -204,7 +214,30 @@ class ServiceManager:
self.log.add(LEVEL.OUTPUT, text)
def _on_exception(self, _service_id: str):
self.log.add(LEVEL.ERROR, "a filter raised; the traffic was forwarded unfiltered")
"""A filter threw, and the chunk went through unfiltered.
Rate-limited, because code that throws throws on every packet: unthrottled this
is the same sentence a few hundred times, and the log is a bounded ring — so the
flood does not merely repeat itself, it evicts everything else, including the
first report and whatever was there before the filter broke.
The suppressed ones are counted rather than dropped. "it is still happening, 412
times since" is the number that says whether this is one odd request or every
request, which is the question the operator actually has.
"""
now = time.time()
self._raised_since += 1
if now - self._raised_told < RAISED_QUIET:
return
first = self._raised_told == 0.0
count, self._raised_since = self._raised_since, 0
self._raised_told = now
self.log.add(
LEVEL.ERROR,
"a filter raised; the traffic was forwarded unfiltered"
if first and count == 1 else
f"a filter is still raising: {count} more since, all forwarded unfiltered",
)
def _on_engine(self, text: str):
"""Whatever the datapath itself says about its own health.
@ -290,11 +323,6 @@ class ServiceManager:
on_over_limit=self._on_over_limit,
)
try:
# Before the engine, which opens its socket to it at startup: an
# interface that appears afterwards is one this process will not find
# until it is restarted.
if self.srv.terminates_tls:
mirror.ensure()
steer = await self.transport.start(self.chain())
self._steer = steer
# Clear first: a stale rule from a crashed run would otherwise sit in
@ -377,13 +405,12 @@ class ServiceManager:
self.reload_addresses()
if not self.active:
return
is_dual = getattr(self.transport, "is_dual_stack", False)
if (
self.srv.transport == transports.TRANSPORT.PROXY
and str(self.srv.proto) != "udp"
and self.srv.has_ipv6
and not was_dual
and not is_dual
and not self.transport.is_dual_stack
):
self.log.add(
LEVEL.INFO,
@ -394,34 +421,20 @@ class ServiceManager:
added = [addr for addr in self.srv.addresses if addr.id == address_id]
if not added:
return
if self.srv.transport == transports.TRANSPORT.PROXY:
from modules.services.nftables import one_address, ip_family
from utils import is_ip_parse, get_interface_ips
for addr in added:
l4 = str(addr.proto or self.srv.proto)
if l4 == "udp":
is_iface = not is_ip_parse(addr.ip_int)
if is_iface:
ips = get_interface_ips(addr.ip_int)
if not ips:
raise transports.UnsupportedChain(
f"interface '{addr.ip_int}' has no IP assigned for UDP proxy relay"
)
target_ip = one_address(ips[0])
else:
target_ip = one_address(addr.ip_int)
target_port = addr.port
family = ip_family(target_ip)
key = (
f"[{target_ip}]:{target_port}"
if family == "ip6" else f"{target_ip}:{target_port}"
)
if "udp_ports" not in self._steer:
self._steer["udp_ports"] = {}
if key not in self._steer["udp_ports"]:
port = await self.transport.add_udp_target(target_ip, target_port)
self._steer["udp_ports"][key] = port
async with self.lock:
# UDP is relayed by one socket per address, so a new address is a new relay
# — opened on the engine that is already running, before the rule that will
# point traffic at it exists. The engine owns the map of them: `_steer`
# carries that same dict, so what is opened here is what the rule finds.
if self.srv.transport == transports.TRANSPORT.PROXY:
for addr in added:
if str(addr.proto or self.srv.proto) != "udp":
continue
try:
host = udp_relay_host(addr.ip_int)
except NoRelayAddress as e:
raise transports.UnsupportedChain(str(e)) from e
await self.transport.add_udp_target(host, addr.port)
nft.add(self.srv, added, **self._steer)
self.log.add(
LEVEL.INFO,
@ -500,6 +513,12 @@ class FirewallManager:
async def init(self):
nft.init()
# Before any engine, which opens its socket to it at startup: an interface that
# appears afterwards is one that process will not find until it is restarted.
# Once, here, rather than per service: a capture tool is attached to it for the
# length of a round, and an interface that comes and goes with the services
# takes the tool with it.
mirror.ensure()
await self.reload()
async def reload(self):
@ -529,27 +548,24 @@ class FirewallManager:
if persist:
# A deleted service's log has nothing left to be about.
forget_log(srv_id)
self.release_capture_if_idle()
def release_capture_if_idle(self) -> None:
"""Take the capture interface away once nothing is decrypting.
Asked of the managers rather than of the database, because what matters is which
services are *running*: a stopped one writes nothing, and an interface that is
there while nothing is decrypting is one somebody points a capture at and watches
stay empty.
"""
if any(m.active and m.srv.terminates_tls for m in self.services.values()):
return
mirror.release()
async def close(self):
"""Stop every service. Deliberately *not* the end of the capture interface.
`reset()` closes and re-initialises without the process going anywhere, so
releasing here would make the device disappear and come back underneath whatever
was capturing from it — the exact failure it was made persistent to avoid.
"""
for key in list(self.services.keys()):
try:
await self.remove(key, persist=False)
except Exception:
self.services.pop(key, None)
def release_capture(self) -> None:
"""Take the capture interface away, at process shutdown and nowhere else."""
mirror.release()
def get(self, srv_id: str) -> ServiceManager:
if srv_id not in self.services:
raise ServiceNotFoundException()

14
backend/modules/services/mirror.py
View File

@ -10,9 +10,15 @@ All that is needed here is the device: a `dummy` interface, which is a sink that
whatever is sent to it and goes nowhere. One for the whole instance, so every TLS
service's plaintext arrives on the same interface and a single capture covers the lot.
It exists only while some TLS service is running. Nothing is lost by removing it — the
engine reopens its socket per process — and an interface that is there when nothing is
decrypting is an interface somebody points a capture at and watches stay empty.
It exists for as long as firegex does, put there by `FirewallManager.init` and taken
away by `close`. It used to come and go with the TLS services themselves, on the argument
that an interface present while nothing is decrypting is one somebody points a capture at
and watches stay empty. Watching it stay empty turned out to be the better failure: a
capture tool is attached once, at the start of a round, and an interface that disappears
underneath it takes the tool with it — Zeek and Suricata exit rather than wait, and the
restart of a single TLS service was enough to do it. So the device now outlives any one
service, and the thing an operator points at in the morning is still there in the
afternoon.
"""
import subprocess
@ -45,5 +51,5 @@ def ensure() -> bool:
def release() -> None:
"""Take it away again."""
"""Take it away again, at shutdown and nowhere else."""
_run("ip", "link", "del", DEVICE)

5
backend/modules/services/models.py
View File

@ -256,11 +256,6 @@ class Service:
"""The transport the kernel sees, which is what a rule has to match."""
return L4.l4_of(self.proto)
@property
def hands_off(self) -> bool:
"""Whether firegex only steers this traffic rather than inspecting it."""
return self.transport == TRANSPORT.EXTERNAL
@property
def has_ipv6(self) -> bool:
"""Whether any of its addresses is IPv6.

61
backend/modules/services/nftables.py
View File

@ -89,6 +89,48 @@ def one_address(ip: str) -> str:
return str(ip).split("/")[0]
class NoRelayAddress(Exception):
"""An address a UDP relay cannot be bound for."""
def udp_relay_host(ip: str) -> str:
"""The concrete address a UDP relay binds for `ip`, which may be an interface name.
A relay has to bind somewhere, and an interface name is not somewhere — so the
interface's own address stands in for it, resolved when it is needed rather than
stored, because an interface's address is the host's to change. IPv4 first and
link-local last: an interface almost always carries a link-local IPv6 address it was
never configured with, and binding the relay to that one would put the service on an
address nobody dials.
"""
if is_ip_parse(ip):
return one_address(ip)
candidates = [one_address(addr.split("%")[0]) for addr in get_interface_ips(ip)]
ordered = sorted(
candidates,
key=lambda a: (ip_family(a) == "ip6", a.lower().startswith(("fe80:", "169.254."))),
)
if not ordered:
raise NoRelayAddress(
f"interface '{ip}' has no IP address assigned, so there is nothing for a "
f"UDP relay to bind to"
)
return ordered[0]
def udp_relay_key(ip: str, port: int) -> str:
"""How a UDP relay's upstream is spelled, everywhere it is spelled.
The engine reports one `UDP <upstream> <port>` line per relay and is asked for a new
one by the same token, so the transport's map of them, the rule that points traffic
at one, and the manager adding an address to a running service all have to agree on
it exactly. Written out at each of those it stops agreeing the first time one of
them is changed, and the symptom is a service whose UDP silently goes nowhere.
"""
host = one_address(ip)
return f"[{host}]:{port}" if ip_family(host) == "ip6" else f"{host}:{port}"
class InstalledRule:
"""A rule already in the ruleset, matched back to the service that owns it.
@ -413,24 +455,7 @@ class FiregexTables(NFTableManager):
# at that address's port rather than at the one shared TCP listener.
port = proxy_port
if l4 == "udp":
if is_iface:
ips = get_interface_ips(target_ip)
if not ips:
raise Exception(
f"the proxy transport has no UDP relay listening for {target_ip}: "
f"interface '{target_ip}' has no IP assigned"
)
resolved_ip = one_address(ips[0])
resolved_family = ip_family(resolved_ip)
key = (
f"[{resolved_ip}]:{target_port}"
if resolved_family == "ip6" else f"{resolved_ip}:{target_port}"
)
else:
key = (
f"[{one_address(target_ip)}]:{target_port}"
if family == "ip6" else f"{one_address(target_ip)}:{target_port}"
)
key = udp_relay_key(udp_relay_host(target_ip), target_port)
port = (udp_ports or {}).get(key)
if port is None:
raise Exception(

85
backend/modules/services/pyworker.py
View File

@ -47,6 +47,7 @@ import json
import re
import struct
import sys
import time
import traceback
VERDICT_ACCEPT = 0
@ -57,6 +58,32 @@ VERDICT_REPLACE = 2
# quietly stopped being filtered.
READY = 0xFF
#: Written to stderr when the user's code raises, beside the traceback itself.
#:
#: The traceback says a filter broke and names its file and line, which is what an
#: operator needs to fix it. It does not say what happened to the *connection*, and that
#: is the part they have to know: it went through unfiltered. The engine cannot report it
#: because it never learns of it — the exception is caught here and answered with ACCEPT,
#: so nothing crosses the frame boundary to tell it apart from a filter that simply
#: agreed. So it is marked on the diagnostics channel instead, and `Transport._pump_stderr`
#: turns the marker back into the one sentence `_on_exception` writes for either network
#: layer. A wire token like `BLOCKED` or `[fatal]`, and spelled the same in both places.
EXCEPTION_MARK = "[exception] [filter]"
#: How long between two tracebacks out of the same worker.
#:
#: A filter that throws usually throws on every packet, and a traceback is a dozen lines.
#: Unthrottled, one broken filter writes thousands of lines a minute into a log that
#: holds a few hundred — so the flood does not merely repeat itself, it pushes out the
#: first traceback, which was the one worth reading, along with everything that was in
#: the log before the filter broke.
#:
#: The mark is *not* throttled with it: it is one short token the engine turns into a
#: single rate-limited sentence on the other side, and it is what carries the count. So
#: the operator still learns how often this is happening, and gets one traceback rather
#: than the same one several hundred times.
TRACEBACK_QUIET = 30
KIND_C2S = 0
KIND_S2C = 1
KIND_CLOSE = 2
@ -144,6 +171,28 @@ def _where_in_source(source: str, exc: BaseException, filename: str) -> dict:
return {"line": 0, "column": 0, "text": ""}
#: The rename this package went through, and the one message worth saying about it.
#:
#: `firegex.nfproxy` was the old spelling of `firegex.pyfilters`, aliased through 4.x and
#: removed in 5.0.0. `from firegex.nfproxy import pyfilter` is the first line of every
#: filter written before that, and the error Python raises for it — "No module named
#: 'firegex.nfproxy'" — is accurate and tells the operator nothing about what to write
#: instead. They meet it here, in the editor, with the line already marked; the
#: alternative is meeting it at the first start after an upgrade, which at a competition
#: is the start of a round.
RENAMED_IMPORT = "firegex.nfproxy"
RENAMED_TO = "firegex.pyfilters"
def _explain(exc: BaseException, message: str) -> str:
"""Add what to do about it, where there is something to say."""
if isinstance(exc, ImportError) and RENAMED_IMPORT in (str(exc) or ""):
return (f"{message}. `{RENAMED_IMPORT}` was renamed to `{RENAMED_TO}` in 5.0.0 "
f"and the old name no longer resolves: change the import at the top of "
f"this file, and the rest of the filter works unchanged")
return message
def check(path: str) -> dict:
"""Would this file load, and what does it define?
@ -173,8 +222,9 @@ def check(path: str) -> dict:
"type": type(e).__name__,
# A SyntaxError's text carries "(<string>, line N)", which names a file
# the operator never wrote and repeats a position reported separately.
"message": re.sub(r"\s*\((?:<[^>]*>|[^()]*\.py), line \d+\)$", "",
str(e) or type(e).__name__),
"message": _explain(e, re.sub(
r"\s*\((?:<[^>]*>|[^()]*\.py), line \d+\)$", "",
str(e) or type(e).__name__)),
**_where_in_source(source, e, path),
"traceback": traceback.format_exc(),
},
@ -326,8 +376,23 @@ def main() -> int:
if len(sys.argv) < 3:
print("usage: pyworker.py --check <filter file>", file=sys.stderr)
return 2
sys.stdout.write(json.dumps(check(sys.argv[2])))
sys.stdout.flush()
# Checking a file means *running* its module body, and `print()` is the most
# natural thing in the world to reach for while writing a filter. On stdout it
# would land in the middle of the answer: the backend parses this output as JSON,
# a stray line makes it unparseable, and the operator is told `the check produced
# no answer` about a file whose only sin was printing while it loaded.
#
# So the real stdout is put aside before the user's code can reach it, and
# anything the file prints goes to the diagnostics channel — the same swap the
# worker itself makes below, for the same reason.
answer = sys.stdout
sys.stdout = sys.stderr
try:
result = check(sys.argv[2])
finally:
sys.stdout = answer
answer.write(json.dumps(result))
answer.flush()
return 0
# `*` (or nothing at all) means every function the file defines, which is what a
# hand-run worker wants; `=a,b` means exactly those, and a bare `=` means none.
@ -358,6 +423,10 @@ def main() -> int:
stdout.write(struct.pack(">I", 2) + bytes([READY, 0]))
stdout.flush()
# Monotonic rather than wall clock: the machine's clock moving is not a reason to
# start printing tracebacks again, or to stop.
last_traceback = -TRACEBACK_QUIET
while True:
header = _read_exactly(stdin, 4)
if header is None:
@ -382,8 +451,14 @@ def main() -> int:
verdict, matched, out = filters.run(connection, payload, kind == KIND_C2S)
except Exception:
# One connection's filter blowing up must not take the process down: the
# other connections it is serving would lose their filtering with it.
# other connections it is serving would lose their filtering with it. The
# chunk is accepted, which is the fail-open half — and the mark is what makes
# that visible rather than silent.
now = time.monotonic()
if now - last_traceback >= TRACEBACK_QUIET:
last_traceback = now
traceback.print_exc()
print(EXCEPTION_MARK, file=sys.stderr, flush=True)
verdict, matched, out = VERDICT_ACCEPT, "", b""
_write(stdout, verdict, matched, out)

73
backend/modules/services/transports.py
View File

@ -25,7 +25,8 @@ import sys
import traceback
from modules.services.models import KIND, L4, PROTO, TRANSPORT, Filter, Regex
from modules.services.nftables import MAX_CHAIN_POSITIONS, one_address
from modules.services.nftables import (MAX_CHAIN_POSITIONS, NoRelayAddress,
resolve_target, udp_relay_host, udp_relay_key)
from utils import DEBUG, nicenessify
MODULES_DIR = os.path.join(os.path.dirname(os.path.abspath(__file__)), "..")
@ -41,6 +42,12 @@ TLS_DIR = "/tmp/firegex_tls"
# The out-of-process worker the proxy engine runs the user's Python in. The engine
# spawns it, not us: it is the one that has to kill it when it stops answering.
PYWORKER = os.path.join(os.path.dirname(os.path.abspath(__file__)), "pyworker.py")
#: What `pyworker.py` writes to stderr when the user's code raises. Spelled here as well
#: as there on purpose: the worker is launched as a script and imports nothing from the
#: backend, which is what keeps the user's code from sharing this process's imports.
#: Kept in step by `tests/integration/test_resilience.py`, which asks for the sentence on
#: both network layers.
PYWORKER_EXCEPTION_MARK = "[exception] [filter]"
#: How long to wait for a binary to acknowledge a configuration push.
ACK_TIMEOUT = 3
@ -127,6 +134,10 @@ class Transport:
#: last report. The engine's own counter resets when it restarts, so somebody
#: above has to add them up if the trace is to outlive the process.
self.on_over_limit = on_over_limit
#: Whether whatever this layer listens on can accept IPv6. Declared here rather
#: than on the one layer that has a listener, so a caller asking a transport
#: what it can accept gets an answer instead of an AttributeError.
self.is_dual_stack: bool = False
self.process: asyncio.subprocess.Process | None = None
self._stderr_pump: asyncio.Task | None = None
@ -204,6 +215,19 @@ class Transport:
text = line.decode(errors="replace")
if "[fatal]" in text:
self._last_fatal = text.split("[fatal]", 1)[1].strip()
# The worker marks a filter that raised, because the verdict it sends
# back is an ordinary ACCEPT and the engine cannot tell that apart
# from a filter that agreed. The traceback beside this line says which
# filter and where; this is what says the traffic went through
# unfiltered, and it is the same sentence the queued layer's
# `EXCEPTION` produces.
if PYWORKER_EXCEPTION_MARK in text:
# A protocol token, like `BLOCKED <id>` on the other channel:
# it is turned into the sentence an operator reads and is not
# forwarded raw, or the log would carry both.
if self.on_exception:
self.on_exception(self.srv.id)
continue
self.on_engine(text)
except (asyncio.CancelledError, asyncio.IncompleteReadError):
pass
@ -629,34 +653,33 @@ class ProxyTransport(Transport):
self._ack: asyncio.Future | None = None
self._cmd_lock = asyncio.Lock()
self.port: int | None = None
self.is_dual_stack: bool = False
#: One relay port per protected address, for UDP. TCP needs none of this: a
#: single listener fronts every address and recovers where each connection was
#: headed from conntrack, which UDP cannot do — see `udp.rs`.
self.udp_ports: dict[str, int] = {}
def _udp_targets(self) -> list:
"""The addresses that each need a relay of their own. Empty for TCP."""
def _udp_targets(self) -> list[tuple[str, int]]:
"""The addresses that each need a relay of their own, resolved. Empty for TCP.
Resolved here and nowhere else, so the relay the engine is asked to open, the
rule that points at it and the key both are filed under are derived from one
answer rather than from three lookups that could each land somewhere different.
"""
if str(self.srv.proto) != L4.UDP:
return []
from modules.services.nftables import resolve_target
from utils import get_interface_ips, is_ip_parse
out = []
for addr in self.srv.addresses:
target = resolve_target(self.srv, addr)
if target:
if not target:
continue
target_ip, target_port = target
if not is_ip_parse(target_ip):
ips = get_interface_ips(target_ip)
if ips:
out.append((ips[0], target_port))
else:
try:
out.append((udp_relay_host(target_ip), target_port))
except NoRelayAddress as e:
raise UnsupportedChain(
f"interface '{target_ip}' has no IP assigned for UDP proxy relay. "
f"Use an IP address or NFQUEUE transport."
)
else:
out.append(target)
f"{e}. Give the service an IP address, or put it on the NFQUEUE "
f"layer, which needs none."
) from e
return out
def _tls_env(self) -> dict:
@ -726,10 +749,7 @@ class ProxyTransport(Transport):
# address is not preserved on that path, which the operator is told
# before choosing it.
"FGEX_PROXY_UDP": ",".join(
f"[{ip}]:{port}" if ":" in str(ip) else f"{ip}:{port}"
for ip, port in (
(one_address(a), p) for a, p in self._udp_targets()
)
udp_relay_key(ip, port) for ip, port in self._udp_targets()
),
# One number for both halves: TCP connections and UDP flows spend
# the same descriptors, and two limits meaning "how much of that may
@ -849,10 +869,13 @@ class ProxyTransport(Transport):
raise Exception(f"the proxy engine rejected the chain: {detail}")
async def add_udp_target(self, ip: str, port: int) -> int:
"""Bind a dedicated UDP relay for a new address without restarting the proxy engine."""
from modules.services.nftables import one_address
clean_ip = one_address(ip)
target = f"[{clean_ip}]:{port}" if ":" in str(clean_ip) else f"{clean_ip}:{port}"
"""Bind a dedicated UDP relay for a new address without restarting the engine.
Idempotent, and deliberately the only way in: the map of relays is this
object's, so a caller that has just added an address asks for the relay and is
handed the port, rather than keeping a second map of its own beside this one.
"""
target = udp_relay_key(udp_relay_host(ip), port)
if target in self.udp_ports:
return self.udp_ports[target]
if not self.process or self.process.returncode is not None:

10
backend/routers/services.py
View File

@ -40,7 +40,6 @@ from modules.services import transports
from modules.services.transports import PROXY_ENGINE, PYWORKER, UnsupportedChain
from utils import (
PortType,
ip_parse,
is_ip_parse,
parse_ip_or_int,
refactor_name,
@ -455,8 +454,6 @@ class DebugResponse(BaseModel):
error: str | None = None
#: Valid, and will run, but cannot be highlighted here.
unscannable: list[DebugErrorModel] = []
#: base64: what the sample becomes, produced by the engine's own rewriting code.
rewritten: str | None = None
truncated: bool
@ -483,6 +480,9 @@ async def startup():
async def shutdown():
await firewall.close()
# The capture interface outlives any one service but not the process: this is the
# only place it goes away, so a `reset()` — which also closes — leaves it alone.
firewall.release_capture()
db.disconnect()
@ -979,7 +979,7 @@ async def edit_service(service_id: str, form: ServiceSettingsForm):
@app.get("/{service_id}/addresses", response_model=list[AddressModel])
async def get_addresses(service_id: str):
row = _service_or_404(service_id)
_service_or_404(service_id)
return [_address_row(a) for a in _addresses(service_id)]
@ -1141,7 +1141,6 @@ async def start_service(service_id: str):
async def stop_service(service_id: str):
_service_or_404(service_id)
await firewall.get(service_id).next(STATUS.STOP)
firewall.release_capture_if_idle()
await refresh_frontend()
return {"status": "ok"}
@ -1963,6 +1962,5 @@ async def debug_regexes(form: DebugForm):
"errors": res.get("errors", []),
"error": res.get("error"),
"unscannable": res.get("unscannable", []),
"rewritten": res.get("rewritten"),
"truncated": res.get("truncated", False),
}

40
backend/utils/__init__.py
View File

@ -11,10 +11,8 @@ except ImportError:
nftables = None
from socketio import AsyncServer
from typing import Annotated, List, Union
from functools import wraps
from pydantic import BaseModel, ValidationError, Field
from pydantic import BaseModel, Field
import traceback
from utils.models import StatusMessageModel
from pathlib import Path
from fastapi import HTTPException, status
@ -234,12 +232,6 @@ def nftables_int_to_json(ip_int):
ip_addr_cidr = int(str(ip_int).split("/")[1])
return {"prefix": {"addr": ip_addr, "len": ip_addr_cidr}}
def nftables_json_to_int(ip_json_int):
if isinstance(ip_json_int,str):
return str(ip_parse(ip_json_int))
else:
return f'{ip_json_int["prefix"]["addr"]}/{ip_json_int["prefix"]["len"]}'
class Singleton(object):
__instance = None
def __new__(class_, *args, **kwargs):
@ -302,36 +294,6 @@ def json_like(obj: BaseModel|List[BaseModel], unset=False, convert_keys:dict[str
return [_json_like(ele, unset=unset, convert_keys=convert_keys, exclude=exclude, mode=mode) for ele in obj]
return _json_like(obj, unset=unset, convert_keys=convert_keys, exclude=exclude, mode=mode)
def register_event(sio_server: AsyncServer, event_name: str, model: BaseModel, response_model: BaseModel|None = None):
def decorator(func):
@sio_server.on(event_name) # Automatically registers the event
@wraps(func)
async def wrapper(sid, data):
try:
# Parse and validate incoming data
parsed_data = model.model_validate(data)
except ValidationError:
return json_like(StatusMessageModel(status=f"Invalid {event_name} request"))
# Call the original function with the parsed data
result = await func(sid, parsed_data)
# If a response model is provided, validate the output
if response_model:
try:
parsed_result = response_model.model_validate(result)
except ValidationError:
traceback.print_exc()
return json_like(StatusMessageModel(status=f"SERVER ERROR: Invalid {event_name} response"))
else:
parsed_result = result
# Emit the validated result
if parsed_result:
if isinstance(parsed_result, BaseModel):
return json_like(parsed_result)
return parsed_result
return wrapper
return decorator
def nicenessify(priority:int, pid:int|None=None):
try:
pid = os.getpid() if pid is None else pid

3
backend/utils/sqlite.py
View File

@ -43,9 +43,6 @@ class SQLite():
if were_active:
self.connect()
def delete_backup(self):
self.__backup = None
def disconnect(self) -> None:
if self.conn:
self.conn.close()

9
docs/pyfilter.md
View File

@ -187,13 +187,13 @@ from firegex.pyfilters.models import HttpRequest
The current HTTP request. This handler is called up to twice: once when the headers are complete, and again once the body is complete (if the whole request arrives in a single TCP packet, it's called only once).
- `method: bytes` — the request method.
- `method: str` — the request method, e.g. `"GET"`. Documented as `bytes` before, which no comparison in a filter ever satisfied.
- `url: str | None` — the request URL.
- `headers: dict[str, str]` — request headers, keys/values exactly as received (case-sensitive); a repeated header becomes a list of values.
- `get_header(header: str, default=None) -> str` — looks up a header case-insensitively; if the header was repeated, its values are joined with a comma (this method never returns a list).
- `user_agent: str`
- `content_encoding: str`
- `content_length: int | None`
- `content_length: int | None` — what the `Content-Length` header declared, or `None` when there was none (a chunked body, or one delimited by the connection closing).
- `body: bytes` — `None` until the body has arrived.
- `body_decoded` — the body decoded according to `content_encoding` (`gzip`, `br`, `deflate` and `zstd` are supported). `False` if decoding failed and `body` isn't `None`.
- `http_version: str`
@ -232,7 +232,8 @@ from firegex.pyfilters.models import HttpResponse
The current HTTP response — same shape and calling convention as `HttpRequest` (up to twice: headers complete, then body complete), plus:
- `status_code: int`
- `status_code: int | None` — the numeric status, e.g. `404`. `None` on a request, and before the response line has been read.
- `status_phrase: str | None` — the reason phrase beside it, e.g. `"Not Found"`. Free text: a server may write anything there, so decide on the code, a header or the body.
All the other fields listed for `HttpRequest` (`headers`, `get_header`, `body`, `body_decoded`, `content_encoding`, `content_length`, `http_version`, `keep_alive`, `should_upgrade`, `upgrading_to_h2`, `upgrading_to_ws`, `ws_stream`, `stream`, `headers_complete`, `message_complete`, `total_size`, `history`, `user_agent`) apply here too.
@ -370,7 +371,7 @@ def mangle_example(packet: RawPacket):
# Higher level of abstraction: parsed HTTP requests.
@pyfilter
def http_filter(http: HttpRequest):
if http.method == b"GET" and http.url and "test" in http.url:
if http.method == "GET" and http.url and "test" in http.url:
return REJECT
# Using history to see previous requests on the same keep-alive stream.

107
docs/services.md
View File

@ -15,7 +15,7 @@ in whatever order you put them, and you can change either without recreating the
A service has a **list of addresses**, not one. The same daemon routinely answers on more
than one — a v4 and a v6 address, a public and an internal one, two ports of the same
process — and all of them deserve the same chain. Add them when you create the service or
afterwards, mixing address families freely.
afterwards, mixing address families — and interface names — freely.
Adding an address to a running service **drops nothing**: the datapath is already up and
already enforcing the chain, so all that happens is that one more address is pointed at
@ -26,15 +26,26 @@ without interrupting existing traffic.
The alternative — one service per address, with the chains copied between them — is how
one of them silently stops being protected the first time a pattern is added to the other.
### Intercepting by IP or network interface
### An address, or the interface it arrives on
An address entry is not limited to a concrete IP (`192.168.1.10:80`, `::1:80`, CIDR `10.0.0.0/24:8080`).
You can also specify a **network interface name directly** — such as `eth0:80`, `wg0:8080`, `tun0:53`, or `lo:80`.
An address entry is a concrete IP (`192.168.1.10:80`, `::1:80`, CIDR `10.0.0.0/24:8080`) or
a **network interface name** (`eth0:80`, `wg0:8080`, `tun0:53`, `lo:80`). The picker lists
both, each interface shown with the addresses it is carrying at that moment, and takes
anything you type that is neither — the address a service will answer on does not have to
exist on this host yet.
Which of the two to use is a question about who decides the address. An IP protects that
one place. An interface protects *whatever address that link is carrying*, matched on the
name, which is what you want when somebody else hands the address out — a VPN the
organisers dial you on, a DHCP lease, a tunnel that comes up mid-round. Nothing has to be
edited when the address changes, and nothing silently stops being protected when it does.
The rules are built differently for each, so the layer matters:
- **On NFQUEUE**: interface matching operates natively in nftables using `meta iifname <iface>` (in prerouting) and `meta oifname <iface>` (in postrouting). Packets arriving at or routed through that interface on the configured port are queued to userspace, regardless of the destination IP.
- **On Proxy (TCP & TLS)**: traffic entering through the interface (`meta iifname <iface>`) is redirected to the proxy listener. The engine connects upstream preserving the destination (`SO_ORIGINAL_DST`) and spoofing the client source IP (`IP_TRANSPARENT`), while policy routing diverts return packets (`meta oifname <iface>`). Local host output redirection is skipped for interfaces to prevent hijacking unrelated outbound host connections.
- **On Proxy (UDP)**: because UDP relays need a concrete destination to bind upstream forwarding, Firegex automatically resolves the interface's primary assigned IP. If the interface has no IP assigned, use NFQUEUE instead.
- **External proxies**: the `external` transport requires specific IP addresses because stateless return rewrites need unambiguous source addresses, so interfaces are disallowed there.
- **External proxies**: the `external` transport takes an IP only. Its return rule recognises your proxy by one address and port to put the original port back, and an interface is not one address — so the interface half of the picker is not offered there at all.
## Transport protocol
@ -63,12 +74,12 @@ allowed to do to the traffic. Pick by those.
| Unit of work | a connection | a packet |
| Transparency | full — dials your service **from the client's own address** | full — the real packets, on their way to your service |
| Reassembly | the kernel's, on two real sockets | rebuilt in userspace with libtins |
| Rewriting | **exact**, any length | patterns: no. Python: yes, unstable on TCP, exact on UDP |
| Rewriting | patterns: no. Python: exact, any length | patterns: no. Python: unstable on TCP, exact on UDP |
| If a filter dies | the engine rebuilds fail-open by hand | **the kernel keeps forwarding**, by itself |
| Cost | terminating a connection, once | **a userspace round trip per packet**, per filter |
| Measured ([how](../tests/README.md#performance)) | **4035 MB/s** at 1 thread, **13 984** at 8 | 1820 at 1 thread, 2956 at 8 |
| Measured ([how](../tests/bench/README.md#performance)) | **4035 MB/s** at 1 thread, **13 984** at 8 | 1820 at 1 thread, 2956 at 8 |
| Short connections | the two are indistinguishable — see below | |
| UDP | yes, fully transparent (exact rewriting, source IP preserved) | yes, fully transparent |
| UDP | yes, fully transparent (source IP preserved, one relay per address) | yes, fully transparent |
| TLS termination | yes | no — decrypting means terminating |
### Proxy
@ -78,8 +89,10 @@ halves.
**What that buys**
- **Exact rewriting.** Two independent connections means a replacement can be any
length: there are no sequence numbers shared with the client to desynchronise. This is
- **Exact rewriting, for a Python filter.** Two independent connections means a
replacement can be any length: there are no sequence numbers shared with the client to
desynchronise. Patterns do not rewrite on any layer — see **Regex** under *Filters* below
for why that was withdrawn. This is
- **Reassembly is the kernel's problem.** Both sides are ordinary sockets, so
out-of-order and retransmitted segments are sorted out before a filter ever sees them
— rather than being rebuilt in userspace with the caveats that come with it.
@ -102,7 +115,7 @@ halves.
win: measured with the layers interleaved, they overlap, and the medians land within
3% of each other. Anyone quoting a number for this axis is quoting noise, this page
included until it was measured properly. Both are in
[tests/README.md](../tests/README.md#performance), with the scripts that produce them.
[tests/bench/README.md](../tests/bench/README.md#performance), with the scripts that produce them.
**What it costs**
@ -136,10 +149,10 @@ Nothing is terminated.
which caps bulk throughput.
- **Reassembly in userspace.** Done via libtins, with memory caps and timeout heuristics.
- **One process per filter.** Chaining requires multiple processes and netfilter queues.
- **Patterns cannot rewrite here.** The matcher reports where a pattern matched; it has
no replacement path. A *Python* filter can mangle on this layer — unstably on TCP,
where changing a payload's length desynchronises the stream (hence `UNSTABLE_MANGLE`),
and **exactly on UDP**, where a datagram carries no sequence numbers to desynchronise.
- **A Python filter's rewriting is unstable on TCP here.** Changing a payload's length
desynchronises the stream, which is what `UNSTABLE_MANGLE` is named after. On **UDP**
it is exact, because a datagram carries no sequence numbers to desynchronise.
Patterns do not rewrite on either layer.
### UDP on the proxy layer
@ -158,9 +171,10 @@ What UDP on the proxy layer provides:
- **Zero-downtime dynamic address addition.** Adding new addresses to a running UDP service
dynamically binds new relay sockets through the control channel without restarting the
engine or dropping existing connections.
- **Exact rewriting, at any length.** A datagram is self-contained, so there are no
sequence numbers for a longer or shorter payload to desynchronise. The caveat that
makes rewriting unstable on the NFQUEUE layer simply does not apply.
- **Exact rewriting, at any length, for a Python filter.** A datagram is
self-contained, so there are no sequence numbers for a longer or shorter payload to
desynchronise. The caveat that makes rewriting unstable on the NFQUEUE layer simply
does not apply.
- **Per-flow filter state.** Each client address is a flow with its own filter state and
its own Python module globals, released after a minute of silence — UDP has no close
to observe, so a timeout is the only thing that can end one.
@ -353,11 +367,11 @@ client's verdict.
Each pattern has a direction — client to service, service to client, or both. A leaked
flag shows up on the way out; an exploit on the way in.
A pattern **blocks** the connection:
A pattern **blocks** the connection. That is the only thing it does:
```
FLAG\{[a-z0-9]+\} service → client → FLAG{redacted}
X-Debug: [^\r\n]* client → service → X-Debug: off
FLAG\{[a-z0-9]+\} service → client block
X-Debug: [^\r\n]* client → service block
../ both ways block
```
@ -371,19 +385,19 @@ it is active — leaves the counters alone. The new pattern is checked by the en
before it is stored, and a rule that would collide with another in the same filter is
refused rather than silently duplicated.
Rewriting is proxy-only, and not because it was left unimplemented: on NFQUEUE a
replacement of a different length desynchronises the stream, and the connection breaks
some time later in a way nobody would trace back to the rule. Starting an NFQUEUE
service with a rewriting pattern is refused, saying so.
**A pattern cannot rewrite, on any layer.** It used to be able to, and the reason it no
longer can is worth stating rather than leaving as a missing feature. Rewriting scanned
one chunk at a time, because bytes already forwarded cannot be taken back — so a match
straddling two chunks was never rewritten. Measured: a pattern sent whole reached the
service redacted, and the *same* pattern split across two TCP segments reached it intact.
Blocking, which scans in hyperscan's stream mode, caught the split version.
Two things to know about a rewrite:
- **The replacement is literal.** No `$1` capture references — hyperscan reports where a
match is, not what its groups captured, and inventing groups would mean a second engine
deciding them.
- **It works within one chunk of the stream.** Bytes already forwarded cannot be taken
back, so a match split across two of them is not rewritten. Blocking has no such limit,
because a block can still refuse the connection after the fact.
What made that unacceptable was not the miss but its silence: no block, no log line, no
counter — just a rule the operator believed was applying. A filter that works on the
sender's segmentation is worse than no filter, because it is trusted. Blocking is the
verdict a pattern can honestly reach on a stream, so it is the only one offered. To
change bytes rather than refuse them, use a Python filter, which sees the reassembled
stream and can say so.
### Python (pyfilter)
@ -438,14 +452,11 @@ pattern against it and find out mid-round that it was never valid. Here, a patte
tester accepts is one you can save, and a pattern it rejects tells you why in hyperscan's
own words.
The same holds for rewriting: set a pattern to rewrite and the tester shows you what the
sample **becomes**, produced by running the engine's own rewriting code over it rather
than by describing what it would do.
Two details follow from being honest rather than approximate. A pattern is judged against
the mode it will actually run in — blocking matches live streams, rewriting scans blocks,
and hyperscan does not accept quite the same patterns in both — so the tester's verdict is
the one the datapath will reach. And occasionally a pattern is valid, will run, and still
One detail follows from being honest rather than approximate. A pattern is judged against
the mode it will actually run in — stream matching, which is how a pattern follows a
connection across chunk boundaries, and hyperscan does not accept quite the same patterns
in every mode — so the tester's verdict is the one the datapath will reach. And
occasionally a pattern is valid, will run, and still
cannot be highlighted here, because showing *where* a match starts needs the scan mode
that accepts slightly less. The tester says so instead of silently showing no matches.
@ -586,9 +597,9 @@ The ruleset file is **the same shape firegex uses**, so one file works in both p
```json
[
{"id": "traversal", "pattern": "\\.\\./", "direction": "c2s", "action": "block"},
{"id": "traversal", "pattern": "\\.\\./", "direction": "c2s"},
{"id": "flag-leak", "pattern": "FLAG\\{[a-z0-9]+\\}", "direction": "s2c",
"action": "rewrite", "with": "FLAG{redacted}"}
"case_sensitive": false}
]
```
@ -597,11 +608,11 @@ pattern it accepts is one firegex accepts, and one it rejects reports hyperscan'
message. Where hyperscan is not installed it says so and stops, rather than quietly
answering with a different engine.
Two honest limits. `fgex regex proxy` matches **per chunk**, while a blocking rule on a
real service matches across the whole stream — so a pattern split over two reads is
caught in production and not in the simulator, never the other way round. And the
simulator terminates the connection itself, so it always has the exact rewriting the
proxy layer has, even when you are aiming at an NFQUEUE service.
One honest limit. `fgex regex proxy` matches **per chunk**, while a rule on a real
service matches across the whole stream — so a pattern split over two reads is caught in
production and not in the simulator, never the other way round. A ruleset that looks
clean here may still block something on a live service; one that blocks here always
would.
## Editing while it runs

4
fgex-lib/MANIFEST.in
View File

@ -1 +1,5 @@
include requirements.txt
recursive-include llhttp *.c *.h
include llhttp/PROVENANCE.md
include llhttp/LICENSE.pyllhttp
include llhttp/lib/LICENSE.llhttp

12
fgex-lib/firegex/cli.py
View File

@ -38,10 +38,6 @@ def test_connection(host, port, use_ipv6=False):
sock.close()
@app.command("pyfilters", help="Run your Python filters against a real service, locally")
# `nfproxy` is what this was called when it was one of several modules. Kept as a second
# name rather than removed: it is in people's shell history and in scripts, and a command
# that has quietly stopped existing is a worse answer than one that still works.
@app.command("nfproxy", hidden=True)
def pyfilters(
filter_file: str = typer.Argument(..., help="The path to the filter file"),
address: str = typer.Argument(..., help="The address of the target to proxy"),
@ -155,16 +151,12 @@ def regex_test(
ruleset = Ruleset(rules)
is_input = not from_service
blocked_by, payload = ruleset.apply(sample, is_input)
blocked_by = ruleset.apply(sample, is_input)
if blocked_by is not None:
print(f"[bold red]blocked[/] by [bold]{escape(blocked_by)}[/]")
close_cli(0)
if payload != sample:
print("[bold yellow]rewritten[/]")
print(escape(payload.decode(errors="replace")))
close_cli(0)
print("[bold green]passed through unchanged[/]")
print("[bold green]passed through[/]")
@regex_app.command("proxy", help="Run a local proxy applying a ruleset, like `fgex pyfilters`")

48
fgex-lib/firegex/nfproxy/__init__.py
View File

@ -1,48 +0,0 @@
"""The old name for `firegex.pyfilters`, kept working.
The feature is called pyfilters. It was called nfproxy when it was one of several
modules, and that name is at the top of every filter file anyone has already written —
including the ones sitting in a running firegex's database, which would stop loading the
moment the import failed. So the old path still resolves, and resolves to *the same
objects*.
That last part is the whole point, and why this is `sys.modules` aliasing rather than a
file of re-exports. `type_annotations_associations` maps model **classes** to protocols,
and both the library and the datapath decide when to call a filter by looking the
annotation up in that table. A shim that built second copies of `HttpRequest` and friends
would leave a filter written against the old path annotating a class the table has never
heard of: it would import cleanly and then never be called, or be refused as speaking an
unknown protocol. Aliasing the modules keeps one class per model, so `firegex.nfproxy`
and `firegex.pyfilters` are two spellings of one thing rather than two libraries.
"""
import sys
from firegex import pyfilters as _pyfilters
# Every submodule, under both names. Listed rather than walked: an explicit list is what
# makes it obvious at review time which import paths are promised to keep working, and a
# walk would quietly start promising whatever happened to be added next.
for _name in (
"",
".models",
".models.http",
".models.tcp",
".internals",
".internals.data",
".internals.models",
".internals.exceptions",
".proxysim",
):
_module = sys.modules.get("firegex.pyfilters" + _name)
if _module is None:
import importlib
_module = importlib.import_module("firegex.pyfilters" + _name)
sys.modules["firegex.nfproxy" + _name] = _module
# `firegex.nfproxy` itself is this module, not the aliased one, or `import firegex.nfproxy`
# would hand back a package whose `__name__` says pyfilters and whose docstring does not
# explain why. Its contents are the same objects either way.
from firegex.pyfilters import * # noqa: F401,F403,E402
from firegex.pyfilters import __all__ # noqa: F401,E402

3
fgex-lib/firegex/pyfilters/internals/data.py
View File

@ -1,5 +1,4 @@
from firegex.pyfilters.internals.models import FilterHandler
from firegex.pyfilters.internals.models import FullStreamAction, ExceptionAction
from firegex.pyfilters.internals.models import ExceptionAction, FullStreamAction
class RawPacket:
"""One chunk of a connection, with what is known about where it came from.

41
fgex-lib/firegex/pyfilters/internals/zstd_compat.py Normal file
View File

@ -0,0 +1,41 @@
"""One `zstd.decompress`, whichever Python is underneath.
`compression.zstd` arrived in the standard library in Python 3.14. The library imported
it directly, while `setup.py` promised 3.10 — so on everything from 3.10 to 3.13 the
package installed cleanly and then raised `ModuleNotFoundError: No module named
'compression'` on the first import of an HTTP model. That was the whole of what stopped
it running on Python 3.12, which is what Ubuntu 24.04 LTS ships.
The `zstandard` package fills the gap, and the shim is not a one-line alias because the
two do **not** answer the same question. `zstandard.decompress()` refuses a frame whose
header carries no content size — which is exactly the shape a server produces when it
compresses a response body on the fly, the common case here — while the standard
library's handles it. A filter that reads a body on 3.14 and gets nothing on 3.12 is the
same class of lie as a tester that disagrees with the engine, so this normalises to the
more capable behaviour rather than to whichever backend is present.
"""
try: # Python 3.14 and later
from compression.zstd import decompress as _decompress
except ImportError: # Python 3.10 - 3.13
import io
import zstandard
def _decompress(data: bytes) -> bytes:
"""Every frame in `data`, streamed.
`stream_reader` rather than `ZstdDecompressor.decompress`: the latter needs the
content size in the frame header and raises without it. `read_across_frames`
matches the standard library, which decompresses a concatenation of frames as
one body — an encoder that flushed mid-response produces exactly that.
"""
reader = zstandard.ZstdDecompressor().stream_reader(
io.BytesIO(data), read_across_frames=True
)
return reader.read()
def decompress(data: bytes) -> bytes:
return _decompress(data)

93
fgex-lib/firegex/pyfilters/models/http.py
View File

@ -1,4 +1,4 @@
import pyllhttp
from firegex import _llhttp
from firegex.pyfilters.internals.exceptions import NotReadyToRun
from firegex.pyfilters.internals.data import DataStreamCtx
from firegex.pyfilters.internals.exceptions import (
@ -10,7 +10,7 @@ from firegex.pyfilters.internals.exceptions import (
from firegex.pyfilters.internals.models import FullStreamAction, ExceptionAction
from dataclasses import dataclass, field
from collections import deque
from compression import zstd
from firegex.pyfilters.internals import zstd_compat
import gzip
import io
import zlib
@ -18,7 +18,7 @@ import brotli
import traceback
from websockets.frames import Frame
from websockets.extensions.permessage_deflate import PerMessageDeflate
from pyllhttp import PAUSED_H2_UPGRADE, PAUSED_UPGRADE
from firegex._llhttp import PAUSED_H2_UPGRADE, PAUSED_UPGRADE
@dataclass
@ -34,7 +34,13 @@ class InternalHTTPMessage:
body_decoded: bool = field(default=False)
headers_complete: bool = field(default=False)
message_complete: bool = field(default=False)
#: The reason phrase, as the server wrote it. Free text.
status: str | None = field(default=None)
#: The numeric status. Separate from the phrase because they are different things,
#: and because for a long time only the phrase was reachable at all: the binding
#: never exposed `llhttp`'s status code, so `status_code` on the public model handed
#: back the phrase while the documentation promised an int.
status_code: int | None = field(default=None)
total_size: int = field(default=0)
user_agent: str = field(default_factory=str)
content_encoding: str = field(default=str)
@ -149,6 +155,7 @@ class InternalCallbackHandler:
self.buffers._current_header_value = b""
self.msg.headers_complete = True
self.msg.method = self.method_parsed
self.msg.status_code = self.status_code
self.msg.content_length = self.content_length_parsed
self.msg.should_upgrade = self.should_upgrade
self.msg.keep_alive = self.keep_alive
@ -173,12 +180,24 @@ class InternalCallbackHandler:
if not enc:
continue
if enc == "deflate":
# Both spellings, because both are on the wire. RFC 7230 defines
# `deflate` as the zlib format of RFC 1950, and plenty of servers send
# the raw stream instead — browsers accept either, and a decoder that
# takes only one lets half the compressed traffic reach a filter still
# compressed, where a pattern finds nothing and the only trace is a line
# on stdout saying it skipped.
for wbits in (-zlib.MAX_WBITS, zlib.MAX_WBITS):
try:
decompress = zlib.decompressobj(-zlib.MAX_WBITS)
decoding_body = decompress.decompress(decoding_body)
decoding_body += decompress.flush()
except Exception as e:
print(f"Error decompressing deflate: {e}: skipping", flush=True)
decompress = zlib.decompressobj(wbits)
decoded = decompress.decompress(decoding_body)
decoded += decompress.flush()
except Exception:
continue
decoding_body = decoded
break
else:
print("Error decompressing deflate: neither raw nor zlib-wrapped: "
"skipping", flush=True)
decode_success = False
break
elif enc == "br":
@ -201,7 +220,7 @@ class InternalCallbackHandler:
break
elif enc == "zstd":
try:
decoding_body = zstd.decompress(decoding_body)
decoding_body = zstd_compat.decompress(decoding_body)
except Exception as e:
print(f"Error decompressing zstd: {e}: skipping", flush=True)
decode_success = False
@ -243,10 +262,15 @@ class InternalCallbackHandler:
@property
def http_version(self) -> str:
if self.major and self.minor:
return f"{self.major}.{self.minor}"
else:
"""`"1.1"`, `"1.0"`, `""` if the parser has not read a version yet.
Written `if self.major and self.minor` before — so a minor of zero is falsy and
HTTP/1.0, and HTTP/2.0, both reported the empty string. A filter keying on the
version never saw either, and nothing said why.
"""
if self.major is None or self.minor is None:
return ""
return f"{self.major}.{self.minor}"
@property
def method_parsed(self) -> str:
@ -376,19 +400,19 @@ class InternalCallbackHandler:
return f"<InternalCallbackHandler msg={self.msg} buffers={self.buffers} save_body={self.save_body} raised_error={self.raised_error} has_begun={self.has_begun} messages={self.messages}>"
class InternalHttpRequest(InternalCallbackHandler, pyllhttp.Request):
class InternalHttpRequest(InternalCallbackHandler, _llhttp.Request):
def __init__(self):
super(InternalCallbackHandler, self).__init__()
super(pyllhttp.Request, self).__init__()
super(_llhttp.Request, self).__init__()
def _is_input(self):
return True
class InternalHttpResponse(InternalCallbackHandler, pyllhttp.Response):
class InternalHttpResponse(InternalCallbackHandler, _llhttp.Response):
def __init__(self):
super(InternalCallbackHandler, self).__init__()
super(pyllhttp.Response, self).__init__()
super(_llhttp.Response, self).__init__()
def _is_input(self):
return False
@ -676,11 +700,19 @@ class InternalBasicHttpMetaClass:
if (
not internal_data.call_mem["headers_were_set"]
and parser.msg.headers_complete
and not parser.msg.message_complete
and parser.release_message_headers
):
messages_tosend.append(
parser.msg
) # Also the current message needs to be sent due to complete headers
# The message still being read, handed over early because its headers are
# already known — which is what lets a filter act before a body arrives.
#
# `not message_complete` is the load-bearing half. `on_message_complete`
# appends the finished message to the queue but leaves `parser.msg` pointing
# at it, so without this a packet carrying two pipelined messages delivered
# the second one twice: once from the queue and once from here. Blocking is
# idempotent and survived that, which is why it went unnoticed — anything
# that counts did not.
messages_tosend.append(parser.msg)
if parser._packet_to_stream():
messages_tosend.append(
@ -776,8 +808,23 @@ class HttpResponse(InternalBasicHttpMetaClass):
return not internal_data.current_pkt.is_input
@property
def status_code(self) -> int:
"""Status code of the response"""
def status_code(self) -> int | None:
"""The numeric status, e.g. `404`.
This used to hand back the reason phrase while being documented and annotated as
an int, so `if res.status_code == 500` never matched and nothing said why. It was
not carelessness: the binding did not expose llhttp's status code, and there was
no way to reach it. The binding ships with this package now and does.
"""
return self._message.status_code
@property
def status_phrase(self) -> str | None:
"""The reason phrase, e.g. `"Not Found"`.
Free text — a server may write anything here — so read it and decide on the code
above, a header, or the body.
"""
return self._message.status
@ -786,7 +833,7 @@ class HttpResponse(InternalBasicHttpMetaClass):
return "http_module"
def __repr__(self):
return f"<HttpResponse status_code={self.status_code} url={self.url} headers={self.headers} body=[{0 if not self.body else len(self.body)} bytes] http_version={self.http_version} keep_alive={self.keep_alive} should_upgrade={self.should_upgrade} headers_complete={self.headers_complete} message_complete={self.message_complete} content_length={self.content_length} stream={self.stream} ws_stream={self.ws_stream}>"
return f"<HttpResponse status_code={self.status_code} status_phrase={self.status_phrase!r} url={self.url} headers={self.headers} body=[{0 if not self.body else len(self.body)} bytes] http_version={self.http_version} keep_alive={self.keep_alive} should_upgrade={self.should_upgrade} headers_complete={self.headers_complete} message_complete={self.message_complete} content_length={self.content_length} stream={self.stream} ws_stream={self.ws_stream}>"
class HttpFullRequest(HttpRequest):
@ -828,7 +875,7 @@ class HttpFullResponse(HttpResponse):
return "http_full"
def __repr__(self):
return f"<HttpFullResponse status_code={self.status_code} url={self.url} headers={self.headers} body=[{0 if not self.body else len(self.body)} bytes] http_version={self.http_version} keep_alive={self.keep_alive} should_upgrade={self.should_upgrade} headers_complete={self.headers_complete} message_complete={self.message_complete} content_length={self.content_length} stream={self.stream} ws_stream={self.ws_stream}>"
return f"<HttpFullResponse status_code={self.status_code} status_phrase={self.status_phrase!r} url={self.url} headers={self.headers} body=[{0 if not self.body else len(self.body)} bytes] http_version={self.http_version} keep_alive={self.keep_alive} should_upgrade={self.should_upgrade} headers_complete={self.headers_complete} message_complete={self.message_complete} content_length={self.content_length} stream={self.stream} ws_stream={self.ws_stream}>"

73
fgex-lib/firegex/regex/__init__.py
View File

@ -126,28 +126,36 @@ class Direction(str, Enum):
return (self is Direction.C2S) == is_input
class Action(str, Enum):
BLOCK = "block"
@dataclass
class Rule:
"""One pattern, in the same shape firegex stores it."""
"""One pattern, in the same shape firegex stores it.
There is no action to choose. A matching pattern refuses the connection, here and in
firegex, because that is the only verdict a pattern can honestly reach on a stream:
rewriting scanned one chunk at a time, so a match straddling two of them was never
rewritten, and the operator got no block, no log and no counter to say so. A ruleset
file may still carry `"action": "block"` — it is read and ignored — but anything else
is refused rather than quietly treated as a block.
"""
id: str
pattern: str
direction: Direction = Direction.BOTH
case_sensitive: bool = True
action: Action = Action.BLOCK
@classmethod
def from_dict(cls, raw: dict) -> "Rule":
action = str(raw.get("action", "block")).lower()
if action != "block":
raise ValueError(
f"rule {raw.get('id', raw.get('regex_id', '?'))!r} asks for "
f"action {action!r}; a pattern can only block"
)
return cls(
id=str(raw.get("id", raw.get("regex_id", "?"))),
pattern=raw["pattern"] if "pattern" in raw else raw["regex"],
direction=Direction(raw.get("direction", "both")),
case_sensitive=bool(raw.get("case_sensitive", True)),
action=Action(raw.get("action", "block")),
)
@ -228,9 +236,10 @@ class Database:
def validate(rule: Rule) -> str | None:
"""Would firegex accept this rule? Returns hyperscan's own message if not.
Checked against the mode it would actually run in: a blocking pattern is compiled
for stream matching and a rewriting one for block scanning, and hyperscan does not
accept quite the same patterns in both.
Checked against the mode it will actually run in — stream matching, which is how a
pattern follows a connection across chunk boundaries — because hyperscan does not
accept quite the same patterns in every mode, and judging by the wrong one would
refuse a rule that works.
"""
mode = HS_MODE_STREAM
try:
@ -251,38 +260,32 @@ class Ruleset:
def __init__(self, rules: list[Rule]):
self.rules = list(rules)
self._cache: dict[tuple[bool, Action], Database | None] = {}
self._cache: dict[bool, Database | None] = {}
def _db(self, is_input: bool, action: Action) -> Database | None:
key = (is_input, action)
if key not in self._cache:
selected = [
r for r in self.rules
if r.action is action and r.direction.covers(is_input)
]
self._cache[key] = Database(selected, HS_MODE_BLOCK) if selected else None
return self._cache[key]
def _db(self, is_input: bool) -> Database | None:
if is_input not in self._cache:
selected = [r for r in self.rules if r.direction.covers(is_input)]
self._cache[is_input] = Database(selected, HS_MODE_BLOCK) if selected else None
return self._cache[is_input]
def apply(self, data: bytes, is_input: bool) -> tuple[str | None, bytes]:
def apply(self, data: bytes, is_input: bool) -> str | None:
"""Run the ruleset over one chunk, as the proxy datapath would.
Returns the id of the rule that refused it, or `None` and the payload to
forward — rewritten if any rewriting rule matched. Blocking is decided before
rewriting, so a chunk that is going to be refused is not rewritten on its way
there.
Returns the id of the rule that refused the chunk, or `None` if it may be
forwarded as it stands. Nothing is ever handed back changed: a pattern blocks,
and that is the whole of what it can do.
One caveat this cannot hide: matching here is per chunk, while a *blocking* rule
on a real service matches across the whole stream. A pattern split over two
reads is caught there and not here, so a rule that does not fire in the
simulator may still fire in production. Rewriting has the same scope in both,
because bytes already forwarded cannot be taken back either way.
One caveat this cannot hide: matching here is per chunk, while a rule on a real
service matches across the whole stream. A pattern split over two reads is
caught there and not here — never the other way round, so a ruleset that looks
clean in the simulator may still block something in production.
"""
blocking = self._db(is_input, Action.BLOCK)
if blocking:
hits = blocking.scan(data, limit=1)
db = self._db(is_input)
if db:
hits = db.scan(data, limit=1)
if hits:
return hits[0].rule.id, data
return None, data
return hits[0].rule.id
return None
def load_rules(raw: list[dict]) -> list[Rule]:

8
fgex-lib/firegex/regex/sim.py
View File

@ -35,19 +35,17 @@ async def _pump(rules: Ruleset, reader, writer, is_input: bool, on_block):
if not data:
break
try:
blocked_by, payload = rules.apply(data, is_input)
blocked_by = rules.apply(data, is_input)
except Exception as e:
# A ruleset that blows up mid-connection must not take the traffic with
# it; the real datapath fails open too, and so does this.
_log("filter", f"failed on this chunk ({escape(str(e))}), forwarding it", "ERROR")
blocked_by, payload = None, data
blocked_by = None
if blocked_by is not None:
on_block(blocked_by)
_log("block", f"connection refused by [bold]{escape(blocked_by)}[/]", "WARNING")
break
if payload != data:
_log("rewrite", f"{escape(repr(data)[:60])} -> {escape(repr(payload)[:60])}")
writer.write(payload)
writer.write(data)
await writer.drain()
finally:
writer.close()

21
fgex-lib/llhttp/LICENSE.pyllhttp Normal file
View File

@ -0,0 +1,21 @@
MIT License
Copyright (c) 2020 Derrick Lyndon Pallas <derrick@pallas.us>
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.

68
fgex-lib/llhttp/PROVENANCE.md Normal file
View File

@ -0,0 +1,68 @@
# Where the code in this directory comes from
Two separate works live here, under two separate MIT notices, and the difference
matters when either is updated.
## `lib/` — llhttp, upstream and unmodified
* Version: **9.4.3**
* Source: <https://github.com/nodejs/llhttp>, tag `release/v9.4.3` (30 July 2026)
* Tarball: `https://github.com/nodejs/llhttp/archive/refs/tags/release/v9.4.3.tar.gz`
(`sha256:1eb813c7437b31a87496a1cd3ed79f00746720f5e7e29c79b42c02cb69f36c39`)
* Licence: MIT, © Fedor Indutny and contributors — `lib/LICENSE.llhttp`
Four files, copied verbatim from that tag, and nothing in them has been edited:
| here | upstream | sha256 |
|---|---|---|
| `lib/llhttp.h` | `include/llhttp.h` | `5bc82fa51b19aa8bee7d921038393fbfc74e8cc9bae0844c2d82c102fb8cfd68` |
| `lib/llhttp.c` | `src/llhttp.c` | `391e7c99912abf3b1c9dd8a9c85fdeaac2663e9fa55b6f3871154d16fef8b892` |
| `lib/api.c` | `src/api.c` | `0f8590206fe2f264db2825401b5fb856b979ee1363dc88b06f36dc9577afb941` |
| `lib/http.c` | `src/http.c` | `a1f2b23168f8e9b5bfa464c89027d3ca259fc649ae3d7e72bfff997d1aeb5d72` |
`llhttp.c` is *generated* upstream, from the TypeScript that describes the state
machine; the `release/*` tags are the branch where that output is committed, which is
why these can be taken as source without a Node toolchain. There is no `api.h`: the
public header is one file and already carries what that used to declare. It was here
until 9.4.3, included by nothing.
### Updating
Replace those four files from the new tag, update the table above, rebuild, and run
`pytest unit/test_http_parsing.py` from `tests/`. The binding needs no change for a
minor release: `parser_settings` is built with designated initialisers, so a callback
added upstream (`on_protocol_complete` arrived in 9.3.0) stays NULL, and the error list
is generated from `HTTP_ERRNO_MAP`, so new codes bring their own exceptions along.
What does need looking at is the other direction: each release changes what the parser
*accepts*. Going from 9.2.1 to 9.4.3, an empty `Transfer-Encoding` stopped being
accepted and tabs around `Content-Length` started being. Both are pinned in
`tests/unit/test_http_parsing.py` — a change there is a decision about what filters see,
not a test to fix.
## `llhttp_module.c` — the Python binding, ours
Derived from [pyllhttp](https://github.com/domysh/pyllhttp) (itself a fork of Derrick
Lyndon Pallas' `pyllhttp`, now archived), and modified. Licence: MIT —
`LICENSE.pyllhttp`.
What differs from that fork, which is what to re-apply if anything is ever taken from it
again:
* It is built as `firegex._llhttp` and declares `Py_MOD_PER_INTERPRETER_GIL_SUPPORTED`,
which is what lets it load in the subinterpreters `cpproxy` runs filters in — the
reason none of the alternatives with prebuilt wheels can be used.
* **Nothing here reaches for `pyllhttp` any more.** The types and the exception classes
carry this module's name, and the error path looks the class up in this module through
`sys.modules`, so each interpreter finds the classes it built. It used to
`PyImport_ImportModule("pyllhttp")` on every parse error: that package is not a
dependency and is installed on no machine the wheel is, so the import failed, printed
its traceback to stderr — the service log, on a filter — and returned *without setting
an exception*, which CPython reports as `SystemError: ... returned NULL without setting
an exception`. A malformed request said that instead of naming the parse error. Pinned
by `tests/unit/test_llhttp_binding.py`, which takes `pyllhttp` away before asking, and
by `wheel_smoke.py`, which `cibuildwheel` runs where it was never there.
* The exception class names are derived into a local buffer instead of being `malloc`ed
once onto a process-global table and never freed — that table is shared by every
interpreter, which is not somewhere a module claiming per-interpreter support may
write.

22
fgex-lib/llhttp/lib/LICENSE.llhttp Normal file
View File

@ -0,0 +1,22 @@
MIT License
Copyright © 2018 Fedor Indutny
Permission is hereby granted, free of charge, to any person obtaining a
copy of this software and associated documentation files (the
"Software"), to deal in the Software without restriction, including
without limitation the rights to use, copy, modify, merge, publish,
distribute, sublicense, and/or sell copies of the Software, and to permit
persons to whom the Software is furnished to do so, subject to the
following conditions:
The above copyright notice and this permission notice shall be included
in all copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS
OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN
NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM,
DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR
OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE
USE OR OTHER DEALINGS IN THE SOFTWARE.

517
fgex-lib/llhttp/lib/api.c Normal file
View File

@ -0,0 +1,517 @@
#include <stdlib.h>
#include <stdio.h>
#include <string.h>
#include "llhttp.h"
#define CALLBACK_MAYBE(PARSER, NAME) \
do { \
const llhttp_settings_t* settings; \
settings = (const llhttp_settings_t*) (PARSER)->settings; \
if (settings == NULL || settings->NAME == NULL) { \
err = 0; \
break; \
} \
err = settings->NAME((PARSER)); \
} while (0)
#define SPAN_CALLBACK_MAYBE(PARSER, NAME, START, LEN) \
do { \
const llhttp_settings_t* settings; \
settings = (const llhttp_settings_t*) (PARSER)->settings; \
if (settings == NULL || settings->NAME == NULL) { \
err = 0; \
break; \
} \
err = settings->NAME((PARSER), (START), (LEN)); \
if (err == -1) { \
err = HPE_USER; \
llhttp_set_error_reason((PARSER), "Span callback error in " #NAME); \
} \
} while (0)
void llhttp_init(llhttp_t* parser, llhttp_type_t type,
const llhttp_settings_t* settings) {
llhttp__internal_init(parser);
parser->type = type;
parser->settings = (void*) settings;
}
#if defined(__wasm__)
extern int wasm_on_message_begin(llhttp_t * p);
extern int wasm_on_url(llhttp_t* p, const char* at, size_t length);
extern int wasm_on_status(llhttp_t* p, const char* at, size_t length);
extern int wasm_on_header_field(llhttp_t* p, const char* at, size_t length);
extern int wasm_on_header_value(llhttp_t* p, const char* at, size_t length);
extern int wasm_on_headers_complete(llhttp_t * p, int status_code,
uint8_t upgrade, int should_keep_alive);
extern int wasm_on_body(llhttp_t* p, const char* at, size_t length);
extern int wasm_on_message_complete(llhttp_t * p);
static int wasm_on_headers_complete_wrap(llhttp_t* p) {
return wasm_on_headers_complete(p, p->status_code, p->upgrade,
llhttp_should_keep_alive(p));
}
const llhttp_settings_t wasm_settings = {
.on_message_begin = wasm_on_message_begin,
.on_url = wasm_on_url,
.on_status = wasm_on_status,
.on_header_field = wasm_on_header_field,
.on_header_value = wasm_on_header_value,
.on_headers_complete = wasm_on_headers_complete_wrap,
.on_body = wasm_on_body,
.on_message_complete = wasm_on_message_complete,
};
llhttp_t* llhttp_alloc(llhttp_type_t type) {
llhttp_t* parser = malloc(sizeof(llhttp_t));
llhttp_init(parser, type, &wasm_settings);
return parser;
}
void llhttp_free(llhttp_t* parser) {
free(parser);
}
#endif // defined(__wasm__)
/* Some getters required to get stuff from the parser */
uint8_t llhttp_get_type(llhttp_t* parser) {
return parser->type;
}
uint8_t llhttp_get_http_major(llhttp_t* parser) {
return parser->http_major;
}
uint8_t llhttp_get_http_minor(llhttp_t* parser) {
return parser->http_minor;
}
uint8_t llhttp_get_method(llhttp_t* parser) {
return parser->method;
}
int llhttp_get_status_code(llhttp_t* parser) {
return parser->status_code;
}
uint8_t llhttp_get_upgrade(llhttp_t* parser) {
return parser->upgrade;
}
void llhttp_reset(llhttp_t* parser) {
llhttp_type_t type = parser->type;
const llhttp_settings_t* settings = parser->settings;
void* data = parser->data;
uint16_t lenient_flags = parser->lenient_flags;
llhttp__internal_init(parser);
parser->type = type;
parser->settings = (void*) settings;
parser->data = data;
parser->lenient_flags = lenient_flags;
}
llhttp_errno_t llhttp_execute(llhttp_t* parser, const char* data, size_t len) {
return llhttp__internal_execute(parser, data, data + len);
}
void llhttp_settings_init(llhttp_settings_t* settings) {
memset(settings, 0, sizeof(*settings));
}
llhttp_errno_t llhttp_finish(llhttp_t* parser) {
int err;
/* We're in an error state. Don't bother doing anything. */
if (parser->error != 0) {
return 0;
}
switch (parser->finish) {
case HTTP_FINISH_SAFE_WITH_CB:
CALLBACK_MAYBE(parser, on_message_complete);
if (err != HPE_OK) return err;
/* FALLTHROUGH */
case HTTP_FINISH_SAFE:
return HPE_OK;
case HTTP_FINISH_UNSAFE:
parser->reason = "Invalid EOF state";
return HPE_INVALID_EOF_STATE;
default:
abort();
}
}
void llhttp_pause(llhttp_t* parser) {
if (parser->error != HPE_OK) {
return;
}
parser->error = HPE_PAUSED;
parser->reason = "Paused";
}
void llhttp_resume(llhttp_t* parser) {
if (parser->error != HPE_PAUSED) {
return;
}
parser->error = 0;
}
void llhttp_resume_after_upgrade(llhttp_t* parser) {
if (parser->error != HPE_PAUSED_UPGRADE) {
return;
}
parser->error = 0;
}
llhttp_errno_t llhttp_get_errno(const llhttp_t* parser) {
return parser->error;
}
const char* llhttp_get_error_reason(const llhttp_t* parser) {
return parser->reason;
}
void llhttp_set_error_reason(llhttp_t* parser, const char* reason) {
parser->reason = reason;
}
const char* llhttp_get_error_pos(const llhttp_t* parser) {
return parser->error_pos;
}
const char* llhttp_errno_name(llhttp_errno_t err) {
#define HTTP_ERRNO_GEN(CODE, NAME, _) case HPE_##NAME: return "HPE_" #NAME;
switch (err) {
HTTP_ERRNO_MAP(HTTP_ERRNO_GEN)
default: abort();
}
#undef HTTP_ERRNO_GEN
}
const char* llhttp_method_name(llhttp_method_t method) {
#define HTTP_METHOD_GEN(NUM, NAME, STRING) case HTTP_##NAME: return #STRING;
switch (method) {
HTTP_ALL_METHOD_MAP(HTTP_METHOD_GEN)
default: abort();
}
#undef HTTP_METHOD_GEN
}
const char* llhttp_status_name(llhttp_status_t status) {
#define HTTP_STATUS_GEN(NUM, NAME, STRING) case HTTP_STATUS_##NAME: return #STRING;
switch (status) {
HTTP_STATUS_MAP(HTTP_STATUS_GEN)
default: abort();
}
#undef HTTP_STATUS_GEN
}
void llhttp_set_lenient_headers(llhttp_t* parser, int enabled) {
if (enabled) {
parser->lenient_flags |= LENIENT_HEADERS;
} else {
parser->lenient_flags &= ~LENIENT_HEADERS;
}
}
void llhttp_set_lenient_chunked_length(llhttp_t* parser, int enabled) {
if (enabled) {
parser->lenient_flags |= LENIENT_CHUNKED_LENGTH;
} else {
parser->lenient_flags &= ~LENIENT_CHUNKED_LENGTH;
}
}
void llhttp_set_lenient_keep_alive(llhttp_t* parser, int enabled) {
if (enabled) {
parser->lenient_flags |= LENIENT_KEEP_ALIVE;
} else {
parser->lenient_flags &= ~LENIENT_KEEP_ALIVE;
}
}
void llhttp_set_lenient_transfer_encoding(llhttp_t* parser, int enabled) {
if (enabled) {
parser->lenient_flags |= LENIENT_TRANSFER_ENCODING;
} else {
parser->lenient_flags &= ~LENIENT_TRANSFER_ENCODING;
}
}
void llhttp_set_lenient_version(llhttp_t* parser, int enabled) {
if (enabled) {
parser->lenient_flags |= LENIENT_VERSION;
} else {
parser->lenient_flags &= ~LENIENT_VERSION;
}
}
void llhttp_set_lenient_data_after_close(llhttp_t* parser, int enabled) {
if (enabled) {
parser->lenient_flags |= LENIENT_DATA_AFTER_CLOSE;
} else {
parser->lenient_flags &= ~LENIENT_DATA_AFTER_CLOSE;
}
}
void llhttp_set_lenient_optional_lf_after_cr(llhttp_t* parser, int enabled) {
if (enabled) {
parser->lenient_flags |= LENIENT_OPTIONAL_LF_AFTER_CR;
} else {
parser->lenient_flags &= ~LENIENT_OPTIONAL_LF_AFTER_CR;
}
}
void llhttp_set_lenient_optional_crlf_after_chunk(llhttp_t* parser, int enabled) {
if (enabled) {
parser->lenient_flags |= LENIENT_OPTIONAL_CRLF_AFTER_CHUNK;
} else {
parser->lenient_flags &= ~LENIENT_OPTIONAL_CRLF_AFTER_CHUNK;
}
}
void llhttp_set_lenient_optional_cr_before_lf(llhttp_t* parser, int enabled) {
if (enabled) {
parser->lenient_flags |= LENIENT_OPTIONAL_CR_BEFORE_LF;
} else {
parser->lenient_flags &= ~LENIENT_OPTIONAL_CR_BEFORE_LF;
}
}
void llhttp_set_lenient_spaces_after_chunk_size(llhttp_t* parser, int enabled) {
if (enabled) {
parser->lenient_flags |= LENIENT_SPACES_AFTER_CHUNK_SIZE;
} else {
parser->lenient_flags &= ~LENIENT_SPACES_AFTER_CHUNK_SIZE;
}
}
void llhttp_set_lenient_header_value_relaxed(llhttp_t* parser, int enabled) {
if (enabled) {
parser->lenient_flags |= LENIENT_HEADER_VALUE_RELAXED;
} else {
parser->lenient_flags &= ~LENIENT_HEADER_VALUE_RELAXED;
}
}
/* Callbacks */
int llhttp__on_message_begin(llhttp_t* s, const char* p, const char* endp) {
int err;
CALLBACK_MAYBE(s, on_message_begin);
return err;
}
int llhttp__on_protocol(llhttp_t* s, const char* p, const char* endp) {
int err;
SPAN_CALLBACK_MAYBE(s, on_protocol, p, endp - p);
return err;
}
int llhttp__on_protocol_complete(llhttp_t* s, const char* p, const char* endp) {
int err;
CALLBACK_MAYBE(s, on_protocol_complete);
return err;
}
int llhttp__on_url(llhttp_t* s, const char* p, const char* endp) {
int err;
SPAN_CALLBACK_MAYBE(s, on_url, p, endp - p);
return err;
}
int llhttp__on_url_complete(llhttp_t* s, const char* p, const char* endp) {
int err;
CALLBACK_MAYBE(s, on_url_complete);
return err;
}
int llhttp__on_status(llhttp_t* s, const char* p, const char* endp) {
int err;
SPAN_CALLBACK_MAYBE(s, on_status, p, endp - p);
return err;
}
int llhttp__on_status_complete(llhttp_t* s, const char* p, const char* endp) {
int err;
CALLBACK_MAYBE(s, on_status_complete);
return err;
}
int llhttp__on_method(llhttp_t* s, const char* p, const char* endp) {
int err;
SPAN_CALLBACK_MAYBE(s, on_method, p, endp - p);
return err;
}
int llhttp__on_method_complete(llhttp_t* s, const char* p, const char* endp) {
int err;
CALLBACK_MAYBE(s, on_method_complete);
return err;
}
int llhttp__on_version(llhttp_t* s, const char* p, const char* endp) {
int err;
SPAN_CALLBACK_MAYBE(s, on_version, p, endp - p);
return err;
}
int llhttp__on_version_complete(llhttp_t* s, const char* p, const char* endp) {
int err;
CALLBACK_MAYBE(s, on_version_complete);
return err;
}
int llhttp__on_header_field(llhttp_t* s, const char* p, const char* endp) {
int err;
SPAN_CALLBACK_MAYBE(s, on_header_field, p, endp - p);
return err;
}
int llhttp__on_header_field_complete(llhttp_t* s, const char* p, const char* endp) {
int err;
CALLBACK_MAYBE(s, on_header_field_complete);
return err;
}
int llhttp__on_header_value(llhttp_t* s, const char* p, const char* endp) {
int err;
SPAN_CALLBACK_MAYBE(s, on_header_value, p, endp - p);
return err;
}
int llhttp__on_header_value_complete(llhttp_t* s, const char* p, const char* endp) {
int err;
CALLBACK_MAYBE(s, on_header_value_complete);
return err;
}
int llhttp__on_headers_complete(llhttp_t* s, const char* p, const char* endp) {
int err;
CALLBACK_MAYBE(s, on_headers_complete);
return err;
}
int llhttp__on_message_complete(llhttp_t* s, const char* p, const char* endp) {
int err;
CALLBACK_MAYBE(s, on_message_complete);
return err;
}
int llhttp__on_body(llhttp_t* s, const char* p, const char* endp) {
int err;
SPAN_CALLBACK_MAYBE(s, on_body, p, endp - p);
return err;
}
int llhttp__on_chunk_header(llhttp_t* s, const char* p, const char* endp) {
int err;
CALLBACK_MAYBE(s, on_chunk_header);
return err;
}
int llhttp__on_chunk_extension_name(llhttp_t* s, const char* p, const char* endp) {
int err;
SPAN_CALLBACK_MAYBE(s, on_chunk_extension_name, p, endp - p);
return err;
}
int llhttp__on_chunk_extension_name_complete(llhttp_t* s, const char* p, const char* endp) {
int err;
CALLBACK_MAYBE(s, on_chunk_extension_name_complete);
return err;
}
int llhttp__on_chunk_extension_value(llhttp_t* s, const char* p, const char* endp) {
int err;
SPAN_CALLBACK_MAYBE(s, on_chunk_extension_value, p, endp - p);
return err;
}
int llhttp__on_chunk_extension_value_complete(llhttp_t* s, const char* p, const char* endp) {
int err;
CALLBACK_MAYBE(s, on_chunk_extension_value_complete);
return err;
}
int llhttp__on_chunk_complete(llhttp_t* s, const char* p, const char* endp) {
int err;
CALLBACK_MAYBE(s, on_chunk_complete);
return err;
}
int llhttp__on_reset(llhttp_t* s, const char* p, const char* endp) {
int err;
CALLBACK_MAYBE(s, on_reset);
return err;
}
/* Private */
void llhttp__debug(llhttp_t* s, const char* p, const char* endp,
const char* msg) {
if (p == endp) {
fprintf(stderr, "p=%p type=%d flags=%02x next=null debug=%s\n", s, s->type,
s->flags, msg);
} else {
fprintf(stderr, "p=%p type=%d flags=%02x next=%02x debug=%s\n", s,
s->type, s->flags, *p, msg);
}
}

170
fgex-lib/llhttp/lib/http.c Normal file
View File

@ -0,0 +1,170 @@
#include <stdio.h>
#ifndef LLHTTP__TEST
# include "llhttp.h"
#else
# define llhttp_t llparse_t
#endif /* */
int llhttp_message_needs_eof(const llhttp_t* parser);
int llhttp_should_keep_alive(const llhttp_t* parser);
int llhttp__before_headers_complete(llhttp_t* parser, const char* p,
const char* endp) {
/* Set this here so that on_headers_complete() callbacks can see it */
if ((parser->flags & F_UPGRADE) &&
(parser->flags & F_CONNECTION_UPGRADE)) {
/* For responses, "Upgrade: foo" and "Connection: upgrade" are
* mandatory only when it is a 101 Switching Protocols response,
* otherwise it is purely informational, to announce support.
*/
parser->upgrade =
(parser->type == HTTP_REQUEST || parser->status_code == 101);
} else {
parser->upgrade = (parser->method == HTTP_CONNECT);
}
return 0;
}
/* Return values:
* 0 - No body, `restart`, message_complete
* 1 - CONNECT request, `restart`, message_complete, and pause
* 2 - chunk_size_start
* 3 - body_identity
* 4 - body_identity_eof
* 5 - invalid transfer-encoding for request
*/
int llhttp__after_headers_complete(llhttp_t* parser, const char* p,
const char* endp) {
int hasBody;
hasBody = parser->flags & F_CHUNKED || parser->content_length > 0;
if (
(parser->upgrade && (parser->method == HTTP_CONNECT ||
(parser->flags & F_SKIPBODY) || !hasBody)) ||
/* See RFC 2616 section 4.4 - 1xx e.g. Continue */
(parser->type == HTTP_RESPONSE && parser->status_code == 101)
) {
/* Exit, the rest of the message is in a different protocol. */
return 1;
}
if (parser->type == HTTP_RESPONSE && parser->status_code == 100) {
/* No body, restart as the message is complete */
return 0;
}
/* See RFC 2616 section 4.4 */
if (
parser->flags & F_SKIPBODY || /* response to a HEAD request */
(
parser->type == HTTP_RESPONSE && (
parser->status_code == 102 || /* Processing */
parser->status_code == 103 || /* Early Hints */
parser->status_code == 204 || /* No Content */
parser->status_code == 304 /* Not Modified */
)
)
) {
return 0;
} else if (parser->flags & F_CHUNKED) {
/* chunked encoding - ignore Content-Length header, prepare for a chunk */
return 2;
} else if (parser->flags & F_TRANSFER_ENCODING) {
if (parser->type == HTTP_REQUEST &&
(parser->lenient_flags & LENIENT_CHUNKED_LENGTH) == 0 &&
(parser->lenient_flags & LENIENT_TRANSFER_ENCODING) == 0) {
/* RFC 7230 3.3.3 */
/* If a Transfer-Encoding header field
* is present in a request and the chunked transfer coding is not
* the final encoding, the message body length cannot be determined
* reliably; the server MUST respond with the 400 (Bad Request)
* status code and then close the connection.
*/
return 5;
} else {
/* RFC 7230 3.3.3 */
/* If a Transfer-Encoding header field is present in a response and
* the chunked transfer coding is not the final encoding, the
* message body length is determined by reading the connection until
* it is closed by the server.
*/
return 4;
}
} else {
if (!(parser->flags & F_CONTENT_LENGTH)) {
if (!llhttp_message_needs_eof(parser)) {
/* Assume content-length 0 - read the next */
return 0;
} else {
/* Read body until EOF */
return 4;
}
} else if (parser->content_length == 0) {
/* Content-Length header given but zero: Content-Length: 0\r\n */
return 0;
} else {
/* Content-Length header given and non-zero */
return 3;
}
}
}
int llhttp__after_message_complete(llhttp_t* parser, const char* p,
const char* endp) {
int should_keep_alive;
should_keep_alive = llhttp_should_keep_alive(parser);
parser->finish = HTTP_FINISH_SAFE;
parser->flags = 0;
/* NOTE: this is ignored in loose parsing mode */
return should_keep_alive;
}
int llhttp_message_needs_eof(const llhttp_t* parser) {
if (parser->type == HTTP_REQUEST) {
return 0;
}
/* See RFC 2616 section 4.4 */
if (parser->status_code / 100 == 1 || /* 1xx e.g. Continue */
parser->status_code == 204 || /* No Content */
parser->status_code == 304 || /* Not Modified */
(parser->flags & F_SKIPBODY)) { /* response to a HEAD request */
return 0;
}
/* RFC 7230 3.3.3, see `llhttp__after_headers_complete` */
if ((parser->flags & F_TRANSFER_ENCODING) &&
(parser->flags & F_CHUNKED) == 0) {
return 1;
}
if (parser->flags & (F_CHUNKED | F_CONTENT_LENGTH)) {
return 0;
}
return 1;
}
int llhttp_should_keep_alive(const llhttp_t* parser) {
if (parser->http_major > 0 && parser->http_minor > 0) {
/* HTTP/1.1 */
if (parser->flags & F_CONNECTION_CLOSE) {
return 0;
}
} else {
/* HTTP/1.0 or earlier */
if (!(parser->flags & F_CONNECTION_KEEP_ALIVE)) {
return 0;
}
}
return !llhttp_message_needs_eof(parser);
}

10279
fgex-lib/llhttp/lib/llhttp.c Normal file
View File

File diff suppressed because it is too large Load Diff

925
fgex-lib/llhttp/lib/llhttp.h Normal file
View File

@ -0,0 +1,925 @@
#ifndef INCLUDE_LLHTTP_H_
#define INCLUDE_LLHTTP_H_
#define LLHTTP_VERSION_MAJOR 9
#define LLHTTP_VERSION_MINOR 4
#define LLHTTP_VERSION_PATCH 3
#ifndef INCLUDE_LLHTTP_ITSELF_H_
#define INCLUDE_LLHTTP_ITSELF_H_
#ifdef __cplusplus
extern "C" {
#endif
#include <stdint.h>
typedef struct llhttp__internal_s llhttp__internal_t;
struct llhttp__internal_s {
int32_t _index;
void* _span_pos0;
void* _span_cb0;
int32_t error;
const char* reason;
const char* error_pos;
void* data;
void* _current;
uint64_t content_length;
uint8_t type;
uint8_t method;
uint8_t http_major;
uint8_t http_minor;
uint8_t header_state;
uint16_t lenient_flags;
uint8_t upgrade;
uint8_t finish;
uint16_t flags;
uint16_t status_code;
uint8_t initial_message_completed;
void* settings;
};
int llhttp__internal_init(llhttp__internal_t* s);
int llhttp__internal_execute(llhttp__internal_t* s, const char* p, const char* endp);
#ifdef __cplusplus
} /* extern "C" */
#endif
#endif /* INCLUDE_LLHTTP_ITSELF_H_ */
#ifndef LLLLHTTP_C_HEADERS_
#define LLLLHTTP_C_HEADERS_
#ifdef __cplusplus
extern "C" {
#endif
enum llhttp_errno {
HPE_OK = 0,
HPE_INTERNAL = 1,
HPE_STRICT = 2,
HPE_LF_EXPECTED = 3,
HPE_UNEXPECTED_CONTENT_LENGTH = 4,
HPE_CLOSED_CONNECTION = 5,
HPE_INVALID_METHOD = 6,
HPE_INVALID_URL = 7,
HPE_INVALID_CONSTANT = 8,
HPE_INVALID_VERSION = 9,
HPE_INVALID_HEADER_TOKEN = 10,
HPE_INVALID_CONTENT_LENGTH = 11,
HPE_INVALID_CHUNK_SIZE = 12,
HPE_INVALID_STATUS = 13,
HPE_INVALID_EOF_STATE = 14,
HPE_INVALID_TRANSFER_ENCODING = 15,
HPE_CB_MESSAGE_BEGIN = 16,
HPE_CB_HEADERS_COMPLETE = 17,
HPE_CB_MESSAGE_COMPLETE = 18,
HPE_CB_CHUNK_HEADER = 19,
HPE_CB_CHUNK_COMPLETE = 20,
HPE_PAUSED = 21,
HPE_PAUSED_UPGRADE = 22,
HPE_PAUSED_H2_UPGRADE = 23,
HPE_USER = 24,
HPE_CR_EXPECTED = 25,
HPE_CB_URL_COMPLETE = 26,
HPE_CB_STATUS_COMPLETE = 27,
HPE_CB_HEADER_FIELD_COMPLETE = 28,
HPE_CB_HEADER_VALUE_COMPLETE = 29,
HPE_UNEXPECTED_SPACE = 30,
HPE_CB_RESET = 31,
HPE_CB_METHOD_COMPLETE = 32,
HPE_CB_VERSION_COMPLETE = 33,
HPE_CB_CHUNK_EXTENSION_NAME_COMPLETE = 34,
HPE_CB_CHUNK_EXTENSION_VALUE_COMPLETE = 35,
HPE_CB_PROTOCOL_COMPLETE = 38
};
typedef enum llhttp_errno llhttp_errno_t;
enum llhttp_flags {
F_CONNECTION_KEEP_ALIVE = 0x1,
F_CONNECTION_CLOSE = 0x2,
F_CONNECTION_UPGRADE = 0x4,
F_CHUNKED = 0x8,
F_UPGRADE = 0x10,
F_CONTENT_LENGTH = 0x20,
F_SKIPBODY = 0x40,
F_TRAILING = 0x80,
F_TRANSFER_ENCODING = 0x200
};
typedef enum llhttp_flags llhttp_flags_t;
enum llhttp_lenient_flags {
LENIENT_HEADERS = 0x1,
LENIENT_CHUNKED_LENGTH = 0x2,
LENIENT_KEEP_ALIVE = 0x4,
LENIENT_TRANSFER_ENCODING = 0x8,
LENIENT_VERSION = 0x10,
LENIENT_DATA_AFTER_CLOSE = 0x20,
LENIENT_OPTIONAL_LF_AFTER_CR = 0x40,
LENIENT_OPTIONAL_CRLF_AFTER_CHUNK = 0x80,
LENIENT_OPTIONAL_CR_BEFORE_LF = 0x100,
LENIENT_SPACES_AFTER_CHUNK_SIZE = 0x200,
LENIENT_HEADER_VALUE_RELAXED = 0x400
};
typedef enum llhttp_lenient_flags llhttp_lenient_flags_t;
enum llhttp_type {
HTTP_BOTH = 0,
HTTP_REQUEST = 1,
HTTP_RESPONSE = 2
};
typedef enum llhttp_type llhttp_type_t;
enum llhttp_finish {
HTTP_FINISH_SAFE = 0,
HTTP_FINISH_SAFE_WITH_CB = 1,
HTTP_FINISH_UNSAFE = 2
};
typedef enum llhttp_finish llhttp_finish_t;
enum llhttp_method {
HTTP_DELETE = 0,
HTTP_GET = 1,
HTTP_HEAD = 2,
HTTP_POST = 3,
HTTP_PUT = 4,
HTTP_CONNECT = 5,
HTTP_OPTIONS = 6,
HTTP_TRACE = 7,
HTTP_COPY = 8,
HTTP_LOCK = 9,
HTTP_MKCOL = 10,
HTTP_MOVE = 11,
HTTP_PROPFIND = 12,
HTTP_PROPPATCH = 13,
HTTP_SEARCH = 14,
HTTP_UNLOCK = 15,
HTTP_BIND = 16,
HTTP_REBIND = 17,
HTTP_UNBIND = 18,
HTTP_ACL = 19,
HTTP_REPORT = 20,
HTTP_MKACTIVITY = 21,
HTTP_CHECKOUT = 22,
HTTP_MERGE = 23,
HTTP_MSEARCH = 24,
HTTP_NOTIFY = 25,
HTTP_SUBSCRIBE = 26,
HTTP_UNSUBSCRIBE = 27,
HTTP_PATCH = 28,
HTTP_PURGE = 29,
HTTP_MKCALENDAR = 30,
HTTP_LINK = 31,
HTTP_UNLINK = 32,
HTTP_SOURCE = 33,
HTTP_PRI = 34,
HTTP_DESCRIBE = 35,
HTTP_ANNOUNCE = 36,
HTTP_SETUP = 37,
HTTP_PLAY = 38,
HTTP_PAUSE = 39,
HTTP_TEARDOWN = 40,
HTTP_GET_PARAMETER = 41,
HTTP_SET_PARAMETER = 42,
HTTP_REDIRECT = 43,
HTTP_RECORD = 44,
HTTP_FLUSH = 45,
HTTP_QUERY = 46
};
typedef enum llhttp_method llhttp_method_t;
enum llhttp_status {
HTTP_STATUS_CONTINUE = 100,
HTTP_STATUS_SWITCHING_PROTOCOLS = 101,
HTTP_STATUS_PROCESSING = 102,
HTTP_STATUS_EARLY_HINTS = 103,
HTTP_STATUS_RESPONSE_IS_STALE = 110,
HTTP_STATUS_REVALIDATION_FAILED = 111,
HTTP_STATUS_DISCONNECTED_OPERATION = 112,
HTTP_STATUS_HEURISTIC_EXPIRATION = 113,
HTTP_STATUS_MISCELLANEOUS_WARNING = 199,
HTTP_STATUS_OK = 200,
HTTP_STATUS_CREATED = 201,
HTTP_STATUS_ACCEPTED = 202,
HTTP_STATUS_NON_AUTHORITATIVE_INFORMATION = 203,
HTTP_STATUS_NO_CONTENT = 204,
HTTP_STATUS_RESET_CONTENT = 205,
HTTP_STATUS_PARTIAL_CONTENT = 206,
HTTP_STATUS_MULTI_STATUS = 207,
HTTP_STATUS_ALREADY_REPORTED = 208,
HTTP_STATUS_TRANSFORMATION_APPLIED = 214,
HTTP_STATUS_IM_USED = 226,
HTTP_STATUS_MISCELLANEOUS_PERSISTENT_WARNING = 299,
HTTP_STATUS_MULTIPLE_CHOICES = 300,
HTTP_STATUS_MOVED_PERMANENTLY = 301,
HTTP_STATUS_FOUND = 302,
HTTP_STATUS_SEE_OTHER = 303,
HTTP_STATUS_NOT_MODIFIED = 304,
HTTP_STATUS_USE_PROXY = 305,
HTTP_STATUS_SWITCH_PROXY = 306,
HTTP_STATUS_TEMPORARY_REDIRECT = 307,
HTTP_STATUS_PERMANENT_REDIRECT = 308,
HTTP_STATUS_BAD_REQUEST = 400,
HTTP_STATUS_UNAUTHORIZED = 401,
HTTP_STATUS_PAYMENT_REQUIRED = 402,
HTTP_STATUS_FORBIDDEN = 403,
HTTP_STATUS_NOT_FOUND = 404,
HTTP_STATUS_METHOD_NOT_ALLOWED = 405,
HTTP_STATUS_NOT_ACCEPTABLE = 406,
HTTP_STATUS_PROXY_AUTHENTICATION_REQUIRED = 407,
HTTP_STATUS_REQUEST_TIMEOUT = 408,
HTTP_STATUS_CONFLICT = 409,
HTTP_STATUS_GONE = 410,
HTTP_STATUS_LENGTH_REQUIRED = 411,
HTTP_STATUS_PRECONDITION_FAILED = 412,
HTTP_STATUS_PAYLOAD_TOO_LARGE = 413,
HTTP_STATUS_URI_TOO_LONG = 414,
HTTP_STATUS_UNSUPPORTED_MEDIA_TYPE = 415,
HTTP_STATUS_RANGE_NOT_SATISFIABLE = 416,
HTTP_STATUS_EXPECTATION_FAILED = 417,
HTTP_STATUS_IM_A_TEAPOT = 418,
HTTP_STATUS_PAGE_EXPIRED = 419,
HTTP_STATUS_ENHANCE_YOUR_CALM = 420,
HTTP_STATUS_MISDIRECTED_REQUEST = 421,
HTTP_STATUS_UNPROCESSABLE_ENTITY = 422,
HTTP_STATUS_LOCKED = 423,
HTTP_STATUS_FAILED_DEPENDENCY = 424,
HTTP_STATUS_TOO_EARLY = 425,
HTTP_STATUS_UPGRADE_REQUIRED = 426,
HTTP_STATUS_PRECONDITION_REQUIRED = 428,
HTTP_STATUS_TOO_MANY_REQUESTS = 429,
HTTP_STATUS_REQUEST_HEADER_FIELDS_TOO_LARGE_UNOFFICIAL = 430,
HTTP_STATUS_REQUEST_HEADER_FIELDS_TOO_LARGE = 431,
HTTP_STATUS_LOGIN_TIMEOUT = 440,
HTTP_STATUS_NO_RESPONSE = 444,
HTTP_STATUS_RETRY_WITH = 449,
HTTP_STATUS_BLOCKED_BY_PARENTAL_CONTROL = 450,
HTTP_STATUS_UNAVAILABLE_FOR_LEGAL_REASONS = 451,
HTTP_STATUS_CLIENT_CLOSED_LOAD_BALANCED_REQUEST = 460,
HTTP_STATUS_INVALID_X_FORWARDED_FOR = 463,
HTTP_STATUS_REQUEST_HEADER_TOO_LARGE = 494,
HTTP_STATUS_SSL_CERTIFICATE_ERROR = 495,
HTTP_STATUS_SSL_CERTIFICATE_REQUIRED = 496,
HTTP_STATUS_HTTP_REQUEST_SENT_TO_HTTPS_PORT = 497,
HTTP_STATUS_INVALID_TOKEN = 498,
HTTP_STATUS_CLIENT_CLOSED_REQUEST = 499,
HTTP_STATUS_INTERNAL_SERVER_ERROR = 500,
HTTP_STATUS_NOT_IMPLEMENTED = 501,
HTTP_STATUS_BAD_GATEWAY = 502,
HTTP_STATUS_SERVICE_UNAVAILABLE = 503,
HTTP_STATUS_GATEWAY_TIMEOUT = 504,
HTTP_STATUS_HTTP_VERSION_NOT_SUPPORTED = 505,
HTTP_STATUS_VARIANT_ALSO_NEGOTIATES = 506,
HTTP_STATUS_INSUFFICIENT_STORAGE = 507,
HTTP_STATUS_LOOP_DETECTED = 508,
HTTP_STATUS_BANDWIDTH_LIMIT_EXCEEDED = 509,
HTTP_STATUS_NOT_EXTENDED = 510,
HTTP_STATUS_NETWORK_AUTHENTICATION_REQUIRED = 511,
HTTP_STATUS_WEB_SERVER_UNKNOWN_ERROR = 520,
HTTP_STATUS_WEB_SERVER_IS_DOWN = 521,
HTTP_STATUS_CONNECTION_TIMEOUT = 522,
HTTP_STATUS_ORIGIN_IS_UNREACHABLE = 523,
HTTP_STATUS_TIMEOUT_OCCURED = 524,
HTTP_STATUS_SSL_HANDSHAKE_FAILED = 525,
HTTP_STATUS_INVALID_SSL_CERTIFICATE = 526,
HTTP_STATUS_RAILGUN_ERROR = 527,
HTTP_STATUS_SITE_IS_OVERLOADED = 529,
HTTP_STATUS_SITE_IS_FROZEN = 530,
HTTP_STATUS_IDENTITY_PROVIDER_AUTHENTICATION_ERROR = 561,
HTTP_STATUS_NETWORK_READ_TIMEOUT = 598,
HTTP_STATUS_NETWORK_CONNECT_TIMEOUT = 599
};
typedef enum llhttp_status llhttp_status_t;
#define HTTP_ERRNO_MAP(XX) \
XX(0, OK, OK) \
XX(1, INTERNAL, INTERNAL) \
XX(2, STRICT, STRICT) \
XX(3, LF_EXPECTED, LF_EXPECTED) \
XX(4, UNEXPECTED_CONTENT_LENGTH, UNEXPECTED_CONTENT_LENGTH) \
XX(5, CLOSED_CONNECTION, CLOSED_CONNECTION) \
XX(6, INVALID_METHOD, INVALID_METHOD) \
XX(7, INVALID_URL, INVALID_URL) \
XX(8, INVALID_CONSTANT, INVALID_CONSTANT) \
XX(9, INVALID_VERSION, INVALID_VERSION) \
XX(10, INVALID_HEADER_TOKEN, INVALID_HEADER_TOKEN) \
XX(11, INVALID_CONTENT_LENGTH, INVALID_CONTENT_LENGTH) \
XX(12, INVALID_CHUNK_SIZE, INVALID_CHUNK_SIZE) \
XX(13, INVALID_STATUS, INVALID_STATUS) \
XX(14, INVALID_EOF_STATE, INVALID_EOF_STATE) \
XX(15, INVALID_TRANSFER_ENCODING, INVALID_TRANSFER_ENCODING) \
XX(16, CB_MESSAGE_BEGIN, CB_MESSAGE_BEGIN) \
XX(17, CB_HEADERS_COMPLETE, CB_HEADERS_COMPLETE) \
XX(18, CB_MESSAGE_COMPLETE, CB_MESSAGE_COMPLETE) \
XX(19, CB_CHUNK_HEADER, CB_CHUNK_HEADER) \
XX(20, CB_CHUNK_COMPLETE, CB_CHUNK_COMPLETE) \
XX(21, PAUSED, PAUSED) \
XX(22, PAUSED_UPGRADE, PAUSED_UPGRADE) \
XX(23, PAUSED_H2_UPGRADE, PAUSED_H2_UPGRADE) \
XX(24, USER, USER) \
XX(25, CR_EXPECTED, CR_EXPECTED) \
XX(26, CB_URL_COMPLETE, CB_URL_COMPLETE) \
XX(27, CB_STATUS_COMPLETE, CB_STATUS_COMPLETE) \
XX(28, CB_HEADER_FIELD_COMPLETE, CB_HEADER_FIELD_COMPLETE) \
XX(29, CB_HEADER_VALUE_COMPLETE, CB_HEADER_VALUE_COMPLETE) \
XX(30, UNEXPECTED_SPACE, UNEXPECTED_SPACE) \
XX(31, CB_RESET, CB_RESET) \
XX(32, CB_METHOD_COMPLETE, CB_METHOD_COMPLETE) \
XX(33, CB_VERSION_COMPLETE, CB_VERSION_COMPLETE) \
XX(34, CB_CHUNK_EXTENSION_NAME_COMPLETE, CB_CHUNK_EXTENSION_NAME_COMPLETE) \
XX(35, CB_CHUNK_EXTENSION_VALUE_COMPLETE, CB_CHUNK_EXTENSION_VALUE_COMPLETE) \
XX(38, CB_PROTOCOL_COMPLETE, CB_PROTOCOL_COMPLETE) \
#define HTTP_METHOD_MAP(XX) \
XX(0, DELETE, DELETE) \
XX(1, GET, GET) \
XX(2, HEAD, HEAD) \
XX(3, POST, POST) \
XX(4, PUT, PUT) \
XX(5, CONNECT, CONNECT) \
XX(6, OPTIONS, OPTIONS) \
XX(7, TRACE, TRACE) \
XX(8, COPY, COPY) \
XX(9, LOCK, LOCK) \
XX(10, MKCOL, MKCOL) \
XX(11, MOVE, MOVE) \
XX(12, PROPFIND, PROPFIND) \
XX(13, PROPPATCH, PROPPATCH) \
XX(14, SEARCH, SEARCH) \
XX(15, UNLOCK, UNLOCK) \
XX(16, BIND, BIND) \
XX(17, REBIND, REBIND) \
XX(18, UNBIND, UNBIND) \
XX(19, ACL, ACL) \
XX(20, REPORT, REPORT) \
XX(21, MKACTIVITY, MKACTIVITY) \
XX(22, CHECKOUT, CHECKOUT) \
XX(23, MERGE, MERGE) \
XX(24, MSEARCH, M-SEARCH) \
XX(25, NOTIFY, NOTIFY) \
XX(26, SUBSCRIBE, SUBSCRIBE) \
XX(27, UNSUBSCRIBE, UNSUBSCRIBE) \
XX(28, PATCH, PATCH) \
XX(29, PURGE, PURGE) \
XX(30, MKCALENDAR, MKCALENDAR) \
XX(31, LINK, LINK) \
XX(32, UNLINK, UNLINK) \
XX(33, SOURCE, SOURCE) \
XX(46, QUERY, QUERY) \
#define RTSP_METHOD_MAP(XX) \
XX(1, GET, GET) \
XX(3, POST, POST) \
XX(6, OPTIONS, OPTIONS) \
XX(35, DESCRIBE, DESCRIBE) \
XX(36, ANNOUNCE, ANNOUNCE) \
XX(37, SETUP, SETUP) \
XX(38, PLAY, PLAY) \
XX(39, PAUSE, PAUSE) \
XX(40, TEARDOWN, TEARDOWN) \
XX(41, GET_PARAMETER, GET_PARAMETER) \
XX(42, SET_PARAMETER, SET_PARAMETER) \
XX(43, REDIRECT, REDIRECT) \
XX(44, RECORD, RECORD) \
XX(45, FLUSH, FLUSH) \
#define HTTP_ALL_METHOD_MAP(XX) \
XX(0, DELETE, DELETE) \
XX(1, GET, GET) \
XX(2, HEAD, HEAD) \
XX(3, POST, POST) \
XX(4, PUT, PUT) \
XX(5, CONNECT, CONNECT) \
XX(6, OPTIONS, OPTIONS) \
XX(7, TRACE, TRACE) \
XX(8, COPY, COPY) \
XX(9, LOCK, LOCK) \
XX(10, MKCOL, MKCOL) \
XX(11, MOVE, MOVE) \
XX(12, PROPFIND, PROPFIND) \
XX(13, PROPPATCH, PROPPATCH) \
XX(14, SEARCH, SEARCH) \
XX(15, UNLOCK, UNLOCK) \
XX(16, BIND, BIND) \
XX(17, REBIND, REBIND) \
XX(18, UNBIND, UNBIND) \
XX(19, ACL, ACL) \
XX(20, REPORT, REPORT) \
XX(21, MKACTIVITY, MKACTIVITY) \
XX(22, CHECKOUT, CHECKOUT) \
XX(23, MERGE, MERGE) \
XX(24, MSEARCH, M-SEARCH) \
XX(25, NOTIFY, NOTIFY) \
XX(26, SUBSCRIBE, SUBSCRIBE) \
XX(27, UNSUBSCRIBE, UNSUBSCRIBE) \
XX(28, PATCH, PATCH) \
XX(29, PURGE, PURGE) \
XX(30, MKCALENDAR, MKCALENDAR) \
XX(31, LINK, LINK) \
XX(32, UNLINK, UNLINK) \
XX(33, SOURCE, SOURCE) \
XX(34, PRI, PRI) \
XX(35, DESCRIBE, DESCRIBE) \
XX(36, ANNOUNCE, ANNOUNCE) \
XX(37, SETUP, SETUP) \
XX(38, PLAY, PLAY) \
XX(39, PAUSE, PAUSE) \
XX(40, TEARDOWN, TEARDOWN) \
XX(41, GET_PARAMETER, GET_PARAMETER) \
XX(42, SET_PARAMETER, SET_PARAMETER) \
XX(43, REDIRECT, REDIRECT) \
XX(44, RECORD, RECORD) \
XX(45, FLUSH, FLUSH) \
XX(46, QUERY, QUERY) \
#define HTTP_STATUS_MAP(XX) \
XX(100, CONTINUE, CONTINUE) \
XX(101, SWITCHING_PROTOCOLS, SWITCHING_PROTOCOLS) \
XX(102, PROCESSING, PROCESSING) \
XX(103, EARLY_HINTS, EARLY_HINTS) \
XX(110, RESPONSE_IS_STALE, RESPONSE_IS_STALE) \
XX(111, REVALIDATION_FAILED, REVALIDATION_FAILED) \
XX(112, DISCONNECTED_OPERATION, DISCONNECTED_OPERATION) \
XX(113, HEURISTIC_EXPIRATION, HEURISTIC_EXPIRATION) \
XX(199, MISCELLANEOUS_WARNING, MISCELLANEOUS_WARNING) \
XX(200, OK, OK) \
XX(201, CREATED, CREATED) \
XX(202, ACCEPTED, ACCEPTED) \
XX(203, NON_AUTHORITATIVE_INFORMATION, NON_AUTHORITATIVE_INFORMATION) \
XX(204, NO_CONTENT, NO_CONTENT) \
XX(205, RESET_CONTENT, RESET_CONTENT) \
XX(206, PARTIAL_CONTENT, PARTIAL_CONTENT) \
XX(207, MULTI_STATUS, MULTI_STATUS) \
XX(208, ALREADY_REPORTED, ALREADY_REPORTED) \
XX(214, TRANSFORMATION_APPLIED, TRANSFORMATION_APPLIED) \
XX(226, IM_USED, IM_USED) \
XX(299, MISCELLANEOUS_PERSISTENT_WARNING, MISCELLANEOUS_PERSISTENT_WARNING) \
XX(300, MULTIPLE_CHOICES, MULTIPLE_CHOICES) \
XX(301, MOVED_PERMANENTLY, MOVED_PERMANENTLY) \
XX(302, FOUND, FOUND) \
XX(303, SEE_OTHER, SEE_OTHER) \
XX(304, NOT_MODIFIED, NOT_MODIFIED) \
XX(305, USE_PROXY, USE_PROXY) \
XX(306, SWITCH_PROXY, SWITCH_PROXY) \
XX(307, TEMPORARY_REDIRECT, TEMPORARY_REDIRECT) \
XX(308, PERMANENT_REDIRECT, PERMANENT_REDIRECT) \
XX(400, BAD_REQUEST, BAD_REQUEST) \
XX(401, UNAUTHORIZED, UNAUTHORIZED) \
XX(402, PAYMENT_REQUIRED, PAYMENT_REQUIRED) \
XX(403, FORBIDDEN, FORBIDDEN) \
XX(404, NOT_FOUND, NOT_FOUND) \
XX(405, METHOD_NOT_ALLOWED, METHOD_NOT_ALLOWED) \
XX(406, NOT_ACCEPTABLE, NOT_ACCEPTABLE) \
XX(407, PROXY_AUTHENTICATION_REQUIRED, PROXY_AUTHENTICATION_REQUIRED) \
XX(408, REQUEST_TIMEOUT, REQUEST_TIMEOUT) \
XX(409, CONFLICT, CONFLICT) \
XX(410, GONE, GONE) \
XX(411, LENGTH_REQUIRED, LENGTH_REQUIRED) \
XX(412, PRECONDITION_FAILED, PRECONDITION_FAILED) \
XX(413, PAYLOAD_TOO_LARGE, PAYLOAD_TOO_LARGE) \
XX(414, URI_TOO_LONG, URI_TOO_LONG) \
XX(415, UNSUPPORTED_MEDIA_TYPE, UNSUPPORTED_MEDIA_TYPE) \
XX(416, RANGE_NOT_SATISFIABLE, RANGE_NOT_SATISFIABLE) \
XX(417, EXPECTATION_FAILED, EXPECTATION_FAILED) \
XX(418, IM_A_TEAPOT, IM_A_TEAPOT) \
XX(419, PAGE_EXPIRED, PAGE_EXPIRED) \
XX(420, ENHANCE_YOUR_CALM, ENHANCE_YOUR_CALM) \
XX(421, MISDIRECTED_REQUEST, MISDIRECTED_REQUEST) \
XX(422, UNPROCESSABLE_ENTITY, UNPROCESSABLE_ENTITY) \
XX(423, LOCKED, LOCKED) \
XX(424, FAILED_DEPENDENCY, FAILED_DEPENDENCY) \
XX(425, TOO_EARLY, TOO_EARLY) \
XX(426, UPGRADE_REQUIRED, UPGRADE_REQUIRED) \
XX(428, PRECONDITION_REQUIRED, PRECONDITION_REQUIRED) \
XX(429, TOO_MANY_REQUESTS, TOO_MANY_REQUESTS) \
XX(430, REQUEST_HEADER_FIELDS_TOO_LARGE_UNOFFICIAL, REQUEST_HEADER_FIELDS_TOO_LARGE_UNOFFICIAL) \
XX(431, REQUEST_HEADER_FIELDS_TOO_LARGE, REQUEST_HEADER_FIELDS_TOO_LARGE) \
XX(440, LOGIN_TIMEOUT, LOGIN_TIMEOUT) \
XX(444, NO_RESPONSE, NO_RESPONSE) \
XX(449, RETRY_WITH, RETRY_WITH) \
XX(450, BLOCKED_BY_PARENTAL_CONTROL, BLOCKED_BY_PARENTAL_CONTROL) \
XX(451, UNAVAILABLE_FOR_LEGAL_REASONS, UNAVAILABLE_FOR_LEGAL_REASONS) \
XX(460, CLIENT_CLOSED_LOAD_BALANCED_REQUEST, CLIENT_CLOSED_LOAD_BALANCED_REQUEST) \
XX(463, INVALID_X_FORWARDED_FOR, INVALID_X_FORWARDED_FOR) \
XX(494, REQUEST_HEADER_TOO_LARGE, REQUEST_HEADER_TOO_LARGE) \
XX(495, SSL_CERTIFICATE_ERROR, SSL_CERTIFICATE_ERROR) \
XX(496, SSL_CERTIFICATE_REQUIRED, SSL_CERTIFICATE_REQUIRED) \
XX(497, HTTP_REQUEST_SENT_TO_HTTPS_PORT, HTTP_REQUEST_SENT_TO_HTTPS_PORT) \
XX(498, INVALID_TOKEN, INVALID_TOKEN) \
XX(499, CLIENT_CLOSED_REQUEST, CLIENT_CLOSED_REQUEST) \
XX(500, INTERNAL_SERVER_ERROR, INTERNAL_SERVER_ERROR) \
XX(501, NOT_IMPLEMENTED, NOT_IMPLEMENTED) \
XX(502, BAD_GATEWAY, BAD_GATEWAY) \
XX(503, SERVICE_UNAVAILABLE, SERVICE_UNAVAILABLE) \
XX(504, GATEWAY_TIMEOUT, GATEWAY_TIMEOUT) \
XX(505, HTTP_VERSION_NOT_SUPPORTED, HTTP_VERSION_NOT_SUPPORTED) \
XX(506, VARIANT_ALSO_NEGOTIATES, VARIANT_ALSO_NEGOTIATES) \
XX(507, INSUFFICIENT_STORAGE, INSUFFICIENT_STORAGE) \
XX(508, LOOP_DETECTED, LOOP_DETECTED) \
XX(509, BANDWIDTH_LIMIT_EXCEEDED, BANDWIDTH_LIMIT_EXCEEDED) \
XX(510, NOT_EXTENDED, NOT_EXTENDED) \
XX(511, NETWORK_AUTHENTICATION_REQUIRED, NETWORK_AUTHENTICATION_REQUIRED) \
XX(520, WEB_SERVER_UNKNOWN_ERROR, WEB_SERVER_UNKNOWN_ERROR) \
XX(521, WEB_SERVER_IS_DOWN, WEB_SERVER_IS_DOWN) \
XX(522, CONNECTION_TIMEOUT, CONNECTION_TIMEOUT) \
XX(523, ORIGIN_IS_UNREACHABLE, ORIGIN_IS_UNREACHABLE) \
XX(524, TIMEOUT_OCCURED, TIMEOUT_OCCURED) \
XX(525, SSL_HANDSHAKE_FAILED, SSL_HANDSHAKE_FAILED) \
XX(526, INVALID_SSL_CERTIFICATE, INVALID_SSL_CERTIFICATE) \
XX(527, RAILGUN_ERROR, RAILGUN_ERROR) \
XX(529, SITE_IS_OVERLOADED, SITE_IS_OVERLOADED) \
XX(530, SITE_IS_FROZEN, SITE_IS_FROZEN) \
XX(561, IDENTITY_PROVIDER_AUTHENTICATION_ERROR, IDENTITY_PROVIDER_AUTHENTICATION_ERROR) \
XX(598, NETWORK_READ_TIMEOUT, NETWORK_READ_TIMEOUT) \
XX(599, NETWORK_CONNECT_TIMEOUT, NETWORK_CONNECT_TIMEOUT) \
#ifdef __cplusplus
} /* extern "C" */
#endif
#endif /* LLLLHTTP_C_HEADERS_ */
#ifndef INCLUDE_LLHTTP_API_H_
#define INCLUDE_LLHTTP_API_H_
#ifdef __cplusplus
extern "C" {
#endif
#include <stddef.h>
#if defined(__wasm__)
#define LLHTTP_EXPORT __attribute__((visibility("default")))
#elif defined(_WIN32)
#define LLHTTP_EXPORT __declspec(dllexport)
#else
#define LLHTTP_EXPORT
#endif
typedef llhttp__internal_t llhttp_t;
typedef struct llhttp_settings_s llhttp_settings_t;
typedef int (*llhttp_data_cb)(llhttp_t*, const char *at, size_t length);
typedef int (*llhttp_cb)(llhttp_t*);
struct llhttp_settings_s {
/* Possible return values 0, -1, `HPE_PAUSED` */
llhttp_cb on_message_begin;
/* Possible return values 0, -1, HPE_USER */
llhttp_data_cb on_protocol;
llhttp_data_cb on_url;
llhttp_data_cb on_status;
llhttp_data_cb on_method;
llhttp_data_cb on_version;
llhttp_data_cb on_header_field;
llhttp_data_cb on_header_value;
llhttp_data_cb on_chunk_extension_name;
llhttp_data_cb on_chunk_extension_value;
/* Possible return values:
* 0 - Proceed normally
* 1 - Assume that request/response has no body, and proceed to parsing the
* next message
* 2 - Assume absence of body (as above) and make `llhttp_execute()` return
* `HPE_PAUSED_UPGRADE`
* -1 - Error
* `HPE_PAUSED`
*/
llhttp_cb on_headers_complete;
/* Possible return values 0, -1, HPE_USER */
llhttp_data_cb on_body;
/* Possible return values 0, -1, `HPE_PAUSED` */
llhttp_cb on_message_complete;
llhttp_cb on_protocol_complete;
llhttp_cb on_url_complete;
llhttp_cb on_status_complete;
llhttp_cb on_method_complete;
llhttp_cb on_version_complete;
llhttp_cb on_header_field_complete;
llhttp_cb on_header_value_complete;
llhttp_cb on_chunk_extension_name_complete;
llhttp_cb on_chunk_extension_value_complete;
/* When on_chunk_header is called, the current chunk length is stored
* in parser->content_length.
* Possible return values 0, -1, `HPE_PAUSED`
*/
llhttp_cb on_chunk_header;
llhttp_cb on_chunk_complete;
llhttp_cb on_reset;
};
/* Initialize the parser with specific type and user settings.
*
* NOTE: lifetime of `settings` has to be at least the same as the lifetime of
* the `parser` here. In practice, `settings` has to be either a static
* variable or be allocated with `malloc`, `new`, etc.
*/
LLHTTP_EXPORT
void llhttp_init(llhttp_t* parser, llhttp_type_t type,
const llhttp_settings_t* settings);
LLHTTP_EXPORT
llhttp_t* llhttp_alloc(llhttp_type_t type);
LLHTTP_EXPORT
void llhttp_free(llhttp_t* parser);
LLHTTP_EXPORT
uint8_t llhttp_get_type(llhttp_t* parser);
LLHTTP_EXPORT
uint8_t llhttp_get_http_major(llhttp_t* parser);
LLHTTP_EXPORT
uint8_t llhttp_get_http_minor(llhttp_t* parser);
LLHTTP_EXPORT
uint8_t llhttp_get_method(llhttp_t* parser);
LLHTTP_EXPORT
int llhttp_get_status_code(llhttp_t* parser);
LLHTTP_EXPORT
uint8_t llhttp_get_upgrade(llhttp_t* parser);
/* Reset an already initialized parser back to the start state, preserving the
* existing parser type, callback settings, user data, and lenient flags.
*/
LLHTTP_EXPORT
void llhttp_reset(llhttp_t* parser);
/* Initialize the settings object */
LLHTTP_EXPORT
void llhttp_settings_init(llhttp_settings_t* settings);
/* Parse full or partial request/response, invoking user callbacks along the
* way.
*
* If any of `llhttp_data_cb` returns errno not equal to `HPE_OK` - the parsing
* interrupts, and such errno is returned from `llhttp_execute()`. If
* `HPE_PAUSED` was used as a errno, the execution can be resumed with
* `llhttp_resume()` call.
*
* In a special case of CONNECT/Upgrade request/response `HPE_PAUSED_UPGRADE`
* is returned after fully parsing the request/response. If the user wishes to
* continue parsing, they need to invoke `llhttp_resume_after_upgrade()`.
*
* NOTE: if this function ever returns a non-pause type error, it will continue
* to return the same error upon each successive call up until `llhttp_init()`
* is called.
*/
LLHTTP_EXPORT
llhttp_errno_t llhttp_execute(llhttp_t* parser, const char* data, size_t len);
/* This method should be called when the other side has no further bytes to
* send (e.g. shutdown of readable side of the TCP connection.)
*
* Requests without `Content-Length` and other messages might require treating
* all incoming bytes as the part of the body, up to the last byte of the
* connection. This method will invoke `on_message_complete()` callback if the
* request was terminated safely. Otherwise a error code would be returned.
*/
LLHTTP_EXPORT
llhttp_errno_t llhttp_finish(llhttp_t* parser);
/* Returns `1` if the incoming message is parsed until the last byte, and has
* to be completed by calling `llhttp_finish()` on EOF
*/
LLHTTP_EXPORT
int llhttp_message_needs_eof(const llhttp_t* parser);
/* Returns `1` if there might be any other messages following the last that was
* successfully parsed.
*/
LLHTTP_EXPORT
int llhttp_should_keep_alive(const llhttp_t* parser);
/* Make further calls of `llhttp_execute()` return `HPE_PAUSED` and set
* appropriate error reason.
*
* Important: do not call this from user callbacks! User callbacks must return
* `HPE_PAUSED` if pausing is required.
*/
LLHTTP_EXPORT
void llhttp_pause(llhttp_t* parser);
/* Might be called to resume the execution after the pause in user's callback.
* See `llhttp_execute()` above for details.
*
* Call this only if `llhttp_execute()` returns `HPE_PAUSED`.
*/
LLHTTP_EXPORT
void llhttp_resume(llhttp_t* parser);
/* Might be called to resume the execution after the pause in user's callback.
* See `llhttp_execute()` above for details.
*
* Call this only if `llhttp_execute()` returns `HPE_PAUSED_UPGRADE`
*/
LLHTTP_EXPORT
void llhttp_resume_after_upgrade(llhttp_t* parser);
/* Returns the latest return error */
LLHTTP_EXPORT
llhttp_errno_t llhttp_get_errno(const llhttp_t* parser);
/* Returns the verbal explanation of the latest returned error.
*
* Note: User callback should set error reason when returning the error. See
* `llhttp_set_error_reason()` for details.
*/
LLHTTP_EXPORT
const char* llhttp_get_error_reason(const llhttp_t* parser);
/* Assign verbal description to the returned error. Must be called in user
* callbacks right before returning the errno.
*
* Note: `HPE_USER` error code might be useful in user callbacks.
*/
LLHTTP_EXPORT
void llhttp_set_error_reason(llhttp_t* parser, const char* reason);
/* Returns the pointer to the last parsed byte before the returned error. The
* pointer is relative to the `data` argument of `llhttp_execute()`.
*
* Note: this method might be useful for counting the number of parsed bytes.
*/
LLHTTP_EXPORT
const char* llhttp_get_error_pos(const llhttp_t* parser);
/* Returns textual name of error code */
LLHTTP_EXPORT
const char* llhttp_errno_name(llhttp_errno_t err);
/* Returns textual name of HTTP method */
LLHTTP_EXPORT
const char* llhttp_method_name(llhttp_method_t method);
/* Returns textual name of HTTP status */
LLHTTP_EXPORT
const char* llhttp_status_name(llhttp_status_t status);
/* Enables/disables lenient header value parsing (disabled by default).
*
* Lenient parsing disables header value token checks, extending llhttp's
* protocol support to highly non-compliant clients/server. No
* `HPE_INVALID_HEADER_TOKEN` will be raised for incorrect header values when
* lenient parsing is "on".
*
* **Enabling this flag can pose a security issue since you will be exposed to
* request smuggling attacks. USE WITH CAUTION!**
*/
LLHTTP_EXPORT
void llhttp_set_lenient_headers(llhttp_t* parser, int enabled);
/* Enables/disables lenient handling of conflicting `Transfer-Encoding` and
* `Content-Length` headers (disabled by default).
*
* Normally `llhttp` would error when `Transfer-Encoding` is present in
* conjunction with `Content-Length`. This error is important to prevent HTTP
* request smuggling, but may be less desirable for small number of cases
* involving legacy servers.
*
* **Enabling this flag can pose a security issue since you will be exposed to
* request smuggling attacks. USE WITH CAUTION!**
*/
LLHTTP_EXPORT
void llhttp_set_lenient_chunked_length(llhttp_t* parser, int enabled);
/* Enables/disables lenient handling of `Connection: close` and HTTP/1.0
* requests responses.
*
* Normally `llhttp` would error on (in strict mode) or discard (in loose mode)
* the HTTP request/response after the request/response with `Connection: close`
* and `Content-Length`. This is important to prevent cache poisoning attacks,
* but might interact badly with outdated and insecure clients. With this flag
* the extra request/response will be parsed normally.
*
* **Enabling this flag can pose a security issue since you will be exposed to
* poisoning attacks. USE WITH CAUTION!**
*/
LLHTTP_EXPORT
void llhttp_set_lenient_keep_alive(llhttp_t* parser, int enabled);
/* Enables/disables lenient handling of `Transfer-Encoding` header.
*
* Normally `llhttp` would error when a `Transfer-Encoding` has `chunked` value
* and another value after it (either in a single header or in multiple
* headers whose value are internally joined using `, `).
* This is mandated by the spec to reliably determine request body size and thus
* avoid request smuggling.
* With this flag the extra value will be parsed normally.
*
* **Enabling this flag can pose a security issue since you will be exposed to
* request smuggling attacks. USE WITH CAUTION!**
*/
LLHTTP_EXPORT
void llhttp_set_lenient_transfer_encoding(llhttp_t* parser, int enabled);
/* Enables/disables lenient handling of HTTP version.
*
* Normally `llhttp` would error when the HTTP version in the request or status line
* is not `0.9`, `1.0`, `1.1` or `2.0`.
* With this flag the invalid value will be parsed normally.
*
* **Enabling this flag can pose a security issue since you will allow unsupported
* HTTP versions. USE WITH CAUTION!**
*/
LLHTTP_EXPORT
void llhttp_set_lenient_version(llhttp_t* parser, int enabled);
/* Enables/disables lenient handling of additional data received after a message ends
* and keep-alive is disabled.
*
* Normally `llhttp` would error when additional unexpected data is received if the message
* contains the `Connection` header with `close` value.
* With this flag the extra data will discarded without throwing an error.
*
* **Enabling this flag can pose a security issue since you will be exposed to
* poisoning attacks. USE WITH CAUTION!**
*/
LLHTTP_EXPORT
void llhttp_set_lenient_data_after_close(llhttp_t* parser, int enabled);
/* Enables/disables lenient handling of incomplete CRLF sequences.
*
* Normally `llhttp` would error when a CR is not followed by LF when terminating the
* request line, the status line, the headers or a chunk header.
* With this flag only a CR is required to terminate such sections.
*
* **Enabling this flag can pose a security issue since you will be exposed to
* request smuggling attacks. USE WITH CAUTION!**
*/
LLHTTP_EXPORT
void llhttp_set_lenient_optional_lf_after_cr(llhttp_t* parser, int enabled);
/*
* Enables/disables lenient handling of line separators.
*
* Normally `llhttp` would error when a LF is not preceded by CR when terminating the
* request line, the status line, the headers, a chunk header or a chunk data.
* With this flag only a LF is required to terminate such sections.
*
* **Enabling this flag can pose a security issue since you will be exposed to
* request smuggling attacks. USE WITH CAUTION!**
*/
LLHTTP_EXPORT
void llhttp_set_lenient_optional_cr_before_lf(llhttp_t* parser, int enabled);
/* Enables/disables lenient handling of chunks not separated via CRLF.
*
* Normally `llhttp` would error when after a chunk data a CRLF is missing before
* starting a new chunk.
* With this flag the new chunk can start immediately after the previous one.
*
* **Enabling this flag can pose a security issue since you will be exposed to
* request smuggling attacks. USE WITH CAUTION!**
*/
LLHTTP_EXPORT
void llhttp_set_lenient_optional_crlf_after_chunk(llhttp_t* parser, int enabled);
/* Enables/disables lenient handling of spaces after chunk size.
*
* Normally `llhttp` would error when after a chunk size is followed by one or more
* spaces are present instead of a CRLF or `;`.
* With this flag this check is disabled.
*
* **Enabling this flag can pose a security issue since you will be exposed to
* request smuggling attacks. USE WITH CAUTION!**
*/
LLHTTP_EXPORT
void llhttp_set_lenient_spaces_after_chunk_size(llhttp_t* parser, int enabled);
/* Enables/disables relaxed handling of unusual characters in header values.
*
* RFC 9110 describes NULL, CR and LF as 'dangerous' and says they MUST be
* rejected, while other control characters are merely 'invalid' and discouraged,
* and are explicitly allowed by other standards (e.g. WHATWG Fetch) and
* in surprisingly common use on the web.
*
* This flag enables these 'invalid but common' characters, aiming to
* maximize compatibility without enabling any potentially dangerous scenarios.
*
* Unlike `llhttp_set_lenient_headers()`, this does NOT enable any other
* potentially unsafe behaviors (like accepting whitespace before colons
* or after the start line).
*/
LLHTTP_EXPORT
void llhttp_set_lenient_header_value_relaxed(llhttp_t* parser, int enabled);
#ifdef __cplusplus
} /* extern "C" */
#endif
#endif /* INCLUDE_LLHTTP_API_H_ */
#endif /* INCLUDE_LLHTTP_H_ */

632
fgex-lib/llhttp/llhttp_module.c Normal file
View File

@ -0,0 +1,632 @@
#define PY_SSIZE_T_CLEAN
#include <Python.h>
#include <stdbool.h>
#include <string.h>
#include <ctype.h>
#include "lib/llhttp.h"
#define STRING(x) #x
#define XSTRING(x) STRING(x)
#define LLHTTP_VERSION XSTRING(LLHTTP_VERSION_MAJOR) "." XSTRING(LLHTTP_VERSION_MINOR) "." XSTRING(LLHTTP_VERSION_PATCH)
/* What this module is called, in one place. It is the name the types and the exception
* classes carry, and the name they are looked back up under at runtime. It used to be
* `pyllhttp` — the archived package this binding was forked from — which is not a
* dependency and is not installed anywhere this is shipped. */
#define MODULE_NAME "firegex._llhttp"
typedef struct{
PyObject_HEAD
llhttp_t llhttp;
} llhttp_obj_t;
typedef struct{
llhttp_errno_t code;
const char *name;
} error_info;
static error_info errors[] = {
#define HTTP_ERRNO_GEN(CODE, NAME, _) {CODE, #NAME},
HTTP_ERRNO_MAP(HTTP_ERRNO_GEN)
#undef HTTP_ERRNO_GEN
};
/*
* `INVALID_METHOD` -> `InvalidMethodError`: the name this module publishes the
* exception for that code under.
*
* Written into the caller's buffer every time, never cached on the table above. That
* table is one array shared by every interpreter in the process, and this module
* declares per-interpreter GIL support — a name stored there is a name two interpreters
* can be writing at the same moment. It used to be `malloc`ed on first use, guarded by
* a NULL check two threads can pass together, and never freed.
*/
static void error_class_name(const char *snake, char *out, size_t out_len) {
size_t written = 0;
bool upper = true;
for (const char *c = snake; *c && written + 1 < out_len; ++c) {
if (isalpha((unsigned char)*c))
out[written++] = (char)(upper ? toupper((unsigned char)*c)
: tolower((unsigned char)*c));
else if (isdigit((unsigned char)*c))
out[written++] = *c;
upper = !isalpha((unsigned char)*c);
}
for (const char *suffix = "Error"; *suffix && written + 1 < out_len; ++suffix)
out[written++] = *suffix;
out[written] = '\0';
}
static const char* methods[] = {
#define HTTP_METHOD_GEN(NUMBER, NAME, STRING) #STRING,
HTTP_METHOD_MAP(HTTP_METHOD_GEN)
#undef HTTP_METHOD_GEN
};
/* New public callback helper that uses method names as C strings */
static int parser_callback(const char *name, llhttp_t *llhttp) {
PyObject *result = PyObject_CallMethod(llhttp->data, name, NULL);
if (result)
Py_DECREF(result);
if (PyErr_Occurred())
return HPE_USER;
if (HPE_PAUSED == llhttp_get_errno(llhttp)) {
llhttp_resume(llhttp);
return HPE_PAUSED;
}
if (HPE_PAUSED_UPGRADE == llhttp_get_errno(llhttp)) {
llhttp_resume_after_upgrade(llhttp);
return HPE_PAUSED_UPGRADE;
}
return HPE_OK;
}
static int parser_data_callback(const char *name, llhttp_t *llhttp, const char *data, size_t length) {
PyObject *payload = PyMemoryView_FromMemory((char*)data, length, PyBUF_READ);
if (!payload)
return HPE_USER;
PyObject *result = PyObject_CallMethod(llhttp->data, name, "O", payload);
Py_DECREF(payload);
if (result)
Py_DECREF(result);
if (PyErr_Occurred())
return HPE_USER;
if (HPE_PAUSED == llhttp_get_errno(llhttp)) {
llhttp_resume(llhttp);
return HPE_PAUSED;
}
if (HPE_PAUSED_UPGRADE == llhttp_get_errno(llhttp)) {
llhttp_resume_after_upgrade(llhttp);
return HPE_PAUSED_UPGRADE;
}
return HPE_OK;
}
/* Macros now simply pass the name as a string literal */
#define PARSER_CALLBACK(type) \
static int parser_ ## type (llhttp_t *llhttp) { \
return parser_callback(#type, llhttp); \
}
#define PARSER_DATA_CALLBACK(type) \
static int parser_ ## type (llhttp_t *llhttp, const char *data, size_t length) { \
return parser_data_callback(#type, llhttp, data, length); \
}
PARSER_CALLBACK(on_message_begin)
PARSER_DATA_CALLBACK(on_url)
PARSER_CALLBACK(on_url_complete)
PARSER_DATA_CALLBACK(on_status)
PARSER_CALLBACK(on_status_complete)
PARSER_DATA_CALLBACK(on_header_field)
PARSER_CALLBACK(on_header_field_complete)
PARSER_DATA_CALLBACK(on_header_value)
PARSER_CALLBACK(on_header_value_complete)
PARSER_CALLBACK(on_headers_complete)
PARSER_DATA_CALLBACK(on_body)
PARSER_CALLBACK(on_message_complete)
PARSER_CALLBACK(on_chunk_header)
PARSER_CALLBACK(on_chunk_complete)
llhttp_settings_t parser_settings = {
.on_message_begin = parser_on_message_begin,
.on_url = parser_on_url,
.on_url_complete = parser_on_url_complete,
.on_status = parser_on_status,
.on_status_complete = parser_on_status_complete,
.on_header_field = parser_on_header_field,
.on_header_field_complete = parser_on_header_field_complete,
.on_header_value = parser_on_header_value,
.on_header_value_complete = parser_on_header_value_complete,
.on_headers_complete = parser_on_headers_complete,
.on_body = parser_on_body,
.on_message_complete = parser_on_message_complete,
.on_chunk_header = parser_on_chunk_header,
.on_chunk_complete = parser_on_chunk_complete,
};
static PyObject *request_new(PyTypeObject *type, PyObject *args, PyObject *kwds) {
PyObject *self = type->tp_alloc(type, 0);
if (self) {
llhttp_t *llhttp = &((llhttp_obj_t*)self)->llhttp;
llhttp_init(llhttp, HTTP_REQUEST, &parser_settings);
llhttp->data = self;
}
return self;
}
static PyObject *response_new(PyTypeObject *type, PyObject *args, PyObject *kwds) {
PyObject *self = type->tp_alloc(type, 0);
if (self) {
llhttp_t *llhttp = &((llhttp_obj_t*)self)->llhttp;
llhttp_init(llhttp, HTTP_RESPONSE, &parser_settings);
llhttp->data = self;
}
return self;
}
/*
* Turn an llhttp error code into the exception this module publishes for it.
*
* The classes are looked up in the module itself — `sys.modules` belongs to the
* interpreter asking, so each one finds the classes it built, which is the whole reason
* they are created per module instance rather than once per process.
*
* This used to import `pyllhttp`, a package that is not a dependency and is installed
* nowhere the wheel is: the import failed, its traceback was printed to stderr (the
* service log, on a filter), and the function returned **without setting an exception**,
* which CPython reports to the caller as `SystemError: ... returned NULL without setting
* an exception`. A malformed request — the traffic this exists to inspect — said that
* instead of naming the parse error. So every path out of here raises something.
*/
static void set_related_exception(llhttp_errno_t eno, llhttp_t *llhttp) {
const char *reason = llhttp_get_error_reason(llhttp);
if (!reason)
reason = "HTTP parse error";
for (size_t i = 0; i < sizeof(errors)/sizeof(errors[0]); ++i) {
if (errors[i].code != eno)
continue;
char class_name[128];
error_class_name(errors[i].name, class_name, sizeof(class_name));
PyObject *module = PyImport_ImportModule(MODULE_NAME);
if (module) {
PyObject *exception = PyObject_GetAttrString(module, class_name);
Py_DECREF(module);
if (exception) {
PyErr_SetString(exception, reason);
Py_DECREF(exception);
return;
}
}
/* The class is missing, which should not happen — but the traffic was still
* malformed, and saying so badly beats saying nothing. */
PyErr_Clear();
PyErr_Format(PyExc_RuntimeError, "%s: %s", class_name, reason);
return;
}
PyErr_Format(PyExc_RuntimeError,
"unknown HTTP parser error %d: %s", (int)eno, reason);
}
static PyObject *parser_execute(PyObject *self, PyObject *payload) {
llhttp_t *llhttp = &((llhttp_obj_t*)self)->llhttp;
Py_buffer buffer;
if (PyObject_GetBuffer(payload, &buffer, PyBUF_SIMPLE)){
return NULL;
}
if (!PyBuffer_IsContiguous(&buffer, 'C')) {
PyErr_SetString(PyExc_TypeError, "buffer is not contiguous");
PyBuffer_Release(&buffer);
return NULL;
}
llhttp_errno_t error = llhttp_execute(llhttp, buffer.buf, buffer.len);
PyBuffer_Release(&buffer);
if (PyErr_Occurred())
return NULL;
switch (error) {
case HPE_OK:
return Py_BuildValue("(nn)", error, buffer.len);
case HPE_PAUSED:
case HPE_PAUSED_UPGRADE:
case HPE_PAUSED_H2_UPGRADE:
return Py_BuildValue("(nn)", error, llhttp->error_pos - (const char*)buffer.buf);
default:
set_related_exception(error, llhttp);
return NULL;
}
}
static PyObject *parser_pause(PyObject *self) {
llhttp_t *llhttp = &((llhttp_obj_t*)self)->llhttp;
llhttp_pause(llhttp);
Py_RETURN_NONE;
}
static PyObject *parser_unpause(PyObject *self) {
llhttp_t *llhttp = &((llhttp_obj_t*)self)->llhttp;
llhttp_resume(llhttp);
Py_RETURN_NONE;
}
static PyObject *parser_upgrade(PyObject *self) {
llhttp_t *llhttp = &((llhttp_obj_t*)self)->llhttp;
llhttp_resume_after_upgrade(llhttp);
Py_RETURN_NONE;
}
static PyObject *parser_finish(PyObject *self) {
llhttp_t *llhttp = &((llhttp_obj_t*)self)->llhttp;
llhttp_errno_t error = llhttp_finish(llhttp);
if (HPE_OK == error)
Py_RETURN_NONE;
set_related_exception(error, llhttp);
return NULL;
}
static PyObject *parser_reset(PyObject *self) {
llhttp_t *llhttp = &((llhttp_obj_t*)self)->llhttp;
llhttp_reset(llhttp);
Py_RETURN_NONE;
}
static PyObject * parser_dummy_noargs(PyObject *self) { Py_RETURN_NONE; }
static PyObject * parser_dummy_onearg(PyObject *self, PyObject *arg) { Py_RETURN_NONE; }
static PyMethodDef parser_methods[] = {
{ "execute", (PyCFunction)parser_execute, METH_O },
{ "pause", (PyCFunction)parser_pause, METH_NOARGS },
{ "unpause", (PyCFunction)parser_unpause, METH_NOARGS },
{ "upgrade", (PyCFunction)parser_upgrade, METH_NOARGS },
{ "finish", (PyCFunction)parser_finish, METH_NOARGS },
{ "reset", (PyCFunction)parser_reset, METH_NOARGS },
{ "on_message_begin", (PyCFunction)parser_dummy_noargs, METH_NOARGS },
{ "on_url", (PyCFunction)parser_dummy_onearg, METH_O },
{ "on_url_complete", (PyCFunction)parser_dummy_noargs, METH_NOARGS },
{ "on_status", (PyCFunction)parser_dummy_onearg, METH_O },
{ "on_status_complete", (PyCFunction)parser_dummy_noargs, METH_NOARGS },
{ "on_header_field", (PyCFunction)parser_dummy_onearg, METH_O },
{ "on_header_field_complete", (PyCFunction)parser_dummy_noargs, METH_NOARGS },
{ "on_header_value", (PyCFunction)parser_dummy_onearg, METH_O },
{ "on_header_value_complete", (PyCFunction)parser_dummy_noargs, METH_NOARGS },
{ "on_headers_complete", (PyCFunction)parser_dummy_noargs, METH_NOARGS },
{ "on_body", (PyCFunction)parser_dummy_onearg, METH_O },
{ "on_message_complete", (PyCFunction)parser_dummy_noargs, METH_NOARGS },
{ "on_chunk_header", (PyCFunction)parser_dummy_noargs, METH_NOARGS },
{ "on_chunk_complete", (PyCFunction)parser_dummy_noargs, METH_NOARGS },
{ NULL }
};
static PyObject *parser_method(PyObject *self, void *closure) {
llhttp_t *llhttp = &((llhttp_obj_t*)self)->llhttp;
if (llhttp->type != HTTP_REQUEST)
Py_RETURN_NONE;
if (!llhttp->http_major && !llhttp->http_minor)
Py_RETURN_NONE;
return PyUnicode_FromString(methods[llhttp->method]);
}
static PyObject *parser_major(PyObject *self, void *closure) {
llhttp_t *llhttp = &((llhttp_obj_t*)self)->llhttp;
if (!llhttp->http_major && !llhttp->http_minor)
Py_RETURN_NONE;
return PyLong_FromUnsignedLong(llhttp->http_major);
}
static PyObject *parser_minor(PyObject *self, void *closure) {
llhttp_t *llhttp = &((llhttp_obj_t*)self)->llhttp;
if (!llhttp->http_major && !llhttp->http_minor)
Py_RETURN_NONE;
return PyLong_FromUnsignedLong(llhttp->http_minor);
}
/* The numeric status of a response.
*
* llhttp has had it all along — it is a field on the parser struct, and
* `llhttp_get_status_code` reads it — but this binding never exposed it, so the only
* thing reachable from Python was the reason phrase the server wrote beside it. A filter
* asking "was this a 500?" could not be answered at all.
*
* `None` for a request, which has no status, and before a response line has been read:
* llhttp leaves the field at zero there, and zero is not a status code. */
static PyObject *parser_status_code(PyObject *self, void *closure) {
llhttp_t *llhttp = &((llhttp_obj_t*)self)->llhttp;
if (llhttp->type != HTTP_RESPONSE)
Py_RETURN_NONE;
if (!llhttp->status_code)
Py_RETURN_NONE;
return PyLong_FromUnsignedLong(llhttp_get_status_code(llhttp));
}
static PyObject *parser_content_length(PyObject *self, void *closure) {
llhttp_t *llhttp = &((llhttp_obj_t*)self)->llhttp;
if (!(llhttp->flags & F_CONTENT_LENGTH))
Py_RETURN_NONE;
return PyLong_FromUnsignedLong(llhttp->content_length);
}
static bool get_lenient(const llhttp_t *llhttp, llhttp_lenient_flags_t flag) {
return llhttp->lenient_flags & flag;
}
static int set_lenient(llhttp_t *llhttp, llhttp_lenient_flags_t flag, bool value) {
if (value) {
llhttp->lenient_flags |= flag;
} else {
llhttp->lenient_flags &= ~flag;
}
return 0;
}
#define LENIENT_FLAG(name) \
static PyObject * \
parser_get_lenient_ ## name(PyObject *self, void *closure) \
{ return PyBool_FromLong(get_lenient(&((llhttp_obj_t*)self)->llhttp, LENIENT_ ## name)); } \
\
static int \
parser_set_lenient_ ## name(PyObject *self, PyObject *value, void *closure) \
{ return set_lenient(&((llhttp_obj_t*)self)->llhttp, LENIENT_ ## name, PyObject_IsTrue(value)); }
LENIENT_FLAG(HEADERS);
LENIENT_FLAG(CHUNKED_LENGTH);
LENIENT_FLAG(KEEP_ALIVE);
LENIENT_FLAG(TRANSFER_ENCODING);
LENIENT_FLAG(VERSION);
static PyObject *parser_get_lenient_headers(PyObject *self, void *closure) {
llhttp_t *llhttp = &((llhttp_obj_t*)self)->llhttp;
return PyBool_FromLong(llhttp->lenient_flags & LENIENT_HEADERS);
}
static int parser_set_lenient_headers(PyObject *self, PyObject *value, void *closure) {
llhttp_t *llhttp = &((llhttp_obj_t*)self)->llhttp;
llhttp_set_lenient_headers(llhttp, PyObject_IsTrue(value));
return 0;
}
static PyObject *parser_get_lenient_chunked_length(PyObject *self, void *closure) {
llhttp_t *llhttp = &((llhttp_obj_t*)self)->llhttp;
return PyBool_FromLong(llhttp->lenient_flags & LENIENT_CHUNKED_LENGTH);
}
static int parser_set_lenient_chunked_length(PyObject *self, PyObject *value, void *closure) {
llhttp_t *llhttp = &((llhttp_obj_t*)self)->llhttp;
llhttp_set_lenient_chunked_length(llhttp, PyObject_IsTrue(value));
return 0;
}
static PyObject *parser_get_lenient_keep_alive(PyObject *self, void *closure) {
llhttp_t *llhttp = &((llhttp_obj_t*)self)->llhttp;
return PyBool_FromLong(llhttp->lenient_flags & LENIENT_KEEP_ALIVE);
}
static int parser_set_lenient_keep_alive(PyObject *self, PyObject *value, void *closure) {
llhttp_t *llhttp = &((llhttp_obj_t*)self)->llhttp;
llhttp_set_lenient_keep_alive(llhttp, PyObject_IsTrue(value));
return 0;
}
static PyObject *parser_message_needs_eof(PyObject *self, void *closure) {
llhttp_t *llhttp = &((llhttp_obj_t*)self)->llhttp;
return PyBool_FromLong(llhttp_message_needs_eof(llhttp));
}
static PyObject *parser_should_keep_alive(PyObject *self, void *closure) {
llhttp_t *llhttp = &((llhttp_obj_t*)self)->llhttp;
return PyBool_FromLong(llhttp_should_keep_alive(llhttp));
}
static PyObject *parser_is_paused(PyObject *self, void *closure) {
llhttp_t *llhttp = &((llhttp_obj_t*)self)->llhttp;
return PyBool_FromLong(HPE_PAUSED == llhttp_get_errno(llhttp));
}
static PyObject *parser_is_upgrading(PyObject *self, void *closure) {
llhttp_t *llhttp = &((llhttp_obj_t*)self)->llhttp;
switch (llhttp_get_errno(llhttp)) {
case HPE_PAUSED_UPGRADE:
case HPE_PAUSED_H2_UPGRADE:
Py_RETURN_TRUE;
break;
default:
Py_RETURN_FALSE;
break;
}
}
static PyObject *parser_is_busted(PyObject *self, void *closure) {
llhttp_t *llhttp = &((llhttp_obj_t*)self)->llhttp;
switch (llhttp_get_errno(llhttp)) {
case HPE_OK:
case HPE_PAUSED:
case HPE_PAUSED_UPGRADE:
Py_RETURN_FALSE;
default:
Py_RETURN_TRUE;
}
}
static PyObject *parser_error(PyObject *self, void *closure) {
llhttp_t *llhttp = &((llhttp_obj_t*)self)->llhttp;
if (HPE_OK == llhttp_get_errno(llhttp))
Py_RETURN_NONE;
return PyUnicode_FromString(llhttp_get_error_reason(llhttp));
}
static PyGetSetDef parser_getset[] = {
{ "method", parser_method },
{ "status_code", parser_status_code },
{ "major", parser_major },
{ "minor", parser_minor },
{ "content_length", parser_content_length },
{ "lenient_headers", parser_get_lenient_HEADERS, parser_set_lenient_HEADERS },
{ "lenient_chunked_length", parser_get_lenient_CHUNKED_LENGTH, parser_set_lenient_CHUNKED_LENGTH },
{ "lenient_keep_alive", parser_get_lenient_KEEP_ALIVE, parser_set_lenient_KEEP_ALIVE },
{ "lenient_transfer_encoding", parser_get_lenient_TRANSFER_ENCODING, parser_set_lenient_TRANSFER_ENCODING },
{ "lenient_version", parser_get_lenient_VERSION, parser_set_lenient_VERSION },
{ "message_needs_eof", parser_message_needs_eof },
{ "should_keep_alive", parser_should_keep_alive },
{ "is_paused", parser_is_paused },
{ "is_upgrading", parser_is_upgrading },
{ "is_busted", parser_is_busted },
{ "error", parser_error },
{ NULL }
};
static void parser_dealloc(PyObject *self) {
llhttp_obj_t *llhttp = (llhttp_obj_t*)self;
Py_TYPE(self)->tp_free(self);
}
static PyType_Slot request_slots[] = {
{Py_tp_doc, "llhttp request parser"},
{Py_tp_new, request_new},
{Py_tp_dealloc, parser_dealloc},
{Py_tp_methods, parser_methods},
{Py_tp_getset, parser_getset},
{0, 0},
};
static PyType_Spec request_spec = {
MODULE_NAME ".Request",
sizeof(llhttp_obj_t),
0,
Py_TPFLAGS_DEFAULT | Py_TPFLAGS_BASETYPE,
request_slots,
};
static PyType_Slot response_slots[] = {
{Py_tp_doc, "llhttp response parser"},
{Py_tp_new, response_new},
{Py_tp_dealloc, parser_dealloc},
{Py_tp_methods, parser_methods},
{Py_tp_getset, parser_getset},
{0, NULL},
};
static PyType_Spec response_spec = {
MODULE_NAME ".Response",
sizeof(llhttp_obj_t),
0, Py_TPFLAGS_DEFAULT | Py_TPFLAGS_BASETYPE,
response_slots,
};
/* One class per error code, per module instance: two subinterpreters must never share
* an exception object, which is what `Py_MOD_PER_INTERPRETER_GIL_SUPPORTED` promises. */
static int add_error_classes(PyObject *m) {
PyObject *base_error = PyErr_NewException(MODULE_NAME ".Error", NULL, NULL);
if (!base_error)
return -1;
if (PyModule_AddObjectRef(m, "Error", base_error) < 0) {
Py_DECREF(base_error);
return -1;
}
for (size_t i = 0; i < sizeof(errors)/sizeof(errors[0]); ++i) {
char class_name[128];
char qualified_name[128 + sizeof(MODULE_NAME) + 1];
error_class_name(errors[i].name, class_name, sizeof(class_name));
snprintf(qualified_name, sizeof(qualified_name), MODULE_NAME ".%s", class_name);
PyObject *exception = PyErr_NewException(qualified_name, base_error, NULL);
if (!exception) {
Py_DECREF(base_error);
return -1;
}
int added = PyModule_AddObjectRef(m, class_name, exception);
Py_DECREF(exception);
if (added < 0) {
Py_DECREF(base_error);
return -1;
}
}
Py_DECREF(base_error);
return 0;
}
static int add_type(PyObject *m, PyType_Spec *spec) {
PyObject *type = PyType_FromSpec(spec);
if (!type)
return -1;
/* The spec carries the dotted name; the module wants the last component of it. */
int added = PyModule_AddObjectRef(m, spec->name + strlen(MODULE_NAME "."), type);
Py_DECREF(type);
return added;
}
static int init_llhttp_module(PyObject *m) {
if (PyModule_AddStringConstant(m, "version", LLHTTP_VERSION) < 0)
goto fail;
for (size_t i = 0; i < sizeof(errors)/sizeof(errors[0]); ++i) {
if (PyModule_AddIntConstant(m, errors[i].name, errors[i].code) < 0)
goto fail;
}
if (add_error_classes(m) < 0)
goto fail;
if (add_type(m, &request_spec) < 0)
goto fail;
if (add_type(m, &response_spec) < 0)
goto fail;
return 0;
fail:
/* The module belongs to the import machinery, which destroys it when a `Py_mod_exec`
* slot fails; releasing it here as well handed back a reference that was never ours.
* Whatever raised first is the useful message, so nothing is overwritten either. */
if (!PyErr_Occurred())
PyErr_SetString(PyExc_RuntimeError, "Failed to initialize the llhttp module");
return -1;
}
static PyModuleDef_Slot llhttp_slots[] = {
{Py_mod_exec, init_llhttp_module},
{Py_mod_multiple_interpreters, Py_MOD_PER_INTERPRETER_GIL_SUPPORTED},
{0, NULL}
};
static struct PyModuleDef llhttp_module = {
PyModuleDef_HEAD_INIT,
.m_name = "firegex._llhttp",
.m_doc = "llhttp wrapper",
.m_slots = llhttp_slots,
.m_size = 0,
};
PyMODINIT_FUNC
PyInit__llhttp(void) {
return PyModuleDef_Init(&llhttp_module);
}

4
fgex-lib/requirements.txt
View File

@ -6,4 +6,6 @@ watchfiles
fgex
websockets
brotli
pyllhttp
# Python 3.14 has `compression.zstd` in the standard library; this fills the gap
# below it. See firegex/pyfilters/internals/zstd_compat.py.
zstandard; python_version < '3.14'

25
fgex-lib/setup.py
View File

@ -1,5 +1,28 @@
import setuptools
# The HTTP parser, built from source that ships with this package.
#
# It is llhttp — the parser Node uses — behind a binding that used to live in its own
# package, `pyllhttp`. Three things made vendoring it the only arrangement that works:
# the package was archived; every alternative binding with prebuilt wheels fails to load
# in the subinterpreters `cpproxy` runs filters in (they need
# `Py_MOD_PER_INTERPRETER_GIL_SUPPORTED`, which this declares and `httptools` does not);
# and binding the system `libllhttp` through ctypes would mean `pip install firegex`
# assuming a package the user had installed beforehand.
#
# So the C comes with the source, and the release workflow builds wheels for every
# platform it can. Where no wheel matches, pip compiles this — which needs a C toolchain,
# and is the deliberate trade: compiling is a thing pip knows how to do, assuming a
# system library is not.
LLHTTP = setuptools.Extension(
"firegex._llhttp",
sources=["llhttp/llhttp_module.c", "llhttp/lib/llhttp.c",
"llhttp/lib/http.c", "llhttp/lib/api.c"],
include_dirs=["llhttp/lib"],
language="c",
extra_compile_args=["-O3"],
)
with open("README.md", "r", encoding="utf-8") as fh:
long_description = fh.read()
@ -22,10 +45,10 @@ setuptools.setup(
long_description_content_type="text/markdown",
url="https://github.com/pwnzer0tt1/firegex",
packages=setuptools.find_packages(),
ext_modules=[LLHTTP],
classifiers=[
"Programming Language :: Python :: 3",
"License :: OSI Approved :: GNU General Public License v3 or later (GPLv3+)",
"Operating System :: OS Independent",
],
python_requires='>=3.10',
)

36
fgex-lib/wheel_smoke.py Normal file
View File

@ -0,0 +1,36 @@
"""What a published wheel has to do, checked where nothing else is installed.
`cibuildwheel` runs this against each built wheel in a clean environment, which is the
one property that matters here: it is the only place in the whole pipeline that looks
like a user's machine rather than a developer's. The error path of the parser binding
once imported `pyllhttp` — archived, not a dependency, present on both machines this was
written on and on no other — and every check we had passed anyway, because every check
we had parsed *valid* traffic.
So: a good exchange, a bad one, and the exception that names it.
"""
import sys
from firegex import _llhttp
response = _llhttp.Response()
response.execute(b"HTTP/1.1 404 Not Found\r\nContent-Length: 0\r\n\r\n")
assert response.status_code == 404, response.status_code
assert _llhttp.Request.__module__ == "firegex._llhttp", _llhttp.Request.__module__
assert _llhttp.Error.__module__ == "firegex._llhttp", _llhttp.Error.__module__
request = _llhttp.Request()
try:
request.execute(b"NOTAMETHOD / HTTP/1.1\r\n\r\n")
except _llhttp.InvalidMethodError as error:
assert "method" in str(error).lower(), error
else:
raise AssertionError("a malformed request was parsed without complaint")
# Not a detail: borrowing the exception from somewhere else is exactly what used to
# happen, and it only ever worked where that somewhere else happened to be installed.
assert "pyllhttp" not in sys.modules, "the parser reached for pyllhttp"
print(f"llhttp {_llhttp.version} ok on {sys.implementation.name} {sys.version.split()[0]}")

393
frontend/src/components/InterfaceInput.tsx
View File

@ -1,123 +1,348 @@
import { Combobox, TextInput, useCombobox } from "@mantine/core";
import { useEffect, useState } from "react";
import { ipInterfacesQuery } from "../js/utils";
import { Badge, Box, Combobox, Group, Loader, Text, TextInput, useCombobox } from "@mantine/core";
import { useViewportSize } from "@mantine/hooks";
import { useMemo, useState } from "react";
import { TbCheck, TbDeviceDesktopSearch, TbPlugConnected } from "react-icons/tb";
import { bareAddress, ipInterfacesQuery, isInterfaceName, isIpAddress } from "../js/utils";
interface ItemProps{
/** A value the caller wants offered beside what the host actually has (`ANY`, a CIDR). */
export interface InterfaceOption {
value: string
netint: string;
/** The badge it is listed under. */
netint: string
label?: string
}
interface InterfaceInputProps {
initialCustomInterfaces?:ItemProps[],
includeInterfaceNames?:boolean,
onChange?: (value:string) => void,
value?:string,
initialCustomInterfaces?: InterfaceOption[]
/** Whether interface *names* are a valid answer here, or only addresses. */
includeInterfaceNames?: boolean
onChange?: (value: string) => void
value?: string
defaultValue?: string
placeholder?: string
error?: React.ReactNode
disabled?: boolean
}
export const InterfaceInput = ({ initialCustomInterfaces, includeInterfaceNames, onChange, value, defaultValue }:InterfaceInputProps) => {
type Kind = { label: string, color: string }
const [customIpInterfaces, setCustomIpInterfaces] = useState<ItemProps[]>(initialCustomInterfaces??[]);
/** One width for the badge everywhere it appears, so the field and the list line up. */
export const BADGE_WIDTH = 86
/**
* The narrowest the note beside a value may be squeezed before it moves to its own line.
* Just wide enough for `on docker0`; the point of having one at all is that without it a
* long value took every pixel and left the note wrapping towards a character per line.
*/
const DETAIL_FLOOR = 80
/**
* How wide the list is: what *this* host needs, between a floor and a ceiling.
*
* Three ways of deciding this were wrong before this one. Left to the content it
* collapsed to the field's width, because content allowed to wrap asks for less than it
* needs to stay on one line — 200px of list in a firewall rule. Pinned to the widest row
* any host could produce, every list was that wide whether or not anything in it was
* long, which beside a field half the size reads as a panel that escaped. A floor alone
* put the one long address back over three lines.
*
* So it is measured from the longest value actually in the list: a host whose addresses
* are `10.60.3.1` and `wg0` gets the floor, and one carrying a full-length IPv6 gets the
* room to show it with its note beside it. Monospace is what makes this honest — every
* character is the same width, so the arithmetic is the layout's, not a guess.
*/
const DROPDOWN_FLOOR = 420
const DROPDOWN_CEILING = 560
//: The value is `size="sm"` monospace, whose advance is 0.6em of 14px.
const CH = 8.5
//: What the row costs before the value: the list's padding, the option's, the badge and
// the two gaps, plus the floor the note keeps for itself.
const ROW_CHROME = 36 + BADGE_WIDTH + 16 + DETAIL_FLOOR
const dropdownWidth = (longestValue: number, viewport: number) => {
const room = Math.max(240, viewport - 24)
const needed = ROW_CHROME + Math.ceil(longestValue * CH)
return Math.min(room, Math.max(
Math.min(DROPDOWN_FLOOR, room),
Math.min(needed, DROPDOWN_CEILING),
))
}
/**
* What the operator has typed, named.
*
* The badge is not decoration: an interface name and an address are matched by
* completely different nftables rules, and a typo in one reads as the other — `eth-0`
* is a perfectly good interface name and a perfectly wrong address. Saying which one
* firegex understood, while it is being typed, is the cheapest place to catch that.
*/
export const addressKind = (value: string, interfacesAllowed = true): Kind => {
//: `/32` and `/128` are how the backend spells one address, not a range.
const v = bareAddress(value.trim())
if (v === "") return { label: "ANY", color: "gray" }
if (isIpAddress(v, { cidr: true }))
return v.includes("/")
? { label: v.includes(":") ? "IPv6 NET" : "IPv4 NET", color: "violet" }
: { label: v.includes(":") ? "IPv6" : "IPv4", color: "teal" }
// Named red where this field does not take one, rather than blue and then refused on
// submit: the two are told apart by eye here, and "that is an interface" is only
// reassuring somewhere an interface is an answer.
if (isInterfaceName(v))
return interfacesAllowed
? { label: "INTERFACE", color: "blue" }
: { label: "INVALID", color: "red" }
return { label: "INVALID", color: "red" }
}
type Row = {
value: string
/** What is written in bold on the option. */
title: string
/** The dimmed half: an interface's addresses, or an address's interface. */
detail: string
badge: string
color: string
group: string
/** Lowercased haystack, so typing an interface name also finds its addresses. */
haystack: string
}
/**
* A link-local address is not a place anybody dials.
*
* Every interface carries an `fe80::` it was never configured with — offering them
* doubles the list with addresses that would put a service where no client will look
* for it, which is the same reason `udp_relay_host()` picks them last on the backend.
* They are still accepted if one is typed; they are just not *offered*, and an
* interface that has nothing else says so rather than listing one.
*/
const isLinkLocal = (addr: string) =>
addr.toLowerCase().startsWith("fe80:") || addr.startsWith("169.254.")
const IFACES_GROUP = "Network interfaces"
const ADDRS_GROUP = "Addresses on this host"
const PRESET_GROUP = "Common choices"
/**
* Where an address — or an interface — is chosen.
*
* Both, because the datapath takes both, and the two are genuinely different things
* rather than two spellings of one: an address is a place, an interface is *wherever
* that cable currently is*, which is what you want for a link whose address is handed
* out by somebody else. So the list says which is which, and an interface carries the
* addresses it holds right now — the one question an operator has when choosing between
* them is "and what is on it".
*
* Free text is still the point: this is a combobox and not a select, because the
* address a service will answer on does not have to exist on this host yet.
*/
export const InterfaceInput = ({
initialCustomInterfaces, includeInterfaceNames, onChange, value, defaultValue,
placeholder, error, disabled,
}: InterfaceInputProps) => {
const interfacesQuery = ipInterfacesQuery()
const { width: viewport } = useViewportSize()
const getInterfaces = () => {
if (interfacesQuery.isLoading || !interfacesQuery.data) return []
if(includeInterfaceNames){
const result = interfacesQuery.data.map(item => ({netint:"IP", value:item.addr})) as ItemProps[]
interfacesQuery.data.map(item => item.name).filter((item, index, arr) => arr.indexOf(item) === index).forEach(item => result.push({netint:"INT", value:item}))
return result
}
return (interfacesQuery.data.map(item => ({netint:item.name, value:item.addr})) as ItemProps[])
const controlled = value !== undefined
const [internal, setInternal] = useState(defaultValue ?? "")
const current = controlled ? value : internal
//: What is being *searched for*, which is only the same as the value while typing.
// Opening the dropdown clears it, so a click always shows the whole host rather
// than the one option whose name is already in the box.
const [query, setQuery] = useState<string | null>(null)
const combobox = useCombobox({ onDropdownClose: () => combobox.resetSelectedOption() })
const rows = useMemo<Row[]>(() => {
const addrs = interfacesQuery.data ?? []
//: One entry per interface, carrying every address it holds.
const byIface = new Map<string, string[]>()
//: …and the other way round, because one address can be on more than one.
const byAddr = new Map<string, string[]>()
for (const { name, addr } of addrs) {
if (!byIface.has(name)) byIface.set(name, [])
if (!byIface.get(name)!.includes(addr)) byIface.get(name)!.push(addr)
if (!byAddr.has(addr)) byAddr.set(addr, [])
if (!byAddr.get(addr)!.includes(name)) byAddr.get(addr)!.push(name)
}
const interfaces = getInterfaces()
const presets: Row[] = (initialCustomInterfaces ?? []).map(item => ({
value: item.value,
title: item.label ?? (item.value === "" ? "Any address" : item.value),
detail: "",
badge: item.netint,
color: "gray",
group: PRESET_GROUP,
haystack: `${item.label ?? ""} ${item.value} ${item.netint}`.toLowerCase(),
}))
const combobox = useCombobox({
onDropdownClose: () => {
combobox.resetSelectedOption()
},
});
const ifaces: Row[] = includeInterfaceNames
? [...byIface.entries()].sort((a, b) => a[0].localeCompare(b[0])).map(([name, ips]) => ({
value: name,
title: name,
// A no-break space before each separator, so a line that has to wrap
// breaks *between* two addresses and never leaves a dot stranded at the
// start of the next one.
detail: ips.some(ip => !isLinkLocal(ip))
? ips.filter(ip => !isLinkLocal(ip)).join("\u00a0· ")
: (ips.length > 0 ? "link-local only" : "no address assigned"),
badge: "INTERFACE",
color: "blue",
group: IFACES_GROUP,
haystack: `${name} ${ips.join(" ")}`.toLowerCase(),
}))
: []
const data = [...customIpInterfaces.filter(v => !interfaces.map(v => v.value).includes(v.value)), ...interfaces]
const [selectedValue, setSelectedValue] = useState<string | null>(null);
const [search, setSearch] = useState('');
const ips: Row[] = [...byAddr.entries()]
.filter(([addr]) => !isLinkLocal(addr))
.sort((a, b) => (a[0].includes(":") === b[0].includes(":"))
? a[0].localeCompare(b[0])
: (a[0].includes(":") ? 1 : -1))
.map(([addr, names]) => ({
value: addr,
title: addr,
detail: `on ${names.join(", ")}`,
badge: addr.includes(":") ? "IPv6" : "IPv4",
color: "teal",
group: ADDRS_GROUP,
haystack: `${addr} ${names.join(" ")}`.toLowerCase(),
}))
const exactOptionMatch: ItemProps|undefined = data.find((item) => item.value === search);
const seen = new Set(presets.map(p => p.value))
return [...presets, ...[...ifaces, ...ips].filter(r => !seen.has(r.value))]
}, [interfacesQuery.data, includeInterfaceNames, initialCustomInterfaces])
const filteredOptions = data.filter((item) => item.value.toLowerCase().includes(search.toLowerCase().trim())).sort((a, b) => {
if (exactOptionMatch != null) {
if (a.value == exactOptionMatch.value) return -1
if (b.value == exactOptionMatch.value) return 1
const search = (query ?? "").toLowerCase().trim()
const visible = search === "" ? rows : rows.filter(r => r.haystack.includes(search))
const exact = rows.some(r => r.value === (query ?? current).trim())
const commit = (v: string) => {
if (!controlled) setInternal(v)
onChange?.(v)
}
return a.value.localeCompare(b.value)
});
const options = filteredOptions.sort( (a, b) => a.value == selectedValue? -1 : b.value == selectedValue? 1: a.value.localeCompare(b.value) ).map((item) => (
<Combobox.Option value={item.value} key={item.value}>
( <b>{item.value == selectedValue ? "SELECTED" : item.netint}</b> ) -{">"} <b>{item.value}</b>
/**
* The badge is a column; everything else lives in the one beside it.
*
* Not three items in one wrapping row, which is what this was: a value too long to
* sit beside the badge wrapped to the *container's* left edge, under the badge,
* leaving a row three lines tall whose address started nowhere near the addresses
* above it. Inside its own column it wraps under itself instead, and the detail
* follows it — the value and its note stay one block, aligned down the list.
*
* Which of the two is the long one changes with the group — the addresses a link
* carries in one, the address itself in the other — so neither is given a fixed
* share: the value takes what it needs and the detail keeps a floor just wide
* enough for `on docker0`, below which it moves to its own line rather than being
* squeezed towards a character per line. The floor is what the longest address on
* a host leaves over at the list's width, so that row stays on one line too.
*
* The tick goes next to the value rather than at the end of the row, because the
* end of the row is a place that moves.
*/
const option = (row: Row) => <Combobox.Option value={row.value} key={`${row.group}/${row.value}`}
active={row.value === current}>
<Group gap="xs" wrap="nowrap" align="flex-start">
<Badge size="xs" variant="light" color={row.color}
style={{ flexShrink: 0, width: BADGE_WIDTH, marginTop: 2 }}>{row.badge}</Badge>
<Group gap="xs" wrap="wrap" align="baseline"
style={{ flex: '1 1 auto', minWidth: 0, rowGap: 0 }}>
<Text size="sm" ff="monospace" style={{ minWidth: 0, overflowWrap: 'anywhere' }}>
{row.title}
</Text>
{row.value === current
? <TbCheck size={14} style={{ flexShrink: 0 }} color="var(--mantine-color-teal-4)" />
: null}
{row.detail ? <Text size="xs" c="dimmed" ff="monospace"
style={{
flex: `1 1 ${DETAIL_FLOOR}px`, minWidth: DETAIL_FLOOR,
textAlign: 'right', overflowWrap: 'anywhere',
}}>{row.detail}</Text> : null}
</Group>
</Group>
</Combobox.Option>
));
useEffect(() => {
setSelectedValue(data.find((item) => item.value === defaultValue)?.value??null)
setSearch(defaultValue??'')
}, [])
return <>
<Combobox
store={combobox}
withinPortal={false}
position="bottom-end"
onOptionSubmit={(value) => {
if (value === '$create') {
const item = { value: search, netint: "CUSTOM" };
setCustomIpInterfaces((current) => [...current, item]);
setSelectedValue(search);
onChange?.(search)
} else {
setSelectedValue(value);
setSearch(value);
onChange?.(value)
const group = (name: string, icon: React.ReactNode) => {
const inside = visible.filter(r => r.group === name)
if (inside.length === 0) return null
return <Combobox.Group label={<Group gap={6}>{icon}<span>{name}</span></Group>}>
{inside.map(option)}
</Combobox.Group>
}
combobox.closeDropdown();
const kind = addressKind(current ?? "", includeInterfaceNames !== false)
const typed = (query ?? "").trim()
const typedKind = addressKind(typed, includeInterfaceNames !== false)
return <Combobox
store={combobox}
width={dropdownWidth(
rows.reduce((longest, r) => Math.max(longest, r.title.length), 0),
viewport,
)}
// Portalled, so the list is not clipped by whatever it sits in: inside a firewall
// rule's card it used to be cut off halfway down the options.
position="bottom-start"
disabled={disabled}
onOptionSubmit={(v) => {
commit(v)
setQuery(null)
combobox.closeDropdown()
}}
>
<Combobox.Target>
<TextInput
style={{ width: "100%" }}
rightSection={<Combobox.Chevron />}
value={value??search}
placeholder="10.1.1.1"
disabled={disabled}
error={error}
value={query ?? current ?? ""}
placeholder={placeholder ?? (includeInterfaceNames
? "IP address or interface name" : "IP address")}
leftSectionWidth={BADGE_WIDTH + 16}
leftSection={<Badge size="xs" variant="light" color={kind.color}
style={{ width: BADGE_WIDTH }}>{kind.label}</Badge>}
rightSection={interfacesQuery.isLoading ? <Loader size={14} /> : <Combobox.Chevron />}
rightSectionPointerEvents="none"
onChange={(event) => {
combobox.openDropdown();
combobox.updateSelectedOptionIndex();
setSearch(event.currentTarget.value)
onChange?.(event.currentTarget.value)
}}
onClick={(e) => {
const v = event.currentTarget.value
setQuery(v)
commit(v)
combobox.openDropdown()
combobox.updateSelectedOptionIndex()
}}
onFocus={(e) => {
combobox.openDropdown()
}}
onBlur={(e) => {
combobox.closeDropdown();
setSearch(selectedValue??'');
}}
// Opening clears the search rather than filtering by whatever is already
// in the box: the value is usually an exact option, and filtering by it
// would show the operator only the thing they were trying to change.
onClick={() => { setQuery(null); combobox.openDropdown() }}
onFocus={() => { setQuery(null); combobox.openDropdown() }}
onBlur={() => { setQuery(null); combobox.closeDropdown() }}
/>
</Combobox.Target>
<Combobox.Dropdown>
<Combobox.Options mah={200} style={{ overflowY: 'auto' }}>
{options}
{(exactOptionMatch==null) && search.trim().length > 0 && (
<Combobox.Option value="$create">+ Use this: {search}</Combobox.Option>
)}
<Combobox.Options mah={260} style={{ overflowY: 'auto' }}>
{group(PRESET_GROUP, null)}
{group(IFACES_GROUP, <TbPlugConnected size={13} />)}
{group(ADDRS_GROUP, <TbDeviceDesktopSearch size={13} />)}
{!exact && typed.length > 0 ? <Combobox.Option value={typed}>
<Group gap="xs" wrap="nowrap">
<Badge size="xs" variant="light" color={typedKind.color}
style={{ flexShrink: 0, width: BADGE_WIDTH }}>{typedKind.label}</Badge>
<Text size="sm">Use <Text span ff="monospace">{typed}</Text></Text>
</Group>
</Combobox.Option> : null}
{visible.length === 0 && typed.length === 0 ? <Combobox.Empty>
{interfacesQuery.isLoading ? "Reading this host's interfaces…"
: "This host reports no address"}
</Combobox.Empty> : null}
</Combobox.Options>
<Box px="xs" py={6} style={{ borderTop: '1px solid var(--mantine-color-dark-4)' }}>
<Text size="xs" c="dimmed">
{includeInterfaceNames
? "An interface name follows the link: whatever address it is given, the rules match on the name. An address protects that one address only."
: "One address here, not an interface name — the rules built from it need a single address to rewrite."}
</Text>
</Box>
</Combobox.Dropdown>
</Combobox>
</>
}

2
frontend/src/components/MarkdownDocs.tsx
View File

@ -9,7 +9,7 @@ function toHighlightLanguage(lang: string | undefined): string {
const GITHUB_BLOB_BASE = "https://github.com/Pwnzer0tt1/firegex/blob/main/";
// Cross-references between doc files (e.g. "nfregex.md", "../fgex-lib/README.md")
// Cross-references between doc files (e.g. "pyfilter.md", "../fgex-lib/README.md")
// are relative to the file's position in the repo, which only resolves correctly
// on GitHub - inside the app they'd resolve against the current page's URL instead.
// Repoint any relative link at the equivalent file on GitHub, given where the

35
frontend/src/components/PortAndInterface.tsx
View File

@ -1,25 +1,14 @@
import { Box, Space, Title } from "@mantine/core"
import React from "react"
import PortInput from "./PortInput";
import { Box, Space, Text, Title } from "@mantine/core"
import { UseFormReturnType } from "@mantine/form";
import { InterfaceInput } from "./InterfaceInput";
type ItemProps = {
value: string;
netint: string;
} & any
const AutoCompleteItem = React.forwardRef<HTMLDivElement, ItemProps>(
({ netint, value, ...props }, ref) => <Box ref={ref} {...props}>
( <b>{netint}</b> ) -{">"} <b>{value}</b>
</Box>
);
import PortInput from "./PortInput";
export default function PortAndInterface({
form,
int_name,
port_name,
label,
description,
orientation,
includeInterfaceNames = true,
}: {
@ -27,22 +16,34 @@ export default function PortAndInterface({
int_name: string,
port_name: string,
label?: string,
/** One line under the label, for what the field takes that is not obvious. */
description?: string,
orientation?: "line" | "column",
includeInterfaceNames?: boolean,
}) {
const line = !orientation || orientation == "line"
// One message under the pair, because either half can be the wrong one and they
// sit on a single line. The address box never received the form's `error` at all,
// so "Invalid IP address or interface name" was computed on every submit and shown
// nowhere; both boxes still turn red, which is what says which half it was.
const error = form.getInputProps(int_name).error ?? form.getInputProps(port_name).error
return <>
{label ? <>
<Title order={6}>{label}</Title>
{description ? <Text size="xs" c="dimmed" mt={2}>{description}</Text> : null}
<Space h="xs" /></> : null}
<Box className={(!orientation || orientation == "line")?'center-flex':"center-flex-row"} style={{width:"100%"}}>
<Box className={line ? 'center-flex' : "center-flex-row"} style={{ width: "100%" }}>
<InterfaceInput
{...form.getInputProps(int_name)}
error={!!form.getInputProps(int_name).error}
includeInterfaceNames={includeInterfaceNames}
/>
{(!orientation || orientation == "line")?
{line ?
<><Space w="sm" /><span style={{ marginTop: "-3px", fontSize: "1.5em" }}>:</span><Space w="sm" /></> :
<Space h="md" />}
<PortInput {...form.getInputProps(port_name)} />
<PortInput {...form.getInputProps(port_name)} error={!!form.getInputProps(port_name).error} />
</Box>
{error ? <Text size="xs" c="red" mt={4}>{error}</Text> : null}
</>
}

48
frontend/src/components/Services/AddEditService.tsx
View File

@ -1,9 +1,9 @@
import { Accordion, ActionIcon, Alert, Box, Button, Group, Modal, NumberInput, SegmentedControl, Space, Switch, Text, TextInput, Tooltip } from '@mantine/core';
import { Accordion, ActionIcon, Alert, Box, Button, Code, Group, Modal, NumberInput, SegmentedControl, Space, Switch, Text, TextInput, Tooltip } from '@mantine/core';
import { useForm } from '@mantine/form';
import { useQueryClient } from '@tanstack/react-query';
import { useEffect, useState } from 'react';
import { BsPlusLg, BsTrashFill } from 'react-icons/bs';
import { errorNotify, okNotify, regex_ipv4_no_cidr, regex_ipv6_no_cidr } from '../../js/utils';
import { errorNotify, isAddressOrInterface, isInterfaceName, okNotify } from '../../js/utils';
import PortAndInterface from '../PortAndInterface';
import LayerChoice from './LayerChoice';
import PemInput from './PemInput';
@ -34,12 +34,6 @@ const emptyAddress = (): AddressValues => ({
ip_int: "127.0.0.1", port: 80, proxy_ip: "127.0.0.1", proxy_port: 8080,
})
const isInterface = (v: string) =>
!v.match(regex_ipv6_no_cidr) && !v.match(regex_ipv4_no_cidr) && v.trim().length > 0 && v.trim().length <= 15 && !!v.trim().match(/^[a-zA-Z0-9_.:-]+$/)
const validAddress = (v: string) =>
!!(v.match(regex_ipv6_no_cidr) || v.match(regex_ipv4_no_cidr) || isInterface(v))
/**
* Create or edit the network layer of a service.
*
@ -97,7 +91,12 @@ export default function AddEditService({ opened, onClose, edit }: {
: "Not a PEM private key: no 'BEGIN PRIVATE KEY' block in it"
},
addresses: {
ip_int: (v, values) => validAddress(v) ? (values.transport === Transport.EXTERNAL && isInterface(v) ? "External transport requires an IP, not an interface" : null) : "Invalid IP address or interface name",
ip_int: (v, values) => !isAddressOrInterface(v,
{ cidr: values.transport !== Transport.EXTERNAL })
? "Not an IP address, and not an interface name either"
: (values.transport === Transport.EXTERNAL && isInterfaceName(v)
? "Your own proxy is handed one address: the return rule has to put the original port back, and an interface is not one address"
: null),
port: v => (v > 0 && v < 65536) ? null : "Invalid port",
proxy_port: (v, values) => (values.transport !== Transport.EXTERNAL || (v > 0 && v < 65536))
? null : "Your proxy's port is required",
@ -250,16 +249,18 @@ export default function AddEditService({ opened, onClose, edit }: {
<Space h="md" />
{edit ? null : <>
<Group justify="space-between" align="center">
<Box>
<Group justify="space-between" align="center" wrap="nowrap">
<Box style={{ minWidth: 0 }}>
<Text size="sm" fw={500}>Addresses to protect</Text>
<Text size="xs" c="dimmed">
One service, one filter chain, as many addresses as it answers on —
IPv4 and IPv6 together if that is how it is reachable.
{isExternal
? "One service, one hand-off, as many addresses as it answers on — IPv4 and IPv6 together if that is how it is reachable."
: "One service, one filter chain, as many places as it answers on — IPv4 and IPv6 together if that is how it is reachable, and an interface name where the address is not yours to know."}
</Text>
</Box>
<Tooltip label="Protect another address with the same chain" position="left">
<ActionIcon variant="light" onClick={() => form.insertListItem('addresses', emptyAddress())}>
<ActionIcon variant="light" style={{ flexShrink: 0 }}
onClick={() => form.insertListItem('addresses', emptyAddress())}>
<BsPlusLg size={14} />
</ActionIcon>
</Tooltip>
@ -270,7 +271,8 @@ export default function AddEditService({ opened, onClose, edit }: {
<Box style={{ flex: 1 }}>
<PortAndInterface form={form}
int_name={`addresses.${index}.ip_int`}
port_name={`addresses.${index}.port`} />
port_name={`addresses.${index}.port`}
includeInterfaceNames={!isExternal} />
</Box>
<ActionIcon variant="subtle" color="red" mb={4}
disabled={form.values.addresses.length === 1}
@ -286,6 +288,22 @@ export default function AddEditService({ opened, onClose, edit }: {
includeInterfaceNames={false} />
</Box> : null}
</Box>)}
{/* Said once under the list rather than as a validation error per row:
the operator is choosing between an address and an interface while they
fill it in, and being told afterwards that half of what the picker
offered was never available here is the worse way to learn it. */}
<Text size="xs" c="dimmed">
{isExternal
? "Interface names are not offered on this layer: the return rule recognises your proxy by one address and port to put the original port back, and an interface is not one address."
: <>
An interface name — <Code>eth0</Code>, <Code>wg0</Code>, <Code>tun0</Code> —
protects whatever address that link currently carries, which is what you
want when somebody else hands it out. An address protects that one alone.
{form.values.proto === L4.UDP && form.values.transport === Transport.PROXY
? " On UDP this layer binds a relay to the interface's own address, so it has to have one; NFQUEUE needs none."
: ""}
</>}
</Text>
<Space h="md" />
</>}

106
frontend/src/components/Services/AddressList.tsx
View File

@ -6,19 +6,14 @@ import { useEffect, useState } from 'react';
import { BsPlusLg, BsTrashFill } from 'react-icons/bs';
import { MdEdit } from 'react-icons/md';
import { TbEye, TbShieldLock } from 'react-icons/tb';
import { errorNotify, okNotify, regex_ipv4_no_cidr, regex_ipv6_no_cidr } from '../../js/utils';
import { bareAddress, errorNotify, ipInterfacesQuery, isAddressOrInterface, isInterfaceName, isIpAddress, okNotify } from '../../js/utils';
import { addressKind, BADGE_WIDTH } from '../InterfaceInput';
import PortAndInterface from '../PortAndInterface';
import YesNoModal from '../YesNoModal';
import { Address, decrypts, Service, serviceQueryKey, services, Transport } from './utils';
type Values = { ip_int: string, port: number, proxy_ip: string, proxy_port: number }
export const isInterface = (v: string) =>
!v.match(regex_ipv6_no_cidr) && !v.match(regex_ipv4_no_cidr) && v.trim().length > 0 && v.trim().length <= 15 && !!v.trim().match(/^[a-zA-Z0-9_.:-]+$/)
export const validAddress = (v: string) =>
!!(v.match(regex_ipv6_no_cidr) || v.match(regex_ipv4_no_cidr) || isInterface(v))
function AddressModal({ opened, onClose, service, edit }: {
opened: boolean, onClose: () => void, service: Service, edit?: Address,
}) {
@ -30,9 +25,13 @@ function AddressModal({ opened, onClose, service, edit }: {
const form = useForm<Values>({
initialValues: { ip_int: "127.0.0.1", port: 80, proxy_ip: "127.0.0.1", proxy_port: 8080 },
validate: {
ip_int: v => validAddress(v) ? (isExternal && isInterface(v) ? "External transport requires an IP, not an interface" : null) : "Invalid IP address or interface name",
ip_int: v => !isAddressOrInterface(v, { cidr: !isExternal })
? "Not an IP address, and not an interface name either"
: (isExternal && isInterfaceName(v)
? "Your own proxy is handed one address: the return rule has to put the original port back, and an interface is not one address"
: null),
port: v => (v > 0 && v < 65536) ? null : "Invalid port",
proxy_ip: v => (!isExternal || (v.match(regex_ipv6_no_cidr) || v.match(regex_ipv4_no_cidr))) ? null : "Invalid proxy IP",
proxy_ip: v => (!isExternal || isIpAddress(v)) ? null : "Not an IP address: your proxy is reached at one, never at an interface",
proxy_port: v => (!isExternal || (v > 0 && v < 65536)) ? null : "Your proxy's port is required",
},
})
@ -41,7 +40,10 @@ function AddressModal({ opened, onClose, service, edit }: {
if (!opened) return
setError(null)
if (edit) form.setValues({
ip_int: edit.ip_int.split("/")[0], port: edit.port,
//: Only the prefix that means "this one address" comes off. Cutting at the
// first slash took a real range down to its first host and saved it as one,
// which is a service quietly protecting a fraction of what it did before.
ip_int: bareAddress(edit.ip_int), port: edit.port,
proxy_ip: edit.proxy_ip ?? "127.0.0.1", proxy_port: edit.proxy_port ?? 8080,
})
else form.reset()
@ -74,7 +76,11 @@ function AddressModal({ opened, onClose, service, edit }: {
title={edit ? "Move this address" : "Protect another address"}>
<form onSubmit={form.onSubmit(submit)}>
<PortAndInterface form={form} int_name="ip_int" port_name="port"
label="Service address and port" />
label="Where the service answers"
description={isExternal
? "One address and its port. This layer rewrites the destination and puts the original port back on the way out, which takes an address it can recognise."
: "An IP address, or the name of an interface — an interface protects whatever address that link is carrying, an address protects that one alone."}
includeInterfaceNames={!isExternal} />
{isExternal ? <>
<Space h="md" />
<PortAndInterface form={form} int_name="proxy_ip" port_name="proxy_port"
@ -112,6 +118,34 @@ export default function AddressList({ service }: { service: Service }) {
const [removing, setRemoving] = useState<Address | null>(null)
const addresses = service.addresses ?? []
const only = addresses.length === 1
//: What each interface is carrying at the moment, so a row named `wg0` can say
// where that actually is. An interface with nothing on it is worth seeing too:
// on the proxy layer a UDP relay has to bind one of these.
const interfaces = ipInterfacesQuery()
const carriedBy = (name: string) => (interfaces.data ?? [])
.filter(i => i.name === name && !i.addr.toLowerCase().startsWith("fe80:"))
.map(i => i.addr)
//: …and the other direction, which is the same question asked of an address row.
const arrivesOn = (addr: string) => (interfaces.data ?? [])
.filter(i => i.addr === addr).map(i => i.name)
/**
* What a row says beside the address, in the words the picker used to choose it.
*
* The same pairing as the list it was chosen from — an interface carries addresses,
* an address sits on an interface — because this is where an operator comes back to
* check what they picked, and finding it described differently is a reason to wonder
* whether it is the same thing.
*/
const rowDetail = (ip_int: string) => {
if (isInterfaceName(ip_int)) {
if (!interfaces.data) return ""
const ips = carriedBy(ip_int)
return ips.length > 0 ? `now ${ips.join("\u00a0· ")}` : "no address on it right now"
}
const names = arrivesOn(bareAddress(ip_int))
return names.length > 0 ? `on ${names.join(", ")}` : ""
}
const remove = async (address: Address) => {
try {
@ -127,32 +161,46 @@ export default function AddressList({ service }: { service: Service }) {
}
return <>
<Group justify="space-between" align="center">
<Box>
<Group justify="space-between" align="center" wrap="nowrap">
<Box style={{ minWidth: 0 }}>
<Text fw={600} size="sm">Protected addresses</Text>
<Text size="xs" c="dimmed">
The same chain runs on every one of them. Adding one costs no connections.
{service.transport === Transport.EXTERNAL
? "Each one handed to your own proxy. Adding one costs no connections."
: "Addresses, or the interfaces they arrive on. The same chain runs on every one of them, and adding one costs no connections."}
</Text>
</Box>
<Button size="xs" variant="light" leftSection={<BsPlusLg size={12} />}
style={{ flexShrink: 0 }}
onClick={() => setAddOpen(true)}>Add an address</Button>
</Group>
<Space h="sm" />
<Stack gap="xs">
{addresses.map(address => <Card key={address.address_id} withBorder radius="md" p="xs"
{addresses.map(address => {
const kind = addressKind(address.ip_int)
const detail = rowDetail(address.ip_int)
return <Card key={address.address_id} withBorder radius="md" p="xs"
bg="transparent" style={{ borderColor: 'var(--mantine-color-dark-4)' }}>
<Group justify="space-between" wrap="nowrap">
<Group gap="xs" wrap="nowrap" style={{ minWidth: 0 }}>
{/* No protocol badge: it is the same on every address, because
it is a property of the service, and the header already says
it once. Repeating it on each row is noise that grows with
the list. */}
<Code>{address.ip_int}:{address.port}</Code>
{isInterface(address.ip_int) ? (
<Badge size="xs" variant="outline" color="blue">
IFACE
{/* The row a service is read from is laid out like the list it was
chosen in: the same badge, at the same width, then the value, then
what is on it. No protocol badge — it is a property of the service,
the header already says it once, and repeating it on every row is
noise that grows with the list. */}
<Group gap="xs" wrap="nowrap" align="flex-start" style={{ minWidth: 0 }}>
<Tooltip position="bottom" disabled={!isInterfaceName(address.ip_int)}
label="An interface, not an address: the rules match on the name, so this follows whatever address the link is carrying.">
<Badge size="xs" variant="light" color={kind.color}
style={{ flexShrink: 0, width: BADGE_WIDTH, marginTop: 4 }}>
{kind.label}
</Badge>
) : null}
</Tooltip>
{/* The badge is a column of its own, exactly as in the list this was
chosen from: everything that has to wrap wraps under the value
rather than under the badge. */}
<Group gap="xs" wrap="wrap" align="center"
style={{ flex: '1 1 auto', minWidth: 0, rowGap: 2 }}>
<Code>{bareAddress(address.ip_int)}:{address.port}</Code>
{decrypts(service) ? <Tooltip position="bottom"
label="The engine decrypts here: clients dial this address as they always did, and the filters see the plaintext inside the process.">
<Badge size="xs" variant="light" color="grape"
@ -163,6 +211,11 @@ export default function AddressList({ service }: { service: Service }) {
{address.proxy_port ? <Badge size="xs" variant="light" color="cyan">
→ {address.proxy_ip ?? "127.0.0.1"}:{address.proxy_port}
</Badge> : null}
{detail ? <Text size="xs" c="dimmed" ff="monospace"
style={{ flex: '1 1 80px', minWidth: 80, overflowWrap: 'anywhere' }}>
{detail}
</Text> : null}
</Group>
</Group>
<Group gap={4} wrap="nowrap">
<Tooltip label="Move this address" position="bottom">
@ -181,7 +234,8 @@ export default function AddressList({ service }: { service: Service }) {
</Tooltip>
</Group>
</Group>
</Card>)}
</Card>
})}
</Stack>
<PlaintextCapture service={service} />

8
frontend/src/components/Services/FilterCard.tsx
View File

@ -169,11 +169,7 @@ export default function FilterCard({ filter, serviceId, dragHandle, position, tr
<Code style={{ overflow: 'hidden', textOverflow: 'ellipsis', whiteSpace: 'nowrap' }}>
{decode(rx.regex)}
</Code>
{false ?
<Badge size="xs" variant="light" color="orange">
→ {decode(rx.replace_with ?? "") || "(removed)"}
</Badge>
: <Badge size="xs" variant="light" color="red">block</Badge>}
<Badge size="xs" variant="light" color="red">block</Badge>
<Badge size="xs" variant="outline" color="gray">{modeLabel(rx.mode)}</Badge>
{rx.case_sensitive ? null :
<Badge size="xs" variant="outline" color="gray">any case</Badge>}
@ -272,8 +268,6 @@ export default function FilterCard({ filter, serviceId, dragHandle, position, tr
initial={(regexes.data ?? []).map(rx => ({
expr: decode(rx.regex),
caseSensitive: rx.case_sensitive,
action: rx.action,
replaceWith: decode(rx.replace_with ?? ""),
}))} />
<YesNoModal
title="Remove this filter?"

8
frontend/src/components/Services/LayerChoice.tsx
View File

@ -27,10 +27,10 @@ type Layer = {
const LAYERS: Record<string, Layer> = {
[Transport.PROXY]: {
line: "Terminates and reopens the connection — exact rewriting, and the faster of the two.",
line: "Terminates and reopens the connection — the faster of the two, and the only one that can decrypt TLS.",
good: [
"Carries bulk traffic 2-5× faster, and scales better with threads",
"Rewriting is exact, at any length",
"Python filters rewrite a payload exactly, at any length",
"The kernel does the reassembly",
"Any number of filters, in one process",
"A slow filter slows the sender; nothing is dropped",
@ -40,7 +40,7 @@ const LAYERS: Record<string, Layer> = {
"Fail-open is rebuilt in userspace, not the kernel's",
],
udpGood: [
"Datagrams are rewritten exactly — no sequence numbers to break",
"One relay socket per address — no per-datagram state to recover",
"Your service still sees the real client address (transparent IP spoofing)",
],
},
@ -55,7 +55,7 @@ const LAYERS: Record<string, Layer> = {
"A userspace round trip per packet: a fraction of the bulk throughput",
"Reassembly is rebuilt in userspace",
"One process per filter, up to eight",
"Patterns can only block, never rewrite",
"A Python filter rewriting a TCP payload desynchronises the stream",
],
udpGood: ["Fully transparent on UDP, and Python rewrites datagrams exactly"],
},

29
frontend/src/components/Services/RegexDebugger.tsx
View File

@ -1,4 +1,4 @@
import { ActionIcon, Switch, Alert, Badge, Box, Button, Checkbox, Code, Group, Modal, ScrollArea, SegmentedControl, Space, Stack, Text, Textarea, TextInput, Tooltip } from '@mantine/core';
import { ActionIcon, Switch, Alert, Box, Button, Code, Group, Modal, ScrollArea, Space, Stack, Text, Textarea, TextInput } from '@mantine/core';
import { useEffect, useMemo, useState } from 'react';
import { BsPlusLg, BsTrashFill } from 'react-icons/bs';
import { DebugResult, services } from './utils';
@ -134,18 +134,6 @@ export default function RegexDebugger({ opened, onClose, initial }: {
}))
const pieces = highlight(sample, ranges)
// The engine hands back what the sample becomes; nothing here reconstructs it.
const rewritten = useMemo(() => {
if (!result?.rewritten) return null
try {
const binary = atob(result.rewritten)
const bytes = Uint8Array.from(binary, c => c.charCodeAt(0))
return new TextDecoder().decode(bytes)
} catch {
return null
}
}, [result?.rewritten])
const update = (key: string, patch: Partial<Draft>) =>
setDrafts(d => d.map(item => item.key === key ? { ...item, ...patch } : item))
@ -208,20 +196,7 @@ export default function RegexDebugger({ opened, onClose, initial }: {
</Code>
</ScrollArea.Autosize>
{rewritten !== null ? <>
<Space h="md" />
<Text size="sm" fw={500}>After rewriting</Text>
<Text size="xs" c="dimmed" mb={6}>
Produced by the engine itself, not a guess at what it would do.
</Text>
<ScrollArea.Autosize mah={160}>
<Code block style={{ whiteSpace: 'pre-wrap', wordBreak: 'break-all' }}>
{rewritten}
</Code>
</ScrollArea.Autosize>
</> : null}
{result?.truncated ?
const update {result?.truncated ?
<Alert color="yellow" mt="md" title="Too many matches to show">
Only the first thousand are listed. A pattern that matches this often will
block on the first connection carrying any of it.

22
frontend/src/components/Services/ServiceRow.tsx
View File

@ -5,7 +5,7 @@ import { FaPlay, FaStop, FaTrash } from 'react-icons/fa';
import { IoSettingsSharp } from 'react-icons/io5';
import { MdChevronRight, MdMoreHoriz } from 'react-icons/md';
import { TbHexagon, TbShieldLock } from 'react-icons/tb';
import { errorNotify, isMediumScreen, okNotify } from '../../js/utils';
import { bareAddress, errorNotify, isMediumScreen, okNotify } from '../../js/utils';
import YesNoModal from '../YesNoModal';
import AddEditService from './AddEditService';
import { Address, decrypts, L4, Service, serviceQueryKey, services, Transport } from './utils';
@ -28,19 +28,16 @@ export const transportSummary = (transport: string, proto: string) => {
if (transport === Transport.EXTERNAL)
return "Firegex only steers this traffic to a proxy you run; nothing here inspects it, so no filter can be attached."
if (transport === Transport.NFQUEUE)
return "Packets are inspected and a verdict handed back; nothing is terminated. Fully transparent, and the kernel keeps forwarding if a filter dies — at the cost of a userspace round trip per packet, userspace reassembly, a process per filter, and patterns that cannot rewrite."
return "Packets are inspected and a verdict handed back; nothing is terminated. Fully transparent, and the kernel keeps forwarding if a filter dies — at the cost of a userspace round trip per packet, userspace reassembly, and a process per filter."
return proto === L4.UDP
? "Each address is relayed by a dedicated socket. Rewriting is exact, filters keep per-flow state, and source IP transparency is preserved. Adding new addresses works dynamically without restarting the service."
: "The connection is terminated and reopened, so rewriting is exact, the kernel reassembles, and the chain has no length limit. It also carries bulk traffic several times faster than NFQUEUE, which pays a userspace round trip per packet; what it costs is fail-open being rebuilt in userspace rather than guaranteed by the kernel. The service still sees the real client address."
? "Each address is relayed by a dedicated socket. Filters keep per-flow state, source IP transparency is preserved, and a new address is relayed without restarting the service."
: "The connection is terminated and reopened, so the kernel reassembles and the chain has no length limit. It also carries bulk traffic several times faster than NFQUEUE, which pays a userspace round trip per packet; what it costs is fail-open being rebuilt in userspace rather than guaranteed by the kernel. The service still sees the real client address."
}
/** Whether this combination loses the client's address, which is worth saying out loud. */
export const losesClientAddress = (_transport: string, _proto: string) => false
/** Where a service is reachable, short enough to sit on one line. */
export const addressSummary = (addresses: Address[]) => {
if (!addresses || addresses.length === 0) return "no address yet"
const first = `${addresses[0].ip_int}:${addresses[0].port}`
const first = `${bareAddress(addresses[0].ip_int)}:${addresses[0].port}`
return addresses.length === 1 ? first : `${first} +${addresses.length - 1} more`
}
@ -140,13 +137,6 @@ export default function ServiceRow({ service, onClick }: { service: Service, onC
{transportLabel(service.transport)}
</Badge>
</Tooltip>
{losesClientAddress(service.transport, service.proto) ?
<Tooltip position="bottom" multiline w={340}
label="Relayed per address, so your service sees firegex's address instead of the client's. The NFQUEUE layer filters UDP with the real packets untouched.">
<Badge color="orange" variant="light" size="xs" radius="sm">
NO CLIENT IP
</Badge>
</Tooltip> : null}
{decrypts(service) ? <Tooltip label="The engine decrypts this service, so the filters see the plaintext" position="bottom">
<Badge color="grape" variant="light" size="xs" radius="sm"
leftSection={<TbShieldLock size={10} />}>TLS</Badge>
@ -154,7 +144,7 @@ export default function ServiceRow({ service, onClick }: { service: Service, onC
</Group>
<Group gap="xs" mt={4}>
<Tooltip position="bottom" disabled={(service.addresses?.length ?? 0) < 2}
label={(service.addresses ?? []).map(a => `${a.ip_int}:${a.port}`).join(", ")}>
label={(service.addresses ?? []).map(a => `${bareAddress(a.ip_int)}:${a.port}`).join(", ")}>
<Text size="xs" c="dimmed" style={{ letterSpacing: 0.5 }}>
{addressSummary(service.addresses)} ON {service.proto.toUpperCase()}
</Text>

12
frontend/src/components/Services/utils.ts
View File

@ -202,9 +202,6 @@ export type Regex = {
mode: string,
case_sensitive: boolean,
active: boolean,
action: string,
/** base64, and only meaningful when the action is `rewrite`. */
replace_with: string | null,
blocked: number,
}
@ -312,8 +309,6 @@ export type DebugResult = {
error?: string | null,
/** Valid and will run, but cannot be highlighted here. */
unscannable: DebugError[],
/** base64: what the sample becomes, computed by the engine's own rewriting code. */
rewritten?: string | null,
truncated: boolean,
}
@ -428,12 +423,10 @@ export const services = {
await getapi(`services/${id}/filters/${fid}/regexes`) as Regex[],
addRegex: async (id: string, fid: string, data: {
regex: string, mode: string, case_sensitive: boolean,
action?: string, replace_with?: string | null,
}) =>
done(await postapi(`services/${id}/filters/${fid}/regexes`, data) as ServerResponse),
editRegex: async (id: string, fid: string, rid: string, data: {
active?: boolean, regex?: string, mode?: string, case_sensitive?: boolean,
action?: string, replace_with?: string | null,
}) =>
done(await putapi(`services/${id}/filters/${fid}/regexes/${rid}`, data) as ServerResponse),
deleteRegex: async (id: string, fid: string, rid: string) =>
@ -463,10 +456,7 @@ export const services = {
/** Ask the matching engine itself what a pattern would do. */
debug: async (
patterns: {
id: string, expr: string, case_sensitive: boolean,
action?: string, replace_with?: string,
}[],
patterns: { id: string, expr: string, case_sensitive: boolean }[],
sample: string,
) =>
await postapi("services/debug-regex", { patterns, sample }) as DebugResult,

48
frontend/src/js/demo.ts
View File

@ -125,6 +125,7 @@ const state = {
{ name: "lo", addr: "127.0.0.1" },
{ name: "eth0", addr: "10.60.3.1" },
{ name: "eth0", addr: "fd66:666:3::1" },
{ name: "wg0", addr: "10.10.0.3" },
],
// One shape for every service: a network layer, and a chain of filters on it.
services: [
@ -135,11 +136,14 @@ const state = {
{ service_id: svcHijack, name: "legacy-ftp", status: "active", proto: "tcp", transport: "external", fail_open: true, max_connections: 0, over_limit_forwards: false, first_byte_timeout: 0, over_limit_hits: 0, over_limit_first: null, over_limit_last: null, filtering_since: demoStart },
] as Json[],
// Where each one is reachable. shop-api answers on both stacks behind one chain,
// which is the case that used to need two services kept in step by hand.
// which is the case that used to need two services kept in step by hand — and on
// `wg0` by name, the other half of the same idea: an address somebody else hands
// out is still one place this service answers.
addresses: [
{ address_id: uuid(), service_id: svcShop, ip_int: "10.60.3.1", port: 9000, proto: "tcp", proxy_ip: null, proxy_port: null },
{ address_id: uuid(), service_id: svcShop, ip_int: "fd66:666:3::1", port: 9000, proto: "tcp", proxy_ip: null, proxy_port: null },
{ address_id: uuid(), service_id: svcScore, ip_int: "10.60.3.1", port: 8080, proto: "tcp", proxy_ip: null, proxy_port: null },
{ address_id: uuid(), service_id: svcShop, ip_int: "10.60.3.1/32", port: 9000, proto: "tcp", proxy_ip: null, proxy_port: null },
{ address_id: uuid(), service_id: svcShop, ip_int: "fd66:666:3::1/128", port: 9000, proto: "tcp", proxy_ip: null, proxy_port: null },
{ address_id: uuid(), service_id: svcShop, ip_int: "wg0", port: 9000, proto: "tcp", proxy_ip: null, proxy_port: null },
{ address_id: uuid(), service_id: svcScore, ip_int: "10.60.0.0/16", port: 8080, proto: "tcp", proxy_ip: null, proxy_port: null },
{ address_id: uuid(), service_id: svcVault, ip_int: "10.60.3.1", port: 8443, proto: "tcp", proxy_ip: null, proxy_port: null },
{ address_id: uuid(), service_id: svcHijack, ip_int: "10.60.3.1", port: 21, proto: "tcp", proxy_ip: "127.0.0.1", proxy_port: 12021 },
] as Json[],
@ -152,14 +156,13 @@ const state = {
{ filter_id: fltVaultRegex, service_id: svcVault, position: 0, kind: "regex", proto: "tcp", name: "patterns", active: true, blocked: 0 },
] as Json[],
regexes: [
{ regex_id: uuid(), filter_id: fltShopRegex, regex: b64("\\.\\./"), mode: "C", case_sensitive: true, active: true, action: "block", replace_with: null, blocked: 341 },
{ regex_id: uuid(), filter_id: fltShopRegex, regex: b64("FLAG\\{[A-Za-z0-9_]+\\}"), mode: "S", case_sensitive: true, active: true, action: "block", replace_with: null, blocked: 27 },
{ regex_id: uuid(), filter_id: fltShopRegex, regex: b64("\\.\\./"), mode: "C", case_sensitive: true, active: true, blocked: 341 },
{ regex_id: uuid(), filter_id: fltShopRegex, regex: b64("FLAG\\{[A-Za-z0-9_]+\\}"), mode: "S", case_sensitive: true, active: true, blocked: 27 },
// A rewriting rule, so the demo shows what only the proxy layer can do.
{ regex_id: uuid(), filter_id: fltShopRegex, regex: b64("X-Debug: [^\\r\\n]*"), mode: "C", case_sensitive: false, action: "rewrite", replace_with: b64("X-Debug: off"), active: true, blocked: 0 },
{ regex_id: uuid(), filter_id: fltScoreRegex, regex: b64("/etc/passwd"), mode: "C", case_sensitive: true, active: true, action: "block", replace_with: null, blocked: 914 },
{ regex_id: uuid(), filter_id: fltScoreRegex, regex: b64("<script>"), mode: "C", case_sensitive: false, active: true, action: "block", replace_with: null, blocked: 233 },
{ regex_id: uuid(), filter_id: fltScoreRegex, regex: b64("/etc/passwd"), mode: "C", case_sensitive: true, active: true, blocked: 914 },
{ regex_id: uuid(), filter_id: fltScoreRegex, regex: b64("<script>"), mode: "C", case_sensitive: false, active: true, blocked: 233 },
{ regex_id: uuid(), filter_id: fltScoreRegex, regex: b64("[A-Z0-9]{31}="), mode: "S", case_sensitive: true, active: false, blocked: 57 },
{ regex_id: uuid(), filter_id: fltVaultRegex, regex: b64("/proc/self/"), mode: "C", case_sensitive: true, active: true, action: "block", replace_with: null, blocked: 0 },
{ regex_id: uuid(), filter_id: fltVaultRegex, regex: b64("/proc/self/"), mode: "C", case_sensitive: true, active: true, blocked: 0 },
] as Json[],
code: { [fltShopPy]: SAMPLE_FILTER } as Record<string, string>,
// One row per @pyfilter the code defines. The code says which exist; these say
@ -308,7 +311,6 @@ const demoDebug = (patterns: Json[], sample: string) => {
try { text = atob(sample) } catch { text = "" }
const matches: Json[] = []
const errors: Json[] = []
const spans: { start: number, end: number, with: string | null }[] = []
for (const p of patterns) {
let re: RegExp
try {
@ -320,30 +322,13 @@ const demoDebug = (patterns: Json[], sample: string) => {
for (const hit of text.matchAll(re)) {
if (hit.index === undefined) continue
matches.push({ id: p.id, start: hit.index, end: hit.index + hit[0].length })
if (p.action === "rewrite") {
spans.push({
start: hit.index, end: hit.index + hit[0].length,
with: p.replace_with ?? "",
})
}
if (hit[0].length === 0) break // a zero-width match would never advance
if (matches.length >= 1000) break
}
}
let rewritten: string | null = null
if (spans.length > 0) {
spans.sort((a, b) => a.start - b.start || b.end - a.end)
let out = "", cursor = 0
for (const span of spans) {
if (span.start < cursor) continue
out += text.slice(cursor, span.start) + (span.with ?? "")
cursor = span.end
}
rewritten = btoa(out + text.slice(cursor))
}
return {
matches, errors, error: null, unscannable: [],
rewritten, truncated: matches.length >= 1000,
truncated: matches.length >= 1000,
}
}
@ -716,8 +701,7 @@ const routes: [string, RegExp, Handler][] = [
try { new RegExp(expr) } catch (err) { throw `Invalid pattern: ${err}` }
state.regexes.push({
regex_id: uuid(), filter_id: m[2], regex: b.regex, mode: b.mode ?? "B",
case_sensitive: b.case_sensitive ?? true, active: b.active ?? true,
action: b.action ?? "block", replace_with: b.replace_with ?? null, blocked: 0,
case_sensitive: b.case_sensitive ?? true, active: b.active ?? true, blocked: 0,
})
emit(["services"]); return ok
}],
@ -738,8 +722,6 @@ const routes: [string, RegExp, Handler][] = [
}
if (b.mode != null) rx.mode = b.mode
if (b.case_sensitive != null) rx.case_sensitive = b.case_sensitive
if (b.action != null) rx.action = b.action
if (b.replace_with !== undefined) rx.replace_with = b.replace_with
emit(["services"]); return ok
}],
["DELETE", /^services\/([^/]+)\/filters\/([^/]+)\/regexes\/([^/]+)$/, m => {

40
frontend/src/js/utils.tsx
View File

@ -22,6 +22,46 @@ export const regex_ipv4 = "^(([0-9]|[1-9][0-9]|1[0-9]{2}|2[0-4][0-9]|25[0-5])\\.
export const regex_ipv4_no_cidr = "^(([0-9]|[1-9][0-9]|1[0-9]{2}|2[0-4][0-9]|25[0-5])\\.){3}([0-9]|[1-9][0-9]|1[0-9]{2}|2[0-4][0-9]|25[0-5])$"
export const regex_port = "^([1-9]|[1-9][0-9]{1,3}|[1-5][0-9]{4}|6[0-4][0-9]{3}|65[0-4][0-9]{2}|655[0-2][0-9]|6553[0-5])?$"
export const regex_range_port = "^(([1-9]|[1-9][0-9]{1,3}|[1-5][0-9]{4}|6[0-4][0-9]{3}|65[0-4][0-9]{2}|655[0-2][0-9]|6553[0-5])(-([1-9]|[1-9][0-9]{1,3}|[1-5][0-9]{4}|6[0-4][0-9]{3}|65[0-4][0-9]{2}|655[0-2][0-9]|6553[0-5])?)?)?$"
/**
* What an operator may type where an address is asked for.
*
* One rule, read by the pickers, the badges and every form that validates one, because
* the backend has exactly one too (`utils.is_interface_name` / `parse_ip_or_int`): an
* interface name is 1-15 characters of `[a-zA-Z0-9_.:-]` that is not already an address.
* The order matters — `::1` fits the charset as happily as it fits IPv6, and IPv6 is
* what it means.
*/
export const isIpAddress = (v: string, { cidr = false } = {}) =>
!!(v.match(cidr ? regex_ipv4 : regex_ipv4_no_cidr) || v.match(cidr ? regex_ipv6 : regex_ipv6_no_cidr))
export const isInterfaceName = (v: string) =>
!isIpAddress(v, { cidr: true }) && !!v.trim().match(/^[a-zA-Z0-9_.:-]{1,15}$/)
/**
* Either of them: what a service address accepts, and what the picker offers.
*
* A range is an address here too — `10.0.0.0/24:8080` is in the documented set and the
* backend parses it — except where one address is the whole point, which is the layer
* that hands traffic to a proxy of the operator's own.
*/
export const isAddressOrInterface = (v: string, { cidr = true } = {}) =>
isIpAddress(v, { cidr }) || isInterfaceName(v)
/**
* An address as the operator typed it.
*
* The backend parses what it is given and stores it back with a prefix, so `127.0.0.1`
* comes out of the database as `127.0.0.1/32` — the same address, spelled in a way
* nobody types and that reads like a range on a row beside real ones. A host-length
* prefix is dropped for display; a prefix that actually covers more than one address is
* left exactly as it is, because there it is the whole point.
*/
export const bareAddress = (v: string) => {
const [addr, prefix] = v.split("/")
if (prefix === undefined) return v
return (addr.includes(":") ? prefix === "128" : prefix === "32") ? addr : v
}
// Where `bun run dev` looks for the backend. Overridable because the backend needs
// Linux and NET_ADMIN, so it is often not on the same machine as the dev server.
export const DEV_IP_BACKEND = import.meta.env.VITE_BACKEND ?? "127.0.0.1:4444"

7
frontend/src/pages/Firewall/index.tsx
View File

@ -248,6 +248,7 @@ export const Firewall = () => {
initialCustomInterfaces={[...src_custom_int, ...customInt]}
value={item.src}
onChange={v => ip_setter(item, v, { src: true })}
placeholder="Any source"
includeInterfaceNames
/>
<Group gap="xs" wrap="nowrap">
@ -279,8 +280,12 @@ export const Firewall = () => {
<Stack gap="xs" style={{ flex: 1, minWidth: isMedium ? 0 : "100%" }}>
<InterfaceInput
initialCustomInterfaces={[...dst_custom_int, ...customInt]}
defaultValue={item.dst}
// Controlled, like the source beside it: uncontrolled, the box kept
// whatever it was first rendered with, so a rule reordered or reset
// under it went on showing the old destination.
value={item.dst}
onChange={v => ip_setter(item, v, { dst: true })}
placeholder="Any destination"
includeInterfaceNames
/>
<Group gap="xs" wrap="nowrap">

9
frontend/src/pages/Services/ServiceDetails.tsx
View File

@ -11,7 +11,7 @@ import AddEditService from '../../components/Services/AddEditService';
import AddressList from '../../components/Services/AddressList';
import FilterCard from '../../components/Services/FilterCard';
import LogPanel from '../../components/Services/LogPanel';
import { addressSummary, losesClientAddress, ServiceMenu, transportLabel, transportSummary } from '../../components/Services/ServiceRow';
import { addressSummary, ServiceMenu, transportLabel, transportSummary } from '../../components/Services/ServiceRow';
import StatsPanel from '../../components/Services/StatsPanel';
import { decrypts, Filter, FilterKind, serviceFiltersQuery, serviceQueryKey, services, servicesQuery, Transport } from '../../components/Services/utils';
import YesNoModal from '../../components/YesNoModal';
@ -82,13 +82,6 @@ export default function ServiceDetails() {
{transportLabel(service.transport)}
</Badge>
</Tooltip>
{losesClientAddress(service.transport, service.proto) ?
<Tooltip position="bottom" multiline w={360}
label="Relayed per address, so your service sees firegex's address instead of the client's. The NFQUEUE layer filters UDP with the real packets untouched.">
<Badge color="orange" variant="light" size="sm" radius="sm">
NO CLIENT IP
</Badge>
</Tooltip> : null}
{decrypts(service) ? <Badge color="grape" variant="light" size="sm" radius="sm"
leftSection={<TbShieldLock size={11} />}>TLS</Badge> : null}
</Group>

349
tests/README.md
View File

@ -2,311 +2,102 @@
## [GO BACK](../README.md)
Two different things live here. The **tests** check that a change to firegex did not break
anything, and they are quick. The **benchmarks** measure how much traffic the filters can
carry, and they are slow, noisy, and easy to misread — the second half of this page is
mostly about how to read them.
One suite, one runner. Everything here is `pytest`; the **benchmarks** live in
[`bench/`](bench/README.md) and are not tests — they measure, so they have no pass or fail
and `pytest` does not collect them.
# Running all the tests
Everything below drives a **live, already-running** firegex on Linux, as root. Start one
first (`python3 run.py start -P testpassword` from a clone), then:
## Running them
```bash
./run_tests.sh [password] # defaults to "testpassword"
./run_tests.sh --set-pass # for an instance still in its initial-setup state
./run_tests.sh # everything this host can run
./run_tests.sh --set-pass # against an instance still in its initial-setup state
./run_tests.sh mypassword # a different password
```
`FIREGEX_ADDRESS` points the whole suite somewhere other than `http://127.0.0.1:4444`.
`run_tests.sh` installs the dependencies, waits for the instance to answer, and hands
everything else straight to pytest — so anything pytest understands works there, and
`pytest` on its own works too once the dependencies are in place.
The suite runs, in order:
| Script | Needs an instance | What it covers |
|---|---|---|
| `regex_lib_test.py` | no (needs `libhs`) | `firegex.regex`: validity per mode, direction, rewriting precedence, the ruleset file format. First, because a broken match library makes everything below fail in a way that is much harder to read. |
| `test_http_history.py` (pytest) | no | the `firegex.pyfilters` library on its own: the HTTP models, `HttpHistory`, and the knobs a filter file may set. |
| `api_test.py` | yes | login, password change and expiry, backup export/import, and authentication being turned off and back on at runtime. |
| `services_test.py` | yes | the whole product: both network layers, both filter kinds, TLS, IPv6, the external hand-off, addresses added to a running service, statistics. |
`services_test.py` is the one that matters, and it is run **eight times** — `proxy`,
`nfqueue` and `external` × IPv4/IPv6, plus both layers behind TLS — because a filter that
works on one layer and not the other is exactly the failure the unified model exists to
prevent.
Part of "in place" is a **build**: `unit/` imports the filter library out of
`../fgex-lib`, and that library carries a C extension — the llhttp binding that parses
HTTP. `run_tests.sh` builds it for you; doing it by hand once is enough:
```bash
./services_test.py -p testpassword -t proxy|nfqueue|external [-6] [--tls] [-P PORT]
./api_test.py -p testpassword [-a http://127.0.0.1:4444/]
pip install -e ../fgex-lib
```
**Run the suite twice in a row** when you have changed anything about addresses or TLS: a
leaked `tls_streams` row only shows up on the second pass.
Two scripts are deliberately **not** in `run_tests.sh`:
- `ipfilter_test.py` — those settings are only read at process startup, so it stops and
starts the instance itself for each scenario. Run it on its own and expect it to bounce
what you have running.
- `stress_test_history.py` — a soak test for `HttpHistory` under load, not a pass/fail check.
The benchmarks (`benchmark.py`, `connbench.py`) are not in it either — they take minutes
and answer a different question. See [Performance](#performance).
# Running a benchmark
Without it those tests cannot be collected at all, and pytest says
`ImportError: cannot import name '_llhttp' from 'firegex'`.
```bash
./benchmark.py -p testpassword -r 50 -d 1 -s 50 -t nfqueue
pytest # the same thing, if you already have the dependencies
pytest unit # the ones that need nothing running at all
pytest -k udp # one subject
pytest -m "not slow" # skip the ones that sit and wait on a real timeout
pytest --layer proxy --no-ipv6 # one network layer, one address family
pytest --fg-address http://box:4444/ --fg-password hunter2
```
| Flag | Meaning |
| Option | What it does |
|---|---|
| `-r, --num-of-regexes` | how many patterns to build up to; one measurement per pattern count |
| `-d, --duration` | seconds per measurement |
| `-s, --num-of-streams` | concurrent iperf3 streams |
| `-t, --transport` | `nfqueue` or `proxy` — which network layer to put in the path |
| `-o, --output-file` | CSV of `pattern count, MB/s` |
| `--fg-address` | where the instance under test is (or `FIREGEX_ADDRESS`) |
| `--fg-password` | its password (or `FIREGEX_PASSWORD`) |
| `--layer` | only these network layers — `proxy`, `nfqueue`, `external`; repeatable |
| `--no-ipv6` | skip the IPv6 half of every parametrised case |
| `--no-tls` | skip the TLS cases |
It stands up a service, measures the throughput with no filter, then adds one random
pattern at a time and measures again, using iperf3 over loopback. All the patterns live in
**one** regex filter, because that is the property worth measuring: they are compiled into
a single hyperscan database, so fifty of them should cost about what one costs.
Markers: `instance` (needs a live firegex), `root` (needs root and a Linux kernel with
nftables), `slow` (waits on a real timeout), `ipv6`, `tls`.
## Reading the numbers
## What is where
Three things decide whether a number here means anything at all.
| Directory | Needs a running instance | What it covers |
|---|---|---|
| `unit/` | no | `firegex.regex` against libhs, the `firegex.pyfilters` models and knobs, and the address-to-nftables translation in the backend's own modules |
| `integration/` | yes | the product: both network layers, both filter kinds, TLS, UDP, IPv6, the hand-off, addresses on a running service, statistics, logs, limits, resilience |
| `standalone/` | yes, and it restarts it | settings that are only read at process startup, so the test has to bounce the instance itself — **not** collected by default |
| `helpers/` | — | the API client, the stand-in services, certificates, and the traffic channel |
| `bench/` | yes | [measurement, not testing](bench/README.md) |
**The filter has to be in the path.** `benchmark.py` creates its service with
`fail_open=False`, and that one argument is worth more than everything else on this page.
With fail-open on, the nft rule carries `bypass` and the queue is configured
`NFQA_CFG_F_FAIL_OPEN`: when the queue fills — which under fifty iperf3 streams it does
constantly — the kernel accepts packets **without inspecting them**. Measured that way the
same build reports ~32 000 MB/s instead of ~3 000, and the extra order of magnitude is
traffic that was never filtered. A benchmark of a filter has to make every packet go
through the filter.
Most of `integration/` is **parametrised over the combinations**, which is the point of
the suite rather than a detail of it: a service is a network layer and a chain of filters
chosen independently, so the same filters are exercised on `proxy` and on `nfqueue`, over
IPv4 and IPv6, with and without TLS. A filter that works on one layer and not the other is
exactly the failure the unified model exists to prevent, and it only shows up if both are
run. Each case is a test with its own name, so a failure says which combination broke:
**A pass is repeatable; a restart is not.** Three passes back to back against the same
running instance land within ~3% of each other. The same configuration measured again
after stopping and starting firegex moved by up to ~30%. So differences smaller than about
a third are not results, and any comparison across versions — each of which needs its own
instance — carries that uncertainty.
```
integration/test_filters_regex.py::test_a_matching_pattern_blocks[nfqueue-ipv6] FAILED
```
**One second per point is noisy on purpose.** `-d 1` is what the archived runs used and
what these use, so the curves are comparable with each other; it also means the wiggle
between adjacent points is measurement, not behaviour. Read the level and the slope.
This used to be a shell script invoking one 1100-line program eight times with different
flags. The first failure in a run ended it, so the seven other combinations were never
reached, and a host without IPv6 on loopback failed rather than skipped.
# Performance
**Run the suite twice in a row** when you have changed anything about addresses or TLS:
state left behind by the first pass only shows up on the second.
Measured on:
- MacBook Air M2, 16 GB
- OrbStack VM, Fedora Linux 43 aarch64, Linux 7.0.14, 7 CPUs
- `./benchmark.py -p testpassword -r 50 -d 1 -s 50 -t nfqueue` — and the same with
`-t proxy` for the proxy line. **The layer is part of the measurement**, so it is part of
every label, every filename (`5.0.0-nfqueue-8T.csv`, `5.0.0-proxy.csv`) and every table
row below: the two differ by more than three times, and a number without its layer is not
a number about firegex.
The `-d 1` in that command used to be load-bearing by accident. `benchmark.py` drove
iperf3 through the `iperf3` python binding, which captures libiperf's output by `dup2`-ing
stdout onto an `os.pipe()` and reads it only *after* the test returns — and a Linux pipe
holds 64 KiB. At fifty streams the JSON is 55 KB for a one-second test and 130 KB for a
five-second one, so anything past a second filled the pipe, blocked libiperf in `write()`,
and hung the benchmark for good: no output, no error, no timeout. It survived only because
the recorded runs happened to use the one duration that fits. iperf3 is now run as a
subprocess against a pipe something is draining, so `-d` is just a duration again.
## By version and thread count
Every line below was measured on that machine, in one sitting, with the filter in the path
for every packet. The versions are all driving their own `tests/benchmark.py` against their
own API — what that API is called changed across 3.x and 4.x, what is being measured did
not: a regex filter, one hyperscan database, throughput over loopback. **5.0.0** is the
tree these tests live in.
One chart per thread count, because the thread count moved a line as much as the version
did and seven lines on one pair of axes was something you decoded rather than read. The
dashed line in each is 5.0.0's **proxy** layer: not another version, the other way of
putting the same filter in the path.
### One thread
![One thread](results/Benchmark-1T.svg)
### Eight threads
![Eight threads](results/Benchmark-8T.svg)
### Medians
| Version | Layer | 1 thread | 8 threads | 8T / 1T |
|---|---|---|---|---|
| 3.5.3 | NFQUEUE | 1601 | 2620 | 1.6× |
| 4.0.5 | NFQUEUE | 1827 | 2461 | 1.3× |
| 5.0.0 | NFQUEUE | 1820 | 2956 | 1.6× |
| **5.0.0** | **Proxy** | **4035** | **13984** | **3.5×** |
What this supports:
- **Throughput is flat in the number of patterns**, on every version, both layers, both
thread counts. Fifty patterns cost about what one costs. That is hyperscan doing what it
is here for, and it is the clearest thing on either chart.
- **The proxy layer is the faster one, and the gap widens with threads**: 2.2× NFQUEUE at
one thread, **4.7× at eight**. NFQUEUE pays a userspace round trip per packet; the proxy
pays once per connection and then the kernel moves the bytes. Both gaps are several
times the measurement uncertainty below, so both are results.
- **The proxy scales better across threads too** — 3.5× from one to eight, against ~1.5×
for NFQUEUE, which is bounded by how much of its work is a per-packet trip through the
kernel rather than something a second core can do.
- **The NFQUEUE versions are not separable at this resolution.** The spread across passes
of the same version is as wide as the gap between versions. 5.0.0 measures highest and
most steadily and that is as much as these numbers say; the charts are not evidence that
it is faster than 4.0.5.
The `3.5.3-8T` pass in `results/3.5.3-8T-outlier.csv` collapsed to ~500 MB/s past the
seventeenth pattern and did not reproduce on the next pass. It is kept because leaving it
out would make the run-to-run spread look smaller than it is.
### Measured again after the nftables tables moved
`results/*-rerun.csv` is the whole table above, taken a second time after the firewall
module moved out of the shared `filter`/`mangle` tables into `fgex_filter` and
`fgex_mangle` — the kind of change that has no business costing throughput, which is why
it is worth showing that it did not. Same Mac, same protocol (`-r 50 -d 1 -s 50`), but
from a container sharing the datapath's network namespace rather than from the Fedora
machine, so read it as a second host and not as a repeat:
| Layer | 1 thread | 8 threads | 8T / 1T |
|---|---|---|---|
| NFQUEUE | 2205 *(was 1820)* | 2940 *(was 2956)* | 1.3× *(was 1.6×)* |
| **Proxy** | **4341** *(was 4035)* | **14590** *(was 13984)* | **3.4×** *(was 3.5×)* |
The proxy is 2.0× NFQUEUE at one thread and 5.0× at eight, against 2.2× and 4.7× before.
Every medians-level claim above survives; none of them survives to the decimal, which the
run-to-run spread in the same table already said.
**Duration is part of the protocol, and the numbers here are only comparable at `-d 1`.**
The same four runs at the default `-d 5` land at 1.9× and 3.9× — the layers' gap narrows
as each reading settles. Neither pair is wrong; quoting one against the other is.
`--threads` reaches **both** layers: it is passed to `cppregex` and `cpproxy`, and — since
5.0.0 — to the Rust proxy engine, which builds its tokio runtime with that many workers
instead of taking every core whatever the operator asked for. Before that the same flag
meant one thing on one layer and nothing on the other, and there could be only one proxy
line on these charts.
### Versions that can no longer be measured
`2.3.3`, `2.4.0`, `2.5.1`, `3.0.0` and `3.2.x` pin no dependency versions and either pin
no base image or pin `fedora:latest`, so building them today resolves to current packages.
Their containers **do not start**: `passlib` 1.7.4 probes its bcrypt backend at import and
today's `bcrypt` answers with `ValueError: password cannot be longer than 72 bytes`, which
kills the application at startup. `3.4.11` builds and starts but the datapath dies partway
through the benchmark under fifty streams.
Their old numbers are still in `results/` and are rendered further down, but they were
measured in 2025 on Fedora 41 and Linux 6.12. They are **not** comparable with anything
above — that machine is roughly an order of magnitude away from this one on the same
benchmark — and they are kept as a record of what was measured then, not as a baseline.
## Short connections
Bulk throughput is the shape a terminating proxy is best at and the shape a CTF service
almost never sees. The cost the proxy is supposed to pay is in *opening* a connection,
which the throughput benchmark never touches, so there is a second script for it:
## Two that are not in the default run
- **`standalone/test_ip_filter.py`.** Access control by CIDR is read once, at process
startup, so there is no way to test it against a running instance — it drives
`run.py stop`/`start` itself for each scenario and restores an unrestricted instance at
the end. Run it on its own, and expect it to bounce whatever you have running:
```bash
./connbench.py -p testpassword [--passes 5] [-c 1500] [-s 32]
pytest standalone
```
- **`bench/`.** Minutes rather than seconds, and it answers a different question. See
[bench/README.md](bench/README.md).
Open, one small request, one small response, close — measured as connections per second
against an unfiltered baseline taken in the same run.
## Adding a test
| 5.0.0, short connections | conn/s | of unfiltered | across passes |
|---|---|---|---|
| unfiltered baseline | 11182 | — | 8437 – 14014 |
| Proxy | 10181 | 91% | 9687 – 10346 |
| NFQUEUE | 9870 | 88% | 7318 – 12117 |
Put it in the directory that matches what it needs, and take the fixtures from the
conftest rather than building a service by hand — `protected` gives you a started-service
shape for whichever layer is being parametrised, `service` cleans up whatever you create,
and `Channel` asks a protected service a question without the test having to know whether
TLS is in the way. A leaked service is not untidiness: it holds a port and an nftables
rule, so the next test to want that port fails for a reason belonging to yours.
**There is no measurable difference.** The medians land 3% apart and the ranges overlap
heavily, so the honest answer on this axis is "the same", and the script says so itself
rather than printing a ratio.
Getting there took three wrong answers, which is the useful part. Measuring every pass of
one layer and then every pass of the other gave the proxy a 1.4× lead twice, then a 1.0×,
then NFQUEUE a lead — because whichever went second inherited the machine's drift.
`connbench.py` now measures both layers **within** each pass and alternates which goes
first, so drift lands on both and shows up as spread instead of as a winner.
None of this is an argument for always choosing the proxy. What it costs is the kernel's
fail-open backstop, and on UDP the client's address; `-s` and `-d` on `benchmark.py` are
there so you can measure the shape you actually care about.
# Archived results (2025)
Everything below was measured on the machine described at the time — a MacBook Air M2 on
OrbStack with **Fedora 41** and Linux 6.12.13 — and is kept as the record it was. Do not
read it against the section above.
Command: `./benchmark.py -p testpassword -r 50 -d 1 -s 50`
NOTE: 8-thread performance before 2.5.0 does not change, because the source and
destination IP are always the same, so the kernel hashes every packet onto the same
thread. See
[this thread](https://netfilter.vger.kernel.narkive.com/sTP7613Y/meaning-of-nfqueue-s-queue-balance-option).
In a CTF there is usually a NAT hiding the real IPs, so this is not a hypothetical.
Firegex 2.5.0 assigns threads in userland instead, which is what makes the distribution
work.
Charts are labelled `[version]-[n_thread]T`, e.g. `2.5.0-8T`.
![Firegex Benchmark](results/Benchmark-chart.svg)
The advantage of multithreading is hard to see there, and much clearer with a fake load in
the filtering path:
```cpp
volatile int x = 0;
for (int i=0; i<50000; i++){
x+=1;
}
```
![Firegex Benchmark](results/Benchmark-chart-with-load.svg)
## nfregex vs nfproxy (2025)
Command: `./comparemark.py nfproxy -p testpassword -d 1 -s 50 -V 100`
> These are **archived**, from back when `nfregex` and `nfproxy` were separate modules.
> Both were folded into `services` — what `nfproxy` did is now the **pyfilters** filter
> kind — so `comparemark.py` drives an API that no longer exists, and the names below are
> the ones the stored data was recorded under, not names anything answers to. The current
> equivalent of this comparison is [By network layer](#by-network-layer) above.
The code under test matched this pattern with Python's `re` module:
```
(?:[a-z0-9!#$%&'*+/=?^_`{|}~-]+(?:\.[a-z0-9!#$%&'*+/=?^_`{|}~-]+)*|"(?:[\x01-\x08\x0b\x0c\x0e-\x1f\x21\x23-\x5b\x5d-\x7f]|\\[\x01-\x09\x0b\x0c\x0e-\x7f])*")@(?:(?:[a-z0-9](?:[a-z0-9-]*[a-z0-9])?\.)+[a-z0-9](?:[a-z0-9-]*[a-z0-9])?|\[(?:(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.){3}(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?|[a-z0-9-]*[a-z0-9]:(?:[\x01-\x08\x0b\x0c\x0e-\x1f\x21-\x5a\x53-\x7f]|\\[\x01-\x09\x0b\x0c\x0e-\x7f])+)\])
```
![nfproxy benchmarks](results/whisker_nfproxy.svg)
![nfproxy benchmarks](results/istogramma_nfproxy.svg)
nfproxy was slower than nfregex, and more flexible.
![nfproxy benchmarks](results/whisker_compare.svg)
![nfproxy benchmarks](results/istrogramma_compare.svg)
# Regenerating the charts
`results_plotter.py` reads the CSVs in `results/` and writes the SVGs beside them:
```bash
cd tests && python3 results_plotter.py
```
That writes the two current charts, through one shared `line_chart()` helper, and stops.
The archived charts are behind `--archived`, because matplotlib has moved on since they
were produced and redrawing them changes six committed SVGs for no reason other than a
different font being available.
If a case cannot run on some hosts, **skip it with a reason** rather than letting it fail.
A red test nobody can act on teaches less than a summary line saying which half of the
suite this machine declined to run and why.

225
tests/api_test.py
View File

@ -1,225 +0,0 @@
#!/usr/bin/env python3
from utils.colors import colors, puts, sep
from utils.firegexapi import FiregexAPI
import argparse
import secrets
if __name__ == "__main__":
parser = argparse.ArgumentParser()
parser.add_argument(
"--address",
"-a",
type=str,
required=False,
help="Address of firegex backend",
default="http://127.0.0.1:4444/",
)
parser.add_argument(
"--password", "-p", type=str, required=True, help="Firegex password"
)
args = parser.parse_args()
sep()
puts("Testing will start on ", color=colors.cyan, end="")
puts(f"{args.address}", color=colors.yellow)
firegex = FiregexAPI(args.address)
# Connect to Firegex
# Half of what follows is about passwords, and an instance started with
# `--unsafe-disable-auth` answers 403 to every one of those endpoints by design. Say
# so and stop, rather than reporting a password that would not change as a failure:
# the rest of the suite runs against such an instance perfectly well, this one alone
# needs authentication to be on.
if firegex.status().get("auth_disabled"):
puts("This instance has authentication turned off, and this suite is mostly "
"about authentication. Start one with a password to run it.", color=colors.yellow)
exit(1)
if firegex.status()["status"] == "init":
if firegex.set_password(args.password):
puts(f"Sucessfully set password to {args.password} ✔", color=colors.green)
else:
puts(
"Test Failed: Unknown response or password already put ✗",
color=colors.red,
)
exit(1)
else:
if firegex.login(args.password):
puts("Sucessfully logged in ✔", color=colors.green)
else:
puts("Test Failed: Unknown response or wrong passowrd ✗", color=colors.red)
exit(1)
if firegex.status()["loggined"]:
puts("Correctly received status ✔", color=colors.green)
else:
puts("Test Failed: Unknown response or not logged in✗", color=colors.red)
exit(1)
# Prepare second instance
firegex2 = FiregexAPI(args.address)
if firegex2.login(args.password):
puts("Sucessfully logged in on second instance ✔", color=colors.green)
else:
puts(
"Test Failed: Unknown response or wrong passowrd on second instance ✗",
color=colors.red,
)
exit(1)
if firegex2.status()["loggined"]:
puts("Correctly received status on second instance✔", color=colors.green)
else:
puts(
"Test Failed: Unknown response or not logged in on second instance✗",
color=colors.red,
)
exit(1)
# Change password
new_password = secrets.token_hex(10)
if firegex.change_password(new_password, expire=True):
puts(f"Sucessfully changed password to {new_password} ✔", color=colors.green)
else:
puts("Test Failed: Coundl't change the password ✗", color=colors.red)
exit(1)
# Check if we are still logged in
if firegex.status()["loggined"]:
puts("Correctly received status after password change ✔", color=colors.green)
else:
puts(
"Test Failed: Unknown response or not logged after password change ✗",
color=colors.red,
)
exit(1)
# Check if second session expired and relog
if not firegex2.status()["loggined"]:
puts("Second instance was expired currectly ✔", color=colors.green)
else:
puts(
"Test Failed: Still logged in on second instance, expire expected ✗",
color=colors.red,
)
exit(1)
if firegex2.login(new_password):
puts("Sucessfully logged in on second instance ✔", color=colors.green)
else:
puts(
"Test Failed: Unknown response or wrong passowrd on second instance ✗",
color=colors.red,
)
exit(1)
# Change it back
if firegex.change_password(args.password, expire=False):
puts("Sucessfully restored the password ✔", color=colors.green)
else:
puts("Test Failed: Coundl't change the password ✗", color=colors.red)
exit(1)
# Check if we are still logged in
if firegex2.status()["loggined"]:
puts("Correctly received status after password change ✔", color=colors.green)
else:
puts(
"Test Failed: Unknown response or not logged after password change ✗",
color=colors.red,
)
exit(1)
# Backups never contain the password/secret (see export_db), so importing one back
# must not lock the current session out or force a re-login.
backup = firegex.export_backup()
if backup:
puts("Sucessfully exported backup ✔", color=colors.green)
else:
puts("Test Failed: Couldn't export backup ✗", color=colors.red)
exit(1)
if firegex.import_backup(backup):
puts("Sucessfully imported backup ✔", color=colors.green)
else:
puts("Test Failed: Couldn't import backup ✗", color=colors.red)
exit(1)
firegex3 = FiregexAPI(args.address)
if firegex3.login(args.password):
puts("Password survived the backup import ✔", color=colors.green)
else:
puts("Test Failed: Password was lost/changed after backup import ✗", color=colors.red)
exit(1)
# Whether firegex asks for a password at all is a property of where it is deployed,
# not of the configuration in a backup. It has to survive an import for the same
# reason the password does — and it did not, once: the key lives in `keys_values`, so
# it rode along in the dump, and the restart at the end of the import re-seeded it
# from the container's environment, reverting a `run.py config` made hours earlier.
#
# Turned off around the import on purpose. Asserting it is still on afterwards would
# pass either way on a container whose environment says the same thing; the bug only
# shows where the running state and the boot-time environment disagree, which is what
# a runtime change is.
if not firegex.set_auth_mode(True):
puts("Test Failed: could not turn authentication off ✗", color=colors.red)
exit(1)
if not firegex.import_backup(firegex.export_backup()):
puts("Test Failed: could not import a backup with authentication off ✗", color=colors.red)
exit(1)
if firegex.status()["auth_disabled"] is True:
puts("Authentication mode survived the backup import ✔", color=colors.green)
else:
puts("Test Failed: the backup import turned authentication back on ✗", color=colors.red)
exit(1)
# Back on, with the session signed before it went off — everything below expects it.
if not firegex.set_auth_mode(False) or firegex.status()["auth_disabled"] is not False:
puts("Test Failed: could not turn authentication back on ✗", color=colors.red)
exit(1)
puts("And it goes back on afterwards ✔", color=colors.green)
# --- authentication is a runtime setting, not a boot-time one ----------------
# It used to be read from the environment once, at startup, which meant a password
# handed to a running instance was stored and never asked for while `run.py config`
# reported "it will take effect immediately".
sep()
puts("Authentication at runtime", color=colors.white, is_bold=True)
def check(name, ok, extra=""):
if ok:
puts(f" [+] {name}", color=colors.green)
else:
puts(f" [-] {name} {extra}", color=colors.red)
exit(1)
check("it starts on", firegex.status()["auth_disabled"] is False)
anonymous = FiregexAPI(args.address)
check("and an anonymous caller is refused",
anonymous.status()["loggined"] is False)
check("an administrator can turn it off", firegex.set_auth_mode(True))
check("the status says so at once", firegex.status()["auth_disabled"] is True)
check("and the same anonymous caller is now let in",
anonymous.status()["loggined"] is True and isinstance(anonymous.get_interfaces(), list))
# Nothing new is signed while it is off, so a token is proof of having been an
# administrator before it was — which is what re-enabling asks for.
why = anonymous.set_auth_mode_error(False)
check("a caller who arrived afterwards cannot put it back",
why is not None and "before it was turned off" in why, str(why))
check("but the session that turned it off can",
firegex.set_auth_mode(False))
check("and it is asked for again",
firegex.status()["auth_disabled"] is False and anonymous.status()["loggined"] is False)
check("while the password still works",
FiregexAPI(args.address).login(args.password))
puts("List of available interfaces:", color=colors.yellow)
for interface in firegex.get_interfaces():
puts(
"name: {}, address: {}".format(interface["name"], interface["addr"]),
color=colors.yellow,
)

277
tests/bench/README.md Normal file
View File

@ -0,0 +1,277 @@
# Firegex benchmarks
## [GO BACK](../README.md)
Nothing here is a test. These scripts **measure** — they have no pass and no fail, they take minutes rather than seconds, and they are easy to misread, which is what
most of this page is about. `pytest` does not collect this directory.
| Script | What it answers |
|---|---|
| `benchmark.py` | how much bulk traffic a layer carries, against the number of patterns |
| `connbench.py` | how many short connections a second, which is the shape a CTF service actually sees |
| `stress_test_history.py` | a soak test for `HttpHistory` under load |
| `results_plotter.py` | redraws the SVGs in `results/` from the CSVs beside them |
These need more than the tests do — `pip3 install -r requirements.txt` here, on top of
the one a directory up, and `iperf3` itself installed as a package because the python
binding links against it.
# Running a benchmark
```bash
./benchmark.py -p testpassword -r 50 -d 1 -s 50 -t nfqueue
```
| Flag | Meaning |
|---|---|
| `-r, --num-of-regexes` | how many patterns to build up to; one measurement per pattern count |
| `-d, --duration` | seconds per measurement |
| `-s, --num-of-streams` | concurrent iperf3 streams |
| `-t, --transport` | `nfqueue` or `proxy` — which network layer to put in the path |
| `-o, --output-file` | CSV of `pattern count, MB/s` |
It stands up a service, measures the throughput with no filter, then adds one random
pattern at a time and measures again, using iperf3 over loopback. All the patterns live in
**one** regex filter, because that is the property worth measuring: they are compiled into
a single hyperscan database, so fifty of them should cost about what one costs.
## Reading the numbers
Three things decide whether a number here means anything at all.
**The filter has to be in the path.** `benchmark.py` creates its service with
`fail_open=False`, and that one argument is worth more than everything else on this page.
With fail-open on, the nft rule carries `bypass` and the queue is configured
`NFQA_CFG_F_FAIL_OPEN`: when the queue fills — which under fifty iperf3 streams it does
constantly — the kernel accepts packets **without inspecting them**. Measured that way the
same build reports ~32 000 MB/s instead of ~3 000, and the extra order of magnitude is
traffic that was never filtered. A benchmark of a filter has to make every packet go
through the filter.
**A pass is repeatable; a restart is not.** Three passes back to back against the same
running instance land within ~3% of each other. The same configuration measured again
after stopping and starting firegex moved by up to ~30%. So differences smaller than about
a third are not results, and any comparison across versions — each of which needs its own
instance — carries that uncertainty.
**One second per point is noisy on purpose.** `-d 1` is what the archived runs used and
what these use, so the curves are comparable with each other; it also means the wiggle
between adjacent points is measurement, not behaviour. Read the level and the slope.
# Performance
Measured on:
- MacBook Air M2, 16 GB
- OrbStack VM, Fedora Linux 43 aarch64, Linux 7.0.14, 7 CPUs
- `./benchmark.py -p testpassword -r 50 -d 1 -s 50 -t nfqueue` — and the same with
`-t proxy` for the proxy line. **The layer is part of the measurement**, so it is part of
every label, every filename (`5.0.0-nfqueue-8T.csv`, `5.0.0-proxy.csv`) and every table
row below: the two differ by more than three times, and a number without its layer is not
a number about firegex.
The `-d 1` in that command used to be load-bearing by accident. `benchmark.py` drove
iperf3 through the `iperf3` python binding, which captures libiperf's output by `dup2`-ing
stdout onto an `os.pipe()` and reads it only *after* the test returns — and a Linux pipe
holds 64 KiB. At fifty streams the JSON is 55 KB for a one-second test and 130 KB for a
five-second one, so anything past a second filled the pipe, blocked libiperf in `write()`,
and hung the benchmark for good: no output, no error, no timeout. It survived only because
the recorded runs happened to use the one duration that fits. iperf3 is now run as a
subprocess against a pipe something is draining, so `-d` is just a duration again.
## By version and thread count
Every line below was measured on that machine, in one sitting, with the filter in the path
for every packet. The versions are all driving their own `tests/benchmark.py` against their
own API — what that API is called changed across 3.x and 4.x, what is being measured did
not: a regex filter, one hyperscan database, throughput over loopback. **5.0.0** is the
tree these tests live in.
One chart per thread count, because the thread count moved a line as much as the version
did and seven lines on one pair of axes was something you decoded rather than read. The
dashed line in each is 5.0.0's **proxy** layer: not another version, the other way of
putting the same filter in the path.
### One thread
![One thread](results/Benchmark-1T.svg)
### Eight threads
![Eight threads](results/Benchmark-8T.svg)
### Medians
| Version | Layer | 1 thread | 8 threads | 8T / 1T |
|---|---|---|---|---|
| 3.5.3 | NFQUEUE | 1601 | 2620 | 1.6× |
| 4.0.5 | NFQUEUE | 1827 | 2461 | 1.3× |
| 5.0.0 | NFQUEUE | 1820 | 2956 | 1.6× |
| **5.0.0** | **Proxy** | **4035** | **13984** | **3.5×** |
What this supports:
- **Throughput is flat in the number of patterns**, on every version, both layers, both
thread counts. Fifty patterns cost about what one costs. That is hyperscan doing what it
is here for, and it is the clearest thing on either chart.
- **The proxy layer is the faster one, and the gap widens with threads**: 2.2× NFQUEUE at
one thread, **4.7× at eight**. NFQUEUE pays a userspace round trip per packet; the proxy
pays once per connection and then the kernel moves the bytes. Both gaps are several
times the measurement uncertainty below, so both are results.
- **The proxy scales better across threads too** — 3.5× from one to eight, against ~1.5×
for NFQUEUE, which is bounded by how much of its work is a per-packet trip through the
kernel rather than something a second core can do.
- **The NFQUEUE versions are not separable at this resolution.** The spread across passes
of the same version is as wide as the gap between versions. 5.0.0 measures highest and
most steadily and that is as much as these numbers say; the charts are not evidence that
it is faster than 4.0.5.
The `3.5.3-8T` pass in `results/3.5.3-8T-outlier.csv` collapsed to ~500 MB/s past the
seventeenth pattern and did not reproduce on the next pass. It is kept because leaving it
out would make the run-to-run spread look smaller than it is.
### Measured again after the nftables tables moved
`results/*-rerun.csv` is the whole table above, taken a second time after the firewall
module moved out of the shared `filter`/`mangle` tables into `fgex_filter` and
`fgex_mangle` — the kind of change that has no business costing throughput, which is why
it is worth showing that it did not. Same Mac, same protocol (`-r 50 -d 1 -s 50`), but
from a container sharing the datapath's network namespace rather than from the Fedora
machine, so read it as a second host and not as a repeat:
| Layer | 1 thread | 8 threads | 8T / 1T |
|---|---|---|---|
| NFQUEUE | 2205 *(was 1820)* | 2940 *(was 2956)* | 1.3× *(was 1.6×)* |
| **Proxy** | **4341** *(was 4035)* | **14590** *(was 13984)* | **3.4×** *(was 3.5×)* |
The proxy is 2.0× NFQUEUE at one thread and 5.0× at eight, against 2.2× and 4.7× before.
Every medians-level claim above survives; none of them survives to the decimal, which the
run-to-run spread in the same table already said.
**Duration is part of the protocol, and the numbers here are only comparable at `-d 1`.**
The same four runs at the default `-d 5` land at 1.9× and 3.9× — the layers' gap narrows
as each reading settles. Neither pair is wrong; quoting one against the other is.
`--threads` reaches **both** layers: it is passed to `cppregex` and `cpproxy`, and — since
5.0.0 — to the Rust proxy engine, which builds its tokio runtime with that many workers
instead of taking every core whatever the operator asked for. Before that the same flag
meant one thing on one layer and nothing on the other, and there could be only one proxy
line on these charts.
### Versions that can no longer be measured
`2.3.3`, `2.4.0`, `2.5.1`, `3.0.0` and `3.2.x` pin no dependency versions and either pin
no base image or pin `fedora:latest`, so building them today resolves to current packages.
Their containers **do not start**: `passlib` 1.7.4 probes its bcrypt backend at import and
today's `bcrypt` answers with `ValueError: password cannot be longer than 72 bytes`, which
kills the application at startup. `3.4.11` builds and starts but the datapath dies partway
through the benchmark under fifty streams.
Their old numbers are still in `results/` and are rendered further down, but they were
measured in 2025 on Fedora 41 and Linux 6.12. They are **not** comparable with anything
above — that machine is roughly an order of magnitude away from this one on the same
benchmark — and they are kept as a record of what was measured then, not as a baseline.
## Short connections
Bulk throughput is the shape a terminating proxy is best at and the shape a CTF service
almost never sees. The cost the proxy is supposed to pay is in *opening* a connection,
which the throughput benchmark never touches, so there is a second script for it:
```bash
./connbench.py -p testpassword [--passes 5] [-c 1500] [-s 32]
```
Open, one small request, one small response, close — measured as connections per second
against an unfiltered baseline taken in the same run.
| 5.0.0, short connections | conn/s | of unfiltered | across passes |
|---|---|---|---|
| unfiltered baseline | 11182 | — | 8437 – 14014 |
| Proxy | 10181 | 91% | 9687 – 10346 |
| NFQUEUE | 9870 | 88% | 7318 – 12117 |
**There is no measurable difference.** The medians land 3% apart and the ranges overlap
heavily, so the honest answer on this axis is "the same", and the script says so itself
rather than printing a ratio.
Getting there took three wrong answers, which is the useful part. Measuring every pass of
one layer and then every pass of the other gave the proxy a 1.4× lead twice, then a 1.0×,
then NFQUEUE a lead — because whichever went second inherited the machine's drift.
`connbench.py` now measures both layers **within** each pass and alternates which goes
first, so drift lands on both and shows up as spread instead of as a winner.
None of this is an argument for always choosing the proxy. What it costs is the kernel's
fail-open backstop — not the client's address, which both layers preserve on TCP and on
UDP alike; `-s` and `-d` on `benchmark.py` are there so you can measure the shape you
actually care about.
# Archived results (2025)
Everything below was measured on the machine described at the time — a MacBook Air M2 on
OrbStack with **Fedora 41** and Linux 6.12.13 — and is kept as the record it was. Do not
read it against the section above.
Command: `./benchmark.py -p testpassword -r 50 -d 1 -s 50`
NOTE: 8-thread performance before 2.5.0 does not change, because the source and
destination IP are always the same, so the kernel hashes every packet onto the same
thread. See
[this thread](https://netfilter.vger.kernel.narkive.com/sTP7613Y/meaning-of-nfqueue-s-queue-balance-option).
In a CTF there is usually a NAT hiding the real IPs, so this is not a hypothetical.
Firegex 2.5.0 assigns threads in userland instead, which is what makes the distribution
work.
Charts are labelled `[version]-[n_thread]T`, e.g. `2.5.0-8T`.
![Firegex Benchmark](results/Benchmark-chart.svg)
The advantage of multithreading is hard to see there, and much clearer with a fake load in
the filtering path:
```cpp
volatile int x = 0;
for (int i=0; i<50000; i++){
x+=1;
}
```
![Firegex Benchmark](results/Benchmark-chart-with-load.svg)
## nfregex vs nfproxy (2025)
> These are **archived**, and the script that produced them is gone. It drove an API
> that no longer exists: back then `nfregex` and `nfproxy` were separate modules, and both
> were folded into `services` — what `nfproxy` did is now the **pyfilters** filter kind.
> A benchmark that cannot be run is not tooling, so only the numbers are kept, under the
> names they were recorded with rather than names anything still answers to. The current
> equivalent of this comparison is [By network layer](#by-network-layer) above.
The code under test matched this pattern with Python's `re` module:
```
(?:[a-z0-9!#$%&'*+/=?^_`{|}~-]+(?:\.[a-z0-9!#$%&'*+/=?^_`{|}~-]+)*|"(?:[\x01-\x08\x0b\x0c\x0e-\x1f\x21\x23-\x5b\x5d-\x7f]|\\[\x01-\x09\x0b\x0c\x0e-\x7f])*")@(?:(?:[a-z0-9](?:[a-z0-9-]*[a-z0-9])?\.)+[a-z0-9](?:[a-z0-9-]*[a-z0-9])?|\[(?:(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.){3}(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?|[a-z0-9-]*[a-z0-9]:(?:[\x01-\x08\x0b\x0c\x0e-\x1f\x21-\x5a\x53-\x7f]|\\[\x01-\x09\x0b\x0c\x0e-\x7f])+)\])
```
![nfproxy benchmarks](results/whisker_nfproxy.svg)
![nfproxy benchmarks](results/istogramma_nfproxy.svg)
nfproxy was slower than nfregex, and more flexible.
![nfproxy benchmarks](results/whisker_compare.svg)
![nfproxy benchmarks](results/istrogramma_compare.svg)
# Regenerating the charts
`results_plotter.py` reads the CSVs in `results/` and writes the SVGs beside them:
```bash
cd tests/bench && python3 results_plotter.py
```
That writes the two current charts, through one shared `line_chart()` helper, and stops.
The archived charts are behind `--archived`, because matplotlib has moved on since they
were produced and redrawing them changes six committed SVGs for no reason other than a
different font being available.

12
tests/bench/_path.py Normal file
View File

@ -0,0 +1,12 @@
"""Make `helpers` importable from a script run out of this directory.
These are standalone CLI tools rather than pytest modules, so nothing has arranged
`sys.path` for them. `import _path` at the top of one is the whole of the arrangement.
"""
import os
import sys
TESTS_DIR = os.path.abspath(os.path.join(os.path.dirname(__file__), ".."))
if TESTS_DIR not in sys.path:
sys.path.insert(0, TESTS_DIR)

5
tests/benchmark.py → tests/bench/benchmark.py
View File

@ -1,6 +1,7 @@
#!/usr/bin/env python3
from utils.colors import colors, puts, sep
from utils.firegexapi import FiregexAPI
import _path # noqa: F401
from helpers.colors import colors, puts, sep
from helpers.firegexapi import FiregexAPI
from time import sleep
import csv
import argparse

5
tests/connbench.py → tests/bench/connbench.py
View File

@ -31,8 +31,9 @@ import threading
import time
from concurrent.futures import ThreadPoolExecutor
from utils.colors import colors, puts, sep
from utils.firegexapi import FiregexAPI
import _path # noqa: F401
from helpers.colors import colors, puts, sep
from helpers.firegexapi import FiregexAPI
parser = argparse.ArgumentParser()
parser.add_argument("--address", "-a", type=str, required=False,

6
tests/bench/requirements.txt Normal file
View File

@ -0,0 +1,6 @@
# On top of ../requirements.txt.
# `iperf3` is a binding against libiperf, so the library has to be installed first
# (`apt-get install iperf3`, `dnf install iperf3`).
iperf3
matplotlib
numpy

0
tests/results/2.3.3-1T.csv → tests/bench/results/2.3.3-1T.csv
View File

0
tests/results/2.3.3-8T.csv → tests/bench/results/2.3.3-8T.csv
View File

0
tests/results/2.4.0-1T.csv → tests/bench/results/2.4.0-1T.csv
View File

0
tests/results/2.4.0-8T.csv → tests/bench/results/2.4.0-8T.csv
View File

0
tests/results/2.5.1-1T-withload.csv → tests/bench/results/2.5.1-1T-withload.csv
View File

0
tests/results/2.5.1-1T.csv → tests/bench/results/2.5.1-1T.csv
View File

0
tests/results/2.5.1-8T-withload.csv → tests/bench/results/2.5.1-8T-withload.csv
View File

0
tests/results/2.5.1-8T.csv → tests/bench/results/2.5.1-8T.csv
View File

0
tests/results/3.5.3-1T.csv → tests/bench/results/3.5.3-1T.csv
View File

0
tests/results/3.5.3-8T-outlier.csv → tests/bench/results/3.5.3-8T-outlier.csv
View File

0
tests/results/3.5.3-8T.csv → tests/bench/results/3.5.3-8T.csv
View File

0
tests/results/4.0.5-1T.csv → tests/bench/results/4.0.5-1T.csv
View File

0
tests/results/4.0.5-8T-run2.csv → tests/bench/results/4.0.5-8T-run2.csv
View File

0
tests/results/4.0.5-8T.csv → tests/bench/results/4.0.5-8T.csv
View File

0
tests/results/5.0.0-nfqueue-1T-rerun.csv → tests/bench/results/5.0.0-nfqueue-1T-rerun.csv
View File

0
tests/results/5.0.0-nfqueue-1T.csv → tests/bench/results/5.0.0-nfqueue-1T.csv
View File

0
tests/results/5.0.0-nfqueue-8T-rerun.csv → tests/bench/results/5.0.0-nfqueue-8T-rerun.csv
View File

0
tests/results/5.0.0-nfqueue-8T-run2.csv → tests/bench/results/5.0.0-nfqueue-8T-run2.csv
View File

0
tests/results/5.0.0-nfqueue-8T.csv → tests/bench/results/5.0.0-nfqueue-8T.csv
View File

0
tests/results/5.0.0-proxy-1T-rerun.csv → tests/bench/results/5.0.0-proxy-1T-rerun.csv
View File

0
tests/results/5.0.0-proxy-1T.csv → tests/bench/results/5.0.0-proxy-1T.csv
View File

0
tests/results/5.0.0-proxy-8T-rerun.csv → tests/bench/results/5.0.0-proxy-8T-rerun.csv
View File

0
tests/results/5.0.0-proxy-8T.csv → tests/bench/results/5.0.0-proxy-8T.csv
View File

0
tests/results/Benchmark-1T.svg → tests/bench/results/Benchmark-1T.svg
View File

Side by Side Swipe Overlay

Before

Width:  |  Height:  |  Size: 50 KiB

After

Width:  |  Height:  |  Size: 50 KiB

0
tests/results/Benchmark-8T.svg → tests/bench/results/Benchmark-8T.svg
View File

Side by Side Swipe Overlay

Before

Width:  |  Height:  |  Size: 57 KiB

After

Width:  |  Height:  |  Size: 57 KiB

0
tests/results/Benchmark-chart-with-load.svg → tests/bench/results/Benchmark-chart-with-load.svg
View File

Side by Side Swipe Overlay

Before

Width:  |  Height:  |  Size: 43 KiB

After

Width:  |  Height:  |  Size: 43 KiB

0
tests/results/Benchmark-chart.svg → tests/bench/results/Benchmark-chart.svg
View File

Side by Side Swipe Overlay

Before

Width:  |  Height:  |  Size: 55 KiB

After

Width:  |  Height:  |  Size: 55 KiB

0
tests/results/comparemark_nfproxy_1T.csv → tests/bench/results/comparemark_nfproxy_1T.csv
View File

0
tests/results/comparemark_nfproxy_8T.csv → tests/bench/results/comparemark_nfproxy_8T.csv
View File

0
tests/results/comparemark_nfregex_1T.csv → tests/bench/results/comparemark_nfregex_1T.csv
View File

0
tests/results/comparemark_nfregex_8T.csv → tests/bench/results/comparemark_nfregex_8T.csv
View File

0
tests/results/istogramma_nfproxy.svg → tests/bench/results/istogramma_nfproxy.svg
View File

Side by Side Swipe Overlay

Before

Width:  |  Height:  |  Size: 52 KiB

After

Width:  |  Height:  |  Size: 52 KiB

0
tests/results/istrogramma_compare.svg → tests/bench/results/istrogramma_compare.svg
View File

Side by Side Swipe Overlay

Before

Width:  |  Height:  |  Size: 68 KiB

After

Width:  |  Height:  |  Size: 68 KiB

0
tests/results/whisker_compare.svg → tests/bench/results/whisker_compare.svg
View File

Side by Side Swipe Overlay

Before

Width:  |  Height:  |  Size: 62 KiB

After

Width:  |  Height:  |  Size: 62 KiB

0
tests/results/whisker_nfproxy.svg → tests/bench/results/whisker_nfproxy.svg
View File

Side by Side Swipe Overlay

Before

Width:  |  Height:  |  Size: 45 KiB

After

Width:  |  Height:  |  Size: 45 KiB

14
tests/results_plotter.py → tests/bench/results_plotter.py
View File

@ -1,3 +1,15 @@
"""Redraws the charts in `results/` from the CSVs beside them.
The archived half reads data recorded by `comparemark.py`, which is gone — it drove the
`nfregex`/`nfproxy` API from before both were folded into `services`. The CSVs it left
are a record of what was measured then, so they are still plottable; there is just no
longer a way to produce new ones, which is why that half is behind `--archived`.
Paths are resolved against this file rather than the working directory, so it draws the
same thing wherever it is run from.
"""
import os
import sys
import matplotlib.pyplot as plt
@ -6,6 +18,8 @@ import csv
from matplotlib.ticker import MaxNLocator
from matplotlib import cm
os.chdir(os.path.dirname(os.path.abspath(__file__)))
plt.style.use("fivethirtyeight")
colors = cm.Set1.colors # Use a different strong color palette
plt.rcParams["figure.facecolor"] = "white"

4
tests/stress_test_history.py → tests/bench/stress_test_history.py
View File

@ -13,12 +13,12 @@ import signal
import socket
import threading
import argparse
import traceback
from typing import Optional, List
# Add project paths
sys.path.insert(0, os.path.abspath(os.path.join(os.path.dirname(__file__), "..")))
sys.path.insert(0, os.path.abspath(os.path.join(os.path.dirname(__file__), "../fgex-lib")))
sys.path.insert(0, os.path.abspath(os.path.join(os.path.dirname(__file__), "../..")))
sys.path.insert(0, os.path.abspath(os.path.join(os.path.dirname(__file__), "../../fgex-lib")))
try:
import psutil

235
tests/comparemark.py
View File

@ -1,235 +0,0 @@
#!/usr/bin/env python3
from utils.colors import colors, puts, sep
from utils.firegexapi import FiregexAPI
from multiprocessing import Process
from time import sleep
import iperf3
import csv
import argparse
parser = argparse.ArgumentParser()
parser.add_argument("module", type=str, help='module to analyse', choices=['nfregex','nfproxy'])
parser.add_argument("--address", "-a", type=str , required=False, help='Address of firegex backend', default="http://127.0.0.1:4444/")
parser.add_argument("--port", "-P", type=int , required=False, help='Port of the Benchmark service', default=1337)
parser.add_argument("--service-name", "-n", type=str , required=False, help='Name of the Benchmark service', default="Benchmark Service")
parser.add_argument("--password", "-p", type=str, required=True, help='Firegex password')
parser.add_argument("--duration", "-d", type=int, required=False, help='Duration of the Benchmark in seconds', default=1)
parser.add_argument("--output-file", "-o", type=str, required=False, help='Output results csv file', default="comparemark.csv")
parser.add_argument("--num-of-streams", "-s", type=int, required=False, help='Number of concurrent streams', default=1)
parser.add_argument("--number-of-values", "-V", type=int, required=False, help='Number of values to generate', default=10)
parser.add_argument("--test-baseline", action='store_true', required=False, help='Test baseline without any filters')
args = parser.parse_args()
sep()
puts("Benchmark for compare firegex features ", color=colors.cyan, end="")
puts(f"{args.address}", color=colors.yellow)
firegex = FiregexAPI(args.address)
#Connect to Firegex
if (firegex.login(args.password)):
puts("Sucessfully logged in ✔", color=colors.green)
else:
puts("Benchmark Failed: Unknown response or wrong passowrd ✗", color=colors.red)
exit(1)
#Cleaning services
srvs = firegex.nfregex_get_services()
for ele in srvs:
if ele['name'] == args.service_name:
firegex.nfregex_delete_service(ele['service_id'])
srvs = firegex.nfproxy_get_services()
for ele in srvs:
if ele['name'] == args.service_name:
firegex.nfproxy_delete_service(ele['service_id'])
args.port = int(args.port)
args.duration = int(args.duration)
args.num_of_streams = int(args.num_of_streams)
args.number_of_values = int(args.number_of_values)
#Start iperf3
def startServer():
try:
server = iperf3.Server()
server.bind_address = '127.0.0.1'
server.port = args.port
server.verbose = False
while True:
server.run()
except Exception as e:
puts(f"Failed to start/run iperf3 server: {e}", color=colors.red)
global server
server = Process(target=startServer)
server.start()
sleep(1)
def getReading(port):
global server
attempt = 1
while True:
try:
client = iperf3.Client()
client.duration = args.duration
client.server_hostname = '127.0.0.1'
client.port = port
client.zerocopy = True
client.verbose = False
client.protocol = 'tcp'
client.num_streams = args.num_of_streams
return round(client.run().json['end']['sum_received']['bits_per_second']/8e+6 , 3)
except Exception as e:
if attempt >= 3:
raise e
else:
server.kill()
server = Process(target=startServer)
server.start()
sleep(1)
puts(f"Faild to run test on attempt: {attempt}, retrying...", color=colors.red)
attempt += 1
baseline_data = []
no_filters = []
test_data = []
if args.test_baseline:
for _ in range(args.number_of_values):
#Get baseline reading
data = getReading(args.port)
baseline_data.append(data)
puts("Baseline without any filter: ", color=colors.blue, end='')
print(f"{data} MB/s")
#Create new Service
test_regex = '(?:[a-z0-9!#$%&\'*+/=?^_`{|}~-]+(?:\\.[a-z0-9!#$%&\'*+/=?^_`{|}~-]+)*|"(?:[\\x01-\\x08\\x0b\\x0c\\x0e-\\x1f\\x21\\x23-\\x5b\\x5d-\\x7f]|\\\\[\\x01-\\x09\\x0b\\x0c\\x0e-\\x7f])*")@(?:(?:[a-z0-9](?:[a-z0-9-]*[a-z0-9])?\\.)+[a-z0-9](?:[a-z0-9-]*[a-z0-9])?|\\[(?:(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\\.){3}(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?|[a-z0-9-]*[a-z0-9]:(?:[\\x01-\\x08\\x0b\\x0c\\x0e-\\x1f\\x21-\\x5a\\x53-\\x7f]|\\\\[\\x01-\\x09\\x0b\\x0c\\x0e-\\x7f])+)\\])'
BASE_FILTER = f"""
from firegex.nfproxy.models import RawPacket
from firegex.nfproxy import pyfilter, REJECT
import re
@pyfilter
def verdict_test(packet:RawPacket):
if re.match({repr(test_regex.encode())}, packet.data):
return REJECT
"""
if args.module == "nfregex":
def exit_test(code):
if service_id:
server.kill()
if(firegex.nfregex_delete_service(service_id)):
puts("Sucessfully deleted service ✔", color=colors.green)
else:
puts("Test Failed: Coulnd't delete serivce ✗", color=colors.red)
exit_test(1)
exit(code)
service_id = firegex.nfregex_add_service(args.service_name, args.port, "tcp", "127.0.0.1/32")
if service_id:
puts(f"Sucessfully created service {service_id} ✔", color=colors.green)
else:
puts("Test Failed: Failed to create service ✗", color=colors.red)
exit(1)
#Start nfregex service
if firegex.nfregex_start_service(service_id):
puts(f"Sucessfully started service with id {service_id} ✔", color=colors.green)
else:
puts("Benchmark Failed: Coulnd't start the service ✗", color=colors.red)
exit_test(1)
#Get no regexes reading
for _ in range(args.number_of_values):
#Get baseline reading
data = getReading(args.port)
no_filters.append(data)
puts("Baseline nfregex with no filter: ", color=colors.blue, end='')
print(f"{data} MB/s")
if firegex.nfregex_add_regex(service_id,test_regex,"B",active=True,is_case_sensitive=False):
puts("Sucessfully added test regex ✔", color=colors.green)
else:
puts("Benchmark Failed: Couldn't add the regex ✗", color=colors.red)
exit_test(1)
for _ in range(args.number_of_values):
#Get baseline reading
data = getReading(args.port)
puts("Baseline nfregex with 1 filter: ", color=colors.blue, end='')
test_data.append(data)
print(f"{data} MB/s")
if(firegex.nfregex_delete_service(service_id)):
puts("Sucessfully deleted service ✔", color=colors.green)
else:
puts("Test Failed: Coulnd't delete serivce ✗", color=colors.red)
exit_test(1)
elif args.module == "nfproxy":
service_id = firegex.nfproxy_add_service(args.service_name, args.port, "http" , "127.0.0.1")
if service_id:
puts(f"Sucessfully created service {service_id} ✔", color=colors.green)
else:
puts("Test Failed: Failed to create service ✗", color=colors.red)
exit(1)
def exit_test(code):
if service_id:
server.kill()
if firegex.nfproxy_delete_service(service_id):
puts("Sucessfully deleted service ✔", color=colors.green)
else:
puts("Test Failed: Coulnd't delete serivce ✗", color=colors.red)
exit(1)
exit(code)
if(firegex.nfproxy_start_service(service_id)):
puts("Sucessfully started service ✔", color=colors.green)
else:
puts("Test Failed: Failed to start service ✗", color=colors.red)
exit_test(1)
#Get no filters
for _ in range(args.number_of_values):
#Get baseline reading
data = getReading(args.port)
puts("Baseline nfproxy with no filter: ", color=colors.blue, end='')
no_filters.append(data)
print(f"{data} MB/s")
if firegex.nfproxy_set_code(service_id, BASE_FILTER):
puts("Sucessfully added filter for test in REJECT mode ✔", color=colors.green)
else:
puts("Test Failed: Couldn't add the filter for test ✗", color=colors.red)
exit_test(1)
for _ in range(args.number_of_values):
#Get baseline reading
data = getReading(args.port)
puts("Baseline nfproxy with 1 filter: ", color=colors.blue, end='')
test_data.append(data)
print(f"{data} MB/s")
if(firegex.nfproxy_delete_service(service_id)):
puts("Sucessfully deleted service ✔", color=colors.green)
else:
puts("Test Failed: Coulnd't delete serivce ✗", color=colors.red)
exit_test(1)
with open(args.output_file,'w') as f:
writer = csv.writer(f)
if baseline_data:
writer.writerow(["Baseline","No filters","test data"])
for data in zip(baseline_data, no_filters, test_data):
writer.writerow(data)
else:
writer.writerow(["No filters","test data"])
for data in zip(no_filters, test_data):
writer.writerow(data)
puts(f"Sucessfully written results to {args.output_file} ✔", color=colors.magenta)
server.terminate()

127
tests/conftest.py Normal file
View File

@ -0,0 +1,127 @@
"""One place that knows where the instance under test is, and what may be asked of it.
Before this, four scripts each grew their own `--address`/`--password` argparse block, a
fifth read the same two things out of the environment, and a shell script held the list
of which combinations to run. The combinations are the interesting part of this suite —
the same filters on both network layers, over IPv4 and IPv6, with and without TLS — and
they belong in the test code as parameters rather than in a shell loop, where nothing can
skip one that this host cannot carry.
Selection is by command line, so a developer can narrow to the case they are fixing:
pytest # everything this host can run
pytest tests/unit # no instance needed
pytest -m "not slow" # skip the ones that wait on timeouts
pytest --layer proxy --no-ipv6 # one layer, one family
pytest --fg-address http://box:4444/ # somewhere other than localhost
"""
import os
import sys
import pytest
TESTS_DIR = os.path.dirname(os.path.abspath(__file__))
BACKEND_DIR = os.path.abspath(os.path.join(TESTS_DIR, "..", "backend"))
LIB_DIR = os.path.abspath(os.path.join(TESTS_DIR, "..", "fgex-lib"))
# `tests/` first so `helpers` resolves here, and the backend after it. The two used to
# collide outright: this directory's package was called `utils`, which is also the name
# of the backend's, so whichever imported first won and the other module failed to
# collect. That is why these could not be run in one pytest process at all.
for path in (TESTS_DIR, LIB_DIR, BACKEND_DIR):
if path not in sys.path:
sys.path.append(path)
sys.path.insert(0, TESTS_DIR)
from helpers.firegexapi import FiregexAPI # noqa: E402
from helpers.net import supports_ipv6 # noqa: E402
def pytest_addoption(parser):
group = parser.getgroup("firegex")
group.addoption("--fg-address", action="store",
default=os.getenv("FIREGEX_ADDRESS", "http://127.0.0.1:4444/"),
help="Where the instance under test is")
group.addoption("--fg-password", action="store",
default=os.getenv("FIREGEX_PASSWORD", "testpassword"),
help="Its password")
group.addoption("--layer", action="append", default=[],
choices=["proxy", "nfqueue", "external"],
help="Only these network layers (repeatable; default: all of them)")
group.addoption("--no-ipv6", action="store_true", default=False,
help="Skip the IPv6 half of every parametrised case")
group.addoption("--no-tls", action="store_true", default=False,
help="Skip the TLS cases")
def pytest_configure(config):
for marker in (
"instance: needs a running firegex to talk to",
"root: needs root and a Linux kernel with nftables",
"slow: waits on a real timeout and takes seconds to do it",
"ipv6: exercises the IPv6 half",
"tls: exercises TLS termination",
):
config.addinivalue_line("markers", marker)
def pytest_collection_modifyitems(config, items):
"""Skip what this host or this invocation cannot carry, rather than failing it.
A skip with a reason beats a red test nobody can act on: a laptop without IPv6 on
loopback is not a firegex bug, and saying so in the summary line is more useful than
a connection error inside an assertion.
"""
wanted_layers = set(config.getoption("--layer"))
no_ipv6 = config.getoption("--no-ipv6") or not supports_ipv6()
no_ipv6_reason = ("--no-ipv6 was given" if config.getoption("--no-ipv6")
else "this host has no IPv6 loopback")
no_tls = config.getoption("--no-tls")
for item in items:
if wanted_layers:
layer = item.callspec.params.get("layer") if hasattr(item, "callspec") else None
name = getattr(layer, "transport", None)
if name and name not in wanted_layers:
item.add_marker(pytest.mark.skip(reason=f"--layer did not ask for {name}"))
if no_ipv6 and "ipv6" in item.keywords:
item.add_marker(pytest.mark.skip(reason=no_ipv6_reason))
if no_tls and "tls" in item.keywords:
item.add_marker(pytest.mark.skip(reason="--no-tls was given"))
@pytest.fixture(scope="session")
def fg_address(pytestconfig) -> str:
address = pytestconfig.getoption("--fg-address")
return address if address.endswith("/") else address + "/"
@pytest.fixture(scope="session")
def fg_password(pytestconfig) -> str:
return pytestconfig.getoption("--fg-password")
@pytest.fixture(scope="session")
def api(fg_address, fg_password) -> FiregexAPI:
"""A logged-in client, shared by everything that needs an instance.
An instance started with `--unsafe-disable-auth` answers 403 to the login endpoint
and accepts every request without a token — `login()` reads that as "there is nothing
to log in to" rather than as a failure, which is what lets the whole suite run
against the one configuration where every request is already allowed.
"""
client = FiregexAPI(fg_address)
if not client.login(fg_password):
pytest.fail(
f"Could not reach or authenticate to firegex at {fg_address}. "
f"Start one with `python3 run.py start -P {fg_password}`, or point the suite "
f"elsewhere with --fg-address."
)
return client
@pytest.fixture(scope="session")
def auth_enabled(api) -> bool:
"""Whether this instance asks for a password at all."""
return not api.status().get("auth_disabled", False)

0
tests/utils/__init__.py → tests/helpers/__init__.py
View File

72
tests/helpers/certs.py Normal file
View File

@ -0,0 +1,72 @@
"""Certificates for the TLS tests, generated in-process.
No `openssl` shelling out and no fixture files: a checked-in certificate expires, and the
morning it does every TLS test fails for a reason that has nothing to do with firegex.
`key_size` and the non-RSA generators are parameters rather than conveniences. The engine
refuses a key its signing backend will not sign with — RSA under 2048 bits — and that
refusal is something to test, so the suite has to be able to produce one. The others are
here because rustls accepts them and an operator will eventually bring one.
"""
import datetime
import ipaddress
from cryptography import x509
from cryptography.hazmat.primitives import hashes, serialization
from cryptography.hazmat.primitives.asymmetric import ec, ed25519, rsa
from cryptography.x509.oid import NameOID
def _build(key, hash_algorithm, common_name: str) -> tuple[str, str]:
subject = issuer = x509.Name([x509.NameAttribute(NameOID.COMMON_NAME, common_name)])
now = datetime.datetime.now(datetime.timezone.utc)
builder = (
x509.CertificateBuilder()
.subject_name(subject)
.issuer_name(issuer)
.public_key(key.public_key())
.serial_number(x509.random_serial_number())
# A day of slack at the front: a container whose clock is a little behind the
# host would otherwise reject a certificate minted a moment ago.
.not_valid_before(now - datetime.timedelta(days=1))
.not_valid_after(now + datetime.timedelta(days=365))
)
# The SAN has to name the address the test actually dials. It used to be hardcoded
# to 127.0.0.1, so an IPv6 run presented a certificate for an address nobody in that
# run was talking to.
try:
alt = x509.IPAddress(ipaddress.ip_address(common_name))
except ValueError:
alt = x509.DNSName(common_name)
builder = builder.add_extension(x509.SubjectAlternativeName([alt]), critical=False)
cert = builder.sign(key, hash_algorithm)
return (
cert.public_bytes(serialization.Encoding.PEM).decode(),
key.private_bytes(
serialization.Encoding.PEM,
serialization.PrivateFormat.PKCS8,
serialization.NoEncryption(),
).decode(),
)
def rsa_cert(common_name: str = "127.0.0.1", key_size: int = 2048) -> tuple[str, str]:
return _build(rsa.generate_private_key(public_exponent=65537, key_size=key_size),
hashes.SHA256(), common_name)
def ecdsa_cert(common_name: str = "127.0.0.1", curve: str = "secp256r1") -> tuple[str, str]:
chosen = ec.SECP384R1() if curve == "secp384r1" else ec.SECP256R1()
return _build(ec.generate_private_key(chosen), hashes.SHA256(), common_name)
def ed25519_cert(common_name: str = "127.0.0.1") -> tuple[str, str]:
return _build(ed25519.Ed25519PrivateKey.generate(), None, common_name)
#: Material whose envelope is right and whose body will not parse. The only way to get
#: a service past the backend's own check and as far as the engine, which is where the
#: refusal being tested comes from.
UNPARSEABLE_CERT = "-----BEGIN CERTIFICATE-----\nnope\n-----END CERTIFICATE-----\n"
UNPARSEABLE_KEY = "-----BEGIN PRIVATE KEY-----\nnope\n-----END PRIVATE KEY-----\n"

0
tests/utils/colors.py → tests/helpers/colors.py
View File

43
tests/utils/firegexapi.py → tests/helpers/firegexapi.py
View File

@ -1,7 +1,7 @@
import string
from requests import Session
import base64
from requests import Session
def verify(req):
try:
assert(req.json()["status"] == "ok")
@ -125,7 +125,8 @@ class FiregexAPI:
tls: bool = False, tls_cert: str | None = None, tls_key: str | None = None,
proxy_ip: str | None = None, proxy_port: int | None = None,
addresses: list | None = None,
max_connections: int = 0, over_limit_forwards: bool = False):
max_connections: int = 0, over_limit_forwards: bool = False,
first_byte_timeout: int = 0):
"""A service takes a list of addresses; the single-address case is the common one.
`tls=True` is kept as a convenience for the callers that read as "and behind
@ -140,6 +141,7 @@ class FiregexAPI:
"proto": "tls" if tls else proto, "fail_open": fail_open,
"max_connections": max_connections,
"over_limit_forwards": over_limit_forwards,
"first_byte_timeout": int(first_byte_timeout),
"tls_cert": tls_cert, "tls_key": tls_key,
})
res = req.json()
@ -371,38 +373,3 @@ class FiregexAPI:
"sample": base64.b64encode(sample).decode(),
})
return req.json()
#PortHijack
def ph_get_services(self):
req = self.s.get(f"{self.address}api/porthijack/services")
return req.json()
def ph_get_service(self,service_id: str):
req = self.s.get(f"{self.address}api/porthijack/services/{service_id}")
return req.json()
def ph_stop_service(self,service_id: str):
req = self.s.post(f"{self.address}api/porthijack/services/{service_id}/stop")
return verify(req)
def ph_start_service(self,service_id: str):
req = self.s.post(f"{self.address}api/porthijack/services/{service_id}/start")
return verify(req)
def ph_delete_service(self,service_id: str):
req = self.s.delete(f"{self.address}api/porthijack/services/{service_id}")
return verify(req)
def ph_rename_service(self,service_id: str,newname: str):
req = self.s.put(f"{self.address}api/porthijack/services/{service_id}/rename" , json={"name":newname})
return verify(req)
def ph_change_destination(self,service_id: str, ip_dst:string , proxy_port: int):
req = self.s.put(f"{self.address}api/porthijack/services/{service_id}/change-destination", json={"ip_dst": ip_dst, "proxy_port": proxy_port})
return verify(req)
def ph_add_service(self, name: str, public_port: int, proxy_port: int, proto: str, ip_src: str, ip_dst: str):
req = self.s.post(f"{self.address}api/porthijack/services" ,
json={"name":name, "public_port": public_port, "proxy_port":proxy_port, "proto": proto, "ip_src": ip_src, "ip_dst": ip_dst})
return req.json()["service_id"] if verify(req) else False

32
tests/helpers/net.py Normal file
View File

@ -0,0 +1,32 @@
"""Ports and the small network primitives the tests share.
Ports are asked for rather than derived from a base. The suite used to take a `--port`
and reach for `port + 2`, `port + 3`, `port + 5`, `port + 7`, `port + 9`, `port + 11`,
`port + 13` — which is fine until two of those runs overlap, or the host already has
something on one of them, and the failure reads as firegex refusing traffic.
"""
import socket
def free_port(ipv6: bool = False) -> int:
"""A port nothing is listening on, on the loopback address the caller will use."""
family = socket.AF_INET6 if ipv6 else socket.AF_INET
with socket.socket(family, socket.SOCK_STREAM) as probe:
probe.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
probe.bind(("::1" if ipv6 else "127.0.0.1", 0))
return probe.getsockname()[1]
def loopback(ipv6: bool = False) -> str:
return "::1" if ipv6 else "127.0.0.1"
def supports_ipv6() -> bool:
"""Whether this host can carry the IPv6 half of the suite at all."""
try:
with socket.socket(socket.AF_INET6, socket.SOCK_STREAM) as probe:
probe.bind(("::1", 0))
return True
except OSError:
return False

Some files were not shown because too many files have changed in this diff Show More