ramjet-ingress

A Kubernetes ingress controller with a native Rust data plane. No nginx, no config file regeneration, no reload: a configuration change swaps an Arc, and in-flight connections never notice.

Feature target is parity with kubernetes/ingress-nginx on the networking.k8s.io/v1 Ingress resource. There is no nginx anywhere in the design.

helm install ramjet deploy/chart/ramjet-ingress \
  --namespace ramjet-ingress --create-namespace

The thesis: swap a pointer, do not reload

ingress-nginx reacts to a configuration change by regenerating nginx.conf and reloading. A reload forks new workers, drains the old ones, and in the process resets upstream state and severs connections that were meant to be long-lived. The cost of a config change is proportional to how much traffic you are carrying, which is backwards: the busier you are, the more a routine deploy hurts.

Here the control plane compiles configuration into an immutable RouteTable and publishes it by storing one pointer into an arc_swap::ArcSwap. The data plane does a single atomic load per request and then reads an immutable snapshot. There is no RwLock, no reader-writer contention, no reload, and no draining.

Kubernetes API                    ArcSwap<RouteTable>              worker
     |                                    |                          |
  watch Ingress/Service/Secret            |                     load() -- 1 atomic
     |                                    |                          |
  RouteTableBuilder --> RouteTable --> store()                  match_request()
     (pure function)      (immutable)   (one pointer)           (borrows, no alloc)

Three properties follow, and they are the point of the design:

• A publish never blocks a reader. Writers and readers never share a lock, so a rebuild cannot add latency to a request.
• In-flight requests are unaffected. A request that loaded generation 7 holds that Arc and finishes against generation 7 even if 8 is published mid-flight. Nothing is rewritten under it.
• Load-balancer state survives the swap. Round-robin cursors and in-flight counts are carried forward by identity, not by position, so adding one Ingress does not make every backend forget how many requests it is currently serving.

The numbers, including the ones we lose

This project’s benchmarks are written to be checkable, and that means reporting the losses with the wins. Performance has the full method, the caveats, and the raw-data paths. The headlines:

The last three rows are not a disclaimer at the bottom of a marketing page. ingress-nginx does not reload for every change — endpoint updates go through its Lua balancer without touching nginx at all — and a report that measured only the changes which force a reload would be describing a system that does not exist.

Two engines

The data plane is selected with --engine, and everything above it — routing, load balancing, canaries, header rewriting, /metrics — is the same code either way.

uring exists to answer one question. Profiling measured where a request goes and found no hot function to fix: 59.4% of a request is the four syscalls a proxy hop cannot avoid, another 9.1% is finding out a socket is ready, and everything this project wrote is about 1%. Getting under that floor is not a tuning exercise, it is an I/O model change — so there is a second data plane that submits those four operations into a ring and enters the kernel once for a batch of them.

Everything it refuses, it refuses with a status code and an explanation naming the other engine, and it prints the same list at startup. A gap that behaves like a bug in whatever is on the other end is worse than a missing feature.

Where to go next

• Quick start — the data plane on a laptop in 60 seconds, then a cluster.
• Deployment — one command per cloud, and the question that decides most of the configuration: where the client’s IP address comes from.
• Annotations reference and Flags reference — every key and every option, verified against the source.
• Limitations — read this before you deploy. The line that matters operationally: there is no leader election yet, so run one replica.

Quick start

Two paths. The first needs no cluster and takes about a minute; the second is the real thing.

Without a cluster, in 60 seconds

--static-routes swaps the API server for a YAML file and changes nothing else about the serving path, which makes it the fastest way to look at the data plane on its own.

Start two throwaway upstreams that say which one they are:

for u in a b; do
    mkdir -p /tmp/ramjet-$u/api /tmp/ramjet-$u/healthz
    echo "upstream-$u /"        > /tmp/ramjet-$u/index.html
    echo "upstream-$u /api"     > /tmp/ramjet-$u/api/index.html
    echo "upstream-$u /healthz" > /tmp/ramjet-$u/healthz/index.html
done
(cd /tmp/ramjet-a && python3 -m http.server 9001) &
(cd /tmp/ramjet-b && python3 -m http.server 9002) &

Then run the daemon against the example route table shipped in the repository:

cargo run -p ramjet-ingressd -- \
    --static-routes crates/ramjet-ingressd/examples/dev-routes.yaml

ramjet-ingressd 0.1.0 — 5 backend(s), 6 endpoint(s), 5 route(s), 0 certificate(s), default backend set
  config   crates/ramjet-ingressd/examples/dev-routes.yaml
  http     0.0.0.0:8080
  https    disabled
  http3    disabled
  admin    0.0.0.0:10254
  probes   http://0.0.0.0:10254/healthz  http://0.0.0.0:10254/readyz  http://0.0.0.0:10254/metrics
  admin    http://0.0.0.0:10254/admin/generations  http://0.0.0.0:10254/admin/routes
 INFO audit: 5 routes added, 3 hosts added, 1 mirror added, default backend now fallback (gen 0→0)

https is disabled because this file declares no certificates and no explicit --https was given. The audit line is the same record every publish gets in Kubernetes mode — dev mode has exactly one generation and nothing is special-cased for it.

And drive it:

curl -H 'Host: shop.example.com' http://127.0.0.1:8080/          # upstream-a
curl -H 'Host: shop.example.com' http://127.0.0.1:8080/api/      # leastConn
curl -H 'Host: sub.example.com'  http://127.0.0.1:8080/          # wildcard
curl -H 'Host: anything.else'    http://127.0.0.1:8080/          # default backend

# `always` forces the canary regardless of the weight; `never` keeps it away.
curl -H 'Host: shop.example.com' -H 'x-canary: always' \
     http://127.0.0.1:8080/api/                                  # upstream-b

curl http://127.0.0.1:10254/readyz
curl http://127.0.0.1:10254/metrics

Ctrl-C (or SIGTERM) drains in-flight requests and exits.

Listeners default to :8080 plaintext, :8443 TLS, and :10254 admin. In dev mode, without an explicit --https or --no-https, the TLS listener is skipped when the configuration declares no certificates — a listener that fails every handshake is not a useful default.

The route file

crates/ramjet-ingressd/examples/dev-routes.yaml is annotated end to end. The shape:

# Answers any request that matches no rule at all. Without this, those are 404s.
defaultBackend: fallback

backends:
  - name: web
    policy: roundRobin           # roundRobin | leastConn | random
    endpoints:
      - 127.0.0.1:9001           # short form means weight 1

  - name: api
    policy: leastConn
    endpoints:
      - 127.0.0.1:9001
      - address: 127.0.0.1:9002  # long form, for a bigger pod
        weight: 2                # or for draining one with `weight: 0`

routes:
  - host: shop.example.com
    path: /healthz
    pathType: Exact              # Exact | Prefix | ImplementationSpecific
    backend: web

  - host: shop.example.com
    path: /api
    pathType: Prefix
    backend: api
    canary:
      backend: api-next
      weight: 20
      header: x-canary

  - host: "*.example.com"        # replaces exactly one label
    path: /
    pathType: Prefix
    backend: web

  - path: /status                # no host: every name not claimed above
    pathType: Prefix
    backend: web

# tls:
#   - host: shop.example.com
#     cert: /tmp/dev-cert.pem
#     key: /tmp/dev-key.pem

This is not a production configuration format, and nothing else in the tree parses YAML. The Kubernetes path builds tables from API objects directly; it does not render configuration and read it back, which is exactly the round trip that makes ingress-nginx’s behaviour hard to predict from its inputs.

The two modes are mutually exclusive by nature — a file and an API server are two writers for one route table, and letting both write would make the winner a race.

On a cluster, with Helm

helm install ramjet deploy/chart/ramjet-ingress \
  --namespace ramjet-system --create-namespace

That is a hostNetwork DaemonSet serving :80 and :443 on every node — the shape that works on a cluster with nothing underneath it. It also installs a ServiceAccount, a ClusterRole and binding, a ClusterIP Service, a separate ClusterIP Service for the admin port, and an IngressClass named ramjet whose controller is ramjet.dev/ingress. On a cloud, add the preset for your provider (see Deployment), which goes back to a Deployment on 8080/8443 behind a LoadBalancer Service.

Point workloads at it with ingressClassName: ramjet:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: shop
spec:
  ingressClassName: ramjet
  rules:
    - host: shop.example.com
      http:
        paths:
          - path: /api
            pathType: Prefix
            backend:
              service:
                name: api
                port:
                  number: 80

Or set ingressClass.isDefaultClass=true to catch Ingresses that name no class at all.

Deployment has a values preset and a rendered, Helm-free manifest for each supported provider, and answers the question that decides most of that configuration: where the client’s IP address comes from.

Before you scale it

replicas: 1 is hard-coded in the chart, and there is no values entry to find at 3am. There is no leader election yet: status writeback reads a Service’s address and server-side-applies it to every managed Ingress, and a second replica would do the same work against the same objects with the same field manager on its own schedule. Scale by making the one replica bigger, or use --no-status-update if you must run more. See Limitations.

Readiness, before you debug a slow rollout

/readyz returns 503 until a route table has actually been compiled from the API server. A replica that has finished starting but not finished its first list is deliberately kept out of the Service, because an empty table would 404 everything sent to it. /healthz, which the liveness probe uses, answers as soon as the process does.

Watching it live

The admin port reports counters, and the question you usually have is about rates. ramjet-top polls /admin/routes, /admin/generations and /metrics, differences the counters, and draws them.

cargo run -p ramjet-top                 # the local admin port
ramjet-top 10.0.0.5:10254               # somewhere else
ramjet-top --once                       # one aligned table, for scripts and CI

See Observability for the keybindings and what the numbers mean.

Deployment

One command per cloud. Every provider below has a Helm values preset and a rendered manifest that needs no Helm at all.

# Helm, with the preset for your provider
helm install ramjet deploy/chart/ramjet-ingress \
  --namespace ramjet-ingress --create-namespace \
  -f deploy/provider/aws/values.yaml

# or the rendered equivalent, no Helm required
kubectl apply -f deploy/static/provider/aws.yaml

Both install the same thing: a DaemonSet (or Deployment), a ServiceAccount, a ClusterRole and binding, a Service for traffic, a separate ClusterIP Service for the admin port, and an IngressClass named ramjet. Point workloads at it with ingressClassName: ramjet.

The static manifests are generated from the presets. Edit the preset, not the manifest — see Regenerating the static manifests.

The default install

No provider preset:

helm install ramjet deploy/chart/ramjet-ingress \
  --namespace ramjet-ingress --create-namespace

That is a hostNetwork DaemonSet serving :80 and :443 on every node. There is no load balancer, no NodePort translation and no address to wait for: point a DNS record at the nodes and the thing is reachable, with the client’s own address on the socket.

It is the default because it is the shape that works on a cluster with nothing underneath it, which is the cluster most people are holding the first time they install an ingress controller. On a cloud, use the preset for your provider — each one turns this off and goes back to a Deployment behind a LoadBalancer Service on 8080/8443, which is where the balancer is what makes the pod reachable.

Three things follow from it, all covered under bare metal below: the node’s ports have to be free, binding them takes a capability the image carries and the chart enables, and Ingress status has no address to read unless you set controller.publishAddress.

Providers

Each preset lives at deploy/provider/<name>/values.yaml and carries the reasoning for every line in it. Read the one you are about to use — several of them turn on behaviour that is unsafe if the other half is missing.

Where the client’s IP address comes from

This is the question that decides most of the configuration above, and getting it wrong is quiet: X-Forwarded-For still gets written, it just contains the load balancer’s address instead of the client’s, and nothing anywhere reports an error.

There are three mechanisms that preserve it, and two shapes that do not need one.

externalTrafficPolicy: Local, and its health check

Every cloud preset here sets Local, and it is worth knowing what that trades.

Cluster (the Kubernetes default) lets any node accept the packet and forward it to a pod on another node. That second hop is a SNAT, so the pod sees the first node’s address. Local removes the hop: only a node that already has a ready pod will serve the packet, and a node without one drops it silently.

What makes that safe is the healthCheckNodePort Kubernetes allocates whenever a LoadBalancer Service is Local. Every node serves /healthz on it, and it answers 200 only where a ready pod actually is. The cloud balancer checks it and stops sending to the silent nodes. Note that this is kube-proxy’s /healthz on a port of its own — it is not this data plane’s, and not the admin port.

The trap is a shape with no balancer doing that check. On bare metal with DNS round-robin or a hand-written upstream list, Local means traffic keeps going to nodes that quietly answer nothing. That is why baremetal-nodeport leaves it at Cluster and says so.

PROXY protocol

Where the balancer supports it, PROXY protocol is the most direct answer: the balancer prepends a header naming the real client, and the listener reads the address out of it.

It comes as a pair, and both halves must be set together:

proxyProtocol:
  enabled: true          # the listeners require the header
service:
  annotations:
    <provider annotation that makes the balancer send it>

Neither half is useful alone. Without the annotation, every connection is rejected for missing a header nothing is sending. Without the value, the daemon reads PROXY TCP4 … as an HTTP request line.

Three things that follow from it, all of which have bitten someone:

The listener has no mixed mode. Once it requires the header, a connection arriving without one is refused. That includes anything inside the cluster dialing the Service or the pod directly.

Which is why the hostname workaround exists. kube-proxy adds the balancer’s external IP to node-local iptables rules, so a pod connecting to that IP is short-circuited straight to a backend and never traverses the balancer — arriving with no header at a listener that demands one. In-cluster clients break while everything from outside works. The fix is to make the Service status report a hostname rather than an IP, which stops the rule being installed: do-loadbalancer-hostname on DigitalOcean, scw-loadbalancer-use-hostname on Scaleway. Both are documented, commented out, in their presets.

A reachable proxy-protocol port is a spoofable one. The header names the client, so anything that can open a connection to that port can claim any address it likes. Only turn this on where the balancer is the sole path in.

The admin port is never covered by any of this: /healthz and /readyz come from the kubelet, which speaks no PROXY protocol, and requiring the header there would take the probes offline the moment the flag was set.

Either engine can sit behind one of these presets. The uring engine reads the header with the same parser, in the same place — ahead of the TLS record layer — and with the same required-not-optional answer, so a preset that sets --proxy-protocol and an --engine uring in controller.extraArgs work together.

Cloud health checks need the same care, and the answer differs by provider. AWS sends the PROXY header on health check connections too once the target group attribute is set — so aws-nlb-proxy deliberately keeps the default TCP check rather than aiming an HTTP check at the admin port, which would reject it and take every target unhealthy. DigitalOcean and Scaleway keep their default TCP checks for the same reason.

HTTP/3, and which load balancers can carry it

http3.enabled=true is experimental and off by default. It adds --http3, a UDP container port and a UDP Service port — both on the same number as https — and makes every HTTPS response carry alt-svc: h3=":<port>"; ma=86400.

That header is the whole mechanism, and it is also the whole constraint. A client that reads it retries the same authority over QUIC, so the port number it is already using for TCP has to answer UDP too, through every hop in front of this Service. Which is a per-provider question with mostly disappointing answers:

Getting it wrong is slow rather than broken. A client whose QUIC attempt fails falls back to TCP by itself — the cost is one wasted attempt per connection until the advertisement expires, which is why ma is a day and not a week.

The presets are deliberately unchanged: none of them turns this on, because whether UDP reaches the pod is a property of an account’s networking rather than of a provider. Turn it on with --set http3.enabled=true on top of a preset once you have checked that it does.

Two more things worth knowing before enabling it in production:

• The PROXY protocol does not apply. It is a preamble on a TCP byte stream and has no UDP form, so a QUIC connection’s client address is whatever the IP header says. On a balancer that forwards UDP without rewriting the source that is the real client; on one that SNATs it, X-Forwarded-For on HTTP/3 requests will name the balancer while the TCP path is still correct. There is no configuration that fixes the difference.
• It is one core. The QUIC endpoint runs on a single dedicated runtime rather than one per core, for the reason set out in crates/ramjet-proxy/src/http3.rs: sharding a UDP port across sockets with SO_REUSEPORT hashes by 4-tuple, and a QUIC connection is deliberately not identified by its 4-tuple. HTTP/1.1 and HTTP/2 keep every core they had.

See HTTP/3 for the protocol-side detail.

Bare metal

The default shape and its fallback. The choice is about which ports you need and whether the node’s :80 is yours to take.

baremetal-nodeport — a NodePort Service pinned to 30080 and 30443. Works on any cluster, needs no load balancer controller, and is the right thing for testing. It cannot serve :80, and the client address is lost to SNAT unless something in front is health-checking the nodes (see externalTrafficPolicy above). The ports are fixed rather than allocated so that firewall rules and upstream configs naming them do not go stale when the Service is recreated.

baremetal-hostnetwork — a DaemonSet on the host network, binding :80 and

443 on every node. No translation, no balancer, and the socket already carries the client’s address. This is the chart’s default, so a helm install with no values file at all gets it; the preset exists to say so explicitly and to be the source of the rendered manifest. The ports belong to the node, so a second copy on the same node cannot start — which is why it is a DaemonSet — and the admin port is now on every node’s external interface, with the node firewall as the only thing keeping it private.

Binding :80

This is the part that is genuinely surprising, and it was this chart’s own bug until a stock node found it.

The process runs as uid 65532, and a port below 1024 needs CAP_NET_BIND_SERVICE in its effective set. Adding the capability in the securityContext is not enough on its own: Kubernetes puts it in the container’s permitted and bounding sets, and the kernel raises a capability into a non-root process’s effective set only from a file capability on the binary. So the obvious configuration is the one that cannot work, and it fails as Permission denied (os error 13) in a CrashLoopBackOff.

The fix is in the image. Its binary carries cap_net_bind_service=+ep, set by a setcap in the Dockerfile’s builder stage and carried into the runtime stage in the layer’s security.capability extended attribute. That is the same mechanism ingress-nginx uses on its nginx binary.

allowPrivilegeEscalation stays false. This is worth stating because the opposite is widely assumed: no_new_privs does not discard a file capability, it downgrades the new permitted set by intersecting it with the one the process already had — and capabilities.add has already put NET_BIND_SERVICE there. Verified on a stock node rather than reasoned about. Nothing about this shape is more privileged than a release on 8080.

The obligation it creates

A file capability with the effective bit set cannot be exec’d at all when that capability is outside the container’s bounding set. The kernel refuses with EPERM — “insufficient to execute correctly” — and the kubelet reports:

exec /usr/local/bin/ramjet-ingressd: operation not permitted

That happens before a line of the program runs, and it does not care what port anything was going to bind. So every pod spec for this image has to keep NET_BIND_SERVICE in securityContext.capabilities.add, on 8080 exactly as much as on 80. The chart does that unconditionally; a hand-written manifest that drops ALL and adds nothing back will not start.

The PodSecurity restricted profile permits exactly this one addition on top of a dropped ALL, so nothing here costs a profile.

Checking that an image you have pulled really carries it (distroless has no shell, so the check happens somewhere that does):

IMAGE=sofelia/ramjet-ingress:0.1.0
docker build -q - <<EOF >/dev/null && echo "capability present in $IMAGE"
FROM debian:bookworm-slim
RUN apt-get update && apt-get install -y --no-install-recommends libcap2-bin
COPY --from=$IMAGE /usr/local/bin/ramjet-ingressd /d
RUN getcap /d | grep -q cap_net_bind_service
EOF

The capability is only in images built from the commit that added the setcap, so a tag older than that fails this check — which is also what a pod running one will tell you in its log.

Two ways out for a cluster that will not run this image at all:

• sysctl -w net.ipv4.ip_unprivileged_port_start=80 on every node, which makes

  80 unprivileged for everything on that node. It has to be set on the node — a host-network pod is in the node’s own network namespace and the kubelet refuses namespaced net.* sysctls for it.

• baremetal-nodeport, or a balancer in front, and leave the listeners above 1024.

A pod that fails to bind says all of this in its log, naming the port, the uid and both remedies — kubectl logs it rather than working back from os error 13.

One reason this went unnoticed for so long: containerd sets net.ipv4.ip_unprivileged_port_start=0 inside every pod sandbox it creates (enable_unprivileged_ports, on by default), so a pod on the pod network binds :80 whatever its capabilities say. hostNetwork is the shape that has no sandbox netns of its own, and therefore the only one where the node’s own setting — 1024 on a stock node — applies.

MetalLB is the third option and often the best one where a real address matters more than the node’s ports: put it in front and install with --set service.type=LoadBalancer --set kind=Deployment --set hostNetwork=false --set ports.http=8080 --set ports.https=8443, which is the cloud shape without a cloud. MetalLB assigns an address out of an IPAddressPool, the Service behaves like a cloud one, and Ingress status writeback works because there is finally an address to publish. https://metallb.universe.tf/

Neither bare-metal preset can populate Ingress status on its own — and neither can the default install, for the same reason: there is no LoadBalancer Service with an address to read. Set controller.publishAddress to whatever clients actually use. Routing is unaffected either way; the status field is advertising, not configuration.

What the chart does not let you configure

Two things, deliberately.

replicas: 1 is hard-coded. There is no leader election yet. Status writeback reads a Service’s address and server-side-applies it to every managed Ingress, and a second replica would do the same work against the same objects with the same field manager on its own schedule. The fix is leader election in the controller, not a values entry — so there is no values entry to find at 3am.

The admin port is on its own ClusterIP Service. /metrics and the probes are never attached to the internet-facing LoadBalancer, and because the split is two objects rather than a list of ports, no value can accidentally publish them.

Chart values

The defaults, as deploy/chart/ramjet-ingress/values.yaml ships them. Every controller.* entry maps to a flag on the flags reference.

image:
  repository: sofelia/ramjet-ingress
  tag: ""                        # defaults to .Chart.AppVersion
  pullPolicy: IfNotPresent
  pullSecrets: []

kind: DaemonSet                  # or Deployment

controller:
  ingressClass: ramjet
  watchNamespace: ""             # "" is every namespace
  updateStatus: true
  publishService: ""             # defaults to "<release-namespace>/<fullname>"
  publishAddress: ""
  defaultBackend: ""
  defaultTlsSecret: ""
  connectTimeout: 5
  responseTimeout: 60
  maxConnectAttempts: 3
  shutdownGrace: 30
  historySize: 10
  auditWebhook: ""
  extraArgs: []
  logLevel: "info,kube=warn"
  extraEnv: []

proxyProtocol:
  enabled: false
http3:
  enabled: false

ports:
  http: 80                       # cloud presets set 8080
  https: 443                     # cloud presets set 8443
  admin: 10254

hostNetwork: true                # cloud presets set false

service:
  type: ClusterIP                # cloud presets set LoadBalancer
  annotations: {}
  externalTrafficPolicy: ""
  http:  { port: 80,  nodePort: "", targetPort: http }
  https: { port: 443, nodePort: "", targetPort: https }

adminService:
  enabled: true
  port: 10254

ingressClass:
  create: true
  name: ""                       # defaults to controller.ingressClass
  isDefaultClass: false

resources:
  requests: { cpu: 100m, memory: 64Mi }
  limits:   { memory: 256Mi }

terminationGracePeriodSeconds: 45

podSecurityContext:
  runAsNonRoot: true
  runAsUser: 65532
  runAsGroup: 65532
  fsGroup: 65532
  seccompProfile: { type: RuntimeDefault }

# NET_BIND_SERVICE is not optional: the image's binary carries a file capability
# and cannot be exec'd at all unless that capability is in the container's
# bounding set — on 8080 as much as on 80. See [Binding :80](#binding-80).
securityContext:
  allowPrivilegeEscalation: false
  readOnlyRootFilesystem: true
  capabilities: { drop: [ALL], add: [NET_BIND_SERVICE] }

metrics:
  scrapeAnnotations: true

The memory limit is load-bearing

resources.limits.memory: 256Mi is not an arbitrary default. An idle keep-alive connection costs this data plane about 20 KiB, so 256Mi is roughly twelve thousand of them. That arithmetic is in the values file rather than left to be rediscovered, and the number comes from a benchmark that originally found the opposite: at 10,000 idle connections the process peaked at 266 MiB and would have been OOM-killed by its own default manifest. That is fixed — the peak is now 200.7 MiB and the memory comes back on close — but the per-connection cost is still 4.6x nginx’s, and raising the default to make room for it would have hidden the finding rather than fixed it. See Performance.

--publish-service and Ingress status

--publish-service defaults to the chart’s own Service. Under a cloud preset that Service is a LoadBalancer, and reading its address is the mechanism by which an Ingress in an unrelated namespace ends up advertising the address traffic really arrives on.

On the default hostNetwork shape there is no such address — the Service is a ClusterIP and traffic does not go through it at all — so set controller.publishAddress to whatever clients actually use, or the ADDRESS column stays empty. Routing is unaffected either way.

RBAC

Exactly the six rules the controller’s five watches and its status writer need. It creates nothing, deletes nothing, and holds no write verb outside the ingresses/status subresource — with two additions that are opt-in behaviour rather than routing:

• events.k8s.io / events / create, patch — for the audit trail’s Kubernetes Events. Without it the Events are skipped at debug level and everything else still works.
• networking.k8s.io / ingresses / patch — for canary auto-promotion, the only write this controller makes to an object an operator authored. Without it, promotion logs a permission error every interval and changes nothing.

Regenerating the static manifests

deploy/static/provider/*.yaml are generated. Edit the preset and re-render:

deploy/render.sh            # regenerate all of them
deploy/render.sh --check    # fail if the committed files are stale

--check is for CI. Without it, a chart change that nobody re-rendered ships a manifest that no longer matches the chart it claims to come from.

Validating a change

deploy/cloud-e2e.sh

Four passes: helm lint against every preset, every static manifest through kubectl apply --dry-run=server, a full install-and-route test of baremetal-nodeport against a local Docker Desktop cluster, and a PROXY protocol test that reinstalls with proxyProtocol.enabled=true and speaks a hand-built v1 header onto the socket. That last one asserts both halves: that the spoofed address in the header reaches the backend as X-Forwarded-For, and that a request arriving without a header is refused — the second being the one that matters, since a listener accepting both shapes would let any client that can reach it claim any address it likes.

The cloud presets are only ever dry-run, and cannot be more than that here — an aws-load-balancer-type annotation means nothing without the AWS Load Balancer Controller watching. What the dry-run does prove is the half that actually breaks: that every object is schema-valid and the API server accepts it, which is where a typo’d annotation key or a malformed port block shows up.

Every kubectl and helm call carries an explicit --context, and the preflight refuses any cluster that does not look local. A developer kubeconfig usually holds production clusters, and a mistyped current-context is exactly how a test script deletes one.

One local-cluster detail worth knowing: Docker Desktop does not publish NodePorts on the host, so the traffic assertions run inside the node container against the node’s own address. That is the more faithful test anyway — it is the real NodePort path, where a port-forward would have proven only that the pod serves.

One thing this cannot prove, and it is the reason the default shape is verified by hand on a real node as well: Docker Desktop’s node ships net.ipv4.ip_unprivileged_port_start=0, so :80 binds for any uid there. The privileged-bind failure the file capability exists to prevent is invisible on this cluster — a chart that could not bind :80 anywhere else would pass here. See Binding :80.

End-to-end proof

deploy/e2e.sh builds the image, installs the chart on a local Docker Desktop cluster, deploys a pair of echo backends behind a production Ingress and a canary Ingress, and asserts host routing, 404 on an unknown host, the canary weight split, TLS with SNI, Ingress status writeback, and metrics movement. It tears everything down afterwards; KEEP=1 leaves it standing.

deploy/e2e.sh

One local-cluster detail the script handles in place: Docker Desktop’s Kubernetes runs a kind-style node whose containerd is a separate image store from the docker daemon’s, so a freshly built image is invisible to the kubelet and a pod referencing it fails with ErrImageNeverPull. The script loads it explicitly, which is what kind load docker-image does under the hood:

docker save ramjet-ingress:e2e | docker exec -i desktop-control-plane ctr -n k8s.io images import -

The node is addressable as desktop-control-plane even though docker ps does not list it — Docker Desktop hides the container from the listing while still allowing exec, so an empty docker ps is not evidence that there is no node to load into. Importing rather than pulling is what lets imagePullPolicy stay Never: the assertions then prove the image this script built is the one that ran, with no path by which the kubelet could quietly substitute a registry copy.

The container image

sofelia/ramjet-ingress on Docker Hub, public, and what the chart installs by default — the quick starts at the top of this page pull it and need no local build.

It is a manifest list covering linux/amd64 and linux/arm64, so the pull resolves to the node’s architecture on its own.

.github/workflows/images.yml publishes them on every push to main and every v* tag. Each architecture builds on a runner of that architecture — a Rust release build with LTO under QEMU runs past the job timeout, so emulation is not a slower version of the same thing but a broken one — and each pushes by digest into the registry untagged. A final job creates the manifest list over both digests, which is the only point at which any tag above starts resolving. The workflow needs DOCKERHUB_USERNAME and DOCKERHUB_TOKEN as repository secrets; see deploy/README.md.

How it is built

Multi-stage: a full Rust toolchain compiles, and gcr.io/distroless/cc-debian12:nonroot carries the result. The runtime has no shell and no package manager, and the process runs as uid 65532. TLS is rustls over ring, so the image needs no OpenSSL and no CA bundle.

The build context is the parent directory, because ramjet-engine depends on the ramjet runtime from a sibling repository by path — the Dockerfile copies both this tree and the enhance-socket sibling, and a context rooted here cannot see the second one:

docker build -f Dockerfile -t ramjet-ingress:0.1.0 ..

The builder uses BuildKit cache mounts for Cargo’s registry and the target directory rather than the usual “build dummy sources first” trick, which for a four-crate workspace would mean maintaining four fabricated source files that mirror the real layout. The tradeoff: the caches live in the builder, not in the image, so the binary is copied out inside the same RUN.

Configuration

There are three places configuration comes from, and they answer different questions.

The dividing line is deliberate: an annotation is a per-route decision made by whoever owns the workload, a flag is a per-replica decision made by whoever owns the ingress controller.

What is deliberately not configurable

The annotation vocabulary is canary, mirroring, auto-promotion, and class. RouteTable has no rewrite, header-mutation, rate-limit, session-affinity, or auth rules, so the corresponding nginx.ingress.kubernetes.io annotations are not read. Those attach to a route when the proxy can act on them; parsing an annotation the data plane ignores is worse than not parsing it, because it looks configured.

If you are migrating from ingress-nginx, Limitations is the page that tells you what will not come across.

Ingress basics

The resource is networking.k8s.io/v1 Ingress, and the semantics follow ingress-nginx wherever there is a choice to be made — including the parts that look like historical accidents, because deviating from them would silently move traffic during a migration.

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: shop
  namespace: prod
spec:
  ingressClassName: ramjet
  rules:
    - host: shop.example.com
      http:
        paths:
          - path: /api
            pathType: Prefix
            backend:
              service:
                name: api
                port:
                  number: 80

Which Ingresses this controller claims

Getting this wrong in either direction is a production incident: claim too much and you fight another controller for the same hostnames, claim too little and traffic silently 404s.

The check runs in this order:

1. kubernetes.io/ingress.class on the Ingress. If present, it is decisive — even if spec.ingressClassName says something else. Ours if the value equals --ingress-class (default ramjet), another controller’s otherwise. This is not the order the Ingress API documents, but it is what ingress-nginx does, and an object that sets both is asking a compatibility question rather than a spec-compliance one.
2. spec.ingressClassName naming an IngressClass whose spec.controller is ramjet.dev/ingress → ours.
3. spec.ingressClassName naming an IngressClass that exists but belongs to someone else → not ours, silently.
4. spec.ingressClassName naming a class that does not exist at all → not ours, and logged. A dangling name is a typo or a missing manifest; either way the Ingress serves nothing and nobody would otherwise be told why.
5. No class named at all → ours only if one of our IngressClass objects carries ingressclass.kubernetes.io/is-default-class: "true". The chart sets that from ingressClass.isDefaultClass, default false.

Because an Ingress naming another class is invisible to this replica, running alongside ingress-nginx during a migration is safe.

Path types

Kubernetes defines three, and Prefix is the one implementations get wrong. It is not a string prefix; it matches whole path elements.

An unrecognised pathType is treated as ImplementationSpecific and warned about, rather than rejected.

Prefix paths are normalized at build time into a match length: trailing slashes are stripped, and the root prefix / becomes a length of zero — which is what makes / match everything without a special case.

Regex paths

ImplementationSpecific compiles the path as a case-insensitive regex, with two deliberate divergences from ingress-nginx:

• Anchoring. ingress-nginx emits location ~* "^<path>", a literal concatenation. This compiles ^(?:<path>). The two differ only for a top-level alternation, where ^a|b anchors just the first branch and routes traffic nobody intended.
• Size. Compiled regexes are limited to 1 MiB. A pathological path should fail validation, not silently consume memory in every replica.

Hosts, wildcards, and precedence

A wildcard host replaces exactly one left-most label, which is the Kubernetes rule:

*.example.com  matches  foo.example.com
               not      foo.bar.example.com
               not      example.com

Host validation is strict: a host containing a port, a path, or a misplaced * is rejected at build time rather than normalized into a guess.

Host selection happens first, then path matching within the selected host. This is nginx’s server-then-location order.

1. exact host        ─┐
2. wildcard host      ├─ pick exactly one virtual host
3. hostless catch-all ─┘
4. default backend    ── if none of the above claimed the request

within the chosen host:
   Exact  >  longest Prefix  >  regex in controller order

The subtle one: a request whose host matches exactly but whose path matches nothing falls to the default backend. It does not reconsider the wildcard.

A rule with no host at all serves every name not claimed by an exact or wildcard entry.

The default backend

Requests matching no rule are 404s unless a default backend is set. There are two ways to set one, and the flag is the replica-wide answer:

--default-backend kube-system/notfound:8080

as namespace/name:port. A malformed value is rejected at startup rather than at the first unmatched request. An Ingress may also carry spec.defaultBackend.

Backends and load balancing

A backend is a Service, resolved through its EndpointSlices to a list of addresses. Three policies exist:

Non-uniform weights are expanded once per generation into a precomputed rotation, interleaved so consecutive requests spread across endpoints rather than bursting weight of them at one. Uniform weights skip it entirely. A live endpoint is never rounded down to zero.

An empty endpoint list is not a build error. A Service whose pods are all unready is normal during a rollout, and failing the whole table for it would turn one bad Deployment into a cluster-wide outage. Selection yields nothing and the proxy answers 503.

Counters survive a rebuild

Round-robin cursors and in-flight counts are not stored in the route table. They live behind shared references that successive tables carry forward by identity — backend name for cursors, socket address for in-flight counts — so adding one Ingress does not make every backend forget how many requests it is currently serving. A request that started under generation 7 and finishes under generation 8 decrements the same counter it incremented.

That is the same class of bug as an nginx reload, just quieter, and it is why the mechanism exists.

When one object is broken

Rebuilds are total: the controller compiles the current state of every watched object, not the event that woke it. One malformed Ingress, one dangling Secret, or one unresolvable Service degrades that route and nothing else, and comes back as a structured warning.

The alternative — refusing to build a table containing one broken object — hands every namespace owner a cluster-wide kill switch.

Warnings worth alerting on: a rejected Ingress, an unresolvable Service, a Secret that will not parse. They go to stderr through tracing, filtered with RUST_LOG.

How fast a change lands

Five watches (Ingress, IngressClass, Service, EndpointSlice, Secret) funnel into one rebuild task with a 200 ms debounce. A fifty-pod rollout produces fifty EndpointSlice events and at most one rebuild, and each rebuild is built from everything known at that instant — so a burst of churn costs what a single change costs.

A publish is suppressed when the compiled digest matches what is already serving. The API server re-sends every object on each watch restart and periodic resync, and without that check each of those would bump the generation and hand the data plane a table it already has.

Measured end to end — kubectl apply to the first request the data plane answers correctly — that is a median of 363 ms on an empty cluster and 507 ms with 500 routes already loaded. See Performance.

Status writeback

By default the controller writes the ingress address into every managed Ingress’s .status.loadBalancer. The address comes from --publish-service (a Service whose own status supplies it, which the chart points at itself) or --publish-address (a literal). --no-status-update turns it off entirely.

Status is advertising, not configuration: routing is unaffected either way.

This is the part that does not tolerate a second replica. See Limitations.

Unsupported backend shapes

Two Service shapes are compiled but answer an error rather than routing, each for a reason rather than a TODO:

• ExternalName Services serve 503. Following a DNS name from the data plane needs a resolver with TTL handling and re-resolution; pointing at whatever the name resolved to at compile time would be a stale-address bug waiting for the first failover.
• A gRPC Service answers 502 until you annotate it. gRPC has no HTTP/1.1 form, and a Service is dialled over HTTP/1.1 unless it says otherwise, so a request with an application/grpc content type is rejected explicitly rather than downgraded into something the backend cannot parse. The 502 names the fix: backend-protocol: GRPC on the Ingress, after which the Service is dialled over h2c and gRPC works end to end.

TLS

TLS is terminated on the --https listener (default 0.0.0.0:8443) by rustls over ring. There is no OpenSSL in the image and no CA bundle.

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: shop
  namespace: prod
spec:
  ingressClassName: ramjet
  tls:
    - hosts:
        - shop.example.com
      secretName: shop-tls        # a kubernetes.io/tls Secret in `prod`
  rules:
    - host: shop.example.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: web
                port:
                  number: 80

The Secret is looked up in the Ingress’s own namespace. There is no cross-namespace reference.

Both engines terminate TLS, and through the same resolver over the same certificate store — so a name resolves to the same certificate, and a rotation reaches both, whichever --engine is serving. Everything below is therefore about this proxy rather than about one lane of it.

SNI resolution

A server name resolves to a certificate using exactly the same precedence as host routing — exact name, then single-label wildcard, then the default certificate. A handshake that picked a different certificate than the request would later be routed by is a confusing way to fail.

1. exact name          shop.example.com
2. wildcard parent     *.example.com
3. --default-tls-secret

The default certificate

A handshake whose SNI matches nothing gets the default certificate, if there is one:

--default-tls-secret ingress/wildcard      # namespace/name

or controller.defaultTlsSecret in the chart. Without it, such a handshake fails.

This is also the supported way to serve a certificate that covers names the Ingress does not list, because:

An IngressTLS entry with no hosts is skipped, with a warning. The controller cannot read a certificate’s SANs to work out which names it covers — that would mean parsing X.509 in the control plane, which is exactly the dependency the layering split exists to avoid.

An entry with no secretName is skipped the same way.

Two Ingresses claiming one host

The older Ingress keeps the host, and the newer one gets a warning naming the holder. Nothing is silently overwritten and nothing is refused.

A certificate that will not parse

Logged and skipped, never fatal. TLS for the names it covers fails until the Secret is fixed; every other host, and all plaintext traffic, is untouched. Refusing the whole generation would let one malformed Secret in one namespace take the cluster’s routing offline.

The same is true one level up: a Secret that cannot be read at all produces prod/shop-tls: <reason>; serving these hosts without it and the rest of the table compiles.

Rotation costs nothing it does not have to

handle_id is derived from the Secret’s namespace, name, and contents, so it changes if and only if the material changes. The daemon keeps its parsed keys in a map keyed by that id and carries forward every id that survives a rebuild, parsing only what actually rotated.

A cluster with 500 certificates does no X.509 work at all when an unrelated Ingress is edited.

The same property makes eviction safe: a key that no longer appears in a new generation is simply not carried over, and dropping it cannot orphan a name, because a name that still resolves still names its id.

Why a rotation never drops a handshake

A generation is applied in two stores, in this order:

1. the certificate store — the whole handle_id → key map at once;
2. the route table that references those ids.

Those are two independent atomic pointers, so a handshake can observe a new table against an older store. Publishing certificates first makes the only possible skew a store holding a key nothing points at yet, which is invisible. The other order leaves a name whose id is missing from the store, which rustls turns into a failed handshake — and every rotation would drop connections for the width of that gap.

The same order is used when a rollback republishes an old generation, for the same reason.

Dev mode

The static route file takes certificates as PEM paths:

tls:
  - host: shop.example.com
    cert: /tmp/dev-cert.pem
    key: /tmp/dev-key.pem
  # An entry with no host (or `host: "*"`) becomes the default certificate,
  # served when SNI matches nothing.

Generate a throwaway pair with:

openssl req -x509 -newkey rsa:2048 -nodes -days 365 \
    -keyout /tmp/dev-key.pem -out /tmp/dev-cert.pem \
    -subj '/CN=shop.example.com' \
    -addext 'subjectAltName=DNS:shop.example.com'

In dev mode, without an explicit --https or --no-https, the TLS listener is skipped when the file declares no certificates — a listener that fails every handshake is not a useful default. In Kubernetes mode it always binds: the certificates arrive over a watch, after the socket.

What TLS does not do here

• There is no TLS to the upstream. The upstream side speaks HTTP/1.1, or cleartext HTTP/2 for a backend annotated backend-protocol: GRPC — and both are cleartext, which is why GRPCS and HTTPS are reported and not honoured.
• HTTP/3 shares this listener’s certificates exactly — the same SNI resolution, the same store, the same rotation, reaching both transports at the same instant because it is the same two pointer stores in the same order. See HTTP/3.

Metrics

ramjet_tls_handshakes_total and ramjet_tls_handshake_failures_total. A failure rate that moves after a deploy is usually a Secret that did not parse or a name nothing covers.

Annotations reference

Every annotation this controller reads, and nothing else. Anything not on this page is ignored — see what is not here.

A value that is read and cannot be used is reported on the object itself, so finding out does not need pod-log access.

Two prefixes, and the rule for choosing

Anything ingress-nginx already spells gets the nginx.ingress.kubernetes.io prefix, on purpose. Compatibility is the whole point: an existing cluster should be able to swap controllers without rewriting every Ingress, so this controller speaks the annotations people already have — the canary family below is transcribed from theirs, semantics included.

Anything ingress-nginx has no equivalent for gets ramjet.dev. Traffic mirroring and canary auto-promotion are both in that group: there is no established spelling to be compatible with, and borrowing their prefix for a key they do not define would be a claim about portability that is not true. An operator reading ramjet.dev/… on an Ingress knows immediately that moving back to ingress-nginx loses that behaviour.

Class

The full claim order is in Ingress basics.

Canary

Transcribed from ingress-nginx, semantics included. Set these on the canary Ingress — a second Ingress with the same host and path as the production one.

Precedence, and what “beats” means

header > cookie > weight. The subtlety is that only the literal values always and never are decisive. A header that is present but says something else is ignored, and evaluation continues to the next rule. Getting this wrong makes every request carrying an unrelated header value bypass the weight split.

Parsing failures are not fatal

An unparseable or negative canary-weight is reported and read as 0. A fat-fingered weight should not take the Ingress out of service.

A canary with canary: "true" and nothing else is inert — weight 0, no header, no cookie — and is reported as such rather than compiled into a rule that can never fire. That is also true in ingress-nginx; it is just said out loud here.

metadata:
  annotations:
    nginx.ingress.kubernetes.io/canary: "true"
    nginx.ingress.kubernetes.io/canary-weight: "20"
    nginx.ingress.kubernetes.io/canary-by-header: x-canary

Backend protocol

How the data plane talks to the pods behind a Service. Set on the Ingress, and it applies to every backend that Ingress’s rules point at.

GRPC is what makes a gRPC Service work: gRPC is defined in terms of HTTP/2 streams and trailers and has no HTTP/1.1 form, so without this the request would be downgraded into something the backend cannot parse. With it, the whole exchange works — unary and streaming, in both directions, with grpc-status arriving in the trailers where the client expects it. The client may speak HTTP/1.1, HTTP/2, or HTTP/3; the version is translated at this hop.

Nothing about it is gRPC-specific. Any Service that speaks h2c — a plain HTTP/2 API, a service mesh sidecar — is reached correctly with the same value.

metadata:
  annotations:
    nginx.ingress.kubernetes.io/backend-protocol: GRPC

The four values ingress-nginx has that this does not

GRPCS, HTTPS, AUTO_HTTP and FCGI are read, reported, and not honoured. The backend stays on HTTP/1.1 and a warning names the value:

default/api [InvalidAnnotation]: `nginx.ingress.kubernetes.io/backend-protocol: GRPCS`
is not supported; only `HTTP` and `GRPC` are, and this backend stays on HTTP/1.1

GRPCS and HTTPS need TLS to the upstream, which this data plane does not do yet; AUTO_HTTP needs per-endpoint scheme detection; FCGI is not HTTP. Treating any of them as HTTP silently would send cleartext at a port expecting TLS, with nothing but connection resets to explain it. Refusing to compile the Ingress would be worse — one namespace owner could take the table out — so the route serves and the warning is the signal.

What an h2c backend sees

Two things differ from the HTTP/1.1 path, both forced by HTTP/2 itself:

• No Host header. HTTP/2 carries the authority in the :authority pseudo-header, and :authority has to name the endpoint because that is what keys the upstream connection pool. Sending a Host that disagrees with it is something RFC 9113 §8.3.1 lets a server treat as malformed. The client’s host name is in X-Forwarded-Host, on this path and the HTTP/1.1 one alike.
• No protocol upgrades. Connection and Upgrade are forbidden in HTTP/2, so a WebSocket handshake is not reconstructed for an h2c backend; it reaches the application as an ordinary request. WebSocket over HTTP/2 (RFC 8441 extended CONNECT) is not implemented. Put WebSocket routes on an HTTP backend.

One Service port is one backend

A backend is a Service port, however many Ingresses point at it, so two Ingresses cannot give the same pods two protocols. If they try, the first claim in route order wins and the other is reported:

default/b [BackendProtocolConflict]: backend default/web:80 is already registered
as `h2c` by another Ingress; this Ingress asked for `http` and was not honoured

Split the Service, or annotate both the same way.

Not on the uring engine

--engine uring dials HTTP/1.1 only. A route whose backend is GRPC answers 502 there, naming the engine, rather than being downgraded — see Engines.

Traffic mirroring

ramjet.dev prefix: there is no ingress-nginx spelling of this. Set these on the production Ingress. A mirror is a property of the route, and the canary Ingress is a second opinion about where a share of that route’s traffic goes — not a second route that could have its own shadow.

An out-of-range or unparseable mirror-percent (101, -5, lots, 50%) is reported and falls back to 100; it never disables the mirror.

Setting mirror-backend on a canary Ingress does nothing, and says so in a warning.

mirror-host looks cosmetic and is not: a shadow deployment usually answers to a different name, and a copy carrying the production Host can be routed by whatever sits in front of it — possibly straight back to production, which is the one outcome a mirror must never produce.

metadata:
  annotations:
    ramjet.dev/mirror-backend: shadow/api:80
    ramjet.dev/mirror-percent: "10"
    ramjet.dev/mirror-host: shadow.example.com

See Traffic mirroring for the invariants and the body cap.

Canary auto-promotion

ramjet.dev prefix. Set these on the canary Ingress. Everything but the opt-in has a default that is safe to run with.

Every bad value falls back and is reported

A misspelled threshold does not stop the promotion; it uses the default and logs which key was unusable. The alternative — refusing to promote because one threshold is misspelled — leaves a canary stuck at its starting weight with no explanation, which is a worse failure than promoting against a default somebody can see.

auto-promote-status is a one-way latch

A rollback writes both auto-promote: "false" and auto-promote-status: "rolled-back: <reason>", and the loop refuses any canary whose status starts with rolled-back even if the enable annotation is somehow still true. Both, because the guard has to survive a restart — the annotation carries it across a rescheduled pod — and because the two are written in one patch that could half-fail.

Re-arming is a human decision: clear the status annotation yourself.

metadata:
  annotations:
    nginx.ingress.kubernetes.io/canary: "true"
    nginx.ingress.kubernetes.io/canary-weight: "5"
    ramjet.dev/auto-promote: "true"
    # ramjet.dev/auto-promote-interval: 60s
    # ramjet.dev/auto-promote-steps: 5,10,25,50,100
    # ramjet.dev/auto-promote-max-5xx-percent: "1"
    # ramjet.dev/auto-promote-max-latency-factor: "1.5"
    # ramjet.dev/auto-promote-min-requests: "50"

See Canary auto-promotion for the state machine and the interlocks.

Written by the controller

Two keys go the other way — this controller writes them onto your Ingresses, and reads neither back.

observed-generation answers the question an operator has straight after an edit: did it land?

$ kubectl get ingress -o custom-columns=\
NAME:.metadata.name,GEN:'.metadata.annotations.ramjet\.dev/observed-generation'
NAME        GEN
web         57
shop        57
api         57

An Ingress stuck a generation behind its neighbours is one the controller compiled and then stopped including — nearly always because it was rejected, in which case the Events above say why.

It is not the generation being served. That is what /admin/routes reports, per replica, and the two differ exactly while a rollback pin is held: the annotation follows what the controller compiled, and the pin lives in one data plane’s memory where no control plane can see it. Two numbers that agree mean a replica is serving what the cluster describes.

The write is a merge patch under the ramjet-ingress field manager, sent only when the value on the object differs from the one being written — a steady cluster rebuilds on every watch event and sends nothing. The key is read by no parser here, so the controller’s own write cannot change a compiled digest and cannot cause the republish that would write it again. A merge patch rather than an apply because the same field manager also writes canary-weight, and an apply states everything a manager owns: each write would delete the other’s key.

--no-status-update switches this off along with the address writeback; it is the flag for “do not write to my Ingresses”. A stale value is left behind on an Ingress that moves to another controller, deliberately: clearing it would cost a write to an object we have just decided is not ours, at the moment somebody else is taking it over, to remove a diagnostic that stops claiming anything anyway. The address — the part everything downstream routes on — is cleared.

A refused value says so on the object

Every annotation above falls back rather than failing the Ingress — a fat-fingered weight must not take a route out of service. The cost of that is that a refused value goes on sitting there looking applied, so each one also becomes a Warning Event on the Ingress that carries it:

kubectl describe ingress web-canary

Events:
  Type     Reason             Age   From            Message
  ----     ------             ----  ----            -------
  Warning  InvalidAnnotation  2m    ramjet-ingress  `nginx.ingress.kubernetes.io/canary-weight` is not a number; using 0
  Warning  MirrorRejected     2m    ramjet-ingress  a canary Ingress cannot also mirror; the mirror is ignored

The Reason is the refusal’s kind, so it is filterable:

kubectl get events -A --field-selector reason=CanaryInert

Only these, and only when they change. Events are written when an object’s set of refusals differs from the last set written for it, not on every rebuild — a rebuild happens on every watch event in the cluster, and re-stating an unchanged complaint would be one Event per Ingress per deploy forever. Fixing one annotation and breaking another in the same edit is a change, so it is reported immediately; a cooldown would have swallowed it.

Warnings that are not about an annotation value stay in the log, where the person who can act on them already is: a Service with no endpoints, a TLS Secret that has not been created, a route another Ingress already claimed. So does EndpointsSkipped, which fires on every healthy rolling update and would train people to ignore the stream.

RBAC: events.k8s.io/events/create, which the chart’s ClusterRole has. Without it these are skipped at debug and the log lines are unaffected.

What is not here

The vocabulary above is the whole vocabulary. The route table has no rewrite, header-mutation, rate-limit, session-affinity, or auth rules, so the corresponding nginx.ingress.kubernetes.io annotations are not read — they are not silently accepted either, they are simply absent from the parser.

Those attach to a route when the proxy can act on them. Parsing an annotation the data plane ignores is worse than not parsing it, because it looks configured.

If you are migrating, this is the list to diff your Ingresses against.

Flags reference

Every option ramjet-ingressd accepts. --help prints the same list.

Every flag has an environment variable twin, because that is how a container is configured. The precedence is the usual one — an explicit flag beats the environment, which beats the default — and a flag always wins, so a kubectl edit of the args cannot be silently overridden by a ConfigMap somebody forgot about.

Both --flag value and --flag=value are accepted. Every option is a --flag; the binary takes no positional arguments.

RUST_LOG sets the log filter. The chart’s controller.logLevel defaults to info,kube=warn.

Mode

With no --static-routes, the daemon watches the Kubernetes API and serves what the controller compiles. With one, it serves that file and never talks to Kubernetes at all.

The two are mutually exclusive by nature: a file and an API server are two writers for one route table, and letting both write would make the winner a race.

Kubernetes

The Kubernetes client is configured the way every Kubernetes tool configures one: the in-cluster ServiceAccount if there is one, otherwise the current context of $KUBECONFIG or ~/.kube/config.

Boolean environment values accept true/1/yes/on and false/0/no/off, case-insensitive. Anything else is an error at startup. On the command line a boolean is a flag — --no-status-update true would be a worse way to say it.

Listeners

An address may be host:port, :port, or a bare port — --http :8080 and --http 8080 both bind every interface. IPv6 takes the bracketed form, [::1]:8443.

--admin-token-file covers POST and DELETE on /admin/ and nothing else. A request without the header, or with the wrong token, is a 401 carrying WWW-Authenticate: Bearer. Trailing whitespace in the file is trimmed, so a Secret written by echo works; a file holding only whitespace is refused at startup rather than accepted as an empty token.

GET is never gated — Prometheus and the kubelet cannot send a header, and a /healthz that 401s is a crash loop. Without the flag, the mutating endpoints accept anything that can reach the port and the daemon says so once at startup:

WARN the mutating /admin/ endpoints accept any caller that can reach the admin
     port; set --admin-token-file to require a bearer token

The token is read once, at startup. Rotating it means restarting the process; on the chart, replace the Secret and kubectl rollout restart. See the admin listener for the whole trust model, and controller.adminToken and networkPolicy in the chart.

The three --no-* flags have no environment twins; setting the corresponding RAMJET_* variable to an address is how you move a listener from the environment.

In dev mode, without an explicit --https or --no-https, the TLS listener is skipped when the configuration declares no certificates. In Kubernetes mode it always binds: the certificates arrive over a watch, after the socket.

Time travel and the audit trail

Each kept generation holds its route table and its parsed certificates alive, which is roughly a hundred bytes per route per generation; the certificates are content-addressed and shared between generations that did not rotate them. Ten generations of a ten-thousand route cluster is a few megabytes.

The webhook is fire-and-forget: one attempt, a 5s timeout, failures logged and never blocking a publish. It speaks http:// only and refuses an https:// URL at startup rather than downgrading, because the control plane does not carry a TLS client for this; point it at a collector inside the cluster.

See Rollback and the audit trail.

Upstreams

--upstream-pool-idle is a ceiling, not a reservation: nothing is opened until a request needs it. Below the concurrent requests an endpoint receives, the surplus connections are closed as they go idle and reopened on the next request, which is a TCP handshake on the request path. Above it, the only cost is file descriptors.

Serving

An unrecognised engine name is an error, not a fallback to the default. Somebody who typed --engine io_uring asked for something specific, and quietly serving on the other engine is the worst possible answer, because it looks like it worked.

Each runtime owns its connections, its upstream connection pool, and its timers, and a connection stays on the one it landed on. available_parallelism reads the cgroup CPU limit, so a pod with limits.cpu: 2 gets two runtimes rather than one per host core. Setting this above the cores the process can actually use makes them compete; setting it to 1 serves everything on one thread.

--max-buf-size bounds the tail, not the common case. hyper allocates the first 8 KiB of each buffer whatever this is set to, and never shrinks one again while the connection lives — so a client that sends a 400 KiB header block would pin 400 KiB until it disconnects. 64 KiB accepts every request nginx’s own 32 KiB limit would and bounds the worst case at a sixth of hyper’s default. Requests over the ceiling are answered 431. A value below 8192 is raised to it, because hyper panics on anything smaller.

What --engine uring refuses

It serves HTTP/1.1 on the ramjet reactor — io_uring on Linux, kqueue elsewhere — and terminates TLS, carries protocol upgrades, reads the PROXY protocol, runs in Kubernetes mode and drains on SIGTERM exactly as the other engine does.

What is left is HTTP/2, at both ends of the hop:

• Downstream, it does not speak HTTP/2 itself. A client that asks for it is handed to a hyper engine in the same process, with its bytes intact, and sees one connection that negotiated HTTP/2. That is on by default; --no-h2-dispatch turns it off, at the cost of not offering HTTP/2 at all.
• Upstream, it does not dial one. A backend annotated backend-protocol: GRPC is answered 502 naming the other engine, and gRPC to it with it.

--http3 is refused at startup rather than ignored, because a UDP listener that silently did not exist is worse than one that says so.

Everything about routing, load balancing, canaries, headers and /metrics is the same on both. Engines has the full parity matrix, and the differential test that keeps it honest.

Behind a load balancer

A cloud L4 load balancer — AWS NLB, DigitalOcean, Scaleway, GCP passthrough — forwards TCP without touching the payload, so without this every request is attributed to the balancer. Turn it on where the balancer is configured to send the header, and set the same option on both sides.

Security. The header is the client identity. Anything that can reach the listener can claim to be any address, and X-Forwarded-For, X-Real-IP and every application decision made from them follow. Enable it only on a listener nothing but the load balancer can reach.

The header is required, not optional: a connection without a valid one is dropped, which is what nginx’s proxy_protocol listener parameter and HAProxy’s accept-proxy both do. A permissive fallback would let an attacker choose per connection whether to be spoofed, which is strictly worse than either fixed answer.

The first such drop on each serving runtime is logged at warn and the rest at debug — so a balancer that is not sending the header says so rather than looking like a network fault, and without a line per occurrence burying the outage under its own logs.

The --admin listener never reads one, because Prometheus and the kubelet do not send one.

Three properties worth knowing: the header is read before the TLS handshake (that is the order the wire has); nothing read past the header is thrown away, so a read that returns the header and the start of a ClientHello replays those bytes intact; and a header that names nobody — a v2 LOCAL from a health checker, a v1 UNKNOWN, a v2 AF_UNSPEC — is consumed with the socket’s own peer standing.

HTTP/3 (experimental)

Off costs nothing: no UDP socket is bound, no thread is started, and no header is added.

Two combinations are refused at startup rather than ignored:

• --http3 with --engine uring, which has neither TLS nor QUIC.
• --http3 with --no-https, which leaves no port to take and no response to advertise on.

--help and --version still print over a refused combination — they are requests for text, and answering one with a usage error about two other flags is the least useful moment to be strict.

See HTTP/3.

Traffic mirroring

0 is legal and is not clamped up, unlike the buffer ceiling: it means “never buffer”, which still mirrors every GET and is a reasonable thing to ask for on a route that carries large uploads.

Mirroring itself is annotation-driven — see Traffic mirroring.

Canary auto-promotion

No flags. It is annotation-driven, per canary Ingress, and off unless ramjet.dev/auto-promote: "true" is set on one. See Canary auto-promotion.

Shutdown

SIGTERM stops the accept loop and closes the listeners immediately, so the load balancer looks elsewhere, and then gives in-flight requests the grace period to finish. Both engines do this, and with HTTP/2 dispatch on both lanes are signalled at once and drain inside the one deadline — see Engines for what counts as in-flight and why tunnels do not.

The chart’s terminationGracePeriodSeconds is 45, deliberately longer than this default, so Kubernetes does not SIGKILL a pod mid-drain.

Other

Why the parser is hand-rolled

This binary’s options are all --name value, and clap would add roughly 200KB and a dozen transitive crates to a data-plane image for the privilege of formatting the help text. The parser is about a hundred lines and every option it accepts is visible in one place — crates/ramjet-ingressd/src/args.rs, which also has a test asserting that every option it accepts appears in --help.

Operations

What this controller gives you that a reload-based one cannot, and how to use it.

Two of these — rollback and mirroring — exist because publishing a configuration is one pointer store. If applying a generation is a pointer store, republishing an old one is the same pointer store; and if the request path never waits on a lock, adding a fire-and-forget copy to it is not a latency decision.

The admin listener

Everything on these pages is reachable on :10254, which is its own port rather than a reserved path on the data plane. A path on the data plane is a path an Ingress can claim, so /metrics would either shadow somebody’s application route or be shadowed by it — and it would be reachable from the internet, which is a way to tell an attacker your request rate.

The chart puts it behind a ClusterIP Service, never the internet-facing LoadBalancer, and because the split is two objects rather than a list of ports, no values entry can accidentally publish it.

kubectl port-forward -n ramjet-ingress svc/ramjet-ingress-admin 10254:10254

Three things stand between the mutating endpoints and an accident or an attacker, and they are deliberately different in kind.

The shape, unconditionally: the mutating endpoint answers to POST and DELETE and nothing else, so a link, a browser prefetch, a scraper following URLs, or a health checker walking paths cannot roll a cluster back by accident.

The network: a ClusterIP Service and nothing in front of it. The chart’s optional networkPolicy.enabled narrows that further, to the release namespace.

A bearer token, with --admin-token-file (chart: controller.adminToken.secretName). Set it and every mutating /admin/ request must carry Authorization: Bearer <token>:

kubectl -n ramjet-ingress create secret generic ramjet-admin \
  --from-literal=token="$(openssl rand -hex 32)"
helm upgrade ramjet ... --set controller.adminToken.secretName=ramjet-admin

curl -X POST -H "Authorization: Bearer $TOKEN" \
  -d '{"generation": 41}' localhost:10254/admin/rollback

Without it, the daemon logs one warning at startup and accepts a rollback from anything that can reach the port. This page used to argue that a token was pointless because anything reaching the port could already read the pod’s ServiceAccount token — which was wrong in one specific way. That token is on our filesystem, not on the network. A pod in some other namespace cannot read it, and until now the only thing stopping that pod from rolling the ingress table back was that it had not thought of it.

GET is never gated. /metrics is scraped by Prometheus and /healthz and /readyz are called by the kubelet, and neither can be taught to send a header — gating them would trade a rollback for a pod that restarts every time its liveness probe is refused. /admin/generations and /admin/routes stay open for the same reason they are GET at all: they report what a replica is serving, which is not a secret from anything that can already send it traffic.

The token is read once, at startup, so rotating it is kubectl rollout restart after replacing the Secret. A read(2) per request to make a yearly event convenient is the wrong trade. ramjet-top sends the token with --token-file, and only on its pin and unpin keys.

Observability

Three surfaces on the admin listener (:10254 by default): a Prometheus page, two probes, and a small JSON API.

Endpoints

The two rollback verbs are covered in Rollback and the audit trail, including the bearer token they need when one is configured.

Both JSON endpoints carry a top-level "version", currently 1. It exists for the day a field’s meaning has to change rather than a field being added — a discriminator introduced at the same time as the break would be one release too late to help anyone. Until then, a reader that ignores it is correct, and one that reads it must treat absent as version 0: every build before this one serves the same shape without the field, and an upgrade is exactly when somebody is watching. ramjet-top parses it and does not branch on it.

The probes answer different questions

/healthz is unconditional. A liveness probe that fails restarts the pod, so anything conditional in it turns a transient dependency problem into a crash loop.

/readyz is gated on the first compiled generation, not on the process being up. The controller seeds its channel with an empty table at generation 0, meaning “nothing has been compiled yet”, and the flag flips only once a generation greater than zero has been published. Without that gate a rolling update would route traffic to a replica whose table is empty, and every request in that window is a 404.

The flag is one-way. A later generation never takes a replica back out of rotation: a table one debounce window stale is far better than 404ing everything while Kubernetes reroutes.

Metrics

ramjet_requests_total
ramjet_route_misses_total
ramjet_active_connections
ramjet_upstream_latency_seconds          (_sum, _count)
ramjet_upstream_connect_failures_total
ramjet_upstream_timeouts_total
ramjet_upstream_retries_total
ramjet_tls_handshakes_total
ramjet_tls_handshake_failures_total
ramjet_route_table_generation
ramjet_pinned
ramjet_mirrored_total
ramjet_mirror_dropped_total
ramjet_mirror_skipped_total
ramjet_mirror_failures_total
ramjet_h3_connections_total
ramjet_h3_requests_total
ramjet_h3_handshake_failures_total
ramjet_engine_unsupported_h2c_total
ramjet_engine_unsupported_grpc_total

Four of them deserve naming:

• ramjet_route_table_generation is how you tell whether a replica is actually serving the configuration you think it is.
• ramjet_pinned is 1 while a rollback is holding publication — so a replica frozen on purpose is distinguishable from one whose control plane has died.
• ramjet_engine_unsupported_h2c_total counts requests refused because their backend is annotated backend-protocol: GRPC and the serving engine dials HTTP/1.1 only. It can only move on --engine uring; the hyper engine emits it at zero, so a dashboard does not lose a line when somebody switches. Anything above zero means those routes are down on this replica and the fix is --engine hyper.
• ramjet_engine_unsupported_grpc_total counts gRPC requests refused because the backend is not annotated. Both engines can move it, and the fix is on the Ingress rather than on the daemon.

Both refusals also carry an x-ramjet-unsupported response header — h2c-upstream and grpc-needs-backend-protocol respectively — so an access log or a client library can tell them from an ordinary 502 without reading the body. The vocabulary is closed and no other response carries the header, which is what makes a check for it a check for equality.

An HTTP/3 request is counted in ramjet_requests_total like any other, because it is one; the h3 series are in addition.

Per-route counters are deliberately not here

/metrics gained exactly one series for the whole per-route feature, and it is a gauge with no labels. Per-route data is served as JSON on /admin/routes instead.

ingress-nginx exports per-route series, and it is the single most common reason its metrics endpoint becomes the most expensive request the pod serves: ten thousand routes means ten thousand series on every scrape, forever, whether or not anybody looks.

The cost of counting is not the reason. On the hot path a route’s counters are four relaxed atomic adds to one cache-line-aligned block, reached by an index the matched rule already carries — no map, no label set, no reference count. Measured on a 10,001-route table, the whole per-request sequence is 4.9 ns against roughly 24 µs for the forwarded request it describes.

/admin/routes

Every route in the serving table with its counters, exactly as served:

{
  "version": 1,
  "generation": 0,
  "routes": [
    {
      "host": "shop.example.com",
      "path": "/api",
      "path_type": "Prefix",
      "backend": "api",
      "endpoints": 2,
      "requests_total": 0,
      "errors_5xx_total": 0,
      "upstream_latency_count": 0,
      "upstream_latency_ms_sum": 0.0,
      "canary": { "backend": "api-next", "weight_percent": 20 },
      "canary_stats": {
        "requests_total": 0,
        "errors_5xx_total": 0,
        "upstream_latency_count": 0,
        "upstream_latency_ms_sum": 0.0
      },
      "mirror": null
    },
    {
      "host": "shop.example.com",
      "path": "/",
      "path_type": "Prefix",
      "backend": "web",
      "endpoints": 1,
      "requests_total": 0,
      "errors_5xx_total": 0,
      "upstream_latency_count": 0,
      "upstream_latency_ms_sum": 0.0,
      "canary": null,
      "canary_stats": null,
      "mirror": { "backend": "shadow", "percent": 100, "host": null }
    }
  ]
}

A hostless rule appears with "host": "*".

Reading canary_stats

canary_stats is null on a route with no canary, and an object of zeroes on a canary nothing has reached yet. That distinction is deliberate — an object full of zeroes could not be told apart from the other case, and it is what an automatic promotion is about to act on.

The totals are totals. A route’s own counters include the requests the canary answered; canary_stats says how much of them was the new backend, and the stable share is one subtraction.

The other arrangement — stable in one block, canary in the other — would make every existing graph of a route’s request rate step down the moment somebody started a canary, which is exactly the graph an operator is watching at that moment.

Counters survive a rebuild

They are carried forward by identity, so adding one Ingress does not reset every neighbour’s numbers. A route’s identity is its host, path, path type and backend — change the backend and it is a different route for accounting purposes, because its latency is no longer comparable to what came before.

/admin/generations

{
  "version": 1,
  "serving": 0,
  "pinned": null,
  "generations": [
    {
      "generation": 0,
      "applied_at": "2026-08-28T13:31:56Z",
      "published": true,
      "digest": "0000000000000000",
      "routes": 5,
      "hosts": 2,
      "certs": 0,
      "diff": {
        "summary": "5 routes added, 3 hosts added, 1 mirror added, default backend now fallback (gen 0→0)",
        "routes_added": ["shop.example.com /api -> api", "…"],
        "routes_removed": [],
        "backends_changed": [],
        "hosts_added": ["shop.example.com", "…"],
        "hosts_removed": [],
        "certs_rotated": [],
        "mirrors_added": ["shop.example.com / -> shadow (100%)"],
        "mirrors_removed": []
      }
    }
  ]
}

published: false marks a generation the controller compiled while a rollback pin was held — it was recorded, and it never reached the wire.

The diff is taken over the two compiled tables, not over the API objects, and that is what makes it useful. An Ingress edited from Prefix: /foo to Prefix: /foo/ compiles to the same route and does not appear; a Deployment scaling from three pods to five changes no Ingress at all and does.

ramjet-top

The admin port reports counters, and the question you usually have is about rates. ramjet-top polls all three endpoints, differences the counters, and draws them.

╭ ramjet-top ─ http://127.0.0.1:10254 ───────────────────────────────────────────────────────╮
│gen 0  routes 5  gens 1  conns 0                                   rps · last 6 polls · peak 420│
│rps 103.9  5xx 0.00%  upstream 0.6  up 5s          █                                        │
│                                                   █▃▄▄▄▃                                   │
╰────────────────────────────────────────────────────────────────────────────────────────────╯
╭ routes 5 ──────────────────────────────────────────────────────────────────────────────────╮
│HOST                   PATH          TYPE     BACKEND        EPS  RPS       5XX     ms   CANARY
│shop.example.com       /             Prefix   web              1      52.0   0.00%   0.6 -
│shop.example.com       /api          Prefix   api              2      52.0   0.00%   0.6 20%→api-next
│*                      /status       Prefix   web              1      0.00       -     - -
│*.example.com          /             Prefix   web              1      0.00       -     - -
╰ sorted by rps desc ────────────────────────────────────────────────────────────────────────╯
 ● live · polling every 1s
 q quit  Tab generations  r rps  e 5xx  l latency  h host  / filter  g refresh

# The default target is the conventional admin port, 127.0.0.1:10254.
cargo run -p ramjet-top

# Anywhere else. A bare host:port is fine; it gets an http:// scheme.
ramjet-top 10.0.0.5:10254
ramjet-top --url http://10.0.0.5:10254

# Against a pod, through a port-forward.
kubectl port-forward -n ingress ds/ramjet-ingress 10254:10254 &
ramjet-top localhost:10254

# Poll faster, or slower.
ramjet-top -i 250ms
ramjet-top --interval 5s

# Somebody else's cluster: watch, but do not touch.
ramjet-top --read-only

# One shot, for a script, a CI log, or an incident channel.
ramjet-top --once
ramjet-top --json | jq '.routes.routes[] | select(.errors_5xx_total > 0)'

--once prints an aligned text table and exits: no terminal required, sorted by host and path so two runs are diffable, and reporting cumulative counters rather than rates, because a rate is a difference between two polls and this mode does one. --json dumps the merged snapshot — both admin responses verbatim plus the series read out of /metrics — and implies --once.

Exit status is 0 on success, 1 if the daemon could not be reached, and 2 if the command line was wrong.

Keys

p and u are the emergency brake — they drive POST/DELETE /admin/rollback. Both need a y to confirm, anything else cancels, and --read-only refuses them outright and stops advertising them.

What the numbers mean

Everything the server exports is cumulative and everything on screen is a rate, so the interesting part is the subtraction. Three things make it harder than it looks, and all three are handled:

• Counters restart. A removed and re-added route, or a restarted data plane, drops a counter below the value held from last poll. Every subtraction saturates at zero, so a restart reads as 0.00 rather than as eighteen quintillion requests per second.
• Routes are not rows. The table is rebuilt every generation, so “the same route” is keyed on host, path and path type — deliberately not on the backend, because a backend swap is the most interesting moment to keep watching a route through.
• A new route has no rate. Dividing a lifetime counter by one poll interval reports an hour’s traffic as if it happened this second. New routes show - for one interval, are flagged green, and report a real rate from the next poll.

The interval divided by is the measured gap between polls, from a monotonic clock — not --interval. A poll that took 900 ms because the server was busy would otherwise inflate every rate on screen at the worst possible moment.

Latency is a windowed mean: the delta of the sum over the delta of the count. On a process that has been up a week, a lifetime mean cannot move, and an upstream that just started taking two seconds would not show up in it at all.

When the daemon goes away

The last good data stays on screen, dimmed and marked STALE, with the status line saying how long ago it was true and why the poll failed. It never clears the screen to print a connection error: the moment the daemon becomes unreachable is the moment its last known state is most worth looking at.

When it comes back, the rate reported for the gap is the true average across it — 600 requests over a 60-second outage is 10/s, not 600/s.

Logs

Through tracing to stderr, info by default, filtered with RUST_LOG. The lines worth alerting on are the per-generation publish record on the audit target and the warnings from translation: a rejected Ingress, an unresolvable Service, a Secret that will not parse.

Events

Not everything worth knowing needs pod-log access, and the things an Ingress’s author can act on should not.

The split is by what the Event is about: a compiled generation belongs to no single Ingress, and a promotion decision or a refused value belongs to exactly one. Per-object Events are written only when what they say changes, so a steady broken state costs one Event rather than one per rebuild.

Which generation reached which Ingress

Every managed Ingress carries ramjet.dev/observed-generation, the compiled generation that last included it:

kubectl get ingress -A -o custom-columns=\
NS:.metadata.namespace,NAME:.metadata.name,GEN:'.metadata.annotations.ramjet\.dev/observed-generation'

That is what the controller compiled. /admin/routes’s top-level generation is what a replica is serving. They agree in the steady state and diverge in exactly one place — while a rollback pin is held, the controller keeps compiling and annotating while the data plane stays where it was put. So the pair is a useful diagnostic on its own: annotation ahead of /admin/routes means something is holding publication back, and the same annotation lagging its neighbours means one Ingress stopped being included.

Rollback and the audit trail

The thesis says a configuration change is one pointer store. Two things follow that a reload-based controller cannot offer, and this page is both of them.

The emergency brake

If publishing a generation is a pointer store, republishing an old one is the same pointer store. So the daemon keeps the last N applied generations — default 10, --history-size — and putting one back on the wire costs what a normal configuration change costs.

# What has been applied, and what changed
curl :10254/admin/generations

# Put 41 back on the wire, now
curl -XPOST :10254/admin/rollback -d '{"generation": 41}'

# Release, and jump to the newest
curl -XDELETE :10254/admin/rollback

DELETE is idempotent.

Where the daemon was started with --admin-token-file — the chart’s controller.adminToken.secretName — both verbs need the token, and the GET above does not:

TOKEN=$(kubectl -n ramjet-ingress get secret ramjet-admin \
  -o jsonpath='{.data.token}' | base64 -d)
curl -XPOST -H "Authorization: Bearer $TOKEN" \
  :10254/admin/rollback -d '{"generation": 41}'

This is the request worth authenticating: it is the one thing an arbitrary pod in the cluster could otherwise send to change what every replica serves. See the admin listener.

On a shared workstation, note that the shell expands $TOKEN before exec, so the token is in curl’s argv and visible to ps for the length of the request. curl -K - reads the header from stdin instead:

printf 'header = "Authorization: Bearer %s"\n' "$TOKEN" |
  curl -K - -XPOST :10254/admin/rollback -d '{"generation": 41}'

It works when the API server is the thing that is wrong. Every alternative route to the same outcome — re-applying the previous Ingress objects, kubectl rollout undo, waiting for a controller to recompile — goes back through the control plane, which is exactly the component an operator reaches for this lever to route around.

ramjet-top drives both verbs from the generation timeline: p to pin the selected generation, u to release. Both ask for a y first, and --read-only refuses them outright. Against a daemon with a token, give it --token-file <PATH> (or RAMJET_TOP_TOKEN_FILE); everything it polls is a GET and needs none.

A rollback is a pin, not a rewind

The controller does not stop. It keeps watching, keeps compiling, and keeps handing generations over; they are recorded with published: false so you can see what is being held back, and nothing reaches the data plane until the pin is released — at which point it publishes the newest generation, not the one that was pinned over.

Draining the controller’s side matters more than it looks: a pin that stopped reading the channel would block the rebuild loop, and releasing it would then jump to whatever was stuck there rather than to the current state of the cluster.

A pinned generation’s certificates go back with its table, in the same certificates-then-table order as a first publish, because a table whose TLS ids are not in the store fails every handshake for the width of the gap.

ramjet_pinned is 1 the whole time.

The pin dies with the process, deliberately

Kubernetes is the source of truth for what this controller serves. A pin is a local override of that, held in memory, by one replica, because something is on fire right now.

Persisting it would create a second source of truth that survives a restart and answers to nobody — a pod that comes back after an eviction still serving a generation from last Tuesday, with no object in the cluster saying why.

Fix the Ingress objects, then release the pin.

What it costs to keep the history

The ring holds each generation’s route table and parsed keys alive instead of letting them drop: roughly a hundred bytes per route per generation. Successive generations share everything that did not change — most importantly the certificates, which are content-addressed and therefore shared by id.

Ten generations of a ten-thousand route cluster is a few megabytes.

The history records generations this replica applied, which is not quite every generation the controller compiled: the channel between them carries the latest value rather than a queue, so publishes closer together than one pass of the applier coalesce. A gap in the numbering is generations that were never on the wire.

--static-routes gets the same endpoints with one generation in the ring. Nothing is special-cased for it — rolling back to it is a no-op that republishes what is already serving.

What changed, in words

A digest tells you that configuration changed, which is all the rebuild loop needs. It cannot answer the question somebody actually asks, which is what changed.

So every publish is diffed against the previous compiled generation: routes added and removed, routes whose backend or endpoint count moved, hosts gained and lost, hosts whose certificate material rotated, mirrors added and removed, and a changed default backend.

The diff is taken over the two compiled tables, not over the API objects, and that is what makes it useful:

• An Ingress edited from Prefix: /foo to Prefix: /foo/ compiles to the same route and does not appear.
• A Deployment scaling from three pods to five changes no Ingress at all and does.

Three ways it is written down

Each publish is recorded for three different readers.

A structured tracing event on the audit target

So a log pipeline can filter to configuration changes and nothing else.

INFO audit: 5 routes added, 3 hosts added, 1 mirror added, default backend now fallback (gen 0→0)
  event="config" generation=0 published=true routes_added=5 routes_removed=0
  backends_changed=0 hosts_added=3 hosts_removed=0 certs_added=0 certs_removed=0
  certs_rotated=0 mirrors_added=1 mirrors_removed=0 default_backend_changed=true

A Kubernetes Event on the IngressClass

Reason ConfigApplied, ConfigPinned, or ConfigResumed, with a message like "3 routes added, 1 cert rotated (gen 41→42)" — so kubectl describe ingressclass answers “what has this controller been doing” without pod-log access.

kubectl describe ingressclass ramjet

Events are written directly rather than through kube’s Recorder, which aggregates same-reason events for six minutes and keeps the first note: three deploys in a minute would become “ConfigApplied ×3” showing only what the first one did, which is precisely the information an audit trail exists to keep.

Canary lifecycle events go on the canary Ingress instead — see canary deployments. The split is by what the Event is about: a compiled generation belongs to no single Ingress, and a promotion decision belongs to exactly one.

RBAC: events.k8s.io / events, create and patch, as a ClusterRole rather than a Role — Events are namespaced and an Ingress can be in any namespace. The chart has it. Without it the Events are skipped at debug and nothing else changes.

An optional webhook

--audit-webhook http://collector.observability.svc:8080/ingress-audit

One fire-and-forget POST of the diff as JSON, five second timeout, failures logged. It does not retry, because it is a copy and not the record — the log line, the Event, and the ring all already have it, and a delivery system with queues and backoff would be a thing to debug during exactly the incidents it exists to describe.

http:// only. An https:// URL is refused at startup rather than silently downgraded, because the control plane does not carry a TLS client for this; point it at a collector inside the cluster.

Canary auto-promotion decisions go down the same three channels.

Interaction with auto-promotion

A rollback pin pauses automatic promotion entirely. An operator holding the emergency brake has taken manual control of what this replica serves; patching Ingresses underneath them would be changing the cluster they are trying to hold still.

Canary auto-promotion

Let a healthy canary promote itself, and pull it back the moment it stops being healthy. It is off unless asked for, and opting in is one annotation.

Annotate the canary Ingress:

metadata:
  annotations:
    nginx.ingress.kubernetes.io/canary: "true"
    nginx.ingress.kubernetes.io/canary-weight: "5"
    ramjet.dev/auto-promote: "true"
    # ramjet.dev/auto-promote-interval: 60s
    # ramjet.dev/auto-promote-steps: 5,10,25,50,100
    # ramjet.dev/auto-promote-max-5xx-percent: "1"
    # ramjet.dev/auto-promote-max-latency-factor: "1.5"
    # ramjet.dev/auto-promote-min-requests: "50"

Every field has a default that is safe to run with. The rest exist because “safe” is a property of a particular service’s error budget, and nobody else can know it. The full table with parsing rules is in the annotations reference.

The state machine

Every interval, per opted-in canary: take the window — this interval’s deltas only, canary side and stable side separately — and decide.

                    ┌─────────────────────────────────┐
                    │  window: canary and stable       │
                    │  requests, 5xx, mean latency     │
                    └───────────────┬─────────────────┘
                                    │
              either side < min-requests?
                        │ yes                  │ no
                        ▼                      ▼
                   ┌────────┐        canary 5xx% > max-5xx-percent
                   │  HOLD  │           or canary mean latency >
                   └────────┘           stable mean × max-latency-factor
                                            │ yes            │ no
                                            ▼                ▼
                                     ┌────────────┐    next step exists?
                                     │  ROLLBACK  │      │ yes      │ no
                                     │ weight → 0 │      ▼          ▼
                                     └────────────┘   ┌──────┐  ┌──────────┐
                                                      │ STEP │  │ PROMOTED │
                                                      └──────┘  └──────────┘

The router counts the two sides apart, which is what makes the comparison possible at all — see canary_stats.

Three things that are easy to get wrong

Holding is not failing. A canary receiving nothing at 03:00 is a quiet service, not a broken one. Gating on both sides — not just the canary’s — also matters: a latency comparison against four stable requests is not a comparison. Rolling back on low traffic would make the feature unusable on anything but the busiest routes.

Windows, not lifetimes. The counters are cumulative and the process may have been up for a week, so a lifetime error rate cannot move fast enough to catch anything. Each pass subtracts the previous pass’s reading.

The first pass after a step spans the moment the weight changed and so mixes two ratios. That is deliberate, and it errs safe: the older and smaller weight is the one over-represented.

Errors are absolute, latency is relative. An error budget is a number somebody actually has, so production being on fire is not a licence to promote a canary that is also on fire. Latency has no such absolute: a service that legitimately takes two seconds would be un-promotable against a fixed threshold, so the canary is compared to what it is replacing.

Interlocks

• A rollback pin pauses everything. An operator holding the emergency brake has taken manual control of what this replica serves; patching Ingresses underneath them would be changing the cluster they are trying to hold still.

• A rollback is one-way. It writes auto-promote: "false" alongside the weight, and the loop refuses any canary whose status says it was rolled back even if the annotation is somehow still true. Both, because the guard has to survive a restart — the annotation carries it across a rescheduled pod — and because the two are written in one patch that could half-fail.

  A canary re-armed automatically after failing once will fail again on the next interval, flapping traffic across a broken backend for as long as nobody is watching. Re-arming is a human decision.

• Reaching the last step is validated before it is accepted. Stepping to 100% and immediately declaring victory would mean full traffic never gets a single window of scrutiny, so promotion happens on the next healthy window at the final weight.

What a rollback writes

nginx.ingress.kubernetes.io/canary-weight: "0"
ramjet.dev/auto-promote: "false"
ramjet.dev/auto-promote-status: "rolled-back: 5xx 4.2% over 1%"

To re-arm after fixing the canary, clear auto-promote-status and set auto-promote back to "true" yourself.

A successful finish writes ramjet.dev/auto-promote-status: promoted and stops.

Where the decisions show up

Logged on the audit target with their numbers, written as a Kubernetes Event on the canary Ingress, and POSTed to --audit-webhook:

kubectl describe ingress web-canary

Events:
  Type     Reason            Age   From            Message
  ----     ------            ----  ----            -------
  Normal   CanaryStepped     4m    ramjet-ingress  canary healthy; weight 5 -> 10
  Normal   CanaryStepped     3m    ramjet-ingress  canary healthy; weight 10 -> 25
  Warning  CanaryRolledBack  2m    ramjet-ingress  rolled back from 25%: 5xx 4.10% over the 1% budget

On the Ingress, and only there. An Event exists to point at the object to go and look at, and after an automatic rollback that object is the canary — a second copy on the IngressClass would make kubectl get events show every promotion twice for a class-level view the audit log and the webhook already carry in full. Configuration-level events (ConfigApplied, ConfigPinned, ConfigResumed) still go on the IngressClass, because a compiled generation belongs to no single Ingress.

Holds are debug only. On a quiet route they are the normal state, and an Event per interval per canary would bury the three that matter.

Why the backend swap stays human

Reaching 100% means every request is served by the canary backend while the production Ingress still names the old one. The obvious next step — rewrite spec.rules[].backend and delete the canary Ingress — is deliberately left to a person, and it looks like the last mile of the same job, so it is worth saying why it is not.

Everything this loop does is reversible by writing one number. Every state it can reach is a weight, and every weight has an inverse the loop already knows how to apply; a rollback is the same mechanism as a step.

Editing the backend is a different kind of change: it is the thing the canary was a rehearsal for, it normally comes with deleting an object, and undoing it means reconstructing an object rather than setting a field. A controller that restructures the resources an operator wrote, on a timer, is a controller people turn off.

So the loop drives the dial to 100, says so in an Event and in the annotation, and stops.

RBAC, and the GitOps warning

This is the only write this controller makes to an object an operator authored. It needs:

- apiGroups: ["networking.k8s.io"]
  resources: ["ingresses"]
  verbs: ["patch"]

Spec-level, because an annotation is metadata and ingresses/status cannot carry it. The chart’s ClusterRole has it. Without the rule, promotion logs a permission error every interval and changes nothing.

The patches are JSON merge patches sent under the ramjet-ingress field manager, so managedFields still answers “who set this weight” and taking the field from whoever created the Ingress — a person, a Helm release, a GitOps reconciler — needs no override, because a merge patch is never refused as a conflict.

Deliberately not a server-side apply. An apply states everything the field manager owns, so the API server deletes whatever that manager’s entry claims and the body omits — and this controller writes two disjoint annotation sets under one manager. As applies, promotion’s canary-weight write erased ramjet.dev/observed-generation, and the status writer’s next write erased canary-weight, leaving a canary with no weight seconds after the controller announced it had stepped it up. Nothing here ever needs to remove an annotation, which is the only thing an apply buys.

In a GitOps cluster, a reconciler that also claims canary-weight will fight this loop and win on its own schedule. Either exclude canary-weight from its managed fields, or do not opt that Ingress in.

What it costs when nobody uses it

Nothing measurable. The candidates are compiled by the controller and arrive on the generation channel; the loop issues no API reads of its own, because the controller has already listed every Ingress and parsed every annotation. A loop doing its own list would cost a cluster-wide read every minute, forever, on every installation, whether or not anybody uses the feature.

With nobody opted in, the list is empty and the loop is a timer that does nothing.

Traffic mirroring

Send a second, fire-and-forget copy of a route’s traffic to a shadow backend and throw the answer away — a rewrite gets production traffic before it gets production responsibility.

Annotate the production Ingress:

metadata:
  annotations:
    ramjet.dev/mirror-backend: shadow/api:80   # namespace optional
    ramjet.dev/mirror-percent: "10"            # default 100
    ramjet.dev/mirror-host: shadow.example.com # optional Host override

Copies carry X-Mirrored-By: ramjet-ingress, so a shadow can tell a copy from the real thing before it decides whether to charge somebody’s card.

A mirror is a property of the route, so it goes on the production Ingress. The canary Ingress is a second opinion about where a share of that route’s traffic goes — not a second route that could have its own shadow. Setting mirror-backend on a canary Ingress does nothing, and says so in a warning.

The invariant

A mirror must never make the primary request slower or more likely to fail.

That is not a goal, it is the property that decides whether the feature can be switched on in front of real traffic, and every mechanism below exists for it.

• Nothing is awaited. The request path hands the copy to a queue and returns. It never waits for a connection, a response, or a timeout.
• The queue is bounded and drops. One channel per serving runtime, 256 deep, entered with a non-blocking send. Per-runtime so a wedged shadow fills one core’s queue rather than contending for a shared one; bounded because the alternative turns a slow mirror into unbounded memory growth on the pod serving production.
• Responses are drained and discarded. Drained rather than dropped so the upstream connection returns to the pool instead of being closed, which would put a TCP handshake on every mirrored request.
• Failures are counted, never propagated. A mirror backend that is down, refusing, absent, or catatonic produces a number on /metrics and nothing else. A five-second deadline, much shorter than the primary’s, bounds the one thing a slow mirror can still affect: the queue behind it.
• No in-flight accounting. A copy does not take the leastConn guard its primary does. Letting shadow traffic move production’s load-balancing decisions would be its own kind of leak.

The body, which is the hard part

Everything above is cheap because a request head is small and already in memory. A body is neither, and this data plane’s whole position on request bodies is that it does not buffer them.

So the cap is real and small: --mirror-max-body, 256 KiB by default.

The primary is never held waiting for more than the cap, and never fails because of the attempt.

--mirror-max-body 0 is legal and is not clamped up: it means “never buffer”, which still mirrors every GET and is a reasonable thing to ask for on a route that carries large uploads.

Four counters, because they have four different fixes

A route’s mirror configuration also appears on /admin/routes:

"mirror": { "backend": "shadow", "percent": 100, "host": null }

and mirrors added or removed show up in the generation diff.

mirror-host is not cosmetic

A shadow deployment usually answers to a different name, and a copy carrying the production Host can be routed by whatever sits in front of it — possibly straight back to production, which is the one outcome a mirror must never produce.

Sampling

ramjet.dev/mirror-percent takes 0–100 and defaults to 100. The opposite default would make an operator who added mirror-backend and saw no traffic conclude the feature does not work; sampling exists to turn mirroring down on a route that cannot afford the duplicate load.

0 is kept rather than defaulted — turning a mirror off without deleting the annotation that says where it points is the whole reason the knob is separate from the backend.

An out-of-range or unparseable value (101, -5, lots, 50%) is reported and falls back to 100. It never disables the mirror.

Trying it locally

The dev-mode route file supports the same thing, and crates/ramjet-ingressd/examples/dev-routes.yaml ships with one wired up:

routes:
  - host: shop.example.com
    path: /
    pathType: Prefix
    backend: web
    mirror:
      backend: shadow
      percent: 100

Kill the shadow upstream and watch nothing change about the response you get, while ramjet_mirror_failures_total moves on :10254/metrics.

HTTP/3

Experimental, and off by default. Off costs nothing: no UDP socket is bound, no thread is started, and no header is added. Read the deployment constraint before turning it on — for most cloud shapes it is the thing that decides whether this works at all.

--http3 serves HTTP/3 over QUIC on the --https port number in UDP, and advertises it on every HTTPS response with alt-svc: h3=":<port>"; ma=86400.

$ ramjet-ingressd --static-routes routes.yaml --http3
ramjet-ingressd 0.1.0 — 1 backend(s), 1 endpoint(s), 1 route(s), 1 certificate(s)
  config   routes.yaml
  http     0.0.0.0:8080
  https    0.0.0.0:8443
  http3    0.0.0.0:8443
  admin    0.0.0.0:10254

In the chart:

helm upgrade ramjet deploy/chart/ramjet-ingress --reuse-values \
  --set http3.enabled=true

which adds --http3, a UDP container port and a UDP Service port — both on the same number as https.

A second way in, not a second proxy

A request that arrives over QUIC is turned into the same types the TCP listeners produce and handed to the same forwarding function, so routing, canary arithmetic, load balancing, header rewriting, retries, per-route counters, mirroring and the upstream pool are the ones already in use and cannot drift from them. What the HTTP/3 module owns is how bytes get on and off the wire, and nothing else.

Two consequences of that reuse are load-bearing:

The certificates are the TLS listener’s. The QUIC crypto configuration is built over the same SNI resolver — the same map in the same route table, the same store — so a name resolves to the same certificate over UDP as over TCP, and a rotation reaches both at the same instant because it is the same two pointer stores in the same order. A handshake that picked differently depending on transport would be a spectacular way to fail.

Bodies. hyper’s incoming-body type has no public constructor, so the forwarding function takes the crate’s own body type and the TCP path converts at the call site. That is the whole reason the signature is what it is.

Deciding whether an HTTP/3 request has a body

HTTP/3 has no Transfer-Encoding and no framing outside the stream: a request has a body if and only if DATA frames arrive before the client finishes the stream. content-length answers it when a client sent one.

When none did — and no GET does — the alternative to guessing is one non-blocking poll of the request stream. A client that has already finished it, which is every ordinary GET by the time its packets arrive, is recognised immediately: the body is known-empty, the request is retryable across endpoints, and the origin sees an ordinary GET rather than one carrying Transfer-Encoding: chunked. A client that has not is not waited for — the poll returns pending, the body streams, and the first DATA frame goes upstream when it arrives.

One endpoint, on one runtime

This is the honest reason the feature is experimental.

The TCP data plane is one runtime per core with SO_REUSEPORT spreading accepts. The obvious transliteration — N UDP sockets on one port, one QUIC endpoint each — is wrong, and quietly.

The kernel chooses which SO_REUSEPORT socket receives a datagram by hashing its 4-tuple. A QUIC connection is not identified by its 4-tuple; it is identified by a connection ID, precisely so it can survive the client’s address changing — a phone moving from wifi to cellular, any NAT rebinding. Under 4-tuple hashing, the moment a client’s address changes its packets land on a socket whose endpoint has never heard of that connection, and the connection dies. Migration is one of the few things QUIC has that TCP does not, and sharding this way trades it away.

Doing it properly needs the kernel to steer by connection ID — on Linux, an eBPF SO_REUSEPORT program. So for now there is one endpoint on one dedicated thread, with an upstream pool of its own, and the ceiling that sets is one core’s worth of QUIC crypto, packet handling and proxying.

That is stated rather than measured. HTTP/1.1 and HTTP/2 keep every core they had, so this is not the path to put peak traffic on yet.

Which load balancers can carry it

The alt-svc header is the whole mechanism, and it is also the whole constraint. A client that reads it retries the same authority over QUIC, so the port number it is already using for TCP has to answer UDP too, through every hop in front of the pod.

Getting it wrong is slow rather than broken. A client whose QUIC attempt fails falls back to TCP by itself — the cost is one wasted attempt per connection until the advertisement expires, which is why ma is a day and not a week.

No provider preset turns this on, because whether UDP reaches the pod is a property of an account’s networking rather than of a provider.

The client address, behind a balancer

The PROXY protocol does not apply. It is a preamble on a TCP byte stream and has no UDP form, so a QUIC connection’s client address is whatever the IP header says.

• On a balancer that forwards UDP without rewriting the source, that is the real client.
• On one that SNATs it, X-Forwarded-For on HTTP/3 requests will name the balancer while the TCP path is still correct.

There is no configuration that fixes the difference.

Draining

SIGTERM stops the endpoint accepting, and each live connection sends GOAWAY and then finishes the requests already on it, inside the same grace period the TCP listeners get.

In-flight requests are counted here rather than left to the HTTP/3 library’s own bookkeeping, and that is not redundancy: its accept loop yields “done” only once every request is complete and a GOAWAY has been received — the peer’s, not ours. A server that sent GOAWAY and then waited for that would be waiting for the client to hang up, and after a GOAWAY every client is idle by definition. Every shutdown with an open HTTP/3 connection would burn the whole grace period and then report a timeout.

What is not supported

• No 0-RTT. Early data is explicitly disabled. It is replayable by anyone who captured it, and which requests are safe to replay is an application’s judgement, not an ingress’s.
• No QUIC upstream. Upstream is HTTP/1.1, as it is for every other downstream protocol here.
• No PROXY protocol, which has no UDP form.
• No protocol upgrades. WebSockets over HTTP/3 are RFC 9220 extended CONNECT, a different mechanism from a 101; an upstream that answers 101 to a request that arrived over QUIC gets the same 502 any half-completable upgrade gets.
• No h3 datagrams, no WebTransport, no server push.
• --engine uring refuses it at startup, because that engine has neither TLS nor QUIC. So does --no-https: there would be no port to take and no response to advertise on.

Metrics

ramjet_h3_connections_total
ramjet_h3_requests_total
ramjet_h3_handshake_failures_total

An HTTP/3 request is also counted in ramjet_requests_total like any other, because it is one.

Engines

There are two data planes. They read the same route table, resolve certificates from the same store, write the same /metrics, and answer requests the same way. What differs is how they move bytes.

--engine uring selects the second one. --engine uring-strict selects it and refuses to start if the host will not run it, rather than falling back.

What each one does

Every row here is covered by a test, and the ones marked same are covered by a differential test that drives both engines with identical traffic and compares the answers.

Two rows are worth reading twice.

Graceful drain. Both engines stop accepting on SIGTERM and then wait up to --shutdown-grace — 30 seconds by default — for what they are already serving to finish. The rules are the same rules on both, because they are the ones a client can observe:

• connections that are idle between requests are closed at once, and the response to a request that is in flight carries Connection: close;
• a request counts as in flight until its exchange ends, in either direction: a body still arriving and a response still streaming are both unfinished;
• upgraded tunnels — WebSockets — are not drained. Once a connection has been upgraded there is no request boundary left to finish at and no bound on how long it will live, so waiting for one would stall every rolling update until the deadline and then kill it anyway;
• a drain that reaches the deadline with connections still open closes them, logs shutdown grace period expired, and still exits zero. A rolling update is not a crash.

With --engine uring and HTTP/2 dispatch on, both lanes are signalled at the same instant and drain inside one deadline rather than one after the other.

crates/ramjet-engine/tests/lifecycle.rs asserts each of those on the reactor, and two cases in the differential test assert that the two engines answer an in-flight request — and report an expired deadline — identically.

HTTP/3. It stays on the hyper engine’s QUIC listener. --http3 with --engine uring is refused at startup rather than ignored.

HTTP/2 on the uring engine

The uring engine speaks HTTP/1.1. Rather than not offering HTTP/2 at all, it offers it and hands those connections to a hyper engine running in the same process.

This is possible because of where rustls lets a server stand. A rustls::server::Acceptor reads the ClientHello and stops — before a configuration is chosen, before a byte is written back — and the ALPN list the client offered is readable at that point. So the decision happens while the connection is still nobody’s:

• the client offered http/1.1, or no ALPN at all → served here;
• the client offered h2 → the socket and every byte read from it go to the other engine, which replays them and finishes the handshake itself.

From the client there is no reset, no second handshake and no retry. It sees one connection, which negotiated HTTP/2. On the plaintext listener the HTTP/2 prior-knowledge preface (PRI * HTTP/2.0) is handed over the same way.

Two counters say which way traffic went:

ramjet_dispatch_uring_total   connections kept after reading the ClientHello
ramjet_dispatch_hyper_total   connections handed over because the client asked for h2

/metrics sums both engines, so the numbers describe the process rather than one half of it.

--no-h2-dispatch turns this off. The TLS listener then advertises http/1.1 alone and an HTTP/2 client negotiates HTTP/1.1 with it — which works, and is what every browser falls back to, but costs multiplexing. It also means the second engine’s threads and upstream pools are never started, which is the reason to turn it off.

HTTP/2 upstreams are the hyper engine’s alone

The dispatch above moves a downstream connection between engines. The backend protocol is a property of the route, and the two engines do not share an upstream pool — the uring engine has its own, written against its own sans-io codec, and it dials HTTP/1.1 only.

So a route whose backend carries backend-protocol: GRPC is refused on the uring lane, in its own words:

502 Bad Gateway: this backend needs an HTTP/2 upstream, which the uring engine
does not dial; use --engine hyper

Distinct from the message a gRPC request gets when its backend was never annotated, and deliberately so: that one says add the annotation, this one says the annotation is right and this engine cannot honour it. Two problems, two fixes, two sentences.

Two tokens, too. A body is for the person who ran curl; it is not in an access log and not in a client library’s error, so each refusal also carries a header and moves a counter:

No other response carries x-ramjet-unsupported, and an ordinary 502 — an upstream that hung up, a connect that failed — carries none, so a check for it is a check for equality rather than a guess. Both counters exist on both engines and the h2c one is permanently zero on hyper, so a dashboard does not lose a line when an operator changes engine.

The refusal is route-level rather than request-level. Any request to that backend gets it, not only the ones with a gRPC content type — because the backend was declared to speak HTTP/2, and sending it HTTP/1.1 anyway is the silent downgrade the annotation exists to prevent.

A cluster serving gRPC wants --engine hyper. The hyper engine carries the whole matrix: HTTP/1.1, HTTP/2 and HTTP/3 clients all reach an h2c backend, with trailers and bidirectional streaming intact.

Falling back

io_uring_setup is blocked by Docker’s default seccomp profile, and by containerd’s. Whether a given cluster allows it depends on the node image, the container runtime, and the pod’s own seccomp profile. The last of those is a chart value; the first two are not something a chart value can know.

So --engine uring asks the host before anything binds — one ring’s setup and teardown — and serves on hyper if the answer is no:

WARN the ramjet reactor will not start on this host; falling back to the hyper
     engine error=Operation not permitted (os error 1) requested="uring"
     serving="hyper"

The reason is always logged with the errno behind it. The two causes an operator actually hits — a kernel older than 5.6, and seccomp — are told apart only by which error comes back.

--engine uring-strict refuses to start instead. That is for a deployment that would rather crash-loop visibly than serve on an engine it did not choose: a silent fallback has none of the properties its operator picked it for, and no obvious sign anything happened.

On macOS and BSD the reactor is kqueue and this never comes up.

/metrics deliberately does not say which engine is serving. A series naming the engine would be the easy way to report it, and it would make every dashboard engine-specific in exchange. The startup log says it once instead.

Checking which engine a replica chose

Both engines name themselves, in the same field, on the line the process writes first. Which engine is serving is never something to infer from a field being absent — absence is also what a truncated line, a log shipper dropping a key, and an older build all look like.

In Kubernetes mode that field is on the startup INFO:

INFO ramjet_ingressd::kubernetes: starting in kubernetes mode version="0.1.0"
     engine="uring" ingress_class=ramjet namespace="<all>" … cores=4

$ kubectl logs -l app.kubernetes.io/name=ramjet-ingress \
    | grep 'starting in kubernetes mode'

With --static-routes it is the startup banner, saying the same thing in prose:

ramjet-ingressd 0.1.0 — engine hyper, 3 backend(s), 6 endpoint(s), 4 route(s), 0 certificate(s)

A replica that fell back reads hyper here — and says why on the line above.

Two clusters where it does fall back

Both measured rather than assumed.

Docker Desktop’s Kubernetes. deploy/e2e.sh was run against it with ENGINE=uring, and the pod reported:

WARN the ramjet reactor will not start on this host; falling back to the hyper
     engine error=Operation not permitted (os error 1) requested="uring"
     serving="hyper"

io_uring_setup returns EPERM inside that kubelet’s containers. The whole suite then passed on the hyper engine — routing, canary split, TLS with SNI, per-route stats, mirroring, auto-promotion — which is the outcome the fallback exists to produce: a replica that serves rather than one that crash-loops because of a syscall policy nobody set deliberately.

k0s on EC2, with containerd 2.3.3. A single-node k0s cluster on a t3.xlarge, Ubuntu, kernel 7.0 — and the same EPERM. This one is worth spelling out because every part of it except the pod’s seccomp profile was willing: the kernel is far newer than the 5.6 the reactor needs, and the host had kernel.io_uring_disabled=0, so io_uring was not switched off anywhere on the machine. containerd’s default seccomp profile — which the chart asks for, as seccompProfile.type: RuntimeDefault — is the whole of what blocked it.

So this is not a Docker Desktop quirk, and not a VM quirk. A stock containerd cluster on real hardware falls back too, and it does so with a kernel that would have run the reactor happily.

The same syscall is permitted in the plain Docker daemon on the same machine once the seccomp profile allows it, which is how bench/engine/ measures the reactor at all. The difference is the pod’s seccomp profile, not the kernel — so on a cluster where you control that profile, uring will run.

Turning it on, and what it costs

On the k0s cluster above, one pod-level value was enough:

podSecurityContext:
  seccompProfile:
    type: Unconfined

or --set podSecurityContext.seccompProfile.type=Unconfined, which Helm merges over the chart’s default and leaves the rest of the pod’s security context (runAsNonRoot, the uid, fsGroup) intact. For the pre-rendered manifests in deploy/static/provider/, it is the pod spec’s securityContext.seccompProfile block. Pod level is sufficient — the container-level securityContext needs no change, and the reactor started with only this.

Be clear about what that value does: it does not unblock io_uring_setup, it removes the syscall filter. RuntimeDefault denies several dozen syscalls, of which the three io_uring ones are a small part; Unconfined denies none of them. Everything else the chart sets still applies — non-root uid, no capabilities, read-only root filesystem — but the kernel-level filter that contains a compromise of this process is gone, and it is the ingress controller, which is to say the process with the most exposure to unauthenticated traffic in the cluster. That is a genuine security tradeoff for the throughput in Performance, and on most clusters it is not worth making.

The narrower fix keeps the filter. A Localhost profile — the runtime’s default deny list plus io_uring_setup, io_uring_enter and io_uring_register — gives the reactor exactly what it needs and nothing else, and is what bench/engine/ runs under. The chart does not ship one because a Localhost profile is a file that has to exist on every node before the pod referencing it will schedule, which is a node-provisioning job rather than a chart value. On a cluster with an opinion about syscall filters, that is the option to reach for; Unconfined is the one that gets you an answer in an afternoon.

Mirroring and the request body

Both engines mirror. They differ in when the copy is taken, and the uring engine’s way is the better one.

The hyper engine reads the request body up to --mirror-max-body before dispatching the primary, because it has to: a body is a stream it can consume only once, so the copy has to be taken before the original is handed to the upstream client. A mirrored request with a body therefore waits for its body to be buffered.

The uring engine already moves those bytes through a buffer on their way upstream, so the copy is taken as they pass and queued when the body ends. The primary waits for nothing.

One case goes the other way. A chunked request body is forwarded verbatim on the uring engine — chunk framing and all — so the bytes going past are the body’s encoding, not the body. Sending those as a self-framed copy would double-encode them, and decoding a body this engine deliberately does not decode is not a trade worth making for a copy. Chunked request bodies are counted in ramjet_mirror_skipped_total and no copy is sent.

The differential test

Two engines that are supposed to be indistinguishable cannot be tested by asserting either one against a literal: that is a test which keeps passing after the other one drifts.

So crates/ramjet-engine/tests/differential.rs starts both, drives them with byte-identical requests against byte-identical route tables, and compares:

1. the status and the body;
2. the whole rewritten head the upstream received, field by field — which is what catches a header written in a different order, a different case, a missing X-Forwarded-Host, or an X-Forwarded-For that replaced the trail instead of extending it;
3. the counter deltas, scraped before and after and subtracted.

The one field that must differ is X-Request-Id when the client sent none: it is 32 random hex characters by design, and two engines agreeing on it would mean the randomness was broken. Its presence and shape are compared; an inbound id is compared exactly, which is what actually matters.

crates/ramjet-engine/tests/exposition.rs does the same for /metrics: both counter sets driven through the same events, and the two strings asserted equal.

Choosing one

Run hyper unless you have a reason not to. It is the default, and it is what the numbers in Performance were measured on for every release before this one.

Run uring when the replica is CPU-bound on a Linux host that permits io_uring, and the traffic is HTTP/1.1 or HTTP/2 over TLS. That is where the completion-based reactor’s fewer syscalls per request show up; see Performance for the measurements and bench/engine/RESULTS.md for the protocol behind them.

Run uring-strict when a fallback would be worse than a crash loop.

Performance

Four benchmark documents live in the repository, and this page condenses them. Every table below is reproduced from a committed measurement; the raw JSON, the version manifests and the diagnostics are in-tree so any of it can be re-derived without re-running anything.

Read this first

Most of these are macOS Docker Desktop VM numbers, on Apple Silicon under a linuxkit guest. They are valid relative to each other under identical conditions; they are not Linux bare-metal absolutes and should not be quoted as such. Two sections are the exception and say so where they appear: the thesis re-run on EC2 and k0s, and the uring engine’s real-Linux check. Both carry their own caveat — a shared, burstable four-vCPU box with the load generator on it — which compresses ratios rather than inflating them.

The competitor is not understated on purpose. ingress-nginx does not reload for every change — endpoint updates go through its Lua balancer without touching nginx at all — and a report that measured only the changes which force a reload would be describing a system that does not exist. The endpoint-churn arm is measured and reported alongside the rest.

The losses are on this page. Idle-connection memory, the kubectl apply write path, upstream connection reuse, and an unexplained 9% gap at high concurrency all belong to the other side.

The thesis: what a configuration change costs

Both controllers installed into the same Docker Desktop cluster at the same time, separate namespaces, separate IngressClasses, one replica each. Load is never sent to both at once. ingress-nginx is the more generously provisioned of the two: it runs with no memory ceiling while ramjet-ingress runs under the 256Mi cap its own chart imposes.

Load reaches the pods through a NodePort on the node’s own bridge address — identically for both. kubectl port-forward was tried and rejected, because a port-forward is one multiplexed stream through a Go proxy on the host and becomes the bottleneck long before either contender does.

Two kinds of churn are measured, every two seconds for 100 seconds:

• Ingress-spec churn adds a differently-named path to a churn Ingress, which forces a reload. Verified from ingress-nginx’s own log, not assumed.
• Endpoint-only churn moves one running pod in and out of a Service’s selector, which goes through the Lua balancer without reloading. The mutation flips a label on a running pod rather than scaling, because scaling would have measured the scheduler’s latency and reported it as the controller’s.

Throughput and latency under churn (oha, c64, 2 runs per cell)

Idle keep-alive connections that survived the window

This is the cleanest result in the whole report, because it does not depend on how fast the machine was. Under spec churn ingress-nginx ended every single run with 0 of 50 idle connections surviving — 0/100 across both rounds, and 0/50 again in each of the two contended replicate rounds. ramjet-ingress kept 100 of 100.

Controller cost of the churn window

Recompiling and republishing a route table 49 times in 110 seconds cost the data plane nothing this benchmark can find. The CPU column is the version of the claim that survives, because it normalises out how fast the machine was.

Where ingress-nginx is fine: endpoint churn. It did not reload once — 0 reloads across every endpoint-churn run, confirmed from its own log — and it kept all 50 idle connections and served zero errors. Its Lua balancer does what it claims, and any characterisation of ingress-nginx as “reloads on every change” is wrong.

The +25% below has since been retracted. This run reported that ingress-nginx’s non-reloading path was the more expensive one for CPU — +25% against its own baseline, where the reloading path cost +10%. It was measured under the arm-ordering confound described in the next section, and when the EC2 run reversed the arm order the figure came back as +1%. The Lua balancer is close to free; treat the +25% as an artifact and not as a finding.

What benchmark 1 does not establish

The -26.8% endpoint-churn throughput figure is not solid. Arms always ran in the order baseline, spec, endpoint within a round, so the endpoint arm was always last and always held whatever drift the machine had accumulated. Rounds 3 and 4 were run specifically to test that, with the arm order reversed — and they were invalidated by the docker daemon: another agent started their own six-container proxy benchmark partway through, and throughput for both contenders fell to roughly a quarter, varying between 27k and 96k rps within a single round.

So the ordering confound on the throughput number is unresolved. The CPU-per-request figure is the version that survives; the connection-survival and reload counts reproduced identically in all four rounds regardless of contention.

The contended rounds are kept in the repository rather than discarded, with their own warning label: they report ramjet-ingress losing 62% of its throughput to spec churn, from a contender whose CPU per request did not move at all under the same churn on a quiet machine. That number is the other agent’s benchmark, not this one’s.

On real Linux, same opponent

Everything above is VM numbers, so the thesis was re-run against the same kubernetes/ingress-nginx 4.15.1 on an EC2 t3.xlarge running k0s v1.36.3 — real Linux, no VM in the path, three controllers standing in one cluster. Full report: bench/thesis/RESULTS-EC2.md.

The thesis transfers, at the same magnitudes.

Steady-state forwarding on the same box, median of three interleaved 30-second runs at c64, with the backend Service’s own NodePort as a no-proxy baseline:

Propagation of a new Ingress is 372–384 ms at the median against 2,762 ms, and the spread matters more than the median: twenty ramjet trials spanned 370–426 ms with no slow mode, while ingress-nginx’s ten spanned 398–2,813 ms in three clusters set by its --sync-rate-limit default of 0.3.

Two claims move, and both move ingress-nginx’s way. Reversing the arm order resolved the confound flagged above: endpoint-only churn costs it +1% CPU per request, not +25%, and −6.3% throughput rather than −26.8% — the same contention tax ramjet-ingress paid in the same arm. And the kubectl apply write path is now a draw, 191 ms against 189, with ingress-nginx still doing nginx -t validation through an admission webhook in that time.

Same caveat as the uring section below, and it is load-bearing. The load generator, the proxies and the upstreams all share four burstable vCPUs, so the proxy is never the sole bottleneck and every ratio on this page is compressed toward 1. Steal stayed under 1.1%. The connection-survival result is the one that does not care: 100 against 0, in both environments.

Propagation latency

kubectl apply to the first request the data plane answers correctly, polled every 20 ms, no other load running. Ten trials of each shape per contender, interleaved with the order flipping every trial. kubectl runs inside the same container as the poller, so “applied” and “served” are two readings of one clock.

~3x faster at the median and ~6x at p95, and the more useful half of that is the spread: ten new-Ingress trials from 324 to 566 ms, against 302 to 3,638 ms.

ingress-nginx’s slow trials cluster just under 3.5 seconds and alternate with fast ones. That shape is a rate limiter, not a queue, and the controller names the number itself: --sync-rate-limit defaults to 0.3, one sync per 3.33 seconds. Raising that flag would shorten this tail — it is a default, not a limit of the design — but the default is what a cluster gets. ramjet-ingress has a fixed 200 ms debounce and no rate limit.

A loss: the write path

The admission webhook is not the reason, and this is where ingress-nginx ties or wins. Median kubectl apply was 138 ms for ingress-nginx against 159 ms for ramjet-ingress — the write path including nginx -t validation of the whole generated configuration is faster than ramjet-ingress’s plain unvalidated write.

500 routes

500 Ingresses with distinct hosts, applied as one batch, one contender at a time.

Both reached 500. Neither choked. That is worth saying first, because the brief allowed for reporting the number at which one of them fell over.

• 5.7x faster convergence, of which most of ingress-nginx’s time is the write path: roughly 117 ms per Ingress against 21 ms. The admission webhook that was free at one Ingress is not free at 500.
• Controller CPU differs by a factor of 100: 0.66 CPU-seconds against 66.7.
• Memory is the sharper result. 21.1 → 20.8 MiB: 500 compiled routes are, within measurement noise, free. ingress-nginx grew 98 MiB, roughly 200 KiB per route. Under the 256Mi limit ramjet-ingress’s own chart ships, ingress-nginx would have been within 40 MiB of being OOM-killed at 500 routes; it survives because its chart ships no limit at all.

Propagation with the routes loaded:

Every ingress-nginx trial at scale was slower than its own worst trial on an empty cluster.

The throughput row of this benchmark is the weakest number in the whole document and is deliberately not reproduced here. 47,535 rps against 16,109 is a 3x gap, but the two measurements were taken minutes apart at 38% and 25% VM CPU idle respectively, on a machine that was doing someone else’s work. It is one run each under unequal conditions and should not be quoted as a throughput result.

Deleting 500 Ingresses took about 105 seconds for each. A tie, and API-server bound rather than controller bound.

Idle-connection memory: the loss

ingress-nginx wins this one decisively, and it is the most important negative result in the report.

10,000 idle keep-alive connections, no Kubernetes, both proxies on the same docker bridge with the same upstream and nginx’s own tuning. Two passes, order reversed between them.

Originally

An idle connection cost nginx 4.4 KiB and ramjet-ingress 27.1 KiB — 6x — and ramjet-ingress did not give the memory back, growing monotonically across connect/disconnect cycles. At 266 MiB peak it would have been OOM-killed by the 256Mi limit its own Helm chart ships, with no traffic flowing.

After the fix

The retention problem is gone: a second full cycle peaked at 201.5 and settled at 11.5 again — the same number, not a higher one. ramjet-ingress now also idles lower than nginx does, 11.5 MiB against 16.5.

The per-connection cost improved by a quarter and ingress-nginx still wins it. 27.1 KiB to 20.3 is real, and 20.3 against 4.4 is still 4.6x. The gap is structural.

The original table is left in the repository exactly as it was — a benchmark that overwrites the evidence it was judged against cannot be checked afterwards.

Where the remaining 20.3 KiB goes

Measured at 2,000 connections:

The ~4.4 KiB gap between the columns on a merely-accepted connection is kernel socket memory, which cgroup v2 charges to the container. That is very nearly nginx’s entire per-connection cost, which is the sharpest way to state the difference: nginx’s 4.4 KiB is, to a first approximation, the socket and nothing else. It hands a connection’s request buffers back to its pool when the connection goes idle and keeps only the connection object. There is no equivalent in hyper.

Two checks that pin it down: sending 6 KiB of request headers instead of 90 bytes moved the figure by two bytes (16,927 against 16,929) — the read buffer is resident whether or not anything is read into it. And patching hyper’s INIT_BUFFER_SIZE from 8192 down to 1024 gave 11.3 KiB cgroup and 7.3 KiB RSS; two 8 KiB buffers becoming two 1 KiB ones accounts for 9.6 KiB, and nothing else moved.

There is no public API that lowers it: max_buf_size caps how far the read buffer may grow, and hyper refuses to set it below INIT_BUFFER_SIZE. So 16 KiB per idle keep-alive connection is this engine’s floor until hyper’s initial allocation follows its configured maximum instead of a constant. The patched measurement is what that change would be worth: roughly 2.5x nginx instead of 4.6x. It is a one-line change in a dependency, and the right place to make it is upstream.

The experimental uring engine was measured on the same harness and is not cheaper: 23.2 KiB per connection, because it allocates per-connection buffers of its own.

What this means for the chart

resources.limits.memory: 256Mi stays, and the values file now carries the arithmetic instead of leaving it to be rediscovered — about 20 KiB per idle keep-alive connection, so 256Mi is roughly twelve thousand of them. Raising the default to make room for a per-connection cost that is still 4.6x nginx’s would have hidden the finding rather than fixed it.

Raw forwarding throughput vs nginx

A forwarding-engine drag race: one route, one host, 128-byte plaintext responses, static configuration. Both proxies pinned to the same two cores with --cpuset-cpus=0,1 (a CPU quota is invisible to sched_getaffinity, so pinning is what makes both see 2 CPUs and start 2 workers), oha 1.16.0 at HTTP/1.1 keep-alive, a discarded 10s warmup, then 3 × 30s at c64 and 1 × 30s at c256, interleaved. The c64 rows are the median-throughput run, not a per-column average, so every number in a row comes from one real 30-second measurement.

Concurrency 64 (median of 3 × 30s runs)

Concurrency 256 (single 30s run)

At c64 the two are level. 85,908 against 86,670 is a 0.9% difference, and nginx’s own three runs spread 4.5% — the gap is smaller than the noise in the measurement, which means this benchmark can no longer tell them apart at this concurrency. It does not mean ramjet-ingress is faster; the honest statement is “the same”. Divide two cores by throughput and both spend 23.6 µs of CPU per request.

At c256 nginx is still 9% ahead, and that gap is outside the noise. nginx’s throughput barely moves between c64 and c256 (+3%) while ramjet’s drops 4%. Latency runs the other way — ramjet’s p99 at c256 is 6,396 µs against nginx’s 7,683 µs — so what this looks like is ramjet trading a little throughput for shorter queues under saturation, not falling over.

Where it started

The first measurement of the same benchmark had nginx 45% ahead, and it is kept in the repository unchanged. What closed it:

That last row is a real cost and is reported as one: one runtime per core means one connection pool, one timer wheel and one set of hyper buffers per core rather than per process. On a 2-core replica that is 14 MiB; on a 64-core node with no CPU limit it would be considerably more, which is an argument for setting --worker-threads deliberately rather than letting it follow the host. (The later memory work brought this to 24.9 MiB.)

Zero errors across all twelve runs, both before and after: 49,114,324 requests, every one a 200, no transport errors from any contender at either concurrency.

Method honesty on this benchmark

Both head-to-head tables were taken with a fixed within-round order. run.sh has since been changed to rotate which contender leads each round, and to wait a 15s cooldown before each warmup — because plain interleaving assumes the machine is steady within a round, and on a laptop it is not: the package heats up as the round proceeds, so a fixed order hands whoever goes first a systematically cooler machine in every round. Neither table has been re-measured under the rotated protocol, so read the numbers as carrying that bias in ramjet’s favour at c64, bounded by the within-round drift (the baseline’s 13.3% spread is the visible upper bound; the contenders’ 1.7–5.3% the likelier scale).

Other stated unfairness:

• The upstream keepalive pools were not equal in the first measurement. nginx held 128 idle upstream connections, ramjet 64 — the edge was nginx’s, and it was left alone rather than patched, because changing the product to win its own benchmark is not a measurement.
• nginx tuning choices were tested, not assumed. reuseport was measured both ways and kept because it is better for nginx. access_log off removes nginx’s default per-request write, which ramjet does not have. proxy_cache was deliberately not enabled: ramjet has no response cache, and serving from nginx’s memory would compare two different jobs.
• Both are round-robin, matching nginx’s default, rather than ramjet’s leastConn.
• A shared docker daemon, and it bit. The reported 30s runs absorb it, and the contender spread is the evidence.

What this does not test

This is a forwarding-engine drag race on the narrowest possible workload: one route, one host, 128-byte responses, plaintext HTTP/1.1, static configuration. It says nothing about the project’s actual thesis — that a config change is a pointer swap rather than an nginx reload. Nothing here exercises TLS termination, HTTP/2, large or streaming bodies, thousands of routes, or configuration churn under live traffic.

That paragraph predates the optimization and still stands unchanged. It is the more important one on the page.

Where a request actually goes

Profiling asked where the 10 µs gap lived, and the answer was not in the forwarding code. Route matching, header rewriting, URI building and the metrics counters together account for about 2% of a request.

The router’s 25 ns match is 0.1% of that. There was no hot function to find.

What the profile found instead was the runtime moving each request’s work between cores:

The same code costs 43% more CPU per request on two threads than on one, held at 33–43% across three interleaved rounds. That is the shape of a work-stealing scheduler under a request that ping-pongs between workers; nginx does not pay it, because its workers are shared-nothing processes.

So the data plane became one current_thread runtime per core with nothing shared between them. Everything else that was tried was inside the noise and is recorded as such:

The floor

After the change the profile reads:

59.4% of a request is the four unavoidable syscalls, and another 9.1% is finding out a socket is ready. That is the floor for this design, and it is not a hyper problem or a tokio problem — it is the I/O model.

Getting under it means fewer syscalls per request, which on Linux means io_uring.

The remaining 9% gap at c256 has not been profiled. Every measurement was taken at c64, and the native harness cannot hold c256 steady enough to be worth reading. Whether it is queueing, the per-runtime pool split, or something else is an open question.

The uring engine

A second data plane on a completion-based reactor, selected with --engine uring. Docker on Linux, --cpuset-cpus=0,1, the same pair of upstreams, oha at c64, three rotated rounds of 30 seconds each. Both ramjet rows are the same image with one flag different.

Concurrency 64 (median of 3 runs)

Concurrency 256 (single run)

+44.7% over nginx at the median, and the proxy hop costs 255 µs where nginx’s costs 469 µs — it keeps 51% of the no-proxy throughput where the other two keep 35%.

The claim that survives the drift

The machine would not sit still: the baseline, which has no moving parts and nothing under test, spread 15.1% across three rounds. Drift makes a median shaky. It does not touch a rank-order claim:

Every measured uring round beat every measured round of both rivals; the ranges do not overlap. “At least 31% ahead of nginx” is the claim that survives, and +44.7% is the median’s reading of the same thing. report.py makes this check itself and refuses the run if the ranges ever overlap.

The hyper row is not under-measured relative to nginx: here it is 0.13% from nginx, so +44.9% for uring over hyper is the same result as +44.7% over nginx rather than an artifact of a cold hyper.

And a cross-day check that cuts the other way

Comparing this session against the committed head-to-head runs — taking both cells from the same run, which an earlier version of the engine document failed to do — puts nginx’s row here on the low side:

The ratio travels across days at the few-percent level — 2.6 points against the nearer comparator, 1.0 against the older one — which is enough to trust this session’s ordering, and not enough to swap an absolute row for another day’s.

The slowdown was not uniform, and that is the part worth carrying: this session’s baseline is within 0.2% of 4f58bd7’s, while its nginx is 6.8% lower and its hyper engine 6.1% lower. Whatever cost the two TCP proxies those points did not cost the no-proxy baseline anything.

So +44.7% is the optimistic end of the margin rather than the middle of it. Against the best committed nginx median, uring’s worst round is +28%; against this session’s own best nginx round, +31%. At least 28% ahead is the figure that survives every pairing, and +44.7% is what you get comparing contenders measured in the same session on the same host.

Why: the syscall counters

cqes_per_waiting_enter    = 21.7 … 39.5   (typically 28–37)
enter_share_of_thread_cpu = 0.81

Between 22 and 40 completions are harvested per trip into the kernel. A request is four operations, so that is roughly seven to ten requests per syscall, against the hyper engine’s four syscalls per request plus a kevent to learn a socket was ready. The 81% is the share of the serving thread’s CPU spent inside io_uring_enter, which is where the kernel actually does the reads and writes — it is not overhead, it is the work.

Cost per request

49% more requests for the same CPU as the hyper engine, and less than half its memory. Two honest readings alongside that:

• nginx did not saturate its cores in this pass (174% of an available 200%), so its 21.2 µs is a fair figure for what it spent but its throughput here may have been limited by something other than CPU.
• A loss: nginx reuses upstream connections six times better — 61,665 requests per connection against 9,947. Per-core pools are the reason, and the price was named when they were introduced: a connection returned to a full pool on one core cannot be reused by another. It is not costing throughput here, but it is a real difference and it is nginx’s win. nginx is also, by a wide margin, the most memory-frugal of the three.

The macOS negative result

The same binary with the flag flipped, on the native macOS harness:

  A (hyper)  median 50,738 rps   spread 30.8%
  B (uring)  median 53,024 rps   spread 18.1%
  B vs A: +4.5%   (inside the noise)

+4.5% against an 18–31% spread is not a result, and the harness says so itself. That is the prediction, not a disappointment: on macOS the reactor’s backend is kqueue, which performs each syscall eagerly at submission. There is no ring, no batch, and nothing to collapse. The whole benefit measured on Linux is io_uring’s, so the platform without io_uring measures none of it.

Caveats on the uring numbers

This is a macOS Docker Desktop VM, and that matters more here than for any other benchmark in the repository. The whole result is about the cost of entering the kernel, and a syscall in a virtualised linuxkit guest is dearer than one on bare metal. io_uring’s advantage is the cost it avoids, so a more expensive syscall flatters it. Treat the margin as an upper bound and the direction, not the size, as the transferable claim.

• nginx runs under Docker’s stock seccomp profile; both ramjet containers run under a pinned one (moby v24.0.7’s default plus the three io_uring syscalls). It is a superset of what nginx needs so it cannot disadvantage nginx, but it is a difference between contenders.
• The two engines were not feature-equivalent when this was measured. The uring engine served HTTP/1.1 plaintext and nothing else. None of the missing features is exercised by this workload, so the comparison is like for like for this traffic. It was not a claim that the engines are interchangeable. Most of that gap has since closed — see TLS, and a tunnel below and Engines.
• c256 is a single run, not a median, and is reported as such.
• The correctness gate compares the two engines’ response headers field by field, so neither can be fast by doing less.

On real Linux

The caveat above says to treat the margin as an upper bound and the direction, not the size, as what transfers. This is the check on that, and it comes out exactly as the caveat predicted. A t3.xlarge EC2 instance running k0s — real Linux, no VM in the path — with the same two engines and the same flag difference:

Baseline with no proxy in the path was 22,363 rps.

+5.8%, against +44.9% in the VM. The direction transferred and the size did not, which is the whole of what the caveat asked to be believed.

The share-of-baseline column is where the mechanism shows:

uring held its share almost exactly; hyper gained twelve points. So the VM was not flattering the reactor so much as it was punishing the other engine. The hyper engine pays four syscalls per request plus a kevent, a virtualised syscall is dearer than a native one, and taking the VM away refunds most of that to the engine making the most calls. The reactor, which was already avoiding those calls, had little to be refunded.

Per busy CPU the gap is smaller still, roughly +2.4%, where busy is 100 − idle:

The us/sy split underneath is uring 35/47 against hyper 41/44 — the reactor spending more of the machine in the kernel and less in userspace, which is the same shape as the io_uring_enter share above and means the same thing: for this engine the kernel time is the work. Those two do not sum to busy; the remainder is wa, st and rounding, which is why the busy column is taken from idle rather than by adding them up.

Every figure in that table counts the load generator and the upstreams as well as the proxy, because all three were on the one box. So +2.4% is a whole-machine efficiency number rather than the engine’s own. Getting the engine’s own needs proxy-only CPU seconds, which this run did not capture — the same reason the run is not the one to quote from at all:

The caveat that matters more than any of the numbers: this run was CPU-contended, and the contention structurally compresses the gap between the engines. Four shared vCPUs, with the load generator on the same instance as the proxy and the upstreams. The proxy was therefore never the sole bottleneck, and a benchmark where the thing under test is not the limiting factor understates every difference between two versions of it. A rerun on pinned, isolated cores with the load off-box is the number worth quoting, and this is not that run. Treat +5.8% as a floor under contention rather than the engine’s ceiling.

The p99 inversion, unexplained

On the same real-Linux run the reactor wins throughput and the median and loses the tail: p99 is about 6.8% worse than the hyper engine’s, and it was worse at both c64 and c256. Consistent enough not to be noise, and not currently explained.

It is not the shape the VM measurements had, where uring led at every percentile including p99.9. Until someone can say why, no tail-latency claim is made for the reactor — see Limitations.

What is not claimed

• Not that io_uring beats epoll in general. This is one proxy workload, one kernel, one VM.
• Not that the uring engine is ready to deploy. At the time of this measurement it had no TLS, no HTTP/2, no upgrades, no Kubernetes mode and no graceful drain. That list is now down to HTTP/2, which is served by dispatch.
• Not that the hyper engine is badly written. Profiling took it to the syscall floor, and this is what is underneath that floor. The difference is the I/O model, which is what was being tested.

TLS, and a tunnel

The measurements above are plaintext HTTP/1.1, which is what the uring engine could serve when they were taken. It terminates TLS now. Same machine, same topology, same pinning, same rules — one certificate added.

Both sides resume sessions, which is the setting that decides a TLS benchmark: nginx ships ssl_session_tickets on and every deployment turns on ssl_session_cache, so a run against a ramjet with resumption off would be measuring a configuration nobody deploys. ECDSA P-256, one certificate generated per run and mounted into all three containers, HTTP/1.1 on all three.

Keep-alive, 30s per run, three rounds, median by throughput:

A new connection per request, so every request pays for a handshake — which is what a rolling deployment or a reconnecting CDN does to a replica:

The margin over hyper survives TLS almost intact: 1.30x at c=64, 1.28x at c=256, against 1.30x on the plaintext run. Crypto is added work, but it is added to both ramjet contenders equally, and what separates them is still how the bytes reach the record layer rather than what happens inside it.

The handshake row narrows to 1.16x between the engines, and it should — a handshake is arithmetic, not syscalls. The 2.52x over nginx there is ring against OpenSSL as much as it is one proxy against another, and should be read as a statement about two TLS stacks.

WebSocket tunnels are level, and that is the expected result

64 tunnels, 128-byte payloads, one echo in flight per connection:

All three within 2.4%. A reader who saw 1.30x on the TLS table would expect a gap here, and there is none, for a reason worth stating: after a 101 there is no request, no routing and no header rewriting — one read and one write per echo on each side, with nothing to batch and nothing to overlap. The rate is bounded by round-trip latency rather than by how many times the kernel is entered, and submission batching is exactly the advantage that has nothing to work on.

What does separate them is steadiness: both ramjet engines hold a 1.2–1.4% spread across runs against nginx’s 7.1%, and the uring engine has the best p99.

Full protocol, raw JSON and the fairness notes are in bench/engine/RESULTS.md.

Route matching

Not a system benchmark — a microbenchmark of the matcher against a table of 1,000 hosts and 10,001 routes, on an Apple M2 Pro, criterion, 100 samples.

For scale, a single uncached main-memory reference is roughly 80 ns — matching a route costs less than one cache miss.

These are laptop numbers taken on a machine that was not otherwise idle, so treat them as an order of magnitude rather than a regression baseline. What the benchmark is really for is the shape: matching does not get slower with table size, because host selection is a hash and a host carries a handful of rules.

match_request performs no heap allocation, and that is enforced rather than asserted in a comment: tests/no_alloc.rs installs a counting global allocator and checks every path through the matcher, including the mixed-case fold, canary resolution, and SNI lookup.

Where ingress-nginx won or tied

Plus, from the raw-forwarding and engine benchmarks: nginx is still 9% ahead at c256, reuses upstream connections 3–6x better, and is by a wide margin the most memory-frugal of everything measured.

Reproducing any of it

./bench/run.sh                                   # ~15 minutes, cleans up after itself
python3 bench/report.py                          # re-render tables from committed JSON
IMAGES="ramjet:before ramjet:after" python3 bench/ab.py

bench/thesis/run-all.sh                          # the whole cluster suite
python3 bench/thesis/report.py
bench/thesis/teardown.sh                         # remove everything, and verify it

bench/thesis/ec2/setup.sh                        # the same thesis on a real k0s node
bench/thesis/ec2/run-all.sh                      # ~50 minutes
python3 bench/thesis/ec2/report.py
bench/thesis/ec2/teardown.sh

./bench/engine/run.sh                            # ~25 minutes
python3 bench/engine/report.py

cargo bench -p ramjet-router                     # the matcher microbenchmark

Raw data lives beside each harness: bench/results/ (current), bench/results/4f58bd7/ and bench/results/before/ (kept verbatim), bench/thesis/results/ including b4/ and b4-after/, bench/thesis/results-ec2/, and bench/engine/results/. Each archived run keeps its own versions.txt, diagnostics.txt and table.md.

Two harness rules worth knowing before you re-run anything:

• Do not shorten WARMUP below 10s, including in the smoke test. A 2s warmup measured 39,000 rps at c64 and 72,549 rps at c128 in the run immediately after — throughput rising with concurrency is the signature of a first run that had not finished warming.
• Check the host before trusting any number. On a quiet host the baseline measures ~248,000 rps; below ~230,000 the host is busy and the run is not worth starting. Host-side contention reaches the guest through vCPU preemption, so container --cpuset pinning does not protect against it. Overriding any tunable redirects output to results/scratch/ so a smoke run cannot overwrite a real measurement.

Limitations

Known gaps, each with the reason it is a gap rather than a bug. Read this before you deploy.

There is no leader election

Every replica watches the API server independently and writes Ingress status independently. Routing is unaffected by that — each replica compiles the same table from the same objects — but the status writes race: several controllers server-side-applying the same subtree under the same field manager will fight over .status.loadBalancer if their --publish-address values differ.

A Deployment from this chart therefore hard-codes replicas: 1, and there is no values entry to find at 3am. Scale by making the one replica bigger, and use --no-status-update if you must run more.

The chart’s default is a DaemonSet, where the node count is the replica count and that hard-coding has nothing to bite on. Every node’s replica is configured identically, so they write the same address through the same field manager and the patches converge — what it costs is volume, not correctness: API traffic of nodes x Ingresses, each on its own debounce. Set controller.updateStatus=false on a large pool and let the address be whatever DNS says it is.

The fix is a coordination.k8s.io Lease and gating the status writer on holding it. The writer is already isolated behind one optional value, so it is a contained change.

The image needs NET_BIND_SERVICE in the bounding set, on every port

The binary carries a cap_net_bind_service file capability, because that is the only way a non-root process binds :80 — see Deployment. The consequence runs the other way too: a file capability with the effective bit set makes execve fail with EPERM when that capability is outside the container’s bounding set, so a pod spec that drops ALL and adds nothing back will not start at all, on 8080 as surely as on 80. The kubelet reports it as exec /usr/local/bin/ramjet-ingressd: operation not permitted.

The chart adds it unconditionally and nothing else is relaxed — allowPrivilegeEscalation stays false, and the PodSecurity restricted profile permits exactly this one addition. It is a constraint on hand-written manifests, not on the chart.

The default install is still outside baseline, but for hostNetwork rather than for anything about capabilities. Where a namespace enforces that profile, use baremetal-nodeport, a cloud preset, or the default with hostNetwork off and the ports back above 1024.

There is no TLS to the upstream

The upstream side speaks two protocols — HTTP/1.1 by default, and cleartext HTTP/2 for a backend annotated backend-protocol: GRPC — and both are cleartext. That is the same default ingress-nginx ships, and inside a cluster it is usually what you want.

The consequence is which annotation values are honoured. HTTP and GRPC are; GRPCS and HTTPS are read, reported in a warning, and not honoured, because both mean “dial this pod over TLS” and there is no code here that does. AUTO_HTTP would need per-endpoint scheme detection, and FCGI is not HTTP. The backend stays on HTTP/1.1 in all four cases and the warning names the value, rather than the request being served against a protocol nobody asked for.

Lifting this means a client-side rustls configuration for upstream connections, with its own trust store and its own answer to what verifies a pod certificate.

gRPC needs one annotation, and is refused without it

gRPC over an HTTP/1.1 backend cannot work — gRPC is defined in terms of HTTP/2 streams and trailers and has no HTTP/1.1 form — so a request with an application/grpc content type whose backend is HTTP/1.1 is answered with a 502 that names the annotation to add:

502 Bad Gateway: gRPC requires an HTTP/2 backend; set
nginx.ingress.kubernetes.io/backend-protocol: GRPC on the Ingress

Add it and the request is forwarded like any other. This is a refusal to guess, not a missing feature.

The response also carries x-ramjet-unsupported: grpc-needs-backend-protocol and increments ramjet_engine_unsupported_grpc_total, so this shows up on a dashboard rather than as an unexplained rise in 502s. Both engines answer identically.

WebSocket does not cross an h2c backend

Connection and Upgrade are forbidden in HTTP/2, so an upgrade request sent to a backend annotated GRPC reaches the application as an ordinary request rather than as a handshake. WebSocket over HTTP/2 (RFC 8441 extended CONNECT) is not implemented in either direction.

This is only a constraint if one Service port serves both WebSocket and gRPC, which is unusual. Otherwise: WebSocket routes go to an HTTP backend, gRPC routes to a GRPC one, and both work.

ExternalName Services serve 503

Following a DNS name from the data plane needs a resolver with TTL handling and re-resolution; pointing at whatever the name resolved to at compile time would be a stale-address bug waiting for the first failover.

The annotation vocabulary is small

Canary, mirroring, auto-promotion, and class. The route table has no rewrite, header-mutation, rate-limit, session-affinity, or auth rules, so the corresponding nginx.ingress.kubernetes.io annotations are not read.

Those attach to a route when the proxy can act on them. Parsing an annotation the data plane ignores is worse than not parsing it, because it looks configured.

The full list of what is read is the annotations reference.

An IngressTLS entry with no hosts is skipped

The controller cannot read a certificate’s SANs to work out which names it covers — that would mean parsing X.509 in the control plane, which is exactly the dependency the layering split exists to avoid.

--default-tls-secret is the supported way to serve a fallback certificate.

The uring engine does not speak HTTP/2 itself

--engine uring reached parity with the hyper engine on TLS, WebSocket upgrades, the PROXY protocol, mirroring, per-route counters, Kubernetes mode and graceful drain. What is left is HTTP/2, at both ends of the hop.

It speaks HTTP/1.1. HTTP/2 is served by handing those connections to a hyper engine in the same process — the ClientHello is read before a configuration is chosen, so a client that offered h2 is passed over with its bytes intact and sees one connection that negotiated HTTP/2. That works, and it is on by default, but it means an HTTP/2-heavy deployment is running both engines and getting the reactor’s benefit on the HTTP/1.1 half only. --no-h2-dispatch turns the dispatch off, at the cost of not offering HTTP/2 at all.

HTTP/3 stays on the hyper engine’s QUIC listener, and --http3 with --engine uring is refused at startup rather than ignored.

HTTP/2 upstreams are the hyper engine’s alone. The uring engine has its own HTTP/1.1 upstream pool rather than sharing hyper’s, so a route whose backend is annotated backend-protocol: GRPC answers 502 there, naming the engine, and gRPC to it is refused with it. The h2 dispatch above does not help: it moves the downstream connection, and the backend protocol is a property of the route. A cluster serving gRPC wants --engine hyper.

That refusal is diagnosable without reading bodies: the 502 carries x-ramjet-unsupported: h2c-upstream and moves ramjet_engine_unsupported_h2c_total, which is the series to alert on if you run --engine uring at all — anything above zero means those routes are down on that replica.

Engines has the full parity matrix, and the differential test that keeps it honest.

The uring engine’s p99 is an open question

On real Linux the reactor wins throughput and the median and loses the tail. On a t3.xlarge k0s cluster it served 5.8% more requests per second at an 18.1% lower p50, and its p99 came out about 6.8% worse than the hyper engine’s — at c64 and at c256 both, consistently enough not to read as noise.

Nobody has explained it. It is also not a known property of the design being rediscovered: the earlier Docker measurements had the reactor ahead at every percentile out to p99.9, so something about this environment or this engine changed and the cause is not identified.

Until it is, no tail-latency claim is made for the reactor. A deployment whose SLO is written against p99 rather than throughput should stay on --engine hyper, or measure its own traffic before switching. The numbers, and the CPU-contention caveat that has to be read with them, are in Performance.

HTTP/3 is experimental and off by default

One QUIC endpoint on one runtime rather than one per core, no 0-RTT, no QUIC upstream, no upgrades, and no PROXY protocol. Each of those has a reason rather than a TODO — they are in HTTP/3.

The deployment-side constraint is separate and larger: alt-svc advertises the TCP port number, so that port has to answer UDP through whatever is in front of the pod, and most cloud load balancers cannot do that. Deployment has the per-provider answer.

Idle-connection memory is 4.6x nginx

An idle keep-alive connection costs this data plane about 20.3 KiB against nginx’s 4.4 KiB. The retention problem is fixed — the memory comes back on close, and a second connect/close cycle settles at the same number rather than a higher one — but the per-connection gap is structural.

About 16 KiB of it is hyper’s two 8 KiB buffers, and there is no public API that lowers it: max_buf_size caps how far the read buffer may grow, and hyper refuses to set it below its initial size. Patching that constant in a local build takes the figure to 11.3 KiB, so the fix is a one-line change in a dependency and the right place to make it is upstream.

The practical consequence: resources.limits.memory: 256Mi is roughly twelve thousand idle keep-alive connections. Budget accordingly, and see Performance.

No Gateway API

The target is parity with kubernetes/ingress-nginx on the networking.k8s.io/v1 Ingress resource.

Deliberate divergences from ingress-nginx

Three, and each changes behaviour you might be relying on:

• Regex anchoring. ingress-nginx emits location ~* "^<path>", a literal concatenation. This compiles ^(?:<path>). The two differ only for a top-level alternation, where ^a|b anchors just the first branch and routes traffic nobody intended. Case-insensitivity is preserved.
• Compiled regexes are size-limited to 1 MiB. A pathological path should fail validation, not silently consume memory in every replica.
• Host validation is strict. A host containing a port, a path, or a misplaced * is rejected at build time rather than normalized into a guess.

Building from source

The sibling repository

ramjet-engine depends on the ramjet runtime and its sans-io HTTP codec from a sibling repository by path, so the workspace expects that checkout beside this one:

.../
  ramjet-ingress/     <- this repository
  enhance-socket/     <- the ramjet runtime and ramjet-http

Without it, cargo refuses to load the workspace at all rather than skipping the crate. It is also why the container builds take the parent directory as their build context.

Build and test

cargo build --release
cargo test --workspace

The release profile is thin LTO, codegen-units = 1, and panic = "abort". That last one is not a size tweak: the data plane has no recovery story for a panicking worker, so unwinding past a half-written connection is worse than dying loudly. Cargo ignores the setting for the test and bench profiles, so cargo test and the criterion harness still build normally.

Minimum supported Rust version is 1.85.

The crates

crates/
  ramjet-router/      sans-io: route table, matcher, LB selection
  ramjet-proxy/       sockets, rustls, HTTP/1.1 + HTTP/2 + HTTP/3, upstream pools
  ramjet-controller/  Kubernetes informers, annotation translation, status
  ramjet-engine/      the experimental completion-based data plane
  ramjet-ingressd/    the daemon binary
  ramjet-top/         the terminal cockpit

Two dependency rules hold the design together, and they are worth understanding before changing anything.

ramjet-router depends on arc-swap, regex, and thiserror. Not on tokio, not on hyper, not on rustls. It never opens a socket, spawns a task, or reads a clock. Certificates are opaque handles, randomness is passed in as a number, and canary decisions take borrowed header values rather than a header collection. That is what makes the matcher testable against string literals and benchmarkable without a network.

ramjet-controller holds no rustls types either, for the mirror-image reason: parsing a certificate means a crypto provider, and a crypto provider in the control plane would mean the translation layer could no longer be unit-tested against objects built in memory.

The daemon is the only crate that depends on both sides — which is also why canary auto-promotion and the rollback-pin bridge live there.

Testing

cargo test --workspace
cargo clippy --workspace --all-targets -- -D warnings
cargo bench -p ramjet-router

Some things worth knowing about the test suite:

• translate is a pure function: cluster snapshot in, compiled config out, no I/O and no clock. Class filtering, path precedence, endpoint resolution, canary merging, and conflict arbitration all have unit tests that construct API objects in memory and assert on the compiled table.
• tests/no_alloc.rs installs a counting global allocator and asserts zero allocations across every path through the matcher. The counters are thread-local, because cargo test runs tests concurrently and a shared counter attributes one test’s allocations to another.
• ramjet-top’s mock-server tests run the real client against a real hyper listener serving canned admin responses, and assert the computed view model rather than anything about pixels. Its --once tests spawn the compiled binary and read its stdout, stderr and exit status.
• The auto-promotion state machine is a pure function — decide(policy, weight, window) — with no clock, no cluster and no counters, so the entire decision table is a unit test.

End to end

deploy/e2e.sh          # build the image, install the chart, assert routing
deploy/cloud-e2e.sh    # lint every preset, dry-run every manifest, PROXY protocol

Every kubectl and helm call in both scripts carries an explicit --context/--kube-context, and they refuse to run against a cluster that does not look local. A developer kubeconfig usually holds production clusters, and a mistyped current-context is exactly how a test script deletes one.

Benchmarks

See Performance for what each harness measures and the two rules that keep a re-run honest (do not shorten the warmup; check the host is quiet first).

Documentation

This site is mdBook. Source is in docs/src, and it is built and published to GitHub Pages by .github/workflows/docs.yml on every push to main that touches docs/.

cargo install mdbook      # or: brew install mdbook
mdbook serve docs         # live reload at http://localhost:3000
mdbook build docs         # output in docs/book

Where the prose and the code disagree, the code wins — the annotation and flag references are transcribed from crates/ramjet-controller/src/annotations.rs and crates/ramjet-ingressd/src/args.rs respectively, and both have tests asserting the vocabulary is complete.

License

Dual-licensed under either of

• Apache License, Version 2.0 (LICENSE-APACHE)
• MIT license (LICENSE-MIT)

at your option.

Unless you explicitly state otherwise, any contribution intentionally submitted for inclusion in the work by you, as defined in the Apache-2.0 license, shall be dual licensed as above, without any additional terms or conditions.