The edge/workload split

This is the whole design in one sentence, and it is worth writing on a board before reading further. There is exactly one cluster with a public address, and it is the small single-node one. The clusters doing the work have no public address, no certificates, and no ACME.

The instinct is that this is a compromise forced by the missing load balancer. It is better than that: a workload cluster with no public ingress cannot be attacked from the internet directly, only through a component whose configuration you control entirely.

HTTP-01 validation answers a plain HTTP request on port 80 at the hostname on the certificate. A workload cluster has no A record, so nothing on the internet resolves to it, so Let’s Encrypt cannot reach it, so it cannot complete a challenge. It could not obtain a public certificate even if someone wanted it to.

Follow that one step further, because this is the good bit: the constraint from hop 3 is doing the enforcement. It is not a policy someone could relax by editing a manifest — it is a property of the validation method. Choose DNS-01 instead and this stops being true, and workload clusters become able to hold public certificates. That is a real argument for keeping HTTP-01 beyond simplicity.

Here is the requirement that turns on the setting flagged in the Traefik values — the two files finally meet.

Traefik will not route to an ExternalName Service unless explicitly permitted, because doing so lets whoever can create a Service choose an arbitrary destination. The edge/workload split needs exactly that capability, for exactly one hostname.

The design question to put to a class: this grants the capability cluster-wide to satisfy one route. What would it take to scope it — provider namespace restrictions, RBAC on Service creation, an admission policy on spec.externalName values? Then have them find where in the repo that scoping happens, and notice it does not.

This must match a SAN on the certificate the workload cluster presents. When it does not, the TLS handshake fails, Traefik has no backend, and the client gets a 502 — with an access log line that says 502 and a general log that, at INFO, says nothing useful at all.

So the symptom points at the backend being down. Every instinct says go and look at the workload cluster, where everything is healthy. This is the highest-value troubleshooting item in the document.

Diagnose it from the far side of the handshake instead of guessing:

openssl s_client -connect ingress.workload-fra.internal.example:443 \
  -servername ingress.workload-fra.internal.example \
  | openssl x509 -noout -text | grep -A1 "Subject Alternative Name"

Compare what comes back to the line above. A mismatch is the answer; anything else and the fault is elsewhere.

The edge presents a client certificate. The workload cluster’s ingress is configured to require one and to accept only this issuer — so the workload cluster does not need a firewall rule, an IP allowlist, or a VPN to know that a connection came from the edge. The certificate is the proof.

Note what that means for rotation: this Secret is a credential with an expiry, and when it expires every workload cluster becomes unreachable at once. It is the single most important expiry date in the fleet and it lives in a Secret the deployer cannot read — so the monitoring for it has to come from somewhere else. Ask a class where they would put that alert.

When the serverName mismatch above produces a 502, this is what a tired engineer at 6pm reaches for. It works — the 502 goes away, traffic flows, the incident closes.

What it actually did: the edge now accepts any certificate from anything that answers on that hostname. Server authentication is gone. The client certificate is still sent, so the workload cluster still authenticates the edge, and the connection still shows as mTLS in every dashboard. The failure is invisible in exactly the place someone would look to check.

It is left in the file as a commented line with a warning rather than deleted, because a deleted option is one somebody rediscovers under pressure, and a warned-about one is a decision they have to overrule in writing. That is a documentation pattern worth naming.

End the session here. A workload cluster can be three zones, nine nodes and a managed control plane with a 99.95% target — and if the only path to it is a single-node edge, the availability of the whole system is the availability of one machine in pl-waw.

Nothing in this document is wrong, and none of it is worth much against that number. It is the right trade for a platform of this size and it should be written down as a trade rather than discovered during an incident.

The question to leave a class with is not "how do we make the edge redundant" — they will say "a second node and a load balancer", which is the constraint they were told to work within. It is: what is the recovery procedure when the edge node is gone? That has an answer that fits inside the constraint, and writing it is a better exercise than removing the constraint.