Fail modes: fail-closed versus fail-open in production

Objective

Decide, per enforcement point, whether it should fail closed or fail open, and justify the choice against blast radius and availability. Then implement the fail-closed decision in the lab.

Concept

Every enforcement point has to answer one question before it ever sees a request: what happens when the decision cannot be made? The PDP is unreachable, the bundle failed to load, the context lookup timed out, the policy errored. There are exactly two answers.

Fail-closed: when in doubt, deny. If the PEP cannot get a positive authorization, the request does not proceed. This protects the resource at the cost of availability, an outage in the authorization path becomes an outage in the protected service.
Fail-open: when in doubt, allow. If the PEP cannot reach a decision, it lets the request through. This protects availability at the cost of the resource, a failure in the authorization path means unauthorized requests can succeed.

There is no universally correct answer; there is a correct answer per enforcement point, and it is a design decision you make deliberately and write down.

How to choose

Weigh two things: the blast radius of a wrong allow, and the cost of a denied-but-legitimate request.

Fail closed when a wrong allow is worse than an outage. A refund tool, a production database, a Kubernetes admission controller guarding what runs in the cluster, an agent invoking a privileged action: here a single unauthorized success can be irreversible or dangerous, so denying during an authorization outage is the lesser harm. Most security-relevant enforcement points fail closed. This is also the safe default: if you have not decided, fail closed.
Fail open only when the guarded action is low-risk and availability is paramount, and you have said so on purpose. A best-effort personalization check that decides whether to show an optional banner can fail open, because showing the banner to someone who should not see it is cheap and blocking the whole page is not. Fail-open is a considered exception, never a silent default.

The dangerous case is fail-open by accident: a PEP that treats "no decision returned" the same as "allow" because nobody specified the behavior. That is how an authorization outage turns into a silent authorization bypass. Make the fail mode explicit in the PEP configuration and in the policy's default.

Fail-closed in the policy: the default matters

In Rego, the policy's contribution to failing closed is the default deny. default allow := false means that if no allow rule fires, for any reason, including a request shaped in a way you did not anticipate, the answer is deny. A policy written the other way, allowing by default and denying specific cases, fails open by construction: anything you forgot to deny is allowed. The default-deny discipline you learned in the foundations track is exactly the fail-closed posture expressed in policy. The PEP's fail mode (what to do when it cannot even reach the PDP) and the policy's default (what the PDP returns when no rule matches) are two layers of the same fail-closed design.

request -> PEP asks PDP
   |            |
   |            +-- PDP reachable? --no--> PEP fail mode decides (closed = deny)
   |                     |
   |                    yes
   |                     v
   |            policy evaluates --> no allow rule fires --> default allow=false (deny)
   v
 enforce the decision

Text version: two independent points can force a deny. If the PEP cannot reach the PDP, its fail mode decides (fail-closed denies). If the PDP evaluates but no allow rule matches, the policy's default allow := false denies. Both must be set correctly to be fail-closed end to end.

Hands-on lab

Now implement the decision in ../lab/policy.rego. The policy is already fail-closed at the default: default allow := false and default reason := "deny-fail-closed" are wired for you. Your job is the two authorizing reason rules that carve out the allowed cases on top of that deny.

Find the # TODO(learner) block and implement:

reason := "same-domain-standard" when input.action == "invoke", input.context.mtls_verified == true, input.subject.trust_domain == input.resource.owner_domain, and input.resource.classification == "standard".
reason := "privileged-tool-approved" when the same identity conditions hold, input.resource.classification == "restricted", and input.context.approval_ref != "".

From the cloned university-labs repo:

git clone https://github.com/EnforceAuth/university-labs.git
cd university-labs
opa test courses/reference-architectures -v

You start at FAIL: 2/11 and finish at PASS: 11/11. Pay attention to the two deny tests test_cross_trust_domain_denied and test_missing_mtls_denied. Neither request matches an authorizing reason, so reason falls to deny-fail-closed and allow stays false. That is fail-closed in action: the request that cannot be positively authorized (wrong trust domain, or no verified mTLS) is denied by the default, not by a rule you had to remember to write. If you had instead written the policy to allow by default and deny specific cases, those two requests would have leaked through.

Check for understanding

Define fail-closed and fail-open in one sentence each, in terms of what the PEP does when it cannot reach a decision.
For each of these, say fail-closed or fail-open and why: a Kubernetes admission controller; an optional "recommended for you" banner; an AI agent invoking a refund tool.
How does default allow := false implement the fail-closed posture in policy, and why does an allow-by-default policy fail open by construction?

Objective​

Concept​

How to choose​

Fail-closed in the policy: the default matters​

Hands-on lab​

Check for understanding​