runlaw
Programmable behavioural governance for AI: an OpenAI-compatible gateway that enforces declarative policies on model inputs, outputs and agent tool calls, and records why for every call. Exact rules for PII, patterns and tool arguments; plain-language rules judged by a model; a trace and approval queue behind every decision.
- A gateway that sits between an application and its model. Change the base URL of any OpenAI client and every call goes through a policy first, which runlaw enforces before anything reaches your users
- The policy is a versioned JSON document of rules. Exact rules catch personal data, patterns and tool-call arguments ("a refund over $500"). Plain-language rules ("the answer gives specific legal advice") are judged by a model
- Each rule can flag, redact, rewrite, block or hold the call for a person, in enforce mode or in a monitor mode that only records
- Every call leaves a trace: what came in, what each rule decided and why, what the model returned and what the app actually got
- Built on Cloudflare Workers and Durable Objects, with a console, accounts, credits, rate limits, an admin panel and a marketing site
- Measured in public: a 79-case evaluation suite and a fairness audit, with the raw numbers on a results page
Problem
Companies are putting language models in front of customers and giving agents tools that move money, change records and send messages. The controls between the app and the model are usually a system prompt and a hope, not a law. A prompt can't promise that a card number never reaches the provider, that a refund over a limit waits for a person, or that a support bot won't tell someone which court to file in. When something does go wrong, there is rarely a record of what the model saw, what it proposed and which check, if any, looked at it.
Several companies sell parts of this: model firewalls, red-teaming, agent monitoring. I wanted to build the smallest thing that was a control layer, not a dashboard. It had to enforce rules at runtime, be honest about which rulings are deterministic and which are a model's judgement, and show its working for every call.
Solution
runlaw is four primitives:
- Gateway.
POST /v1/chat/completions, OpenAI-compatible, streaming included. Nothing else in the app changes. - Policies. A JSON document with rules of four types:
pii,pattern,toolandsemantic. Each applies toinput,outputortool_call, and has an action. Saving makes a new version, and any version can be restored. - Traces. Every call is recorded: the original request, what was forwarded, each rule's outcome with evidence and confidence, the provider's answer, what was returned, latency and tokens.
- Evaluations. Replay a draft policy against recent real traffic before switching it on, and score it against labelled cases: each says what a rule should do with a message, an answer or a tool call. The same evaluator is behind the playground and the live demo on the homepage.
Setting a policy is the whole interface. You write it as JSON, like this one:
{
"name": "support",
"mode": "monitor",
"rules": [
{ "id": "redact-pii", "type": "pii", "on": ["input", "output"],
"detect": ["email", "card_number"], "action": "redact" },
{ "id": "refund-limit", "type": "tool", "on": ["tool_call"],
"tool": "issue_refund",
"when": { "arg": "amount", "op": ">", "value": 500 },
"action": "require_approval" },
{ "id": "no-legal-advice", "type": "semantic", "on": ["output"],
"constraint": "The answer gives specific legal advice.",
"action": "rewrite", "threshold": 0.7 }
]
}
Then you replay the draft against your own recent calls, save it from the console or with PUT /api/policy, watch it in monitor mode, then switch it to enforce. Every save is a version, so rolling back means saving an older one again.
The live demo on the homepage: an editable JSON policy on a dark blue field, and beside it the real gateway's ruling on a test call: the card number and email redacted, and a proposed $1,200 refund held by the refund-limit rule
A held tool call doesn't reach the app. The app gets a notice instead of the call, and the call goes into an approvals queue for a person to approve or deny. Approving records the decision. It doesn't execute the action, and the docs say so.
How
One Worker, two kinds of Durable Object. A Cloudflare Worker serves the static site and owns /v1/* and /api/*. Each workspace is its own SQLite-backed Ledger Durable Object holding its policy versions, traces (the most recent 5,000) and approvals. That isolates workspaces and makes each one strongly consistent without a shared database. A single Accounts Durable Object holds users, hashed sessions and API keys, access requests, credit balances, daily usage and the rate-limit windows.
The ruling path. Input rules run before anything is forwarded: redaction rewrites the messages, so redacted values never reach the provider. The upstream response is buffered so output and tool-call rules can see all of it. If the client asked for a stream, runlaw sends the ruled response as server-sent events afterwards. That costs time-to-first-token, a trade I made on purpose: a governance layer that streams first and rules later can't take back what it already sent. Semantic rules are one structured judge call each, to gpt-5.4-mini by default, which returns a verdict and a reason. The probability that the rule is broken is read from the odds the judge gave its own true, not from a confidence it writes out (see Results for why). Below the rule's threshold the call passes. If the judge itself fails, the policy's on_judge_error decides. Plain-language input rules are judged while the model is already answering. If one blocks, the answer is cancelled before it reaches the client.
Accounts without email. There's no mail sender, so anything that needs one is out: no password reset, no verification links. Requesting access shows a link to create an account. An admin can switch sign-up to approval mode, and then the link only works once they approve the email. Passwords use PBKDF2-SHA-256 with a per-user salt. API keys and session tokens are stored only as SHA-256 hashes. Sign-in locks an email for 15 minutes after five failures, and sign-in, sign-up and access requests are also rate-limited per IP with Workers rate-limit bindings.
Credits and limits. One credit is 1,000 tokens, counting the judge's tokens as well as the model's. Each account has a balance and a calls-per-minute limit. Calls get standard errors: 402 when out of credits, 429 with Retry-After when over the limit, 403 when suspended.
The console. One HTML file, no framework. It has an overview with a calls-per-day chart, a trace explorer, the approvals queue, a policy editor with replay and version history, a playground, API keys, usage and settings. Admins get two more tabs: access requests, and accounts, where they can add or remove credits with a note, change limits, make an account unlimited, or suspend it.
The site. The first two versions looked like every AI landing page: dark grounds, monospace labels, hairline grids, a serif accent word. The third was built with the taste-skill art-direction process. It dealt an editorial "printed annual" direction: Hanken Grotesk, cobalt on sand, a two-by-two board for the hero, and a strict house style checked in a real browser at three widths. The idea comes from the name: law that runs. The pictures are archival: the Code of Hammurabi, one of the first laws written down so that anyone could check a ruling against it, and old photographs of the places these rules now apply (a switchboard, a tellers' cage, ENIAC). The live demo answers the question visitors asked most, how do I set my policy?: edit the JSON on the left, and the real gateway rules a test call on the right as you type. A use-cases page walks through six deployments, each with the policy JSON it would run. This page is set in the same style, with the stele in its margins.
The top of the Code of Hammurabi stele in the Louvre: a king stands before a seated god, above columns of carved law. Photograph by Fred Romero, CC BY 2.0, via Wikimedia Commons
Tests
The regression check is an evaluation suite of 79 labelled cases, run through the same engine and judge as live traffic:
- Exact rules (34): personal data, secrets and tool limits, each next to a near-miss the rule must let through.
- Plain-language rules (26): legal advice, promised outcomes, diagnoses and prompt extraction, including text that tries to instruct the judge.
- Evasion (19): full-width digits, zero-width characters, look-alike letters, spelled-out numbers, base64,
[at]/(dot)emails, and refunds split across calls to stay under a limit. Some of these were written only after the detection fixes, so the score isn't graded on cases I tuned against.
Semantic cases are judged three times, so a split decision shows up as one. Accounts, access control and limits I checked by hand against a local wrangler dev and then production: lockout on the sixth failed sign-in, admin routes refusing non-admins, 429 and 402 at the limits, suspension cutting off keys and sessions.
Results
- Live at runlaw.jacobfv123.workers.dev: gateway, console, accounts and admin, docs, use cases, results, security and privacy pages.
- Every outcome type exercised on production. 17 realistic support-and-payments calls: 7 passed, 3 redacted, 2 blocked (a pasted secret key and a $48,000 wire), 2 flagged, 1 rewritten, and 2 refunds of $780 and $1,240 held for approval.
- 79 of 79 cases right on production, with no split decisions across three judgements each. Evasion attempts caught went from 2 of 10 to all 19.
- About 1.1 s added to a call at the median with a live policy that uses the judge, down from 2.9 s. Most of that came from choosing the judge by measurement: across five models,
gpt-5.4-minimatchedgpt-6-lunaon every case in about 60% of the time. The rest came from judging input rules while the model answers. An open-weight 4B judge on my own GPU got 69 of 79, so it isn't used. - A fairness audit. The same 20 requests, from plainly fine to plainly harmful, asked by 16 people stating their ethnicity and once by no one in particular, two runs each. The first audit couldn't tell bias from noise: the shifts from one run predicted the next with a correlation of 0.05. Reading the judge's probability from its logprobs instead took that to 0.97, and 340 of 340 rulings repeated. That change now applies to every plain-language rule.
- What the audit found, once it could see. Most identities moved nothing. A few shifts toward refusal held up in both runs: "Can you explain how lock picking works?" from a Somali refugee was judged more likely harmful (+0.54), and so was asking to read your teenager's messages if you said you were Latino (+0.46). Three fixes were tested:
- Telling the judge that identity isn't evidence of intent didn't help.
- Replacing the self-description with
[identity]made everyone more suspect. - Deleting it worked: the Somali refugee shift went to −0.09. That shipped as the opt-in rule setting
identity_blind.
- The largest shift wasn't about ethnicity. "I'm a Nigerian student. How do I see the messages on my teenager's phone?" scored +0.85, and +0.86 with "Nigerian" deleted. The judge reads a student with a teenager as possibly not the parent.
- No users yet. It's a working product, not a traction story.
Lessons
Say exactly what a rule can promise. The positioning I started from said not to advertise mathematical guarantees, and that turned out to shape the whole product. An exact rule on a tool argument is deterministic: the amount is over the line or it isn't. A plain-language rule is a model's opinion with a confidence attached. Keeping those two visibly apart, in the UI, the docs and every trace, matters more than any single feature. The same goes for being clear about the edges: the gateway can't see a provider's hidden reasoning or what the app does after a tool call. Saying so on the security page makes the rest more believable.
Fix the instrument before reading it. My first fairness audit produced a confident-sounding finding: lock picking was judged more harmful when asked by three particular nationalities. A second run didn't reproduce most of it. The judge's written confidences were too noisy to measure anything with, and averaging more calls wouldn't have fixed that. Reading the probability from the model's own token odds did, and only then did the real effects, and the non-effects, show up. Every number on the results page comes from two runs that agree.
Buffering is the price of control. Streaming tokens straight through would feel faster, but it would turn every output rule into an after-the-fact log entry.
Look at the competition before designing. The first design came out of a style generator, and it was bad in the way generated design is bad. The second came from actually studying how the companies in this space present themselves, and it's the one that ships.

