BichengWang/ai-decision-system

AI Decision System

Replayable, offline-verifiable records for AI-assisted go/no-go decisions.

Why

Model output is not authority. Promotions, approvals and releases need a record that a third party can re-check against frozen inputs and rules. Dashboards and logs can describe what happened; re-verifiable evidence also preserves what is needed to recompute the decision.

flowchart LR
    J["Typed model answer
jev-decision
answer + replay evaluation"] --> D["Policy gate + human approval
decision-control
checksummed review record"]
    D --> E["Online experiment promotion
experiment-gate
embedded-input decision record"]
    E --> R["Model release/update gate
RELIA
results + MANIFEST"]

These are independent packages, not one integrated runtime. The arrows show possible decision points, not implemented cross-package connections. Each emits a machine-checkable record; verification mechanisms and retained-input needs vary. See Decision record conventions.

Components

| Component | Decision point | Package | Record and offline check | Status | | --- | --- | --- | --- | --- | | Jev decision | Typed yes/no, choice and score answers; labeled quality gates | jevdecision | Typed answer, evaluation.json, provenance; jev-decision-eval --responses … with retained questions/cases/requirements | 0.1.0 (GitHub release; PyPI pending) | | decision-control | Frozen routing policy, human approval and local ticket action | decisioncontrol | Exported record; decision-control replay --record … | 0.1.0 (GitHub release; PyPI pending) | | Experiment gate | SHIP / HOLD / ROLLBACK / INVALID from exported A/B aggregates | expgate | decision.json; experiment-gate-verify … | 0.1.0 (GitHub release; PyPI pending) | | RELIA | Synthetic model release and guarded update | Not published | results.json, MANIFEST.json; python -m relia.compare … --protocol … in its environment | Unreleased; human-gated |

The three small packages require Python 3.11+ and use only the standard library at runtime. decision-control is a trusted local SDK/CLI, not an identity service or external executor. Experiment gate consumes summaries; it does not run an experiment or deploy a model. Jev supports Jev and OpenAI Decisions backends; provider-reported confidence is not a calibrated safety guarantee.

Quickstart from a clone

Use Python 3.11+ in a virtual environment. These commands work before PyPI publication by installing each independent project from Git:

pip install "git+https://github.com/BichengWang/ai-decision-system#subdirectory=ai-workflows/experiment-gate"
pip install "git+https://github.com/BichengWang/ai-decision-system#subdirectory=ai-workflows/jev-decision"
pip install "git+https://github.com/BichengWang/ai-decision-system#subdirectory=ai-workflows/decision-control"

Run the examples below from a clone's repository root, with new output paths. Examples are in the source tree/sdist, not installed as wheel package data.

experiment-gate --input ai-workflows/experiment-gate/examples/offer-ranker-v2.json --out /tmp/expgate-new
experiment-gate-verify /tmp/expgate-new/decision.json
cat /tmp/expgate-new/DECISION.md

jev-decision --questions ai-workflows/jev-decision/examples/support-ticket.json --state "My payouts failed" --dry-run
jev-decision-eval --questions ai-workflows/jev-decision/examples/support-ticket.json \
  --cases ai-workflows/jev-decision/examples/support-ticket-cases.jsonl \
  --responses ai-workflows/jev-decision/examples/support-ticket-responses.jsonl \
  --require ai-workflows/jev-decision/examples/support-ticket-requirements.json --out /tmp/jev-new
cat /tmp/jev-new/EVALUATION.md

decision-control demo --out /tmp/control-new
decision-control replay --record /tmp/control-new/record-review.json
cat /tmp/control-new/summary.json

After 0.1.0 is published, pip install expgate jevdecision decisioncontrol will be available; publication is not claimed here. Live Jev calls require TYPESAFE_API_KEY (Jev) or OPENAI_API_KEY (OpenAI). The examples above make no provider calls and use hand-written/synthetic fixtures. They are software checks, not live-model, calibration, adoption or production evidence. Hashes detect change, not forgery; identities are caller-asserted and provider origin is not authenticated.

How this differs

These tools complement existing platforms; the bounded contribution is retained, re-checkable decision evidence, not a replacement for their operational features.

| Category | Typical role | This repository's scope | | --- | --- | --- | | Experiment platforms (GrowthBook, Statsig, Eppo) | Compute and manage experiment statistics in a platform | expgate consumes exported aggregates and emits a re-verifiable promotion decision | | Policy engines / human review (OPA, Cedar, LangGraph interrupts) | Decide authorization or pause a workflow for review | decision-control freezes policy, proposer and review into replayable evidence with label-based separation of duties | | Evaluation tools (promptfoo, OpenAI Evals) | Score prompts and model behavior | Jev gates typed answers with explicit PASS/FAIL/UNGATED and refusal/unknown-confidence semantics | | Registries / monitoring (MLflow, Evidently) | Track models and monitor performance | RELIA provides byte-reproducible synthetic release gates in its pinned environment |

RELIA uses its own Python 3.13 environment (3.13.7 pinned for reproduction). Its reproduction protocol requires fresh runs and separate reference comparison/reviewer attestations. Synthetic results do not establish real-world commercial performance, fairness, legal compliance or consumer benefit. Thresholds are proposed engineering gates, not industry/government standards; no agency endorsement is claimed. No accepted RELIA public release is claimed. Rights clearance, unaffiliated reproduction and acceptance remain human-gated.

Workspace navigation and validation

The workspace guide and manifest register independent environments, ownership, CI checks and commands. With Python 3.11+ resolved as python3:

make workspace-list
make workspace-check
make affected BASE=origin/main
make affected-test BASE=origin/main
make workspace-test COMPONENTS="experiment-gate jev-decision decision-control"
make release-status
make records-demo DEMO_OUT=/abs/new/dir

workspace-check tests tooling and checks syntax, boundaries, CI consistency and Markdown links; it does not run every behavioral suite. affected-test needs prepared component environments. release-status inspects candidate metadata (Python 3.11+). records-demo writes and verifies experiment records, runs Jev fixture replay, then exports/replays a local review record; it requires an absolute new directory and prints retained record paths. See the Makefile. For RELIA use make relia-sync, make relia-test and make relia-reproduce RELIA_OUT=/abs/new/dir; a local pass is not release acceptance.

The older src/ research workbench has a separate root environment. Its limited service-free make test baseline uses Python 3.11 (the pinned pandas 1.5.3 lacks Python 3.12+ wheels); broker, browser and ML probes need separate services/environments. Root tests do not validate the whole repo.

Licensing and project guides

The three small packages carry their own MIT licenses; the root MIT license covers the research material within its scope. RELIA's license status grants no current rights; intended Apache-2.0 publication requires rights clearance. See SECURITY, CONTRIBUTING, CITATION, records, releasing and delivery plan.

Portfolio and publication records

The author-produced portfolio report records local checks and evidence gaps for an EB-2 NIW assessment. It is not independent recognition, a legal eligibility finding or an accepted RELIA release. News/reviewer briefing, publication record, USCIS/NIST source notes, publication protocol and local evidence summary retain the claim boundaries. Fixtures and self-produced reports are not live-model quality, adoption, production impact or national-importance evidence; RELIA reuse remains subject to rights, reproduction and acceptance review.