# Specification: using AI to end up using less AI

Use AI to build a documented, tested Python analysis in a local VS Code project. Then run the saved code without another model call. Check the numbers and figure before interpreting the results.

## Core idea

The workflow separates scientific reasoning, software design, implementation,
execution, and interpretation:

![Six WAILAI stages: frame the inquiry, specify the computation, define milestones and acceptance tests, implement code and tests, execute and verify locally, and interpret the evidence. Cards identify contributions, artifacts, and human approvals.](figures/wailai-overview-en.svg)

Each stage leaves a file or a checkable result for the next. Together they form a reusable analysis.

The numbers match the six stages below. Review and approve the specifications
and acceptance tests before implementation. Interpret only verified results.
Stage 5 can be repeated from the saved code without another AI request.

## Project folder organization

Start with four empty folders: `.agents/`, `analysis/`, `data/` and `results/`. Put the data or an approved local access path in `data/` before execution. Keep source data read-only. Do not copy sensitive records into prompts or outputs unnecessarily.

### Expected final structure

Specifications and plans go in `.agents/`, code with dependencies and tests in `analysis/`, inputs in `data/`, and outputs in `results/`:

```text
project/
├── .agents/
│   ├── ANALYSIS.md
│   ├── PYTHON.md
│   ├── PLAN.md
│   └── WORKFLOW.md
├── analysis/
│   ├── <analysis implementation>
│   ├── <dependency or environment definition>
│   └── <automated tests>
├── data/
│   └── <source dataset or datasets>
└── results/
    ├── <result figure>
    └── <result table>.csv
```

`ANALYSIS.md`, `PYTHON.md`, and `PLAN.md` are distinct artifacts:

- `ANALYSIS.md` says what should be learned and which statistical reasoning is
  appropriate.
- `PYTHON.md` says what the software must do to implement that analysis.
- `PLAN.md` says how implementation will proceed and how correctness will be
  tested.

`WORKFLOW.md` documents the reusable method itself. In Challenge 2, `PLAN.md` is called `CODE_PLAN.md` to distinguish it from the report plan. Result filenames may vary, but a figure and its underlying CSV table are required. The table is the
machine-readable source for the values shown in the figure.

## Workflow stages

### Stage 1 — Frame the inquiry

**Purpose:** Move from a dataset or subject area to a worthwhile, answerable
question and a statistical analysis specification.

**Example prompt:**

> Using the available variables, suggest an interesting research question. For
> the selected question, write a rigorous analysis specification to
> `.agents/ANALYSIS.md`.

For a privacy-preserving workflow, “available variables” should be supplied as
a data dictionary, schema, permitted value ranges, aggregate description, or
synthetic example—not raw sensitive records. If the AI operates entirely in an
approved local environment, it may instead inspect the local dataset without
its contents leaving that environment.

**The analysis specification should define:**

- The question of interest.
- Inputs, exposures, outcomes, and derived variables.
- Descriptive and inferential analyses.
- Covariates, reference groups, and comparison directions.
- Assumptions, sensitivity checks, and limitations.
- Required analytical deliverables.
- Criteria for considering the analysis complete.

### Stage 2 — Specify the computation

**Purpose:** Translate the conceptual statistical analysis into an unambiguous
software contract.

**Example prompt:**

> Translate `.agents/ANALYSIS.md` into a complete specification for Python code.
> Define data validation, statistical calculations, reproducibility controls,
> the output CSV contract, and the result figure. Save it as
> `.agents/PYTHON.md`.

**The software specification should define:**

- Input paths and required schema.
- Data-quality checks and failure behavior.
- Clear failure when data validation or statistical model fitting is invalid.
- Derived variables and exact statistical models.
- Reference levels, contrast directions, confidence intervals, and random
  seeds.
- Command-line behavior and dependencies.
- The schema, ordering, and precision of the results table.
- The content and design requirements of the figure.
- Numerical, structural, and visual verification criteria.

### Stage 3 — Plan milestones and acceptance tests

**Purpose:** Decide how the software will be built and how correctness will be
recognized before implementation begins.

**Example prompt:**

> Create `.agents/PLAN.md` with small implementation milestones. For every
> milestone, define its expected output and high-value acceptance tests covering
> validation, statistical calculations, reproducibility, and output contracts.

**The plan should include milestones such as:**

1. Load and validate the data.
2. Construct derived variables.
3. Reproduce simple calculations with independently checkable examples.
4. Fit the specified statistical models.
5. Assemble and validate the result table.
6. Generate the figure from the result table.
7. Run the complete workflow and inspect the artifacts.

**Acceptance tests should be designed for:**

- Rejection of missing, duplicated, malformed, or out-of-range data.
- Correct comparison directions and threshold definitions.
- Known calculations on small synthetic examples.
- Deterministic results under a fixed seed.
- Preservation of the original source data.
- Required CSV schema, row counts, value ranges, and confidence-interval
  invariants.
- Creation of exactly the required result artifacts.
- Agreement between values in the CSV and marks in the figure.
- Successful end-to-end execution from a documented command.

Tests are specified in this stage so that they act as prior acceptance criteria,
not as convenient checks invented after seeing the implementation.

### Stage 4 — Build to the contracts

**Purpose:** Generate the implementation and the already-planned tests.

**Example prompt:**

> Implement `.agents/PYTHON.md` by following `.agents/PLAN.md`. Keep the complete
> application in `analysis/`, create the planned automated tests, and make them
> pass without weakening the specifications or acceptance criteria.

**Implementation requirements:**

- Code resides in `analysis/`.
- Data are read from `data/` without modification.
- Dependencies and execution instructions are recorded.
- Automated tests implement the acceptance criteria from Stage 3.
- Randomized procedures use explicit seeds.
- Statistical estimates are calculated once and placed in a structured table.
- The figure is generated from that result table rather than from a separate,
  potentially inconsistent calculation.

### Stage 5 — Execute and verify locally

**Purpose:** Run the tests and analysis in the controlled execution environment,
then verify the generated artifacts.

**Minimal representative prompt or local instruction:**

> Run the automated tests and execute the analysis locally. Do not modify the
> source data. Confirm that `results/` contains the required figure and CSV,
> inspect the figure, and verify that its values agree with the table.

**Execution should establish:**

- All automated tests pass.
- The documented analysis command exits successfully.
- The input data remain unchanged.
- Repeated runs with the same inputs, seed, and dependency versions produce
  identical numerical output.
- The result table satisfies its schema and numerical invariants.
- The figure is legible, complete, and consistent with the table.
- `results/` contains only the expected analytical deliverables.

The diagram under “Privacy and security boundary” shows the local data flow
and the return to implementation when a check fails.

### Stage 6 — Interpret the evidence

**Purpose:** Turn the verified numerical output into a clear explanation while
respecting the analysis design and its limitations.

**Example prompt:**

> Using only the generated figure, aggregate result table, and analysis
> specification, explain the findings in plain language. Report magnitudes and
> uncertainty, distinguish association from causation, and describe the main
> limitations.

**Interpretation should:**

- Lead with the answer to the original question.
- Explain important effect sizes in meaningful units.
- Explain confidence intervals without relying only on significance labels.
- Identify whether adjustment materially changes the descriptive comparison.
- Distinguish descriptive results, adjusted associations, and causal claims.
- State the limitations of measurement, confounding, selection, and
  generalizability.
- Avoid exposing or quoting individual-level records.

**Output after this stage:**

- A human-readable explanation grounded in the saved results.
- Optionally, a separately saved interpretation or report when the project
  requires one. Saving such a report is not necessary for the minimal workflow.

The interpretation may be created with AI assistance using only aggregated
outputs. It can also be written entirely by a human; the reproducible analytical
result does not depend on AI interpretation.

## Cumulative outputs by stage

| Stage | Durable artifact or verified outcome |
|---|---|
| Initial setup | Empty `.agents/`, `analysis/`, `data/`, and `results/` directories |
| 1. Frame the inquiry | `.agents/ANALYSIS.md` |
| 2. Specify the computation | `.agents/PYTHON.md` |
| 3. Plan and test | `.agents/PLAN.md`, including milestones and acceptance-test definitions |
| 4. Build | Executable code, dependency definition, and automated tests under `analysis/` |
| 5. Execute | Verified figure and underlying CSV under `results/` |
| 6. Interpret | Plain-language explanation tied to the verified aggregate results |

## Reuse

Another person should be able to inspect the specifications, install dependencies, run tests and code, and reproduce the outputs. Interpretation can then change without rerunning the analysis.

## Privacy and security boundary

![AI-assisted development uses a data description and synthetic examples; local execution uses unchanged raw data. Python produces the CSV table and generates the figure from it. The human checks the results, approves interpretation, or requests a correction using non-sensitive evidence followed by another verification.](figures/wailai-local-execution-en.svg)

This diagram shows the route without AI access to sensitive raw data. The
analytical figure is generated from the CSV table. If a check fails, correct
the affected implementation and verify again; keep the agreed acceptance
criteria intact.

To avoid sending sensitive data to the AI:

1. Give the AI a schema, data dictionary, synthetic examples, and analytical
   requirements rather than the sensitive dataset.
2. Generate specifications, plans, code, and tests from that non-sensitive
   description.
3. Move the generated application into a controlled local environment.
4. Run it locally against the real data.
5. Share only reviewed aggregate results for interpretation, if AI assistance
   is still desired.

Using an AI agent that can read a local file is not automatically equivalent to
keeping the data private from the AI system. The environment, connector,
retention policy, and institutional rules must be understood before granting
access. The workflow provides a way to avoid such access; it does not by itself
guarantee privacy.

Sensitive values must not be written to logs, exception messages, figures, test
fixtures, or generated documentation. Synthetic fixtures should be used for
calculation and validation tests whenever possible.

## Check reproducibility

Stages 2–5 specify the checks. Keep expected test results independent of the code where possible. The same inputs, seeds and dependencies should produce the same numbers. Reproducibility does not establish correctness; check the calculation and its interpretation separately.

## Concise formulation

The workflow can be summarized in one sentence:

> Use AI to turn a clearly defined question into tested local software, then run
> that software reproducibly against protected data and interpret only the
> verified aggregate results.

An even shorter label is:

```text
Specify → Plan and test → Build → Run locally → Interpret
```