# Testing

Greph is validated with three layers of tests:

1. **Unit tests** for isolated components (parsers, walkers, codecs, options).
2. **Oracle regression corpus** that diffs Greph output against the canonical `grep`, `ripgrep`, and `ast-grep` binaries on a fixed scenario set.
3. **Probe-driven feature matrix** that asserts the verified surface of the `rg` and `sg` wrappers.

CI runs on every push to `main` and on every pull request (`.github/workflows/ci.yml`). It runs coding standards (`composer cs`), static analysis (`composer analyse`), the PHPUnit unit and integration suites (`composer test`) and `composer audit` on PHP 8.4. The oracle corpus (`composer test:oracle`) is not part of CI, and the feature matrix is regenerated by hand with `php bin/feature-matrix`. The PHPUnit suite contains a test that generates the matrix with the upstream `rg` and `sg` marked unavailable and checks the report's shape, so it does not fail when a single probe regresses.

## Working from a clone

Everything on this page needs a clone of the repository. Since 0.2.3 the Composer package contains only `src/` and the four runtime executables; `tests/`, `scenarios/`, `benchmarks/` and the development scripts in `bin/` are left out of it.

```bash
git clone https://github.com/inline0/greph
cd greph
composer install
```

## Running the tests

```bash
# PHPUnit unit and integration suites
composer test

# Oracle regression corpus
composer test:oracle

# Static analysis
composer analyse

# Coding standards
composer cs
composer cs:fix

# Full release-grade verification (analyse + cs + test + test:oracle)
composer verify
```

`composer verify` is the bar for any release. If it does not pass, the change is not done.

## The oracle model

Greph follows an oracle-driven verification model. For every scenario, three commands run:

1. **Oracle**: `grep`, `ripgrep`, or `ast-grep`, captured against the scenario corpus.
2. **Actual**: Greph, captured against the same corpus.
3. **Comparator**: a structured diff that checks line-by-line or semantic equivalence.

If Greph disagrees with the oracle, Greph is wrong. If `grep` and `rg` disagree with each other, the scenario records the disagreement under `oracle_disagreement`, names the tool Greph follows in `greph_follows`, and sets the comparison against the other tool to `skip`. Both directions occur in the corpus.

Text scenarios are compared against `grep` and `rg`, AST and rewrite scenarios against `sg`. Oracle output is captured once and committed under each scenario's `oracle/` directory, so a normal run needs only PHP; the external tools are needed only to capture or refresh oracle output.

The implementation lives under `tests/Oracle/` and the scenarios live under `scenarios/text/`, `scenarios/ast/`, `scenarios/rewrite/`, and `scenarios/edge/`.

## Scenario layout

Each scenario is a self-contained directory:

```
scenarios/text/literal-simple/
├── scenario.json     # name, category, pattern, flags, path
├── setup/            # corpus to search through
├── oracle/           # captured oracle output (grep.txt, rg.txt, rg.json, ...)
├── actual/           # captured greph output (generated, git-ignored)
└── reports/          # comparison results (generated, git-ignored)
```

`scenario.json` declares the pattern, the flags and the search path. An optional `expectations` object sets the comparison mode per oracle (`exact`, `semantic` or `skip`); without it, text scenarios compare `grep` and `rg` exactly and the `rg` JSON semantically, and AST and rewrite scenarios compare against `sg`.

## Running a single scenario

The repository has a small set of CLIs for working with individual scenarios. They are development tools, not runtime entrypoints: they live alongside the runtime binaries in `bin/` but are not part of the Composer package. Each one takes the scenario name relative to `scenarios/`.

```bash
# Capture oracle output for a scenario (needs grep and rg, or sg)
./bin/oracle text/literal-simple

# Run greph against a scenario, capture its output
./bin/actual text/literal-simple

# Diff oracle vs actual
./bin/compare text/literal-simple

# Full pipeline: oracle (only if missing) -> actual -> compare
./bin/test-scenario text/literal-simple

# Recapture the oracle output first
./bin/test-scenario text/literal-simple --refresh
```

## Running the full regression suite

```bash
# All scenarios (the same as composer test:oracle)
./bin/test-regression

# Filter by category
./bin/test-regression --category text
./bin/test-regression --category ast
./bin/test-regression --category rewrite
./bin/test-regression --category edge

# Print only failures and the summary
./bin/test-regression --fast

# Recapture every oracle before comparing
./bin/test-regression --refresh
```

Scenarios always run one at a time. `--jobs N` is accepted and echoed in the summary line, but it does not run anything in parallel ([#7](https://github.com/inline0/greph/issues/7)). In a fresh clone `text/no-ignore` and `text/explicit-file-input` fail, because their fixture files under `setup/vendor/` are not committed ([#6](https://github.com/inline0/greph/issues/6)).

The regression suite is the gate for new behavior. A new flag, a new edge case, or a new bug fix requires a scenario that captures the expected behavior against an oracle.

## Compliance reporting

```bash
./bin/verify-compliance
```

`verify-compliance` does not run anything itself. It reads the oracle, actual and report files that the last scenario runs left behind and prints which scenarios pass, which fail, and which have no report yet. Run `./bin/test-regression` first. `--output report.json` also writes the report as JSON. It is the right command to run when you want to know "is the current branch ready for release".

## Test-driven development workflow

When adding a feature:

1. Write a scenario directory with the corpus and flags that exercise the new behavior.
2. Run `./bin/oracle <scenario>` to capture the canonical tool's output.
3. Implement the feature in `src/`.
4. Run `./bin/test-scenario <scenario>` to verify Greph matches the oracle.
5. Run `./bin/test-regression --fast` to make sure nothing else regressed.
6. Run `composer verify` before committing.

When fixing a bug:

1. Capture the failing input as a scenario.
2. Confirm the oracle produces the expected output.
3. Confirm Greph does not.
4. Fix the bug.
5. Confirm Greph now matches the oracle.
6. Run `composer verify`.

The scenario stays in the corpus permanently, so the bug cannot regress without `composer test:oracle` (and `composer verify`) failing.