Testing

Greph is validated with three layers of tests:

1. Unit tests for isolated components (parsers, walkers, codecs, options).
2. Oracle regression corpus that diffs Greph output against the canonical grep, ripgrep, and ast-grep binaries on a fixed scenario set.
3. Probe-driven feature matrix that asserts the verified surface of the rg and sg wrappers.

CI runs on every push to main and on every pull request (.github/workflows/ci.yml). It runs coding standards (composer cs), static analysis (composer analyse), the PHPUnit unit and integration suites (composer test) and composer audit on PHP 8.4. The oracle corpus (composer test:oracle) is not part of CI, and the feature matrix is regenerated by hand with php bin/feature-matrix. The PHPUnit suite contains a test that generates the matrix with the upstream rg and sg marked unavailable and checks the report's shape, so it does not fail when a single probe regresses.

Working from a clone

Everything on this page needs a clone of the repository. Since 0.2.3 the Composer package contains only src/ and the four runtime executables; tests/, scenarios/, benchmarks/ and the development scripts in bin/ are left out of it.

git clone https://github.com/inline0/greph
cd greph
composer install

Running the tests

# PHPUnit unit and integration suites
composer test

# Oracle regression corpus
composer test:oracle

# Static analysis
composer analyse

# Coding standards
composer cs
composer cs:fix

# Full release-grade verification (analyse + cs + test + test:oracle)
composer verify

composer verify is the bar for any release. If it does not pass, the change is not done.

The oracle model

Greph follows an oracle-driven verification model. For every scenario, three commands run:

1. Oracle: grep, ripgrep, or ast-grep, captured against the scenario corpus.
2. Actual: Greph, captured against the same corpus.
3. Comparator: a structured diff that checks line-by-line or semantic equivalence.

If Greph disagrees with the oracle, Greph is wrong. If grep and rg disagree with each other, the scenario records the disagreement under oracle_disagreement, names the tool Greph follows in greph_follows, and sets the comparison against the other tool to skip. Both directions occur in the corpus.

Text scenarios are compared against grep and rg, AST and rewrite scenarios against sg. Oracle output is captured once and committed under each scenario's oracle/ directory, so a normal run needs only PHP; the external tools are needed only to capture or refresh oracle output.

The implementation lives under tests/Oracle/ and the scenarios live under scenarios/text/, scenarios/ast/, scenarios/rewrite/, and scenarios/edge/.

Scenario layout

Each scenario is a self-contained directory:

scenarios/text/literal-simple/
├── scenario.json     # name, category, pattern, flags, path
├── setup/            # corpus to search through
├── oracle/           # captured oracle output (grep.txt, rg.txt, rg.json, ...)
├── actual/           # captured greph output (generated, git-ignored)
└── reports/          # comparison results (generated, git-ignored)

scenario.json declares the pattern, the flags and the search path. An optional expectations object sets the comparison mode per oracle (exact, semantic or skip); without it, text scenarios compare grep and rg exactly and the rg JSON semantically, and AST and rewrite scenarios compare against sg.

Running a single scenario

The repository has a small set of CLIs for working with individual scenarios. They are development tools, not runtime entrypoints: they live alongside the runtime binaries in bin/ but are not part of the Composer package. Each one takes the scenario name relative to scenarios/.

# Capture oracle output for a scenario (needs grep and rg, or sg)
./bin/oracle text/literal-simple

# Run greph against a scenario, capture its output
./bin/actual text/literal-simple

# Diff oracle vs actual
./bin/compare text/literal-simple

# Full pipeline: oracle (only if missing) -> actual -> compare
./bin/test-scenario text/literal-simple

# Recapture the oracle output first
./bin/test-scenario text/literal-simple --refresh

Running the full regression suite

# All scenarios (the same as composer test:oracle)
./bin/test-regression

# Filter by category
./bin/test-regression --category text
./bin/test-regression --category ast
./bin/test-regression --category rewrite
./bin/test-regression --category edge

# Print only failures and the summary
./bin/test-regression --fast

# Recapture every oracle before comparing
./bin/test-regression --refresh

Scenarios always run one at a time. --jobs N is accepted and echoed in the summary line, but it does not run anything in parallel (#7). In a fresh clone text/no-ignore and text/explicit-file-input fail, because their fixture files under setup/vendor/ are not committed (#6).

The regression suite is the gate for new behavior. A new flag, a new edge case, or a new bug fix requires a scenario that captures the expected behavior against an oracle.

Compliance reporting

./bin/verify-compliance

verify-compliance does not run anything itself. It reads the oracle, actual and report files that the last scenario runs left behind and prints which scenarios pass, which fail, and which have no report yet. Run ./bin/test-regression first. --output report.json also writes the report as JSON. It is the right command to run when you want to know "is the current branch ready for release".

Test-driven development workflow

When adding a feature:

1. Write a scenario directory with the corpus and flags that exercise the new behavior.
2. Run ./bin/oracle <scenario> to capture the canonical tool's output.
3. Implement the feature in src/.
4. Run ./bin/test-scenario <scenario> to verify Greph matches the oracle.
5. Run ./bin/test-regression --fast to make sure nothing else regressed.
6. Run composer verify before committing.

When fixing a bug:

1. Capture the failing input as a scenario.
2. Confirm the oracle produces the expected output.
3. Confirm Greph does not.
4. Fix the bug.
5. Confirm Greph now matches the oracle.
6. Run composer verify.

The scenario stays in the corpus permanently, so the bug cannot regress without composer test:oracle (and composer verify) failing.