Metadata-Version: 2.4
Name: ipw-assay
Version: 0.1.0
Summary: Random program generation for RISC-V cores, with shrinking and coverage feedback
Author: IPweave contributors
License: Apache-2.0
Project-URL: Homepage, https://github.com/IPWeave/ipw-assay
Project-URL: Source, https://github.com/IPWeave/ipw-assay
Project-URL: Issues, https://github.com/IPWeave/ipw-assay/issues
Keywords: riscv,verification,torture,random,fuzzing,lockstep,spike,sail,rvfi,riscof,arch-test,shrinking,coverage,profiles,rva23,hypervisor,compliance
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Programming Language :: Python :: 3
Classifier: Topic :: Scientific/Engineering :: Electronic Design Automation (EDA)
Classifier: Topic :: Software Development :: Testing
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
License-File: NOTICE
Provides-Extra: sim
Requires-Dist: ipweave>=0.1.1; extra == "sim"
Provides-Extra: cgf
Requires-Dist: pyyaml>=6; extra == "cgf"
Provides-Extra: dev
Requires-Dist: pytest>=8; extra == "dev"
Requires-Dist: pyyaml>=6; extra == "dev"
Requires-Dist: ipweave>=0.1.1; extra == "dev"
Dynamic: license-file

# ipw-assay

**Version 0.1.0** — preview. **Verification for RISC-V cores: generate, compare,
reduce.** Apache-2.0.

An assay determines what something actually *is*, by measurement, against a standard.
That is the job: run a design and a reference model over the same bytes and hold them to
agreeing — and when they do not, say which instruction, and cut the program down until
the answer is obvious.

Four ways in, all sharing one comparison engine, and a fifth that asks a different
question — *is this core an RVA23U64?*

| | |
| --- | --- |
| **`torture`** | random programs from a weighted mix, interleaved so unrelated work collides in the pipeline. The heritage is UC Berkeley's [riscv-torture](https://github.com/ucb-bar/riscv-torture); see [`PROVENANCE.md`](PROVENANCE.md). |
| **`archtest`** | the official [riscv-arch-test](https://github.com/riscv-non-isa/riscv-arch-test) suite, run directly or through RISCOF |
| **`cover`** | programs *aimed* at coverpoints in a CGF coverage file — what riscv-ctg does, closing gaps and checking a design in one pass |
| **`export`** | this tool's own generated tests, written in the architectural format so RISCOF, `riscv-isac` and any conforming reference model can read them |
| **`profile`** | the ratified RISC-V profiles — RVI20, RVA20, RVA22, RVA23, RVB23 — as a requirement-by-requirement ledger, and a run of the part of one that can be run |

```bash
python assay.py selftest                       # can anything here establish a result?
python assay.py limits                         # what it will not produce, and why
python assay.py generate --isa rv64gc --seed 7 -o test.S
python assay.py --path examples --path ../ipw-jacquard \
    run --core jacquard_target:lite --config rv32im --seed 7
python assay.py --path examples --path ../ipw-jacquard \
    campaign --core jacquard_target:lite --minutes 60 --corpus corpus/
python assay.py profile RVA23U64 --isa rv64gcb_zicond_zcb   # does the claim hold?
```

No Scala, no sbt, no second checkout. Generating, shrinking and comparing are the standard
library; simulating a core in-process is the framework; assembling and the reference model
are the RISC-V toolchain and Spike.

```bash
python gate.py --core examples.jacquard_target:lite --path ../ipw-jacquard
python examples/shrink_demo.py       # watch a 95-instruction failure become three
```

`gate.py` is what CI runs: the selftest, the suite, the integration tests and the corpus,
as one thing — non-zero when a gate fails *or when a gate could not run*.

## Coverage-directed generation

Random generation is very good at the middle of the distribution and hopeless at its
edges. A weighted mix will emit `add` ten thousand times tonight and never once with both
sources at the most negative number, because there is no reason it would.

```bash
python assay.py cover --cgf coverage/rv32i_m/I/add.cgf --dataset coverage/dataset.cgf     --isa rv32imzicsr --rounds 8 --core mycore:make -o measured.cgf
```

CGF is the coverage-group format `riscv-ctg` generates from and `riscv-isac` measures
against, so coverage measured here reads next to coverage measured by anything else — and
`-o` writes it back in the same form.

The loop is: measure what a run reached, *solve* the coverpoints nothing reached for
operands that would reach them, generate a directed program that does exactly that, run
it, measure again. A coverpoint is a Python expression, so solving one means finding
operands that make it true; what makes that practical without a constraint solver is
where the candidates come from — the expression's own literals are tried first, so
`rs1_val == (-2**(xlen-1))` is satisfied on the first attempt.

On the real `rv32e` covergroup — 5,028 coverpoints — this reaches **98%** in nine rounds
and stops there, and where it stops is the interesting part. Every one of the seventy-odd
it does not reach asks for a branch that goes *backwards* and is *taken*, which is a
program that does not terminate; a run that hangs says nothing about the core. A backward
branch that the operands show is **not** taken is written, because that one is safe.

Coverpoints it cannot solve or cannot write are **counted, named and reported**, never
quietly dropped: a coverage-directed run that reported only what it managed would look
like a suite with fewer gaps than it has.

A coverage run installs no trap handler unless asked. The handler reserves three
registers, and a covergroup wants every register as every operand.

With `--core`, each generated program is also run against a design, so closing coverage
and checking a core are one pass rather than two.

## Profiles

A profile is the RISC-V answer to combinatorial explosion: a base, a set of mandatory
extensions, and a set of options the software ecosystem undertakes to cope with, so that
`RVA23U64` says in eight characters what an ISA string says in two hundred. "Is this core
an RVA23U64?" is a question people actually ask, and it is not the same question as "do
these two models agree".

```bash
python assay.py profile                                # the ten ratified ones
python assay.py profile RVA23U64 --all                 # every requirement, one per line
python assay.py profile RVA23U64 --isa rv64gcb_zcb     # hold a claim to the mandates
python assay.py profile RVA22U64 --core mycore:make    # run the part that can be run
```

Every requirement is reported as exactly one of four things, and **two of them are not
passes**:

| | |
| --- | --- |
| **generated** | there are instructions for it here; a torture run at the profile's ISA executes and compares them |
| **probed** | no instructions of its own, but a configuration here reaches it — `Sv39` is a body running translated, `Zicntr` is a counter read whose value is destroyed before anything can compare it |
| **declared** | nothing executed here can establish it. `Zic64b` says cache blocks are 64 bytes; `Zkt` says certain instructions take a data-independent time. Two models can agree perfectly and neither be measuring that |
| **absent** | it could be established and nothing here does it yet. A gap, named, so that it is countable |

As of version 0.1.0 **every one of the ten ratified profiles has `absent` at zero** — there
is no mandate this package could establish and does not. That is not the same as a
compliance claim, and the report says so on its own last line: `declared` requirements are
still the implementer's word, and always will be.

```
RVA22U64  (RISC-V Profiles, version 1.0, 2023-04-02 (ratified))
  generated   14      probed    3      declared   6      absent   0

verdict: the 17 mandate(s) this package can establish held.
         6 could not be established here, so this is not a claim of RVA22U64 compatibility
         - it is a statement about the part that was tested.
```

Which bucket a requirement lands in is **computed, not tabulated**: whether a mandate has
instructions is a question about the instruction table, so the answer improves by itself
the day a row is added. A hand-written list of "things we test" goes stale silently.

One thing worth knowing before it surprises somebody: a profile can make something *more*
testable than the bare architecture. `assay.py limits` refuses to compare misaligned
behaviour because trapping and completing are both correct — but a profile mandating
`Zicclsm` has taken that freedom away, so under a profile a misaligned trap is a real
failure and is reported as one.

[`docs/profiles.md`](docs/profiles.md) is the whole story, including why there are two
ISA-string readers in this package and why the toolchain is part of the answer.

## Open tools, and what this replaces

The package is Apache-2.0 and it is meant to be usable by anyone, which is a design
constraint and not a licence header. The policy, in five lines:

1. **No licensed tool is ever on the path to a result.** Not for generating, not for
   comparing, not for coverage, not for a profile report. If something here needed a
   simulator licence to establish a result, it would not be here.
2. **Formats in, formats out.** Coverage is CGF, the format `riscv-ctg` generates from
   and `riscv-isac` measures against — read *and* written. Tests export in the
   architectural format. Configuration exports as `riscv-config` YAML. The trace port is
   RVFI, the reference trace is Spike's commit log, the exit protocol is HTIF `tohost`.
3. **Drive good tools; do not reimplement them.** RISCOF, Spike, `riscv-config` and the
   GNU toolchain are invoked as separate programs. `assay.py riscof` writes the whole
   plugin directory rather than replacing the harness.
4. **Nothing is vendored.** No source from any other generator is in this package; see
   [`PROVENANCE.md`](PROVENANCE.md).
5. **Anything produced here must be readable by something else.** A format only this
   package can read is a format that makes its users dependent, which is the opposite of
   the point.

Where that leaves it next to the tools people are using now:

| | what it is | what it costs to run | here |
| --- | --- | --- | --- |
| **riscv-torture** | the original: random sequences over a register pool, interleaved | JDK, sbt, a second checkout | replaced. Same idea, same families, same default weights — plus shrinking, coverage steering and instruction-by-instruction lockstep |
| **riscv-ctg** | directed tests aimed at CGF coverpoints | Python, `riscv-config` | replaced by `assay.py cover`, in the same format, and it closes coverage and checks a design in one pass rather than two |
| **riscv-dv** | constrained-random generation in SystemVerilog/UVM | a UVM-capable simulator for the main flow — a licence, in practice | overlapping. This needs no simulator licence at all; riscv-dv still reaches interrupts and multi-hart, which are refused here *with reasons* rather than missing |
| **Force-RISCV** | a generator carrying its own architectural model | build it, and learn it | overlapping. This borrows Spike and Sail instead of maintaining a third opinion about what `fnmsub.d` does |
| **RISCOF / riscv-arch-test** | the conformance harness and the official suite | `riscof`, `riscv-config`, `riscv-isac` | not replaced — driven. `assay.py riscof` writes the plugin, the ISA YAML and the model header from one ISA string |

The honest summary: it replaces riscv-torture outright and does riscv-ctg's job in
riscv-ctg's own format; against riscv-dv and Force-RISCV it wins on cost, on shrinking
and on saying what it did not test, and it does not yet reach everything they do.
`assay.py limits` is the list, and it distinguishes what *cannot* be compared this way
from what merely has not been built.

## A second reference model

"The design disagrees with Spike" and "the design is wrong" are not the same sentence,
and the difference is worth a second model.

```bash
./tools/fetch-sail.sh          # one binary, pinned and checksummed
python assay.py selftest       # to see it found
```

There is nothing to build. Sail once meant an OCaml toolchain, an opam switch and z3;
the released model is a single executable linked against libc, libm and libpthread. The
script pins a version, checks a checksum and puts it beside the package, and that is the
whole installation.

What it changes is what a run can say:

```
seed 2 (rv32imzicsr, 40 sequences), 79 instructions in 8.0s
  passed
    note: and Sail agrees, over 344 signature word(s) - two models and the design
          saying the same thing

seed 0 (rv32imzicsr, 40 sequences), 79 instructions in 8.9s
  DIVERGED: 2192 instruction(s) compared by lockstep, 0 signature word(s)
  trap: after 105 matched, at 0x80000160  csrw    instret,zero
      reference  pc=0x80000160 insn=0xc0201073 trapped (illegal instruction)
      design     pc=0x80000160 insn=0xc0201073 no write
      from sequence 19 (xtrap: write to instret, cause 2 (illegal instruction))
      note: Sail disagrees with the design as well, first at signature word 281
            - two models against it
```

Three things can happen and each is a different sentence. Both models agreeing with the
design is the strongest statement this package can make; both disagreeing is almost
certainly the design. The two models disagreeing with *each other* is a finding about
the models rather than a verdict on the core, and it is said out loud and left alone
— the run's own result stays Spike's.

The second model is asked when the first comparison **failed**, which is where the whole
value is: a campaign spends its night on seeds that pass. `assay.py run` asks always,
having only one seed to be sure about.

**And it is compared instruction by instruction, not only at the end.** The signature is
an end state, so a disagreement in it says the memory came out different and nothing
about where — "somewhere in three thousand instructions" is not a bug report. Sail's
trace is read back into the same structure Spike's commit log is read into, so a run that
fails can say which address the two models parted at, which instruction word was there,
and what each of them did with it. On a thirty-sequence program with traps in it: 2313
retirements each, zero mismatches.

**Sail does not take an ISA string.** It takes a JSON configuration whose default has
every extension it knows switched on, and a model that implements more than the run
asked for cannot catch an instruction the generator should never have emitted — it
would execute it, agree with itself, and say nothing. So the configuration is built from
the run's own ISA string, and built by reading the model's own default rather than from
a list here, so that a newer Sail with more in it still works. The model validates what
it is given and refuses an incoherent one by name, which is how two of the rules in
`sail.py` were found rather than guessed.

One thing it cannot arbitrate: version 0.13.1 has **no hypervisor**. Everything this
package generates for that is Spike's word alone, and the arbitration says nothing
rather than saying the wrong thing.

## The document somebody else reads

Everything above prints to a terminal. That is the right shape for working and the wrong
shape for what verification actually produces: a piece of evidence that outlives the
session, that a colleague can review, that goes in a release or a design record.

```bash
ipw-assay report --core mycore:make --isa rv32imc_zicsr --seeds 50 -o report.md
ipw-assay report --core mycore:make --isa rv64gc --profile RVA22U64 --html -o report.html
```

It runs, shrinks every failure, and writes one self-contained document. Markdown by
default because it is diffable and reviewable; `--html` is a standalone page with no
external anything, for sending to somebody who will not open a terminal.

Three things it does that the terminal does not:

**It says what was *not* established.** Every section pairs what was settled with what
was not, and the two are never added together. A report listing only what passed would be
advertising, and would be read as a claim about the whole design rather than about the
part that was tested.

**It keeps three numbers, not two.** Agreed, disagreed, and *established nothing* - a
seed the design never finished, which is neither a pass nor a failure. Folding those into
either column is the one arithmetic mistake this package exists to make impossible.

**It says what it was.** Which design at which revision, which reference models, which
toolchain, which assembler and what that assembler could not encode, and the command that
produced it. None of it is typed in, because a field somebody fills in is a field
somebody gets wrong.

Nothing in it is maintained by hand: the profile ledger comes from `profiles.py`, the
limits from the table in `config.py`, the coverage from the counters a run filled in, and
what the assembler refused from asking the assembler. A report kept in step by hand would
drift from the tool it describes, and be worth less than no report at all.

## Picking this up

[`docs/architecture.md`](docs/architecture.md) is why the tool is shaped the way it is —
the seven decisions behind it and the reasoning for each. [`docs/profiles.md`](docs/profiles.md)
is the profile ledger in full: what each requirement means, and why a requirement nothing
here can establish is counted apart from the passes rather than inside them.

## The official tests

Two ways, and they answer different questions.

```bash
# One test, one process, about a second. For the twentieth look at a failing test.
python assay.py archtest --core mycore:make     --suite riscv-arch-test/riscv-test-suite --march rv32imc --filter rv32i_m/I

# The whole suite, properly, with a reference model and a report anybody will accept.
python assay.py riscof --core mycore:make --isa rv32imzicsr_zifencei --outdir riscof/ipweave
riscof run --config config.ini --suite riscv-arch-test/riscv-test-suite            --env riscv-arch-test/riscv-test-suite/env
```

`assay.py riscof` writes the whole plugin directory — the RISCOF plugin, `model_test.h`,
the linker script, and an ISA yaml generated from your ISA string. The yaml is what
decides which tests apply, so it is computed from one place rather than written twice;
the three shipped shapes validate clean against `riscv-config`.

The direct runner exists because RISCOF rebuilds the world each time and you will look at
one failing test more than once. When it disagrees it names the signature word and the
address, and the design's own trace is still in hand — so the lockstep and the shrinker
that work on generated programs work on an architectural test that fails.

It also *diagnoses* rather than shrugs. A design compiled without `c` running a suite
whose macros align with `c.nop` padding gets told exactly that, not "the test did not
finish".

## What it does

Builds a program out of short **sequences** — a few instructions each, drawn from a
weighted **mix** — that claim their registers from a shared pool, then shuffles the
sequences together so that unrelated work collides in the pipeline. Runs the result on
Spike and on your core. Compares.

The shuffle is the point. A program with its sequences laid end to end tests each
sequence; the same sequences interleaved test the machine *between* them — the forwarding
path that has to notice a register two instructions back, the scoreboard entry that has to
still be there when the divide finishes, the store buffer that has to answer a load. The
register pool is what makes the shuffle safe: a sequence owns its registers until its last
instruction is placed, so no two overlapping sequences can touch the same one.

## What is the same as upstream

| | |
| --- | --- |
| Sequences over a register pool, interleaved | the same idea, the same reason |
| A weighted mix by family | the same families, and the same default weights from `config/default.config` |
| End-of-run signature, diffed against Spike | the same check |
| An overnight run that keeps going until *n* failures | `campaign --minutes` |

## What is different

**Failures are shrunk.** A torture program that diverges is thousands of instructions
long and every one of them is suspect. Delta debugging cuts it down to the smallest
program that still fails *the same way* — same kind of divergence, same instruction word —
and that is usually not just a bug report but a diagnosis. Upstream has no shrinker.

`examples/shrink_demo.py`, against a Jacquard deliberately made wrong about `srai` and
nothing else:

```
generated 95 instructions from 40 sequences

  DIVERGED: 262 instruction(s) compared by lockstep
  result: after 60 matched, at 0x800000f0  srai  t0,s3,0x14
      reference  pc=0x800000f0 insn=0x4149d293 x5=0x61
      design     pc=0x800000f0 insn=0x4149d293 x5=0x60
      from sequence 10 (xalu)

  [  3] down to 46      [  9] down to 4
  [  5] down to 24      [ 11] down to 3
  [  7] down to 11      [ 13] down to 1

95 -> 1 instructions (minimal) in 14 candidate(s)

the body that still fails:
    srai x5, x19, 20
```

One instruction, from fourteen candidates. That is the whole argument.

**The comparison is instruction by instruction.** With an RVFI trace port the lockstep
stops at the first divergence and names it: this program counter, this instruction, your
core wrote *this* and Spike wrote *that*, and it came from the load-use hazard at distance
one that sequence 37 was built to provoke. Without a trace port it falls back to the
signature, which is what upstream compares and works against any core, an external
simulator, or a board. Both run when both can.

**Hazards are aimed at.** A `hazard` family generates a write and the read of it at a
chosen distance, a load and its immediate use, a long-latency result read while it is
still in flight, a write in a branch's shadow, a store and a load of one address. Weighted
random reaches all of these eventually and none of them reliably.

**Coverage steers the mix.** What a run has and has not reached — opcodes generated,
opcodes actually *executed*, branch directions taken, hazard distances produced — pushes
the weights towards the gaps. Upstream generates from fixed weights all night and cannot
tell you that in eleven hours it never once emitted `remu` or took a backward branch.

**Failures become a corpus.** Every one is written out as a seed, a fingerprint and the
shrunk program, and `assay.py gate` replays the lot. The program is the regression test;
the seed answers a different question — does this generator still make it? — and a drift
is reported as news rather than as a pass.

**Traps are taken and returned from.** A handler is installed wherever the ISA allows one
— not only for the `xtrap` family, but because without it `mtvec` is zero and *any* trap
runs off to address zero and dies there looking like a core that stopped for no reason.
Only the three causes two implementations must agree about are generated deliberately;
see `assay.py limits` for why the rest are not.

**More instructions.** RV32 and RV64, `I M A F D C V`, `Zicsr`, `Zifencei`,
`Zba Zbb Zbc Zbs Zbkb`, `B`, `Zmmul`, and the profile extensions — `Zicond`, `Zimop`,
`Zcmop`, `Zcb`, `Zihintpause`, `Zihintntl`, `Zicbom Zicbop Zicboz`, `Zfhmin`, `Zfh`,
`Zfa`, `Zacas`, `Zabha`, `Zvbb`, `Zvfhmin`, `Zawrs`, `Svinval`, `H`. Over eight
hundred rows, including the
compressed forms an assembler would never choose, and all of RVV 1.0's arithmetic:
integer, floating point, fixed point, bit manipulation, and every combination of access
shape. Adding one is adding a
line to a table, because there are no encodings in this package: the generator emits
assembly, the toolchain encodes it, and both models run the *same image*. A second encoder
in here would be a third opinion about what `fnmsub.d` looks like, and the first thing it
would find is its own bugs.

**The privileged extensions a profile mandates.** `Svinval` takes `sfence.vma` apart
— a fence, invalidations naming a page each, a fence — so an implementation can
overlap them instead of stopping the pipeline once per page. `Svpbmt` puts a memory type
in two bits of a page-table entry, on the pages the body only loads and stores through
and never on the one it fetches instructions from. And `Supm` is the architecture
agreeing to look the other way: with `PMLEN=7` the top seven bits of a data address are
ignored, so a store through a pointer with rubbish up there has to land at the untagged
address. Which register carries that grant is the whole of the difference between `Supm`
and `Ssnpm`, and putting the enable in the wrong one is a store access fault at the first
tagged pointer.

**The hypervisor, including a guest.** `hlv` and `hsv` are not loads
and stores with a different name: what makes them different is *whose* translation they
use. An ordinary load from HS-mode goes through `satp`; these go through `vsatp` and then
`hgatp` — the guest's tables and then the host's — which is how a hypervisor reaches
into a guest's memory without becoming the guest. `hlvx` reads as if *fetching*, so it
wants execute permission rather than read permission, which is how one emulates an
instruction it trapped on. Both fences go in too, one per stage.

Beside them, the three guest registers a profile pins the contents of. They are WARL, so
writing a value and reading it back would ordinarily say nothing at all — an
implementation may keep what it likes — and it says something only because a mandate
has taken the freedom away: `vstvec` must hold Direct, and `vsatp` and `hgatp` must
support every mode `satp` does. Both address registers go back to Bare afterwards,
because one left holding a mode with a page-table pointer of zero would send the next
`hlv` walking a table at physical address zero.

And a **guest**, which is what the last two mandates needed. VS-mode is not a fourth
privilege level — it is supervisor mode with `mstatus.MPV` set — and in it the name
`stval` *is* `vstval`, `stvec` is `vstvec`, `satp` is `vsatp`. So `virtualized=True`
reuses the trap handler, the page tables and the faulting families exactly as they are,
and only the way in differs. What proves the trap went to VS-mode is negative and all the
stronger for it: the program never writes `stvec`, so a handler that ran can only have
been reached through `vstvec`.

Beside it, `guest_pages=True` gives the **second** stage of translation one gigapage and
a hole everywhere else, and a hypervisor load into the hole is the only thing that writes
`htval`. The hypervisor rather than the guest, which sounds backwards until you read
`hedeleg`: bits 20 to 23 are read-only zero, so a guest can never be handed the fault
that says its own memory is missing.

**Sixteen entries that all say the same thing.** `Svnapot` maps sixty-four kilobytes
through sixteen page-table entries holding one physical page number between them, and the
hardware puts the address's own low bits back on the way out. It is not a bigger page,
which is the whole test: a core that ignored the `N` bit sends all sixteen to the same
place, and fifteen sixteenths of the region reads back the wrong bytes. The region is
built two levels below the leaves the rest of the tables use, and the test memory is
aligned to sixty-four kilobytes so that an offset into one is the same offset into the
other.

**What a trap left behind.** A profile mandating `Sstvala` promises that `stval` holds
the address that faulted, so `stval_in_log=` puts it in the supervisor trap log beside
the cause. It records a zero for every cause the specification does *not* pin — an
illegal instruction may leave the instruction bits or zero, and both are correct — so
the comparison never sees a number two conforming cores could differ about. There is no
machine-mode equivalent to claim, which is why `mtval` is never recorded: the privileged
architecture says it *may* hold the faulting address, and one that writes zero is
conforming.

**Address translation.** `paging=` takes `sv32`, `sv39`, `sv48` or `sv57`: it builds the
page table, turns it on, and runs the body in supervisor mode. Under Sv48 and Sv57 a root
entry covers half a terabyte and a quarter of a petabyte, and a leaf entry has to be
aligned to the page it maps — so the walk has to *descend*, and a core whose walker was
written for three levels passes everything above and stops there.

Four regions are described, and three of them are the point: the program's own memory,
the same memory at a second virtual address so that a store through one has to be visible
through the other, the same memory again with no write permission, and a fourth address
range mapped by nothing at all. The `xfault` family reaches for the last two — a store to
a page that is not writable and a load from an address nothing maps have cause numbers
the specification pins down exactly, unlike an access fault in a bare memory system,
where what is mapped is whatever somebody wired up.

**Privilege.** `privilege=` takes `machine`, `supervisor` or `user`, independently of
paging in both directions — a body can run in user mode with translation off, which is
the cheapest way to find a core that checks privilege in the wrong place. Below machine
mode the generator's own behaviour changes with it: an `ecall` reports a different cause,
the `xcsr` family stops offering registers the body may no longer name, and `xtrap` gains
the instructions the body is no longer allowed to execute — `mret`, `sfence.vma`, and a
machine-mode CSR named from anywhere below.

**Sub-programs and a call stack.** The `xcall` family calls into generated sub-programs,
and they call further into the ones after them. A call is the only structure here that has
to come *back*, and the only one a return-address predictor has anything to predict about.
Each sub-program saves and restores every register it touches, which is what makes a call
safe to interleave into a body where several other sequences have live values in flight.

**Vector.** Most of RVV is not more table rows, and each part is a family that controls
`vtype` itself rather than inheriting it — because every one of them can produce an
instruction an assembler accepts and a machine refuses:

- **Floating point.** An eight-bit float is not a format RISC-V has, so a vector
  floating-point instruction executed while `vtype` says `e8` is *reserved*. `vfgen` sets
  a width it can vouch for. Almost every row is `delicate`: there is nowhere in the
  encoding to put a rounding mode, so every one of them reads `frm`.
- **Masking.** `vmask` runs arithmetic only where a bit in `v0` says so. It pins the
  policy to `mu`, because under `ma` an element the mask turned off may be left
  undisturbed *or* set to all ones and both are correct — so nothing generated under `ma`
  can be compared at all.
- **Addresses that are not consecutive.** `vstride` walks memory at a constant distance —
  including zero, which reads one address `vl` times, and backwards. `vindexed` gathers
  and scatters with one address per element, and it *builds* its index vector with
  `vid.v` rather than finding one, because a vector register here starts from the test
  block and an index vector of arbitrary 64-bit numbers points nowhere the program owns.
  A *store* is offered neither a stride of zero nor a shared index: element accesses
  within one of those are unordered with respect to each other, so which of the values
  survives is the implementation's, and a divergence there would not be a bug.
- **Register groups.** `vlmul` writes one instruction across two, four or eight registers,
  and down to an eighth of one. A group of four does not begin at `v6`; naming one is
  reserved, and the assembler cannot tell. Reductions and comparisons go in too, and they
  are the awkward ones — a reduction folds a group down to one number and a comparison
  produces a mask, so some of their operands are groups and some are not. `SEW / LMUL` may
  not exceed ELEN either, so only eight-bit elements go all the way down.
- **Segments.** `vlseg3e32.v` reads x, y, z, x, y, z … and leaves all the x's in one
  register, all the y's in the next. The one access shape needing several register groups
  allocated in step, and they have to be *consecutive* rather than aligned — a weaker
  constraint than a group's, and a different one, so the pool has a `run()` beside its
  `group()`.
- **Segments at addresses that are not consecutive**, which is the *product* of the two
  above and a family rather than a row in either, because each parent contributes a
  constraint and the two are different shapes. `vlsseg3e32.v` puts a whole segment at
  each of `vl` strided addresses; `vluxseg3ei32.v` puts one at each address a vector
  register names, and its index vector is scaled by a *multiply* rather than a shift
  because three fields of four bytes is not a power of two. An index element is only
  SEW bits wide, so at eight-bit elements it cannot hold an offset past 255 however
  large the block is — a bound the unsegmented gather never reaches and this one does.
- **Widening, which is narrowing backwards.** `vwsll` packs narrow elements into wide
  ones and the two half-precision conversions turn a half into a single and back, and
  all three write a destination twice the width of their sources — so at LMUL one the
  destination is the register *pair*, and a pair starts at an even register. `Zvfhmin`
  is exactly those two conversions, which is the whole of what RVA23 makes mandatory:
  a core must be able to convert a half, and is not required to add two.
- **Fixed point.** Saturating adds, averaging adds, the fractional multiply, the scaling
  shifts and the narrowing clip — the integer operations that keep a number's *scale*.
  Four of the six round, there is nowhere in the encoding to say how, and so they read
  `vxrm`, which the manual says may hold anything at all after reset. `vfixed` writes it
  and no other family may emit them. The clip is also the one instruction here whose two
  vector operands are different sizes: its source is a pair of registers where its
  destination is one, and a pair starts at an even register.
- **A vector shorter than the register.** `vsetvli` with `x0` asks for the longest one
  there is, which is what leaves no tail; `vsetivli` and `vsetvl` name a length instead,
  and both are comparable only at or below VLMAX — above it the manual pins `vl` to a
  *range* and lets an implementation choose inside it. A named length leaves a tail, so
  the tail policy stops being a choice and is pinned to `tu`. `vsetvl` takes its whole
  `vtype` from a register, and that value is read back out of the `vtype` CSR rather than
  assembled here: a wrong field position would not trap, it would set `vill`, and the
  first sign of trouble would be some later instruction being illegal.

**Trap delegation.** `delegate=` names the exception causes that go to supervisor mode
instead of machine mode. There is then a second handler at `stvec` and a second log, and
that is the whole of how a routing mistake shows up: the record a delegated trap leaves is
identical either way — same cause, same address — so only *where it landed* says which
handler wrote it.

One architectural constraint is worth knowing before it costs an afternoon: delegation
cannot be combined with a user-mode body under a page table. Every page is marked
user-accessible so the body can fetch through it, and supervisor mode may not fetch from a
user page at all. `sstatus.SUM` does not help — it covers supervisor loads and stores and
has no effect on instruction fetches — so the delegated handler would fault on its own
first instruction, in machine mode, which looks exactly like a core that cannot delegate.
That combination is refused with the reason.

**Physical memory protection.** `pmp_regions=16` configures one region nothing below
machine mode may touch and one below it in priority that permits everything else — and
the `xpmp` family reaches into the first. This is the other half of "what may this program
touch", and a different half from address translation: translation says where an address
*goes*, protection says whether the place it went to may be touched at all, and it is
checked afterwards and separately. A core that implemented either in terms of the other
passes both alone and fails `rv64gc-pmp`, which does both at once.

The count is stated rather than discovered, for the same reason `vlen` is: how many
entries a core implements is implementation-defined, and a write to one above that number
goes nowhere. Asking for it with the body in machine mode is refused rather than quietly
testing nothing — an unlocked entry does not apply to machine mode.

**Hints.** The encodings that are architecturally required to do nothing. No compiler
emits `c.mv x0, x5`, so no compiler-shaped test reaches the decoder that way — and a core
that trapped on one, or let one write a register, is wrong in a way nothing else here
would catch.

**Self-modifying code.** The `xfencei` family writes an instruction, fences, and executes
it. Neither word is encoded here: the replacement is assembled into a block the program
jumps over and loaded out of it as data.

**Campaigns run in parallel.** `--workers 8` puts seeds in separate processes; coverage is
folded back between batches and steering carries on. `--coverage-file` carries what has
been reached from one campaign into the next, so an overnight run continues rather than
starting blind.

## What it refuses to do

**Read a CSR the two sides have no reason to agree about.** `cycle`, `time` and `instret`
count something Spike and your core will differ about; `mvendorid` and friends are
*supposed* to differ; `mstatus` and `mtvec` change what every later instruction means. Most
CSRs are also WARL — an implementation may legally refuse a value it does not like and
read back something else — so two correct cores can disagree about what a write left
behind. `mscratch` and `sscratch` are the exceptions the specification actually pins down:
each is an XLEN-bit read/write register with no further constraint. Those, and the
floating-point control registers when the configuration says the two sides agree about
rounding. Nothing else.

The read-only ones are still *written to*, on purpose: writing a read-only CSR is an
illegal instruction at every privilege level, and that is a comparable trap and a real
test. Reading a machine-mode one from below machine mode is another.

**Generate a program that might not terminate.** Control flow goes forward, never back. A
loop, when a configuration asks for one, is written by the scaffolding where its trip
count is known — because a run that hangs tells you nothing about the core.

**Report a check that did not run as a pass.** `ran` and `passed` are separate everywhere.
No toolchain, no Spike, a design that ran out of cycles, a reference model that never
reached the end of the program: each is reported as what it is and counted apart from both
the passes and the failures. A campaign that reports nine hundred passes and a hundred
that could not run says so on the first line.

## Naming your core

`--core module:name`, where *name* is a callable taking **no arguments** that returns an
elaborated component:

```python
# examples/jacquard_target.py
from ipw_jacquard import config
from ipw_jacquard.core import Jacquard

def lite():
    return Jacquard(config.LITE.with_(rvfi=True))
```

No arguments deliberately: a core's constructor takes a configuration object this package
knows nothing about, and guessing at one is how a torture run comes to be testing a
different core from the one you meant.

For a design that is not elaborated here — a Verilator build, a licensed simulator, a
board — `ipw_assay.targets.CommandTarget` drives a command line and hands its output to
a parser you supply. Signature comparison works against anything with a memory bus.

## The configurations we ship

| | ISA | sequences | memory | segments |
| --- | --- | --- | --- | --- |
| `rv32i` | rv32i | 200 | 1 KiB | 8 |
| `rv32im` | rv32im + Zicsr | 200 | 1 KiB | 8 |
| `rv64ima` | rv64ima + Zicsr | 300 | 1 KiB | 8 |
| `rv64gc` | rv64gc + Zba/Zbb/Zbs | 400 | 4 KiB | 16 |
| `rv64gc-dyn` | the same, rounding read from `frm` | 400 | 4 KiB | 16 |
| `rv64gcv` | + RVV 1.0, VLEN 128, masked, grouped and fixed point | 400 | 4 KiB | 16 |
| `rv32im-sv32` | rv32im under Sv32, in supervisor mode | 200 | 1 KiB | 8 |
| `rv64gc-sv39` | rv64gc under Sv39, in supervisor mode | 400 | 4 KiB | 16 |
| `rv64gc-sv48` | four levels, so the root cannot hold the leaves | 400 | 4 KiB | 16 |
| `rv64gc-sv57` | five levels, as deep as RISC-V goes | 400 | 4 KiB | 16 |
| `rv32im-user` | rv32im in user mode, untranslated | 200 | 1 KiB | 8 |
| `rv64gc-user` | rv64gc in user mode under Sv39 | 400 | 4 KiB | 16 |
| `rv64gc-pmp` | the same, with protection regions as well | 400 | 4 KiB | 16 |
| `rv64gc-deleg` | everything delegable sent to supervisor mode | 400 | 4 KiB | 16 |

Everything is overridable — `--isa`, `--nseqs`, `--memsize`, `--segments`, `--loop`,
`--mix xmuldiv=40`. A configuration that cannot produce a program is refused when it is
made, not when the assembler fails three minutes into a run.

## State

What has actually been run, rather than what is intended:

* **Generation** — eleven configurations generate, assemble under
  `riscv64-unknown-elf-gcc`, run to their own end on Spike, and leave a signature exactly
  as long as the program said it would be: RV32I, RV32IM, RV64IMA, RV64GC, RV64GC with
  dynamic rounding, RV32E, misaligned accesses, deliberate compression, deliberate traps,
  RVV at VLEN 128 and 512, and Sv32 and Sv39 with the body in supervisor mode. Every one
  of the 800-odd rows in the instruction table is assembled once as its own check.
  Nothing is stored outside the memory the program owns.
* **Address translation** — thousands of instructions retired in supervisor mode, and
  every store through one virtual address read back through its alias.
* **Comparison** — a design mutated to be wrong about one instruction is caught, and the
  report names that instruction and the sequence that produced it. A comparison that could
  not happen is never reported as a pass; there is a test for each way it can fail to
  happen.
* **Shrinking** — a failing program is reduced to a handful of instructions that still
  contain the one responsible and still fail with the same fingerprint.
* **Corpus** — record, read back, replay, and a gate that reports a fixed bug as fixed and
  an entry that could not run as a failure.
* **ipw-jacquard** — 12 seeds of `rv32im` at 40 sequences each, and the whole `slow` suite:
  no disagreement with Spike. That is not a claim that the core is correct; it is 12 seeds.
* **Profiles** — every ratified profile reads back as an ISA string and checks against a
  claim. **Nine of the ten ratified profiles have no `absent` mandates left**, or none
  that is ours: every one of RVI20, RVA20, RVA22 and the 2023 user-mode profiles is
  clear, and RVB23S64's last gap is `Sstc`, which the reference model segfaults on.
  RVA23S64 has that and two more — `Shvstvala` and `Shtvala`, which are about what a
  *guest* fault leaves behind and need a guest that is not built. The tool names every
  one.

Things the generator got wrong and now does not, all worth knowing if you write another
one:

* **`mstatus.FS` starts Off.** Every floating-point instruction — including a load into a
  float register — is an illegal instruction until it is written. A generated program that
  skips it traps on its first `fld`, and the symptom looks exactly like a core with no FPU.
  See `program.py`, and the refusal of `f` without `Zicsr` in `config.py`: the unit could
  not be turned on at all.
* **A narrow store's lane comes from the bus width, not the access width.** A halfword
  store to `...4` on a 64-bit bus is four lanes up. Taking the lane from the access size
  writes the wrong two bytes on RV64 and is correct on RV32, which is the worst place for
  a bug to be.

**Not built, and named rather than quietly omitted:** run `python assay.py limits`. It
distinguishes the two cases that matter — things that *cannot* be compared this way
(interrupts, multiple harts, misaligned-access behaviour, a vector store whose elements
write one address) from things nobody has done yet. As of 0.1.0 the second list is
**empty**: every extension this package names has instructions for it. Everything left on
the list is there because two correct implementations may legitimately differ about it,
and the reason is written down beside each one.

## Installing it

```bash
pip install ipw-assay                 # the generator, the comparison, the profiles
pip install "ipw-assay[cgf]"          # ...and the coverage and riscv-config formats
pip install "ipw-assay[sim]"          # ...and simulating a core in this process
```

Installed, the command is `ipw-assay`; from a checkout it is `python assay.py`, and they
are the same thing. Nothing else is bundled: the toolchain and the reference models are
external on purpose, and `ipw-assay selftest` says which of them it found.

The second reference model is fetched rather than built — one pinned, checksummed
binary, no compiler:

```bash
$(python -c "import ipw_assay.plugins as p; print(p.path('fetch-sail.sh'))")
./tools/fetch-sail.sh                 # the same script, from a checkout
```

## What it needs

* `riscv64-unknown-elf-{gcc,objcopy,objdump,nm}` — to assemble, link and place.
* `spike` — the reference model.
* `ipweave` — only to simulate a core in this process. A run against an external simulator
  does not need it.

On Windows these are reached through WSL; `python assay.py selftest` says what was found
and, when something is missing, what to install. **There is no fallback.** Without the
toolchain there is no test, and this package says so rather than doing something it can
manage on its own and calling it a result.

## Layout

| | |
| --- | --- |
| [`isa.py`](ipw_assay/isa.py) | every instruction that may be emitted, as a table |
| [`config.py`](ipw_assay/config.py) | what a run is, and the named ones we ship |
| [`pool.py`](ipw_assay/pool.py) | register ownership — why the shuffle is safe |
| [`sequences.py`](ipw_assay/sequences.py) | the families, and what each provokes |
| [`generate.py`](ipw_assay/generate.py) | the mix, and the interleave |
| [`program.py`](ipw_assay/program.py) | the scaffolding, and the signature |
| [`toolchain.py`](ipw_assay/toolchain.py) | the external tools, and whether they are there |
| [`reference.py`](ipw_assay/reference.py) | Spike's commit log, as a trace |
| [`targets.py`](ipw_assay/targets.py) | running an image on a design |
| [`compare.py`](ipw_assay/compare.py) | the verdict, and who to blame for it |
| [`shrink.py`](ipw_assay/shrink.py) | delta debugging down to a reproducer |
| [`coverage.py`](ipw_assay/coverage.py) | what was reached, and what to generate next |
| [`corpus.py`](ipw_assay/corpus.py) | failures, kept and replayable |
| [`run.py`](ipw_assay/run.py) | one test, and a campaign of them |
| [`parallel.py`](ipw_assay/parallel.py) | many seeds at once, in separate processes |
| [`selfcheck.py`](ipw_assay/selfcheck.py) | the reference model as a design, to prove the rig |
| [`archtest.py`](ipw_assay/archtest.py) | signatures, the model header, and the exporter |
| [`suite.py`](ipw_assay/suite.py) | running the official tests, and diagnosing them |
| [`riscof.py`](ipw_assay/riscof.py) | everything RISCOF has to be told, from one ISA string |
| [`cgf.py`](ipw_assay/cgf.py) | the coverage format: reading it, measuring it, aiming at it |
| [`profiles.py`](ipw_assay/profiles.py) | the ratified profiles, and what can be established about each requirement |
| [`report.py`](ipw_assay/report.py) | the document somebody else reads, and why it is shaped the way it is |
| [`sail.py`](ipw_assay/sail.py) | the second reference model, and the configuration it has to be given |
| [`directed.py`](ipw_assay/directed.py) | programs aimed at something rather than programs that happened |
| [`riscof/`](riscof/) | the RISCOF plugin itself |

## Provenance and licence

**Apache-2.0.** See [`LICENSE`](LICENSE).

See [`PROVENANCE.md`](PROVENANCE.md) for what was read to write this, what was not, and
the citations — which matter more in an open-source package than in a private one. In
short: no source from riscv-torture, riscv-dv, riscv-ctg or Force-RISCV was translated,
copied or consulted line by line while writing this, and this package contains no code
from any of them. The external tools it drives — the RISC-V GNU toolchain, Spike, Sail —
are invoked as separate programs, are not distributed with it, and nothing here is a
derivative work of any of them.
