Browsing the trial store

The trial store is the project's memory: every backtest, optimization, walk-forward, and verdict across every session, with provenance, dedup identity, and each trial's out-of-sample return series. trials list, trials show, and trials best are how you ask it questions without writing SQL.

It is deliberately a CLI browser, not a UI — the store is SQLite, the queries are cheap, and what was missing was filters, a detail view, and honest presentation.

The browse commands are read-only: list, show, best, campaign, analyze, compare, status and archives change nothing. There is no delete and no edit of a trial: the journal is append-only history, and the browser must not make corrupting it convenient. The commands that do write are named where they appear: promote writes a config file, mark-contaminated records a quarantine and archive retires the journal to a manifest (see Maintenance), and rebuild reconstructs the derived index from the journal.

trials list — what have we tried?

quantred trials list --strategy demo_trend --min-sharpe 1.0 --sort dsr --limit 20

ID            KIND        STRATEGY           SHARPE BASIS          DSR  PROMO  ACCT  TS
a1b2c3d4e5f6  walkforward demo_trend          1.512 oos agg      0.903    yes     3  2025-03-01 09:14:22 EST
…

Showing 20 of 4318 matching trials (use --limit/--offset for more).
3 in-sample row(s) (optimize/backtest/alpha) left out of this ranking — none was validated out of sample. Pass --include-in-sample to see them.

The SHARPE column is not one quantity. BASIS says which: oos agg is a walk-forward's out-of-sample aggregate over its folds, oos median a research round's median fold, and IN-SAMPLE is either a backtest's one pass over the whole window or an optimize row's best-of-N candidate. trials show spells the basis out in full, and --json rows carry sharpe_basis and in_sample.

Flag	Meaning
--strategy / --kind	Filter to one strategy, or one kind (backtest, optimize, walkforward, alpha, research, verdict)
--symbols	Filter to one universe — matched on the normalized universe, so NVDA,MSFT and msft, nvda find the same trials
--since / --until	Filter by when the trial was recorded. A date is a New York market date and both ends include it: --until 2026-08-01 includes trials from that evening
--min-sharpe	Only trials with a recorded Sharpe at or above this; in-sample kinds are left out unless --include-in-sample
--gates-passed	Only trials recorded as promotable
--sort date|sharpe|dsr	Sort order (default date). Sorting by sharpe or dsr is a ranking, so optimize, backtest and alpha rows are left out of it unless --include-in-sample, as trials best does
--include-in-sample	Rank in-sample rows too. They stay marked IN-SAMPLE. Naming one with --kind optimize shows exactly those rows without it
--limit / --offset	Paging — done in SQL, so a store with tens of thousands of rows stays fast
--accounting / --all-accounting	Stay within one accounting version (default), or span them deliberately
--json	Emit the rows as JSON

Two rules the listing follows:

• Absent is not zero. A trial from before a field existed, or of a kind that never produces one, renders as —. A pre-return-series trial did not fail to persist returns, and a forecast did not score a Sharpe of 0.00.
• Truncation is never silent. The footer always says how many rows matched versus how many were shown.
• Exclusions are counted. A ranking says how many in-sample rows it left out, and a --since/--until filter says how many rows it could not place because their recorded time is unknown. Those rows are not guessed a date; drop the bound to see them.

Rows from the research agent are worth reading with that in mind. Their SHARPE is the median fold, while trials campaign prints the aggregate over all folds under EVIDENCE, and says so. A round that proposed new code is listed under draft:<Class>:<hash> as its strategy, never the base strategy's name.

trials query still works as an alias for trials list for one release.

trials show — what did trial X actually run?

quantred trials show a1b2c3d4e5f6
quantred trials show a1b2          # a unique prefix of at least four characters works too

Anywhere a trial id is taken - show, analyze, compare, campaign, promote, mark-contaminated --id, and the MCP tools - a unique prefix of four or more characters stands for the full id. A prefix two trials share is refused with both listed, never guessed.

Prints everything the store knows about one trial: its full params (including the folded _cost, _limits and data-vintage keys that make up its dedup identity — _limits is the book the run was given, so two otherwise identical runs at different position caps are different trials rather than one served twice), provenance (git SHA, timestamp, accounting version), headline metrics, and what was stored alongside it —

  Stored alongside this trial:
    return series : 361 periods, 2024-01-27 → 2025-01-21
    proposed book : 4 names, with factor exposures
    trade table   : — (not recorded — re-run with --record-trades, or record_trades=true over MCP)

  Reused by 2 later trial(s) with the same identity:
    c3d4e5f6a1b2  2025-04-02T08:11:40  (backtest)

That last section is the reverse of the memoization lookup: a number served from the store is traceable in both directions, from the reuse back to its origin and from the origin forward to everything it was reused for.

For a trial that proposed a portfolio (a verdict), show lists the proposed book itself — each symbol and its weight, largest first, plus active weights when stored — and says how many names it left out past 25. --json carries the whole book.

A walk-forward's show and campaign also report its holdout, read from the journal (the index never kept it): the days, Sharpe, CAGR and trade count it was scored at, or one of three honest absences — none requested, requested but produced no result, or requested (N days) but its result was not retained for a trial recorded before the journal kept holdout results. A reused walk-forward (CLI, or the MCP tool's holdout key) repeats the same statement rather than dropping it.

An unknown id exits non-zero with a plain message.

Promoting a validated trial

walkforward --save-config writes the chosen config after a validation, so saving a config you have already validated means validating it again — and the memo only serves an identical recipe, which it is not once a seed has changed to ask a different question.

trials promote reads the recorded trial instead:

quantred trials promote 50fd06209f49 --save-config alpha.json

Promoted trial 50fd06209f49 -> ~/.tradeflow/configs/alpha.json
  recipe   : embargo_days=5, holdout_days=0, max_evals=50, method=grid, mode=anchored, objective=sharpe_ratio, seed=42, test_days=63, train_days=252
  evidence : accounting v6, 1 trial id(s), family has tried 50
  strategy 'demo_trend'  universe 61 symbols resolved from 85 candidates
  Saving a config never trades it - a human promotes it to live.

This is not a fast path through validation. The trial store holds what a config needs because a real validation put it there, so reading from it cannot bless state that was never validated. A --skip-validation flag could, and would be reached for exactly when someone is in a hurry — which is when it matters most that a saved config means what it says.

A trial that did not clear its gates is refused, because promoting one silently would put a config on disk whose own provenance says it failed. --force saves it anyway with that verdict recorded in the file, and prints WARNING: this trial did not clear its gates; the config records that verdict. (a trial that was never gated, such as a backtest, gets its own warning saying it was not validated out of sample). A bare --save-config filename lands in configs/ under the state root, as for walkforward.

The universe comes from the journal, not the store: the store records a universe hash, not the symbols. That keeps the store passive and derived, and means a trial journaled before candidate lists were recorded reads as less complete rather than being backfilled into looking more authoritative than it is — such a config says so, and --re-resolve-universe will tell you it has no candidates to re-scan.

trials best — the honest leaderboard

quantred trials best --strategy demo_trend

  Top 5 by deflated Sharpe (validated runs only):
    #  ID            KIND        STRATEGY             DSR   SHARPE BASIS       FAMILY n_trials
    1  a1b2c3d4e5f6  walkforward demo_trend         0.903    1.512 oos agg                 87
    …

  FAMILY n_trials = configs tried so far on this row's strategy over its universe (this
  accounting version), counted now. A row's DSR was deflated by the trials of its own run,
  so two rows' DSRs are not comparable when their runs searched different amounts.

  Ranked by DEFLATED Sharpe, which already discounts for how many configs the
  family tried. Each row's family n_trials is shown: the larger it is, the more
  of the leader's edge is selection.

  This family has tried 87 configs. At that count the best raw Sharpe is largely
  selection — read the deflated column, not the rank.

A leaderboard is the most dangerous view in the project. Sorting a research campaign's trials by raw Sharpe and showing the winner is precisely the selection-bias trap the evaluation machinery exists to fight — and it would be worse coming from our own tooling, which lends it authority. So:

• Ranking is by deflated Sharpe by default.
• In-sample rows are excluded. An optimize row is the winner of a search — best-of-N by construction — so ranking one ranks the selection bias rather than any skill. A plain backtest is excluded too: it is one pass over its window, never split into a part it was fitted on and a part it was judged on, so its Sharpe is description rather than out-of-sample evidence. --include-in-sample opts back in and says plainly what that means; either way the count of excluded rows is reported, so the exclusion is never silent.
• Every row names its kind, because a validated walk-forward and a search's winning candidate look identical as numbers and mean opposite things as evidence.
• Every row shows its family's n_trials, and a large count gets its own warning line. That count is the configs tried so far on the row's strategy over its universe, read now; a row's DSR was deflated by the trials of its own run, so two rows' DSRs are not comparable when their runs searched different amounts. This is why a row from a short research round can outrank a longer walk-forward on DSR while showing a worse Sharpe: the ranking is by the recorded DSR, and each row also carries a BASIS cell saying what its Sharpe measures.
• A row with nothing to rank by is not listed. A verdict records neither a Sharpe nor a DSR; it used to sort last but still sit in the table. The count of such rows is reported (unranked_excluded).
• --rank-by sharpe is available and prints a caveat saying, in as many words, that the ordering does not correct for how many configs were tried.

These rules live in the payload, not only in the terminal formatting: --json carries the family counts and the caveat too, so an agent reading this over MCP sees the same context a human reads on screen.

trials campaign — what validated this?

quantred trials campaign a1b2c3d4e5f6

A walk-forward is not one row. It is a validation recipe — the folds, the embargo, the objective, the search method, and the cost model and book limits folded into its identity — plus a chosen parameter set plus the universe that was actually resolved. Promoting the winning row captures the middle one, so the config could not say what validated it.

All of it was already recorded. The recipe is in the journal, written as the trial's dedup identity; the resolved universe is there too, in full, because the store keeps only its hash. This reads them back — nothing is re-run.

The universe line names what the evidence covers, and considered says how many the scanner looked at to get there — the gap between the two is how the book was resolved. A long list elides with a count of what it hid, never silently. The seed is read from the validation recipe, which is where it lives; a run that recorded none says so rather than showing a default.

=== What validated a1b2c3d4e5f6 (walkforward) ===

  RECIPE — how it was validated. Survives an accounting bump.
    window          2024-01-01 → 2024-06-29
    embargo_days    5
    holdout_days    0
    max_evals       50
    method          grid
    mode            anchored
    objective       sharpe_ratio
    seed            42
    test_days       63
    train_days      252
    _cost           {'gross': False, 'commission_bps': 1.0, 'impact_eta': 0.3, 'borrow_bps': 50.0}   (folded into its identity)
    _limits         {'max_positions': 8}   (folded into its identity)
    _run            {'capital': 100000.0, 'benchmark': None}   (folded into its identity)
    universe        8 symbol(s): AAA, BBB, CCC, DDD, EEE, FFF, GGG, HHH
    considered      31 candidate(s) before the scan

  SET UP WITH — how the run was configured. Not part of its identity.
    capital         100,000.00
    scanner         none
    scan_as_of      2024-06-29T00:00:00-04:00
    cache           cache=False, offline=False, workers=1

  EVIDENCE — what was measured, under accounting v6. Valid only there.
    deflated_sharpe_ratio 0.903
    max_drawdown    4.10%
    profit_factor   1.31
    sharpe_ratio    1.12
    total_return    9.40%
    total_trades    88
    Holdout : none requested (--holdout-days 0)
    promotable      yes
    family n_trials 50
    trial ids       a1b2c3d4e5f6
    probes          — not recorded

  METADATA — about the record, not about the strategy.
    recorded        2026-03-01 09:14:22 EST
    git             a1b2c3d
    seed            42   (from the validation recipe)
    notes           — not recorded
    return series   quantred trials compare a1b2c3d4e5f6 <other-trial-id>
    trade table     — not recorded
    proposed book   — not recorded

(Illustrative figures.)

The four labels are the feature. They age differently, and a reader who cannot tell them apart will carry a stale number forward beside a recipe that is still perfectly good:

Section	What it is	What an accounting bump does to it
recipe	How it was validated	Nothing — it stays true
set up with	How the run was configured (capital, scanner, cache)	Nothing — it records context (capital also keys the run, through its own identity; see below)
evidence	What was measured	Invalidates all of it
metadata	About the record	Nothing

A section that could not be assembled says so with a reason. Most trial kinds have no separate recipe — a backtest's identity is its parameters — and the recipe section says that rather than coming back empty, which would read as "validated with no settings".

What the run was set up with

Beside the recipe, a trial records the context it ran under — the capital, the scanner and as-of clock its universe was resolved through, the cache policy its bars came under, the probes that ran and what they said, and any note left on the run.

Recording these facts changed no dedup key and invalidated nothing. Capital is the exception that keys a run anyway, through the run's own identity rather than this record: whole-share rounding makes a run at $4,000 a different book from one at $100,000, so a backtest or walk-forward at one capital is never served for a request at another.

Every fact reports whether it was recorded, not just its value. A recorded fact prints as its value; one that was not prints as — not recorded:

  capital         250,000.00
  scanner         — not recorded
  cache           — not recorded

A trial written before this existed reads as not recorded for all of them, which is what it is. Nothing is defaulted: a config that never recorded its capital is not a config that ran at zero, and filling one in would put a number in the record that nobody chose. trials promote carries the capital into the saved config when it was recorded, and writes no capital key when it was not.

The promoted config trades the book that was validated

position_limits is written into the config from the book the trial recorded, because omitting it does not leave the book unspecified — it resolves at load to the strategy class's default of one position. A config whose provenance says eight positions were validated would otherwise trade one, and nothing would say so.

A trial that recorded no limits at all — one from before they entered the dedup identity — writes no position_limits, rather than a book it never had.

It lands in the config, not beside it

trials promote writes this block into the config's own provenance.campaign. There is deliberately no separate campaign export format: save_config is the portability format, campaign material is a field of it, and configs written before this existed load unchanged and read as not materialised from a campaign.

Promotion also warns at promotion time, rather than only when the file is later loaded:

  WARNING: this trial was measured under accounting v3 and this engine is v6. The recipe still
  applies; the recorded metrics do not, and must not be compared with a fresh run.

Over MCP, get_campaign_material returns the same block for an agent to pass through to save_config verbatim — one schema on both surfaces.

trials analyze — what did those trades actually do?

quantred trials analyze a1b2c3d4e5f6
quantred trials analyze a1b2c3d4e5f6 --json

Exit-reason P&L, win and loss by reason, holding period, and per-trade excursion for one recorded run — the questions that otherwise mean opening SQLite.

=== Trades of a1b2c3d4e5f6 (backtest) ===
  strategy demo_trend | accounting v6 | recorded 2026-03-01T09:14:22
  table    : 40 trades

  Overall:
    Trades          40
    Wins / losses   28 / 12
    Win rate        70.0%
    Net P&L         $8,483
    Average win     $409
    Average loss    -$247
    Expectancy      $212
    Profit factor   3.86

  By exit reason:
    exit              trades   share       net P&L  win rate     avg win    avg loss
    TAKE_PROFIT           26   65.0%        $4,528     65.4%        $431       -$312
    STOP_LOSS             14   35.0%        $3,955     78.6%        $374        -$54

  Held     : median 6.5 days, p25 5.8, p75 7.2, max 8.0 (40 measured, 0 not)
  Excursion: per-trade excursion — not the book's aggregate open drawdown
    adverse     median 2.84%, p90 3.85%, max 5.23% (40 measured, 0 not)
    favourable  median 4.88%, p90 7.06%, max 9.04% (40 measured, 0 not)

(Illustrative figures.) It grades nothing — the register is the one execution-report sets: report the number, say what it does not cover, leave the judgement alone.

Three things it will not do:

• Sum a capped table. If the run's trades were truncated at the storage ceiling, there are no totals and the command exits non-zero saying why. --allow-partial computes them anyway, labels every number as covering the stored rows only, and still draws no concentration verdict — which exit carried a run is a claim about all of its trades.
• Report a missing column as a zero distribution. A table with no entry_time says the holding period is not computable from it, rather than reporting that trades lasted no time.
• Confuse per-trade excursion with the book's. A position deep underwater that is a small fraction of the book did not put the book that far underwater. The label is on every excursion figure because that conflation is what makes people rewrite a result they had measured correctly.

trials compare — are these two results one result?

quantred trials compare a1b2c3d4e5f6 b2c3d4e5f6a1

=== Return-series comparison ===
  a1b2c3d4e5f6     300 periods, 2024-01-02 → 2024-10-27  (v5, demo_trend, walkforward: stitched out-of-sample folds)
  b2c3d4e5f6a1     300 periods, 2024-01-02 → 2024-10-27  (v5, demo_trend, walkforward: stitched out-of-sample folds)

  a1b2c3d4e5f6 vs b2c3d4e5f6a1  overlap 300 periods, 2024-01-02..2024-10-27
      correlation +0.98 [+0.98, +0.99]

  1 of 1 pair(s) compared; 0 refused. Minimum overlap 60 periods.
  Highest: a1b2c3d4e5f6 vs b2c3d4e5f6a1 at +0.98.
  At that level they are one bet held twice, however differently they were
  parameterised — promoting both counts a single result as two.

(Illustrative figures.) Each series names its kind, and a pair of unlike series — a stitched out-of-sample walk-forward against a full-window in-sample backtest — is flagged UNLIKE SERIES, because the overlap between them is arithmetic over two different things. Two candidates that correlate near 1.0 are one candidate. A campaign that promotes both believes it has two findings and has one.

Pairs get refused, not caveated. A correlation is a claim about a relationship and there is no partial version of one:

Refusal	Why
Fewer than --min-overlap shared dates (default 60)	A correlation over a handful of dates is an error bar wearing two decimals
Different accounting versions	The two series came from engines that compute different things. --across-accounting computes it anyway and marks the pair incomparable
Either series not recorded	Not every trial kind persists one
Either series flat over the overlap	Nothing to correlate

Every correlation carries a 95% interval, so one resting on a thin overlap arrives visibly wide rather than merely short of decimals. In --json, read pairs alongside matrix: the matrix holds null where nothing was computed, because a zero there is the strong claim that two results move independently — exactly what a refusal cannot say.

Both commands are also MCP tools (analyze_trial, compare_trials), and both journal nothing.

Keeping trade tables (--record-trades)

backtest and walkforward accept --record-trades, which journals the run's trade table (entries, exits, per-trade P&L, costs, MAE/MFE) with its trial so trials show can render it.

quantred backtest --symbols NVDA,AAPL --start 2024-01-01 --end 2024-12-31 --record-trades

It is opt-in on purpose: a long optimization campaign multiplying thousands of candidates by hundreds of trades each is exactly the storage nobody asked for. Use it for the runs you intend to open again. Without the flag, the trial row is identical to what it would otherwise have been, and show reports the trade table as not recorded — never as zero trades.

Over MCP the same opt-in is record_trades=true on run_backtest and run_walk_forward; analyze_trial and compare_trials then read the table.

Asking for the table is not answered by a memo that has none: if an identical prior trial exists without a stored table, the run is repeated and a new trial recorded. That trial counts toward the multiple-testing total like any other: a backtest adds one, and a walk-forward adds every configuration its inner search evaluated, so asking a large validation for its table after the fact raises the deflation bar for its whole family. Pass --record-trades on the first run you intend to open again. An identical prior trial that did store one is still served.

Persistence follows the same journal-first pattern as the return series and the proposed book: the journal is the source of truth, and trials rebuild reconstructs the table from it alone.

A stored table says whether it is all of them

Very large tables are capped at a storage ceiling, and the cap travels with the rows. A trade table has four states and they never render alike:

What trials show prints	What it means
— (not recorded — re-run with --record-trades, or record_trades=true over MCP)	The run did not opt in. Nothing is known about its trades
0 trades	It opted in and made none
1,204 trades	All of them
5,000 of 18,432 trades — TRUNCATED at the storage cap…	A prefix. Any total over these rows is short by the rest

A fifth reads as whether that was all of them was not recorded: a table stored before the count was kept. It did not prove completeness, so it is never rendered as though it had.

The distinction is not cosmetic — everything that aggregates a stored table (exit-reason P&L, win rates, duration) sums exactly these rows, and a capped table that looks complete turns a partial sum into a confident wrong number. --trades-limit truncates the view and reports itself separately (display_truncated in the JSON); that one is undone by raising the flag, and the storage cap is not undone by anything.

Maintenance: quarantine a subset, or retire an era

Two commands, deliberately not one. They answer different questions and differ in how reversible they are, and collapsing them is a mistake in both directions — quarantining an era leaves thousands of rows carrying a caveat nobody can act on, archiving a subset throws away the evidence around it.

trials mark-contaminated — a suspect subset

For when something is learned about some trials after they were recorded: a scanner that used a stale universe, a vendor's data correction, a config that turned out not to be the one you thought.

quantred trials mark-contaminated --reason "vendor split adjustment was wrong"     --strategy demo_trend --before 2026-08-01 --dry-run
quantred trials mark-contaminated --reason "vendor split adjustment was wrong" --id abc123def456

Select by --id or by filters, not both: ids name trials outright, and combining them is refused. An --id that matches no trial is refused (exit 1, naming the id, nothing written even for the other ids beside it); a filter that matches nothing is an ordinary empty answer.

--reason is required. Rows excluded from every leaderboard with nothing saying why cannot be judged later, which is worse than not excluding them.

Nothing is rewritten. The quarantine is an appended event naming the affected trials; the trial records stay exactly as written. trials rebuild replays the event and reaches the same rows, so the quarantine survives the store being thrown away — which it is, routinely, at every schema bump.

What it changes, and what it deliberately does not:

	Quarantined trial
Served as a memo for an identical run	No — a fresh run is always safe, a suspect number never is
Ranked in trials best	No, and the excluded count is reported
Counted toward the family's multiple-testing total	Yes

That last row is the one that could most easily have gone the flattering way. The search still happened — you did look at that configuration — so dropping it would lower the deflated-Sharpe bar for the family, and this store never moves that bar down on its own.

trials archive — a whole era

For when everything recorded becomes incommensurable rather than suspect. An accounting bump changes what the engine computes, so every stored metric was measured with a different instrument; that is not a subset and no annotation fixes it.

quantred trials archive --reason "accounting v6 invalidated v5 walk-forwards" --label pre-v6 --dry-run
quantred trials archive --reason "accounting v6 invalidated v5 walk-forwards" --label pre-v6
quantred trials archives     # what has been retired, and why

Both files move together, always. The store is an index over the journal, so moving one alone leaves the other describing something that is not there — or an era's rows sitting beside a fresh journal, still reporting a multiple-testing count for evidence that is gone, with nothing erroring because both files are individually valid. That is the hand-rolled state this replaces.

Each archive keeps a manifest (what, when, why, which accounting version and commit), and the new journal opens with a record of the archive, so its emptiness reads as "an era was retired here" rather than "nothing has ever been run".

There is no reset

Deliberately. The destructive version of archive is what the append-only rule forbids: the journal has nothing behind it to rebuild from, so a record removed is a record gone. Archive moves the files; they are still on disk under a name that says when and why.

Both are CLI-only

Neither is an MCP tool, and that is a decision rather than an omission. Quarantining evidence and retiring an era are operator decisions about a campaign's record, not run configuration: one changes what every later leaderboard and memo reports, the other moves your files. An agent that believes a trial is contaminated should say so and let you act.

Index health, and how it repairs itself

The database is derived. The journal is the record; the index is a cache over it, and deleting the file loses nothing that trials rebuild cannot put back.

That is what makes the repair safe. Every time the store is opened it compares the tables actually in the file against the schema this build declares, and rebuilds from the journal if they differ — so a store written by an older version gets the columns a newer one needs, rather than reporting the new version over the old shape. A version stamp alone could not catch that: the stamp is written by the rebuild, and the old rebuild emptied the tables instead of recreating them.

One case it will not repair, on purpose:

$ quantred trials rebuild
418 recorded trial(s) are indexed here but the journal they came from
(~/.tradeflow/logs/research_journal.jsonl) cannot be read. Rebuilding would replace
them with an empty index and the journal is the only other copy. Restore the journal,
or point --journal at the one this store indexes.

Replaying an absent journal produces an empty index, and an empty index reads as a campaign that tried nothing — which would quietly lower the deflation bar every one of those trials paid for. trials status reports any schema mismatch it could not fix, and reports a quarantine count it cannot read as unknown rather than as zero.

Not covered here

Retention and pruning are deliberately out of scope — append-only history is the point. trials status and trials rebuild handle index health, and the maintenance commands above never delete; see the walk-forward page for what the store's campaign count feeds.

From a source checkout (contributors)

The commands above are the installed quantred command. In a checkout, run them as uv run python main.py <command> .... Extras are installed with uv sync --inexact --extra <name> (see optional extras).