Fix live option guards, grounding operand evidence, source-bound benchmarks, trade artifacts, attribution, drawdowns, valuation tolerances, and SEC point-in-time factor inputs.
Add Brazil research data and market support, 13 fundamental and valuation factors, channel progress forwarding, and opt-in ZimaOS templates. Update seven README News sections with three visible dates.
Validation: 17,635 backend tests passed; 1,084 final targeted tests and 103 frontend tests passed. Frontend build, package build, documentation checks, and CI gates passed.
Four ways a figure could be grounded by something that did not measure it:
- The active run's metrics.csv was read as the current backtest's output, so
the previous run's Sharpe could be credited to this one. That copy counts
only once the loop's archive manifest names the directory the call declared
-- which also covers a detached backtest whose own run dir sits outside the
active run dir, where no own_dir resolves at all.
- metrics.csv/metrics.json were ingested even when the model wrote them
itself: _is_model_written covered the summary files and the table glob but
not the metrics files, so a model-authored metrics.csv became engine
evidence -- a gap in the path d2dd7006 introduced three days earlier.
- A date cell carrying an explicit year matched on month and day alone, so a
2026 bar could ground a claim dated 2025.
- A repeat resolution that failed, was refused or came back empty replaced a
locked identity: no evidence about the entity, yet it erased the lock and the
contradiction anchor with it. A locked or conflicting record now survives any
non-conclusive step, and a repeat resolution agreeing with one side of a
contradiction no longer promotes it to locked -- re-answering the same query
through the same resolver is not evidence that settles it. A successful
resolution that disagrees still surfaces as conflicting, which is what
identity_status documents.
Corrections after review of this branch's first push: the item-B pin wrote an
escaped newline, so its CSV was a single line and _record_metrics_file returned
early -- it passed against main as well and pinned nothing; a conflicting
record was still overwritten by a repeat resolution; and the manifest gate was
skipped for a detached backtest, because own_dir stays None when the declared
run dir resolves outside the active run, so that call was credited with the
active run's copy whatever the manifest said.
Tests: five pins -- a model-written metrics.csv, a detached call that cannot
inherit the active run's copy, a vouched-for archive that still counts, a
repeat resolution that cannot clear a conflict, and a timeout that cannot clear
one either -- beside the explicit-claim-year and lock-survival cases. The
item-B pin fails on main and passes here; the three new ones fail on this
branch before the corrections above. Grounding suites 242 passed.
Signed-off-by: cgycorey <cgycorey@users.noreply.github.com>
Signed-off-by: t <t@t>
#1576 kept Yahoo's declared quote currency on every frame. Probing what it
declares for .BA showed the case the ar_equity market (#1543) had missed. BYMA
lists an issuer in pesos and again in dollars. Yahoo declares GGAL.BA / YPFD.BA
/ GOOGL.BA in ARS, and SPYD.BA / GGALD.BA / AAPLD.BA in USD (13.37 / 4.20 /
17.53 on 2026-09-24). The engine books every .BA line in one static ARS pool,
so a dollar line was priced as pesos. The trailing D is not a rule: YPFD.BA is
a peso line.
This is the LSE contract (#1206) on a second venue. normalize_declared_quote_
currency admits a .BA frame only when it is declared ARS and still sends .L
through the GBp/GBP normalizer. Every other declared currency is recorded as
#1576 intended. The Yahoo loader calls it for every frame. The yfinance loader
reads the declared currency for .L and .BA only (a metadata request each) and
passes it through the same function. The ar_equity chain is yahoo -> yfinance
-> local, so a dollar line can no longer reach the pool through a fallback.
Grounding, same commit:
- evidence.py: a market-data record takes the currency the provenance
declares and falls back to the suffix. The answer must name the record's
currency, so the suffix guess was the one enforced before.
- policies.py: GBP / VND / ARS aliases (£, ₫, AR$, 英镑, 越南盾, 阿根廷比索).
The previous commit gave .L / .VN / .BA a currency. Without the aliases an
answer writing £1.17 took a correction round for not naming GBP.
Mutants each fail: no ARS check (4 tests), yfinance not probing .BA (3), the
record ignoring provenance (1), no aliases (4).
Each item was reproduced on the branch before this commit before fixing; two reviewer residuals
did not reproduce (a plain 56.7 against a 0.567 low is already refused; a
bare-ticker call already grounds its canonical symbol) and are not changed.
- A table cell declared with a call's ref is compared with that call's values
for its own column: a close cell holding the high passed with `ref c1`.
- `var` and `es` count as tail-risk evidence only as the whole leaf name;
`var_explained` and `sales_es` were read as a VaR.
- A buffered run that hit the tool-call-syntax fallback streamed its message
twice (pre-existing).
- `thinking_done` no longer carries the figures block in an unbuffered run, and
a held-back last line in a tool-call turn is shown once the turn ends.
- The `_PrivateCompanySkillTool` test stub (named load_skill) takes a required
constructor argument, so registry discovery cannot instantiate it as the
real tool for later tests.
Three read-only reviews of the branch up to its frontend commit (adversary, correctness, spec) found an
unobserved figure reaching released text through the declaration channel and
through shape gaps; each HIGH was reproduced on the live tree before fixing.
On the adversary's sparse-ledger probe set, leaks went from 38 of 53 at
that point to 6 before the last two rules below.
Shape layer (figures.py, release.py):
- numerals are read the way a renderer shows them: zero-width characters,
full-width and Arabic decimal points, NBSP before a percent, U+2212 minus,
backslash escapes, HTML entities, and an ISO currency code glued to digits;
- decimal comma ("0,666 CNY", "12,5 %") is a decimal, while a valid grouping
("-1,250.00") stays grouped (#1418);
- fences follow CommonMark (indent, info string, closer length), and an
unterminated non-figures fence is prose, so it fails closed;
- a year-shaped number with a currency mark or under a price column is
measured; compact dates, dotted dates and times are structure; date / code
table columns exempt only structure-shaped numbers;
- the block parser accepts header and separator rows, keeps an empty note
field empty, and reads values with units, currency, magnitude, pp and bp;
every figures block is merged and stripped;
- 円 and 원 are currency marks;
- the redaction sweep also removes fenced and escaped restatements of a cut
figure, and a malformed block is stripped and re-validated instead of
forcing the canned fallback.
Role layer (policies.py, evidence.py):
- count covers counts, weights, thresholds and probabilities; a currency-marked
figure declared count is checked as observed;
- cited must name its source on the figure's line (CJK character pairs or
English words of four letters or more), because the block is stripped;
- derived: added or subtracted operands must be observed, the result matches
the prose's written precision and sign, and a currency-marked result derives
only from money-denominated values;
- no non-observed role in a price column; proposed is a price of its own
symbol; a declaration cannot move a figure to an instrument the sentence
does not name;
- ref may name a tool; referenced values are filtered to the figure's symbol
and, by shape, to price or non-price kinds;
- a currency-marked figure is not grounded by a volume or a metric;
- a measurement left out of a present block is checked as observed first;
- metadata counts (return_observations, vol_lookback, aligned_days, ...) are
not metrics, and VaR / ES leaves are tail-risk evidence (#1420, #1426);
- "nearest observed" comes from the figure's own symbol (#1433).
Integration: precision and sign use the normalized reading; correction
templates for the new reasons; a general answer (no instrument asked about)
with no tool evidence is released cut with a range-less footnote, while a
market answer with no observed price still falls back.
Grounding tests: 714 passed. Mutation: 53/53 (shape), 38/38 (roles).
A number's role — observed print, derived level, proposal, citation, count —
was inferred from the prose around it, against 22 masks and a dozen phrase
lists: price words, target/stop/trigger words, ATH and 52-week spellings, FX
bases, signal values, line references, ratios, indicator windows, observation
binders, formula markers, attribution verbs, forecast frames, definition
frames, strategy subjects. A catalogue is only ever as complete as the day it
was typed, and it was not complete for one day: the same sentence got opposite
verdicts in its two translations, a level word the list had missed cost a
revision round, and every phrasing it did not know was either a rejected
correct answer or a released fabrication.
The model declares each figure now, and the gate verifies the declaration
against evidence. What is left to infer is SHAPE, which is language-independent.
```figures
0.666 | observed | 159516.SZ close 2026-09-09 | c1
0.646 | derived | 0.666 × 0.97 | c1
0.95 | proposed | entry
1.8 | cited | Fama-French 2024, table 3
3 | count | 月
```
- figures.py: block parse (lenient — full-width pipes, run-on spacing, missing
ref; an unreadable line is a `figures_block_malformed` issue, never a silent
skip), strip, and the three shapes. Structural exemptions (date, year, symbol
digits, line-leading ordinal, fenced block, a table's own date/symbol column)
need no declaration; measurement-shaped figures (decimal, percent, touching
currency mark, table cell) must have one; bare integers are not checked.
- policies.py: one check per role, replacing _validate_price_claims /
_validate_price_tables / _validate_analysis_claims and every helper under
them. Symbol attribution is declaration ref, then the table's symbol column,
then the one canonical symbol in the figure's punctuation-delimited segment,
then the document's.
- evidence.py: price-denominated indicator leaves are REGISTERED per tool
(technical_indicators: latest_close, indicators.sma_*, ema_*, bollinger.
{upper,middle,lower}) instead of guessed from path tokens, which had been
admitting ma_diff, sma_cross and supertrend_direction as prices.
- ledger.py: validate → figures → policies; ValidationResult carries
released_text (the draft minus the block) and the artifact keeps the raw block.
- release.py: cuts each flagged figure at its own span. The document-wide sweep
is gone with the unscanned surfaces it patched — every occurrence is now
located and checked where it stands, so an unflagged one is one the evidence
supports.
Measured: package 5508 → 4585 lines, 65 → 12 compiled regexes, all twelve
shape or symbol patterns except two that decide identity scope rather than a
number's role (_PRIVATE_ASSERTION_RE, _ACTIONABLE_MARKET_RE).
Tests: K=184 A=35 D=73, plus 41 replacements and 33 new contract tests.
Full suite 12859 passed, 138 skipped. 30 mutations, 29 killed; the survivor is
the fail-closed postcondition on redacted_release's bounded loop, which
per-figure spans leave with no reachable input.
RELAXATIONS — each detection this moves, and where it moves to:
- A fabricated INTEGER price with no currency mark, outside a table, is no
longer checked ("close was 2500"). Moves to shape: the same figure written
the way an answer actually quotes a price — 2500 CNY, $2500, or a table cell
— is measurement-shaped and still checked.
- A categorical window claim carrying only a bare integer ("所有历史 12 个月
窗口均实现正收益") is no longer refused. Same trade; _CATEGORICAL_WINDOW_RE
named the claim by its words.
- An analysis metric equal to an observed price is no longer separated from it
when written without a percent sign ("策略夏普比率 1.171"). Moves to the
declaration's ref, which scopes the check to one call. A PERCENT figure is
structurally immune: the price pool is not consulted for one at all.
- A strategy-subject metric can be grounded by endpoint arithmetic if the model
writes that arithmetic in the note. _STRATEGY_SUBJECT_RE decided this by the
word beside the figure; the note is now an explicit, persisted claim.
- A derived percentage is compared by magnitude, so its sign is not checked.
The direction moved into the note: (low−high)/high and (high−low)/low are
different notes and ground different figures.
- In prose, a registered indicator level can answer a figure declared observed
without a ref. The scoping survives on the two surfaces that carry it
structurally — a table's OHLC column, and a declared ref.
- `count` and `cited` are unverified by contract, so a model that misdeclares a
fabricated figure as one is not caught. Strictly narrower than what it
replaces: 预计 / 目标位 / ATH bought the same exemption for free, with no
declaration at all. It now costs an auditable line in the artifact.
TIGHTENINGS in the same change: every table cell is checked (the unchecked
price-column hole, and _has_unchecked_price_column with it, are gone); an RSI
or other unregistered indicator reading is no longer exempt by name; a
provenance demand now follows any figure checked against price evidence rather
than only one a price word introduced.
src/agent/grounding.py (5183 lines) becomes src/agent/grounding/ along the
data flow the v2 spec names: identity locks what a number is about, evidence
records what the run observed, figures holds the shape primitives that find
numbers in a draft, policies decides whether each one is grounded, release
turns a rejection into a correction or a discounted release, and ledger is the
facade the loop drives. GroundingLedger keeps its name, its public signatures
and its behaviour — it is now composed from four mixins, one per module.
This stage is the skeleton only. No detector, no regex and no word list was
touched; the deletions the spec calls for land in the stages that follow.
Proof that nothing moved but the text:
* all 80 GroundingLedger method bodies are byte-identical to the previous commit after
dedent, except five that call a sibling staticmethod through the class name.
``GroundingLedger._pipe_tables`` etc. cannot resolve from policies/release
without importing ledger, so those five call sites now name the mixin that
owns the method (_check_table_cell, _direct_price_values, _pipe_tables,
_numbers_without_dates_or_percent, _has_unchecked_price_column). The
resolved function object is the same one.
* all 131 module-level definitions are byte-identical and none were dropped or
added: 65 compiled regexes before, 65 after.
* the import graph is a DAG in that order, so no module imports downward and
there is no cycle.
Package total is 5586 lines against 5183 — the difference is six module
docstrings and six import headers. The budget the spec sets (<= 2200 lines,
<= 12 shape regexes) is a target for the deletion stages, not for this one.
Test imports follow the code. Two monkeypatch targets had to move to the
module whose global is actually read, because a ``from x import y`` binding is
not patched by patching x: ``_normalize_symbol`` is read by identity, and the
recovery caps are read by release.recovery_action.
Full suite: 12903 passed, 138 skipped — the same counts as the previous commit, measured
on a worktree of that commit with the same interpreter.
A ten-minute run for 159516.SZ ended in the three-sentence safe_fallback
refusal after the gate rejected four drafts. Three things were wrong: the
fallback discarded the whole draft, the gate rejected derivations its own
correction prompt asks for, and a missing provenance word cost a full model
round. This makes the rejected draft releasable with only its unverified
figures cut out, and closes the escapes an adversarial audit of that change
found.
Which evidence backs which claim
- _is_price_denominated_indicator decides on the LEAF. Under the old
any-token-anywhere rule indicators.ma_diff, .sma_cross, .ema_gap,
.support_strength and .supertrend_direction all classified as prices, and
any of them set to 1.30 let the answer print "现价 1.30 元". Pivot levels
r1/r2/r3/s1/s2/s3 were unreachable dead entries (the digit strip turned r1
into r) and now match; window/period/length/span/cross join the denylist.
- An indicator answers a price LEVEL claim and nothing else. The rule was
first written the other way round — admissible unless the clause names an
OHLC field — against a hand-written list of field phrases, and a
hand-written denylist is only as complete as the day it was typed. With
sma_20 at 1.150 and the observed close at 1.171, "收盘价为 1.150 元" was
caught while "closed at 1.150" / "the close was 1.150" / "highest price was
1.150" were released, and all ten spellings of a spot quote (现价 / 最新价 /
成交价 / 报价 / 股价 / current price / the quote is) leaked in both scripts,
because a spot quote names the latest print and names no OHLC field. The
allowlist is 支撑 / 阻力 / 均线 / band / pivot and their kin; everything else
is compared against OHLC evidence at the unchanged 0.5% band. A level word
the allowlist misses costs a correction round — a denylist gap released a
fabricated print.
- An indicator is symbol-bound. With evidence for two instruments and a
clause naming neither, 562500's sma_20 grounded a 600519 price claim.
- A metric figure is not grounded by happening to equal an observed price. A
run holds hundreds of observed values spanning the instrument's range, so
any percent-free metric written in that range collides with one:
"策略夏普比率 1.171" and "| 夏普比率 | 1.171 |" validated against an observed
close with no backtest in the session. The figure must also be WRITTEN as a
price, and a metric cell is exempt only when it restates a PAIR of observed
prints ("| 峰值→当前 | 1.180 → 1.110 |"), which is the shape the exemption
was added for.
Derivations and endpoint arithmetic
- Only a formula's own operands and result are exempt, at the position where
they sit. Value-scoped, "建议买入价 0.95 元（0.95 × 1.171 = 1.11245 参考）"
released an entry price 19% below every observed bar: arithmetic that
derives nothing, whose free operand inherited the exemption its own copy
inside the bracket earned. The occurrence inside the formula is exempt; the
one beside it is compared like any other claim.
- A formula's RESULT may not be a market print. "2026-06-23 收盘价 = 1.171 ×
0.80 = 0.937" claims a close the ledger holds as 1.137. When a print word
labels the equation itself, the operands stay exempt and the result is
compared; "基于收盘价 1.171 × 0.80 = 0.937 作为买入价" — the word on an
operand, the result proposed as an entry — still passes.
- The result separators are one shared list for both languages: "≈" passed
while "approximately" burned a revision round.
- _ARITHMETIC_EQUATION_RE is deleted. _evaluate_formula's two-operand rule
already subsumes it (mutation-verified: forcing the precondition false
changes no verdict), and it removes one entry from the regex catalogue.
- An English period ends a clause, exactly as 。 does. Two English sentences
sharing a line were one clause, so the header's "last close" labelled the
level claim after it and a derivation in the first sentence exempted a
fabricated price in the second — while the Chinese translation of the same
text got the opposite verdict.
- A drawdown may only be grounded by the falling direction. Measured on a
1.053/0.666 ledger (true fall −36.75%, inverse rise +58.11%):
"最大回撤约 58.1%" and "58%" both validated.
- The written-precision half unit is applied in the units the figure was
written in. A percent figure is compared only against the percentage-point
derivation; running it against the fraction gave an integer percent a
0.5 band in FRACTION units — 50 percentage points — so "区间收益率约 0%"
validated against a real +58% move.
- A STRATEGY-subject metric gets neither exemption. "回测显示该策略最大回撤
5.9%（从 1.180 跌至 1.110）" left a session with no backtest in it at all, on
the exemption written for a price drawdown against an observed high; the
subject word (策略 / 回测 / 组合 / 净值 / strategy / backtest / portfolio /
NAV) is what decides, and "股价较 5 月高点 1.053 元已回撤约 37%" keeps it.
- The endpoint exemption is applied figure by figure and the derivation
exemption is measured in the clause the claim was measured in. One correct
ratio used to carry every percentage in its clause out with it ("…down about
37% from the 1.053 May high with a max drawdown of 60%"), and any true
arithmetic anywhere on the line grounded a metric in the next clause.
Release path
- Cuts are located by the clause's character SPAN, recorded on the issue,
not by substring-searching the normalised claim text. Measured before:
a price written "1,450.50" was unlocatable and the release silently fell
back to the refusal; a metric table ("| 最大回撤 | 12% |") likewise, because
the claim is the synthetic "label: value"; and a bare cell claim "1.10"
matched inside "21.10 亿元", rewriting an untouched turnover figure into
"2（略※）". All three release correctly now, and the substring fallback is
deleted rather than left carrying a rationale the span had already settled.
- After cutting the flagged spans, the same figure is swept out of the rest
of the document. Before: "建议买入价 1.30 元" was cut while "| 第一档 | 1.30 |"
and "- 关键位：1.30 / 1.45" survived under a footnote asserting the figure
had been removed. Two kinds of occurrence are never swept — one the ledger
observed, and one sitting inside a valid formula. The formula protection is
scoped to its own character SPANS: scoped to the value, the rejected entry
price 0.95 survived in "| 第一档 | 0.95 |" as soon as the model also wrote
"基于收盘价 1.171 × 0.95 = 1.112" beside it, protected by the derivation. A
percent literal this same gate accepted in the same run is protected too —
a validated "区间收益率约 5.5%" was cut and footnoted as unmatched because an
unrelated "策略年化波动率 5.5%" was flagged in the next clause.
- A figure the run observed is no longer reported as an invented analysis
metric, in prose or in a table cell. "最新收盘 1.171 元且策略最大回撤 12%"
reported 1.171 (the first unmatched number), so the release cut the
correctly quoted close and footnoted it as unverifiable.
- The released document is validated WITH its footnote; the footnote prints
the symbol the way the answer spells it (562500.SS, not the canonical
562500.SH) and a large price in full (".10g", not "%g"'s 1.23457e+06).
- The redaction footnote is the only redaction statement in the document. A
draft writing its own "※ 略去 0 处……" shipped two contradictory footnotes;
both it and any marker pasted into the body are stripped before the cuts.
- The marker follows the script of the LINE it cuts, so a Chinese report
containing an English table no longer releases "| First |（略※） |", and the
column padding after a table pipe survives.
- 元 is only swallowed when it is a unit: "0.95 元宵节后关注" no longer
becomes "（略※）宵节后关注", and 元/股 is one compound unit — cutting half of
it left "建议买入价（略※）/股".
- The release path's own rechecks run through _validate(record=False).
One successful redacted_release raised validation_count 1 → 2 with no model
round, shrinking the revision budget and overstating the rejected-draft
count in the run reason and the artifact; the safe_fallback branch now
captures its count where the redacted branch does. The zero-round
provenance repair takes the same non-recording path, and the document that
actually shipped is recorded once under "released" so the artifact carries
evidence for the fail-closed promise instead of nothing at all.
- repair_provenance fills in the source and the currency only.
canonical_symbol_not_surfaced was repaired too: a draft reading
"贵州茅台（雅虎，CNY）最新收盘价 1.171 元" was released in zero model rounds
under a note naming 562500.SH. Neither is a draft carrying a price COLUMN no
validator reads ("| 档位 | 挂单价 |"): the note says where this run's prices
came from, and attaching it in zero rounds spends the round that was the
model's last chance to drop that figure.
Verification
- tests/test_grounding_release.py: 28 → 83 cases (144 with parametrisation),
every behaviour two-sided.
- Full suite green: 12903 passed, 138 skipped, 0 failed.
- 27 mutants, 26 killed, including one per finding above. The survivor is
equivalent by construction: with the analysis derivation exemption now
measured on the clause's own values, its `_DERIVATION_RE` keyword pre-filter
can be widened back to the line without changing any verdict.
- Real deepseek-v4-pro E2E on 159516.SZ (run 20260909_223146_43_233953,
10 calls / 551k tokens): status success, NOT degraded, one correction round,
and the released answer names an entry price (0.650, the observed
2026-09-04 intraday low) — the question the screenshot run refused.
Resolved against main and reworked where main had moved past the branch:
* the candidate filter no longer keys on "candidate is a canonical crypto
pair". Main's later hardening of _canonical_crypto_pair (the USD base
whitelist) makes that predicate None for exactly the near-string hits the
PR set out to drop, so the rule was a no-op and its three headline tests
failed on merge. A metal/FX pair query is now treated the same way the
crypto branch treats a pair query — an exact instrument assertion — and
keeps only candidates naming the same two legs, across every spelling
(XAUUSD / XAU-USD / XAU/USD / XAUUSD=X).
* the venue/instrument tables are named constants instead of two inline
literals, with a test pinning them against the resolver's own tables.
* a crypto-quoted pair (ETH-BTC, SOL-ETH) stays crypto: the branch as
written classified everything but a stablecoin quote as forex.
Verified on main before the change: search_symbol("XAUUSD") returned exactly
one candidate, VALOUR-BTC-0-SEK.ST (a Swedish Bitcoin ETP), and XAU/USD
returned AETHUSDT-USD. Both now return no candidate, leaving the identity
unresolved rather than locked to the wrong instrument.
#1384 locked HK.06693 onto 06693.HK, but the connector writes US.AAPL,
SH.600519, SZ.000001 and BJ.430047 the same way — those still scanned as
no symbol at all, so a user who pasted one got identity_required from
every market tool, which is the bug #1384 set out to fix.
The US branch is case-sensitive: with the file's IGNORECASE flag a
case-folded match turns a "…/us.reuters/…" host inside a cited source URL
into the symbol REUTERS.US.
Replace the per-phrasing denylist with one structural rule. A number after a
price-context word is an observed claim only when nothing formula-like binds
it instead:
- a closed formula-marker set (comparison/division operators, or an indicator
identifier followed by digits: SMA/EMA/.../VWAP + digits, plus MA for the
Chinese MA20 convention; re.ASCII so a CJK-adjacent indicator like 上穿MA20
still matches) sitting between the price word and the number marks the
number an operand;
- an observation binder (was/at/报收/收于/... or =/==/≈) after the last marker
re-attaches the number to the price word, so 'close above SMA50 and was
2500' stays a claim while 'close/SMA50 > 1' claims nothing;
- an ASCII sentence boundary ('. ' / '! ' / '? ' followed by a capital) ends
the price word's reach, so a bare year two sentences later is not claimed
(the capital lookahead keeps 'close approx. 2500' gated).
The binder guard is what keeps it a strict narrowing: the canonical
'close was 2500' (no marker) still rejects with no matching evidence, and
'close above SMA50 and was 2500' / 'close above SMA50 at 2450' stay gated.
Numbers with no preceding price word keep main behavior.
The analysis gate only examined pipe-table rows when the table HEADER
resolved to a metric kind. A `| 指标 | 数值 |` / `| Metric | Value |`
generic-header table with the metric kind in the row escaped the whole
#1336 gate — a one-line formatting dodge of the exact bug it exists to
block.
Add a generic-header fallback built on (label, value) adjacency, with
PROSE PARITY as the bar: a table cell must never get a different
verdict than the same claim written as prose. A cell naming a metric
kind is a row label; it claims exactly one adjacent value cell (right
first for label-first layouts, then left for value-first ones), and its
own text is validated too so a label cannot smuggle its measurement
("| 年化收益率 18.2% | - |" is prose-identical to "年化收益率为
18.2%"). Cells claimed by no label (annotation columns, "备注 |
较去年提升 2%") stay unvalidated, exactly as prose never attributes
them to the metric word.
No comparison/criterion exemption, deliberately: prose has none —
"夏普比率 > 1.0 较好" is rejected without evidence exactly like
"夏普比率为 1.0", and passes once the figure is kind-backed. Exempting
"> 1.0" in tables would reintroduce the formatting dodge this PR
closes. The only cell exemptions are the frames prose already applies:
forecast ("预计 12.4%") and contiguous definitional ("通常认为大于
1.0 较好").
年化/annualized alone resolve to the return kind, closing the fragment
dodge ("| 年化 | 18.2% |") that the prose detector already treats as
a metric word. Price tables (日期|收盘) are still the price gate's job.
Regression tests: the dodge (ZH + EN, value-first, bare fragment,
multi-pair rows, metric-headed comparison cells, label-smuggled
measurements) rejected; value-backed, forecast, contiguous-definitional
rows and backed-claim-plus-annotation-column accepted — each mirroring
the prose verdict.
Two blockers from the #1338 review.
1. 95e2c2f8 applied _ATTRIBUTION_RE inside _validate_price_claims, so any
fabricated price passed once it carried a citation subject. With zero tool
evidence, main rejects and that branch accepted all five of
"Analysts say TSLA.US last traded at 412.35 USD.",
"分析师指出特斯拉现价 412.35 美元。",
"The paper reports that AAPL.US closed at 189.20.",
"据研究机构报告，AAPL.US 收盘价为 189.20。",
"The filing reports the stock closed at 412.35.".
The exemption is right in the analysis gate — a paper's Sharpe is a figure
this run could never observe — and wrong in the price gate, where the
quantity IS what the run observes. Removed there, kept there.
2. The derived-return exemption was gated on _GROWTH_FRAME_RE, a growth-phrase
list (从…到 / from…to). Any wording the list missed was rejected, so
"第一日收盘 100.0 美元，第二日收盘 112.4 美元，收益率 12.4%" failed while
the from/to spelling of the same arithmetic passed — the per-language drift
test_grounding_language_parity exists to stop. Replaced with the structural
fact the review asked for: the operands must themselves appear in the clause
and be observed values. Strictly narrower than the phrase list — a bare
"cumulative return of 12.4%" carries no operands — and language-independent.
_GROWTH_FRAME_RE had no other caller and is deleted.
Both pinned, both mutations killed. 370 grounding/discipline/parity tests pass.
Adversarial re-check found `研究` (research) alone was an external
subject in _ATTRIBUTION_RE, so "研究报告显示…" / "回测研究显示…" — the
model dressing up its own analysis — was exempted and re-opened #1336.
Removed bare `研究`; the unambiguous external subjects remain
(论文/文献/研究机构/学者 + English paper/study/analyst/filings).
`该研究显示` ("this study shows") now falls back to the measurement
gate, which is the correct conservative position: `研究` is exactly how
a model labels its own output. Verified no regressions: 276 tests pass;
33-case adversarial probe (self-attribution, laundering, kind-scoping,
metric-header tables) all hold.
Generic-header tables ("| Metric | Value |" with kind only in the row)
are a pre-existing main limitation, not introduced here.
Addresses the standing item of review on #1338: "The paper reports a
Sharpe ratio of 1.8" was rejected with zero tool results. Attributing a
figure to a named external source is the opposite of an unsourced claim
— the model is citing, not inventing.
- New `_ATTRIBUTION_RE` keys on who the source is, not on citation
shape: only subjects that cannot be this run's own output count (the
paper/study/researchers/analysts/surveys/regulators/literature/
authors/filings/rating agencies + reporting verb; Chinese 论文/文献/
研究/分析师/研究机构/学者 + 报告/显示/表明/指出/称/估计/发现).
"The backtest data shows an annualized return of 25%" — the exact
#1336 attack — does NOT count: data/backtest/strategy/report subjects
are how a model dresses up its own numbers.
- Scoped to the clause. A line-level check would let a citation in one
clause ("The paper reports a Sharpe of 1.8,") launder an invented
sibling metric in the next (", and our strategy achieved 47.3%").
- Both gates that can fire on such a sentence check it:
`_validate_analysis_claims` and `_validate_unsourced_symbols`.
- Tests: accept column covers English + Chinese restatements; reject
column pins the review's pair (TSLA.US traded / 特斯拉现价), the
non-attributed "the strategy reports" metric, the #1336-shaped
data/backtest subjects, and cross-clause laundering.
Addresses review of #1338: three answer shapes that main accepts were
rejected because the analysis gate treated an anchored derivation or an
attributed paper figure as an invented metric.
- A return figure stated as growth between two observed endpoints
("从 2026-08-03 的 100.0 涨到 2026-09-02 的 112.4，区间收益率为
12.4%" / "rose from 100.0 to 112.4, a cumulative return of 12.4%")
is arithmetic on sourced inputs. The gate now reuses the price gate's
_is_explicit_derivation for formula-shaped lines and adds a
from/to-frame check whose claimed value must match an observed
endpoint pair exactly (±0.5%). An unanchored return claim, or one
whose arithmetic no observed pair supports, stays rejected.
- Compound evidence leaves resolve their metric kind by token split
("reported_annualized_return" -> return, "strategy_max_drawdown" ->
drawdown, "benchmark_return_vol" -> vol), scanned from the right so
the head noun wins. A successful research_papers result therefore
grounds attributed paper performance instead of blocking it.
- Tests: derivation accept-cases (Chinese + English), unanchored and
wrong-arithmetic reject-cases, paper attribution, and unit coverage
for the token-split resolution.
Addresses review of #1338: a successful analysis result no longer
blanket-authorizes every metric claim, and observed price evidence no
longer stands in for an invented volatility figure.
- Record an analysis completion only when the result actually produced
metric figures: backtest parses run-dir artifacts/metrics.csv|json
(containment-checked, plus the archived copy in the active run), and
factor_analysis/run_shadow_backtest/quantlib_call record numeric leaves
whose keys name a metric kind. Skipped/deduplicated calls and
quantlib_call list/describe never count.
- Match claims kind-scoped (vol/drawdown/sharpe/win_rate/probability/
return) against analysis evidence or any tool's observed numeric
output; percent/fraction scaling and drawdown magnitude compared.
- Reject categorical "all N-month windows were profitable" claims
without a window-producing analysis result.
- Do not gate definitional prose ("夏普比率大于 1.0 通常被认为较好").
- Parse metric tables per cell against header kinds, so a forecast
annotation in one cell cannot exempt a measured neighbor cell.
- Persist analysis_completed / analysis_evidence in grounding_evidence.json.
Adds seven regression tests; the successful-backtest fixture now
requires a real metrics.csv artifact.
Backtest-style metric claims (Return vol, MaxDD, probability of hitting
target) never matched the price-context regex, so after failed or
deduplicated market-data calls the agent could still state analysis
figures as measured facts. Record completed analysis tools (backtest,
factor_analysis, run_shadow_backtest, quantlib_call) and reject
measurement-shaped figures (decimal or percent with an analysis-context
marker) when none completed in the run. Definitional prose such as
'252 个交易日年化' stays untouched.
Cherry-picked onto current main rather than merged: main has since gained
canonical_fx_pair and index-symbol recognition, and a merge silently resolved
several of those hunks in this branch's favour — dropping the canonical_fx_pair
first-check, the index branch of _CANONICAL_SYMBOL_RE and _CLAUSE_SEPARATOR_RE.
The test suite caught it (104 failures), but a deletion-closure check against
main is what showed how much had gone. On the rebase there is one real
conflict, the regex alternation, resolved as a union.
Ordering matters and is now explicit: canonical_fx_pair runs before the
joined-pair rewrite, so GBP/USD and GBPUSD both land on GBPUSD=X instead of
being folded to the crypto spelling.
The defect: XPTUSD is XPT + USD (spot platinum), but it also ends in "TUSD",
one of the quote suffixes, and stripping that leaves the alpha base "XP" —
so platinum normalized to XP-TUSD, a crypto pair that does not exist. That is
the same class of misresolution the deliberate exclusion of bare "USD" exists
to prevent; the suffix list alone cannot catch it, because the collision is
inside the suffix rather than at its boundary. Guarded with an explicit
metal-pair check.
The USD exclusion had no test either — adding "USD" back passed every other
test in the file while folding spot gold to XAU-USD. Both mechanisms are now
pinned separately, plus a third test that genuine TrueUSD/BUSD/USDT pairs
still fold, so the guard cannot be widened into a blanket refusal.
178 tests pass.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Pin the offending symbol in the assertion (000858.SZ, not the sourced
600519.SH) and add a negative arm proving the gate stays silent for an
unsourced symbol with no nearby figure — so the figure-presence guard
in _validate_unsourced_symbols cannot be dropped unnoticed.
Known tradeoffs, noted on the PR:
- （000858.SZ）公司有 3 大业务板块：the count is compared as a figure by
the unsourced gate (ASCII parity, pre-existing class of behavior).
- Internal ，/；/、 inside full-width parens still split symbol from
figure: residual false negative, out of scope for #1260.
（）【】 were in the clause-separator class, so the idiomatic Chinese form
公司名（代码）价格 split the symbol from its figure and the
unsourced_symbol_figures gate never fired (false negative). Drop the
paired full-width brackets from the separator regex so they behave
like ASCII parentheses, which were already excluded for the formula
check.
Regression test parametrized over full/half-width parens, verified to
fail on the old regex. Note: 公司名（代码，价格） with an internal
separator still splits (pre-existing residual, see #1260).
Code review of the alignment change found integration gaps; all fixed:
- Grounding identity gate: search_symbol('GBP/USD') locked GBPUSD=X but
the query-as-asserted scan normalized GBP/USD to the crypto spelling
GBP-USD -> conflicting identity, which outranks every later lock and
blocks subsequent get_market_data (unrecoverable for the session).
FIAT_CODES/canonical_fx_pair now live in src.market_data (single
source of truth) and grounding._normalize_symbol applies it BEFORE the
slash->hyphen rewrite; ^ symbols with INDEX type no longer read as
listed_security, and =X pairs infer their quote currency.
- Runner/index: ^SPX routed to ChinaAEngine/CryptoEngine (index market
class had no engine wiring). _MARKET_TO_SOURCE['index'], explicit
GlobalEquityEngine branch in _create_market_engine, and a composite
rule engine; JPY=X (3-letter =X abbreviation) classifies as forex with
honest UNKNOWN:forex currency instead of CNY.
- Restored the GC=F/CL=F/EURUSD=X/JPY=X source-pin rows dropped by the
first edit; env_schema + docs updated for the index market and the
fx_normalizer source; integration tests added for every fix.
Two related bugs caused ``search_symbol("XAUUSD")`` to surface wrong
assets in the runtime identity registry, and caused ``XAU-USD``,
``XAU/USD``, and ``XAUUSD=X`` to surface as ``instrument_type =
crypto, venue = crypto_or_fx``.
## Problem
Runtime validation against the Vibe-Trading agent (Mimo) exposed two
distinct issues that PR #1280 did not cover:
1. ``SymbolSearchTool.execute`` had an inverted filter gate. The
PR #1234 filter (added in ``SymbolSearchTool.execute``) was:
if crypto_pair is not None:
candidates = [
c for c in candidates
if _canonical_crypto_pair(c["symbol"]) == crypto_pair
]
This only fired for queries the resolver recognized as a crypto
pair. A bare ``XAUUSD`` query produces ``crypto_pair = None``
(the upstream ``_canonical_crypto_pair`` correctly returns ``None``
after PR #1280's ``USD`` whitelist safeguard). The filter then does
NOT run, and a near-string Yahoo free-text hit such as
``VALOUR-BTC-0-SEK.ST`` (a Swedish Bitcoin ETP) survives to the
identity ledger as a candidate for a gold query. The runtime
observation: ``search_symbol("XAUUSD")`` returns 1 candidate
``VALOUR-BTC-0-SEK.ST`` that is not gold.
2. ``_infer_venue`` and ``_infer_instrument_type`` in
``src/agent/grounding.py`` had a shape-based fallback that
classified any symbol containing ``-`` or ``/`` as
``crypto_or_fx`` / ``crypto``. ``XAU-USD`` and ``XAU/USD`` were
both mis-classified. The runtime registry reported
``instrument_type = crypto, venue = crypto_or_fx`` for
``XAU-USD``, contradicting the engine classifier
(``_detect_market`` returns ``forex``).
## Root Cause
Both bugs are shape-vs-semantic mismatches. The fix has to know
which dashed / slashed / ``=F`` / ``=X`` symbols are crypto and which
are forex / futures.
## Fix
* ``src/tools/symbol_search_tool.py`` (19 lines): broaden the
PR #1234 filter to also fire when the query is a non-crypto
ticker shape (separator, ``=F``, ``=X``). For those queries,
drop any candidate whose canonical form IS a crypto pair. A
bare name query (``apple``, ``tesla``, ``Gold``) falls through
to the existing free-text behavior because it has no separator
or ``=F``/``=X`` form.
* ``src/agent/grounding.py`` (72 lines): mirror the metal/FX/
futures patterns in ``_infer_venue`` and
``_infer_instrument_type``. The dashed/slashed fallback now
consults the same 26-entry ``_CRYPTO_USD_BASES`` whitelist
and the 5-entry ``_STABLECOIN_QUOTES`` set that
``_canonical_crypto_pair`` already uses, so
``XAU-USD`` / ``EUR-USD`` / ``GBP-USD`` resolve as ``forex``
and ``BTC-USDT`` / ``XAUT-USDT`` keep resolving as ``crypto``.
This is the third copy of these patterns. PR #1280 has two
(``_market_hooks._MARKET_PATTERNS`` and ``correlation.infer_market``).
A future refactor can lift them into a shared module.
## Risk
* Additive only: the new ``elif`` branch in
``SymbolSearchTool.execute`` fires only when the existing
``if crypto_pair is not None:`` branch does not. Existing
behavior is preserved for crypto queries and free-text name
queries.
* The whitelist constants are duplicated a third time. Drift
risk is low (the upstream ``_canonical_crypto_pair`` is
ground truth) and the audit has already noted this as a
future refactor.
* No new data providers, no new asset class, no engine
changes, no new skill, no changes to ``ForexEngine`` /
``GlobalFuturesEngine`` / loader layer.
* Tokenized gold (``XAUT-USDT`` / ``PAXG-USDT``) keeps
resolving as ``crypto``; the whitelist explicitly includes
``XAUT`` and ``PAXG``.
* US-equity and free-text name behavior is unchanged. The
test ``test_free_text_name_query_unaffected`` pins this.
## Tests
* ``test_agent_grounding.py::test_runtime_registry_classifies_gold_fx_futures_consistently``:
parametrized over 10 symbols (XAUUSD, XAU-USD, XAU/USD, XAUUSD=X,
GC=F, XAUT-USDT, PAXG-USDT, BTC-USDT, GLD, AAPL.US) and asserts
``_infer_venue`` and ``_infer_instrument_type`` for each.
* ``test_symbol_search_tool.py::TestSpotGoldCandidateFilter``: 7
tests pinning the filter behavior end-to-end:
- ``XAUUSD`` drops near-string BTC ETP.
- ``XAU/USD`` drops near-string crypto hits.
- ``XAUUSD=X`` drops near-string crypto hits.
- ``GC=F`` keeps the correct futures hit.
- ``XAUT-USDT`` keeps tokenized gold.
- ``PAXG-USDT`` keeps tokenized gold.
- Free-text name query (``apple``) is unaffected.
## Behavior matrix
| Query | Old: count | New: count | Old: instrument_type | New: instrument_type |
| ------------ | ---------- | ---------- | ---------------------- | -------------------- |
| XAUUSD | 1 (wrong) | 0 | listed_security | forex |
| XAU-USD | 0 | 0 | crypto | forex |
| XAU/USD | 0 | 0 | crypto | forex |
| XAUUSD=X | 0 | 0 | listed_security | forex |
| GC=F | 1 | 1 | future | future |
| XAUT-USDT | 1 | 1 | crypto | crypto |
| PAXG-USDT | 1 | 1 | crypto | crypto |
| BTC-USDT | 1 | 1 | crypto | crypto |
| GLD | n/a | n/a | listed_security | listed_security |
| AAPL.US | n/a | n/a | listed_security | listed_security |
| apple (free) | n/a | n/a | (existing behavior) | (existing behavior) |
832 tests pass across 21 modules. No regressions.
The first cut of the order mask carried a bare "@ <price>" branch. Dates are
masked before it runs, so an observed close written "收盘 2026-08-10 @ 8.20"
reached it as "@ 8.20" and stopped being checked — the mask would have made
the gate blind to exactly the claim it exists to verify. An order level now
has to arrive with its quantity or under an order label, and a case pins the
observed-close form.
Two ways a run's own evidence went unseen.
A report writes the trading day as "08-10(一)" or "08-10(周一)盘中". The
yearless-claim pattern was anchored end-to-end with no room for a weekday or
session suffix, so the cell matched no evidence row and every price in that
row came back numeric_claim_unavailable — the bar had been fetched and the
quote was right.
The CSV stem map knew _V, _TO and _F but not _US, while ".US" is the venue
suffix the rest of the project resolves on. A US OHLC file the agent wrote
itself contributed no evidence, with the same result.
"GTC 100 @ $3.50" extracted 100 and 3.50 as candidate prices and compared
both against the session's OHLC range: 3.50 is a limit the report proposes,
and 100 is a share count that was never a price. A weekly update carrying a
GTC summary was rejected on quotes that were correct.
The label set is deliberately narrow. 买入价 / 卖出价 are NOT in it: in
running prose they name the price a report says it observed, and including
them stopped a genuinely ungrounded quote from being rejected — the existing
correction test caught that on the first attempt, and a case now pins it.
Two ways a correct draft was still refused.
A report writes the day as "08-10(一)" or "08-10盘中", and only the slash
form was masked, so 8 and 10 went to the OHLC check as candidate prices.
The dash is not symmetric with the slash — it also separates a range — so
the mask splits the two: a zero-padded month is formatting intent no price
range imitates, while 10/11/12 must carry a weekday or session marker.
"区间 8-10 元" stays checkable.
A level stated as a range fared worse than being unchecked. The separator
touches the second number, so masking "目标价 10" left "-20" behind, and a
negative price is outside every observed window — the draft could not pass
at all. Both bounds are masked now, mirroring what _PERCENT_RANGE_RE
already does for "1-2%".
_DATE_RE used \b as a word boundary, but \b is Unicode-aware so CJK
letters count as word characters. A date glued to Chinese text -- e.g.
"(2026-07-14最低)" -- had no boundary after "14" and was left
unmasked, contributing 2026/7/14 as candidate prices that rejected a
correct report. Compile the pattern with re.ASCII so \b is a byte
boundary.
Closes#1122
Signed-off-by: Robin1987China <41602358+Robin1987China@users.noreply.github.com>
Grounding gate (agent/src/agent/grounding.py):
- accept report-style weekday-suffixed dates ("08-10(一)", "08-10盘中") and
dash-form dates in prose, so correct drafts stop being rejected as
numeric_claim_unavailable
- map "_US" CSV filenames to ".US" so bash-written US OHLC CSVs are ingested
- mask order/limit levels ("100 @ $3.50", GTC, 挂单/限价单) as prospective,
not observed, quotes
- correction prompt now tells the model to remove rejected claims
Final-answer gate (agent/src/agent/loop.py):
- never release tool-call DSL (plain, ASCII-bar, or fullwidth-bar forms) as a
final answer; retry with a plain-text instruction, else deterministic fallback
- retry a single empty LLM response with a nudge; fail only on a second
consecutive empty response
- allow one extra revision when iteration budget remains
- mark degraded runs (safe-fallback released) with degraded=true + reason
- wrap-up nudge prioritizes finishing outstanding file writes
LLM timeouts (agent/src/config/env_schema.py, loop.py, providers/chat.py):
- add VIBE_TRADING_LLM_TIMEOUT_SECONDS (default 300s) bounding stream_chat
per-call and idle-delta, and the auto-compact summary call
- auto-compact degrades to hard truncation instead of freezing the run
Tests: regression coverage added for each fix across six test modules.
The pair branch of _FX_RATE_RE masked every ``AAA/BBB <number>``, and
``EUR/USD`` is this project's canonical forex symbol — forex.py names its pairs
that way and akshare_loader explicitly accepts the slash form. So the mask
covered exactly how an FX instrument quote is written, and an invented rate on a
first-class market would have passed the price gate unchecked.
Answering the question every mask has to answer: this one relaxed price
validation for all of forex. Narrowed so the pair form requires an
approximation marker — which a conversion basis carries and a quote does not.
The real case that motivated the mask, "远小于当前 USD/CAD≈1.36 的量级", still
masks; bare "USD/CAD 1.36" and asserted "usd/cad=1.36" go back to being checked
against observed evidence. The labelled 汇率 / "exchange rate" form is
unambiguous and keeps working without a marker.
The SPCX double-check report was rejected for its own structure and
conversion references:
- Markdown section numbers: "### 6. 关键价位" parsed the 6 as a price
claim (the ordered-list mask stops at the "### " prefix).
- Conversion ratios: "6:1 平价" and "6:1 折算" parsed 6 and 1.
- Forex rates: "USD/CAD≈1.36" parsed 1.36 as an instrument quote.
- Row counts: "25/25 行" and "243 行" (行 was missing from the
quantity units).
Mask numbered markdown headings, N:M ratios, currency-pair/exchange-rate
references, and add 行 to the quantity units. Observed quotes beside them
stay checked. Tests cover each class; the full agent suite keeps the same
environmental failure set with no new regressions.
The SPCX validation run had correct reports refused by the numeric gate
again — this time for document citations and counts, not prices:
- "~line 206" parsed the line number 206 as a price claim.
- "1 项事实错误" and "16 个交易日" parsed the counts 1 and 16 (项 and
个交易日 were missing from the quantity units).
- "7/10(150.57)以来最高" / "highest since 7/10 (150.57)" left the
earlier extreme (150.57) to be compared against the session's
observed window.
Mask line-number citations (line/行/第 N 行, including ranges), add
项/个交易日 to the quantity units, and add a since-reference branch
(以来最高/最低 and "highest/lowest since") for date-anchored extremes.
Observed quotes beside them stay checked. Tests cover each class plus an
end-to-end validation summary that previously failed.
The SPCX.US weekly-plan update was rejected by the numeric gate three
times even though the plan file was correct. Two masking gaps caused
legitimate claims to be read as observed prices:
- Reference levels survived extraction: "6/16 ATH 225.64 以来最高"
compared the all-time high (225.64) against the session's observed
OHLC range and reported numeric_claim_conflict. Same for 52-week and
historical extremes.
- A currency token between the anchor and the number broke every
prospective branch: "≥ $135", "≥ $119.68", "目标位 $6.80", and
"收盘 <$2.86" left their thresholds unmasked.
Add a shared optional currency token to branches (a)/(b)/(c) of
_PROSPECTIVE_LEVEL_RE, and a span-local _REFERENCE_LEVEL_RE for
ATH / all-time / 52-week / historical extremes. Observed quotes beside
them ("8/12 高 149.60") stay checked. Tests cover both mask classes
plus an end-to-end SPCX.US scenario that previously failed.
Markdown ordered-list markers (e.g. "1. **标题**") were parsed as floats
by _NUMBER_RE and rejected downstream as numeric_claim_conflict against the
observed OHLC range.
Add _MD_LIST_ITEM_RE and strip "^\s*\d+[.)]\s+" at the head of
_numbers_without_dates_or_percent's mask chain, so list markers are removed at
the number-extraction entry point across all numeric paths, not just price
comparison. In-text decimals like 1.5 are unaffected (digit after the dot).
Add tests in test_agent_grounding.py covering marker masking, conflict check
not weakened, and in-text decimal survival.
#1037 gave Eastmoney's Canada skip the status "unsupported: ...", while the
non-ASCII Yahoo skip already used "skipped: ...". The ledger recognizes a
non-failure by prefix only, so the second spelling read as a source outage:
a Canadian query returning no candidates became `invalidated` — a blocking
identity — instead of `not_found`, which is the state #1037 exists to avoid.
Measured before this change, with the real tool and the real validator:
eastmoney="unsupported: ...", yahoo=ok -> invalidated, honest answer REJECTED
eastmoney="skipped: ...", yahoo=ok -> not_found, honest answer accepted
eastmoney=ok, yahoo="HTTP 429" -> invalidated (a real outage, correct)
Both skips now share one `_SKIPPED` constant, and a cross-module test asserts
every skip status the resolver can emit is read as a non-failure by the ledger,
so the two sides cannot drift apart again silently.
Also add "C$" to the CAD aliases. A model writes a Toronto quote as "C$211.37",
not "211.37 CAD", and the currency check had the same gap that "¥" had for CNY.
The gate could reach a state where no market tool could run and no draft could
pass, so the run spent its whole iteration budget and returned the fail-closed
"could not safely lock the instrument identity" message. Seven defects, each
reproduced against the live resolver, the live validator, and real quotes.
Identity resolution
- Fold Shanghai's `.SS` onto `.SH`. Eastmoney and Yahoo publish one listing
under both spellings, so `search_symbol("600519")` returned two candidates
for one instrument and no tie-break could choose; every Shanghai query
resolved `ambiguous` and blocked every consumer for the rest of the run.
Symbols now canonicalize once (`.SS`, `sh600519`, `700.HK`, `BTC/USDT`), and
the deleted `.SS`/`.SH` alias-conflict check is replaced by one that fires on
an actual cross-exchange swap, which it never covered.
- Skip Yahoo for a non-ASCII query instead of letting it fail. That endpoint
answers any CJK query with HTTP 400, and "not found" required two clean
sources, so a Chinese name that simply is not listed came back `invalidated`
— a blocking state — and even the honest "I did not find it" was refused.
"Not found" now needs every source that could answer to have answered.
- Let a lock outrank a stale `invalidated`/`unresolved` side query.
`identity_status` took the worst status across all records, with no recovery
path, so one flaky resolver call ended the session's ability to answer.
`conflicting` still outranks everything; per-symbol safety was never carried
by this aggregate.
Consumer arguments
- Replace the nine-tool bare-ticker whitelist with canonicalization plus a
uniqueness constraint. Eleven documented argument spellings across the
registry were rejected while being the first example in their own tool's
schema (`get_market_data(["700.HK"])`, `get_fundamentals(["AAPL"])`,
`get_margin_trading("sh600519")`, ...). A bare code still resolves only when
exactly one locked identity has it as its base.
Final-answer validation
- Accept a localized source and currency. The answer follows the user's
language, but the check demanded the ASCII loader id verbatim, so
"数据来源：腾讯财经" failed against `tencent`, and CNY accepted 人民币 but not
元 or ¥ (港元/美元 still do not satisfy CNY).
- Stop splitting a grouped number. The ASCII comma is both a clause separator
and a thousands separator and the split ran first, so "¥1,309.22" left a
clause ending in "¥1" whose 1 was compared against the observed
1300.01–1363.35 range — every price above 999 written the ordinary way.
- Check an unattributed price against the union of observed quotes instead of
refusing it. Every comparison report names its second subject in words, and
that clause was rejected although the value was the observed close.
- Require a locked identity only once the run has an identity record, and drop
`ambiguous` from the blocking set to match `_RESOLUTION_INCOMPLETE_STATUSES`.
"什么是市盈率估值法？" matched the trigger phrase and could then never produce
a passing answer, and #955 unblocked skill selection for a shortlist while
leaving the final answer blocked. Consumers stay blocked on `ambiguous`, and
an unevidenced figure is still caught by the price and unsourced-symbol
checks.
33 tests added, two-sided: correct Shanghai/Shenzhen/HK/US/crypto/A+H
resolution, conceptual questions, localized prose and grouped prices all pass,
while invented quotes, cross-exchange rewrites, another currency's 元, partial
resolver outages and shortlist-driven quote fetches stay blocked. A mutation
test asserts Shanghai regresses to `ambiguous` without canonicalization.
The bash+yfinance workaround writes per-symbol OHLC CSVs into the run directory (data/raw/BYN_V.csv) instead of returning bars through get_market_data. Those prices are genuine observed tool output, but the grounding ledger only ingested JSON tool payloads, so the final-answer gate rejected every bash-derived price as numeric_claim_unavailable and withheld otherwise-correct answers.
Scan the run dir for OHLC CSVs and register their open/high/low/close/price rows as observed evidence, keyed to the symbol derived from the filename (BYN_V.csv -> BYN.V, PDI_TO.csv -> PDI.TO, GC_F.csv -> GC=F). Only files whose filename maps to a symbol already entitled in the run are accepted, rows are bounded by _MAX_GENERIC_EVIDENCE, and each file is ingested once. CSV cells are strings, so numeric cells are coerced with a dedicated helper.
Adds regression tests for CSV grounding, TSX filename mapping, fabricated-price rejection, and stray-symbol rejection.
Signed-off-by: Will Liao <liaotoca@yahoo.com>
Scope #1024 to its Canadian-market objective, retain the contributor history, and complete symbol routing, CAD-aware backtesting, Yahoo fallback, grounding, benchmark selection, documentation, and regression coverage.
Co-authored-by: Will Liao <liaotoca@yahoo.com>
Haozhe Wu