calc_metrics measures total_return and drawdown from initial_cash, but
the return series behind sharpe/sortino/vol/IR/beta started at a hard
zero: pct_change().fillna(0.0) drops the move from cash to the first
recorded bar, which carries real PnL whenever a warm-up boundary fills
at that bar's open. A 3-bar path that loses 10% on the first bar and
recovers reported sharpe 18.3 and sortino 2.9e9 while total_return was
0.0, and the equity.csv ret column compounded to +11.1%.
Add period_returns(equity, initial_cash) and use it in calc_metrics,
the equity.csv writer, and _calc_options_metrics, so prod(1+ret)-1
reproduces total_return and the ret column agrees with the drawdown
column row by row. bootstrap_sharpe_ci gains an optional
initial_capital, passed by run_validation, so observed_sharpe matches
the report's own sharpe; the standalone default keeps the legacy
adjacent-pair contract. Walk-forward windows keep their documented
observed-pairs convention.
No behavior change when the first recorded bar carries no fill: the
seed is 0 and the series matches the old one exactly.
Signed-off-by: Yufeng He <40085740+he-yufeng@users.noreply.github.com>
A local Parquet file whose bar dates live on a DatetimeIndex only had the
date column repointed when no date_format was configured. With a format
set, the mapping was never updated, the date column was not found, and
every bar was silently dropped.
The index handling moves into the shared column normalizer: an explicitly
mapped date column wins over the index (which also sidesteps the
reset_index crash when index and column share a name), otherwise the
index becomes the date column regardless of date_format. The caller's
column mapping is never mutated.
Supersedes the narrower report in #1819; thanks to rudycelekli for
flagging it.
Signed-off-by: Yufeng He <40085740+he-yufeng@users.noreply.github.com>
Every range check in these two families is a comparison, and a comparison
with NaN is False, so undefined inputs walked through and came out as
confidence numbers:
- bond_price(freq=inf) repriced any bullet bond to a plausible par: every
coupon collapses to zero and each discount factor to one. freq now joins
face and coupon_rate in the schedule admission, which also covers the
duration, convexity, dv01 and key-rate-duration paths that inherit it.
- accrued_interest admitted no numeric inputs at all, and its
on-coupon-date shortcuts return constants, laundering a NaN face into a
clean 0.0. Admission now runs before the shortcuts.
- ytm_solve compared NaN bracket bounds, failed silently inside the solver
and blamed the bracket for an unreachable price; price and both bounds
are now admitted first.
- key_rate_duration accepted a NaN tenor (neither empty nor non-increasing)
and returned a NaN key; dv01 scaled a NaN par_amount into a NaN hedge.
- probabilistic_sharpe_ratio checked only the observed Sharpe: a NaN
benchmark returned NaN as "confidence" and an infinite one a clean 0.0
or 1.0. All five summary inputs are now admitted.
- expected_maximum_sharpe accepted NaN and infinite trial counts, so
deflated_sharpe_ratio reported the resulting NaN or infinite bar as an
ordinary pass/fail verdict.
Supersedes the narrower reports in #1821 and #1822; thanks to rudycelekli
for flagging the pattern. Verified through the quantlib_call contract:
freq=1e309 and a NaN benchmark now come back ok=false naming the input.
Signed-off-by: Yufeng He <40085740+he-yufeng@users.noreply.github.com>
A-share rows carry fields literally named ``changepercent`` whose values
are already percentage points (0.52 means 0.52%). A formula over such
fields evaluates in points with no trailing ``*100``, but the grounding
gate compared a percent figure only against ``result * 100``: the
correct "-0.89%" was refused while the hundredfold-wrong "-89%" passed,
teaching the model to write the wrong number (#1820).
The unit now follows the evidence. When every data operand anchors only
to percent-named fields (``percent``/``pct``/trailing ``%``), the result
is read as points and meets the figure directly. An operand whose
magnitude also appears under a plain field keeps the fraction reading,
so the fix never widens acceptance where the unit is ambiguous, and an
explicit trailing ``*100`` still normalizes exactly as before.
Closes#1820
Signed-off-by: Yufeng He <40085740+he-yufeng@users.noreply.github.com>
monkeypatch.setattr on _pool._local.refcount/ib required the attributes
to already exist, but the pool creates them lazily per thread. Any run
that reached the reset before an acquire on a fresh thread (single-test
or -k-filtered runs) died with AttributeError, e.g.
pytest tests/test_ibkr_local.py::test_tier_hint_is_best_effort_against_old_sdk.
Pass raising=False so the reset works whether or not state exists.
The bench tool and tushare fallbacks both strip before use, so a
whitespace-only token is inert downstream but was still reported as
tushare_ready here.
Signed-off-by: Yufeng He <40085740+he-yufeng@users.noreply.github.com>
The options metrics built the drawdown peak from recorded equity only,
so the loss from initial cash to the first recorded bar never showed up:
the open on that bar pays commission, which guarantees the dip. A path
that falls 100 -> 95 -> 90 read as a 5.3% drawdown instead of 10%, and
a single losing bar read as no drawdown at all. calc_metrics seeds the
high-water mark with initial cash for the same reason; the options
engine now matches it. Invalid initial cash keeps the observed-peak
fallback and its warning.
Closes#1749
Signed-off-by: Yufeng He <40085740+he-yufeng@users.noreply.github.com>
Jackzigen's review on #1631: loop.py still imported copy although the
extraction left no user of it, and tests that monkeypatch the collapse
constants had no hint that the knobs now live in context_compact.py.
Signed-off-by: Yufeng He <40085740+he-yufeng@users.noreply.github.com>
Two more regressions from 2620a396 surfaced once the gate let tests run:
- infer_market deferred to the engine classifier before the crypto check,
so ETH/USD (a slash pair the engine reads as FX) came back "forex".
canonical_fx_pair already rejects it (ETH is not fiat); the crypto block
now runs before the engine fallback, matching the documented order.
- _validate_bench_data_requirements read registry.get(aid).meta directly,
which crashes on registry entries without a meta attribute (the symlink
regression test uses such a stub). getattr with a default keeps the
fundamental-column validation without the new assumption.
Signed-off-by: Yufeng He <40085740+he-yufeng@users.noreply.github.com>
2620a396 added a raw os.getenv() in the readiness probe, which trips CI
gate e (env reads must go through agent/src/config/). Route it through
get_env_config().data.tushare_token like the other API modules.
Signed-off-by: Yufeng He <40085740+he-yufeng@users.noreply.github.com>
american_waterfall takes as_of as a mapping of deal_id -> date, but a
non-mapping, non-None value (most plausibly a bare date, which is the
shape the sibling european_waterfall accepts for the same parameter
name) was silently treated as None. The caller's measurement date was
dropped and the preferred return accrued to each deal's own latest
flow date instead, returning a plausible-looking wrong number: on the
canonical 100-in/150-out deal at an 8% pref, as_of=date(2024, 1, 1)
priced the preferred tier at 8.0228 (the 2021 fallback) instead of
36.0776, and with a smaller distributable the error moves gp_total
and unpaid_preferred while every internal reconciliation still passes.
Refuse it the way _validate_deals refuses a non-mapping deals, which
test_american_waterfall_refuses_a_non_mapping already pins for the
neighbor argument. Also add a positive test that a per-deal mapping
is honored (hand-computed 36.077585).
Signed-off-by: Yufeng He <40085740+he-yufeng@users.noreply.github.com>
TDX quote servers stopped answering the client protocol mootdx/tdxpy
speak (2026-07, mootdx/mootdx#157): every kline category, market and
server returns empty, realtime quotes return zero rows, and get_k_data
dies on a KeyError inside mootdx. Upstream has been dormant since 2024,
and the weekly health lane has been red on mootdx since it started.
Each auto-mode fetch still spent ~9.5s inside mootdx before falling
through to eastmoney, so the dead slot was pure latency tax on every
A-share call.
Drop mootdx from FALLBACK_CHAINS["a_share"] and from the canary set
(tracked under EXCLUDED_PUBLIC_SOURCES with the upstream issue as the
reason). The loader stays registered: explicit source="mootdx" calls
keep working if a compatible fork ever restores the protocol. Docs
(en/zh/ja/ko/es/ar/id), tool schema hints, env examples and the
data-routing/mootdx skills updated to match.
Closes#1729
Signed-off-by: Yufeng He <40085740+he-yufeng@users.noreply.github.com>
joblib 1.6.0 pulls in cloudpickle>=3.0 and langsmith 0.14.2
switches from httpx to httpx2<3,>=2 (which itself needs
httpcore2==2.13.1 and truststore>=0.10). None of them were in
requirements-lock.txt, so the hash-locked install check fails
with "all requirements must have their versions pinned".
Pin all four with their PyPI artifact hashes; h11 0.16.0 is
already locked and satisfies httpcore2.
Signed-off-by: Yufeng He <40085740+he-yufeng@users.noreply.github.com>
Second slice of the #1622 migration after #1628: identity_not_locked,
figures_block_malformed, and unsourced_symbol_figures move out of the
inline rule list into declared registry rows, called at the exact points
they used to run so issue order is unchanged.
The ledger now stashes the parsed figures block and scanned figures for
the duration of a validation walk, which is what the registry contract
means by figure state living on the ledger; predicates receive only
(ledger, content). The per-figure role checks and the price-provenance
family stay inline on purpose: provenance gating depends on the loop's
checked_price state, and the role checks are the loop.
Golden pass: the full suite runs unchanged (16406 passed, 148 skipped).
Run cards now name all three checks in fired_checks when they fire.
Signed-off-by: Yufeng He <40085740+he-yufeng@users.noreply.github.com>
MARKET_SPECS only modelled full-day holidays, so on NYSE early-close
days (day after Thanksgiving, weekday Christmas Eve) the market-hours
gate stayed open until 16:00 ET and the runner ticked a closed market
all afternoon: reconcile, agent invoke, audit.
_MarketSpec now carries an early_closes date -> bell mapping and
market_is_open_at resolves the close per date. The static table covers
the same 2026/2027 window as the holiday set.
Signed-off-by: Yufeng He <40085740+he-yufeng@users.noreply.github.com>
A mixed rejection used to name only the failing figures. With 65 issues
on the table and nothing saying which values the gate had already
passed, the safest-looking play for the model was to rewrite whole
numeric sections as qualitative prose. zeus229's production run on #1622
lost a valid Risk X-Ray block and a portfolio-vs-benchmarks table that
way: the corrected draft passed, minus most of its numbers.
The gate now carries the measured figures that checked clean on the
ValidationResult, and the correction prompt tells the model to keep
them exactly as written: "only the figures listed above need work". A
spelling that fails anywhere in the draft is endorsed nowhere, and bare
integers stay off the list since an unchecked pass would vouch for
nothing.
Tests: three new pins in test_grounding_release.py, red before the fix
and green after (the keep list names the clean figures in document
order and none of the rejected ones; no section renders when nothing
passed; the list caps at 24 values and counts the rest). The 667
grounding tests and the full agent suite pass.
Ref #1622
Signed-off-by: Yufeng He <40085740+he-yufeng@users.noreply.github.com>
Main now stamps the price basis on every FMP/Tiingo frame and the loader
cache whitelists the attr across warm hits, but nothing fails if the stamp
disappears: the loader tests assert prices stay on one basis, and the cache
round-trip runs with an attrless frame, so the provenance table could fall
back to the static split_dividend default on raw prices again with every
test green.
Assert the explicit stamp on both bases for both loaders, and round-trip
an adjustment="raw" frame through a real duckdb cache.
Signed-off-by: Yufeng He <40085740+he-yufeng@users.noreply.github.com>
Upstream grew the replay-visibility lease (preserve_tool_call_ids) through
_microcompact and _context_collapse after this branch moved both helpers to
context_compact.py. Re-apply the extraction on top: loop.py keeps the
upstream AgentLoop changes and drops the moved helpers again, and the two
helpers in context_compact.py take the new keyword and its guards so the
upstream call sites keep working.
Signed-off-by: Yufeng He <40085740+he-yufeng@users.noreply.github.com>
On a host with no usable CJK TTF the first probe correctly rendered
with Helvetica, but it also marked the font probe as done, so every
later render in the same process asked platypus for the unregistered
VibeShadowCJK family and reportlab refused the paragraph. The report
then silently degraded to HTML-only from the second render on, which
is exactly what CI showed on the second reportlab test in a process.
Cache the resolved family instead of a bare probed flag, and add a
regression test that renders twice on a fontless host.
Also add reportlab to the desktop embedded-runtime lock; the wheel
metadata check failed packaging because the new dependency was not
installed there.
Signed-off-by: Yufeng He <40085740+he-yufeng@users.noreply.github.com>
Refs #1677. The shadow report's PDF half depends on weasyprint's system
pango/cairo libraries, which pip cannot install; hosts without them
silently degraded to HTML-only output. Add a pure-Python fallback that
renders the same 8-section payload (plus charts) through reportlab
platypus, with CJK font registration from the local system list. Engine
order is weasyprint, then reportlab, then html-only; the weasyprint path
and its one-shot import probe are unchanged.
Signed-off-by: Yufeng He <40085740+he-yufeng@users.noreply.github.com>
When write_file/edit_file reject a path outside the allowed roots, the
error only said the path escaped, so neither the model nor the user
could tell how to widen the boundary (#1677). resolve_safe_path now
names the purpose's widening knob, VIBE_TRADING_ALLOWED_WRITE_ROOTS on
the write side and VIBE_TRADING_ALLOWED_FILE_ROOTS on the read side,
with the usual MCP scope hint, matching the sibling run-dir and
import-root rejections.
The knob itself already worked; it was just undocumented. Add it to
.env.example and the README table next to the other two root lists,
and drop an unused import in the touched module.
Tests: three rejections pin that the write side names WRITE_ROOTS (both
the bare and the run_dir-fallback path) and the read side names
FILE_ROOTS instead.
Signed-off-by: Yufeng He <40085740+he-yufeng@users.noreply.github.com>
When write_file/edit_file reject a path outside the allowed roots, the
error only said the path escaped, so neither the model nor the user
could tell how to widen the boundary (#1677). resolve_safe_path now
names the purpose's widening knob, VIBE_TRADING_ALLOWED_WRITE_ROOTS on
the write side and VIBE_TRADING_ALLOWED_FILE_ROOTS on the read side,
with the usual MCP scope hint, matching the sibling run-dir and
import-root rejections.
The knob itself already worked; it was just undocumented. Add it to
.env.example and the README table next to the other two root lists,
and drop an unused import in the touched module.
Tests: three rejections pin that the write side names WRITE_ROOTS (both
the bare and the run_dir-fallback path) and the read side names
FILE_ROOTS instead.
Signed-off-by: Yufeng He <40085740+he-yufeng@users.noreply.github.com>
#1486 added the tool to six READMEs but missed the Indonesian one, so
test_readme_counts fails on every PR merge ref: the enumerated list is
missing get_southbound_flow and the three counts still say 74/74/107.
Signed-off-by: Yufeng He <40085740+he-yufeng@users.noreply.github.com>
Financial reports write negatives as (25.3) / （25.3）. The extractor
searched the bare digits out of the parentheses, so a loss was audited
as a profit, and the closing paren swallowed the unit: (25.3)亿 came
out as +25.3 with no unit at all (#1660).
Tables now treat a parenthesized number as negative and read the unit
past the closing paren, per accounting convention. Prose label:value
lines do the same only when a unit follows, so a bare (2024) stays a
dropped year instead of becoming a data point.
Refs #1660. Touches the same extraction regexes as #1654 (sign
preservation); the two compose, and I will rebase whichever lands
second.
Signed-off-by: Yufeng He <40085740+he-yufeng@users.noreply.github.com>
#1628 gave checks stable names so a run card can say what fired, but the
artifact still recorded only issue codes; naming a fired check meant
re-running the gate. Each recorded validation now carries fired_checks,
the declared checks whose code appears in the attempt's issues, resolved
through the registry in registration order. Rules that still run inline
emit codes only, so the list stays empty until their check is migrated.
The gate itself is untouched: names are derived from the already collected
issues after the verdict. Grounding suites (922 passed) and the full
offline suite (15996 passed) are green, and new tests pin the reverse
lookup order, the named fire, and the empty list for inline-rule and
clean attempts.
Refs #1622
Signed-off-by: Yufeng He <40085740+he-yufeng@users.noreply.github.com>
loop.py was 3761 lines and one of the most-touched files in the tree. The
context-compaction and message-normalization cluster (chunked summary
folding, microcompact, context collapse, tool-pair repair, thought-signature
attachment) reads no AgentLoop state, so it moves verbatim to
agent/src/agent/context_compact.py. loop.py re-exports every moved name, so
existing imports keep working; the split is a pure code move with identical
behavior. First slice of the roadmap split (#1624); the tool-call and
streaming seams follow in their own PRs.
Signed-off-by: Yufeng He <40085740+he-yufeng@users.noreply.github.com>
The edge fold could not tell a quote wobble from a dividend landing on the
first or last bar of the window: a 0.03 dividend on the final bar of a flat
10.00 series converted to the exact same series as without it. Only interior
one-bar plateaus whose two neighbors agree still fold as wobble.
Also drops the mechanical line-length reformat that had swollen the diff of
runner.py, registry.py, market_data.py and the three touched loaders; the
change is now the conversion, the frame caliber, and their tests only.
Signed-off-by: Yufeng He <40085740+he-yufeng@users.noreply.github.com>
warren618's review of the additive conversion:
1. A window with no corporate action inside it now stays as served: with no
plateau boundary there is no factor, so the converter returns None and
the loader keeps today's series and static stamp. This also un-regresses
A-share indices (000300.SH has qfq == raw on every bar, so it always took
the identity path and got mislabeled split_dividend).
2. The frame stamp is now read on both paths: frame_caliber(frame, source,
market, symbol) prefers the frame's own adjustment attribute and falls
back to the static table; market_data._emit and the five runner sites all
go through it, so a converted series is labeled split_dividend inside a
backtest too and no longer trips the mixed-caliber warning against a
genuinely multiplicative source.
3. The 0.05 merge tolerance is gone. A one-bar plateau folds only when the
neighboring level(s) match within tolerance (a quote wobble), and a
one-bar plateau that cannot be classified that way refuses the window
instead of swallowing a sub-5-fen step and shipping a wrong series with a
clean stamp.
4. _plateau_spans' docstring now matches reality: it returns None for an
unclassifiable one-bar plateau, and the module docstring states the
window-end anchoring and that 送转 windows drift with the price level and
are refused by construction.
5. The ETF guard is now uniform across tencent, eastmoney, and akshare.
6. Measured on the live endpoint: a 10-year window costs 1.32s for qfq plus
1.29s for the raw companion (two paginated series), roughly doubling
requests and wall time; a cached refetch is 0.52s.
7. Tencent's raw companion moved to the kline endpoint: fqkline without the
qfq suffix serves no bars at all, while kline/kline returns the
unadjusted day series. Verified live: 600519.SH 2019 converts to
split_dividend with close ratio 2.0043 (+100.43%), the reference number.
Suites: 149 scoped tests green (conversion + three loaders + caliber +
runtime runner), full agent suite 14755 passed / 148 skipped, ruff check
and format clean on the touched files.
Tencent, Eastmoney and AKShare serve A-share forward-adjusted prices that
are dividend-ADDITIVE: a return computed on them is not a total return
(600519.SH 2019 buy-and-hold reads +184.07% additive against +100.43%
multiplicative), and old levels can go negative (its 2012 additive closes
reach -167.36). #1497 stamped the series and warned; #1541 asks for the
conversion.
Between corporate actions qfq - raw is a constant offset, and each
down-step at an ex-date is the cash dividend per share. The loaders now
also fetch the raw companion (tencent day, eastmoney fqt=0, akshare
adjust="") and, when the offsets form valid plateaus, apply the
(prev_close - dividend) / prev_close factors cumulatively so the series
lands on the multiplicative basis, stamped split_dividend. cn_adjust's
conventions carry over: OHLC scale by the ratio, volume divides by it,
amount is untouched.
Validation is the contract: misaligned calendars, offsets that drift bar
to bar (a ratio-adjusted series' signature), sub-cent noise steps,
non-positive or over-price dividends, and windows dominated by one-bar
plateaus all fail closed to the additive series, stamp and warning. A
failed raw fetch degrades the same way, never fails the symbol.
Measured on the live Tencent endpoint for 600519.SH: 2019 buy-and-hold
reads +100.43% after conversion (additive +184.07%, raw +97.50%, all
three matching the issue's references), 2016 converts (+62.54% against
the additive -148.52%), and the 2012/2015 windows whose offsets do not
form plateaus keep the additive series and warning.
Tests: additive_conversion suite 15/15 (conversion math, inverse volume
scaling, fail-closed shapes, loader wiring, provenance stamp preference,
raw-failure degrade); the full backend suite is green. Existing
tencent/eastmoney loader tests now follow the dual fetch and the new
_request_page signature.
Risk: a window that converts changes its served prices from the additive
levels to multiplicative ones, which is the point; downstream consumers
already reading the adjustment provenance see split_dividend and the
additive warning goes quiet for the right reason. Rollback: revert the
loader fetch blocks; the static caliber table still stamps the sources
additive and everything behaves as before.
Signed-off-by: Yufeng He <40085740+he-yufeng@users.noreply.github.com>
policies.py grew one inline rule per incident; adding a check meant editing
shared control flow. This adds GROUNDING_CHECKS: each check is a declared row
with a stable name, the issue code it emits (release decisions still key on
codes), a description, and the predicate. Migrated checks run exactly where
they used to run inline, preserving issue order.
First migrated check: listed-identity-relabelled-private, moved verbatim into
identity_checks.py. Golden pass: the grounding suites and the full offline
suite (15992 passed) are green unchanged, and new registry tests pin duplicate
rejection, walk order, and the identical fire through the gate.
Refs #1622
Signed-off-by: Yufeng He <40085740+he-yufeng@users.noreply.github.com>
The contract mostly existed (BaseChannel abstracts, config_meta field hints
with fail-safe secret masking, token_probe) but nothing enforced it, so a new
channel could ship a scalar config key with no form hint or an unmasked
credential-shaped value. test_channel_authoring_contract.py walks every
registry-discovered channel and fails on exactly those, and CONTRIBUTING gets
the "Adding a Channel" quickstart that names the recipe pieces.
Writing the suite surfaced two documented subtleties now pinned by name:
dict-valued config stays file-configured (the generic form cannot edit it),
and a hand-written hint may authoritatively subtract a regex over-match
(websocket's token_* path names).
Refs #1625
Signed-off-by: Yufeng He <40085740+he-yufeng@users.noreply.github.com>
Which read and order paths each connector exposes lived in per-connector
code. The profiles already declare it, so render them into a marked block in
the README's connector onboarding section; a drift test fails CI the moment
a profile changes without `python -m src.trading.capability_matrix`, so the
table cannot go stale.
Refs #1626
Signed-off-by: Yufeng He <40085740+he-yufeng@users.noreply.github.com>
Which read tools, quote paths and order kinds each connector actually has
lived in per-connector code and tribal knowledge. The profiles already
declare it, so render them: docs/broker-capabilities.md is generated by
`python -m src.trading.capability_matrix`, and a drift test fails CI the
moment a profile changes without a regenerate, so the table cannot go stale.
README's connector onboarding section links to it.
Refs #1626
Signed-off-by: Yufeng He <40085740+he-yufeng@users.noreply.github.com>
The last few loader failures were all found by users rather than by us:
the northbound feed died silently (#1481), baostock can hang forever
(#1492), the A-share qfq caliber drifted (#1541). This adds a weekly lane
that hits every no-auth loader with one known-liquid symbol and asserts a
non-empty, correctly shaped, fresh frame, so a dead or drifted endpoint
goes red on CI instead of in someone's run.
The canary module is env-gated (VIBE_TRADING_LOADER_HEALTH=1) and
integration-marked, so the default CI lane skips it entirely. Loaders that
cannot be reached from the runner at all are reported as unreachable, not
failed; empty or stale answers fail, with one retry to ride out flaky
connections.
Closes#1623
Signed-off-by: Yufeng He <40085740+he-yufeng@users.noreply.github.com>
baostock 0.9.3's send_msg reads with no socket timeout and loops until a
terminator that never comes on a closed connection: a silent server hangs
the loader forever, a closed one spins at 100% CPU. The loader now swaps in
a bounded send_msg and a connect with a deadline for the lifetime of each
fetch. The replacement is byte-compatible on a healthy server and returns
None on timeout, EOF, or a malformed reply, which every baostock caller
already maps to BSERR_RECVSOCK_FAIL, so a dead server degrades into an
ordinary per-symbol failure that falls through the loader chain.
The inactivity deadline defaults to 30s per read and can be tuned with
VIBE_TRADING_BAOSTOCK_TIMEOUT. Fake-server tests cover the silent-server,
close-immediately, healthy-roundtrip, malformed-reply, and refused-connect
cases.
Closes#1492
Signed-off-by: Yufeng He <40085740+he-yufeng@users.noreply.github.com>
The backtest sandbox re-exposed only cache/data-bridge/qveris config, so
the MT5 loader found no mt5.json inside it, called initialize() bare, and
the backtest silently fell back to akshare while get_market_data worked
in-process. Worse, a path-less initialize() lets the SDK pick a terminal
itself, which on multi-install hosts can be a different installation
logged into a different account.
The sandbox now writes a secret-free subset of the host's mt5.json
(terminal_path and timeout only) so the loader attaches to the
already-running, already-logged-in terminal, while login/password/server
stay out of strategy code per VT-001. The bare-attach path itself stays
as documented, with a warning that names the wrong-terminal risk and how
to pin it.
Signed-off-by: Yufeng He <40085740+he-yufeng@users.noreply.github.com>
The latest-only report (RPT_HOLDERNUMLATEST) answers exactly one row per
symbol regardless of pageSize, so max_periods did nothing, and its
"quarter-over-quarter" label was wrong for the 27.7% of rows whose
PRE_END_DATE is not the previous quarter-end (#1503 measured nine months
to nine years).
The history now comes from RPT_HOLDERNUM_DET, which pages one row per
disclosure. Each row states its own prev_period_end (the next returned
row's end date; the latest-only report's PRE_END_DATE on the fallback
single row) instead of claiming quarter-over-quarter. The average-holding
and market-cap columns exist only on the latest-only report, so its row
joins onto the newest detail row when the two agree on the end date; a
symbol absent from the detail report falls back to that single row. The
join is best-effort and never fails a working history answer.
Signed-off-by: Yufeng He <40085740+he-yufeng@users.noreply.github.com>
Rework after review: the prose-side frame parsing is out (the gate reads no
natural-language words), and a figure's measure/confidence identity now comes
from the declaration the model already writes. ref may name an exact evidence
field (var.var_95), or call_id::field when a field repeats across calls; a
field-only ref that repeats across calls fails closed instead of pooling. The
es_95/es_99/cvar_95/cvar_99 aliases stay: those leaves resolved to nothing
before. Two-sided coverage: field-ref scoping tests plus the nine adversarial
prose rows all rejecting.
Ref-field scoping shape prototyped by zeus229 in
contrib/grounding-tail-risk-ref-identity-v2.
Signed-off-by: Yufeng He <40085740+he-yufeng@users.noreply.github.com>
portfolio_risk_xray called fetch_market_data without max_rows, so once a window crossed the shared 250-row cap the series came back sampled: consecutive-return inputs broke and longer windows failed with 'no close prices returned'. Ask for max_rows=0 like technical_indicators already does; volatility, VaR/ES and drawdown need consecutive bars. Regression test drives 300 daily bars through a capped fetcher and asserts all of them reach the report.
Signed-off-by: Yufeng He <40085740+he-yufeng@users.noreply.github.com>
Tail-risk evidence resolved to a flat "tail_risk" kind, so a var_95 reading
grounded a "VaR 99%" or "ES 95%" claim as long as the number matched, and the
confidence level in "VaR 95%: 1.57%" was itself checked as a measurement and
rejected against the evidence.
The prose frame is now read at scan time: a percent figure in a tail-risk
frame carries its (measure, confidence), the confidence figure in the frame
is flagged and never checked, and the metric pool narrows to evidence of the
same measure, plus the same confidence when the claim states one. The frame
ends at the next measure word or clause end, so a following drawdown claim is
not swallowed. A declared role still wins over the frame mask.
The evidence side resolves the same identity from leaf names, with es_95 /
es_99 / cvar_95 / cvar_99 added to the alias table (they used to fall out of
the exact-only guard). Compound leaves keep their confidence
("portfolio_cvar_95" -> ES at 95%).
Closes#1425
Signed-off-by: Yufeng He <40085740+he-yufeng@users.noreply.github.com>
Tushare's ts_code suffixes (SHF/CZC/CFX/GFE) never matched the
canonical exchange set (SHFE/ZCE/CFFEX/GFEX), so CU1811.SHF classified
as an A-share at market detection and as global futures at the engine
layer, picking up a USD 50x multiplier and USD commission on a CNY
contract. Normalize the suffix to the canonical exchange before the
set-membership test, and admit the Tushare spellings in the futures
market pattern.
Signed-off-by: Yufeng He <40085740+he-yufeng@users.noreply.github.com>
MT5: _sized_volume clamped an oversize order to the symbol's
volume_max and sent it while the below-min side errors. Return an
error naming the cap instead, same style as the min branch.
eToro: a limit order without limit_price went out as orderType=mit
with no TriggerRate and rested untriggered while reading as
accepted. Error before any HTTP call, matching the side/type guards.
futu: get_historical_bars inherited the SDK's implicit autype
default. Pass AuType.QFQ explicitly and declare adjustment=qfq in
the response so consumers know the served caliber. No behavior
change.
market hooks: separator-less crypto pairs (BTCUSDT) fell through to
the a_share default and picked up T+1, no-shorting and CNY
settlement. Route the trade-journal quote-asset table
(USDT/USDC/BUSD) to crypto; metals/FX still end in USD and stay
forex.
Each new test is red on current main and green here.
Signed-off-by: Yufeng He <40085740+he-yufeng@users.noreply.github.com>
Broker exports carry cash-dividend rows (同花顺 红利入账, 富途 Side=Dividend).
_normalize_side rejected them, so one dividend row killed the whole journal
parse and the shadow attribution chain silently returned all zeros (#1207
item 15).
The parsers now normalize those rows to side="dividend" records whose amount
field carries the cash, and drop known non-trade corporate-action rows
(bonus shares, rights issues, interest, split-style events) instead of
raising; split effects on covered symbols are already restated by the #1311
frame-caliber adjustment. Truly unknown sides still raise.
pair_trades_fifo skips non-buy/sell rows explicitly, and the profile and
behavior stats are computed on trade rows only, with total_dividends summed
separately and folded into total_pnl. In the shadow backtest the dividend
cash joins real_pnl only for symbols the run's adjusted frames do not cover;
for covered symbols the dividend is already embedded in the adjusted price
caliber, so adding cash there would double count it.
Signed-off-by: Yufeng He <40085740+he-yufeng@users.noreply.github.com>
The layer-1 microcompact replaces old tool results with [cleared] under
token pressure, but the dedup ledger kept the tool name in _called_ok,
so a re-call was blocked with "use the previous result" that the model
can no longer read. Financial data tools hit this hard: fetch once,
lose the result to compaction, and every retry is intercepted, leaving
the model to report fundamentals as unavailable (#1343).
_microcompact now returns the names whose every result just became
unreadable, and the caller drops them from _called_ok and writes a
microcompact_cleared trace event (the clearing used to be silent, which
is why the trace showed the retries but not the cause). A tool keeps
its block as long as some result of it is still in context, including a
short one under the 100-char clearing floor.
AI-assisted: yes (Kimi K3); mechanism verified against a trace-evidenced
repro, red-green tests added.
Signed-off-by: Yufeng He <40085740+he-yufeng@users.noreply.github.com>
Stooq answers non-browser clients with a JavaScript proof-of-work
challenge page (HTTP 200, HTML body). The loader parsed that as no data,
so the us_equity fallback chain treated a never-serving source as a
healthy position and nothing in the run log said otherwise.
Detect the challenge shape before parsing, treat the source as
unavailable for the call, and warn once per process with the operator
guidance (reorder via VIBE_TRADING_MARKET_DATA_ORDER_* or drop stooq).
Signed-off-by: Yufeng He <40085740+he-yufeng@users.noreply.github.com>
Once the bounded ffill limit expires, the aligned close goes NaN and
_safe_price fell back to pos.entry_price, so a position held through a
long halt was re-marked at entry cost: the equity curve printed a
phantom drawdown mid-halt, rebalance sizing ran against the phantom,
and the curve recovered on resume.
_align now also emits an unbounded-ffill valuation view, and the
valuation call sites (open equity used for rebalance sizing, equity
snapshots, actual weights, terminal forced liquidation) mark from it.
Trading paths are unchanged: order pricing still reads the bounded
matrix, so a halted symbol still cannot fill at a stale price.
Signed-off-by: Yufeng He <40085740+he-yufeng@users.noreply.github.com>
A partially successful loader stopped the whole walk: one 404 out of
five symbols left the other four in _unresolved with no per-symbol retry
(the venue-alias rescue was the only exception). The walk now tracks the
missing symbols and keeps going with just those, merging each source's
coverage, so a partial batch resolves what it can instead of stranding
the rest. Provenance now names the source each symbol actually came from
instead of pinning the group's first success on every symbol, and a
source that returns nothing no longer gets recorded as the serving
source.
Refs #1207 (Phase 1, item 7).
Signed-off-by: Yufeng He <40085740+he-yufeng@users.noreply.github.com>
Every other loader on the fallback chain returns adjusted prices (qfq),
but both Yahoo paths served raw OHLC, so a split read as a crash and
every ex-dividend gap booked as a fake loss, and two runs over the same
market could mix calibers depending on which source won the chain. The
v8 chart client now requests events=div,splits and scales OHLC to the
adjclose series when Yahoo provides it (raw volume kept, matching the
qfq convention); a missing or implausible ratio falls back to raw rather
than corrupting the bar. The yfinance path flips to auto_adjust=True,
which is the same caliber decision on that side.
Refs #1207 (Phase 1, item 6).
Signed-off-by: Yufeng He <40085740+he-yufeng@users.noreply.github.com>
_jobs_from_triggers downgraded a MARKET trigger to a plain 60s interval
job, and the tick never re-checked the session: due_now sat imported
with zero call sites, so weekend ticks ran the full pipeline and
submitted orders that queue into the Monday open. The runner now gates
on the trigger layer the jobs were synthesized from: with MARKET
triggers attached, a tick proceeds only when at least one of their
markets is open at the tick instant. Interval-only and 24/7 (crypto)
channels never block, and the skip stops before reconcile, so a closed
session costs no broker calls and writes no audit records.
Refs #1207 (Phase 0, item 5).
Signed-off-by: Yufeng He <40085740+he-yufeng@users.noreply.github.com>
The HALT sentinel is a file and survives a restart, but _flatten_fired
does not. A restart with flatten orders still working replayed the whole
sweep on the next tick: a cancel pass plus a fresh market order per
position, which can flip the account from long to net short. The latch
now lives on disk next to the per-broker HALT sentinel, bound to the
halt episode (tripped_at, or the file mtime for a hand-touched
sentinel). A fresh trip writes a new sentinel, so clearing HALT and
re-tripping re-arms the sweep without anyone deleting the latch.
Refs #1207 (Phase 0, item 4).
Signed-off-by: Yufeng He <40085740+he-yufeng@users.noreply.github.com>
The MCP adapter converts a failed remote call into a {"status": "error"}
envelope instead of raising, but the preemptive sweep treated any
non-exception response as success. A shaky broker connection during a
halt therefore wrote a compliant-looking audit trail while the resting
order stayed live, which is the exact failure the sweeper exists to
prevent. Error envelopes now land in report["errors"] with an "error"
audit outcome, for both the cancel pass and the flatten pass.
Refs #1207 (Phase 0, item 3).
Signed-off-by: Yufeng He <40085740+he-yufeng@users.noreply.github.com>
stream_chat() never asked the endpoint for usage, so every streamed call
fell back to len(content)//4 estimates and run.json under-reported output
tokens by an order of magnitude (issue #1224). Set stream_usage=True on
the OpenAI-compatible branch so the final stream chunk carries real
counts, and self-heal endpoints that reject stream_options the same way
the Anthropic temperature path does: classify the 400, retry once
without the field, and remember the model process-wide.
Signed-off-by: Yufeng He <40085740+he-yufeng@users.noreply.github.com>
_attribution_or_zero read total_return_abs/total_pnl, keys the runner never
emits (its metrics schema is final_value, total_return, ...), so every
successful backtest was reported as Shadow PnL = 0 and the Delta comparison
came out inverted. The or-chain also swallowed a genuine zero PnL.
Resolve PnL as explicit keys (is-not-None) -> final_value - initial_capital
-> total_return * initial_capital, and carry an explicit unknown state
(None) through the result, cache, tool payload, waterfall and HTML instead
of rendering 0.00 and a signed delta for failed runs.
Signed-off-by: Yufeng He <40085740+he-yufeng@users.noreply.github.com>
OKX and Futu flattened non-zero business codes into an empty position or
balance list with status ok, so the mandate gate evaluated limits against
an empty book whenever the broker API rejected the call. Both connectors
now return an explicit error envelope on a failed read.
Part of #1207 Phase 0.
Signed-off-by: Yufeng He <40085740+he-yufeng@users.noreply.github.com>
Alpaca reports qty as an absolute magnitude with direction in side, but
the mandate gate reads quantity as signed (its own fixtures use negative
quantities for shorts). An Alpaca short therefore booked as positive
exposure: every additional sell moved the computed exposure toward zero,
so max_total_exposure_usd and max_leverage stopped binding, and a
buy-to-cover read as adding long. Negate qty when side is short, matching
the convention the gate and the tiger connector already share.
Part of #1207 Phase 0.
Signed-off-by: Yufeng He <40085740+he-yufeng@users.noreply.github.com>
With VT_MEMORY_HIERARCHY on, add() passed the bare slug to
route_entry(), which uses it as the leaf filename verbatim, while every
discovery path filters on the .md suffix. New entries landed on disk
invisible to list_entries()/find()/auto-recall. The existing routing
test only asserted the file exists, which is why it shipped.
_create_market_engine's yfinance branch handed every symbol to
GlobalEquityEngine on the loader name alone, so crypto pairs fetched
from Yahoo ran under zero-commission equity rules while the
CryptoEngine fee keys (taker_rate/maker_rate/slippage) sat ignored in
config. Two runs differing only in fees produced byte-identical
metrics (#965).
Route that branch on the instrument market like the fallback branch
already does, and teach the market classifier yfinance's native crypto
spelling (BTC-USD) next to the USDT forms so the routing actually
covers what Yahoo serves.
Fixes#965
Signed-off-by: Yufeng He <40085740+he-yufeng@users.noreply.github.com>
* feat(runs): surface Portfolio Studio artifacts in run detail
The backtest emits risk_xray.json and rebalance_notes.json for every
portfolio run, but nothing could read them short of opening the run
directory. The run endpoint now returns both payloads as optional
fields, and the run detail page renders them in a Portfolio Studio tab
(x-ray stats and weights, rebalance summary and per-rebalance table),
hidden for runs without them. i18n for all five locales, route tests
for both payload shapes, and a render test for the tab.
Signed-off-by: Yufeng He <40085740+he-yufeng@users.noreply.github.com>
* test(runs): use distinct symbols in the studio tab render test so getByText matches uniquely
Signed-off-by: Yufeng He <40085740+he-yufeng@users.noreply.github.com>
* test(runs): expect the repeated date and turnover strings in the studio tab test
The rebalance date renders in both the table and the largest-rebalance
card, and the 35% turnover renders in the row plus the mean and max
cards, so bare getByText throws on multiple matches. Assert the exact
counts instead, which also pins where each value is supposed to show up.
---------
Signed-off-by: Yufeng He <40085740+he-yufeng@users.noreply.github.com>
* feat(backtest): emit risk x-ray artifacts in portfolio runs (Portfolio Studio slice)
Signed-off-by: Yufeng He <40085740+he-yufeng@users.noreply.github.com>
* fix(backtest): refuse the average-basket x-ray for long-short books
compute_risk_xray is long-only by contract, but average_invested_weights
filtered negative mean weights out before the check could fire — a
long-short run would emit an x-ray describing only its long half while
presenting it as the whole strategy. Net short average exposure now
raises, taking the same no-artifact path as flat and short-history runs.
---------
Signed-off-by: Yufeng He <40085740+he-yufeng@users.noreply.github.com>
Co-authored-by: Haozhe Wu <haozhe_wu@connect.hku.hk>
Portfolio Studio step 2 (#456): per-name cap, per-name floor, and per-group exposure caps as a shared layer on top of every optimizer, not just turnover_aware's built-in bounds. Configured via constraints in config.json, off by default.
Signed-off-by: Yufeng He <40085740+he-yufeng@users.noreply.github.com>
The strict bench runner (mandatory same-universe random control plus an
optional train/test OOS split) has shipped since 0.1.9 but nothing
invoked it: no CLI flag, no API route, no tool reference, so the only
gate users could run was the raw-IC one its own docstring critiques.
This wires it into the CLI: --strict routes the bench through
bench_runner_strict with the random control on, --oos-split and
--random-seeds tune the gate, the pre-flight banner shows the mode, and
the JSON envelope carries the strict category counts. Passing
--oos-split/--random-seeds without --strict is rejected up front so
they cannot silently no-op.
Fixes#773.
Signed-off-by: Yufeng He <40085740+he-yufeng@users.noreply.github.com>
Last backend slice of the Portfolio Studio epic: per-rebalance turnover
and weight-drift notes computed from the target position frame, so they
land for every optimizer and for the no-optimizer baseline. Writes
artifacts/rebalance_notes.json (strict JSON, same writer discipline as
validation.json) and a Markdown twin, and surfaces rebalance_count /
rebalance_turnover_mean / rebalance_turnover_max as scalar metrics that
the run card picks up on its own.
Signed-off-by: Yufeng He <40085740+he-yufeng@users.noreply.github.com>
First slice of the Portfolio Studio roadmap item ("risk x-ray ... rest
Planned"): a pure-computation risk x-ray for weighted baskets, plus an
agent-facing tool that fetches closes through the standard loader
fallback chain.
backtest/risk_xray.py computes, over trailing returns only (no look-ahead
by construction): concentration (HHI / effective N / top-1 / top-3),
annualized and downside volatility, max drawdown with dates, historical
VaR + expected shortfall at 95/99, diversification ratio, and pairwise
correlation / beta vs an equal-weight proxy. Weights are validated and
renormalized with a warning; thin symbols fall out through a min-history
filter; every emitted float is finite-guarded so the payload always
survives json.dumps(allow_nan=False). Long-only for now — a short leg
changes concentration semantics and deserves its own slice.
The portfolio_risk_xray tool handles argument validation, shapes the
fetch envelope into a close panel (records with missing or mixed
timestamps fall back to loader order), and returns a strict-JSON
envelope with unresolved/skipped bookkeeping. Data fetching is
injectable for tests.
17 new tests cover the math on hand-built panels, the edge matrix
(renormalization, thin symbols, single asset, zero variance, bad
args), and the tool end to end with a stub fetcher. Full suite:
6037 passed, 12 skipped.
Signed-off-by: Yufeng He <40085740+he-yufeng@users.noreply.github.com>
Yufeng He