kenneth/Auto-Trader
1
0 Fork 0
Code Issues Pull Requests Actions 16 Packages Projects Releases Wiki Activity

Compare commits

base: kenneth/Auto-Trader:a8144e9acfef981933c15e8d6ef04f37acf987bb
kenneth/Auto-Trader:main
kenneth/Auto-Trader:v0.1.16 kenneth/Auto-Trader:v0.1.15 kenneth/Auto-Trader:v0.1.14 kenneth/Auto-Trader:v0.1.13 kenneth/Auto-Trader:v0.1.12 kenneth/Auto-Trader:v0.1.11 kenneth/Auto-Trader:v0.1.10 kenneth/Auto-Trader:v0.1.9 kenneth/Auto-Trader:v0.1.8 kenneth/Auto-Trader:v0.1.7 kenneth/Auto-Trader:v0.1.6 kenneth/Auto-Trader:v0.1.5 kenneth/Auto-Trader:v0.1.4
...
compare: kenneth/Auto-Trader:73a21b1560bfcc90afb3ee8f3a7e200ba38a5cf6
kenneth/Auto-Trader:main
kenneth/Auto-Trader:v0.1.16 kenneth/Auto-Trader:v0.1.15 kenneth/Auto-Trader:v0.1.14 kenneth/Auto-Trader:v0.1.13 kenneth/Auto-Trader:v0.1.12 kenneth/Auto-Trader:v0.1.11 kenneth/Auto-Trader:v0.1.10 kenneth/Auto-Trader:v0.1.9 kenneth/Auto-Trader:v0.1.8 kenneth/Auto-Trader:v0.1.7 kenneth/Auto-Trader:v0.1.6 kenneth/Auto-Trader:v0.1.5 kenneth/Auto-Trader:v0.1.4
965 Commits
a8144e9acf ... 73a21b1560
Author SHA1 Message Date
Haozhe Wu 73a21b1560 docs: add the 2026-08-06 news entry to all five READMEs
CI / test (push) Waiting to run
Details
CI / Windows 11 ARM64 / Python 3.14 background lifecycle (push) Waiting to run
Details
CI / Windows Server 2025 x64 / Python 3.14 background lifecycle (push) Waiting to run
Details
Wiki / validate (push) Waiting to run
Details
Deploy Wiki / Deploy to Cloudflare Pages (push) Canceled after 0s
Details
2026-08-06 17:24:18 +08:00
Haozhe Wu ab82e4fb19 fix: three defects found by testing the fixes 
Option leg types accept the spellings real configs carry -- case, C/P, the
plural, and 认购/认沽, which is the official SSE/SZSE wording an A-share
config will use. A genuine typo still raises: defaulting an unrecognised type
is what priced a call leg on entry and settled it as a put.

Comparable-company exclusion is now two-sided. A net-cash-rich peer can carry
a negative enterprise value with healthy EBITDA, and -3.2x is not a cheap
company but one the multiple cannot describe. The old rule guarded the
denominator only and documented the gap in a docstring; a guarantee a reader
has to read the source to discover is not one.

Tests no longer leak process environment into each other. The settings write
path applies a written .env to the running process, which is correct in
production and meant a settings test writing a placeholder token leaked it
into every test that followed -- four live-data tests failed with "token is
wrong" while passing in isolation, and monkeypatch could not undo it because
the test never set it through monkeypatch. conftest now snapshots and restores
os.environ, closing the class rather than the instance.

The Tushare loader also gains backoff on quota rejections only; ordinary
failures still fail on the first attempt rather than stalling for a minute to
return the same error.
 2026-08-06 17:06:00 +08:00
Haozhe Wu a7d180fe61 feat(governance): connect the model-risk primitives to the running system 
The governance pieces existed as tested libraries that nothing called, which
is the same amount of governance as none: a Session.owner that is always None
carries no more information than a missing field.

  Principal      Threaded from require_auth through the session route into
                 SessionService, and persisted. It records HOW a request was
                 authorised and refuses to overstate it: a shared bearer key
                 and loopback trust both authorise without identifying, so
                 attributable is False and is recomputed on load rather than
                 read from storage. "Owner unknown" (None) and "authorised but
                 unattributable" stay distinct states.
  ModelRecord    Reachable through sdm_register/sdm_status. Supplying any
                 governance field makes intended_use and limitations
                 mandatory, because a registration missing either cannot be
                 reviewed. Four-eyes conflicts are reported, not blocked --
                 that call belongs to policy, not to a tool.
  Run manifest   Written beside each run's trace: prompt hash, tool registry,
                 package versions. Hashes only, never prompt text, and it
                 states that mid-run load_skill content is outside its
                 coverage rather than implying otherwise.
  Chained ledger On by default, written alongside the classic audit.jsonl so
                 every existing reader is unaffected. The classic record lands
                 and is fsynced first; a chain failure is logged and swallowed,
                 because a detectable gap beats an undetectable half-executed
                 trade.

Multiple-testing control now feeds the alpha bench under a new key, leaving
every existing key untouched.
 2026-08-06 17:05:48 +08:00
Haozhe Wu 11f97f9dfe feat(quantlib): add the analytics the finance-math layer was missing 
Nine new modules, each with a tested implementation rather than a formula
in a markdown code block:

  var_backtest     Kupiec POF, Christoffersen independence + conditional
                   coverage, Basel traffic light. The zone cut reproduces the
                   1996 250-day table exactly.
  eventstudy       Estimation-window market model, AR/CAR/CAAR, Patell and
                   BMP. BMP is reported alongside Patell because events raise
                   volatility and Patell assumes they do not.
  factormodel      Cross-sectional style exposures (cap-weighted mean,
                   equal-weighted dispersion), factor returns, style drift.
                   Public definitions only -- this is not Barra and says so.
  multipletesting  Deflated Sharpe, CSCV/PBO, BH-FDR. Wired into the alpha
                   bench, where n_alphas_tested had been carried and only ever
                   displayed.
  crossvalidation  Purged + embargoed splits for overlapping labels, plus a
                   leakage detector so the off-by-one class of bug is
                   assertable rather than reviewable.
  valuation/       run_dcf (dual terminal value that cross-checks each method
                   against the other's implied multiple/growth), run_comps
                   (two-sided multiple exclusion), three-statement projection
                   with a hard balance assertion and an explicit revolver plug.
                   contracts.py holds the one shared rule: a missing input
                   makes a model not runnable and is never silently defaulted.
  governance/      Run manifest (prompt/skill/tool/version hashes) and a
                   hash-chained ledger whose rotation never deletes history.

Plus entities FX translation and panel ingest, private-markets PME and the
American waterfall with GP clawback, an ADV/participation cost model with
borrow, and a crypto L2 depth tool.

quantlib_call reaches all of it from every transport. The shell gate that
previously hid src/quantlib from the API and MCP paths is a fix for two
published advisories, so this routes around it instead of widening it: module
allowlist, __all__ only, file writers refused.
 2026-08-06 17:05:33 +08:00
Haozhe Wu c103708b15 fix(swarm): make preset deliverables match what can actually be computed 
Swarm workers get a strict tool whitelist, and several shipped presets ordered
deliverables their whitelisted tools could not produce -- so the agent filled
the gap from its own head and returned an institution-shaped report with
invented numbers in it. All thirty presets were re-read against what the tree
can now actually compute, and each claim resolved one of three ways.

Twenty-three claims were backed by a capability the node was simply never
granted; those got the tool. financial_rigor already implements a three-scenario
valuation and was not in fundamental_research_team's list. portfolio_review_board
ordered VaR, CVaR, concentration and days-to-liquidate from a node holding no
data tool at all, and its execution analyst was asked for implementation
shortfall with nothing to compute it from.

Eight are backed by the new quantlib modules and now name them.

Seven have no backing and say so, rather than being deleted. There is no DCF or
WACC engine, so fundamental_research_team now orders an assumption set with
every input tagged sourced or assumed and forbids quoting a valuation; the same
applies to event_driven_task_force and investment_committee. There is no Barra
model, so fund_selection_panel must label its style profile a proxy and define
the factors. There is no crypto options source of any kind -- the only chain
tool is Yahoo-backed US-listed equity -- so crypto_research_lab no longer asks
for a 25-delta risk reversal.

test_preset_claim_backing.py holds the mechanical half: every snake_case tool
identifier named in a prompt must be in that node's tools list, and every skills
entry must have a SKILL.md on disk. Matching bare English words instead of
snake_case identifiers is what made an earlier attempt at this gate almost
entirely false positives -- sentiment, pattern, backtest and remember are all
tool names and ordinary words.

Note for whoever reads the audit that prompted this: its headline example is
false. It reported that derivatives_strategy_desk grants get_options_chain to no
agent; the pre-edit file grants it to all three, at lines 55, 120 and 181.
 2026-08-06 01:57:23 +08:00
Haozhe Wu b8276f2abf feat(entities): add the entity and irregular cash-flow spine 
The instrument model was a daily OHLCV bar, enforced at three independent
chokepoints: runner.py caps the interval, local_loader returns None for any
file lacking {open,high,low,close} and discards every other column, and metrics
annualises off a hardcoded per-source trading-day table. The bring-your-own-data
escape hatch was therefore closed -- a nav, a called_capital or a coupon column
could not enter the system at all, which is the single root cause of the
critical gaps on the fixed-income, private-markets and asset-management desks.

src/entities is a deliberately parallel path. It does not widen BaseEngine, does
not relax the OHLC gate and does not touch the runner: if a nav column were
allowed to flow into a bar engine it would be priced as a close, which is the
class of silent wrong answer this whole effort exists to remove. Frozen
dataclasses for instruments and entities, an ordered immutable CashFlowSeries
with a stated sign convention, and a delimited-text ingest that raises on a
missing column rather than returning None. Mixed currencies are refused outright
-- the composite engine silently summing five of them into one equity curve was
a shipped defect, and it is not repeated here.

On that substrate: XIRR (with an explicit no-solution result rather than a
garbage root), MOIC/DPI/RVPI/TVPI, a European distribution waterfall, and both
true-daily TWR and Modified Dietz alongside money-weighted return, so an
external client contribution no longer makes the reported return meaningless.
cashflow_analytics_tool exposes the family to the agent, read-only.
 2026-08-06 01:57:09 +08:00
Haozhe Wu 62b8d9d7ad feat(quantlib): promote the finance-math layer out of markdown 
Until now the formulas were markdown inside skills, and the model retyped them
into throwaway Python on every run. That delivery model is not reproducible
(different code each run), not reviewable (no tests, no version) and not
auditable (no artifact), which is why an audit of the twelve institutional
desks found "market-data plumbing, a backtest harness and an agent, but almost
no financial-mathematics layer" -- a fifteen-term grep for wacc|dcf|cva|xirr
and friends across every .py returned two hits, both acronym string literals.

src/quantlib now holds one tested implementation of each: Black-Scholes price,
Greeks and implied-volatility inversion; bond pricing, YTM, duration/convexity/
DV01 and Nelson-Siegel/Svensson curve fitting; Altman Z and Merton/KMV;
stationarity, cointegration, Granger, GARCH and bootstrap; VaR/CVaR, drawdown,
Monte Carlo and EVT tail fitting; Brinson-Fachler attribution; impact models.
The skills now import and call these instead of carrying the source.

Two of them were not promotions:

- Black-Scholes existed three times over -- in the options engine, in
  options_pricing_tool, and in markdown. The tool's copy was the worst of the
  three: it guarded T and sigma but never S or K, so a zero strike raised
  ZeroDivisionError and a zero spot returned gamma=nan. All three now share
  one implementation and both defects are gone.
- Brinson-Fachler had no code anywhere; performance-attribution/SKILL.md
  carried prose algebra and zero python blocks. It is written here, with
  allocation + selection + interaction asserted to sum to the active return
  exactly rather than to two decimal places.

statsmodels and arch were being imported while declared in none of the three
dependency files, so eight of the eleven econometric functions would raise
ImportError on a clean install. They are now the `stats` extra, and the lazy
import names it.

A folded-in review pass fixed what the promotion itself introduced or exposed:
a mixed-case leg type that priced as a call and settled as a put; implied
volatility returning an arbitrary number where the price is flat in sigma
(now refused, with the threshold scaled to the requested tolerance); put-call
delta parity breaking at exactly the money; a negative face silently producing
a positive DV01; a hand-built CurveFit reporting a perfect fit it never
achieved; impact models returning a negative fill price for a very large sell;
and altman_z_score made keyword-only, because its six interchangeable floats
no longer sit in the same order as the markdown they replaced.
 2026-08-06 01:56:57 +08:00
Haozhe Wu c33606a140 docs: close the four section-level gaps between the five READMEs 
Until now the five files documented different products. English was missing
the MetaTrader 5 connector entirely while its own tables advertised MT5; the
four translations were missing the Longbridge setup while their data-source
tables advertised longbridge; and the whole MCP-client half of the product —
how to point the built-in agent at your own MCP servers, the IBKR read-only
OAuth probe, the local TWS path, TAP credential isolation, the config
reference and the v1 limits — existed only in English, as did the SWARM
external-tool allowlist that explains why caller-supplied variables cannot
inject an MCP URL.

All five files now carry the same fifty-two headings at the same levels in the
same order.

The new English MT5 section is translated from the Chinese one, so it was
checked against the connector rather than trusted: the four profiles, the
per-call demo/live re-verification, the guards, lot sizing, the hedging-account
close path and the loader's config sharing all match the code. Two claims did
not survive that check and are corrected in every locale:

* the guards are per-order size limits, not position limits — nothing caps
  accumulated exposure on a paper profile, and "position guards" invited the
  opposite reading. Chinese had said 仓位护栏 since the section was written;
  Japanese, Korean and Arabic had it right.
* `close_position` was offered as a way to close a hedge, but it has no
  user-facing path: `service.py` dispatches only place and cancel, and the
  agent surface is `trading_place_order` / `trading_cancel_order`. The
  ticket-pinned `trading_cancel_order` route, which does work, is what remains.

Also backported the guard's fail-closed behaviour on an unpriceable notional
to the four translations, which had dropped it.
 2026-08-05 17:19:27 +08:00
Haozhe Wu a768b67452 test: assert the README claims that actually drifted 
Five of the six defects fixed alongside this were the same shape: a sentence
that was true when written and silently stopped being true. Counting them by
hand is what let them accumulate, so each now has a test that derives the
answer from the code.

* the slash table against `SLASH_COMMANDS`, scoped to the TUI table because
  the IM-channels section further down has its own unrelated one;
* the skill count in every prose position, not just the badge — dated news
  entries and credit lines are excluded as frozen history, and so is the MCP
  tool list, which contains `list_skills` beside a different number;
* the skill category table against the `category:` frontmatter, summing to
  the loader's count;
* the brokers with no paper account against `list_profiles()`, which is the
  check that would have caught Robinhood.

Verified by breaking each one: removing Robinhood from the English paragraph
fails the broker test and passes again when restored.
 2026-08-05 16:32:35 +08:00
Haozhe Wu 99d06ddf79 docs: bring all five READMEs back in line with what the code does 
The slash-command table was the worst of it. It documented seven commands
the router does not resolve — `/skills`, `/list`, `/code`, `/trace`,
`/continue`, `/sessions`, `/settings` — which survive only in the legacy REPL
reached when prompt_toolkit fails to import, and it omitted twenty-one real
ones including the `/halt` kill switch. All five tables are now generated from
`SLASH_COMMANDS`: same twenty-seven commands, same order, descriptions taken
from the code.

Corrections, each checked against the code rather than against another
document:

* "18 read-only data tools" is 22, and the sentence listed only 17 labels,
  missing `get_stock_profile` and `iwencai_search` on top of the four that
  landed yesterday.
* The skill count was 88 in six places per file and the Analysis category row
  said 22; the loader reports 89 and the frontmatter tally gives analysis 23.
* "Each connector does read + paper-account order placement" was wrong for
  three of twelve. IBKR is read-only, Trading 212 refuses order placement
  outright, and Robinhood has exactly one profile and it is live — which
  matters most of the three, since a reader who believes Robinhood has a
  paper mode places a real order while trying to rehearse.
* The API endpoint table was missing the three playbook routes the section
  goes on to curl forty lines later.

New documentation for what shipped yesterday: the five scheduled-research
templates with their CLI, REST and `/playbook` entry points; a capability-table
row for the institutional data tools; `vibe-trading playbook` in the CLI
reference.

Locale parity, all pre-existing: two Korean capability rows and the Japanese
broker-connector block were still in English; `provider doctor` was documented
only in English; Japanese used 定時リサーチ against the file's own 定期リサーチ
everywhere else. The Chinese and Japanese renderings of `/memo`'s "variant
view" both said "a changing/variable perspective" rather than a view that
differs from consensus.

Three section-level gaps are deliberately left alone: "Using Longbridge
explicitly" (en/zh only), "MetaTrader 5" (missing from en), and the two
MCP-client sections (en only). They predate this work and come to roughly 340
lines of translation, which is not something to land unreviewed.
 2026-08-05 16:32:26 +08:00
Haozhe Wu eabb72366b fix(config): route the new tools' env reads through the config layer 
CI has been red since yesterday's merge landed: `tools/ci_grep_gates.sh`
gate (e) forbids raw `os.getenv` outside `agent/src/config/`, and six reads
in the new code bypassed it. The gate is a shell script the pytest suite
never invokes, so a full local run stayed green the whole time.

Seven variables move into the schema — the three 13F scan bounds, the SEC
fails-to-deliver overrides, the OpenAlex mailto, the playbook directory, and
the slash-argument cap — and their readers go through `get_env_config()`.

Behaviour is preserved rather than tightened. `_EnvBase` already drops an
unparseable numeric env value so the field default applies, which is exactly
what the hand-rolled try/except did; the surviving `_positive` helper keeps
the separate guard against a parseable-but-nonsensical zero or negative
bound. No `gt=0` constraint is declared, since that would raise a
ValidationError at config load where the old code fell back to the default.

The OpenAlex test changes an env var mid-test and now resets the cached
singleton first — the same reset `settings_routes.py` performs after it
rewrites `.env`. Without it the second phase reads the value cached by the
first.
 2026-08-05 15:49:35 +08:00
Haozhe Wu 95a572331b docs: cut the 2026-08-05 news entry back to the length of its neighbours 
The entry ran 2,569 characters in English against roughly 1,250 for the two
below it, and worse in the translations. The convention is about three lines
per entry; this one narrated every design decision instead of saying what
shipped. It keeps the four tools, the one number worth quoting (342 positions
at 98.7% of net assets, against a quarterly top ten), and a single closing
clause for the templates, commands, lenses and agent core.
 2026-08-05 15:29:58 +08:00
Haozhe Wu 85de202786 docs: date the feature news to the day it reached main 
The institutional-holdings / ETF-look-through / prediction-market / research
entry was written on 2026-08-04 but merged and pushed on 2026-08-05. The news
section records when something became available to readers, so it takes the
landing date. This also separates it from the correctness-pass entry, which
really did ship on 08-04 and keeps its date.
 2026-08-05 15:18:12 +08:00
Haozhe Wu 75395771ed docs: correct the MCP tool and skill counts, and hold them there with a test 
Every README stated an MCP tool count in three places, and all three had
drifted: the prose said 54, the enumerated list said 55 in English and 54
elsewhere, and the repository tree already said 59. The real number is 59.
The four non-English lists were also missing `analyze_options_payoff`, which
landed on 2026-08-01, on top of the four data tools from yesterday. The skills
badge said 88 across all five files and has said so since `investor-lenses`
shipped.

The counts a reader uses to size up the project are exactly the ones nobody
notices going stale, so they now have a test. It reads the numbers back out of
all five READMEs and compares them with what the code ships — MCP tools by
name and by count, bundled skills, swarm presets, alphas, engines.

The checks anchor on structure rather than wording, since four of the five
files are translated: the tool list is the line with the most backticked
identifiers, the badges are the ones inside a <summary>, and the MCP paragraph
is the non-news line mentioning stdio. Two anchors that looked obvious are
wrong and are documented in the module: a 2026-04-14 news entry also mentions
stdio, and the loose "Plus 20+ specialist presets" badge wraps onto its own
line in English but not in the other four, which puts bare <sub> counting out
of step across locales.
 2026-08-05 15:18:05 +08:00
Haozhe Wu 8b56e6ed94 fix(swarm): attribute committee verdicts to the framework, not the person 
The four worker agents keep their first-person prompts — the role-play is
what makes the four analyses genuinely disagree, and removing it collapses
them into one voice. What changes is only the chair, whose report is the
artifact the user reads and saves: it now labels each position by its lens
rather than asserting that a named living investor holds an opinion about a
specific security, and it must state once that these are reconstructions of
publicly documented methodologies rather than the individuals' views.

This also removes a contradiction introduced by the investor-lenses skill,
which lists "Buffett would buy this" as an anti-pattern while this preset
was emitting exactly that.
 2026-08-05 15:03:46 +08:00
Haozhe Wu c95e0be410 feat(mcp): expose the four new data tools 
Institutional holdings, ETF look-through, prediction markets and
research papers were reachable from the agent and the CLI but not from
MCP, so OpenClaw and Claude Desktop could not use them. MCP goes from
55 tools to 59.

Schemas are read off the tool classes rather than restated here, so the
MCP surface cannot drift from the agent surface as the tools change.

A regression test pins the order-placing tools as absent from MCP.
 2026-08-04 19:43:29 +08:00
Haozhe Wu 9eb681f8ff feat(scheduled): surface research playbooks on CLI, REST and slash 
The five templates shipped loadable from Python only, so nothing in the
product pointed at them. Three entry points now do.

REST adds list / show / create-from-template beside the existing
scheduled-runs CRUD and behind the same auth dependency; there is no
path that lists or creates a job without it. CLI adds a playbook
subcommand, and the REPL gains /playbook for list, run-now and
schedule.

A job created from a template stores exactly what render() produced,
so the prompt that runs is the template that was reviewed. An
undeclared variable is rejected rather than silently ignored, and a
schedule still goes through the existing validator.
 2026-08-04 19:38:39 +08:00
Haozhe Wu 4cfc1f87c0 docs: add the 2026-08-04 feature news to all five READMEs 
Covers the four new read-only data tools, the five research playbooks,
the six institutional commands, the investor lenses skill, and the
agent-core output principles plus progressive skill loading.

Each locale keeps three entries above the fold, so the 2026-08-02 row
moves into the folded section.
 2026-08-04 19:38:39 +08:00
Haozhe Wu 0db64ec982 feat(agent): output discipline and progressive skill loading 
Two changes to the agent core.

Output principles are stated in the system prompt and declared not
overridable by a single session: every number must trace to the tool
that produced it, data carries its as-of time, anything a tool did not
return is reported as missing rather than recalled, analysis is not
investment advice, and a request to break one of these is refused
explicitly instead of quietly followed.

Skill loading becomes progressive. Tool results are capped at 10k
chars, so a long skill arrived truncated and effectively broken. A
skill over the cap now returns its structure first with per-section
summaries and expands a section on request. Short skills are returned
whole exactly as before, and the cap itself is unchanged so no other
tool result grows.
 2026-08-04 19:36:35 +08:00
Haozhe Wu 382fadc0ca feat(cli): add institutional research slash commands 
Adds /comps, /dcf, /attrib, /memo, /earnings and /screen. These
workflows were previously reachable only as prose inside swarm presets,
where a user could not discover them and the agent got no procedure.

Each command carries an ordered execution skeleton naming the inputs,
the computation and the artifact per step, plus a worked example whose
arithmetic ties out: the Brinson decomposition sums exactly to active
return and the earnings bridge sums exactly to the EPS delta. A user
and the agent can both reproduce them, which is what makes the
skeleton checkable rather than decorative.

Specs live in a data-only playbooks module that imports nothing from
cli, so slash_router reads them at its own import time without a
cycle, and help plus tab-completion see the commands regardless of
import order. Registration is name-checked and idempotent: an existing
command is never overwritten and a repeated call never duplicates.
 2026-08-04 19:36:35 +08:00
Haozhe Wu fecddcf9b4 feat(tools): add research paper search and signal extraction 
Searches arXiv and OpenAlex and reads a paper's structured record, then
extracts what a factor implementation would need: the proposed signal,
the inputs it requires, the tested market and window, and the checks
that would falsify it. Data requirements come out as capabilities
rather than tool names, so the result routes to whatever is configured.

Extraction is anchored to the source. A field that is not literally
present is left empty and marked not stated in source; nothing is
inferred, completed, or filled from general knowledge, and each claim
can be traced back to the sentence it came from.

A paper's own reported performance is labelled as the paper's claim.
It is never presented as a backtest result of ours.
 2026-08-04 19:36:35 +08:00
Haozhe Wu 7018083223 feat(skills): add investor lenses as stackable overlays 
Named-investor reasoning frameworks existed only inside swarm presets,
bound to a committee run. They could not be applied to an ordinary
analysis. This lifts them out as an independent skill so the same
evidence can be re-read through a chosen framework.

Each lens is an operating procedure, not a biography: the signals it
weighs first, the conditions that disqualify a candidate outright, and
where the framework is known to fail. Lenses name no tool, so they stay
usable regardless of which data sources are configured.

Every lens states that it is an analytical frame and not investment
advice, and that its verdict is conditional on that frame.
 2026-08-04 19:36:35 +08:00
Haozhe Wu e50f960a0f feat(scheduled): ship five research playbooks 
The scheduling runtime has been end-to-end since #278 but shipped with
nothing to run, so the feature was invisible on arrival. This adds five
ready-to-schedule templates: premarket brief, earnings-season tracker,
portfolio checkup, A-share money flow, and institutional-holdings diff.

Templates state their data needs in natural language instead of naming
tools. Coverage can then grow without editing a single template, and a
template stays valid when a tool is renamed or replaced.

Each one mandates the same behaviour on a gap: name the missing input
and continue on retrieved evidence only, never fill it from memory.

Loadable from Python; not yet wired to CLI or REST.
 2026-08-04 19:36:35 +08:00
Haozhe Wu 1817cd6baa feat(tools): add prediction market data 
Event-contract data as an alternative input for event-driven research:
search events, read an event and its markets, and pull a market's
price history. Public endpoints, no key required.

Contract prices are converted to implied probability and the unit is
labelled, so a downstream reader cannot take 0.63 for a dollar price.
Responses carry the event's status and settlement date so a resolved
market is never read as a live one.

Read-only by construction. There is no order path here and none is
reachable from it: research and backtest stay structurally barred from
live endpoints.
 2026-08-04 19:36:35 +08:00
Haozhe Wu bbff62b345 feat(tools): add cross-market ETF holdings look-through 
Resolves an ETF and returns its constituent book: SEC N-PORT for US
funds (full portfolio) and akshare for A-share funds. Market is
detected from the symbol, so .SH/.SZ and US tickers go to different
sources through one interface.

A-share coverage is top-10 holdings only. That is a hard limit of the
upstream disclosure, not a bug, and the response says so explicitly
through coverage, missing_fields and pct_of_net_assets_disclosed
rather than presenting a partial book as complete.

Every response stamps the disclosure as-of date. ETF holdings are
published on a lag; without the date a caller can mistake a quarter-old
book for the live portfolio.
 2026-08-04 19:36:35 +08:00
Haozhe Wu 375b833ee3 feat(tools): add SEC 13F institutional holdings 
Reads 13F-HR filings straight from EDGAR full-text search and the
INFORMATION TABLE XML, with three modes behind one tool: a manager's
holdings, who holds a given ticker, and the largest managers by
reported value. Quarter-over-quarter position diffs (new/added/
trimmed/exited) are available so the series can feed factor work.

Cover-page totals carry an explicit value_units field and its basis:
filings before the 2023 rule change report in thousands, and reading
one as dollars understates a portfolio 1000x, which silently
mis-orders any ranking by size. When a cover-page total lands below
the USD 100M level that compels filing at all, the filing's own
information table decides the unit rather than the filing-date prior.

Free public data, no key. SEC requires a contact User-Agent and
rate-limits, so requests are throttled.
 2026-08-04 19:36:35 +08:00
Haozhe Wu 3a752d5a8e docs: add the 2026-08-04 correctness-pass news to all five READMEs 
Covers the SEC period-span fix, A-share corporate-action adjustment,
point-in-time CSI300 membership, the mixed-currency composite guard, the
option-leg volatility fix, whole-record paging of oversized tool results,
and the new tracking-error and benchmark-beta metrics. 2026-08-01 moves into
the collapsed Earlier-news block so three entries stay visible.
 2026-08-04 14:59:08 +08:00
Haozhe Wu 9a806223bd fix(engines): stop the silent-wrong-answer paths in composite, options, metrics 
The composite engine holds one shared capital pool and summed CNY, USD, HKD,
INR and KRW positions into a single equity curve with no warning and no FX
translation, so every metric derived from it was meaningless. It now resolves a
settlement currency per symbol and refuses a mixed set at run_backtest — where
the equity curve is actually produced, so cross-market rule routing stays
legal. Global futures resolve to USD because that engine's margin, commission
and multiplier tables are USD throughout and carry no non-USD product; .EUREX
is mapped separately.

The options engine opened legs at smile-adjusted vol and marked them at flat
at-the-money vol, booking a profit the instant a position existed: +16.7% of
premium on a 30-day 10%-OTM call at skew=-0.15, +93.0% at 20% OTM, feeding
straight into Sharpe, Calmar and drawdown. The Greeks and the American
early-exercise continuation value used the same wrong vol. A single leg_iv()
now serves all four call sites, with a day-zero-P&L regression test.

calc_metrics computed the annualised active-return standard deviation to form
the information ratio and discarded it, so the return dict carried no tracking
error and no benchmark beta — the two numbers a benchmark-relative mandate is
written around. Both are now returned. get_stock_profile also parsed and threw
away epsTrend, epsRevisions, enterpriseToEbitda and enterpriseToRevenue.

Two presets shipped promises their workers could not keep: three skills that
were never bundled (load_skill returns a soft error, so the worker ran without
its methodology and said nothing), and a derivatives desk ordered to report
25-delta risk reversals with get_options_chain granted to no agent. The skills
references are gone, the chain tool is granted, and a CI gate now asserts every
referenced skill exists and every tool named in a prompt is granted to that node.
 2026-08-04 14:52:14 +08:00
Haozhe Wu d52199ff6d fix(loaders): adjust A-share prices and use point-in-time index membership 
Tushare's daily and fund_daily endpoints return raw prices, so a close-to-close
return taken across an ex-date spans the mechanical gap of a split or bonus
issue rather than the stock's actual move. Checked against Tushare's own
pct_chg over 2020-2024: 300750.SZ on 2023-04-26 reads -41.82% raw against a
true +5.40%, 300124.SZ -34.26% against -1.01%, 601012.SH -24.21% against
+6.46%. Every sampled name was hit and the error is always negative, so this
was a systematic contaminant of returns rather than noise.

The akshare and tencent loaders already request qfq; only the Tushare path did
not, and it feeds both the factor bench and A-share backtests. Adds
backtest/loaders/cn_adjust.py as the single implementation and routes both
through it. After the fix, adjusted returns match pct_chg to within 1e-5 and
the amount-derived VWAP keeps its relationship to close to 2e-16. A symbol with
no usable factors is dropped with a warning rather than served raw; indices
need no adjustment and Tushare publishes no HK factor series.

The CSI300 bench also took its roster from the newest index_weight snapshot and
applied it across the whole window, so a name was present only because it
survived to the end and every IC was measured on a hindsight-selected set. The
panel now carries every name that was ever a member and masks each date's
cross-section to the roster in force then.
 2026-08-04 14:52:13 +08:00
Haozhe Wu fde1aa58f4 fix(sec): key reporting periods on their span, and page oversized results 
SEC companyfacts files a true quarter and the year-to-date frame that
contains it under the same end date, fy, fp, form and accn. Keying a period
on anything narrower than the (start, end) span therefore lets one silently
overwrite the other — 36 of 81 keys collide on live AAPL revenue.

The damage ran both ways: period="annual" returned 64,698,000,000 for FY2020
against a filed 274,515,000,000, and period="quarter" served the full-year
figure in every fiscal-Q4 slot. get_sec_filings dropped `start` altogether and
emitted a series mixing quarters, year-to-date frames and full years with no
way to tell them apart.

Adds backtest/loaders/sec_frames.py as the single definition of the span rule;
fundamentals_loader now references it rather than its own literals. Fiscal Q4
is derived as FY - (Q1+Q2+Q3) and labelled DERIVED so a computed figure is
never mistaken for a filed one.

cik_for strips a .US market suffix and falls back to the SEC table's dashed
share-class form, so get_fundamentals("AAPL.US") — its own documented example —
no longer answers ok:true with an all-null panel. A batch that resolves nothing
now fails closed, and the advertised source list matches what the loader takes.

Tool results were also cut at 10,000 characters with no marker, mid-JSON:
get_financial_statements(income, quarter) serialized to 28,498 characters for
40 periods and delivered about 12, while a parser-side cap of 40 put the older
quarters out of reach of any request. Results are now paged by whole record
with a paging block carrying the total and the resume offset, and the generic
cut appends a notice. TOOL_RESULT_LIMIT moves to src/config/limits.py so the
agent loop, the swarm worker and the tools share one definition instead of a
duplicated literal.
 2026-08-04 14:51:51 +08:00
Haozhe Wu 652917e74e fix(providers): isolate relay HTTP headers 2026-08-04 03:13:31 +08:00
Haozhe Wu 61c3e5a785 chore: ignore repo-root runtime artifacts 
get_runtime_root() resolves to the repository directory when the CLI or
server is started from there, so sessions.db, sessions/, runs/, live/ and
friends land at the root — where the existing agent/-anchored ignore rules
do not reach. sessions.db holds real session transcripts and live/ holds
broker audit and kill-switch state, so a single `git add -A` would publish
them.
 2026-08-04 02:29:39 +08:00
Haozhe Wu 539b9ab69c fix(tools): name the allowed roots when a path is refused 
A run_dir outside the sandbox was refused with only the name of an
environment variable, so the user had no way to see where the boundary
actually was. Under an MCP client it was a dead end in a second way: the
client spawns the server itself, so exporting the variable in a shell
never reaches it, and none of the READMEs documented an env block for the
MCP config.

Rejections from safe_run_dir, safe_run_id, and the import-root helper now
list the roots that are allowed and say where the variable belongs under
an MCP client. All five READMEs document the env block in the MCP Plugin
section, with VIBE_TRADING_ALLOWED_RUN_ROOTS as the worked example.

Closes #963
 2026-08-04 02:18:55 +08:00
Haozhe Wu 1e3600b047 docs: update multilingual README news 2026-08-03 23:17:02 +08:00
Haozhe Wu 011faeea5a fix(agent): let screening runs reach a grounded answer 
A stock-screening request resolves to many candidates by design, but the
grounding ledger treated that shortlist as a stalled resolution: load_skill was
refused with identity_required for the rest of the run, and the ambiguous record
blocked every final answer even after a candidate was locked. The price checker
compounded it by reading a ticker's digits, a localized date, a holding period,
and a position cost as price claims, so a correct draft was rejected on 543, 3,
100, and 820 (closes #955).

- Workflow selection is blocked only while resolution is in flight or
  contradicted; ambiguous is an answer, not a stall.
- A shortlist is retired once a later resolution locks one of its candidates.
- Symbol, calendar, aggregate-amount, and unit-bearing digits are masked before
  price extraction, and full-width brackets delimit clauses so a position cost
  is no longer read as a quote. ASCII parentheses stay intact for derivations.

The contradiction check is unchanged: a quote outside the observed OHLC range is
still refused.
 2026-08-03 22:06:25 +08:00
Pham Van Ngoan 516ae67ef7 feat(scheduled-research): add timezone-aware local schedules (#954) 
Evaluate cron schedules in typed IANA zones with documented DST gap/fold semantics, preserve legacy UTC and interval behavior, and add Scheduled Research Web/i18n/docs coverage.

Closes #953.
 2026-08-03 20:46:52 +08:00
Santh fed8468188 fix(memory): keep find_relevant within max_results when expanding links (#957) 
Enforce max_results before appending semantic-link expansion results and cover the boundary.
 2026-08-03 20:46:31 +08:00
Santh 620a85826f fix(memory): match exact title anchors when updating MEMORY.md (#956) 
Require an exact title-anchor prefix when updating MEMORY.md and cover description/title collisions.
 2026-08-03 20:46:11 +08:00
Haozhe Wu bfa0133c0d docs: update multilingual README news 2026-08-02 17:06:49 +08:00
Gao Fengkai 8d84c11ab6 feat(llm): add model discovery and runtime identity UX (#924) 
Adds provider/model discovery, immutable runtime identity metadata, five-locale model controls, and session-scoped reply details across Agent, API, and Web UI.
 2026-08-02 16:54:02 +08:00
dependabot[bot] 4da304fe0c build(deps): bump the pip-minor-patch group with 9 updates (#949) 
Bumps the pip-minor-patch group:
- langchain-core 1.5.1 → 1.5.2
- langgraph 1.2.9 → 1.2.10
- akshare 1.18.80 → 1.18.81
- ccxt 4.5.68 → 4.5.70
- fastapi 0.140.7 → 0.141.1
- uvicorn 0.51.0 → 0.52.0
- annotated-doc 0.0.4 → 0.0.5
- langsmith 0.10.10 → 0.10.12
- openai 2.49.0 → 2.50.0

Validated on synthetic merge 78188181: exact-version import checks and 330 focused tests passed. GitHub Linux CI passed the repository hash-lock check.

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2026-08-02 14:02:41 +08:00
dependabot[bot] b7218bb503 build(deps-dev): bump the npm-minor-patch group in /frontend with 2 updates (#948) 
Bumps the frontend npm-minor-patch group:
- jsdom 30.0.0 → 30.0.1
- postcss 8.5.23 → 8.5.25

Validated on synthetic merge a09b902a: npm ci, production build, and 373 tests passed.

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2026-08-02 13:54:17 +08:00
Haozhe Wu bec189f2ee docs(readme): add 2026-08-01 news across all 5 READMEs 2026-08-01 23:09:12 +08:00
Haozhe Wu 39579413cc Merge pull request #946 from HKUDS/codex/pr-883-options-payoff-salvage 
feat(options): add analytic payoff and scenario tool
 2026-08-01 22:01:49 +08:00
Haozhe WuandYufeng He 73967e476c feat(options): add analytic payoff and scenario tool 
Rebuild the useful core of PR #883 on clean history with analytic piecewise-linear extrema and breakevens, engine-aligned entry commission, Agent and MCP surfaces, localized Web labels, and regression coverage.

Co-authored-by: Yufeng He <40085740+he-yufeng@users.noreply.github.com>
 2026-08-01 21:39:31 +08:00
Haozhe Wu 22bfd087a7 fix(mcp): harden result serialization errors 2026-08-01 18:13:17 +08:00
Haozhe Wu f7ab8a0111 fix(api): migrate lifecycle hooks to lifespan 2026-08-01 17:52:35 +08:00
Haozhe Wu 757e8ba9f7 fix(scheduler): retry transient scheduled research failures 2026-08-01 17:52:31 +08:00
Haozhe Wu 93edbd7028 fix(agent): gate market answers on verified evidence 
Resolve and lock instrument identity before market-sensitive tools, record structured evidence, reject contradictory final numbers, and prepare the 0.1.13 Robinhood serialization patch release.
 2026-08-01 16:43:00 +08:00
卫斯李 6ad0e8d259 feat(tools): add sentiment analysis tool (#939) 
Signed-off-by: Robin1987China <41602358+Robin1987China@users.noreply.github.com>
 2026-08-01 15:51:34 +08:00
Hongi Nagaputra ef13b3169f feat(backtest): persist USD-M fidelity evidence (#936) 
Signed-off-by: zygora <122336305+honginp@users.noreply.github.com>
 2026-08-01 15:29:23 +08:00
Haozhe Wu e0b236caae docs(readme): add 2026-07-31 news across all 5 READMEs 2026-07-31 21:37:35 +08:00
Xiaotian JinandClaude Fable 5 c7d2ebaab8 fix: store session and run history under the user runtime root, not the code (#925) 
* fix: store session and run history under the user runtime root, not the code (#904)

Sessions, runs, swarm runs, and uploads were resolved relative to the
installed code, so a pip install wrote user history into site-packages
(lost on upgrade) and dev checkouts partitioned history per clone.

- src/config/paths.py: VIBE_TRADING_HOME env override for
  get_runtime_root() (explicit config_path still wins), plus
  get_sessions_dir / get_runs_dir / get_swarm_runs_dir / get_uploads_dir
  as the single source of truth.
- Redirect every code-relative definition site: cli/_legacy.py,
  src/agent/loop.py, src/api/helpers.py, src/api/uploads_routes.py,
  src/swarm/store.py (swarm_runs_root), src/tools/swarm_tool.py, and
  the TraceWriter.find_trace_dir defaults. .swarm/runs becomes
  swarm/runs under the already-hidden runtime root.
- src/config/migrate.py: idempotent one-time move of legacy
  <install>/{sessions,runs,.swarm/runs,uploads} into the runtime root,
  wired into the CLI entry, api_server startup, and vibe-trading-mcp;
  when both sides hold data the new location wins and the legacy dir is
  left in place with a warning, never silently merged.
- path_utils: browser-upload handles (uploads/<name>) resolve to the
  user-level uploads dir, and the file/write/run sandbox allow-lists
  follow VIBE_TRADING_HOME.
- Welcome banner Workspace row now reports the runtime root, so the UI
  shows which store is active.

Fixes #904.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix: close the gaps a local multi-agent review of #925 found

- Redirect the eighth definition site the first pass missed:
  src/api/swarm_routes.py built its SwarmStore from the code-relative
  .swarm/runs, so API-started swarms still wrote into site-packages,
  split from CLI/MCP history, and landed outside the run-root sandbox
  allow-list (P03-A regression). It now shares swarm_runs_root().
- Rework migration to child-level merges: autopilot runs and channel
  media already populate ~/.vibe-trading/{runs,uploads}, so the old
  whole-directory "both non-empty -> skip" rule silently stranded all
  legacy history for those users. Each session/run/upload entry now
  moves individually via a .migrating-* staging name plus atomic
  rename (interrupted moves are recovered, never surfaced half-copied),
  only same-named entries are skipped, a directory containing
  __init__.py is never touched (site-packages/runs may be someone
  else's distribution), one failing directory no longer blocks the
  rest, and the emptied legacy .swarm shell is removed.
- Route the sessions.db FTS index, the goal store default, and the CLI
  banner probe through get_runtime_root() so VIBE_TRADING_HOME no
  longer splits the index from the sessions it indexes.
- Reject UNC paths in VIBE_TRADING_HOME, matching the hardening on the
  other root-widening env vars, and keep legacy agent/.swarm/runs in
  the run-root allow-list like the other legacy roots.
- Correct the .env.example rationale (constants bind at process start,
  before .env is read), scope the README claim to what actually
  relocates, sync the zh/ja/ko/ar READMEs, and refresh stale docstrings.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
 2026-07-31 21:30:52 +08:00
Hongi Nagaputra 04739840a7 feat(backtest): execute USD-M funding and liquidation lifecycle (#903) 
* feat(backtest): add engine lifecycle hooks

Signed-off-by: zygora <122336305+honginp@users.noreply.github.com>

* feat(backtest): apply perpetual funding before fills

Signed-off-by: zygora <122336305+honginp@users.noreply.github.com>

* feat(backtest): execute perpetual liquidations

Signed-off-by: zygora <122336305+honginp@users.noreply.github.com>

---------

Signed-off-by: zygora <122336305+honginp@users.noreply.github.com>
 2026-07-31 21:17:44 +08:00
卫斯李 da37633fc2 feat(tools): add technical indicator computation tool (RSI, MACD, Bollinger, SMA, EMA) (#921) 
Signed-off-by: Robin1987China <41602358+Robin1987China@users.noreply.github.com>
 2026-07-31 21:17:32 +08:00
Santh 890ac578b9 fix(financial-statements): classify Yahoo .SS suffix as A-share in _classify_market (#935) 
Signed-off-by: santhreal <64453045+santhreal@users.noreply.github.com>
 2026-07-31 21:14:24 +08:00
Santh e701c089be fix(trade-journal): recognize slash-delimited crypto trading pairs in market inference (#934) 
Signed-off-by: santhreal <64453045+santhreal@users.noreply.github.com>
 2026-07-31 21:14:12 +08:00
Santh e23e1d2ff6 fix(financial-rigor): check math.isfinite in benford_check to prevent OverflowError on infinity (#932) 
Signed-off-by: santhreal <64453045+santhreal@users.noreply.github.com>
 2026-07-31 21:12:51 +08:00
Santh 0af65ff5b3 fix(trade-journal): coerce string datetime cells in pair_trades_fifo before computing timedelta (#931) 
Signed-off-by: santhreal <64453045+santhreal@users.noreply.github.com>
 2026-07-31 21:12:39 +08:00
Santh 2e21ad98ff fix(margin-trading): extract 6-digit code for prefix-style A-share symbols in _extract_code (#930) 
Signed-off-by: santhreal <64453045+santhreal@users.noreply.github.com>
 2026-07-31 21:12:27 +08:00
Santh 569a79ae69 fix(block-trades): support bare 6-digit A-share code resolution in _resolve_code (#929) 
Signed-off-by: santhreal <64453045+santhreal@users.noreply.github.com>
 2026-07-31 21:12:15 +08:00
Santh 50aa514311 fix(trade-journal): return default float when parsing non-finite string inputs in _to_float (#928) 
Signed-off-by: santhreal <64453045+santhreal@users.noreply.github.com>
 2026-07-31 21:12:03 +08:00
Santh cd0fd87b68 fix(metrics): guard calendar_days calculation when equity_curve has non-datetime index (#927) 
Signed-off-by: santhreal <64453045+santhreal@users.noreply.github.com>
 2026-07-31 21:11:43 +08:00
Santh 2a50c5563a fix(pattern-tool): use abs(avg) when evaluating double top peak difference (#926) 
Signed-off-by: santhreal <64453045+santhreal@users.noreply.github.com>
 2026-07-31 21:11:31 +08:00
Santh 6454236f0d fix(lockup-expiry): handle integer symbol code in get_lockup_expiry (#919) 2026-07-31 21:11:19 +08:00
Haozhe Wu 261f007c41 fix(packaging): cap requires-python below 3.14 
llvmlite publishes no cp314 wheel, so on Python 3.14 pip falls back to
building it from source and dies on a missing cmake with a 103-line
traceback. The dependency is not optional or obscure: smartmoneyconcepts
-> numba -> llvmlite, all in the base install.

The metadata said ">=3.11" with no upper bound, so pip happily attempted
the install and the user saw a compiler error instead of an unsupported
Python version. Reported in discussion #702 on macOS.

The 3.14 CI job is unaffected: it installs pytest/pydantic/pyyaml/
python-dotenv and runs two test files over PYTHONPATH, never the package,
so requires-python is not evaluated there.

Also declares 3.13, which is what the development box runs.
 2026-07-30 19:11:41 +08:00
Haozhe Wu c095c5948a docs(readme): add 2026-07-30 news and sync the body with this batch 
News entry across all five READMEs: the WebUI overhaul, LLM session
titles, Korea (KRX) equity, the OpenBB Workspace bridge, the Taiwan
snapshot tool, execution-time price bands, one-run-per-session, and the
trace/redaction/fail-closed batch. 07-27 rolls into <details>.

The body was stale the moment KRX landed, so it is corrected here too:
markets list gains India and Korea, free data sources 22 -> 23 with a
pykrx row and a Korea fallback chain, backtest engines 8 -> 9 with a
KoreaEquity row, VIBE_TW_STOCK_DB and VIBE_TRADING_EXTRA_CORS_ORIGINS
join the env table, the API table gains /sessions/{id}/cancel,
/sessions/{id}/title/auto, /correlation/regime and the OpenBB bridge
routes, and the project tree reads 8 engines / 24 sources.
 2026-07-30 18:01:11 +08:00
Haozhe Wu 4df1862ecc fix: close the gaps an independent pre-push audit found 
Three independent reviews of this batch's own delta each returned
DO-NOT-PUSH. Every finding below was reproduced against the code before
being fixed, and each now has a regression test.

Session lifecycle
- The in-flight claim could be stranded permanently. mark_running, the
  attempt persist and the started event ran outside the releasing
  try/finally, so a disk error there left the session answering 409 for
  good. The whole body is now inside the try.
- A run could not be cancelled before its AgentLoop existed. _active_loops
  is only populated once the registry is built, so a hung MCP discovery
  held the claim with no way out. The task handle is kept from the moment
  the run is scheduled and cancel_current falls back to it; CancelledError
  is handled explicitly, since it is a BaseException and would otherwise
  slip past the failure branch and leave the attempt stuck RUNNING.
- A cancelled run wrote its artifact with mark_failure, so the Reports view
  called it failed while the session transcript called it cancelled. The
  state store, the runs API and the run detail page now agree.

Redaction
- Only the result sink pattern-scrubbed string leaves, so a credential
  embedded in an otherwise-benign argument — a bearer token inside a shell
  command, a broker error string bound for the audit ledger — passed
  through. Both sinks scrub now.
- A bare token with no key label in front of it was never matched. The
  well-known issuer formats (OpenAI, GitHub, Slack, AWS, PyPI, JWT) are now
  recognised by shape, anchored and length-bounded so ordinary output such
  as "tokens: 1204 in / 318 out" is untouched.

Trace durability
- os.write may write fewer bytes than requested, so a large sidecar could
  be renamed into place holding only a prefix. It now loops until every
  byte lands, cleans up its temp file on failure, and the docstring states
  what fsync does and does not guarantee instead of over-claiming.

Price limits
- A close is booked with the opposite of the position's direction, so
  slippage moves it the other way. Checking the raw open approved fills
  that were then booked outside the band on all four banded engines.

Elsewhere
- OpenBB treats attempt.cancelled as terminal; without it a cancelled run
  left the SSE stream emitting heartbeats until the client gave up.
- reasoning_effort is gated on the resolved OpenAI endpoint, not just the
  provider label: a base-URL override points the same client at Ollama or a
  proxy that need not accept the field.

Pre-existing weaknesses the audit surfaced but this commit does not
address, since they predate the batch: composite sub-engines see neither
positions nor the close panel, so their band checks are weaker than a
single-market run's; and end-of-backtest liquidation bypasses can_execute
entirely.
 2026-07-30 15:36:46 +08:00
Haozhe Wu 3eb86996f5 docs: sync CLAUDE.md and .env.example with this batch 
Loaders 23 -> 24 (pykrx), engines 8 -> 9 (KoreaEquity), plus the two new
environment variables (VIBE_TW_STOCK_DB, VIBE_TRADING_EXTRA_CORS_ORIGINS),
the OpenBB Workspace bridge and its CORS opt-in, the conditional Taiwan
snapshot tool, the execution-time price-limit rule now shared by every
banded engine, the one-run-per-session 409 contract with cancellation as
its own terminal state, and the sink-aware trace redaction policy.
 2026-07-30 15:06:28 +08:00
Haozhe Wu e855a3fc54 fix(backtest): judge price limits at execution time, not from the bar's close 
Every engine with a daily band — A-share, India equity, China futures and
global futures — decided whether a fill was legal from the CURRENT bar's
close-to-close move. BaseEngine calls can_execute before filling at that
same bar's OPEN, so the close is not information the decision can use.

The error ran in both directions. A name that opened locked at the limit
but drifted back by the close was allowed to trade at the locked open — a
fill that could not have happened, which inflates results. A name that
opened freely but closed at the limit was refused a fill it would have
got. Both were reproduced directly against the engines before the change.

The band is now derived from a base price the market knew before the
order — pre_close, the previous row of the run's own close panel, or a
prior close reconstructed from tushare's pct_chg — and compared against
the price the engine actually fills at, this bar's open plus slippage.
Futures consult pre_settle first, since exchanges set their bands off the
previous settlement rather than the previous close. When no historical
base price is reachable (first bar of a run, or a composite sub-engine
with no panel of its own) the check allows the trade rather than
fabricating a block.

Closing a futures position now picks the correct side of the band from
the position's own direction: closing a long is a sell and is blocked at
the lower band, closing a short is a buy and is blocked at the upper one.

The Korea engine already worked this way; the shared helpers now live on
BaseEngine so all five engines answer the question the same way, and the
per-engine _calc_pct_change copies are gone with their callers.

Backtest results for these four markets will shift where a band was
touched intraday. That is the point: the previous numbers included fills
the simulation could not have obtained.
 2026-07-30 15:03:32 +08:00
tyj147454413-cmd 5353a18445 fix(session): one run per session, honest terminal states (#676) 
Reworked from the contribution. The defects it identified were all real and
still present; the fixes are placed differently.

- Concurrency. There was no guard at all: two concurrent sends both
  appended a user message and created an attempt, so two AgentLoops ran
  against one session. The contribution checked _active_loops, but that map
  is only populated after the registry is built — long after both sends
  have persisted — so the race stayed open. The session is now claimed
  synchronously before anything is written, and released in exactly one
  terminal path.
- A busy session is HTTP 409, not 500. The contribution's error subclassed
  RuntimeError while the endpoint converts only ValueError, so a rejection
  surfaced as an unhandled 500. The IM channel path shares send_message and
  now answers "still working on your previous message" instead of leaking an
  exception name; the live runner already funnels errors into an audited
  tick error, and the scheduled-research and OpenBB paths each create a
  fresh session, so they cannot collide.
- Cancellation is its own terminal state. AttemptStatus.CANCELLED existed
  but was dead code — every non-success landed in the failure branch, so a
  user stop was recorded and reported as a failure. It now has its own
  status, its own attempt.cancelled event, and its own reply text, and the
  frontend consumes that event live and on history hydration instead of
  relying on a client-side "stopped" marker that did not survive reload.
- Metrics reached the attempt. They were loaded from the run directory into
  the result dict but never assigned, so the reply metadata check for them
  was always false and the UI never saw them.
- One oversized history item no longer empties the window. The trim loop
  broke on the first message over budget, so a single large newest message
  left the agent with no history at all; the newest turn now survives
  truncated.
- An empty successful answer says so instead of claiming a strategy run
  completed.

The contribution's partial-JSONL read retry was dropped: it slept 10 ms and
then returned the older text anyway, so the record was skipped regardless,
and the store already writes with flush+fsync and validates record shapes
on read.
 2026-07-30 14:23:25 +08:00
Junghoon eb511ad0aa feat(backtest): Korea equity (KRX: KOSPI/KOSDAQ) market support (#693) 
Adds a KRX daily cash-equity engine, an optional pykrx loader behind a
`krx` extra, and equity_kr factor coverage.

Maintainer repairs on top of the contribution:

- The +/-30% price-limit guard was inert and would have been lookahead if
  fed naively. It needed pct_chg or pre_close, which neither pykrx (OHLCV
  only, by design) nor Yahoo supplies, so it received None and permitted
  orders at locked limits; and the borrowed helper derives the move from
  the CURRENT bar's close while fills happen at that bar's open. The band
  is now derived from a strictly historical base price and compared against
  the prospective fill price, warning once when no history is reachable.
- KRX tick quantization was missing entirely, so the engine could produce
  impossible fill prices and misjudge locked bars. Limit prices now follow
  KRX's published three-step truncation and fills snap to the tick grid
  away from the trader. KRX unified the KOSPI and KOSDAQ tables in its
  2023-01-25 reform, so this is one table, not two.
- The loader accepted any interval, fetched daily bars and cached them
  under the requested key, so a 1m/1H/4H backtest silently ran on daily
  data. Non-daily intervals are now refused so the runner falls through.
- The securities transaction tax default was the 2025 rate. From 2026-01-01
  the aggregate sell-side rate is 0.20%; the composition is documented and
  the rate stays config-driven.
- Costs were keyed to open/close rather than trade side, which is only
  correct for long positions. The side is now derived from direction plus
  open/close, and because KRX covered-short and uptick rules are not
  representable on daily bars the engine refuses allow_short outright
  rather than silently mis-modelling shorts.
- Run cards no longer claim a source that never served the data: pykrx is
  optional, and when absent the resolver returns Yahoo under a pykrx
  request. The loader that actually served is now recorded, including in
  auto mode and on per-symbol runtime fallback.
- equity_kr was added to factor metadata but every surface rejected or hid
  it. It now works through REST, the CLI, the get_market_data source enum
  and the Web Alpha Zoo, with labels in all five locales. There is no KRX
  bench panel, so the browse filter and the bench selectors are kept
  distinct instead of offering a universe that cannot run.
- The cost smoke test compared Korea against US with different slippage
  and could pass on slippage alone; it now asserts exact per-leg costs and
  the closing cash identity.
- pykrx throttling moved to the shared host throttle at its documented 1s
  interval (the module-global timestamp was not concurrency-safe), and the
  adjusted-price path is now explicit and documented as Naver-backed, since
  the unadjusted KRX path requires credentials this loader does not take.

The contributor's 255 one-line factor-zoo metadata edits are kept as
written; they are mechanically uniform. That sweep is the last one this
repo should accept — zoo-wide market capability belongs in a registry-level
default, not in 255 modules per new market.
 2026-07-30 14:02:55 +08:00
Roger Ye 9c0712b493 feat(openbb): OpenBB Workspace agent bridge (#817) 
Exposes the agent to OpenBB Workspace over the openbb-ai SDK, behind an
optional `openbb` extra.

Security repairs on top of the contribution:

- pro.openbb.co is no longer a default CORS origin. CORS is installed
  globally with credentials and keyless loopback mode admits safe GETs by
  peer address, so that default would have let scripts on the OpenBB
  origin read /sessions and /runs. Extra origins are now an additive
  opt-in (VIBE_TRADING_EXTRA_CORS_ORIGINS) declared in the typed config
  layer; the loopback-only defaults are unchanged.
- POST /v1/query carries require_auth. It reaches the full agent registry,
  which auto-registers the order tools, so an unauthenticated caller could
  have driven actions inside an already-active mandate.
- The three unadvertised LLM routes (chat title, dashboard title, prompt
  enhancement) are removed rather than authenticated: they were absent
  from the manifest and duplicated the existing authenticated title path.
- Conversation identity no longer hashes the first message. Any two
  conversations opening with the same text mapped onto one persistent
  session, leaking history between them, and later requests' full history
  was ignored. openbb-ai 2.1.0 exposes no conversation id, and the SDK
  contract is stateless with full history per request, so each request now
  runs an ephemeral session replaying the supplied history.
- The SDK is wired into the dev install and CI, with an explicit import
  check. It previously landed only in agent/requirements.txt while CI
  installs the extras, so every bridge test silently skipped and the
  feature could look green while being uninstallable.
- Workspace context is now ingested under an 8k budget instead of ignored,
  and widget-dashboard-search is advertised as false: widget groups carry
  metadata only, and the value round-trip is not implemented, so the model
  is told values are absent rather than left to invent them.

The contributor's requirements-lock.txt rewrite was dropped (stale
Python-3.12 regen with unrelated downgrades; the runtime targets 3.11).

Note for operators: a browser POST from a remote Workspace origin is still
refused by the pre-existing cross-site guard, independently of CORS and of
a valid token. Self-hosted same-origin Workspace and server-side callers
work; that boundary is pinned by a test.
 2026-07-30 14:01:46 +08:00
Haozhe Wu 205a85fcbb fix(config): route VIBE_TW_STOCK_DB through the config layer (#848) 
The Taiwan snapshot tool read its path with raw os.getenv, which the
env-var AST gate rejects for anything under agent/src/ — CI runs that gate,
so #848 as merged would have failed it. Declared the variable in DataConfig
alongside the other data-source settings and resolved it through
get_env_config(), with the container default unchanged.
 2026-07-30 13:33:18 +08:00
santhreal 7c0f35fbdb fix(tools): fail closed on malformed tool arguments (#913, #911) 
Lands both sibling hardening PRs as one unit, with the coercion direction
reversed where an action or a charge is involved.

The contributions turned malformed values into None (or 0.0) and let the
call proceed. For read-only helpers that is a reasonable fallback; for
these three it was not:

- trading_connector_tool: quantity="bad" with notional=100 became a valid
  $100 notional order, and invalid port/client_id silently fell back to
  profile defaults. Malformed or non-finite quantity/notional/limit_price
  and overrides are now rejected before place_order or cancel_order is
  reached; the numeric helpers also reject NaN and Infinity, which the
  contribution still accepted.
- qveris_tool: a missing required `parameters` became {} and a missing
  query became "", so a schema-invalid request could reach client.execute
  and spend credits. Validation now runs before the quote call, before the
  budget reservation and before execute. An explicitly supplied empty
  parameters mapping stays valid.
- report_audit_tool (#911): mapping a missing reported_value to 0.0 made
  it compare equal to a fetched 0 and count as a PASS, so unverifiable
  evidence could be certified. A missing, None or non-finite reported
  value is now an explicit failing item carrying a reason, while a genuine
  reported 0 stays valid.

options_pricing_tool additionally catches OverflowError, which escaped the
error envelope for very large integer inputs; that block subsumes #910,
whose risk_free_rate=None default is preserved here.

The regression tests were rewritten to be hermetic and branch-effective:
they no longer perform six real web searches or touch the user's session
index, they import through the same path as the rest of the suite (the
original imports would have failed collection outright), and each one now
reaches the branch it claims to cover.

The GroupExposure hunk from #913 was dropped: caps go through
_bounded_fraction, which constrains them to (0, 1], and the frame path
filters zero-weight rows, so the zero-division branch is unreachable for
validated input.
 2026-07-30 13:28:36 +08:00
tyj147454413-cmd 9efd9a1581 fix(redaction): scrub free-text tool results, keep readable payloads (#675) 
Plain-text tool results were persisted and streamed unredacted, and path
redaction over-matched shared prefixes so traces lost the payload they
exist to explain.

Reworked from the contribution:

- redact_tool_result() is now the single choke point. JSON results go
  through redact_payload, non-JSON through redact_text, once per result —
  every downstream consumer reads the already-scrubbed preview. The same
  hole in the swarm worker's preview path is wired to it too. The
  contribution added redact_text() but never called it.
- Classification is sink-aware. The contribution deleted 'content' and
  'env' from the sensitive set outright, which would have persisted whole
  write_file bodies — user documents, generated credentials, private skill
  text — into traces. 'content' is now released only in the explicit
  result sink and stays redacted in the fail-closed default used by
  tool-call arguments and the live audit ledger; 'env' is never released.
  An unrecognised sink degrades to the strict set.
- Results in the result sink also get their surviving string leaves
  pattern-scrubbed: shell output arrives as a JSON envelope, so key-based
  classification alone cannot reach secrets inside stdout.
- Secret patterns now catch quoted values, quoted JSON keys and bearer
  tokens, all idempotently; the contribution's pattern missed
  api_key="secret" and "api_key": "secret".
- Exact-root path redaction is restored (the contribution only matched
  root + separator) while keeping its real fix, which was to stop
  redacting unrelated paths that merely share a prefix. Root lookup is
  memoised per cwd instead of recomputed on the hot path.

Adds end-to-end coverage proving a planted secret in a plain-text result
and in a write_file content argument reaches neither the trace files nor
the streamed preview, while the LLM-facing message keeps the raw text.
 2026-07-30 13:25:31 +08:00
1anter 7af56b93eb feat(providers): send top-level reasoning_effort to direct OpenAI (#755) 
Direct OpenAI needs an explicit top-level reasoning_effort — including the
literal "none" — which the agent could not express: the setting was
documented as relay-only and never reached the client constructor.

Reworked from the contribution, which gated on "every provider whose
openrouter_reasoning_body is false". That would have sent the field to
DeepSeek, Gemini, Groq, DashScope, Zhipu, NVIDIA, Spark and MiniMax, any
of which may reject an unknown field, and it contradicted the existing
DeepSeek invariant asserted in test_provider_diagnostics.py. Gating is now
a positive allowlist covering direct OpenAI only.

The allowlist deliberately checks BOTH the resolved capability name and
the configured provider name, because either check alone leaks the field:
model inference maps provider=openai + a deepseek model onto DeepSeek
capabilities, and unknown providers fall back to OpenAI capabilities, so a
flag on the capability record alone would start sending the field to any
unrecognised gateway.

OpenRouter/Requesty keep their extra_body.reasoning path untouched. The
settings allowlist accepts "none" and its rejection message now lists the
values it actually permits. Settings exposes the option through t() with
labels in all five locales, and provider_diagnostics reports the resolved
capability so `doctor` can explain why an effort is or is not sent.
 2026-07-30 13:17:32 +08:00
JerryTseng f736c106a3 feat(tools): read-only Taiwan stock snapshot tool (#848) 
Adds a read-only agent tool over a published Taiwan-market SQLite
snapshot (stdlib + sqlite3 only, mode=ro&immutable=1, no network).

Maintainer repairs on top of the contribution:

- Bulk responses are now budgeted to 9,500 serialized characters so they
  survive the agent loop's 10,000-char tool-result cut as valid JSON. At
  the documented maximum (200 rows x 50 stocks) the response was 5.4 MB
  and the prefix cut produced a JSON parse error. Rows are retained
  newest-first, round-robin across stocks, so the newest bar of every
  stock survives trimming; responses carry total_rows, returned_rows,
  truncated and a narrowing hint. The budget covers history, universe,
  latest and lookup — the size overflow was not limited to history.
- Indicators are no longer mis-dated. The tool picked the newest price
  row and the newest feature row independently but reported a single
  date, so 07-23 indicators were served under date=07-24. latest and
  universe now return price_date, feature_date and a stale_features
  flag.
- The Compose snapshot mount no longer defaults inside the checkout
  (./data/tw-stock was also un-gitignored). The default host path is
  now ${HOME}/.vibe-trading/tw-stock, overridable via
  VIBE_TW_STOCK_DATA_DIR; no market data can land in the working tree.
- The snapshot schema (4 tables, 37 columns) is validated on open, so an
  unrelated or stale SQLite file reports an actionable error instead of
  registering successfully and failing mid-query.
- Fixed a connection leak: sqlite3's connection context manager commits
  but never closes, leaking a handle per call.

The contributor's unrelated requirements-lock.txt rewrite (21 stale pin
changes, including FastAPI and OpenAI downgrades) was dropped; the tool
needs no dependency change.
 2026-07-30 13:15:16 +08:00
tyj147454413-cmd 002d04beb5 fix(trace): make trace records and their sidecars durable (#662) 
TraceWriter.write() only flushed, so a host crash or container kill lost
the last records. Each record is now fsynced, and — because a fsynced
record must never point at a blob that is not on disk — sidecar writes go
through temp-write -> fsync -> os.replace -> parent-directory fsync
before the referencing record is written. New trace.jsonl and sidecar
directory entries are also fsynced.

fsync failures (unsupported filesystem) log one warning per writer and
degrade to flush-only rather than breaking the run.
 2026-07-30 13:01:53 +08:00
santhreal 0157409f0d fix(options-engine): tolerate malformed trade PnL and disclose exclusions (#907) 
Trade records carrying pnl=None or a non-numeric value raised TypeError
while computing trade statistics. Skip those records instead, and — since
trade_count still reports every record — append a warning naming how many
were excluded so win_rate and profit_loss_ratio are never silently
derived from a subset.
 2026-07-30 13:01:53 +08:00
santhreal e816448e30 fix(regime): return NaN edge density when there are no asset pairs (#908) 
compute_edge_density divided by n_pairs, which is zero for a zero- or
one-column frame. Correlation edge density is undefined without at least
one distinct asset pair, so return an index-aligned all-NaN series. The
timeline entrypoint already requires two fetched assets, so this does not
mask a user-facing error.

Supersedes the duplicate guard bundled in #912.
 2026-07-30 13:01:53 +08:00
santhreal 58d3026171 fix(risk-xray): raise when surviving symbols have zero total weight (#909) 
The input validator requires the original basket total to be positive but
permits individual zero weights, so the history filter can leave only
zero-weight survivors and the renormalization divided by zero. Raise a
clear ValueError instead: allocating capital to zero-weight survivors
would invent holdings the user does not own, and an all-zero x-ray would
contradict the fully-invested-basket contract.
 2026-07-30 13:01:52 +08:00
Haozhe Wu 3ed96c32ba feat: LLM auto session titles + live reasoning whisper 
- POST /sessions/{id}/title/auto — summarize the first exchange into a short
  title in the user's language; never clobbers a manual rename
- Frontend fires it after the first completed exchange and refreshes the sidebar
- reasoning_delta now carries a bounded 600-char rolling tail on the existing
  throttled emit; ActivityLine whispers the live thought (serif italic, 3-line
  clamp) while the attempt is active
 2026-07-30 11:58:19 +08:00
Haozhe Wu 8f40de5a5c feat: FW-3 live-test fix wave — connector resume + i18n, hero tabs, activity chain, pro validation charts 
- ConnectionBanner: surface only during an active reconnect loop (boot-state false alarm fix)
- Trading-monitor wording replaces literal 'runtime' jargon across zh/ja/ko/ar
- Hero: vertical centering + category-tab example library; About relocated to sidebar footer
- Brand: ascending-candles mark in BrandMark/favicon/logo.svg (replaces VT lettermark + stale FIN-ANYTHING wordmark)
- Dark theme: surface separation and border/muted-foreground contrast lift
- Connector panel: /live/resume wired with a resume button; RunnerStatus internals fully i18n'd
- Activity chain: coalesced xN steps with argument summaries, chronological order, sub-second durations
- RunDetail: trades side-normalization fix + pnl/return/holding columns + filters/pagination/totals; code copy button; run-card validation rendered via ValidationPanel
- Validation: Monte Carlo fan chart + Sharpe distribution histograms + walk-forward bars; validation.py emits sharpe samples and path envelopes
- Compare: centered layout, rebase-to-100 overlay toggle, in-cell metric bars, swap button
 2026-07-30 11:44:31 +08:00
Haozhe Wu d8c2a600c7 feat: FW-2 wave 2 — durable activity object, Agent decomposition, full ja/ko/ar parity, serif reading voice 
One durable activity object per turn rendered by a single ActivityLine
(verb + newest step + 1s tabular timer; userOpen ?? auto disclosure with
900ms dwell; cancel/timeout/failure leave terminal rows with Continue and
re-attach; scattered spinners and the pulse bar removed). Escape-to-stop
and full + menu keyboard contract; follow-up placeholder after runs;
hidden-tab completion title marker. Agent.tsx decomposed 2353→1686 via
memoized Composer (byte-faithful IME guard + render-isolation test),
GoalPanel, LiveRuntimePanel. ja/ko/ar translated to full 929-key parity.
Assistant narrative paragraphs read in a CJK-safe serif at 15.5px; tables,
metrics, and code stay sans/mono. Remaining page-level hardcoded strings
localized.
 2026-07-30 00:50:35 +08:00
Haozhe Wu 4b69f49809 feat: FW-2 wave 1 — end-to-end tool call_id, tool-trail persistence, 84-tool i18n, copy polish, a11y/robustness sweep 
Backend (surgical, additive): call_id on tool_call/result/progress/heartbeat
(loop.py, swarm worker); tool_trail serialized onto assistant messages at
attempt completion and exposed through session history. Frontend: exact
call_id matching with FIFO replay fallback, trail rebuild on reload via the
same helper as the live path, 84 localized tool labels, MetricsCard plain-
language glosses, focus-trap dialogs, overlay connection banner with
terminal reload state, cross-tab prefs sync, idle echarts prefetch, mandate
zh glossing (交易授权), swarm vocabulary unification, '/' product-first
routing with /about.
 2026-07-30 00:13:17 +08:00
Haozhe Wu 71933ecf31 feat(frontend): guided-minimalism WebUI overhaul — 12-way audit + fix fleet 
Land the pre-talk WebUI campaign: first-frame stability (pre-paint theme
boot via CSP-safe public/theme-boot.js, self-hosted fonts + OFL license),
streaming performance (rAF token batching, throttled flush, streaming
markdown with deferred highlight, 160-row windowing, swarm state in its
own store slice with per-run coalescing), correctness (FIFO tool matching,
retry preserves request semantics, display/request separation, cache-hit
session refresh, multi-tab message.received), guided-minimalism welcome
(time-aware greeting + brand lockup + 4 quick actions + collapsible
library, fill-first affordance), SwarmDashboard rebuild with honest
indeterminate progress, five-locale copy de-jargoning with lazy locale
chunks, accessibility Level-A fixes (color-blind metric glyphs, aria-live
streaming, skip link, focus-visible affordances), robustness (markdown
link/table/KaTeX overflow, safe localStorage, portal modal), and '/' now
opens the product with marketing moved to /about.
 2026-07-29 23:07:35 +08:00
Haozhe Wu c8741e34bf docs(readme): add 2026-07-29 news 2026-07-29 16:04:30 +08:00
Yufeng HeandHaozhe Wu 59fa3181b0 feat(backtest): emit risk x-ray artifacts in portfolio runs (Portfolio Studio slice) (#900) 
* feat(backtest): emit risk x-ray artifacts in portfolio runs (Portfolio Studio slice)

Signed-off-by: Yufeng He <40085740+he-yufeng@users.noreply.github.com>

* fix(backtest): refuse the average-basket x-ray for long-short books

compute_risk_xray is long-only by contract, but average_invested_weights
filtered negative mean weights out before the check could fire — a
long-short run would emit an x-ray describing only its long half while
presenting it as the whole strategy. Net short average exposure now
raises, taking the same no-artifact path as flat and short-history runs.

---------

Signed-off-by: Yufeng He <40085740+he-yufeng@users.noreply.github.com>
Co-authored-by: Haozhe Wu <haozhe_wu@connect.hku.hk>
 2026-07-29 15:51:56 +08:00
Hongi Nagaputra 7f8f52d8bf feat(backtest): model isolated and cross margin liquidation risk (#889) 
* feat(backtest): calculate perpetual maintenance risk

Signed-off-by: zygora <122336305+honginp@users.noreply.github.com>

* feat(backtest): model isolated and cross liquidation

Signed-off-by: zygora <122336305+honginp@users.noreply.github.com>

* fix(backtest): enforce strict cross liquidation threshold

Signed-off-by: zygora <122336305+honginp@users.noreply.github.com>

* test(backtest): trim liquidation risk test budget

Signed-off-by: zygora <122336305+honginp@users.noreply.github.com>

* fix(backtest): value risk margin at mark

Signed-off-by: zygora <122336305+honginp@users.noreply.github.com>

---------

Signed-off-by: zygora <122336305+honginp@users.noreply.github.com>
 2026-07-29 15:42:44 +08:00
ALI EMADIAli EmadiHaozhe Wu
 		89b15ac806 fix(backtest): keep bar_returns bit-identical for gapped positive series (#895) 
* fix(backtest): keep bar_returns bit-identical for gapped positive series

#872 replaced close.pct_change().fillna(0.0) with bar_returns and stated that
results for strictly positive series are bit-identical, so ordinary equity and
crypto runs are unchanged. That was verified only on gapless series.

On the pinned pandas (>=2.0,<3.0) pct_change defaults to fill_method='pad' and
forward-fills the divisor. bar_returns divided by the raw shifted series, so any
positive series with an interior NaN diverged. _align deliberately limits its
ffill to 5 bars (base.py:166-179) to keep long suspensions visible, so a halt
longer than that reaches bar_returns as a NaN prior and the real move across it
is reported as 0.0 -- silently, because the undefined-bar counter excludes NaN
priors by construction. On a real _align run an 8-bar halt turned a +17.1429%
resumed-trading bar into +0.0000%.

Injecting a zero where a large move occurred shrinks dispersion. Return deltas
cancel by sign; that damage does not. On a 20-symbol universe with two halted
symbols and news-shaped gaps, annualised vol fell 4.9619% -> 4.4673% and Sharpe
rose 3.0502 -> 3.3817.

Carry the prior price forward explicitly. This restores the pre-#872 answer and,
unlike the deprecated default it replaces, gives the same result on every
supported pandas -- the old behaviour flips silently under pandas 3.0, where
fill_method is None. The #872 contract is untouched: a non-positive prior is
still undefined, still 0.0, still logged.

Second, smaller item: inf > 0 is True, so where(prev > 0) kept an infinite prior
and the bar after it read as a clean -1.0 with no warning. buy_and_hold_return
had the same asymmetry -- it checked isfinite(last) but only first > 0, so an
infinite entry price returned -100% instead of None. Both now assert finiteness.

Signed-off-by: Ali Emadi <ali@gatekeeperai.com>

* test(backtest): lock the _align boundary and align the undefined-bar warning

- add an engine-boundary regression: a halt longer than _align's ffill
  limit must still credit the across-gap move to the resumed bar
- the undefined-bar warning now also fires for non-finite prior prices,
  so say so in the message (and in the assertions that pin it)

---------

Signed-off-by: Ali Emadi <ali@gatekeeperai.com>
Co-authored-by: Ali Emadi <ali@gatekeeperai.com>
Co-authored-by: Haozhe Wu <haozhe_wu@connect.hku.hk>
 2026-07-29 15:34:17 +08:00
Xiaotian Jin 6479fb6428 fix(cli): load ~/.vibe-trading/.env on the connector command path (#902) 
`main()` routes the `connector` subcommand group straight to
`_dispatch_connector`, which — unlike the interactive TUI — never runs
preflight, so nothing populated `os.environ` from `~/.vibe-trading/.env`.
`EnvConfig` reads only `os.environ`, so every connector resolving
credentials through `get_env_config()` reported them as missing despite a
valid file on disk: Longbridge's app key / secret / access token (the only
broker credential set sourced from env vars rather than a JSON config
file), and Futu's `FUTU_TRADE_PWD_MD5` live-unlock hash.

Call the existing `_ensure_cli_env()` helper at the dispatcher entry.
`_ensure_dotenv()` is idempotent and loads with `override=False`, so a real
environment variable still outranks the file, and connectors that read JSON
config are unaffected.

Distinct from #477 / #541, which were the stale-`EnvConfig`-singleton
variant: there `_ensure_dotenv()` ran but no `reset_env_config()` followed.
Here it never ran at all, so that fix could not reach this path.

Closes #901

Signed-off-by: Xiaotian Jin <jin.xiaotian.kim@gmail.com>
 2026-07-29 15:28:27 +08:00
卫斯李 348a25b8f8 fix(metrics): add missing annualisation entries for 15 data sources (#891) 
* fix(metrics): add missing annualisation entries for 15 data sources

13 of 24 VALID_SOURCES entries had no trading_days or bars_per_day
lookup, causing calc_bars_per_year to silently return 252 for every
interval — intraday bars for these sources were never annualised
correctly (e.g. Longbridge 1m returned 252 instead of 252×390).

Add all 15 missing sources to _TRADING_DAYS and _BARS_PER_DAY:
- crypto: binance (365 days)
- A-share: baostock, tencent, eastmoney, sina (252 days, 240-min)
- US/international: yahoo, finnhub, alphavantage, tiingo, fmp,
  stooq, longbridge (252 days, 390-min)
- Indian: india_broker (252 days, 375-min)
- daily-only: qveris, local (252 days, 1 bar/day for all intervals)

24 new test assertions across 16 tests, including a cross-VALID_SOURCES
sweep that verifies every source returns >0 for both daily and intraday.

Closes #884

Signed-off-by: Robin1987China <41602358+Robin1987China@users.noreply.github.com>

* fix(metrics): restore _SOURCE_ALIASES, _normalize_interval, and related tests

These were accidentally dropped during stash conflict resolution;
restore the alias/interval-normalisation layer that loaders depend on
and re-add the 3 corresponding tests (lowercase hour/day matching,
yahoo→yfinance alias).

Signed-off-by: Robin1987China <41602358+Robin1987China@users.noreply.github.com>

* fix(metrics): address review — reclassify local/qveris, harden drift guard

Three changes from review (#891):
1. Reclassify local and qveris from daily-only to US equity session
   (252d/390-min) — both loaders support all intervals via resampling.
   Updated all 7 interval layers of _BARS_PER_DAY.
2. Already fixed in d6770bd: source_key lower normalisation via
   str(source).strip().lower() + _SOURCE_ALIASES guard — no change needed.
3. Strengthen test_no_intraday_source_falls_to_default with a
   set-coverage assertion that fails the moment a loader is registered
   without a _TRADING_DAYS + _BARS_PER_DAY entry.

Signed-off-by: Robin1987China <41602358+Robin1987China@users.noreply.github.com>

---------

Signed-off-by: Robin1987China <41602358+Robin1987China@users.noreply.github.com>
 2026-07-29 15:21:22 +08:00
Santh a9e50e9bc1 fix(channels): preserve leading indent when splitting messages (#867) 
* fix(channels): preserve leading indent when splitting messages

After a newline cut, lstrip() removed indentation on the next chunk, so
code blocks and nested markdown lost spaces across Discord/Slack/Telegram
splits. Drop only the separator character.

Signed-off-by: santhreal <64453045+santhreal@users.noreply.github.com>

* fix(channels): keep Signal style offsets aligned after indent-preserving splits

split_message no longer blanket-lstrips; mirror that in _partition_styles so
indented chunks keep UTF-16 style ranges.

Signed-off-by: santhreal <64453045+santhreal@users.noreply.github.com>

* fix(channels): advance style cursor on space separator preceding indented chunk

Signed-off-by: santhreal <64453045+santhreal@users.noreply.github.com>

---------

Signed-off-by: santhreal <64453045+santhreal@users.noreply.github.com>
 2026-07-29 15:20:08 +08:00
Santh 72f6985825 fix(skills): parse frontmatter when closing fence is at EOF (#861) 
* fix(skills): parse frontmatter when closing fence is at EOF

SKILL.md / memory files that end at the closing --- without a trailing
newline (common from editors and LLM skill saves) were treated as plain
text, so name/description were dropped. Empty ---/--- blocks had the
same miss. Accept EOF or empty meta between fences.

Signed-off-by: santhreal <64453045+santhreal@users.noreply.github.com>

* fix(skills): require newline before closing frontmatter fence

Signed-off-by: santhreal <64453045+santhreal@users.noreply.github.com>

---------

Signed-off-by: santhreal <64453045+santhreal@users.noreply.github.com>
 2026-07-29 15:19:35 +08:00
Haozhe Wu 749c37b09f chore(frontend): declare the Node 22 requirement 
React Router 8 requires Node >=22.22.0, so `npm install` now fails for anyone
still on Node 20 with an error that does not name the cause. Declare it in
package.json engines, and say it on the Quick Start line that tells people to
run it, in all five READMEs.
 2026-07-28 15:59:33 +08:00
Haozhe Wu bd9a9b70b8 docs(readme): add 2026-07-28 news 2026-07-28 15:58:42 +08:00
Haozhe Wu 1e5b79dac6 chore(frontend): move to Node 22 and React Router 8 
React Router 7.12.0 through 8.2.x carry a high-severity RSC-mode CSRF bypass
(GHSA-qwww-vcr4-c8h2). We were on 7.18.1, inside that range. The advisory is not
reachable here — the app is a client-side SPA on createBrowserRouter with no
@react-router/server and no RSC entry point, and the frontend ships as static
Vite output — but 7.18.1 is the final 7.x release, no backport exists, and 8.3.0
is the only patched version, so staying put means staying on an EOL line with a
permanent open alert.

react-router-dom has no 8.x at all; the DOM exports moved into react-router
itself. So the dependency is swapped and every import rewritten. All ten APIs in
use here are unchanged in 8.3.0, and none of the data-layer surface that changed
most between majors is used.

react-router 8 requires Node >=22.22.0, which is the same wall that had jsdom 30
and @testing-library/jest-dom 7 sitting in the Dependabot ignore list. Node 20
reached EOL on 2026-04-30, so the toolchain moves to Node 22 across CI, the wiki
workflow, the Dockerfile and docker-compose. That unblocks all three at once:
jsdom and jest-dom are taken here too, and their ignores are removed rather than
left carrying a rationale that no longer holds.

Verified on the frontend suite: tsc -b and vite build clean, 310 tests across 35
files pass before and after the jsdom/jest-dom bumps.
 2026-07-28 14:39:59 +08:00
dependabot[bot] cd04c93851 build(deps): bump the pip-minor-patch group across 1 directory with 6 updates (#893) 
Bumps the pip-minor-patch group with 6 updates in the / directory:

| Package | From | To |
| --- | --- | --- |
| [akshare](https://github.com/akfamily/akshare) | `1.18.79` | `1.18.80` |
| [fastapi](https://github.com/fastapi/fastapi) | `0.140.0` | `0.140.7` |
| [fastmcp](https://github.com/PrefectHQ/fastmcp) | `3.4.4` | `3.4.5` |
| [fastmcp-slim](https://github.com/PrefectHQ/fastmcp) | `3.4.4` | `3.4.5` |
| [openai](https://github.com/openai/openai-python) | `2.48.0` | `2.49.0` |
| [tqdm](https://github.com/tqdm/tqdm) | `4.69.1` | `4.70.0` |



Updates `akshare` from 1.18.79 to 1.18.80
- [Release notes](https://github.com/akfamily/akshare/releases)
- [Changelog](https://github.com/akfamily/akshare/blob/main/docs/changelog.md)
- [Commits](https://github.com/akfamily/akshare/compare/release-v1.18.79...release-v1.18.80)

Updates `fastapi` from 0.140.0 to 0.140.7
- [Release notes](https://github.com/fastapi/fastapi/releases)
- [Commits](https://github.com/fastapi/fastapi/compare/0.140.0...0.140.7)

Updates `fastmcp` from 3.4.4 to 3.4.5
- [Release notes](https://github.com/PrefectHQ/fastmcp/releases)
- [Changelog](https://github.com/PrefectHQ/fastmcp/blob/main/docs/changelog.mdx)
- [Commits](https://github.com/PrefectHQ/fastmcp/compare/v3.4.4...v3.4.5)

Updates `fastmcp-slim` from 3.4.4 to 3.4.5
- [Release notes](https://github.com/PrefectHQ/fastmcp/releases)
- [Changelog](https://github.com/PrefectHQ/fastmcp/blob/main/docs/changelog.mdx)
- [Commits](https://github.com/PrefectHQ/fastmcp/compare/v3.4.4...v3.4.5)

Updates `openai` from 2.48.0 to 2.49.0
- [Release notes](https://github.com/openai/openai-python/releases)
- [Changelog](https://github.com/openai/openai-python/blob/main/CHANGELOG.md)
- [Commits](https://github.com/openai/openai-python/compare/v2.48.0...v2.49.0)

Updates `tqdm` from 4.69.1 to 4.70.0
- [Release notes](https://github.com/tqdm/tqdm/releases)
- [Commits](https://github.com/tqdm/tqdm/compare/v4.69.1...v4.70.0)

---
updated-dependencies:
- dependency-name: akshare
  dependency-version: 1.18.80
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: pip-minor-patch
- dependency-name: fastapi
  dependency-version: 0.140.7
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: pip-minor-patch
- dependency-name: fastmcp
  dependency-version: 3.4.5
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: pip-minor-patch
- dependency-name: fastmcp-slim
  dependency-version: 3.4.5
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: pip-minor-patch
- dependency-name: openai
  dependency-version: 2.49.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: pip-minor-patch
- dependency-name: tqdm
  dependency-version: 4.70.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: pip-minor-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2026-07-28 14:30:09 +08:00
yagnikpipaliya d2f12a3e72 fix(anthropic): self-heal models that deprecate the temperature field (#890) 
Problem
-------
Next-gen Claude models (e.g. claude-opus-5, claude-opus-4-8, claude-sonnet-5)
reject the "temperature" request field with HTTP 400 "temperature is deprecated
for this model." for ANY value, while older models (claude-opus-4-5,
claude-sonnet-4-5, ...) still accept it. _build_anthropic unconditionally
forwarded the global default temperature=0.0, so every run on a next-gen model
failed before producing output. Model names are not predictable (opus-4-5
accepts it, opus-4-8 does not) and the rejection is value-independent, so
neither a fixed model allowlist nor an "omit when 0.0" shortcut is a correct
fix.

Solution
--------
Add a temperature-safe ChatAnthropic subclass (_make_temperature_safe_anthropic)
that wraps all four generation entry points (_generate, _agenerate, _stream,
_astream). When the API reports temperature as unsupported it drops the field
and retries once, then records the model in a process-wide set so subsequent
requests omit it up front (no repeated failed request). Models that accept
temperature keep their configured deterministic value unchanged. The subclass
is built from the lazily-imported class and cached per base class, preserving
the optional langchain-anthropic dependency boundary and existing test doubles.

Correct for any temperature value, current and future model names, and the
streaming path the Web UI relies on.

Tests
-----
Add regression tests in agent/tests/test_llm.py: self-heal retry drops
temperature and remembers the model, temperature is preserved for supporting
models, and the error matcher only triggers on temperature-unsupported errors.
Full test_llm.py suite: 58 passed.
 2026-07-28 13:38:11 +08:00
Haozhe Wu c63713c587 chore(deps): ignore the transitive versions ccxt exact-pins 
ccxt does not use ranges — it exact-pins its entire transitive closure. All 23
of those packages that appear in our lock sit at precisely ccxt's pin, with no
mismatches, so none can move until ccxt itself releases, and ccxt 4.5.68 is
already the latest. Any Dependabot PR touching one is unmergeable by
construction: the pip group PR proposed five of them and died with
ResolutionImpossible on certifi against the ccxt line in the lock.

Re-running the resolver confirms this is the ceiling rather than a stale lock —
pip-compile --upgrade under python:3.11-slim, matching CI, reproduces every one
of those versions, so there is no lock change to make. pydantic-core is in the
same position via pydantic's own == pin, and was the sixth package in that PR.

Ignore the set. ccxt itself is deliberately not ignored: it is the package whose
bump legitimately carries all of them. This suppresses only the automated PRs —
Dependabot alerts come from the dependency graph and are unaffected by this
file, so a CVE in cryptography, urllib3, requests or certifi still surfaces in
the Security tab, where the remedy is a ccxt bump.
 2026-07-28 12:57:46 +08:00
Haozhe Wu 452653c81c fix(backtest): make the buy-and-hold return path sign-safe 
close.pct_change() silently assumes price[t-1] > 0. That holds for equities but
not for instruments that can print zero or negative prices, such as European
day-ahead power. As the divisor approaches zero an ordinary absolute move reads
as an enormous percentage move, so the compounded benchmark explodes — a
reported +39,560% where the honest figure was -42.7%. At exactly zero the next
bar is inf, which fillna(0.0) does not neutralise because it fills NaN, and
(1 + r).prod() then collapses to nan.

Adopt an explicit contract. A return is defined only when the previous price is
finite and strictly positive; otherwise it is 0.0, never inf or nan. Buy-and-
hold total return becomes the price relative P_end / P_start - 1 rather than the
compounded product, and is omitted entirely when the entry price is not
positive, because no honest percentage exists. Undefined bars are logged rather
than silently swallowed: a wrong return is a wrong reported number, not a crash.

bar_returns keeps the exact `close / prev - 1` expression pct_change uses, so
results for strictly positive series are bit-identical and ordinary equity and
crypto runs are unchanged.

The loader still rejects an exact-zero close, but the reason has narrowed to the
one that survives: sizing is target_notional / abs(price), undefined at zero.

Closes #872
 2026-07-28 12:57:37 +08:00
Haozhe Wu 6a859902e9 fix(cli): inject a host session id into non-interactive runs 
The research-goal tools are registered unconditionally and resolve their
session from the host runtime, so an entry point that calls _run_agent without
an id makes every goal call fail validation with "session_id is required" while
the run still reports SUCCESS. Reported for `vibe-trading run`; six of the seven
_run_agent call sites had the same omission, including cmd_session_chat, which
already received the session id as a parameter and simply never forwarded it.

Add _ensure_session_id, which builds a Session (whose id is populated at
construction) and persists it best effort, so a store or index failure still
yields a usable id rather than the empty string that caused the bug.
cmd_continue reuses one session across continuations so goals accumulate: a
session-backed run_id already is a session id, since sessions and their traces
share SESSIONS_DIR, so it is used directly; a plain run gets a derived id.
Re-registering an existing id is idempotent. The interactive REPL had the same
latent hole whenever _new_session returned None, and now falls back to an
unpersisted id.

Also align the MCP contract. All four MCP goal tools declared session_id
required while the in-process registry injects it and leaves it optional — the
opposite contract, which asked the model to invent an internal identifier it
has no way to know. session_id is now optional there too, defaulting to one
stable id per server process.

Closes #885
 2026-07-28 12:57:28 +08:00
Haozhe Wu d528066646 docs(readme): fix Windows install steps in all five READMEs (#888) 
CMD has no `cp`, so `cp agent/.env.example agent/.env` fails there; add the
`copy` form and note that PowerShell's `cp` alias already works. Also add the
CMD activation line and the execution-policy hint that blocks Activate.ps1.

Reported by @brady-cec in discussion #29.
 2026-07-28 01:54:41 +08:00
Haozhe Wu e8c3bf6ee5 Merge pull request #881 from HKUDS/chore/unblock-dep-lock 
chore(deps): unblock the ccxt/setuptools lock deadlock and cut dependabot churn
 2026-07-27 18:11:00 +08:00
Haozhe Wu db3ed92cef chore(deps): ignore jsdom majors until the Node 22 migration 
jsdom 30.x requires Node ^22.22.2 || ^24.15.0 || >=26 while CI and Docker are
on Node 20. On Node 20 the incompatibility surfaces indirectly as vitest
failing to start its pool workers rather than as a clean engine error, so #877
reads like a flaky frontend failure. Same wall as @testing-library/jest-dom 7.
 2026-07-27 18:03:57 +08:00
Haozhe Wu 58363fc01f chore(deps): unblock the ccxt/setuptools lock deadlock and cut dependabot churn 
Dependabot could not land either half of a mutually-blocking pair: the
minor/patch group wanted ccxt 4.5.68 (which pins setuptools==83.0.0) while
keeping setuptools at 82.0.1, and the separate setuptools 83.0.0 PR conflicted
with the ccxt 4.5.67 already in the lock. Neither can go green alone because
setuptools is a major bump and is excluded from the minor/patch group by
design, so it cannot ride along.

Regenerate the lock so both move together. Verified with pip-compile under
Python 3.11 on Linux (matching CI) and checked with CI's own gate,
pip install --dry-run --require-hashes. Note the lock is platform-sensitive:
regenerating on macOS drops jeepney/secretstorage and swaps py-mini-racer for
mini-racer, so it must be produced on Linux.

Also ignore websockets, capped at <16 by langgraph-sdk 0.4.2, and move all
three ecosystems to monthly with 2 concurrent PRs. Security updates are a
separate channel and are unaffected.
 2026-07-27 18:03:14 +08:00
Haozhe Wu 2d0192a98b Merge pull request #876 from HKUDS/dependabot/npm_and_yarn/frontend/npm-minor-patch-8de1194a4f 
build(deps): bump lucide-react from 1.26.0 to 1.27.0 in /frontend in the npm-minor-patch group
 2026-07-27 17:51:22 +08:00
dependabot[bot] 2dc45d7ecf build(deps): bump lucide-react in /frontend in the npm-minor-patch group 
Bumps the npm-minor-patch group in /frontend with 1 update: [lucide-react](https://github.com/lucide-icons/lucide/tree/HEAD/packages/lucide-react).


Updates `lucide-react` from 1.26.0 to 1.27.0
- [Release notes](https://github.com/lucide-icons/lucide/releases)
- [Commits](https://github.com/lucide-icons/lucide/commits/1.27.0/packages/lucide-react)

---
updated-dependencies:
- dependency-name: lucide-react
  dependency-version: 1.27.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: npm-minor-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
 2026-07-27 09:33:03 +00:00
Haozhe Wu 218d453859 build(deps): ignore bumps blocked by upstream or infra constraints 
caio: aiofile 3.11.1 (latest) requires caio~=0.9.0, so bumping caio to 0.10.x
makes pip unresolvable and the grouped PR fails CI. Ignore only caio, not the
whole minor/patch group.

@vitejs/plugin-react 6.x peers vite@^8 (we are on 6.x) and
@testing-library/jest-dom 7.x requires Node >=22 (CI/Docker use 20). Both are
migrations, not dependency bumps.
 2026-07-27 17:31:41 +08:00
Haozhe Wu 612004243c fix(loaders): keep a stubbed config from writing cache into the working tree 
loader_cache_enabled() returned the raw config value, so a MagicMock config
read truthy and switched the opt-in cache on. loader_cache_root() then passed
that MagicMock to Path(), which resolves via __fspath__ to a *relative* path
and put real market-data parquet under agent/MagicMock/ inside the repo.

Validate both at the boundary: require a genuine True to enable the cache, and
honor the root override only when it is a real non-blank str.
 2026-07-27 17:31:41 +08:00
Haozhe Wu a69c55944b docs(readme): add 2026-07-27 news 2026-07-27 17:15:06 +08:00
Haozhe Wu 5222f418cf Merge pull request #866 from santhreal/fix/journal-to-float-strip-currency 
fix(journal): strip currency symbols before float coerce
 2026-07-27 15:49:02 +08:00
Haozhe Wu 5c690c7211 Merge branch 'main' into fix/journal-to-float-strip-currency 
Resolves the import-list conflict in agent/tests/test_trade_journal.py
introduced when #862 landed on main: keep both _to_float (this branch)
and load_dataframe (#862).
 2026-07-27 15:41:42 +08:00
Haozhe Wu 5904e615a1 Merge pull request #862 from santhreal/fix/journal-load-utf16-csv 
fix(journal): accept UTF-16 BOM CSV exports in load_dataframe
 2026-07-27 15:40:12 +08:00
Haozhe Wu 8c0369727d Merge pull request #868 from santhreal/fix/journal-infer-crypto-suffix-quotes 
fix(journal): treat BTCUSDT-style symbols as crypto markets
 2026-07-27 15:40:07 +08:00
Haozhe Wu fde1179bb1 Merge pull request #873 from ddy4633/fix-correlation-forward-fill 
fix(backtest): stop forward-filling missing closes in the correlation matrix
 2026-07-27 15:40:02 +08:00
Haozhe Wu c8fa7a263c Merge pull request #869 from y85998607/fix/vnpy-export-import-path 
fix(vnpy-export): update CtaTemplate/BacktestingEngine imports to vn.py 4.0 layout
 2026-07-27 15:39:56 +08:00
Haozhe Wu 7b2eda9636 Merge pull request #863 from santhreal/fix/skill-sanitize-keep-cjk-slug 
fix(skills): keep CJK characters in skill directory slugs
 2026-07-27 15:39:51 +08:00
Haozhe Wu 7fe894df86 Merge pull request #864 from santhreal/fix/metrics-bars-per-year-lowercase-interval 
fix(metrics): annualize lowercase 1h/1d and yahoo like loaders
 2026-07-27 15:39:47 +08:00
Haozhe Wu ded3e52ebc Merge pull request #865 from santhreal/fix/doc-reader-utf16-before-latin1 
fix(tools): decode UTF-16 BOM text before latin-1 fallback
 2026-07-27 15:39:42 +08:00
ddy4633 b913ef67eb fix(backtest): stop forward-filling missing closes in the correlation matrix 
Under the project's pandas>=2,<3 pin, pct_change defaults to
fill_method='pad'. A halted session (NaN close) is forward-filled into a
fabricated 0% return, which survives the inner join and gets paired against
the peer's real move for that day.

regime.py:113 performs the identical computation with fill_method=None, and
its docstring states it "Mirrors the alignment in
_rolling_correlation_matrix" — the sibling was written from this function and
fixed there while the original was left behind. risk_xray.py:144 also passes
fill_method=None. The intended behaviour is already established in-tree.

Passing it explicitly also silences the FutureWarning; the default flips
again under pandas 3.
 2026-07-27 03:15:41 +08:00
Haozhe Wu 5140803980 Merge pull request #871 from HKUDS/codex/fix-windows-background-self-kill 
fix(background): prevent Windows host self-termination
 2026-07-27 02:32:27 +08:00
Haozhe Wu f56a3d398f fix(background): prevent Windows host self-termination 
Add task-scoped cancellation, reject broad Python process-name kills, expose timeout context, and cover the lifecycle on Windows 11 and Windows x64 with Python 3.14.

Signed-off-by: Haozhe Wu <haozhe_wu@connect.hku.hk>
 2026-07-27 02:20:50 +08:00
Haozhe Wu d5ea0ec519 docs(readme): add 2026-07-26 news and ignore workbuddy 2026-07-26 19:13:36 +08:00
y85998607 9a7f60aca5 fix(vnpy-export): update CtaTemplate/BacktestingEngine imports to vn.py 4.0 layout 
vn.py 4.0 split every app (CTA strategy, CTA backtester, ...) into its own
pip-installable package. The vnpy-export skill's generated strategy template
and SKILL.md docs still used the pre-4.0 import path
(`vnpy.app.cta_strategy`), which no longer exists — a strategy file exported
by this skill would fail to import in any current vn.py install.

Verified against a real vn.py 4.0 + vnpy_ctastrategy install: the old import
raises ModuleNotFoundError; the corrected import resolves and the exported
MaCrossStrategy class loads successfully.

Signed-off-by: y85998607 <y85998607@gmail.com>
 2026-07-26 14:40:59 +08:00
Haozhe Wu 3470c0cb21 Merge pull request #859 from HKUDS/codex/fix-845-universe-metadata 
fix(alpha): disclose benchmark universe degradation
 2026-07-26 13:23:24 +08:00
Haozhe Wu 96e43b6e1d Merge pull request #858 from HKUDS/codex/fix-847-hash-lock 
fix(deps): repair cold hash-locked installs
 2026-07-26 13:23:01 +08:00
Haozhe Wu 0a690913c2 Merge pull request #852 from HKUDS/dependabot/npm_and_yarn/frontend/npm-minor-patch-3c68efc824 
build(deps): bump the npm-minor-patch group in /frontend with 5 updates
 2026-07-26 13:13:31 +08:00
Haozhe Wu 10d1b69aaf Merge pull request #851 from HKUDS/dependabot/github_actions/actions/setup-python-7.0.0 
build(deps): bump actions/setup-python from 6.3.0 to 7.0.0
 2026-07-26 13:13:08 +08:00
Haozhe Wu 453951ee1a Merge pull request #850 from HKUDS/dependabot/github_actions/actions-minor-patch-7a5a078ad4 
build(deps): bump actions/checkout from 7.0.0 to 7.0.1 in the actions-minor-patch group
 2026-07-26 13:12:37 +08:00
Haozhe Wu 2c1ccd7678 fix(alpha): disclose benchmark universe degradation 2026-07-26 13:03:57 +08:00
santhreal 5ed1eb5472 fix(journal): treat BTCUSDT-style symbols as crypto markets 
Concatenated pairs like BTCUSDT are alphabetic, so isalpha() labeled them
as US equities. Recognize USDT/USDC/BUSD suffixes before that branch.

Signed-off-by: santhreal <64453045+santhreal@users.noreply.github.com>
 2026-07-25 21:55:00 -07:00
Haozhe Wu 8aa7a13bfe fix(deps): repair cold hash-locked installs 2026-07-26 12:53:22 +08:00
santhreal 0a59f57f9e fix(journal): strip currency symbols before float coerce 
Broker CSV cells like $1,234.56 or ¥1000 failed float() after only commas
were stripped, so price/fee silently became 0.0. Strip common currency
tokens and unicode minus too.

Signed-off-by: santhreal <64453045+santhreal@users.noreply.github.com>
 2026-07-25 21:52:41 -07:00
santhreal 8a4f6260bb fix(tools): decode UTF-16 BOM text before latin-1 fallback 
Notepad/Excel Unicode text exports start with a UTF-16 BOM; latin-1 never
raises so those bytes were accepted as mojibake. Detect BOM and decode as
utf-16 first.

Signed-off-by: santhreal <64453045+santhreal@users.noreply.github.com>
 2026-07-25 21:50:16 -07:00
santhreal 34a9fd4302 fix(metrics): annualize lowercase 1h/1d and yahoo like loaders 
After loaders started accepting 1h/4h/1d, calc_bars_per_year still keyed
only 1H/1D, so hourly runs were annualised as daily (Sharpe/vol wrong).
Normalize interval case and alias yahoo→yfinance / binance→ccxt.

Signed-off-by: santhreal <64453045+santhreal@users.noreply.github.com>
 2026-07-25 21:48:25 -07:00
santhreal fe04869194 fix(skills): keep CJK characters in skill directory slugs 
Non-ASCII skill names such as 动量策略 were scrubbed to "--", so distinct
Chinese names collided into one directory and overwrote each other.
Allow the same non-Latin ranges as persistent memory, and hash when the
slug would otherwise be only hyphens.

Signed-off-by: santhreal <64453045+santhreal@users.noreply.github.com>
 2026-07-25 21:46:55 -07:00
santhreal b13243afd6 fix(journal): accept UTF-16 BOM CSV exports in load_dataframe 
Excel "CSV UTF-16" / Unicode CSV writes a BOM that utf-8/gbk cannot
decode, so analyze_trade_journal rejected the file. Try utf-16 in the
encoding fallback before raising.

Signed-off-by: santhreal <64453045+santhreal@users.noreply.github.com>
 2026-07-25 21:46:53 -07:00
dependabot[bot] d8accd822c build(deps): bump the npm-minor-patch group in /frontend with 5 updates 
Bumps the npm-minor-patch group in /frontend with 5 updates:

| Package | From | To |
| --- | --- | --- |
| [lucide-react](https://github.com/lucide-icons/lucide/tree/HEAD/packages/lucide-react) | `1.25.0` | `1.26.0` |
| [react](https://github.com/react/react/tree/HEAD/packages/react) | `19.2.7` | `19.2.8` |
| [react-dom](https://github.com/react/react/tree/HEAD/packages/react-dom) | `19.2.7` | `19.2.8` |
| [react-i18next](https://github.com/i18next/react-i18next) | `17.0.10` | `17.0.11` |
| [postcss](https://github.com/postcss/postcss) | `8.5.20` | `8.5.23` |


Updates `lucide-react` from 1.25.0 to 1.26.0
- [Release notes](https://github.com/lucide-icons/lucide/releases)
- [Commits](https://github.com/lucide-icons/lucide/commits/1.26.0/packages/lucide-react)

Updates `react` from 19.2.7 to 19.2.8
- [Release notes](https://github.com/react/react/releases)
- [Changelog](https://github.com/react/react/blob/main/CHANGELOG.md)
- [Commits](https://github.com/react/react/commits/v19.2.8/packages/react)

Updates `react-dom` from 19.2.7 to 19.2.8
- [Release notes](https://github.com/react/react/releases)
- [Changelog](https://github.com/react/react/blob/main/CHANGELOG.md)
- [Commits](https://github.com/react/react/commits/v19.2.8/packages/react-dom)

Updates `react-i18next` from 17.0.10 to 17.0.11
- [Changelog](https://github.com/i18next/react-i18next/blob/master/CHANGELOG.md)
- [Commits](https://github.com/i18next/react-i18next/compare/v17.0.10...v17.0.11)

Updates `postcss` from 8.5.20 to 8.5.23
- [Release notes](https://github.com/postcss/postcss/releases)
- [Changelog](https://github.com/postcss/postcss/blob/main/CHANGELOG.md)
- [Commits](https://github.com/postcss/postcss/compare/8.5.20...8.5.23)

---
updated-dependencies:
- dependency-name: lucide-react
  dependency-version: 1.26.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: npm-minor-patch
- dependency-name: react
  dependency-version: 19.2.8
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: npm-minor-patch
- dependency-name: react-dom
  dependency-version: 19.2.8
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: npm-minor-patch
- dependency-name: react-i18next
  dependency-version: 17.0.11
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: npm-minor-patch
- dependency-name: postcss
  dependency-version: 8.5.23
  dependency-type: direct:development
  update-type: version-update:semver-patch
  dependency-group: npm-minor-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
 2026-07-25 18:13:25 +00:00
dependabot[bot] ed61a2aaeb build(deps): bump actions/setup-python from 6.3.0 to 7.0.0 
Bumps [actions/setup-python](https://github.com/actions/setup-python) from 6.3.0 to 7.0.0.
- [Release notes](https://github.com/actions/setup-python/releases)
- [Commits](https://github.com/actions/setup-python/compare/ece7cb06caefa5fff74198d8649806c4678c61a1...5fda3b95a4ea91299a34e894583c3862153e4b97)

---
updated-dependencies:
- dependency-name: actions/setup-python
  dependency-version: 7.0.0
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
 2026-07-25 18:12:37 +00:00
dependabot[bot] 422d2f918c build(deps): bump actions/checkout in the actions-minor-patch group 
Bumps the actions-minor-patch group with 1 update: [actions/checkout](https://github.com/actions/checkout).


Updates `actions/checkout` from 7.0.0 to 7.0.1
- [Release notes](https://github.com/actions/checkout/releases)
- [Changelog](https://github.com/actions/checkout/blob/main/CHANGELOG.md)
- [Commits](https://github.com/actions/checkout/compare/9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0...3d3c42e5aac5ba805825da76410c181273ba90b1)

---
updated-dependencies:
- dependency-name: actions/checkout
  dependency-version: 7.0.1
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: actions-minor-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
 2026-07-25 18:12:34 +00:00
Haozhe Wu 93d666c5e6 docs(readme): add 2026-07-25 news across all 5 READMEs, keep 3 visible entries 2026-07-25 23:01:23 +08:00
Gordon Lee 84eff84a6a fix(backtest): consume historical USD-M funding + round settlement jitter (#819) 
Two fixes that make historical funding actually flow into perp backtests:

1. Loader: Binance funding settlement timestamps carry millisecond
   jitter (e.g. 2026-05-05T08:00:00.011Z). The raw timestamps never
   intersect bar timestamps, so the missing-settlement guard raises
   'funding settlement data is missing' for most real fetches. Round
   settlement timestamps to the second before aligning.

2. Engine: the loader writes a historical funding_rate column into the
   frame, but calc_crypto_funding_fee still charges the fixed config
   rate (default 0.0001) — the fetched data is never consumed. Prefer
   the bar's historical rate when present and finite; fall back to the
   fixed rate otherwise (spot-proxy and daily-fallback paths keep
   their existing behaviour).

Verified against live Binance USD-M data (BTC/ETH -USDT-PERP, 4H,
Jan-Jul 2026): fixed-rate short returned +44.5%, historical-funding
short +39.1% — the ~5pp delta matches 2026's predominantly negative
funding regime where shorts pay.

Tests: jitter alignment test in test_ccxt_perpetual_loader.py; three
override/fallback tests in test_crypto_engine.py.

AI-assisted-by: Claude (Anthropic)

Signed-off-by: Gordon Lee <xnegra80@gmail.com>
 2026-07-25 22:54:00 +08:00
Amir Fathi 283752d908 fix(cli): forward universe meta into alpha bench JSON/HTML output (#841) 
bench_runner.run_bench forwards the sp500 loader's survivorship_bias flag
as result["meta"] (bench_runner.py L319-347), and
alpha_routes._result_for_wire already keeps that key for the SSE/frontend
path (alpha_routes.py L777). cmd_alpha_bench built its own JSON envelope
and HTML report context from hand-enumerated dict literals that never
included "meta", so the CLI and the HTML report silently dropped the
disclosure while the web UI kept it.

Forward result.get("meta") into both dict literals when present, mirroring
how the wire path already treats it: an opaque passthrough, not a new
rendered field. Verified with a fresh regression test
(agent/tests/test_alpha_bench_meta_survivorship.py) that fails on main and
passes on this branch, plus a third case asserting the key stays absent
when the loader supplies no meta.

Fixes #797

Signed-off-by: Amir Fathi <amirfathi.me@gmail.com>
 2026-07-25 22:53:57 +08:00
ShaunMandShaunM b2bac34888 fix(mcp): handle dataclass tool results in _make_jsonable (#849) 
_make_jsonable only converted Pydantic BaseModel instances (via
model_dump), list, and dict. fastmcp's client auto-types
CallToolResult.data from a tool's output schema via
json_schema_to_type, which produces a plain stdlib dataclass (not a
BaseModel) whenever the schema doesn't set additionalProperties: true
-- the common case for a typical Pydantic-model tool return type. Such
an instance fell through to the final `return value` unconverted, so
json.dumps's `default` callback received the same unconvertible object
back and raised ValueError: Circular reference detected (its cycle
detector can't distinguish "default() gave up" from an actual cycle).

Adds a dataclasses.is_dataclass branch that recurses per-field (not a
raw asdict(), which would leave nested datetime/dataclass values
unconverted), plus a datetime/date branch, since a date-time schema
field validates to a real datetime object and would hit the same
failure one level down.

Reproduced against the real fastmcp dependency using an MCP tool's
output schema shaped like this (plain object properties, no
additionalProperties, nested datetime field): old _make_jsonable raises
the reported Circular reference detected error; the patched version
serializes correctly, verified both from direct construction and via
pydantic.TypeAdapter validation (the path fastmcp's client actually
uses).

Signed-off-by: ShaunM <echoandelementwebsites@gmail.com>
Co-authored-by: ShaunM <echoandelementwebsites@gmail.com>
 2026-07-25 22:53:54 +08:00
Hongi Nagaputra c5540a92e3 feat(backtest): define perpetual margin state contracts (#798) 
Pure, immutable state and risk contracts for historical USD-M perpetual
backtests: MaintenanceBracket/MaintenanceSchedule, PositionState,
AccountState, ExecutionFrame, MarketRiskFrame. No engine wiring yet — that's
a later slice.

Reapplied against the current data contract rather than cherry-picked from
the old codex/perp-engine-fidelity branch, which predates #757 and still
carries the pre-#470 loader shape. Two real changes from that branch's
version:

- MaintenanceBracket gains bracket_tier (ordering) and optional
  notional_coefficient, matching the loader's validated artifact tier
  fields exactly (backtest.loaders.ccxt_loader._validate_bracket_artifact).
- MaintenanceSchedule drops its own independent SHA-256 hash computation
  in favor of a single caller-supplied version string — the loader's
  maintenance_bracket_version content hash. Content integrity is already
  checked once, at the artifact-validation boundary; this class only
  re-checks structural invariants (tier and notional-cap ordering) as
  defense in depth, not a second, incompatible hash scheme.
  MaintenanceSchedule.from_loader_columns(symbol, maintenance_brackets,
  maintenance_bracket_version) builds a schedule straight from the
  loader's DataFrame columns.

MarketRiskFrame.schedule is Optional: a -PERP fetch without a supplied
bracket artifact still produces a usable execution/mark/funding frame.

Part of #462's PR 2 (margin-risk model), sliced small per maintainer
guidance. Isolated/cross liquidation logic and engine integration are
separate follow-up slices.

Signed-off-by: zygora <122336305+honginp@users.noreply.github.com>
 2026-07-25 22:31:37 +08:00
Santh 3f7e9a1c2a fix(tools): keep empty cells when parsing report_audit tables (#807) 
Dropping blank cells shifted later values onto earlier column headers.

Signed-off-by: santhreal <64453045+santhreal@users.noreply.github.com>
 2026-07-25 22:31:32 +08:00
卫斯李 6da76b7e01 fix(cli): show account label in broker_sdk balances view (#843) 
broker_sdk connectors (Longbridge, etc.) return a balances list but no
accounts key, causing the CLI to print "Accounts: (none)" before the
balances table. Fall back to result.profile_id — which _with_profile()
always injects — so the label is informative.

Closes #846

Signed-off-by: Robin1987China <41602358+Robin1987China@users.noreply.github.com>
 2026-07-25 22:31:28 +08:00
Santh e109ae69c8 fix(channels): keep empty edge columns in Slack md tables (#801) 
str.strip('|') dropped leading/trailing empty cells, so Slack table
conversion mislabeled row-label columns and dropped trailing cell values.

Signed-off-by: santhreal <64453045+santhreal@users.noreply.github.com>
 2026-07-25 22:31:25 +08:00
Santh 8374fc4cf1 fix(channels): keep empty edge columns in Signal md tables (#800) 
str.strip('|') dropped leading/trailing empty cells in pipe tables, so
row-label columns and trailing blanks collapsed when rendering for Signal.

Signed-off-by: santhreal <64453045+santhreal@users.noreply.github.com>
 2026-07-25 22:31:21 +08:00
Santh 1b6420d804 fix(channels): keep empty edge columns in Telegram md tables (#799) 
str.strip('|') dropped leading/trailing empty cells in pipe tables, so
row-label columns and trailing blanks collapsed when rendering for Telegram.

Signed-off-by: santhreal <64453045+santhreal@users.noreply.github.com>
 2026-07-25 22:31:17 +08:00
Santh e04c08bf79 fix(connectors): map MT5 1H/4H/1D periods to hour/day bars (#806) 
Project-style period tokens were falling through to daily TIMEFRAME_D1.

Signed-off-by: santhreal <64453045+santhreal@users.noreply.github.com>
 2026-07-25 22:29:40 +08:00
Santh 96f7bc58b4 fix(connectors): map Binance 1H/4H periods to ccxt timeframes (#805) 
Project-style hour tokens were passed raw to ccxt, which expects lowercase unified timeframes.

Signed-off-by: santhreal <64453045+santhreal@users.noreply.github.com>
 2026-07-25 22:29:37 +08:00
Santh 7b6d219a53 fix(loaders): keep lowercase 1h/4h/1w as ccxt hour/week bars (#804) 
* fix(loaders): keep lowercase 1h/4h/1w as ccxt hour/week bars

Lowercase project tokens previously missed _INTERVAL_MAP and fell through to silent daily bars.

Signed-off-by: santhreal <64453045+santhreal@users.noreply.github.com>

* style(test): drop narrating comment on ccxt interval aliases

Signed-off-by: santhreal <64453045+santhreal@users.noreply.github.com>

---------

Signed-off-by: santhreal <64453045+santhreal@users.noreply.github.com>
 2026-07-25 22:29:33 +08:00
Santh 457ebebdaa fix(connectors): reject unsupported Dhan history periods (#803) 
Unknown periods such as 1H were silently served as daily candles. Fail
closed with an explicit unsupported-period error instead.

Signed-off-by: santhreal <64453045+santhreal@users.noreply.github.com>
 2026-07-25 22:29:29 +08:00
Santh 4e8271a0b9 fix(connectors): map Futu 1H/4H periods to hour bars (#802) 
Project-style 1H fell through to daily K lines, and 4h incorrectly mapped
to K_60M instead of K_240M.

Signed-off-by: santhreal <64453045+santhreal@users.noreply.github.com>
 2026-07-25 22:29:26 +08:00
Santh dfdbefd0ea fix(journal): normalize Excel serials in generic datetime columns (#810) 
Generic CSV/XLSX loads stringify serial day numbers, which then failed
to parse and became NaT in the standardized journal frame.

Signed-off-by: santhreal <64453045+santhreal@users.noreply.github.com>
 2026-07-25 22:28:54 +08:00
Santh b71427a829 fix(journal): normalize stringified Excel serials in Futu dates (#809) 
load_dataframe reads CSV/XLSX as strings, so Date serials never hit the
numeric Excel path and stayed as raw "45321.0".

Signed-off-by: santhreal <64453045+santhreal@users.noreply.github.com>
 2026-07-25 22:27:57 +08:00
Santh ebc1ff8d85 fix(journal): normalize stringified Excel serials in Tonghuashun dates (#808) 
load_dataframe reads CSV/XLSX as strings, so serial day numbers never
hit the numeric Excel path and stayed as raw "45321.375".

Signed-off-by: santhreal <64453045+santhreal@users.noreply.github.com>
 2026-07-25 22:27:08 +08:00
Haozhe Wu 65bab10f32 docs(readme): add 2026-07-24 news across all 5 READMEs 2026-07-24 22:46:19 +08:00
shadowinlifeandQoder b9507ee12b feat(memory): implement Tier 2 structural organization (#815) 
* feat(memory): implement Tier 2 structural organization

Add four independent modules for memory structural organization:
- H-MEM hierarchical directory routing (hierarchy.py)
- A-MEM semantic linking via BM25 (semantic_links.py)
- Auto-compression pipeline via TF-IDF (compression.py)
- FTS5 full-text search index (search_index.py)

Integration:
- 4 feature flags in MemoryConfig (default: disabled)
- MemoryEntry extended with category + compression_level fields
- persistent.py wired with hierarchy routing, FTS5 search, link expansion
- lifecycle.py triggers compression in GC cycle
- remember_tool.py attaches related links in recall results
- FTS5 CJK bigram+unigram indexing for better Chinese recall
- Lazy auto-rebuild on first FTS5 search

Testing:
- 163 unit tests + 9 integration tests + 23 benchmarks (all passing)
- Benchmark evaluated on 200 real production memories + 50 labeled queries
- Compression retention: daily 91.4%, digest 97.1%
- Hierarchy routing: 93.3% P@5 parity with full scan

Closes #732 (Tier 2 scope)

Signed-off-by: shadowinlife <shadowinlife@gmail.com>

Co-Authored-By: Qoder <noreply@qoder.com>
AI-Contributed/Feature: 1582/1871
AI-Contributed/UT: 2123/2159

* feat(memory): add VT_MEMORY=off|on|full business-friendly preset

Simplify memory configuration from 7 individual boolean flags to a
single business-friendly environment variable:
  - off:  all features disabled (default, backward compatible)
  - on:   Tier 1 quality + decay + garbage collection
  - full: Tier 1 + Tier 2 hierarchy + links + compression + FTS

Individual VT_MEMORY_* flags still work as explicit overrides with
higher priority than the preset baseline.

Also adds a dependency-conflict warning in lifecycle.py when
compression is enabled without GC.

Signed-off-by: shadowinlife <shadowinlife@gmail.com>

AI-Contributed/Feature: 0/138
AI-Contributed/UT: 0/0

* docs: add Memory Tier 2 and VT_MEMORY preset to CHANGELOG

Document the new structural organization features (hierarchy, semantic
links, compression, FTS5) and the business-friendly VT_MEMORY=off|on|full
configuration preset.

Signed-off-by: shadowinlife <shadowinlife@gmail.com>

AI-Contributed/Feature: 0/15
AI-Contributed/UT: 0/0

---------

Co-authored-by: Qoder <noreply@qoder.com>
 2026-07-24 22:13:16 +08:00
Yufeng He 97083c618b feat(backtest): composable weight constraints for any optimizer (#818) 
Portfolio Studio step 2 (#456): per-name cap, per-name floor, and per-group exposure caps as a shared layer on top of every optimizer, not just turnover_aware's built-in bounds. Configured via constraints in config.json, off by default.

Signed-off-by: Yufeng He <40085740+he-yufeng@users.noreply.github.com>
 2026-07-24 22:13:10 +08:00
ALI EMADIAli EmadiClaude Fable 5
 		dffa8fe03f Add allow_nonpositive_prices: open on negative-price bars, still reject zero (#816) 
Markets like European day-ahead power clear negative routinely, but the
loader's validate_ohlc drops any bar with a non-positive OHLC value and the
engine rejects any open at price <= 0. On real EU power data this silently
deletes ~27% of DE-LU days (every bar with a sub-zero hourly-derived low),
punching holes in rolling statistics (e.g. a z-score entry signal) with no
warning.

Add an opt-in `allow_nonpositive_prices` flag (default False — existing
behavior byte-identical for all positive-price markets):

- loaders/base.py: validate_ohlc gains the flag. Structural invariants
  (high<low, high/low bracketing open/close) always enforced; when the flag
  is set, negative prices pass through and only an exactly-zero price is
  rejected. Zero is genuinely undefined for `size = notional / price` and
  margin; a negative price is well-defined via abs().
- engines/base.py: BaseEngine reads the flag from config. _plan_open_order
  rejects an exactly-zero open always, negatives only when the flag is off.
  _calc_raw_size / _calc_margin divide by abs(price) so a negative entry
  price yields a positive size magnitude (direction carries the sign) and
  positive collateral — unchanged for positive prices. _calc_pnl already
  uses a price difference, well-defined through zero.

Tests open a position on a negative-price bar and assert size/margin/PnL are
well-defined through zero, that an exactly-zero price is still rejected with
the flag on, and that the default still rejects negatives. A regression test
pins *why* zero stays dropped rather than kept: a surviving 0.00 close yields
inf at the `close.pct_change().fillna(0.0)` sites in benchmark.py and
engines/base.py (fillna fills NaN, not inf), so keeping it would corrupt the
return series. Fixture carries the 5 real non-positive-close bars from a 2025
NO2/DE-LU window (ENTSO-E via Energy-Charts / Bundesnetzagentur | SMARD,
CC BY 4.0). Existing engine and OHLC-validation suites pass unchanged
(140 tests total).

Co-authored-by: Ali Emadi <ali@gatekeeperai.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
 2026-07-24 22:13:05 +08:00
Santh 830122cffa fix(journal): normalize Excel day-count serials in Eastmoney dates (#811) 
load_dataframe reads cells as strings, so day-count serials like
"44941.0" skipped the YYYYMMDD float path and stayed as raw date text.

Signed-off-by: santhreal <64453045+santhreal@users.noreply.github.com>
 2026-07-24 22:12:59 +08:00
Santh e5a99c778e fix(loaders): map Yahoo 4H to 1h like yfinance (#812) 
Yahoo chart has no 4h interval; project 4H was lowercased to 4h and
sent unsupported to the API. Match yfinance's 4H → 1h approximation.

Signed-off-by: santhreal <64453045+santhreal@users.noreply.github.com>
 2026-07-24 22:12:55 +08:00
Santh 6d9079c162 fix(loaders): reject unsupported BaoStock history intervals (#813) 
Docs say daily-only, but 1H/4H still logged in and returned frequency=d
bars. Reject non-daily intervals before login (tencent/fmp parity).

Signed-off-by: santhreal <64453045+santhreal@users.noreply.github.com>
 2026-07-24 22:12:50 +08:00
Santh 9fdb316f3a fix(loaders): reject unsupported Alpha Vantage history intervals (#814) 
TIME_SERIES_DAILY is daily-only, but 1H/4H still hit the API and
returned day bars. Reject non-daily intervals before the request.

Signed-off-by: santhreal <64453045+santhreal@users.noreply.github.com>
 2026-07-24 22:12:43 +08:00
Santh 960f595f31 fix(loaders): reject unsupported Finnhub history intervals (#823) 
Finnhub always called resolution=D even for 1H/4H, so intraday runs
silently got day bars and stopped the fallback chain.

Reject unsupported intervals like the other daily-only loaders.

Signed-off-by: santhreal <64453045+santhreal@users.noreply.github.com>
 2026-07-24 22:12:38 +08:00
Santh bcdaaf5e8f fix(loaders): reject unsupported Tiingo history intervals (#824) 
Tiingo always hit the daily EOD endpoint even for 1H/4H, so intraday
runs silently got day bars and stopped the fallback chain.

Reject unsupported intervals like the other daily-only loaders.

Signed-off-by: santhreal <64453045+santhreal@users.noreply.github.com>
 2026-07-24 22:12:33 +08:00
Santh eac1e92755 fix(loaders): reject unsupported Stooq history intervals (#825) 
Stooq always fetched the daily CSV even for 1H/4H, so intraday runs
silently got day bars and stopped the fallback chain.

Reject unsupported intervals like tencent/fmp/india-broker.

Signed-off-by: santhreal <64453045+santhreal@users.noreply.github.com>
 2026-07-24 22:12:28 +08:00
Santh 4ab478cb56 fix(loaders): map OKX lowercase 1h/4h instead of silent daily (#826) 
Connector-style 1h/4h fell through the uppercase-only allowlist and were
substituted with 1D. Map the aliases like the OKX connector bar map, and
reject unknown intervals instead of silently fetching daily bars.

Signed-off-by: santhreal <64453045+santhreal@users.noreply.github.com>
 2026-07-24 22:12:24 +08:00
Santh e14e627d9b fix(loaders): treat QVeris 1m/5m/15m/30m as intraday granularity (#827) 
Lowercasing first left 1m/5m/15m/30m with empty wanted tokens, so daily
OHLCV capabilities outranked minute ones. Match minute tokens
case-sensitively and keep 1M as monthly.

Signed-off-by: santhreal <64453045+santhreal@users.noreply.github.com>
 2026-07-24 22:12:17 +08:00
Santh a4e9222235 fix(loaders): reject non-daily intervals for AKShare US/HK/ETF/forex (#828) 
* fix(loaders): reject non-daily intervals for AKShare US/HK/ETF/forex

A-share already raised on unsupported intervals, but US/HK/ETF/forex
always called the daily endpoints, so interval=1H returned day bars
under a 1H cache key and stopped the fallback chain.

Signed-off-by: santhreal <64453045+santhreal@users.noreply.github.com>

* fix(loaders): tighten AKShare daily-only interval gate comment

---------

Signed-off-by: santhreal <64453045+santhreal@users.noreply.github.com>
 2026-07-24 22:12:12 +08:00
Santh c3b284055f fix(loaders): accept lowercase 1h/1d in India broker interval map (#829) 
Connector-style 1h/1d were rejected while project 1H/1D mapped to the
same broker tokens, so market_data and mixed-vocabulary callers got
empty frames for valid hour/day bars.

Signed-off-by: santhreal <64453045+santhreal@users.noreply.github.com>
 2026-07-24 22:12:07 +08:00
Santh bca8ecf91b fix(loaders): accept lowercase 1h/4h/1d in Futu interval map (#830) 
Connector-style hour/day tokens raised unsupported while 1H/4H/1D
already mapped to K_60M/K_240M/K_DAY, so mixed-vocabulary callers lost
intraday Futu history.

Signed-off-by: santhreal <64453045+santhreal@users.noreply.github.com>
 2026-07-24 22:12:02 +08:00
Santh be32143bc4 fix(loaders): accept daily aliases in FMP interval gate (#831) 
* fix(loaders): accept daily aliases in FMP interval gate

The gate compared interval != \"1D\" exactly, so connector-style 1d/day
returned empty while sina/tencent treat those as daily.

Signed-off-by: santhreal <64453045+santhreal@users.noreply.github.com>

* fix(loaders): drop narrating FMP daily-alias gate comment

---------

Signed-off-by: santhreal <64453045+santhreal@users.noreply.github.com>
 2026-07-24 22:11:57 +08:00
Santh ae1cb2d19e fix(loaders): accept lowercase 1h/4h in local resample rules (#832) 
Connector-style 1h/4h missed _RESAMPLE_RULES and returned native bars
unchanged, while 1H/4H correctly downsampled.

Signed-off-by: santhreal <64453045+santhreal@users.noreply.github.com>
 2026-07-24 22:11:52 +08:00
Santh c4e73777fe fix(loaders): map yfinance lowercase 4h/1w like 4H/1W (#833) 
Lowercase 4h/1w fell through to .lower() as invalid Yahoo tokens while 4H/1W already mapped to 1h/1wk.

Signed-off-by: santhreal <64453045+santhreal@users.noreply.github.com>
 2026-07-24 22:11:47 +08:00
Santh 211baefbf9 fix(loaders): accept lowercase 1d/1w in Longbridge interval map (#834) 
Loader rejected 1d/1w while the connector already maps them to Day/Week.

Signed-off-by: santhreal <64453045+santhreal@users.noreply.github.com>
 2026-07-24 22:11:43 +08:00
Santh 32290a22c8 fix(loaders): map MT5 1W/1M and reject unknown intervals (#835) 
* fix(loaders): map MT5 1W/1M and reject unknown intervals

Unknown intervals silently fell back to daily bars; week/month tokens missed the map while the connector already exposes W1/MN1.

Signed-off-by: santhreal <64453045+santhreal@users.noreply.github.com>

* fix(loaders): tighten MT5 unknown-interval reject comment

---------

Signed-off-by: santhreal <64453045+santhreal@users.noreply.github.com>
 2026-07-24 22:11:39 +08:00
Santh 7d4767c96d fix(loaders): accept lowercase 1d/1w in AKShare daily map (#836) 
1d/1w missed the daily period map while 1D/1W already mapped to daily/weekly.

Signed-off-by: santhreal <64453045+santhreal@users.noreply.github.com>
 2026-07-24 22:11:33 +08:00
Santh 6c743bb4d2 fix(loaders): treat lowercase 1d as Tushare daily bars (#837) 
* fix(loaders): treat lowercase 1d as Tushare daily bars

interval=1d missed the exact 1D daily gate and fell into the minute path, which returned empty.

Signed-off-by: santhreal <64453045+santhreal@users.noreply.github.com>

* fix(loaders): tighten Tushare daily-alias gate comment

---------

Signed-off-by: santhreal <64453045+santhreal@users.noreply.github.com>
 2026-07-24 22:11:29 +08:00
Santh 6d5c1678a6 fix(loaders): accept lowercase 1d/1w in Eastmoney KLT map (#838) 
1d/1w missed KLT_BY_INTERVAL while 1D/1W already mapped to daily/weekly codes.

Signed-off-by: santhreal <64453045+santhreal@users.noreply.github.com>
 2026-07-24 22:11:24 +08:00
Dvir Arad 170722e31b docs: fix broken Contributing nav anchor in README (#840) 
The header nav links to #-contributing, but the 'Contributing' heading
has no leading emoji, so GitHub renders its anchor as #contributing (no
leading hyphen). Clicking 'Contributing' in the nav therefore does not
scroll to the section. README_ja.md already uses the correct #contributing
target; align README.md and README_zh.md with it.

Signed-off-by: dvir arad <dvirarad@gmail.com>
 2026-07-24 22:10:46 +08:00
Haozhe Wu 43c444292c docs(readme): add 2026-07-23 news across all 5 READMEs 2026-07-23 13:44:46 +08:00
Haozhe Wu 27e6dbde0f Merge PR #796: feat(alpha): expose the strict bench gate via alpha bench --strict 
feat(alpha): expose the strict bench gate via alpha bench --strict
 2026-07-23 11:59:55 +08:00
Haozhe Wu 33448d7d7a Merge PR #733: feat(memory): implement Phase 1 quality-driven lifecycle management 
feat(memory): implement Phase 1 quality-driven lifecycle management
 2026-07-23 11:39:34 +08:00
Haozhe Wu 71ed54d91f Merge PR #794: fix(journal): treat numpy int serials as Excel days in Tonghuashun 
fix(journal): treat numpy int serials as Excel days in Tonghuashun
 2026-07-23 11:24:05 +08:00
Haozhe Wu f1ed0cbba6 Merge PR #793: fix(journal): treat numpy int serials as Excel days in Futu Date 
fix(journal): treat numpy int serials as Excel days in Futu Date
 2026-07-23 11:23:59 +08:00
Haozhe Wu f4c5a78223 Merge PR #778: fix(journal): normalize Excel float YYYYMMDD dates in eastmoney parser 
fix(journal): normalize Excel float YYYYMMDD dates in eastmoney parser
 2026-07-23 11:23:54 +08:00
Haozhe Wu 6e4b5aedb4 Merge PR #795: feat(backtest): rebalance notes artifact + metrics (Portfolio Studio slice) 
feat(backtest): rebalance notes artifact + metrics (Portfolio Studio slice)
 2026-07-23 11:23:50 +08:00
Haozhe Wu e6148b0938 Merge PR #792: fix(loaders): accept lowercase 1h/4h/1d MT5 intervals 
fix(loaders): accept lowercase 1h/4h/1d MT5 intervals
 2026-07-23 11:23:46 +08:00
Haozhe Wu e10fd912dd Merge PR #791: fix(longbridge): accept project-style 1H/4H period tokens 
fix(longbridge): accept project-style 1H/4H period tokens
 2026-07-23 11:23:42 +08:00
Haozhe Wu 599e714a7e Merge PR #790: fix(shoonya): map 4h/1H periods and reject unknown intervals 
fix(shoonya): map 4h/1H periods and reject unknown intervals
 2026-07-23 11:23:39 +08:00
Haozhe Wu 5ad54fd16d Merge PR #789: fix(connectors): map OKX 1H/4H instead of falling to 1D 
fix(connectors): map OKX 1H/4H instead of falling to 1D
 2026-07-23 11:23:35 +08:00
Haozhe Wu 2cb4f90f71 Merge PR #788: fix(connectors): keep Alpaca 1H as hour bars, not day 
fix(connectors): keep Alpaca 1H as hour bars, not day
 2026-07-23 11:23:30 +08:00
Haozhe Wu 2db128c862 Merge PR #787: fix(cli): keep empty edge columns in terminal md tables 
fix(cli): keep empty edge columns in terminal md tables
 2026-07-23 11:23:27 +08:00
Haozhe Wu ceaa899487 Merge PR #786: fix(channels): keep empty edge columns in Feishu md tables 
fix(channels): keep empty edge columns in Feishu md tables
 2026-07-23 11:23:23 +08:00
Haozhe Wu 2e82d8e2f6 Merge PR #785: fix(loaders): reject unsupported akshare intervals instead of daily 
fix(loaders): reject unsupported akshare intervals instead of daily
 2026-07-23 11:23:19 +08:00
Haozhe Wu f359b71154 Merge PR #784: fix(connectors): keep Tiger 1H/4H as hour bars, not day 
fix(connectors): keep Tiger 1H/4H as hour bars, not day
 2026-07-23 11:23:15 +08:00
Haozhe Wu f10cb68660 Merge PR #783: fix(loaders): map CCXT 1W/1M intervals instead of silent daily 
fix(loaders): map CCXT 1W/1M intervals instead of silent daily
 2026-07-23 11:23:12 +08:00
Haozhe Wu 17414f7415 Merge PR #782: fix(loaders): keep yfinance 1M as monthly, not minute 
fix(loaders): keep yfinance 1M as monthly, not minute
 2026-07-23 11:23:08 +08:00
Haozhe Wu bebdb2dc61 Merge PR #781: fix(loaders): reject unsupported india-broker intervals 
fix(loaders): reject unsupported india-broker intervals
 2026-07-23 11:23:04 +08:00
Haozhe Wu f401c3640a Merge PR #780: fix(validation): accept blank holding_days in trades.csv 
fix(validation): accept blank holding_days in trades.csv
 2026-07-23 11:23:01 +08:00
Haozhe Wu 7cf43195ff Merge PR #779: fix(tools): keep report_audit verdict JSON finite 
fix(tools): keep report_audit verdict JSON finite
 2026-07-23 11:22:58 +08:00
Haozhe Wu fec31c1ce5 Merge PR #776: fix(mt5): convert numpy scalars to native Python types for JSON serialization 
fix(mt5): convert numpy scalars to native Python types for JSON serialization
 2026-07-23 11:22:54 +08:00
Haozhe Wu 08c84e8f0f Merge PR #772: fix(fundamentals): dedup restated rows and prevent period regression in PIT snapshots 
fix(fundamentals): dedup restated rows and prevent period regression in PIT snapshots
 2026-07-23 11:22:51 +08:00
Yufeng He 23174f28b7 feat(alpha): expose the strict bench gate via alpha bench --strict 
The strict bench runner (mandatory same-universe random control plus an
optional train/test OOS split) has shipped since 0.1.9 but nothing
invoked it: no CLI flag, no API route, no tool reference, so the only
gate users could run was the raw-IC one its own docstring critiques.
This wires it into the CLI: --strict routes the bench through
bench_runner_strict with the random control on, --oos-split and
--random-seeds tune the gate, the pre-flight banner shows the mode, and
the JSON envelope carries the strict category counts. Passing
--oos-split/--random-seeds without --strict is rejected up front so
they cannot silently no-op.

Fixes #773.

Signed-off-by: Yufeng He <40085740+he-yufeng@users.noreply.github.com>
 2026-07-23 11:06:34 +08:00
Yufeng He a6a57b2562 feat(backtest): rebalance notes artifact + metrics (Portfolio Studio slice) 
Last backend slice of the Portfolio Studio epic: per-rebalance turnover
and weight-drift notes computed from the target position frame, so they
land for every optimizer and for the no-optimizer baseline. Writes
artifacts/rebalance_notes.json (strict JSON, same writer discipline as
validation.json) and a Markdown twin, and surfaces rebalance_count /
rebalance_turnover_mean / rebalance_turnover_max as scalar metrics that
the run card picks up on its own.

Signed-off-by: Yufeng He <40085740+he-yufeng@users.noreply.github.com>
 2026-07-23 07:06:56 +08:00
santhreal 0f7983e595 fix(journal): treat numpy int serials as Excel days in Futu Date 
iterrows yields np.int64 for int64 Date columns; use is_number so Excel
day serials convert instead of stringifying or ns-epoch parsing.

Signed-off-by: santhreal <64453045+santhreal@users.noreply.github.com>
 2026-07-22 12:44:46 -07:00
santhreal b11ccc7c8b fix(journal): treat numpy int serials as Excel days in Tonghuashun 
iterrows yields np.int64 for int64 成交时间 columns; isinstance(int)
missed them and bare pd.to_datetime treated the value as ns-epoch.

Signed-off-by: santhreal <64453045+santhreal@users.noreply.github.com>
 2026-07-22 12:44:43 -07:00
santhreal ae435887ee fix(journal): normalize Tonghuashun Excel serial 成交时间 to ISO 
同花顺 exports opened via Excel leave 成交时间 as a float serial; stringifying
it produced values like "45321.375" instead of a real timestamp. Convert
Excel day serials with pd.to_datetime before storing datetime.

Signed-off-by: santhreal <64453045+santhreal@users.noreply.github.com>
 2026-07-22 12:39:35 -07:00
santhreal b7b942fb1d fix(journal): normalize Futu Excel serial Date/Time to ISO 
Futu exports opened via Excel leave Date/Time as float serials; stringifying
them produced values like "45321.0 0.375" instead of a real timestamp.
Convert Excel day serials with pd.to_datetime before building datetime.

Signed-off-by: santhreal <64453045+santhreal@users.noreply.github.com>
 2026-07-22 12:39:33 -07:00
santhreal 98f8544ac6 fix(loaders): accept lowercase 1h/4h/1d MT5 intervals 
The MT5 loader only listed 1H/4H/1D, so trading-convention 1h fell through
to the daily fallback while the MT5 connector already maps 1h to H1.
Add the lowercase aliases.

Signed-off-by: santhreal <64453045+santhreal@users.noreply.github.com>
 2026-07-22 12:39:32 -07:00
santhreal 64b751503e fix(longbridge): accept project-style 1H/4H period tokens 
Connector _candlestick_enums only listed lowercase 1h/4h, so runner-style
1H fell through to Day while the Longbridge loader already maps 1H to
Min_60. Align the connector map with the loader.

Signed-off-by: santhreal <64453045+santhreal@users.noreply.github.com>
 2026-07-22 12:39:30 -07:00
santhreal 5e5c5bb7de fix(shoonya): map 4h/1H periods and reject unknown intervals 
Documented connector periods 4h/1H fell through to daily D. Map them to
Noren 240/60 minute candles and return an error for unsupported tokens
such as 1w/1M instead of silently substituting daily bars.

Signed-off-by: santhreal <64453045+santhreal@users.noreply.github.com>
 2026-07-22 12:39:29 -07:00
santhreal a30271ddf9 merge(main): sync before thermo hold 2026-07-22 12:21:58 -07:00
santhreal 2dddec63f4 merge(main): sync before thermo hold 2026-07-22 12:21:57 -07:00
santhreal 9d05d5856e merge(main): sync before thermo hold 2026-07-22 12:21:31 -07:00
santhreal c2da481bff merge(main): sync before thermo hold 2026-07-22 12:21:29 -07:00
santhreal 98038eeae7 merge(main): sync before thermo hold 2026-07-22 12:21:27 -07:00
santhreal 74cdab3b28 merge(main): sync before thermo hold 2026-07-22 12:21:26 -07:00
santhreal a88de6caab merge(main): sync before thermo hold 2026-07-22 12:21:24 -07:00
santhreal 805a75713c merge(main): sync before thermo hold 2026-07-22 12:21:23 -07:00
santhreal 5432364770 merge(main): sync before thermo hold 2026-07-22 12:21:21 -07:00
santhreal 298e3ec1d0 merge(main): sync before thermo hold 2026-07-22 12:21:20 -07:00
santhreal c671be6fba merge(main): sync before thermo hold 2026-07-22 12:21:18 -07:00
santhreal 87cab5ffa5 merge(main): sync before thermo hold 2026-07-22 12:21:15 -07:00
Haozhe Wu ef04bee26c Merge PR #777: feat(backtest): portfolio risk x-ray core + agent tool (Portfolio Studio slice) 
feat(backtest): portfolio risk x-ray core + agent tool (Portfolio Studio slice)
 2026-07-22 23:30:59 +08:00
Haozhe Wu 329b054128 docs(readme): backfill longbridge data-source parity in ja/ar 
ja/ar were missing the longbridge Data Sources table row and its place in
the US/HK fallback chains (a pre-0.1.12 drift; en/ko/zh already had them).
Now all 5 locales agree on the 23-source table + chains. (The en-only
'Using Longbridge explicitly' config subsection remains en-only.)
 2026-07-22 23:19:10 +08:00
Haozhe Wu 761d52155c docs(readme): sync all 5 READMEs to v0.1.12 features + counts 
News: 🚀 v0.1.12 entry (3-visible, demote 07-19). Counts synced to code:
462 alphas (academic 12), 88 skills, 23 data sources, 12 broker connectors.
Data Sources: add binance + mt5 rows + forex/metals fallback chain. New
Broker Connectors (12) and Backtest Engines (8) capability tables. Providers
list + native Anthropic / SiliconFlow / NVIDIA. Roadmap: SDM + Correlation
Regime rows. en/ja/ko English descriptions; zh/ar translated.
 2026-07-22 22:47:39 +08:00
Yufeng He 9889a74320 feat(backtest): portfolio risk x-ray core + agent tool (Portfolio Studio slice) 
First slice of the Portfolio Studio roadmap item ("risk x-ray ... rest
Planned"): a pure-computation risk x-ray for weighted baskets, plus an
agent-facing tool that fetches closes through the standard loader
fallback chain.

backtest/risk_xray.py computes, over trailing returns only (no look-ahead
by construction): concentration (HHI / effective N / top-1 / top-3),
annualized and downside volatility, max drawdown with dates, historical
VaR + expected shortfall at 95/99, diversification ratio, and pairwise
correlation / beta vs an equal-weight proxy. Weights are validated and
renormalized with a warning; thin symbols fall out through a min-history
filter; every emitted float is finite-guarded so the payload always
survives json.dumps(allow_nan=False). Long-only for now — a short leg
changes concentration semantics and deserves its own slice.

The portfolio_risk_xray tool handles argument validation, shapes the
fetch envelope into a close panel (records with missing or mixed
timestamps fall back to loader order), and returns a strict-JSON
envelope with unresolved/skipped bookkeeping. Data fetching is
injectable for tests.

17 new tests cover the math on hand-built panels, the edge matrix
(renormalization, thin symbols, single asset, zero variance, bad
args), and the tool end to end with a stub fetcher. Full suite:
6037 passed, 12 skipped.

Signed-off-by: Yufeng He <40085740+he-yufeng@users.noreply.github.com>
 2026-07-22 22:07:41 +08:00
shadowinlife 7adb2c6fb8 fix(mt5): convert numpy scalars to native Python types for JSON serialization 
MetaTrader5 SDK copy_rates_from_pos() returns numpy structured arrays
containing numpy.int64/float64 scalars. These types are not natively
JSON-serializable, causing TypeError when trading_history responses
are serialized downstream.

Add _int_or_none() and _float_or_none() helpers in reads.py and apply
them to all numeric fields in _rate_to_dict(). Add regression tests
covering numpy, plain-dict, and None input scenarios.

Closes #774

Signed-off-by: shadowinlife <shadowinlife@gmail.com>
 2026-07-22 21:25:31 +08:00
Haozhe Wu 43331c3221 docs(readme): v0.1.12 contributor credits across all 5 READMEs 
Add the v0.1.12 cycle cohort (28 contributors) and demote the v0.1.11 list
into a collapsible block, in en/ja/ko/zh/ar. en/ja/ko keep English
descriptions; zh/ar translated.
 2026-07-22 21:13:25 +08:00
Haozhe Wu 8cb536fa61 chore(release): 0.1.12 — version bump + CHANGELOG 
Bump version 0.1.11 → 0.1.12 across pyproject, frontend package(+lock),
i18n app.version, SKILL.md, and Dockerfile OCI label. CHANGELOG [0.1.12]
covers the 0.1.11→0.1.12 window (correlation regime timeline, 3 new LLM
providers, MT5 connector, SDM skill, USD-M perps, ~90 correctness fixes,
and the #758 zai streaming/base_url fix).
 2026-07-22 13:14:11 +08:00
Haozhe Wu a066e5545e fix(providers): resolve zai streaming + base_url resolution (#758) 
Three fixes for the native zai provider on glm-5.1:

- get_llm_credentials now falls back to the provider catalog's
  default_base_url when no *_BASE_URL is set, so CLI/manual-.env users
  reach the canonical endpoint (Z.ai /api/coding/paas/v4) instead of
  silently defaulting to api.openai.com. Web Settings already did this;
  the backend credential path did not. Benefits all 22 providers.
- stream_chat falls back to a non-streaming invoke when a provider
  streams zero chunks ('No generation chunks were returned'), instead of
  raising ProviderStreamError. Scoped to the empty-stream case; genuine
  provider errors still raise.
- ProviderStreamError appends a base-URL hint when the endpoint returns
  an HTML error page (a common misconfiguration signal).

Fixes #758.
 2026-07-22 12:59:07 +08:00
klmtseng 4e7ddc5e4a fix(fundamentals): dedup restated rows and prevent period regression in PIT snapshots 
Two point-in-time correctness bugs in tushare_fundamentals.py:

1. `query_fundamentals` did not deduplicate after the PIT cut, so the same
   (ts_code, end_date) could appear twice when both the original filing and a
   later restatement were visible as of the query date.  Fix: after the PIT
   cut, keep only the row with the latest effective pit date per
   (ts_code, end_date); ties preserve the last row in input order (natural
   Tushare ordering where restatements follow originals).

2. `enrich_price_frames_with_fundamentals` fed the deduped snapshot directly
   into merge_asof.  This caused two problems: (a) the per-day PIT visibility
   was evaluated on already-deduped data, making a restatement whose pit_date
   was in the future appear as if it were always visible; (b) when a late
   restatement for an *older* period (end_date < currently-visible end_date)
   arrived, merge_asof picked it as the nearest-prior announcement, regressing
   every subsequent snapshot to the older period.

   Fix: enrich now calls the new internal `_query_pit_cut` (raw PIT cut, no
   dedup) and builds an explicit forward-only effective timeline — a row is
   only accepted into the timeline when its end_date >= the currently-visible
   end_date.  Same-period restatements (same end_date, later pit_date) pass
   this check and appear as additional timeline entries; merge_asof then
   surfaces the restatement value only from its own pit_date onward.

Fixes #771

Signed-off-by: klmtseng <20399135+klmtseng@users.noreply.github.com>
 2026-07-22 12:52:45 +08:00
Haozhe Wu f2c7256d56 Merge PR #756: feat(correlation): regime timeline endpoint and UI (edge density + hysteresis) 2026-07-22 12:51:20 +08:00
Haozhe Wu 93fa241fc8 Merge PR #757: feat(data): decouple maintenance-bracket acquisition from the historical fetch path 2026-07-22 12:51:19 +08:00
Haozhe Wu f06d399388 Merge PR #770: fix(journal): normalize Excel float-stringified A-share codes 2026-07-22 12:51:08 +08:00
Haozhe Wu 5eb45502c7 Merge PR #769: fix(tools): accept unicode dashes in PDF page ranges 2026-07-22 12:51:07 +08:00
Haozhe Wu d538bfa369 Merge PR #768: fix(api): treat export KEY= as KEY in dotenv settings helpers 2026-07-22 12:51:04 +08:00
Haozhe Wu 3f0e6b8c94 Merge PR #767: fix(factors): keep single-day IC std finite in factor analysis JSON 2026-07-22 12:51:03 +08:00
Haozhe Wu f1697117c1 Merge PR #766: fix(backtest): print scalar metrics as strict JSON 2026-07-22 12:51:02 +08:00
Haozhe Wu 566eb86ba3 Merge PR #765: fix(metrics): keep Sharpe/Sortino finite when equity hits zero mid-path 2026-07-22 12:51:02 +08:00
Haozhe Wu be0d0a7c2c Merge PR #764: fix(pattern): emit strict JSON when trend_slope mean is undefined 2026-07-22 12:51:01 +08:00
Haozhe Wu 6fffd52541 Merge PR #763: fix(session): skip schema-bad messages.jsonl lines when reading 2026-07-22 12:50:59 +08:00
Haozhe Wu e8241034e9 Merge PR #762: fix(session): skip corrupt session.json when listing sessions 2026-07-22 12:50:58 +08:00
Haozhe Wu c350cddf8c Merge PR #761: fix(loaders): keep yahoo 1m as minute bars, not monthly 2026-07-22 12:50:57 +08:00
santhreal 65e624ab8d fix(connectors): accept project 1H/4H periods on OKX bars 
_BAR_MAP only listed lowercase 1h/4h, so runner-style 1H/4H fell
through to bar 1D and silently returned daily candles.

Signed-off-by: santhreal <64453045+santhreal@users.noreply.github.com>
 2026-07-21 21:27:34 -07:00
santhreal d4d4e2762b fix(connectors): accept project 1H/4H periods on Alpaca bars 
_rest_timeframe only listed lowercase 1h/4h, so runner-style 1H/4H
fell through to 1Day and silently returned daily bars.

Signed-off-by: santhreal <64453045+santhreal@users.noreply.github.com>
 2026-07-21 21:27:33 -07:00
santhreal 92f0f6e35d fix(cli): keep empty edge columns when parsing markdown tables 
Signed-off-by: santhreal <64453045+santhreal@users.noreply.github.com>
 2026-07-21 21:26:40 -07:00
santhreal 90def21855 fix(feishu): keep empty edge columns when parsing markdown tables 
Signed-off-by: santhreal <64453045+santhreal@users.noreply.github.com>
 2026-07-21 21:26:39 -07:00
santhreal 3ab407d799 fix(loaders): reject unsupported akshare a-share intervals 
Signed-off-by: santhreal <64453045+santhreal@users.noreply.github.com>
 2026-07-21 21:26:37 -07:00
santhreal 78650f74aa fix(tiger): accept project-style 1H/4H period tokens 
Signed-off-by: santhreal <64453045+santhreal@users.noreply.github.com>
 2026-07-21 21:26:36 -07:00
santhreal d69be31f10 fix(loaders): map ccxt 1W/1M intervals instead of silent daily 
Signed-off-by: santhreal <64453045+santhreal@users.noreply.github.com>
 2026-07-21 21:26:34 -07:00
santhreal f14e701501 fix(loaders): map yfinance 1M to monthly bars, not minutes 
Unmapped intervals were lowercased, so project monthly token 1M became
yfinance 1m (one minute). Map 1M/1W explicitly like the yahoo loader.

Signed-off-by: santhreal <64453045+santhreal@users.noreply.github.com>
 2026-07-21 21:25:16 -07:00
santhreal 19c1cbc6e4 fix(loaders): reject unsupported india broker intervals instead of daily 
Unknown intervals (including runner 4H) fell through to broker period 1d,
so hourly requests silently returned daily bars. Reject and return no data.

Signed-off-by: santhreal <64453045+santhreal@users.noreply.github.com>
 2026-07-21 21:25:15 -07:00
santhreal 0f3a2d3aa0 fix(validation): tolerate blank holding_days cells in trades.csv 
Blank holding_days become NaN under pandas and int(NaN) aborted the
validation CLI. Coerce non-numeric/blank values to 0 holding bars.

Signed-off-by: santhreal <64453045+santhreal@users.noreply.github.com>
 2026-07-21 21:25:13 -07:00
santhreal 9563c1a110 fix(tools): emit JSON-safe diffs when report audit hits a zero reported value 
When reported_value is 0 and fetched_value is not, verdict used float('inf')
and json.dumps emitted non-RFC Infinity tokens. Record null percents instead
and serialize with allow_nan=False.

Signed-off-by: santhreal <64453045+santhreal@users.noreply.github.com>
 2026-07-21 21:25:12 -07:00
santhreal 4c6f23cb25 fix(journal): normalize Excel float YYYYMMDD dates in eastmoney parser 
Signed-off-by: santhreal <64453045+santhreal@users.noreply.github.com>
 2026-07-21 21:19:15 -07:00
santhreal 2b01bf46d0 fix(journal): normalize float-stringified A-share codes before qualifying 
Signed-off-by: santhreal <64453045+santhreal@users.noreply.github.com>
 2026-07-21 21:19:15 -07:00
santhreal 91c7cb23da fix(journal): normalize Excel float-stringified A-share codes 
Numeric CSV/Excel cells often stringify as 600519.0; the '.' made
_qualify_a_share treat them as already-qualified and skip .SH/.SZ.

Signed-off-by: santhreal <64453045+santhreal@users.noreply.github.com>
 2026-07-21 21:18:35 -07:00
santhreal ab3e8076d6 fix(tools): accept unicode dashes in PDF page ranges 
Word/LLM paste often uses en/em/minus dashes in ranges like 1–10;
those were silently dropped because only ASCII '-' was recognized.

Signed-off-by: santhreal <64453045+santhreal@users.noreply.github.com>
 2026-07-21 21:18:22 -07:00
santhreal c0f794c2c9 fix(api): treat export KEY= as KEY in dotenv settings helpers 
Shell-style export KEY=value lines were read as a literal key named export KEY, so Settings missed configured secrets and upserts appended a duplicate KEY= instead of updating the export line. Strip the export prefix on read/write like python-dotenv.

Signed-off-by: santhreal <64453045+santhreal@users.noreply.github.com>
 2026-07-21 21:17:28 -07:00
santhreal e76e1dc6fb fix(factors): keep single-day IC std finite in factor analysis JSON 
One IC observation makes Series.std() (ddof=1) NaN, which leaked into ic_summary.json and the tool envelope. Treat single-sample std as 0.0 (same guard as metrics vol) and dump with allow_nan=False.

Signed-off-by: santhreal <64453045+santhreal@users.noreply.github.com>
 2026-07-21 21:17:25 -07:00
santhreal d88a591272 fix(backtest): print scalar metrics as strict JSON 
Explosive equity paths can set annual_return to +inf. The engine stdout
dump used default allow_nan=True and emitted bare Infinity tokens.
Sanitize non-finite scalars to null and dump with allow_nan=False,
matching options and run_card.

Signed-off-by: santhreal <64453045+santhreal@users.noreply.github.com>
 2026-07-21 21:16:54 -07:00
santhreal 3074b684e1 fix(metrics): keep Sharpe/Sortino finite when equity hits zero mid-path 
A 100 → 0 → 50 equity path yields non-finite pct_change values that
poisoned Sharpe/Sortino/IR with NaN/inf. Skip those ratios (report 0)
when bar returns are non-finite, matching the options metrics path.

Signed-off-by: santhreal <64453045+santhreal@users.noreply.github.com>
 2026-07-21 21:16:52 -07:00
santhreal bd610d0b67 fix(pattern): emit strict JSON when trend_slope mean is undefined 
Halt-style NaN gaps can leave every polyfit window empty, so mean_slope
became NaN and leaked a bare NaN token into the tool JSON. Treat that as
0.0 and serialize with allow_nan=False, matching options_pricing.

Signed-off-by: santhreal <64453045+santhreal@users.noreply.github.com>
 2026-07-21 21:16:48 -07:00
santhreal afd0046cbd fix(session): skip schema-bad messages.jsonl lines when reading 
get_messages already skipped JSONDecodeError lines but still raised on
valid JSON that was not a Message mapping (null, extra keys). Skip those
too so later good lines remain readable.

Signed-off-by: santhreal <64453045+santhreal@users.noreply.github.com>
 2026-07-21 21:13:47 -07:00
santhreal 0b9df59cfe fix(session): skip corrupt session.json when listing sessions 
One unreadable or schema-invalid session.json raised during list_sessions
and aborted the whole listing; skip that entry and keep siblings.

Signed-off-by: santhreal <64453045+santhreal@users.noreply.github.com>
 2026-07-21 21:13:46 -07:00
santhreal 13f71e5a6a fix(loaders): keep yahoo 1m as minute bars, not monthly 
_to_yahoo_interval uppercased intervals before the map lookup, so project
interval 1m collided with monthly 1M and was sent to Yahoo as 1mo.

Signed-off-by: santhreal <64453045+santhreal@users.noreply.github.com>
 2026-07-21 21:13:46 -07:00
shadowinlife d92e362cd6 style(memory): apply black formatting per CONTRIBUTING.md 
Signed-off-by: shadowinlife <shadowinlife@gmail.com>
 2026-07-21 17:13:39 +08:00
shadowinlife 6f9e304987 test(memory): add retrieval quality benchmark suite 
Add P@5/MRR/NDCG@5 evaluation framework for memory retrieval A/B testing.
Benchmark compares Baseline (pure token overlap) vs Treatment (BM25 + importance
decay) on a 200-entry corpus with 50 annotated queries.

The benchmark suite is NOT intended for CI — it requires corpus files
(tmp/benchmark_corpus/) that are not version-controlled. The suite
automatically skips when corpus files are absent.

Corpus is derived from anonymized production agent sessions.

Usage:
    python -m pytest agent/tests/memory/benchmarks/ -v

Refs: #732
Signed-off-by: shadowinlife <shadowinlife@gmail.com>
 2026-07-21 16:18:21 +08:00
zygora 5f9144946d feat(data): decouple maintenance-bracket acquisition from the historical fetch path 
binanceusdm.fetch_leverage_tiers() routes to Binance's signed
GET /fapi/v1/leverageBracket USER_DATA endpoint and requires an API key,
even though the response carries no account-specific data. #716 called it
unconditionally for every -PERP fetch, so on merged main today any
perpetual fetch hard-crashes for callers without a Binance key.

- Remove the unconditional fetch_leverage_tiers() call (and the now-unused
  _fetch_maintenance_brackets live-fetch method entirely). Execution/mark
  OHLC and funding settlements go back to zero-credential, matching the
  rest of the stack.
- Introduce a caller-supplied, versioned bracket artifact as the only way
  to attach maintenance_brackets/maintenance_bracket_version: schema
  version, symbol, provenance timestamp, per-tier fields plus an optional
  notional_coefficient, and a content hash validated against the
  canonical bracket JSON. Never fetched live.
- fetch(..., bracket_artifacts={code: artifact}) attaches brackets only
  when a valid matching artifact is supplied for that code.
- fetch(..., require_brackets=True) makes a strict margin-risk consumer
  fail closed with an actionable error before any network call when no
  valid artifact is supplied for a -PERP code.

Per #462 / #716 follow-up discussion. No real bracket snapshot is
committed; test fixtures are synthetic. No Binance credentials are added
to the backtest path.

Signed-off-by: zygora <122336305+honginp@users.noreply.github.com>
 2026-07-21 15:50:38 +08:00
shadowinlife 9ad5c9acf4 chore: remove test_hotness_decay.py (hotness decay out of scope) 
Signed-off-by: shadowinlife <shadowinlife@gmail.com>
 2026-07-21 15:22:31 +08:00
shadowinlife fc6ee69f4b fix(memory): gate features behind flags per maintainer review 
- Gate find_relevant importance multiplier behind VT_MEMORY_DECAY
- Gate content deduplication behind VT_MEMORY_QUALITY
- Remove compute_hotness / VT_MEMORY_HOTNESS_DECAY (out of scope)
- Remove hotness_decay test file
- Update tests to enable required feature flags

Refs: #732
Signed-off-by: shadowinlife <shadowinlife@gmail.com>
 2026-07-21 15:17:29 +08:00
shadowinlife b2953a86bb chore: remove benchmark suite from Phase 1 contribution 
Evaluation framework (benchmarks, eval tests) moved to separate
contribution scope. Community discussion will determine whether
these are wanted as part of the memory lifecycle work.

Refs: #732
Signed-off-by: shadowinlife <shadowinlife@gmail.com>
 2026-07-21 15:01:46 +08:00
Haozhe Wu a44e1be357 docs: add 2026-07-21 news across all 5 READMEs 2026-07-21 14:58:18 +08:00
shadowinlife a8bf593d94 test(memory): skip benchmark suite when corpus files are absent 
Benchmark corpus lives outside version control (tmp/benchmark_corpus/).
CI runners do not have these files. Skip the entire benchmark module at
collection time rather than failing individual tests.

Refs: #732
Signed-off-by: shadowinlife <shadowinlife@gmail.com>
 2026-07-21 14:58:11 +08:00
shadowinlife 8033844364 fix(memory): address important-level code review findings 
- Move _HOTNESS_HALF_LIFE_DAYS and _HOTNESS_DECAY_RATE to module-level constants
- Move datetime import to module level in persistent.py
- Remove capacity gate from run_gc (caller decides when to trigger)
- Log warnings when _update_frontmatter_field cannot find frontmatter delimiters

Refs: #732
Signed-off-by: shadowinlife <shadowinlife@gmail.com>
 2026-07-21 14:46:38 +08:00
shadowinlife c79d9a3e56 fix(memory): address code review findings for Phase 1 
- Handle add() returning None (dedup blocked) in remember_tool.py
- Call track_access() in _recall() to enable importance decay
- Periodically clean up expired entries from _recent_hashes
- Remove duplicated is_hotness_decay_enabled() from lifecycle.py
- Use atomic write-then-rename in _update_frontmatter_field
- Protect _EVENT_DELTAS with MappingProxyType

Refs: #732
Signed-off-by: shadowinlife <shadowinlife@gmail.com>
 2026-07-21 14:36:21 +08:00
ebujinovch 555dcec560 feat(correlation): regime timeline endpoint and UI (edge density + hysteresis) 
Implements the proposal in #719: Mode 1 of the correlation-regime skill
(#557) wired into the /correlation surface. New GET /correlation/regime
(same auth, shared rate-limit budget) returns edge density, causally
smoothed density, hysteresis FUSED states and episodes over the requested
window; the Correlation tab gains an opt-in timeline strip above the
matrix. The loader fallback-chain fetch and close-series extraction are
extracted from compute_correlation_matrix for reuse (pure code motion);
existing /correlation behavior is unchanged.

Signed-off-by: ebujinovch <ch.ebujinov@gmail.com>
 2026-07-21 08:22:28 +02:00
shadowinlife 4cbb9c3243 chore: remove bench_report.json and revert .gitignore changes 
These files are not part of the community contribution scope.
Benchmark results are kept locally for validation only.

Refs: #732
Signed-off-by: shadowinlife <shadowinlife@gmail.com>
 2026-07-21 14:15:50 +08:00
shadowinlife 6ffa63be8d feat(memory): add content dedup, hotness decay and P@5 benchmark 
- Implement SHA-256 content-hash deduplication with 30s sliding window
- Add compute_hotness() with sigmoid frequency × exponential recency decay
- Register VT_MEMORY_HOTNESS_DECAY feature flag (default off)
- Add P@5/MRR/NDCG@5 evaluation framework with A/B comparison
- Benchmark shows +12.9% P@5 improvement with lifecycle-aware retrieval
- 35 new tests (17 dedup + 8 hotness + 10 benchmark)

Refs: #732
Signed-off-by: shadowinlife <shadowinlife@gmail.com>
 2026-07-21 14:12:30 +08:00
Haozhe Wu 251e7e0987 Merge PR #689: complete partial market-data results through fallback 
Rebased onto main and deduped — reuse the existing _restore_original_codes helper instead of the PR's parallel _map_result_to_requested; apply the same missing-symbol completion to both _fetch_auto and the explicit-source path. Fixes #681.
 2026-07-21 13:42:04 +08:00
xkam7ar 89ac9eb896 fix(data): fall back for missing symbols 
Signed-off-by: xkam7ar <kmoteallemi@icloud.com>
 2026-07-21 13:38:21 +08:00
Haozhe Wu bcf318d676 Merge pull request #644 from tyj147454413-cmd/codex/pr543-okx-loader-robustness 
fix(okx): bound history fetches and handle rate limits
 2026-07-21 12:26:29 +08:00
Haozhe Wu cab87bd40a Merge pull request #731 from octo-patch/octopatch/minimax-cn-endpoint 
feat: expose MiniMax regional API endpoints
 2026-07-21 12:26:23 +08:00
Haozhe Wu 2fe20faaf9 Merge pull request #748 from Robin1987China/shadow-daily-entry-hour-gate 
fix(shadow): skip the entry-hour gate on daily bars
 2026-07-21 12:26:18 +08:00
Haozhe Wu 857ceac313 Merge pull request #749 from Robin1987China/fix-journal-parsers-empty-code 
fix(tools): skip blank/NaN symbol rows in futu and generic journal parsers
 2026-07-21 12:26:12 +08:00
Haozhe Wu 171185ffd5 Merge pull request #750 from Robin1987China/fix-mcp-host-guard-ipv6 
fix(mcp): accept IPv6 and case-variant hosts in the network guard
 2026-07-21 12:26:07 +08:00
Haozhe Wu 2867365845 Merge pull request #725 from HKUDS/dependabot/pip/websockets-16.1.1 
build(deps): bump websockets from 15.0.1 to 16.1.1
 2026-07-21 12:17:13 +08:00
Haozhe Wu 9216e4baa3 Merge pull request #753 from HKUDS/dependabot/pip/pip-minor-patch-5ba2c1a785 
build(deps): bump the pip-minor-patch group across 1 directory with 20 updates
 2026-07-21 12:16:35 +08:00
Haozhe Wu f3069b2f03 Merge pull request #754 from HKUDS/dependabot/npm_and_yarn/frontend/npm-minor-patch-0b632d9036 
build(deps): bump the npm-minor-patch group across 1 directory with 6 updates
 2026-07-21 12:15:30 +08:00
dependabot[bot] 5066f99f4d build(deps): bump the npm-minor-patch group across 1 directory with 6 updates 
Bumps the npm-minor-patch group with 6 updates in the /frontend directory:

| Package | From | To |
| --- | --- | --- |
| [@fontsource/inter](https://github.com/fontsource/font-files/tree/HEAD/fonts/google/inter) | `5.2.8` | `5.3.0` |
| [@fontsource/jetbrains-mono](https://github.com/fontsource/font-files/tree/HEAD/fonts/google/jetbrains-mono) | `5.2.8` | `5.3.0` |
| [lucide-react](https://github.com/lucide-icons/lucide/tree/HEAD/packages/lucide-react) | `1.24.0` | `1.25.0` |
| [react-i18next](https://github.com/i18next/react-i18next) | `17.0.9` | `17.0.10` |
| [autoprefixer](https://github.com/postcss/autoprefixer) | `10.5.2` | `10.5.4` |
| [postcss](https://github.com/postcss/postcss) | `8.5.18` | `8.5.20` |



Updates `@fontsource/inter` from 5.2.8 to 5.3.0
- [Changelog](https://github.com/fontsource/font-files/blob/main/CHANGELOG.md)
- [Commits](https://github.com/fontsource/font-files/commits/HEAD/fonts/google/inter)

Updates `@fontsource/jetbrains-mono` from 5.2.8 to 5.3.0
- [Changelog](https://github.com/fontsource/font-files/blob/main/CHANGELOG.md)
- [Commits](https://github.com/fontsource/font-files/commits/HEAD/fonts/google/jetbrains-mono)

Updates `lucide-react` from 1.24.0 to 1.25.0
- [Release notes](https://github.com/lucide-icons/lucide/releases)
- [Commits](https://github.com/lucide-icons/lucide/commits/1.25.0/packages/lucide-react)

Updates `react-i18next` from 17.0.9 to 17.0.10
- [Changelog](https://github.com/i18next/react-i18next/blob/master/CHANGELOG.md)
- [Commits](https://github.com/i18next/react-i18next/compare/v17.0.9...v17.0.10)

Updates `autoprefixer` from 10.5.2 to 10.5.4
- [Release notes](https://github.com/postcss/autoprefixer/releases)
- [Changelog](https://github.com/postcss/autoprefixer/blob/main/CHANGELOG.md)
- [Commits](https://github.com/postcss/autoprefixer/compare/10.5.2...10.5.4)

Updates `postcss` from 8.5.18 to 8.5.20
- [Release notes](https://github.com/postcss/postcss/releases)
- [Changelog](https://github.com/postcss/postcss/blob/main/CHANGELOG.md)
- [Commits](https://github.com/postcss/postcss/compare/8.5.18...8.5.20)

---
updated-dependencies:
- dependency-name: "@fontsource/inter"
  dependency-version: 5.3.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: npm-minor-patch
- dependency-name: "@fontsource/jetbrains-mono"
  dependency-version: 5.3.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: npm-minor-patch
- dependency-name: lucide-react
  dependency-version: 1.25.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: npm-minor-patch
- dependency-name: react-i18next
  dependency-version: 17.0.10
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: npm-minor-patch
- dependency-name: autoprefixer
  dependency-version: 10.5.4
  dependency-type: direct:development
  update-type: version-update:semver-patch
  dependency-group: npm-minor-patch
- dependency-name: postcss
  dependency-version: 8.5.20
  dependency-type: direct:development
  update-type: version-update:semver-patch
  dependency-group: npm-minor-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
 2026-07-20 13:13:25 +00:00
Haozhe Wu 585b8f69cd fix(frontend): render LaTeX math in chat via remark-math + KaTeX 
LLM answers with \(...\)/\[...\] or $$ formulas rendered as plain text --
the markdown pipeline had no math plugin. Normalize LLM delimiters to $$
outside code segments (streaming-safe for unclosed fences), parse with
remark-math (single-$ disabled so dollar amounts like "$150 to $120"
never become formulas), render with rehype-katex.
 2026-07-20 20:56:55 +08:00
Haozhe Wu 7abc2a8a03 docs: add 2026-07-20 news across all 5 READMEs 
Providers (Anthropic native/SiliconFlow/iFlytek), MetaTrader 5 connector + mt5
source, llm-vision OCR, 80x signal vectorization, USD-M funding data, reliability
consolidation (#584, 13 issues), options partial-close fix, and more. Keeps 3
entries visible; 2026-07-17 moved into the earlier-news details.
 2026-07-20 19:07:11 +08:00
dependabot[bot] 76e8f518f0 build(deps): bump the pip-minor-patch group across 1 directory with 20 updates 
Bumps the pip-minor-patch group with 20 updates in the / directory:

| Package | From | To |
| --- | --- | --- |
| [langchain](https://github.com/langchain-ai/langchain) | `1.3.13` | `1.3.14` |
| [pypdfium2](https://github.com/pypdfium2-team/pypdfium2) | `5.11.0` | `5.12.1` |
| [ccxt](https://github.com/ccxt/ccxt) | `4.5.64` | `4.5.67` |
| [fastapi](https://github.com/fastapi/fastapi) | `0.139.0` | `0.139.2` |
| [matplotlib](https://github.com/matplotlib/matplotlib) | `3.11.0` | `3.11.1` |
| [caio](https://github.com/mosquito/caio) | `0.9.25` | `0.10.2` |
| [cyclopts](https://github.com/BrianPugh/cyclopts) | `4.21.0` | `4.22.0` |
| [jaraco-functools](https://github.com/jaraco/jaraco.functools) | `4.5.0` | `4.6.0` |
| [joserfc](https://github.com/authlib/joserfc) | `1.7.3` | `1.7.4` |
| [langsmith](https://github.com/langchain-ai/langsmith-sdk) | `0.10.2` | `0.10.6` |
| [narwhals](https://github.com/narwhals-dev/narwhals) | `2.23.0` | `2.24.0` |
| [openai](https://github.com/openai/openai-python) | `2.45.0` | `2.46.0` |
| [opentelemetry-api](https://github.com/open-telemetry/opentelemetry-python) | `1.43.0` | `1.44.0` |
| [peewee](https://github.com/coleifer/peewee) | `4.2.1` | `4.2.6` |
| [platformdirs](https://github.com/tox-dev/platformdirs) | `4.10.0` | `4.10.1` |
| [pydantic-core](https://github.com/pydantic/pydantic) | `2.46.4` | `2.47.0` |
| [regex](https://github.com/mrabarnett/mrab-regex) | `2026.7.10` | `2026.7.19` |
| [soupsieve](https://github.com/facelessuser/soupsieve) | `2.8.4` | `2.9` |
| [tqdm](https://github.com/tqdm/tqdm) | `4.68.4` | `4.69.0` |
| [yarl](https://github.com/aio-libs/yarl) | `1.24.2` | `1.24.5` |



Updates `langchain` from 1.3.13 to 1.3.14
- [Release notes](https://github.com/langchain-ai/langchain/releases)
- [Commits](https://github.com/langchain-ai/langchain/compare/langchain==1.3.13...langchain==1.3.14)

Updates `pypdfium2` from 5.11.0 to 5.12.1
- [Release notes](https://github.com/pypdfium2-team/pypdfium2/releases)
- [Commits](https://github.com/pypdfium2-team/pypdfium2/compare/5.11.0...5.12.1)

Updates `ccxt` from 4.5.64 to 4.5.67
- [Release notes](https://github.com/ccxt/ccxt/releases)
- [Commits](https://github.com/ccxt/ccxt/compare/v4.5.64...v4.5.67)

Updates `fastapi` from 0.139.0 to 0.139.2
- [Release notes](https://github.com/fastapi/fastapi/releases)
- [Commits](https://github.com/fastapi/fastapi/compare/0.139.0...0.139.2)

Updates `matplotlib` from 3.11.0 to 3.11.1
- [Release notes](https://github.com/matplotlib/matplotlib/releases)
- [Commits](https://github.com/matplotlib/matplotlib/compare/v3.11.0...v3.11.1)

Updates `caio` from 0.9.25 to 0.10.2
- [Release notes](https://github.com/mosquito/caio/releases)
- [Commits](https://github.com/mosquito/caio/compare/0.9.25...0.10.2)

Updates `cyclopts` from 4.21.0 to 4.22.0
- [Release notes](https://github.com/BrianPugh/cyclopts/releases)
- [Commits](https://github.com/BrianPugh/cyclopts/compare/v4.21.0...v4.22.0)

Updates `jaraco-functools` from 4.5.0 to 4.6.0
- [Release notes](https://github.com/jaraco/jaraco.functools/releases)
- [Changelog](https://github.com/jaraco/jaraco.functools/blob/main/NEWS.rst)
- [Commits](https://github.com/jaraco/jaraco.functools/compare/v4.5.0...v4.6.0)

Updates `joserfc` from 1.7.3 to 1.7.4
- [Release notes](https://github.com/authlib/joserfc/releases)
- [Changelog](https://github.com/authlib/joserfc/blob/main/docs/changelog.rst)
- [Commits](https://github.com/authlib/joserfc/compare/1.7.3...1.7.4)

Updates `langsmith` from 0.10.2 to 0.10.6
- [Release notes](https://github.com/langchain-ai/langsmith-sdk/releases)
- [Commits](https://github.com/langchain-ai/langsmith-sdk/compare/v0.10.2...v0.10.6)

Updates `narwhals` from 2.23.0 to 2.24.0
- [Release notes](https://github.com/narwhals-dev/narwhals/releases)
- [Commits](https://github.com/narwhals-dev/narwhals/compare/v2.23.0...v2.24.0)

Updates `openai` from 2.45.0 to 2.46.0
- [Release notes](https://github.com/openai/openai-python/releases)
- [Changelog](https://github.com/openai/openai-python/blob/main/CHANGELOG.md)
- [Commits](https://github.com/openai/openai-python/compare/v2.45.0...v2.46.0)

Updates `opentelemetry-api` from 1.43.0 to 1.44.0
- [Release notes](https://github.com/open-telemetry/opentelemetry-python/releases)
- [Changelog](https://github.com/open-telemetry/opentelemetry-python/blob/main/CHANGELOG.md)
- [Commits](https://github.com/open-telemetry/opentelemetry-python/compare/v1.43.0...v1.44.0)

Updates `peewee` from 4.2.1 to 4.2.6
- [Release notes](https://github.com/coleifer/peewee/releases)
- [Changelog](https://github.com/coleifer/peewee/blob/master/CHANGELOG.md)
- [Commits](https://github.com/coleifer/peewee/compare/4.2.1...4.2.6)

Updates `platformdirs` from 4.10.0 to 4.10.1
- [Release notes](https://github.com/tox-dev/platformdirs/releases)
- [Changelog](https://github.com/tox-dev/platformdirs/blob/main/docs/changelog.rst)
- [Commits](https://github.com/tox-dev/platformdirs/compare/4.10.0...4.10.1)

Updates `pydantic-core` from 2.46.4 to 2.47.0
- [Release notes](https://github.com/pydantic/pydantic/releases)
- [Changelog](https://github.com/pydantic/pydantic/blob/main/HISTORY.md)
- [Commits](https://github.com/pydantic/pydantic/commits)

Updates `regex` from 2026.7.10 to 2026.7.19
- [Changelog](https://github.com/mrabarnett/mrab-regex/blob/hg/changelog.txt)
- [Commits](https://github.com/mrabarnett/mrab-regex/compare/2026.7.10...2026.7.19)

Updates `soupsieve` from 2.8.4 to 2.9
- [Release notes](https://github.com/facelessuser/soupsieve/releases)
- [Commits](https://github.com/facelessuser/soupsieve/compare/2.8.4...2.9)

Updates `tqdm` from 4.68.4 to 4.69.0
- [Release notes](https://github.com/tqdm/tqdm/releases)
- [Commits](https://github.com/tqdm/tqdm/compare/v4.68.4...v4.69.0)

Updates `yarl` from 1.24.2 to 1.24.5
- [Release notes](https://github.com/aio-libs/yarl/releases)
- [Changelog](https://github.com/aio-libs/yarl/blob/master/CHANGES.rst)
- [Commits](https://github.com/aio-libs/yarl/compare/v1.24.2...v1.24.5)

---
updated-dependencies:
- dependency-name: langchain
  dependency-version: 1.3.14
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: pip-minor-patch
- dependency-name: pypdfium2
  dependency-version: 5.12.1
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: pip-minor-patch
- dependency-name: ccxt
  dependency-version: 4.5.67
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: pip-minor-patch
- dependency-name: fastapi
  dependency-version: 0.139.2
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: pip-minor-patch
- dependency-name: matplotlib
  dependency-version: 3.11.1
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: pip-minor-patch
- dependency-name: caio
  dependency-version: 0.10.2
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: pip-minor-patch
- dependency-name: cyclopts
  dependency-version: 4.22.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: pip-minor-patch
- dependency-name: jaraco-functools
  dependency-version: 4.6.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: pip-minor-patch
- dependency-name: joserfc
  dependency-version: 1.7.4
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: pip-minor-patch
- dependency-name: langsmith
  dependency-version: 0.10.6
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: pip-minor-patch
- dependency-name: narwhals
  dependency-version: 2.24.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: pip-minor-patch
- dependency-name: openai
  dependency-version: 2.46.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: pip-minor-patch
- dependency-name: opentelemetry-api
  dependency-version: 1.44.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: pip-minor-patch
- dependency-name: peewee
  dependency-version: 4.2.6
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: pip-minor-patch
- dependency-name: platformdirs
  dependency-version: 4.10.1
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: pip-minor-patch
- dependency-name: pydantic-core
  dependency-version: 2.47.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: pip-minor-patch
- dependency-name: regex
  dependency-version: 2026.7.19
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: pip-minor-patch
- dependency-name: soupsieve
  dependency-version: '2.9'
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: pip-minor-patch
- dependency-name: tqdm
  dependency-version: 4.69.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: pip-minor-patch
- dependency-name: yarl
  dependency-version: 1.24.5
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: pip-minor-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
 2026-07-20 11:03:53 +00:00
Haozhe Wu aaa1061b78 fix(options): honor partial-close quantity instead of flattening the lot 
The options engine close branch computed cash/PnL from the full matched lot and
removed the whole position, ignoring the leg's requested qty — any partial close
wiped the entire lot with full PnL. Now an explicit leg qty closes only that many
contracts (clamped to the open size), cash/PnL scale to the closed fraction, and
the remainder stays open as a reduced-qty position; a close leg with no qty still
closes the whole lot (legacy). Closes #577.
 2026-07-20 19:00:53 +08:00
Haozhe Wu c1f9f96a60 Merge PR #548: feat(ocr): pluggable engine architecture + LLM vision OCR + configurable threshold 
Self-fix on merge: the qwen-vl → llm-vision deprecation warning now calls out the
credential-source change (DASHSCOPE_API_KEY → configured LANGCHAIN_PROVIDER) so
existing DashScope users aren't surprised. Generalizes the shipped qwen-vl engine
into a provider-agnostic llm-vision engine + entry-point plugin discovery + OCR
threshold + quality metrics. Closes #547.
 2026-07-20 18:56:15 +08:00
Haozhe Wu 2400bbcbaa Merge PR #695: feat(providers): add native Anthropic Messages API support 
Self-fix on merge: moved langchain-anthropic from core dependencies to an
[anthropic] optional extra (like [deepseek]) — _build_anthropic lazy-imports it
and raises an actionable install hint, so the base install stays lean. Resolved
llm.py conflicts against #563's centralized get_llm_credentials + the anthropic
native adapter-mode branch. Closes #694.
 2026-07-20 18:55:21 +08:00
dependabot[bot] 8d27badecd build(deps): bump websockets from 15.0.1 to 16.1.1 
Bumps [websockets](https://github.com/python-websockets/websockets) from 15.0.1 to 16.1.1.
- [Release notes](https://github.com/python-websockets/websockets/releases)
- [Commits](https://github.com/python-websockets/websockets/compare/15.0.1...16.1.1)

---
updated-dependencies:
- dependency-name: websockets
  dependency-version: 16.1.1
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
 2026-07-20 10:54:10 +00:00
Haozhe Wu 0c86356a77 docs(skill): market-data source count 22 → 23 (mt5 loader, #481) 2026-07-20 18:51:50 +08:00
Haozhe Wu 11b4f62340 Merge PR #481: feat: MetaTrader 5 (Exness) broker connector + mt5 market-data source 
Red-lines verified PASS (mandate gate, structural trade_mode discriminator, no MCP
order exposure, sandbox-barred loader). Resolved additive conflicts: registry (mt5
loader alongside longbridge), pyproject (mt5 extra), CHANGELOG (kept both entries);
README/SKILL counts reconciled in the news pass. market_data =F/=X (yahoo) and mt5
forex routing coexist.
 2026-07-20 18:48:32 +08:00
Haozhe Wu 5278469d0d Merge PR #584: Fix reliability across packaging, web, scheduler, swarm, and CLI 
Consolidation resolving #603 #605 #611 #613 #617 #639 (named) + folded #583 #585 #587 #589 #591 #595 #599.
Kept both dev-extra and slack-version packaging tests at merge.
 2026-07-20 18:45:20 +08:00
Haozhe Wu 949ccad594 Merge PR #726: fix: Robinhood connector account_number wiring + remote-MCP display shape 
Reconciled the connector renderers with #735 (Longbridge schema-tolerant balances):
_print_connector_account now handles balances (broker_sdk) → remote-MCP nested
(Robinhood) → IBKR summary; positions use both remote-MCP shape and _first_present aliasing.
 2026-07-20 18:44:45 +08:00
Haozhe Wu b7c62c68a1 Merge PR #716: feat(data): add historical USD-M funding settlements + maintenance brackets 
Stray docs/superpowers/pr-drafts/pr1b-funding-brackets.md dropped at merge (repo never carries docs/ from contributor PRs).
 2026-07-20 18:42:41 +08:00
Haozhe Wu c2151a2269 fix(backtest): make vectorized _align writable (follow-up #698) 
The vectorized _align did in-place np.nan_to_num/clip on arrays taken from
pandas via .values/astype(copy=False), which are read-only views under pandas
copy-on-write, raising 'ValueError: assignment destination is read-only' on
current main. Force a writable copy for the per-symbol signal array and use the
copy-returning nan_to_num for the position matrix. Vectorization stays numerically
equivalent to the scalar path (assert_frame_equal tests pass).
 2026-07-20 18:39:21 +08:00
Haozhe Wu 099948ec01 Merge PR #698: perf: Signal alignment vectorization — 80x speedup 2026-07-20 18:37:45 +08:00
Haozhe Wu 8c627b00f7 test(cli): update provider-menu indices after SiliconFlow(#565)+Spark(#537) inserts 
Both PRs insert providers ahead of ollama/openai-codex in the init menu; the
combined order puts ollama at 17 and openai-codex at 18.
 2026-07-20 18:33:20 +08:00
Haozhe Wu add7b3f364 Merge PR #563: refactor(providers): centralize credentials + fix gateway misroute (closes #549, #553) 
# Conflicts:
#	agent/src/providers/capabilities.py
#	agent/tests/test_llm.py
 2026-07-20 18:32:08 +08:00
Haozhe Wu 7ba3bf2f5d Merge PR #537: feat(providers): add iFlytek Spark as an OpenAI-compatible provider 2026-07-20 18:31:33 +08:00
Haozhe Wu 3b9dba55eb Merge PR #565: feat: add SiliconFlow CN and Global providers 2026-07-20 18:31:33 +08:00
Haozhe Wu e796a34dd6 Merge PR #663: fix(codex): classify stream HTTP failures 2026-07-20 18:31:33 +08:00
Haozhe Wu 0802a535a6 Merge PR #641: Honor cancellation before the first AgentLoop iteration (closes #638) 2026-07-20 18:31:33 +08:00
Haozhe Wu 205c940c99 Merge PR #704: perf(swarm): cache MCP tool discovery specs to avoid redundant RPC calls 2026-07-20 18:31:33 +08:00
Haozhe Wu 015e1f7e3e Merge PR #717: fix(frontend): prevent insertBefore DOM race during streaming output 2026-07-20 18:31:33 +08:00
Haozhe Wu 7eff24ddf0 test(api): preserve #738 commented-key upsert coverage after folding into #742 2026-07-20 17:49:20 +08:00
Haozhe Wu 8b4dcaf313 fix(data): resolve Yahoo futures (=F) and forex (=X) suffix tickers 
get_market_data could not resolve tickers like GC=F or EURUSD=X: detect_source
matched no pattern and fell to the 'tushare' default, routing them to
China-market loaders that cannot serve them; and the yahoo direct loader's
_is_supported guard rejected any non-.US/.HK/.NS/.BO symbol even though Yahoo's
public chart endpoint serves =F/=X verbatim.

Add =F/=X source patterns (route to yahoo) and accept =F/=X in the yahoo loader.

Closes #718
 2026-07-20 17:46:14 +08:00
Haozhe Wu 2b31effe13 fix(cli): make connector positions/account renderers schema-tolerant 
The shared 'connector positions'/'connector account' CLI renderers assumed the
IBKR result shape (position/avg_cost/sec_type/summary). broker_sdk connectors
such as Longbridge emit quantity/cost_price/market/balances, so every
non-matching key rendered as an empty cell.

Read key aliases (position|quantity, avg_cost|cost_price, sec_type|market) via a
None-aware _first_present helper (keeps a real zero qty), and render a balances
table when a connector returns 'balances' instead of IBKR 'summary' rows.

Closes #735
 2026-07-20 17:46:14 +08:00
Haozhe Wu 57034a0fbc Merge PR #743: fix(api): keep hash characters inside quoted dotenv values 
# Conflicts:
#	agent/tests/test_api_infrastructure.py
 2026-07-20 17:36:16 +08:00
Haozhe Wu 7558b8b86a Merge PR #742: fix(api): upsert last duplicate active dotenv key (also skips commented keys, folding in #738) 2026-07-20 17:35:24 +08:00
Haozhe Wu 707ceb1c2b Merge PR #744: fix(options): treat non-positive spot/strike as intrinsic in BS helpers 2026-07-20 17:35:16 +08:00
Haozhe Wu 45b21ff516 Merge PR #741: fix(pattern): reject window=1 for trend_line_slope 2026-07-20 17:35:16 +08:00
Haozhe Wu 2c679b2a23 Merge PR #740: fix(options): catch OverflowError in options metrics annualization 2026-07-20 17:35:16 +08:00
Haozhe Wu dd5494f92c Merge PR #739: fix(metrics): avoid OverflowError annualizing explosive equity paths 2026-07-20 17:35:16 +08:00
Haozhe Wu f72f3ff822 Merge PR #737: fix(ui): skip non-positive MA periods in indicator overlays 2026-07-20 17:35:16 +08:00
Haozhe Wu 2d35e8000a Merge PR #736: fix(channels): avoid hang splitting telegram fenced markdown chunks 2026-07-20 17:35:16 +08:00
Haozhe Wu 3bd64c3b1f Merge PR #734: fix(composite): fall back to first available sub-engine for unknown symbols 2026-07-20 17:35:16 +08:00
Robin1987China 63dee9a081 fix(mcp): accept IPv6 and case-variant hosts in the network guard 
The GHSA-p3c9 guard wrapped network MCP transports in Starlette's
TrustedHostMiddleware, whose plain split(":") Host parsing mangles
bracketed IPv6 ([::1]:8900 -> "[") and matches case-sensitively. An
MCP server bound to --host ::1 rejected every request with 400, and no
VIBE_TRADING_MCP_ALLOWED_HOSTS entry ([::1] or ::1) could fix it — the
only way out was '*', disabling the protection entirely. LOCALHOST was
rejected the same way, while the Origin guard (urlparse().hostname)
already lowercased, so the two layers disagreed.

Replace the middleware with a repo-owned _HostGuardMiddleware that
normalizes the header (brackets/port stripped, bare IPv6 kept whole,
lowercased) before matching the allow-list, normalize configured
entries the same way, and add ::1 to the loopback default. Allow-list
semantics (exact, *. wildcard, *) and the 400 rejection are unchanged;
the Origin guard is untouched.

Refs GHSA-p3c9

Signed-off-by: Robin1987China <41602358+Robin1987China@users.noreply.github.com>
 2026-07-20 14:41:09 +08:00
santhreal d84c00f778 fix(api): scan quoted dotenv values before stripping hash comments 
Signed-off-by: santhreal <64453045+santhreal@users.noreply.github.com>
 2026-07-19 22:35:01 -07:00
santhreal 1260ce4704 fix(options): clarify non-positive S/K intrinsic fallback comment 
Signed-off-by: santhreal <64453045+santhreal@users.noreply.github.com>
 2026-07-19 22:35:01 -07:00
santhreal 6ffdc220f4 fix(api): preserve trailing comments after closed dotenv quotes 
Signed-off-by: santhreal <64453045+santhreal@users.noreply.github.com>
 2026-07-19 22:24:40 -07:00
santhreal d1f83c096c fix(api): upsert last duplicate active dotenv key 
Signed-off-by: santhreal <64453045+santhreal@users.noreply.github.com>
 2026-07-19 22:20:47 -07:00
santhreal 3fda400b76 fix(pattern): reject window=1 for trend_line_slope 
Signed-off-by: santhreal <64453045+santhreal@users.noreply.github.com>
 2026-07-19 22:20:47 -07:00
santhreal be7cf96383 fix(api): unquote dotenv values before stripping hash comments 
Signed-off-by: santhreal <64453045+santhreal@users.noreply.github.com>
 2026-07-19 22:20:47 -07:00
santhreal b52f2edd74 fix(options): catch OverflowError in options metrics annualization 
Signed-off-by: santhreal <64453045+santhreal@users.noreply.github.com>
 2026-07-19 22:20:36 -07:00
santhreal a0546c5b19 fix(metrics): avoid OverflowError annualizing explosive equity paths 
Signed-off-by: santhreal <64453045+santhreal@users.noreply.github.com>
 2026-07-19 21:44:39 -07:00
santhreal aeff42d776 fix(options): treat non-positive spot/strike as intrinsic in BS helpers 
Signed-off-by: santhreal <64453045+santhreal@users.noreply.github.com>
 2026-07-19 21:44:38 -07:00
santhreal a63a880c92 fix(ui): skip non-positive MA periods in indicator overlays 
Planner/design artifacts can emit ma_window=0; build_indicator_series then divided by zero. Drop non-positive periods when inferring and when building series.

Signed-off-by: santhreal <64453045+santhreal@users.noreply.github.com>
 2026-07-19 20:53:14 -07:00
santhreal 497274c688 fix(channels): avoid hang splitting telegram fenced markdown chunks 
Long or mid-size cuts inside an opening fence could land on the fence line and reopen the same chunk forever. Advance past the fence header and bail when reopen does not shrink the buffer.

Signed-off-by: santhreal <64453045+santhreal@users.noreply.github.com>
 2026-07-19 20:51:47 -07:00
Jelech 39ea99a4d9 feat(providers): support ANTHROPIC_MAX_TOKENS configuration 2026-07-20 10:51:58 +08:00
Marnie0415 1ba3565819 fix(composite): add unit tests and edge case guard for empty sub-engines 
Address review feedback:
- Add pytest tests for unknown symbol handling (round_size, commission, leverage)
- Add defensive check for empty _rule_engines dict
- Verify known symbols still route to correct sub-engine

Signed-off-by: Marnie0415 <Marnie0415@users.noreply.github.com>
 2026-07-20 04:05:08 +08:00
Marnie0415 4834bddaed fix(composite): fall back to first available sub-engine for unknown symbols 
CompositeEngine._rule_for() defaults to 'a_share' for symbols whose market
type is not in _symbol_market. When _rule_engines does not contain an
'a_share' key (because no A-share codes were in the original request),
the method raises KeyError: 'a_share'.

Replace the hard index with .get() and fall back to the first available
sub-engine when the resolved market has no dedicated rule engine.

Signed-off-by: Marnie0415 <Marnie0415@users.noreply.github.com>
 2026-07-20 03:57:35 +08:00
shadowinlife 57087d2ac5 test(memory): add lifecycle latency benchmarks and GC accuracy tests 
Signed-off-by: shadowinlife <shadowinlife@gmail.com>
 2026-07-20 00:23:36 +08:00
shadowinlife cfca5504b1 docs(memory): add lifecycle feature flags to .env.example 
Document VT_MEMORY_QUALITY, VT_MEMORY_GC, and VT_MEMORY_DECAY
environment variables with descriptions and default values.

Signed-off-by: shadowinlife <shadowinlife@gmail.com>
 2026-07-19 23:28:42 +08:00
shadowinlife 0a6a6344dd fix(memory): route feature flags through centralized config layer 
Move VT_MEMORY_QUALITY, VT_MEMORY_GC, VT_MEMORY_DECAY reads from raw
os.environ.get() to src.config.env_schema.MemoryConfig, satisfying the
CI environment-variable encapsulation gate.

Signed-off-by: shadowinlife <shadowinlife@gmail.com>
 2026-07-19 23:23:07 +08:00
shadowinlife 00adec5256 test(memory): fix related_memories test to assert hex-only validation 
Update test_related_memories_filters_invalid_ids to verify that
non-hex 6-char IDs (e.g. 'ok33ff') are correctly rejected by the
hex charset validator introduced in the previous commit.

Signed-off-by: shadowinlife <shadowinlife@gmail.com>
 2026-07-19 21:39:55 +08:00
shadowinlife 57f08ebe50 fix(memory): address code review findings for Phase 1 
- Move memory_lock to persistent.py; guard add/remove/rebuild with lock
- Compute importance dynamically in _scan_entries via Ebbinghaus decay
- Add exception handling in GC delete path and gc.log writes
- Conditional fcntl import for Windows compatibility
- Validate hex charset for related_memories field

Signed-off-by: shadowinlife <shadowinlife@gmail.com>
 2026-07-19 21:36:48 +08:00
octo-patch 6dc76e1ad3 feat: expose MiniMax regional endpoints 2026-07-19 20:06:43 +08:00
Haozhe Wu 97e762e51c docs: add 2026-07-19 news across all 5 READMEs 
Real US/HK Yahoo Finance news articles (#730), MCP factor_analysis contract
fix (#715/#635), Kimi K-series temperature=1 forcing (#701), and inverted/
degenerate-input fail-fast across split_message/PDF pages/journal filters
(#727-#729). Keeps 3 entries visible; 07-16 moves into the details block.
 2026-07-19 19:50:02 +08:00
Haozhe Wu cf51bc322b ci: correct setup-node pin comment to v7.0.0 (follow-up #720) 
Dependabot bumped the setup-node SHA to v7.0.0 (820762786...) but left the
pin comment reading # v6. Align the comment with the actual pinned version.
 2026-07-19 19:41:57 +08:00
Haozhe Wu 668389bdea fix(tools): surface inverted journal date filter as error envelope (follow-up #729) 
#729 made _apply_filter raise on inverted/degenerate date ranges (fail-fast
over silently returning an empty frame). But analyze_trade_journal called it
unwrapped, so the ValueError escaped as a raw traceback instead of the
{status: error} envelope the rest of the function uses; and the pre-existing
test_apply_filter_preserves_full_date_and_mixed_precision still asserted the
old silent-empty behavior. Wrap the call to return the error envelope and
update the stale assertion to expect the raise.
 2026-07-19 19:41:57 +08:00
Haozhe Wu 1845e41122 Merge pull request #730 from yxhuang/fix/us-hk-stock-news 
fix(tools): return real Yahoo news articles for US/HK stock news
 2026-07-19 19:40:53 +08:00
Haozhe Wu 9d185ba357 Merge pull request #729 from santhreal/fix/journal-inverted-date-filter 
fix(tools): reject inverted trade-journal date filters
 2026-07-19 19:40:48 +08:00
Haozhe Wu 7b2b75fb5c Merge pull request #728 from santhreal/fix/doc-reader-inverted-pages 
fix(tools): reject inverted PDF page ranges in doc_reader
 2026-07-19 19:40:45 +08:00
Haozhe Wu 41ba61bcc8 Merge pull request #727 from santhreal/fix/channels-split-message-nonpositive 
fix(channels): avoid hang in split_message when max_len <= 0
 2026-07-19 19:40:41 +08:00
Haozhe Wu f42d1b3af0 Merge pull request #720 from HKUDS/dependabot/github_actions/actions/setup-node-7.0.0 
build(deps): bump actions/setup-node from 6.4.0 to 7.0.0
 2026-07-19 19:40:38 +08:00
Haozhe Wu 5c6ec6c4c3 Merge pull request #715 from Robin1987China/fix-mcp-factor-analysis-contract 
fix(mcp): align factor_analysis wrapper with the registered tool contract
 2026-07-19 19:40:34 +08:00
Haozhe Wu c1277e197b Merge pull request #701 from sambazhu/fix/kimi-k-series-temperature 
fix(providers): force temperature=1 for the whole Kimi K-series
 2026-07-19 19:40:30 +08:00
Haozhe Wu d55de48d42 fix(skills): use API-native decimal funding rate in perp-funding-basis 
The annualized example used 0.01 with a '0.01%' comment, which reads as
1% (→1095% annualized), not the intended 0.01% (→10.95%). Standardize the
whole skill on the per-8h decimal convention OKX/Binance actually return
(0.0001 = 0.01%): fix the annualized example, funding_regime and
oi_funding_matrix thresholds, the cross-exchange dict, and the carry-trade
example; add an explicit units-convention note. Human-readable % tables
kept as-is.
 2026-07-19 19:23:23 +08:00
shadowinlife 6628c4a833 feat(memory): implement Phase 1 lifecycle management 
- Extend MemoryEntry with quality_score, importance, access tracking
- Add lifecycle.py: compute_importance, reinforce(), run_gc()
- Add file locking (fcntl) for single-writer concurrency
- Extend RememberTool with reinforce action
- Feature flags: VT_MEMORY_QUALITY, VT_MEMORY_GC, VT_MEMORY_DECAY
- Add 35 unit tests covering all new functionality

All features disabled by default; enable via environment variables.

Signed-off-by: shadowinlife <shadowinlife@gmail.com>
 2026-07-19 16:24:40 +08:00
Robin1987China 1af03bb312 fix(tools): skip blank/NaN symbol rows in futu and generic journal parsers 
parse_tonghuashun and parse_eastmoney already drop rows with empty
securities codes, but parse_futu stringified a NaN Symbol cell into a
literal "NAN" symbol (classified as a US trade since "NAN".isalpha())
and parse_generic did the same with a "nan" symbol. Rows with
blank/NaN symbols now skip through the same _is_empty_code guard,
keeping phantom symbols out of FIFO pairing and journal stats.

Refs #714

Signed-off-by: Robin1987China <41602358+Robin1987China@users.noreply.github.com>
 2026-07-19 14:32:41 +08:00
Robin1987China bb7a2cc00e fix(shadow): skip the entry-hour gate on daily bars 
Rules mined from real journals carry an entry_hour window from actual
fill times (p10-p90, e.g. 9-11 for market-hours trading). The generated
signal engine replays on daily bars stamped at midnight, so gating on
bar_hour rejected every bar: any profile mined from real intraday fills
produced a silently flat strategy with zero entries and no error.

On daily (or coarser) bars the mined hour window carries no
information, so the generated engine now applies it only when the bar
index has more than one distinct time-of-day (intraday data). Other
rule conditions (RSI, prior return, holding) are unchanged.

Regression tests cover: market-hours window entering on midnight daily
bars, intraday data still enforcing the window, and date-only journal
windows ({0, 0}) still entering.

Signed-off-by: Robin1987China <41602358+Robin1987China@users.noreply.github.com>
 2026-07-19 13:43:00 +08:00
Carls a08525ed18 fix(tools): return real Yahoo news articles for US/HK stock news 
The stock news tool documented that US/.HK codes could not get articles
because yahoo_client.search() only exposes the search endpoint's
instrument quotes, so it returned related-instrument matches instead of
news. The same v1 search endpoint also carries a news array; expose it
via a new throttled yahoo_client.search_news() helper and project the
items into the same article shape the Eastmoney path already returns
(title/url/source/published/snippet).

The stopgap matches contract (result_type='matches') is removed; no
other code in the repo consumes it.

Signed-off-by: Carls <yxhuang@uibe.edu.cn>
 2026-07-19 13:24:26 +08:00
santhreal a4c3652935 test(tools): fix journal filter fixture month window 
Use Jan-Feb so the exclusive upper MonthBegin bound matches two rows.

Signed-off-by: santhreal <64453045+santhreal@users.noreply.github.com>
 2026-07-18 17:59:58 -07:00
santhreal 159bffbef9 fix(tools): reject inverted trade-journal date filters 
filter_expr like "2026-03 to 2026-01" silently emptied the frame and
ran analysis on zero rows. Raise ValueError, matching period inversion.

Signed-off-by: santhreal <64453045+santhreal@users.noreply.github.com>
 2026-07-18 17:59:40 -07:00
santhreal b8ccf2bc3f fix(tools): reject inverted PDF page ranges in doc_reader 
pages like 10-5 produced an empty index list and silently returned no
text. Raise ValueError, matching alpha_bench period inversion.

Signed-off-by: santhreal <64453045+santhreal@users.noreply.github.com>
 2026-07-18 17:53:50 -07:00
santhreal 78d51b4eb5 fix(channels): avoid hang in split_message when max_len <= 0 
When max_len is 0 or negative, the cut pointer never advances and split_message loops forever.
Return the content unsplit so outbound IM adapters cannot wedge.

Signed-off-by: santhreal <64453045+santhreal@users.noreply.github.com>
 2026-07-18 17:09:32 -07:00
Naresh d4beae4b59 fix(cli): render Robinhood's real remote-MCP account/positions shape 
See PR description for full detail.
 2026-07-18 12:47:16 -07:00
Naresh b64d73ea32 fix(trading): forward account_number to Robinhood remote MCP calls 
See PR description for full detail.
 2026-07-18 12:47:16 -07:00
dependabot[bot] 743a498255 build(deps): bump actions/setup-node from 6.4.0 to 7.0.0 
Bumps [actions/setup-node](https://github.com/actions/setup-node) from 6.4.0 to 7.0.0.
- [Release notes](https://github.com/actions/setup-node/releases)
- [Commits](https://github.com/actions/setup-node/compare/48b55a011bda9f5d6aeb4c2d9c7362e8dae4041e...820762786026740c76f36085b0efc47a31fe5020)

---
updated-dependencies:
- dependency-name: actions/setup-node
  dependency-version: 7.0.0
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
 2026-07-18 18:12:45 +00:00
shadowinlife fe141023da fix(test): relax perf gate to 50ms for CI runner variance 
Design target remains 15-35ms on dev machines. The 50ms gate still
guarantees >40x improvement over the pre-optimization 2-2.5s baseline
while accommodating GitHub Actions shared-runner variability.

Signed-off-by: shadowinlife <shadowinlife@gmail.com>

AI-Contributed/Feature: 0/0
AI-Contributed/UT: 0/18
 2026-07-18 22:47:59 +08:00
Haozhe Wu b79e4c54bd docs: add July 18 news (Binance loader + parallel-exec / correctness fixes) 2026-07-18 22:39:31 +08:00
shadowinlife 0b6bf15b65 test: add E2E fund panel compatibility tests for vectorized _align() 
Signed-off-by: shadowinlife <shadowinlife@gmail.com>

AI-Contributed/Feature: 0/0
AI-Contributed/UT: 0/225
 2026-07-18 22:39:31 +08:00
shadowinlife e743a41de2 test: add gold standard regression test for _align() vectorization 
Add TestAlignGoldStandard class that runs both the old pandas scalar
path and new vectorized path with identical inputs, asserting
element-wise output equivalence.

Covers: basic sizes (100-2000 bars), high NaN gaps (20%),
cross-market (ffill_limit=10), all-NaN column drops, and optimizer
callback scenarios.

Ref: reviewer request on PR #698
Signed-off-by: shadowinlife <shadowinlife@gmail.com>


Signed-off-by: shadowinlife <shadowinlife@gmail.com>
AI-Contributed/Feature: 0/0
AI-Contributed/UT: 0/232
 2026-07-18 22:18:05 +08:00
shadowinlife 5efdc7c0f5 fix: ensure array column order matches codes parameter 
The numpy arrays extracted from target_pos and close_df must be
reindexed by the codes parameter to ensure _code_to_col mapping
is consistent regardless of DataFrame internal column order.

Fixes test_rotation_is_independent_of_close_open_symbol_order

Signed-off-by: shadowinlife <shadowinlife@gmail.com>

AI-Contributed/Feature: 0/6
AI-Contributed/UT: 0/0
 2026-07-18 22:18:02 +08:00
shadowinlife c304485dd1 perf: vectorize signal alignment and bar execution (~80× speedup) 
Replace pandas reindex/at operations with numpy array indexing:
- _align(): searchsorted + direct fill replaces per-symbol reindex
- _execute_bars(): pre-extract .values arrays, use row/col indexing
- _safe_price(): add keyword-only fast-path params (_arr/_row/_col)

Performance: 5000 bars × 50 symbols from ~2.5s to 29ms median
Tests: 20/20 existing + 16/17 new regression tests pass

Closes #697

Signed-off-by: shadowinlife <shadowinlife@gmail.com>

AI-Contributed/Feature: 0/175
AI-Contributed/UT: 0/372
 2026-07-18 22:17:59 +08:00
Haozhe Wu ec052ffab2 fix(backtest): block getattr indirection onto os in signal-engine sandbox (GHSA-jqmf F8) 
The AST validator scanned os.<attr> but not getattr(os, "system")(...). Reject
getattr/setattr/delattr whose target is os or a forbidden module — keyed off the
target, not the attr string, so getattr(os, "sys"+"tem") is caught too.
Legit getattr(obj, name, default) on user objects (as the bundled harmonic
example uses) is unaffected.
 2026-07-18 21:46:44 +08:00
Haozhe Wu a8f75628d6 fix(channels): operator-gate IM /pairing control plane (GHSA-fwpw) 
IM /pairing ran with no authorization, so any allow-listed chat participant could
take over pairing across channels (+ Signal group members bypassed the sender
check, + code/sender_id leak). Add a config-driven operators allowlist (global +
per-channel, fail-closed when empty); gate every /pairing on operator identity in
runtime; scope store list/approve/revoke by channel; require per-sender auth for
control-plane slash-commands in Signal groups (ordinary /new /reset unchanged).
README documents the operator requirement.
 2026-07-18 21:46:44 +08:00
Haozhe Wu a1ec1c0cb4 fix(security): neutralize chat-template control tokens in external content (GHSA-v8fm) 
Untrusted web/doc/search content entered the LLM context with no special-token
filtering, letting it forge system/assistant role boundaries. Add
neutralize_special_tokens() (hooked into with_security_warnings, applied to the 3
untrusted ingestion tools) that ZWSP-defangs the ChatML <|...|> family, the
DeepSeek fullwidth-bar <｜...｜> family (the shipped default model), Llama
[INST]/<<SYS>>/<s>, and Gemma role tokens. Text stays visually identical.
 2026-07-18 21:46:44 +08:00
Haozhe Wu e4498d403f fix(mcp): Host/Origin guard on network transports (GHSA-p3c9) 
sse / streamable-http bound a TCP port with no Host/Origin validation (fastmcp
3.2.4 ships none), so a DNS-rebinding browser page could reach every MCP tool.
Wrap the network ASGI app (mcp.http_app) with TrustedHostMiddleware + an Origin
allow-list (VIBE_TRADING_MCP_ALLOWED_HOSTS, default loopback) and serve via
uvicorn. stdio (default) untouched.
 2026-07-18 21:46:44 +08:00
Haozhe Wu 2f35bce832 fix(api): require the configured key on loopback too (GHSA-7wgj) 
_validate_api_auth / require_event_stream_auth trusted the loopback peer BEFORE
checking API_AUTH_KEY, so a same-host reverse-proxy/sidecar reached key-protected
routes with no token. Invert to key-first (mirrors require_settings_write_auth):
when a key is set every peer incl. loopback must present a valid bearer/ticket;
only the no-key dev mode keeps loopback trust. No-key default UX unchanged; the
bundled frontend already sends the bearer/ticket when a key is stored. Updates
3 tests that codified the loopback-bypass behavior.
 2026-07-18 21:46:44 +08:00
Haozhe Wu 106edbdb41 fix(mcp): stop stdio transport from force-enabling shell tools (GHSA-6wjh) 
MCP stdio previously force-enabled bash/background_run with no opt-out and the
module-level default was fail-open. Default to fail-closed and gate every
transport behind explicit opt-in (--enable-shell-tools flag or
VIBE_TRADING_ENABLE_SHELL_TOOLS). Local CLI is unaffected (still opts in
explicitly). Also shrinks the network-reachable surface of the bash
OS-command-injection sink (GHSA-m768). Adds regression tests + doc alignment.
 2026-07-18 21:46:43 +08:00
Haozhe Wu 7cd998da1f docs(skill): bump market-data source count to 22 for Binance loader (#643) 2026-07-18 19:28:33 +08:00
Haozhe Wu 4275b85fe6 Merge PR #643 2026-07-18 19:25:53 +08:00
Haozhe Wu 1ad07f316c Merge PR #645 2026-07-18 19:25:52 +08:00
Haozhe Wu eb518947aa Merge PR #636 2026-07-18 19:25:52 +08:00
Haozhe Wu 9cc7d273ab Merge PR #634 2026-07-18 19:25:52 +08:00
Haozhe Wu dbaefaa00e Merge PR #705 2026-07-18 19:25:52 +08:00
Haozhe Wu 0f88c50fbe Merge PR #707 2026-07-18 19:25:52 +08:00
Haozhe Wu 96f7a9174c Merge PR #714 2026-07-18 19:25:52 +08:00
Haozhe Wu 3aa7aebef7 Merge PR #713 2026-07-18 19:25:52 +08:00
Haozhe Wu 0bcc2d5937 Merge PR #712 2026-07-18 19:25:52 +08:00
Haozhe Wu b1964cdedb Merge PR #711 2026-07-18 19:25:52 +08:00
Haozhe Wu 7d69947327 Merge PR #710 2026-07-18 19:25:52 +08:00
Haozhe Wu ef680a4213 Merge PR #709 2026-07-18 19:25:52 +08:00
Marnie 64e748035f test(frontend): add @testing-library/react component render test for streaming DOM stability 
Add 6 component-level tests that render the streaming area and
simulate state transitions to verify no insertBefore errors occur:
- placeholder → streaming text transition
- streaming text → tool calls transition
- rapid reasoning → text → tools transitions (insertBefore monkey-patch)
- clean unmount when streaming ends
- stream_reset keeps wrapper mounted

All 287 tests pass (277 existing + 10 new).

Signed-off-by: Marnie <Marnie0415@users.noreply.github.com>
 2026-07-18 17:42:55 +08:00
Marnie a8af0ab7ef test(frontend): add streaming DOM stability regression tests 
Add 4 regression tests for the insertBefore DOM race condition fix:
- streaming wrapper stays mounted across state transitions
- rapid delta accumulation does not cause state inconsistency
- stream_reset clears streaming text without crashing
- concurrent tool calls and streaming text coexist

All 281 tests pass (277 existing + 4 new).

Signed-off-by: Marnie <Marnie0415@users.noreply.github.com>
 2026-07-18 17:37:04 +08:00
Marnie 80afd09c8f fix(frontend): prevent insertBefore DOM race during streaming output 
During streaming, the chat UI rendered two separate conditional blocks
(pre-stream placeholder and live streaming area) that mounted/unmounted
independently based on state transitions. When reasoningActive switched
to streamingText, React had to simultaneously unmount one div and mount
another, while scrollToBottom requestAnimationFrame inserted DOM
operations between React reconciliation cycles, causing the reference
node to disappear and triggering an insertBefore NotFoundError.

Fix by merging both blocks into a single stable wrapper div that only
checks status === streaming, with inner conditionals controlling
which content is shown. The wrapper itself never unmounts during
streaming, eliminating the DOM race.

Also debounce ConversationTimeline scroll handler DOM queries with
requestAnimationFrame to avoid conflicts with React batched updates.

Signed-off-by: Marnie <Marnie0415@users.noreply.github.com>
 2026-07-18 17:30:48 +08:00
Robin1987China 5089373025 fix(mcp): align factor_analysis wrapper with the registered tool contract 
The MCP factor_analysis wrapper forwarded codes/factor_name/dates/source/
top_n/bottom_n, but the registered FactorAnalysisTool requires pre-built
CSV artifact paths (factor_csv, return_csv, output_dir). Every MCP call
died on KeyError: 'factor_csv' before any analysis ran.

Mirror the registered tool's real contract in the wrapper signature and
forward exactly those keys, and pin the contract with regression tests
(including an end-to-end call through the real registry on synthetic
CSVs).

Closes #635

Signed-off-by: Robin1987China <41602358+Robin1987China@users.noreply.github.com>
 2026-07-18 15:11:16 +08:00
santhreal a43c519fbb fix(journal): treat NaN securities codes as empty 
CSV/Excel blank cells arrive as NaN; str(nan) was truthy and still
produced garbage tickers. Centralize empty-code detection with pd.isna.

Signed-off-by: santhreal <64453045+santhreal@users.noreply.github.com>
 2026-07-17 21:13:55 -07:00
santhreal e227a8b735 fix(validation): drop single-column equity.csv fallback 
Only accept explicit equity/nav/value columns so an unrelated lone
column is not silently treated as the equity curve.

Signed-off-by: santhreal <64453045+santhreal@users.noreply.github.com>
 2026-07-17 21:13:53 -07:00
santhreal bae719209c fix(factors): validate n_groups before writing artifacts 
Reject non-positive n_groups at the tool entrypoint instead of catching
any ValueError from compute_group_equity after IC files are written.

Signed-off-by: santhreal <64453045+santhreal@users.noreply.github.com>
 2026-07-17 21:13:52 -07:00
santhreal 1c34a5b725 fix(journal): reject empty A-share codes instead of 000000.SZ 
Blank securities codes were zfill(6)-mapped to 000000.SZ. Reject empty codes in _qualify_a_share and skip blank-code rows when parsing Tonghuashun/Eastmoney exports.

Signed-off-by: santhreal <64453045+santhreal@users.noreply.github.com>
 2026-07-17 21:10:12 -07:00
santhreal fb4b7a365a fix(validation): accept equity.csv nav/value column aliases 
validation CLI hardcoded df['equity'], so artifacts that store nav or value raised KeyError. Accept equity/nav/value (or a single numeric column).

Signed-off-by: santhreal <64453045+santhreal@users.noreply.github.com>
 2026-07-17 21:10:09 -07:00
santhreal b9325075ee fix(pattern): reject non-positive detection window 
pattern detectors passed window<=0 into peak scans and polyfit, causing IndexError, TypeError, or every-bar peak/valley garbage. Require window >= 1 and return a JSON error from run_pattern.

Signed-off-by: santhreal <64453045+santhreal@users.noreply.github.com>
 2026-07-17 21:10:06 -07:00
santhreal f684178dd2 fix(backtest): handle unnamed DatetimeIndex in correlation matrix 
\_rolling_correlation_matrix assumed a trade_date column when the index was not named trade_date, so standard OHLCV frames with an unnamed DatetimeIndex raised KeyError. Accept DatetimeIndex close series directly.

Signed-off-by: santhreal <64453045+santhreal@users.noreply.github.com>
 2026-07-17 21:10:03 -07:00
santhreal 29a6ec2c89 fix(alpha): reject inverted period ranges in _parse_period 
\_parse_period accepted inverted ranges like 2024-2020 and returned a reversed (start, end) pair. Reject start > end with ValueError, matching validate_date_range.

Signed-off-by: santhreal <64453045+santhreal@users.noreply.github.com>
 2026-07-17 21:10:00 -07:00
santhreal fddcd5b2b0 fix(factors): reject non-positive n_groups in factor analysis 
compute_group_equity passed n_groups=-1 into pandas.qcut and raised
IndexError. Validate n_groups >= 1 and return a JSON error from the
factor_analysis tool instead of crashing.

Signed-off-by: santhreal <64453045+santhreal@users.noreply.github.com>
 2026-07-17 21:02:58 -07:00
shadowinlife af6e77a078 fix(swarm): include enabled_tools in MCP specs cache key 
The cache key for MCP tool discovery results must include the
enabled_tools configuration. Without it, calls to the same server
with different enabledTools filters incorrectly return cached specs
from a prior call with a different filter, bypassing tool filtering.

This fixes the CI failures in test_mcp_stdio_integration,
test_mcp_sse_integration, and test_mcp_streamable_http_integration
where enabledTools=["echo"] was not being respected.

Signed-off-by: shadowinlife <shadowinlife@gmail.com>

AI-Contributed/Feature: 0/1
AI-Contributed/UT: 0/0
 2026-07-18 11:14:44 +08:00
sambazhuandClaude d76b819a5c fix(api): add fundamental zoo to _VALID_ZOOS whitelist 
The fundamental factor zoo (zoo/fundamental/, 4 PIT-safe alphas) shipped
in v0.1.11 and has a frontend card in AlphaZoo, but the API whitelist was
never updated — GET /alpha/list?zoo=fundamental and POST /alpha/bench both
reject it with 400 "unknown zoo 'fundamental'".

Add "fundamental" to _VALID_ZOOS so both endpoints accept it.

Co-Authored-By: Claude <noreply@anthropic.com>
 2026-07-18 09:55:37 +08:00
Michele Cerreto c4aff5e10a fix(ibkr): replace fixed 2s sleep with ib.sleep() polling loop that exits on real data 
The single ib.sleep(2.0) call in _wait_for_tick was unreliable when
multiple quotes ran in parallel via ThreadPoolExecutor (8 workers).
Each thread has its own IB connection (thread-local pool), but the
fixed 2s sleep doesn't account for varying network latency -- some
ticks arrive in 200ms, others take longer, and 2s wasn't always
enough, causing empty/NaN fields in 7/10 parallel quote requests.

Replace with incremental ib.sleep(0.1) polling that checks
_tick_has_data after each pump and exits as soon as real market data
arrives, with a 5s total timeout. This gives each ticker dedicated
event-loop time and avoids the race where the LLM reads partial data.

Key: uses ib.sleep() NOT waitOnUpdate() for polling -- ib.sleep
reliably processes all pending socket messages, while waitOnUpdate
can return True for unrelated messages on the same connection.

Changes:
- Re-add _tick_has_data() with None + NaN rejection
- Rewrite _wait_for_tick() as ib.sleep(0.1) polling loop (5s timeout)
- Add test_quote_waits_for_delayed_tick_arrival: tick arrives after
  5 pump cycles, verifies correct bid/ask/last values
- Add test_quote_keeps_polling_when_ticker_is_nan: fields start NaN,
  continue polling until real data (200.0) arrives after 8 pumps
 2026-07-18 00:20:11 +02:00
Michele Cerreto 81b8d0d331 fix(ibkr): replace waitOnUpdate polling with ib.sleep(2.0), remove _tick_has_data, raise on qualify failure 
The incremental waitOnUpdate() polling in _wait_for_tick was unreliable
under ThreadPoolExecutor (8 workers). waitOnUpdate() returns True for any
socket message, not necessarily the targeted ticker, which meant parallel
quote requests across threads would often time out (2s) with NaN fields.

Replace the 20x100ms loop with a single ib.sleep(2.0), which processes
all pending socket messages in one shot -- the approach the original
pre-pool code used successfully.

Changes in local.py:
- Remove _tick_has_data() -- no longer needed (no polling loop)
- Simplify _wait_for_tick() to a single _sleep(ib, 2.0) call
- Update _wait_for_tick docstring to explain the single-sleep rationale
- Fix _qualify_contract() to raise IBKRConnectionError instead of
  silently swallowing exceptions (contracts like Samsung KRX fail
  qualification and previously returned garbage data)

Changes in test_ibkr_local.py:
- Remove the _wait_for_tick timing regression test section (3 tests:
  test_quote_waits_for_tick_arrival, test_quote_receives_tick_after_event_loop_pump,
  and the delayed-ticker/pump-count fixtures)
- Remove the NaN rejection test section (test_quote_rejects_nan_and_waits_for_real_data
  and _NaNTickerIB fixtures)
- Add test_quote_calls_sleep_for_tick_arrival: clean test verifying
  get_quote calls ib.sleep(2.0) and returns valid quote data
 2026-07-18 00:03:05 +02:00
Michele Cerreto cf26d30aab fix(ibkr): reject NaN ticker values in _wait_for_tick 
The _wait_for_tick loop checked `value is not None` before exiting,
but ib_async tickers return `float("nan")` for fields that have been
processed but contain no trade data (e.g. after-hours, illiquid
symbols).  Since NaN is not None, the loop exited immediately and
returned NaN bid/ask/last -- breaking all quote-dependent analysis.

## Fix

- Added _tick_has_data(ticker) helper that rejects both None and NaN.
  Only returns True when at least one of bid/ask/last is a non-NaN
  numeric value.
- _wait_for_tick now calls _tick_has_data() instead of inline None
  checks.
- Added test_quote_rejects_nan_and_waits_for_real_data -- mock returns
  NaN ticker; verifies the wait loop pumps until timeout (does not exit
  early), keeping the safety ceiling.

All 16 tests pass.
 2026-07-17 23:18:17 +02:00
Michele Cerreto 6b61f9402d fix(ibkr): restore tick-wait in get_quote and add timing + concurrency tests 
## Problem 1: Empty ticker in get_quote (timing regression)

Commit a1070a1 changed reqMktData(..., snapshot=True) to avoid the
2-second sleep and thread-safety deadlock, but the ib_async documentation
states: "The ticker will initially be empty and gradually (after a couple
of seconds) be filled" -- even with snapshot mode.  Reading the ticker
fields immediately after reqMktData returned None for bid/ask/last in
production, silently breaking all quote-dependent analysis.

## Problem 2: Existing test hid the bug

The _FakeIB.reqMktData mock returned a fully-populated ticker instantly
(bid=100.0, ask=100.2, ...), so the timing regression never surfaced in
CI.  The existing test_quote_and_history_are_readonly test passed despite
the bug.

## Solution

### agent/src/trading/connectors/ibkr/local.py

- Added _wait_for_tick(ib, ticker) helper that pumps the ib_async event
  loop via ib.waitOnUpdate() in 100 ms increments until at least one of
  bid/ask/last is populated, with a configurable 2-second timeout.
  Falls back to ib.sleep() for ib_async versions without waitOnUpdate.
- get_quote now calls _wait_for_tick() immediately after
  reqMktData(snapshot=True), ensuring the ticker fields are filled
  before serializing the quote response.
- Moved import itertools to the top-level import block (was previously
  added inline before the _TwsPool class in commit a1070a1).

### agent/tests/test_ibkr_local.py -- 5 new tests

Timing tests (2):
- test_quote_waits_for_tick_arrival -- mock returns an initially-empty
  ticker; the wait loop runs without hanging, exits gracefully (fields
  stay None if data never arrives -- no deadlock).
- test_quote_receives_tick_after_event_loop_pump -- simulates the real
  async-tick flow: ticker starts empty, after 2 event-loop pumps
  _obj_get is patched to return populated values.  Asserts bid=150.0,
  ask=150.5, last=150.25 are correctly read after the wait loop.

Thread-local pool concurrency tests (3):
- test_pool_thread_local_isolation -- 4 concurrent threads acquire the
  pool via ThreadPoolExecutor with threading.Barrier to force distinct
  worker threads.  Each thread gets a unique client ID (> base 77) and
  a distinct ib_async.IB object.  Verifies no cross-thread leakage.
- test_pool_refcount_disconnects_on_last_release -- nested
  acquire/release in the same thread: disconnect is called only when
  the refcount reaches zero after the final release.
- test_pool_release_idempotent_no_connection -- calling release() on an
  empty pool is a safe no-op.

## Impact

- Quotes now work correctly in production -- ticker fields are populated
  before reading.
- All 10 existing + 5 new = 15 tests pass.
- The wait loop adds ~100-300 ms per quote in the common case (tick
  arrives quickly), with a 2-second safety ceiling.
 2026-07-17 22:34:31 +02:00
xkam7ar 69ad82b4e5 fix(web): ignore stale authenticated SSE connections 
Signed-off-by: xkam7ar <kmoteallemi@icloud.com>
 2026-07-17 10:51:11 -07:00
xkam7ar b95c1c2a1a fix(cli): return failures from channel commands 
Signed-off-by: xkam7ar <kmoteallemi@icloud.com>
 2026-07-17 10:50:52 -07:00
xkam7ar df24ecf955 fix(cli): clean up development server children 
Signed-off-by: xkam7ar <kmoteallemi@icloud.com>
 2026-07-17 10:50:33 -07:00
xkam7ar d824b502e9 fix(cli): keep continuation history in one trace 
Signed-off-by: xkam7ar <kmoteallemi@icloud.com>
 2026-07-17 10:50:20 -07:00
xkam7ar b7cd2c98b2 fix(swarm): redact task details in the REST API 
Signed-off-by: xkam7ar <kmoteallemi@icloud.com>
 2026-07-17 10:50:14 -07:00
xkam7ar c2a20d2209 fix(swarm): save nested output files safely 
Signed-off-by: xkam7ar <kmoteallemi@icloud.com>
 2026-07-17 10:49:23 -07:00
xkam7ar cd8acfc814 fix(background): stop timed-out process trees 
Signed-off-by: xkam7ar <kmoteallemi@icloud.com>
 2026-07-17 10:48:39 -07:00
xkam7ar 23d87084d8 fix(scheduler): match standard cron day rules 
Signed-off-by: xkam7ar <kmoteallemi@icloud.com>
 2026-07-17 10:47:05 -07:00
xkam7ar 435f1b85ff fix(web): clear stale correlation output 
Signed-off-by: xkam7ar <kmoteallemi@icloud.com>
 2026-07-17 10:46:52 -07:00
xkam7ar 691a9bcc85 fix(web): isolate run detail state by route 
Signed-off-by: xkam7ar <kmoteallemi@icloud.com>
 2026-07-17 10:46:39 -07:00
xkam7ar 2d82a300bf fix(web): ignore stale comparison responses 
Signed-off-by: xkam7ar <kmoteallemi@icloud.com>
 2026-07-17 10:46:26 -07:00
xkam7ar 3705564ed3 fix(web): proxy auth routes in Vite 
Signed-off-by: xkam7ar <kmoteallemi@icloud.com>
 2026-07-17 10:45:27 -07:00
xkam7ar adb5b4d6c6 fix(packaging): fix the Slack install version 
Signed-off-by: xkam7ar <kmoteallemi@icloud.com>
 2026-07-17 10:44:55 -07:00
ebujinovch 919e96caf1 fix(factors): corr_rewire must keep missing bars missing under pandas 2.x 
pct_change() without fill_method forward-fills gaps on pandas 2.x (the CI
pin), silently defeating the coverage guard that test_corr_rewire's
low-coverage case asserts. Explicit fill_method=None matches the bench
runner's own forward-return convention and behaves identically on pandas 3.

Signed-off-by: ebujinovch <ch.ebujinov@gmail.com>
 2026-07-17 17:56:10 +02:00
ebujinovch eafdd4bf9f feat(factors): add academic_corr_rewire (correlation-rewiring stability score) 
Signed-off-by: ebujinovch <ch.ebujinov@gmail.com>

# Conflicts:
#	agent/SKILL.md
 2026-07-17 17:35:43 +02:00
shadowinlife d95626d4e6 perf(swarm): cache MCP tool discovery specs to avoid redundant RPC calls 
Add a module-level thread-safe cache in src/tools/mcp.py that stores
MCPRemoteToolSpec lists keyed by (server_name, command, args, env).
Within a single Swarm run, identical MCP servers are discovered only
once regardless of how many workers or layers execute.

SwarmRuntime._execute_run() calls invalidate_mcp_specs_cache() at the
start of each run to ensure fresh discovery when operator config may
have changed between runs.

Reduces tool discovery overhead from O(Workers x Layers x Servers) to
O(Servers), yielding ~6x speedup in typical multi-layer DAG scenarios.

Closes #703

Signed-off-by: shadowinlife <shadowinlife@gmail.com>
 2026-07-17 23:35:38 +08:00
Haozhe Wu d9cbce553b docs: add July 17 project news (correlation-regime skill + correctness pass) 2026-07-17 23:05:35 +08:00
Haozhe Wu ac9889a5ba revert(loaders): remove feitu / FTShare data source 
#567 added feitu, a client SDK for the contributor-operated FTShare data
service. Reverting it: a 3-week-old, author-affiliated backend whose free tier
is at the operator's discretion is not a neutral public data source, and the
A/HK fallback chains already carry ample free coverage (tencent, mootdx,
eastmoney, baostock, akshare, yahoo, futu). Removed the loader, its test,
registry entries, fallback-chain slots, and doc/source-count references
(sources 22->21, free 20->19).
 2026-07-17 22:41:55 +08:00
Haozhe Wu e1d6a3558b Merge PR #632 2026-07-17 22:15:12 +08:00
Haozhe Wu 3bb1ff2f84 Merge PR #630 2026-07-17 22:14:50 +08:00
Haozhe Wu c500ae93e0 Merge PR #686 2026-07-17 22:14:50 +08:00
Haozhe Wu 358166f3c7 Merge PR #685 2026-07-17 22:14:50 +08:00
Haozhe Wu 90f5dd0a25 Merge PR #628 2026-07-17 22:14:50 +08:00
Haozhe Wu 7d39cf3b49 Merge PR #626 2026-07-17 22:14:50 +08:00
Haozhe Wu 0476b710d6 Merge PR #594 2026-07-17 22:14:50 +08:00
Haozhe Wu cc51644b90 fix(swarm): redact absolute preset path in not-found error (P10 CWE-209) 2026-07-17 21:42:51 +08:00
Haozhe Wu 30a533add2 Merge PR #610 2026-07-17 21:34:39 +08:00
Haozhe Wu 0e814a8b43 Merge PR #616 2026-07-17 21:33:29 +08:00
Haozhe Wu 9a64a4f249 Merge PR #673 2026-07-17 21:33:29 +08:00
Haozhe Wu 682efbb40e Merge PR #672 2026-07-17 21:33:29 +08:00
Haozhe Wu 41785db633 Merge PR #660 2026-07-17 21:33:12 +08:00
Haozhe Wu 7ce5765bca Merge PR #659 2026-07-17 21:32:37 +08:00
Haozhe Wu 593dbe4f94 Merge PR #658 2026-07-17 21:32:24 +08:00
Haozhe Wu ee776133b7 Merge PR #657 2026-07-17 21:31:16 +08:00
Haozhe Wu 10e3be933c Merge PR #576 2026-07-17 21:31:03 +08:00
Haozhe Wu 10e9a8d9e5 Merge PR #574 2026-07-17 21:30:18 +08:00
Haozhe Wu 9577f2daca Merge PR #670 2026-07-17 21:30:03 +08:00
Haozhe Wu 8a8132aab8 Merge PR #669 2026-07-17 21:29:25 +08:00
Haozhe Wu facc97cec1 Merge PR #570 2026-07-17 21:29:25 +08:00
Haozhe Wu 618650c298 Merge PR #557 2026-07-17 21:29:12 +08:00
Haozhe Wu ff4effb16d Merge PR #569 2026-07-17 21:27:58 +08:00
Haozhe Wu 65446854f8 Merge PR #567 2026-07-17 21:26:15 +08:00
Haozhe Wu 06a90715f2 Merge PR #688 2026-07-17 21:25:53 +08:00
Haozhe Wu fd137d697d Merge PR #654 2026-07-17 21:25:02 +08:00
Haozhe Wu 2056e8fb57 Merge PR #687 2026-07-17 21:25:02 +08:00
Haozhe Wu 0f24c456df Merge PR #656 2026-07-17 21:25:01 +08:00
Haozhe Wu 1b2a84c34c Merge PR #552 2026-07-17 21:25:01 +08:00
Haozhe Wu 0fe877e1e2 Merge PR #692 2026-07-17 21:24:42 +08:00
Haozhe Wu a8289ac7ae Merge PR #691 2026-07-17 21:24:42 +08:00
Haozhe Wu f92c1a2c5c Merge PR #690 2026-07-17 21:24:42 +08:00
Haozhe Wu d892a53134 Merge PR #671 2026-07-17 21:24:42 +08:00
Haozhe Wu 3ca02b56fc Merge PR #655 2026-07-17 21:24:42 +08:00
Haozhe Wu c536ef5ef7 Merge PR #624 2026-07-17 21:24:42 +08:00
Haozhe Wu 20fdfb135a Merge PR #622 2026-07-17 21:24:42 +08:00
Haozhe Wu ef6712a83f Merge PR #620 2026-07-17 21:24:42 +08:00
Haozhe Wu 5d813c5953 Merge PR #608 2026-07-17 21:24:42 +08:00
Haozhe Wu 7fc5fbf43f Merge PR #602 2026-07-17 21:24:42 +08:00
Haozhe Wu 12b55a8b73 Merge PR #598 2026-07-17 21:24:42 +08:00
Haozhe Wu 2fb6dd7666 Merge PR #582 2026-07-17 21:24:42 +08:00
Haozhe Wu c5897e6758 Merge PR #580 2026-07-17 21:24:42 +08:00
sambazhuandClaude 03adce85a3 fix(providers): force temperature=1 for the whole Kimi K-series 
kimi-k3 rejects any temperature other than 1 ("invalid temperature:
only 1 is allowed for this model"), same as kimi-k2, but the forcing
guard only matched the kimi-k2 prefix — so a default
LANGCHAIN_TEMPERATURE=0.0 config fails on every request with kimi-k3.

Replace the hardcoded prefix tuple with a K-family regex
(kimi-k<digits> + kimi-for-coding) so kimi-k3 and future K-series
reasoning models are covered without another whack-a-mole patch.

Verified against the Kimi for Coding endpoint: both kimi-k3 and its
k3 alias return the temperature error; moonshot-v1-* models are
unaffected. Note: bare aliases like "k3" are intentionally out of
scope for the name-based guard.

Co-Authored-By: Claude <noreply@anthropic.com>
 2026-07-17 16:05:02 +08:00
Jelech f21d8620fd feat(providers): support ANTHROPIC message 2026-07-17 13:31:44 +08:00
xkam7ar f73aa90e55 fix(qveris): reserve credits before paid calls 
Signed-off-by: xkam7ar <kmoteallemi@icloud.com>
 2026-07-16 19:29:24 -07:00
tyj147454413-cmd c7bff740b8 fix(codex): classify stream HTTP failures 2026-07-17 10:27:31 +08:00
xkam7ar 4c4f7df53a fix(backtest): enforce the QVeris credit budget 
Signed-off-by: xkam7ar <kmoteallemi@icloud.com>
 2026-07-16 19:26:58 -07:00
xkam7ar 5c976c3f08 fix(data): fail incomplete mootdx pagination 
Signed-off-by: xkam7ar <kmoteallemi@icloud.com>
 2026-07-16 19:25:17 -07:00
xkam7ar 70eaf95a4f fix(data): reject incomplete CCXT history 
Signed-off-by: xkam7ar <kmoteallemi@icloud.com>
 2026-07-16 19:24:10 -07:00
xkam7ar e238a6b296 fix(data): reject unsupported Tencent intervals 
Signed-off-by: xkam7ar <kmoteallemi@icloud.com>
 2026-07-16 19:22:41 -07:00
tyj147454413-cmd 2962790a43 fix(state): fsync persisted run state 2026-07-17 10:21:48 +08:00
tyj147454413-cmd 5ce9a04dca fix(okx): bound history fetches and handle rate limits 2026-07-17 10:21:29 +08:00
xkam7ar 0142f0d4f7 fix(live): halt on corrupt runtime state 
Signed-off-by: xkam7ar <kmoteallemi@icloud.com>
 2026-07-16 19:21:08 -07:00
xkam7ar 0940c15b33 fix(live): halt on recorded-only positions 
Signed-off-by: xkam7ar <kmoteallemi@icloud.com>
 2026-07-16 19:21:08 -07:00
xkam7ar 6a38df34ff fix(live): serialize daily-cap admission and submit 
Signed-off-by: xkam7ar <kmoteallemi@icloud.com>
 2026-07-16 19:21:08 -07:00
xkam7ar 12aec9b257 fix(live): calculate gross exposure after each order 
Signed-off-by: xkam7ar <kmoteallemi@icloud.com>
 2026-07-16 19:21:08 -07:00
xkam7ar ec86d9b90f fix(live): publish consent before mandate authority 
Signed-off-by: xkam7ar <kmoteallemi@icloud.com>
 2026-07-16 19:21:08 -07:00
tyj147454413-cmd b364d793e4 feat(loaders): add Binance crypto fallback 2026-07-17 10:21:07 +08:00
xkam7ar f429063e61 fix(data): preserve requested crypto symbols 
Signed-off-by: xkam7ar <kmoteallemi@icloud.com>
 2026-07-16 19:19:32 -07:00
xkam7ar 8d61745504 fix(backtest): size China futures by symbol margin 
Signed-off-by: xkam7ar <kmoteallemi@icloud.com>
 2026-07-16 19:18:27 -07:00
xkam7ar 74c54de5b8 fix(backtest): route futures by submarket 
Signed-off-by: xkam7ar <kmoteallemi@icloud.com>
 2026-07-16 19:16:42 -07:00
xkam7ar b0793637fe fix(portfolio): constrain risk-parity weights 
Signed-off-by: xkam7ar <kmoteallemi@icloud.com>
 2026-07-16 19:15:40 -07:00
xkam7ar 869e78b294 fix(backtest): reuse the validated data snapshot 
Signed-off-by: xkam7ar <kmoteallemi@icloud.com>
 2026-07-16 19:13:59 -07:00
xkam7ar 8dc13d0a79 fix(alpha): enforce sample floors per split 
Signed-off-by: xkam7ar <kmoteallemi@icloud.com>
 2026-07-16 19:12:24 -07:00
xkam7ar 26af89352e fix(alpha): require the strict training threshold 
Signed-off-by: xkam7ar <kmoteallemi@icloud.com>
 2026-07-16 19:10:58 -07:00
xkam7ar e5a7b70f52 fix(shadow): exclude incomplete buy-day bars 
Signed-off-by: xkam7ar <kmoteallemi@icloud.com>
 2026-07-16 19:09:04 -07:00
xkam7ar e62582b554 fix(shadow): skip rules for unmatched markets 
Signed-off-by: xkam7ar <kmoteallemi@icloud.com>
 2026-07-16 19:07:52 -07:00
xkam7ar b51b9413ce fix(agent): preserve queued cancellation signals 
Signed-off-by: xkam7ar <kmoteallemi@icloud.com>
 2026-07-16 19:07:31 -07:00
Michele Cerreto a1070a1b12 fix(ibkr): use snapshot quotes and thread-local connection pool 
## Problem

The previous shared-connection pool was incompatible with the agent's
parallel tool executor (ThreadPoolExecutor, up to 8 workers).  When the
agent fired 15 trading_quote calls in one iteration every worker thread
raced into the same `ib_async.IB` socket.  `ib_async` is not
thread-safe — its internal event loop is bound to one thread — so the
parallel access deadlocked the market-data stream.  Every session trace
ended with tool_call events but zero tool_result events for quotes.

Additionally, `get_quote` called `reqMktData` with `snapshot=False`
(streaming subscription) followed by a 2-second sleep and manual cancel.
This held the connection for 2+ seconds per symbol even when it worked.

## Solution

Two changes in `agent/src/trading/connectors/ibkr/local.py`:

1. **`reqMktData(..., snapshot=True)`** — requests a one-shot snapshot
   instead of a streaming subscription.  No sleep needed; the ticker
   delivers one tick and auto-cancels.  Drops the `_sleep(2.0)` and
   `cancelMktData` dance entirely.

2. **Thread-local pool** — replaces the shared `_TwsPool` with a
   `threading.local()`-based design:

   - Each worker thread gets its own `ib_async.IB` socket.
   - Unique client IDs via `itertools.count` + base config ID.
   - Per-thread reference counting so the connection persists across
     multiple tool calls in the same thread and disconnects only when
     the refcount drops to zero.
   - No socket is ever shared across threads — respects ib_async's
     single-thread-per-socket design.

## Impact

- No more hangs when the agent fires parallel trading_quote calls.
- Quotations complete in ~100 ms instead of 2+ seconds.
- All 10 `test_ibkr_local.py` tests pass without modification.
 2026-07-16 23:03:55 +02:00
Michele Cerreto 3896822a14 fix(ibkr): replace per-call connect/disconnect with thread-safe connection pool 
## Problem
The local IBKR connector opened a fresh `ib_async.IB()` connection and
disconnected it on every single tool call (get_account_snapshot,
get_positions, get_open_orders, get_quote, get_historical_bars).  When
the agent invoked multiple trading tools in parallel every call reused
the same hard-coded client ID (77), which caused TWS / IB Gateway to
reject the duplicates with error 326:

    Unable to connect as the client id is already in use.
    Peer closed connection. clientId 77 already in use?

This made the connector unreliable under the agent's parallel execution
model and surfaced as intermittent `TimeOutError` and `Could not
connect to TWS / IB Gateway` errors in the CLI and Web UI.

## Solution
Replace the module-level `_connect()` / `_disconnect()` helpers with
a thread-safe connection pool class `_TwsPool` backed by a
`threading.Lock` and reference counting.

- `_pool.acquire(config)` connects once on first use; subsequent
  concurrent callers reuse the same `ib_async.IB` socket without
  reconnecting.
- `_pool.release()` decrements the refcount; the underlying socket is
  disconnected only when the last active caller releases.
- Guards against mid-flight profile switching (different host / port /
  profile) with a clear error instead of a silent misconfiguration.

All five public functions updated to use `_pool.acquire(cfg)` /
`_pool.release()`.  Call-site contract is unchanged — same
`try: … finally: _pool.release()` pattern.

## Impact
- No more client-ID collision errors under parallel agent call patterns.
- No resource leak from orphaned `ib_async.IB` instances.
- All 10 existing `test_ibkr_local.py` tests pass without modification.
 2026-07-16 22:01:49 +02:00
xkam7ar df10ac1eeb fix(cli): authenticate live API requests 
Signed-off-by: xkam7ar <kmoteallemi@icloud.com>
 2026-07-16 11:43:23 -07:00
xkam7ar dcebbfe523 fix(cli): send complete mandate requests 
Signed-off-by: xkam7ar <kmoteallemi@icloud.com>
 2026-07-16 11:42:58 -07:00
xkam7ar 41ad3396e3 ci: run environment gate unit tests 
Signed-off-by: xkam7ar <kmoteallemi@icloud.com>
 2026-07-16 11:41:39 -07:00
xkam7ar 1e214ef1c8 fix(swarm): resume and close event streams correctly 
Signed-off-by: xkam7ar <kmoteallemi@icloud.com>
 2026-07-16 11:41:35 -07:00
xkam7ar 086acc4230 fix(background): report failed command exit codes 
Signed-off-by: xkam7ar <kmoteallemi@icloud.com>
 2026-07-16 11:39:29 -07:00
xkam7ar f6e88870c6 fix(options): handle unusual account values safely 
Signed-off-by: xkam7ar <kmoteallemi@icloud.com>
 2026-07-16 11:39:17 -07:00
xkam7ar 165c0aded5 fix(futu): load every history page 
Signed-off-by: xkam7ar <kmoteallemi@icloud.com>
 2026-07-16 11:38:23 -07:00
xkam7ar 0fecf31d73 fix(futu): use the requested minute interval 
Signed-off-by: xkam7ar <kmoteallemi@icloud.com>
 2026-07-16 11:38:08 -07:00
xkam7ar 2ad7036d1c build: install Black and Ruff for contributors 
Signed-off-by: xkam7ar <kmoteallemi@icloud.com>
 2026-07-16 11:37:48 -07:00
xkam7ar 60f3c63059 ci: run all repository safety checks 
Signed-off-by: xkam7ar <kmoteallemi@icloud.com>
 2026-07-16 11:37:22 -07:00
xkam7ar 2248737bff fix(journal): reject unknown trade directions 
Signed-off-by: xkam7ar <kmoteallemi@icloud.com>
 2026-07-16 11:37:22 -07:00
xkam7ar 9747eec2a1 fix(journal): include full ending months in filters 
Signed-off-by: xkam7ar <kmoteallemi@icloud.com>
 2026-07-16 11:37:22 -07:00
xkam7ar abc4300dd9 fix(hypotheses): reject blank required updates 
Signed-off-by: xkam7ar <kmoteallemi@icloud.com>
 2026-07-16 11:37:22 -07:00
xkam7ar a5e446e8e3 fix(journal): allocate entry fees across partial exits 
Signed-off-by: xkam7ar <kmoteallemi@icloud.com>
 2026-07-16 11:37:22 -07:00
xkam7ar ae2e110b4f fix(data): match local data result names 
Signed-off-by: xkam7ar <kmoteallemi@icloud.com>
 2026-07-16 11:37:22 -07:00
xkam7ar f39ec95ec6 fix(data): normalize local timestamps to UTC 
Signed-off-by: xkam7ar <kmoteallemi@icloud.com>
 2026-07-16 11:37:22 -07:00
xkam7ar 4e0e312b2b fix(swarm): report tool failures as errors 
Signed-off-by: xkam7ar <kmoteallemi@icloud.com>
 2026-07-16 11:37:22 -07:00
xkam7ar 22867b01d9 fix(alpha): stop filling missing prices 
Signed-off-by: xkam7ar <kmoteallemi@icloud.com>
 2026-07-16 11:37:21 -07:00
Ali EmadiandClaude Fable 5 e04c98ca5b feat(swarm): discover user presets from ~/.vibe-trading/swarm/presets 
Preset YAMLs dropped into ~/.vibe-trading/swarm/presets/ are now discovered
alongside the bundled roster, mirroring the user-skills pattern
(src/agent/skills.py): the user directory is searched first, same-name files
override bundled presets, and user files survive pip install -U (previously
the only way to add a preset was copying into site-packages, which upgrades
wipe).

- load_preset()/resolve_preset_path(): user dir first, then bundled; the
  not-found error names both locations.
- list_presets(): union with user-wins-by-stem override; entries gain a
  source: "user"|"bundled" field (additive — CLI/MCP/API consumers read
  keys individually or serialize the whole dict).
- Preset names are validated to a single path segment before any filesystem
  lookup (rejects separators and '..'); the bundled lookup had the same
  latent exposure, so validation covers both paths.
- run_swarm(preset_name=...): an explicitly named user preset now resolves;
  keyword auto-routing intentionally stays limited to the curated table, so
  a user preset is reachable only by naming it.
- Packaging regression suite extended (6 new tests: discovery, override,
  absent-dir no-op, traversal rejection, error message, tool gating); the
  roster-count test now filters to bundled entries so a developer's own user
  presets cannot make it flap.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Ali Emadi <ali@gatekeeperai.com>
 2026-07-16 16:50:15 +02:00
Haozhe Wu db2ff578ae docs: add July 16 project news 2026-07-16 20:12:02 +08:00
root eecb86e1bc feat: add Longbridge connector runtime connection card 
- Runtime page: add Longbridge connection card with status, capability summary, verify/inspect
- RunnerStatus: filter connector-specific SDK-only connectors from top bar
- API: expose /live/connectors/{id}/verify with SDK credential validation
- i18n: add Longbridge runtime strings (en/zh-CN/ja/ko/ar) — parity test enforced
- Tests: Runtime.test.tsx (27 tests), RunnerStatus.test.tsx (5 tests), i18n parity (20 tests)
 2026-07-16 19:31:22 +08:00
root ea89dd17d5 feat: Longbridge SDK security hardening and data source integration 
- SDK: sanitize runtime metadata, reject non-readonly profiles, suppress unrepresentable timestamps
- API: expose Longbridge connector runtime status via /live/status, redact credentials in error paths
- Loader: redact credential boundaries, eliminate __context__ secret leakage
- Market data tool: share Longbridge credentials across data and trading
- Tests: cover safe SDK metadata vocabularies, loader redaction, credential flow
- README: document Longbridge as optional extra installation
 2026-07-16 19:24:02 +08:00
root 0de5a78ccd build: add Longbridge credentials module and optional dependency 
- Add credentials.py: unified credential resolution for data and trading
- Add test_longbridge_credentials.py: coverage for credential resolution
- Remove longbridge from requirements.txt (optional extra only)
- Update requirements-lock.txt
- Update .env.example with Longbridge credential hints
 2026-07-16 19:23:54 +08:00
FenjuFu c5f6afabaf feat(providers): add iFlytek Spark as an OpenAI-compatible provider 
Wire iFlytek Spark's OpenAI-compatible HTTP endpoint
(https://spark-api-open.xf-yun.com/v1, Bearer APIPassword) through the
provider registry, capabilities, legacy CLI onboarding, .env.example, and
the README provider list, with "iflytek" accepted as an alias. Spark rides
the generic OpenAI-compatible path with no special capability flags; the
default model is 4.0Ultra, which supports function calling.

Signed-off-by: FenjuFu <92919259+FenjuFu@users.noreply.github.com>
 2026-07-16 18:11:28 +08:00
shadowinlife b5770489f3 fix(ci): align noqa env-gate markers with os.getenv call lines 
The CI env var gate checks the AST node's start line for the noqa
comment. Move markers from the closing parenthesis line to the
os.getenv( call line so the gate recognizes the exemption.

Signed-off-by: shadowinlife <shadowinlife@gmail.com>
 2026-07-16 17:52:08 +08:00
shadowinlife d3aa5847a9 refactor(providers): use get_llm_credentials in _build_native_deepseek 
Replace manual credential resolution in _build_native_deepseek() with
the centralized get_llm_credentials() helper, completing the
consolidation goal of #553.

Signed-off-by: shadowinlife <shadowinlife@gmail.com>
 2026-07-16 17:38:46 +08:00
JellyfishKnight bca9d49593 feat(data): add FTShare market data source 
Signed-off-by: JellyfishKnight <knightjellyfish@gmail.com>
 2026-07-16 16:49:52 +08:00
shadowinlife 021bcd502d fix(ocr): address review feedback — selective retry, remove unused confidence(), deduplicate import 
- Only retry transient errors (timeout/connection/5xx) in LLM vision OCR;
  deterministic 4xx failures now raise immediately without delay
- Remove unused confidence() method that hid a redundant OCR call
- Remove duplicate model_validator import in env_schema.py

Signed-off-by: shadowinlife <shadowinlife@gmail.com>

AI-Contributed/Feature: 0/46
AI-Contributed/UT: 0/0
 2026-07-16 16:26:47 +08:00
shadowinlife 606f20d3c2 fix(ocr): address review comments — quality_flag misreport, magic constants, nit 
- Track ocr_attempted_pages to distinguish "OCR attempted but empty"
  from "no OCR needed"; map former to quality_flag="degraded"
- Promote inline magic numbers to named module constants with rationale
  comments (_OCR_MAX_TOKENS, _OCR_TEMPERATURE, _OCR_JPEG_QUALITY)
- Replace no-op round(len(text) / 1, 1) with float(len(text))

Signed-off-by: shadowinlife <shadowinlife@gmail.com>

AI-Contributed/Feature: 0/40
AI-Contributed/UT: 0/0
 2026-07-16 15:49:18 +08:00
xor_xe 8e4fd16646 fix(backtest): guard local benchmark against swapped fallback loader 
fetch_data_map()'s empty-result fallback can silently replace the local
loader with a network source (e.g. yahoo) while config['source'] still
reads local, so resolve_benchmark() fetched the benchmark through a
network loader before the allow_fallback check applied. Verify the
handed-in loader actually belongs to the local source before calling
fetch() on it, and fail closed otherwise.

Signed-off-by: xor_xe <bedauri.it@gmail.com>
 2026-07-16 11:37:26 +04:00
shadowinlife d35e6d8f9b refactor(providers): centralize credentials + fix gateway misroute (#549, #553) 
Two related provider architecture improvements:

1. Keep explicit provider choice when it is a gateway (OpenRouter/Requesty)
   (#549): The existing explicit-provider check already protects gateway
   users from silent misrouting — provider="openrouter" returns
   openrouter capabilities regardless of model name. Added regression
   tests covering all gateway + model-name-prefix combinations.

2. Centralize credential resolution (#553): Extract the duplicated
   provider→env_name→os.getenv→credential chain that was copy-pasted
   across _sync_provider_env() and provider_diagnostics() into a
   single get_llm_credentials() in capabilities.py. The new function
   inherits the existing provider resolution logic, so every consumer
   benefits from consistent behavior.

Additional fixes from code review:
- Remove unused base_env variable in provider_diagnostics() — use
  caps.api_key_env directly, eliminating the redundant
  provider_env_names() call.
- Add 3 base_url resolution tests covering the fallback chain
  (provider-specific → OPENAI_BASE_URL → OPENAI_API_BASE).
- Run black to fix formatting violations on import lines.

Note: _sync_runtime_env() in settings_routes.py is a WRITE operation
(saves Web UI settings to env vars) and follows a fundamentally
different pattern from the READ path above — intentionally left out
of this refactor.

Backward compatibility: The default provider="openai" + model-name
inference (e.g. model="glm-4.6" → zhipu) is preserved. Users who
rely on this convenience behavior are unaffected.

Closes #549, closes #553

Signed-off-by: shadowinlife <shadowinlife@gmail.com>
 2026-07-16 15:30:48 +08:00
ebujinovch 038bf64ccb fix(skills): correlation-regime — cite only the public 8-event regression; mark internal replay numbers as unpublished 
Signed-off-by: ebujinovch <ch.ebujinov@gmail.com>
 2026-07-16 08:52:59 +02:00
Haozhe Wu 8b4bfc0da0 Merge pull request #561 from CRui5in/agent/windows-settings-fchmod 
fix: support Web UI settings persistence on Windows
 2026-07-16 14:42:46 +08:00
Haozhe Wu 2f49127a36 Merge pull request #564 from tianrking/agent/regenerate-runtime-lock 
fix(deps): regenerate runtime lock
 2026-07-16 14:42:38 +08:00
zhanghuiying 58bf8818a4 feat: add SiliconFlow CN and Global providers 
Signed-off-by: zhanghuiying <20434979+UNHNQ@users.noreply.github.com>
 2026-07-16 14:34:34 +08:00
tianrking 3eb598faf5 fix(deps): regenerate runtime lock 
Signed-off-by: tianrking <10758833+tianrking@users.noreply.github.com>
 2026-07-16 12:58:47 +08:00
QinRX baaf07d1ee fix: support Web UI settings persistence on Windows 
Signed-off-by: QinRX <2474250045@qq.com>
 2026-07-16 09:42:29 +08:00
ebujinovch c1dab8331f feat(skills): add correlation-regime skill (regime detection + crisis attribution) 
Signed-off-by: ebujinovch <ch.ebujinov@gmail.com>
 2026-07-15 20:38:36 +02:00
shadowinlife f036a2f29d fix(ocr, 547): backward-compat aliases + ollama-only fallback 
Implement the backward compatibility layer mandated by Issue #547
(design-decision table):

- engine.py: add _CHOICE_ALIASES table ("qwen-vl" -> "llm-vision") applied
  in get_ocr_engine() before engine lookup, emitting a deprecation warning.
- env_schema.py: add Pydantic model_validator(mode="before") on OcrConfig
  that reads legacy VIBE_TRADING_OCR_QWEN_MODEL, forwards to the new
  VIBE_TRADING_OCR_LLM_MODEL field, and emits a deprecation warning.
  New env var takes precedence when both are set.
- llm_vision_ocr.py: restrict the "ollama" placeholder api_key to the
  ollama provider only. Unknown providers now get empty string so they
  no longer silently misconfigure (previously any unknown provider
  without OPENAI_API_KEY was given "ollama" as api_key, corrupting the
  OpenAI client for non-ollama providers).
- test_ocr_engine.py: TestBackwardCompatAliases (6 new tests): alias
  mapping, deprecation warning emission, env-var precedence,
  new-takes-precedence over legacy, ollama-only fallback, and
  unknown-provider empty-key behavior.

Tests: 30 passed, 1 skipped (rapidocr_onnxruntime not installed).
Full suite: 5412 passed, 4 failed (pre-existing tushare E2E auth, unrelated).

Part of #547.

Signed-off-by: shadowinlife <shadowinlife@gmail.com>
 2026-07-16 00:12:14 +08:00
xor_xe 100a832a42 fix(backtest): fail closed on local benchmark instead of yfinance fallback 
An explicit source=local run whose loader returned no benchmark data
(or raised) silently fell back to yfinance, breaking the offline
contract with an unexpected network request. Local now returns no
external benchmark instead; other sources keep the yfinance fallback.

Signed-off-by: xor_xe <bedauri.it@gmail.com>
 2026-07-15 19:10:45 +04:00
xor_xe 80b564436b Merge remote-tracking branch 'upstream/main' into fix/550-local-source-engine-benchmark 
Signed-off-by: xor_xe <bedauri.it@gmail.com>

# Conflicts:
#	CHANGELOG.md
 2026-07-15 19:07:34 +04:00
shadowinlife a45b02486e feat(ocr): pluggable OCR engine architecture with entry_points discovery 
Replace hard-coded QwenVisionOCR with a plugin-based engine that discovers
OCR backends via Python entry_points. Ships with:
- LLMDocumentOCR: multi-modal LLM vision (Qwen/OpenAI/Gemini)
- RapidDocumentOCR: offline RapidOCR adapter
- Extensible base class for custom engines

Includes comprehensive test coverage, env-schema config for engine selection,
and updated doc-reader skill documentation.

Signed-off-by: shadowinlife <shadowinlife@gmail.com>
 2026-07-15 21:09:31 +08:00
Haozhe Wu bd51782647 docs: add July 15 project news 2026-07-15 21:06:04 +08:00
Haozhe Wu 682467cfcf Merge pull request #554 from HKUDS/codex/issue-541-envconfig-reset 
fix(config): refresh cached EnvConfig after dotenv load
 2026-07-15 20:37:15 +08:00
Haozhe Wu 1ece159b61 fix(config): refresh cached env after dotenv load 
Closes #541.
 2026-07-15 20:33:25 +08:00
xor_xe 83e482ef46 fix(backtest): honor explicit local source for engine routing and benchmark 
Sources without a dedicated routing branch (e.g. local) sent US/HK
equity datasets to the crypto engine, and explicit benchmarks were
always fetched via yfinance even when all required price data was
available locally.

Route unbranched sources by detected instrument market, and fetch the
explicit benchmark through the configured source's loader first,
keeping yfinance as the fallback. source: auto keeps the yfinance
default.

Fixes #550

Signed-off-by: xor_xe <bedauri.it@gmail.com>
 2026-07-15 14:12:24 +04:00
Haozhe Wu 5f5d97bf03 Merge pull request #532 from YZY0108/agent/realized-turnover 
fix(backtest): report turnover from executed trades
 2026-07-15 12:09:44 +08:00
Haozhe Wu 2eceb1ed80 Merge pull request #540 from Robin1987China/portfolio-exposure-caps 
feat(optimizers): add per-name and per-group exposure caps
 2026-07-15 12:06:01 +08:00
Haozhe Wu 89f1883016 Merge remote-tracking branch 'origin/main' into codex/pr-532-followup 
# Conflicts:
#	CHANGELOG.md
 2026-07-15 12:04:57 +08:00
Haozhe Wu 8d71bd74a3 Merge pull request #531 from YZY0108/agent/terminal-close-costs 
fix(backtest): account for terminal close costs
 2026-07-15 12:04:29 +08:00
Haozhe Wu c7975ea7f5 Merge pull request #539 from santhreal/fix/backtest-nonfinite-hardening 
fix(backtest): keep backtest outputs finite and validation.json strict
 2026-07-15 12:02:38 +08:00
Haozhe Wu 57c019453f Merge remote-tracking branch 'origin/main' into codex/pr-540-followup 2026-07-15 11:59:46 +08:00
Haozhe Wu c382d614b3 fix(optimizers): preserve uncapped initialization 2026-07-15 11:59:42 +08:00
Haozhe Wu 9fa3a641c0 Merge remote-tracking branch 'origin/main' into codex/pr-531-followup 
# Conflicts:
#	CHANGELOG.md
 2026-07-15 11:57:57 +08:00
Haozhe Wu fbcd335fcf Merge remote-tracking branch 'origin/main' into codex/pr-539-followup 
# Conflicts:
#	agent/tests/test_validation.py
 2026-07-15 11:57:33 +08:00
Haozhe Wu 92728ae3bc Merge pull request #530 from YZY0108/agent/open-rebalance-causality 
fix(backtest): make open rebalances causal and order-independent
 2026-07-15 11:55:45 +08:00
Haozhe Wu dcb2f757b8 Merge pull request #538 from santhreal/fix/validation-config-knob-guards 
fix(backtest): validate validation-config knobs instead of crashing
 2026-07-15 11:55:40 +08:00
Haozhe Wu e6d0617f83 Merge pull request #544 from Marnie0415/fix/marketable-tools-repeatable 
fix(tools): mark core query tools as repeatable to prevent silent dedup
 2026-07-15 11:51:06 +08:00
Haozhe Wu 415422ca1c Merge pull request #535 from xkam7ar/agent/vibe-preserve-run-source 
fix: preserve run data source during chart reconstruction
 2026-07-15 11:51:01 +08:00
Haozhe Wu e6032cc57d Merge pull request #536 from santhreal/fix/metrics-nan-sharpe-single-bar 
fix(backtest): keep Sharpe and information ratio finite for single-bar runs
 2026-07-15 11:50:38 +08:00
Haozhe Wu 624f496925 test(agent): cover every repeatable query tool 2026-07-15 11:21:51 +08:00
Haozhe Wu f9971f617e fix(data): isolate shared fetch configuration 2026-07-15 11:21:00 +08:00
Haozhe Wu d1a6c90434 fix(optimizer): reject boolean exposure caps 2026-07-15 11:20:31 +08:00
Haozhe Wu b09917debc fix(backtest): serialize numpy validation values 2026-07-15 11:19:41 +08:00
Haozhe Wu 63a26adf83 fix(backtest): isolate opening-plan failures 2026-07-15 11:10:56 +08:00
Haozhe Wu c8be539f8b fix(optimizer): fail closed on infeasible exposure caps 2026-07-15 11:08:19 +08:00
Haozhe Wu a5cb2c4827 fix(backtest): plan opening baskets without order bias 2026-07-15 11:07:29 +08:00
Haozhe Wu c95f64fbb4 test(backtest): define realized turnover across engines 2026-07-15 11:04:40 +08:00
Haozhe Wu 2a06987982 test(backtest): cover terminal costs across engines 2026-07-15 11:03:41 +08:00
Haozhe Wu 73c8ba9bca fix(ui): reconstruct charts through loader registry 2026-07-15 11:02:12 +08:00
Haozhe Wu e2a158f167 test(agent): cover repeatable queries across iterations 2026-07-15 10:58:16 +08:00
Haozhe Wu 772b5c090a fix(backtest): reject non-finite initial cash 2026-07-15 10:57:36 +08:00
Haozhe Wu 1b8f843505 fix(backtest): validate validation parameter types 2026-07-15 10:56:57 +08:00
Marnie0415 53d790a397 fix(tools): mark core query tools as repeatable to prevent silent dedup 
Signed-off-by: Marnie0415 <xzh110415@gmail.com>
 2026-07-15 07:08:38 +08:00
Robin1987China 1d4c0740a6 feat(optimizers): add per-name and per-group exposure caps 
Add optional max_per_name, groups, and max_per_group parameters to the
turnover_aware optimizer. When set, these become additional SLSQP
constraints on each rebalance:

  w_i <= max_per_name                  (per-name cap)
  sum_{i in group_g} w_i <= max_per_group[g]  (per-group cap)

The caps are pure linear constraints, so they compose with the existing
turnover penalty without changing the objective. When max_per_name is
None and groups/max_per_group are empty, behavior is unchanged.

Config example:
  optimizer: turnover_aware
  optimizer_params:
    turnover_penalty: 0.1
    max_per_name: 0.25
    groups:
      AAPL: tech
      MSFT: tech
    max_per_group:
      tech: 0.5

Refs #456

Signed-off-by: Robin1987China <41602358+Robin1987China@users.noreply.github.com>
 2026-07-14 22:47:41 +08:00
Santh 5c299c0ce4 fix(backtest): keep backtest outputs finite and validation.json strict 
Two related non-finite-output robustness gaps:

1. initial_cash is not validated. BacktestConfigSchema uses extra="allow"
   and never declares initial_cash, so a config with initial_cash=0 (or
   negative) passes validation and reaches calc_metrics, where
   `equity[-1] / initial_cash - 1` yields inf total_return / annual_return.
   Declare initial_cash: float = Field(default=1_000_000, gt=0) so a
   non-positive value is rejected at the config boundary (the CLI runner and
   the MCP backtest tool both run through this schema).

2. The in-engine validation.json write bypassed strict-JSON serialization.
   run_card and the standalone `python -m backtest.validation` both sanitize
   with _json_safe + allow_nan=False, but BaseEngine.run_backtest wrote
   json.dumps(v_results, ...) with the default allow_nan=True. A non-finite
   validation metric therefore produced bare NaN/Infinity tokens (invalid
   RFC-8259 JSON) only on the automatic in-engine path. Extract the strict
   write into a shared write_validation_json helper and call it from both the
   engine and the standalone main so every writer of validation.json emits
   the same valid JSON (non-finite -> null).

Tests: reject non-positive initial_cash (and default/positive accepted);
write_validation_json serializes inf/NaN as null and parses under a strict
(parse_constant) loader.

Signed-off-by: Santh <64453045+santhreal@users.noreply.github.com>
 2026-07-14 06:24:23 -07:00
Santh 2c72ff8f31 fix(backtest): validate validation-config knobs instead of crashing 
The three validation tools read count/seed knobs straight from user
config["validation"] via run_validation, but none validated them:

  - walk_forward n_windows=0  -> ZeroDivisionError (len // n_windows)
  - walk_forward n_windows<0  -> silently returns empty windows with a
                                 NaN return_mean and negative consistency_rate
  - monte_carlo n_simulations<1 -> ZeroDivisionError (count / n_simulations)
  - monte_carlo/bootstrap seed<0 -> ValueError from np.random.default_rng
  - bootstrap n_bootstrap<1   -> IndexError from np.percentile on empty array
  - bootstrap confidence outside (0,1) -> ValueError from np.percentile

A single misconfigured knob either aborted the whole backtest with an
opaque error or returned NaN-laden garbage with no signal. Guard each knob
at function entry and return the same {"error": ...} shape these functions
already use for insufficient data, so a bad knob surfaces cleanly in the
validation results instead of crashing or silently corrupting the run.

Signed-off-by: Santh <64453045+santhreal@users.noreply.github.com>
 2026-07-14 06:11:30 -07:00
Santh 960f186c04 fix(backtest): keep Sharpe and information ratio finite for single-bar runs 
Series.std() uses ddof=1, so a single-observation return series (a one-bar
backtest, e.g. start_date == end_date) returns NaN. vol (Sharpe) and
active_std (information ratio) consumed it unguarded and produced NaN,
while downside_std (Sortino) was already guarded with the same pattern.
Apply the len > 1 guard to vol and active_std so all three risk metrics
stay finite; a one-bar run now reports 0.0 instead of NaN.

Signed-off-by: Santh <64453045+santhreal@users.noreply.github.com>
 2026-07-14 04:14:57 -07:00
xkam7ar 8f7405a319 fix: preserve run data source 
Signed-off-by: xkam7ar <kmoteallemi@icloud.com>
 2026-07-13 22:55:28 -07:00
Haozhe Wu 7cd7dad3f1 docs: add July 14 project news 2026-07-14 13:46:31 +08:00
Haozhe Wu b4d859d8e9 fix: harden provider settings and NVIDIA NIM support 
Refs #528

Refs #529
 2026-07-14 13:02:03 +08:00
Haozhe Wu 8ff1a12fa0 Merge pull request #534 from HKUDS/codex/rehearsal-priority-integration 
Integrate verified priority fixes and Longbridge data loader
 2026-07-14 12:30:05 +08:00
Haozhe Wu 322fed6631 chore: clean integration lint findings 2026-07-14 11:45:26 +08:00
ShiroKSH 92e07677e2 fix(frontend): authenticate correlation requests 
Signed-off-by: ShiroKSH <kushidashiro@gmail.com>
 2026-07-14 11:38:36 +08:00
Haozhe Wu dde6e13a76 fix(longbridge): enforce complete historical data contracts 2026-07-14 11:38:36 +08:00
fanfpyandClaude Opus 4.6 < 0088fba682 fix(registry): move longbridge to key-gated section of fallback chains 
Longbridge requires API credentials (key-gated), so it belongs in the
key-gated REST section of the fallback chain (after alphavantage), not
after yfinance (free public). Update test_chains_ordered_by_ip_ban_risk
to match the new ordering.

Fixes CI failure in PR #521.

Co-Authored-By: Claude Opus 4.6 <<EMAIL>>
 2026-07-14 11:38:36 +08:00
fanfpyandClaude Opus 4.6 < 350e216bda fix(longbridge loader): remove ctx.close(), add date-window splitting, tz-naive index 
- Remove all ctx.close() calls: LongPort QuoteContext has no close() method
  (verified against installed SDK). Calling it raised AttributeError.
- Add _date_windows() to split wide ranges into 180-day windows, avoiding
  the LongPort ~1000-bar per-call cap that silently truncated long backtests.
- Standardise _normalize_frame output to timezone-naive DatetimeIndex
  (LongPort returns UTC timestamps; downstream code assumes tz-naive).
- Replace is_available() probe from ctx.quote() to ctx.candlesticks() for
  a more reliable liveness check.
- Remove unused _SYMBOL_FORCE_US_SUFFIX empty frozenset.
- Use PEP 585 built-in generics (dict vs Dict) for consistency.

Co-Authored-By: Claude Opus 4.6 <<EMAIL>>
 2026-07-14 11:38:36 +08:00
fanfpyandfanfpy 57c54a2b4e feat(backtest): add Longbridge historical OHLCV loader 
Adds a backtest data loader wrapping the LongPort QuoteContext
(longbridge SDK) for US and HK equity historical bars. Registers
longbridge in VALID_SOURCES and the us_equity / hk_equity fallback
chains alongside yfinance and futu.

Requires LONGBRIDGE_APP_KEY, LONGBRIDGE_APP_SECRET and
LONGBRIDGE_ACCESS_TOKEN (declared in env_schema DataConfig).

Co-authored-by: fanfpy <fanfpy@gmail.com>
 2026-07-14 11:38:36 +08:00
Haozhe Wu 3454e6a2bb test(config): cover Longbridge credentials 2026-07-14 11:38:36 +08:00
fanfpyandfanfpy eb0f71d7dd feat(config): add LONGBRIDGE_APP_KEY/APP_SECRET/ACCESS_TOKEN env vars 
Declares the three LongPort credentials in DataConfig so they flow
through the centralized Pydantic env-schema layer. Also documents them
in agent/.env.example.

These fields back the new longbridge backtest loader
(backtest.loaders.longbridge) which reads credentials via
get_env_config().data.

Co-authored-by: fanfpy <fanfpy@gmail.com>
 2026-07-14 11:38:36 +08:00
Haozhe Wu 63043ea4a1 fix(packaging): keep Longbridge SDK opt-in 2026-07-14 11:38:36 +08:00
fanfpy 5bf48ee854 feat: add longbridge SDK as a runtime dependency 
Longbridge (>=0.2.5) is the official LongPort OpenAPI Python SDK
used by the longbridge trading connector. It was previously undeclared
(assumed to be installed by the connector's own requirements.txt),
which breaks pip install vibe-trading-ai users whose envs pins the lock
file and Docker builds that rely solely on requirements-lock.txt.

Changes:
  - agent/requirements.txt: add longbridge>=0.2.5
  - pyproject.toml: add longbridge>=0.2.5 to dependencies
  - requirements-lock.txt: regenerated (resolves to longbridge==4.3.3)

Connector code lives at agent/src/trading/connectors/longbridge/.
History data loader is out of scope (tracked for PR2).

Relates to: feat/longbridge-support (PR1 of 4)
Co-authored-by: fanfpy <fanfpy@gmail.com>
 2026-07-14 11:38:36 +08:00
asahikiko 5fe512dcdf fix(tools): add tushare fallbacks for flow tools 2026-07-14 11:38:36 +08:00
HUANG Cheng 06d742c4b6 feat(mcp): add Streamable HTTP transport for modern clients 
Add --transport http exposing the MCP Streamable HTTP transport (spec 2025-03-26+) as a single POST/GET /mcp endpoint, alongside the existing stdio and legacy SSE transports. The legacy sse transport is unchanged.

Modern clients (e.g. QwenPaw) negotiate by POSTing an InitializeRequest; the deprecated two-endpoint SSE transport rejects that POST on /sse with 405 Method Not Allowed. Streamable HTTP accepts POST on its single endpoint.

FastMCP 3.4.4 (already a dependency) provides the transport via mcp.run(transport=streamable-http). Adds a --host option scoped to the http transport (default 127.0.0.1).

Signed-off-by: HUANG Cheng <cheng@duck.com>
 2026-07-14 11:38:36 +08:00
Haozhe Wu 93220830dd test(tools): cover recoverable write-file arguments 2026-07-14 11:38:36 +08:00
abhishekjaisinghaniandClaude Opus 4.8 7d7f294164 fix(tools): make write_file robust to aliased/missing path arg 
write_file did `kwargs["path"]`, raising an uncaught KeyError when a model
emits the call with the path under an alias (file_path/filepath/filename/file)
or omits it entirely. In swarm runs this crash-loops the worker with no error
the model can read and self-correct from, so the DAG never completes.

Accept the common aliases for both `path` and `content`, and return a
structured {"status":"error", ...} the model can recover from instead of
raising an uncaught KeyError. Reproduced with deepseek-v4-pro on the
commodity_research_team preset; after this change the 3-agent DAG completes.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
 2026-07-14 11:38:36 +08:00
sTunnaSu ac426cc440 fix: filter unsupported hypothesis update fields 2026-07-14 11:38:36 +08:00
Santh fd59b4a2df Fix crash in backtest metrics on negative final equity 
calc_metrics computed the annualised return as
(1 + total_return) ** (bpy / n). A leveraged or short book can finish at or
below zero equity, making total_return <= -1 and the base non-positive. A
negative base raised to a fractional power evaluates to a Python complex, so
the wrapping float(...) raised TypeError and took down the whole metrics
computation (and the backtest run reporting it).

Guard the non-positive case: a total wipeout annualises to -100%. Normal
runs are unchanged. Adds regression tests for a zero-equity wipeout and a
negative-final-equity book.

Signed-off-by: Santh <64453045+santhreal@users.noreply.github.com>
 2026-07-14 11:38:36 +08:00
YZY0108 6b26f6bb0e fix(backtest): report turnover from executed trades 
Signed-off-by: YZY0108 <u3629247@connect.hku.hk>
 2026-07-14 09:33:51 +08:00
YZY0108 46e2e6cd7f fix(backtest): account for terminal close costs 
Signed-off-by: YZY0108 <u3629247@connect.hku.hk>
 2026-07-14 09:28:59 +08:00
YZY0108 cdd231cb41 fix(backtest): make open rebalances causal 
Signed-off-by: YZY0108 <u3629247@connect.hku.hk>
 2026-07-14 09:24:13 +08:00
zygora cfe579086f feat(data): add historical USD-M funding settlements and maintenance brackets 
Part of #462, follow-up to merged #470 per maintainer guidance: "Still open
for a follow-up PR (1b): historical funding-rate settlement and
maintenance-margin brackets. Same constraints apply — opt-in, fail-closed,
spot untouched."

Extends the perpetual-only fetch path in ccxt_loader.py:

- Fetch historical funding-rate settlements for *-USDT-PERP symbols and
  align them to the required 00:00/08:00/16:00 UTC bars. Fails closed on a
  missing required settlement or a duplicate settlement timestamp.
- Fetch the current per-symbol maintenance-margin bracket schedule
  (notional_cap, maintenance_rate, cumulative_maintenance_amount per
  bracket, per #462's accounting boundary) via fetch_leverage_tiers, stamped
  with a content-hash version since CCXT/Binance expose no bracket history.
  Fails closed on a missing, incomplete, or non-monotonic bracket schedule.

Both are additive DataFrame columns on the existing perpetual frame, so they
survive the opt-in loader cache; spot symbols and the pre-existing
execution/mark-only perpetual path are unchanged.

Signed-off-by: zygora <122336305+honginp@users.noreply.github.com>
 2026-07-13 17:45:41 +08:00
dependabot[bot] f82a21d843 chore(deps): bump the npm-minor-patch group in /frontend with 4 updates (#515) 
Verified locally: build + full frontend suite (246/246) pass. First grouped minor/patch bundle under the new dependabot.yml config — react-i18next 17.0.8->17.0.9, zustand 5.0.12->5.0.14, @types/react patch, autoprefixer 10.4.27->10.5.2.
 2026-07-13 13:07:59 +08:00
Haozhe Wu 7b3dc47107 chore: group Dependabot minor/patch bumps, lower open-PR limit 10->4 
Closing a slot (merge/close) was causing Dependabot to immediately
backfill from its backlog rather than waiting for the next weekly
scan, producing a same-day cascade of individual one-dependency PRs.
Bundling minor/patch updates into one PR per ecosystem (majors stay
excluded and arrive individually, since they need real review) and
capping concurrent PRs lower should keep this from recurring.
 2026-07-13 13:01:10 +08:00
Haozhe Wu 8855e654ed docs: log today's security-hardening + PR batch in CHANGELOG and news 
Covers issue #476 (all 10 audit findings), #377/#478/#480/#487/#479/#484.
Propagated to all 5 locale READMEs' News section (3 visible + moved
2026-07-10 into <details>, per convention).
 2026-07-13 12:55:36 +08:00
Yogesh ModiandYogesh Modi 4f0d5aa04a feat(factors): add Frazzini-Pedersen betting-against-beta (BAB) to ac… (#480) 
Adds Frazzini-Pedersen (2014) "Betting Against Beta" to the academic zoo,
following the existing OHLCV-only proxy pattern: rolling 252-day per-stock
beta against an equal-weighted cross-sectional market return, cross-sectional
z-scored negative (so low-beta names rank highest, matching the long leg of
the original long-short construction). Proxy limitations vs. the original
(equal- vs value-weighted market, single window vs separate 1y-vol/5y-corr,
no explicit leverage) are disclosed in both the module docstring and
academic/LICENSE.md, matching the zoo's existing disclosure convention.

Verified locally: full purity + lookahead suites green (923 passed, 5
pre-existing skips), env-var CI gate passes, ruff clean.

One thing the PR's own test plan missed (it only ran the purity/lookahead
suites): SKILL.md has a manifest guard test
(test_alpha_zoo_counts_match_factor_modules) asserting the documented alpha
count matches the actual factor-module count. Adding a 461st alpha without
updating it fails that test. Folded in a fix bumping the count in SKILL.md
(460->461, academic 10->11) and propagated to all 5 locale READMEs' current-
state references (left the historical changelog entries describing past
counts untouched).

Co-authored-by: Yogesh Modi <YogeshModi24@users.noreply.github.com>
 2026-07-13 12:47:58 +08:00
卫斯李and卫斯李 43f803dee2 feat(metrics): surface realized portfolio turnover in backtest metrics (#478) 
Refs #456. After #466, turnover_aware's realized_turnover was computed but
discarded — the engine's optimizer loader keeps only the returned positions
frame, never the optimizer instance that accumulated it.

Rather than thread the optimizer instance out of the loader lambda, this
computes turnover generically from the executed position frame the engine
already holds (calc_turnover_series: per-bar 0.5*sum|w_t - w_t-1|), so it
works for every strategy/optimizer, not just turnover_aware, and measures the
actual post-normalization weight path that drives transaction cost. Purely
additive: calc_metrics gains an optional positions arg, no existing metric
values change.

Verified locally: full backend suite green (same pre-existing unrelated
failures as main), env-var CI gate passes, ruff clean, 63/63
metrics-and-turnover tests pass. Only one calc_metrics call site
(engines/base.py), inherited by every engine.

Co-authored-by: 卫斯李 <Robin1987China@users.noreply.github.com>
 2026-07-13 12:39:35 +08:00
dependabot[bot] 8b00697a8c chore(deps): bump i18next from 26.3.1 to 26.3.6 in /frontend (#507) 
Verified locally: merged onto current main, npm run build succeeds, full frontend suite 246/246 passes. Patch-level bump (26.3.1->26.3.6).
 2026-07-13 12:27:08 +08:00
dependabot[bot] 41d971efac chore(deps-dev): bump @tailwindcss/typography in /frontend (#508) 
Verified locally: merged onto current main, npm run build succeeds, full frontend suite 246/246 passes. Patch-level bump (0.5.19->0.5.20).
 2026-07-13 12:24:19 +08:00
dependabot[bot] fc5d873d9d chore(deps): bump tailwind-merge from 2.6.1 to 3.6.0 in /frontend (#505) 
Verified locally: merged onto current main, npm run build succeeds, full frontend suite 246/246 passes. v3 is primarily Tailwind v4 support + minor API additions; no usage of removed APIs in this codebase.
 2026-07-13 12:23:51 +08:00
Haozhe Wu bd25a6dcfe chore(deps): bump postcss from 8.5.14 to 8.5.18 in /frontend 
Applied manually — #501's branch was based on a stale main (predates
the other 11 dependabot merges from this batch) and GitHub's own
mergeability check got stuck reporting a conflict even after a clean
local rebase, so this recreates the identical bump directly against
current main instead.
 2026-07-13 12:22:12 +08:00
dependabot[bot] a45f465e9e chore(deps): bump pydantic-core from 2.46.4 to 2.47.0 (#503) 
CI green (test + build). Verified mergeable/clean against current main.
 2026-07-13 12:18:33 +08:00
dependabot[bot] 9ce904a388 chore(deps): bump anyio from 4.14.1 to 4.14.2 (#500) 
CI green (test + build). Verified mergeable/clean against current main.
 2026-07-13 12:18:27 +08:00
dependabot[bot] b15c90640e chore(deps): bump react-router-dom from 7.17.0 to 7.18.1 in /frontend (#499) 
CI green (test + build). Verified mergeable/clean against current main.
 2026-07-13 12:18:23 +08:00
dependabot[bot] eda3e767dd chore(deps-dev): bump @vitest/coverage-v8 in /frontend (#498) 
CI green (test + build). Verified mergeable/clean against current main.
 2026-07-13 12:18:19 +08:00
dependabot[bot] b5defc41fa chore(deps): bump lucide-react from 0.460.0 to 1.24.0 in /frontend (#497) 
CI green (test + build). Verified mergeable/clean against current main.
 2026-07-13 12:18:14 +08:00
dependabot[bot] cef21a58f7 chore(deps-dev): bump jsdom from 27.0.1 to 29.1.1 in /frontend (#496) 
CI green (test + build). Verified mergeable/clean against current main.
 2026-07-13 12:18:10 +08:00
dependabot[bot] 4669bf1810 chore(deps): bump websockets from 15.0.1 to 16.1 (#495) 
CI green (test + build). Verified mergeable/clean against current main.
 2026-07-13 12:18:06 +08:00
dependabot[bot] bddbce9fef chore(deps): bump peewee from 4.1.2 to 4.2.1 (#491) 
CI green (test + build). Verified mergeable/clean against current main.
 2026-07-13 12:18:02 +08:00
dependabot[bot] f1f2822351 chore(deps): bump actions/checkout from 6.0.3 to 7.0.0 (#490) 
CI green (test + build). Verified mergeable/clean against current main.
 2026-07-13 12:17:58 +08:00
dependabot[bot] 5cd3a26419 chore(deps): bump caio from 0.9.25 to 0.10.2 (#489) 
CI green (test + build). Verified mergeable/clean against current main.
 2026-07-13 12:17:54 +08:00
dependabot[bot] 9c5cbc4506 chore(deps): bump cloudflare/wrangler-action from 3.15.0 to 4.0.0 (#488) 
CI green (test + build). Verified mergeable/clean against current main.
 2026-07-13 12:17:50 +08:00
dependabot[bot] c5a4b9bca8 chore(deps): bump react-dom from 19.2.4 to 19.2.7 in /frontend (#493) 
Verified locally: merged onto current main, `npm run build` succeeds, full frontend suite 246/246 passes. npm's lockfile resolver bumped react's locked version to 19.2.7 alongside react-dom to keep the pair in sync (React requires react + react-dom to match exactly), so this single PR resolves the pair cleanly.
 2026-07-13 12:16:22 +08:00
Haozhe Wu 985f0e1b18 chore: ignore pandas major-version bumps in Dependabot 
The <3.0.0 ceiling in pyproject.toml/requirements.txt is a deliberate
pin (pandas 3.x has real breaking API changes we haven't vetted), not
an oversight Dependabot should "fix" by widening it. #494 closed.
 2026-07-13 12:14:44 +08:00
Yuan ZhiyiandYuan Zhiyi 9daccb427f fix(optimizers): exclude decision-bar returns (#487) 
BaseOptimizer.optimize() built each estimation window with ret.loc[:dt], which
includes ret[dt] — a close-to-close return for the decision date. Since
weights at dt execute at that bar's open, ret[dt] isn't actually observable
yet at execution time, so every optimizer built on this base (mean-variance,
equal-volatility, max-diversification, risk-parity, turnover-aware) was
leaking one bar of future information into its own decision.

Fix restricts the window to ret.index < dt. Two regression tests prove it:
the window's latest timestamp always precedes the decision date, and shocking
only the decision-bar return leaves that bar's executed weights unchanged.
Backtests without an optimizer are unaffected; this does change historical
optimized results as intended (removing a look-ahead leak, not introducing
one).

Verified locally: full backend suite green (only 2 pre-existing/unrelated
failures, same as main), env-var CI gate passes, ruff clean, 18/18
optimizer-specific tests pass including both new causality tests.

Co-authored-by: Yuan Zhiyi <YZY0108@users.noreply.github.com>
 2026-07-13 12:06:58 +08:00
Andrew BarnesandAndrew Barnes 7d912f7623 fix(providers): replace stale OpenAI base URL (#484) 
_sync_provider_env() resolves the correct provider-specific base URL (e.g.
switching to OpenRouter) but wrote it to OPENAI_BASE_URL with setdefault, so a
stale value from a previous provider configuration silently won. The OpenAI
SDK reads OPENAI_BASE_URL directly, so requests kept hitting the old endpoint
with the new (still-valid) key attached — surfacing as a 401 "User not found"
from the wrong provider, not an actual key problem. Fixes #482.

Verified locally: full backend suite green (same pre-existing unrelated
failures as main), env-var CI gate passes, ruff clean, 13/13 provider-sync
tests pass including the new regression case.

Co-authored-by: Andrew Barnes <Bortlesboat@users.noreply.github.com>
 2026-07-13 12:06:15 +08:00
AnanayandAnanay b410eba3cf fix(preflight): reset EnvConfig after dotenv load to pick up .env values (#479) 
The EnvConfig singleton can get cached before _ensure_dotenv() loads the
project .env (e.g. theme.py's import-time _is_dark_terminal call runs first),
so `vibe-trading run` preflight would resolve provider/model against a stale
empty snapshot even though `provider doctor` (which calls get_env_config()
fresh) worked fine. Fixes #477 by resetting the EnvConfig singleton right
after dotenv loads, with a regression test + an explanatory comment on the
caching gotcha.

Verified locally: full backend suite green (same pre-existing unrelated
failures as main), env-var CI gate passes, ruff clean.

Note: #483 landed the identical one-line fix independently around the same
time — closing that one in favor of this one since it also carries the
regression test.

Co-authored-by: Ananay <ananaymital@users.noreply.github.com>
 2026-07-13 12:01:34 +08:00
0xZKnwand0xZKnw fb6713e303 feat(trading): opt-in TAP mode — full Alpaca key isolation (reads + orders + cancel) (#377) 
Opt-in TAP (Tool Authorization Protocol) mode for Alpaca: when TAP_PROXY_URL +
TAP_AGENT_KEY are configured, all broker egress (reads, order placement,
cancel) routes through a self-hosted TAP proxy instead of the direct SDK, so
the agent process never holds the raw Alpaca key — only a policy-scoped TAP
agent key. Writes block on human approval; the direct-SDK path is unchanged
when TAP isn't configured.

Reviewed: self-hosted deployment (holonym-foundation/tap-oss) resolves the
custody question; the TAP write path sits strictly downstream of the existing
mandate gate; with TAP env vars unset the direct path is byte-for-byte
unchanged; SSRF guard on the poll URL confirmed. 60/60 tests green
(agent/tests/test_tap_forward.py + test_alpaca_tap_routing.py).

Maintainer fix folded in before merge: tools/ci_env_var_gate.py (#440) landed
after this branch was cut and flagged TAP's raw os.environ reads — annotated
with the established `# noqa: env-gate` escape hatch (module is deliberately
independent of EnvConfig so it keeps working outside the app's normal
startup/config bootstrap).

Co-authored-by: 0xZKnw <0xZKnw@users.noreply.github.com>
 2026-07-13 11:57:57 +08:00
Haozhe Wu 4e3c91c77a security: address all 10 findings from external audit (issue #476) 
Implements VT-001 through VT-010 from the 2026-07-10 external security
audit (Trivy/Semgrep/ZAP/Gitleaks/pip-audit against a real deployment):

- VT-001 (Critical): AST hardening in backtest/runner.py rejects
  network/subprocess/eval/os.environ/unsafe-open calls reachable from
  generated SignalEngine method bodies (previously only checked
  top-level statements). Defense-in-depth: ephemeral per-run HOME,
  RLIMIT_AS/NOFILE caps, best-effort UID-drop to a new vibe-sandbox
  user (currently inert under Docker's cap model - see session doc;
  accepted as documented residual gap, primary AST defense unaffected).
- VT-002: multi-stage Dockerfile split so build-essential never
  reaches the runtime image.
- VT-003: SSE auth moved from a long-lived API key in the URL query
  string to short-lived single-use tickets; access-log redaction for
  any remaining query secrets.
- VT-004: agent/.env split-brain fixed (bind mount + atomic 0600
  writes) between docker-compose and Web UI settings writes.
- VT-005: /correlation now requires auth + per-IP rate limit; generic
  exceptions no longer leak raw text to clients.
- VT-006: CSP-Report-Only + security headers middleware; Google Fonts
  replaced with self-hosted @fontsource packages.
- VT-007: docker-compose hardening (cap_drop ALL + minimal cap_add,
  no-new-privileges, read_only rootfs, resource limits).
- VT-008: /health split into /live (liveness) and /ready (real
  provider-config readiness check); /health kept as a compat alias.
- VT-009: base images pinned by digest, GitHub Actions pinned by
  commit SHA, dependabot.yml added, hash-verified requirements-lock.txt
  now actually wired into the Docker build via --require-hashes.
- VT-010: universe-panel pickle cache sidecar upgraded from an
  unkeyed SHA-256 (forgeable by anyone with cache-dir write access)
  to a keyed HMAC.

Implemented via a 5-parcel Workflow (file-disjoint, independent
adversarial verification per parcel, then a holistic final audit).
The final audit caught a real regression (CI env-var-gate violation)
and confirmed a dependency-pinning gap, both fixed and re-verified
here, including a latent transitive-dependency bug the fix surfaced
(smartmoneyconcepts<0.0.27 pulls an unbuildable zigzag package) - see
docs/sessions/2026-07-13_session_vt476_security_hardening.md (local,
gitignored) for full evidence trail.

Full suite: 5191 passed, 6 pre-existing/environmental failures
(unchanged from main). Docker build + hardened compose boot verified
end-to-end.
 2026-07-13 01:40:53 +08:00
staniell 7296ef9ca7 docs: propagate MT5 connector + data source to all 5 locale READMEs 
zh/ja/ko/ar gain the mt5 data-source table row, the forex/metals
fallback-chain bullet, the source-count bump (19 -> 20), and the full
translated MetaTrader 5 connector section. zh/ja/ko/ar lack the English
MCP Client Mode chapter, so the connector section anchors at the end of
the MCP Plugin section in all four (consistent placement). Also bumps
the QVeris paragraph built-in-source count to 20 in all 5 READMEs.

Signed-off-by: staniell <giostaniell12@gmail.com>
 2026-07-13 00:37:15 +08:00
staniell b159284af4 docs+packaging: mt5 extra, source/connector docs, CHANGELOG 
- pyproject: [mt5] extra with a sys_platform == win32 marker so ubuntu
  CI and non-Windows installs are unaffected.
- README: data-source table + forex fallback chain + MetaTrader 5
  connector section documenting mt5.json, the four profiles, the
  demo/live identity guard, lot-aware mandate sizing, hedging
  close-by-ticket semantics, and the halt path.
- agent/SKILL.md + data-routing SKILL.md: sources 20 -> 21 (test-
  enforced counts), forex routing priority mt5 > akshare.
- CHANGELOG [Unreleased]: connector (11 -> 12) + data source (20 -> 21).

Signed-off-by: staniell <giostaniell12@gmail.com>
 2026-07-13 00:21:39 +08:00
staniell 84c57e8829 feat(data): MetaTrader 5 as a market-data source (mt5 loader) 
Backtests and get_market_data can now use a local MT5 terminal feed
(e.g. Exness) with the broker exact symbols. Sources: 20 -> 21.

- New backtest/loaders/mt5_loader.py: copy_rates_range history with
  tz-aware UTC ranges, Exness-style suffix discovery via symbols_get
  (shortest-then-alpha, memoized), results keyed by the original input
  code, settled-range parquet cache participation, per-symbol failures
  degrade instead of raising so runner fallback engages.
- Routing: mt5 heads FALLBACK_CHAINS[forex]; forex/metal patterns
  (EUR/USD, XAUUSD.FX) added to market_data._SOURCE_PATTERNS -- they
  previously fell through to tushare and failed.
- akshare _is_forex/_fetch_forex now accept the canonical slash form,
  so the mt5 -> akshare degradation actually works for EUR/USD.
- metrics: mt5 annualisation columns (260 trading days, 24x5 bars).

Signed-off-by: staniell <giostaniell12@gmail.com>
 2026-07-13 00:11:07 +08:00
staniell 9a746b9e95 feat(trading): MetaTrader 5 broker connector (Exness-style MT5 brokers) 
New broker_sdk connector talking to a locally running MT5 terminal via
the Windows-only MetaTrader5 package (lazy-imported; ubuntu CI never
needs it). Connectors: 11 -> 12.

- Reads: check_status, account, positions (USD market_value per row,
  None-when-unpriceable so the gate fails closed), pending orders +
  optional deals, suffix-aware quote (zero last omitted), history.
- Writes: place_order (market/pending-limit, volume_step flooring,
  filling-mode negotiation, order_check pre-flight) and risk-reducing
  cancel_order that removes a pending ticket OR closes a position by
  ticket via an opposite deal pinned to position= (hedging-safe).
- Safety: bidirectional identity guard on every session (paper profile
  requires DEMO trade_mode, live requires REAL, contest rejected,
  login pinned); connector-level max_order_volume /
  max_order_notional_usd guards on demo AND live; live orders route
  through the mandate gate with the lot-aware quantity_notional_usd
  sizing hook; per-symbol forex/CFD order classification.
- Profiles: mt5-paper-sdk, mt5-live-sdk-readonly, mt5-paper-trade,
  mt5-live-trade (requires_mandate). Config in ~/.vibe-trading/mt5.json
  (0600 best-effort, redacted in every payload).

Signed-off-by: staniell <giostaniell12@gmail.com>
 2026-07-13 00:06:15 +08:00
staniell 1b295344d2 feat(live): forex/CFD mandate vocabulary + lot-aware gate sizing hook 
Groundwork for the MetaTrader 5 connector:

- InstrumentType gains FOREX and CFD; AssetClass gains FOREX. No schema
  bump: the store parses enums by value, so v1 mandates keep loading.
  CFD follows the OPTION precedent (no universe bucket; admitted only
  via an explicit cfd instrument allowance).
- _INSTRUMENT_ASSET_CLASS maps FOREX to its bucket; forex universe
  floors fail closed (no loader market wired).
- sdk_order_gate._normalize_notional now defers to an optional,
  authoritative quantity_notional_usd connector hook. Lot-sized
  quantities (1 lot EURUSD == 100k EUR) must never be priced as
  quantity x quote — that under-states notional ~contract-size-fold
  and would silently disarm every USD cap. Without the hook the legacy
  path is byte-identical.

Signed-off-by: staniell <giostaniell12@gmail.com>
 2026-07-13 00:05:58 +08:00
Haozhe Wu 2f4ba89733 docs: remove dead Star History badge from all 5 READMEs 
GitHub restricted the stargazers API on 2026-06-30 to repo admins/
collaborators only; api.star-history.com's SVG badge renders server-side
and can't authenticate as a collaborator on every embedding repo, so the
badge now fails with a server timeout instead of rendering (confirmed
live, no fix path per star-history.com's own postmortem).
 2026-07-12 18:41:59 +08:00
Haozhe Wu 87b17daa55 docs: warn about unauthorized Vibe-Trading token 2026-07-12 17:35:11 +08:00
Haozhe Wu 9af8cb8c1e docs: log today's PR batch in CHANGELOG [Unreleased] 
#457 SDM, #474 Requesty provider, #470 Binance USD-M slice 1,
#472 correlation bare tickers, #467 local loader interval, #469 FastMCP compat
 2026-07-12 16:58:04 +08:00
Haozhe Wu c0e3bbd1ec docs: propagate Requesty provider (#474) to all 5 locale READMEs + today's news entry 2026-07-12 16:49:08 +08:00
Haozhe Wu dc5b901a75 Merge PR #474: add Requesty as an OpenAI-compatible provider 2026-07-12 16:47:22 +08:00
Thibault Jaigu 4b21586e10 feat(providers): add Requesty as an OpenAI-compatible provider 
Mirror the existing OpenRouter wiring for Requesty (router.requesty.ai/v1,
REQUESTY_API_KEY/REQUESTY_BASE_URL, provider/model naming, opt-in
extra_body.reasoning). Add it to the capability registry, llm_providers.json,
onboarding + legacy CLI provider lists, .env.example, README, and add a
capability mirror test.

Signed-off-by: Thibault Jaigu <thibault.jaigu@gmail.com>
 2026-07-12 09:36:11 +01:00
Haozhe Wu 39241e0ac3 docs: 2026-07-12 news entry + skill counts 86→87 for strategy-dev-manager (#457) 2026-07-12 15:54:41 +08:00
Haozhe Wu b92eecdb77 Merge PR #457: Strategy Development Manager skill with artifact store and decay monitoring (closes #455) 2026-07-12 15:54:35 +08:00
Haozhe Wu 75bcd0cf3e fix(sdm/ocr): cloud OCR strictly opt-in + duplicate-name guard + all-None metrics guard 
- OCR auto mode is local-only (RapidOCR); qwen-vl requires explicit
  VIBE_TRADING_OCR_ENGINE=qwen-vl — document pages must never leave the
  machine without an explicit user choice
- register_artifact rejects duplicate (name, universe) with a friendly
  ValueError in both sqlite and in-memory stores
- bench histories with no evaluable metric family (all ic_mean/sharpe None)
  report insufficient_data instead of defaulting to HEALTHY
- tests for each (+ new test_ocr_engine.py)
 2026-07-12 15:54:26 +08:00
shadowinlife 68b8434b1a fix: address maintainer review + CI env var gate failures 
1. CI env var gate: replace os.getenv() with centralized config accessor
   in sqlite_store.py, ocr/engine.py, ocr/qwen_vision_ocr.py — add
   OcrConfig + dashscope_api_key + strategy_store_db_path to EnvConfig
2. is_readonly=False on sdm_decay_scan and sdm_status (maintainer #1)
3. read_document: skip low-text pages instead of hard-failing on first
   scanned page; only error when entire doc is image-only (maintainer #2)
4. Wire Sharpe metric into evaluate_decay() for strategy artifacts
   (maintainer #3)
5. UNIQUE(name) → UNIQUE(name, universe) in sqlite schema (maintainer minor)
6. Remove unused imports: asdict, DecaySignal x3 (maintainer minor)
7. Wire consecutive_warnings field in decay snapshots (maintainer minor)
8. Add non-dry-run active→monitoring transition test (maintainer minor)
9. Extract shared compute_decay_metrics() to strategy_store/metrics.py
   eliminating duplication between sdm_decay_scan and sdm_status
10. Update skill count 86→87 in SKILL.md and README.md

All 5055 tests pass. CI env var gate passes. LSP diagnostics clean.

Co-Authored-By: OpenCode <noreply@opencode.ai>
AI-Model: alibaba-cn/glm-5.2
Co-Authored-By: opencode <noreply@ai-tool.com>
Co-Authored-By: Claude Code <noreply@anthropic.com>
AI-Contributed/Feature: 314/314
AI-Contributed/UT: 49/49
 2026-07-12 13:17:06 +08:00
Haozhe Wu 5cd46bc521 Merge PR #470: Binance USD-M symbol routing + execution/mark split (#462 slice 1) 2026-07-12 12:16:18 +08:00
Haozhe Wu 4d81d6e647 Merge PR #469: support legacy FastMCP transport imports 2026-07-12 12:16:18 +08:00
Haozhe Wu ebb98a0a0b Merge PR #467: honor interval in local loader by resampling 2026-07-12 12:16:18 +08:00
Haozhe Wu e55b2c3a6e Merge PR #472: resolve bare tickers in the correlation matrix (closes #471) 2026-07-12 12:16:18 +08:00
shadowinlife 4f78bf9e3a Merge remote-tracking branch 'origin/main' into feat/strategy-dev-manager 2026-07-12 11:17:14 +08:00
CarlsandClaude Opus 4.8 e854606586 fix(correlation): resolve bare tickers so the matrix stops failing with "Fetched: []" 
The /correlation endpoint passed user input straight to the data loaders,
which key US/HK/A-share instruments by exchange-suffixed symbols
(AAPL.US / 0700.HK / 600000.SH). Bare tickers — exactly what the UI hint
suggests (e.g. "BTC-USDT,ETH-USDT,AAPL,SPY") — matched no loader and the
request failed with "Could not fetch price data for at least 2 assets.
Fetched: []".

Three layered fixes:

1. infer_market: classify by explicit suffix first, then bare numeric
   codes by digit length (6 digits = A-share, <=5 = HK). The old prefix
   heuristics misrouted bare HK codes (0700 -> SZ, 9988 -> US), ChiNext
   codes (300750 -> HK), and even suffixed 830799.BJ (-> US).
2. _normalize_symbol (new): canonicalize bare tickers to the loader
   symbol form before fetching, keeping the user's input as the label.
3. Fetch loop: walk the market's full FALLBACK_CHAIN until a loader
   actually returns data instead of stopping at the first *available*
   one, and log every per-source failure instead of the bare
   "except: continue" that made this failure mode undiagnosable.

Tests: +10 cases (bare/suffixed classification, symbol normalization,
chain fall-through); 22 pass.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
 2026-07-11 23:40:47 +08:00
shadowinlife 65fb112c5f feat(ocr): pluggable OCR engine interface with RapidOCR and Qwen-VL backends 
Replace hardcoded RapidOCR dependency with a pluggable OcrEngine Protocol:

- OcrEngine Protocol (engine.py): abstract interface with recognize(image) method
- RapidOcrEngine (rapid_ocr.py): local ONNX-based OCR (existing capability)
- QwenVisionOcrEngine (qwen_vision_ocr.py): cloud vision via DashScope API
  using qwen-vl-plus model with OpenAI-compatible endpoint

Engine selection via VIBE_TRADING_OCR_ENGINE env var:
  auto (default) -> try rapid first, then qwen-vl
  rapid          -> local RapidOCR only
  qwen-vl        -> Qwen-VL cloud vision only
  none           -> disable OCR entirely

doc_reader_tool.py refactored to use the pluggable interface.
Fail-close behavior preserved: scanned pages error out with install hints
when no OCR engine is available.

E2E verified:
- Text PDF (21-page Chinese broker report): 28,130 chars extracted, 0 OCR pages
- Synthetic scanned PDF (3 image-only pages): all 3 pages OCR'd via Qwen-VL
- 60 existing unit tests pass

Co-Authored-By: OpenCode <noreply@opencode.ai>
AI-Model: qwen3.7-max
Signed-off-by: shadowinlife <shadowinlife@gmail.com>
Co-Authored-By: opencode <noreply@ai-tool.com>
Co-Authored-By: Claude Code <noreply@anthropic.com>
AI-Contributed/Feature: 323/323
AI-Contributed/UT: 0/0
 2026-07-11 21:03:35 +08:00
zygora 59aa5ee909 feat(data): separate USD-M execution and mark prices 
Signed-off-by: zygora <122336305+honginp@users.noreply.github.com>
 2026-07-11 13:19:39 +08:00
zygora dc8337d6c3 feat(data): add explicit Binance USD-M symbol contract 
Signed-off-by: zygora <122336305+honginp@users.noreply.github.com>
 2026-07-11 13:19:33 +08:00
Robert Tidball 0ec7f96fd7 fix: support legacy FastMCP transport imports 2026-07-11 15:18:47 +10:00
Lanre Shittu 2b55b41dfd chore(cache): bump loader cache version to 3 for interval + end-date fix 2026-07-10 19:36:47 +01:00
Haozhe Wu fa5e098143 fix(docs): propagate the fundamental zoo family + 0.1.11 counts to frontend, skills, and wiki 
Post-release cross-audit findings: the 5th Alpha Zoo family never reached the
frontend (AlphaZoo card grid missing fundamental, academic still 6, '4 zoos'
i18n strings x5 locales), factors/__init__ docstring, or the asset-allocation
skill ('4 optimizers'). Wiki: 29->30 presets, docs version selector bumped to
0.1.11 (labels only; content is unversioned). README x5: locale parity for the
07-04 UTC news entry, india_broker in the loaders tree comment (20 sources),
ja/ko contributor headings translated.
 2026-07-11 01:31:22 +08:00
Haozhe Wu 3a9e72f2f6 chore(release): sync infra version strings + requirements for 0.1.11 
- Dockerfile OCI version label 0.1.10 -> 0.1.11
- frontend package.json / package-lock.json / 5 i18n locale UI versions -> 0.1.11
- agent/requirements.txt: add bottleneck, defusedxml, prompt_toolkit to match
  pyproject (Docker installs -e . afterwards so images were functionally fine,
  but the cached requirements layer and the file itself had drifted)
 2026-07-11 01:17:12 +08:00
Haozhe Wu ecd43ad7d2 release: v0.1.11 — India equity, fundamental factors, IM channels roll-up 
Version bump + release paperwork: detailed CHANGELOG for 0.1.11 (plus the
missing 0.1.10 backfill), README news + counts across all 5 locales
(30 presets / 460 alphas across 5 families / 19 free sources + india_broker
row and India fallback chain / roadmap), v0.1.11 contributor cycle credits,
SKILL.md version, and the opt-in scheduler flag documented in .env.example.
 2026-07-11 01:11:35 +08:00
Haozhe Wu 85599573cd polish: post-merge review fixups for #461/#464/#466 
- manifest guard test: ignore __pycache__ when counting zoo modules
- analyze_image: state the vision-capable-model requirement in the tool
  description; point the example path at the new uploads media root
- turnover_aware: drop the mathematically inert risk_free knob (additive
  constant on the simplex), fix the max-Sharpe docstring mislabel, document
  realized_turnover as class-API-only
- skills: surface turnover_aware in asset-allocation + strategy-generate
  so the agent can actually propose it
 2026-07-11 00:29:08 +08:00
Haozhe Wu 3e0d95b469 fix(channels): save inbound IM media under the allowed uploads root (#465) 
Inbound media now lands in ~/.vibe-trading/uploads/<channel>/ so the agent's
file-reading tools can open it without VIBE_TRADING_ALLOWED_FILE_ROOTS
configuration. The Matrix E2E store (credentials) moves to the runtime dir
instead, with a legacy-path fallback so existing E2E sessions survive.
 2026-07-11 00:29:08 +08:00
卫斯李 d16716ee1f feat(optimizers): add turnover-aware portfolio optimizer (#466) 
Fifth optimizer: mean-variance utility with an L1 turnover penalty against the previous rebalance (SLSQP, long-only simplex), so the portfolio only trades when expected improvement outweighs churn. Includes 12 seeded tests.
 2026-07-11 00:28:52 +08:00
fei-moss a40f680b00 feat(tools): add analyze_image vision tool for charts and screenshots (#464) 
Read-only tool that sends a local image to the session model as a multimodal message. Path-validated against the allowed file roots; complements read_document's OCR with semantic vision analysis. Requires a vision-capable model.
 2026-07-11 00:28:48 +08:00
asahikiko 3090c418b7 fix(docs): sync packaged skill capability counts (#461) 
Update agent/SKILL.md to the live counts (8 engines / 460 alphas / 86 skills / 30 presets / 20 sources) and add a guard test that keeps the packaged manifest in sync with source.
 2026-07-11 00:28:44 +08:00
烦fpy 430b10f939 fix(longbridge): convert Decimal to float for JSON serialization (#459) 
The Longbridge SDK returns Decimal for prices/quantities/balances; serialize them as floats across quotes, bars, balances, positions, orders, and executions.
 2026-07-11 00:28:40 +08:00
fei-moss 2b27aed383 fix(napcat): send pairing codes for private messages (#463) 
Pass is_dm for private messages so the pairing-code flow fires, matching every other channel adapter.
 2026-07-11 00:28:35 +08:00
Haozhe Wu aaa9945792 docs: add July 10 news — India equity support, centralized env config, UX guard batch 2026-07-10 12:50:25 +08:00
Haozhe Wu ff04624d13 fix(providers): capture GLM reasoning content on the zhipu provider 
GLM thinking models (glm-4.5+/glm-5.x) stream chain-of-thought as
reasoning_content; without capture the capability layer dropped it at the
ChatOpenAI boundary. Enable capture_reasoning (DeepSeek posture: no replay
on assistant turns until verified live against bigmodel).

Fixes #458. Refs discussion #337.
 2026-07-10 12:48:23 +08:00
Haozhe Wu 1c3953ff05 Merge PR #305: first-class Indian equity (NSE/BSE) support 
IndiaEquityEngine (T+1 delivery, opt-in shorts, circuit bands, config-driven STT/stamp/exchange/SEBI/GST cost stack), .NS/.BO routing to Yahoo with yfinance + read-only Shoonya/Dhan broker-data bridge fallbacks, equity_in factor universe (255 alpha101/qlib158 factors opted in), mandate asset-class wiring. Dhan/Shoonya stay paper + read-only per the runtime-discriminator red line. Conflict resolved in loaders/registry.py (keep both qveris + india_broker).

Closes #305.
 2026-07-10 11:28:25 +08:00
wison1717-maker 8e7e16313e feat(frontend): clarify inputs, unify error feedback, guard mandate commit 
- Add aria-labels and a Enter/Shift+Enter hint to the Agent chat input,
  goal editor, and previously label-less send button.
- Add a lightweight ConfirmDialog (no new dependency) and require a
  second confirmation step before committing a real trading mandate,
  with a limits summary shown before the user commits.
- Surface previously swallowed errors in Compare.tsx (api.listRuns /
  api.getRun failures now toast instead of silently showing "no data").
- Remove a duplicate toast+inline error in QVerisSettings' save flow.
- Add a "How it works" 4-step section to the Home page.
- Give the Agent page's kill-switch row a border to visually separate
  it from the lighter decorative status chips above it.
- Route several already-translated mandate strings (Cancel, Send
  adjustment, Committing…, error messages) through i18n instead of
  hardcoded English; add matching keys across all 5 locales.
- Fix a flex min-width bug that caused long Home page step titles to
  overflow instead of wrap.

No new dependencies. No changes to trading/API/auth logic beyond the
mandate confirm step and the two error-surfacing fixes noted above.
 2026-07-10 11:24:54 +08:00
Haozhe Wu dbad71d7d9 Merge PR #440: centralize environment variable management with Pydantic schema + CI gate 
EnvConfig Pydantic schema (60+ vars, 6 sub-models) + thread-safe lazy singleton accessor replacing scattered os.getenv across 50+ files; AST-based CI gate (tools/ci_env_var_gate.py, noqa: env-gate suppression) wired into CI. Conflict with main resolved in agent/src/providers/llm.py: kimi-coding UA override (848d139) now reads moonshot_user_agent via get_env_config().

Closes #438.
 2026-07-10 11:22:41 +08:00
shadowinlife d593377668 fix(doc-reader): fail close when OCR engine is missing instead of silent degradation 
Previously, when rapidocr_onnxruntime was not installed and a scanned/image
page was encountered, the tool silently returned empty text. Users had no
indication that content was lost.

Now:
- PDF pages with <50 chars of extractable text return an error with clear
  install instructions (pip install rapidocr_onnxruntime)
- Standalone image files return the same error when OCR is unavailable
- Text-based PDFs and text files are unaffected (no OCR needed)
- _ocr_available() caches the import check for performance

Co-Authored-By: OpenCode <noreply@opencode.ai>
AI-Model: qwen3.7-max
Signed-off-by: shadowinlife <shadowinlife@gmail.com>
Co-Authored-By: opencode <noreply@ai-tool.com>
Co-Authored-By: Claude Code <noreply@anthropic.com>
AI-Contributed/Feature: 42/42
AI-Contributed/UT: 0/0
 2026-07-10 10:58:43 +08:00
卫斯李 df92a1a0be test(api): cover the scheduled research routes (#452) 
Route-level contracts for POST/GET/DELETE /scheduled-runs: create/validation/422, newest-first list + status filter + limit bounds, delete/404/unsafe-id 400. Store singleton redirected to a temp file so tests never touch the runtime root. Verified locally against main: 9 passed.
 2026-07-10 10:43:28 +08:00
shadowinlife 5b8aaaf124 feat(strategy-dev-manager): add Strategy Development Manager skill with artifact store and decay monitoring 
Add a new 'strategy-dev-manager' skill that orchestrates the full lifecycle
from academic paper to validated factor/strategy: ingest, extract, implement,
evaluate, and monitor with automated decay detection.

New modules:
- strategy_store/: StrategyStoreProtocol + InMemoryStore + SqliteStrategyStore
  (WAL mode, FK constraints, PRAGMA user_version migration)
- strategy_store/decay.py: DecayEvaluator state machine with configurable
  thresholds (IC ratio, IR, Sharpe-based transitions)
- tools/sdm_register_tool.py: Register extracted factors/strategies
- tools/sdm_status_tool.py: Query/update lifecycle (list/detail/disable/enable/decay_check)
- tools/sdm_decay_scan_tool.py: Batch decay monitoring with per-artifact scanning

New skill (agent/src/skills/strategy-dev-manager/):
- SKILL.md: 5-phase orchestration workflow
- examples.md: Paper-to-factor and paper-to-strategy examples
- templates/: Factor and strategy SignalEngine templates, decay report template
- references/: Decay thresholds, scheduled scan config, extraction guide, metrics

Tests: 60 tests (41 store/tool + 19 SQLite-specific), all passing.

Closes #455

Co-Authored-By: OpenCode <noreply@opencode.ai>
AI-Model: qwen3.7-max
Signed-off-by: shadowinlife <shadowinlife@gmail.com>
Co-Authored-By: opencode <noreply@ai-tool.com>
Co-Authored-By: Claude Code <noreply@anthropic.com>
AI-Contributed/Feature: 3404/3404
AI-Contributed/UT: 753/753
 2026-07-10 10:37:29 +08:00
Haozhe Wu f0213c14bc docs: add July 9 news 2026-07-09 13:48:52 +08:00
Haozhe Wu 848d1392cb fix: integrate startup and provider quick wins 
Fix Docker/server startup when mounted router-like routes do not expose path.
Preserve the first CLI turn entered at the resume prompt, align loader fetch signatures, refresh Codex OAuth defaults, add Kimi for Coding and opencode-compatible provider mappings, and clean up Tushare reference fence typos.

Fixes #450.
Integrates #437, #448, #446, #435, #449, #444.
Reported-by: Penn-Live
Co-authored-by: shadowinlife <shadowinlife@gmail.com>
Co-authored-by: morluto <williamlin1327@gmail.com>
Co-authored-by: Carls <yxhuang@uibe.edu.cn>
Co-authored-by: Sankalp Singh <iamsankz@gmail.com>
Co-authored-by: flash1234pku <53250768+flash1234pku@users.noreply.github.com>
 2026-07-09 13:35:45 +08:00
shadowinlife 8c2fd3e127 fix: complete env-var migration — CI gate exit 0, 9 schema fields, noqa mechanism 
- Add 9 missing env vars to EnvConfig schema (QVERIS_API_KEY,
  VIBE_TRADING_API_URL, OPENAI_MODEL, FUTU_TRADE_PWD_MD5, etc.)
- Migrate 26 os.getenv calls to get_env_config() accessor across 17 files
- Add # noqa: env-gate inline suppression to ci_env_var_gate.py (21 lines)
- Refactor security.py _env_flag_enabled to direct schema access
- Wire CI gate into .github/workflows/test.yml
- Fix api_server.py import of deleted constants
- Restore _legacy.py timeout positive-value validation
- Add reset_env_config() to test_security_auth_api.py for singleton freshness

CI gate: exit 0 (was exit 1 with 54 violations)
Tests: 4953 passed, 4 pre-existing tushare failures, 0 regressions

Co-Authored-By: OpenCode <noreply@opencode.ai>
AI-Model: qwen3.7-max
Co-Authored-By: opencode <noreply@ai-tool.com>
Co-Authored-By: Claude Code <noreply@anthropic.com>
AI-Contributed/Feature: 188/188
AI-Contributed/UT: 39/39
 2026-07-08 22:42:14 +08:00
shadowinlife 8234058562 refactor: centralize environment variable management with Pydantic schema + CI gate 
- Add agent/src/config/env_schema.py: Pydantic models for ~50 env vars across 6 sub-models (LLMConfig, DataConfig, APIConfig, SwarmConfig, AgentTuningConfig, PathConfig)
- Add agent/src/config/accessor.py: thread-safe lazy singleton get_env_config() + unified _parse_bool() + reset_env_config() for test isolation
- Fix TIMEOUT_SECONDS CLI display default (2400 -> 120, matching runtime)
- Fix MAX_RETRIES CLI display default ('(not set)' -> 2, matching runtime)
- Fix VIBE_TRADING_API_KEY server-side alias (security.py now resolves it via get_env_config().api.api_auth_key)
- Fix VIBE_TRADING_CHANNELS_AUTO_START=on silently ignored (unified _parse_bool accepts 'on')
- Migrate 207 scattered os.getenv/os.environ.get calls across 50+ files to centralized accessor
- Convert 12 module-level import-time fixations to lazy accessor pattern (loop.py, swarm_tool.py, _backend.py, worker.py, security.py)
- Add reset_env_config() calls after settings_routes.py env mutations (13 mutation points)
- Replace print('[WARN]...') with logger.warning() in tushare/okx/yfinance loaders
- Add tools/ci_env_var_gate.py: AST-based CI gate preventing new os.getenv calls outside src/config/
- Add tools/test_ci_env_var_gate.py: 18 test cases for the CI gate
- Update tools/ci_grep_gates.sh to call the new env var gate
- Add agent/tests/test_env_schema.py: 81 test cases covering defaults, type coercion, singleton, thread safety, bool parser
- Update agent/.env.example with schema cross-references
- Update agent/tests/conftest.py with reset_env_config autouse fixture

Closes HKUDS/Vibe-Trading#438

Co-Authored-By: OpenCode <noreply@opencode.ai>
AI-Model: alibaba-cn/glm-5.2

Signed-off-by: shadowinlife <shadowinlife@gmail.com>
Co-Authored-By: opencode <noreply@ai-tool.com>
Co-Authored-By: Claude Code <noreply@anthropic.com>
AI-Contributed/Feature: 1383/1383
AI-Contributed/UT: 735/735
 2026-07-08 19:36:24 +08:00
Haozhe Wu ef0335286c docs: add July 8 news — fundamental factor layer Phase 1, QVeris optional premium data, maintainer day 2026-07-08 15:06:24 +08:00
Haozhe Wu 9c128a6544 feat(factors): fundamental factor layer Phase 1 — PIT-safe SEC fundamentals as daily panels 
The translator layer: sparse point-in-time filing events become dense
daily panels shaped like price/volume, so existing factor primitives and
backtest engines consume fundamentals with zero changes.

- loaders/_fundamental_schema.py: 12 raw fields, SEC XBRL concept alias
  table (new-standard revenue concepts first), derived-field formulas
  (roe/roa/gross_profitability/asset_growth/accruals/leverage)
- loaders/fundamentals_loader.py: filed-date-anchored PIT ffill (never
  period_end), first-filed-wins restatement policy, duration-span frame
  selection (true quarters only — YTD and full-year frames excluded),
  fiscal Q4 synthesized as FY minus three quarters anchored on the 10-K
  filed date, TTM = rolling four-quarter sum
- factors/registry.py: PanelColumn opens to fund:-prefixed columns
  (price cols unchanged, unknown cols still rejected); decay_horizon
  bound raised to 512 for annual-report-scale factors
- backtest/runner.py: panel injection when a selected factor requires
  fund:* columns; lazy import, zero overhead otherwise, data-source
  failure degrades to NaN frames instead of crashing the run
- tools/get_fundamentals_tool.py: read-only auto-discovered facade
- factors/zoo/fundamental/: fund_roe, fund_earnings_yield,
  fund_gross_profitability, fund_asset_growth (registry now 460 alphas);
  cross-sectional zscore added to factors/base.py (purity-gate home)

Live-verified against SEC EDGAR: AAPL TTM revenue $385.7B and ROE 1.36
match independent hand calculation; the YTD double-count this caught is
pinned by a regression test. Full suite 4817 passed.
 2026-07-08 15:00:41 +08:00
Haozhe Wu 7f5a6ea766 fix(cli): reject extra --swarm-run tokens and explain invalid vars JSON 
--swarm-run was declared nargs="+" but the handler silently dropped
every token after the second, and a non-JSON vars argument died with a
bare "Invalid JSON". Extra tokens now fail fast naming the dropped
values, and JSON errors report the offending string, the parser error,
correct usage, and a shell-quoting hint.

Closes #428
 2026-07-08 15:00:41 +08:00
Haozhe Wu 2c62367ed8 test(frontend): fix flaky QVeris settings save tests on slow CI 
The base-URL input renders its default value before the config GET
resolves, so findByDisplayValue could match while the save button was
still loading-disabled — the click was swallowed and the PUT assertion
timed out on slow runners. Wait for the button to be enabled instead.
 2026-07-08 14:33:24 +08:00
Haozhe Wu b37da118fb Merge pull request #424 from shadowinlife/refactor/api-server-complete-modular 
refactor(api): extract shared infrastructure into focused modules

# Conflicts:
#	agent/api_server.py
 2026-07-08 14:18:29 +08:00
Haozhe Wu 0320f050ce Merge pull request #429 from isaveall/fix/validation-json-missing-artifacts-dir 
fix(backtest): create artifacts dir before writing validation.json (closes #427)
 2026-07-08 14:16:44 +08:00
Haozhe Wu 035673b065 revert: back out IRR-AGL reliability/governance stack (#405, #416) 
Reverts merge commits 0b867a7 (#405) and 474f5e4 (#416), and removes the
now-orphaned #406 artifact metadata store (src/reliability + tests, zero
references left outside its own tests).

Why: the stack wrapped the tool registry in every session path (even
governance mode=off) without exposing the registry surface AgentContext
reads, breaking all session-runtime chats within a day (#433, #431). It
also modified protected core files (src/agent/loop.py, src/session/
service.py), shipped 12 internal-convention docs/ files into the public
tree, and its enforcement layer already has reported bypass paths (#420).
Trading safety stays on the single mandate-gate design (#31, #144).

README news entries for 2026-07-07/07-06 updated accordingly.
 2026-07-08 14:15:56 +08:00
Haozhe Wu be117a24b4 feat: QVeris paid data marketplace integration (pluggable, single-commit revertable) 
Provider: qveris.ai capability marketplace (search/inspect/execute + usage/ledger).
All logic lives in new files; every touch to an existing file is a tagged
QVERIS-INTEGRATION line so 'git revert' of this commit removes the feature cleanly.

- agent tools: qveris_search / qveris_inspect (free) / qveris_execute
  (paid; preview-by-default, allow_paid gated by per-session credit budget)
- REST: GET/PUT /qveris/config + GET /qveris/status (masked key, signup url)
- CLI: vibe-trading qveris status|enable|disable|usage (+ /qveris slash)
- backtest loader: source="qveris" (explicit-only; structurally excluded
  from every auto fallback chain; granularity-aware capability selection;
  top-3 candidate fallback; provider-named series container parsing;
  truncated-result full-content download)
- frontend: Settings QVeris card (toggle/base URL/key/mode/budget/balance/
  recent usage/referral signup CTA), 5 locales aligned (21 keys)
- skill: qveris SKILL.md + references (rest-api, coverage map) + data-routing row
- README x5: free-vs-premium dual-track data section (QVERIS-START/END)

Live-verified end to end against the real API: CLI enable/status, REST config,
search/inspect/preview-refusal/paid execute (charged 1-2 credits, ledger
reconciled), loader fetched 9 real AAPL daily bars. Full backend suite
5025 passed with no regressions; frontend vitest + tsc clean.
 2026-07-08 13:42:31 +08:00
isaveall 7c15396a79 fix(backtest): create artifacts dir before writing validation.json 
When config["validation"] is set, run_backtest() writes
run_dir/artifacts/validation.json in step 7, but run_dir/artifacts is
only created by _write_artifacts() in step 8. If the directory does not
already exist (e.g. a swarm agent workspace), the write raises
FileNotFoundError and aborts the backtest.

Ensure the parent directory exists before the write. Adds a regression
test that runs a backtest with validation enabled against a run_dir
with no pre-created artifacts/ and asserts validation.json is written.

Fixes #427

Signed-off-by: isaveall <isaveall0@gmail.com>
 2026-07-07 21:45:00 +08:00
shadowinlife 8bbadc4ece fix(api): resolve channel runtime singleton staleness across tests 
The host-check logic in _get_channel_runtime/_get_session_service used
_host_attr(name, None) which returns None when the host module sets the
attribute to None (e.g. monkeypatch reset). The elif branch checking the
local singleton was unreachable because api_server always exists and
hasattr is True for re-exported symbols, so stale local instances from
prior tests were returned instead of re-initializing.

Replace _host_attr with direct sys.modules getattr so that an explicit
None on the host falls through to the initialization path, matching the
original api_server.py behavior before extraction.

Co-Authored-By: OpenCode <noreply@opencode.ai>
AI-Model: qwen3.7-max
Co-Authored-By: opencode <noreply@ai-tool.com>
Co-Authored-By: Claude Code <noreply@anthropic.com>
AI-Contributed/Feature: 23/23
AI-Contributed/UT: 0/0
 2026-07-07 17:59:54 +08:00
shadowinlife 39627475af refactor(api): extract shared compat layer, fix session service writeback 
- Create src/api/_compat.py with shared host_attr/set_host_attr
- Remove duplicated _host_attr from security.py, helpers.py, state.py
- Fix _get_session_service missing _set_host_attr writeback to api_server
- Simplify host-check logic in state.py using _compat helpers
- Change _DEFAULT_CORS_ORIGINS from mutable list to immutable tuple
- Add middleware registration comment in api_server.py
- Expand test coverage: 7 → 33 tests (CORS, path validation, SPA routes,
  dotenv round-trip, compat layer, session service writeback)

Co-Authored-By: OpenCode <noreply@opencode.ai>
AI-Model: qwen3.7-max
Co-Authored-By: opencode <noreply@ai-tool.com>
Co-Authored-By: Claude Code <noreply@anthropic.com>
AI-Contributed/Feature: 104/104
AI-Contributed/UT: 199/199
 2026-07-07 17:14:04 +08:00
shadowinlife 976a1c71f3 refactor(api): extract shared infrastructure into focused modules (#331) 
Move auth/CORS/security helpers, Pydantic models, dotenv/SPA helpers,
and lazy-init service singletons out of api_server.py into four focused
modules under src/api/. api_server.py becomes a thin assembler that
creates the FastAPI app, mounts middleware, registers route modules,
and re-exports symbols for test compatibility.

- security.py: auth deps, CORS parsing, DNS-rebinding guard, loopback
  host validation, shell-tools gating (379 lines)
- models.py: Artifact, BacktestMetrics, RAGSelection, RunInfo,
  RunResponse Pydantic schemas (91 lines)
- helpers.py: path constants, dotenv I/O, SPA deep-link middleware,
  path-parameter validation (188 lines)
- state.py: lazy-init session service and channel runtime (124 lines)

api_server.py: 1,103 → 363 lines. Zero behavior change.
All 171 existing API tests pass unchanged.

Closes #331

Co-Authored-By: OpenCode <noreply@opencode.ai>
AI-Model: alibaba-cn/glm-5.2
Co-Authored-By: opencode <noreply@ai-tool.com>
Co-Authored-By: Claude Code <noreply@anthropic.com>
AI-Contributed/Feature: 1718/1718
AI-Contributed/UT: 57/57
 2026-07-07 16:36:27 +08:00
Haozhe Wu a2b68e8e6b docs: add July 7 contributor batch news 2026-07-07 13:24:51 +08:00
Haozhe Wu 0b867a7444 Merge pull request #405 from Elfsa-Miranda/upstream-pr/00-rfc-tool-inventory 
docs: add IRR-AGL reliability RFC and tool inventory
 2026-07-07 13:10:57 +08:00
Haozhe Wu 474f5e4361 Merge pull request #416 from Elfsa-Miranda/upstream-pr/08-hardening-docs 
docs: harden IRR-AGL migration, security, and regression coverage
 2026-07-07 13:09:24 +08:00
Haozhe Wu 74d69e8498 Merge pull request #407 from sambazhu/feat/ai-berkshire-skills 
Feat/ai berkshire skills
 2026-07-07 13:07:35 +08:00
Haozhe Wu 824c4d8e47 Merge pull request #397 from mustafakamal88/fix/aware-utc-timestamps-395 
Use timezone-aware UTC timestamps
 2026-07-07 13:06:16 +08:00
Haozhe Wu 2157738d85 Merge pull request #417 from irfanallana-oss/fix/place-order-zero-size 
fix(trading): treat zero quantity/notional as unset in trading_place_order
 2026-07-07 13:06:12 +08:00
Haozhe Wu 0ec8f2400e Merge pull request #393 from kadaliao/codex/beginner-guide-tutorial 
docs: Add beginner Vibe-Trading tutorial
 2026-07-07 13:05:14 +08:00
Haozhe Wu c00de34676 Merge pull request #396 from kadaliao/codex/alpha-library-social-card 
Add Alpha Library social card
 2026-07-07 13:04:29 +08:00
Haozhe Wu 8324ed070a Merge pull request #413 from SyntaxSawdust/codex/im-channel-reply-timeout 
fix(channels): make IM reply timeout configurable
 2026-07-07 13:04:25 +08:00
mustafa88 c15437c441 Merge main into timezone-aware timestamp fix 2026-07-07 13:03:31 +08:00
sambazhu a6b82efba8 fix(frontend): add value investing locale parity 2026-07-07 12:58:22 +08:00
irfan allana e2f02a6e4f test(trading): cover zero unused order sizing fields 2026-07-07 12:58:22 +08:00
Miranda a974372a1a fix: remove IRR-AGL trailing EOF blanks 2026-07-07 12:53:50 +08:00
irfan allanaandClaude Opus 4.8 9f2be9dc61 fix(trading): treat zero quantity/notional as unset in trading_place_order 
The agent's trading_place_order tool could never place a share-quantity
order. LLMs routinely populate both sizing fields, leaving the unused one
at 0. _num_or_none(0) returned 0.0, so both quantity and notional reached
the connector as non-None and place_order rejected the request with
"provide exactly one of quantity or notional".

A zero order size is never valid, so coerce a 0 quantity/notional to None,
preserving the "exactly one of quantity/notional" contract regardless of
which field the model over-fills.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
 2026-07-06 18:33:41 -05:00
Miranda a9835776ad Merge upstream/main into upstream-pr/08-hardening-docs 2026-07-06 23:07:06 +08:00
Miranda bb2c2b538e fix: keep phase5 factor registry registration compatible 2026-07-06 21:58:21 +08:00
Dustin Persek f4da7e6523 fix(channels): configure IM reply timeout 
Signed-off-by: Dustin Persek <dustin.persek@protonmail.com>
 2026-07-06 08:57:32 -04:00
Miranda c906d2fe93 docs: harden IRR-AGL migration and regression coverage 
Signed-off-by: Miranda <tu07918382691@gmail.com>
 2026-07-06 19:37:06 +08:00
Haozhe Wu c19da874f3 docs: add July 6 README news 2026-07-06 17:31:49 +08:00
sambazhuandClaude 6c09d25c34 feat(skills): English skill bodies + macro constraints per upstream review 
Responding to upstream review feedback:

1. All skill bodies converted to English (matching bundled skills like
   akshare). Some Chinese wording was ambiguous in an international project.
2. Added macro constraints (demographic clock / consumption trend /
   generational shift) per reviewer's point that a consumer-name pick
   should weigh long-term macro headwinds:
   - thesis-tracker: new red line on macro-trend reversal for
     consumer/property/education names.
   - value_investing_committee Li Lu agent: mandatory demographic-clock +
     consumption-trend + generational-shift section for consumer-dependent
     names; a consumer name cannot score high 10-year certainty without
     addressing demographics.

No structural changes; counts unchanged (85 skills, 70 tools, 30 presets).

Co-Authored-By: Claude <noreply@anthropic.com>
 2026-07-06 17:25:39 +08:00
Haozhe Wu 760a490b87 test(search): cover CN fallback without network 
Keep web_search fallback tests deterministic by mocking CN fallback providers and covering the disabled fallback error path.

Clean trailing blank lines from the new reliability artifact files so diff checks stay green.
 2026-07-06 17:10:56 +08:00
Miranda 60bc141f95 feat(reliability): add artifact metadata store 
Add a local artifact metadata foundation with canonical hashing, secret redaction, lineage helpers, and SQLite-backed records.\n\nPreserve trace and run-card compatibility by making new artifact references optional.
 2026-07-06 17:08:24 +08:00
sambazhu b3531409e4 feat(search): add CN fallback search backends 
Add optional Alibaba Cloud IQS, Sogou, and cn.bing fallback paths for environments where ddgs engines are unavailable or slow.\n\nKeep the fallback configurable and preserve the existing web_search JSON envelope.
 2026-07-06 17:08:15 +08:00
shadowinlife fa52c0069b refactor(api): extract remaining routes into modules 
Move runs, scheduled research, swarm, sessions, goals, and live routes from api_server.py into focused src/api route modules.\n\nKeep compatibility re-exports for existing tests and preserve route behavior while reducing api_server.py to the app shell and shared wiring.
 2026-07-06 17:07:47 +08:00
Dustin Persek c3f440ee55 fix(preflight): avoid following provider probe redirects 
Treat provider preflight redirects as successful reachability responses by disabling redirect following on the stripped base URL probe.\n\nAdd regression coverage for redirect responses and request errors.
 2026-07-06 17:07:37 +08:00
Miranda b054afe9de feat: add research cards and reliability panels 
Signed-off-by: Miranda <tu07918382691@gmail.com>
 2026-07-06 17:02:44 +08:00
Miranda b9eaa31656 feat: add deterministic agent eval harness 
Signed-off-by: Miranda <tu07918382691@gmail.com>
 2026-07-06 15:57:55 +08:00
sambazhuandClaude 5d9fd088ea feat(frontend): add Value Investing examples to welcome screen 
Adds a 'Value Investing' category to WelcomeScreen with 4 one-click
examples surfacing the ai-berkshire toolkit:
- Value Investing Committee (run_swarm value_investing_committee, Tencent)
- Supply-Chain Bottleneck Hunter (bottleneck-hunter, AI infra)
- Investment Thesis Tracker (thesis-tracker, Moutai)
- Exact Valuation Check (financial_rigor PE/PB/ROE + three-scenario)

i18n: en + zh-CN (title + desc + prompt). dist is gitignored — CI builds it.

Co-Authored-By: Claude <noreply@anthropic.com>
 2026-07-06 15:35:36 +08:00
Miranda 23004d5303 feat: add quant reliability scorecards 
Signed-off-by: Miranda <tu07918382691@gmail.com>
 2026-07-06 15:18:42 +08:00
sambazhuandClaude bab15351f1 feat(skills): add deep-company-series (P3) 
An 8-part publication-grade deep-dive on a single company (~120k words):
cognitive reset / moat / profit engine / hidden assets / era variable /
financials / management / valuation+redlines. The core IP is the strict
fact-check checklist (5 pseudo-precision traps + 7 revision checks) and
the cross-article consistency scan.

Adapted from ai-berkshire's deep-company-series.md. Cross-article scan
reuses report_audit (extract numbers across 8 articles) + financial_rigor
cross_validate (flag >1% deviations) — closes the loop with P1's tools.

Co-Authored-By: Claude <noreply@anthropic.com>
 2026-07-06 15:02:09 +08:00
sambazhuandClaude fdc7e198d1 feat(skills): add data verification discipline + research bias self-check (P3) 
P3 of the ai-berkshire fusion — small, broad-impact methodology additions:

- data-routing: add a 'Data Verification Discipline' section — cross-check
  material figures across >=2 sources, flag >1% deviations as caliber
  mismatches, use financial_rigor's cross_validate for exact comparison,
  mark unverified numbers. Pairs with the cross_validate tool added in P1.
- research-discipline (new skill, analysis): a 60-second self-bias checklist
  run at the start of any research task — leader-bias, English-bias,
  narrative-bias, confirmation-bias, recency-bias, with concrete corrections.
  Covers the reasoning layer (vs cross_validate=numbers, report_audit=output).

No code changes, no new deps. Existing data-routing tests still pass (4/4).

Co-Authored-By: Claude <noreply@anthropic.com>
 2026-07-06 14:46:41 +08:00
Miranda ab782c9189 feat: add research protocol registry and trial ledger 
Signed-off-by: Miranda <tu07918382691@gmail.com>
 2026-07-06 12:38:11 +08:00
Miranda e3afb93f4b test: add IRR-AGL readiness gate 
Signed-off-by: Miranda <tu07918382691@gmail.com>
 2026-07-06 12:35:15 +08:00
Miranda 6f23860dc9 docs: add IRR-AGL phase 0 inventory baseline 
Signed-off-by: Miranda <tu07918382691@gmail.com>
 2026-07-06 12:35:15 +08:00
Miranda b74a179c53 feat: add governed tool registry and policy decisions 
Signed-off-by: Miranda <tu07918382691@gmail.com>
 2026-07-06 12:34:24 +08:00
Miranda 4551fa2f08 test: stabilize local suite on Windows 
Signed-off-by: Miranda <tu07918382691@gmail.com>
 2026-07-06 12:32:01 +08:00
Miranda f3640e976a fix: preserve audited loader fetch compatibility 
Signed-off-by: Miranda <tu07918382691@gmail.com>
 2026-07-06 12:28:58 +08:00
Miranda 5dd1194be4 feat: add data audit and PIT observe mode 
Signed-off-by: Miranda <tu07918382691@gmail.com>
 2026-07-06 12:28:57 +08:00
Miranda dcb42d2e5d feat: add IRR-AGL artifact store and trace refs 
Signed-off-by: Miranda <tu07918382691@gmail.com>
 2026-07-06 12:18:35 +08:00
sambazhuandClaude 5c438c0d37 fix(swarm): register value_investing_committee in the routing table 
The preset YAML was on disk and loadable (inspect_preset valid=True), but
run_swarm refused preset_name='value_investing_committee' as 'Unknown preset'
because _PRESET_NAMES is derived from _PRESET_KEYWORDS, which the new preset
was not added to. The LLM then fuzzy-matched to the similarly-named built-in
investment_committee and ran the wrong team.

Register value_investing_committee in _PRESET_KEYWORDS (value-investing /
Buffett / Munger / Duan Yongping / Li Lu keywords) and add its
{company}/{market} variables to _build_variables.

Co-Authored-By: Claude <noreply@anthropic.com>
 2026-07-06 12:15:04 +08:00
Miranda a265a0a174 test: add IRR-AGL readiness gate 
Signed-off-by: Miranda <tu07918382691@gmail.com>
 2026-07-06 12:07:13 +08:00
Miranda 1b674eaf1a docs: add IRR-AGL phase 0 inventory baseline 
Signed-off-by: Miranda <tu07918382691@gmail.com>
 2026-07-06 12:07:01 +08:00
sambazhuandClaude 93da172460 fix(tools): normalize three_scenario growth > 1 as percent 
LLMs routinely pass '15%' as 15 instead of 0.15, which made
three_scenario_valuation compound at 1500% and return absurd target prices
(23.5 x 16^3 = 96256 instead of 35.74). The math was correct for the input
given, but the tool silently accepted an unreasonable growth rate.

Now |growth| > 1 is treated as a percent and divided by 100, with a
growth_normalized_from_percent flag and note in the result. Description
also clarified ('0.15 for 15%').

Co-Authored-By: Claude <noreply@anthropic.com>
 2026-07-06 09:11:15 +08:00
sambazhuandClaude 366e82385b fix(tools): make financial_rigor + report_audit repeatable 
loop.py dedups non-repeatable tools by name (a tool already in _called_ok
is refused on a second call). Both tools are read-only and idempotent but
were left at the default repeatable=False, so calling financial_rigor
twice in one session (e.g. verify_valuation then three_scenario) hit a
'cached result, cannot re-invoke' refusal. Set repeatable=True on both.

Co-Authored-By: Claude <noreply@anthropic.com>
 2026-07-06 08:55:26 +08:00
Haozhe Wu 2442fa2e1f docs: add July 5 README news 2026-07-05 23:38:42 +08:00
Haozhe Wu 1cb15d1f3c Merge remote-tracking branch 'origin/pr/401' 2026-07-05 23:26:15 +08:00
Haozhe Wu 5cc9c44baa Merge remote-tracking branch 'origin/pr/390' 2026-07-05 23:25:43 +08:00
Haozhe Wu 5c4e1ae1b3 Merge remote-tracking branch 'origin/pr/382' 2026-07-05 23:25:20 +08:00
Haozhe Wu cfe397c360 Merge remote-tracking branch 'origin/pr/399' 2026-07-05 23:25:12 +08:00
Miranda 7a1c7c9d42 fix(tests): bypass proxies for loopback MCP fixtures 2026-07-05 21:14:41 +08:00
Miranda 76725699b9 fix(tests): skip fork-dependent bench mock test on Windows 2026-07-05 21:04:43 +08:00
Miranda b84518066c fix(tests): make oauth token cache permission checks portable 2026-07-05 21:03:35 +08:00
Miranda 3be0738f1e fix(tests): isolate loader cache root on Windows 2026-07-05 21:02:17 +08:00
sambazhuandClaude ef74a22370 feat(skills): add value-investing toolkit (rigor + audit + 4 skills + committee preset) 
Ports the ai-berkshire value-investing methodology into first-class
Vibe-Trading tools / skills / swarm preset, rewritten for the BaseTool +
SKILL.md + swarm model (no Claude Code slash-commands or TeamCreate).

Tools (auto-discovered BaseTool subclasses, pure stdlib):
- financial_rigor: exact-decimal verification — verify_market_cap,
  verify_valuation, cross_validate, benford, calc, three_scenario. The calc
  evaluator walks an AST in the Decimal domain (0.1 + 0.2 == 0.3 exactly),
  so there is no eval and no IEEE-754 drift.
- report_audit: markdown data-point extraction -> random sample -> PASS/FAIL
  verdict gate. A single-source failure now correctly FAILs the report
  (the source logic silently WARNed instead).

Skills (markdown methodology):
- bottleneck-hunter: supply-chain bottleneck arbitrage — decompose a
  super-trend to Layer 2/3 choke points with mandatory valuation gates.
- thesis-tracker: post-buy discipline — 5-sentence thesis, falsifiable
  assumptions, red lines, 1-10 health score.
- management-deep-dive: management integrity / ability / capital-allocation
  / governance with Duan Yongping's 3-question verdict.
- private-company-research: 6-lens pre-IPO framework (business / financial /
  competitive / risk / tech / alternative-data) with confidence labeling
  and signal-consistency cross-validation.

Swarm preset:
- value_investing_committee: four adversarial master perspectives
  (Buffett / Munger / Duan Yongping / Li Lu) plus a chair synthesizing
  consensus and contradictions into a verdict.

63 new tests; EXPECTED_PRESET_COUNT bumped 29 -> 30.

Inspired by https://github.com/xbtlin/ai-berkshire.

Co-Authored-By: Claude <noreply@anthropic.com>
 2026-07-05 13:39:34 +08:00
shadowinlife b1c6fcec23 fix(ashare-mootdx): replace bare except with except Exception 
The batch_fetch_daily function in a_mootdx_fetcher.py used a bare
'except:' clause that caught KeyboardInterrupt and SystemExit in
addition to ordinary exceptions. This prevented users from interrupting
long-running A-share data pulls with Ctrl+C and silently swallowed
OOM/SIGTERM signals.

Replace with 'except Exception as e:' and log a warning so that:
- KeyboardInterrupt and SystemExit propagate to the caller
- fetch failures are visible in logs instead of being silent

Signed-off-by: shadowinlife <shadowinlife@gmail.com>
 2026-07-05 13:05:06 +08:00
mustafa88 5084e03fd6 Use timezone-aware UTC timestamps 2026-07-04 21:16:11 +01:00
Kada Liao 63f825ca77 fix(wiki): add alpha library social card 
Signed-off-by: Kada Liao <kadaliao@gmail.com>
 2026-07-05 02:09:09 +08:00
Haozhe Wu 5f6e572219 docs: add July 4 README news 2026-07-04 14:06:43 +08:00
shadowinlife 43be0ca4d1 refactor(api): extract settings routes 
Move LLM and data-source settings routes plus their exclusive helpers into src/api/settings_routes.py, preserving existing api_server re-exports for test monkeypatch compatibility.

Local verification: 63 focused settings/security tests passed; compileall, diff check, settings route registration, and monkeypatch target checks passed.
 2026-07-04 13:11:06 +08:00
aeonframework 4d71bd60df fix(deps): raise vulnerable dependency floors 
Raise Pillow and LangChain floors past disclosed advisories, and align the direct langgraph constraint with the newer LangChain 1.3 dependency graph so the package remains installable.

Local verification: Python 3.13 venv pip dry-run for editable install resolved; Pillow/langchain/langgraph/langchain-openai installed and imported; pip check passed.
 2026-07-04 13:10:41 +08:00
Kada Liao 2e5ea4fc17 docs: add Chinese beginner tutorial 
Add a Chinese beginner-facing wiki tutorial and link it from the tutorial index, keeping broker/trading guidance conservative around read-only, paper, mandate, kill switch, and fail-closed boundaries.

Local verification: diff check, HTML/resource/id validation, sensitive-claim scan, and local wiki HTTP fetches passed.
 2026-07-04 13:07:09 +08:00
shadowinlife bae9d21828 refactor(api): extract channels routes 
Move the IM channel HTTP routes and channel runtime lifecycle wrappers from api_server.py into src/api/channels_routes.py while preserving the existing registration path and test monkeypatch compatibility.

Local verification: 85 focused API/channel tests passed; compileall, diff check, and route duplicate checks passed.
 2026-07-04 13:06:21 +08:00
Kada Liao 936480b1f1 docs: add beginner Vibe-Trading tutorial 2026-07-04 11:47:34 +08:00
Haozhe Wu 077ceae759 Merge pull request #392 from HKUDS/codex/20260703-readme-news 
docs: add July 3 README news
 2026-07-03 16:49:21 +08:00
Haozhe Wu c3af4a9dd2 docs: add July 3 README news 2026-07-03 16:42:04 +08:00
Haozhe Wu 180aa2d053 Merge pull request #391 from HKUDS/codex/20260703-pr-issue-audit 
fix: align Robinhood MCP and env startup
 2026-07-03 16:35:31 +08:00
Haozhe Wu 44f8cad928 fix: align Robinhood MCP and env startup 2026-07-03 16:28:24 +08:00
Haozhe Wu f96795c6ee Merge pull request #389 from hobostay/fix/ssrf-cgnat-qq-redirect 
fix(security): block CGNAT/mesh ranges in SSRF guard; close QQ media redirect-to-internal
 2026-07-03 16:21:23 +08:00
Haozhe Wu 6af4c72e4d Merge pull request #378 from shadowinlife/refactor/pr-1-system-routes 
refactor(api) [Patch 2/9]: extract system routes to src/api/system_routes.py
 2026-07-03 16:21:16 +08:00
aeonframework 7b9d519386 fix(deps): raise Pillow and langchain floors past disclosed CVEs 
Pillow >=10.0.0 -> >=12.2.0 clears PYSEC-2026-457/GHSA-3f63-hfp8-52jq (arbitrary code execution), GHSA-44wm-f244-xhp3 (buffer overflow), GHSA-j7hp-h8jx-5ppr (libwebp OOB write), PYSEC-2023-175, PYSEC-2026-165/GHSA-r73j-pqj5-w3x7 (PDF-parse infinite-loop DoS), GHSA-wjx4-4jcj-g98j (font integer overflow).

langchain >=1.0.0,<2 -> >=1.3.9,<2 clears GHSA-gr75-jv2w-4656 (path traversal / sandbox escape, fixed 1.3.9); stays within the existing <2 cap.

Detected by osv-scanner. Manifest-only change (pyproject.toml + agent/requirements.txt).
 2026-07-03 08:13:38 +00:00
hobostayandClaude 3ce2ee3ed1 fix(security): block CGNAT/mesh ranges in SSRF guard; close QQ media redirect-to-internal 
Two SSRF gaps in the channel media-fetch path:

1. channels/utils._is_private only checked is_loopback | is_link_local |
   is_private. ipaddress.is_private is False for RFC 6598 shared address
   space (100.64.0.0/10 — the default Tailscale/mesh range), so those hosts
   passed validate_url_target and every channel (QQ, DingTalk, NapCat,
   Telegram, channelsui) would fetch them as media. Switch to
   `not addr.is_global or addr.is_multicast`, which blocks all
   non-globally-routable ranges uniformly and also closes an IPv4-multicast
   gap (is_global is True for multicast). Matches web_reader_tool._url_allowed.

2. qq._read_media_bytes validated media_ref once, then fetched with
   allow_redirects=True. media_ref is agent-controlled (from msg.media, which
   is routinely influenced by untrusted external content), so a public URL
   returning 302 -> http://169.254.169.254/... would be followed unchecked and
   the internal body uploaded as a media attachment. Set allow_redirects=False
   and fail closed on 3xx, mirroring napcat.py. (The separate inbound download
   at qq.py:618 fetches a trusted QQ CDN URL and is intentionally left as-is.)

Adds agent/tests/test_url_target_security.py covering both fixes. The central
guard (validate_url_target / _is_private) previously had no tests at all.

Co-Authored-By: Claude <noreply@anthropic.com>
 2026-07-03 10:50:26 +08:00
shadowinlife 199c59113b refactor(api): extract settings routes to src/api/settings_routes.py 
Move 4 settings routes (LLM GET/PUT, data-sources GET/PUT) and
7 exclusive helpers from api_server.py into a self-contained
route module following the PR #375 pattern.

- settings_routes.py defines all Pydantic models locally (no shared modules)
- register_settings_routes() resolves auth deps via sys.modules fallback
- Late-access closures for monkeypatch targets (ENV_PATH, ENV_EXAMPLE_PATH)
- Re-exports: ENV_PATH, ENV_EXAMPLE_PATH, _baostock_supported,
  _baostock_installed, _load_llm_providers for test compatibility
- api_server.py: -294 lines (3449 -> 3155)

Co-Authored-By: OpenCode <noreply@opencode.ai>
AI-Model: idealab/Qwen3.7-Max-DogFooding
AI-Contributed/Feature: 0/756
AI-Contributed/UT: 0/0
 2026-07-03 08:35:58 +08:00
shadowinlife 88b5945ee2 refactor(api): extract channels routes to src/api/channels_routes.py 
Move 4 channel routes (status, start, stop, pairing/command) and the
_start/_stop_channel_runtime lifecycle helpers from api_server.py into
a self-contained route module following PR #375 pattern.

- channels_routes.py defines ChannelPairingCommandRequest locally
- Lifecycle helpers access host state via sys.modules for monkeypatch compat
- _get_channel_runtime() stays in api_server.py (uses global on module state)
- Re-exports lifecycle functions for startup/shutdown hooks

Co-Authored-By: OpenCode <noreply@opencode.ai>
AI-Model: idealab/Qwen3.7-Max-DogFooding
Co-Authored-By: opencode <noreply@ai-tool.com>
Co-Authored-By: Claude Code <noreply@anthropic.com>
AI-Contributed/Feature: 72/190
AI-Contributed/UT: 0/0
 2026-07-03 08:24:50 +08:00
shadowinlife b0baa9c211 refactor(api): extract system routes to src/api/system_routes.py 
Move 5 system routes (health, correlation, shutdown, skills, api-info)
and the _terminate_current_process helper from api_server.py into a
self-contained route module following the PR #375 pattern.

- system_routes.py defines HealthResponse locally (no shared modules)
- register_system_routes() resolves _security and
  _require_shutdown_authorization via sys.modules fallback
- Re-exports _terminate_current_process for test monkeypatch compat
- api_server.py: -92 lines (3552 → 3471)

Co-Authored-By: OpenCode <noreply@opencode.ai>
AI-Model: idealab/Qwen3.7-Max-DogFooding
Co-Authored-By: opencode <noreply@ai-tool.com>
Co-Authored-By: Claude Code <noreply@anthropic.com>
AI-Contributed/Feature: 270/270
AI-Contributed/UT: 0/0
 2026-07-03 08:07:24 +08:00
Haozhe Wu 01e211853f docs: add July 2 news 2026-07-02 16:48:01 +08:00
Haozhe Wu 88dfc97877 perf(factors): optimize rolling operators and bench path 
Closes #339
Refs #342

Original performance implementation from #342 by @shadowinlife. Maintainer integration keeps the contribution and tightens the parallel execution and dependency edges before merge.

Validation:
- PYTHONPATH=/tmp/vibe-pr342-bn-only:agent pytest agent/tests/test_factor_operators.py agent/tests/test_bench_parallel.py agent/tests/test_equity_regression.py -q
- VIBE_TRADING_DISABLE_BOTTLENECK=1 PYTHONPATH=/tmp/vibe-pr342-bn-only:agent pytest agent/tests/test_factor_operators.py -q
- python -m compileall -q agent/backtest/engines/base.py agent/scripts/bench_performance.py agent/src/factors/_backend.py agent/src/factors/base.py agent/src/factors/bench_runner.py
- git diff --check
- GitHub Actions CI test passed
 2026-07-02 16:13:47 +08:00
Haozhe Wu b574af8528 refactor(api): extract upload routes 
Refs #331
Refs #358

Original route extraction from #358 by @shadowinlife. Maintainer integration narrows the first API modularization step to the upload and Shadow report routes so no unused duplicate API modules land in main.

Validation:
- PYTHONPATH=agent pytest agent/tests/test_upload_api.py agent/tests/test_upload_security.py agent/tests/test_security_auth_api.py agent/tests/test_settings_api.py agent/tests/test_spa_deep_link.py agent/tests/test_run_card.py -q
- python -m compileall -q agent/api_server.py agent/src/api/uploads_routes.py
- git diff --check
- GitHub Actions CI test passed
 2026-07-02 16:12:28 +08:00
Haozhe Wu 2cd47cd166 fix(core): narrow backtest subprocess environment 
Closes #332

Summary:
- Narrow generated-backtest subprocess environment inheritance to OS/Python basics, proxy/certificate settings, allowed run-root config, and read-only market-data settings.
- Avoid forwarding LLM provider keys, API auth tokens, shell-tool switches, broker trading secrets, and live/advisory toggles into generated strategy code.
- Document the generated backtest trust boundary in README and SECURITY.

Validation:
- PYTHONPATH=agent pytest agent/tests/test_runner_env.py agent/tests/test_run_card.py agent/tests/test_path_safety.py agent/tests/test_file_tool_sandbox_security.py -q
- python -m ruff check agent/src/core/runner.py agent/tests/test_runner_env.py
- python -m compileall -q agent/src/core/runner.py agent/tests/test_runner_env.py
- git diff --check
- GitHub Actions CI passed
 2026-07-02 15:23:41 +08:00
shadowinlife 6ec6bd1314 fix(channels): add IM session reset commands 
Closes #371.

- Add channel-agnostic /new, /reset, and /newsession handling in ChannelRuntime so IM users can start a fresh session without leaving the chat.
- Preserve existing session history while clearing only the channel session mapping.
- Document in-chat slash commands across README locales.
- Normalize pairing slash-command parsing so the documented case-insensitive command behavior matches runtime behavior.

Validation:
- PYTHONPATH=agent pytest agent/tests/test_channels_runtime.py -q
- python -m compileall -q agent/src/channels/runtime.py agent/src/channels/pairing/store.py agent/tests/test_channels_runtime.py
- git diff --check
- GitHub Actions CI: test
 2026-07-02 14:45:09 +08:00
Haozhe Wu 4709f4c33c docs: add July 1 news 2026-07-01 16:21:19 +08:00
Haozhe Wu 87e675c344 fix(frontend): update echarts security patch 2026-07-01 15:49:20 +08:00
Matt Van Horn d8f88d45da fix(wiki): harden alpha-library CSP by externalizing inline scripts 2026-07-01 15:40:27 +08:00
Haozhe Wu e22e06ac89 fix(frontend): update vulnerable frontend dependencies 2026-07-01 15:40:07 +08:00
Haozhe Wu 041e1cf643 fix(frontend): reconcile channel proxy and zh-CN i18n 2026-07-01 15:26:04 +08:00
hoc.s 18d97e8bef fix(i18n): fix zh-CN language switching and LanguageSwitcher display 
1. Remove nonExplicitSupportedLngs — it strips region codes from compound
   language keys (zh-CN → zh) causing isSupportedCode to reject them and
   fall back to English.

2. Add /channels to Vite proxy paths — Settings page calls /channels/status
   via Promise.all with getLLMSettings and getDataSourceSettings; without a
   proxy rule these requests return index.html causing JSON parse errors.

3. Fix LanguageSwitcher always showing 'English' — i18n.languages (plural)
   always includes the fallback 'en', and SUPPORTED_LANGUAGES has 'en' first
   so .find() always matched 'en'. Fixed by checking i18n.language (singular)
   first.

Signed-off-by: hoc.s <a615828a615828@outlook.com>
 2026-07-01 15:23:34 +08:00
opj161 80c811eadd fix(frontend): make channel settings robust against API failures and proxy channel requests 
Signed-off-by: opj161 <opj161@outlook.com>
 2026-07-01 15:23:34 +08:00
Haozhe Wu 1b9a6700cb Merge pull request #349 from digger-yu/feat/cli-setup-dev 
fix(cli): dev pre-checks frontend deps to surface clear error
 2026-07-01 14:59:03 +08:00
Haozhe Wu fda61aa4c5 Merge pull request #351 from shadowinlife/fix/frontend-loopback-port-334 
fix(docker): bind frontend dev port to loopback for consistency with backend
 2026-07-01 14:58:57 +08:00
Haozhe Wu 3b78745b1f Merge pull request #338 from Robin1987China/safe-bind-default 
fix(api): default bind to 127.0.0.1 and warn on non-loopback without API_AUTH_KEY
 2026-07-01 14:58:50 +08:00
Robin1987China 3234a82e45 fix(api): default bind to 127.0.0.1 and warn on non-loopback without API_AUTH_KEY 
Default the --host bind address to 127.0.0.1 instead of 0.0.0.0 to
match the secure-by-default posture already used in docker-compose.
Add a startup warning when binding to a non-loopback address without
API_AUTH_KEY configured — the existing loopback peer-IP and
DNS-rebinding checks still reject remote requests, but the warning
surfaces the misconfiguration at startup.

Also default the Typer `serve` command's --host to 127.0.0.1 so every
serve entry point agrees (the console script and `python -m cli` already
bound loopback; this covers the `python -m cli.main` path too).

Tests: _is_loopback_bind_host truth table (IPv4/IPv6/hostname/edge),
loopback default, and the warning firing only on a non-loopback bind
without a configured key.

Refs #333

Signed-off-by: Robin1987China <41602358+Robin1987China@users.noreply.github.com>
 2026-06-30 19:50:20 +08:00
shadowinlife 2b9647a5be fix(docker): bind frontend dev port to loopback for consistency with backend 
The frontend dev service published port 5899 on all interfaces
(0.0.0.0) while the backend was correctly bound to 127.0.0.1:8899.
On a shared/LAN network, this exposed the Vite dev server (source,
module graph, HMR websocket) to other hosts unnecessarily.

Add the 127.0.0.1 prefix to the frontend port mapping so both
services share the same loopback-only exposure discipline.

Closes #334

Co-Authored-By: OpenCode <noreply@opencode.ai>
AI-Model: qwen3.7-max
Co-Authored-By: opencode <noreply@ai-tool.com>
Co-Authored-By: Claude Code <noreply@anthropic.com>
AI-Contributed/Feature: 2/2
AI-Contributed/UT: 0/0
 2026-06-30 17:05:15 +08:00
digger yu df6b760ea4 fix(cli): dev pre-checks frontend deps to surface clear error 
`vibe-trading dev` previously assumed the user had already run
`vibe-trading setup`. When they hadn't, the Vite spawn failed with
npm's confusing "vite is not a command" message and the user had to
know to install dependencies separately.

Add a quick `frontend/node_modules/.bin/vite` (or vite.cmd) existence
check before spawning. If missing, fail fast with a clear panel that
points the user at `vibe-trading setup`.

Signed-off-by: digger yu <digger-yu@outlook.com>
 2026-06-30 16:30:05 +08:00
Haozhe Wu 2d08b9550c docs: add IM channel README coverage (#343) 2026-06-30 15:09:46 +08:00
Haozhe Wu 101306e916 feat: wire IM channel runtime 
Merge IM channel runtime wiring, controls, dependency gates, tests, and legacy-name cleanup.
 2026-06-30 14:31:26 +08:00
Haozhe Wu 6f6b1794a3 docs: add 2026-06-29 news to all 5 READMEs (live advisory + Trading 212 + fixes) 2026-06-29 17:15:49 +08:00
Haozhe Wu 4ad497dfc4 Merge pull request #321 from mvanhorn/fix/309-trading212-readonly-connector 
feat: read-only Trading 212 broker connector
 2026-06-29 16:55:33 +08:00
Matt Van Horn eb1c23db48 feat: read-only Trading 212 broker connector 
Fixes #309
 2026-06-29 16:44:19 +08:00
Haozhe Wu 2416a82373 Merge branch 'fix/gemini-thought-signature-replay-local' 2026-06-29 16:12:30 +08:00
Haozhe Wu 59a1dc1853 Merge branch 'patch1' 2026-06-29 16:12:23 +08:00
Haozhe Wu ddb44e0be2 Merge branch 'feat/advisory-interface' 2026-06-29 16:12:18 +08:00
mvanhorn 4ac49e8582 fix: preserve Gemini thought_signature across AgentLoop dict-history replay 2026-06-29 16:09:40 +08:00
digger yu b17ad71761 fix(i18n): polish language switcher semantics 2026-06-29 16:02:41 +08:00
shadowinlife 6ed89f6a94 test(live): assert advisory reviews stay observational 2026-06-29 15:59:59 +08:00
Haozhe Wu b4430b3b40 fix(tools): route US statements to SEC EDGAR 2026-06-29 15:51:47 +08:00
Haozhe Wu 399b5b4e23 fix(wiki): harden alpha library hydration 2026-06-29 15:51:47 +08:00
Haozhe Wu 7ec94c6d2a Merge pull request #329 from hannibal-lee/fix/pandas-3-windows-crash 
Fix pandas version constraint
 2026-06-29 15:51:23 +08:00
hannibal-lee d46db51750 Fix pandas version constraint 2026-06-29 13:58:57 +08:00
shadowinlife e5ca46aa6a fix(live): advisory provider registry, public enforcement aliases, exception sanitization 
Address review findings from PR #328:

- Add module-level provider registry (register/clear/get_advisory_providers)
  replacing the hardcoded AdvisoryOrchestrator([]) dead-code path
- Add public aliases in enforcement.py for 3 private helpers used by advisory
- Replace monkeypatch __init__ in test #9 with registry-based injection
- Sanitize exception messages: str(exc) → type(exc).__name__ in audit dicts
- Add exc_info=True to logger.warning for full traceback in logs only
- Add logger.info when advisory enabled but no providers registered
- Rename account_drawdown → utilization_ratio (semantic accuracy)

Co-Authored-By: OpenCode <noreply@opencode.ai>
AI-Model: qwen3.7-max
Co-Authored-By: opencode <noreply@ai-tool.com>
Co-Authored-By: Claude Code <noreply@anthropic.com>
AI-Contributed/Feature: 85/85
AI-Contributed/UT: 57/57
 2026-06-29 12:02:38 +08:00
shadowinlife 866d4e5683 test(live): add advisory interface and gate integration tests (#317) 
9 test cases covering:
- Verdict enum members (4 values matching invinoveritas contract)
- AdvisoryContext frozen dataclass
- AdvisoryResult auto-timestamp
- MockAdvisory default approve, configurable verdict, raise_on_review
- AdvisoryOrchestrator fail-open on provider exception
- Gate advisory disabled by default (no advisory in gate_decision)
- Gate advisory enabled with mock provider (verdict embedded in audit)

All tests pass, ruff clean, no network dependencies.


Signed-off-by: shadowinlife <shadowinlife@gmail.com>
 2026-06-29 11:36:59 +08:00
shadowinlife 20ea6d6f5a feat(live): wire advisory layer into order gate (opt-in, observational) (#317) 
Add advisory review hook to LiveOrderGuardTool._allow() behind an opt-in
environment variable (VIBE_TRADING_ENABLE_ADVISORY). Advisory verdicts are
embedded in gate_decision['advisory'] for audit visibility but never block
or alter order execution.

- _advisory_review() builds AdvisoryContext from intent/positions/balance
- Fail-open: any exception → REVIEW_UNAVAILABLE, order proceeds
- Default off: env var unset = no advisory field in gate_decision
- _allow() accepts positions/balance kwargs threaded from execute()


Signed-off-by: shadowinlife <shadowinlife@gmail.com>
 2026-06-29 11:36:53 +08:00
shadowinlife 2c92714340 feat(live): add MockAdvisory for advisory interface testing (#317) 
Configurable in-memory advisory provider for test suites. Supports verdict
injection, optional delay, forced failure, and call-history recording —
everything needed to exercise the orchestrator and gate integration without
any network dependency.


Signed-off-by: shadowinlife <shadowinlife@gmail.com>
 2026-06-29 11:36:47 +08:00
shadowinlife 1bcb8ee336 feat(live): add PreTradeAdvisoryInterface and advisory data models (#317) 
Add broker-agnostic advisory interface for external risk assessment services
to provide observational pre-trade opinions. Purely additive — never blocks
or alters order execution.

- Verdict enum: APPROVE, APPROVE_WITH_CONCERNS, REJECT, REVIEW_UNAVAILABLE
- AdvisoryContext: frozen dataclass with 8 normalized fields
- AdvisoryResult: frozen dataclass with auto-filled UTC timestamp
- PreTradeAdvisoryInterface: abstract base with provider_id + review()
- AdvisoryOrchestrator: sequential provider execution, fail-open aggregation
- AggregatedVerdict: worst-case verdict with flattened concerns


Signed-off-by: shadowinlife <shadowinlife@gmail.com>
 2026-06-29 11:36:41 +08:00
digger yu ae9aee9282 feat(i18n): add ja/ko/ar locales, RTL layout, LanguageSwitcher, and i18n tests 
Add Japanese, Korean, and Arabic locales to match the README set,
replace the two-option zh/en toggle with a full language dropdown,
and add comprehensive i18n parity tests.

i18n core
- New ja/ko/ar.json (634 keys each, 100% parity with en.json).
- SUPPORTED_LANGUAGES registry + isRtl()/applyDocumentDirection()
  helpers; navigator as secondary detector with Node.js 21+ guard;
  nonExplicitSupportedLngs for regional variant matching.
- New i18next.d.ts for strict t() key type-checking at compile time.

Components
- LanguageSwitcher dropdown in Layout.tsx: position: fixed + JS-measured
  coords (resists overflow/clipping/RTL direction), full ARIA, touchstart
  + Escape dismiss, i18n.languages array for regional code matching.
- WelcomeScreen: fix double-prefix bug — CATEGORIES/CAPABILITY keys now
  carry full i18n paths with `as const`.
- AlphaZoo: 10 dynamic t() calls use `as any` + defaultValue fallback.
- APP_VERSION constant removed; version string sourced from t('app.version').

RTL & theme
- Logical properties throughout (border-e, ps/pe, border-s-2) +
  rtl:flex-row-reverse on main layout.
- --popover CSS variable defined (light + dark); bg-popover was producing
  no color before, making every popover transparent.
- [dir="rtl"] .rtl\:flip-x rule for mirroring directional icons.

Tests (i18n.test.ts — 231 total, all passing)
- Key parity: 5 locales × 2 directions (missing + extra) + sanity check.
- Interpolation variable parity across all locales.
- isRtl: base codes, regional variants (ar-EG/ar-SA), unknown codes.
- SUPPORTED_LANGUAGES registry integrity.

zh-CN.json
- Translation refinements ("A股回测"→"多市场回测", description alignment
  with en.json); key ordering aligned to en.json structure.

Signed-off-by: digger yu <digger-yu@outlook.com>
 2026-06-29 09:28:59 +08:00
Haozhe Wu 5bde656fbf docs: update June 28 news 2026-06-28 22:18:03 +08:00
Haozhe Wu e1feee31af fix(tools): clean file tool sandbox imports 2026-06-28 17:02:16 +08:00
Haozhe Wu 8633bc51dd Merge pull request #299 from skloxo/contrib/fix-file-tools-tilde 2026-06-28 17:01:23 +08:00
Haozhe Wu b5246ca999 fix(frontend): guard runtime status polling 
Fixes #322
 2026-06-28 16:39:51 +08:00
Haozhe Wu cb1aa92f7c Merge pull request #292 from digger-yu/patch1 
feat(cli): add `setup` and `dev` commands for cross-platform frontend…
 2026-06-28 16:36:33 +08:00
Haozhe Wu 1a3cee4012 fix(mcp): sanitize OAuth cache keys 
Fixes #313
 2026-06-28 11:31:11 +08:00
Haozhe Wu d1aab2be76 fix(cli): align legacy OpenAI default model 2026-06-28 03:35:54 +08:00
Haozhe Wu 53135139f3 Merge remote-tracking branch 'refs/remotes/pr/319' 2026-06-28 03:35:09 +08:00
Haozhe Wu efff920eee Merge remote-tracking branch 'refs/remotes/pr/320' 2026-06-28 03:35:03 +08:00
Matt Van Horn 9a9e4da5b2 fix: clearer Robinhood agent.json validation and safe seed config surfacing 
Fixes #312
 2026-06-27 09:44:36 -07:00
Matt Van Horn 454bceed6c fix: resolve default OpenAI model to an available id (gpt-5.5-instant unavailable) 
Fixes #311
 2026-06-27 09:39:00 -07:00
Haozhe Wu 3f7a3c8b4d docs: update June 27 news 2026-06-27 21:40:11 +08:00
shadowinlife 022d042c6a fix(providers): tolerate content-filtered LLM responses 
Skip individual content-filtered LLM responses instead of aborting event-driven analysis or swarm workers, surface high filter-rate warnings in run cards, and recognize Gemini safety finish reasons alongside OpenAI content_filter.

Closes #307.

Thanks @shadowinlife.
 2026-06-27 21:39:13 +08:00
卫斯李 f5900c5f33 refactor(shadow-account): centralize price feature contract 
Centralize Shadow Account price feature names in models.PRICE_FEATURES so extractor and codegen cannot drift, and keep price-context bounds at four decimals to avoid losing small return thresholds.

Includes regression coverage for feature-name sharing and prior_5d_return precision.

Thanks @Robin1987China.
 2026-06-27 21:39:02 +08:00
Haozhe Wu 0cf7d691f8 docs(news): add 2026-06-26 update 2026-06-26 22:52:04 +08:00
Haozhe Wu 93efeb76e0 refactor(loaders): drop unused _ETF_PREFIXES import after #315 2026-06-26 22:52:04 +08:00
shadowinlife 28f9a26a1e fix(loaders): route tushare ETF/index/HK to correct API endpoints (#315) 
Tushare loader now routes ETF/LOF to fund_daily(), index to index_daily(), HK to hk_daily() instead of always calling daily() (empty for non-stocks). Adds per-symbol empty warnings + runner partial-fetch warning. Closes #310. Thanks @shadowinlife.
 2026-06-26 22:42:07 +08:00
卫斯李 632a6778f2 feat(shadow-account): carry price-condition bounds into extracted rules (#314) 
Promoted entry_rsi14 / prior_5d_return bounds now flow into extracted ShadowRules and the generated SignalEngine for real conditional entry. Follows #302. Thanks @Robin1987China.
 2026-06-26 22:41:57 +08:00
skloxo ab50a0afca refactor(tools): isolate read/write allowed roots, deduplicate path resolution, and add comprehensive security tests 2026-06-26 17:33:41 +08:00
skloxo 049a5afa82 Merge remote-tracking branch 'upstream/main' into contrib/fix-file-tools-tilde 2026-06-25 21:31:47 +08:00
Haozhe Wu 1792715145 docs(news): add 2026-06-24 and 2026-06-25 updates 2026-06-25 19:57:29 +08:00
skloxo 955a1d02d2 feat(i18n): expand reactive zh-CN localization 
Moves more main-panel UI copy through reactive English/zh-CN translations.
 2026-06-25 19:53:42 +08:00
MarkfuGod df731256f4 fix(agent): sync data-source count and gate microcompact 
Closes #282. Derives the data-source count dynamically and avoids microcompaction below the token threshold.
 2026-06-25 19:53:39 +08:00
gyx09212214-prog ec7e65015f fix(validation): write strict JSON outputs 
Normalizes non-finite validation payload values before writing validation.json or stdout.
 2026-06-25 19:53:36 +08:00
卫斯李 1cadda4199 feat(shadow-account): add as-of price-context features 
Adds PIT-safe entry RSI and prior return features for Shadow Account strategy extraction.
 2026-06-25 19:53:32 +08:00
Hinotobi a351a258ae fix(api): allow authenticated remote same-origin UI requests 
Follow-up to #293. Allows authenticated remote same-origin browser requests while preserving cross-site rejection.
 2026-06-25 19:53:29 +08:00
muku314115 987630e7f8 adding support for india 2026-06-24 13:22:54 +05:30
skloxo 3241aa736c chore: update uv.lock to fix zigzag build Cython dependency issue 2026-06-24 10:20:15 +08:00
skloxo 91764e74eb fix: safe_path tilde expansion at source + widen file roots to / 
Root cause: safe_path() did (workdir / p).resolve() which treats ~ as a
literal directory name. This was masked before VIBE_TRADING_ALLOWED_FILE_ROOTS
was added because tools only accessed run_dir + skills/ (relative paths).

Fix: expand tildes in safe_path() itself so ALL callers benefit. Revert
per-tool workarounds (read/write/edit) in favor of the centralized fix.

Also:
- Set VIBE_TRADING_ALLOWED_FILE_ROOTS=/ for full filesystem access
- Set VIBE_TRADING_ALLOWED_RUN_ROOTS=/ (same)
- Clean revert of 629c2cf per-tool patches, keeping ALLOWED_FILE_ROOTS support
 2026-06-24 10:19:34 +08:00
skloxo 5561a548d7 fix: tilde expansion in file tools — safe_path doesn't expand ~, causing paths like ~/.vibe-trading/ to resolve to root/~/.vibe-trading/ (nonsense). Now all three tools (read/write/edit) expand tildes before path validation. 
Also set VIBE_TRADING_ALLOWED_FILE_ROOTS=/home/skloxo in vibe-trading-api.service
so the API agent has full filesystem access like the user requested.
 2026-06-24 10:19:32 +08:00
skloxo 360a0696c6 fix: expand read_file/write_file/edit_file to access configured extra file roots 
The file tools were sandboxed to run_dir only. Now they also check
VIBE_TRADING_ALLOWED_FILE_ROOTS (from path_utils._allowed_file_roots)
so the agent can read/write ~/.vibe-trading/scripts/, db/, etc.
 2026-06-24 10:19:29 +08:00
digger yu 88bfc6d419 feat(cli): add cross-platform setup and dev commands 
Add ``vibe-trading setup`` and ``vibe-trading dev`` to fix the
cross-platform frontend build / dev workflow.

The Windows npx tsc footgun (npm pulls the abandoned ``tsc@2.0.4``
package from the registry) is solved by always invoking tsc via
``npm exec --package=typescript tsc -b`` on Windows. POSIX uses the
standard ``npm run build`` since npm prepends ``./node_modules/.bin``
to PATH for local scripts.

``dev`` launches the backend from ``AGENT_DIR`` (so ``python -m
cli._legacy serve`` resolves the in-repo ``cli`` package) and the
Vite dev server with the port from ``vite.config.ts`` (5899, not the
5173 we used to hardcode). Both children receive SIGINT/SIGTERM and
are force-killed after a 5 s grace period.

On Windows, ``subprocess.Popen`` does not consult ``PATHEXT``, so the
npm path is taken from ``shutil.which`` (which returns the ``.cmd``
form) and used directly — without this, ``Popen(["npm", ...])``
raises ``FileNotFoundError`` even when ``which`` succeeds.

argparse subparsers are registered in ``_legacy.py``; typer
commands ``setup`` / ``dev`` are registered in ``main.py``'s
``_build_typer_app()``. ``init`` behavior is unchanged.

Tests: ``agent/tests/test_cli_setup_dev.py`` covers platform-aware
build commands, the node/npm-missing path, the dev 5899-vs-5173
regression, and the AGENT_DIR-cwd regression. The existing
``cmd_live_authorize`` flow (OAuth handshake) is untouched.

Signed-off-by: digger yu <digger-yu@outlook.com>
 2026-06-24 08:59:59 +08:00
Haozhe Wu 45df314d2f docs(news): add 2026-06-23 entry across all 5 READMEs (#298) 
Local API CSRF hardening (#293). Keeps 3 visible news entries; 2026-06-20 moves under the fold.
 2026-06-23 20:28:58 +08:00
Haozhe Wu 5155016c4b docs(readme): sync Roadmap shipped status (Research Autopilot / Data Bridge / Research Delivery) (#297) 
Mark Research Autopilot (Phase 1-3, #260/#267), the local Data Bridge loader (#252), and the scheduled-research scheduler/executor (#272/#278/#288) as shipped in the Roadmap status column across all 5 READMEs. Feature descriptions left intact so SQL connectors and multi-channel outbound delivery still read as pending. Closes #294.
 2026-06-23 19:13:14 +08:00
Hinotobi dafdb036fa fix(api): reject cross-site unsafe browser requests before loopback auth (#293) 
Run the existing cross-site guard for unsafe methods (non GET/HEAD/OPTIONS) before honoring loopback dev-mode trust, so a malicious page cannot drive local POST/PUT/DELETE routes via blind cross-origin requests. Adds upload-API regressions for cross-site rejection and the local non-browser path.
 2026-06-23 18:49:41 +08:00
Haozhe Wu 57f73fe583 docs(news): add 2026-06-22 entry across all 5 READMEs (#291) 
Live-authorize OAuth fix (#281, closes #259), Alpha Zoo headline
double-count fix (#287, closes #286), and scheduled-research usage
docs (#288). Rolls the 0.1.10 (06-19) entry into the Earlier-news
fold to keep three entries visible.
 2026-06-22 23:24:33 +08:00
Haozhe Wu 9b127f453b docs(readme): sync current counts across all 5 READMEs (#289) 
Update the live capability counts to match the code, verified against
the registries:
- Alpha Zoo 452 -> 456 (registry.list() == 456); academic zoo 6 -> 10
  (Jegadeesh / George-Hwang / Amihud / Harvey-Siddique added in #277)
- Finance skills 77 -> 79
- MCP tools 36 -> 54 (mcp_server has 54 @mcp.tool fns); the exposed-tool
  list is completed with the 18 read-only data tools that were missing
- agent tools tree comment 31 -> 68 (build_registry().tool_names == 68)

Only current-state claims are touched. Dated News entries (the 0.1.8 and
v0.1.9 release notes), the "Shipped 0.1.8" roadmap row, and contributor
credits are point-in-time records and are deliberately left unchanged.

Supersedes #285.
 2026-06-22 22:54:10 +08:00
卫斯李 1c897e3642 fix(cli,mcp): keep live authorize alive through the OAuth flow (#281) 
The connector authorize OAuth handshake timed out before the token was
persisted (#259). Two root causes:

1. The OAuth flow is driven lazily by the list_tools discovery request,
   which is bounded by the per-call tool_timeout (default 30s), not
   init_timeout. The prior fix widened only init_timeout, so a Robinhood
   face-scan sign-in (>30s) still tripped the handshake.
2. The 30s timeout was retried, and the retry opened a fresh client
   context that started a SECOND OAuth callback server on a new port,
   orphaning the sign-in the user had just completed.

Fix:
- cmd_live_authorize now raises BOTH tool_timeout and init_timeout to a
  configurable authorize deadline (VIBE_LIVE_AUTHORIZE_TIMEOUT_SECONDS,
  default 300s), raise-only so a larger user value is preserved.
- The authorize handshake runs a single non-retried list_tools attempt
  via build_mcp_tool_wrappers(..., max_list_tools_attempts=1), so a
  transient failure cannot spawn a duplicate OAuth flow.
- _run_with_retry takes a configurable attempts cap; runtime/registry
  callers keep the default 2 (one transient retry) unchanged.

Closes #259

Signed-off-by: Robin1987China <41602358+Robin1987China@users.noreply.github.com>
 2026-06-22 22:33:42 +08:00
Haozhe Wu 02c65b818e docs(readme): document Scheduled Research usage across all 5 READMEs (#288) 
The scheduled-research feature (#254/#278) shipped without user-facing
docs, so users couldn't tell how to drive it. Add a "Scheduled research"
subsection to the API Server section plus the three /scheduled-runs
endpoint rows in the endpoint table, in en/zh/ja/ko/ar: the
VIBE_TRADING_ENABLE_SCHEDULER flag, the interval-ms / 5-field-cron
schedule format, and create/list/cancel curl examples.
 2026-06-22 22:22:03 +08:00
Haozhe Wu 86b98214ec fix(frontend): stop Alpha Zoo headline rendering the count twice (#286) (#287) 
The AlphaZoo headline rendered a standalone `{total}` and then the
translated `prebuiltAlpha` string, which already interpolates
`{{count}}` — so the page showed the number twice (e.g. "456 456
pre-built quant alphas across 4 zoos"). It also carried a hardcoded,
now-stale `452` fallback.

Render only the translated string with the live (filter-aware) total,
and show a count-less loading variant while alphas load. New
`alphaZoo.prebuiltAlphaLoading` key added to both en and zh-CN.
 2026-06-22 22:12:55 +08:00
Haozhe Wu dc8a5eb060 docs(news): add 2026-06-21 entry across all 5 READMEs 2026-06-21 19:24:27 +08:00
shadowinlife 4a64f0e2b9 feat(prompt): add layered post-backtest attribution guidance (#280) 
Adds a 4-layer post-backtest attribution framework to the agent system prompt (trade attribution, beta regression, regime analysis, Monte Carlo permutation test) with strategy routing (At-risk / Sub-optimal / Healthy) and self-check rules. Prompt-only change to the Backtest task routing — no engine or tool code. Attribution is gated by data availability and explicitly marked secondary to strategy correctness.

Verified against live artifact schemas (trades.csv pnl!=0 exit rows with holding_days/return_pct, validation.json monte_carlo section, config.validation wiring) and prompt .format() brace integrity; 11 new tests pass.

Closes #279
 2026-06-21 19:10:20 +08:00
LemonCANDY42 8f7dc0aa3d feat(reports): add backtest report library page (#224) 
Adds a dedicated /reports Run Library page (product shape agreed in #216): lists report-worthy runs with run id, status, created time, truncated prompt, symbols, date range, return, Sharpe, and links to Run Detail + Compare. Search over run id / prompt / symbol / status; v1 status + created-date filters; newest-first sort with oldest/return/Sharpe options. Sidebar + page copy localized in en + zh-CN. listRuns gains an optional limit aligned with the existing GET /runs?limit contract.

Verified open-box: npm run build clean, full frontend suite 209/209 passing.

Closes #216
 2026-06-21 19:07:08 +08:00
Matt Van Horn 974ad59a68 feat(scheduled-research): add executor that fires due jobs on schedule (#278) 
Adds the background executor that polls the scheduled-research store and dispatches due jobs (interval-ms + cron) through the existing session runtime. Gated behind VIBE_TRADING_ENABLE_SCHEDULER (default off), so existing deployments are unchanged. Resets stale RUNNING jobs on startup, re-reads each record before completion writes to survive delete/replace races, rejects impossible cron schedules without blocking the loop, and dispatches with a config copy so saved jobs are never mutated.

Completes the scheduled-research feature begun in #272 (data model + store + CRUD).

Closes #254
 2026-06-21 19:06:46 +08:00
Haozhe Wu eeb11bf303 docs(news): add 2026-06-20 entry across all 5 READMEs 
Autopilot Phase 3 (loop closure) + loader OHLC integrity guard + 4
academic alphas. Keeps the 3-latest-visible convention (2026-06-17 moved
into the Earlier-news details).
 2026-06-20 18:30:05 +08:00
Matt Van HornandMatt Van Horn f3bdc3b9ad feat: add scheduled research job data model and CRUD API (#272) 
* feat: add scheduled research job data model and CRUD API

Implements the first slice of issue #254 (scheduled execution of agent
prompts and backtests): a durable ScheduledResearchJob data model and
ScheduledResearchJobStore with crash-safe atomic persistence (write-temp
+ fsync + replace + fsync-dir), mirroring src/live/runtime/jobstore.py.
Wires three new API endpoints to api_server.py (POST /scheduled-runs,
GET /scheduled-runs, DELETE /scheduled-runs/{job_id}). Execution wiring
is intentionally deferred to a follow-up PR once the schedule types and
output channels are confirmed. 24 unit tests included.

Refs #254

* fix(scheduled-research): root job store under the user runtime dir

_default_store_path() wrote scheduled_research_jobs.json into the repo
working tree (agent/data/), which is not gitignored — running the API
would leave job state in the checkout. Root it under get_runtime_root()
(~/.vibe-trading/scheduled_research/) instead, the same root the live
runtime, swarm config, and persistent memory resolve. Also range-check
cron fields (minute 0-59 / hour 0-23 / dom 1-31 / month 1-12 / dow 0-6)
so an out-of-range expression like '99 * * * *' is rejected.

---------

Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
 2026-06-20 18:00:39 +08:00
Lanre Shittu 9addc95a4b feat(loaders): validate OHLC invariants at the loader boundary (#274) 
* feat(loaders): validate OHLC invariants at the loader boundary

Loaders only dropped NaN rows, so structurally invalid bars (high<low, a
non-positive price, or a high/low that fails to bracket open/close) passed
straight into the backtest and surfaced downstream as NaN/inf metrics that
break the strict allow_nan=False JSON serializers.

Add a shared validate_ohlc() helper in loaders/base.py and call it after the
existing dropna in the yfinance and local loaders. The default drops offending
rows and logs a count, matching the loaders' existing silent-clean convention;
"warn" and "raise" strategies are available for stricter callers.

Signed-off-by: Lanre Shittu <136805224+Shizoqua@users.noreply.github.com>

* feat(loaders): apply OHLC validation centrally for every source

Wire validate_ohlc into the runner's single fetch-convergence point so
auto, single-source, runtime-fallback, and any future loader are guarded
uniformly — not just the two loaders wired at source. Adds
_sanitize_data_map(data_map) in runner.py, applied right after the data
map is assembled and before the empty-data check, plus a test that a dirty
bar from an unwired loader is dropped before it reaches the backtest.

---------

Signed-off-by: Lanre Shittu <136805224+Shizoqua@users.noreply.github.com>
 2026-06-20 18:00:35 +08:00
卫斯李andRobin1987China f87f94db88 test(market-data): cover the shared loader-resolution layer (#276) 
src/market_data.py is the source-detection + normalization layer shared
by the MCP server and the agent get_market_data tool (added with the #270
global data layer), but shipped without dedicated tests. Adds 24
network-free tests using an injected stub loader:

- detect_source: A-share/US/HK/crypto symbol routing + tushare fallback.
- cap_rows: under-limit passthrough, max_rows=0 (no cap), negative ->
  default, and stride sampling with the final bar always pinned.
- _json_safe: non-finite floats -> None, Timestamp isoformat, numpy
  scalar unwrap, plain passthrough.
- fetch_market_data: explicit-source normalization, auto-grouping by
  detected source, loader-exception fallthrough to _unresolved, and
  partially-resolved symbols listed in _unresolved.
- fetch_market_data_json: strict-JSON output and NaN sanitization under
  allow_nan=False.

Test-only; no production changes.

Signed-off-by: Robin1987China <fanrenzhige@163.com>
Co-authored-by: Robin1987China <fanrenzhige@163.com>
 2026-06-20 17:55:36 +08:00
卫斯李andRobin1987China 2e7761a554 feat(autopilot): scaffold signal engine + auto-link backtest metrics (Phase 3) (#267) 
Two new tools close the hypothesis -> backtest -> evidence loop that
Phase 1+2 left half-wired:

- scaffold_signal_engine: writes a contract-correct code/signal_engine.py
  stub (no-arg SignalEngine, generate(self, data_map) -> dict of pd.Series)
  with a runnable flat-signal default and the hypothesis signal_definition
  embedded as a docstring. Eliminates the runner's most common pre-flight
  validation failure. Refuses to overwrite an existing engine unless
  overwrite=true.
- link_autopilot_backtest: reads run_card.json metrics from a run dir and
  links them to the hypothesis via link_backtest in one step, instead of
  making the agent hand-extract the metrics dict.

Both resolve run_dir through safe_run_dir, inheriting the allowed-run-root
boundary (fail-closed on path traversal). Adds 12 tests, including a check
that the generated stub satisfies the backtest runner contract by importing
it and running generate().

Signed-off-by: Robin1987China <fanrenzhige@163.com>
Co-authored-by: Robin1987China <fanrenzhige@163.com>
 2026-06-20 17:55:33 +08:00
卫斯李andRobin1987China 33527a7f77 feat(factors): add 4 canonical academic alphas (#277) 
Expands the thin academic zoo (6 -> 10) with widely-cited, OHLCV-only
factors, each following the existing academic template (__alpha_meta__ +
cross-sectional z-score compute, src.factors.base primitives only):

- academic_strev: Jegadeesh (1990) one-month reversal — inverse 21d return.
- academic_high52w: George & Hwang (2004) 52-week-high momentum —
  close / ts_max(close, 252).
- academic_illiq: Amihud (2002) illiquidity — 21d mean of |return| per
  dollar volume.
- academic_retskew: Harvey & Siddique (2000) skewness premium — inverse
  60d return skewness.

All four auto-register via the registry AST scan and pass the
alpha_purity and lookahead gates (no future peeking, whitelisted imports
only). Verified: no inf, fully-valid latest row, factors suite green.

Signed-off-by: Robin1987China <fanrenzhige@163.com>
Co-authored-by: Robin1987China <fanrenzhige@163.com>
 2026-06-20 17:55:29 +08:00
卫斯李andRobin1987China 94f03c9fc2 test(tools): add coverage for trade journal analyzer + parsers (#268) 
trade_journal_tool.py and trade_journal_parsers.py shipped with no
dedicated tests despite carrying deterministic, user-facing numeric
logic. Adds 48 tests:

- parsers: A-share symbol qualification (.SH/.SZ/.BJ by prefix), buy/sell
  token normalization (CN + EN), market inference, format detection
  (tonghuashun/eastmoney/futu/generic/unknown), and end-to-end parse_file
  on CSV fixtures.
- tool: FIFO roundtrip pairing (exact PnL, proportional fee allocation,
  partial-fill splitting, unmatched-sell handling), profile win-rate/PnL,
  disposition-effect diagnostic, filter expressions, and the
  analyze_trade_journal dispatch + error paths (missing file, bad
  extension, profile/behavior/strategy modes).

Test-only; no production changes.

Signed-off-by: Robin1987China <fanrenzhige@163.com>
Co-authored-by: Robin1987China <fanrenzhige@163.com>
 2026-06-20 17:55:25 +08:00
卫斯李andRobin1987China f855be3103 test(tools): add coverage for chart pattern recognition (#269) 
pattern_tool.py (370 LOC) shipped with no dedicated tests despite
holding deterministic numeric detection logic that feeds strategy
design and post-backtest analysis. Adds 21 tests:

- find_peaks_valleys: exact peak/valley indices, too-short guard, NaN
  center handling.
- trend_line_slope: exact slope on a linear series, NaN prefix, flat=0.
- candlestick_patterns: doji (neutral), hammer (bullish), bullish and
  bearish engulfing.
- support_resistance: peak/valley clustering separation, flat-series shape.
- head_and_shoulders: <3-peak guard + a constructed positive case flagged
  at the head.
- double_top_bottom: exact double-top and double-bottom detection.
- triangle / broadening: output shape/dtype + flat-series zero guards.
- run_pattern: path-safety rejection, missing-OHLCV error, invalid pattern
  name, and an end-to-end run over an ohlcv CSV fixture.

Test-only; no production changes.

Signed-off-by: Robin1987China <fanrenzhige@163.com>
Co-authored-by: Robin1987China <fanrenzhige@163.com>
 2026-06-20 17:55:21 +08:00
Lanre Shittu 12bdca5c1c fix(loaders): honor interval in local loader by resampling 
The local loader accepted an ``interval`` argument but passed it only to the
cache key, so requesting e.g. 4H against hourly data silently returned the
file's native bars.

Resample to the requested interval using the same OHLCV aggregation as the
yfinance loader (open=first, high=max, low=min, close=last, volume=sum). A
finer-than-source interval cannot be fabricated from a file, so the source
bars are returned unchanged with a warning; requests already matching the
source granularity are passed through. The end-date filter is now inclusive
of the whole end day so intraday bars on that day survive for sub-daily
intervals.

Signed-off-by: Lanre Shittu <136805224+Shizoqua@users.noreply.github.com>
 2026-06-19 20:20:03 +01:00
Haozhe Wu 01d31f589f docs: add in-place-upgrade note for the 0.1.10 LangChain 1.x bump 
0.1.10 moved to LangChain 1.x. Upgrading in place with `pip install -U` over a pre-0.1.10 install can leave orphaned dist-info and break imports (e.g. langgraph). Add a short "Upgrading from an older version?" note to all five READMEs (en/zh/ja/ko/ar) pointing users to recreate the venv or `--force-reinstall`. Docs-only; a fresh install is unaffected.
 2026-06-19 23:19:18 +08:00
Haozhe Wu 2a5557edb6 chore(release): 0.1.10 — version sync + wheel packaging + Docker weasyprint libs (#271) 
* chore(release): 0.1.10 — version sync, wheel packaging, Docker weasyprint libs

Release-prep for 0.1.10 (bundles the global data layer + everything since 0.1.9):

- version 0.1.9 -> 0.1.10 synced across all surfaces: pyproject, frontend
  package.json/lock, sidebar footer + i18n, SKILL.md, Dockerfile LABEL. REST
  API + MCP server now derive the version from cli._version (were hardcoded
  "5.0.0" / unset).
- pyproject package-data ships skill example scripts (skills/**/*.py) so the
  wheel + sdist include them; fastmcp floor -> 2.14.0 (first release where the
  MCP version= kwarg applies).
- Dockerfile installs weasyprint native libs (Pango/HarfBuzz/Fontconfig/Cairo/
  gdk-pixbuf + DejaVu) so PDF reports render in the image; .dockerignore *.md
  -> /*.md (root-anchored, skill markdown still ships).
- README news (5 locales): 06-19 reframed as the v0.1.10 release; 06-19/06-18
  trimmed.

Verified: clean-room pip install imports all new loaders/tools; wheel + sdist
carry the eastmoney/sec-edgar skills incl. UTF-8 references + scripts; 4167
backend tests pass (weasyprint PDF path covered on CI).

* docs(release): credit v0.1.10 cycle contributors across READMEs

Refresh the Contributors section in all 5 READMEs from the v0.1.9 cycle to
the v0.1.10 cycle (PRs #162–#260): @Hinotoi-agent, @mvanhorn, @gyx09212214-prog,
@BillDin, @Robin1987China, @LemonCANDY42, @zwrong, @forge-builder, @skloxo,
@LeeCQiang, @KaiLuettmann, @ngoanpv, @ShahNewazKhan, @sambazhu, @bhlt, @octo-patch.
 2026-06-19 20:54:20 +08:00
Haozhe Wu adfbc82303 feat(data): global data layer — 8 new sources + 18 read-only data tools (#270) 
- 8 loaders (eastmoney/sina/stooq/yahoo free + finnhub/alphavantage/tiingo/fmp
  optional-key); registry now 18 sources, fallback chains re-ordered by IP-ban
  risk behind a shared throttled HTTP gate
- 18 read-only data tools (fund flow, dragon-tiger, northbound, margin, block
  trades, shareholder count, lockup, sector, research, news, SEC filings,
  financial statements, options chain, stock profile, screener, search, FRED
  macro, iwencai), exposed over MCP
- 3 shared clients (eastmoney/yahoo/sec_edgar); consolidated data-routing skill
  + per-source eastmoney/sec-edgar skills; read_file skill-ref resolution fix
- docs: 5 READMEs (Data Sources & Smart Fallback section + counts + 06-19 news),
  .env.example, SKILL.md

1200 tests green; live e2e LLM-routing + data smoke verified.
 2026-06-19 18:19:12 +08:00
Haozhe Wu ca9ded74fd docs: remove exposed scam discord invite code 2026-06-19 00:08:09 +08:00
Haozhe Wu 84bbf871c5 docs: add June 18 news (Research Autopilot Phase 1 + local Data Bridge loader + Discord security notice) 2026-06-18 23:54:02 +08:00
Haozhe Wu 0d0ef439bb chore: finish proposal_id hardening + document MCP initTimeout (#264) 
Completes work the 2026-06-18 self-merged batch left half-applied:
- Tighten the mandate proposal_id relay/preview parsers in api_server.py
  and cli/_legacy.py to the strict mp_[0-9a-f]{32} format already enforced
  by commit.py (#256), so the relay and the commit gate agree on format.
- Document the MCP initTimeout config key (shipped in #263) in README and
  wire its base-default passthrough in config/loader.py.
- Add the registered pytest.mark.unit marker to test_mandate_commit_security.py.
 2026-06-18 23:36:32 +08:00
Haozhe Wu e7a95106da fix(autopilot,loaders): repair zero-review runtime defects in #260 + #252 (#266) 
#260 Research Autopilot shipped two reproduced runtime-breaking bugs:
- run_research_autopilot never received the host session_id (it read only
  kwargs, which the LLM cannot know), so every call errored 'session_id is
  required'. Inject default_session_id via build_registry like the goal tools.
- generate_backtest_config returned a run_dir under ~/.vibe-trading/runs,
  which safe_run_dir rejected, breaking the advertised write_file -> backtest
  handoff. Add ~/.vibe-trading/runs to the allowed run roots.
Also validate the derived backtest source against VALID_SOURCES (fall back to
'auto' with a warning) and document the goal-replace semantics.

#252 local loader silently emptied tz-aware CSV/Parquet inputs (a tz-naive vs
tz-aware Timestamp comparison raised TypeError that was swallowed). Normalize
the timezone in _normalize_columns.

Adds registry-level + end-to-end tests for both autopilot tools and a tz
regression test for the loader.
 2026-06-18 23:36:27 +08:00
Haozhe Wu 83a33dfef1 docs(security): document official Discord + wallet-impersonation warning 
Adds an 'Official channels & impersonation' section pointing to the only
official Discord (HKUDS server) and warning that any wallet 'verification'
prompt is a scam, after the 2026-06-18 impostor-Discord incident.
 2026-06-18 22:33:34 +08:00
Haozhe Wu b20af90b8d docs: point community Discord to the official HKUDS server 
Re-add the Discord badge (5 READMEs) and an issue-template contact removed
in 8346def, now pointing at the org-controlled HKUDS community server
(discord.gg/6TdQnT5xcF, guild 1296348098003734629, ~1.9k members) — the same
server linked from the HKUDS org profile and LightRAG — instead of the
compromised invite. Keeps GitHub Discussions as a second contact.
 2026-06-18 22:18:36 +08:00
Haozhe Wu 8346def2aa fix(security): remove compromised Discord invite from README + issue template 
The discord.gg/2vDYc2w5 invite (in all 5 READMEs and the issue-template
contact links) resolves to a "Vibe-Trading" Discord server we no longer
control. It runs a fake Collab.Land wallet "verification" via
collab.layers-land.com — a throwaway domain registered 2026-06-18 — that
phishes users' crypto wallets. Remove every public pointer to it and replace
the issue-template community link with GitHub Discussions.
 2026-06-18 22:13:12 +08:00
Haozhe Wu 69545dbabd fix(live): extend Robinhood OAuth init timeout (#263) 2026-06-18 21:42:28 +08:00
卫斯李Robin1987ChinaHaozhe Wu
 		83ec532564 feat(autopilot): add goal-hypothesis bridge for Research Autopilot Phase 1 (#260) 
* feat(autopilot): add goal-hypothesis bridge for Research Autopilot Phase 1

New `run_research_autopilot` agent tool that connects the Hypothesis
Registry to the Research Goal runtime. Given a hypothesis_id, the tool
reads the hypothesis, creates a research goal with the thesis as
objective and backtest-relevant criteria, and returns a goal snapshot
with next-step hints.

Auto-registered via BaseTool subclass discovery.

Signed-off-by: Robin1987China <fanrenzhige@163.com>

* feat(autopilot): add generate_backtest_config tool for Phase 2

New `generate_backtest_config` tool auto-generates backtest config.json
from hypothesis metadata. Maps universe names (CSI 300, S&P 500, etc.)
to standard index codes and uses hypothesis.data_sources for the source
field. Writes config.json to a run directory and returns the path so the
agent can proceed to create signal_engine.py and call backtest.

Signed-off-by: Robin1987China <fanrenzhige@163.com>

* fix: harden research autopilot config generation

---------

Signed-off-by: Robin1987China <fanrenzhige@163.com>
Co-authored-by: Robin1987China <fanrenzhige@163.com>
Co-authored-by: Haozhe Wu <haozhe_wu@connect.hku.hk>
 2026-06-18 21:39:49 +08:00
Haozhe Wu d6bf209303 fix(provider): parse DeepSeek DSML tool calls 
Fixes #261.
 2026-06-18 21:38:57 +08:00
卫斯李Robin1987ChinaHaozhe Wu
 		cf32a6dfaa feat(loaders): add local data loader for CSV/Parquet/DuckDB files (#252) 
* feat(loaders): add local data loader for CSV/Parquet/DuckDB files

- New 'local' loader reads OHLCV from user-configured CSV/Parquet/DuckDB sources
- Config file at ~/.vibe-trading/data-bridge/config.yaml maps symbols to files
- Symbols prefixed with 'local:' auto-route to the local data source
- Integrated into loader registry with fallback chain support

Signed-off-by: Robin1987China <fanrenzhige@163.com>

* fix: allow duckdb local loader without path

---------

Signed-off-by: Robin1987China <fanrenzhige@163.com>
Co-authored-by: Robin1987China <fanrenzhige@163.com>
Co-authored-by: Haozhe Wu <haozhe_wu@connect.hku.hk>
 2026-06-18 21:37:39 +08:00
HinotobiandHaozhe Wu 43d33a036f [security] fix(live): contain mandate proposal identifiers (#256) 
* fix: contain live mandate proposal identifiers

* test: use valid mandate proposal id in consent helper

---------

Co-authored-by: Haozhe Wu <haozhe_wu@connect.hku.hk>
 2026-06-18 21:34:49 +08:00
Hinotobi 078ce6fcd2 fix(security): contain MCP swarm run IDs (#258) 
Signed-off-by: hinotoi-agent <paperlantern.agent@gmail.com>
 2026-06-18 21:30:30 +08:00
Hinotobi 27c74932b1 fix(memory): validate persistent memory types (#257) 
Signed-off-by: hinotoi-agent <paperlantern.agent@gmail.com>
 2026-06-18 21:30:23 +08:00
Haozhe Wu 1387b213d6 docs: update June 16 and 17 news 2026-06-17 01:30:35 +08:00
Haozhe Wu 7aa6b1d960 test: cover assistant-prefill handoff paths (#251) 2026-06-17 01:19:52 +08:00
卫斯李andwesley d0ff3599c1 fix(loop,providers): prevent assistant prefill errors + configurable Kimi User-Agent (#248) 
- Replace assistant-role handoff messages with user-role in agent loop
  to avoid "assistant message prefill" rejection on Anthropic Claude Opus 4.8+
- Allow MOONSHOT_USER_AGENT env var to override Kimi/Moonshot User-Agent header
  for compatibility with Kimi Coding API endpoint

Closes #246, Closes #204

Signed-off-by: wesley <wesley87@wesley87deMacBook-Pro.local>
Co-authored-by: wesley <wesley87@wesley87deMacBook-Pro.local>
 2026-06-17 01:17:24 +08:00
Matt Van HornandMatt Van Horn 43ffd041c0 feat(providers): add glm/zhipu provider alias and model-name inference (#247) 
Add a first-class `glm` alias of the existing `zhipu` provider so a model
name like `glm-4.6`/`glm-5.1` resolves to the Zhipu capability record and
base URL instead of silently falling back to the OpenAI slot.

- capabilities.py: share one `_ZHIPU_CAPABILITIES` record between `zhipu`
  and the new `glm` alias (mirrors kimi/moonshot), and infer `zhipu` for
  model ids starting with `glm`.
- llm_providers.json: register a `glm` entry aliasing the Zhipu env names
  and default model so onboarding/diagnostics surface it.
- tests: cover the alias, model-name inference, env-name resolution, the
  unchanged openai fallback, and the new registry default.

Signed-off-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
 2026-06-17 01:17:20 +08:00
Haozhe Wu ecbbd72eff fix: make harmonic backend optional (#250) 2026-06-17 01:11:51 +08:00
Hinotobi 97c67c9bb5 [security] fix(api): require auth for settings writes (#245) 
Require a valid bearer token (not just loopback peer IP) for settings WRITE endpoints (/settings/llm, /settings/data-sources) when API_AUTH_KEY is configured, closing a DNS-rebinding / loopback-trust bypass while keeping zero-config local dev writes.

Also reconciles the DNS-rebound security tests across the #241-#245 sprint: now that #242's rebound-host middleware 403s loopback requests with an attacker-controlled Host before they reach the route, the end-to-end rebound tests for /swarm/runs and /sessions/messages (#243) and /settings/llm (#245) assert the request is rejected at the host-gate layer (403) and never reaches the shell-tool / settings-write decision. Full suite: 3217 passed.
 2026-06-16 00:51:58 +08:00
Hinotobi 0568527cd6 [security] fix(api): require explicit opt-in for agent shell tools (#243) 
* fix(api): require explicit opt-in for swarm shell tools

* test: cover session shell tool opt-in boundary
 2026-06-16 00:37:50 +08:00
Hinotobi e5ee704e2e fix(api): reject untrusted loopback hosts (#242) 2026-06-16 00:37:18 +08:00
Hinotobi 73d3e1153b fix: require explicit auth for local shutdown (#241) 
Signed-off-by: hinotoi-agent <paperlantern.agent@gmail.com>
 2026-06-16 00:36:27 +08:00
sambazhu 205e4c1843 fix(frontend): sync completed chat attempts (#236) 2026-06-16 00:36:07 +08:00
gyx09212214-prog 9ba1f7bf7b fix(loaders): tolerate malformed RSSHub timeout env vars (#240) 
Signed-off-by: gyx09212214-prog <gyx09212214@gmail.com>
 2026-06-16 00:36:03 +08:00
gyx09212214-prog f4ce8b44d4 test(search): cover ddgs retry fallback behavior (#239) 2026-06-16 00:35:58 +08:00
gyx09212214-prog d8c55f3112 fix(run-card): write strict JSON for non-finite metrics (#238) 
Signed-off-by: gyx09212214-prog <gyx09212214@gmail.com>
 2026-06-16 00:35:52 +08:00
Haozhe Wu f63849e55e docs: news for 06-15 — web_search multi-engine resilience + Web UI run-continuity (#229) 2026-06-15 18:38:37 +08:00
Haozhe Wu f0ba648b1f fix(agent): make Stop responsive mid-stream and between tools (#229) (#235) 
Symptom 2 of #229: clicking Stop during a run did nothing, or only took
effect after a long delay. The cancel signal (_cancelled bool) was polled
only at the top of the ReAct iteration loop, so a Stop pressed while the
agent was mid-LLM-stream or executing tools was ignored until the entire
current iteration — the whole streamed answer plus all its tool calls —
finished and the next iteration check ran.

Add cooperative cancel checkpoints at the granularity the user actually
waits on:

- Switch _cancelled to a threading.Event (_cancel_event) so the flag is
  safe to poll from the streaming thread.
- Thread a should_cancel predicate into ChatLLM.stream_chat; the per-chunk
  loop breaks as soon as it is set, so a live stream stops at the next
  token instead of running to completion.
- After the stream returns, check the flag before processing tool calls:
  a cancel during streaming ends the run immediately and discards the
  partial turn (no tools run).
- Check the flag at the top of _process_tool_calls and between tool batches
  in _batch_execute, so a multi-tool turn stops after the current tool
  rather than launching the rest.

Limitation (documented in the PR): a connection that hangs before emitting
any chunk leaves the worker blocked in a socket read, which Python cannot
interrupt cooperatively from another thread; that case stays bounded by the
existing client timeout (TIMEOUT_SECONDS). Cancel now lands promptly in all
the normal "working" states (tokens streaming, tools running).

Tests: chat-level should_cancel breaks the chunk loop (and absence is a
no-op); loop-level cancel mid-stream ends the run as cancelled and skips
tool execution. Full suite: 3199 passed, 2 skipped.
 2026-06-15 17:41:43 +08:00
Haozhe Wu 9b9e9f50a0 fix(web): reconnect SSE when returning to a running session (#229) (#234) 
Symptom 1 of #229: navigating away from the chat page and back often made
the agent "stop working". The page run is server-side and the frontend only
subscribes via SSE; the unmount cleanup tore down that stream, but the
navigation effect only (re)subscribed when the URL session differed from the
store session. Returning to the SAME session hit neither branch, so SSE was
never reopened — in-flight events stopped arriving and the run looked frozen
until the 90s safety timeout flipped it to idle.

The backend already buffers events per session and replays them on connect
(GET /sessions/{id}/events?replay=active), and setupSSE already requests it;
the only defect was the missing re-subscribe on same-session remount.

Add that branch: re-hydrate like a reload — reset the transient streaming
view first (so replay=active rebuilds the in-flight turn from the ring buffer
instead of duplicating deltas onto the preserved text), refresh committed
history (covers an attempt that finished while away), then re-subscribe.

Symptom 2 (Stop ineffective during a hung/long turn) is cooperative-cancel
granularity in the agent core and is tracked separately.

Verified: tsc clean, frontend build passes, existing store/useSSE/Runtime
tests green (42).
 2026-06-15 17:40:56 +08:00
Haozhe Wu 541c4e5d8c ci: bump first-party actions to Node 24 (checkout/setup-python/setup-node v6) (#233) 
GitHub forces Node 20 actions onto the Node 24 runtime starting 2026-06-16
and removes Node 20 from runners on 2026-09-16. Bump the three first-party
actions to their latest Node-24-native majors across all workflows:

- actions/checkout v4 -> v6
- actions/setup-python v5 -> v6
- actions/setup-node v4 -> v6

Inputs are unchanged (node-version "20" is the frontend JS runtime, not the
action runtime). Third-party cloudflare/wrangler-action is out of scope.
 2026-06-15 17:08:04 +08:00
Haozhe Wu 04480e8a03 fix(tools): make web_search resilient via multi-engine fallback (#232) 
web_search relied on ddgs's default 'auto' backend, which leads with
DuckDuckGo — heavily rate-limited from cloud/shared IPs. When DDG
threw, the whole tool returned ❌ (issue #231) even though Google/
Bing/Brave/Mojeek/Yahoo were reachable.

- Pass an explicit ordered backend list so ddgs falls through a
  throttled engine to the next; override via VIBE_TRADING_SEARCH_BACKENDS
- Retry transient failures with linear backoff (3 attempts)
- Treat "No results found" as an ok+empty envelope, not an error
- On persistent failure, return an actionable message (retry / set
  backend env / use read_url) instead of a bare exception string
- Report the backend list used in the ok envelope

Tests: new tests/test_web_search_tool.py (6 cases, mocked ddgs) cover
multi-backend pass-through, env override, retry-then-succeed,
no-results, persistent-failure message, and max_results cap.
 2026-06-15 17:01:32 +08:00
bhlt 5399a078e8 fix(baostock): support baostock-native code format (sh.601398) (#230) 
The baostock loader previously only accepted tushare-style codes (601398.SH)
and silently rejected baostock-native codes (sh.601398) — the format baostock
uses in its own docs. _is_a_share() now accepts both (case-insensitive), and
_fetch_one() passes native codes through unchanged while still converting
tushare-style suffixes. Adds 10 unit tests.

Thanks to @bhlt for the fix.
 2026-06-15 16:38:49 +08:00
Haozhe Wu 59d98a741b docs: news for 06-14 — per-run token usage + progressive Run Detail charts 2026-06-14 23:03:30 +08:00
gyx09212214-prog 9f5b8afe7b fix(loaders): tolerate malformed crypto timeout env vars (#227) 
Malformed values like CCXT_TIMEOUT_MS=abc / OKX_TIMEOUT_S=abc raised ValueError at import time and could block startup. Add shared positive_env_int/float helpers in loaders/base.py; invalid/blank/non-positive timeout/budget values now warn and fall back to defaults. Valid values still honored.

Closes #227.
 2026-06-14 22:25:29 +08:00
gyx09212214-prog 9e7c1a8772 fix(loaders): include requested yfinance end date (#226) 
yfinance download(end=...) is exclusive while the loader API treats end_date as inclusive, so the final trading day was dropped. Pass end_date + 1 day to the yfinance call (bulk + single-symbol fallback); cache lookups/writes keep the original requested range.

Closes #226.
 2026-06-14 22:25:26 +08:00
LemonCANDY42 88dd54c7c3 perf(run-detail): add opt-in chart payloads (#225) 
Add opt-in chart payload modes for the Run Detail endpoint and UI. Default /runs/{id} stays byte-for-byte unchanged; ?chart_payload=summary returns metadata + chart_symbols (no bars/markers); ?chart_symbol=SYM returns one symbol's bars+markers. Run Detail UI renders summary first, then progressively loads selected symbols with loading/progress + cancel. New copy localized en + zh-CN.

Refs #215.
 2026-06-14 22:14:50 +08:00
LemonCANDY42 8678c289ce feat(agent): persist AgentLoop usage artifacts (#223) 
Persist provider-reported AgentLoop usage as a run-scoped artifact (<run_dir>/llm_usage.json) and surface it additively on /runs/{id}. Provider-reported tokens only — no prompt/content capture, no price estimation. Behind the existing usage guard so providers without usage_metadata write nothing. Live llm_usage SSE event keys unchanged.

Refs #214.
 2026-06-14 21:52:11 +08:00
Haozhe WuandLanre Shittu ccfe50fff2 fix(loaders): land RSSHub event/sentiment provider with explicit feed config 
Builds on @Shizoqua's verified provider core (#201) and resolves the three
open-box blockers from review:

- No built-in feed catalogue: the shipped DEFAULT_FEEDS routes (/stock/notice,
  /stock/news) don't exist in RSSHub, so a default run scored every bar 0.0.
  Feeds are now declared explicitly in config (event_feeds = list of
  {name, route_template, event_type, code_style}); DEFAULT_FEEDS is empty, so
  there is nothing to silently rot as RSSHub routes evolve.
- Per-symbol code formatting: a feed declares a code_style
  (raw / bare / exchange_prefix), so dotted backtest codes (600519.SH) are
  rewritten to the form a route needs (e.g. SH600519 for xueqiu-style routes).
- No silent all-zero scores: each fetch failure is logged per feed/code, and
  query_events raises EventProviderError when every fetch failed (configured
  but fully unreachable) instead of returning an all-zero event_score that
  reads as "sentiment considered, no signal". A reachable feed that returned
  no items stays a legitimate empty result.

PIT discipline, defusedxml hardening, the pluggable scorer, and the
engine/runner wiring are otherwise unchanged. Adds 12 tests for the code
styles, config parsing/validation, and the loud-failure path.

Thanks @Shizoqua (#201) for the provider core.

Co-authored-by: Lanre Shittu <136805224+Shizoqua@users.noreply.github.com>
 2026-06-14 06:50:02 +08:00
Haozhe Wuandskloxo f470947c81 feat(loaders,i18n): add baostock/tencent A-share sources + zh-CN frontend i18n 
A-share data sources (free, no-auth, bypass eastmoney CDN IP blocks):
- baostock loader (TCP protocol) + tencent loader (HTTP), both wrapped with
  cached_loader_fetch like the mootdx/akshare loaders so they honor the opt-in
  local cache
- registered into the a_share fallback chain (tushare -> mootdx -> baostock ->
  tencent -> akshare); auto-detect default source stays tushare
- pyproject [ashare] extra for the optional baostock dependency

Frontend i18n (English default, sidebar toggle to Chinese, persisted):
- i18next + react-i18next + browser-languagedetector; first visit defaults to
  English (localStorage-only detection, no navigator sniff), the toggle persists
  the choice across reloads
- en / zh-CN locales at full key parity, including the Runtime page
- i18n initialized in the test setup so useTranslation resolves real strings;
  restored the exported METRIC_LABELS so formatters consumers/tests keep working

Thanks @skloxo (#217) for the i18n layer and the A-share data sources.

Co-authored-by: skloxo <skloxo@users.noreply.github.com>
 2026-06-14 06:50:02 +08:00
LemonCANDY42 b215e99805 feat(frontend): add read-only runtime status dashboard (#210) 
Adds a dedicated /runtime page + sidebar entry rendering read-only /live/status: global halt, broker auth presence, runner liveness, active mandate limits/expiry, and derived runtime risk state. Fail-closed unavailable + empty-profile states. No write controls (no order/mandate/authorize/start/stop).

Verified locally: tsc --noEmit clean, Runtime.test.tsx 3/3 pass, vite build OK.
 2026-06-14 05:12:04 +08:00
Haozhe Wu 4207392433 docs: news for 06-13 — resume past session by ID from the CLI (#218) 2026-06-14 00:12:03 +08:00
Haozhe Wu b2573b4689 Merge pull request #218 from zwrong/pr/trace-offload 
feat(cli): show session-id on exit, add `resume <session-id>` subcommand
 2026-06-14 00:01:51 +08:00
zwrong 8a02b7c87f refactor(cli): extract _build_session_history, add resume-by-id tests 
Simplify / Brooks-Lint follow-up:

- Extract _build_session_history() helper to eliminate the duplicated
  get_messages -> role/content filter -> _HISTORY_RETAINED_TURNS slice
  pattern from both _maybe_resume_last_session and the resume path.
- Use _extract_max_iter instead of hardcoded max_iter=50 for the resume
  dispatch so future default changes propagate consistently.
- Add TestResumeByIdDispatch (3 tests) covering the vibe-trading resume
  <session-id> CLI dispatch path — previously uncovered.

Signed-off-by: zwrong <1003912034@qq.com>
 2026-06-13 17:25:56 +08:00
zwrong 2317947c4e feat(cli): add vibe-trading resume <session-id> and exit session-id display 
Exit interactive CLI shows the session ID so users can copy-paste
`vibe-trading resume <id>` to resume later. The resume subcommand
loads a specific session by ID and enters the interactive loop with
full history — messages are persisted through the existing
_run_one_turn / _append_message path.

Signed-off-by: zwrong <1003912034@qq.com>
 2026-06-13 17:25:33 +08:00
Haozhe Wu 8b0199d4b5 Merge pull request #206 from zwrong/pr/trace-offload 
feat(agent): complete trace.jsonl overhaul — zero truncation, offload, message entries, iter consistency
 2026-06-13 01:22:00 +08:00
Haozhe Wuandzwrong 4f70258d40 fix(trace): rebase trace offload onto current main 
Co-authored-by: zwrong <1003912034@qq.com>
 2026-06-13 01:21:24 +08:00
Haozhe Wu 4c4565a50f fix(web): clear partial stream text on provider retry 2026-06-13 01:02:00 +08:00
Haozhe Wu 65660e4087 docs: news for 06-12 — provider reliability overhaul (capability layer, provider doctor, streaming liveness) 2026-06-12 23:41:25 +08:00
Haozhe Wu fa2ad4b664 fix(providers): capability-gated payloads, explicit stream errors, liveness + provider doctor 
A cluster of provider reports traced to one root: every OpenAI-compatible
provider shared a single shim that applied DeepSeek/Kimi/Gemini quirks
globally and silently swallowed stream failures.

- Provider capability layer (capabilities.py): reasoning capture/replay,
  Gemini thought signatures, Kimi User-Agent (version from package
  metadata), OpenRouter reasoning body — gated per provider, no
  cross-contamination
- Optional native DeepSeek adapter via VIBE_TRADING_DEEPSEEK_ADAPTER
  (auto/native/openai-compatible) + 'deepseek' extra (langchain-deepseek)
- ChatLLM.stream_chat: silent fallback to chat() removed; failures raise
  ProviderStreamError with redacted provider/model context; one retry for
  transient errors in AgentLoop + swarm worker (deterministic 4xx fail fast)
- Empty model response diagnosed as empty_model_response instead of
  'reached max iterations'
- reasoning_delta SSE event (throttled to 1/s, first chunk immediate) +
  frontend 'Reasoning…' indicator; reasoningActive reset on safety timeout
- SSE heartbeats carry no replay id; replay falls back to full buffer on
  unknown last-event-id; read-only tools get a hard timeout
  (VIBE_TRADING_TOOL_TIMEOUT_SECONDS) instead of hiding behind heartbeats
- 'vibe-trading provider doctor' prints redacted provider/model/package/
  proxy diagnostics; preflight includes the same hint
- kimi-k2.x temperature=1 requirement auto-applied; Kimi path verified
  end-to-end against the live API (kimi-k2.6 tool calls + strict
  multi-turn reasoning replay)

Refs #208 #203 #195 #204
 2026-06-12 23:41:19 +08:00
zwrong 38582c564d fix(trace): all entries carry iter right after type, resume before start/message 
Three interrelated fixes for trace.jsonl field consistency:

1. Field order — every entry now has `iter` immediately after `type`.
   This was already the convention for tool_call, tool_result, thinking,
   answer, and cancelled. Fixed start, message (user+assistant), and end
   to follow the same pattern.

2. Iteration values — start and message (user) previously used
   `self._run_iteration` (which is 0 before the loop begins), so the
   first turn showed iter=0 while the first loop iteration showed iter=1.
   Changed to `self._run_iteration + 1` so they share the same iter as
   the first loop iteration. end now carries `iter` in addition to
   `iterations` (the per-run count), so it participates in iteration
   grouping like every other entry.

3. Resume ordering — the iteration-counter restore logic (reading max
   iter from an existing trace to continue a session) was placed AFTER
   start/message were written, so resumed sessions had iter=0 on those
   entries. Moved the resume block before the write calls.

Tests added (TestTraceFieldConsistency):
- test_tool_result_iter_second — write_tool_result() dict order
- test_generic_write_preserves_type_then_iter — generic write order
- test_all_entry_types_carry_iter — 10 entry types all have iter at
  position 1; catches future regressions

Closes the long-standing gap where start/message/end could not be
attributed to a specific turn in multi-turn sessions.

Signed-off-by: zwrong <1003912034@qq.com>
 2026-06-12 13:44:19 +08:00
zwrong c4dec47576 feat(agent): complete trace.jsonl — zero truncation, message entries, tool result offload 
- TraceWriter: add write_tool_result() with 50K offload threshold + 500-char preview
- TraceWriter: read() auto-resolves offloaded results from tool-results/*.txt
- TraceWriter: add find_trace_dir() — sessions/ first, runs/ fallback
- loop.py: sessions/<id>/trace.jsonl when session_id valid, runs/ fallback for -p
- loop.py: remove all [:N] truncation from trace entries (prompt, thinking, answer, args, summary)
- loop.py: add message type entries (user/assistant) for full conversation replay
- loop.py: use write_tool_result() for proper inline/offload routing
- loop.py: auto_compact transcript now uses trace.dir_path
- _legacy.py: find_trace_dir() in _build_history_from_trace, cmd_show, cmd_trace, cmd_continue
- _legacy.py: cmd_trace display supports new fields (result_path, result_preview, result_size)
- tests: 18 new TraceWriter tests covering inline, offload, boundary, mixed, unicode, find_trace_dir

Signed-off-by: zwrong <1003912034@qq.com>
 2026-06-12 13:44:15 +08:00
Haozhe Wu c390aa8462 docs: news for 06-10/06-11 — swarm market-data hardening, Docker Ollama + persistence 
Two entries across all 5 READMEs (3 visible + earlier rotated into
details): 06-11 swarm loader-backed get_market_data rollout (#198/#199/
#200, bare-ticker grounding, 48 tools) bundled with Docker state
persistence (#197); 06-10 host-side Ollama reachability (#196).
 2026-06-11 13:34:12 +08:00
Haozhe Wu 2ba6ae2f0c fix(docker): persist user agent state across container rebuilds 
All user-level agent state lives under ~/.vibe-trading in the container
(persistent memory, sessions.db search index, user-created skills,
shadow accounts, hypothesis registry, broker connector config), but
docker-compose only mounted volumes for web sessions and backtest runs.
Every `docker compose up --build` therefore recreated the container and
silently wiped the user's memory store (#197). Swarm run history and
uploads were lost the same way.

- Mount vibe-home -> /home/vibe/.vibe-trading, plus vibe-swarm-runs and
  vibe-uploads.
- Pre-create /home/vibe/.vibe-trading owned by the service user in the
  Dockerfile so the named volume inherits writable permissions (a named
  volume mounted onto a missing path is created root-owned and the
  non-root user cannot write).
- README: document what persists across updates.

Verified end-to-end: memory written via PersistentMemory in one
container is intact in a fresh container on the same volume; the
counterfactual run without the volume reproduces the wipe.

Fixes #197.
 2026-06-11 13:17:04 +08:00
Shah Newaz KhanClaude Opus 4.8Shah Newaz Khan
 		b6fd9b1fd4 fix(docker): reach host Ollama via host.docker.internal (#196) 
* fix(docker): reach host Ollama via host.docker.internal

Inside the container, OLLAMA_BASE_URL=http://localhost:11434 resolves to
the container itself, so the agent's preflight check and AgentLoop fail
with "Connection refused" even when Ollama is running on the host.

Override OLLAMA_BASE_URL to http://host.docker.internal:11434 in the
compose service (leaving agent/.env correct for bare-metal runs) and add
an extra_hosts host-gateway mapping so the same file also works on Linux.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* fix(docker): parameterize OLLAMA_BASE_URL and document host networking

Address PR review feedback:
- Make OLLAMA_BASE_URL overridable via Compose variable substitution
  (${OLLAMA_BASE_URL:-http://host.docker.internal:11434}) so it can be
  changed without editing the compose file when Ollama runs elsewhere.
- Note the Docker Engine >= 20.10 / Compose v2 requirement for the
  host-gateway mapping, both inline and in the README Docker section.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

---------

Co-authored-by: Shah Newaz Khan <shah.khan@lexful.ai>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
 2026-06-11 12:55:48 +08:00
Haozhe Wu dfd71b4bc2 feat(swarm): ground bare US tickers mentioned in swarm prompts 
Grounding previously only matched loader-suffixed symbols (NVDA.US),
so an investment_committee run on "long or short on NVDA" got no
pre-fetched ground-truth prices and workers fell back to ad-hoc
scripts (#198, fix direction 5).

Promote bare all-caps tokens to <TOKEN>.US under guards:
- 2-5 uppercase letters only, dotted compounds rejected on both sides
  (FOO.USDA promotes neither side; sentence-ending "NVDA." still works)
- stopword list for finance/English acronyms (ETF, CEO, GDP, USD,
  bare crypto symbols, index providers like MSCI, ...)
- suffixed matches are blanked before the bare scan, so BTC-USDT can
  never leak a bogus BTC.US
- promotions sort after explicit symbols, so explicit symbols win the
  max-symbols cap; the per-symbol fetch remains the final validator

Closes the last open item of #198.
 2026-06-11 12:22:51 +08:00
Haozhe Wu d7d1654f32 fix(swarm): expose get_market_data across all market-data presets 
PR #199 added the loader-backed get_market_data tool to
investment_committee and equity_research_team. Extend it to the 16
remaining workers (across 11 presets) whose skills include tushare,
yfinance, or okx-market, so every OHLCV-capable worker gets the
loader-first path and the worker prompt policy.

Add a regression test locking the convention: any preset worker with a
market-data skill must whitelist get_market_data.

Part of #198.
 2026-06-11 12:14:15 +08:00
Chengcheng DingandChengcheng Ding d1ed1019c0 fix(swarm): preserve preset context for continuations (#200) 
Signed-off-by: Chengcheng Ding <billdin1999@gmail.com>
Co-authored-by: Chengcheng Ding <billdin1999@gmail.com>
 2026-06-11 12:11:37 +08:00
Chengcheng DingandChengcheng Ding 7436986246 fix(swarm): expose loader-backed market data tool (#199) 
Signed-off-by: Chengcheng Ding <billdin1999@gmail.com>
Co-authored-by: Chengcheng Ding <billdin1999@gmail.com>
 2026-06-11 12:10:52 +08:00
Haozhe Wu a80b4431bf fix(web): arm SSE safety timeout so a no-event turn can't hang forever 
The streaming safety timeout guarded on lastEventRef.current being truthy, but
it starts at 0 and only real SSE events (not the keep-alive heartbeat) update
it. A turn whose very first event never arrives — e.g. the LLM provider hangs
before emitting a token — left it at 0, so the guard short-circuited and the UI
stayed on "Agent is working…" indefinitely. Arm the clock on entering the
streaming state so the timeout fires after sse_timeout and resets to idle with a
notice. Turns issue #195's hang into a recoverable timeout.
 2026-06-09 20:33:24 +08:00
Haozhe Wu f735e4ca51 fix(backtest): make mootdx & futu first-class config sources 
Two whitelists/maps had drifted as mootdx and futu loaders were added:

1. BacktestConfigSchema._VALID_SOURCES rejected an explicit
   source="mootdx"/"futu" even though both are registered loaders (and in
   FALLBACK_CHAINS), while src/tools/backtest_tool.py already allowed them.
   Fix: one canonical VALID_SOURCES in backtest.loaders.registry, consumed by
   both backtest.runner and src.tools.backtest_tool so they cannot diverge.

2. metrics annualisation maps (_TRADING_DAYS / _BARS_PER_DAY) lacked mootdx and
   futu, so intraday backtests on them fell back to bars_per_day=1 and
   mis-annualised vol/Sharpe. Fix: add both as equity sources (252 days, 240-min
   session, mirroring tushare/akshare).

Regression tests: mootdx/futu accepted by the config schema, every registered
loader covered by VALID_SOURCES, and intraday bars_per_year correct for both.
 2026-06-09 19:17:21 +08:00
Matt Van HornandMatt Van Horn 05e492a99a docs: add custom data loader guide (#194) 
Signed-off-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
 2026-06-09 19:06:58 +08:00
Haozhe Wu bbd8466ded fix(web): surface auth-required reason on remote 403 (#191) (#192) 
Chat sends from a non-loopback client (another machine, VM host, LAN
device) without API_AUTH_KEY returned 403 on every sensitive endpoint,
but the UI only showed a generic "Failed to send message, please retry."
The send path now surfaces AUTH_REQUIRED_MESSAGE so users know to set
API_AUTH_KEY (and enter it once in Settings) or browse via localhost.

Also document the localhost-vs-LAN rule in the web-UI setup and add a
2026-06-09 news entry across all 5 locale READMEs.
 2026-06-09 11:10:02 +08:00
Haozhe Wu 6b37c040c1 revert: drop GHCR pre-built image path (#187) 
The Path E "pull a pre-built image, no clone" deploy path was open-box
broken: the GHCR package defaults to private, so anonymous `docker pull`
fails, and the path was never verified end-to-end against the main repo
(the contributor tested a self-built image, not ghcr.io/hkuds/vibe-trading).

Removes the publish workflow, docker-compose.prod.yml, the Dockerfile
VERSION arg, and the README Path E section/table row/tree note. The
2026-06-08 news entry is rewritten to cover only the Gemini 3.x
dict-path fix (#184). Also drops the now-orphan edge-build paths-ignore.

If a buildless deploy path is wanted later, do it properly: make the
package public, verify an anonymous pull works, then document it.
 2026-06-08 10:45:25 +08:00
Haozhe Wu 4389246350 docs(readme): add 2026-06-07 + 2026-06-08 news in 5 locales 
06-07 live swarm status in chat (#188) + preset routing (#189); 06-08 one-pull GHCR image (#187) + Gemini 3.x dict-path fix (#184). Older 06-05/06-04 folded into Earlier news to keep 3 visible.
 2026-06-08 10:29:02 +08:00
Haozhe Wu 7dd9e58870 ci(docker): skip GHCR edge build for doc-only main pushes 
Release builds still always fire via the release trigger; only the edge-on-main-push build is skipped when a commit touches nothing but .md / docs/ / wiki/.
 2026-06-08 10:29:02 +08:00
Chengcheng DingandChengcheng Ding 0f0fcb6280 fix(swarm): honor explicit preset names (#189) 
Signed-off-by: Chengcheng Ding <billdin1999@gmail.com>
Co-authored-by: Chengcheng Ding <billdin1999@gmail.com>
 2026-06-08 10:13:12 +08:00
Chengcheng DingandChengcheng Ding f194e51f4d feat(web): show swarm agent status in chat (#188) 
Signed-off-by: Chengcheng Ding <billdin1999@gmail.com>
Co-authored-by: Chengcheng Ding <billdin1999@gmail.com>
 2026-06-08 10:13:07 +08:00
Kai LüttmannandCopilot d77f6b3379 feat: publish pre-built image to GHCR on release (closes #183) (#1) (#187) 
* feat: publish pre-built image to GHCR on release (#183)

* docs: remove Watchtower references from README

* Add workflow_dispatch trigger to Docker publish workflow

* Allow Docker publish on additional branch

* Update Docker publish workflow triggers

Removed manual trigger and restricted push branches.

---------
authored-by: Kai Lüttmann <33122946+KaiLuettmann@users.noreply.github.com>

Co-authored-by: Copilot <198982749+Copilot@users.noreply.github.com>
 2026-06-08 10:12:59 +08:00
Pham Van Ngoan 0e4ffc8c66 fix(llm): preserve Gemini thought_signature through the AgentLoop dict path (#184) 
#176 added the Gemini thought_signature round-trip but only covered the
in-memory AIMessage path its unit tests exercise. The AgentLoop replays
conversation history as OpenAI-format dicts, stamping the signature into
tool_calls[i].extra_content.google.thought_signature
(loop.py _attach_tool_call_thought_signatures).

LangChain's _convert_dict_to_message discards extra_content during the early
_convert_input step of invoke()/stream() -- before _get_request_payload
re-injects -- so _signature_maps reads an empty additional_kwargs and the
outbound request goes out unsigned. Gemini then rejects the next tool-using
turn with HTTP 400 "Function call is missing a thought_signature in
functionCall parts". This broke every tool-using loop turn on gemini-3.x
(single or parallel), e.g. the parallel load_skill case.

Override ChatOpenAIWithReasoning._convert_input -- the single chokepoint both
invoke and stream call while input is still raw dicts -- to lift signatures
back onto the converted AIMessage in the same
additional_kwargs["tool_call_thought_signatures"] shape the in-memory path
uses, so the existing _signature_maps / _inject_tool_call_thought_signatures
machinery handles both paths identically. Idempotent via an
isinstance(raw, dict) guard. No parallel-specific handling needed: Gemini
signs only the first of N parallel calls and that signature covers the block.

Add regression tests in TestChatOpenAIWithReasoningOutboundPayload that drive
the real dict path (the gap #176's tests missed): red without the fix
(KeyError: 'extra_content'), green with it.
 2026-06-08 10:12:56 +08:00
Haozhe Wu b31e9b4af1 docs(readme): add 2026-06-06 alpha compare news in 5 locales 2026-06-07 01:08:47 +08:00
Haozhe Wu b110bdc3b2 feat(factors): add alpha compare across CLI, REST, Web UI and agent tool 
Head-to-head comparison of hand-picked Alpha Zoo alphas, sharing one core
(compare_runner.compare_alphas) across four surfaces:

- CLI: `vibe-trading alpha compare <id1> <id2> ... --sort ir`
- REST: POST /alpha/compare + SSE progress/result stream
- Web UI: Compare view in Alpha Zoo (catalogue multi-select -> ranked table)
- Agent: read-only `alpha_compare` tool (now 47 tools)

Adds a backward-compatible `only=` subset filter to run_bench so comparing
a few alphas no longer benches the whole zoo. Ranks by IC mean/std, IR,
IC-positive ratio or sample count, each with its gap to the leader.

39 new backend tests; frontend tsc + 197 vitest + build green.
 2026-06-07 01:08:42 +08:00
Haozhe Wu 9b9e83fe6c docs(readme): add 2026-06-05 news (Dhan/Shoonya, Gemini, docstrings, tests) in 5 locales 2026-06-05 21:52:59 +08:00
Haozhe Wu 36c5a4457c fix(deps): patch 7 Dependabot alerts (react-router XSS/DoS, vitest RCE) 
Bump react-router-dom ^7.1.0 -> ^7.15.0 (resolves to 7.17.0), clearing 6 react-router advisories (XSS, open redirect, DoS, turbo-stream arbitrary constructor; patched across 7.13.2-7.15.0).

Bump vitest + @vitest/coverage-v8 ^3.2.6 -> ^4.1.0 (resolves to 4.1.8), clearing the critical Vitest UI-server arbitrary file read/exec (patched in 4.1.0) and aligning the coverage peer dependency on the v4 line. Frontend suite (197 tests) green on vitest 4; production build passes.
 2026-06-05 20:56:40 +08:00
sambazhu d17faacefc test: add frontend test suite + backend security/coverage tests 
Cherry-picked the test-infra commit from #175 (sambazhu): vitest setup + 19 frontend test files (197 tests), backend auth/path-traversal/CORS tests, yfinance crypto fallback (BTC-USDT->BTC-USD), and CI coverage + frontend-test steps. Dropped the PR's separate i18n commit to stay consistent with #163.

Fix: align @vitest/coverage-v8 to ^3.2.6 (was ^4.1.8, incompatible with vitest ^3.2.6 - blocked npm install and the new npm ci CI step).
 2026-06-05 20:41:14 +08:00
leeadmin 44f14b67f8 docs(factors): add Chinese docstrings to factor zoo 
Cherry-picked the factor-module Chinese docstrings from #180 (alpha101 / gtja191 / qlib158 / academic, 452 files, purely additive). Dropped the i18n parts — react-i18next deps + 29 swarm-preset name_zh/description_zh (currently unused) — to stay consistent with the #163 decision to not bundle i18n yet.
 2026-06-05 20:25:36 +08:00
Matt Van HornandMatt Van Horn 51b207f0ba fix: round-trip Gemini thoughtSignature through OpenAI-compat tool calls (#176) 
Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
 2026-06-05 20:15:42 +08:00
Panditvirchandra77-lgtmandHaozhe Wu dda7a5739b feat(trading): add Dhan + Shoonya Indian broker connectors (paper-capped) 
Two broker_sdk connectors for the Indian equity and F&O markets (NSE/BSE):
Dhan (dhanhq) and Shoonya / Finvasia (NorenRestApiPy). Read access plus
locally simulated paper-account order placement.

Both brokers expose no sandbox and no runtime paper/live discriminator — a
single token / TOTP login reaches the same real account — so, following the
Longbridge precedent, the order path is structurally capped at paper:

- place_order / cancel_order refuse any non-paper config at the first line, so
  a flipped `profile` override can never reach a live order;
- no live order-placing profile is registered (paper-sdk read-only,
  paper-trade local-sim, live-sdk-readonly read-only — three profiles each,
  mirroring Longbridge);
- DHAN/SHOONYA tool-class maps wired into the live classifier registry;
- order ops pinned WRITE; secrets redacted; pyotp made an optional dependency;
  Shoonya sessions cached per user_id to avoid cross-account reuse.

Tests cover profile registration, the non-paper fail-closed guard, paper
simulation, WRITE-pinned order ops, secret redaction, and a regression guard
that these no-discriminator brokers never expose a *-live-trade profile.

Maintainer follow-up to PR #174: the original submission shipped live order
placement with no structural paper/live discriminator (and an invalid asset
class that crashed the live path); this revision caps both connectors to
paper + read-only to keep the paper/live boundary structural, never an
agent-flippable config flag.

Co-authored-by: Haozhe Wu <haozhe_wu@connect.hku.hk>
 2026-06-05 15:45:11 +08:00
Haozhe Wu c2257fe6f3 fix(wiki): make footer counter all-time cumulative, not a 30-day window 
Sum the full visits history instead of windowing to 30 days, so the agent/human counts grow monotonically since launch (data is never GC'd).
 2026-06-05 14:54:43 +08:00
Haozhe Wu ee28390a2c fix(wiki): cache-bust main.js so browsers load the traffic-counter JS immediately 
main.js is browser-cached up to 4h; the fresh HTML still pointed at the unversioned /main.js, so returning visitors kept running the pre-counter JS and the footer never populated. Versioned query forces an immediate fetch of the new module.
 2026-06-05 14:47:39 +08:00
Haozhe Wu cb82c6262c fix(wiki): align live-trading copy with shipped Robinhood Agentic support; hide footer counter until data loads 
The wiki still claimed 'no live trading / does not execute live trades', which contradicts the shipped opt-in bounded Robinhood Agentic channel (and the README). Aligns home, docs, and alpha-library copy with the README's framing. Also fixes a CSS specificity bug where .footer-traffic{display:flex} overrode the [hidden] attribute, showing placeholder dashes before /api/stats data arrived.
 2026-06-05 14:39:55 +08:00
Haozhe Wu e2a42aa4ab fix(wiki): cache PyPI installs in D1, drop redundant GitHub call from /api/stats 
pypistats/GitHub calls from the Worker intermittently rate-limit, blanking the footer. PyPI now reads through a D1 last-good cache so it never goes empty; GitHub stars are dropped here since main.js already loads them client-side.
 2026-06-05 14:14:07 +08:00
Haozhe Wu 6fac55d795 fix(ci): deploy wiki in config mode so Pages Functions compile 
pages deploy <dir> from repo root uploads functions/ as static files and never compiles them, so /api/stats (the agent-vs-human counter) was never live. Run wrangler from wiki/ (workingDirectory) with the wrangler.toml present so functions/ is detected and compiled.
 2026-06-05 14:10:04 +08:00
Haozhe Wu 25b6ee5d72 feat(wiki): add agent-vs-human traffic counter to vibetrading.wiki 
First-party, anonymous analytics for the public wiki — no client-side telemetry in the pip package. Pages Function middleware classifies each page request (AI agent / bot / human) by User-Agent server-side and counts it into D1; /api/stats serves the footer's aggregate counts alongside public PyPI install and GitHub star numbers.
 2026-06-05 14:03:17 +08:00
Haozhe Wu 1cd4ba19df docs(readme): add 2026-06-04 opt-in data cache news entry across 5 locales 2026-06-04 23:40:06 +08:00
Matt Van Horn ff56134b91 feat(loaders): opt-in local data cache for all backtest data sources (#177) 
Add an opt-in local market-data cache (VIBE_TRADING_DATA_CACHE) for the backtest
loaders. When enabled, every data source — tushare, okx, ccxt, akshare, mootdx,
yfinance, futu — caches settled historical bars under ~/.vibe-trading/cache
(user home, never the repo tree) so repeated and long-horizon / cross-market
backtests skip the network and avoid provider rate limits.

- Generic loader_cache_get / loader_cache_put / cached_loader_fetch helpers in
  the loader base; batch/connection loaders (yfinance, futu) skip the bulk
  download / FutuOpenD connection entirely on a full cache hit.
- Staleness guard: ranges ending today or in the future are never cached, since
  the key is content-addressed on end_date, not fetch time.
- Cached frames round-trip byte-identical to freshly fetched ones (columns name
  and per-level index dtype preserved across the duckdb parquet round-trip).
- pid + uuid tmp filenames for concurrency-safe atomic writes; cache version 2.
- Documented in .env.example; staleness, real-duckdb, and batch-loader tests.

Closes #171.

Signed-off-by: Haozhe Wu <haozhe_wu@connect.hku.hk>
 2026-06-04 23:14:32 +08:00
Roger 1a5f096496 docs: add AI contributor guide (#173) 
Add AGENT_CONTRIBUTOR_GUIDE.md for AI/automation-assisted contributors: safe
local checks, high-risk broker/MCP/credential surfaces, targeted test hints,
documentation and security rules, PR expectations, and rollback notes. Linked
from CONTRIBUTING.md so agent-assisted contributors see it before changing code.

Adds a guard test (agent/tests/test_agent_guide_paths.py) that parses the guide
for in-repo paths and asserts each still exists, so the guide turns a renamed
test/entry-point into a red CI run instead of silently rotting. pytest --ignore
targets (gitignored e2e suites) and .env do-not-write references are excluded by
design.

Signed-off-by: Haozhe Wu <haozhe_wu@connect.hku.hk>
 2026-06-04 18:42:26 +08:00
Haozhe Wu 2c2c072319 docs: add trendshift badge to readmes 2026-06-04 10:43:21 +08:00
Haozhe Wu d1309b4f73 docs(readme): add 2026-06-03 community-triage news entry across 5 locales 
Adds the 2026-06-03 News entry (trace call_id correlation #168, dead-doc
reference cleanup #166, langchain-community install-warning clarification
#167, Gemini thoughtSignature scoped to help-wanted #170) to the English,
Chinese, Japanese, Korean, and Arabic READMEs, and collapses the 2026-05-31
entry into the "Earlier news" section to keep three latest entries visible.
 2026-06-03 23:39:18 +08:00
Haozhe Wuandzwrong 65a14c632e feat(loop): add call_id to trace entries for tool-call correlation 
Adds the tool call_id to the tool_call and tool_result trace records so a
result can be correlated back to its originating call when replaying a trace.
Keeps the existing [:200] truncation on serialized args to avoid bloating
trace files with large tool inputs.

Equivalent to #168 (arg truncation preserved).

Co-authored-by: zwrong <1003912034@qq.com>
 2026-06-03 22:59:25 +08:00
Haozhe Wu b5efee66ab docs: remove dead docs/live-trading/SPEC.md references from source comments 
The live-trading source files referenced docs/live-trading/SPEC.md in
docstrings, comments, and tests, but docs/ is gitignored (internal planning),
so external contributors hit a dead path when they grep / ls-tree for it.
Genericize the references to "the live-trading SPEC"; docs/ stays internal.

Closes #166
 2026-06-03 22:59:21 +08:00
Haozhe Wu 0df32f2eb3 feat(live): direct-SDK bounded-autonomy order gate + HK/CN universe 
Function-based mandate gate for direct-SDK live orders mirroring the MCP gate:
mandate load + schema/expiry check, kill-switch halt, quantity→notional pricing,
fail-closed pre-trade checks via check_mandate, per-broker audit, and a daily
count consumed only on a confirmed placement. Shared daily counter (order_guard
now delegates to it). AssetClass gains HK_EQUITY/CN_EQUITY and OrderIntent gains
an explicit asset_class so the mandate gate buckets HK/A-share correctly. The 6
connector classification maps are wired into the live registry for fail-closed
WRITE coverage. Covered by two adversarial audit rounds; full suite 2979 passed.
 2026-06-02 12:52:14 +08:00
Haozhe Wu 8563d5e78b feat(trading): add 6 SDK broker connectors with paper+live order placement 
Tiger/Longbridge/Alpaca/OKX/Binance/Futu connectors under a new broker_sdk
transport — read-only account/positions/orders/quote/history + paper-account
order placement, plus bounded mandate-gated order placement for five of them.
Longbridge is paper + read-only only (its API exposes no runtime paper/live
discriminator). Per-broker structural paper/live guards: Tiger 17-digit account,
Alpaca/Binance host separation, OKX flag + uid pin, Futu trd_env via get_acc_list.
New trading_place_order/trading_cancel_order tools; generic period/limit wired
through the tool and CLI. News updated across all 5 READMEs.
 2026-06-02 12:51:50 +08:00
Haozhe Wu ede8a8b33b fix(ui): clarify dev-jargon labels, drop drift-prone chip counts, add Compare loading state 
Addresses UX feedback from #163: rename systems jargon leaking into chrome
(ev -> evidence, Run bench -> Run benchmark, Decay -> Decay (days), Compact
context -> Summarize conversation), add a hint for Reasoning effort, remove
hardcoded skill/preset/tool counts that go stale under frequent releases, and
show a skeleton on the Compare route while run data loads.
 2026-06-02 10:37:20 +08:00
Roger de2a180ebd docs: clarify OpenClaw MCP smoke test (#165) 
Closes #164
 2026-06-02 10:03:56 +08:00
Octopus bcc3ae6405 feat: upgrade MiniMax default model to M3 (#162) 
Add MiniMax-M3 as the new default model and retain MiniMax-M2.7 as
the legacy alternative (incl. -highspeed). Update llm_providers.json
registry, cli onboard provider choices, .env.example example, and the
provider-default regression tests.

Signed-off-by: Octopus <liyuan851277048@icloud.com>
Signed-off-by: octo-patch <liyuan851277048@icloud.com>
 2026-06-02 00:52:45 +08:00
Haozhe Wu 15c7a9b577 docs(release): credit v0.1.9 cycle contributors across READMEs 
Refresh the Contributors section in all 5 READMEs from the v0.1.8 cycle
list to the v0.1.9 cycle (PRs #124–#160): @toanalien, @ai7eam-dev,
@shadowinlife, @DoubleSky123, @ArthurXi, @omcdecor-cyber, @Soli22de,
@ruok808, @faizack, @fightZy, @lcwSeven, @Teerapat-Vatpitak.
 2026-06-01 23:07:06 +08:00
Haozhe Wu 5a06371a94 chore(release): 0.1.9 
Roll-up release since 0.1.8 — connector-first broker profiles (IBKR +
Robinhood Agentic Trading), Research Goal runtime, swarm reconcile/MCP
keepalive + worker MCP tools + strict alpha-bench + retry_run, the
agent/cli/ package refactor, the mootdx A-share loader, a robustness pass,
and the session/version fixes (#150 #156 #157 #158 + UI indicator).

Version → 0.1.9 across pyproject, SKILL.md, frontend APP_VERSION, and the
wiki docs version selector. MCP tool count corrected to 36 (adds retry_run)
across all 5 READMEs, with the ar/ko tool lists rebuilt from a stale 22 and
the ar skill count fixed 75→77. CHANGELOG [0.1.9] + 0.1.9 news entry added
to all 5 READMEs. Home backtest card data-source count corrected 3→7.
 2026-06-01 23:00:43 +08:00
Haozhe Wu df70288dee fix(cli): derive version from pyproject, drop hardcoded fallback (#156) 
The shipped 0.1.8 wheel reported 0.1.7 because a hardcoded constant was
not bumped on release. The metadata path already fixed installed packages;
this removes the remaining hardcoded fallback so an un-installed checkout
reads the version straight from pyproject.toml — making pyproject the one
and only place the version is written, with no constant left to drift.

- _version: tomllib-based pyproject fallback (>=3.11 guaranteed), "unknown"
  sentinel only if the tree is moved away from its pyproject.toml
- tests: pyproject fallback matches declared version, resolved-not-unknown,
  and a guard against reintroducing any hardcoded version literal
 2026-06-01 22:39:57 +08:00
Haozhe Wuandai7eam-dev d2f853233f feat(swarm): retry failed/stale/cancelled runs (HTTP + MCP) 
Completes the swarm-retry capability extracted from #159 and makes it
usable end-to-end instead of dead API surface.

- api_server: POST /swarm/runs/{id}/retry — reconciles a stale "running"
  run first, refuses a genuinely active run (409), else re-launches a new
  run with the original preset + user_vars
- mcp_server: retry_run(run_id) tool so the list_runs -> spot-failure ->
  retry loop works from OpenClaw/Claude Desktop (the frontend has no swarm
  dashboard; MCP + HTTP are the real consumers)
- api.ts: retrySwarmRun client method (parity with cancelSwarmRun)
- tests: missing-run / running-refused / happy-path relaunch, MCP well-known
  tool name, and HTTP traversal rejection for the new endpoint
- docs: bump current MCP tool count 35 -> 36 across the 5 READMEs (ar/ko
  prose still carries a stale 22-tool list — tracked separately)

Co-authored-by: ai7eam-dev <ai.7eam@gmail.com>
 2026-06-01 22:39:57 +08:00
Haozhe Wuandai7eam-dev a0da6c7727 fix(ui): reliable session running-status indicator 
Scoped extraction of the UI running-status indicator from #159 (the
remaining commits there overlapped with #157/#158 or touched the
protected providers/ module).

- useSSE: add attempt.created/started, message.received, session.created,
  tool_heartbeat, llm_usage to knownTypes so a mid-stream reconnect/replay
  still surfaces the working state
- agent store: track streamingSessionId, preserved across switchSession so
  the sidebar spinner persists when navigating away from a running session
- Agent.tsx: handle attempt.created/started (re-affirm streaming on reconnect
  and page reload), keep streaming alive on tool_heartbeat, add a persistent
  pulse bar, reword "Thinking…" to "Agent is working…"
- Layout: replace the dead-code (status === "failed") dot with a spinning
  Loader2 for the actively streaming session
- index.css: pulse-slide keyframe for the running bar

Co-authored-by: ai7eam-dev <ai.7eam@gmail.com>
 2026-06-01 22:39:57 +08:00
1592 changed files with 249397 additions and 10524 deletions
Expand all files Collapse all files

No files matched your search

.dockerignore
+4 -1
View File
@@ -43,8 +43,11 @@ build/
dist/
# Project docs
# NOTE: `*` never crosses `/` in Docker ignore matching, so `/*.md` (anchored)
# excludes only ROOT-level markdown. Nested skill docs (agent/src/skills/**/*.md)
# and agent/SKILL.md are intentionally NOT excluded — they ship in the image.
CLAUDE.md
CONTRIBUTING.md
*.md
/*.md
!README.md
!LICENSE
.github/ISSUE_TEMPLATE/config.yml
+4 -1
View File
@@ -1,7 +1,10 @@
blank_issues_enabled: true
contact_links:
- name: Discord Community
url: https://discord.gg/2vDYc2w5
url: https://discord.gg/6TdQnT5xcF
about: Join the HKUDS community Discord
- name: GitHub Discussions
url: https://github.com/HKUDS/Vibe-Trading/discussions
about: Ask questions, share strategies, get help
- name: Documentation
url: https://github.com/HKUDS/Vibe-Trading#-quick-started
.github/dependabot.yml
+107
View File
@@ -0,0 +1,107 @@
version: 2
updates:
# Root Python project (pyproject.toml).
- package-ecosystem: "pip"
directory: "/"
schedule:
interval: "monthly"
open-pull-requests-limit: 2
groups:
# Minor/patch bumps are low-risk and reviewed as a single batch; majors
# are excluded so they still arrive individually and get real scrutiny
# (a batched PR mixing 10 patch bumps with 1 breaking major is worse
# than either alone — you can't reject just the major).
pip-minor-patch:
applies-to: version-updates
update-types: ["minor", "patch"]
ignore:
# pandas 3.x has real breaking changes (Copy-on-Write always-on, removed
# deprecated APIs) we haven't vetted; the <3.0.0 ceiling is deliberate,
# not an oversight. Revisit as a dedicated migration, not a dep bump.
- dependency-name: "pandas"
update-types: ["version-update:semver-major"]
# caio is capped by our own dependency tree, not by choice: aiofile
# 3.11.1 (latest) requires caio~=0.9.0, so any bump to caio 0.10.x makes
# `pip install` unresolvable. Dependabot kept proposing it and the group
# PR failed CI with ResolutionImpossible. Drop this ignore once aiofile
# ships a release that accepts caio 0.10.
- dependency-name: "caio"
# websockets is likewise capped by our tree: langgraph-sdk 0.4.2 requires
# websockets<16, so a bump to 16.x is unresolvable. Drop this once
# langgraph-sdk widens its range.
- dependency-name: "websockets"
# ccxt does not use ranges — it exact-pins (``==``) its entire transitive
# closure, and every one of the 23 packages below is currently held by it
# at precisely the version in our lock. None of them can move until ccxt
# itself releases, so a Dependabot PR touching any of them is unmergeable
# by construction: PR #882 proposed five of them and died with
# ResolutionImpossible. Verified by re-running the full resolver
# (`pip-compile --upgrade`, Python 3.11 on Linux) — it reproduces these
# exact versions, so this is the resolver's ceiling and not a stale lock.
#
# This suppresses only the automated PRs. Dependabot *alerts* come from
# the dependency graph and are unaffected by this file, so a CVE in
# cryptography/urllib3/requests/certifi still surfaces in the Security
# tab — the remedy there is to bump ccxt, which carries the whole set.
# Drop an entry here if ccxt ever loosens that pin.
- dependency-name: "aiohappyeyeballs"
- dependency-name: "aiohttp"
- dependency-name: "aiohttp-fast-zlib"
- dependency-name: "aiosignal"
- dependency-name: "attrs"
- dependency-name: "certifi"
- dependency-name: "cffi"
- dependency-name: "charset-normalizer"
- dependency-name: "coincurve"
- dependency-name: "cryptography"
- dependency-name: "frozenlist"
- dependency-name: "idna"
- dependency-name: "multidict"
- dependency-name: "orjson"
- dependency-name: "propcache"
- dependency-name: "pycparser"
- dependency-name: "requests"
- dependency-name: "setuptools"
- dependency-name: "typing-extensions"
- dependency-name: "urllib3"
- dependency-name: "uvloop"
- dependency-name: "yarl"
- dependency-name: "zlib-ng"
# pydantic-core is exact-pinned by pydantic (``pydantic-core==2.46.4``),
# so it only ever moves as part of a pydantic release. It was the sixth
# and last package in #882.
- dependency-name: "pydantic-core"
# Frontend npm dependencies.
- package-ecosystem: "npm"
directory: "/frontend"
schedule:
interval: "monthly"
open-pull-requests-limit: 2
groups:
npm-minor-patch:
applies-to: version-updates
update-types: ["minor", "patch"]
ignore:
# @vitejs/plugin-react 6.x peers vite@^8 and we are on vite 6.x, so it can
# only sit red. Same rationale as the pandas pin above: revisit as a
# dedicated migration, not a dependency bump.
#
# The jsdom and @testing-library/jest-dom majors used to be ignored here
# too, both blocked on Node >=22 while CI and Docker ran Node 20. That
# wall is gone — Node 20 reached EOL on 2026-04-30 and the toolchain moved
# to Node 22 — so both were taken and the ignores removed rather than left
# with a rationale that no longer holds.
- dependency-name: "@vitejs/plugin-react"
update-types: ["version-update:semver-major"]
# GitHub Actions workflow pins.
- package-ecosystem: "github-actions"
directory: "/"
schedule:
interval: "monthly"
open-pull-requests-limit: 2
groups:
actions-minor-patch:
applies-to: version-updates
update-types: ["minor", "patch"]
.github/workflows/test.yml
+71 -6
View File
@@ -12,15 +12,37 @@ jobs:
timeout-minutes: 10
steps:
- uses: actions/checkout@v4
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
- uses: actions/setup-python@v5
- uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
with:
python-version: "3.11"
cache: pip
- name: Verify hash-locked dependencies
run: >-
pip install
--dry-run
--ignore-installed
--require-hashes
-r requirements-lock.txt
- name: Install dependencies
run: pip install -e ".[dev]"
# `openbb` is included so agent/tests/test_openbb_bridge actually runs:
# those tests importorskip on openbb_ai, so without the extra the whole
# OpenBB Workspace bridge would report green while never being exercised.
run: pip install -e ".[dev,openbb]"
- name: Verify optional extras under test are importable
# Fails loudly if the openbb extra stops resolving, instead of letting
# the bridge suite silently degrade to skips.
run: python -c "import openbb_ai, importlib.metadata as m; print('openbb-ai', m.version('openbb-ai'))"
- name: Repository safety gates
run: bash tools/ci_grep_gates.sh
- name: Test environment variable gate
run: pytest tools/test_ci_env_var_gate.py -q
- name: Syntax check
run: |
@@ -36,11 +58,11 @@ jobs:
# test_e2e_harness_v2.py is gated by VIBE_TRADING_RUN_LIVE_E2E
# (real-LLM only); list --ignore here so CI's intent is explicit and
# the suite never silently switches state on env presence.
run: pytest --ignore=agent/tests/e2e_backtest --ignore=agent/tests/test_e2e_harness_v2.py --tb=short -q
run: pytest --ignore=agent/tests/e2e_backtest --ignore=agent/tests/test_e2e_harness_v2.py --cov=agent --cov-report=term-missing --cov-report=xml --tb=short -q
- uses: actions/setup-node@v4
- uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0
with:
node-version: "20"
node-version: "22"
cache: npm
cache-dependency-path: frontend/package-lock.json
@@ -49,3 +71,46 @@ jobs:
cd frontend
npm ci
npm run build
- name: Frontend tests
run: |
cd frontend
npx vitest run --reporter=verbose
windows-background:
name: ${{ matrix.name }} / Python 3.14 background lifecycle
runs-on: ${{ matrix.runner }}
timeout-minutes: 10
strategy:
fail-fast: false
matrix:
include:
- name: Windows 11 ARM64
runner: windows-11-arm
architecture: arm64
- name: Windows Server 2025 x64
runner: windows-2025
architecture: x64
steps:
# actions/checkout v7.0.1
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1
# actions/setup-python v7.0.0
- uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97
with:
python-version: "3.14"
architecture: ${{ matrix.architecture }}
cache: pip
- name: Install focused test dependencies
run: python -m pip install pytest pydantic pyyaml python-dotenv
- name: Run Windows background safety regressions
env:
PYTHONPATH: agent
run: >-
python -m pytest
agent/tests/test_background_tools.py
agent/tests/test_bash_tool.py
-q
.github/workflows/wiki-deploy.yml
+9 -3
View File
@@ -21,12 +21,18 @@ jobs:
deployments: write
steps:
- name: Checkout
uses: actions/checkout@v4
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
- name: Deploy wiki
uses: cloudflare/wrangler-action@v3
uses: cloudflare/wrangler-action@ebbaa1584979971c8614a24965b4405ff95890e0 # v4.0.0
with:
apiToken: ${{ secrets.CLOUDFLARE_API_TOKEN }}
accountId: ${{ secrets.CLOUDFLARE_ACCOUNT_ID }}
command: pages deploy wiki --project-name=vibetrading-wiki --branch=main
# Deploy from inside wiki/ (config mode) so wrangler reads
# wiki/wrangler.toml and compiles wiki/functions/ as Pages Functions.
# Deploying the dir positionally (`pages deploy wiki`) uploads the
# functions as static files and never compiles them.
workingDirectory: wiki
wranglerVersion: "4.98.0"
command: pages deploy --project-name=vibetrading-wiki --branch=main
gitHubToken: ${{ secrets.GITHUB_TOKEN }}
.github/workflows/wiki.yml
+11 -5
View File
@@ -18,13 +18,13 @@ jobs:
timeout-minutes: 5
steps:
- uses: actions/checkout@v4
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
- uses: actions/setup-node@v4
- uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0
with:
node-version: "20"
node-version: "22"
- uses: actions/setup-python@v5
- uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
with:
python-version: "3.11"
@@ -38,6 +38,9 @@ jobs:
- name: Check locale JSON
run: python -m json.tool wiki/locales/en.json >/dev/null
- name: Check social card metadata
run: python wiki/scripts/check_social_cards.py
- name: Check static site files
run: |
test -f wiki/index.html
@@ -49,4 +52,7 @@ jobs:
test -f wiki/_redirects
test -f wiki/_headers
test -f wiki/assets/icon.png
! grep -R "localhost\\|nanobot" wiki --exclude=README.md
test -f wiki/assets/alpha-library-og.png
blocked="localhost\\|nano"
blocked="${blocked}bot"
! grep -R "$blocked" wiki --exclude=README.md
.gitignore
+26
View File
@@ -4,6 +4,16 @@ __pycache__/
.pytest_cache/
*.pyc
# Coverage reports
.coverage
coverage.xml
# Node dependencies (root level)
node_modules/
# Backtest output artifacts
results/
.env
agent/.env
@@ -17,6 +27,19 @@ agent/runs/
agent/.ui_runtime/
agent/.tasks/
agent/uploads/
# Same runtime artifacts, but written to the repo root when the CLI or server
# is started from there — sessions.db holds real session transcripts and
# live/ holds broker audit and kill-switch state.
/sessions.db
/sessions.db-shm
/sessions.db-wal
/sessions/
/runs/
/shadow_runs/
/swarm/runs/
/live/
/uploads/
/imports/
agent/tests/*_result.json
.ruff_cache/
frontend/dist/
@@ -26,6 +49,7 @@ wiki/dist/
wiki/.astro/
wiki/.vite/
wiki/.wrangler/
.wrangler/
wiki/.cache/
.vibe-dev/
.vibe-record/
@@ -70,6 +94,7 @@ agent/tests/test_e2e_*.py
# Local working notes
NOTES.md
.workbuddy/
# Agent tool aliases (auto-generated mirror of CLAUDE.md)
AGENTS.md
@@ -79,3 +104,4 @@ AGENTS.md
# Alpha Zoo wiki artifacts (generated, not source)
wiki/alpha-library/manifest.json
wiki/alpha-library/content/
bench_report.json
AGENT_CONTRIBUTOR_GUIDE.md
+139
View File
@@ -0,0 +1,139 @@
# Agent Contributor Guide
This guide is for AI-assisted and automation-assisted contributors working on
Vibe-Trading. It does not replace `CONTRIBUTING.md`, `SECURITY.md`, or the pull
request template. It makes the repo's agent-facing safety and verification
expectations explicit.
## Repository Shape
- Backend and package code live under `agent/`.
- Frontend code lives under `frontend/`.
- Public wiki content lives under `wiki/` and has separate GitHub Actions checks.
- MCP entry point: `vibe-trading-mcp` / `agent/mcp_server.py`.
- CLI entry point: `vibe-trading` / `agent/cli/`.
- Broker connector, mandate, order gate, halt, and audit-ledger logic are safety
critical even when a change appears small.
- Community commits must include the DCO `Signed-off-by:` trailer; see
`CONTRIBUTING.md`.
Do not copy local `.env` files, token caches, broker exports, run artifacts,
private notebooks, generated reports, or local agent memory into this repository
unless they are explicitly sanitized fixtures.
## Safe Local Checks
These commands are normally safe for local validation:
```bash
git status --short --branch
git diff --check
python -m compileall -q agent/cli
python -m py_compile agent/api_server.py agent/mcp_server.py
pytest --ignore=agent/tests/e2e_backtest --ignore=agent/tests/test_e2e_harness_v2.py --tb=short -q
pytest agent/tests/test_sdk_order_gate.py agent/tests/test_mandate_enforcement.py -q
cd frontend && npm ci && npm run build
```
Use the narrowest test command that matches the changed files when a full suite
is too expensive, and state what was not run.
## High-Risk Surfaces
Get explicit maintainer or operator approval before running commands that:
- place, cancel, approve, flatten, or otherwise affect broker orders,
- authorize broker, OAuth, MCP, exchange, payment, wallet, or cloud accounts,
- write real credentials into `agent/.env`, `~/.vibe-trading/`, token caches, or
external services,
- start externally reachable API, MCP, SSE, webhook, or dashboard servers,
- deploy the wiki, publish packages, trigger releases, or change CI secrets,
- rewrite history, force-push shared branches, delete backups, or remove
persistent run or memory data.
Never run live trading, payment, wallet, contract, or broker-write flows as part
of routine PR validation.
## Targeted Test Hints
For general Python changes:
```bash
pytest --ignore=agent/tests/e2e_backtest --ignore=agent/tests/test_e2e_harness_v2.py --tb=short -q
```
For live/order safety changes:
```bash
pytest agent/tests/test_sdk_order_gate.py \
agent/tests/test_mandate_enforcement.py \
agent/tests/test_killswitch_blocks_orders.py \
agent/tests/test_readonly_default.py -q
```
For factor-zoo changes:
```bash
pytest agent/tests/factors/test_alpha_purity.py agent/tests/factors/test_lookahead.py -q
```
For frontend changes:
```bash
cd frontend && npm ci && npm run build
```
## Documentation Rules
- Update `README.md` for user-facing CLI, Web, MCP, connector, provider, or
safety-boundary changes.
- Update `CONTRIBUTING.md` when contributor workflow, DCO expectations, style
rules, or test expectations change.
- Update `SECURITY.md` only for vulnerability reporting policy changes.
- Keep wiki edits under `wiki/`; avoid mixing generated assets or local drafts
into public docs.
- If a change affects live/order behavior, document the safety boundary and the
rollback or halt path.
## Security Rules
- Do not commit API keys, access tokens, OAuth caches, session cookies, private
keys, seed phrases, broker credentials, `.env` files with real values, or
private trading exports.
- Treat secrets as access, not text. If one appears, redact it, stop repeating
it, and recommend rotation.
- API or Web deployments beyond loopback must use `API_AUTH_KEY`.
- External MCP servers are operator-trust surfaces. Do not allow caller-provided
MCP command, URL, environment, or allowlist injection unless the code path
explicitly documents and tests that opt-in.
- Broker connector writes must remain mandate-gated, kill-switch-aware,
fail-closed, and audit-logged.
- Prefer sanitized fixtures over real financial data.
## Pull Request Expectations
Every PR should explain:
- the goal,
- the affected areas,
- what is deliberately out of scope,
- the test plan,
- any live, broker, MCP, network, secret, or deployment risk,
- rollback or close path.
Every community commit must be signed:
```bash
git commit -s -m "docs(agent): add contributor guide"
```
Do not add AI-assistant attribution trailers; follow `CONTRIBUTING.md`.
## Rollback
- Documentation-only changes can usually be reverted with `git revert <sha>`.
- Code changes touching order gates, mandate enforcement, OAuth, providers, MCP,
settings, or external network behavior need a targeted regression test before
merge and a clear revert path in the PR.
- If a safety regression is found after merge, prefer a small revert or
fail-closed hotfix over broad refactors.
CHANGELOG.md
+426 -6
View File
@@ -5,16 +5,436 @@ This project adheres to [Semantic Versioning](https://semver.org/).
## [Unreleased]
### UI
- Redesigned the interactive CLI startup and run-state visuals with a large
figlet-style banner plus a Claude Code-style activity rail using `●`, `│`,
and `·` glyphs. Non-TTY prompt runs now fall back to plain text output.
### Added
- **Memory Tier 2: Structural Organization** — four independently-gated modules for memory lifecycle enhancement:
- H-MEM hierarchical directory routing (`VT_MEMORY_HIERARCHY`)
- A-MEM semantic linking via BM25 (`VT_MEMORY_LINKS`)
- Auto-compression pipeline with TF-IDF key-sentence extraction (`VT_MEMORY_COMPRESSION`)
- FTS5 full-text search index with CJK bigram tokenization (`VT_MEMORY_FTS_INDEX`)
- **`VT_MEMORY=off|on|full` preset** — one-line business-friendly configuration replacing 7 individual flags
- `off`: no lifecycle management (default)
- `on`: quality scoring + auto-decay + garbage collection (Tier 1)
- `full`: all features including hierarchy, links, compression, and FTS (Tier 1 + Tier 2)
- Individual `VT_MEMORY_*` flags still work as explicit overrides
- Dependency-conflict warnings when `VT_MEMORY_COMPRESSION=true` without `VT_MEMORY_GC`
- Lazy FTS5 auto-rebuild on first search when index is empty
- 9 end-to-end integration tests + 23 benchmark tests against real production corpus
## [0.1.13] — 2026-08-01
### Added
### Changed
- **Read-only sentiment tool** (#939): local lexicon scoring for arbitrary text
plus the crypto Fear & Greed Index from Alternative.me's free, no-auth API.
### Fixed
- **Agent identity ordering and evidence grounding** (#887/#886): market-sensitive
consumers now require a canonical symbol/venue locked before their assistant
tool-call batch starts, so a resolver and its consumer cannot race in one
parallel batch. Silent exchange-suffix rewrites are rejected, listed-company
evidence cannot be overwritten by model memory, and ambiguous/not-found/
conflicting/invalidated identities are persisted as first-class states.
- **Final numeric consistency gate** (#886): full untruncated OHLC tool results
are recorded as structured evidence with symbol, venue, currency, actual data
source, and explicit conversion status. Contradictory final prices are rejected
before streaming; derived levels require a visible, arithmetically correct
formula anchored to observed inputs, and repeated bad drafts fail closed to
verified observations.
Tool envelopes with `ok: false`, `success: false`, or failed status are no
longer recorded as successful calls.
- **Robinhood Agentic MCP response serialization** (#922): nested FastMCP
dataclasses containing `date`/`datetime` values serialize to JSON instead of
surfacing the misleading `Circular reference detected` error. A representative
Robinhood portfolio/account shape now has an end-to-end regression test.
## [0.1.12] — 2026-07-22
### Added
- **User swarm-presets directory**: preset YAMLs dropped into
`~/.vibe-trading/swarm/presets/` are discovered alongside the bundled
roster (same-name files override it — the same rule as user skills) and
survive `pip install -U`. `list_presets()` entries now carry a
`source: "user" | "bundled"` field; explicitly named user presets run
through `run_swarm(preset_name=...)`, while keyword auto-routing stays
limited to the curated table. Preset names are validated to a single path
segment before any filesystem lookup.
- **Security hardening**: all 10 findings from the 2026-07-10 external audit
closed (#476, tracking discussion #468) — Docker multi-stage rebuild with
digest-pinned base images, AST-hardened backtest sandbox (blocks
network/subprocess/eval/os.environ/unsafe-open reachable from generated
code, including inside nested function bodies), short-lived single-use SSE
auth tickets replacing a long-lived key in the URL/logs, hardened Compose
(`read_only`, dropped capabilities, `no-new-privileges`, resource limits),
auth + rate limiting on `/correlation`, security headers (CSP
Report-Only, `X-Content-Type-Options`, `Permissions-Policy`), `/live` +
`/ready` health split, hash-locked dependencies wired into the Docker
build, GitHub Actions pinned by commit SHA, and an HMAC-authenticated
factor cache.
- Opt-in **TAP mode** for Alpaca (#377, thanks @0xZKnw) — routes all broker
egress through a self-hosted TAP proxy so the agent process never holds
the raw API key, with writes blocking on human approval.
- Realized portfolio turnover (`avg_turnover` / `total_turnover`) surfaced
in backtest metrics for every optimizer (#478, thanks @Robin1987China).
- **Frazzini-Pedersen betting-against-beta** academic factor (#480, thanks
@YogeshModi24) — Alpha Zoo: 460 → **461**.
- **MetaTrader 5 connector + data source** (Exness-style MT5 brokers,
Windows-only `pip install "vibe-trading-ai[mt5]"`). Broker connectors:
11 → **12** — full read surface plus order placement against a locally
running terminal, with a bidirectional identity guard (paper profile ⇔
demo `trade_mode`, login pinned, contest rejected), connector-level
`max_order_volume`/`max_order_notional_usd` guards on demo AND live, and
hedging-safe position close by ticket. The live mandate gate gains
`forex`/`cfd` instrument vocabulary (schema v1 unchanged) and a lot-aware
`quantity_notional_usd` sizing hook so USD caps bind on lot-sized orders
(0.1 lot EURUSD ≈ $10,800, never 0.1 × quote). Market-data sources:
20 → **21** — the `mt5` loader heads the forex fallback chain (broker-exact
symbols with Exness suffix discovery, 1m–1D bars), `get_market_data` learns
forex/metal symbol routing (`EUR/USD`, `XAUUSD.FX` previously fell through
to tushare), and akshare's forex path accepts the canonical slash form so
degradation off-Windows keeps working.
- **Strategy Development Manager** skill (#457, thanks @shadowinlife, closes
#455) — `sdm_register` / `sdm_status` / `sdm_decay_scan` turn academic
papers and broker research into registered factors/strategies with a
persistent SQLite artifact store (`UNIQUE(name, universe)`) and automated
IC/Sharpe decay monitoring driving an active → monitoring → decayed →
disabled lifecycle. Pluggable OCR for `read_document` (local RapidOCR by
default; cloud Qwen-VL is explicit opt-in only via
`VIBE_TRADING_OCR_ENGINE=qwen-vl`, never auto-selected). Skills: 86 → **87**.
- **Requesty** as an OpenAI-compatible LLM gateway provider (#474, thanks
@Thibaultjaigu) — same `provider/model` naming and capability shape as
OpenRouter, wired through CLI onboarding, provider menu, and Settings.
- Binance USD-M perpetual routing, slice 1 of #462 (#470, thanks @honginp) —
explicit `BTC-USDT-PERP` symbol contract with execution/mark price
separation, fail-closed when the two aren't timestamp-synchronized.
- **Correlation regime timeline** (#756, thanks @ebujinovch, closes #719) — a
new additive `GET /correlation/regime` endpoint plus an opt-in "Regime
timeline" strip on the Correlation tab: rolling pairwise correlations reduce
to an edge-density series, causally smoothed, and run through a two-threshold
hysteresis state machine that marks FUSED episodes ("when did the market fuse
into one bloc?"). Descriptive risk context, not a trading signal; shares
`/correlation`'s auth + rate-limit budget. Backed by the **correlation-regime
skill** (#557, thanks @ebujinovch).
- **Three new LLM providers** — SiliconFlow CN + Global (#565, thanks @UNHNQ),
iFlytek Spark (#537, thanks @FenjuFu), and a **native Anthropic Messages API**
adapter (#695, thanks @jelech; `pip install "vibe-trading-ai[anthropic]"`).
MiniMax now exposes its regional API endpoints (#731, thanks @octo-patch).
- **Historical USD-M funding settlements** for Binance perpetuals (#716, thanks
@honginp); maintenance brackets are supplied as a validated, versioned
artifact rather than a live authenticated fetch (#757, thanks @honginp), so a
plain `-PERP` backtest stays zero-credential.
- **Pluggable OCR** for `read_document` with optional LLM-vision extraction and
a configurable text-density threshold (#548, thanks @shadowinlife) — local
RapidOCR by default; cloud engines are explicit opt-in, never auto-selected.
- New academic factor `academic_corr_rewire` (#705, thanks @ebujinovch) and the
fundamental zoo wired into the `_VALID_ZOOS` whitelist (#707, thanks
@sambazhu). Binance crypto fallback loader (#643) and bounded OKX history
fetches with rate-limit handling (#644, thanks @tyj147454413-cmd).
- QVeris premium-track hardening — session budget applied to backtest data
calls (#685) and atomic credit accounting (#686, thanks @xkam7ar).
### Changed
- Correlation tab accepts bare tickers like `AAPL,SPY` and walks the full
loader fallback chain instead of failing with `Fetched: []` (#472, thanks
@yxhuang, closes #471).
- `local` loader honors the requested interval via OHLCV resampling instead
of silently returning daily bars (#467, thanks @Shizoqua).
- Provider credentials are resolved through one centralized path, fixing a
gateway misroute (#563, thanks @shadowinlife, closes #549/#553). When no
`*_BASE_URL` is set, the backend now falls back to the provider catalog's
canonical `default_base_url` (the same default Web Settings already used), so
a CLI / manual-`.env` user reaches the right endpoint instead of defaulting
to `api.openai.com`.
- Signal alignment is vectorized for an ~80× speedup on wide panels (#698,
thanks @shadowinlife); swarm workers cache MCP tool-discovery specs to avoid
redundant RPC round-trips (#704, thanks @shadowinlife).
### Fixed
- Explicit `source: local` backtests now route US/HK equities to the
global-equity engine instead of the crypto default, and explicit benchmarks
are fetched through the configured source's loader — `local` fails closed
(no yfinance fallback) so offline runs stay offline (#550).
- Loading `.env` now invalidates an `EnvConfig` singleton cached during early
CLI imports, so the welcome panel, `/settings`, and dotenv diagnostic report
the configured provider and model consistently (#541).
- FastMCP transport imports work across both module layouts (#469, thanks
@roberttidball).
- Portfolio optimizers no longer include the decision bar's close-to-close
return in weights executed at that bar's open (#487, thanks @YZY0108).
- Backtest turnover metrics now use actual filled and rounded position sizes;
targets rejected by market rules no longer inflate reported turnover.
- End-of-backtest liquidations now apply exit slippage and include their
commission in the final reported equity.
- Open-price rebalances no longer use the decision bar's close for sizing or
depend on whether a replacement symbol sorts before the position it closes.
- Preflight (`vibe-trading run`) no longer resolves provider/model against a
stale `EnvConfig` snapshot cached before dotenv loads (#479, thanks
@ananaymital, closes #477).
- Switching providers no longer leaves a stale `OPENAI_BASE_URL` from a
previous configuration silently overriding the newly-resolved endpoint
(#484, thanks @Bortlesboat, closes #482).
- **Strict-JSON / finite-number hardening** across the backtest + tools stack
(thanks @santhreal): risk ratios stay finite when equity crosses zero mid-path
(#765) or annualizes an explosive path (#739/#740); scalar backtest metrics
(#766), factor IC std (#767), and pattern trend-slope (#764) emit strict
RFC-8259 JSON (`null`, never bare `NaN`/`Infinity`); Black-Scholes helpers
treat non-positive spot/strike as intrinsic (#744).
- **Loader / data correctness** — yahoo `1m` stays minute bars instead of being
uppercased to monthly (#761, @santhreal); the composite engine falls back to
the first available sub-engine for unknown symbols (#734, thanks @Marnie0415);
mootdx history that doesn't reach the requested start is rejected (#692, thanks
@xkam7ar).
- **Session / journal robustness** — one corrupt `session.json` (#762) or a
schema-bad `messages.jsonl` line (#763) no longer aborts listing/reading;
Excel float-stringified A-share codes (#770), unicode-dash PDF page ranges
(#769), and `export KEY=` dotenv lines (#768) all parse correctly (@santhreal).
- **Native `zai` provider on glm-5.1** (#758) — endpoints that stream zero
chunks fall back to a non-streaming invoke instead of raising, and an HTML
error page surfaces an actionable base-URL hint.
- Partial market-data results are completed through the loader fallback chain
instead of silently shrinking the universe (#689, closes #681).
- Cancellation is honored before the first AgentLoop iteration (#641, thanks
@xkam7ar, closes #638); streaming output no longer triggers an `insertBefore`
DOM race in the frontend (#717, thanks @Marnie0415); codex stream HTTP
failures are classified for correct retry (#663, thanks @tyj147454413-cmd).
- Robinhood connector `account_number` wiring and remote-MCP display shape
(#726, thanks @nareshkps).
- Broad reliability fixes across packaging, web, scheduler, swarm, and CLI
(#584, thanks @xkam7ar).
## [0.1.11] — 2026-07-11
### Added
- **Indian equity (NSE/BSE) as a first-class market** (#305, thanks
@muku314115). A dedicated `IndiaEquityEngine` — T+1 delivery, no overnight
shorts (opt-in intraday), configurable circuit bands, 1-share lots, and a
config-driven STT / stamp-duty / exchange / SEBI / GST cost stack — with
`.NS`/`.BO` symbol routing (`yahoo → yfinance → india_broker → local`), an
opt-in read-only Shoonya/Dhan `india_broker` data bridge, and 255
alpha101/qlib158 factors opted into the new `equity_in` universe. Backtest
engines: 7 → **8**; market-data sources: 19 → **20**.
- **Fundamental factor layer, Phase 1.** PIT-safe SEC fundamentals flow into
dense daily `fund:*` factor panels — filed-date anchoring, first-filed
restatement policy, true-quarter `(start, end)` frame selection with Q4
synthesis (so YTD/annual frames can't contaminate TTM), and rolling TTM —
plus a `get_fundamentals` tool and 4 quality/value factors in a new
`fundamental` zoo family. Alpha Zoo: 456 → **460** across **5** families.
- **Research Autopilot Phase 3 — the loop closes** (#267, thanks
@Robin1987China). `scaffold_signal_engine` writes a contract-correct signal
engine from a hypothesis and `link_autopilot_backtest` runs it, completing
hypothesis → signal-engine → backtest end to end.
- **4 canonical academic alphas** (#277, thanks @Robin1987China) — Jegadeesh
short-term reversal, George–Hwang 52-week high, Amihud illiquidity, and
Harvey–Siddique co-skewness join the academic family (452 → 456), with a
**central OHLC-invariant guard** at the runner fetch boundary dropping
malformed bars from every loader (#274, thanks @Shizoqua).
- **Scheduled research runs end to end** (#278 closing #254, thanks
@mvanhorn). A default-off background executor
(`VIBE_TRADING_ENABLE_SCHEDULER`) fires due interval/cron jobs through the
session runtime, on top of a crash-safe atomic job store, 3 auth-gated
`/scheduled-runs` REST routes, a Reports library, and post-backtest
attribution. Route test coverage followed in #452 (thanks @Robin1987China).
- **IM channel runtime — research delivery over 16 adapters.** The agent
session runtime now attaches to 16 built-in message adapters (WebSocket,
Telegram, Slack, Discord, Matrix, WhatsApp, Signal, QQ/NapCat,
WeChat/WeCom, Feishu, DingTalk, email, MS Teams, MoChat), dependency-gated
with install hints, configurable via `AgentConfig.channels`, and surfaced in
REST (`/channels/*`), CLI (`vibe-trading channels ...`), and Web Settings —
in all 5 UI locales.
- **QVeris optional premium data track.** The 19 free sources stay the
default; an explicit-only QVeris mode (Settings → QVeris or
`vibe-trading data mode paid`) unlocks 63+ providers behind 3 key-gated
tools (`qveris_search` / `qveris_inspect` / `qveris_execute`) with
preview-by-default and a session budget gate. Never enters auto-fallback.
- **Trading 212 read-only connector** (#321, thanks @mvanhorn) — 11 brokers
total. Trading 212 exposes no runtime paper/live discriminator, so the
connector is fully read-only: `place_order`/`cancel_order` hard-refuse
every order, paper included. Live order guards also gained an opt-in,
broker-agnostic `PreTradeAdvisoryInterface` that records advisory reviews
without bypassing the mandate gate.
- **Turnover-aware portfolio optimizer** (#466, thanks @Robin1987China) —
fifth optimizer: mean-variance utility with an L1 penalty on weight changes
versus the previous rebalance (SLSQP, long-only simplex), so the portfolio
only trades when expected improvement outweighs churn. Optimizers: 4 → **5**.
- **`analyze_image` vision tool** (#464, thanks @fei-moss) — send a local
chart / K-line screenshot / app screenshot to the session model as a
multimodal message and get a semantic read (complements `read_document`'s
OCR). Path-validated against the allowed file roots; requires a
vision-capable model. Tools: 71 → **72** free-mode (75 with QVeris).
- **Value-investing toolkit** (thanks @sambazhu): financial-rigor +
report-audit tools, 4 skills, and a `value_investing_committee` swarm
preset. Swarm presets: 29 → **30**.
- **CN-friendly search fallbacks** — `web_search` gains China-reachable
backends in the ordered no-key engine chain.
- **Provider roster additions**: Kimi for Coding as a distinct provider
(#435, thanks @yxhuang), opencode provider mappings (#444, thanks
@imsankz), Codex OAuth default model bumped to `openai-codex/gpt-5.4`
(#446, thanks @morluto).
- **SKILL.md manifest guard test** (#461, thanks @asahikiko) — the packaged
skill's capability counts (skills / presets / zoo / sources / MCP tools /
engines) are now asserted against source, so distribution paperwork can't
silently drift again.
### Changed
- **`api_server.py` modularization completed** — 1,103 → 371 lines (#424
closing #331, thanks @shadowinlife), after route slices for channels,
settings, and the remaining route groups, plus a shared compat layer with
session-service writeback fixes.
- **Centralized environment variable management** (#440 closing #438, thanks
@shadowinlife) — every env var flows through a single Pydantic `EnvConfig`
schema, enforced by an AST-based CI gate that rejects raw
`os.getenv`/`os.environ` outside `agent/src/config/`.
- **Factor engine acceleration** — hot rolling operators use
`bottleneck`/NumPy fast paths, and alpha-bench parallelism stops resending
large panel payloads to workers.
- **Robinhood Agentic MCP refresh** — current MCP tool names across generic
reads, live-runner plumbing, default read-only seeds, and mandate-gate
tests; interactive OAuth holds the handshake open through multi-minute
broker sign-ins (`VIBE_LIVE_AUTHORIZE_TIMEOUT_SECONDS`).
- **Loader `fetch()` signatures** now match the loader protocol across
OKX / Tushare / yfinance (#437, thanks @shadowinlife).
- **Timezone-aware UTC timestamps** across session, goal, channel, and API
paths (#397, thanks @mustafakamal88).
- **Inbound IM media now lands under `~/.vibe-trading/uploads/<channel>/`**
(fixes #465, thanks @fei-moss for the report) — inside the default allowed
file roots, so the agent can read what users send over IM channels with
zero configuration. The Matrix E2E store moves to the runtime dir (legacy
path honored) so credentials never enter the readable root.
### Fixed
- **Docker/server startup crash** when FastAPI route iteration hit an
included-router entry without `path` (#450, thanks @Penn-Live).
- **GLM thinking models on the zhipu provider** no longer lose their
reasoning stream (#458).
- **Trading mandate UX guards** — a second-confirmation dialog before
committing a real mandate, unified error toasts, and clarified inputs
(#453, thanks @wison1717-maker).
- **`trading_place_order` treats zero quantity/notional as unset** instead of
passing a zero-size order to the broker (#417, thanks @irfanallana-oss).
- **Longbridge Decimal values serialize as floats** across quotes, bars,
balances, positions, orders, and executions (#459, thanks @fanfpy).
- **NapCat private messages now trigger pairing codes** (#463, thanks
@fei-moss).
- **Backtest validation artifacts**: `validation.json` no longer requires a
pre-existing artifacts dir (#429, thanks @isaveall), and nested
`NaN`/`Infinity` values are normalized before writing, so strict JSON
parsers don't choke.
- **CLI**: `resume` preserves the first user message (#448 closing #447,
thanks @morluto); `--swarm-run` rejects extra tokens with a clear error
(#428, thanks @isaveall); the interactive CLI prints the session-id on
exit with a copy-paste resume hint.
- **Shadow Account**: extracted rules carry RSI / prior-return entry bounds
computed from PIT-safe context fetched through the loader registry, so
generated engines enter on real conditions (#302/#314/#316, thanks
@Robin1987China); tushare ETF/index/HK symbol routing fixed along the way.
- **Content-filter resilience** — event-driven and swarm runs skip individual
LLM content-moderation hits and warn when filter rates are high; Gemini
safety finish-reasons recognized.
- **IM channel reply timeout is configurable** (#413, thanks @dpersek).
- **Provider preflight no longer follows redirects** (#404 closing #402,
thanks @dpersek).
- **Windows baseline green** — `vibe-trading setup`/`dev` handle Windows
TypeScript builds, correct cwd, the Vite 5899 port, and child-process
shutdown; mootdx batch pulls let `KeyboardInterrupt`/`SystemExit`
propagate.
- **Security hardening** — loopback API CSRF protection (cross-site POSTs
can no longer drive side effects on the local API), SSRF guards on
interactive fetch paths, tightened API/Docker/frontend dev defaults, and
cleared frontend dependency/CSP alerts.
- **Reverted the IRR-AGL reliability/governance stack** (#405/#416) after it
broke session chats on day 1 (#433 — thanks @yxhuang for the precise
diagnosis); the evidence-bound research-pipeline direction continues in
reviewable slices on #442.
## [0.1.10] — 2026-06-19
Roll-up release; see the
[v0.1.10 release notes](https://github.com/HKUDS/Vibe-Trading/releases/tag/v0.1.10)
for the full narrative.
### Added
- **Global data layer** — market-data sources 10 → 18 (direct-API Eastmoney /
Sina / Stooq / Yahoo + key-gated Finnhub / Alpha Vantage / Tiingo / FMP)
with ban-risk-ordered fallback chains behind a shared throttled HTTP gate,
plus **18 read-only data tools** (fund flow, dragon-tiger, northbound,
margin, block trades, shareholder count, lockup, sector, research, news,
SEC filings, financial statements, options chains, institutional holdings,
screening, symbol search, FRED macro, iwencai) — all MCP-exposed.
- **10 broker SDK connectors** — Tiger / Longbridge / Alpaca / OKX / Binance /
Futu / Dhan / Shoonya join IBKR (local read-only) and Robinhood (Agentic
MCP); direct-SDK live orders pass the fail-closed bounded-autonomy gate;
brokers without a runtime paper/live discriminator are structurally capped
at paper + read-only.
- **Alpha Zoo `alpha compare`** across CLI, REST, Web UI, and agent tool.
- **Research Autopilot Phase 1** (`run_research_autopilot`,
`generate_backtest_config`) and the local Data Bridge loader.
- **Opt-in local data cache** (`VIBE_TRADING_DATA_CACHE`) for settled bars
under `~/.vibe-trading/cache/`.
- Per-run token usage (`llm_usage.json`) + progressive Run Detail charts;
CLI `resume <session-id>`.
### Changed
- **Provider reliability overhaul** — DeepSeek hang fixes, Kimi access,
streaming liveness watchdog, Gemini 3.x multi-turn tool-calling fix.
- Swarm workers pull market data through the loader layer; live swarm status
cards stream in the chat timeline.
- Baseline install slimmed (`pyharmonics`/`ta` behind
`vibe-trading-ai[harmonic]`).
### Fixed
- Community security-hardening wave (#241–#258): Settings write auth, shell
tools opt-in, LAN-access 403 clarity, Docker-to-host Ollama URL rewrite,
web_search multi-engine fallback, and more.
## [0.1.9] — 2026-06-01
### Added
- **Connector-first broker profiles (IBKR + Robinhood).** Trading access now
starts from a selectable connector profile rather than separate broker/live
entry points; `vibe-trading connector list/use/check/account/positions/orders/quote/history`
and the MCP `trading_*` tools share the selected profile, with paper/live as
a property under the connector. IBKR is usable immediately as a local
read-only TWS / IB Gateway profile; the official IBKR remote MCP path is
seeded as an OAuth `mcp.read` probe until stable read tool names ship.
Robinhood Agentic Trading is a bounded connector behind OAuth, a committed
mandate, an order guard, an audit ledger, and an instant halt switch.
- **Research Goal runtime.** Long-running, research-only goals with auditable
checklist criteria, budgets, and a `/goal` CLI command, plus REST + MCP
endpoints (`start_research_goal`, `get_research_goal`, `add_goal_evidence`,
`update_research_goal_status`) and a Web `GoalDrawer`.
- **Swarm `retry_run`.** Re-launch a failed/stale/cancelled run with the
original preset + variables; exposed as both `POST /swarm/runs/{id}/retry`
and an MCP `retry_run` tool (the `list_runs → retry` loop). 36 MCP tools now.
- **Operator-configured external MCP tools in swarm workers** (#142) and
**remote MCP transports** for the built-in agent.
- **`mootdx` A-share OHLCV loader** — native 通达信 TCP, no token, sits between
tushare and akshare in the fallback chain. CCXT loader now reads proxy env
for restricted networks (#126).
- **Hypothesis Registry CLI** — `list / show / invalidate`.
- **Strict alpha-bench mode** with a mandatory random control (#143).
### Changed
- **CLI split into the `agent/cli/` package** (from a 3216-LOC single file),
with a refreshed interactive terminal UI (figlet banner + activity rail) and
a single `cli/_version.py` version source.
- Swarm status reconciles from live task files on every read; `run_swarm`
sends MCP progress heartbeats, and the stale-run reaper uses per-run
thresholds (#132).
- Refreshed provider default model ids; bumped `langgraph` for CVE-2026-28277.
### Fixed
- **`--version` no longer drifts (#156).** The version derives from package
metadata, falling back to reading `pyproject.toml` directly — no hardcoded
constant left to forget on release.
- **Session running-status indicator** now survives reconnect / page reload /
sidebar navigation; **swarm DAG** blocks downstream tasks when an upstream
task fails (#145).
- **Robustness pass:** pre-flight validation for LLM-generated signal engines
with clean JSON errors (#149), graceful agent-loop exit at the iteration
budget instead of an output-less `failed` (#148), `flush + fsync` session
message writes that skip corrupted JSONL lines on read (#147), and IME Enter
handling in the Web composer (#146).
- **Full Report** link now always renders when a `runId` exists, even cross-browser
(#150); SSE idle timeout is configurable via `VIBE_TRADING_SSE_TIMEOUT` (#157);
cross-market correlation normalizes timestamps so crypto-vs-equity pairs align (#158).
## [0.1.8] — 2026-05-17
CONTRIBUTING.md
+13
View File
@@ -10,6 +10,11 @@ For general project setup (`pip install -e ".[dev]"`, dev servers,
`pytest --ignore=agent/tests/e2e_backtest`), see the README. For bug reports
and feature requests, use the GitHub issue templates.
For AI-assisted or automation-assisted contributions, also see
[`AGENT_CONTRIBUTOR_GUIDE.md`](AGENT_CONTRIBUTOR_GUIDE.md). It summarizes safe
local checks, higher-risk broker/MCP/credential surfaces, and the expected PR
risk notes for agent-authored changes.
## Developer Certificate of Origin (DCO)
Every commit in a community pull request MUST carry a `Signed-off-by:`
@@ -135,6 +140,14 @@ merging. Authors are strongly encouraged to self-check first.
## Code Style
- Format with `black`; lint with `ruff` (config in `pyproject.toml`).
- Install both tools with the development extra: `pip install -e ".[dev]"`.
- Run them on the Python files you changed, for example:
```bash
black --check agent/src/example.py agent/tests/test_example.py
ruff check agent/src/example.py agent/tests/test_example.py
```
The repository does not yet enforce a whole-tree Black or Ruff check, so
avoid mixing unrelated formatting cleanup into a focused pull request.
- Type-annotate all public function and method signatures.
- Google-style docstrings (`Args:` / `Returns:` / `Raises:`).
- Keep files under 400 lines where practical, 800 hard cap.
Dockerfile
+69 -21
View File
@@ -1,7 +1,8 @@
# ============================================================================
# Stage 1: Build frontend
# ============================================================================
FROM node:20-slim AS frontend-build
FROM node:22-slim@sha256:6c74791e557ce11fc957704f6d4fe134a7bc8d6f5ca4403205b2966bd488f6b3 AS frontend-build
# node:22-slim digest resolved 2026-07-28
WORKDIR /app/frontend
COPY frontend/package.json frontend/package-lock.json ./
@@ -10,50 +11,97 @@ COPY frontend/ ./
RUN npm run build
# ============================================================================
# Stage 2: Python runtime
# Stage 2: Python builder — compiles wheels + builds a self-contained venv.
# build-essential lives ONLY here; it never reaches the runtime image.
# ============================================================================
FROM python:3.11-slim AS runtime
FROM python:3.11-slim@sha256:e031123e3d85762b141ad1cbc56452ba69c6e722ebf2f042cc0dc86c47c0d8b3 AS builder
# python:3.11-slim digest resolved 2026-07-13
RUN apt-get update && apt-get install -y --no-install-recommends \
build-essential \
&& rm -rf /var/lib/apt/lists/*
# Isolated venv we can copy wholesale into the runtime stage.
ENV VIRTUAL_ENV=/opt/venv
RUN python -m venv "$VIRTUAL_ENV"
ENV PATH="$VIRTUAL_ENV/bin:$PATH"
WORKDIR /app
# Python deps first for layer caching. Installed from the hash-pinned lock
# (agent/requirements.txt is the human-edited source; regenerate the lock
# with the command documented at the top of requirements-lock.txt whenever
# agent/requirements.txt changes).
COPY agent/requirements.txt agent/requirements.txt
COPY requirements-lock.txt requirements-lock.txt
RUN pip install --no-cache-dir --require-hashes -r requirements-lock.txt
# Copy project + install the CLI entrypoint (editable — the runtime stage
# re-creates the same /app/agent source tree the .pth file points at).
COPY pyproject.toml LICENSE README.md ./
COPY agent/ agent/
RUN pip install --no-cache-dir -e .
# ============================================================================
# Stage 3: Runtime — carries the prebuilt venv only, no compilers/dev headers.
# ============================================================================
FROM python:3.11-slim@sha256:e031123e3d85762b141ad1cbc56452ba69c6e722ebf2f042cc0dc86c47c0d8b3 AS runtime
# python:3.11-slim digest resolved 2026-07-13
LABEL org.opencontainers.image.title="Vibe-Trading" \
org.opencontainers.image.description="Natural-language finance research AI agent with backtesting" \
org.opencontainers.image.version="0.1.7" \
org.opencontainers.image.version="0.1.13" \
org.opencontainers.image.source="https://github.com/HKUDS/Vibe-Trading" \
org.opencontainers.image.licenses="MIT"
WORKDIR /app
# System deps
ENV PYTHONDONTWRITEBYTECODE=1 \
PYTHONUNBUFFERED=1
# Runtime-only native libs. NO build-essential here — these are weasyprint's
# shared libraries (Pango/HarfBuzz/Fontconfig/Cairo/gdk-pixbuf) per its official
# Debian install list; without them the lazy `from weasyprint import HTML` in
# reporter.py fails and PDF rendering silently downgrades to HTML-only.
# fonts-dejavu-core gives non-blank PDFs.
RUN apt-get update && apt-get install -y --no-install-recommends \
build-essential \
libpango-1.0-0 \
libpangoft2-1.0-0 \
libharfbuzz0b \
libfontconfig1 \
libgdk-pixbuf-2.0-0 \
libcairo2 \
fonts-dejavu-core \
&& rm -rf /var/lib/apt/lists/*
# Python deps (install before copying code for layer caching)
COPY agent/requirements.txt agent/requirements.txt
RUN pip install --no-cache-dir -r agent/requirements.txt
# Bring in the prebuilt venv from the builder stage.
ENV VIRTUAL_ENV=/opt/venv
ENV PATH="$VIRTUAL_ENV/bin:$PATH"
COPY --from=builder /opt/venv /opt/venv
# Copy project
# Re-materialize the source tree the editable install references, plus the
# built frontend static assets.
COPY pyproject.toml LICENSE README.md ./
COPY agent/ agent/
# Copy built frontend
COPY --from=frontend-build /app/frontend/dist frontend/dist
# Install CLI entrypoint
RUN pip install --no-cache-dir -e .
# Runtime should not run as root. Keep writable app data directories owned by
# the service user so named Docker volumes inherit usable permissions.
# Runtime should not run as root. `vibe` owns the writable app-data dirs so
# named volumes inherit usable permissions. `vibe-sandbox` is an unprivileged
# system account (no home, no shell) that runner.py drops into via
# subprocess.run(user="vibe-sandbox") to execute LLM-generated code with the
# least privilege — created here by fixed contract, not otherwise used.
RUN useradd --create-home --shell /usr/sbin/nologin vibe \
&& mkdir -p agent/runs agent/sessions agent/uploads agent/.swarm/runs \
&& chown -R vibe:vibe /app
&& useradd --system --no-create-home --shell /usr/sbin/nologin --uid 10001 vibe-sandbox \
&& mkdir -p agent/runs agent/sessions agent/uploads agent/.swarm/runs /home/vibe/.vibe-trading \
&& chown -R vibe:vibe /app /home/vibe/.vibe-trading
USER vibe
# Default port
EXPOSE 8899
# Health check
# Health check — hits /live (liveness probe; /health remains a legacy alias).
HEALTHCHECK --interval=30s --timeout=5s --start-period=10s --retries=3 \
CMD python -c "import urllib.request; urllib.request.urlopen('http://localhost:8899/health')" || exit 1
CMD python -c "import urllib.request; urllib.request.urlopen('http://localhost:8899/live')" || exit 1
# Run API server (serves frontend/dist as static files)
CMD ["vibe-trading", "serve", "--host", "0.0.0.0", "--port", "8899"]
NOTICE
+3
View File
@@ -20,3 +20,6 @@ repository as factual mathematical content (formulas are not copyrightable):
Prose, tables, and figures from the source papers and reports are NOT
reproduced in this repository. Only the mathematical formulas — which are
factual content and not subject to copyright — are reimplemented here.
This product bundles Inter and JetBrains Mono fonts from @fontsource 5.3.0
under the SIL Open Font License 1.1. See frontend/public/fonts/LICENSE.
README.md
+700 -76
View File
File diff suppressed because it is too large. Load diff
README_ar.md
+810 -74
View File
File diff suppressed because it is too large. Load diff
README_ja.md
+817 -75
View File
File diff suppressed because it is too large. Load diff
README_ko.md
+812 -74
View File
File diff suppressed because it is too large. Load diff
README_zh.md
+805 -75
View File
File diff suppressed because it is too large. Load diff
SECURITY.md
+21
View File
@@ -20,6 +20,27 @@ We will acknowledge your report within **5 business days** and work with you to
This policy applies to the [HKUDS/Vibe-Trading](https://github.com/HKUDS/Vibe-Trading) repository.
## Generated backtest code
Backtest runs may execute generated Python strategy code locally. Treat generated strategies as local code you review before running. The runner validates run directories and uses a narrow subprocess environment: it preserves OS/Python basics, proxy/certificate settings, allowed run-root configuration, and read-only market-data credentials needed by loaders, but it does not forward LLM provider keys, API server bearer tokens, shell-tool opt-ins, broker trading secrets, or live/advisory toggles by default.
The backtest subprocess is still network-capable so loaders can fetch public or user-authorized market data. Do not run untrusted generated strategies in an environment that exposes sensitive files, secret-bearing proxy variables, or network services you would not trust local code to access.
## Official channels & impersonation
Vibe-Trading is an open-source finance **research** tool. We will **never** ask you to
"verify", connect, or sign with a crypto wallet to join our community, claim an airdrop, or
unlock features — any such prompt is a scam.
- Our only official Discord is **https://discord.gg/6TdQnT5xcF** (the HKUDS community server,
also linked from the README). Treat any other "Vibe-Trading" Discord as an impostor.
- If a Discord or website asks you to connect/sign a wallet for "verification", do not do it.
If you already did, move your funds to a fresh wallet and revoke approvals (e.g. via
[revoke.cash](https://revoke.cash)).
See the pinned security announcement in [Discussions](https://github.com/HKUDS/Vibe-Trading/discussions)
for the 2026-06-18 impostor-Discord incident.
## Disclosure
- Please do not publicly disclose the vulnerability until we have released a fix.
agent/.env.example
+178 -13
View File
@@ -1,24 +1,40 @@
# NOTE: All defaults are defined in agent/src/config/env_schema.py (EnvConfig).
# ============================================================================
# LLM Provider — uncomment ONE provider block below
# LLM Provider (LLMConfig) — uncomment ONE provider block below
# ============================================================================
# --- OpenRouter (recommended, multi-model gateway) ---
LANGCHAIN_PROVIDER=openrouter
LANGCHAIN_MODEL_NAME=deepseek/deepseek-v4-pro
OPENROUTER_API_KEY=sk-or-v1-your-key-here
OPENROUTER_API_KEY=sk-or-...here
OPENROUTER_BASE_URL=https://openrouter.ai/api/v1
# --- Requesty (OpenAI-compatible multi-model gateway) ---
# Same provider/model naming as OpenRouter. Get a key at https://app.requesty.ai/api-keys
# EU endpoint: https://router.eu.requesty.ai/v1
# LANGCHAIN_PROVIDER=requesty
# LANGCHAIN_MODEL_NAME=openai/gpt-4o-mini
# REQUESTY_API_KEY=xxx
# REQUESTY_BASE_URL=https://router.requesty.ai/v1
# --- OpenAI ---
# LANGCHAIN_PROVIDER=openai
# LANGCHAIN_MODEL_NAME=gpt-5.5-instant
# OPENAI_API_KEY=sk-xxx
# OPENAI_BASE_URL=https://api.openai.com/v1
# --- Anthropic Messages API / compatible proxy ---
# LANGCHAIN_PROVIDER=anthropic
# LANGCHAIN_MODEL_NAME=claude-sonnet-4-6
# ANTHROPIC_API_KEY=sk-ant-xxx
# ANTHROPIC_BASE_URL=https://api.anthropic.com
# ANTHROPIC_MAX_TOKENS=16384 # Recommended for proxy model names unknown to LangChain
# --- OpenAI Codex (ChatGPT OAuth; not the OpenAI API key path) ---
# Login first: vibe-trading provider login openai-codex
# Requires a ChatGPT account with Codex access. OAuth tokens are stored by oauth-cli-kit.
# LANGCHAIN_PROVIDER=openai-codex
# LANGCHAIN_MODEL_NAME=openai-codex/gpt-5.3-codex
# LANGCHAIN_MODEL_NAME=openai-codex/gpt-5.4
# OPENAI_CODEX_BASE_URL=https://chatgpt.com/backend-api/codex/responses
# --- DeepSeek ---
@@ -26,6 +42,17 @@ OPENROUTER_BASE_URL=https://openrouter.ai/api/v1
# LANGCHAIN_MODEL_NAME=deepseek-v4-pro
# DEEPSEEK_API_KEY=sk-xxx
# DEEPSEEK_BASE_URL=https://api.deepseek.com/v1
# DeepSeek adapter selection: auto (default; use langchain-deepseek if installed),
# native (require langchain-deepseek, error if missing), or openai-compatible
# (force the legacy ChatOpenAI path). Install the adapter with:
# pip install "vibe-trading-ai[deepseek]"
# VIBE_TRADING_DEEPSEEK_ADAPTER=auto
# --- NVIDIA NIM (hosted API catalog) ---
# LANGCHAIN_PROVIDER=nvidia
# LANGCHAIN_MODEL_NAME=nvidia/nemotron-3-ultra-550b-a55b
# NVIDIA_API_KEY=nvapi-xxx
# NVIDIA_BASE_URL=https://integrate.api.nvidia.com/v1
# --- Gemini ---
# LANGCHAIN_PROVIDER=gemini
@@ -51,17 +78,26 @@ OPENROUTER_BASE_URL=https://openrouter.ai/api/v1
# ZHIPU_API_KEY=xxx
# ZHIPU_BASE_URL=https://open.bigmodel.cn/api/paas/v4
# --- Moonshot / Kimi ---
# --- Moonshot / Kimi (open platform, pay-as-you-go) ---
# LANGCHAIN_PROVIDER=moonshot
# LANGCHAIN_MODEL_NAME=kimi-k2.6
# MOONSHOT_API_KEY=sk-xxx
# MOONSHOT_BASE_URL=https://api.moonshot.ai/v1
# --- Kimi for Coding (subscription plan; NOT the open platform above) ---
# Create the key in the Kimi Code console (kimi.com/code/docs). A coding-plan
# key is rejected by api.moonshot.ai and vice-versa. temperature is forced to 1.
# LANGCHAIN_PROVIDER=kimi-coding
# LANGCHAIN_MODEL_NAME=kimi-for-coding
# KIMI_CODING_API_KEY=sk-kimi-xxx
# KIMI_CODING_BASE_URL=https://api.kimi.com/coding/v1
# --- MiniMax ---
# LANGCHAIN_PROVIDER=minimax
# LANGCHAIN_MODEL_NAME=MiniMax-M2.7 # or MiniMax-M2.7-highspeed (faster)
# LANGCHAIN_MODEL_NAME=MiniMax-M3 # or MiniMax-M2.7 / MiniMax-M2.7-highspeed
# MINIMAX_API_KEY=xxx
# MINIMAX_BASE_URL=https://api.minimax.io/v1
# MINIMAX_BASE_URL=https://api.minimax.io/v1 # Global
# MINIMAX_BASE_URL=https://api.minimaxi.com/v1 # China
# Note: MiniMax requires temperature > 0. Set LANGCHAIN_TEMPERATURE=1.0 (default when using MiniMax)
# --- Xiaomi MIMO ---
@@ -70,6 +106,13 @@ OPENROUTER_BASE_URL=https://openrouter.ai/api/v1
# MIMO_API_KEY=xxx
# MIMO_BASE_URL=https://api.xiaomimimo.com/v1
# --- iFlytek Spark ---
# APIPassword from the iFlytek console (https://console.xfyun.cn), sent as a Bearer token.
# LANGCHAIN_PROVIDER=spark
# LANGCHAIN_MODEL_NAME=4.0Ultra # or generalv3.5 / pro-128k / lite
# SPARK_API_KEY=xxx
# SPARK_BASE_URL=https://spark-api-open.xf-yun.com/v1
# --- Z.ai (for Coding Plan) ---
# LANGCHAIN_PROVIDER=zai
# LANGCHAIN_MODEL_NAME=glm-5.1
@@ -86,13 +129,17 @@ LANGCHAIN_TEMPERATURE=0.0
TIMEOUT_SECONDS=120
MAX_RETRIES=2
# Enable reasoning on relays that require opt-in (e.g. OpenRouter thinking models).
# Forwarded as extra_body.reasoning.effort = low / medium / high / max.
# Leave unset for Moonshot / DeepSeek official APIs — they return reasoning_content by default.
# Reasoning effort: none / low / medium / high / max. Two delivery paths:
# * Relays requiring opt-in (OpenRouter, Requesty) get extra_body.reasoning.effort.
# * Direct OpenAI gets a top-level reasoning_effort — gpt-5.6-* models require an
# explicit value (including "none") before they accept function tools.
# Leave unset for every other provider: Moonshot / DeepSeek official APIs return
# reasoning_content by default, and other OpenAI-compatible providers may reject
# the field outright.
# LANGCHAIN_REASONING_EFFORT=medium
# ============================================================================
# Data Sources
# Data Sources (DataConfig)
# ============================================================================
# A-shares: Tushare Pro token (https://tushare.pro)
TUSHARE_TOKEN=your-tushare-token
@@ -104,15 +151,89 @@ TUSHARE_TOKEN=your-tushare-token
# Download FutuOpenD: https://www.futunn.com/download/openAPI
# FUTU_HOST=127.0.0.1
# FUTU_PORT=11111
# US / HK equities via LongPort OpenAPI (optional, requires LongPort developer account)
# Sign up: https://open.longbridge.com
# LONGBRIDGE_APP_KEY=your-app-key
# LONGBRIDGE_APP_SECRET=your-app-secret
# LONGBRIDGE_ACCESS_TOKEN=your-access-token
# Install the optional SDK before using Longbridge:
# pip install "vibe-trading-ai[longbridge]"
# To force Longbridge for a backtest, set "source": "longbridge" in config.json.
# In an Agent conversation, ask: "Use Longbridge to fetch QQQ.US historical data."
# Free direct-API sources need NO key and auto-join the fallback chain: Eastmoney / Sina / Stooq / Yahoo.
# Optional API-key data sources (enabled only when the key is set; silently skipped otherwise)
# FINNHUB_API_KEY=xxx # optional US-equity OHLCV fallback provider (selectable get_market_data source)
# ALPHAVANTAGE_API_KEY=xxx # optional US-equity OHLCV fallback provider (selectable get_market_data source)
# TIINGO_API_KEY=xxx # optional US-equity OHLCV fallback provider (selectable get_market_data source)
# FMP_API_KEY=xxx # optional US-equity OHLCV fallback provider (selectable get_market_data source)
# FRED_API_KEY=xxx # macro series (get_macro_series tool)
# VIBE_TRADING_IWENCAI_KEY=xxx # A-share natural-language research search (iwencai_search tool)
# VIBE_TRADING_SEC_UA="Your Name your@email.com" # SEC EDGAR compliant contact User-Agent (optional; a built-in default UA also works)
# Optional per-source request-spacing overrides for batch jobs — VIBE_TRADING_{EASTMONEY,SINA,STOOQ,YAHOO,SEC,FINNHUB,ALPHAVANTAGE,TIINGO,FMP,FRED,IWENCAI,THS}_MIN_INTERVAL (seconds).
# OCR engine for scanned PDF pages and images (auto by default).
# auto — local engines only (privacy: never sends pages to cloud)
# rapid — force RapidOCR (local, ONNX; pip install rapidocr_onnxruntime)
# llm-vision — force LLM vision OCR (cloud; any OpenAI-compatible vision model)
# none — disable OCR entirely
# VIBE_TRADING_OCR_ENGINE=auto
# Override the model used for LLM vision OCR (defaults to LANGCHAIN_MODEL_NAME).
# Example: qwen3.7-plus, gpt-5.5, gemini-3.5-flash, claude-sonnet-4.6
# VIBE_TRADING_OCR_LLM_MODEL=
# ============================================================================
# API Server (optional)
# Storage root
# ============================================================================
# Chat sessions, run artifacts, swarm runs, uploads, and the sessions.db
# search/goal index live under ~/.vibe-trading by default. Set
# VIBE_TRADING_HOME to relocate them, e.g. to keep deliberately separate
# stores per project. (Other state — memory, reports, caches, live configs —
# is still anchored to ~/.vibe-trading regardless.) Set it in your shell
# environment (export VIBE_TRADING_HOME=...), NOT in this file: path
# constants are resolved when the process starts, before this file is read,
# so an override here would apply to some code paths and not others.
# VIBE_TRADING_HOME=~/.vibe-trading
# Opt-in local market-data cache for backtest loaders (off by default).
# When enabled, every data source caches settled historical bars under
# ~/.vibe-trading/cache/loaders/ so repeated/long-horizon backtests skip the
# network. Only fully-elapsed days are cached (ranges ending today are always
# re-fetched). Clear it any time with: rm -rf ~/.vibe-trading/cache
# VIBE_TRADING_DATA_CACHE=1
# Path to a published Taiwan-market SQLite snapshot. The read-only Taiwan stock
# tool registers only when this points at a database with the expected schema,
# so leaving it unset simply omits the tool. Keep the file outside the checkout —
# no market data may land in the working tree.
# VIBE_TW_STOCK_DB=~/.vibe-trading/tw-stock/latest.db
# Opt-in background scheduler for saved research jobs (off by default).
# When enabled, the API server runs a background poller that fires due
# interval/cron scheduled-research jobs through the session runtime. Cron
# jobs may carry an IANA `timezone` (evaluated on that zone's wall clock,
# DST-aware); jobs without one evaluate in UTC.
# VIBE_TRADING_ENABLE_SCHEDULER=1
# A dispatch failure stays retryable until this many consecutive failures.
# Retry timing is max(next scheduled time, exponential delay), capped below.
# VIBE_TRADING_SCHEDULER_MAX_CONSECUTIVE_FAILURES=3
# VIBE_TRADING_SCHEDULER_RETRY_BASE_DELAY_MS=60000
# VIBE_TRADING_SCHEDULER_RETRY_MAX_DELAY_MS=3600000
# ============================================================================
# API Server (APIConfig, optional)
# ============================================================================
# Bearer token for API auth.
# Empty dev mode is loopback-only. Set this before exposing port 8899 beyond localhost.
# API_AUTH_KEY=
# CORS allowed origins (comma-separated)
# CORS allowed origins (comma-separated). Setting this REPLACES the loopback
# defaults.
# CORS_ORIGINS=http://localhost:3000,http://localhost:5173,http://localhost:8000
# Extra CORS origins, ADDED to the loopback defaults instead of replacing them.
# Needed to reach the API from a hosted console such as OpenBB Workspace. A
# remote browser origin is still refused by the cross-site guard, so this is for
# same-host or server-side callers.
# VIBE_TRADING_EXTRA_CORS_ORIGINS=https://pro.openbb.co
# Enable session runtime
# ENABLE_SESSION_RUNTIME=true
# Docker compose sets this together with a 127.0.0.1 port bind so browser
@@ -130,7 +251,7 @@ TUSHARE_TOKEN=your-tushare-token
# VIBE_TRADING_ALLOWED_RUN_ROOTS=
# ============================================================================
# Agent Tuning (optional, defaults are sensible)
# Agent Tuning (AgentTuningConfig, optional, defaults are sensible)
# ============================================================================
# SWARM_WORKER_TIMEOUT=300
# SWARM_WORKER_MAX_ITER=50
@@ -139,6 +260,50 @@ TUSHARE_TOKEN=your-tushare-token
# SUBAGENT_TIMEOUT=300
# SUBAGENT_MAX_ITER=25
# TOKEN_THRESHOLD=40000
# Hard timeout (seconds) for read-only tool execution; write tools only warn.
# Default 1800; set 0 to disable.
# VIBE_TRADING_TOOL_TIMEOUT_SECONDS=1800
# Frontend SSE idle timeout in seconds before showing "Execution timed out".
# Increase when using slow local models (e.g. Ollama on CPU/limited VRAM).
# VIBE_TRADING_SSE_TIMEOUT=90
# Content filter warning threshold (default: 0.05 = 5%)
# When the ratio of LLM responses blocked by content moderation exceeds this
# threshold, the run card will warn you to switch providers.
# CONTENT_FILTER_WARNING_THRESHOLD=0.05
# ============================================================================
# Memory System (optional, OFF by default)
# ============================================================================
# One-line preset (recommended):
# off = no lifecycle management (default)
# on = quality scoring + auto-decay + garbage collection
# full = on + hierarchy + semantic links + compression + FTS search
# VT_MEMORY=off
# Advanced: override individual features (takes priority over VT_MEMORY)
# VT_MEMORY_QUALITY=true
# VT_MEMORY_DECAY=true
# VT_MEMORY_GC=true
# VT_MEMORY_HIERARCHY=true
# VT_MEMORY_LINKS=true
# VT_MEMORY_COMPRESSION=true
# VT_MEMORY_FTS_INDEX=true
# ============================================================================
# TAP Mode — credential isolation for broker orders (optional, OFF by default)
# ============================================================================
# Route Alpaca order placement through the TAP proxy (https://tap.human.tech)
# instead of the local broker SDK: the agent process holds no Alpaca key, and
# every order is held for HUMAN APPROVAL before it reaches the broker. Leave
# these unset for the default (direct broker SDK) behaviour.
#
# Setup: in the TAP dashboard create a multi-secret credential named `alpaca`
# with fields `key_id` + `secret_key`, allowed host `paper-api.alpaca.markets`,
# assigned to your agent. See the README "TAP Mode" section for details.
#
# TAP_PROXY_URL=https://proxy.tap.human.tech
# TAP_AGENT_KEY=your-tap-agent-key
# Credential name configured in TAP (default: alpaca)
# TAP_ALPACA_CREDENTIAL=alpaca
# Seconds to wait for a human approval decision (default: 300)
# TAP_APPROVAL_TIMEOUT=300
agent/SKILL.md
+51 -19
View File
@@ -1,7 +1,7 @@
---
name: vibe-trading
version: 0.1.8
description: Professional finance research toolkit — backtesting (7 engines + benchmark comparison panel), factor analysis, Alpha Zoo (452 pre-built alphas across qlib158/alpha101/gtja191/academic), options pricing, 77 finance skills, 29 multi-agent swarm teams, Trade Journal analyzer, and Shadow Account (extract → backtest → render) across 7 data sources (tushare, yfinance, okx, akshare, mootdx, ccxt, futu).
version: 0.1.13
description: Professional finance research toolkit — backtesting (9 engines + benchmark comparison panel), factor analysis, Alpha Zoo (462 pre-built alphas across qlib158/alpha101/gtja191/academic/fundamental), options pricing, 89 finance skills, 30 multi-agent swarm teams, Trade Journal analyzer, and Shadow Account (extract → backtest → render) across 24 market-data sources (tushare, yfinance, okx, binance, akshare, baostock, tencent, mootdx, ccxt, futu, mt5, local, eastmoney, sina, stooq, yahoo, pykrx, india_broker, qveris, longbridge, plus optional-key finnhub/alphavantage/tiingo/fmp).
dependencies:
python: ">=3.11"
pip:
@@ -23,7 +23,7 @@ mcp:
# Vibe-Trading
Professional finance research toolkit with AI-powered backtesting (7 engines), multi-agent teams, 77 specialized skills, the **Alpha Zoo** (452 pre-built quantitative alphas across qlib158 / alpha101 / gtja191 / academic with one-line CLI benchmarking), and the Shadow Account loop — extract your implicit trading rules from a journal, backtest them across A股/港股/美股/crypto, then see where they would have served you better.
Professional finance research toolkit with AI-powered backtesting (9 engines), multi-agent teams, 89 specialized skills, the **Alpha Zoo** (462 pre-built quantitative alphas across qlib158 / alpha101 / gtja191 / academic / fundamental with one-line CLI benchmarking), and the Shadow Account loop — extract your implicit trading rules from a journal, backtest them across A股/港股/美股/crypto, then see where they would have served you better.
## Setup
@@ -53,12 +53,13 @@ Add to your agent's MCP config:
### API Key Requirements
Core research MCP tools work with zero API keys for HK/US/crypto. After `pip install`, backtesting, market data, factor analysis, options pricing, chart patterns, web search, document reading, trade journal analysis, shadow-account extraction/backtest/report, the Alpha Zoo (452 pre-built alphas), and all 77 skills are ready to use. IBKR tools require a local TWS / IB Gateway session; `run_swarm` requires an LLM key.
Core research MCP tools work with zero API keys for HK/US/crypto. After `pip install`, backtesting, market data, factor analysis, options pricing, chart patterns, web search, document reading, trade journal analysis, shadow-account extraction/backtest/report, the Alpha Zoo (462 pre-built alphas), and all 89 skills are ready to use. IBKR tools require a local TWS / IB Gateway session; `run_swarm` requires an LLM key.
| Feature | Key needed | When |
|---------|-----------|------|
| HK/US equities & crypto | None | Always free (yfinance + OKX) |
| China A-share data | `TUSHARE_TOKEN` | Only if you query A-share symbols |
| HK/US equities & crypto | None | Always free (yfinance / stooq / yahoo + OKX) |
| China A-share data | None | Free via akshare / baostock / tencent / sina / eastmoney / mootdx fallback (`TUSHARE_TOKEN` optional for premium quality) |
| Premium US fundamentals/quotes | `FINNHUB_API_KEY` / `ALPHAVANTAGE_API_KEY` / `TIINGO_API_KEY` / `FMP_API_KEY` | Only for optional-key providers (graceful fallback to free sources) |
| Multi-agent swarm (`run_swarm`) | `OPENAI_API_KEY` + `LANGCHAIN_MODEL_NAME` | Swarm spawns internal LLM workers |
## What You Can Do
@@ -73,12 +74,19 @@ Feed a CSV broker export (同花顺 / 东财 / 富途 / generic), and the agent
5. `scan_shadow_signals` — list today's symbols that match your shadow's entry cadence (research only).
### Backtesting
Create and run quantitative strategies across 7 engines (ChinaA, GlobalEquity, Crypto, ChinaFutures, GlobalFutures, Forex + options) with 7 data sources:
- **HK/US equities** via yfinance (free, no API key)
Create and run quantitative strategies across 9 engines (ChinaA, GlobalEquity, IndiaEquity, KoreaEquity, Crypto, ChinaFutures, GlobalFutures, Forex + options) with 24 market-data sources (auto-detect + ordered fallback):
- **HK/US equities** via yfinance / stooq / yahoo (free, no API key); optionally via **Longbridge** historical OHLCV (`longbridge`, requires the optional SDK and `LONGBRIDGE_APP_KEY` / `LONGBRIDGE_APP_SECRET` / `LONGBRIDGE_ACCESS_TOKEN`). To force it for a run, set `"source": "longbridge"` in `config.json`.
- **India equities (NSE/BSE)** via yahoo / yfinance using `<SYMBOL>.NS` (NSE, e.g. `RELIANCE.NS`) or `<SCRIP>.BO` (BSE, e.g. `500325.BO`) — free, no API key. The `IndiaEquityEngine` models T+1 delivery, no overnight shorts (set `allow_short` for intraday), configurable circuit bands, 1-share lots, and the STT/stamp-duty/exchange/GST cost stack. Optionally back-fill from your live broker via the `india_broker` source (Shoonya/Dhan; requires broker login).
- **Korea equities (KRX: KOSPI/KOSDAQ)** via pykrx using `<CODE>.KS` (KOSPI, e.g. `005930.KS`) or `<CODE>.KQ` (KOSDAQ, e.g. `247540.KQ`) — free, no API key (`pip install "vibe-trading-ai[krx]"`; yahoo/yfinance fallback needs no extra). pykrx serves **daily bars only** (an intraday request falls through to another source) and its adjusted series is Naver-backed rather than a verbatim KRX print. The `KoreaEquityEngine` models same-day round trips (no T+1), the ±30% daily price limit measured from the previous close and quantized to the KRX tick grid, tick-rounded fills, the 0.20% sell-side transaction tax (2026 rate), and 1-share lots. It is **long-only**: `allow_short` is refused, because KRX covered-short and uptick rules cannot be enforced on daily bars.
- **Cryptocurrency** via OKX or CCXT/100+ exchanges (free, no API key)
- **China A-shares** via Tushare (token) or AKShare (free fallback)
- **China A-shares** via AKShare / baostock / tencent / sina / eastmoney / mootdx (free, no API key) — `TUSHARE_TOKEN` optional for premium quality
- **Futures, forex, macro** via AKShare (free, no API key)
- **HK & A-share equities** via Futu (broker login required, optional)
- **Local CSV/parquet bars** via the `local` loader (offline, no network)
- **Premium cross-market data** via QVeris (optional API key)
- **Premium US data** via optional-key finnhub / alphavantage / tiingo / fmp (graceful fallback to free sources)
Factors: the Alpha101 and QLib158 zoos are tagged for the `equity_in` and `equity_kr` universes, so they compute on NSE/BSE and KRX bars (the GTJA191 zoo stays China-only). Live/paper India trading uses the Shoonya / Dhan connectors (paper + read-only live; live order placement is structurally disabled because those brokers expose no paper/live switch).
Example workflow:
1. Use `list_skills()` to discover strategy patterns
@@ -87,7 +95,7 @@ Example workflow:
4. Use `backtest()` to run and get metrics (Sharpe, return, drawdown, etc.)
### Multi-Agent Swarm Teams
29 pre-built agent teams for complex research:
30 pre-built agent teams for complex research:
- **Investment Committee**: bull/bear debate → risk review → PM decision
- **Global Equities Desk**: A-share + HK/US + crypto → global strategist
- **Crypto Trading Desk**: funding/basis + liquidation + flow → risk manager
@@ -95,20 +103,21 @@ Example workflow:
- **Macro/Rates/FX Desk**: rates + FX + commodities → macro PM
- **Quant Strategy Desk**: screening → factor research → backtest → risk audit
- **Risk Committee**: drawdown, tail risk, regime analysis
- And 22 more specialized teams
- And 23 more specialized teams
Use `list_swarm_presets()` to see all teams, then `run_swarm()` to execute.
### Alpha Zoo (452 pre-built alphas)
One-line cross-sectional IC / IR / alive-reversed-dead categorisation across four bundled zoos:
### Alpha Zoo (462 pre-built alphas)
One-line cross-sectional IC / IR / alive-reversed-dead categorisation across five bundled zoos:
- **qlib158** (154 alphas) — Microsoft Qlib's `Alpha158` feature handler, Apache-2.0 with pinned commit SHA.
- **alpha101** (101 alphas) — Kakushadze (2015) "101 Formulaic Alphas" (arXiv:1601.00991), written from the paper appendix.
- **gtja191** (191 alphas) — Guotai Junan 2014 "191 Short-period Trading Alpha Factors" research report.
- **academic** (6 factors) — Fama-French 5 + Carhart momentum (honest price-based proxies).
- **academic** (12 factors) — Fama-French 5 + Carhart momentum + Jegadeesh reversal + George-Hwang 52-week-high + Amihud illiquidity + Harvey-Siddique skew + Frazzini-Pedersen betting-against-beta (price-based proxies) + a correlation-rewiring stability score (from the in-repo correlation-regime skill).
- **fundamental** (4 factors) — PIT-safe earnings yield, ROE, gross profitability, and asset growth from daily fundamental panels.
Each alpha ships with `__alpha_meta__` (formula LaTeX + theme + universe + warmup + columns required), guarded by an AST purity gate + 300-row lookahead sentinel test. Use the `vibe-trading alpha {list,show,bench,compare,export-manifest}` CLI, the `/alpha/*` REST routes (browser at `/alpha-zoo`), or compose multi-factor signals via `ZooSignalEngine.from_zoo(...)`.
### Finance Skills (77)
### Finance Skills (88)
Comprehensive knowledge base covering:
- Technical analysis (candlestick, Elliott wave, Ichimoku, SMC, harmonic, chanlun)
- Quantitative methods (factor research, ML strategy, pair trading, multi-factor)
@@ -121,11 +130,11 @@ Comprehensive knowledge base covering:
Use `load_skill(name)` to access full methodology docs with code templates.
## Available MCP Tools (35)
## Available MCP Tools (55)
| Tool | Description | API Key |
|------|-------------|---------|
| `list_skills` | List all 77 finance skills | None |
| `list_skills` | List all 89 finance skills | None |
| `load_skill` | Load full skill documentation | None |
| `start_research_goal` | Create an auditable research goal | None |
| `get_research_goal` | Read the current research goal | None |
@@ -134,8 +143,27 @@ Use `load_skill(name)` to access full methodology docs with code templates.
| `backtest` | Run vectorized backtest engine | None* |
| `factor_analysis` | IC/IR analysis + layered backtest | None* |
| `analyze_options` | Black-Scholes price + Greeks | None |
| `analyze_options_payoff` | Multi-leg expiry payoff + spot/IV scenarios | None |
| `pattern_recognition` | Detect chart patterns (H&S, double top, etc.) | None |
| `get_market_data` | Fetch OHLCV data across 6 sources (auto-detect + fallback) | None* |
| `get_market_data` | Fetch OHLCV data (auto-detect + ordered fallback across 24 sources) | None* |
| `get_fund_flow` | Capital fund-flow (main/retail net inflow) | None* |
| `get_dragon_tiger` | Dragon-tiger list (龙虎榜) top buyer/seller seats | None* |
| `get_northbound_flow` | Northbound (Stock Connect) net flow | None* |
| `get_margin_trading` | Margin trading & short-selling balances | None* |
| `get_block_trades` | Block-trade (大宗交易) records | None* |
| `get_shareholder_count` | Shareholder-count history per symbol | None* |
| `get_lockup_expiry` | Restricted-share lockup release schedule | None* |
| `get_sector_info` | Sector / industry constituents & performance | None* |
| `get_research_reports` | Sell-side analyst research reports | None* |
| `get_stock_news` | Market & company news headlines | None* |
| `get_sec_filings` | SEC EDGAR filings (10-K/10-Q/8-K, etc.) | None |
| `get_financial_statements` | Income / balance / cash-flow statements | None* |
| `get_options_chain` | Options chain (strikes, IV, OI, Greeks) | None* |
| `get_stock_profile` | Valuation, analyst estimates & institutional holdings (US/HK) | None |
| `screen_market` | Market screener with fundamental/technical filters | None* |
| `search_symbol` | Symbol / ticker search across markets | None |
| `get_macro_series` | FRED macroeconomic series | FRED_API_KEY |
| `iwencai_search` | A-share natural-language research search | IWENCAI_KEY |
| `web_search` | Search the web via DuckDuckGo | None |
| `read_url` | Fetch web page as Markdown | None |
| `read_document` | Extract text from PDF/DOCX/XLSX/PPTX/images | None |
@@ -152,6 +180,7 @@ Use `load_skill(name)` to access full methodology docs with code templates.
| `get_run_result` | Get final report and task summaries | None |
| `list_runs` | List recent swarm runs with metadata | None |
| `reap_stale_runs` | Finalize stale swarm runs | None |
| `retry_run` | Re-run a failed/stale swarm run | LLM key |
| `trading_connections` | List selectable connector profiles | None |
| `trading_select_connection` | Select the default connector profile | None |
| `trading_check` | Check connector readiness | Connector app/OAuth |
@@ -169,7 +198,7 @@ Use `load_skill(name)` to access full methodology docs with code templates.
pip install vibe-trading-ai
```
That's it — no API keys needed for HK/US/crypto markets. Start using `backtest`, `get_market_data`, `analyze_options`, `analyze_trade_journal`, `extract_shadow_strategy`, `web_search`, the **Alpha Zoo** (`vibe-trading alpha bench --zoo gtja191 --universe csi300 --period 2018-2025`), and all 77 skills immediately.
That's it — no API keys needed for HK/US/crypto markets. Start using `backtest`, `get_market_data`, `analyze_options`, `analyze_trade_journal`, `extract_shadow_strategy`, `web_search`, the **Alpha Zoo** (`vibe-trading alpha bench --zoo gtja191 --universe csi300 --period 2018-2025`), and all 89 skills immediately.
## Loading Tools from External MCP Servers
@@ -325,3 +354,6 @@ Without `ALLOW_SESSION_MCP_SERVERS=1`, any `mcpServers` key in `session.config`
**Options analysis:**
> Use analyze_options: spot=100, strike=105, 90 days, vol=25%, rate=3%
**Multi-leg options payoff:**
> Use analyze_options_payoff for a 95/105 bull call spread at spot 100 with 30 days remaining: long one 95 call at premium 8, short one 105 call at premium 3, multiplier 100, commission rate 0.001.
agent/api_server.py
+261 -2965
View File
File diff suppressed because it is too large. Load diff
agent/backtest/benchmark.py
+56 -9
View File
@@ -7,11 +7,12 @@ data given a set of strategy codes and a data source.
from __future__ import annotations
from dataclasses import dataclass
from typing import Optional
from typing import Any, Optional
import pandas as pd
from backtest.loaders.yfinance_loader import DataLoader as YfinanceLoader
from backtest.metrics import bar_returns, buy_and_hold_return
# -------------------------------------------------------------------
@@ -42,6 +43,7 @@ def resolve_benchmark(
end_date: str,
interval: str = "1D",
explicit: Optional[str] = None,
loader: Optional[Any] = None,
) -> Optional[BenchmarkResult]:
"""Resolve the appropriate benchmark ticker and fetch its return series.
@@ -52,6 +54,10 @@ def resolve_benchmark(
end_date: Backtest end date.
interval: Bar interval (1m / 5m / 15m / 30m / 1H / 4H / 1D).
explicit: Override ticker (e.g. "SPY" passed via config).
loader: Loader of the configured data source. When given, the
benchmark is fetched through it first, falling back to
yfinance if it yields no data — except ``local``,
which fails closed to keep offline runs offline.
Returns:
BenchmarkResult with return series and total return, or None if no
@@ -61,8 +67,19 @@ def resolve_benchmark(
if ticker is None:
return None
offline = source == "local"
if offline and getattr(loader, "name", None) != source:
# The runtime fallback chain in fetch_data_map() may have swapped in a
# network loader while config["source"] still says local — never fetch
# the benchmark through it. Fail closed instead.
loader = None
try:
bench_df = _fetch_benchmark(ticker, start_date, end_date, interval)
bench_df = _fetch_benchmark(
ticker, start_date, end_date, interval,
loader=loader,
allow_fallback=not offline,
)
except Exception:
return None
@@ -73,10 +90,15 @@ def resolve_benchmark(
if len(close) < 2:
return None
ret_series = close.pct_change().fillna(0.0)
total_ret = float((1 + ret_series).prod() - 1)
ret_series = bar_returns(close, label=f"benchmark {ticker}")
# Price relative, not the compounded product: identical while prices are
# positive, and it is the only one of the two that stays honest once they
# are not (#872).
total = buy_and_hold_return(close)
if total is None:
return None
return BenchmarkResult(ticker=ticker, ret_series=ret_series, total_ret=total_ret)
return BenchmarkResult(ticker=ticker, ret_series=ret_series, total_ret=total)
# -------------------------------------------------------------------
@@ -134,11 +156,36 @@ def _fetch_benchmark(
start_date: str,
end_date: str,
interval: str,
loader: Optional[Any] = None,
allow_fallback: bool = True,
) -> pd.DataFrame:
"""Fetch benchmark OHLCV data via yfinance (single symbol, no auth)."""
loader = YfinanceLoader()
result = loader.fetch([ticker], start_date, end_date, interval=interval)
"""Fetch benchmark OHLCV data.
Tries the configured source's loader first (when given). Falls back to
yfinance (single symbol, no auth) when no loader is given or it yields
no data — unless ``allow_fallback`` is False (offline sources fail
closed instead of making a network request).
"""
if loader is not None:
try:
df = _extract_frame(
loader.fetch([ticker], start_date, end_date, interval=interval),
ticker,
)
except Exception:
df = pd.DataFrame()
if not df.empty:
return df
if not allow_fallback:
return pd.DataFrame()
result = YfinanceLoader().fetch([ticker], start_date, end_date, interval=interval)
return _extract_frame(result, ticker)
def _extract_frame(result: Any, ticker: str) -> pd.DataFrame:
"""Normalise a loader fetch result to a single DataFrame."""
if isinstance(result, dict):
df = result.get(ticker)
elif isinstance(result, pd.DataFrame):
@@ -149,4 +196,4 @@ def _fetch_benchmark(
if df is None or (isinstance(df, pd.DataFrame) and df.empty):
return pd.DataFrame()
return df
return df
agent/backtest/constraints.py
+205
View File
@@ -0,0 +1,205 @@
"""Composable weight constraints applied on top of any optimizer.
Portfolio Studio step 2 (#456): per-name cap, per-name floor, and per-group
exposure caps that work with every optimizer, not only the bounds baked into
``turnover_aware``. Configured via ``constraints`` in config.json::
"constraints": [
{"type": "max_weight", "cap": 0.25},
{"type": "min_weight", "floor": 0.02},
{"type": "group_exposure",
"groups": {"AAPL": "tech", "MSFT": "tech", "XOM": "energy"},
"caps": {"tech": 0.6}}
]
The layer runs on the optimizer's output frame (signed weights, one row per
rebalance date). Magnitudes are adjusted and signs are preserved, so
long/short books work too. Constraints apply in config order, and the layer
only acts on names that are already active: it reshapes allocations, it
never opens or closes positions. Off by default, so existing configs behave
exactly as before.
"""
from __future__ import annotations
from typing import Any, Dict, List, Mapping, Sequence
import numpy as np
import pandas as pd
_EPS = 1e-9
_TOL = 1e-12
_MAX_PASSES = 50
def _bounded_fraction(value: Any, name: str) -> float:
"""Validate a (0, 1] fraction from config, rejecting bools and junk."""
if isinstance(value, (bool, np.bool_)):
raise ValueError(f"{name} must be numeric, not boolean")
try:
v = float(value)
except (TypeError, ValueError) as exc:
raise ValueError(f"{name} must be numeric") from exc
if not np.isfinite(v) or not 0.0 < v <= 1.0:
raise ValueError(f"{name} must be finite and in (0, 1]")
return v
class MaxWeight:
"""Per-name cap with pro-rata redistribution of the clipped excess.
Clipped weight is handed to the names still under the cap, in
proportion to their current size, until everything fits. When the cap
cannot hold the book (n_active * cap < gross), every name lands on the
cap and gross exposure shrinks accordingly.
"""
def __init__(self, cap: float) -> None:
self.cap = cap
def apply(self, w: np.ndarray, codes: Sequence[str]) -> np.ndarray:
del codes # per-name rule, codes unused
w = w.astype(float).copy()
for _ in range(_MAX_PASSES):
over = w > self.cap + _TOL
if not over.any():
break
excess = float((w[over] - self.cap).sum())
w[over] = self.cap
room = w < self.cap - _TOL
base = float(w[room].sum())
if not room.any() or base <= _TOL:
break
w[room] += w[room] / base * excess
return w
class MinWeight:
"""Per-name floor: active names below ``floor`` are lifted to it.
The lift is funded pro-rata by the names above the floor (each gives
up weight in proportion to how far it sits above the floor), so gross
exposure is preserved. If the book cannot fund the floor
(floor * n_active > gross), the date degrades to equal weights.
"""
def __init__(self, floor: float) -> None:
self.floor = floor
def apply(self, w: np.ndarray, codes: Sequence[str]) -> np.ndarray:
del codes
w = w.astype(float).copy()
for _ in range(_MAX_PASSES):
below = (w > _EPS) & (w < self.floor - _TOL)
if not below.any():
break
need = float((self.floor - w[below]).sum())
above = w > self.floor + _TOL
avail = float((w[above] - self.floor).sum())
if avail <= _TOL:
n = len(w)
return np.full(n, w.sum() / n) if n else w
w[below] = self.floor
w[above] -= (w[above] - self.floor) / avail * need
return w
class GroupExposure:
"""Cap the summed weight of each configured group.
A group over its cap is scaled down pro-rata; the freed exposure is
not redistributed (pushing it elsewhere could break another group's
cap), so gross exposure shrinks by the clipped amount. Names absent
from ``groups`` are unconstrained.
"""
def __init__(self, groups: Mapping[str, str], caps: Mapping[str, float]) -> None:
self.groups: Dict[str, str] = dict(groups)
self.caps: Dict[str, float] = dict(caps)
def apply(self, w: np.ndarray, codes: Sequence[str]) -> np.ndarray:
w = w.astype(float).copy()
for group, cap in self.caps.items():
idx = [i for i, c in enumerate(codes) if self.groups.get(c) == group]
if not idx:
continue
total = float(w[idx].sum())
if total > cap + _TOL:
w[idx] *= cap / total
return w
_TYPES = ("max_weight", "min_weight", "group_exposure")
def _build_constraint(spec: Mapping[str, Any]) -> Any:
if not isinstance(spec, Mapping):
raise ValueError(f"constraint spec must be a mapping, got {type(spec).__name__}")
kind = spec.get("type")
if kind == "max_weight":
if "cap" not in spec:
raise ValueError("max_weight constraint requires 'cap'")
return MaxWeight(_bounded_fraction(spec["cap"], "max_weight cap"))
if kind == "min_weight":
if "floor" not in spec:
raise ValueError("min_weight constraint requires 'floor'")
return MinWeight(_bounded_fraction(spec["floor"], "min_weight floor"))
if kind == "group_exposure":
groups = spec.get("groups")
caps = spec.get("caps")
if not isinstance(groups, Mapping) or not groups:
raise ValueError("group_exposure constraint requires a non-empty 'groups' mapping")
if any(not isinstance(c, str) or not isinstance(g, str) for c, g in groups.items()):
raise ValueError("groups must map string asset codes to string group names")
if not isinstance(caps, Mapping) or not caps:
raise ValueError("group_exposure constraint requires a non-empty 'caps' mapping")
unknown = set(caps) - set(groups.values())
if unknown:
raise ValueError(
"caps reference groups with no mapped assets: " + ", ".join(sorted(unknown))
)
return GroupExposure(
groups,
{g: _bounded_fraction(c, f"cap for group {g!r}") for g, c in caps.items()},
)
raise ValueError(
f"unknown constraint type {kind!r}; expected one of {', '.join(_TYPES)}"
)
def load_constraints(config: Mapping[str, Any]) -> List[Any]:
"""Build the constraint list from ``config['constraints']`` (empty if unset)."""
raw = config.get("constraints")
if raw in (None, []):
return []
if not isinstance(raw, list):
raise ValueError("constraints must be a list of constraint specs")
return [_build_constraint(spec) for spec in raw]
def apply_constraints_frame(
frame: pd.DataFrame, constraints: Sequence[Any]
) -> pd.DataFrame:
"""Apply constraints row by row to a signed weight frame.
Args:
frame: Optimizer output (dates x codes), signed weights.
constraints: Constraint objects from ``load_constraints``.
Returns:
Adjusted frame with signs preserved and zero rows/cells untouched.
"""
if not constraints:
return frame
out = frame.copy()
for dt in frame.index:
row = frame.loc[dt]
codes = [c for c in row.index if abs(row[c]) > _EPS]
if not codes:
continue
signs = np.sign(row[codes].to_numpy(dtype=float))
mags = np.abs(row[codes].to_numpy(dtype=float))
for con in constraints:
mags = con.apply(mags, codes)
out.loc[dt, codes] = signs * mags
return out
agent/backtest/correlation.py
+164 -53
View File
@@ -6,32 +6,111 @@ over a configurable lookback window. Used by the /correlation API endpoint.
from __future__ import annotations
import logging
from typing import Dict, Literal
import pandas as pd
import numpy as np
from scipy.stats import spearmanr
logger = logging.getLogger(__name__)
def infer_market(code: str) -> str:
"""Infer market key from a ticker symbol."""
code_upper = code.upper()
"""Infer market key from a ticker symbol.
Resolution order:
1. Crypto pair spellings (``BTC-USDT``, ``ETH/USD`` …).
2. Explicit exchange suffix — always authoritative (``.HK``, ``.SH``/
``.SZ``/``.BJ``, ``.US``). Bare HK and A-share codes are both purely
numeric, so the suffix is the only reliable disambiguator.
3. Bare numeric codes by digit length: A-share codes are exactly 6 digits
(600000, 000001, 300750, 688981, 830799); HK codes are at most 5
(700, 0700, 9988, 3690). Prefix alone cannot tell them apart — both
markets use leading 0 and 3.
4. Anything else (alphabetic tickers) is a US equity.
"""
code_upper = code.strip().upper()
crypto_suffixes = ("USDT", "BTC", "ETH", "BNB", "SOL", "ADA", "DOGE")
if any(code_upper.endswith(s) for s in crypto_suffixes) or "/" in code:
return "crypto"
# Check .HK suffix FIRST so leading-zero tickers like 0700.HK / 0005.HK
# are correctly classified before the A-share prefix checks
if code_upper.endswith(".HK"):
return "hk_equity"
if code_upper.startswith(("6", "000", "001", "002")):
if code_upper.endswith((".SH", ".SZ", ".BJ")):
return "a_share"
if code_upper.startswith(("0", "399")):
return "a_share"
if code_upper.startswith(("0", "1", "2", "3", "4")):
return "hk_equity"
if code_upper.endswith((".KS", ".KQ")):
return "kr_equity"
if code_upper.endswith(".US"):
return "us_equity"
if code_upper.isdigit():
if len(code_upper) == 6:
return "a_share"
if len(code_upper) <= 5:
return "hk_equity"
return "us_equity"
def _normalize_symbol(code: str, market: str) -> str:
"""Convert a user-supplied code to the project's canonical loader symbol.
The data loaders key US/HK/A-share instruments by an exchange-suffixed
symbol (``AAPL.US``, ``0700.HK``, ``600000.SH``); a bare ticker such as
``AAPL`` or ``600000`` matches no loader and fetches nothing. Crypto pairs
(``BTC-USDT``) are already canonical, and any code that already carries a
``.`` suffix is left untouched.
Args:
code: The raw code as typed by the user (e.g. ``AAPL``, ``600000``).
market: The market key from :func:`infer_market`.
Returns:
The canonical symbol the market's loaders expect.
"""
cleaned = code.strip()
# Crypto pairs and anything already exchange-qualified pass through as-is.
if market == "crypto" or "." in cleaned:
return cleaned
upper = cleaned.upper()
if market == "us_equity":
return f"{upper}.US"
if market == "hk_equity":
return f"{upper}.HK"
if market == "a_share":
# 6xxxxx -> Shanghai; 4xxxxx / 8xxxxx -> Beijing; else (0/3) Shenzhen.
if upper[:1] == "6":
return f"{upper}.SH"
if upper[:1] in ("4", "8"):
return f"{upper}.BJ"
return f"{upper}.SZ"
return cleaned
def _close_series(code: str, df: pd.DataFrame) -> pd.Series:
"""Extract the close-price series from a loader frame, date-indexed and sorted.
Supports ``trade_date`` as the index name, as a column, or a plain
DatetimeIndex — the shapes real loaders return.
"""
if df.empty:
raise ValueError(f"Price series for '{code}' is empty")
if "close" not in df.columns and "close" not in df.index.names:
raise ValueError(f"No 'close' column in price series for '{code}'")
# Support trade_date as index name, column, or a plain DatetimeIndex.
if "trade_date" in df.columns:
ts = df.set_index("trade_date")["close"]
elif "close" in df.columns and (
"trade_date" in df.index.names
or isinstance(df.index, pd.DatetimeIndex)
):
ts = df["close"]
else:
raise ValueError(
f"No trade_date index/column for price series '{code}'"
)
return ts.sort_index()
def _rolling_correlation_matrix(
price_series: Dict[str, pd.DataFrame],
window: int,
@@ -57,16 +136,7 @@ def _rolling_correlation_matrix(
returns_frames = []
closes = {}
for code, df in price_series.items():
if df.empty:
raise ValueError(f"Price series for '{code}' is empty")
if "close" not in df.columns and "close" not in df.index.names:
raise ValueError(f"No 'close' column in price series for '{code}'")
# Support both column-based and index-based trade_date
if "trade_date" in df.index.names and "trade_date" not in df.columns:
ts = df["close"]
else:
ts = df.set_index("trade_date")["close"]
closes[code] = ts.sort_index()
closes[code] = _close_series(code, df)
for code in codes:
ts = closes[code]
@@ -74,7 +144,10 @@ def _rolling_correlation_matrix(
# (e.g. crypto via OKX/CCXT at UTC midnight vs US equity via
# yfinance at EDT midnight = 04:00 UTC) align correctly.
ts.index = ts.index.normalize()
rets = ts.pct_change().dropna()
# ``fill_method=None`` is explicit because under the project's
# pandas>=2,<3 pin the ``pct_change`` default forward-fills missing
# prices, silently manufacturing 0% returns on halted sessions.
rets = ts.pct_change(fill_method=None).dropna()
rets.name = code
returns_frames.append(rets)
@@ -119,6 +192,75 @@ def _rolling_correlation_matrix(
return labels, matrix
def _fetch_price_series(
codes: list[str],
start_date: str,
end_date: str,
) -> Dict[str, pd.DataFrame]:
"""Fetch daily price frames for each code via the loader fallback chains.
Args:
codes: Asset codes as supplied by the caller (used as result keys).
start_date: Fetch range start, ``YYYY-MM-DD``.
end_date: Fetch range end, ``YYYY-MM-DD``.
Returns:
Mapping of original code -> OHLCV DataFrame; codes no loader could
serve are omitted (with a warning logged).
"""
# Import here to avoid circular
from backtest.loaders import registry
registry._ensure_registered()
price_series: Dict[str, pd.DataFrame] = {}
for code in codes:
market = infer_market(code)
# Loaders key instruments by the canonical exchange-suffixed symbol
# (AAPL.US / 600000.SH); a bare ticker fetches nothing. Fetch under the
# normalized symbol but keep the user's original code as the label.
symbol = _normalize_symbol(code, market)
# Walk the market's full fallback chain until a loader actually
# returns data. A loader can be "available" yet still serve nothing
# (network error, unsupported symbol), so stopping at the first
# available loader — as resolve_loader does — would silently drop
# the asset even when a later loader could serve it.
for name in registry.FALLBACK_CHAINS.get(market, []):
loader_cls = registry.LOADER_REGISTRY.get(name)
if loader_cls is None:
continue
try:
loader = loader_cls()
except Exception as exc:
logger.debug("correlation: loader %s failed to construct: %s", name, exc)
continue
if not loader.is_available():
continue
try:
result = loader.fetch(
codes=[symbol],
start_date=start_date,
end_date=end_date,
interval="1D",
fields=["trade_date", "open", "high", "low", "close", "volume"],
)
except Exception as exc:
logger.warning("correlation: %s fetch via %s failed: %s", symbol, name, exc)
continue
if symbol in result and not result[symbol].empty:
price_series[code] = result[symbol]
break
logger.warning("correlation: %s returned no data via %s", symbol, name)
else:
logger.warning(
"correlation: no loader in the %s chain returned data for %s "
"(normalized from %r)", market, symbol, code,
)
return price_series
def compute_correlation_matrix(
codes: list[str],
days: int = 90,
@@ -139,38 +281,7 @@ def compute_correlation_matrix(
end_date = datetime.now().strftime("%Y-%m-%d")
start_date = (datetime.now() - timedelta(days=days + 60)).strftime("%Y-%m-%d")
# Import here to avoid circular
from backtest.loaders.registry import resolve_loader
price_series: Dict[str, pd.DataFrame] = {}
for code in codes:
market = infer_market(code)
try:
loader = resolve_loader(market)
except Exception:
# Fall back to yfinance for us_equity / hk_equity
try:
from backtest.loaders.registry import LOADER_REGISTRY
if "yfinance" in LOADER_REGISTRY:
loader = LOADER_REGISTRY["yfinance"]()
else:
continue
except Exception:
continue
try:
result = loader.fetch(
codes=[code],
start_date=start_date,
end_date=end_date,
interval="1D",
fields=["trade_date", "open", "high", "low", "close", "volume"],
)
if code in result and not result[code].empty:
price_series[code] = result[code]
except Exception:
continue
price_series = _fetch_price_series(codes, start_date, end_date)
if len(price_series) < 2:
raise ValueError(
@@ -184,4 +295,4 @@ def compute_correlation_matrix(
"matrix": matrix,
"window": days,
"method": method,
}
}
agent/backtest/engines/_market_hooks.py
+65 -1
View File
@@ -27,6 +27,11 @@ _MARKET_PATTERNS = [
(re.compile(r"^(51|15|56)\d{4}\.(SZ|SH)$", re.I), "a_share"),
(re.compile(r"^[A-Z]+\.US$", re.I), "us_equity"),
(re.compile(r"^\d{3,5}\.HK$", re.I), "hk_equity"),
# India equities: NSE (RELIANCE.NS) / BSE (500325.BO); tickers may carry
# '&' and '-' (e.g. M&M.NS, BAJAJ-AUTO.NS).
(re.compile(r"^[A-Z0-9&.\-]+\.(NS|BO)$", re.I), "india_equity"),
# Korea equities: KOSPI (005930.KS) / KOSDAQ (247540.KQ), 6-digit codes.
(re.compile(r"^\d{6}\.(KS|KQ)$", re.I), "kr_equity"),
(re.compile(r"^[A-Z]+-USDT$", re.I), "crypto"),
(re.compile(r"^[A-Z]+/USDT$", re.I), "crypto"),
# China futures: product+delivery.exchange (e.g. IF2406.CFFEX, rb2410.SHFE)
@@ -44,6 +49,58 @@ _MARKET_PATTERNS = [
_CHINA_EXCHANGES = {"CFFEX", "SHFE", "DCE", "ZCE", "INE", "GFEX"}
# Settlement currency per market. A composite backtest holds one shared capital
# pool, so a code set spanning two of these would add CNY to USD to KRW as if
# they were the same unit.
_MARKET_CURRENCY = {
"a_share": "CNY",
"us_equity": "USD",
"hk_equity": "HKD",
"india_equity": "INR",
"kr_equity": "KRW",
# Every crypto pattern in _MARKET_PATTERNS is USDT-quoted, and USDT is
# carried at its USD peg. This is the one approximation in the table: a
# depeg would make a crypto+US book wrong by the depeg amount, which is
# orders of magnitude below the CNY/USD-style unit error this guard exists
# to catch.
"crypto": "USD",
}
# Non-US futures venues. The GlobalFuturesEngine is USD-denominated end to end
# — margin, commission and contract multipliers are all in USD and it carries
# no EUR or JPY product — so anything it handles settles in USD unless the
# symbol names a venue that does not.
_FUTURES_EXCHANGE_CURRENCY = {"EUREX": "EUR"}
def code_currency(code: str) -> str:
"""Return the currency a symbol settles in.
Args:
code: Ticker / symbol string.
Returns:
A currency code such as ``"CNY"``. A forex pair resolves to its quote
currency and Chinese futures to ``"CNY"``. A symbol whose currency
cannot be established returns a ``"UNKNOWN:<market>"`` marker rather
than a guess, so a homogeneous set still compares equal while a mixed
one cannot pass a same-currency check by accident.
"""
market = _detect_market(code)
if market in _MARKET_CURRENCY:
return _MARKET_CURRENCY[market]
if market == "forex":
pair = code.upper().replace("/", "")
if pair.endswith(".FX"):
pair = pair[:-3]
return pair[3:6] if len(pair) == 6 else "UNKNOWN:forex"
if market == "futures":
if _is_china_futures(code):
return "CNY"
exchange = code.upper().rpartition(".")[2]
return _FUTURES_EXCHANGE_CURRENCY.get(exchange, "USD")
return f"UNKNOWN:{market}"
# Known Chinese-futures product codes — used as a heuristic when a symbol
# lacks an exchange suffix (e.g. bare ``RB2410``, ``IF2406``). Without this
# table composite.py was misrouting such bare codes to GlobalFutures.
@@ -161,7 +218,8 @@ def calc_crypto_funding_fee(
bar: Current bar data.
timestamp: Bar timestamp.
positions: Shared positions dict.
funding_rate: Fixed rate per settlement.
funding_rate: Fallback fixed rate per settlement, used when the bar
carries no historical ``funding_rate`` column.
applied_set: (symbol, date, hour) dedup set — mutated.
daily_done_set: (symbol, date) dedup set — mutated.
@@ -191,6 +249,12 @@ def calc_crypto_funding_fee(
mark_price = float(bar.get("close", pos.entry_price))
notional = pos.size * mark_price
# Prefer the bar's historical funding rate when the loader supplied one
# (USD-M perpetual data via BASE-USDT-PERP); fall back to the fixed
# config rate otherwise so spot-proxy runs keep their behaviour.
hist = bar.get("funding_rate")
if hist is not None and pd.notna(hist):
funding_rate = float(hist)
return notional * funding_rate * pos.direction
agent/backtest/engines/base.py
+701 -95
View File
File diff suppressed because it is too large. Load diff
agent/backtest/engines/china_a.py
+59 -22
View File
@@ -61,14 +61,9 @@ class ChinaAEngine(BaseEngine):
if bar_date is not None and entry_date is not None and bar_date == entry_date:
return False
# 3. Price limits
pct_chg = _calc_pct_change(bar)
if pct_chg is not None:
limit = _price_limit(symbol)
if direction == 1 and pct_chg >= limit - 0.001:
return False # limit-up: can't buy
if direction == 0 and pct_chg <= -limit + 0.001:
return False # limit-down: can't sell
# 3. Price limits, tested at execution time (see _blocked_by_limit).
if _blocked_by_limit(self, symbol, direction, bar, _price_limit(symbol)):
return False
return True
@@ -117,21 +112,63 @@ def _bar_date(bar: pd.Series):
return None
# Note: china_futures and global_futures have variants that prioritise
# settle/pre_settle (futures-native); see those modules for the
# futures-specific logic.
def _calc_pct_change(bar: pd.Series):
"""Calculate price change percentage from bar data."""
if "pct_chg" in bar.index:
val = bar["pct_chg"]
if pd.notna(val):
return float(val) / 100.0 # tushare pct_chg is in percentage points
def _blocked_by_limit(
engine,
symbol: str,
direction: int,
bar: pd.Series,
limit: float,
position_direction: int | None = None,
) -> bool:
"""Whether a price-limit band blocks a fill on this bar.
close = bar.get("close")
pre_close = bar.get("pre_close")
if close is not None and pre_close is not None and pre_close > 0:
return (float(close) - float(pre_close)) / float(pre_close)
return None
Shared by every engine with a daily band (A-share, India, China futures,
global futures). The band comes from a base price the market knew before
the order — ``pre_close``, else the prior bar's close — and is compared
against the price the engine would actually fill at, which is this bar's
open plus slippage.
The earlier implementation derived the day's move from the CURRENT bar's
close, which is lookahead and wrong in both directions: a name that opened
locked but drifted back by the close was allowed to trade at the locked
open, and a name that opened freely but closed limit-up was refused a fill
it would have got.
Args:
engine: Engine instance (needs the BaseEngine band helpers).
symbol: Symbol being traded.
direction: 1 (buy / open long), -1 (sell short), 0 (close).
bar: Current bar.
limit: Band half-width as a fraction (0.1 for +/-10%).
position_direction: For ``direction == 0``, the direction of the
position being closed: 1 closes a long (a sell, blocked at the
lower band), -1 closes a short (a buy, blocked at the upper band).
Defaults to a long close, which is the cash-equity case.
Returns:
True when the band blocks the fill. False when it does not, and also
when no historical base price is reachable — an unknown band must not
fabricate a block.
"""
band = engine.limit_band(symbol, bar, limit)
if band is None:
return False
lower, upper = band
# BaseEngine books a close with the OPPOSITE of the position's direction,
# so slippage moves the price the other way. Checking the raw open here
# would approve a fill that is then booked outside the band.
fill_direction = -(position_direction or 1) if direction == 0 else direction
fill = engine.prospective_fill_price(bar, fill_direction)
if fill is None:
return False
# Relative tolerance: a fill within a rounding step of the band counts as
# touching it, matching the old check's 0.1pp slack in percentage terms.
tol = 1e-9 * max(abs(lower), abs(upper), 1.0)
buying = direction == 1 or (direction == 0 and position_direction == -1)
if buying:
return fill >= upper - tol
return fill <= lower + tol
def _price_limit(symbol: str) -> float:
agent/backtest/engines/china_futures.py
+24 -48
View File
@@ -16,6 +16,7 @@ import re
import pandas as pd
from backtest.engines.china_a import _blocked_by_limit
from backtest.engines.futures_base import FuturesBaseEngine
@@ -151,7 +152,10 @@ class ChinaFuturesEngine(FuturesBaseEngine):
leverage = 10.0 # ~10% margin default
config = {**config, "leverage": leverage}
super().__init__(config)
# Futures bands come off the previous settlement, not the previous close.
self.base_price_fields = ("pre_settle", "pre_close")
self.slippage_rate: float = config.get("slippage", 0.0005)
self._margin_rate_override = margin_override if margin_override else None
self._commission_override = config.get("commission_override")
def can_execute(self, symbol: str, direction: int, bar: pd.Series) -> bool:
@@ -168,24 +172,20 @@ class ChinaFuturesEngine(FuturesBaseEngine):
# T+0: no same-day sell restriction
# Both long and short allowed
# Price limit check
pct_chg = _calc_pct_change(bar)
if pct_chg is not None:
product = _extract_product(symbol)
limit = _PRICE_LIMIT.get(product, _DEFAULT_PRICE_LIMIT)
if direction == 1 and pct_chg >= limit - 0.001:
return False # limit-up: can't open long / can't buy
if direction == -1 and pct_chg <= -limit + 0.001:
return False # limit-down: can't open short
if direction == 0:
pos = self.positions.get(symbol)
if pos is not None:
# Can't close long at limit-down, can't close short at limit-up
if pos.direction == 1 and pct_chg <= -limit + 0.001:
return False
if pos.direction == -1 and pct_chg >= limit - 0.001:
return False
return True
# Price limit, tested at execution time (see _blocked_by_limit).
product = _extract_product(symbol)
limit = _PRICE_LIMIT.get(product, _DEFAULT_PRICE_LIMIT)
pos = self.positions.get(symbol) if direction == 0 else None
if pos is None and direction == 0:
return True
return not _blocked_by_limit(
self,
symbol,
direction,
bar,
limit,
position_direction=pos.direction if pos is not None else None,
)
def round_size(self, raw_size: float, price: float) -> float:
"""Minimum 1 contract, integer lots only."""
@@ -243,36 +243,12 @@ class ChinaFuturesEngine(FuturesBaseEngine):
product = _extract_product(symbol)
return _MARGIN_RATE.get(product, 0.10)
def _leverage_for_symbol(self, symbol: str) -> float:
"""Derive leverage from this contract's own margin requirement."""
if self._margin_rate_override is not None:
return 1.0 / float(self._margin_rate_override)
return 1.0 / self.get_margin_rate(symbol)
# ── Helpers ──
# Note: china_a uses close/pre_close-only; global_futures prioritises
# close/pre_close before settle. This China-futures variant prefers
# settle/pre_settle because tushare reports settlement as the canonical
# daily price for domestic contracts. See those modules for the equity /
# global-futures equivalents.
def _calc_pct_change(bar: pd.Series):
"""Calculate bar price change percentage.
Priority: settle/pre_settle (futures native) > close/pre_close > pct_chg.
pct_chg from tushare is always in percentage points (0.5 = 0.5%).
"""
# Prefer settlement prices (unambiguous for futures)
settle = bar.get("settle")
pre_settle = bar.get("pre_settle")
if settle is not None and pre_settle is not None and pre_settle > 0:
return (float(settle) - float(pre_settle)) / float(pre_settle)
close = bar.get("close")
pre_close = bar.get("pre_close")
if close is not None and pre_close is not None and pre_close > 0:
return (float(close) - float(pre_close)) / float(pre_close)
# tushare pct_chg is in percentage points (e.g. 0.5 = 0.5%)
if "pct_chg" in bar.index:
val = bar["pct_chg"]
if pd.notna(val):
return float(val) / 100.0
return None
agent/backtest/engines/composite.py
+77 -8
View File
@@ -16,6 +16,7 @@ from backtest.engines.base import BaseEngine
from backtest.engines._market_hooks import (
_detect_market,
_is_china_futures,
code_currency,
calc_crypto_funding_fee,
check_crypto_liquidation,
calc_forex_swap,
@@ -37,6 +38,12 @@ def _build_rule_engines(config: dict, codes: List[str]) -> Dict[str, BaseEngine]
elif market == "hk_equity":
from backtest.engines.global_equity import GlobalEquityEngine
engines["hk_equity"] = GlobalEquityEngine(config, market="hk")
elif market == "india_equity":
from backtest.engines.india_equity import IndiaEquityEngine
engines["india_equity"] = IndiaEquityEngine(config)
elif market == "kr_equity":
from backtest.engines.korea_equity import KoreaEquityEngine
engines["kr_equity"] = KoreaEquityEngine(config)
elif market == "crypto":
from backtest.engines.crypto import CryptoEngine
engines["crypto"] = CryptoEngine(config)
@@ -44,19 +51,51 @@ def _build_rule_engines(config: dict, codes: List[str]) -> Dict[str, BaseEngine]
from backtest.engines.forex import ForexEngine
engines["forex"] = ForexEngine(config)
elif market == "futures":
china = any(
_is_china_futures(c) for c in codes if _detect_market(c) == "futures"
)
if china:
futures_codes = [c for c in codes if _detect_market(c) == "futures"]
if any(_is_china_futures(c) for c in futures_codes):
from backtest.engines.china_futures import ChinaFuturesEngine
engines["futures"] = ChinaFuturesEngine(config)
else:
engines["china_futures"] = ChinaFuturesEngine(config)
if any(not _is_china_futures(c) for c in futures_codes):
from backtest.engines.global_futures import GlobalFuturesEngine
engines["futures"] = GlobalFuturesEngine(config)
engines["global_futures"] = GlobalFuturesEngine(config)
return engines
def _reject_mixed_currency(codes: List[str]) -> None:
"""Refuse a code set whose members do not settle in one currency.
The shared capital pool holds a single scalar of cash and sums position
values into a single equity curve. With codes from two currency zones that
curve adds CNY to USD to KRW as if the units matched, and every metric
derived from it — return, Sharpe, drawdown — is meaningless. There is no FX
translation layer yet, so this fails closed rather than reporting a number
that looks fine.
Args:
codes: Instrument codes for the backtest.
Raises:
ValueError: If the codes span more than one settlement currency.
"""
by_currency: Dict[str, List[str]] = {}
for code in codes:
by_currency.setdefault(code_currency(code), []).append(code)
if len(by_currency) <= 1:
return
breakdown = "; ".join(
f"{currency}: {', '.join(sorted(members))}"
for currency, members in sorted(by_currency.items())
)
raise ValueError(
"composite backtest requires one settlement currency across all codes, "
f"but got {len(by_currency)} — {breakdown}. The shared capital pool has "
"no FX translation, so a mixed-currency equity curve would sum "
"different units. Split the run by currency, or convert the inputs "
"to one currency before loading."
)
class CompositeEngine(BaseEngine):
"""Cross-market engine with shared capital pool.
@@ -84,10 +123,37 @@ class CompositeEngine(BaseEngine):
# Forex dedup state
self._last_swap_dates: dict = {}
def run_backtest(self, config: dict, *args, **kwargs):
"""Run the pipeline, refusing a code set that spans currencies.
The check lives here rather than in ``__init__`` because the damage is
in the shared equity curve, not in constructing the rule-book engines.
Args:
config: Backtest configuration dict.
*args: Forwarded to :meth:`BaseEngine.run_backtest`.
**kwargs: Forwarded to :meth:`BaseEngine.run_backtest`.
Returns:
The metrics dictionary from :meth:`BaseEngine.run_backtest`.
Raises:
ValueError: If the codes span more than one settlement currency.
"""
_reject_mixed_currency(config.get("codes") or list(self._symbol_market))
return super().run_backtest(config, *args, **kwargs)
def _rule_for(self, symbol: str) -> BaseEngine:
"""Get the sub-engine that provides rules for this symbol."""
market = self._symbol_market.get(symbol, "a_share")
return self._rule_engines[market]
if market == "futures":
market = "china_futures" if _is_china_futures(symbol) else "global_futures"
engine = self._rule_engines.get(market)
if engine is None:
if not self._rule_engines:
raise ValueError("No sub-engines available for composite backtest")
engine = next(iter(self._rule_engines.values()))
return engine
# ── Stateless method dispatch ──
@@ -152,6 +218,9 @@ class CompositeEngine(BaseEngine):
) -> float:
return self._rule_for(symbol)._calc_raw_size(symbol, target_notional, price)
def _leverage_for_symbol(self, symbol: str) -> float:
return self._rule_for(symbol)._leverage_for_symbol(symbol)
# ── Stateful hooks (implemented directly, NO delegation) ──
def on_bar(self, symbol: str, bar: pd.Series, timestamp: pd.Timestamp) -> None:
agent/backtest/engines/crypto.py
+410 -2
View File
@@ -10,6 +10,9 @@ Market rules:
from __future__ import annotations
from pathlib import Path
from typing import Any
import pandas as pd
from backtest.engines.base import BaseEngine
@@ -17,6 +20,21 @@ from backtest.engines._market_hooks import (
calc_crypto_funding_fee,
check_crypto_liquidation,
)
from backtest.perpetual_evidence import (
SCHEMA_VERSION,
build_perpetual_summary,
write_perpetual_evidence,
)
from backtest.perpetual_risk import (
AccountState,
CrossMarginRiskModel,
ExecutionFrame,
MaintenanceSchedule,
MarketRiskFrame,
PositionState,
RiskSnapshot,
evaluate_isolated,
)
class CryptoEngine(BaseEngine):
@@ -37,12 +55,57 @@ class CryptoEngine(BaseEngine):
self.taker_rate: float = config.get("taker_rate", 0.0005)
self.slippage_rate: float = config.get("slippage", 0.0005)
self.funding_rate: float = config.get("funding_rate", 0.0001)
self.perpetual_strict = bool(config.get("perpetual_strict", False))
self.funding_mode = str(config.get("funding_mode", "fixed"))
self.margin_mode = str(config.get("margin_mode", "isolated"))
self.liquidation_fee_rate = float(config.get("liquidation_fee_rate", 0.0))
if self.perpetual_strict and self.funding_mode != "data":
raise ValueError("perpetual_strict requires funding_mode='data'")
if self.perpetual_strict and self.margin_mode not in {"isolated", "cross"}:
raise ValueError("margin_mode must be 'isolated' or 'cross'")
if self.perpetual_strict and not 0 <= self.liquidation_fee_rate < 1:
raise ValueError("liquidation_fee_rate must be between zero and one")
self._validate_strict_resolution(config)
self.terminal_status = "active"
self._perpetual_events: list[dict[str, Any]] = []
self._run_interval = str(config.get("interval", "1D"))
self._maintenance_bracket_versions: dict[str, str] = {}
self._market_risk_sources: set[str] = set()
self._liquidation_price_source: str | None = None
self._strict_funding_applied: set[tuple[str, pd.Timestamp]] = set()
self._isolated_margins: dict[str, float] = {}
self._schedule_cache: dict[tuple[str, str], MaintenanceSchedule] = {}
self._risk_frames: dict[str, MarketRiskFrame] = {}
self._blocked_symbols: set[str] = set()
self._funding_applied: set = set() # (symbol, date, hour) — per-slot dedup
self._funding_daily_done: set = set() # (symbol, date) — daily fallback dedup
def _validate_strict_resolution(self, config: dict[str, Any]) -> None:
if not self.perpetual_strict or self.default_leverage < 100:
return
interval = str(config.get("interval", "1D")).strip().lower()
if interval not in {"1m", "5m", "15m", "30m", "1h"}:
raise ValueError(
"strict 100x requires a supported interval <= 1H "
"(1m/5m/15m/30m/1H); 1H is only a resolution boundary, "
"not a liquidation-sequence guarantee"
)
def run_backtest(
self,
config: dict[str, Any],
loader: Any,
signal_engine: Any,
run_dir: Path,
bars_per_year: int = 252,
) -> dict[str, Any]:
self._validate_strict_resolution(config)
self._run_interval = str(config.get("interval", "1D"))
return super().run_backtest(config, loader, signal_engine, run_dir, bars_per_year)
def can_execute(self, symbol: str, direction: int, bar: pd.Series) -> bool:
"""Crypto: 24/7, long/short/close all allowed."""
return True
return not (self.perpetual_strict and symbol in self._blocked_symbols)
def round_size(self, raw_size: float, price: float) -> float:
"""Crypto supports fractional sizes, round to 6 decimals."""
@@ -54,13 +117,358 @@ class CryptoEngine(BaseEngine):
``_direction`` is unused — reserved for future funding-rate asymmetry
between long/short legs on perp swaps.
"""
rate = self.taker_rate if is_open else self.maker_rate
rate = self.taker_rate if self.perpetual_strict or is_open else self.maker_rate
return size * price * rate
def apply_slippage(self, price: float, direction: int) -> float:
"""Slippage: unfavourable direction."""
return price * (1 + direction * self.slippage_rate)
def execution_open(self, bar: pd.Series) -> float:
if not self.perpetual_strict:
return super().execution_open(bar)
return ExecutionFrame(bar.name, float(bar["execution_open"])).execution_open
def valuation_open(self, bar: pd.Series) -> float:
if not self.perpetual_strict:
return super().valuation_open(bar)
return float(bar["mark_open"])
def _schedule(self, symbol: str, bar: pd.Series) -> MaintenanceSchedule:
version = bar["maintenance_bracket_version"]
if pd.isna(version):
raise ValueError(f"missing maintenance bracket version for {symbol}")
version = str(version)
previous = self._maintenance_bracket_versions.setdefault(symbol, version)
if previous != version:
raise ValueError(f"maintenance bracket version changed for {symbol}")
key = (symbol, version)
if key not in self._schedule_cache:
self._schedule_cache[key] = MaintenanceSchedule.from_loader_columns(
symbol, bar["maintenance_brackets"], version
)
return self._schedule_cache[key]
def _build_strict_frames(
self,
timestamp: pd.Timestamp,
data_map: dict[str, pd.DataFrame],
codes: list[str],
) -> None:
risks: dict[str, MarketRiskFrame] = {}
for symbol in codes:
frame = data_map.get(symbol)
if frame is None or timestamp not in frame.index:
raise ValueError(f"missing synchronized frame for {symbol} at {timestamp}")
bar = frame.loc[timestamp]
if not isinstance(bar, pd.Series):
raise ValueError(f"duplicate frame timestamp for {symbol} at {timestamp}")
ExecutionFrame(timestamp, float(bar["execution_open"]))
rate = bar["funding_rate"]
settlement = bar["funding_settlement_time"]
if pd.isna(rate):
raise ValueError(f"missing funding rate for {symbol} at {timestamp}")
if pd.isna(settlement):
if float(rate) != 0.0:
raise ValueError(
f"funding rate without settlement for {symbol} at {timestamp}"
)
funding_rate = funding_time = None
else:
funding_rate = float(rate)
funding_time = pd.Timestamp(settlement)
risks[symbol] = MarketRiskFrame(
timestamp=timestamp,
mark_open=float(bar["mark_open"]),
mark_high=float(bar["mark_high"]),
mark_low=float(bar["mark_low"]),
mark_close=float(bar["mark_close"]),
funding_rate=funding_rate,
funding_settlement_time=funding_time,
schedule=self._schedule(symbol, bar),
source=str(self.config.get("market_risk_source", "ccxt:binanceusdm")),
)
self._market_risk_sources.add(risks[symbol].source)
self._risk_frames = risks
def _record_event(self, timestamp: pd.Timestamp, event_type: str, **details: Any) -> None:
if not self.perpetual_strict:
return
self._perpetual_events.append(
{
"schema_version": SCHEMA_VERSION,
"sequence": len(self._perpetual_events) + 1,
"timestamp": pd.Timestamp(timestamp).isoformat(),
"event_type": event_type,
**details,
}
)
def _record_risk_snapshot(
self,
timestamp: pd.Timestamp,
price_field: str,
snapshot: RiskSnapshot,
) -> None:
self._record_event(
timestamp,
"risk_snapshot",
phase="pre_fill" if price_field == "mark_open" else "post_fill",
price_source=("mark_open" if price_field == "mark_open" else "adverse_mark_extrema"),
status=snapshot.status,
margin_balance=snapshot.margin_balance,
initial_margin=snapshot.initial_margin,
maintenance_margin=snapshot.maintenance_margin,
available_balance=snapshot.available_balance,
liquidation_targets=list(snapshot.liquidation_targets),
fidelity_flags=list(snapshot.fidelity_flags),
positions=[
{
"symbol": risk.symbol,
"mark_price": risk.mark_price,
"notional": risk.notional,
"unrealized_pnl": risk.unrealized_pnl,
"initial_margin": risk.initial_margin,
"maintenance_margin": risk.maintenance_margin,
"margin_balance": risk.margin_balance,
}
for risk in snapshot.per_position
],
)
def _apply_data_funding(self) -> None:
for symbol, position in self.positions.items():
frame = self._risk_frames[symbol]
settlement = frame.funding_settlement_time
if settlement is None or position.entry_time >= settlement:
continue
key = (symbol, settlement)
if key in self._strict_funding_applied:
continue
payment = (
position.direction * position.size * frame.mark_open
* float(frame.funding_rate)
)
self.capital -= payment
if self.margin_mode == "isolated":
self._isolated_margins[symbol] -= payment
self._strict_funding_applied.add(key)
self._record_event(
settlement,
"funding_settlement",
symbol=symbol,
signed_quantity=position.direction * position.size,
mark_price=frame.mark_open,
price_source="mark_open",
funding_rate=frame.funding_rate,
funding_pnl=-payment,
)
def _account_state(self) -> AccountState:
positions = tuple(
PositionState(
symbol=symbol,
quantity=position.direction * position.size,
entry_price=position.entry_price,
leverage=position.leverage,
accumulated_entry_fee=position.entry_commission,
isolated_margin=self._isolated_margins.get(symbol),
)
for symbol, position in self.positions.items()
)
locked_margin = sum(
self._calc_margin(
position.symbol, position.size, position.entry_price, position.leverage
)
for position in self.positions.values()
)
return AccountState(
wallet_balance=self.capital + locked_margin,
positions=positions,
margin_mode=self.margin_mode,
terminal_status=self.terminal_status,
)
def _evaluate_and_liquidate(
self, timestamp: pd.Timestamp, price_field: str
) -> bool:
if not self.positions:
return False
account = self._account_state()
snapshot = (
evaluate_isolated(account, self._risk_frames, price_field)
if self.margin_mode == "isolated"
else CrossMarginRiskModel().evaluate(account, self._risk_frames, price_field)
)
self._record_risk_snapshot(timestamp, price_field, snapshot)
if snapshot.status == "healthy":
return False
prices = {risk.symbol: risk.mark_price for risk in snapshot.per_position}
price_source = "mark_open" if price_field == "mark_open" else "adverse_mark_extrema"
liquidation_details = []
for symbol in snapshot.liquidation_targets:
position = self.positions[symbol]
price = prices[symbol]
fee = position.size * price * self.liquidation_fee_rate
self._liquidation_price_source = price_source
try:
self._close_position(symbol, price, timestamp, snapshot.status)
self.capital -= fee
finally:
self._liquidation_price_source = None
liquidation_details.append((symbol, price, fee))
if snapshot.status == "account_liquidation":
self.terminal_status = "account_liquidation"
self._record_event(
timestamp,
"account_liquidation",
symbols=sorted(symbol for symbol, _, _ in liquidation_details),
liquidation_prices={symbol: price for symbol, price, _ in liquidation_details},
price_source=price_source,
liquidation_fee=sum(fee for _, _, fee in liquidation_details),
fidelity_flags=list(snapshot.fidelity_flags),
)
return True
for symbol, price, fee in liquidation_details:
self._record_event(
timestamp,
"position_liquidation",
symbol=symbol,
liquidation_price=price,
price_source=price_source,
liquidation_fee=fee,
fidelity_flags=list(snapshot.fidelity_flags),
)
self._blocked_symbols.update(snapshot.liquidation_targets)
return False
def before_rebalance_bar(
self,
timestamp: pd.Timestamp,
data_map: dict[str, pd.DataFrame],
codes: list[str],
) -> bool:
if not self.perpetual_strict:
return super().before_rebalance_bar(timestamp, data_map, codes)
self._blocked_symbols.clear()
self._build_strict_frames(timestamp, data_map, codes)
self._apply_data_funding()
return self._evaluate_and_liquidate(timestamp, "mark_open")
def after_rebalance_bar(
self,
timestamp: pd.Timestamp,
data_map: dict[str, pd.DataFrame],
codes: list[str],
) -> bool:
if not self.perpetual_strict:
return super().after_rebalance_bar(timestamp, data_map, codes)
return self._evaluate_and_liquidate(timestamp, "adverse")
def _execute_bars(self, dates, data_map, close_df, target_pos, codes) -> None:
if self.perpetual_strict:
try:
close_df = pd.DataFrame(
{symbol: data_map[symbol]["mark_close"] for symbol in codes}, index=dates
)
except KeyError as exc:
raise ValueError(f"missing strict mark-close data: {exc}") from exc
super()._execute_bars(dates, data_map, close_df, target_pos, codes)
if self.perpetual_strict and self.terminal_status == "active":
self.terminal_status = "completed"
def _execute_open_order(self, order, timestamp: pd.Timestamp) -> None:
super()._execute_open_order(order, timestamp)
if self.perpetual_strict:
self._record_event(
timestamp,
"market_fill",
action="open",
symbol=order.symbol,
side="buy" if order.direction == 1 else "sell",
signed_quantity=order.direction * order.size,
execution_price=order.price,
execution_price_source="execution_open",
trading_fee=order.commission,
reason="signal",
)
if self.perpetual_strict and self.margin_mode == "isolated":
self._isolated_margins[order.symbol] = order.margin
def _close_position(
self,
symbol: str,
exit_price: float,
exit_time: pd.Timestamp,
reason: str,
) -> None:
position = self.positions.get(symbol)
super()._close_position(symbol, exit_price, exit_time, reason)
if self.perpetual_strict and position is not None:
trade = self.trades[-1]
price_source = self._liquidation_price_source
if price_source is None:
price_source = "mark_close" if reason == "end_of_backtest" else "execution_open"
self._record_event(
exit_time,
"market_fill",
action="close",
symbol=symbol,
side="sell" if position.direction == 1 else "buy",
signed_quantity=-position.direction * position.size,
execution_price=exit_price,
execution_price_source=price_source,
trading_fee=trade.commission - position.entry_commission,
reason=reason,
)
self._isolated_margins.pop(symbol, None)
def _write_artifacts(
self,
run_dir: Path,
data_map: dict[str, pd.DataFrame],
dates: pd.DatetimeIndex,
equity_series: pd.Series,
bench_equity: pd.Series,
bench_ret: pd.Series,
target_pos: pd.DataFrame,
metrics: dict,
codes: list[str],
) -> None:
summary = None
if self.perpetual_strict:
summary = build_perpetual_summary(
config={**self.config, "interval": self._run_interval},
events=self._perpetual_events,
trades=self.trades,
terminal_status=self.terminal_status,
maintenance_bracket_versions=self._maintenance_bracket_versions,
market_risk_sources=sorted(self._market_risk_sources),
)
metrics.update(
{
"perpetual_funding_settlements": summary["funding_settlement_count"],
"perpetual_funding_pnl": summary["total_funding_pnl"],
"perpetual_liquidation_events": summary["liquidation_event_count"],
"perpetual_liquidated_positions": summary["liquidated_position_count"],
"perpetual_trading_fees": summary["total_trading_fee"],
"perpetual_liquidation_fees": summary["total_liquidation_fee"],
}
)
super()._write_artifacts(
run_dir,
data_map,
dates,
equity_series,
bench_equity,
bench_ret,
target_pos,
metrics,
codes,
)
if summary is not None:
write_perpetual_evidence(run_dir, self._perpetual_events, summary)
def on_bar(self, symbol: str, bar: pd.Series, timestamp: pd.Timestamp) -> None:
"""Crypto per-bar hooks: funding fee + liquidation check."""
fee = calc_crypto_funding_fee(
agent/backtest/engines/global_equity.py
+3
View File
@@ -11,6 +11,9 @@ Market rules:
- Stamp tax 0.1% bilateral + levies
- Lot-size rounding (simplified to 100 shares)
- Higher slippage than US
India (NSE/BSE) is handled by the dedicated ``backtest.engines.india_equity``
``IndiaEquityEngine`` (T+1 delivery, circuit bands, STT/stamp/GST stack).
"""
from __future__ import annotations
agent/backtest/engines/global_futures.py
+15 -44
View File
@@ -16,6 +16,7 @@ import re
import pandas as pd
from backtest.engines.china_a import _blocked_by_limit
from backtest.engines.futures_base import FuturesBaseEngine
@@ -141,6 +142,8 @@ class GlobalFuturesEngine(FuturesBaseEngine):
leverage = config.get("leverage", 10.0)
config = {**config, "leverage": leverage}
super().__init__(config)
# Futures bands come off the previous settlement, not the previous close.
self.base_price_fields = ("pre_settle", "pre_close")
self.slippage_rate: float = config.get("slippage", 0.0003)
self._comm_override = config.get("commission_per_contract")
@@ -160,20 +163,18 @@ class GlobalFuturesEngine(FuturesBaseEngine):
if limit is None:
return True # no price limit for most commodities
pct_chg = _calc_pct_change(bar)
if pct_chg is not None:
if direction == 1 and pct_chg >= limit - 0.001:
return False # limit-up
if direction == -1 and pct_chg <= -limit + 0.001:
return False # limit-down
if direction == 0:
pos = self.positions.get(symbol)
if pos is not None:
if pos.direction == 1 and pct_chg <= -limit + 0.001:
return False
if pos.direction == -1 and pct_chg >= limit - 0.001:
return False
return True
# Tested at execution time (see _blocked_by_limit).
pos = self.positions.get(symbol) if direction == 0 else None
if pos is None and direction == 0:
return True
return not _blocked_by_limit(
self,
symbol,
direction,
bar,
limit,
position_direction=pos.direction if pos is not None else None,
)
def round_size(self, raw_size: float, price: float) -> float:
"""Integer contracts, minimum 1."""
@@ -219,33 +220,3 @@ class GlobalFuturesEngine(FuturesBaseEngine):
# ── Helpers ──
# Note: china_a uses close/pre_close-only; china_futures prioritises
# settle/pre_settle. This global-futures variant prefers close/pre_close
# because CME data feeds (yfinance/IB) expose continuous close more
# reliably than settlement. See those modules for the equity /
# China-futures equivalents.
def _calc_pct_change(bar: pd.Series):
"""Calculate bar price change percentage.
Priority: close/pre_close > settle/pre_settle > pct_chg.
Falls back to pct_chg only when price fields are absent.
"""
close = bar.get("close")
pre_close = bar.get("pre_close")
if close is not None and pre_close is not None and pre_close > 0:
return (float(close) - float(pre_close)) / float(pre_close)
settle = bar.get("settle")
pre_settle = bar.get("pre_settle")
if settle is not None and pre_settle is not None and pre_settle > 0:
return (float(settle) - float(pre_settle)) / float(pre_settle)
if "pct_chg" in bar.index:
val = bar["pct_chg"]
if pd.notna(val):
raw = float(val)
# Heuristic: values > 1 are likely percentage points
return raw / 100.0 if abs(raw) > 1.0 else raw
return None
agent/backtest/engines/india_equity.py
+141
View File
@@ -0,0 +1,141 @@
"""India equity (NSE / BSE) backtest engine.
Models the Indian cash-equity **delivery** segment on daily bars. Intraday
(MIS) mechanics are not represented by a daily-bar engine, so the defaults
reflect overnight delivery rules; the knobs below let advanced users approximate
intraday behaviour.
Market rules:
- T+1 settlement: shares bought today cannot be sold the same bar (delivery).
- No short selling by default: retail cannot hold overnight short delivery
positions. Set ``allow_short=True`` to model intraday (MIS) shorting.
- Circuit bands: per-scrip price bands vary (2/5/10/20%); the exact band is
not derivable from the symbol alone, so a single configurable band applies
(default ±20%, the widest common band). Set ``price_limit`` to ``0`` /
``None`` to disable.
- Lot size: 1 share for cash equity (F&O lot sizes are not modelled here).
Cost stack (delivery, discount-broker defaults; all config-driven). NOTE: SEBI/
exchange tariffs change periodically — verify ``in_*`` rates against a current
broker schedule before relying on absolute cost figures:
- Brokerage: ₹0 (delivery on discount brokers) [in_brokerage]
- STT: 0.1% on buy + 0.1% on sell (bilateral) [in_stt]
- Exchange transaction charge: NSE ~0.00297% (bilateral) [in_exchange_txn]
- SEBI turnover fee: ₹10/crore = 0.0001% (bilateral) [in_sebi_fee]
- Stamp duty: 0.015% on buy only [in_stamp_duty]
- GST: 18% on (brokerage + exchange txn + SEBI fee) [in_gst]
- DP charge: flat per-scrip on sell (default ₹0) [in_dp_charge]
"""
from __future__ import annotations
import pandas as pd
from backtest.engines.base import BaseEngine
from backtest.engines.china_a import _blocked_by_limit
class IndiaEquityEngine(BaseEngine):
"""NSE / BSE cash-equity (delivery) engine.
Config keys (all optional; defaults shown in the module docstring):
- allow_short: bool, default False
- price_limit: float fraction or None, default 0.20
- slippage: default 0.001
- in_brokerage / in_stt / in_exchange_txn / in_sebi_fee /
in_stamp_duty / in_gst / in_dp_charge
"""
def __init__(self, config: dict):
config = {**config, "leverage": 1.0} # cash delivery: no leverage
super().__init__(config)
self.allow_short: bool = bool(config.get("allow_short", False))
self.price_limit = config.get("price_limit", 0.20)
self.slippage_rate: float = config.get("slippage", 0.001)
# Cost stack
self.in_brokerage: float = config.get("in_brokerage", 0.0)
self.in_stt: float = config.get("in_stt", 0.001)
self.in_exchange_txn: float = config.get("in_exchange_txn", 0.0000297)
self.in_sebi_fee: float = config.get("in_sebi_fee", 0.000001)
self.in_stamp_duty: float = config.get("in_stamp_duty", 0.00015)
self.in_gst: float = config.get("in_gst", 0.18)
self.in_dp_charge: float = config.get("in_dp_charge", 0.0)
def can_execute(self, symbol: str, direction: int, bar: pd.Series) -> bool:
"""India delivery execution rules.
Args:
symbol: NSE/BSE symbol (e.g. ``RELIANCE.NS``).
direction: 1 (buy), -1 (short), 0 (sell/close).
bar: Current bar (needs ``close`` + ``pre_close``/``pct_chg`` for
circuit checks).
Returns:
True if the trade is allowed.
"""
# 1. Short selling: blocked unless explicitly modelling intraday (MIS).
if direction == -1 and not self.allow_short:
return False
# 2. T+1: can't sell shares bought today (delivery).
if direction == 0:
pos = self.positions.get(symbol)
if pos is not None:
bar_date = _bar_date(bar)
entry_date = pos.entry_time.date() if hasattr(pos.entry_time, "date") else None
if bar_date is not None and entry_date is not None and bar_date == entry_date:
return False
# 3. Circuit bands, tested at execution time (see _blocked_by_limit).
if self.price_limit and _blocked_by_limit(
self, symbol, direction, bar, float(self.price_limit)
):
return False
return True
def round_size(self, raw_size: float, price: float) -> float:
"""Cash equity trades in 1-share lots."""
return float(max(int(raw_size), 0))
def calc_commission(self, size: float, price: float, _direction: int, is_open: bool) -> float:
"""India delivery cost stack (see module docstring).
``_direction`` is unused — reserved for future asymmetric long/short
(intraday MIS) schedules.
"""
notional = size * price
brokerage = notional * self.in_brokerage
exchange_txn = notional * self.in_exchange_txn # bilateral
sebi_fee = notional * self.in_sebi_fee # bilateral
gst = (brokerage + exchange_txn + sebi_fee) * self.in_gst
stt = notional * self.in_stt # bilateral (delivery)
comm = brokerage + exchange_txn + sebi_fee + gst + stt
if is_open:
comm += notional * self.in_stamp_duty # stamp duty: buy-only
else:
comm += self.in_dp_charge # DP charge: sell-only, flat
return comm
def apply_slippage(self, price: float, direction: int) -> float:
"""India slippage (configurable)."""
return price * (1 + direction * self.slippage_rate)
# ── Helpers ──
def _bar_date(bar: pd.Series):
"""Extract date from bar, handling various column names."""
for col in ("trade_date", "date"):
if col in bar.index:
val = bar[col]
if hasattr(val, "date"):
return val.date()
try:
return pd.Timestamp(val).date()
except Exception:
pass
if hasattr(bar, "name") and hasattr(bar.name, "date"):
return bar.name.date()
return None
agent/backtest/engines/korea_equity.py
+316
View File
@@ -0,0 +1,316 @@
"""Korea equity (KRX: KOSPI / KOSDAQ) backtest engine.
Models the Korean cash-equity segment on daily bars. Prices are assumed to be
quoted in KRW, which KRX prints as whole won on a coarse tick grid — the grid is
modelled explicitly below because a percentage-only model produces fill prices
that cannot exist and mis-identifies locked-limit bars.
Market rules:
- Same-day round trips are allowed: shares bought today CAN be sold the same
day (unlike China A-share T+1 or India delivery), so there is no same-bar
sell block.
- **Long-only by construction.** KRX short selling is covered-only and
uptick-constrained (차입공매도 / 업틱룰), and neither constraint is
representable on daily bars, so ``allow_short=True`` is *refused* at
construction rather than silently mis-modelled.
- Price limit: ±30% daily band on both KOSPI and KOSDAQ (since June 2015),
measured from the base price (기준가격 — normally the previous trading
day's close) and quantized to the tick grid. Set ``price_limit`` to ``0`` /
``None`` to disable.
- Lot size: 1 share (KRX abolished the 10-share odd-lot rule in 2014).
Cost stack (discount-broker defaults; all config-driven):
- Brokerage: 0.015% per side (typical online rate) [kr_brokerage]
- Securities transaction tax: 0.20% on the SELL side [kr_tax_sell]
From transfers on/after 2026-01-01 the aggregate sell-side rate is 0.20%
on both boards, restored after the financial-investment-income tax was
scrapped: KOSPI = 0.05% 증권거래세 + 0.15% 농어촌특별세, KOSDAQ = 0.20%
증권거래세 with no surtax (증권거래세법 시행령 제5조 탄력세율 as amended
by 대통령령 제36001호; 농어촌특별세법 제5조). KONEX stays at 0.10% — pass
``kr_tax_sell=0.001`` for KONEX names. Rates are revised periodically;
verify ``kr_*`` against the current schedule before trusting absolute cost
figures.
Tick grid source: KRX regulation portal (호가가격단위) —
https://regulation.krx.co.kr/contents/RGL/03/03010100/RGL03010100T3.jsp (KOSPI)
and https://regulation.krx.co.kr/contents/RGL/03/03020100/RGL03020100.jsp
(KOSDAQ). The 2023-01-25 reform **unified** the two boards' tables and removed
the old high-price divergence, so one table now serves both. Note that KRX's
English "Guide to Trading in the Korean Stock Market" PDF still prints the
pre-2023 bands (1,000 / 10,000 / 100,000 boundaries, KOSDAQ capped at 100 KRW)
— it is stale; the Korean regulation portal above is authoritative.
Price-limit arithmetic follows the published KRX procedure rather than a naive
``base * 1.3``: the 30% band amount is truncated to the *base price's* tick
unit, then added to / subtracted from the base, then truncated again to the
resulting price's own tick unit. KRX worked example (기준가격 9,980):
9,980 x 0.3 = 2,994 -> 2,990 -> upper 12,970 / lower 6,990.
"""
from __future__ import annotations
import logging
from typing import Optional
import pandas as pd
from backtest.engines.base import BaseEngine
logger = logging.getLogger(__name__)
# KRX quotation price units (호가가격단위): (lower bound inclusive, tick).
# Unified across KOSPI / KOSDAQ / KONEX since 2023-01-25. Every coarser tick is
# an exact multiple of every finer one, which is what makes the truncation
# arithmetic below stable across band boundaries.
KRX_TICK_TABLE: tuple[tuple[float, float], ...] = (
(0.0, 1.0),
(2_000.0, 5.0),
(5_000.0, 10.0),
(20_000.0, 50.0),
(50_000.0, 100.0),
(200_000.0, 500.0),
(500_000.0, 1_000.0),
)
# Tick-grid comparisons only need to survive float representation error: every
# quantized price is a whole number of won, so 1e-9 is far below one tick.
_PRICE_EPS = 1e-9
def krx_tick_size(price: float) -> float:
"""Return the KRX tick unit applicable at *price*.
Args:
price: Price in KRW. Non-positive input returns the finest tick.
Returns:
The tick unit (KRW) for the band containing *price*; bands are
lower-bound inclusive (``이상`` / ``미만``), so 2,000 KRW -> 5 KRW.
"""
tick = KRX_TICK_TABLE[0][1]
for lower, unit in KRX_TICK_TABLE:
if price >= lower:
tick = unit
else:
break
return tick
def krx_round_down(price: float) -> float:
"""Truncate *price* down onto the tick grid (KRX ``절사``)."""
tick = krx_tick_size(price)
return float(int((price + _PRICE_EPS) / tick) * tick)
def krx_round_up(price: float) -> float:
"""Round *price* up onto the tick grid (KRX ``절상``), never below one tick."""
tick = krx_tick_size(price)
steps = int((price - _PRICE_EPS) / tick)
if steps * tick < price - _PRICE_EPS:
steps += 1
return float(max(steps, 1) * tick)
def krx_price_limits(base_price: float, limit: float) -> tuple[float, float]:
"""Return the (upper, lower) KRX limit prices for a base price.
Implements the published three-step procedure: the band amount is
``base * limit`` truncated to the base price's tick unit, and each limit
price is then truncated to its own tick unit (the lower limit is already
grid-aligned because coarser ticks are multiples of finer ones, so only the
upper limit can move).
Args:
base_price: Base price (기준가격), normally the previous close, already
snapped onto the tick grid by the caller.
limit: Band width as a fraction (0.30 for the standard ±30%).
Returns:
``(upper_limit, lower_limit)`` in KRW.
"""
base_tick = krx_tick_size(base_price)
band_amount = max(int((base_price * limit + _PRICE_EPS) / base_tick), 1) * base_tick
return (
krx_round_down(base_price + band_amount),
krx_round_down(max(base_price - band_amount, base_tick)),
)
class KoreaEquityEngine(BaseEngine):
"""KRX (KOSPI / KOSDAQ) cash-equity engine — long-only.
Config keys (all optional; defaults shown in the module docstring):
- price_limit: float fraction or None, default 0.30
- slippage: default 0.001
- kr_brokerage / kr_tax_sell
Raises:
ValueError: When ``allow_short`` is truthy. KRX covered-short and
uptick rules are not modelled, so shorting is refused instead of
being simulated incorrectly. A cross-market composite run shares one
config across its sub-engines, so pairing ``allow_short`` with a KRX
symbol fails the whole run — deliberately, since the alternative is
a KRX leg that silently never shorts while the rest of the book
does.
"""
def __init__(self, config: dict):
config = {**config, "leverage": 1.0} # cash equity: no leverage
super().__init__(config)
if config.get("allow_short"):
raise ValueError(
"KoreaEquityEngine is long-only: KRX short selling is "
"covered-only (차입공매도) and uptick-constrained, and neither "
"rule can be enforced on daily bars. Remove 'allow_short' "
"instead of running a mis-modelled short book."
)
self.price_limit = config.get("price_limit", 0.30)
self.slippage_rate: float = config.get("slippage", 0.001)
# Cost stack
self.kr_brokerage: float = config.get("kr_brokerage", 0.00015)
self.kr_tax_sell: float = config.get("kr_tax_sell", 0.0020)
# One warning per engine when no base price is derivable (see _base_price).
self._limit_base_warned = False
def can_execute(self, symbol: str, direction: int, bar: pd.Series) -> bool:
"""KRX execution rules.
The price-limit test compares the *prospective fill price* — this bar's
open plus slippage, which is what :class:`BaseEngine` fills at — against
limit prices derived from the previous bar's close. Deriving the move
from the current bar's close instead (the china_a/india pattern) would
be lookahead: ``BaseEngine`` calls this hook before the fill, and the
decision bar's close is not known then.
Args:
symbol: KRX symbol (e.g. ``005930.KS``, ``247540.KQ``).
direction: 1 (buy), -1 (short), 0 (sell/close).
bar: Current bar (needs ``open``; the base price comes from
``pre_close`` when the source supplies it, else from the prior
bar of the run's own close panel).
Returns:
True if the trade is allowed.
"""
# 1. Short selling: structurally unavailable (see class docstring).
if direction == -1:
return False
# 2. Same-day sell is allowed on KRX — no T+1 interception.
# 3. Daily price limit ±30% (disabled when falsy).
if not self.price_limit:
return True
base_price = self._base_price(symbol, bar)
if base_price is None:
if not self._limit_base_warned:
self._limit_base_warned = True
logger.warning(
"%s: no base price available (no 'pre_close' column and no "
"prior close panel row) — the KRX ±%.0f%% limit check is "
"inactive for this run",
symbol,
float(self.price_limit) * 100,
)
return True
open_price = float(bar.get("open", bar.get("close", 0.0)) or 0.0)
if open_price <= 0:
return True # order sizing rejects non-positive prices downstream
upper, lower = krx_price_limits(base_price, float(self.price_limit))
fill_price = self.apply_slippage(open_price, direction if direction else -1)
if direction == 1 and fill_price >= upper - _PRICE_EPS:
return False # limit-up (상한가): no ask to buy from
if direction == 0 and fill_price <= lower + _PRICE_EPS:
return False # limit-down (하한가): no bid to sell into
return True
def _base_price(self, symbol: str, bar: pd.Series) -> Optional[float]:
"""Resolve the KRX base price (기준가격) for *symbol* on this bar.
Both sources are strictly historical, so neither leaks the decision
bar's own close into the pre-fill check:
1. ``pre_close`` on the bar, when the data source supplies one.
2. The previous row of the close panel that :class:`BaseEngine`
pre-extracts for the run (unavailable in a cross-market composite
run, whose sub-engines are stateless rule books).
An off-grid close (pykrx's adjusted series is Naver-rebased and need not
land on a tick) is rounded up onto the grid, which is KRX's own rule for
a base price that falls between ticks (``절상``).
Args:
symbol: Symbol whose base price is wanted.
bar: Current bar.
Returns:
The base price in KRW, or ``None`` when no historical close is
reachable (first bar of a run, or a composite sub-engine).
"""
candidate: Optional[float] = None
if "pre_close" in bar.index:
raw = bar["pre_close"]
if pd.notna(raw) and float(raw) > 0:
candidate = float(raw)
if candidate is None:
close_arr = getattr(self, "_close_arr", None)
col = getattr(self, "_code_to_col", {}).get(symbol)
row = getattr(self, "_bar_idx", 0) - 1
if close_arr is not None and col is not None and row >= 0:
raw = close_arr[row, col]
if pd.notna(raw) and float(raw) > 0:
candidate = float(raw)
return None if candidate is None else krx_round_up(candidate)
def round_size(self, raw_size: float, price: float) -> float:
"""Cash equity trades in 1-share lots."""
return float(max(int(raw_size), 0))
def calc_commission(self, size: float, price: float, direction: int, is_open: bool) -> float:
"""Korea cost stack: bilateral brokerage + transaction tax on sells.
The tax follows the actual trade side rather than open/close: a closing
trade on a long book is a sell (taxable), while an opening short would
be the taxable leg and its buy-to-cover would not be. ``BaseEngine``
passes the position direction on both hooks, so the side is derivable
even though this engine is long-only today.
Args:
size: Share count.
price: Fill price in KRW.
direction: 1 for a long book, -1 for a short book.
is_open: True for the opening leg, False for the closing leg.
Returns:
Total cost in KRW for this leg.
"""
notional = size * price
comm = notional * self.kr_brokerage
is_sell = (direction < 0) if is_open else (direction > 0)
if is_sell:
comm += notional * self.kr_tax_sell
return comm
def apply_slippage(self, price: float, direction: int) -> float:
"""Korea slippage (configurable), quantized onto the KRX tick grid.
A slipped price is snapped away from the trader — buys up to the next
valid ask, sells down to the next valid bid — because no fill can land
between ticks. The residual rounding is bounded by one tick unit
(<= 0.25% of price across the whole table).
Args:
price: Reference price in KRW.
direction: 1 to buy, -1 to sell.
Returns:
A valid KRX price, never below one tick.
"""
slipped = price * (1 + direction * self.slippage_rate)
if direction > 0:
return krx_round_up(slipped)
return max(krx_round_down(slipped), krx_tick_size(slipped))
agent/backtest/engines/options_portfolio.py
+231 -132
View File
@@ -12,6 +12,10 @@ v2 enhancements over v1:
Signal interface: OptionsSignalEngine.generate(data_map) returns a list of trade instructions.
Artifacts: equity.csv, metrics.csv, trades.csv, greeks.csv.
Black-Scholes price and Greeks come from ``src.quantlib.options``. What stays
here is the engine's own volatility surface -- historical vol, the smile, and
the per-leg vol every pricing site must agree on.
"""
import json
@@ -21,94 +25,8 @@ from typing import Any, Dict, List, Optional
import numpy as np
import pandas as pd
from scipy.stats import norm
# --- Black-Scholes pricing ---
def bs_price(S: float, K: float, T: float, r: float, sigma: float,
option_type: str = "call") -> float:
"""Black-Scholes European option pricing.
Args:
S: Underlying spot price.
K: Strike price.
T: Time to expiry in years.
r: Risk-free rate (annualised).
sigma: Annualised volatility.
option_type: Option type, "call" or "put".
Returns:
Theoretical option price.
Example:
>>> round(bs_price(100, 100, 1.0, 0.05, 0.2, "call"), 2)
10.45
"""
if T <= 0 or sigma <= 0:
# Expired: return intrinsic value
if option_type == "call":
return max(S - K, 0.0)
return max(K - S, 0.0)
d1 = (np.log(S / K) + (r + sigma ** 2 / 2) * T) / (sigma * np.sqrt(T))
d2 = d1 - sigma * np.sqrt(T)
if option_type == "call":
return float(S * norm.cdf(d1) - K * np.exp(-r * T) * norm.cdf(d2))
return float(K * np.exp(-r * T) * norm.cdf(-d2) - S * norm.cdf(-d1))
# --- Greeks ---
def bs_greeks(S: float, K: float, T: float, r: float, sigma: float,
option_type: str = "call") -> Dict[str, float]:
"""Calculate Black-Scholes Greeks.
Args:
S: Underlying spot price.
K: Strike price.
T: Time to expiry in years.
r: Risk-free rate (annualised).
sigma: Annualised volatility.
option_type: Option type, "call" or "put".
Returns:
Dict containing delta, gamma, theta, vega.
"""
if T <= 0 or sigma <= 0:
intrinsic_call = 1.0 if S > K else 0.0
delta = intrinsic_call if option_type == "call" else intrinsic_call - 1.0
return {"delta": delta, "gamma": 0.0, "theta": 0.0, "vega": 0.0}
sqrt_T = np.sqrt(T)
d1 = (np.log(S / K) + (r + sigma ** 2 / 2) * T) / (sigma * sqrt_T)
d2 = d1 - sigma * sqrt_T
nd1_pdf = float(norm.pdf(d1))
# Delta
if option_type == "call":
delta = float(norm.cdf(d1))
else:
delta = float(norm.cdf(d1) - 1.0)
# Gamma (same for call and put)
gamma = float(nd1_pdf / (S * sigma * sqrt_T))
# Theta (daily)
theta_common = -(S * nd1_pdf * sigma) / (2 * sqrt_T)
if option_type == "call":
theta = theta_common - r * K * np.exp(-r * T) * norm.cdf(d2)
else:
theta = theta_common + r * K * np.exp(-r * T) * norm.cdf(-d2)
theta = float(theta / 365.0) # convert to daily
# Vega (per 1% change in volatility)
vega = float(S * nd1_pdf * sqrt_T / 100.0)
return {"delta": delta, "gamma": gamma, "theta": theta, "vega": vega}
from src.quantlib.options import bs_greeks, bs_price, normalise_option_type
# --- Historical volatility ---
@@ -155,6 +73,31 @@ def iv_smile_adjustment(S: float, K: float, base_iv: float,
return max(adj, 0.01)
def leg_iv(S: float, K: float, base_iv: float, skew: float, curvature: float) -> float:
"""Return the implied vol a single leg is priced at.
Every site that prices a leg must go through this — opening, marking to
market, Greeks, and the American continuation value. Opening a leg on the
smile and marking it at flat at-the-money vol books a fictitious profit the
instant the position exists: on a 30-day 10%-OTM call at ``skew=-0.15`` the
gap is +16.7% of premium, and +93.0% at 20% OTM, which then contaminates
Sharpe, Calmar and drawdown.
Args:
S: Spot price.
K: Strike price.
base_iv: At-the-money implied volatility.
skew: Slope of the smile; ``0`` with ``curvature`` disables the smile.
curvature: Curvature of the smile.
Returns:
The leg's implied volatility.
"""
if skew == 0 and curvature == 0:
return base_iv
return iv_smile_adjustment(S, K, base_iv, skew, curvature)
# --- Option positions ---
@@ -162,19 +105,24 @@ class OptionPosition:
"""A single option leg position.
Attributes:
option_type: "call" or "put".
option_type: "call" or "put", folded to lower case on construction so
that settlement here and pricing in ``src.quantlib.options`` cannot
disagree about a leg typed ``"Call"``.
strike: Strike price.
expiry: Expiry date.
qty: Quantity (positive = long, negative = short).
entry_price: Theoretical option price at entry.
entry_date: Entry date string.
underlying_code: Underlying instrument code.
Raises:
ValueError: If ``option_type`` is neither call nor put.
"""
def __init__(self, option_type: str, strike: float, expiry: str,
qty: int, entry_price: float, entry_date: str,
underlying_code: str):
self.option_type = option_type
self.option_type = normalise_option_type(option_type)
self.strike = strike
self.expiry = pd.Timestamp(expiry)
self.qty = qty
@@ -327,7 +275,10 @@ def run_options_backtest(
if T_ex <= 0:
continue
intrinsic = pos.intrinsic_value(spot)
continuation = bs_price(spot, pos.strike, T_ex, risk_free_rate, iv_val_ex, pos.option_type)
# The continuation value must use the same vol the leg is
# marked at, or early exercise triggers off a mispriced hold.
iv_ex = leg_iv(spot, pos.strike, iv_val_ex, iv_skew, iv_curvature)
continuation = bs_price(spot, pos.strike, T_ex, risk_free_rate, iv_ex, pos.option_type)
if intrinsic > 0 and intrinsic > continuation * 1.02:
# Early exercise is optimal
settlement = intrinsic * pos.qty * contract_multiplier
@@ -384,7 +335,10 @@ def run_options_backtest(
iv_val = ivs.get(underlying, 0.3)
for leg in legs:
leg_type = leg.get("type", "call")
# Fold before it is priced, matched and recorded: config comes
# from the user, and a raw "Call" would price as a call and
# settle as a put.
leg_type = normalise_option_type(leg.get("type", "call"))
strike = leg.get("strike", spot)
expiry = leg.get("expiry", "")
qty = leg.get("qty", 1)
@@ -392,12 +346,7 @@ def run_options_backtest(
expiry_ts = pd.Timestamp(expiry)
T = max((expiry_ts - ts).days / 365.0, 0.001)
# Apply IV smile adjustment (v2) if configured
adj_iv = iv_val
if iv_skew != 0 or iv_curvature != 0:
adj_iv = iv_smile_adjustment(spot, strike, iv_val, iv_skew, iv_curvature)
# Black-Scholes price (with smile-adjusted IV if enabled)
adj_iv = leg_iv(spot, strike, iv_val, iv_skew, iv_curvature)
opt_price = bs_price(spot, strike, T, risk_free_rate, adj_iv, leg_type)
if action == "open":
@@ -432,12 +381,26 @@ def run_options_backtest(
})
elif action == "close":
# Close: find matching position
# Close: find matching position, honoring a partial-close qty.
matched = _find_matching_position(
positions, underlying, leg_type, strike, expiry)
if matched:
pnl = (opt_price - matched.entry_price) * matched.qty * contract_multiplier
abs_close = opt_price * abs(matched.qty) * contract_multiplier
# An explicit leg ``qty`` closes only that many contracts
# (clamped to the open size); a close leg with no ``qty``
# closes the whole lot (legacy behavior). Cash/PnL and the
# remaining position all scale to the amount actually closed
# so a partial close no longer flattens the lot (#577).
requested = leg.get("qty")
full_mag = abs(matched.qty)
close_mag = full_mag if requested is None else min(abs(requested), full_mag)
if close_mag <= 0:
continue
sign = 1 if matched.qty > 0 else -1
closed_qty = sign * close_mag
remaining_qty = matched.qty - closed_qty
pnl = (opt_price - matched.entry_price) * closed_qty * contract_multiplier
abs_close = opt_price * close_mag * contract_multiplier
if matched.qty > 0:
# Long close: sell to recover
cash += abs_close * (1 - commission)
@@ -453,11 +416,24 @@ def run_options_backtest(
"expiry": expiry,
"side": "close",
"price": round(opt_price, 4),
"qty": matched.qty,
"qty": closed_qty,
"pnl": round(pnl, 4),
"entry_date": matched.entry_date,
})
positions.remove(matched)
if abs(remaining_qty) < 1e-9:
positions.remove(matched)
else:
# Reduce the open lot to its remainder instead of
# removing it (new object; positions stay immutable).
positions[positions.index(matched)] = OptionPosition(
option_type=matched.option_type,
strike=matched.strike,
expiry=matched.expiry,
qty=remaining_qty,
entry_price=matched.entry_price,
entry_date=matched.entry_date,
underlying_code=matched.underlying_code,
)
# 4. Compute portfolio mark-to-market value and Greeks
portfolio_value = cash
@@ -465,20 +441,24 @@ def run_options_backtest(
total_gamma = 0.0
total_theta = 0.0
total_vega = 0.0
total_rho = 0.0
for pos in positions:
spot = spot_prices.get(pos.underlying_code, 0.0)
iv_val = ivs.get(pos.underlying_code, 0.3)
T = pos.time_to_expiry(ts)
mark_price = bs_price(spot, pos.strike, T, risk_free_rate, iv_val, pos.option_type)
mark_iv = leg_iv(spot, pos.strike, iv_val, iv_skew, iv_curvature)
mark_price = bs_price(spot, pos.strike, T, risk_free_rate, mark_iv, pos.option_type)
portfolio_value += mark_price * pos.qty * contract_multiplier
greeks = bs_greeks(spot, pos.strike, T, risk_free_rate, iv_val, pos.option_type)
greeks = bs_greeks(spot, pos.strike, T, risk_free_rate, mark_iv, pos.option_type)
total_delta += greeks["delta"] * pos.qty * contract_multiplier
total_gamma += greeks["gamma"] * pos.qty * contract_multiplier
total_theta += greeks["theta"] * pos.qty * contract_multiplier
total_vega += greeks["vega"] * pos.qty * contract_multiplier
total_rho += greeks["rho"] * pos.qty * contract_multiplier
equity_records.append({
"timestamp": date_str,
@@ -493,6 +473,7 @@ def run_options_backtest(
"gamma": round(total_gamma, 6),
"theta": round(total_theta, 6),
"vega": round(total_vega, 6),
"rho": round(total_rho, 6),
"num_positions": len(positions),
})
@@ -529,9 +510,10 @@ def run_options_backtest(
metrics,
data_sources=[str(getattr(loader, "name", config.get("source", "")))],
strategy_path=run_dir / "code" / "signal_engine.py",
warnings=config.get("content_filter_warnings") or None,
)
print(json.dumps(metrics, indent=2))
print(json.dumps(metrics, indent=2, allow_nan=False))
return metrics
@@ -550,7 +532,9 @@ def _find_matching_position(
Args:
positions: Current open positions.
underlying: Underlying instrument code.
option_type: Option type.
option_type: Option type, already folded by ``normalise_option_type``;
``OptionPosition`` folds its own, so both sides compare in lower
case.
strike: Strike price.
expiry: Expiry date string.
@@ -584,34 +568,148 @@ def _calc_options_metrics(
Returns:
Metrics dictionary.
"""
warnings: List[str] = []
n = len(equity)
equity_vals = pd.to_numeric(equity, errors="coerce").astype(float)
path_is_finite = bool(n and np.isfinite(equity_vals.to_numpy()).all())
final_raw: float | None = None
final_value: float | None = None
if n:
terminal = float(equity_vals.iloc[-1])
if np.isfinite(terminal):
final_raw = terminal
final_value = round(terminal, 2)
else:
warnings.append(
"Final equity is non-finite; final and return metrics are undefined."
)
else:
warnings.append(
"No equity observations were produced; equity metrics are undefined."
)
valid_initial_cash = np.isfinite(initial_cash) and initial_cash > 0
total_ret: float | None = None
if final_raw is not None and valid_initial_cash:
total_ret = final_raw / float(initial_cash) - 1
elif final_raw is not None:
warnings.append(
"Total return is undefined because initial cash is not positive and finite."
)
ann_ret: float | None = None
if n < 2:
return {
"final_value": initial_cash, "total_return": 0, "annual_return": 0,
"max_drawdown": 0, "sharpe": 0, "calmar": 0, "sortino": 0,
"trade_count": len(trades), "win_rate": 0, "profit_loss_ratio": 0,
}
warnings.append("Annual return requires at least two equity observations.")
elif total_ret is None:
warnings.append(
"Annual return is undefined because total return is unavailable."
)
elif final_raw is not None and final_raw < 0:
warnings.append("Annual return is undefined when final equity is negative.")
elif bars_per_year <= 0:
warnings.append(
"Annual return is undefined because bars_per_year is not positive."
)
else:
growth = final_raw / float(initial_cash)
# Explosive paths (e.g. 1m bars) can OverflowError before isfinite.
try:
candidate = float(growth ** (bars_per_year / (n - 1)) - 1)
except OverflowError:
candidate = float("inf")
if np.isfinite(candidate):
ann_ret = candidate
else:
warnings.append("Annual return is non-finite for this equity path.")
equity_vals = equity.astype(float)
returns = equity_vals.pct_change().fillna(0.0)
returns: pd.Series | None = None
if n >= 2 and path_is_finite:
candidate_returns = equity_vals.pct_change(fill_method=None).iloc[1:]
if np.isfinite(candidate_returns.to_numpy()).all():
returns = candidate_returns
else:
warnings.append(
"Risk ratios are undefined because equity returns are non-finite."
)
elif n >= 2:
warnings.append(
"Path-dependent metrics are undefined because equity contains non-finite values."
)
total_ret = float(equity_vals.iloc[-1] / initial_cash - 1)
ann_ret = float((1 + total_ret) ** (bars_per_year / max(n, 1)) - 1)
max_dd: float | None = None
if path_is_finite:
peak = equity_vals.cummax()
if bool((peak > 0).all()):
dd = (equity_vals - peak) / peak
max_dd = float(dd.min())
else:
warnings.append(
"Maximum drawdown is undefined because peak equity is not positive."
)
vol = float(returns.std())
sharpe = float(returns.mean() / (vol + 1e-10) * np.sqrt(bars_per_year))
sharpe: float | None = None
if returns is not None and len(returns) > 1 and bars_per_year > 0:
vol = float(returns.std())
if np.isfinite(vol) and vol > 1e-12:
sharpe = float(returns.mean() / vol * np.sqrt(bars_per_year))
else:
warnings.append(
"Sharpe ratio is undefined because return volatility is zero."
)
elif bars_per_year <= 0:
warnings.append("Sharpe ratio requires a positive bars_per_year value.")
else:
warnings.append("Sharpe ratio requires at least two finite returns.")
peak = equity_vals.cummax()
dd = (equity_vals - peak) / peak.replace(0, 1)
max_dd = float(dd.min())
calmar = ann_ret / abs(max_dd) if abs(max_dd) > 1e-10 else 0.0
calmar: float | None = None
if ann_ret is not None and max_dd is not None and abs(max_dd) > 1e-12:
calmar = ann_ret / abs(max_dd)
else:
warnings.append(
"Calmar ratio requires a defined annual return and a nonzero drawdown."
)
downside = returns[returns < 0]
downside_std = float(downside.std()) if len(downside) > 1 else 1e-10
sortino = float(returns.mean() / (downside_std + 1e-10) * np.sqrt(bars_per_year))
sortino: float | None = None
if returns is not None and bars_per_year > 0:
downside = returns[returns < 0]
if len(downside) > 1:
downside_std = float(downside.std())
if np.isfinite(downside_std) and downside_std > 1e-12:
sortino = float(returns.mean() / downside_std * np.sqrt(bars_per_year))
if sortino is None:
warnings.append(
"Sortino ratio requires at least two varying downside returns."
)
elif bars_per_year <= 0:
warnings.append("Sortino ratio requires a positive bars_per_year value.")
else:
warnings.append("Sortino ratio requires finite returns.")
# Trade statistics
closed_pnl = [t["pnl"] for t in trades if t.get("pnl", 0) != 0]
closed_pnl: List[float] = []
ignored_pnl_records = 0
for t in trades:
raw_pnl = t.get("pnl")
if raw_pnl is None:
ignored_pnl_records += 1
continue
try:
val = float(raw_pnl)
except (TypeError, ValueError):
ignored_pnl_records += 1
continue
if not np.isfinite(val):
ignored_pnl_records += 1
continue
if val != 0:
closed_pnl.append(val)
if ignored_pnl_records:
warnings.append(
f"Ignored PnL for {ignored_pnl_records} trade records "
"(missing or non-numeric pnl); win rate and profit/loss ratio "
"are computed from the remaining trades only."
)
wins = [p for p in closed_pnl if p > 0]
losses = [p for p in closed_pnl if p < 0]
win_rate = len(wins) / len(closed_pnl) if closed_pnl else 0.0
@@ -620,14 +718,15 @@ def _calc_options_metrics(
pl_ratio = avg_win / avg_loss if avg_loss > 1e-10 else 0.0
return {
"final_value": round(float(equity_vals.iloc[-1]), 2),
"total_return": round(total_ret, 6),
"annual_return": round(ann_ret, 6),
"max_drawdown": round(max_dd, 6),
"sharpe": round(sharpe, 4),
"calmar": round(calmar, 4),
"sortino": round(sortino, 4),
"final_value": final_value,
"total_return": round(total_ret, 6) if total_ret is not None else None,
"annual_return": round(ann_ret, 6) if ann_ret is not None else None,
"max_drawdown": round(max_dd, 6) if max_dd is not None else None,
"sharpe": round(sharpe, 4) if sharpe is not None else None,
"calmar": round(calmar, 4) if calmar is not None else None,
"sortino": round(sortino, 4) if sortino is not None else None,
"trade_count": len(trades),
"win_rate": round(win_rate, 4),
"profit_loss_ratio": round(pl_ratio, 4),
"warnings": warnings,
}
agent/backtest/factor_costs.py
+416
View File
@@ -0,0 +1,416 @@
"""Trading-cost model for factor backtests: ADV caps, impact, borrow.
A long-short factor backtest that rebalances daily on a 500-name universe is
implicitly claiming it can trade every name, in any size, at the close, for a
flat fee. Three things break that claim, and this module prices all three:
**Capacity.** You cannot buy more of a stock in a day than the market trades.
The standard bound is a participation rate -- a fraction of average daily volume
you are willing to be. Beyond it you are the market, not a participant in it.
When a target weight needs more than the cap allows, the position moves *part of
the way* and the shortfall carries into the next period. It is not silently
filled, and it is not silently dropped: both would flatter the backtest, in
opposite directions.
**Impact.** The part you do trade moves the price against you, and the amount is
not linear in size. :mod:`src.quantlib.impact` already owns those models --
this module calls them rather than carrying a second copy.
**Borrow.** The short leg is not free. Someone lends you the stock and charges
for it, and for hard-to-borrow names that fee can exceed the alpha. A long-short
factor reported without borrow cost is reporting a portfolio nobody could have
held.
WHY THIS IS NOT IN THE ENGINE
-----------------------------
``backtest/engines/base.py`` prices costs per *fill*, which is the right model
when there is an order book and a fill sequence. A factor bench has neither: it
has a weight vector per rebalance date. Forcing one abstraction onto the other
would mean inventing fills that never happened. This module therefore works in
weight space, and the two never share a code path.
SIGN AND UNIT CONVENTIONS
-------------------------
* Weights are fractions of capital; a short is negative.
* Every cost this module returns is a POSITIVE drag, to be subtracted from a
gross return. A function that returned a signed adjustment would eventually be
added somewhere by mistake.
* ``adv`` is in the same currency as ``capital``, not in shares. Mixing those is
the classic error here, so the argument names say ``_value`` and the
docstrings repeat it.
* Rates named ``annual_*`` are annualised decimals; the periods-per-year
argument converts them, and it has no default because a daily and a monthly
bench would silently differ by 21x.
"""
from __future__ import annotations
from collections.abc import Sequence
from dataclasses import dataclass
import numpy as np
import pandas as pd
from src.quantlib.impact import DEFAULT_SLIPPAGE_BPS, linear_impact, sqrt_impact
__all__ = [
"DEFAULT_MAX_PARTICIPATION",
"IMPACT_MODELS",
"MARKET_BORROW_RATES",
"CapacityResult",
"PeriodCost",
"apply_adv_capacity",
"borrow_cost",
"rebalance_cost",
]
#: Fraction of a name's average daily value a strategy is allowed to be. 10% is
#: the conventional institutional ceiling; above it the participation itself
#: starts to define the price and the impact models stop being descriptive.
DEFAULT_MAX_PARTICIPATION: float = 0.10
#: Impact models callable by name. Both come from ``src.quantlib.impact``; this
#: module owns no impact formula of its own.
IMPACT_MODELS: tuple[str, ...] = ("fixed", "linear", "sqrt")
#: Indicative annual stock-borrow rates for a general-collateral name, by
#: market. These are STARTING POINTS for a caller who has no better number, not
#: measurements: a hard-to-borrow name can cost 20% or more, and nothing in this
#: repository knows which names those are. Pass your own rate when you have one.
MARKET_BORROW_RATES: dict[str, float] = {
"US": 0.0030,
"CN": 0.0850,
"HK": 0.0100,
"JP": 0.0050,
}
@dataclass(frozen=True)
class CapacityResult:
"""Weights actually achievable under an ADV participation cap.
Attributes:
achieved_weights: Weights after capping, indexed by symbol.
requested_weights: The weights that were asked for.
unfilled_weights: ``requested - achieved`` per symbol. Non-zero means
the strategy wanted a position the market could not give it that
period; carry this into the next rebalance rather than discarding it.
capped_symbols: Symbols whose trade hit the cap.
participation: Realised participation per symbol, as a fraction of ADV.
max_participation: The cap that was applied.
"""
achieved_weights: pd.Series
requested_weights: pd.Series
unfilled_weights: pd.Series
capped_symbols: tuple[str, ...]
participation: pd.Series
max_participation: float
@dataclass(frozen=True)
class PeriodCost:
"""All-in trading drag for one rebalance period.
Attributes:
impact_cost: Drag from market impact and spread, as a fraction of
capital. Positive.
borrow_cost: Drag from borrowing the short leg, as a fraction of
capital. Positive.
total_cost: ``impact_cost + borrow_cost``. Positive.
turnover: One-way turnover actually executed, as a fraction of capital.
unfilled_turnover: Turnover the ADV cap prevented.
capped_symbols: Symbols that hit the cap.
impact_model: Which model priced the impact.
"""
impact_cost: float
borrow_cost: float
total_cost: float
turnover: float
unfilled_turnover: float
capped_symbols: tuple[str, ...]
impact_model: str
def _as_aligned(
target: pd.Series, current: pd.Series
) -> tuple[pd.Series, pd.Series, pd.Index]:
"""Put two weight vectors on one index, filling absent names with zero.
A name in the target but not the current book is a new position; a name in
the current book but not the target is a full exit. Both are trades, so
neither may be dropped by an inner join.
Args:
target: Desired weights.
current: Weights held before the rebalance.
Returns:
Tuple of ``(target, current, index)`` on the union index.
"""
index = target.index.union(current.index)
return (
target.reindex(index).fillna(0.0).astype(float),
current.reindex(index).fillna(0.0).astype(float),
index,
)
def apply_adv_capacity(
target_weights: pd.Series,
current_weights: pd.Series,
adv_value: pd.Series,
capital: float,
max_participation: float = DEFAULT_MAX_PARTICIPATION,
) -> CapacityResult:
"""Cap each name's trade at a fraction of its average daily traded value.
Args:
target_weights: Desired weights by symbol, shorts negative.
current_weights: Weights held before this rebalance.
adv_value: Average daily traded VALUE by symbol, in the same currency as
``capital`` -- not a share count.
capital: Portfolio capital in that currency.
max_participation: Largest fraction of a name's ADV the strategy may be.
Returns:
A :class:`CapacityResult`. A symbol with no ADV entry, or a non-positive
one, is treated as untradeable this period: its weight stays where it
was and the whole requested change lands in ``unfilled_weights``. That
is deliberately the conservative reading -- assuming a name with unknown
volume is infinitely liquid is how a backtest acquires capacity it never
had.
Raises:
ValueError: If ``capital`` is not positive or ``max_participation`` is
outside ``(0, 1]``.
"""
if not capital > 0:
raise ValueError(f"capital must be positive, got {capital}")
if not 0.0 < max_participation <= 1.0:
raise ValueError(
f"max_participation must be in (0, 1], got {max_participation}"
)
target, current, index = _as_aligned(target_weights, current_weights)
adv = pd.Series(adv_value, dtype=float).reindex(index)
requested_change = target - current
# Largest weight change a name can absorb: participation x ADV, expressed
# as a fraction of capital.
capacity = (adv * max_participation / capital).fillna(0.0)
capacity[~np.isfinite(capacity)] = 0.0
capacity[capacity < 0.0] = 0.0
allowed_change = requested_change.clip(lower=-capacity, upper=capacity)
achieved = current + allowed_change
unfilled = target - achieved
capped = tuple(
str(sym)
for sym in index
if abs(requested_change[sym]) > capacity[sym] + 1e-15
and abs(requested_change[sym]) > 0.0
)
with np.errstate(divide="ignore", invalid="ignore"):
participation = (allowed_change.abs() * capital / adv).replace(
[np.inf, -np.inf], np.nan
)
return CapacityResult(
achieved_weights=achieved,
requested_weights=target,
unfilled_weights=unfilled,
capped_symbols=capped,
participation=participation,
max_participation=max_participation,
)
def borrow_cost(
weights: pd.Series,
periods_per_year: int,
annual_rate: float | pd.Series | None = None,
market: str | None = None,
) -> float:
"""Cost of borrowing the short leg for one period.
Only negative weights are charged: a long position borrows nothing. The
charge is the absolute short exposure times the pro-rated annual fee.
Args:
weights: Portfolio weights, shorts negative.
periods_per_year: Rebalance periods in a year -- 252 for daily, 12 for
monthly. No default: a daily and a monthly bench charging the same
per-period rate differ by 21x, and nothing downstream would notice.
annual_rate: Annual borrow fee as a decimal, either one rate for the
whole book or a per-symbol Series. Takes precedence over ``market``.
market: Key into :data:`MARKET_BORROW_RATES` when no explicit rate is
supplied. Those values are indicative general-collateral rates, not
measurements -- see the constant's own note.
Returns:
Positive cost as a fraction of capital.
Raises:
ValueError: If ``periods_per_year`` is not positive, if neither
``annual_rate`` nor ``market`` is supplied, if ``market`` is
unknown, or if any rate is negative.
"""
if periods_per_year <= 0:
raise ValueError(f"periods_per_year must be positive, got {periods_per_year}")
if annual_rate is None:
if market is None:
raise ValueError(
"supply annual_rate, or market to fall back on an indicative "
f"general-collateral rate from {sorted(MARKET_BORROW_RATES)}"
)
if market not in MARKET_BORROW_RATES:
raise ValueError(
f"unknown market {market!r}; known: {sorted(MARKET_BORROW_RATES)}"
)
annual_rate = MARKET_BORROW_RATES[market]
shorts = pd.Series(weights, dtype=float)
shorts = -shorts[shorts < 0.0]
if shorts.empty:
return 0.0
if isinstance(annual_rate, pd.Series):
rates = annual_rate.reindex(shorts.index)
if rates.isna().any():
missing = sorted(str(s) for s in rates.index[rates.isna()])
raise ValueError(
f"annual_rate is missing a rate for shorted symbol(s): {missing}"
)
else:
rates = pd.Series(float(annual_rate), index=shorts.index)
if (rates < 0.0).any():
raise ValueError("borrow rates must be non-negative")
return float((shorts * rates).sum() / periods_per_year)
def rebalance_cost(
target_weights: pd.Series,
current_weights: pd.Series,
capital: float,
periods_per_year: int,
adv_value: pd.Series | None = None,
max_participation: float = DEFAULT_MAX_PARTICIPATION,
impact_model: str = "sqrt",
slippage_bps: float = DEFAULT_SLIPPAGE_BPS,
impact_coefficient: float = 0.1,
volatility: float | pd.Series | None = None,
borrow_annual_rate: float | pd.Series | None = None,
borrow_market: str | None = None,
) -> tuple[PeriodCost, CapacityResult | None]:
"""Price one rebalance: capacity, then impact, then borrow.
Args:
target_weights: Desired weights, shorts negative.
current_weights: Weights held before the rebalance.
capital: Portfolio capital.
periods_per_year: Rebalance periods in a year, for the borrow charge.
adv_value: Average daily traded VALUE by symbol. When None, no capacity
cap is applied and ``capped_symbols`` comes back empty -- which is
an assumption of infinite liquidity and should be stated wherever
the result is reported.
max_participation: Participation ceiling for the capacity cap.
impact_model: One of :data:`IMPACT_MODELS`.
slippage_bps: Half-spread plus fees, in basis points, charged on every
unit of turnover regardless of model.
impact_coefficient: Coefficient handed to the linear or sqrt model.
volatility: Per-period volatility, required by the ``sqrt`` model.
borrow_annual_rate: Annual borrow fee, decimal.
borrow_market: Fallback market key when no explicit rate is given.
Returns:
Tuple of the :class:`PeriodCost` and the :class:`CapacityResult` (None
when no ``adv_value`` was supplied).
Raises:
ValueError: If ``impact_model`` is unknown, if the ``sqrt`` model is
chosen without ``volatility``, or from the functions this delegates
to.
"""
if impact_model not in IMPACT_MODELS:
raise ValueError(
f"impact_model must be one of {IMPACT_MODELS}, got {impact_model!r}"
)
target, current, index = _as_aligned(target_weights, current_weights)
capacity: CapacityResult | None = None
if adv_value is not None:
capacity = apply_adv_capacity(
target, current, adv_value, capital, max_participation
)
achieved = capacity.achieved_weights
unfilled_turnover = float(capacity.unfilled_weights.abs().sum())
capped = capacity.capped_symbols
else:
achieved = target
unfilled_turnover = 0.0
capped = ()
traded = (achieved - current).abs()
turnover = float(traded.sum())
# Spread and fees are charged on every unit of turnover under every model.
spread_cost = turnover * slippage_bps / 10_000.0
if impact_model == "fixed" or turnover == 0.0:
impact = 0.0
elif impact_model == "linear":
participation = (
capacity.participation.reindex(index).fillna(0.0)
if capacity is not None
else pd.Series(0.0, index=index)
)
# linear_impact prices a fill; here it prices the weighted average
# participation, and the drag scales with the traded fraction.
impact = float((traded * participation * impact_coefficient).sum())
else: # sqrt
if volatility is None:
raise ValueError(
"the sqrt impact model needs a volatility; supply one or choose "
"impact_model='linear' or 'fixed'"
)
vol = (
pd.Series(volatility, dtype=float).reindex(index).fillna(0.0)
if isinstance(volatility, pd.Series)
else pd.Series(float(volatility), index=index)
)
participation = (
capacity.participation.reindex(index).fillna(0.0)
if capacity is not None
else pd.Series(0.0, index=index)
)
impact = float(
(traded * impact_coefficient * vol * np.sqrt(participation.clip(lower=0.0))).sum()
)
impact_cost = spread_cost + impact
borrow = borrow_cost(
achieved,
periods_per_year=periods_per_year,
annual_rate=borrow_annual_rate,
market=borrow_market,
) if (borrow_annual_rate is not None or borrow_market is not None) else 0.0
return (
PeriodCost(
impact_cost=impact_cost,
borrow_cost=borrow,
total_cost=impact_cost + borrow,
turnover=turnover,
unfilled_turnover=unfilled_turnover,
capped_symbols=capped,
impact_model=impact_model,
),
capacity,
)
agent/backtest/loaders/_fundamental_schema.py
+226
View File
@@ -0,0 +1,226 @@
"""Unified fundamental field schema and SEC XBRL concept aliases.
This module is deliberately metadata-only. Callers use ``SEC_CONCEPT_MAP`` to
extract sparse reported facts; when no concept hits, the caller must emit NaN
for that field and log the missing alias coverage. This module does not log or
fabricate fallback values.
"""
from __future__ import annotations
from collections.abc import Callable
from typing import Literal
import pandas as pd
FieldKind = Literal["raw", "derived"]
RawFieldSpec = dict[str, object]
DerivedCompute = Callable[[dict[str, pd.Series]], pd.Series]
DerivedFieldSpec = dict[str, object]
def _safe_divide(numerator: pd.Series, denominator: pd.Series) -> pd.Series:
"""Divide two aligned series, returning NaN where the denominator is zero."""
clean_denominator = denominator.astype("float64").mask(denominator == 0)
return numerator.astype("float64") / clean_denominator
RAW_FIELDS: dict[str, RawFieldSpec] = {
"revenue": {
"statement": "IS",
"description": "Revenue / net sales.",
},
"cogs": {
"statement": "IS",
"description": "Cost of goods sold or cost of revenue.",
},
"gross_profit": {
"statement": "IS",
"description": "Gross profit; derived as revenue minus cogs when not directly reported.",
"dependencies": ["revenue", "cogs"],
"compute": lambda data: data["revenue"] - data["cogs"],
},
"operating_income": {
"statement": "IS",
"description": "Operating income or loss.",
},
"net_income": {
"statement": "IS",
"description": "Net income or loss attributable to the reporting entity.",
},
"total_assets": {
"statement": "BS",
"description": "Total assets.",
},
"total_equity": {
"statement": "BS",
"description": "Total shareholders' equity.",
},
"total_debt": {
"statement": "BS",
"description": "Interest-bearing debt; aggregate debt concept when present, otherwise loader may union current and noncurrent debt concepts.",
},
"cash": {
"statement": "BS",
"description": "Cash and cash equivalents.",
},
"shares_diluted": {
"statement": "IS/cover",
"description": "Weighted-average diluted shares outstanding.",
},
"cfo": {
"statement": "CF",
"description": "Net cash provided by or used in operating activities.",
},
"capex": {
"statement": "CF",
"description": "Capital expenditures for property, plant, equipment, and similar productive assets.",
},
}
# Ordered by priority. SEC extraction should union concepts across aliases, then
# keep the first filed value per period_end for PIT safety. The revenue order is
# intentionally new-standard first: recent AAPL filings report revenue under
# RevenueFromContractWithCustomer... rather than the old Revenues concept.
SEC_CONCEPT_MAP: dict[str, list[str]] = {
"revenue": [
"RevenueFromContractWithCustomerExcludingAssessedTax",
"RevenueFromContractWithCustomerIncludingAssessedTax",
"Revenues",
"SalesRevenueNet",
],
"cogs": [
"CostOfGoodsAndServicesSold",
"CostOfRevenue",
"CostOfGoodsSold",
],
"gross_profit": [
"GrossProfit",
],
"operating_income": [
"OperatingIncomeLoss",
],
"net_income": [
"NetIncomeLoss",
"ProfitLoss",
],
"total_assets": [
"Assets",
],
"total_equity": [
"StockholdersEquity",
"StockholdersEquityIncludingPortionAttributableToNoncontrollingInterest",
],
# US-GAAP has both aggregate and split debt concepts. Prefer aggregate debt
# when reported; otherwise a future loader should combine current and
# noncurrent concepts instead of treating one leg as total debt.
"total_debt": [
"LongTermDebtAndFinanceLeaseObligations",
"LongTermDebtAndCapitalLeaseObligations",
"LongTermDebt",
"LongTermDebtNoncurrent",
"LongTermDebtCurrent",
"DebtCurrent",
"ShortTermBorrowings",
],
# Cash concept choice varies by industry and restricted-cash presentation.
# The first concept is the common operating-company aggregate; the rest keep
# banks and restricted-cash filers from hard-missing in Phase 1.
"cash": [
"CashAndCashEquivalentsAtCarryingValue",
"CashCashEquivalentsRestrictedCashAndRestrictedCashEquivalents",
"Cash",
"CashAndDueFromBanks",
],
"shares_diluted": [
"WeightedAverageNumberOfDilutedSharesOutstanding",
"WeightedAverageDilutedSharesOutstanding",
],
"cfo": [
"NetCashProvidedByUsedInOperatingActivities",
"NetCashProvidedByUsedInOperatingActivitiesContinuingOperations",
],
# Capex is usually reported as PPE purchases. Some filers include broader
# productive-asset or PPE-plus-intangibles concepts; they are lower priority
# because they may not be strictly comparable to plain PPE capex.
"capex": [
"PaymentsToAcquirePropertyPlantAndEquipment",
"PaymentsToAcquireProductiveAssets",
"PaymentsToAcquirePropertyPlantAndEquipmentAndIntangibleAssets",
"CapitalExpendituresIncurredButNotYetPaid",
],
}
DERIVED_FIELDS: dict[str, DerivedFieldSpec] = {
"roe": {
"dependencies": ["net_income", "total_equity"],
"compute": lambda data: _safe_divide(data["net_income"], data["total_equity"]),
},
"roa": {
"dependencies": ["net_income", "total_assets"],
"compute": lambda data: _safe_divide(data["net_income"], data["total_assets"]),
},
"gross_profitability": {
"dependencies": ["gross_profit", "total_assets"],
"compute": lambda data: _safe_divide(data["gross_profit"], data["total_assets"]),
},
# Annual semantics: input is one observation per annual period_end, sorted by
# period_end. Growth is period-over-period, value / value.shift(1) - 1.
# Quarterly or TTM callers must pre-aggregate before applying this formula.
"asset_growth": {
"dependencies": ["total_assets"],
"compute": lambda data: data["total_assets"].astype("float64")
/ data["total_assets"].astype("float64").shift(1)
- 1,
},
"accruals": {
"dependencies": ["net_income", "cfo", "total_assets"],
"compute": lambda data: _safe_divide(
data["net_income"] - data["cfo"],
data["total_assets"],
),
},
"leverage": {
"dependencies": ["total_debt", "total_equity"],
"compute": lambda data: _safe_divide(data["total_debt"], data["total_equity"]),
},
}
# Price-dependent value factors need market_cap = price * shares and therefore
# live in factor files after fundamental and price panels have been aligned.
# Do not add earnings_yield or book_to_market to DERIVED_FIELDS here.
def resolve_field(field: str) -> tuple[FieldKind, RawFieldSpec | DerivedFieldSpec]:
"""Resolve a supported fundamental field to its schema entry.
Args:
field: Unified fundamental field name.
Returns:
A ``("raw", spec)`` or ``("derived", spec)`` tuple.
Raises:
ValueError: If ``field`` is not part of the unified schema.
"""
if field in RAW_FIELDS:
return "raw", RAW_FIELDS[field]
if field in DERIVED_FIELDS:
return "derived", DERIVED_FIELDS[field]
raise ValueError(f"unknown fundamental field: {field}")
def list_supported_fields() -> list[str]:
"""Return all supported raw and schema-derived fundamental field names."""
return sorted(set(RAW_FIELDS) | set(DERIVED_FIELDS))
__all__ = [
"DERIVED_FIELDS",
"RAW_FIELDS",
"SEC_CONCEPT_MAP",
"list_supported_fields",
"resolve_field",
]
agent/backtest/loaders/_http.py
+179
View File
@@ -0,0 +1,179 @@
"""Shared HTTP helpers for direct-API loaders: per-host throttling + JSON/CSV GET.
Several free providers — Eastmoney most notably — rate-limit by source IP and
will temporarily ban a client that bursts requests. Rather than scatter sleep
calls across loaders, every ban-prone call routes through :func:`throttled_get`
here, which enforces a minimum spacing between consecutive requests to the same
*host bucket* (plus a little jitter so concurrent workers don't lock-step) and
reuses one :class:`requests.Session` per process so TCP/TLS setup is amortized.
This module is intentionally provider-agnostic: it knows nothing about
Eastmoney/Sina/Stooq field layouts, only how to space requests politely. A
loader picks its own ``host_key`` and ``min_interval`` and stays ignorant of
the locking mechanics.
All spacing is best-effort and process-local — it does not coordinate across
machines. For batch jobs raise the relevant ``*_MIN_INTERVAL`` env var.
"""
from __future__ import annotations
import logging
import random
import threading
import time
from typing import Any
import requests
from backtest.loaders.base import positive_env_float
logger = logging.getLogger(__name__)
# Default User-Agent. Many free quote endpoints reject the bare urllib/requests
# UA, so we present a normal desktop browser string. Loaders may override.
DEFAULT_USER_AGENT = (
"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 "
"(KHTML, like Gecko) Chrome/124.0 Safari/537.36"
)
# Upper bound on the random jitter (seconds) added on top of the configured
# minimum interval, so parallel callers de-synchronize instead of all firing
# the instant the interval elapses.
_JITTER_MAX_S = 0.4
class HostThrottle:
"""Process-wide minimum-spacing gate keyed by an arbitrary host bucket.
One instance guards all callers; ``wait(bucket, min_interval)`` blocks until
at least ``min_interval`` seconds (plus jitter) have elapsed since the last
request tagged with the same ``bucket``. The lock is held only for the
bookkeeping arithmetic, not across the sleep, so distinct buckets never block
one another.
"""
def __init__(self) -> None:
self._last: dict[str, float] = {}
self._lock = threading.Lock()
def wait(self, bucket: str, min_interval: float) -> None:
"""Block until ``bucket`` is allowed to fire again, then record the slot.
The *reserved fire time* — jitter included — is what gets stored, so the
next caller spaces off this caller's actual fire instant rather than an
earlier un-jittered one. This keeps consecutive requests at least
``min_interval`` apart even when many callers burst concurrently (the
exact scenario the throttle exists for); the jitter only ever pushes a
slot later, never earlier.
"""
if min_interval <= 0:
return
with self._lock:
now = time.monotonic()
last = self._last.get(bucket)
if last is None or now >= last + min_interval:
# Slot is free right now — fire immediately, no jitter needed.
fire_at = now
else:
# Chain off the previous reservation and add jitter to desync
# concurrent callers, baking the jitter into the stored slot.
fire_at = last + min_interval + random.uniform(0.0, _JITTER_MAX_S)
self._last[bucket] = fire_at
sleep_for = fire_at - time.monotonic()
if sleep_for > 0:
time.sleep(sleep_for)
# One shared gate for the whole process.
_THROTTLE = HostThrottle()
# Per-process session reuse, keyed by host bucket so different providers keep
# independent connection pools and cookie jars.
_SESSIONS: dict[str, requests.Session] = {}
_SESSIONS_LOCK = threading.Lock()
def _session_for(bucket: str) -> requests.Session:
with _SESSIONS_LOCK:
session = _SESSIONS.get(bucket)
if session is None:
session = requests.Session()
_SESSIONS[bucket] = session
return session
def resolve_min_interval(env_name: str, default: float) -> float:
"""Resolve a per-provider minimum request interval from the environment.
Args:
env_name: Env var carrying an override in seconds (e.g.
``VIBE_TRADING_EASTMONEY_MIN_INTERVAL``).
default: Fallback interval when the env var is absent or invalid.
Returns:
The override when it parses to a positive float, else ``default``.
"""
return positive_env_float(env_name, default)
def throttled_get(
url: str,
*,
host_key: str,
min_interval: float,
params: dict[str, Any] | None = None,
headers: dict[str, str] | None = None,
timeout: float = 15.0,
) -> requests.Response:
"""GET ``url`` after waiting out the per-host minimum interval.
Args:
url: Fully-qualified request URL.
host_key: Throttle/session bucket. All calls sharing a key are spaced
by ``min_interval`` and reuse one session.
min_interval: Minimum seconds between consecutive calls to ``host_key``.
params: Optional query parameters.
headers: Optional headers merged over the default browser UA.
timeout: Per-request socket timeout in seconds.
Returns:
The :class:`requests.Response`; the caller decides how to parse it.
Raises:
requests.RequestException: Propagated unchanged for the caller's retry
policy to classify as transient.
"""
merged_headers = {"User-Agent": DEFAULT_USER_AGENT}
if headers:
merged_headers.update(headers)
_THROTTLE.wait(host_key, min_interval)
session = _session_for(host_key)
return session.get(url, params=params, headers=merged_headers, timeout=timeout)
def throttled_get_json(
url: str,
*,
host_key: str,
min_interval: float,
params: dict[str, Any] | None = None,
headers: dict[str, str] | None = None,
timeout: float = 15.0,
) -> Any:
"""Throttled GET that decodes the response body as JSON.
Same contract as :func:`throttled_get`, plus ``response.raise_for_status()``
and ``response.json()``. A non-2xx status or undecodable body raises, which
the caller's bounded-retry wrapper treats as transient.
"""
response = throttled_get(
url,
host_key=host_key,
min_interval=min_interval,
params=params,
headers=headers,
timeout=timeout,
)
response.raise_for_status()
return response.json()
agent/backtest/loaders/_symbol_utils.py
+20
View File
@@ -0,0 +1,20 @@
"""Shared symbol-type detection utilities for loaders.
Exchange-listed ETF / LOF prefix codes (same pattern across loaders):
SH: 50/51/52/56/58 (ETFs), SZ: 15/16 (ETFs + LOFs).
"""
from __future__ import annotations
_ETF_PREFIXES = frozenset({"15", "16", "50", "51", "52", "56", "58"})
def _is_etf_listed(code: str) -> bool:
"""Detect exchange-listed ETF / LOF symbols (e.g. 510050.SH, 159915.SZ)."""
upper = code.upper()
if not upper.endswith((".SH", ".SZ")):
return False
digits = upper.split(".")[0]
if len(digits) != 6 or not digits.isdigit():
return False
return digits[:2] in _ETF_PREFIXES
agent/backtest/loaders/akshare_loader.py
+37 -20
View File
@@ -11,17 +11,30 @@ from typing import Dict, List, Optional
import pandas as pd
from backtest.loaders.base import validate_date_range
from backtest.loaders._symbol_utils import _is_etf_listed
from backtest.loaders.base import cached_loader_fetch, validate_date_range
from backtest.loaders.registry import register
logger = logging.getLogger(__name__)
_INTERVAL_MAP_DAILY = {
"1D": "daily",
"1d": "daily",
"1W": "weekly",
"1w": "weekly",
"1M": "monthly",
}
# US/HK/ETF/forex serve daily bars only.
_DAILY_ONLY_ALIASES = frozenset({"1d", "d", "day", "daily"})
def _require_daily_interval(interval: str, market: str) -> None:
if str(interval).strip().lower() not in _DAILY_ONLY_ALIASES:
raise ValueError(
f"Unsupported interval {interval!r}; akshare {market} supports daily bars only"
)
def _is_a_share(code: str) -> bool:
return code.upper().endswith((".SZ", ".SH", ".BJ"))
@@ -39,23 +52,8 @@ def _is_crypto(code: str) -> bool:
return "-USDT" in code.upper() or "/USDT" in code.upper()
# Exchange-listed ETF / LOF prefix codes:
# SH: 50/51/52/56/58 (ETFs), SZ: 15/16 (ETFs + LOFs).
# Issue #50 — these symbols look like A-shares (.SH / .SZ) but stock_zh_a_hist
# can't price them; route through fund_etf_hist_sina instead.
_ETF_PREFIXES = frozenset({"15", "16", "50", "51", "52", "56", "58"})
def _is_etf_listed(code: str) -> bool:
"""Detect exchange-listed ETF / LOF symbols (e.g. 518880.SH, 159915.SZ)."""
upper = code.upper()
if not upper.endswith((".SH", ".SZ")):
return False
digits = upper.split(".")[0]
if len(digits) != 6 or not digits.isdigit():
return False
return digits[:2] in _ETF_PREFIXES
def _is_forex(code: str) -> bool:
"""Detect forex pairs by matching against AKShare's symbol_market_map.
@@ -63,7 +61,9 @@ def _is_forex(code: str) -> bool:
Issue #54 — forex symbols (EURUSD, GBPUSD, etc.) have no exchange suffix
and previously fell through to the A-share endpoint.
"""
upper = code.upper().removesuffix(".FX")
# Accept the canonical slash form (EUR/USD) too, so the forex fallback
# chain (mt5 → akshare) actually engages for project-style codes.
upper = code.upper().removesuffix(".FX").replace("/", "")
try:
from akshare.forex.cons import symbol_market_map
except Exception:
@@ -116,7 +116,15 @@ class DataLoader:
result: Dict[str, pd.DataFrame] = {}
for code in codes:
try:
df = self._fetch_one(code, start_date, end_date, interval)
df = cached_loader_fetch(
source=self.name,
symbol=code,
timeframe=interval,
start_date=start_date,
end_date=end_date,
fields=None,
fetch=lambda code=code: self._fetch_one(code, start_date, end_date, interval),
)
if df is not None and not df.empty:
result[code] = df
except Exception as exc:
@@ -131,14 +139,18 @@ class DataLoader:
# ETF check must precede A-share — 518880.SH ends with .SH but is an ETF.
if _is_etf_listed(code):
_require_daily_interval(interval, "etf")
return self._fetch_etf(ak, code, start_date, end_date)
if _is_a_share(code):
return self._fetch_a_share(ak, code, start_date, end_date, interval)
if _is_us(code):
_require_daily_interval(interval, "us")
return self._fetch_us(ak, code, start_date, end_date)
if _is_hk(code):
_require_daily_interval(interval, "hk")
return self._fetch_hk(ak, code, start_date, end_date)
if _is_forex(code):
_require_daily_interval(interval, "forex")
return self._fetch_forex(ak, code, start_date, end_date)
# Default: try A-share
return self._fetch_a_share(ak, code, start_date, end_date, interval)
@@ -148,7 +160,12 @@ class DataLoader:
) -> Optional[pd.DataFrame]:
"""Fetch A-share via stock_zh_a_hist."""
symbol = code.split(".")[0]
period = _INTERVAL_MAP_DAILY.get(interval, "daily")
period = _INTERVAL_MAP_DAILY.get(interval)
if period is None:
raise ValueError(
f"Unsupported interval {interval!r}; akshare a-share supports "
f"{sorted(_INTERVAL_MAP_DAILY)}"
)
sd = start_date.replace("-", "")
ed = end_date.replace("-", "")
df = ak.stock_zh_a_hist(
@@ -204,7 +221,7 @@ class DataLoader:
— note ``最新价`` (latest) plays the role of close. Volume isn't reported,
so we synthesize a zero column to satisfy the OHLCV contract.
"""
symbol = code.upper().removesuffix(".FX")
symbol = code.upper().removesuffix(".FX").replace("/", "")
df = ak.forex_hist_em(symbol=symbol)
if df is None or df.empty:
return None
agent/backtest/loaders/alphavantage_loader.py
+259
View File
@@ -0,0 +1,259 @@
"""Alpha Vantage loader: US-equity daily OHLCV via the free key-gated REST API.
Alpha Vantage serves daily bars at ``/query?function=TIME_SERIES_DAILY`` and
gates access behind a free API key (env ``ALPHAVANTAGE_API_KEY``). The endpoint
rate-limits per IP, so every request routes through
:mod:`backtest.loaders._http` for per-host throttling and session reuse rather
than calling :mod:`requests` directly.
API format::
https://www.alphavantage.co/query?function=TIME_SERIES_DAILY
&symbol=AAPL&outputsize=full&apikey=<KEY>
The JSON payload nests the bars under ``"Time Series (Daily)"`` keyed by
``YYYY-MM-DD`` date strings, each mapping to numbered OHLCV fields
(``"1. open"`` .. ``"5. volume"``). Throttle/quota responses arrive as a plain
``"Note"`` / ``"Information"`` envelope, and bad symbols as ``"Error Message"``;
all three are surfaced as a transient-classified error so one symbol's failure
never aborts the batch.
"""
from __future__ import annotations
import logging
from typing import Dict, List, Optional
import pandas as pd
from backtest.loaders._http import resolve_min_interval, throttled_get_json
from backtest.loaders.base import cached_loader_fetch, validate_date_range
from backtest.loaders.registry import register
logger = logging.getLogger(__name__)
_BASE_URL = "https://www.alphavantage.co/query"
# Throttle/session bucket shared by all Alpha Vantage calls. The free tier is
# quota-limited per key/IP, so requests are spaced out by default.
_HOST_KEY = "alphavantage"
_MIN_INTERVAL_ENV = "VIBE_TRADING_ALPHAVANTAGE_MIN_INTERVAL"
_DEFAULT_MIN_INTERVAL = 1.0
_API_KEY_ENV = "ALPHAVANTAGE_API_KEY"
# Env values that are present-but-meaningless and must be treated as absent.
_API_KEY_PLACEHOLDERS = {"", "your-alphavantage-api-key", "demo"}
# Bars live under this top-level key; the per-bar OHLCV fields are numbered.
_TIME_SERIES_KEY = "Time Series (Daily)"
_FIELD_MAP = {
"open": "1. open",
"high": "2. high",
"low": "3. low",
"close": "4. close",
"volume": "5. volume",
}
_OHLCV_COLUMNS = ["open", "high", "low", "close", "volume"]
def _resolve_api_key() -> str:
"""Return the configured Alpha Vantage API key, or empty when unset.
Returns:
The trimmed key, or ``""`` when the env var is absent or a placeholder.
"""
from src.config.accessor import get_env_config
key = get_env_config().data.alphavantage_api_key.strip()
return "" if key in _API_KEY_PLACEHOLDERS else key
def _min_interval() -> float:
"""Resolve the per-call minimum Alpha Vantage request spacing in seconds."""
return resolve_min_interval(_MIN_INTERVAL_ENV, _DEFAULT_MIN_INTERVAL)
def _extract_provider_message(payload: object) -> Optional[str]:
"""Return any rate-limit / error envelope message from a payload.
Alpha Vantage signals quota exhaustion and bad requests in-band with a 200
status, using one of three string envelopes instead of the time series.
Args:
payload: The decoded JSON body.
Returns:
The envelope message when present, else ``None``.
"""
if not isinstance(payload, dict):
return None
for key in ("Error Message", "Note", "Information"):
message = payload.get(key)
if isinstance(message, str) and message.strip():
return message.strip()
return None
@register
class DataLoader:
"""Alpha Vantage US-equity daily OHLCV loader (key-gated REST, throttled)."""
name = "alphavantage"
markets = {"us_equity"}
requires_auth = True
def is_available(self) -> bool:
"""Available when a non-placeholder ``ALPHAVANTAGE_API_KEY`` is set."""
return bool(_resolve_api_key())
def __init__(self) -> None:
pass
def fetch(
self,
codes: List[str],
start_date: str,
end_date: str,
*,
interval: str = "1D",
fields: Optional[List[str]] = None,
) -> Dict[str, pd.DataFrame]:
"""Fetch daily OHLCV bars for ``codes`` within ``[start_date, end_date]``.
Args:
codes: US-equity tickers (e.g. ``["AAPL", "MSFT"]``).
start_date: Inclusive start date ``YYYY-MM-DD``.
end_date: Inclusive end date ``YYYY-MM-DD``.
interval: Bar interval; only ``"1D"`` (daily) is supported.
fields: Ignored — the full OHLCV column set is always returned.
Returns:
Mapping ``{symbol: DataFrame}`` with a ``trade_date`` DatetimeIndex
and float ``open/high/low/close/volume`` columns. Symbols that fail
or carry no in-range bars are omitted.
"""
validate_date_range(start_date, end_date)
# Daily-only TIME_SERIES_DAILY; do not silently return day bars for ``1H``.
if str(interval).strip().lower() not in {"1d", "d", "day", "daily"}:
logger.warning(
"alphavantage supports daily bars only; rejecting interval=%r",
interval,
)
return {}
api_key = _resolve_api_key()
if not api_key:
logger.warning("alphavantage skipped: %s not set", _API_KEY_ENV)
return {}
result: Dict[str, pd.DataFrame] = {}
for code in codes:
try:
df = cached_loader_fetch(
source=self.name,
symbol=code,
timeframe=interval,
start_date=start_date,
end_date=end_date,
fields=None,
fetch=lambda code=code: self._fetch_one(
code, start_date, end_date, api_key
),
)
if df is not None and not df.empty:
result[code] = df
except Exception as exc: # noqa: BLE001 - one bad symbol must not abort the batch
logger.warning("alphavantage failed for %s: %s", code, exc)
return result
def _fetch_one(
self, code: str, start_date: str, end_date: str, api_key: str,
) -> Optional[pd.DataFrame]:
"""Fetch and slice one symbol's daily bars into an OHLCV DataFrame.
Args:
code: US-equity ticker.
start_date: Inclusive start date ``YYYY-MM-DD``.
end_date: Inclusive end date ``YYYY-MM-DD``.
api_key: Resolved Alpha Vantage API key.
Returns:
The in-range OHLCV DataFrame, or ``None`` when the payload carries
no usable bars.
Raises:
ValueError: The provider returned a rate-limit / error envelope and
no usable time series (a data-bearing payload that also carries
an upsell note is parsed, not rejected).
requests.RequestException: Network failure, propagated unchanged.
"""
symbol = code.strip().upper()
if not symbol:
return None
payload = throttled_get_json(
_BASE_URL,
host_key=_HOST_KEY,
min_interval=_min_interval(),
params={
"function": "TIME_SERIES_DAILY",
"symbol": symbol,
"outputsize": "full",
"apikey": api_key,
},
)
series = payload.get(_TIME_SERIES_KEY) if isinstance(payload, dict) else None
if not isinstance(series, dict) or not series:
# No usable series: a premium-upsell payload can carry both the data
# key (empty/absent) and a Note/Information envelope, so only now —
# with nothing to parse — do we surface the envelope as an error.
message = _extract_provider_message(payload)
if message is not None:
raise ValueError(f"alphavantage rejected {symbol}: {message}")
return None
rows = [
row
for date, bar in series.items()
if (row := _parse_bar(date, bar)) is not None
]
if not rows:
return None
df = pd.DataFrame(rows)
df["trade_date"] = pd.to_datetime(df["trade_date"])
df = df.set_index("trade_date").sort_index()
df = df[_OHLCV_COLUMNS].dropna(subset=["open", "high", "low", "close"])
start = pd.Timestamp(start_date)
end = pd.Timestamp(end_date)
df = df.loc[(df.index >= start) & (df.index <= end)]
return df
def _parse_bar(date: str, bar: object) -> Optional[dict]:
"""Parse one ``{date: {"1. open": ...}}`` entry into an OHLCV row dict.
Args:
date: The bar's ``YYYY-MM-DD`` date string.
bar: The per-bar field mapping from the payload.
Returns:
A ``{trade_date, open, high, low, close, volume}`` dict, or ``None``
when the entry is malformed or non-numeric.
"""
if not isinstance(bar, dict):
return None
try:
return {
"trade_date": date,
"open": float(bar[_FIELD_MAP["open"]]),
"high": float(bar[_FIELD_MAP["high"]]),
"low": float(bar[_FIELD_MAP["low"]]),
"close": float(bar[_FIELD_MAP["close"]]),
"volume": float(bar[_FIELD_MAP["volume"]]),
}
except (KeyError, TypeError, ValueError):
return None
agent/backtest/loaders/baostock_loader.py
+163
View File
@@ -0,0 +1,163 @@
"""BaoStock loader: free, no-auth A-share data via TCP protocol.
BaoStock (http://baostock.com) uses its own TCP protocol (not HTTP),
bypassing CDN IP blocks that affect HTTP-based data sources like
eastmoney.com. Completely free, no API token required.
Covers: A-shares (SH/SZ), does NOT cover HK/US/crypto.
"""
from __future__ import annotations
import logging
from typing import Dict, List, Optional
import pandas as pd
from backtest.loaders.base import cached_loader_fetch, validate_date_range
from backtest.loaders.registry import register
logger = logging.getLogger(__name__)
def _is_a_share(code: str) -> bool:
# Support both baostock native format (sh.601398) and tushare-style suffix (601398.SH)
code_lower = code.lower()
return (
code_lower.startswith(("sh.", "sz."))
or code.upper().endswith((".SZ", ".SH"))
)
@register
class DataLoader:
"""BaoStock A-share OHLCV loader (free, TCP protocol, no auth)."""
name = "baostock"
markets = {"a_share"}
requires_auth = False
def is_available(self) -> bool:
"""Available if baostock is installed."""
try:
import baostock # noqa: F401
return True
except ImportError:
return False
def __init__(self) -> None:
pass
def fetch(
self,
codes: List[str],
start_date: str,
end_date: str,
*,
interval: str = "1D",
fields: Optional[List[str]] = None,
) -> Dict[str, pd.DataFrame]:
"""Fetch OHLCV data via BaoStock.
Args:
codes: Symbol list (e.g. ["601595.SH", "000001.SZ"]).
start_date: YYYY-MM-DD.
end_date: YYYY-MM-DD.
interval: Bar size (only 1D supported).
fields: Ignored.
Returns:
Mapping symbol -> OHLCV DataFrame.
"""
validate_date_range(start_date, end_date)
# Daily-only API; do not silently return day bars for runner ``1H``/``4H``.
if str(interval).strip().lower() not in {"1d", "d", "day", "daily"}:
logger.warning(
"baostock supports daily bars only; rejecting interval=%r",
interval,
)
return {}
import baostock as bs
lg = bs.login()
if lg.error_code != "0":
logger.error("baostock login failed: %s", lg.error_msg)
return {}
result: Dict[str, pd.DataFrame] = {}
try:
for code in codes:
try:
df = cached_loader_fetch(
source=self.name,
symbol=code,
timeframe=interval,
start_date=start_date,
end_date=end_date,
fields=None,
fetch=lambda code=code: self._fetch_one(bs, code, start_date, end_date),
)
if df is not None and not df.empty:
result[code] = df
except Exception as exc:
logger.warning("baostock failed for %s: %s", code, exc)
finally:
bs.logout()
return result
def _fetch_one(
self, bs, code: str, start_date: str, end_date: str,
) -> Optional[pd.DataFrame]:
"""Fetch a single A-share symbol."""
if not _is_a_share(code):
return None
# Support baostock native format (sh.601398 / sz.000001)
# and tushare-style suffix (601398.SH / 000001.SZ)
code_lower = code.lower()
if code_lower.startswith("sh.") or code_lower.startswith("sz."):
bs_code = code_lower
else:
parts = code.upper().split(".")
symbol = parts[0]
suffix = parts[1] if len(parts) > 1 else ""
if suffix == "SH":
bs_code = f"sh.{symbol}"
elif suffix == "SZ":
bs_code = f"sz.{symbol}"
else:
return None
rs = bs.query_history_k_data_plus(
bs_code,
"date,open,high,low,close,volume,amount",
start_date=start_date,
end_date=end_date,
frequency="d",
adjustflag="2", # 前复权
)
if rs.error_code != "0":
logger.warning("baostock query failed for %s: %s", code, rs.error_msg)
return None
rows = []
while rs.next():
rows.append(rs.get_row_data())
if not rows:
return None
df = pd.DataFrame(rows, columns=["date", "open", "high", "low", "close", "volume", "amount"])
df["date"] = pd.to_datetime(df["date"])
for col in ["open", "high", "low", "close", "volume", "amount"]:
df[col] = pd.to_numeric(df[col], errors="coerce")
df = df.rename(columns={"date": "trade_date"})
df = df.set_index("trade_date").sort_index()
df = df[["open", "high", "low", "close", "volume"]].dropna(
subset=["open", "high", "low", "close"]
)
return df
agent/backtest/loaders/base.py
+493 -1
View File
@@ -1,4 +1,4 @@
"""DataLoader Protocol, shared exceptions, and bounded-retry helpers.
"""DataLoader Protocol, shared exceptions, retry helpers, and loader cache.
The retry/budget helpers are the canonical pattern for any loader that calls
a flaky external API: a wall-clock deadline plus a small backoff schedule
@@ -9,11 +9,20 @@ re-implementing the loop.
from __future__ import annotations
import datetime as dt
import hashlib
import json
import logging
import os
import time
import uuid
from pathlib import Path
from typing import Callable, Protocol, TypeVar, runtime_checkable
import pandas as pd
logger = logging.getLogger(__name__)
class NoAvailableSourceError(Exception):
"""Raised when no data source is available for a given market."""
@@ -38,6 +47,78 @@ def validate_date_range(start_date: str, end_date: str) -> None:
raise ValueError(f"start_date ({start_date}) > end_date ({end_date})")
def validate_ohlc(
frame: pd.DataFrame,
*,
strategy: str = "drop",
allow_nonpositive_prices: bool = False,
) -> pd.DataFrame:
"""Drop, flag, or reject bars that violate OHLC invariants.
Loaders only drop NaN rows, so structurally dirty bars — ``high < low``,
a non-positive price, or a high/low that fails to bracket open/close —
flow straight into the backtest and surface downstream as NaN/inf metrics
that break the strict (``allow_nan=False``) JSON serializers. This is the
canonical loader-boundary check; call it after the existing ``dropna`` so a
single sanity pass guards every source.
Structural invariants (``high < low`` and high/low failing to bracket
open/close) are always enforced. The *positivity* invariant is
configurable: some markets clear at or below zero legitimately (European
day-ahead power routinely prints negative), and a rolling statistic over a
silently gap-filled series is worse than a well-defined negative bar. When
``allow_nonpositive_prices`` is set, negative prices pass through and only
an exactly-zero price is rejected — zero is genuinely undefined for
notional sizing (``size = notional / price``) and margin, whereas a
negative price is handled by ``abs()``-based sizing in the engine.
Args:
frame: OHLCV frame with at least ``open``/``high``/``low``/``close``
columns. NaN handling is left to the caller's ``dropna``.
strategy: ``"drop"`` (remove offending rows, default), ``"warn"``
(log and keep), or ``"raise"`` (raise on any violation).
allow_nonpositive_prices: when ``True``, keep bars with negative
prices and reject only exact zeros; when ``False`` (default,
unchanged behavior) reject any price ``<= 0``.
Returns:
The frame with invalid rows removed (``"drop"``) or unchanged
(``"warn"``). A frame that is empty or lacks OHLC columns is returned
as-is.
Raises:
ValueError: ``strategy="raise"`` and at least one bar is invalid.
"""
required = ("open", "high", "low", "close")
if frame.empty or not all(col in frame.columns for col in required):
return frame
open_, high, low, close = (frame[c] for c in required)
structural = (
(high < low)
| (high < open_)
| (high < close)
| (low > open_)
| (low > close)
)
if allow_nonpositive_prices:
nonpositive = (open_ == 0) | (high == 0) | (low == 0) | (close == 0)
else:
nonpositive = (open_ <= 0) | (high <= 0) | (low <= 0) | (close <= 0)
invalid = structural | nonpositive
n_invalid = int(invalid.sum())
if n_invalid == 0:
return frame
if strategy == "raise":
raise ValueError(f"{n_invalid} bar(s) violate OHLC invariants")
if strategy == "warn":
logger.warning("OHLC validation: %d bar(s) violate invariants (kept)", n_invalid)
return frame
logger.warning("OHLC validation: dropping %d invalid bar(s)", n_invalid)
return frame[~invalid]
# ---------------------------------------------------------------------------
# Bounded retry / budget helpers (shared by ccxt_loader, okx, and any future
# loader calling a flaky external API).
@@ -47,6 +128,38 @@ DEFAULT_BACKOFF: tuple[float, ...] = (0.5, 1.5, 4.0)
DEFAULT_MAX_RETRIES = 3
def positive_env_int(name: str, default: int) -> int:
"""Read a positive integer env var, warning and falling back on invalid values."""
raw = os.getenv(name) # noqa: env-gate — generic env var helper
if raw is None or not raw.strip():
return default
try:
value = int(raw)
except ValueError:
logger.warning("invalid %s=%r, using default %s", name, raw, default)
return default
if value <= 0:
logger.warning("non-positive %s=%r, using default %s", name, raw, default)
return default
return value
def positive_env_float(name: str, default: float) -> float:
"""Read a positive float env var, warning and falling back on invalid values."""
raw = os.getenv(name) # noqa: env-gate — generic env var helper
if raw is None or not raw.strip():
return default
try:
value = float(raw)
except ValueError:
logger.warning("invalid %s=%r, using default %s", name, raw, default)
return default
if value <= 0:
logger.warning("non-positive %s=%r, using default %s", name, raw, default)
return default
return value
def check_budget(deadline: float, label: str, budget_s: float | None = None) -> None:
"""Raise :class:`TimeoutError` if the monotonic clock has crossed ``deadline``.
@@ -123,6 +236,385 @@ def retry_with_budget(
raise AssertionError("unreachable: retry loop must return or raise") # pragma: no cover
# ---------------------------------------------------------------------------
# Opt-in local loader cache.
# ---------------------------------------------------------------------------
LOADER_CACHE_ENV = "VIBE_TRADING_DATA_CACHE"
LOADER_CACHE_ROOT_ENV = "VIBE_TRADING_DATA_CACHE_ROOT"
_LOADER_CACHE_TRUE_VALUES = {"1", "true", "yes", "on"}
# Bump when the key payload or on-disk layout changes so stale entries are
# simply never matched (old files become unreachable garbage, safe to delete).
_LOADER_CACHE_VERSION = 3
def loader_cache_enabled() -> bool:
"""Return whether the local market-data cache is explicitly enabled.
Returns:
True only when the config yields a real ``True``. Any other value —
including a truthy non-bool from a stubbed config — leaves the opt-in
cache off.
"""
from src.config.accessor import get_env_config
return get_env_config().data.vibe_trading_data_cache is True
def loader_cache_root() -> Path:
"""Return the root directory for opt-in loader cache files.
The configured override is honored only when it is a genuine non-blank
``str``. A non-string value (e.g. a stubbed config in tests) would
otherwise reach ``Path()`` via ``__fspath__`` and yield a *relative* path,
which resolves against the CWD and writes market data inside the working
tree — the repository forbids caching data in the repo.
Returns:
The configured cache root, or the default under the user's home.
"""
from src.config.accessor import get_env_config
root = get_env_config().data.vibe_trading_data_cache_root
if isinstance(root, str) and root.strip():
return Path(root).expanduser()
return Path.home() / ".vibe-trading" / "cache" / "loaders"
def make_loader_cache_key(
*,
source: str,
symbol: str,
timeframe: str,
start_date: str,
end_date: str,
fields: list[str] | tuple[str, ...] | None = None,
) -> str:
"""Build a stable content-addressed key for one loader payload."""
payload = _loader_cache_payload(
source=source,
symbol=symbol,
timeframe=timeframe,
start_date=start_date,
end_date=end_date,
fields=fields,
)
blob = json.dumps(payload, sort_keys=True, separators=(",", ":")).encode("utf-8")
return hashlib.sha256(blob).hexdigest()
def loader_cache_path(
*,
source: str,
symbol: str,
timeframe: str,
start_date: str,
end_date: str,
fields: list[str] | tuple[str, ...] | None = None,
) -> Path:
"""Return the parquet cache path for one loader payload."""
key = make_loader_cache_key(
source=source,
symbol=symbol,
timeframe=timeframe,
start_date=start_date,
end_date=end_date,
fields=fields,
)
source_dir = _sanitize_cache_segment(source)
return loader_cache_root() / source_dir / f"{key}.parquet"
def loader_cache_range_is_final(end_date: str) -> bool:
"""Return whether ``end_date`` is settled enough to cache.
The key is content-addressed on ``end_date`` but not on wall-clock fetch
time, so caching a range whose last bar is still forming (``end_date`` today
or in the future) would pin a provisional bar and serve it on every later
run. Only fully-elapsed days (strictly before today) are cacheable.
"""
try:
end = pd.Timestamp(end_date).normalize().date()
except Exception: # noqa: BLE001 - an unparseable date is treated as not cacheable
return False
return end < dt.date.today()
def loader_cache_get(
*,
source: str,
symbol: str,
timeframe: str,
start_date: str,
end_date: str,
fields: list[str] | tuple[str, ...] | None = None,
) -> pd.DataFrame | None:
"""Return a cached DataFrame for one payload, or ``None`` on any miss.
Misses include: cache disabled, range not yet settled, entry absent, or a
corrupt entry. A corrupt entry is non-fatal — the caller falls back to the
live provider.
"""
if not loader_cache_enabled() or not loader_cache_range_is_final(end_date):
return None
cache_path = loader_cache_path(
source=source,
symbol=symbol,
timeframe=timeframe,
start_date=start_date,
end_date=end_date,
fields=fields,
)
return _read_loader_cache_frame(cache_path)
def loader_cache_put(
*,
source: str,
symbol: str,
timeframe: str,
start_date: str,
end_date: str,
fields: list[str] | tuple[str, ...] | None,
frame: pd.DataFrame | None,
) -> None:
"""Write one non-empty DataFrame to the cache; a no-op when not cacheable.
Skips a disabled cache, an unsettled range, and empty/non-DataFrame results.
Write failures are swallowed so a fetch never fails because of the cache.
"""
if not loader_cache_enabled() or not loader_cache_range_is_final(end_date):
return
if not isinstance(frame, pd.DataFrame) or frame.empty:
return
cache_path = loader_cache_path(
source=source,
symbol=symbol,
timeframe=timeframe,
start_date=start_date,
end_date=end_date,
fields=fields,
)
_write_loader_cache_frame(cache_path, frame)
def cached_loader_fetch(
*,
source: str,
symbol: str,
timeframe: str,
start_date: str,
end_date: str,
fields: list[str] | tuple[str, ...] | None,
fetch: Callable[[], pd.DataFrame | None],
) -> pd.DataFrame | None:
"""Fetch one DataFrame through the opt-in local cache.
Convenience wrapper over :func:`loader_cache_get` / :func:`loader_cache_put`
for the common per-symbol loader loop: return the cached frame when present,
otherwise call ``fetch`` and cache a non-empty result. Cache read/write
failures are non-fatal and fall back to ``fetch``.
"""
cached = loader_cache_get(
source=source,
symbol=symbol,
timeframe=timeframe,
start_date=start_date,
end_date=end_date,
fields=fields,
)
if cached is not None:
return cached
frame = fetch()
loader_cache_put(
source=source,
symbol=symbol,
timeframe=timeframe,
start_date=start_date,
end_date=end_date,
fields=fields,
frame=frame,
)
return frame
def _loader_cache_payload(
*,
source: str,
symbol: str,
timeframe: str,
start_date: str,
end_date: str,
fields: list[str] | tuple[str, ...] | None,
) -> dict[str, object]:
return {
"version": _LOADER_CACHE_VERSION,
"source": str(source),
"symbol": str(symbol),
"timeframe": str(timeframe),
"start_date": _normalize_cache_date(start_date),
"end_date": _normalize_cache_date(end_date),
"fields": [str(field) for field in (fields or ())],
}
def _normalize_cache_date(value: str) -> str:
return pd.Timestamp(value).strftime("%Y-%m-%d")
def _sanitize_cache_segment(value: str) -> str:
cleaned = "".join(ch if ch.isalnum() or ch in {"-", "_"} else "_" for ch in value.strip().lower())
return cleaned or "unknown"
def _loader_cache_metadata_path(cache_path: Path) -> Path:
return cache_path.with_suffix(cache_path.suffix + ".json")
def _read_loader_cache_frame(cache_path: Path) -> pd.DataFrame | None:
if not cache_path.is_file():
return None
metadata_path = _loader_cache_metadata_path(cache_path)
try:
metadata = json.loads(metadata_path.read_text(encoding="utf-8"))
except Exception as exc: # noqa: BLE001 - local cache miss is non-fatal
logger.warning("loader cache metadata read failed for %s: %s", cache_path.name, exc)
return None
con = None
try:
import duckdb
con = duckdb.connect(database=":memory:")
frame = con.execute(
f"SELECT * FROM read_parquet({_duckdb_sql_string(cache_path)})"
).fetchdf()
except Exception as exc: # noqa: BLE001 - corrupt cache falls back to provider
logger.warning("loader cache read failed for %s: %s", cache_path.name, exc)
return None
finally:
if con is not None:
con.close()
index_columns = metadata.get("index_columns") or []
if index_columns:
missing = [column for column in index_columns if column not in frame.columns]
if missing:
logger.warning("loader cache %s missing index column(s): %s", cache_path.name, missing)
return None
frame = frame.set_index(index_columns)
frame.index.names = metadata.get("index_names") or index_columns
frame = _restore_cache_index_dtypes(frame, metadata.get("index_dtypes"))
frame.columns.name = metadata.get("columns_name")
return frame
def _restore_cache_index_dtypes(frame: pd.DataFrame, index_dtypes: object) -> pd.DataFrame:
"""Best-effort restore of the per-level index dtypes recorded at write time.
Cosmetic and non-fatal: duckdb parquet may rewrite datetime resolution, so
we cast each level back to its original dtype. A failed cast leaves the
duckdb-provided dtype rather than failing the read.
"""
if not isinstance(index_dtypes, list) or frame.index.nlevels != len(index_dtypes):
return frame
try:
if frame.index.nlevels == 1:
frame.index = frame.index.astype(index_dtypes[0])
else:
for level, dtype in enumerate(index_dtypes):
frame.index = frame.index.set_levels(
frame.index.levels[level].astype(dtype), level=level
)
except Exception: # noqa: BLE001 - index dtype restore is cosmetic
logger.debug("loader cache index dtype restore skipped: %s", index_dtypes)
return frame
def _write_loader_cache_frame(cache_path: Path, frame: pd.DataFrame) -> None:
metadata_path = _loader_cache_metadata_path(cache_path)
# pid + uuid so two concurrent writers of the same key never share a tmp
# path; os.replace then swaps each file in atomically.
unique = f"{os.getpid()}.{uuid.uuid4().hex}"
tmp_path = cache_path.with_name(f"{cache_path.name}.{unique}.tmp")
tmp_metadata_path = metadata_path.with_name(f"{metadata_path.name}.{unique}.tmp")
try:
cache_path.parent.mkdir(parents=True, exist_ok=True)
cache_frame, metadata = _frame_for_loader_cache(frame)
import duckdb
con = duckdb.connect(database=":memory:")
try:
con.register("cache_frame", cache_frame)
con.execute(f"COPY cache_frame TO {_duckdb_sql_string(tmp_path)} (FORMAT PARQUET)")
finally:
con.close()
tmp_metadata_path.write_text(
json.dumps(metadata, sort_keys=True, separators=(",", ":")),
encoding="utf-8",
)
os.replace(tmp_path, cache_path)
os.replace(tmp_metadata_path, metadata_path)
except Exception as exc: # noqa: BLE001 - cache write failures should not fail fetches
logger.warning("loader cache write failed for %s: %s", cache_path.name, exc)
for path in (tmp_path, tmp_metadata_path):
try:
path.unlink()
except FileNotFoundError:
pass
except OSError:
pass
def _frame_for_loader_cache(frame: pd.DataFrame) -> tuple[pd.DataFrame, dict[str, object]]:
cache_frame = frame.copy()
original_index_names = list(cache_frame.index.names)
columns_name = cache_frame.columns.name
index_dtypes = [
str(cache_frame.index.get_level_values(level).dtype)
for level in range(cache_frame.index.nlevels)
]
index_columns = _cache_index_columns(cache_frame)
cache_frame.index = cache_frame.index.set_names(index_columns)
metadata: dict[str, object] = {
"version": _LOADER_CACHE_VERSION,
"index_columns": index_columns,
"index_names": original_index_names,
# Preserve the columns-axis name (e.g. yfinance leaves "Price") and the
# per-level index dtypes so a cached frame round-trips byte-identical to
# a freshly fetched one (duckdb parquet otherwise rewrites datetime
# resolution, e.g. [s] -> [us]).
"columns_name": None if columns_name is None else str(columns_name),
"index_dtypes": index_dtypes,
}
return cache_frame.reset_index(), metadata
def _cache_index_columns(frame: pd.DataFrame) -> list[str]:
columns = {str(column) for column in frame.columns}
used: set[str] = set()
index_columns: list[str] = []
for pos, name in enumerate(frame.index.names):
base = str(name) if name is not None else f"__vibe_loader_index_{pos}__"
candidate = base
suffix = 1
while candidate in columns or candidate in used:
candidate = f"{base}_{suffix}"
suffix += 1
index_columns.append(candidate)
used.add(candidate)
return index_columns
def _duckdb_sql_string(path: Path) -> str:
return "'" + str(path).replace("'", "''") + "'"
@runtime_checkable
class DataLoaderProtocol(Protocol):
"""Interface that every data source loader must satisfy."""
agent/backtest/loaders/binance_loader.py
+44
View File
@@ -0,0 +1,44 @@
"""Binance spot / USD-M perpetual OHLCV loader (via CCXT).
Dedicated source name ``binance`` so it can sit **alongside** ``okx`` in the
crypto auto-fallback chain — not as a replacement. Public market data only;
no API key required.
Use explicitly with ``source="binance"``, or let ``source="auto"`` fall through
to Binance when OKX is unavailable.
"""
from __future__ import annotations
from typing import Any
from backtest.loaders.ccxt_loader import (
_CCXT_TIMEOUT_MS,
_ccxt_proxy_config,
)
from backtest.loaders.ccxt_loader import DataLoader as CcxtDataLoader
from backtest.loaders.registry import register
@register
class DataLoader(CcxtDataLoader):
"""Binance-only crypto OHLCV loader (public REST via CCXT)."""
name = "binance"
markets = {"crypto"}
requires_auth = False
def _get_exchange(self, instrument_type: str = "spot") -> Any:
"""Always use Binance spot or Binance USD-M (swap), ignore CCXT_EXCHANGE."""
import ccxt
if instrument_type == "swap":
exchange_cls = ccxt.binanceusdm
else:
exchange_cls = ccxt.binance
config: dict[str, Any] = {"enableRateLimit": True, "timeout": _CCXT_TIMEOUT_MS}
proxies = _ccxt_proxy_config()
if proxies:
config["proxies"] = proxies
return exchange_cls(config)
agent/backtest/loaders/ccxt_loader.py
+330 -13
View File
@@ -7,15 +7,21 @@ No API key required for public market data.
from __future__ import annotations
import hashlib
import json
import logging
import os
import re
import time
from typing import Dict, List, Optional
import pandas as pd
from backtest.loaders.base import (
cached_loader_fetch,
check_budget,
positive_env_float,
positive_env_int,
retry_with_budget,
validate_date_range,
)
@@ -25,7 +31,20 @@ logger = logging.getLogger(__name__)
_INTERVAL_MAP = {
"1m": "1m", "5m": "5m", "15m": "15m", "30m": "30m",
"1H": "1h", "4H": "4h", "1D": "1d",
"1H": "1h", "1h": "1h", "4H": "4h", "4h": "4h",
"1D": "1d", "1d": "1d", "1W": "1w", "1w": "1w", "1M": "1M",
}
_TIMEFRAME_DELTA = {
"1m": pd.Timedelta(minutes=1),
"5m": pd.Timedelta(minutes=5),
"15m": pd.Timedelta(minutes=15),
"30m": pd.Timedelta(minutes=30),
"1h": pd.Timedelta(hours=1),
"4h": pd.Timedelta(hours=4),
"1d": pd.Timedelta(days=1),
"1w": pd.Timedelta(weeks=1),
"1M": pd.Timedelta(days=31),
}
# P12-b: ccxt had no request timeout and an unbounded paginated fetch with
@@ -33,13 +52,27 @@ _INTERVAL_MAP = {
# minutes. Cap each HTTP call, bound transient retries, and enforce a hard
# wall-clock budget so the fetch fails fast instead of hanging. Retry
# scheduling is delegated to :mod:`backtest.loaders.base`.
_CCXT_TIMEOUT_MS = int(os.getenv("CCXT_TIMEOUT_MS", "15000"))
_CCXT_FETCH_BUDGET_S = float(os.getenv("CCXT_FETCH_BUDGET_S", "60"))
_CCXT_TIMEOUT_MS = positive_env_int("CCXT_TIMEOUT_MS", 15_000)
_CCXT_FETCH_BUDGET_S = positive_env_float("CCXT_FETCH_BUDGET_S", 60.0)
_FUNDING_HOURS = {0, 8, 16}
def _parse_ccxt_symbol(code: str) -> tuple[str, str]:
"""Return the canonical CCXT symbol and instrument type for ``code``."""
normalized = code.strip().upper()
if normalized.endswith("-PERP"):
match = re.fullmatch(r"([A-Z0-9]+)-USDT-PERP", normalized)
if match is None:
raise ValueError(
"USD-M perpetual symbol must use BASE-USDT-PERP, e.g. BTC-USDT-PERP"
)
return f"{match.group(1)}/USDT:USDT", "swap"
return normalized.replace("-", "/"), "spot"
def _first_proxy_env(*names: str) -> str:
for name in names:
value = os.getenv(name, "").strip()
value = os.getenv(name, "").strip() # noqa: env-gate — system proxy vars
if value:
return value
return ""
@@ -59,6 +92,95 @@ def _ccxt_proxy_config() -> dict[str, str]:
return proxies
_BRACKET_SCHEMA_VERSION = 1
def _validate_bracket_artifact(artifact: dict, *, expected_symbol: str) -> tuple[list[dict], str]:
"""Validate a caller-supplied maintenance-bracket artifact.
``binanceusdm.fetch_leverage_tiers()`` routes to Binance's signed
``GET /fapi/v1/leverageBracket`` USER_DATA endpoint — it requires an API
key even though the response carries no account-specific data. This
loader no longer calls it: maintenance brackets are supplied out of band
as a versioned artifact and validated here, never fetched live. Fails
closed (raises ``ValueError``) on any schema, symbol, provenance,
ordering, or content-hash problem.
"""
if not isinstance(artifact, dict):
raise ValueError("bracket artifact must be a dict")
schema_version = artifact.get("schema_version")
if schema_version != _BRACKET_SCHEMA_VERSION:
raise ValueError(
f"bracket artifact schema_version must be {_BRACKET_SCHEMA_VERSION}, "
f"got {schema_version!r}"
)
symbol = artifact.get("symbol")
if symbol != expected_symbol:
raise ValueError(
f"bracket artifact symbol mismatch: expected {expected_symbol!r}, got {symbol!r}"
)
provenance_timestamp = artifact.get("provenance_timestamp")
if not provenance_timestamp:
raise ValueError("bracket artifact is missing provenance_timestamp")
try:
pd.Timestamp(provenance_timestamp)
except (ValueError, TypeError) as exc:
raise ValueError(
f"bracket artifact provenance_timestamp is not a valid timestamp: "
f"{provenance_timestamp!r}"
) from exc
brackets = artifact.get("brackets")
if not brackets:
raise ValueError("bracket artifact has no brackets")
normalized: list[dict] = []
for tier in brackets:
try:
record = {
"bracket_tier": int(tier["bracket_tier"]),
"notional_cap": float(tier["notional_cap"]),
"maintenance_rate": float(tier["maintenance_rate"]),
"cumulative_maintenance_amount": float(tier["cumulative_maintenance_amount"]),
}
except (KeyError, TypeError, ValueError) as exc:
raise ValueError(
f"bracket artifact tier for {expected_symbol} is missing a required field: {exc}"
) from exc
# Optional: reserved for future risk-model calibration, not part of
# Binance's own bracket schema. Validated only if present.
coefficient = tier.get("notional_coefficient")
if coefficient is not None:
try:
record["notional_coefficient"] = float(coefficient)
except (TypeError, ValueError) as exc:
raise ValueError(
f"bracket artifact notional_coefficient must be numeric: {coefficient!r}"
) from exc
normalized.append(record)
normalized.sort(key=lambda row: row["bracket_tier"])
caps = [row["notional_cap"] for row in normalized]
if caps != sorted(caps) or len(caps) != len(set(caps)):
raise ValueError(
f"bracket artifact notional caps for {expected_symbol} are not strictly increasing"
)
blob = json.dumps(normalized, sort_keys=True, separators=(",", ":")).encode("utf-8")
computed_hash = hashlib.sha256(blob).hexdigest()[:16]
content_hash = artifact.get("content_hash")
if content_hash != computed_hash:
raise ValueError(
f"bracket artifact content_hash mismatch for {expected_symbol}: "
f"expected {computed_hash}, artifact declares {content_hash!r}"
)
return normalized, computed_hash
@register
class DataLoader:
"""CCXT-backed crypto OHLCV loader (100+ exchanges)."""
@@ -78,10 +200,18 @@ class DataLoader:
def __init__(self) -> None:
pass
def _get_exchange(self):
"""Create exchange instance."""
def _get_exchange(self, instrument_type: str = "spot"):
"""Create an exchange instance for spot or Binance USD-M swaps."""
import ccxt
exchange_id = os.getenv("CCXT_EXCHANGE", "binance").lower()
from src.config.accessor import get_env_config
exchange_id = get_env_config().data.ccxt_exchange.lower()
if instrument_type == "swap":
if exchange_id not in {"binance", "binanceusdm"}:
raise ValueError(
"BASE-USDT-PERP currently requires CCXT_EXCHANGE=binance"
)
exchange_id = "binanceusdm"
exchange_cls = getattr(ccxt, exchange_id, None)
if exchange_cls is None:
logger.warning("Unknown CCXT exchange %s, falling back to binance", exchange_id)
@@ -101,6 +231,8 @@ class DataLoader:
*,
interval: str = "1D",
fields: Optional[List[str]] = None,
bracket_artifacts: Optional[Dict[str, dict]] = None,
require_brackets: bool = False,
) -> Dict[str, pd.DataFrame]:
"""Fetch crypto OHLCV via CCXT.
@@ -110,31 +242,196 @@ class DataLoader:
end_date: End date (YYYY-MM-DD).
interval: Bar size.
fields: Ignored.
bracket_artifacts: Optional ``{code: artifact}`` map of caller-supplied,
versioned maintenance-bracket artifacts (see
``_validate_bracket_artifact``) for ``-PERP`` codes. Normal
execution/mark/funding data never needs this — it stays
zero-credential regardless. This loader does not fetch brackets
live: ``binanceusdm.fetch_leverage_tiers()`` requires a Binance
API key, which this historical/backtest path never carries.
require_brackets: When True, a ``-PERP`` code without a matching,
valid artifact in ``bracket_artifacts`` fails closed instead of
silently returning data without bracket columns. Intended for
strict margin-risk consumers.
Returns:
Mapping symbol -> OHLCV DataFrame.
"""
validate_date_range(start_date, end_date)
exchange = self._get_exchange()
timeframe = _INTERVAL_MAP.get(interval, "1d")
since_ms = int(pd.Timestamp(start_date).timestamp() * 1000)
end_ms = int((pd.Timestamp(end_date) + pd.Timedelta(days=1)).timestamp() * 1000)
# Build the exchange lazily so a full cache hit never imports ccxt or
# opens an exchange object.
exchange_holder: Dict[str, object] = {}
def get_exchange(instrument_type: str):
if instrument_type not in exchange_holder:
exchange_holder[instrument_type] = self._get_exchange(instrument_type)
return exchange_holder[instrument_type]
result: Dict[str, pd.DataFrame] = {}
for code in codes:
instrument_type = "spot"
try:
ccxt_symbol = code.replace("-", "/").upper()
df = self._fetch_one(exchange, ccxt_symbol, timeframe, since_ms, end_ms)
ccxt_symbol, instrument_type = _parse_ccxt_symbol(code)
exchange = get_exchange(instrument_type)
artifact = (bracket_artifacts or {}).get(code)
def fetch_frame():
if instrument_type == "swap":
return self._fetch_perpetual(
exchange, ccxt_symbol, timeframe, since_ms, end_ms,
bracket_artifact=artifact, require_brackets=require_brackets,
)
return self._fetch_one(
exchange, ccxt_symbol, timeframe, since_ms, end_ms
)
df = cached_loader_fetch(
source=self.name,
symbol=code,
timeframe=interval,
start_date=start_date,
end_date=end_date,
fields=None,
fetch=fetch_frame,
)
if df is not None and not df.empty:
result[code] = df
except Exception as exc:
if instrument_type == "swap" or code.strip().upper().endswith("-PERP"):
raise
logger.warning("CCXT failed for %s: %s", code, exc)
return result
@classmethod
def _fetch_perpetual(
cls, exchange, symbol: str, timeframe: str, since_ms: int, end_ms: int,
*, bracket_artifact: dict | None = None, require_brackets: bool = False,
) -> pd.DataFrame:
"""Fetch aligned trade-price and mark-price candles for one USD-M swap.
Maintenance brackets are never fetched live (see
``_validate_bracket_artifact``): they're attached only when the caller
supplies a validated artifact. A strict caller (``require_brackets``)
fails closed before any network call when no artifact is supplied.
"""
if require_brackets and bracket_artifact is None:
raise ValueError(
f"strict margin-risk fetch for {symbol} requires a maintenance-bracket "
"artifact, but none was supplied. This loader does not perform a live "
"authenticated bracket fetch (binanceusdm.fetch_leverage_tiers requires "
"a Binance API key this backtest path does not carry) — pass a validated "
"artifact via DataLoader.fetch(bracket_artifacts={code: artifact})."
)
trade = cls._fetch_one(exchange, symbol, timeframe, since_ms, end_ms)
mark = cls._fetch_one(
exchange,
symbol,
timeframe,
since_ms,
end_ms,
params={"price": "mark"},
)
if trade is None or mark is None or not trade.index.equals(mark.index):
raise ValueError(f"mark-price timestamps are incomplete or unsynchronized for {symbol}")
result = trade.copy()
result["execution_open"] = trade["open"]
for column in ("open", "high", "low", "close"):
result[f"mark_{column}"] = mark[column]
funding = cls._fetch_funding_history(exchange, symbol, since_ms, end_ms)
if funding.index.has_duplicates:
raise ValueError(f"duplicate funding settlement for {symbol}")
required = result.index[result.index.hour.isin(_FUNDING_HOURS)]
missing = required.difference(funding.index)
if not missing.empty:
raise ValueError(
f"funding settlement data is missing for {symbol}: "
f"{', '.join(str(ts) for ts in missing)}"
)
result["funding_rate"] = 0.0
result["funding_settlement_time"] = pd.NaT
aligned = funding.index.intersection(result.index)
if not aligned.empty:
result.loc[aligned, "funding_rate"] = funding.loc[aligned, "funding_rate"]
result.loc[aligned, "funding_settlement_time"] = aligned
if bracket_artifact is not None:
brackets, version = _validate_bracket_artifact(
bracket_artifact, expected_symbol=symbol
)
result["maintenance_brackets"] = json.dumps(brackets)
result["maintenance_bracket_version"] = version
return result
@staticmethod
def _fetch_funding_history(
exchange, symbol: str, since_ms: int, end_ms: int,
) -> pd.DataFrame:
"""Fetch bounded historical funding settlements for one USD-M swap."""
import ccxt
rows: list[dict] = []
cursor = since_ms
limit = 1000
deadline = time.monotonic() + _CCXT_FETCH_BUDGET_S
label = f"ccxt funding fetch for {symbol}"
for _ in range(200):
check_budget(deadline, label, budget_s=_CCXT_FETCH_BUDGET_S)
page = retry_with_budget(
lambda: exchange.fetch_funding_rate_history(
symbol, since=cursor, limit=limit
),
transient=ccxt.NetworkError,
deadline=deadline,
label=label,
)
if not page:
break
rows.extend(page)
last_ts = int(page[-1]["timestamp"])
if last_ts >= end_ms or len(page) < limit:
break
cursor = last_ts + 1
if not rows:
return pd.DataFrame(
{"funding_rate": pd.Series(dtype=float)},
index=pd.DatetimeIndex([], name="trade_date"),
)
frame = pd.DataFrame({
# Binance settlement timestamps carry millisecond jitter
# (e.g. 08:00:00.011); round to the second so they align with
# bar timestamps instead of failing the missing-settlement check.
"trade_date": pd.to_datetime(
[row["timestamp"] for row in rows], unit="ms"
).round("s"),
"funding_rate": pd.to_numeric(
[row["fundingRate"] for row in rows], errors="raise"
),
}).set_index("trade_date").sort_index()
start_dt = pd.Timestamp(since_ms, unit="ms")
end_dt = pd.Timestamp(end_ms, unit="ms")
return frame[(frame.index >= start_dt) & (frame.index < end_dt)]
@staticmethod
def _fetch_one(
exchange, symbol: str, timeframe: str, since_ms: int, end_ms: int,
exchange,
symbol: str,
timeframe: str,
since_ms: int,
end_ms: int,
*,
params: dict[str, str] | None = None,
) -> Optional[pd.DataFrame]:
"""Paginated OHLCV fetch for one symbol."""
import ccxt
@@ -144,23 +441,32 @@ class DataLoader:
limit = 1000
deadline = time.monotonic() + _CCXT_FETCH_BUDGET_S
label = f"ccxt fetch for {symbol}"
hit_page_cap = True
for _ in range(200):
check_budget(deadline, label, budget_s=_CCXT_FETCH_BUDGET_S)
# ``ccxt.NetworkError`` covers RequestTimeout / DDoSProtection /
# ExchangeNotAvailable — the transient family. Anything else
# (e.g. ``ExchangeError`` for a bad symbol) is not retried.
def fetch_page():
kwargs = {"since": cursor, "limit": limit}
if params is not None:
kwargs["params"] = params
return exchange.fetch_ohlcv(symbol, timeframe, **kwargs)
ohlcv = retry_with_budget(
lambda: exchange.fetch_ohlcv(symbol, timeframe, since=cursor, limit=limit),
fetch_page,
transient=ccxt.NetworkError,
deadline=deadline,
label=label,
)
if not ohlcv:
hit_page_cap = False
break
all_rows.extend(ohlcv)
last_ts = ohlcv[-1][0]
if last_ts >= end_ms or len(ohlcv) < limit:
hit_page_cap = False
break
cursor = last_ts + 1
@@ -181,4 +487,15 @@ class DataLoader:
df = df[["open", "high", "low", "close", "volume"]].dropna(
subset=["open", "high", "low", "close"]
)
return df if not df.empty else None
if df.empty:
return None
tolerance = _TIMEFRAME_DELTA.get(timeframe)
if tolerance is None:
raise ValueError(f"unsupported CCXT timeframe: {timeframe}")
if hit_page_cap and df.index[-1] < end_dt - tolerance:
raise ValueError(
f"incomplete CCXT history for {symbol}: requested "
f"[{start_dt}, {end_dt}), received [{df.index[0]}, {df.index[-1]}]"
)
return df
agent/backtest/loaders/cn_adjust.py
+77
View File
@@ -0,0 +1,77 @@
"""Corporate-action adjustment for Tushare A-share and fund bars.
Tushare's ``daily`` and ``fund_daily`` endpoints return *unadjusted* prices, so
a close-to-close return taken across an ex-date spans the mechanical price drop
of a split, bonus issue or dividend rather than the instrument's actual move.
Measured against Tushare's own ``pct_chg`` over 2020-2024: 300750.SZ on
2023-04-26 reads -41.82% raw against a true +5.40%, 300124.SZ -34.26% against
-1.01%, 601012.SH -24.21% against +6.46%. The error is always negative, because
prices only ever gap down on an ex-date, so it is a systematic contaminant of
returns rather than noise.
The matching factor series comes from ``pro.adj_factor`` for equities and
``pro.fund_adj`` for funds; both carry the same ``(trade_date, adj_factor)``
schema. Factors are cumulative and rise across a corporate action — verified on
600519.SH 2022-06-30, where the factor steps 7.4740 -> 7.5546 as the stock goes
ex-dividend.
This module is the single implementation; the backtest loader and the alpha
bench both route through it rather than re-deriving the convention.
"""
from __future__ import annotations
import pandas as pd
def apply_qfq(df: pd.DataFrame, factor: pd.DataFrame | None) -> pd.DataFrame | None:
"""Forward-adjust raw Tushare daily bars for corporate actions.
``pro.daily`` returns *unadjusted* prices, so a close-to-close return taken
across an ex-date spans the mechanical price drop of a split, bonus issue or
dividend rather than the stock's actual move. Measured against Tushare's own
``pct_chg`` on CSI300 members: 300750.SZ on 2023-04-26 reads -41.82% raw
against a true +5.40%, 300124.SZ on 2021-06-04 reads -34.26% against -1.01%,
601012.SH on 2022-06-06 reads -24.21% against +6.46%. The error is always
negative, so it is a systematic contaminant of every cross-sectional IC the
bench reports, not noise.
Prices are scaled to the last bar in the window (前复权), matching the
``adjust="qfq"`` convention the akshare and tencent loaders already use.
``volume`` is divided by the same ratio so share counts stay on one basis
and the ``amount``-derived VWAP keeps the same relationship to ``close``;
``amount`` is a cash figure and is left untouched.
Args:
df: Raw daily bars indexed by trade date.
factor: ``pro.adj_factor`` rows for the same symbol and window.
Returns:
The adjusted frame, or ``None`` when the adjustment factors are missing
or unusable — the caller drops the symbol rather than benching it on
contaminated prices.
"""
if factor is None or getattr(factor, "empty", True):
return None
if "adj_factor" not in factor.columns or "trade_date" not in factor.columns:
return None
series = factor.copy()
series["trade_date"] = pd.to_datetime(series["trade_date"], errors="coerce")
series = (
series.dropna(subset=["trade_date"])
.set_index("trade_date")["adj_factor"]
.astype(float)
.sort_index()
)
series = series.reindex(df.index).ffill().bfill()
if series.isna().any() or (series <= 0).any():
return None
ratio = series / series.iloc[-1]
out = df.copy()
for col in ("open", "high", "low", "close"):
if col in out.columns:
out[col] = out[col] * ratio
if "volume" in out.columns:
out["volume"] = out["volume"] / ratio
return out
agent/backtest/loaders/eastmoney_client.py
+324
View File
@@ -0,0 +1,324 @@
"""Shared Eastmoney HTTP client: secid resolution + push2his kline fetch.
Eastmoney exposes free, no-auth quote endpoints but rate-limits aggressively by
source IP and will temporarily ban a bursting client, so every request here
routes through :mod:`backtest.loaders._http` for per-host throttling and session
reuse. This module is provider-internal plumbing shared by the Eastmoney-backed
loaders; it only knows Eastmoney's ``secid`` addressing scheme and the
``push2his`` kline JSON layout, not any loader's DataFrame conventions.
Eastmoney addresses every instrument by a ``secid`` of the form ``<market>.<code>``:
* A-shares — Shanghai (``.SH``, plus ``.BJ`` Beijing exchange) use market ``1``
for SH and ``0`` for SZ/BJ.
* Hong Kong (``.HK``) uses market ``116`` with the numeric code zero-padded to
five digits.
* US (``.US``) markets (NASDAQ ``105`` / NYSE ``106`` / AMEX ``107``) are not
derivable from the ticker alone, so the market prefix is discovered once via
Eastmoney's search/suggest endpoint and cached for the life of the process.
"""
from __future__ import annotations
import json
import logging
import re
from typing import Any
from backtest.loaders._http import resolve_min_interval, throttled_get_json
logger = logging.getLogger(__name__)
# Eastmoney kline endpoints. push2his serves historical bars; searchapi resolves
# a free-text ticker to its fully-qualified secid (needed for US tickers).
_KLINE_URL = "https://push2his.eastmoney.com/api/qt/stock/kline/get"
_SEARCH_URL = "https://searchapi.eastmoney.com/api/suggest/get"
# Throttle/session bucket shared by all Eastmoney calls.
_HOST_KEY = "eastmoney"
_MIN_INTERVAL_ENV = "VIBE_TRADING_EASTMONEY_MIN_INTERVAL"
_DEFAULT_MIN_INTERVAL = 1.0
# Eastmoney kline period codes (``klt``) keyed by our interval labels.
KLT_BY_INTERVAL: dict[str, int] = {
"1D": 101,
"1d": 101,
"1W": 102,
"1w": 102,
"1M": 103,
"1m": 1,
"5m": 5,
"15m": 15,
"30m": 30,
"1H": 60,
"60m": 60,
}
# push2his column selectors: fields1 picks the per-request meta, fields2 picks
# the per-bar columns in order: date, open, close, high, low, volume, amount.
_FIELDS1 = "f1,f2,f3,f4,f5,f6"
_FIELDS2 = "f51,f52,f53,f54,f55,f56,f57"
# Process-level cache of resolved US secids so a repeated ticker never re-hits
# the search endpoint. Keyed by the upper-cased bare ticker (e.g. "AAPL").
_US_SECID_CACHE: dict[str, str | None] = {}
# A JSONP envelope is a JS callback identifier followed by a parenthesized body,
# optionally terminated by ';'. The identifier is restricted to legal JS names
# (incl. dotted namespaces like "jQuery.cb") so a plain JSON array/object body —
# which never starts this way — is left untouched.
_JSONP_WRAPPER = re.compile(
r"^[A-Za-z_$][\w$]*(?:\.[A-Za-z_$][\w$]*)*\s*\((.*)\)\s*;?\s*$",
re.DOTALL,
)
def _min_interval() -> float:
"""Resolve the per-call minimum Eastmoney request spacing in seconds."""
return resolve_min_interval(_MIN_INTERVAL_ENV, _DEFAULT_MIN_INTERVAL)
def get_json(url: str, *, params: dict[str, Any]) -> Any:
"""Issue a throttled Eastmoney GET and decode the body as JSON.
Args:
url: Fully-qualified Eastmoney endpoint URL.
params: Query parameters for the request.
Returns:
The decoded JSON payload (typically a ``dict``).
Raises:
requests.RequestException: Network failure, propagated for the caller's
retry policy.
requests.HTTPError: Non-2xx response status.
ValueError: Body is not valid JSON.
"""
return throttled_get_json(
url,
host_key=_HOST_KEY,
min_interval=_min_interval(),
params=params,
)
def _resolve_a_share_secid(code: str, suffix: str) -> str | None:
"""Map an A-share ``code`` + exchange ``suffix`` to its Eastmoney secid.
SH instruments live on market ``1``; SZ and BJ (Beijing exchange) on ``0``.
"""
if suffix == "SH":
return f"1.{code}"
if suffix in ("SZ", "BJ"):
return f"0.{code}"
return None
def _parse_us_secid(payload: Any) -> str | None:
"""Extract a US ``<market>.<code>`` secid from a search/suggest payload.
Eastmoney's suggest endpoint returns candidates under
``QuotationCodeTable.Data``, each carrying a ``QuoteID`` already in
``<market>.<code>`` form. The first US-market (105/106/107) candidate wins.
Args:
payload: Decoded JSON from the search endpoint.
Returns:
The first matching US secid, or ``None`` when no US candidate is found.
"""
if not isinstance(payload, dict):
return None
table = payload.get("QuotationCodeTable")
if not isinstance(table, dict):
return None
candidates = table.get("Data")
if not isinstance(candidates, list):
return None
for candidate in candidates:
if not isinstance(candidate, dict):
continue
quote_id = candidate.get("QuoteID")
if not isinstance(quote_id, str) or "." not in quote_id:
continue
market = quote_id.split(".", 1)[0]
if market in ("105", "106", "107"):
return quote_id
return None
def _strip_jsonp(text: str) -> Any:
"""Decode a possibly JSONP-wrapped body to a Python object.
The suggest endpoint sometimes wraps its JSON in a ``callback(...)`` envelope.
A plain JSON body is parsed as-is first; only a body that actually matches a
leading callback identifier is unwrapped. This avoids corrupting plain JSON
whose string values contain parentheses (an earlier first-``(`` / last-``)``
slice mangled such bodies and returned ``None``).
Args:
text: Raw response body.
Returns:
The decoded object, or ``None`` when nothing parseable is found.
"""
stripped = text.strip()
try:
return json.loads(stripped)
except (ValueError, TypeError):
pass
match = _JSONP_WRAPPER.match(stripped)
if match is None:
return None
try:
return json.loads(match.group(1).strip())
except (ValueError, TypeError):
return None
def _resolve_us_secid(code: str) -> str | None:
"""Resolve a US ticker to its Eastmoney secid via search, with caching.
Args:
code: Bare upper-cased US ticker (e.g. ``"AAPL"``).
Returns:
The resolved ``<market>.<code>`` secid, or ``None`` when unresolvable.
"""
if code in _US_SECID_CACHE:
return _US_SECID_CACHE[code]
secid: str | None = None
try:
payload = get_json(
_SEARCH_URL,
params={"input": code, "type": "14", "count": "10"},
)
if isinstance(payload, str):
payload = _strip_jsonp(payload)
secid = _parse_us_secid(payload)
except Exception as exc: # noqa: BLE001 - an unresolved ticker is non-fatal
logger.warning("eastmoney secid search failed for %s: %s", code, exc)
secid = None
_US_SECID_CACHE[code] = secid
return secid
def resolve_secid(symbol: str) -> str | None:
"""Map a Vibe-Trading symbol to its Eastmoney secid.
Supported suffixes: ``.SH`` / ``.SZ`` / ``.BJ`` (A-share), ``.HK`` (Hong
Kong, code zero-padded to five digits), ``.US`` (resolved via search and
cached). A symbol with no recognized suffix, or a US ticker the search
cannot place, returns ``None``.
Args:
symbol: Symbol such as ``"600519.SH"``, ``"00700.HK"`` or ``"AAPL.US"``.
Returns:
The Eastmoney secid (e.g. ``"1.600519"``), or ``None`` if unresolvable.
"""
if not symbol or "." not in symbol:
return None
code, _, suffix = symbol.rpartition(".")
code = code.strip().upper()
suffix = suffix.strip().upper()
if not code:
return None
if suffix in ("SH", "SZ", "BJ"):
return _resolve_a_share_secid(code, suffix)
if suffix == "HK":
return f"116.{code.zfill(5)}"
if suffix == "US":
return _resolve_us_secid(code)
return None
def _parse_kline_row(raw: str) -> dict[str, Any] | None:
"""Parse one comma-joined push2his kline row into an OHLCV dict.
Column order follows ``fields2``: date, open, close, high, low, volume,
amount.
Args:
raw: One row string from ``data.klines``.
Returns:
A dict ``{trade_date, open, high, low, close, volume, amount}``, or
``None`` when the row is malformed.
"""
parts = raw.split(",")
if len(parts) < 7:
return None
try:
return {
"trade_date": parts[0],
"open": float(parts[1]),
"close": float(parts[2]),
"high": float(parts[3]),
"low": float(parts[4]),
"volume": float(parts[5]),
"amount": float(parts[6]),
}
except (ValueError, TypeError):
return None
def fetch_kline(
secid: str,
*,
klt: int,
fqt: int = 1,
beg: str = "0",
end: str = "20500101",
) -> list[dict]:
"""Fetch ascending OHLCV bars for one ``secid`` from push2his.
Args:
secid: Eastmoney secid (e.g. ``"1.600519"``).
klt: Period code from :data:`KLT_BY_INTERVAL`.
fqt: Adjustment mode (0 raw, 1 forward-adjusted, 2 back-adjusted).
beg: Inclusive start date ``YYYYMMDD`` or ``"0"`` for earliest.
end: Inclusive end date ``YYYYMMDD``.
Returns:
Ascending list of ``{trade_date, open, high, low, close, volume,
amount}`` dicts. Empty when the payload carries no bars.
Raises:
requests.RequestException: Network failure.
requests.HTTPError: Non-2xx response status.
ValueError: Body is not valid JSON.
"""
payload = get_json(
_KLINE_URL,
params={
"secid": secid,
"klt": str(klt),
"fqt": str(fqt),
"beg": beg,
"end": end,
"fields1": _FIELDS1,
"fields2": _FIELDS2,
"rev": "1",
"lmt": "1000000",
},
)
data = payload.get("data") if isinstance(payload, dict) else None
if not isinstance(data, dict):
return []
klines = data.get("klines")
if not isinstance(klines, list):
return []
rows: list[dict] = []
for raw in klines:
if not isinstance(raw, str):
continue
parsed = _parse_kline_row(raw)
if parsed is not None:
rows.append(parsed)
return rows
agent/backtest/loaders/eastmoney_loader.py
+176
View File
@@ -0,0 +1,176 @@
"""Eastmoney loader: free, no-auth OHLCV across A-share, HK and US equities.
Eastmoney's ``push2his`` quote endpoints are free and require no token, but the
service rate-limits aggressively by source IP. All HTTP goes through the shared
:mod:`backtest.loaders.eastmoney_client`, which routes every call through the
per-host throttle in :mod:`backtest.loaders._http`. This loader only maps our
symbol/interval/DataFrame conventions onto that client; it owns no HTTP itself.
Symbol routing is delegated to :func:`eastmoney_client.resolve_secid`:
* A-share — ``600519.SH`` / ``000001.SZ`` / ``430139.BJ``
* Hong Kong — ``00700.HK`` (numeric code zero-padded to five digits)
* US — ``AAPL.US`` (market prefix discovered via Eastmoney search, cached)
"""
from __future__ import annotations
import logging
from typing import Dict, List, Optional
import pandas as pd
from backtest.loaders import eastmoney_client
from backtest.loaders.base import cached_loader_fetch, validate_date_range
from backtest.loaders.registry import register
logger = logging.getLogger(__name__)
# OHLCV columns the engine consumes, in canonical order.
_OHLCV_COLUMNS = ["open", "high", "low", "close", "volume"]
def _to_compact_date(value: str) -> str:
"""Convert a ``YYYY-MM-DD`` date into Eastmoney's ``YYYYMMDD`` form.
Args:
value: Date string in any pandas-parseable form.
Returns:
The date rendered as ``YYYYMMDD``.
Raises:
ValueError: ``value`` is not a parseable date.
"""
try:
return pd.Timestamp(value).strftime("%Y%m%d")
except Exception as exc: # noqa: BLE001 - surfaced as a clear ValueError
raise ValueError(f"Invalid date for eastmoney: {value!r}") from exc
@register
class DataLoader:
"""Eastmoney OHLCV loader (free, throttled HTTP, no auth)."""
name = "eastmoney"
markets = {"a_share", "hk_equity", "us_equity"}
requires_auth = False
def is_available(self) -> bool:
"""Always available — uses unauthenticated throttled HTTP."""
return True
def fetch(
self,
codes: List[str],
start_date: str,
end_date: str,
*,
interval: str = "1D",
fields: Optional[List[str]] = None,
) -> Dict[str, pd.DataFrame]:
"""Fetch OHLCV for each symbol; a single failure never aborts the batch.
Args:
codes: Symbols such as ``"600519.SH"``, ``"00700.HK"``, ``"AAPL.US"``.
start_date: Inclusive start date (``YYYY-MM-DD``).
end_date: Inclusive end date (``YYYY-MM-DD``).
interval: Bar interval label (e.g. ``"1D"``, ``"1H"``, ``"5m"``).
fields: Accepted for protocol parity; the OHLCV columns are fixed.
Returns:
Mapping ``{symbol: DataFrame}`` for every symbol that yielded bars.
Each DataFrame has a ``trade_date`` ``DatetimeIndex`` and float
columns ``open/high/low/close/volume``. Symbols that resolve to no
data are omitted.
Raises:
ValueError: ``start_date``/``end_date`` are malformed or inverted.
"""
validate_date_range(start_date, end_date)
result: Dict[str, pd.DataFrame] = {}
for code in codes:
try:
df = cached_loader_fetch(
source=self.name,
symbol=code,
timeframe=interval,
start_date=start_date,
end_date=end_date,
fields=None,
fetch=lambda code=code: self._fetch_one(
code, start_date, end_date, interval
),
)
if df is not None and not df.empty:
result[code] = df
except Exception as exc: # noqa: BLE001 - one bad symbol must not abort
logger.warning("eastmoney failed for %s: %s", code, exc)
return result
def _fetch_one(
self, code: str, start_date: str, end_date: str, interval: str,
) -> Optional[pd.DataFrame]:
"""Resolve one symbol and build its OHLCV frame, or ``None`` on a miss.
Args:
code: A single Vibe-Trading symbol.
start_date: Inclusive start date (``YYYY-MM-DD``).
end_date: Inclusive end date (``YYYY-MM-DD``).
interval: Bar interval label.
Returns:
An OHLCV DataFrame, or ``None`` when the symbol/interval is
unsupported or Eastmoney returns no bars.
"""
klt = eastmoney_client.KLT_BY_INTERVAL.get(interval)
if klt is None:
logger.warning("eastmoney unsupported interval %r for %s", interval, code)
return None
secid = eastmoney_client.resolve_secid(code)
if not secid:
return None
rows = eastmoney_client.fetch_kline(
secid,
klt=klt,
fqt=1,
beg=_to_compact_date(start_date),
end=_to_compact_date(end_date),
)
return self._frame_from_rows(rows)
@staticmethod
def _frame_from_rows(rows: List[dict]) -> Optional[pd.DataFrame]:
"""Assemble the canonical OHLCV DataFrame from client kline rows.
Args:
rows: Ascending ``{trade_date, open, high, low, close, volume, ...}``
dicts from :func:`eastmoney_client.fetch_kline`.
Returns:
A DataFrame indexed by ``trade_date`` with float OHLCV columns, or
``None`` when ``rows`` is empty or fully unparseable.
"""
if not rows:
return None
df = pd.DataFrame(rows)
if df.empty or "trade_date" not in df.columns:
return None
df["trade_date"] = pd.to_datetime(df["trade_date"])
df = df.set_index("trade_date").sort_index()
df.index.name = "trade_date"
for column in _OHLCV_COLUMNS:
if column not in df.columns:
return None
df[column] = pd.to_numeric(df[column], errors="coerce").astype(float)
df = df[_OHLCV_COLUMNS].dropna(subset=["open", "high", "low", "close"])
if df.empty:
return None
return df
agent/backtest/loaders/finnhub_loader.py
+279
View File
@@ -0,0 +1,279 @@
"""Finnhub loader: key-gated US-equity OHLCV via the public stock-candle API.
Finnhub (https://finnhub.io) serves daily stock candles from a single REST
endpoint, gated by a free API key passed as the ``token`` query parameter:
https://finnhub.io/api/v1/stock/candle?symbol=AAPL&resolution=D&from=...&to=...&token=...
Like other free quote providers it rate-limits by source IP, so every request
routes through :func:`backtest.loaders._http.throttled_get_json` for per-host
spacing plus session reuse. The endpoint answers with parallel arrays
(``o``/``h``/``l``/``c``/``v`` plus epoch ``t``) and a status field ``s`` that
reads ``"ok"`` on a hit or ``"no_data"`` for an empty window.
Symbol convention (Vibe-Trading -> Finnhub):
* US ``AAPL.US`` -> ``AAPL`` (Finnhub carries US tickers bare)
* Anything else is passed through uppercased (e.g. a bare ``MSFT``).
No market data is ever persisted in the repo; settled ranges may be cached to
the user-home loader cache via :func:`cached_loader_fetch`.
"""
from __future__ import annotations
import logging
from typing import Any, Dict, List, Optional
import pandas as pd
from backtest.loaders._http import resolve_min_interval, throttled_get_json
from backtest.loaders.base import cached_loader_fetch, validate_date_range
from backtest.loaders.registry import register
logger = logging.getLogger(__name__)
_API_KEY_ENV = "FINNHUB_API_KEY"
_CANDLE_URL = "https://finnhub.io/api/v1/stock/candle"
# Throttle bucket + default minimum spacing (free tier is ~30 req/s but bursts
# get a 429; one shared bucket keeps the whole process polite).
_HOST_KEY = "finnhub"
_MIN_INTERVAL_ENV = "VIBE_TRADING_FINNHUB_MIN_INTERVAL"
_DEFAULT_MIN_INTERVAL_S = 0.4
# Only daily candles are wired here; Finnhub also exposes intraday resolutions
# but the project's us_equity fallback chain is daily-only.
_RESOLUTION = "D"
# Finnhub packs parallel arrays under these keys; ``t`` carries epoch seconds.
_FINNHUB_TO_OHLCV = {"o": "open", "h": "high", "l": "low", "c": "close", "v": "volume"}
_OHLCV_COLUMNS = ["open", "high", "low", "close", "volume"]
def _min_interval() -> float:
"""Resolve the per-call minimum spacing, honoring the env override."""
return resolve_min_interval(_MIN_INTERVAL_ENV, _DEFAULT_MIN_INTERVAL_S)
def _to_finnhub_symbol(code: str) -> str:
"""Translate a project symbol into Finnhub's ticker convention.
Args:
code: Project-side symbol, e.g. ``AAPL.US`` or a bare ``MSFT``.
Returns:
The Finnhub ticker: the ``.US`` suffix stripped, otherwise the
uppercased symbol unchanged.
"""
upper = code.strip().upper()
if upper.endswith(".US"):
return upper[: -len(".US")]
return upper
def _to_epoch_seconds(date_str: str, *, end_of_day: bool) -> int:
"""Convert a ``YYYY-MM-DD`` date to a UTC epoch-second bound.
Args:
date_str: Inclusive date string.
end_of_day: When ``True`` push the instant to 23:59:59 so the day's
bar falls inside Finnhub's inclusive ``[from, to]`` window.
Returns:
Epoch seconds (UTC) for the requested bound.
"""
ts = pd.Timestamp(date_str).normalize()
if end_of_day:
ts = ts + pd.Timedelta(hours=23, minutes=59, seconds=59)
return int(ts.value // 1_000_000_000)
def _rows_from_payload(payload: Any) -> List[Dict[str, Any]]:
"""Zip Finnhub's parallel candle arrays into ascending row dicts.
Args:
payload: Decoded JSON body from the stock-candle endpoint.
Returns:
Ascending list of ``{trade_date, open, high, low, close, volume}`` dicts
where ``trade_date`` is the raw epoch-second ``int`` (normalized to a
``datetime64[ns]`` index later in :meth:`DataLoader._fetch_one`).
Empty when the status is not ``"ok"`` or the timestamp array is missing,
and per-bar gaps (a ``None`` in any OHLC slot) are skipped.
"""
if not isinstance(payload, dict) or payload.get("s") != "ok":
return []
timestamps = payload.get("t") or []
if not isinstance(timestamps, list):
return []
rows: List[Dict[str, Any]] = []
for index, epoch in enumerate(timestamps):
values = {
field: _at(payload.get(key), index)
for key, field in _FINNHUB_TO_OHLCV.items()
}
if any(values[field] is None for field in ("open", "high", "low", "close")):
continue
# Keep the raw epoch here; the resolution is normalized in one place
# (``_fetch_one``) so the index dtype matches the other loaders.
row: Dict[str, Any] = {"trade_date": int(epoch)}
row.update({field: _to_float(values[field]) for field in _OHLCV_COLUMNS})
rows.append(row)
return rows
def _at(series: Any, index: int) -> Any:
"""Return ``series[index]`` when in range, else ``None``."""
if isinstance(series, list) and 0 <= index < len(series):
return series[index]
return None
def _to_float(value: Any) -> Optional[float]:
"""Coerce a Finnhub numeric (or ``None``) to ``float``; ``None`` on failure."""
if value is None:
return None
try:
return float(value)
except (TypeError, ValueError):
return None
@register
class DataLoader:
"""Finnhub US-equity daily OHLCV loader (key-gated, throttled HTTP)."""
name = "finnhub"
markets = {"us_equity"}
requires_auth = True
def __init__(self) -> None:
"""Initialize the loader without touching the network or credentials.
Construction never raises on a missing key; availability is reported
separately via :meth:`is_available` so the fallback chain can keep
walking when the key is absent.
"""
pass
def is_available(self) -> bool:
"""Return whether a Finnhub API key is present in the environment."""
from src.config.accessor import get_env_config
return bool(get_env_config().data.finnhub_api_key)
def fetch(
self,
codes: List[str],
start_date: str,
end_date: str,
*,
interval: str = "1D",
fields: Optional[List[str]] = None,
) -> Dict[str, pd.DataFrame]:
"""Fetch daily OHLCV history keyed by the original project symbols.
Each symbol is fetched independently through the opt-in loader cache;
a single failing symbol is logged and skipped so it never aborts the
batch.
Args:
codes: Project symbols such as ``AAPL.US``.
start_date: Inclusive start date in ``YYYY-MM-DD`` format.
end_date: Inclusive end date in ``YYYY-MM-DD`` format.
interval: Backtest interval; only daily (``1D``) is supported.
fields: Ignored; included for interface compatibility.
Returns:
Mapping of input symbol to a DataFrame indexed by a ``trade_date``
:class:`~pandas.DatetimeIndex` with float ``open``/``high``/``low``/
``close``/``volume`` columns. Symbols without data are omitted.
Raises:
ValueError: If ``start_date`` > ``end_date``.
"""
del fields
validate_date_range(start_date, end_date)
# Daily-only candle resolution; do not silently return day bars for ``1H``.
if str(interval).strip().lower() not in {"1d", "d", "day", "daily"}:
logger.warning(
"finnhub supports daily bars only; rejecting interval=%r",
interval,
)
return {}
from src.config.accessor import get_env_config
api_key = get_env_config().data.finnhub_api_key
if not api_key:
logger.warning("finnhub fetch skipped: %s not set", _API_KEY_ENV)
return {}
result: Dict[str, pd.DataFrame] = {}
for code in codes:
try:
df = cached_loader_fetch(
source=self.name,
symbol=code,
timeframe=interval,
start_date=start_date,
end_date=end_date,
fields=None,
fetch=lambda code=code: self._fetch_one(
code, start_date, end_date, api_key
),
)
if df is not None and not df.empty:
result[code] = df
except Exception as exc: # noqa: BLE001 - one symbol must not abort the batch
logger.warning("finnhub failed for %s: %s", code, exc)
return result
def _fetch_one(
self, code: str, start_date: str, end_date: str, api_key: str
) -> Optional[pd.DataFrame]:
"""Fetch and normalize one symbol's daily candles.
Args:
code: Project-side symbol.
start_date: Inclusive start date (``YYYY-MM-DD``).
end_date: Inclusive end date (``YYYY-MM-DD``).
api_key: Finnhub API token for the ``token`` query parameter.
Returns:
A normalized OHLCV DataFrame, or ``None`` when Finnhub reports no
usable data for the window.
"""
payload = throttled_get_json(
_CANDLE_URL,
host_key=_HOST_KEY,
min_interval=_min_interval(),
params={
"symbol": _to_finnhub_symbol(code),
"resolution": _RESOLUTION,
"from": _to_epoch_seconds(start_date, end_of_day=False),
"to": _to_epoch_seconds(end_date, end_of_day=True),
"token": api_key,
},
)
rows = _rows_from_payload(payload)
if not rows:
return None
df = pd.DataFrame(rows)
# Epoch seconds -> nanosecond-resolution DatetimeIndex so the index
# dtype (``datetime64[ns]``) stays consistent with the other loaders
# rather than collapsing to the second resolution pandas infers from a
# ``unit="s"`` conversion.
df["trade_date"] = pd.to_datetime(df["trade_date"], unit="s").astype(
"datetime64[ns]"
)
df = df.set_index("trade_date").sort_index()
df = df[_OHLCV_COLUMNS].astype(float).dropna(
subset=["open", "high", "low", "close"]
)
return df
agent/backtest/loaders/fmp_loader.py
+221
View File
@@ -0,0 +1,221 @@
"""Financial Modeling Prep (FMP) loader: key-gated US-equity OHLCV via HTTP.
FMP exposes a daily historical-price endpoint that, like other free quote
providers, rate-limits by source IP and must be throttled. Every request here
routes through :mod:`backtest.loaders._http` so calls share one process-wide
minimum-spacing gate and a reused session.
API format (public, documented):
https://financialmodelingprep.com/api/v3/historical-price-full/{SYMBOL}
?from=YYYY-MM-DD&to=YYYY-MM-DD&apikey=KEY
The JSON body is ``{"symbol": "AAPL", "historical": [{date, open, high, low,
close, volume}, ...]}`` in descending date order; an unknown symbol or empty
window yields an empty/absent ``historical`` array.
Auth: set ``FMP_API_KEY`` in the environment. Covers US equities only.
"""
from __future__ import annotations
import logging
from typing import Any, Dict, List, Optional
import pandas as pd
from backtest.loaders._http import resolve_min_interval, throttled_get_json
from backtest.loaders.base import cached_loader_fetch, validate_date_range
from backtest.loaders.registry import register
logger = logging.getLogger(__name__)
_API_KEY_ENV = "FMP_API_KEY"
_BASE_URL = "https://financialmodelingprep.com/api/v3/historical-price-full"
# Shared throttle/session bucket for every FMP request in this process.
_HOST_KEY = "fmp"
_MIN_INTERVAL_ENV = "VIBE_TRADING_FMP_MIN_INTERVAL"
_DEFAULT_MIN_INTERVAL_S = 0.3
# FMP daily bars carry these numeric fields; emitted in this column order.
_OHLCV_FIELDS = ("open", "high", "low", "close", "volume")
def _api_key() -> str:
"""Return the FMP API key from the environment, stripped (``""`` if unset)."""
from src.config.accessor import get_env_config
return get_env_config().data.fmp_api_key.strip()
def _min_interval() -> float:
"""Resolve the per-call minimum spacing, honoring the env override."""
return resolve_min_interval(_MIN_INTERVAL_ENV, _DEFAULT_MIN_INTERVAL_S)
def _fmp_symbol(code: str) -> str:
"""Translate a project symbol into FMP's bare-ticker convention.
FMP carries US tickers bare, so a trailing ``.US`` suffix (the project's
US-equity marker) is dropped; everything else is upper-cased and passed
through unchanged.
Args:
code: Project-side symbol, e.g. ``AAPL`` or ``AAPL.US``.
Returns:
The FMP ticker (suffix stripped, upper-cased).
"""
cleaned = code.strip().upper()
if cleaned.endswith(".US"):
cleaned = cleaned[: -len(".US")]
return cleaned
@register
class DataLoader:
"""Financial Modeling Prep US-equity OHLCV loader (key-gated, HTTP)."""
name = "fmp"
markets = {"us_equity"}
requires_auth = True
def __init__(self) -> None:
pass
def is_available(self) -> bool:
"""Available when ``FMP_API_KEY`` is set to a non-empty value."""
return bool(_api_key())
def fetch(
self,
codes: List[str],
start_date: str,
end_date: str,
*,
interval: str = "1D",
fields: Optional[List[str]] = None,
) -> Dict[str, pd.DataFrame]:
"""Fetch daily OHLCV bars from FMP, one symbol at a time.
A single failing symbol is logged and skipped so it never aborts the
rest of the batch.
Args:
codes: Project symbols (e.g. ``["AAPL", "MSFT.US"]``).
start_date: Inclusive start date, ``YYYY-MM-DD``.
end_date: Inclusive end date, ``YYYY-MM-DD``.
interval: Bar size; only ``"1D"`` is supported (others skipped).
fields: Ignored — FMP returns a fixed OHLCV schema.
Returns:
Mapping ``{symbol: DataFrame(trade_date, open, high, low, close,
volume)}`` for every symbol that returned non-empty data.
Raises:
ValueError: If ``start_date`` > ``end_date`` (via
:func:`validate_date_range`).
"""
validate_date_range(start_date, end_date)
if str(interval).strip().lower() not in {"1d", "d", "day", "daily"}:
logger.warning("fmp only supports 1D bars; got interval=%r", interval)
return {}
if not self.is_available():
logger.warning("fmp fetch skipped: %s not set", _API_KEY_ENV)
return {}
result: Dict[str, pd.DataFrame] = {}
for code in codes:
try:
df = cached_loader_fetch(
source=self.name,
symbol=code,
timeframe=interval,
start_date=start_date,
end_date=end_date,
fields=None,
fetch=lambda code=code: self._fetch_one(code, start_date, end_date),
)
if df is not None and not df.empty:
result[code] = df
except Exception as exc:
logger.warning("fmp failed for %s: %s", code, exc)
return result
def _fetch_one(
self, code: str, start_date: str, end_date: str,
) -> Optional[pd.DataFrame]:
"""Fetch and parse one symbol's daily bars; ``None`` on no data.
Args:
code: Project symbol to fetch.
start_date: Inclusive start date, ``YYYY-MM-DD``.
end_date: Inclusive end date, ``YYYY-MM-DD``.
Returns:
An ascending OHLCV DataFrame indexed by ``trade_date``, or ``None``
when FMP reports no bars for the symbol/window.
Raises:
RuntimeError: If the API key is missing at call time.
requests.RequestException: Propagated from the HTTP layer.
"""
api_key = _api_key()
if not api_key:
raise RuntimeError(f"{_API_KEY_ENV} is not set")
symbol = _fmp_symbol(code)
if not symbol:
return None
payload = throttled_get_json(
f"{_BASE_URL}/{symbol}",
host_key=_HOST_KEY,
min_interval=_min_interval(),
params={"from": start_date, "to": end_date, "apikey": api_key},
)
return _parse_historical(payload)
def _parse_historical(payload: Any) -> Optional[pd.DataFrame]:
"""Convert an FMP historical-price body into an ascending OHLCV frame.
Args:
payload: Decoded JSON body from the historical-price endpoint.
Returns:
DataFrame indexed by ``trade_date`` with float ``open/high/low/close/
volume`` columns, or ``None`` when no usable rows are present.
"""
historical = (payload or {}).get("historical") if isinstance(payload, dict) else None
if not historical:
return None
rows = []
for bar in historical:
if not isinstance(bar, dict) or "date" not in bar:
continue
rows.append(
{
"trade_date": bar["date"],
**{field: bar.get(field) for field in _OHLCV_FIELDS},
}
)
if not rows:
return None
df = pd.DataFrame(rows)
df["trade_date"] = pd.to_datetime(df["trade_date"])
for field in _OHLCV_FIELDS:
# Cast to float (not just to_numeric) so integer volume from the API
# does not leave the column int64 and break the float-OHLCV contract.
df[field] = pd.to_numeric(df[field], errors="coerce").astype(float)
df = df.set_index("trade_date").sort_index()
df = df[list(_OHLCV_FIELDS)].dropna(subset=["open", "high", "low", "close"])
if df.empty:
return None
return df
agent/backtest/loaders/fundamentals_loader.py
+500
View File
@@ -0,0 +1,500 @@
"""Translate sparse SEC facts into PIT-safe daily fundamental panels.
The loader's core invariant is point-in-time safety: values become visible on
their SEC ``filed`` date, never on ``period_end``. Phase 1 implements the SEC
branch only. ``freq="ttm"`` uses a pragmatic Phase-1 approximation: known
income-statement and cash-flow concepts are rolling four-quarter sums keyed by
``period_end`` and anchored by the latest ``filed`` date in the rolling window;
balance-sheet and share-count concepts use the latest quarterly value. SEC
companyfacts can mix true-quarter, year-to-date, and annual facts, so upstream
schema normalization may still be needed for issuer-specific precision.
"""
from __future__ import annotations
import importlib
import logging
from collections.abc import Iterable
from typing import Any
import pandas as pd
from backtest.loaders import sec_edgar_client
from backtest.loaders.base import cached_loader_fetch, validate_date_range
from backtest.loaders.sec_frames import ANNUAL_SPAN_DAYS, QUARTER_SPAN_DAYS
logger = logging.getLogger(__name__)
_VALID_FREQS = {"annual", "quarterly", "ttm"}
_ANNUAL_FORMS = {"10-K"}
_QUARTERLY_FORMS = {"10-Q", "10-K"}
_FLOW_CONCEPTS = {
"RevenueFromContractWithCustomerExcludingAssessedTax",
"RevenueFromContractWithCustomerIncludingAssessedTax",
"Revenues",
"SalesRevenueNet",
"CostOfGoodsAndServicesSold",
"CostOfRevenue",
"OperatingIncomeLoss",
"NetIncomeLoss",
"ProfitLoss",
"NetCashProvidedByUsedInOperatingActivities",
"PaymentsToAcquirePropertyPlantAndEquipment",
}
def _extract_concept_series(
facts: dict[str, Any],
concepts: list[str],
freq: str,
*,
pit: bool = True,
) -> pd.DataFrame:
"""Extract a sparse ``period_end``/``filed``/``value`` series.
Args:
facts: SEC companyfacts payload.
concepts: Ordered us-gaap concept aliases to union.
freq: ``"annual"``, ``"quarterly"``, or ``"ttm"``.
pit: When true, duplicate ``period_end`` rows keep the earliest filing;
when false, they keep the latest filing for research mode.
Returns:
DataFrame with ``period_end``, ``filed``, and ``value`` columns.
Raises:
ValueError: If ``freq`` is unsupported.
"""
if freq not in _VALID_FREQS:
raise ValueError(f"unsupported fundamental freq: {freq}")
rows: list[dict[str, Any]] = []
us_gaap = facts.get("facts", {}).get("us-gaap", {})
for concept_order, concept in enumerate(concepts):
concept_node = us_gaap.get(concept, {})
units = concept_node.get("units", {}) if isinstance(concept_node, dict) else {}
if not isinstance(units, dict):
continue
for unit_items in units.values():
if not isinstance(unit_items, list):
continue
for item in unit_items:
if not isinstance(item, dict):
continue
if not _form_allowed(str(item.get("form", "")), freq):
continue
rows.append(
{
"period_start": item.get("start"),
"period_end": item.get("end"),
"filed": item.get("filed"),
"value": item.get("val"),
"_concept_order": concept_order,
}
)
columns = ["period_end", "filed", "value"]
if not rows:
return pd.DataFrame(columns=columns)
df = pd.DataFrame(rows)
df["period_start"] = pd.to_datetime(df["period_start"], errors="coerce").dt.normalize()
df["period_end"] = pd.to_datetime(df["period_end"], errors="coerce").dt.normalize()
df["filed"] = pd.to_datetime(df["filed"], errors="coerce").dt.normalize()
df["value"] = pd.to_numeric(df["value"], errors="coerce")
df = df.dropna(subset=["period_end", "filed", "value"])
if df.empty:
return pd.DataFrame(columns=columns)
keep = "first" if pit else "last"
if _is_flow_concept_series(concepts):
# Duration entries mix true quarters, year-to-date frames sharing the
# same end date, and full-year frames; summing them blindly double
# counts. Select by (start, end) span before any dedupe or rolling.
df = df.dropna(subset=["period_start"])
df = df.sort_values(["period_start", "period_end", "filed", "_concept_order"])
df = df.drop_duplicates(subset=["period_start", "period_end"], keep=keep)
duration = (df["period_end"] - df["period_start"]).dt.days
if freq == "annual":
df = df[duration.between(*ANNUAL_SPAN_DAYS)]
else:
df = _quarterly_flow_frames(df, duration)
else:
df = df.sort_values(["period_end", "filed", "_concept_order"])
df = df.drop_duplicates(subset=["period_end"], keep=keep).sort_values("period_end")
if freq == "ttm" and _is_flow_concept_series(concepts):
df = _to_ttm_flow_series(df)
return df.loc[:, columns].sort_values("period_end").reset_index(drop=True)
def _quarterly_flow_frames(df: pd.DataFrame, duration: pd.Series) -> pd.DataFrame:
"""Return true-quarter flow frames, deriving fiscal Q4 from 10-K full years.
Filers like AAPL report Q4 only inside the 10-K full-year frame, so a
missing quarter ending at the fiscal year end is synthesized as
``FY - (Q1 + Q2 + Q3)``. The synthesized row is anchored on the 10-K
``filed`` date, which keeps it PIT-safe.
"""
quarters = df[duration.between(*QUARTER_SPAN_DAYS)].copy()
annuals = df[duration.between(*ANNUAL_SPAN_DAYS)]
quarter_ends = set(quarters["period_end"])
synthesized: list[dict[str, Any]] = []
for annual in annuals.to_dict("records"):
if annual["period_end"] in quarter_ends:
continue
inside = quarters[
(quarters["period_start"] >= annual["period_start"])
& (quarters["period_end"] < annual["period_end"])
]
if len(inside) != 3:
continue
synthesized.append(
{
"period_start": inside["period_end"].max(),
"period_end": annual["period_end"],
"filed": max(annual["filed"], inside["filed"].max()),
"value": annual["value"] - inside["value"].sum(),
"_concept_order": annual["_concept_order"],
}
)
if synthesized:
quarters = pd.concat([quarters, pd.DataFrame(synthesized)], ignore_index=True)
return quarters
def _ffill_pit(series: pd.DataFrame, index: pd.DatetimeIndex) -> pd.Series:
"""Forward-fill sparse values onto ``index`` using ``filed`` as the anchor.
Args:
series: DataFrame containing ``filed`` and ``value`` columns.
index: Target dates.
Returns:
Dense Series indexed like ``index``.
"""
target_index = pd.DatetimeIndex(index)
if series.empty:
return pd.Series(index=target_index, dtype="float64")
sparse = series.copy()
sparse["filed"] = pd.to_datetime(sparse["filed"], errors="coerce").dt.normalize()
sparse["value"] = pd.to_numeric(sparse["value"], errors="coerce")
sparse = sparse.dropna(subset=["filed", "value"]).sort_values("filed")
if sparse.empty:
return pd.Series(index=target_index, dtype="float64")
values = sparse.set_index("filed")["value"].sort_index()
values = values[~values.index.duplicated(keep="last")]
aligned_index = values.index.union(target_index)
return values.reindex(aligned_index).ffill().reindex(target_index)
def load_fundamental_panel(
symbols: list[str],
fields: list[str],
start: str,
end: str,
freq: str = "ttm",
pit: bool = True,
source: str = "auto",
index: pd.DatetimeIndex | None = None,
) -> dict[str, pd.DataFrame]:
"""Load SEC fundamental fields as dense date-by-symbol panels.
Args:
symbols: Tickers to load.
fields: Unified schema field names, raw or derived.
start: Inclusive start date.
end: Inclusive end date.
freq: ``"annual"``, ``"quarterly"``, or ``"ttm"``.
pit: True keeps first filed facts per period; false keeps latest filed.
source: ``"auto"`` or ``"sec"``. Phase 1 routes both to SEC.
index: Optional target date index. When omitted, calendar days from
``start`` through ``end`` are used.
Returns:
Mapping ``field -> DataFrame(index=dates, columns=symbols)``.
Raises:
ValueError: If source or frequency is unsupported.
RuntimeError: If the fundamental schema module is unavailable.
"""
validate_date_range(start, end)
if freq not in _VALID_FREQS:
raise ValueError(f"unsupported fundamental freq: {freq}")
if source not in {"auto", "sec"}:
raise ValueError("Phase 1 fundamentals loader supports source='auto' or 'sec' only")
target_index = _target_index(start, end, index)
schema = _load_schema()
requested_fields = [_resolve_field_name(schema, field) for field in fields]
raw_fields = sorted(_collect_raw_fields(schema, requested_fields))
symbol_list = list(symbols)
ciks = _resolve_ciks(symbol_list)
symbol_raw_frames = {
symbol: _load_symbol_raw_frame(
symbol=symbol,
cik=ciks.get(symbol),
raw_fields=raw_fields,
schema=schema,
start=start,
end=end,
freq=freq,
pit=pit,
index=target_index,
)
for symbol in symbol_list
}
panels: dict[str, pd.DataFrame] = {}
for field in raw_fields:
panels[field] = pd.DataFrame(
{symbol: symbol_raw_frames[symbol][field] for symbol in symbol_list},
index=target_index,
columns=symbol_list,
)
def panel_for(field: str) -> pd.DataFrame:
if field in panels:
return panels[field]
derived = _derived_field(schema, field)
if derived is None:
panels[field] = _empty_panel(target_index, symbol_list)
return panels[field]
deps = {
_resolve_field_name(schema, dep): panel_for(_resolve_field_name(schema, dep))
for dep in _derived_dependencies(derived)
}
computed = _compute_derived(derived, deps)
panels[field] = computed.reindex(index=target_index, columns=symbol_list)
return panels[field]
return {field: panel_for(field) for field in requested_fields}
def _form_allowed(form: str, freq: str) -> bool:
if freq == "annual":
return form in _ANNUAL_FORMS
if freq in {"quarterly", "ttm"}:
return form in _QUARTERLY_FORMS
return False
def _is_flow_concept_series(concepts: Iterable[str]) -> bool:
return any(concept in _FLOW_CONCEPTS for concept in concepts)
def _to_ttm_flow_series(df: pd.DataFrame) -> pd.DataFrame:
ttm = df.sort_values("period_end").copy()
ttm["value"] = ttm["value"].rolling(window=4, min_periods=4).sum()
filed_values = list(ttm["filed"])
ttm["filed"] = [
pd.NaT if pos < 3 else max(filed_values[pos - 3 : pos + 1])
for pos in range(len(filed_values))
]
return ttm.dropna(subset=["filed", "value"])
def _target_index(
start: str,
end: str,
index: pd.DatetimeIndex | None,
) -> pd.DatetimeIndex:
if index is not None:
return pd.DatetimeIndex(index)
return pd.date_range(pd.Timestamp(start), pd.Timestamp(end), freq="D")
def _load_schema() -> Any:
try:
return importlib.import_module("backtest.loaders._fundamental_schema")
except ModuleNotFoundError as exc:
raise RuntimeError(
"backtest.loaders._fundamental_schema is required for fundamentals loading"
) from exc
def _resolve_field_name(schema: Any, field: str) -> str:
normalized = str(field).removeprefix("fund:")
resolver = getattr(schema, "resolve_field", None)
if not callable(resolver):
return normalized
resolved = resolver(normalized)
if isinstance(resolved, str):
return resolved.removeprefix("fund:")
name = getattr(resolved, "name", None)
if isinstance(name, str):
return name.removeprefix("fund:")
return normalized
def _collect_raw_fields(schema: Any, fields: Iterable[str]) -> set[str]:
raw: set[str] = set()
visiting: set[str] = set()
def visit(field: str) -> None:
if field in visiting:
raise ValueError(f"cyclic derived fundamental field: {field}")
derived = _derived_field(schema, field)
if derived is None:
raw.add(field)
return
visiting.add(field)
for dep in _derived_dependencies(derived):
visit(_resolve_field_name(schema, dep))
visiting.remove(field)
for field in fields:
visit(field)
return raw
def _derived_field(schema: Any, field: str) -> Any | None:
return getattr(schema, "DERIVED_FIELDS", {}).get(field)
def _spec_get(derived: Any, key: str) -> Any:
"""Read a derived-field spec entry from either a mapping or an object."""
if isinstance(derived, dict):
return derived.get(key)
return getattr(derived, key, None)
def _derived_dependencies(derived: Any) -> list[str]:
deps = _spec_get(derived, "dependencies") or ()
return [str(dep) for dep in deps]
def _compute_derived(derived: Any, dependencies: dict[str, pd.DataFrame]) -> pd.DataFrame:
compute = _spec_get(derived, "compute")
if not callable(compute):
raise ValueError(f"derived field has no callable compute: {derived!r}")
try:
result = compute(dependencies)
except TypeError:
result = compute(**dependencies)
if not isinstance(result, pd.DataFrame):
result = pd.DataFrame(result)
return result
def _resolve_ciks(symbols: list[str]) -> dict[str, str | None]:
"""Map each symbol to its SEC CIK.
Args:
symbols: Requested symbols.
Returns:
A symbol -> CIK mapping; the value is ``None`` for a symbol the SEC
ticker table does not carry.
Raises:
ValueError: If no symbol resolves. Returning an all-null panel here
reads as "this issuer reports nothing" rather than "this loader
cannot serve this market", which is the more damaging of the two.
"""
ciks: dict[str, str | None] = {}
missing: list[str] = []
for symbol in symbols:
cik = sec_edgar_client.cik_for(symbol)
ciks[symbol] = cik
if cik is None:
missing.append(symbol)
if missing:
logger.warning("No SEC CIK for symbols: %s", ", ".join(missing))
if symbols and len(missing) == len(symbols):
raise ValueError(
f"no SEC CIK resolved for any of: {', '.join(missing)}. The "
"fundamentals loader is US-only (SEC XBRL); pass a US ticker with "
"or without a .US suffix, e.g. 'AAPL' or 'AAPL.US'."
)
return ciks
def _load_symbol_raw_frame(
*,
symbol: str,
cik: str | None,
raw_fields: list[str],
schema: Any,
start: str,
end: str,
freq: str,
pit: bool,
index: pd.DatetimeIndex,
) -> pd.DataFrame:
if cik is None:
return _empty_field_frame(index, raw_fields)
def fetch() -> pd.DataFrame:
return _fetch_symbol_raw_frame(
symbol=symbol,
cik=cik,
raw_fields=raw_fields,
schema=schema,
freq=freq,
pit=pit,
index=index,
)
cached = cached_loader_fetch(
source="sec_fundamentals",
symbol=symbol,
timeframe=f"{freq}:pit={int(pit)}",
start_date=start,
end_date=end,
fields=raw_fields,
fetch=fetch,
)
if cached is None:
return _empty_field_frame(index, raw_fields)
return cached.reindex(index=index, columns=raw_fields)
def _fetch_symbol_raw_frame(
*,
symbol: str,
cik: str,
raw_fields: list[str],
schema: Any,
freq: str,
pit: bool,
index: pd.DatetimeIndex,
) -> pd.DataFrame:
try:
facts = sec_edgar_client.get_company_facts(cik)
except Exception as exc: # noqa: BLE001 - one bad symbol should not abort the panel
logger.warning("SEC companyfacts failed for %s: %s", symbol, exc)
return _empty_field_frame(index, raw_fields)
frame = _empty_field_frame(index, raw_fields)
concept_map = getattr(schema, "SEC_CONCEPT_MAP", {})
for field in raw_fields:
concepts = list(concept_map.get(field, ()))
if not concepts:
logger.warning("SEC concept miss for %s field %s: no aliases configured", symbol, field)
continue
sparse = _extract_concept_series(facts, concepts, freq, pit=pit)
if sparse.empty:
logger.warning(
"SEC concept miss for %s field %s: aliases=%s",
symbol,
field,
",".join(concepts),
)
continue
frame[field] = _ffill_pit(sparse, index)
return frame
def _empty_field_frame(index: pd.DatetimeIndex, fields: list[str]) -> pd.DataFrame:
return pd.DataFrame(index=index, columns=fields, dtype="float64")
def _empty_panel(index: pd.DatetimeIndex, symbols: list[str]) -> pd.DataFrame:
return pd.DataFrame(index=index, columns=symbols, dtype="float64")
agent/backtest/loaders/futu.py
+111 -22
View File
@@ -3,12 +3,16 @@
from __future__ import annotations
import logging
import os
from typing import Dict, List, Optional
import pandas as pd
from backtest.loaders.base import NoAvailableSourceError, validate_date_range
from backtest.loaders.base import (
NoAvailableSourceError,
loader_cache_get,
loader_cache_put,
validate_date_range,
)
from backtest.loaders.registry import register
logger = logging.getLogger(__name__)
@@ -16,10 +20,18 @@ logger = logging.getLogger(__name__)
_OHLCV_COLUMNS = ["open", "high", "low", "close", "volume"]
_INTERVAL_MAP: dict[str, str] = {
"1m": "K_1M",
"5m": "K_5M",
"15m": "K_15M",
"30m": "K_30M",
"1D": "K_DAY",
"1d": "K_DAY",
"1H": "K_60M",
"1h": "K_60M",
"4H": "K_240M",
"4h": "K_240M",
"1W": "K_WEEK",
"1w": "K_WEEK",
"1M": "K_MON",
}
@@ -47,10 +59,24 @@ def _to_futu_ktype(interval: str):
"""Map project interval string to a futu KLType enum value.
Lazy-imports futu so the module can be imported without futu installed.
Unknown intervals fall back to K_DAY.
Unsupported intervals fail explicitly so requested bar fidelity is never
changed silently.
"""
from futu import KLType # noqa: PLC0415
return getattr(KLType, _INTERVAL_MAP.get(interval.strip(), "K_DAY"))
token = interval.strip()
attr = _INTERVAL_MAP.get(token)
if attr is None:
raise NoAvailableSourceError(
f"unsupported Futu interval: {interval!r}; "
f"supported intervals: {sorted(_INTERVAL_MAP)}"
)
try:
return getattr(KLType, attr)
except AttributeError as exc:
raise NoAvailableSourceError(
f"installed Futu SDK does not expose KLType.{attr}"
) from exc
def _normalize_frame(df: pd.DataFrame) -> pd.DataFrame:
@@ -73,7 +99,8 @@ def _normalize_frame(df: pd.DataFrame) -> pd.DataFrame:
result = result.apply(pd.to_numeric, errors="coerce")
result["volume"] = result["volume"].fillna(0.0)
result = result.dropna(subset=["open", "high", "low", "close"])
return result.sort_index()
result = result.sort_index()
return result[~result.index.duplicated(keep="last")]
@register
@@ -89,12 +116,18 @@ class FutuLoader:
requires_auth = True
def __init__(self) -> None:
self._host = os.environ.get("FUTU_HOST", "127.0.0.1")
self._port = int(os.environ.get("FUTU_PORT", "11111"))
from src.config.accessor import get_env_config
cfg = get_env_config().data
self._host = cfg.futu_host
self._port = cfg.futu_port
def is_available(self) -> bool:
"""Return True if env vars are set and FutuOpenD is reachable."""
if not os.environ.get("FUTU_HOST") or not os.environ.get("FUTU_PORT"):
"""Return True if FutuOpenD is reachable."""
from src.config.accessor import get_env_config
cfg = get_env_config().data
if not cfg.futu_host or not cfg.futu_port:
return False
try:
import futu # noqa: PLC0415
@@ -119,7 +152,7 @@ class FutuLoader:
codes: Project symbols such as ``700.HK`` or ``000001.SZ``.
start_date: Start date in ``YYYY-MM-DD`` format.
end_date: End date in ``YYYY-MM-DD`` format.
interval: Backtest interval — ``1D``, ``1H``, or ``4H``.
interval: Backtest interval supported by :data:`_INTERVAL_MAP`.
fields: Ignored; included for interface compatibility.
Returns:
@@ -132,9 +165,38 @@ class FutuLoader:
if not codes:
return {}
validate_date_range(start_date, end_date)
if interval.strip() not in _INTERVAL_MAP:
raise NoAvailableSourceError(
f"unsupported Futu interval: {interval!r}; "
f"supported intervals: {sorted(_INTERVAL_MAP)}"
)
results: Dict[str, pd.DataFrame] = {}
# Serve cached symbols first; only open a FutuOpenD connection when at
# least one symbol is uncached, so a fully-cached request needs no
# running gateway.
pending: List[str] = []
for code in codes:
cached = loader_cache_get(
source=self.name,
symbol=code,
timeframe=interval,
start_date=start_date,
end_date=end_date,
fields=None,
)
if cached is not None and not cached.empty:
results[code] = cached.copy()
else:
pending.append(code)
if not pending:
return results
try:
import futu # noqa: PLC0415
ktype = _to_futu_ktype(interval)
ctx = futu.OpenQuoteContext(host=self._host, port=self._port)
except Exception as exc:
@@ -142,21 +204,48 @@ class FutuLoader:
f"Cannot connect to FutuOpenD at {self._host}:{self._port}: {exc}"
) from exc
results: Dict[str, pd.DataFrame] = {}
try:
for code in codes:
for code in pending:
futu_code = _to_futu_symbol(code)
ret, data = ctx.request_history_kline(
futu_code,
start=start_date,
end=end_date,
ktype=ktype,
max_count=10_000,
)
if ret != futu.RET_OK:
logger.warning("Futu returned error for %s: %s", futu_code, data)
page_key = None
pages: List[pd.DataFrame] = []
failed = False
while True:
ret, data, next_page_key = ctx.request_history_kline(
futu_code,
start=start_date,
end=end_date,
ktype=ktype,
max_count=10_000,
page_req_key=page_key,
)
if ret != futu.RET_OK:
logger.warning(
"Futu returned error for %s: %s", futu_code, data
)
failed = True
break
if isinstance(data, pd.DataFrame) and not data.empty:
pages.append(data)
if not next_page_key:
break
page_key = next_page_key
if failed or not pages:
continue
results[code] = _normalize_frame(data)
normalized = _normalize_frame(pd.concat(pages, ignore_index=True))
if normalized.empty:
continue
loader_cache_put(
source=self.name,
symbol=code,
timeframe=interval,
start_date=start_date,
end_date=end_date,
fields=None,
frame=normalized,
)
results[code] = normalized
finally:
ctx.close()
agent/backtest/loaders/india_broker_loader.py
+194
View File
@@ -0,0 +1,194 @@
"""India broker data bridge: feed Shoonya / Dhan history into the backtest layer.
The Shoonya (Finvasia) and Dhan connectors already expose live-account market
data via ``get_historical_bars`` (read path). This loader adapts that envelope
into the standard OHLCV frame so a user's *broker* history can back the same
backtests as the public Yahoo feed — useful when matching a live account exactly
or pulling symbols Yahoo lacks.
It is an OPT-IN source (``requires_auth``): ``is_available`` is True only when a
broker SDK is importable AND a broker is configured, so it trails Yahoo/yfinance
in the ``india_equity`` fallback chain and never fires in CI / unconfigured runs.
Symbol convention: project ``RELIANCE.NS`` / ``500325.BO`` → broker ``RELIANCE``
on exchange ``NSE`` / ``BSE`` (the suffix selects the exchange; the base symbol
is passed bare).
Limitation: broker endpoints return a bounded window of *recent* bars (period +
limit), not an arbitrary historical start. This loader requests enough bars to
cover the window and clips to ``[start_date, end_date]``; very old ranges may
come back short. For deep history prefer Yahoo.
"""
from __future__ import annotations
import logging
from typing import Any, Dict, List, Optional
import pandas as pd
from backtest.loaders.base import validate_date_range
from backtest.loaders.registry import register
logger = logging.getLogger(__name__)
_OUTPUT_COLUMNS = ["open", "high", "low", "close", "volume"]
# Project interval -> broker ``period`` token (both connectors share this set).
# ``1h``/``1d`` alias the project-style tokens; ``1m`` vs ``1M`` stays case-sensitive.
_PERIOD_MAP = {
"1D": "1d",
"1d": "1d",
"1H": "1h",
"1h": "1h",
"5m": "5m",
"15m": "15m",
"30m": "30m",
"1m": "1m",
}
def _resolve_broker():
"""Return ``(broker_key, sdk_module)`` for the first available India broker.
Prefers Shoonya, then Dhan. Returns ``(None, None)`` when neither the SDK
nor a config is present. Import is deferred and defensive so a missing
``src.trading`` package or broker SDK simply means "unavailable", never a
crash in the loader registry.
"""
try:
from src.trading.connectors.shoonya import sdk as shoonya_sdk
if shoonya_sdk.shoonya_available():
return "shoonya", shoonya_sdk
except Exception as exc: # noqa: BLE001 — optional dependency / config
logger.debug("shoonya bridge unavailable: %s", exc)
try:
from src.trading.connectors.dhan import sdk as dhan_sdk
if dhan_sdk.dhan_available():
return "dhan", dhan_sdk
except Exception as exc: # noqa: BLE001
logger.debug("dhan bridge unavailable: %s", exc)
return None, None
def _exchange_for(code: str) -> str:
"""Map the project suffix to the broker exchange code (NSE / BSE)."""
return "BSE" if code.strip().upper().endswith(".BO") else "NSE"
def _base_symbol(code: str) -> str:
"""Strip the ``.NS`` / ``.BO`` suffix, leaving the broker's bare symbol."""
cleaned = code.strip()
upper = cleaned.upper()
if upper.endswith((".NS", ".BO")):
return cleaned[:-3]
return cleaned
def _bars_to_frame(bars: list[dict], start_date: str, end_date: str) -> Optional[pd.DataFrame]:
"""Convert the broker ``bars`` list into a clipped OHLCV frame, or ``None``."""
if not bars:
return None
frame = pd.DataFrame(bars)
if "time" not in frame.columns:
return None
# ``time`` is epoch seconds (Shoonya ssboe / Dhan candle[0]) or an ISO string.
ts = pd.to_numeric(frame["time"], errors="coerce")
if ts.notna().any():
index = pd.to_datetime(ts, unit="s", errors="coerce")
else:
index = pd.to_datetime(frame["time"], errors="coerce")
frame = frame.drop(columns=["time"])
frame.index = pd.DatetimeIndex(index).tz_localize(None)
frame.index.name = "trade_date"
for col in _OUTPUT_COLUMNS:
if col not in frame.columns:
frame[col] = 0.0 if col == "volume" else pd.NA
frame = frame[_OUTPUT_COLUMNS].apply(pd.to_numeric, errors="coerce")
frame = frame.dropna(subset=["open", "high", "low", "close"]).sort_index()
lower = pd.Timestamp(start_date).normalize()
upper = pd.Timestamp(end_date).normalize() + pd.Timedelta(days=1)
frame = frame[(frame.index >= lower) & (frame.index < upper)]
return frame.astype(float) if not frame.empty else None
@register
class DataLoader:
"""Shoonya / Dhan history adapter for the ``india_equity`` market."""
name = "india_broker"
markets = {"india_equity"}
requires_auth = True
def __init__(self) -> None:
pass
def is_available(self) -> bool:
"""True only when an India broker SDK is importable and configured."""
broker, _ = _resolve_broker()
return broker is not None
def fetch(
self,
codes: List[str],
start_date: str,
end_date: str,
*,
interval: str = "1D",
fields: Optional[List[str]] = None,
) -> Dict[str, pd.DataFrame]:
"""Fetch OHLCV history for ``codes`` from the configured India broker."""
del fields
if not codes:
return {}
validate_date_range(start_date, end_date)
broker, sdk = _resolve_broker()
if sdk is None:
return {}
period = _PERIOD_MAP.get(str(interval).strip())
if period is None:
# Do not silently substitute daily bars for an unsupported interval
# (e.g. runner ``4H`` used to fall through to ``1d``).
logger.warning(
"india broker only supports %s; rejecting interval=%r",
sorted(_PERIOD_MAP),
interval,
)
return {}
# Request enough bars to cover the window (business days + headroom).
span_days = max((pd.Timestamp(end_date) - pd.Timestamp(start_date)).days, 1)
limit = min(max(span_days, 30), 2000)
result: Dict[str, pd.DataFrame] = {}
for code in codes:
try:
envelope = sdk.get_historical_bars(
_base_symbol(code),
exchange=_exchange_for(code),
period=period,
limit=limit,
)
except TypeError:
# Dhan uses ``exchange_segment``; retry without the ``exchange`` kw.
try:
envelope = sdk.get_historical_bars(
_base_symbol(code), period=period, limit=limit
)
except Exception as exc: # noqa: BLE001 — one bad symbol never aborts
logger.warning("%s bridge failed for %s: %s", broker, code, exc)
continue
except Exception as exc: # noqa: BLE001
logger.warning("%s bridge failed for %s: %s", broker, code, exc)
continue
if not isinstance(envelope, dict) or str(envelope.get("status", "")).lower() != "ok":
continue
frame = _bars_to_frame(envelope.get("bars", []), start_date, end_date)
if frame is not None and not frame.empty:
result[code] = frame
return result
agent/backtest/loaders/local_loader.py
+353
View File
@@ -0,0 +1,353 @@
"""Local data loader: reads CSV, Parquet, and DuckDB files from user config.
Configuration lives at ``~/.vibe-trading/data-bridge/config.yaml``.
Each entry maps a symbol to a data file with optional column-name overrides,
date format, and (for DuckDB) an SQL query.
Example config::
sources:
- symbol: "AAPL.US"
type: csv
path: "~/data/aapl_2024.csv"
columns:
date: "Date"
open: "Open"
high: "High"
low: "Low"
close: "Close"
volume: "Volume"
date_format: "%Y-%m-%d"
- symbol: "BTC-USDT"
type: parquet
path: "~/data/btc.parquet"
- symbol: "MYINDEX"
type: duckdb
db_path: "~/data/market.duckdb"
query: "SELECT * FROM prices WHERE ticker = 'MYINDEX'"
"""
from __future__ import annotations
import logging
from pathlib import Path
from typing import Any, Dict, List, Optional
import pandas as pd
import yaml
from backtest.loaders.base import cached_loader_fetch, validate_date_range, validate_ohlc
from backtest.loaders.registry import register
logger = logging.getLogger(__name__)
_CONFIG_DIR = Path.home() / ".vibe-trading" / "data-bridge"
_CONFIG_PATH = _CONFIG_DIR / "config.yaml"
_DEFAULT_COLUMNS = {
"date": "date",
"open": "open",
"high": "high",
"low": "low",
"close": "close",
"volume": "volume",
}
# Map the project's bar intervals (see ``backtest.runner._VALID_INTERVALS``) to
# pandas offset aliases used for resampling. All entries are fixed-duration so
# ``pd.Timedelta`` of the alias is well defined.
_RESAMPLE_RULES = {
"1m": "1min",
"5m": "5min",
"15m": "15min",
"30m": "30min",
"1H": "1h",
"1h": "1h",
"4H": "4h",
"4h": "4h",
"1D": "1D",
"1d": "1D",
}
_OHLCV_AGG = {
"open": "first",
"high": "max",
"low": "min",
"close": "last",
"volume": "sum",
}
def _resample_to_interval(df: pd.DataFrame, interval: str, symbol: str) -> pd.DataFrame:
"""Resample an OHLCV frame to the requested bar ``interval``.
Local files can hold any native granularity, so a requested interval the
file does not already match must be honored explicitly. Previously
``interval`` was passed only to the cache key, so requesting e.g. ``4H``
against hourly data silently returned the file's native bars.
Coarser-than-source requests are downsampled with the standard OHLCV
aggregation. A finer-than-source request cannot be fabricated from a file,
so the source bars are returned unchanged with a warning. Requests that
already match the source granularity are returned unchanged.
"""
rule = _RESAMPLE_RULES.get(interval)
if df.empty:
return df
if rule is None:
logger.warning(
"local loader: unsupported interval %r for %s; returning source bars",
interval,
symbol,
)
return df
target = pd.Timedelta(rule)
if len(df.index) >= 2:
source = df.index.to_series().diff().dropna().median()
if pd.notna(source):
if target < source:
logger.warning(
"local loader: cannot upsample %s source bars to %s for %s; "
"returning source bars",
source,
interval,
symbol,
)
return df
if target == source:
return df
resampled = df.resample(rule).agg(_OHLCV_AGG)
resampled = resampled.dropna(subset=["open", "high", "low", "close"])
resampled.index.name = df.index.name
return resampled
def _load_config() -> dict[str, Any] | None:
if not _CONFIG_PATH.exists():
return None
with open(_CONFIG_PATH, "r", encoding="utf-8") as f:
return yaml.safe_load(f) or {}
def _normalize_columns(
df: pd.DataFrame,
col_map: dict[str, str],
date_fmt: str | None,
) -> pd.DataFrame | None:
rename: dict[str, str] = {}
for std_name, src_name in col_map.items():
if std_name == "date":
continue
if src_name in df.columns:
rename[src_name] = std_name
df = df.rename(columns=rename)
required = {"open", "high", "low", "close"}
if not required.issubset(df.columns):
return None
date_col = col_map.get("date", "date")
if date_col not in df.columns:
return None
if date_fmt:
parsed_dates = pd.to_datetime(
df[date_col], format=date_fmt, errors="coerce", utc=True
)
else:
parsed_dates = pd.to_datetime(df[date_col], errors="coerce", utc=True)
# Normalize every input to the loader's UTC-naive index contract. Parsing
# as UTC first also handles files that span daylight-saving offset changes.
df["trade_date"] = parsed_dates.dt.tz_convert(None)
df = df.dropna(subset=["trade_date"])
df = df.set_index("trade_date").sort_index()
ohlcv_cols: list[str] = ["open", "high", "low", "close"]
if "volume" in df.columns:
ohlcv_cols.append("volume")
df = df[ohlcv_cols]
for col in ohlcv_cols:
df[col] = pd.to_numeric(df[col], errors="coerce")
df = df.dropna(subset=["open", "high", "low", "close"])
df = validate_ohlc(df)
for col in ohlcv_cols:
df[col] = df[col].astype("float64")
if "volume" not in df.columns:
df["volume"] = 0.0
return df
def _read_csv(path: str, col_map: dict[str, str], date_fmt: str | None) -> pd.DataFrame | None:
df = pd.read_csv(path)
return _normalize_columns(df, col_map, date_fmt)
def _read_parquet(path: str, col_map: dict[str, str], date_fmt: str | None) -> pd.DataFrame | None:
df = pd.read_parquet(path)
if isinstance(df.index, pd.DatetimeIndex):
df = df.reset_index()
if date_fmt is None and col_map.get("date", "date") not in df.columns:
col_map = dict(col_map)
col_map["date"] = df.columns[0]
return _normalize_columns(df, col_map, date_fmt)
def _read_duckdb(
db_path: str, query: str, col_map: dict[str, str], date_fmt: str | None
) -> pd.DataFrame | None:
import duckdb
with duckdb.connect(db_path, read_only=True) as conn:
df = conn.execute(query).df()
return _normalize_columns(df, col_map, date_fmt)
_READERS = {
"csv": _read_csv,
"parquet": _read_parquet,
"duckdb": _read_duckdb,
}
@register
class DataLoader:
"""Config-driven local data loader for CSV, Parquet, and DuckDB."""
name = "local"
markets = {"us_equity", "a_share", "hk_equity", "crypto", "futures", "forex", "macro", "fund"}
requires_auth = False
def __init__(self) -> None:
self._config: dict[str, Any] | None = None
self._source_by_symbol: dict[str, dict[str, Any]] = {}
def is_available(self) -> bool:
"""Return True when the YAML config file exists and has sources."""
config = _load_config()
if config is None:
return False
sources = config.get("sources")
return isinstance(sources, list) and len(sources) > 0
def _ensure_config(self) -> None:
if self._config is not None:
return
self._config = _load_config() or {}
self._source_by_symbol = {}
for entry in self._config.get("sources", []):
symbol = entry.get("symbol", "").strip()
if symbol:
self._source_by_symbol[symbol] = entry
def fetch(
self,
codes: List[str],
start_date: str,
end_date: str,
*,
interval: str = "1D",
fields: Optional[List[str]] = None,
) -> Dict[str, pd.DataFrame]:
"""Fetch OHLCV data for each code from configured local sources.
Args:
codes: Symbol list, optionally prefixed with ``local:``.
start_date: YYYY-MM-DD.
end_date: YYYY-MM-DD.
interval: Bar size (all intervals supported if data contains them).
fields: Ignored.
Returns:
Mapping clean_symbol -> OHLCV DataFrame.
"""
validate_date_range(start_date, end_date)
self._ensure_config()
result: Dict[str, pd.DataFrame] = {}
for code in codes:
clean = code.split(":", 1)[1] if code.startswith("local:") else code
entry = self._source_by_symbol.get(clean)
if entry is None:
logger.warning("local loader: no config entry for symbol %s", clean)
continue
try:
df = cached_loader_fetch(
source=self.name,
symbol=clean,
timeframe=interval,
start_date=start_date,
end_date=end_date,
fields=None,
fetch=lambda c=clean: self._fetch_one(c, start_date, end_date, interval),
)
if df is not None and not df.empty:
result[clean] = df
except Exception as exc:
logger.warning("local loader failed for %s: %s", clean, exc)
return result
def _fetch_one(
self, symbol: str, start_date: str, end_date: str, interval: str = "1D",
) -> pd.DataFrame | None:
entry = self._source_by_symbol.get(symbol)
if entry is None:
return None
src_type: str = entry.get("type", "csv").strip().lower()
reader = _READERS.get(src_type)
if reader is None:
logger.warning("local loader: unsupported type %r for symbol %s", src_type, symbol)
return None
col_map: dict[str, str] = dict(_DEFAULT_COLUMNS)
user_cols = entry.get("columns")
if isinstance(user_cols, dict):
for k, v in user_cols.items():
if isinstance(v, str):
col_map[k] = v
date_fmt: str | None = None
user_fmt = entry.get("date_format")
if isinstance(user_fmt, str) and user_fmt.strip():
date_fmt = user_fmt.strip()
if src_type == "duckdb":
db_path = str(Path(entry.get("db_path", "")).expanduser())
query = entry.get("query", "").strip()
if not db_path or not query:
logger.warning(
"local loader: missing db_path or query for duckdb symbol %s", symbol
)
return None
df = _read_duckdb(db_path, query, col_map, date_fmt)
else:
path = entry.get("path", "").strip()
if not path:
logger.warning("local loader: missing path for symbol %s", symbol)
return None
expanded = str(Path(path).expanduser())
df = reader(expanded, col_map, date_fmt)
if df is None:
return None
start = pd.Timestamp(start_date)
# Treat ``end_date`` as inclusive of the whole day so intraday bars on
# the end day survive the filter (a bare midnight bound dropped them,
# which would defeat any sub-daily ``interval``).
end = pd.Timestamp(end_date) + pd.Timedelta(days=1) - pd.Timedelta(seconds=1)
df = df[(df.index >= start) & (df.index <= end)]
if df.empty:
return None
df = _resample_to_interval(df, interval, symbol)
if df.empty:
return None
return df
agent/backtest/loaders/longbridge.py
+411
View File
@@ -0,0 +1,411 @@
"""LongPort (Longbridge) OpenAPI-backed loader for US and HK equity OHLCV data.
Wraps the ``longbridge`` SDK :class:`~longbridge.openapi.QuoteContext` to fetch
historical candlesticks for backtesting. Supports US and HK equities.
Auth requires the three LongPort credentials declared in
``src.config.env_schema.DataConfig``:
``LONGBRIDGE_APP_KEY``, ``LONGBRIDGE_APP_SECRET``, ``LONGBRIDGE_ACCESS_TOKEN``.
Paper-vs-live identity guard: this loader does **not** discriminate between
paper and live environments (LongPort exposes no API field for it). The loaded
Access Token implicitly selects the environment. For backtest purposes the
historical bars are identical regardless of source account.
The LongPort ``history_candlesticks_by_date`` endpoint caps responses at
~1000 bars per call. Date ranges longer than ~4 years of daily bars are
automatically split into sequential 180-day windows. Requests wider than the
bounded window budget fail explicitly instead of returning truncated history.
This module is for backtest data only; live trading uses the separate
``src.trading.connectors.longbridge.sdk`` module.
"""
from __future__ import annotations
import datetime as dt
import logging
from typing import Any, List, Optional
import pandas as pd
from backtest.loaders.base import (
NoAvailableSourceError,
loader_cache_get,
loader_cache_put,
validate_date_range,
)
from backtest.loaders.registry import register
from src.trading.connectors.longbridge.credentials import (
LongbridgeCredentialError,
resolve_longbridge_credentials,
)
logger = logging.getLogger(__name__)
_OHLCV_COLUMNS = ["open", "high", "low", "close", "volume"]
_INTERVAL_MAP: dict[str, str] = {
"1D": "Day",
"1d": "Day",
"1W": "Week",
"1w": "Week",
"1M": "Month",
"1H": "Min_60",
"1h": "Min_60",
"1m": "Min_1",
"5m": "Min_5",
"15m": "Min_15",
"30m": "Min_30",
}
# LongPort returns at most ~1000 bars per call. For wide ranges we split into
# sequential windows of this many days to avoid silent truncation.
_MAX_WINDOW_DAYS = 180
# Cap on the number of windows so a pathological request can't loop forever.
_MAX_WINDOWS = 20
class LongbridgeDependencyError(RuntimeError):
"""Raised when the ``longbridge`` SDK is not installed."""
def _require_longbridge():
"""Import and return the ``longbridge.openapi`` module.
Raises:
LongbridgeDependencyError: If the SDK is not installed.
"""
try:
from longbridge import openapi # noqa: PLC0415
except ImportError as exc:
raise LongbridgeDependencyError(
"The 'longbridge' SDK is not installed. "
"Run: pip install 'vibe-trading-ai[longbridge]'"
) from exc
return openapi
def _to_longport_symbol(code: str) -> str:
"""Convert a project symbol to LongPort format.
LongPort accepts ``AAPL.US``, ``700.HK``, ``000001.SZ``, ``600519.SH``.
Bare codes without a ``.`` suffix get ``.US`` appended so the resolver
treats them as US equities (loader ``markets`` is us_equity + hk_equity;
ambiguous codes lean US).
Examples:
AAPL -> AAPL.US
AAPL.US -> AAPL.US
700.HK -> 700.HK
0700.HK -> 0700.HK
000001.SZ -> 000001.SZ
600519.SH -> 600519.SH
"""
upper = code.strip().upper()
if "." in upper:
return upper
return f"{upper}.US"
def _to_longport_period(interval: str):
"""Map a project interval string to a LongPort ``Period`` enum value.
Lazy-imports the SDK so this module can be imported without it installed.
Unsupported intervals fail explicitly so requested bar fidelity is never
changed silently.
"""
openapi = _require_longbridge()
period_cls = getattr(openapi, "Period")
token = interval.strip()
attr = _INTERVAL_MAP.get(token)
if attr is None:
raise NoAvailableSourceError(
f"unsupported Longbridge interval: {interval!r}; "
f"supported intervals: {sorted(_INTERVAL_MAP)}"
)
try:
return getattr(period_cls, attr)
except AttributeError as exc:
raise NoAvailableSourceError(
f"installed Longbridge SDK does not expose Period.{attr}"
) from exc
def _date_windows(start: dt.date, end: dt.date) -> list[tuple[dt.date, dt.date]]:
"""Split a wide date range into sequential windows to respect the
LongPort ~1000-bar per-call cap.
Each window spans at most ``_MAX_WINDOW_DAYS`` days. Windows are capped
at ``_MAX_WINDOWS`` so a pathological request cannot loop forever.
"""
requested_days = (end - start).days + 1
maximum_days = _MAX_WINDOW_DAYS * _MAX_WINDOWS
if requested_days > maximum_days:
raise NoAvailableSourceError(
f"Longbridge date range spans {requested_days} days and exceeds "
f"the {maximum_days}-day window limit"
)
windows: list[tuple[dt.date, dt.date]] = []
cursor = start
while cursor <= end and len(windows) < _MAX_WINDOWS:
window_end = min(cursor + dt.timedelta(days=_MAX_WINDOW_DAYS - 1), end)
windows.append((cursor, window_end))
cursor = window_end + dt.timedelta(days=1)
return windows
def _normalize_frame(bars: list[Any]) -> pd.DataFrame:
"""Normalise a list of LongPort Candlestick objects to OHLCV schema.
Args:
bars: List of candlestick objects returned by the SDK.
Returns:
DataFrame with columns [open, high, low, close, volume] indexed
by ``trade_date`` (timezone-naive DatetimeIndex), sorted ascending.
"""
if not bars:
return pd.DataFrame(columns=_OHLCV_COLUMNS)
rows = []
for bar in bars:
ts = getattr(bar, "timestamp", None)
rows.append({
"open": float(getattr(bar, "open", 0) or 0),
"high": float(getattr(bar, "high", 0) or 0),
"low": float(getattr(bar, "low", 0) or 0),
"close": float(getattr(bar, "close", 0) or 0),
"volume": float(getattr(bar, "volume", 0) or 0),
"trade_date": pd.to_datetime(ts) if ts is not None else pd.NaT,
})
result = pd.DataFrame(rows)
result.index = result["trade_date"]
result.index.name = "trade_date"
result = result[_OHLCV_COLUMNS].copy()
# Standardise to timezone-naive UTC. Aware timestamps may be represented
# in an exchange timezone by SDK versions even when the instant is UTC.
if isinstance(result.index, pd.DatetimeIndex) and result.index.tz is not None:
result.index = result.index.tz_convert("UTC").tz_localize(None)
result = result.dropna(subset=["open", "high", "low", "close"])
result["volume"] = result["volume"].fillna(0.0)
return result.sort_index()
@register
class LongbridgeLoader:
"""Fetch US and HK equity bars from LongPort OpenAPI.
Resolves one atomic credential set from the ``LONGBRIDGE_*`` environment
variables or the backward-compatible runtime file.
"""
name = "longbridge"
markets = {"us_equity", "hk_equity"}
requires_auth = True
def __init__(self) -> None:
try:
resolution = resolve_longbridge_credentials()
except LongbridgeCredentialError as exc:
self._credential_source = None
self._credential_error = exc
self._app_key = self._app_secret = self._access_token = ""
return
self._credential_source = resolution.source
self._credential_error: LongbridgeCredentialError | None = None
if resolution.credentials is None:
if resolution.conflict_fields:
code = "credentials_conflict"
fields = resolution.conflict_fields
else:
code = (
"credentials_missing"
if resolution.source is None
else "credentials_partial"
)
fields = resolution.missing_fields
self._credential_error = LongbridgeCredentialError(code, fields)
self._app_key = self._app_secret = self._access_token = ""
return
credentials = resolution.credentials
self._app_key = credentials.app_key
self._app_secret = credentials.app_secret
self._access_token = credentials.access_token
def is_available(self) -> bool:
"""Return True if the LongPort SDK is installed and credentials exist.
Availability checks are side-effect free: they validate configured
credentials and SDK importability without consuming quote API quota.
"""
if not (self._app_key and self._app_secret and self._access_token):
return False
try:
_require_longbridge()
return True
except Exception:
return False
def fetch(
self,
codes: List[str],
start_date: str,
end_date: str,
*,
interval: str = "1D",
fields: Optional[List[str]] = None,
) -> dict[str, pd.DataFrame]:
"""Fetch OHLCV history from LongPort OpenAPI.
Args:
codes: Project symbols such as ``AAPL``, ``AAPL.US``, ``700.HK``,
``000001.SZ`` or ``600519.SH``.
start_date: Start date in ``YYYY-MM-DD`` format.
end_date: End date in ``YYYY-MM-DD`` format.
interval: Backtest interval — ``1D``, ``1W``, ``1M``, ``1H``.
fields: Ignored; included for interface compatibility.
Returns:
Mapping of input symbol to normalised OHLCV dataframe.
Raises:
NoAvailableSourceError: If the SDK connection fails or
credentials are missing.
"""
del fields
if not codes:
return {}
validate_date_range(start_date, end_date)
results: dict[str, pd.DataFrame] = {}
# Serve cached symbols first; only open an SDK connection when at
# least one symbol is uncached, so a fully-cached request needs no
# network call.
pending: List[str] = []
for code in codes:
cached = loader_cache_get(
source=self.name,
symbol=code,
timeframe=interval,
start_date=start_date,
end_date=end_date,
fields=None,
)
if cached is not None:
results[code] = cached.copy()
else:
pending.append(code)
if not pending:
return results
credential_error = getattr(self, "_credential_error", None)
if credential_error is not None:
if credential_error.code == "credentials_missing":
message = (
"Longbridge credentials are not configured; missing fields: "
+ ", ".join(credential_error.fields)
)
elif credential_error.code == "credentials_partial":
message = (
"Longbridge credentials_partial; missing fields: "
+ ", ".join(credential_error.fields)
)
else:
message = (
"Longbridge credentials_conflict; differing fields: "
+ ", ".join(credential_error.fields)
)
raise NoAvailableSourceError(message) from None
if not (self._app_key and self._app_secret and self._access_token):
raise NoAvailableSourceError(
"Longbridge credentials are not configured; set "
"LONGBRIDGE_APP_KEY, LONGBRIDGE_APP_SECRET, and "
"LONGBRIDGE_ACCESS_TOKEN"
)
openapi = _require_longbridge()
_init_error: str | None = None
try:
cfg = openapi.Config(
self._app_key, self._app_secret, self._access_token,
)
ctx = openapi.QuoteContext(cfg)
except LongbridgeDependencyError:
raise
except Exception:
# Capture only stable text; do NOT raise inside the handler so
# the original exception (which may contain secrets) cannot
# leak into __context__.
_init_error = "Longbridge SDK initialization failed."
if _init_error is not None:
raise NoAvailableSourceError(_init_error)
period = _to_longport_period(interval)
adjust_type = getattr(openapi, "AdjustType").NoAdjust
try:
start = dt.date.fromisoformat(start_date)
end = dt.date.fromisoformat(end_date)
except (TypeError, ValueError):
raise NoAvailableSourceError(
"Invalid Longbridge date range."
) from None
windows = _date_windows(start, end)
try:
for code in pending:
lp_symbol = _to_longport_symbol(code)
all_bars: list[Any] = []
_window_error: str | None = None
for w_start, w_end in windows:
try:
bars = ctx.history_candlesticks_by_date(
lp_symbol, period, adjust_type,
start=w_start, end=w_end,
)
if isinstance(bars, (list, tuple)):
all_bars.extend(bars)
elif bars is not None:
all_bars.append(bars)
except Exception:
_window_error = "Longbridge history request failed."
break
else:
_window_error = None
if _window_error is not None:
raise NoAvailableSourceError(_window_error)
if not all_bars:
logger.warning(
"LongPort returned no data for %s in [%s, %s]",
lp_symbol, start_date, end_date,
)
continue
normalized = _normalize_frame(all_bars)
loader_cache_put(
source=self.name,
symbol=code,
timeframe=interval,
start_date=start_date,
end_date=end_date,
fields=None,
frame=normalized,
)
results[code] = normalized
finally:
# LongPort QuoteContext has no explicit close(); the SDK manages
# its own connection pool. No cleanup needed.
pass
return results
agent/backtest/loaders/mootdx_loader.py
+14 -5
View File
@@ -17,7 +17,7 @@ from typing import Dict, List, Optional
import pandas as pd
from backtest.loaders.base import validate_date_range
from backtest.loaders.base import cached_loader_fetch, validate_date_range
from backtest.loaders.registry import register
logger = logging.getLogger(__name__)
@@ -132,7 +132,15 @@ class DataLoader:
)
continue
try:
df = self._fetch_one(code, start_date, end_date, interval)
df = cached_loader_fetch(
source=self.name,
symbol=code,
timeframe=interval,
start_date=start_date,
end_date=end_date,
fields=None,
fetch=lambda code=code: self._fetch_one(code, start_date, end_date, interval),
)
if df is not None and not df.empty:
result[code] = df
except Exception as exc:
@@ -183,9 +191,10 @@ class DataLoader:
if first_dt <= start_ts:
break
else:
logger.warning(
"mootdx: %s %s pagination hit cap (%d pages) without reaching %s",
symbol, freq, _MAX_PAGES, start_date,
raise ValueError(
"incomplete mootdx history: "
f"{symbol} frequency={freq} hit {_MAX_PAGES} pages "
f"without reaching {start_date}"
)
if not chunks:
return None
agent/backtest/loaders/mt5_loader.py
+250
View File
@@ -0,0 +1,250 @@
"""MetaTrader 5 data loader — forex/metals history from a local MT5 terminal.
Feeds backtests and ``get_market_data`` from the user's own broker feed
(e.g. Exness), with the broker's exact symbols and session times. Requires
Windows, the optional ``MetaTrader5`` package (``pip install
"vibe-trading-ai[mt5]"``), and a running, logged-in MT5 terminal — when any of
those are missing :meth:`DataLoader.is_available` is ``False`` and the forex
fallback chain degrades to akshare/yfinance/local.
Config is shared with the MT5 trading connector via ``~/.vibe-trading/mt5.json``
(read directly — this module deliberately does not import the connector).
With no config file, ``mt5.initialize()`` attaches to the last-used, already
logged-in terminal, which is the primary read-only path.
History depth is bounded by the terminal's "Max bars in chart" setting.
Broker account-type suffixes (Exness ``EURUSDm``) are discovered via
``symbols_get`` and memoized; results are keyed by the caller's ORIGINAL
code so the runner's coverage check (``set(codes) - set(data_map)``) holds.
"""
from __future__ import annotations
import json
import logging
from datetime import datetime, timedelta, timezone
from pathlib import Path
from types import ModuleType
from typing import Any
import pandas as pd
from backtest.loaders.base import cached_loader_fetch, validate_date_range, validate_ohlc
from backtest.loaders.registry import register
logger = logging.getLogger(__name__)
#: Shared file contract with the MT5 trading connector (extra keys tolerated).
_MT5_CONFIG_PATH = Path.home() / ".vibe-trading" / "mt5.json"
#: Canonical interval token → MetaTrader5 timeframe constant name.
#: Lowercase ``1h``/``4h``/``1d``/``1w`` alias project-style tokens (connector parity).
#: ``1m`` (minute) and ``1M`` (month) differ by case.
_INTERVAL_MAP = {
"1m": "TIMEFRAME_M1", "5m": "TIMEFRAME_M5", "15m": "TIMEFRAME_M15",
"30m": "TIMEFRAME_M30",
"1H": "TIMEFRAME_H1", "1h": "TIMEFRAME_H1",
"4H": "TIMEFRAME_H4", "4h": "TIMEFRAME_H4",
"1D": "TIMEFRAME_D1", "1d": "TIMEFRAME_D1",
"1W": "TIMEFRAME_W1", "1w": "TIMEFRAME_W1",
"1M": "TIMEFRAME_MN1",
}
#: Process-lifetime attach cache: ``initialize`` can take seconds (it may even
#: launch the terminal), and chain resolution probes ``is_available`` often.
_init_state: bool | None = None
#: base symbol → resolved broker symbol memo.
_symbol_cache: dict[str, str] = {}
def _import_mt5() -> ModuleType | None:
"""Lazy-import the Windows-only SDK; ``None`` when absent."""
try:
import MetaTrader5 # type: ignore[import-not-found]
except ImportError:
return None
return MetaTrader5
def _read_mt5_config() -> dict[str, Any]:
"""Best-effort read of the shared mt5.json ({} on missing/corrupt)."""
try:
raw = json.loads(_MT5_CONFIG_PATH.read_text(encoding="utf-8"))
return raw if isinstance(raw, dict) else {}
except (OSError, ValueError):
return {}
def _ensure_initialized() -> bool:
"""Attach to the terminal once per process (result cached, even negative)."""
global _init_state
if _init_state is not None:
return _init_state
mt5 = _import_mt5()
if mt5 is None:
_init_state = False
return False
config = _read_mt5_config()
kwargs: dict[str, Any] = {}
if config.get("login"):
kwargs["login"] = int(config["login"])
kwargs["password"] = str(config.get("password") or "")
kwargs["server"] = str(config.get("server") or "")
if config.get("timeout"):
kwargs["timeout"] = int(float(config["timeout"]) * 1000)
args = (str(config["terminal_path"]),) if config.get("terminal_path") else ()
try:
ok = bool(mt5.initialize(*args, **kwargs))
except Exception as exc: # noqa: BLE001 - availability probe must not raise
logger.warning("mt5: initialize raised: %s", exc)
ok = False
if not ok:
try:
logger.warning("mt5: terminal attach failed: %s", mt5.last_error())
except Exception: # noqa: BLE001
pass
_init_state = ok
return ok
def _to_query_base(code: str) -> str:
"""``EUR/USD`` / ``EURUSD.FX`` → ``EURUSD`` (upper, separators stripped)."""
token = code.strip().upper()
if token.endswith(".FX"):
token = token[: -len(".FX")]
for separator in ("/", "-", "_", " "):
token = token.replace(separator, "")
return token
def _resolve_broker_symbol(mt5: ModuleType, code: str) -> str | None:
"""Resolve a project code to the broker's symbol name (memoized).
Order: the raw token exactly (case-sensitive broker names pass through),
the normalized base, then suffix discovery via ``symbols_get(f"{base}*")``
— shortest name first, then alphabetical, which deterministically picks
``EURUSDm`` over ``EURUSDz`` on Exness-style brokers.
"""
base = _to_query_base(code)
cached = _symbol_cache.get(base)
if cached is not None:
return cached
raw = code.strip()
for candidate in dict.fromkeys((raw, base)): # ordered, deduped
if candidate and mt5.symbol_info(candidate) is not None:
_symbol_cache[base] = candidate
return candidate
matches = sorted(
(getattr(info, "name", "") for info in (mt5.symbols_get(group=f"{base}*") or ())),
key=lambda name: (len(name), name),
)
matches = [m for m in matches if m]
if not matches:
logger.warning("mt5: symbol %r not offered by this broker (no match for %s*)", code, base)
return None
_symbol_cache[base] = matches[0]
return matches[0]
def _rates_to_frame(rates: Any, start_date: str, end_date: str) -> pd.DataFrame | None:
"""Map a ``copy_rates_range`` structured array to the loader frame contract."""
if rates is None or len(rates) == 0:
return None
frame = pd.DataFrame(rates)
if "time" not in frame.columns or "close" not in frame.columns:
return None
frame["trade_date"] = pd.to_datetime(frame["time"], unit="s")
frame = frame.set_index("trade_date").sort_index()
# Forex real_volume is zero on most brokers; tick_volume is the standard proxy.
frame["volume"] = frame.get("tick_volume", 0)
# Bars carry intraday timestamps; the end date is inclusive of its whole day.
start = pd.Timestamp(start_date)
end = pd.Timestamp(end_date) + pd.Timedelta(days=1)
frame = frame[(frame.index >= start) & (frame.index < end)]
frame = frame[["open", "high", "low", "close", "volume"]].dropna()
frame = validate_ohlc(frame)
return frame if not frame.empty else None
@register
class DataLoader:
"""Forex/metals OHLCV from the local MT5 terminal (Windows, opt-in extra)."""
name = "mt5"
markets = {"forex"}
#: A running, logged-in terminal is the auth surface (futu precedent).
requires_auth = True
def __init__(self) -> None: # never raises — registry availability contract
pass
def is_available(self) -> bool:
"""SDK importable AND the terminal attach succeeded (cached)."""
return _import_mt5() is not None and _ensure_initialized()
def fetch(
self,
codes: list[str],
start_date: str,
end_date: str,
*,
interval: str = "1D",
fields: list[str] | None = None,
) -> dict[str, pd.DataFrame]:
"""Fetch OHLCV frames keyed by the original input codes.
Per-symbol failures log and skip (never raise) so the runner's
runtime fallback can engage for the missing symbols.
"""
validate_date_range(start_date, end_date)
timeframe_name = _INTERVAL_MAP.get(str(interval).strip())
if timeframe_name is None:
# Reject unknown tokens; do not fetch TIMEFRAME_D1 under the caller's key.
logger.warning("mt5 unsupported interval %r; rejecting", interval)
return {}
result: dict[str, pd.DataFrame] = {}
for code in codes:
clean = code.strip()
if not clean:
continue
try:
frame = cached_loader_fetch(
source=self.name,
symbol=_to_query_base(clean),
timeframe=interval,
start_date=start_date,
end_date=end_date,
fields=None,
fetch=lambda c=clean: self._fetch_one(c, start_date, end_date, timeframe_name),
)
except Exception as exc: # noqa: BLE001 - one symbol never poisons the batch
logger.warning("mt5: fetch failed for %s: %s", clean, exc)
continue
if frame is not None and not frame.empty:
result[code] = frame
return result
def _fetch_one(
self, code: str, start_date: str, end_date: str, timeframe_name: str
) -> pd.DataFrame | None:
"""Fetch one symbol from the terminal (``None`` on any failure)."""
mt5 = _import_mt5()
if mt5 is None or not _ensure_initialized():
return None
name = _resolve_broker_symbol(mt5, code)
if name is None:
return None
try:
mt5.symbol_select(name, True)
timeframe = getattr(mt5, timeframe_name)
# tz-aware UTC is mandatory: the MT5 API shifts naive datetimes by
# the local timezone, silently offsetting the requested range.
date_from = datetime.fromisoformat(start_date).replace(tzinfo=timezone.utc)
date_to = datetime.fromisoformat(end_date).replace(tzinfo=timezone.utc) + timedelta(days=1)
rates = mt5.copy_rates_range(name, timeframe, date_from, date_to)
except Exception as exc: # noqa: BLE001 - terminal hiccups degrade, not raise
logger.warning("mt5: rates fetch failed for %s (%s): %s", code, name, exc)
return None
return _rates_to_frame(rates, start_date, end_date)
agent/backtest/loaders/okx.py
+253 -49
View File
@@ -1,10 +1,27 @@
"""OKX spot candle loader (crypto).
Uses OKX V5 public REST API (no auth).
Supports 1m/5m/15m/30m/1H/4H/1D.
Up to 300 bars per request; paginates with ``after`` for longer history.
Endpoints
---------
- ``/market/candles`` — recent bars only (limited depth; not enough for multi-year
backtests).
- ``/market/history-candles`` — multi-year history; used whenever the requested
range is older than a few months or recent endpoint returns empty.
Hardening (2026-07 local audit)
--------------------------------
- Explicit proxy support (same env vars as CCXT loader); required on networks
that time out direct HTTPS to www.okx.com.
- Raise / retry on HTTP 429/5xx and OKX business ``code != "0"``.
- Prefer ``history-candles`` for deep ranges so 2020-era backtests no longer
return empty frames.
- ``is_available()`` does a short probe instead of always returning True.
"""
from __future__ import annotations
import logging
import os
import time
from typing import Dict, List, Optional
@@ -12,21 +29,73 @@ from typing import Dict, List, Optional
import pandas as pd
import requests
logger = logging.getLogger(__name__)
# Project / connector period tokens -> OKX candle ``bar`` strings.
# ``1m`` vs ``1M`` stays case-sensitive; hour/day accept either case.
_INTERVAL_MAP = {
"1m": "1m",
"5m": "5m",
"15m": "15m",
"30m": "30m",
"1h": "1H",
"1H": "1H",
"4h": "4H",
"4H": "4H",
"1d": "1D",
"1D": "1D",
}
from backtest.loaders.base import (
cached_loader_fetch,
check_budget,
positive_env_float,
positive_env_int,
retry_with_budget,
validate_date_range,
)
from backtest.loaders.registry import register
BASE_URL = "https://www.okx.com/api/v5"
CANDLES_PATH = f"{BASE_URL}/market/candles"
HISTORY_CANDLES_PATH = f"{BASE_URL}/market/history-candles"
_MAX_PER_PAGE = 300
# P12-b parity: OKX already sets a per-request timeout but had no retry
# budget, so a transient blip dropped the whole symbol and a slow tier
# could stall ~max_pages*timeout. Bound it like the ccxt loader; retry
# scheduling is delegated to :mod:`backtest.loaders.base`.
_OKX_TIMEOUT = int(os.getenv("OKX_TIMEOUT_S", "15"))
_OKX_FETCH_BUDGET_S = float(os.getenv("OKX_FETCH_BUDGET_S", "60"))
# Recent endpoint typically only covers ~months of 1D bars; beyond this age
# always hit history-candles first.
_RECENT_ONLY_DAYS = 400
_OKX_TIMEOUT = positive_env_int("OKX_TIMEOUT_S", 20)
_OKX_FETCH_BUDGET_S = positive_env_float("OKX_FETCH_BUDGET_S", 90.0)
_OKX_PROBE_TIMEOUT = positive_env_int("OKX_PROBE_TIMEOUT_S", 8)
def _first_proxy_env(*names: str) -> str:
for name in names:
value = os.getenv(name, "").strip() # noqa: env-gate — system proxy vars
if value:
return value
return ""
def _okx_proxy_config() -> dict[str, str]:
"""Build requests proxies from conventional env vars (parity with CCXT)."""
all_proxy = _first_proxy_env("ALL_PROXY", "all_proxy")
http_proxy = _first_proxy_env("HTTP_PROXY", "http_proxy") or all_proxy
https_proxy = _first_proxy_env("HTTPS_PROXY", "https_proxy") or all_proxy or http_proxy
proxies: dict[str, str] = {}
if http_proxy:
proxies["http"] = http_proxy
if https_proxy:
proxies["https"] = https_proxy
return proxies
def _okx_session() -> requests.Session:
session = requests.Session()
proxies = _okx_proxy_config()
if proxies:
session.proxies.update(proxies)
return session
@register
@@ -38,8 +107,22 @@ class DataLoader:
requires_auth = False
def is_available(self) -> bool:
"""Always available (public API, no auth)."""
return True
"""Probe public candles with a short timeout (honours proxy env)."""
try:
session = _okx_session()
resp = session.get(
CANDLES_PATH,
params={"instId": "BTC-USDT", "bar": "1D", "limit": "1"},
timeout=_OKX_PROBE_TIMEOUT,
)
if resp.status_code != 200:
logger.warning("OKX probe HTTP %s", resp.status_code)
return False
data = resp.json()
return data.get("code") == "0" and bool(data.get("data"))
except Exception as exc: # noqa: BLE001 — availability probe
logger.warning("OKX probe failed: %s", exc)
return False
def __init__(self) -> None:
"""No credentials required for public candles."""
@@ -50,8 +133,9 @@ class DataLoader:
codes: List[str],
start_date: str,
end_date: str,
fields: Optional[List[str]] = None,
*,
interval: str = "1D",
fields: Optional[List[str]] = None,
) -> Dict[str, pd.DataFrame]:
"""Fetch crypto OHLCV via OKX public API.
@@ -60,7 +144,7 @@ class DataLoader:
start_date: Start date (YYYY-MM-DD).
end_date: End date (YYYY-MM-DD).
fields: Ignored (OKX has no extra fields).
interval: Bar size (1m/5m/15m/30m/1H/4H/1D), default ``1D``.
interval: Bar size (1m/5m/15m/30m/1h/1H/4h/4H/1d/1D), default ``1D``.
Returns:
Mapping symbol -> DataFrame.
@@ -68,50 +152,132 @@ class DataLoader:
validate_date_range(start_date, end_date)
if fields:
print(f"[WARN] OKX ignores extra fields: {fields}")
logger.warning("OKX ignores extra fields: %s", fields)
valid_intervals = {"1m", "5m", "15m", "30m", "1H", "4H", "1D"}
if interval not in valid_intervals:
print(f"[WARN] unsupported OKX interval {interval}, using 1D")
interval = "1D"
# Case aliases: connector-style ``1h``/``4h`` must not fall through to daily.
mapped = _INTERVAL_MAP.get(interval.strip())
if mapped is None:
logger.warning(
"unsupported OKX interval %r; rejecting (supported: %s)",
interval,
sorted(set(_INTERVAL_MAP.values())),
)
return {}
interval = mapped
codes = [c.replace("/", "-").upper() for c in codes]
start_ts = int(pd.Timestamp(start_date).timestamp() * 1000)
end_ts = int((pd.Timestamp(end_date) + pd.Timedelta(days=1)).timestamp() * 1000)
max_pages = 200 if interval in ("1m", "5m") else 50 if interval in ("15m", "30m") else 20
# More pages for minute bars; history endpoint still needs walk-back.
if interval in ("1m", "5m"):
max_pages = 200
elif interval in ("15m", "30m"):
max_pages = 80
else:
max_pages = 40
use_history = self._should_use_history(start_date)
session = _okx_session()
result: Dict[str, pd.DataFrame] = {}
for symbol in codes:
try:
df = self._fetch_candles(symbol, start_ts, end_ts, interval, max_pages)
df = cached_loader_fetch(
source=self.name,
symbol=symbol,
timeframe=interval,
start_date=start_date,
end_date=end_date,
fields=None,
fetch=lambda symbol=symbol, use_history=use_history: self._fetch_candles(
session,
symbol,
start_ts,
end_ts,
interval,
max_pages,
prefer_history=use_history,
),
)
if df is not None and not df.empty:
result[symbol] = df
except Exception as exc:
print(f"[WARN] failed to fetch {symbol}: {exc}")
logger.warning("failed to fetch %s: %s", symbol, exc)
return result
@staticmethod
def _should_use_history(start_date: str) -> bool:
"""True when the range starts older than recent-only window."""
try:
start = pd.Timestamp(start_date)
age_days = (pd.Timestamp.utcnow().tz_localize(None) - start).days
return age_days > _RECENT_ONLY_DAYS
except Exception:
return True
def _fetch_candles(
self, inst_id: str, start_ts: int, end_ts: int,
bar: str = "1D", max_pages: int = 20,
self,
session: requests.Session,
inst_id: str,
start_ts: int,
end_ts: int,
bar: str = "1D",
max_pages: int = 20,
*,
prefer_history: bool = True,
) -> Optional[pd.DataFrame]:
"""Paginated candle download.
"""Paginated candle download (history endpoint for deep ranges)."""
endpoints: list[str] = []
if prefer_history:
endpoints = [HISTORY_CANDLES_PATH, CANDLES_PATH]
else:
endpoints = [CANDLES_PATH, HISTORY_CANDLES_PATH]
Args:
inst_id: OKX instrument id.
start_ts: Start time (ms).
end_ts: End time (ms).
bar: Bar size.
max_pages: Max pagination rounds.
last_error: Exception | None = None
for endpoint in endpoints:
try:
df = self._paginate(
session,
endpoint,
inst_id,
start_ts,
end_ts,
bar,
max_pages,
)
if df is not None and not df.empty:
return df
except Exception as exc:
last_error = exc
logger.warning(
"OKX %s failed for %s: %s — trying next endpoint",
endpoint.rsplit("/", 1)[-1],
inst_id,
exc,
)
Returns:
OHLCV DataFrame or None.
"""
if last_error is not None:
logger.warning("OKX empty/failed for %s: %s", inst_id, last_error)
else:
logger.warning("OKX empty response: %s", inst_id)
return None
def _paginate(
self,
session: requests.Session,
endpoint: str,
inst_id: str,
start_ts: int,
end_ts: int,
bar: str,
max_pages: int,
) -> Optional[pd.DataFrame]:
all_rows: list = []
after = str(end_ts)
deadline = time.monotonic() + _OKX_FETCH_BUDGET_S
label = f"OKX fetch for {inst_id}"
label = f"OKX fetch for {inst_id} via {endpoint.rsplit('/', 1)[-1]}"
for _ in range(max_pages):
check_budget(deadline, label, budget_s=_OKX_FETCH_BUDGET_S)
@@ -122,47 +288,85 @@ class DataLoader:
"after": after,
}
def _do_request() -> dict:
resp = requests.get(
f"{BASE_URL}/market/candles",
def _do_request(params=params) -> dict:
resp = session.get(
endpoint,
params=params,
timeout=_OKX_TIMEOUT,
)
return resp.json()
# Transient gateway / rate-limit → raise for retry_with_budget
if resp.status_code in {429, 500, 502, 503, 504}:
raise requests.HTTPError(
f"OKX HTTP {resp.status_code}",
response=resp,
)
resp.raise_for_status()
try:
data = resp.json()
except ValueError as exc:
raise requests.RequestException(
f"OKX non-JSON response HTTP {resp.status_code}"
) from exc
code = str(data.get("code", ""))
if code != "0":
# Business errors are not always transient; still surface.
msg = data.get("msg") or data.get("error_message") or code
raise requests.RequestException(f"OKX API code={code} msg={msg}")
return data
data = retry_with_budget(
_do_request,
transient=requests.RequestException,
transient=(requests.RequestException, TimeoutError),
deadline=deadline,
label=label,
)
if data.get("code") != "0" or not data.get("data"):
raw_rows = data.get("data") or []
if not raw_rows:
break
rows = data["data"]
rows = [r for r in rows if r[8] == "1"]
# Keep confirmed bars (confirm=="1"); also keep unconfirmed when
# it is the only data returned so live partial days are not empty.
confirmed = [r for r in raw_rows if len(r) > 8 and str(r[8]) == "1"]
rows = confirmed if confirmed else list(raw_rows)
all_rows.extend(rows)
oldest_ts = int(rows[-1][0]) if rows else start_ts
if oldest_ts <= start_ts or len(data["data"]) < _MAX_PER_PAGE:
oldest_ts = int(raw_rows[-1][0])
if oldest_ts <= start_ts or len(raw_rows) < _MAX_PER_PAGE:
break
after = str(oldest_ts)
if not all_rows:
print(f"[WARN] OKX empty response: {inst_id}")
return None
columns = ["ts", "open", "high", "low", "close", "vol", "volCcy", "volCcyQuote", "confirm"]
df = pd.DataFrame(all_rows, columns=columns)
df["trade_date"] = pd.to_datetime(df["ts"].astype("int64"), unit="ms")
columns = [
"ts", "open", "high", "low", "close",
"vol", "volCcy", "volCcyQuote", "confirm",
]
# Rows may be shorter if API schema changes — pad safely
normalized = []
for r in all_rows:
row = list(r) + [""] * (len(columns) - len(r))
normalized.append(row[: len(columns)])
df = pd.DataFrame(normalized, columns=columns)
# OKX daily open is UTC+8 midnight (= 16:00 UTC). Keep absolute UTC
# timestamps so multi-source merges stay consistent; floor to second.
df["trade_date"] = pd.to_datetime(
pd.to_numeric(df["ts"], errors="coerce"),
unit="ms",
utc=True,
).dt.tz_convert(None)
for col in ["open", "high", "low", "close"]:
df[col] = pd.to_numeric(df[col], errors="coerce")
df["volume"] = pd.to_numeric(df["vol"], errors="coerce").fillna(0)
df = df.set_index("trade_date").sort_index()
df = df.dropna(subset=["trade_date"]).set_index("trade_date").sort_index()
df = df[~df.index.duplicated(keep="last")]
start_dt = pd.Timestamp(start_ts, unit="ms")
end_dt = pd.Timestamp(end_ts, unit="ms")
df = df[(df.index >= start_dt) & (df.index < end_dt)]
df = df[["open", "high", "low", "close", "volume"]].dropna(subset=["open", "high", "low", "close"])
df = df[["open", "high", "low", "close", "volume"]].dropna(
subset=["open", "high", "low", "close"]
)
return df if not df.empty else None
agent/backtest/loaders/pykrx_loader.py
+193
View File
@@ -0,0 +1,193 @@
"""pykrx loader: free, no-auth KRX (KOSPI/KOSDAQ) EOD OHLCV.
Fetches Korean equity **daily** bars through the `pykrx
<https://github.com/sharebook-kr/pykrx>`_ package. No API key.
Provenance caveat (be precise about what this data is): ``pykrx`` exposes both
a raw KRX path and an adjusted-price path, and
``stock.get_market_ohlcv_by_date`` defaults to ``adjusted=True``. The adjusted
path is served by Naver rather than by KRX's own endpoint (upstream discussion:
https://github.com/sharebook-kr/pykrx/issues/270). This loader passes
``adjusted=True`` **explicitly** — split/consolidation-adjusted bars are what a
backtest needs — so be clear that the series is *Naver-backed adjusted* data,
not a verbatim KRX print. Nothing here silently inherits that choice.
Symbol convention (Vibe-Trading -> pykrx):
* ``005930.KS`` (KOSPI) / ``247540.KQ`` (KOSDAQ) -> bare 6-digit ticker
``005930``. The ``.KS``/``.KQ`` suffix follows the Yahoo convention already
used for market inference elsewhere; pykrx itself takes the bare code and
does not care which board it trades on.
pykrx returns a DataFrame indexed by date with Korean column names
(시가/고가/저가/종가/거래량), renamed here to the project's canonical
``open/high/low/close/volume``.
Politeness: pykrx's README asks callers to keep at least ~1 second between
bulk requests, so spacing defaults to 1.0s and runs through the shared,
lock-protected :class:`~backtest.loaders._http.HostThrottle` rather than a
module-global timestamp (which raced under concurrent fetches). Override with
``VIBE_TRADING_PYKRX_MIN_INTERVAL``.
"""
from __future__ import annotations
import logging
from typing import Dict, List, Optional
import pandas as pd
from backtest.loaders._http import HostThrottle, resolve_min_interval
from backtest.loaders.base import cached_loader_fetch, validate_date_range
from backtest.loaders.registry import register
logger = logging.getLogger(__name__)
_COLUMN_MAP = {
"시가": "open",
"고가": "high",
"저가": "low",
"종가": "close",
"거래량": "volume",
}
_OUTPUT_COLUMNS = ["open", "high", "low", "close", "volume"]
# Daily-only upstream API. Accepting anything else would cache day bars under an
# intraday key and run a 1m/1H/4H backtest on daily data.
_DAILY_INTERVALS = frozenset({"1d", "d", "day", "daily"})
# pykrx owns its own HTTP session, so requests cannot route through
# ``throttled_get``; the spacing gate is reused on its own.
_HOST_KEY = "pykrx"
_MIN_INTERVAL_ENV = "VIBE_TRADING_PYKRX_MIN_INTERVAL"
_DEFAULT_MIN_INTERVAL_S = 1.0
_THROTTLE = HostThrottle()
def map_symbol(symbol: str) -> str:
"""``005930.KS`` / ``247540.KQ`` -> pykrx's bare 6-digit ticker."""
return symbol.strip().upper().removesuffix(".KS").removesuffix(".KQ")
def _min_interval() -> float:
"""Resolve the per-call minimum spacing, honoring the env override."""
return resolve_min_interval(_MIN_INTERVAL_ENV, _DEFAULT_MIN_INTERVAL_S)
@register
class DataLoader:
"""KRX (KOSPI/KOSDAQ) EOD OHLCV loader via pykrx (free, no auth)."""
name = "pykrx"
markets = {"kr_equity"}
requires_auth = False
def is_available(self) -> bool:
"""Available when the optional ``pykrx`` package is importable."""
try:
import pykrx # noqa: F401
except ImportError:
return False
return True
def fetch(
self,
codes: List[str],
start_date: str,
end_date: str,
*,
interval: str = "1D",
fields: Optional[List[str]] = None,
) -> Dict[str, pd.DataFrame]:
"""Fetch daily OHLCV bars for ``codes`` over ``[start_date, end_date]``.
Args:
codes: Project-side symbols (e.g. ``["005930.KS", "247540.KQ"]``).
start_date: Inclusive start date (``YYYY-MM-DD``).
end_date: Inclusive end date (``YYYY-MM-DD``).
interval: Only daily (``"1D"``) is supported. Any other value is
rejected with an empty result so the runner's fallback chain can
reach a source that actually serves that bar size.
fields: Unused — pykrx always returns the full OHLCV set.
Returns:
Mapping ``{symbol: DataFrame}`` for symbols that returned data,
each indexed by ``trade_date`` with float OHLCV columns ascending.
A failing or empty symbol is omitted, never aborting the batch.
Empty when ``interval`` is not daily.
"""
validate_date_range(start_date, end_date)
# Daily-only API; do not silently return day bars for runner ``1H``/``4H``.
if str(interval).strip().lower() not in _DAILY_INTERVALS:
logger.warning(
"pykrx supports daily bars only; rejecting interval=%r",
interval,
)
return {}
result: Dict[str, pd.DataFrame] = {}
for code in codes:
try:
df = cached_loader_fetch(
source=self.name,
symbol=code,
timeframe=interval,
start_date=start_date,
end_date=end_date,
fields=None,
fetch=lambda code=code: self._fetch_one(code, start_date, end_date),
)
if df is not None and not df.empty:
result[code] = df
except Exception as exc: # noqa: BLE001 - one bad symbol must not abort the batch
logger.warning("pykrx failed for %s: %s", code, exc)
return result
def _fetch_one(
self, code: str, start_date: str, end_date: str,
) -> Optional[pd.DataFrame]:
"""Fetch and normalize one symbol; ``None`` when KRX has no data.
``adjusted=True`` is passed explicitly: it is pykrx's own default, but
stating it keeps the Naver-backed adjusted-price provenance (see the
module docstring) visible at the call site instead of implied.
"""
from pykrx import stock
_THROTTLE.wait(_HOST_KEY, _min_interval())
frame = stock.get_market_ohlcv_by_date(
pd.Timestamp(start_date).strftime("%Y%m%d"),
pd.Timestamp(end_date).strftime("%Y%m%d"),
map_symbol(code),
adjusted=True,
)
return _normalize(frame)
def _normalize(frame: Optional[pd.DataFrame]) -> Optional[pd.DataFrame]:
"""Convert a raw pykrx OHLCV frame into the project's canonical shape.
Args:
frame: pykrx output — date-indexed with Korean column names
(시가/고가/저가/종가/거래량) — or ``None``/empty when KRX has no
data for the symbol/window.
Returns:
A frame indexed by ``trade_date`` with float OHLCV columns sorted
ascending, or ``None`` when the input carries no usable rows.
"""
if frame is None or frame.empty:
return None
frame = frame.rename(columns=_COLUMN_MAP)
if not all(col in frame.columns for col in _OUTPUT_COLUMNS):
return None
frame.index = pd.to_datetime(frame.index)
frame.index.name = "trade_date"
frame = frame[_OUTPUT_COLUMNS].sort_index()
for col in _OUTPUT_COLUMNS:
frame[col] = pd.to_numeric(frame[col], errors="coerce")
frame = frame.dropna(subset=["open", "high", "low", "close"])
if frame.empty:
return None
return frame.astype(float)
agent/backtest/loaders/qveris_loader.py
+684
View File
@@ -0,0 +1,684 @@
"""QVeris loader: explicit, key-gated OHLCV fetches through QVeris tools.
This loader is intentionally self-contained for the QVeris integration parcel:
it reads the shared ``~/.vibe-trading/qveris.json`` config schema, applies the
``QVERIS_API_KEY`` / ``QVERIS_BASE_URL`` environment overrides, and embeds the
small HTTP client it needs for ``POST /search``, ``POST /tools/execute``, and
truncated-result downloads.
QVeris is a paid-capability router, so it must only run when the user explicitly
requests ``source="qveris"``. The registry keeps it out of every auto fallback
chain.
"""
from __future__ import annotations
import json
import logging
import math
import os
import time
from dataclasses import dataclass
from pathlib import Path
from typing import Any, Dict, Iterable, List, Optional
import pandas as pd
import requests
from backtest.loaders.base import (
cached_loader_fetch,
validate_date_range,
validate_ohlc,
)
from backtest.loaders.registry import register
logger = logging.getLogger(__name__)
_CONFIG_PATH = Path.home() / ".vibe-trading" / "qveris.json"
_DEFAULT_BASE_URL = "https://qveris.ai/api/v1"
_API_KEY_ENV = "QVERIS_API_KEY"
_BASE_URL_ENV = "QVERIS_BASE_URL"
_MIN_INTERVAL_ENV = "VIBE_TRADING_QVERIS_MIN_INTERVAL"
_DEFAULT_MIN_INTERVAL_S = 0.5
_HTTP_TIMEOUT_S = 30.0
_MAX_RETRIES = 3
_OHLCV_COLUMNS = ["open", "high", "low", "close", "volume"]
_DATE_KEYS = (
"trade_date",
"date",
"datetime",
"timestamp",
"time",
"period",
)
_FIELD_ALIASES = {
"open": ("open", "o", "1. open"),
"high": ("high", "h", "2. high"),
"low": ("low", "l", "3. low"),
"close": ("close", "c", "adj_close", "adjusted_close", "4. close"),
"volume": ("volume", "vol", "v", "5. volume", "6. volume"),
}
@dataclass(frozen=True)
class QVerisConfig:
"""Resolved QVeris loader configuration."""
enabled: bool
base_url: str
api_key: str
mode: str
budget_credits_per_session: float
def _load_config() -> QVerisConfig:
"""Read QVeris config with environment overrides.
Returns:
Resolved config. Missing or malformed config files fall back to the
disabled default; env vars only override matching fields and do not
implicitly enable the integration.
"""
raw: dict[str, Any] = {
"enabled": False,
"base_url": _DEFAULT_BASE_URL,
"api_key": "",
"mode": "free",
"budget_credits_per_session": 50.0,
}
try:
if _CONFIG_PATH.is_file():
loaded = json.loads(_CONFIG_PATH.read_text(encoding="utf-8"))
if isinstance(loaded, dict):
raw.update(loaded)
except Exception as exc: # noqa: BLE001 - config read failures mean unavailable
logger.warning("qveris config ignored: %s", exc)
from src.config.accessor import get_env_config
api_key = (get_env_config().data.qveris_api_key or str(raw.get("api_key") or "")).strip()
base_url = (get_env_config().data.qveris_base_url or str(raw.get("base_url") or _DEFAULT_BASE_URL)).strip()
try:
budget = float(raw.get("budget_credits_per_session", 50.0))
except (TypeError, ValueError):
budget = 50.0
if not math.isfinite(budget):
budget = 50.0
return QVerisConfig(
enabled=bool(raw.get("enabled")),
base_url=(base_url or _DEFAULT_BASE_URL).rstrip("/"),
api_key=api_key,
mode=_normalize_mode(str(raw.get("mode") or "free")),
budget_credits_per_session=max(budget, 0.0),
)
def _normalize_mode(mode: str) -> str:
"""Normalize QVeris paid-route mode."""
return {"preview": "free", "allow_paid": "paid", "free": "free", "paid": "paid"}.get(mode.strip(), "free")
def _min_interval() -> float:
"""Resolve the minimum interval between QVeris requests."""
raw = os.getenv(_MIN_INTERVAL_ENV) # noqa: env-gate — loader-specific rate limit
if raw is None or not raw.strip():
return _DEFAULT_MIN_INTERVAL_S
try:
value = float(raw)
except ValueError:
logger.warning(
"invalid %s=%r, using default %s",
_MIN_INTERVAL_ENV,
raw,
_DEFAULT_MIN_INTERVAL_S,
)
return _DEFAULT_MIN_INTERVAL_S
if value < 0:
logger.warning(
"negative %s=%r, using default %s",
_MIN_INTERVAL_ENV,
raw,
_DEFAULT_MIN_INTERVAL_S,
)
return _DEFAULT_MIN_INTERVAL_S
return value
class QVerisClient:
"""Minimal QVeris HTTP client for search, execute, and full-result GET."""
def __init__(self, config: QVerisConfig) -> None:
"""Initialize a session-bound client.
Args:
config: Resolved QVeris config containing base URL and API key.
"""
self._config = config
self._session = requests.Session()
self._last_request_at = 0.0
def search(self, query: str, *, limit: int = 20) -> dict[str, Any]:
"""Call ``POST /search``.
Args:
query: Natural-language capability query.
limit: Maximum result count.
Returns:
Decoded response body.
"""
return self._post_json("/search", {"query": query, "limit": limit})
def execute(
self,
tool_id: str,
parameters: dict[str, Any],
*,
search_id: str | None = None,
) -> dict[str, Any]:
"""Call ``POST /tools/execute`` and hydrate truncated results.
Args:
tool_id: QVeris tool identifier returned by search.
parameters: Provider parameters.
search_id: Optional search correlation id.
Returns:
Decoded execute response, with ``result`` replaced by downloaded
full JSON when QVeris returned ``full_content_file_url``.
"""
body: dict[str, Any] = {
"parameters": parameters,
"max_response_size": 20480,
}
if search_id:
body["search_id"] = search_id
payload = self._post_json(f"/tools/execute?tool_id={tool_id}", body)
if isinstance(payload, dict) and isinstance(payload.get("result"), dict):
full_url = payload["result"].get("full_content_file_url")
if isinstance(full_url, str) and full_url:
payload = dict(payload)
payload["result"] = self._get_json(full_url)
return payload
def _post_json(self, path: str, body: dict[str, Any]) -> dict[str, Any]:
url = f"{self._config.base_url}{path}"
response = self._request("post", url, json=body, auth=True)
decoded = response.json()
return decoded if isinstance(decoded, dict) else {}
def _get_json(self, url: str) -> Any:
response = self._request("get", url, auth=False)
try:
return response.json()
except ValueError:
return json.loads(response.text)
def _request(self, method: str, url: str, *, auth: bool, **kwargs: Any) -> requests.Response:
headers = {"User-Agent": "Vibe-Trading/1.0"}
if auth:
headers["Authorization"] = f"Bearer {self._config.api_key}"
for attempt in range(_MAX_RETRIES + 1):
self._wait()
response = self._session.request(
method,
url,
headers=headers,
timeout=_HTTP_TIMEOUT_S,
**kwargs,
)
if response.status_code != 429:
response.raise_for_status()
return response
if attempt == _MAX_RETRIES:
response.raise_for_status()
time.sleep(_retry_after_seconds(response))
raise AssertionError("unreachable: retry loop must return or raise")
def _wait(self) -> None:
interval = _min_interval()
if interval <= 0:
return
now = time.monotonic()
sleep_for = self._last_request_at + interval - now
if sleep_for > 0:
time.sleep(sleep_for)
self._last_request_at = time.monotonic()
def _retry_after_seconds(response: requests.Response) -> float:
"""Parse a Retry-After header, falling back to the default interval."""
raw = response.headers.get("Retry-After", "")
try:
value = float(raw)
except (TypeError, ValueError):
value = _min_interval()
return max(value, 0.0)
@register
class DataLoader:
"""QVeris OHLCV loader, available only when explicitly configured."""
name = "qveris"
markets = {"us_equity", "hk_equity", "a_share", "crypto", "forex", "fund", "macro"}
requires_auth = True
def __init__(self) -> None:
"""Initialize without network access."""
self._config = _load_config()
def is_available(self) -> bool:
"""Return whether paid QVeris routing is enabled and keyed."""
return self._config.enabled and bool(self._config.api_key) and self._config.mode == "paid"
def fetch(
self,
codes: List[str],
start_date: str,
end_date: str,
*,
interval: str = "1D",
fields: Optional[List[str]] = None,
) -> Dict[str, pd.DataFrame]:
"""Fetch daily OHLCV history through selected QVeris capabilities.
Args:
codes: Symbols to fetch.
start_date: Inclusive ``YYYY-MM-DD`` start date.
end_date: Inclusive ``YYYY-MM-DD`` end date.
interval: Bar interval. QVeris selection is optimized for daily
bars; non-daily values are passed through when a tool accepts an
interval-like parameter.
fields: Ignored; QVeris loader returns the standard OHLCV columns.
Returns:
Mapping of input symbol to normalized OHLCV DataFrames.
Raises:
ValueError: If ``start_date`` > ``end_date``.
"""
del fields
validate_date_range(start_date, end_date)
if not self.is_available():
logger.warning("qveris fetch skipped: disabled or %s not set", _API_KEY_ENV)
return {}
client = QVerisClient(self._config)
budget_state = {"spent": 0.0}
result: Dict[str, pd.DataFrame] = {}
for code in codes:
try:
df = cached_loader_fetch(
source=self.name,
symbol=code,
timeframe=interval,
start_date=start_date,
end_date=end_date,
fields=None,
fetch=lambda code=code: self._fetch_one(
client,
code,
start_date,
end_date,
interval,
budget_state,
),
)
if df is not None and not df.empty:
result[code] = df
except Exception as exc: # noqa: BLE001 - one symbol must not abort the batch
logger.warning("qveris failed for %s: %s", code, exc)
return result
def _fetch_one(
self,
client: QVerisClient,
code: str,
start_date: str,
end_date: str,
interval: str,
budget_state: dict[str, float],
) -> Optional[pd.DataFrame]:
search_payload = client.search(_search_query(code, interval))
candidates = _select_capabilities(search_payload.get("results"), interval)
search_id = _str_or_none(search_payload.get("search_id"))
for capability in candidates:
tool_id = str(capability.get("tool_id") or "").strip()
if not tool_id:
continue
quoted_cost = _expected_cost(capability.get("expected_cost"))
if (
not math.isfinite(quoted_cost)
or quoted_cost < 0.0
or budget_state["spent"] + quoted_cost
> self._config.budget_credits_per_session
):
logger.warning(
"QVeris paid capability skipped for %s: credit budget exceeded",
code,
)
continue
parameters = _build_parameters(capability, code, start_date, end_date, interval)
# Reserve the quoted cost before the request. If transport fails
# after the provider accepted it, later symbols still fail closed.
budget_state["spent"] += quoted_cost
execute_payload = client.execute(tool_id, parameters, search_id=search_id)
try:
actual_cost = float(execute_payload.get("cost"))
except (TypeError, ValueError):
actual_cost = quoted_cost
if math.isfinite(actual_cost) and actual_cost > quoted_cost:
budget_state["spent"] += actual_cost - quoted_cost
if execute_payload.get("success") is False:
logger.warning("QVeris execute failed for %s via %s", code, tool_id)
continue
frame = _result_to_frame(execute_payload.get("result"), start_date, end_date)
if frame is not None:
return frame
logger.warning("QVeris result for %s via %s had no parseable bars", code, tool_id)
return None
_MAX_CANDIDATES = 3
def _search_query(symbol: str, interval: str) -> str:
"""Build a capability-search query for one symbol."""
return f"daily OHLCV historical market data for {symbol.strip().upper()} interval {interval}"
def _select_capabilities(results: Any, interval: str) -> list[dict[str, Any]]:
"""Rank OHLCV-like capabilities, excluding wrong-granularity series."""
if not isinstance(results, list):
return []
wanted, unwanted = _granularity_tokens(interval)
candidates = []
for item in results:
if not isinstance(item, dict) or not _looks_ohlcv(item):
continue
text = _capability_text(item)
if any(token in text for token in unwanted):
continue
priority = 0 if any(token in text for token in wanted) else 1
candidates.append((priority, item))
ranked = sorted(candidates, key=lambda pair: (pair[0],) + _capability_rank(pair[1]))
return [item for _, item in ranked[:_MAX_CANDIDATES]]
def _capability_text(item: dict[str, Any]) -> str:
return " ".join(
str(item.get(key) or "").lower() for key in ("tool_id", "name", "description")
)
def _granularity_tokens(interval: str) -> tuple[tuple[str, ...], tuple[str, ...]]:
"""Return (wanted, unwanted) capability-text tokens for a bar interval.
``1m``/``5m``/``15m``/``30m`` are case-sensitive minute tokens; ``1M`` is
month. Lowercasing first would collapse ``1m`` into an empty match and let
daily capabilities outrank minute ones.
"""
token = interval.strip()
norm = token.lower()
intraday = ("intraday", "minute", "1min", "5min", "15min", "30min", "60min", "hourly")
if norm in ("", "1d", "d", "day", "daily"):
return ("daily", "eod", "end-of-day", "end of day"), ("monthly", "weekly") + intraday
# Case-sensitive: ``1M`` (month) must not be treated as ``1m`` (minute).
if token == "1M" or "month" in norm:
return ("monthly",), ("weekly",) + intraday
if token in {"1m", "5m", "15m", "30m"} or "min" in norm or norm in ("1h", "4h") or "hour" in norm:
return intraday, ("monthly", "weekly")
if "w" in norm or "week" in norm:
return ("weekly",), ("monthly",) + intraday
if "mo" in norm:
return ("monthly",), ("weekly",) + intraday
return (), ()
def _looks_ohlcv(item: dict[str, Any]) -> bool:
text_parts = [
item.get("name"),
item.get("description"),
item.get("provider_name"),
json.dumps(item.get("params") or "", default=str),
json.dumps(item.get("examples") or "", default=str),
]
text = " ".join(str(part or "").lower() for part in text_parts)
has_price = any(token in text for token in ("ohlcv", "open", "high", "low", "close", "candle", "historical price"))
has_symbol = any(token in text for token in ("symbol", "ticker", "instrument", "code"))
return has_price and has_symbol
def _capability_rank(item: dict[str, Any]) -> tuple[float, float, str]:
success_rate = _success_rate(item.get("stats"))
cost = _expected_cost(item.get("expected_cost"))
return (-success_rate, cost, str(item.get("tool_id") or item.get("name") or ""))
def _success_rate(stats: Any) -> float:
if not isinstance(stats, dict):
return 0.0
value = stats.get("success_rate", 0.0)
try:
parsed = float(value)
except (TypeError, ValueError):
return 0.0
return parsed / 100.0 if parsed > 1.0 else parsed
def _expected_cost(value: Any) -> float:
if value is None:
return float("inf")
text = str(value)
number = ""
for char in text:
if char.isdigit() or char == ".":
number += char
elif number:
break
try:
return float(number)
except ValueError:
return float("inf")
def _build_parameters(
capability: dict[str, Any],
code: str,
start_date: str,
end_date: str,
interval: str,
) -> dict[str, Any]:
"""Build an execute parameter object from examples plus known date keys."""
parameters = _sample_parameters(capability)
for param in capability.get("params") or []:
if not isinstance(param, dict):
continue
name = str(param.get("name") or "").strip()
if not name:
continue
lower = name.lower()
if _is_symbol_param(lower):
parameters[name] = _provider_symbol(code)
elif _is_start_param(lower):
parameters[name] = start_date
elif _is_end_param(lower):
parameters[name] = end_date
elif _is_interval_param(lower):
parameters[name] = _interval_value(param, interval)
return parameters
def _sample_parameters(capability: dict[str, Any]) -> dict[str, Any]:
examples = capability.get("examples")
if not isinstance(examples, dict):
return {}
sample = examples.get("sample_parameters")
return dict(sample) if isinstance(sample, dict) else {}
def _provider_symbol(code: str) -> str:
"""Normalize common project US suffixes while preserving other markets."""
upper = code.strip().upper()
if upper.endswith(".US"):
return upper[: -len(".US")]
return upper
def _is_symbol_param(name: str) -> bool:
return any(token in name for token in ("symbol", "ticker", "instrument", "code"))
def _is_start_param(name: str) -> bool:
return any(token in name for token in ("start", "from", "begin")) and "end" not in name
def _is_end_param(name: str) -> bool:
return any(token in name for token in ("end", "to", "until"))
def _is_interval_param(name: str) -> bool:
return any(token in name for token in ("interval", "timeframe", "resolution", "frequency"))
def _interval_value(param: dict[str, Any], interval: str) -> str:
enum = param.get("enum")
if isinstance(enum, list):
lowered = {str(value).lower(): value for value in enum}
for candidate in (interval, interval.lower(), "1d", "d", "daily"):
if candidate.lower() in lowered:
return str(lowered[candidate.lower()])
return "daily" if interval.upper() == "1D" else interval
def _result_to_frame(result: Any, start_date: str, end_date: str) -> Optional[pd.DataFrame]:
rows = [_normalize_record(record) for record in _iter_ohlcv_records(result)]
cleaned = [row for row in rows if row is not None]
if not cleaned:
return None
df = pd.DataFrame(cleaned)
df["trade_date"] = pd.to_datetime(df["trade_date"], errors="coerce").astype(
"datetime64[ns]"
)
for field in _OHLCV_COLUMNS:
if field not in df.columns:
df[field] = None
df[field] = pd.to_numeric(df[field], errors="coerce").astype(float)
df = df.dropna(subset=["trade_date", "open", "high", "low", "close"])
if df.empty:
return None
df = df.set_index("trade_date").sort_index()
df = df[_OHLCV_COLUMNS]
start = pd.Timestamp(start_date)
end = pd.Timestamp(end_date)
df = df.loc[(df.index >= start) & (df.index <= end)]
df = validate_ohlc(df)
return None if df.empty else df
def _iter_ohlcv_records(payload: Any) -> Iterable[dict[str, Any]]:
"""Yield dict-like OHLCV records from common QVeris/provider shapes."""
if isinstance(payload, list):
for item in payload:
yield from _iter_ohlcv_records(item)
return
if not isinstance(payload, dict):
return
if _record_has_ohlc(payload):
yield payload
date_keyed = _date_keyed_records(payload)
if date_keyed is not None:
yield from date_keyed
return
yielded = False
for key in (
"data",
"results",
"result",
"historical",
"prices",
"items",
"rows",
"candles",
"values",
"time_series",
"Time Series (Daily)",
):
if key in payload:
for record in _iter_ohlcv_records(payload[key]):
yielded = True
yield record
if yielded:
return
# Provider-specific series containers ("Weekly Adjusted Time Series",
# "Time Series (5min)", ...) — any nested dict of date-keyed records.
for value in payload.values():
if isinstance(value, dict):
date_keyed = _date_keyed_records(value)
if date_keyed:
yield from date_keyed
def _date_keyed_records(payload: dict[str, Any]) -> list[dict[str, Any]] | None:
rows: list[dict[str, Any]] = []
for key, value in payload.items():
if not isinstance(value, dict) or not _looks_like_date(key):
return None
row = dict(value)
row.setdefault("trade_date", key)
rows.append(row)
return rows if rows else None
def _looks_like_date(value: Any) -> bool:
if not isinstance(value, str):
return False
try:
pd.Timestamp(value)
except Exception:
return False
return True
def _record_has_ohlc(record: dict[str, Any]) -> bool:
lowered = {str(key).lower() for key in record}
return all(any(alias in lowered for alias in aliases) for aliases in (
_FIELD_ALIASES["open"],
_FIELD_ALIASES["high"],
_FIELD_ALIASES["low"],
_FIELD_ALIASES["close"],
))
def _normalize_record(record: dict[str, Any]) -> dict[str, Any] | None:
lowered = {str(key).lower(): value for key, value in record.items()}
date = _first_value(lowered, _DATE_KEYS)
if date is None:
return None
row = {"trade_date": date}
for field, aliases in _FIELD_ALIASES.items():
row[field] = _first_value(lowered, aliases)
if any(row[field] is None for field in ("open", "high", "low", "close")):
return None
return row
def _first_value(mapping: dict[str, Any], keys: Iterable[str]) -> Any:
for key in keys:
if key in mapping:
return mapping[key]
return None
def _str_or_none(value: Any) -> str | None:
if value is None:
return None
text = str(value).strip()
return text or None
agent/backtest/loaders/registry.py
+92 -8
View File
@@ -24,6 +24,40 @@ LOADER_REGISTRY: dict[str, Type[Any]] = {}
_registered = False
# Canonical set of accepted data-source names: every registered loader plus the
# ``"auto"`` cross-market selector. Single source of truth shared by the backtest
# config schema (``backtest.runner.BacktestConfigSchema``) and the agent-facing
# backtest tool (``src.tools.backtest_tool``) so the two can never drift apart.
# Keep in sync with ``_loader_modules`` below — the regression test
# ``test_valid_sources_covers_all_registered_loaders`` enforces full coverage.
VALID_SOURCES: set[str] = {
"tushare",
"okx",
"binance",
"yfinance",
"akshare",
"baostock",
"tencent",
"mootdx",
"ccxt",
"futu",
"eastmoney",
"sina",
"stooq",
"yahoo",
"finnhub",
"alphavantage",
"tiingo",
"fmp",
"qveris", # QVERIS-INTEGRATION
"india_broker",
"pykrx",
"longbridge",
"mt5",
"local",
"auto",
}
def register(cls: Type[Any]) -> Type[Any]:
"""Class decorator: register a loader into the global registry.
@@ -49,11 +83,28 @@ def _ensure_registered() -> None:
_loader_modules = [
"backtest.loaders.tushare",
"backtest.loaders.okx",
"backtest.loaders.binance_loader",
"backtest.loaders.yfinance_loader",
"backtest.loaders.akshare_loader",
"backtest.loaders.baostock_loader",
"backtest.loaders.tencent_loader",
"backtest.loaders.mootdx_loader",
"backtest.loaders.ccxt_loader",
"backtest.loaders.futu",
"backtest.loaders.eastmoney_loader",
"backtest.loaders.sina_loader",
"backtest.loaders.stooq_loader",
"backtest.loaders.yahoo_loader",
"backtest.loaders.finnhub_loader",
"backtest.loaders.alphavantage_loader",
"backtest.loaders.tiingo_loader",
"backtest.loaders.fmp_loader",
"backtest.loaders.qveris_loader", # QVERIS-INTEGRATION
"backtest.loaders.india_broker_loader",
"backtest.loaders.pykrx_loader",
"backtest.loaders.longbridge",
"backtest.loaders.mt5_loader",
"backtest.loaders.local_loader",
]
import importlib
for mod in _loader_modules:
@@ -63,19 +114,39 @@ def _ensure_registered() -> None:
pass
# Sources that must NEVER silently fall through to a network loader when the
# caller asked for them explicitly. ``local`` reads the user's own configured
# files (``~/.vibe-trading/data-bridge/config.yaml``); its ``markets`` set spans
# every market only so the cross-market auto-resolver can *reach* it, not so an
# unavailable ``local`` request can degrade into an unrelated network source.
# An explicit ``local`` request that is unavailable is a config problem the user
# must see, not something to paper over with a Yahoo/Tencent fetch.
_NO_NETWORK_FALLBACK_SOURCES: frozenset[str] = frozenset({"local", "qveris"}) # QVERIS-INTEGRATION
# ---------------------------------------------------------------------------
# Fallback chains: market_type -> ordered list of source names
# ---------------------------------------------------------------------------
# Chains are ordered by IP-ban risk first (lighter, throttle-tolerant public
# endpoints lead; key-gated REST and rate-limit-prone sources trail), then by
# data quality. Eastmoney/Sina/Stooq/Yahoo are unauthenticated public sources
# that must be politely throttled; Finnhub/AlphaVantage/Tiingo/FMP are key-gated
# REST fallbacks placed deeper in the chain.
FALLBACK_CHAINS: dict[str, list[str]] = {
"a_share": ["tushare", "mootdx", "akshare"],
"us_equity": ["yfinance", "akshare"],
"hk_equity": ["yfinance", "futu", "akshare"],
"crypto": ["okx", "ccxt"],
"futures": ["tushare", "akshare"],
"fund": ["tushare", "akshare"],
"macro": ["akshare", "tushare"],
"forex": ["akshare", "yfinance"],
"a_share": ["tencent", "mootdx", "eastmoney", "baostock", "akshare", "tushare", "local"],
"us_equity": ["yahoo", "stooq", "sina", "eastmoney", "yfinance", "tiingo", "fmp", "finnhub", "alphavantage", "longbridge", "akshare", "local"],
"hk_equity": ["eastmoney", "yahoo", "futu", "yfinance", "akshare", "longbridge", "local"],
"india_equity": ["yahoo", "yfinance", "india_broker", "local"],
"kr_equity": ["pykrx", "yahoo", "yfinance", "local"],
# OKX first (native), then dedicated Binance, then generic CCXT / Yahoo.
"crypto": ["okx", "binance", "ccxt", "yfinance", "local"],
"futures": ["tushare", "akshare", "local"],
"fund": ["tushare", "akshare", "local"],
"macro": ["akshare", "tushare", "local"],
# mt5 leads when a local MetaTrader 5 terminal is attached (Windows-only,
# broker feed); otherwise it reports unavailable and the chain proceeds.
"forex": ["mt5", "akshare", "yfinance", "local"],
}
@@ -142,6 +213,19 @@ def get_loader_cls_with_fallback(source: str) -> Type[Any]:
if instance is not None and instance.is_available():
return loader_cls
# Some sources must never silently degrade to an unrelated network loader
# when explicitly requested. ``local`` is the canonical case: its broad
# ``markets`` set exists only to make it reachable from the cross-market
# auto-resolver, so falling back through it would fetch network data the
# user never asked for and mask a Data Bridge config problem. Fail loudly.
if source in _NO_NETWORK_FALLBACK_SOURCES:
raise NoAvailableSourceError(
f"Data source '{source}' is unavailable and does not fall back to a "
f"network source. Check your local Data Bridge config "
f"(~/.vibe-trading/data-bridge/config.yaml) — it must exist and list "
f"at least one source."
)
# Source unavailable — try same-market fallback
for market in loader_cls.markets:
try:
agent/backtest/loaders/rsshub_events.py
+568
View File
@@ -0,0 +1,568 @@
"""RSSHub event/sentiment provider with point-in-time safeguards.
A news / announcement / sentiment provider that runs parallel to the Tushare
fundamental layer (:mod:`backtest.loaders.tushare_fundamentals`). It pulls feeds
from a self-hosted `RSSHub <https://docs.rsshub.app>`_ instance, normalises each
item into the ``event-driven`` skill schema (``date, event_type, score, source,
summary``), and attaches a point-in-time-safe ``event_score`` column to daily
price frames.
Point-in-time discipline (mirrors the fundamentals enricher): every item is
stamped with a *knowable date* — the date a backtest could first have acted on
it (items published after the close roll to the next session) — and only items
with ``knowable_date <= as_of`` are ever returned. No feed item can leak into a
bar dated before it became knowable.
Scoring is pluggable. A deterministic, dependency-free lexicon scorer is used by
default; callers may pass any ``scorer(title, summary) -> float`` (e.g. an LLM
judge, as the ``event-driven`` skill describes) to override it.
"""
from __future__ import annotations
import logging
import os
import time
from dataclasses import dataclass
from email.utils import parsedate_to_datetime
from typing import Any, Callable, Iterable, Mapping
from xml.etree.ElementTree import ParseError
import numpy as np
import pandas as pd
from defusedxml import ElementTree as ET
from defusedxml.common import DefusedXmlException
from backtest.loaders.base import positive_env_float, retry_with_budget
logger = logging.getLogger(__name__)
RSSHUB_BASE_URL_ENV = "RSSHUB_BASE_URL"
RSSHUB_BASE_URL_PLACEHOLDERS = {"", "https://rsshub.example.com"}
RSSHUB_TIMEOUT_ENV = "RSSHUB_TIMEOUT_S"
RSSHUB_BUDGET_ENV = "RSSHUB_FETCH_BUDGET_S"
DEFAULT_TIMEOUT_S = 15.0
DEFAULT_BUDGET_S = 60.0
#: Hour (local to the feed timestamps) at/after which a publication is treated as
#: knowable only on the next calendar day, matching the event-driven skill's
#: "released after market close -> use the next trading day" rule.
DEFAULT_CLOSE_CUTOFF_HOUR = 16
#: Canonical column order for the tidy event frame.
EVENT_COLUMNS: tuple[str, ...] = (
"ts_code",
"knowable_date",
"event_type",
"score",
"source",
"summary",
)
ScorerFn = Callable[[str, str], float]
class EventProviderError(Exception):
"""Base error for event-provider failures."""
class UnknownFeedError(EventProviderError):
"""Raised when a requested feed name is not registered."""
#: Supported ``code_style`` values for per-symbol routes.
CODE_STYLES: frozenset[str] = frozenset({"raw", "exchange_prefix", "bare"})
def format_code_for_route(code: str, style: str) -> str:
"""Convert a backtest code (e.g. ``"600519.SH"``) to a route's expected form.
RSSHub routes disagree on instrument formatting: some take the exchange-
prefixed form (``SH600519``, e.g. xueqiu), some the bare number (``600519``),
some a vendor-specific string. The backtest engine always passes the dotted
``"600519.SH"`` form, so a feed declares the shape its route needs.
Args:
code: Dotted instrument code (``"600519.SH"``); passed through if it has
no ``.`` suffix.
style: One of :data:`CODE_STYLES` — ``"raw"`` (unchanged), ``"bare"``
(symbol only, ``"600519"``), or ``"exchange_prefix"`` (``"SH600519"``).
Returns:
The code formatted for the route.
Raises:
ValueError: If ``style`` is not a recognised code style.
"""
if style not in CODE_STYLES:
raise ValueError(f"Unknown code_style {style!r}; expected one of {sorted(CODE_STYLES)}")
if style == "raw":
return code
symbol, _, suffix = code.partition(".")
if style == "bare":
return symbol
return f"{suffix.upper()}{symbol}" if suffix else symbol
@dataclass(frozen=True)
class FeedSpec:
"""Machine-readable metadata for one RSSHub feed route.
Attributes:
name: Stable feed identifier used in configs (e.g. ``"sina_announcements"``).
route_template: RSSHub route with a ``{code}`` placeholder, joined to the
base URL (e.g. ``"/sina/announcement/{code}"``). Routes without a
``{code}`` placeholder are treated as market-wide feeds.
event_type: Event class emitted for items from this feed; one of the
event-driven taxonomy (``earnings``/``macro``/``policy``/
``sentiment``/``insider``/``technical_break``).
code_style: How the dotted backtest code is rewritten before it is
substituted into a per-symbol ``route_template`` — one of
:data:`CODE_STYLES`. Ignored for market-wide routes.
"""
name: str
route_template: str
event_type: str
code_style: str = "raw"
def __post_init__(self) -> None:
if self.code_style not in CODE_STYLES:
raise ValueError(
f"FeedSpec {self.name!r}: unknown code_style {self.code_style!r}; "
f"expected one of {sorted(CODE_STYLES)}"
)
@property
def is_per_symbol(self) -> bool:
"""Whether the route is parameterised by instrument code."""
return "{code}" in self.route_template
def feed_specs_from_config(entries: Iterable[Mapping[str, Any]]) -> list[FeedSpec]:
"""Build :class:`FeedSpec` objects from backtest-config dicts.
Each entry must carry ``name``, ``route_template`` and ``event_type``; an
optional ``code_style`` defaults to ``"raw"``. There is intentionally no
built-in feed catalogue — routes evolve and silently rot, so feeds are always
declared explicitly in config (or passed to the provider constructor).
Args:
entries: Iterable of mapping objects, one per feed.
Returns:
Parsed feed specs.
Raises:
ValueError: If an entry is missing a required key or has a duplicate name.
"""
specs: list[FeedSpec] = []
seen: set[str] = set()
for entry in entries:
if not isinstance(entry, Mapping):
raise ValueError("each event feed must be a mapping with name/route_template/event_type")
missing = [k for k in ("name", "route_template", "event_type") if not str(entry.get(k, "")).strip()]
if missing:
raise ValueError(f"event feed is missing required field(s): {', '.join(missing)}")
name = str(entry["name"]).strip()
if name in seen:
raise ValueError(f"duplicate event feed name: {name!r}")
seen.add(name)
specs.append(
FeedSpec(
name=name,
route_template=str(entry["route_template"]).strip(),
event_type=str(entry["event_type"]).strip(),
code_style=str(entry.get("code_style", "raw")).strip() or "raw",
)
)
return specs
#: No built-in feed catalogue: RSSHub routes change and break, so feeds are always
#: supplied explicitly (via config ``event_feeds`` or the provider constructor).
DEFAULT_FEEDS: tuple[FeedSpec, ...] = ()
# ── Default dependency-free lexicon scorer ───────────────────────────────────
_POSITIVE_TERMS: frozenset[str] = frozenset(
{
"beat",
"beats",
"surge",
"surges",
"soar",
"record",
"growth",
"profit",
"upgrade",
"upgraded",
"outperform",
"bullish",
"approval",
"approved",
"win",
"wins",
"buyback",
"dividend",
"expansion",
"breakthrough",
}
)
_NEGATIVE_TERMS: frozenset[str] = frozenset(
{
"miss",
"misses",
"plunge",
"plunges",
"fall",
"falls",
"loss",
"losses",
"downgrade",
"downgraded",
"underperform",
"bearish",
"fraud",
"probe",
"lawsuit",
"recall",
"default",
"bankruptcy",
"halt",
"warning",
"decline",
}
)
def default_lexicon_scorer(title: str, summary: str) -> float:
"""Score text in ``[-1, 1]`` by net positive/negative term frequency.
A deterministic, dependency-free baseline. It is intentionally simple — the
provider is the data layer; richer scoring (e.g. an LLM judge) is plugged in
via the ``scorer`` argument on :meth:`RSSHubEventProvider.query_events`.
Args:
title: Item headline.
summary: Item summary/description (may be empty).
Returns:
Sentiment score in ``[-1.0, 1.0]``; ``0.0`` when no terms match.
"""
tokens = f"{title} {summary}".lower().replace("/", " ").split()
if not tokens:
return 0.0
pos = sum(1 for tok in tokens if tok.strip(".,;:!?\"'()") in _POSITIVE_TERMS)
neg = sum(1 for tok in tokens if tok.strip(".,;:!?\"'()") in _NEGATIVE_TERMS)
hits = pos + neg
if hits == 0:
return 0.0
return max(-1.0, min(1.0, (pos - neg) / hits))
def _knowable_date(published: pd.Timestamp, cutoff_hour: int) -> pd.Timestamp:
"""Return the date an item becomes actionable (post-cutoff rolls to next day).
Args:
published: Parsed publication timestamp (tz dropped to naive local).
cutoff_hour: Hour at/after which publication rolls to the next day.
Returns:
Normalised (midnight) timestamp of the knowable date.
"""
stamp = published.tz_localize(None) if published.tzinfo is not None else published
base = stamp.normalize()
if stamp.hour >= cutoff_hour:
base = base + pd.Timedelta(days=1)
return base
def _clean_summary(text: str | None) -> str:
"""Collapse whitespace and strip commas so the value is CSV-safe."""
if not text:
return ""
return " ".join(text.replace(",", " ").split())
class RSSHubEventProvider:
"""Point-in-time-safe event/sentiment provider backed by RSSHub.
Attributes:
base_url: Root of the RSSHub instance (no trailing slash).
feeds: Registered feed specs keyed by name.
close_cutoff_hour: Hour after which items roll to the next knowable day.
"""
def __init__(
self,
base_url: str | None = None,
*,
feeds: Iterable[FeedSpec] | None = None,
client: Any | None = None,
close_cutoff_hour: int = DEFAULT_CLOSE_CUTOFF_HOUR,
) -> None:
"""Initialise the provider.
Args:
base_url: RSSHub root URL; defaults to ``$RSSHUB_BASE_URL``.
feeds: Feed specs to register; defaults to :data:`DEFAULT_FEEDS`.
client: Optional injected HTTP client exposing ``get(url, timeout=...)``
that returns an object with ``.text`` and ``.raise_for_status()``
(an ``httpx.Client``-like). Injectable for offline testing.
close_cutoff_hour: Knowable-date roll cutoff (0-23).
"""
from src.config.accessor import get_env_config
resolved = (base_url if base_url is not None else get_env_config().data.rsshub_base_url).strip()
self.base_url = resolved.rstrip("/")
specs = tuple(feeds) if feeds is not None else DEFAULT_FEEDS
self.feeds: dict[str, FeedSpec] = {spec.name: spec for spec in specs}
self.close_cutoff_hour = close_cutoff_hour
self._client = client
def is_available(self) -> bool:
"""Whether a usable RSSHub base URL is configured."""
return self.base_url not in {p.rstrip("/") for p in RSSHUB_BASE_URL_PLACEHOLDERS}
def list_feeds(self) -> list[str]:
"""Return registered feed names in stable order."""
return sorted(self.feeds)
def describe_feed(self, feed: str) -> FeedSpec:
"""Return the spec for a registered feed.
Raises:
UnknownFeedError: If ``feed`` is not registered.
"""
try:
return self.feeds[feed]
except KeyError as exc:
raise UnknownFeedError(f"Unknown RSSHub feed: {feed}") from exc
def query_events(
self,
codes: Iterable[str],
*,
as_of: str | pd.Timestamp,
feeds: Iterable[str] | None = None,
scorer: ScorerFn | None = None,
) -> pd.DataFrame:
"""Fetch and normalise feed items, dropping anything not yet knowable.
Args:
codes: Instrument codes to query.
as_of: Point-in-time boundary; items with ``knowable_date > as_of``
are excluded (no look-ahead).
feeds: Feed names to query; defaults to all registered feeds.
scorer: Optional ``scorer(title, summary) -> float`` override;
defaults to :func:`default_lexicon_scorer`.
Returns:
Tidy frame with :data:`EVENT_COLUMNS`, sorted by
``(ts_code, knowable_date)`` and de-duplicated.
"""
score_fn = scorer or default_lexicon_scorer
feed_names = list(feeds) if feeds is not None else self.list_feeds()
as_of_date = pd.Timestamp(as_of).normalize()
code_list = list(codes)
rows: list[dict[str, Any]] = []
attempted = 0
failed = 0
for feed_name in feed_names:
spec = self.describe_feed(feed_name)
for code in code_list:
attempted += 1
xml_text = self._fetch_feed(spec, code)
if not xml_text:
failed += 1
continue
rows.extend(self._parse_items(xml_text, code, spec, score_fn))
# A configured-but-fully-unreachable provider must fail loudly rather than
# silently scoring every bar 0.0 (which reads as "sentiment considered, no
# signal"). A reachable feed that simply returned no items is legitimate.
if attempted and failed == attempted:
raise EventProviderError(
f"All {attempted} RSSHub feed fetch(es) failed for base_url "
f"{self.base_url!r}; verify the instance is reachable and the "
f"configured routes exist (feeds={feed_names})"
)
if not rows:
return pd.DataFrame(columns=EVENT_COLUMNS)
frame = pd.DataFrame(rows, columns=list(EVENT_COLUMNS))
frame = frame[frame["knowable_date"] <= as_of_date]
frame = frame.drop_duplicates(subset=["ts_code", "knowable_date", "event_type", "summary"])
frame = frame.sort_values(["ts_code", "knowable_date"]).reset_index(drop=True)
return frame
def _fetch_feed(self, spec: FeedSpec, code: str) -> str:
"""Fetch raw RSS XML for one feed/code, bounded by a wall-clock budget.
Returns the response body on success, or ``""`` when the fetch could not
complete within the budget — the caller logs and counts that as a failed
feed so an all-feeds-unreachable run can be surfaced loudly.
"""
if spec.is_per_symbol:
route = spec.route_template.format(code=format_code_for_route(code, spec.code_style))
else:
route = spec.route_template
url = f"{self.base_url}{route}"
timeout = positive_env_float(RSSHUB_TIMEOUT_ENV, DEFAULT_TIMEOUT_S)
budget = positive_env_float(RSSHUB_BUDGET_ENV, DEFAULT_BUDGET_S)
client = self._client or self._default_client()
deadline = time.monotonic() + budget
def _do() -> str:
response = client.get(url, timeout=timeout)
response.raise_for_status()
return response.text
try:
return retry_with_budget(
_do,
transient=Exception,
deadline=deadline,
label=f"rsshub fetch {spec.name} for {code}",
)
except TimeoutError:
logger.warning(
"RSSHub feed %r for %s unreachable within %.0fs budget (%s); "
"no items contributed from this feed",
spec.name,
code,
budget,
url,
)
return ""
@staticmethod
def _default_client() -> Any:
"""Construct a default httpx client (imported lazily)."""
import httpx
return httpx.Client(follow_redirects=True)
def _parse_items(
self,
xml_text: str,
code: str,
spec: FeedSpec,
score_fn: ScorerFn,
) -> list[dict[str, Any]]:
"""Parse RSS 2.0 ``<item>`` nodes into normalised event rows.
Uses ``defusedxml`` to neutralise XXE / entity-expansion attacks in the
untrusted feed payload; malformed or hostile XML yields no rows.
"""
try:
root = ET.fromstring(xml_text)
except (ParseError, DefusedXmlException):
return []
rows: list[dict[str, Any]] = []
for item in root.iter("item"):
title = (item.findtext("title") or "").strip()
summary = _clean_summary(item.findtext("description"))
pub_raw = item.findtext("pubDate")
published = _parse_pubdate(pub_raw)
if published is None:
continue
rows.append(
{
"ts_code": code,
"knowable_date": _knowable_date(published, self.close_cutoff_hour),
"event_type": spec.event_type,
"score": float(score_fn(title, summary)),
"source": spec.name,
"summary": summary or title,
}
)
return rows
def _parse_pubdate(value: str | None) -> pd.Timestamp | None:
"""Parse an RFC-822 (RSS) or ISO ``pubDate`` into a Timestamp, or None."""
if not value or not value.strip():
return None
try:
return pd.Timestamp(parsedate_to_datetime(value))
except (TypeError, ValueError):
pass
try:
return pd.Timestamp(value)
except (TypeError, ValueError):
return None
def enrich_price_frames_with_events(
data_map: dict[str, pd.DataFrame],
provider: RSSHubEventProvider,
*,
as_of: str | pd.Timestamp,
feeds: Iterable[str] | None = None,
decay_lambda: float = 0.1,
lookback: int = 30,
min_abs_score: float = 0.0,
scorer: ScorerFn | None = None,
) -> dict[str, pd.DataFrame]:
"""Attach a point-in-time-safe ``event_score`` column to price frames.
For each bar ``t`` the score aggregates only events whose ``knowable_date``
falls in ``(t - lookback, t]``, exponentially decayed by age — the same
formula as the ``event-driven`` skill — so no future item can influence an
earlier bar. Two columns are added: ``event_score`` (decayed sum, clipped to
``[-1, 1]``) and ``event_count`` (events in the window).
Args:
data_map: Mapping ``{code: OHLCV DataFrame}`` (DatetimeIndex).
provider: Configured :class:`RSSHubEventProvider`.
as_of: Point-in-time boundary passed through to the provider.
feeds: Feed names to query; defaults to all registered feeds.
decay_lambda: Exponential decay per day (higher decays faster).
lookback: Window in days; older events are excluded.
min_abs_score: Drop events with ``|score|`` below this threshold.
scorer: Optional scoring override forwarded to the provider.
Returns:
A new mapping with the same frames plus ``event_score``/``event_count``.
"""
if not data_map:
return data_map
codes = list(data_map)
events = provider.query_events(codes, as_of=as_of, feeds=feeds, scorer=scorer)
enriched = {code: frame.copy() for code, frame in data_map.items()}
if events.empty:
for frame in enriched.values():
frame["event_score"] = 0.0
frame["event_count"] = 0
return enriched
if min_abs_score > 0.0:
events = events[events["score"].abs() >= min_abs_score]
for code, frame in enriched.items():
rows = events[events["ts_code"] == code]
bar_dates = pd.DatetimeIndex(pd.to_datetime(frame.index)).normalize()
scores = pd.Series(0.0, index=frame.index)
counts = pd.Series(0, index=frame.index)
if not rows.empty and len(frame) > 0:
ev_dates = rows["knowable_date"].to_numpy(dtype="datetime64[ns]")
ev_scores = rows["score"].to_numpy(dtype=float)
window = np.timedelta64(lookback, "D")
for pos, bar in enumerate(bar_dates):
bar64 = np.datetime64(bar, "ns")
mask = (ev_dates <= bar64) & (ev_dates > bar64 - window)
if not mask.any():
continue
ages = (bar64 - ev_dates[mask]) / np.timedelta64(1, "D")
decayed = float((ev_scores[mask] * np.exp(-decay_lambda * ages)).sum())
scores.iloc[pos] = max(-1.0, min(1.0, decayed))
counts.iloc[pos] = int(mask.sum())
frame["event_score"] = scores
frame["event_count"] = counts
return enriched
agent/backtest/loaders/sec_edgar_client.py
+234
View File
@@ -0,0 +1,234 @@
"""Shared SEC EDGAR REST client: ticker->CIK mapping + filings/facts fetch.
The U.S. SEC publishes free, no-auth JSON endpoints for company filings and
XBRL financial facts. Three public facts drive this module:
* ``https://www.sec.gov/files/company_tickers.json`` maps every reporting
ticker to its numeric CIK (Central Index Key).
* ``https://data.sec.gov/submissions/CIK##########.json`` returns a company's
recent filing index, keyed by the **zero-padded 10-digit** CIK.
* ``https://data.sec.gov/api/xbrl/companyfacts/CIK##########.json`` returns the
full set of reported XBRL financial concepts for that same padded CIK.
SEC rate-limits by source IP and asks every client to send a descriptive
``User-Agent`` carrying a contact address; bursting without one earns a
temporary block. Every request therefore routes through the shared
:mod:`backtest.loaders._http` throttle under the ``"sec"`` host bucket, and the
UA defaults to a Vibe-Trading contact string overridable via
``VIBE_TRADING_SEC_UA``.
This module is a thin transport client, not a :class:`DataLoaderProtocol`
implementation: it returns raw decoded JSON for downstream loaders/tools to
shape. The ticker->CIK table is fetched once per process and memoized.
"""
from __future__ import annotations
import logging
import threading
from typing import Any, Dict, Optional
from backtest.loaders._http import (
DEFAULT_USER_AGENT,
resolve_min_interval,
throttled_get_json,
)
logger = logging.getLogger(__name__)
_HOST_KEY = "sec"
# SEC asks for no more than ~10 requests/second; 0.12s spacing keeps us well
# under that even with jitter, and is the floor mandated by the parcel contract.
_MIN_INTERVAL_DEFAULT = 0.12
_MIN_INTERVAL_ENV = "VIBE_TRADING_SEC_MIN_INTERVAL"
_TICKERS_URL = "https://www.sec.gov/files/company_tickers.json"
# Symbol suffixes that denote a market rather than a share class, so they can be
# stripped before a ticker lookup.
_MARKET_SUFFIXES = frozenset({"US"})
_SUBMISSIONS_URL = "https://data.sec.gov/submissions/CIK{cik}.json"
_COMPANY_FACTS_URL = "https://data.sec.gov/api/xbrl/companyfacts/CIK{cik}.json"
_UA_ENV = "VIBE_TRADING_SEC_UA"
# SEC's fair-access policy wants a real contact in the UA. Override via env to
# present your own address; this default keeps the client compliant out of box.
_DEFAULT_SEC_UA = f"Vibe-Trading/1.0 (contact: vibe-trading@example.com) {DEFAULT_USER_AGENT}"
# Process-wide memoized ticker->CIK map ("AAPL" -> "0000320193"), built lazily.
_TICKER_CACHE: Optional[Dict[str, str]] = None
_TICKER_CACHE_LOCK = threading.Lock()
def _min_interval() -> float:
"""Resolve the effective per-request spacing, never below the 0.12s floor."""
return max(_MIN_INTERVAL_DEFAULT, resolve_min_interval(_MIN_INTERVAL_ENV, _MIN_INTERVAL_DEFAULT))
def _user_agent() -> str:
"""Return the compliant contact UA, honoring the ``VIBE_TRADING_SEC_UA`` override."""
from src.config.accessor import get_env_config
override = get_env_config().data.vibe_trading_sec_ua
if override and override.strip():
return override.strip()
return _DEFAULT_SEC_UA
def _sec_get_json(url: str) -> Any:
"""GET ``url`` as JSON through the shared SEC-bucket throttle.
Args:
url: Fully-qualified SEC endpoint URL.
Returns:
The decoded JSON body (typically a ``dict``).
Raises:
requests.RequestException: On a non-2xx status or undecodable body,
propagated unchanged for the caller to classify.
"""
return throttled_get_json(
url,
host_key=_HOST_KEY,
min_interval=_min_interval(),
headers={"User-Agent": _user_agent(), "Accept": "application/json"},
)
def _pad_cik(cik: str | int) -> str:
"""Normalize a CIK to the SEC's zero-padded 10-digit string form.
Args:
cik: A CIK as an int or a string that may already be padded or carry a
leading ``"CIK"`` prefix.
Returns:
The CIK as exactly 10 digits, e.g. ``"0000320193"``.
Raises:
ValueError: If ``cik`` contains no digits.
"""
digits = "".join(ch for ch in str(cik) if ch.isdigit())
if not digits:
raise ValueError(f"CIK has no digits: {cik!r}")
return digits.zfill(10)
def _build_ticker_map(payload: Any) -> Dict[str, str]:
"""Build an uppercase ticker->padded-CIK map from the tickers payload.
The SEC payload is a dict of positional string keys to records shaped like
``{"cik_str": 320193, "ticker": "AAPL", "title": "Apple Inc."}``. A
malformed row is skipped rather than aborting the whole map.
Args:
payload: Decoded ``company_tickers.json`` body.
Returns:
Mapping of upper-cased ticker to zero-padded 10-digit CIK.
"""
records = payload.values() if isinstance(payload, dict) else payload
mapping: Dict[str, str] = {}
for record in records or []:
if not isinstance(record, dict):
continue
ticker = record.get("ticker")
cik = record.get("cik_str", record.get("cik"))
if not ticker or cik is None:
continue
try:
mapping[str(ticker).strip().upper()] = _pad_cik(cik)
except ValueError:
logger.debug("skipping ticker row with bad CIK: %r", record)
return mapping
def _ticker_map() -> Dict[str, str]:
"""Return the process-wide ticker->CIK map, fetching once and memoizing.
Returns:
Mapping of upper-cased ticker to zero-padded 10-digit CIK.
Raises:
requests.RequestException: If the one-time tickers fetch fails.
"""
global _TICKER_CACHE
if _TICKER_CACHE is not None:
return _TICKER_CACHE
with _TICKER_CACHE_LOCK:
if _TICKER_CACHE is None:
payload = _sec_get_json(_TICKERS_URL)
_TICKER_CACHE = _build_ticker_map(payload)
return _TICKER_CACHE
def cik_for(ticker: str) -> Optional[str]:
"""Resolve a ticker symbol to its zero-padded 10-digit CIK.
Args:
ticker: A U.S. equity ticker (case-insensitive), e.g. ``"AAPL"``.
Returns:
The padded CIK string (``"0000320193"``) or ``None`` when the ticker is
empty or absent from the SEC table.
Raises:
requests.RequestException: If the one-time tickers fetch fails.
"""
if not ticker or not ticker.strip():
return None
table = _ticker_map()
candidate = ticker.strip().upper()
if candidate in table:
return table[candidate]
# ``AAPL.US`` is this project's US symbol convention while the SEC table
# carries the bare ticker. Only a known market suffix may be stripped — a
# trailing ``.B`` is a share class, not a market.
head, _, tail = candidate.rpartition(".")
if head and tail in _MARKET_SUFFIXES:
candidate = head
if candidate in table:
return table[candidate]
# The SEC table writes share classes with a dash (``BRK-B``, ``MOG-A``) and
# contains no dots at all; users and vendors commonly write a dot.
return table.get(candidate.replace(".", "-"))
def get_submissions(cik: str | int) -> Dict[str, Any]:
"""Fetch the filing-index ("submissions") JSON for a CIK.
Args:
cik: A CIK as int or string; padded to 10 digits internally.
Returns:
The decoded submissions document (company metadata + recent filings).
Raises:
ValueError: If ``cik`` contains no digits.
requests.RequestException: On a non-2xx status or undecodable body.
"""
return _sec_get_json(_SUBMISSIONS_URL.format(cik=_pad_cik(cik)))
def get_company_facts(cik: str | int) -> Dict[str, Any]:
"""Fetch the XBRL ``companyfacts`` JSON for a CIK.
Args:
cik: A CIK as int or string; padded to 10 digits internally.
Returns:
The decoded company-facts document (all reported XBRL concepts).
Raises:
ValueError: If ``cik`` contains no digits.
requests.RequestException: On a non-2xx status or undecodable body.
"""
return _sec_get_json(_COMPANY_FACTS_URL.format(cik=_pad_cik(cik)))
def _reset_ticker_cache_for_tests() -> None:
"""Clear the memoized ticker map. Test-only hook; never called in prod."""
global _TICKER_CACHE
with _TICKER_CACHE_LOCK:
_TICKER_CACHE = None
agent/backtest/loaders/sec_frames.py
+128
View File
@@ -0,0 +1,128 @@
"""Shared period-frame selection for SEC XBRL ``companyfacts`` rows.
``companyfacts`` interleaves several kinds of fact row that can share an ``end``
date:
* **instant** facts (balance-sheet concepts) carry no ``start`` at all;
* **true-quarter** duration frames span roughly 90 days;
* **year-to-date** duration frames span roughly 180 or 270 days;
* **full-year** duration frames span roughly 365 days.
A 10-Q reports the true quarter *and* the year-to-date frame for the same
``end``, under the same ``fy``, ``fp``, ``form`` and ``accn``. Any consumer that
identifies a period by ``end`` — with or without those four companions —
therefore silently conflates a nine-month figure with the quarter that ends on
the same day. On live AAPL revenue data 36 of 81 such keys collide.
The only field that separates them is ``start``, so the identity of a reporting
period is exactly the ``(start, end)`` span. This module is the single
implementation of that rule; ``fundamentals_loader`` and the SEC tools all route
through it rather than re-deriving the thresholds.
"""
from __future__ import annotations
from typing import Any
import pandas as pd
# Inclusive day-count windows. Quarters run 89-93 days but issuers with 4-4-5
# fiscal calendars stretch further, and 52/53-week years land either side of
# 365, so both windows are deliberately generous.
QUARTER_SPAN_DAYS = (60, 120)
ANNUAL_SPAN_DAYS = (330, 380)
#: Period-type labels emitted by :func:`classify_span`.
INSTANT = "instant"
QUARTER = "quarter"
ANNUAL = "annual"
YTD = "ytd"
def span_days(row: dict[str, Any]) -> int | None:
"""Return the number of days a duration fact covers.
Args:
row: One SEC ``companyfacts`` unit row.
Returns:
The day count between ``start`` and ``end``, or ``None`` when the row is
an instant fact (no ``start``) or either date fails to parse.
"""
start, end = row.get("start"), row.get("end")
if not start or not end:
return None
start_ts = pd.to_datetime(start, errors="coerce")
end_ts = pd.to_datetime(end, errors="coerce")
if pd.isna(start_ts) or pd.isna(end_ts):
return None
return int((end_ts - start_ts).days)
def is_quarter_span(days: int | None) -> bool:
"""Return whether ``days`` falls in the true-quarter window."""
return days is not None and QUARTER_SPAN_DAYS[0] <= days <= QUARTER_SPAN_DAYS[1]
def is_annual_span(days: int | None) -> bool:
"""Return whether ``days`` falls in the full-year window."""
return days is not None and ANNUAL_SPAN_DAYS[0] <= days <= ANNUAL_SPAN_DAYS[1]
def classify_span(days: int | None) -> str:
"""Label a span as instant, quarter, annual, or year-to-date.
Args:
days: Output of :func:`span_days`.
Returns:
One of :data:`INSTANT`, :data:`QUARTER`, :data:`ANNUAL`, :data:`YTD`.
Anything that is a duration but matches neither the quarter nor the
annual window is year-to-date or a stub period, and is labelled
:data:`YTD` so a caller never mistakes it for a reporting period.
"""
if days is None:
return INSTANT
if is_quarter_span(days):
return QUARTER
if is_annual_span(days):
return ANNUAL
return YTD
def frame_key(row: dict[str, Any]) -> tuple[Any, Any]:
"""Return the identity of the reporting period a fact row belongs to.
``(start, end)`` and nothing else. Including ``fy`` would split one period
into a separate row per filing that reports it as a comparative, and
including only ``end`` would collide a year-to-date frame with the true
quarter ending the same day.
Args:
row: One SEC ``companyfacts`` unit row.
Returns:
The ``(start, end)`` pair; ``start`` is ``None`` for instant facts.
"""
return (row.get("start"), row.get("end"))
def matches_cadence(row: dict[str, Any], period: str) -> bool:
"""Return whether a fact row is a reporting period at the requested cadence.
Args:
row: One SEC ``companyfacts`` unit row.
period: ``"annual"`` or ``"quarter"``.
Returns:
``True`` for instant facts (they are valid at any cadence and are keyed
on ``end``), for full-year frames when ``period`` is ``"annual"``, and
for true-quarter frames when ``period`` is ``"quarter"``. Year-to-date
frames never match.
"""
kind = classify_span(span_days(row))
if kind == INSTANT:
return True
if period == "annual":
return kind == ANNUAL
return kind == QUARTER
agent/backtest/loaders/sina_loader.py
+200
View File
@@ -0,0 +1,200 @@
"""Sina Finance loader: free, no-auth US daily OHLCV via the JSONP K-line API.
Sina exposes US daily candles through a JSONP endpoint that wraps a JSON array
of ``{d,o,h,l,c,v}`` bars in a JavaScript variable assignment. We request it,
strip the JSONP wrapper, and reshape into the loader's standard OHLCV frame.
API format (JSONP)::
https://stock.finance.sina.com.cn/usstock/api/jsonp_v2.php/var%20x=/US_MinKService.getDailyK?symbol=AAPL
Like Eastmoney, Sina rate-limits by source IP, so every request routes through
the shared per-host throttle in :mod:`backtest.loaders._http`.
"""
from __future__ import annotations
import json
import logging
import re
from typing import Dict, List, Optional
import pandas as pd
from backtest.loaders._http import resolve_min_interval, throttled_get
from backtest.loaders.base import cached_loader_fetch, validate_date_range
from backtest.loaders.registry import register
logger = logging.getLogger(__name__)
_BASE_URL = (
"https://stock.finance.sina.com.cn/usstock/api/jsonp_v2.php/"
"var%20x=/US_MinKService.getDailyK"
)
_HOST_KEY = "sina"
_MIN_INTERVAL_ENV = "VIBE_TRADING_SINA_MIN_INTERVAL"
_DEFAULT_MIN_INTERVAL = 0.5
# Sina wraps the payload as ``var x=([...]);`` — the JSON array sits inside an
# optional ``(...)``. Capture the outermost bracketed array, ignoring the
# assignment/paren scaffolding around it.
_JSONP_ARRAY_RE = re.compile(r"(\[.*\])", re.DOTALL)
def _is_us_equity(code: str) -> bool:
"""Return whether ``code`` is a US-equity symbol this loader handles."""
return bool(code) and code.upper().endswith(".US")
def _to_sina_symbol(code: str) -> str:
"""Map an internal code to Sina's bare ticker (``"AAPL.US"`` -> ``"AAPL"``)."""
return code.upper().rsplit(".", 1)[0]
def _strip_jsonp(raw: str) -> list:
"""Extract the JSON array embedded in Sina's JSONP wrapper.
Args:
raw: Raw response body, e.g. ``var x=([{"d":"2024-01-02",...}]);``.
Returns:
The decoded list of bar dicts.
Raises:
ValueError: If no JSON array can be located or decoded.
"""
match = _JSONP_ARRAY_RE.search(raw.strip())
if not match:
raise ValueError("no JSON array found in Sina JSONP response")
bars = json.loads(match.group(1))
if not isinstance(bars, list):
raise ValueError("Sina JSONP payload is not a list of bars")
return bars
def _bars_to_frame(bars: list, start_date: str, end_date: str) -> Optional[pd.DataFrame]:
"""Reshape Sina ``{d,o,h,l,c,v}`` bars into the standard OHLCV frame.
Args:
bars: List of per-day dicts with ``d/o/h/l/c/v`` keys.
start_date: Inclusive window start (YYYY-MM-DD).
end_date: Inclusive window end (YYYY-MM-DD).
Returns:
A DatetimeIndex (named ``trade_date``) frame with float columns
``open/high/low/close/volume`` clipped to the window, or ``None`` when
no usable rows survive parsing.
"""
rows = []
for bar in bars:
if not isinstance(bar, dict) or "d" not in bar:
continue
try:
rows.append(
{
"trade_date": bar["d"],
"open": float(bar["o"]),
"high": float(bar["h"]),
"low": float(bar["l"]),
"close": float(bar["c"]),
"volume": float(bar["v"]),
}
)
except (KeyError, TypeError, ValueError):
continue
if not rows:
return None
frame = pd.DataFrame(rows)
frame["trade_date"] = pd.to_datetime(frame["trade_date"])
frame = frame.set_index("trade_date").sort_index()
frame = frame[["open", "high", "low", "close", "volume"]].dropna(
subset=["open", "high", "low", "close"]
)
window = frame.loc[
(frame.index >= pd.Timestamp(start_date))
& (frame.index <= pd.Timestamp(end_date))
]
return window if not window.empty else None
@register
class DataLoader:
"""Sina Finance US-equity daily OHLCV loader (free, HTTP/JSONP, no auth)."""
name = "sina"
markets = {"us_equity"}
requires_auth = False
def __init__(self) -> None:
pass
def is_available(self) -> bool:
"""Always available — uses plain throttled HTTP, no credentials."""
return True
def fetch(
self,
codes: List[str],
start_date: str,
end_date: str,
*,
interval: str = "1D",
fields: Optional[List[str]] = None,
) -> Dict[str, pd.DataFrame]:
"""Fetch US daily OHLCV for each code; skip and log per-symbol failures.
Args:
codes: Symbols in ``TICKER.US`` form (e.g. ``"AAPL.US"``).
start_date: Inclusive window start (YYYY-MM-DD).
end_date: Inclusive window end (YYYY-MM-DD).
interval: Bar interval; only daily (``"1D"``) is supported.
fields: Ignored — the standard OHLCV columns are always returned.
Returns:
Mapping ``{code: DataFrame}`` for every code that yielded bars.
Raises:
ValueError: If ``interval`` is not daily or the date range is invalid.
"""
if interval.upper() not in {"1D", "D", "DAY", "DAILY"}:
raise ValueError(f"Unsupported interval {interval!r}; sina is daily-only")
validate_date_range(start_date, end_date)
result: Dict[str, pd.DataFrame] = {}
for code in codes:
try:
df = cached_loader_fetch(
source=self.name,
symbol=code,
timeframe=interval,
start_date=start_date,
end_date=end_date,
fields=None,
fetch=lambda code=code: self._fetch_one(code, start_date, end_date),
)
if df is not None and not df.empty:
result[code] = df
except Exception as exc:
logger.warning("sina failed for %s: %s", code, exc)
return result
def _fetch_one(
self, code: str, start_date: str, end_date: str,
) -> Optional[pd.DataFrame]:
"""Fetch and parse one symbol's daily bars, or ``None`` if non-US/empty."""
if not _is_us_equity(code):
return None
symbol = _to_sina_symbol(code)
response = throttled_get(
_BASE_URL,
host_key=_HOST_KEY,
min_interval=resolve_min_interval(_MIN_INTERVAL_ENV, _DEFAULT_MIN_INTERVAL),
params={"symbol": symbol},
headers={"Referer": "https://stock.finance.sina.com.cn/"},
)
response.raise_for_status()
bars = _strip_jsonp(response.text)
return _bars_to_frame(bars, start_date, end_date)
agent/backtest/loaders/stooq_loader.py
+198
View File
@@ -0,0 +1,198 @@
"""Stooq loader: free, no-auth US-equity EOD OHLCV via CSV download.
Stooq publishes free end-of-day bars from a plain CSV endpoint
(``https://stooq.com/q/d/l/``) with no API key. Like other free quote
providers it rate-limits by source IP and must be throttled, so every request
routes through :mod:`backtest.loaders._http` (shared per-host spacing + session
reuse) under the ``"stooq"`` host bucket.
API format:
https://stooq.com/q/d/l/?s=aapl.us&d1=20240101&d2=20240131&i=d
Symbol convention (Vibe-Trading -> Stooq):
* ``AAPL.US`` -> ``aapl.us`` (Stooq tickers are lowercase; the ``.US`` market
suffix is kept, just lower-cased).
Response body is CSV ``Date,Open,High,Low,Close,Volume``. An unknown symbol or
empty window yields the literal ``"N/D"`` or an empty body, both treated as
"no data" for that symbol (skipped, never fatal to the batch).
"""
from __future__ import annotations
import io
import logging
from typing import Dict, List, Optional
import pandas as pd
from backtest.loaders._http import resolve_min_interval, throttled_get
from backtest.loaders.base import cached_loader_fetch, validate_date_range
from backtest.loaders.registry import register
logger = logging.getLogger(__name__)
_BASE_URL = "https://stooq.com/q/d/l/"
HOST_KEY = "stooq"
_MIN_INTERVAL_ENV = "VIBE_TRADING_STOOQ_MIN_INTERVAL"
_DEFAULT_MIN_INTERVAL_S = 0.6
# Stooq's CSV header columns mapped to our output field names.
_COLUMN_MAP = {
"Open": "open",
"High": "high",
"Low": "low",
"Close": "close",
"Volume": "volume",
}
_OUTPUT_COLUMNS = ["open", "high", "low", "close", "volume"]
def _min_interval() -> float:
"""Resolve the per-call minimum spacing, honoring the env override."""
return resolve_min_interval(_MIN_INTERVAL_ENV, _DEFAULT_MIN_INTERVAL_S)
def map_symbol(symbol: str) -> str:
"""Translate a Vibe-Trading symbol into Stooq's ticker convention.
Args:
symbol: Project-side symbol, e.g. ``AAPL.US``.
Returns:
The lower-cased Stooq ticker, e.g. ``aapl.us``.
"""
return symbol.strip().lower()
def _compact_date(value: str) -> str:
"""Render a date string as Stooq's ``YYYYMMDD`` form."""
return pd.Timestamp(value).strftime("%Y%m%d")
@register
class DataLoader:
"""Stooq US-equity EOD OHLCV loader (free, HTTP CSV, no auth)."""
name = "stooq"
markets = {"us_equity"}
requires_auth = False
def __init__(self) -> None:
pass
def is_available(self) -> bool:
"""Always available — uses plain throttled HTTP, no credentials."""
return True
def fetch(
self,
codes: List[str],
start_date: str,
end_date: str,
*,
interval: str = "1D",
fields: Optional[List[str]] = None,
) -> Dict[str, pd.DataFrame]:
"""Fetch daily OHLCV bars for ``codes`` over ``[start_date, end_date]``.
Args:
codes: Project-side symbols (e.g. ``["AAPL.US", "MSFT.US"]``).
start_date: Inclusive start date (``YYYY-MM-DD``).
end_date: Inclusive end date (``YYYY-MM-DD``).
interval: Bar size; only daily (``"1D"``) is supported. Other
intervals return an empty map so the fallback chain can continue.
fields: Unused — the CSV always carries the full OHLCV set.
Returns:
Mapping ``{symbol: DataFrame}`` for symbols that returned data. Each
frame has a ``DatetimeIndex`` named ``trade_date`` and float columns
``open/high/low/close/volume`` in ascending date order. A symbol that
errors or has no data is omitted, never aborting the batch.
Raises:
ValueError: If ``start_date > end_date`` or a date is unparseable.
"""
validate_date_range(start_date, end_date)
# Daily-only CSV endpoint; do not silently return day bars for ``1H``.
if str(interval).strip().lower() not in {"1d", "d", "day", "daily"}:
logger.warning(
"stooq supports daily bars only; rejecting interval=%r",
interval,
)
return {}
result: Dict[str, pd.DataFrame] = {}
for code in codes:
try:
df = cached_loader_fetch(
source=self.name,
symbol=code,
timeframe=interval,
start_date=start_date,
end_date=end_date,
fields=None,
fetch=lambda code=code: self._fetch_one(code, start_date, end_date),
)
if df is not None and not df.empty:
result[code] = df
except Exception as exc: # noqa: BLE001 - one bad symbol must not abort the batch
logger.warning("stooq failed for %s: %s", code, exc)
return result
def _fetch_one(
self, code: str, start_date: str, end_date: str,
) -> Optional[pd.DataFrame]:
"""Fetch and parse one symbol's CSV; ``None`` when Stooq has no data."""
params = {
"s": map_symbol(code),
"d1": _compact_date(start_date),
"d2": _compact_date(end_date),
"i": "d",
}
response = throttled_get(
_BASE_URL,
host_key=HOST_KEY,
min_interval=_min_interval(),
params=params,
)
response.raise_for_status()
return _parse_csv(response.text)
def _parse_csv(body: str) -> Optional[pd.DataFrame]:
"""Convert a Stooq EOD CSV body into our OHLCV frame, ``None`` on no data.
Args:
body: Raw CSV text ``Date,Open,High,Low,Close,Volume``. An empty body or
the sentinel ``"N/D"`` (unknown symbol / empty window) means no data.
Returns:
A frame indexed by ``trade_date`` with float OHLCV columns sorted
ascending, or ``None`` when the body carries no usable rows.
"""
text = (body or "").strip()
if not text or text.upper().startswith("N/D"):
return None
frame = pd.read_csv(io.StringIO(text))
if frame.empty or "Date" not in frame.columns:
return None
rename = {col: _COLUMN_MAP[col] for col in _COLUMN_MAP if col in frame.columns}
frame = frame.rename(columns=rename)
if not all(col in frame.columns for col in _OUTPUT_COLUMNS):
return None
frame["trade_date"] = pd.to_datetime(frame["Date"], errors="coerce")
frame = frame.dropna(subset=["trade_date"]).set_index("trade_date").sort_index()
frame.index.name = "trade_date"
for col in _OUTPUT_COLUMNS:
frame[col] = pd.to_numeric(frame[col], errors="coerce")
frame = frame[_OUTPUT_COLUMNS].dropna(subset=["open", "high", "low", "close"])
if frame.empty:
return None
return frame.astype(float)
agent/backtest/loaders/tencent_loader.py
+150
View File
@@ -0,0 +1,150 @@
"""Tencent Finance loader: free, no-auth A-share data via HTTP API.
Uses Tencent's ifzq.gtimg.cn API which is not blocked by eastmoney's CDN.
Covers: A-shares (SH/SZ). No API token required.
API format:
https://web.ifzq.gtimg.cn/appstock/app/fqkline/get?param=sh601595,day,2026-06-01,2026-06-13,500,qfq
"""
from __future__ import annotations
import json
import logging
from typing import Dict, List, Optional
import pandas as pd
from backtest.loaders.base import cached_loader_fetch, validate_date_range
from backtest.loaders.registry import register
logger = logging.getLogger(__name__)
_BASE_URL = "https://web.ifzq.gtimg.cn/appstock/app/fqkline/get"
def _is_a_share(code: str) -> bool:
return code.upper().endswith((".SZ", ".SH"))
@register
class DataLoader:
"""Tencent Finance A-share OHLCV loader (free, HTTP, no auth)."""
name = "tencent"
markets = {"a_share"}
requires_auth = False
def is_available(self) -> bool:
"""Always available — uses plain HTTP."""
return True
def __init__(self) -> None:
pass
def fetch(
self,
codes: List[str],
start_date: str,
end_date: str,
*,
interval: str = "1D",
fields: Optional[List[str]] = None,
) -> Dict[str, pd.DataFrame]:
validate_date_range(start_date, end_date)
del fields
if interval.strip().lower() not in {"1d", "d", "day", "daily"}:
logger.warning(
"tencent supports daily bars only; rejecting interval=%s",
interval,
)
return {}
result: Dict[str, pd.DataFrame] = {}
for code in codes:
try:
df = cached_loader_fetch(
source=self.name,
symbol=code,
timeframe=interval,
start_date=start_date,
end_date=end_date,
fields=None,
fetch=lambda code=code: self._fetch_one(code, start_date, end_date),
)
if df is not None and not df.empty:
result[code] = df
except Exception as exc:
logger.warning("tencent failed for %s: %s", code, exc)
return result
def _fetch_one(
self, code: str, start_date: str, end_date: str,
) -> Optional[pd.DataFrame]:
if not _is_a_share(code):
return None
parts = code.upper().split(".")
symbol = parts[0]
suffix = parts[1] if len(parts) > 1 else ""
if suffix == "SH":
tencent_code = f"sh{symbol}"
elif suffix == "SZ":
tencent_code = f"sz{symbol}"
else:
return None
url = (
f"{_BASE_URL}?param={tencent_code},day,"
f"{start_date},{end_date},500,qfq"
)
import urllib.request
req = urllib.request.Request(url, headers={
"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36",
"Referer": "https://web.ifzq.gtimg.cn/",
})
with urllib.request.urlopen(req, timeout=15) as resp:
raw = resp.read().decode("utf-8")
data = json.loads(raw)
# Response: {"code":0,"data":{"sh601595":{"day":[["2026-06-01","21.32",...], ...]}}}
stock_data = data.get("data", {})
if not stock_data:
return None
# Get the first (only) key
stock_key = next(iter(stock_data), None)
if not stock_key:
return None
# Try "day" first, then "qfqday" (forward-adjusted)
klines = stock_data[stock_key].get("qfqday") or stock_data[stock_key].get("day")
if not klines:
return None
# Each row: ["date", "open", "close", "high", "low", "volume"]
rows = []
for k in klines:
if len(k) >= 6:
rows.append({
"trade_date": k[0],
"open": float(k[1]),
"close": float(k[2]),
"high": float(k[3]),
"low": float(k[4]),
"volume": float(k[5]),
})
if not rows:
return None
df = pd.DataFrame(rows)
df["trade_date"] = pd.to_datetime(df["trade_date"])
df = df.set_index("trade_date").sort_index()
df = df[["open", "high", "low", "close", "volume"]].dropna(
subset=["open", "high", "low", "close"]
)
return df
agent/backtest/loaders/tiingo_loader.py
+245
View File
@@ -0,0 +1,245 @@
"""Tiingo loader: US-equity daily OHLCV via the Tiingo REST API (key-gated).
Tiingo serves end-of-day US-equity bars from a documented public endpoint:
GET https://api.tiingo.com/tiingo/daily/{symbol}/prices
?startDate=YYYY-MM-DD&endDate=YYYY-MM-DD&token={KEY}
The response is a JSON array of per-day objects, each carrying ``date`` plus
``open``/``high``/``low``/``close``/``volume`` (and adjusted variants we ignore).
A token is required; it is read from the ``TIINGO_API_KEY`` environment variable
and is never hard-coded. Like every other ban-prone HTTP loader in this package,
all requests route through :mod:`backtest.loaders._http` for per-host throttling
and session reuse — Tiingo rate-limits by client and rejects unspaced bursts.
"""
from __future__ import annotations
import logging
from typing import Dict, List, Optional
import pandas as pd
from backtest.loaders._http import resolve_min_interval, throttled_get_json
from backtest.loaders.base import cached_loader_fetch, validate_date_range
from backtest.loaders.registry import register
logger = logging.getLogger(__name__)
_BASE_URL = "https://api.tiingo.com/tiingo/daily"
# Throttle/session bucket plus the per-provider minimum spacing override.
_HOST_KEY = "tiingo"
_MIN_INTERVAL_ENV = "VIBE_TRADING_TIINGO_MIN_INTERVAL"
_DEFAULT_MIN_INTERVAL = 0.5
_AUTH_ENV = "TIINGO_API_KEY"
# Placeholder values that must not count as a real key (mirrors how other
# key-gated loaders treat blank/example tokens as "not configured").
_KEY_PLACEHOLDERS = {"", "your_tiingo_api_key", "changeme", "xxx"}
# Target OHLCV schema, in column order.
_OHLCV_COLUMNS = ["open", "high", "low", "close", "volume"]
def _resolve_key() -> str:
"""Return the configured Tiingo API key, or an empty string when unset.
Returns:
The trimmed ``TIINGO_API_KEY`` value, or ``""`` when absent or a known
placeholder.
"""
from src.config.accessor import get_env_config
key = get_env_config().data.tiingo_api_key.strip()
return "" if key.lower() in _KEY_PLACEHOLDERS else key
def _to_tiingo_symbol(code: str) -> Optional[str]:
"""Map a project symbol to a Tiingo ticker, or ``None`` when unsupported.
Args:
code: Project symbol such as ``AAPL.US`` or ``AAPL``.
Returns:
The bare lower-cased ticker Tiingo expects (e.g. ``aapl``), or ``None``
for symbols that are clearly not US equities (HK/crypto/A-share).
"""
upper = code.strip().upper()
if not upper:
return None
if upper.endswith(".US"):
upper = upper[:-3]
# Reject non-US market suffixes and crypto pairs outright.
if "." in upper or "-" in upper:
return None
if not upper.isalnum():
return None
return upper.lower()
def _rows_to_frame(rows: List[dict]) -> Optional[pd.DataFrame]:
"""Convert Tiingo's per-day records into the normalized OHLCV frame.
Args:
rows: JSON array decoded from the prices endpoint; each item carries a
``date`` plus ``open``/``high``/``low``/``close``/``volume``.
Returns:
A DataFrame indexed by a ``trade_date`` :class:`~pandas.DatetimeIndex`
with float ``open``/``high``/``low``/``close``/``volume`` columns, or
``None`` when no usable bar is present.
"""
parsed: List[dict] = []
for row in rows:
date = row.get("date")
if date is None:
continue
parsed.append(
{
"trade_date": date,
"open": row.get("open"),
"high": row.get("high"),
"low": row.get("low"),
"close": row.get("close"),
"volume": row.get("volume"),
}
)
if not parsed:
return None
frame = pd.DataFrame(parsed)
# Tiingo stamps each bar with an ISO-8601 datetime (e.g.
# "2024-01-02T00:00:00.000Z"); normalize to a tz-naive midnight index.
index = pd.DatetimeIndex(pd.to_datetime(frame["trade_date"], utc=True, errors="coerce"))
if getattr(index, "tz", None) is not None:
index = index.tz_convert(None)
frame = frame.drop(columns=["trade_date"])
frame.index = index.normalize()
frame.index.name = "trade_date"
frame = frame.apply(pd.to_numeric, errors="coerce")
# Coerce every OHLCV column to float64 so the schema is uniform: integer
# JSON volumes would otherwise land as int64, breaking downstream code that
# assumes a single numeric dtype.
frame = frame.astype("float64")
frame["volume"] = frame["volume"].fillna(0.0)
frame = frame.loc[:, _OHLCV_COLUMNS].sort_index()
frame = frame.dropna(subset=["open", "high", "low", "close"])
return frame if not frame.empty else None
@register
class DataLoader:
"""Fetch US-equity daily OHLCV bars from Tiingo (key-gated REST)."""
name = "tiingo"
markets = {"us_equity"}
requires_auth = True
def __init__(self) -> None:
"""Initialize the loader (no network or SDK touched at construction)."""
pass
def is_available(self) -> bool:
"""Return whether a usable Tiingo API key is configured."""
return bool(_resolve_key())
def fetch(
self,
codes: List[str],
start_date: str,
end_date: str,
*,
interval: str = "1D",
fields: Optional[List[str]] = None,
) -> Dict[str, pd.DataFrame]:
"""Fetch OHLCV history keyed by the original project symbols.
Each symbol is fetched independently so one failing ticker (bad symbol,
transient HTTP error) never aborts the rest of the batch.
Args:
codes: Project symbols such as ``AAPL.US`` or ``MSFT``.
start_date: Inclusive start date in ``YYYY-MM-DD`` format.
end_date: Inclusive end date in ``YYYY-MM-DD`` format.
interval: Backtest interval; only ``1D`` is served by this endpoint.
fields: Ignored; included for interface compatibility.
Returns:
Mapping of input symbol to a normalized OHLCV DataFrame. Symbols
with no usable data are omitted.
Raises:
ValueError: If ``start_date`` > ``end_date`` or dates are unparseable.
"""
del fields
validate_date_range(start_date, end_date)
# Daily-only EOD endpoint; do not silently return day bars for ``1H``.
if str(interval).strip().lower() not in {"1d", "d", "day", "daily"}:
logger.warning(
"tiingo supports daily bars only; rejecting interval=%r",
interval,
)
return {}
key = _resolve_key()
if not key:
logger.warning("tiingo skipped: %s not configured", _AUTH_ENV)
return {}
result: Dict[str, pd.DataFrame] = {}
for code in codes:
try:
frame = cached_loader_fetch(
source=self.name,
symbol=code,
timeframe=interval,
start_date=start_date,
end_date=end_date,
fields=None,
fetch=lambda code=code: self._fetch_one(code, start_date, end_date, key),
)
if frame is not None and not frame.empty:
result[code] = frame
except Exception as exc: # noqa: BLE001 - one symbol must not kill the batch
logger.warning("tiingo failed for %s: %s", code, exc)
return result
def _fetch_one(
self, code: str, start_date: str, end_date: str, key: str,
) -> Optional[pd.DataFrame]:
"""Fetch and normalize a single symbol's daily bars.
Args:
code: Original project symbol.
start_date: Inclusive start date in ``YYYY-MM-DD`` format.
end_date: Inclusive end date in ``YYYY-MM-DD`` format.
key: Resolved Tiingo API key.
Returns:
Normalized OHLCV DataFrame, or ``None`` when the symbol is not a US
equity or the endpoint returns no usable bars.
"""
ticker = _to_tiingo_symbol(code)
if ticker is None:
return None
url = f"{_BASE_URL}/{ticker}/prices"
payload = throttled_get_json(
url,
host_key=_HOST_KEY,
min_interval=resolve_min_interval(_MIN_INTERVAL_ENV, _DEFAULT_MIN_INTERVAL),
params={
"startDate": start_date,
"endDate": end_date,
"token": key,
},
)
# The prices endpoint returns a JSON array; anything else is empty/error.
if not isinstance(payload, list) or not payload:
return None
return _rows_to_frame(payload)
agent/backtest/loaders/tushare.py
+230 -29
View File
@@ -4,18 +4,115 @@ Supports ``interval``: 1D (default) / 1m / 5m / 15m / 30m / 1H.
Minute data uses ``pro.stk_mins()`` (Tushare points >= 2000).
"""
import os
from typing import Dict, List, Optional
import logging
import time
from typing import Any, Callable, Dict, List, Optional
import pandas as pd
from backtest.loaders.base import validate_date_range
from backtest.loaders._symbol_utils import _is_etf_listed
from backtest.loaders.base import cached_loader_fetch, validate_date_range
from backtest.loaders.cn_adjust import apply_qfq
from backtest.loaders.registry import register
#: Substrings that identify a Tushare per-minute quota rejection rather than a
#: real failure. Tushare returns these as an ordinary exception message, so the
#: only way to tell "you are going too fast" from "this symbol does not exist"
#: is to read the text.
_RATE_LIMIT_MARKERS: tuple[str, ...] = (
"每分钟",
"每天",
"抽取",
"访问该接口",
"频率",
"rate limit",
"too many requests",
)
#: Backoff schedule in seconds. The quota window is a minute, so the last wait
#: has to cross one; three attempts totalling ~65s clear it without turning a
#: genuinely broken call into a two-minute hang.
_RATE_LIMIT_BACKOFF_SECONDS: tuple[float, ...] = (5.0, 20.0, 40.0)
def _is_rate_limited(exc: Exception) -> bool:
"""Return whether an exception is a quota rejection rather than a failure.
Args:
exc: The exception raised by a Tushare endpoint call.
Returns:
True when the message carries any marker in :data:`_RATE_LIMIT_MARKERS`.
"""
message = str(exc).lower()
return any(marker.lower() in message for marker in _RATE_LIMIT_MARKERS)
def _call_with_backoff(fn: Callable[..., Any], /, **kwargs: Any) -> Any:
"""Call a Tushare endpoint, waiting out a per-minute quota rejection.
Only quota rejections are retried. Retrying an ordinary failure would turn
a broken symbol or a bad date range into a silent multi-second stall and
then the same error anyway, so those propagate on the first attempt.
Args:
fn: The endpoint callable, e.g. ``api.daily``.
**kwargs: Passed straight through.
Returns:
Whatever the endpoint returns.
Raises:
Exception: The endpoint's own exception, either immediately (not a
quota issue) or after the backoff schedule is exhausted.
"""
for delay in _RATE_LIMIT_BACKOFF_SECONDS:
try:
return fn(**kwargs)
except Exception as exc: # noqa: BLE001 - classified immediately below
if not _is_rate_limited(exc):
raise
logger.warning(
"tushare quota hit (%s); waiting %.0fs before retrying", exc, delay
)
time.sleep(delay)
return fn(**kwargs)
logger = logging.getLogger(__name__)
TUSHARE_TOKEN_PLACEHOLDERS = {"", "your-tushare-token"}
def _is_index(code: str) -> bool:
"""Detect A-share index symbols (000xxx.SH, 000300.SH, 399xxx.SZ)."""
upper = code.upper()
if upper.endswith(".SH"):
digits = upper.split(".")[0]
return len(digits) == 6 and digits.isdigit() and digits.startswith("000")
if upper.endswith(".SZ"):
digits = upper.split(".")[0]
return len(digits) == 6 and digits.isdigit() and digits.startswith("399")
return False
def _is_hk_equity(code: str) -> bool:
"""Detect Hong Kong equity symbols (e.g. 00700.HK)."""
return code.upper().endswith(".HK")
def _is_us_equity(code: str) -> bool:
"""Detect US equity symbols (e.g. AAPL.US)."""
return code.upper().endswith(".US")
def _is_crypto(code: str) -> bool:
"""Detect crypto symbols (e.g. BTC-USDT, ETH/USDT)."""
upper = code.upper()
return upper.endswith("-USDT") or upper.endswith("/USDT")
@register
class DataLoader:
"""Tushare-backed OHLCV loader."""
@@ -26,13 +123,17 @@ class DataLoader:
def is_available(self) -> bool:
"""Available when TUSHARE_TOKEN is set."""
return os.getenv("TUSHARE_TOKEN", "").strip() not in TUSHARE_TOKEN_PLACEHOLDERS
from src.config.accessor import get_env_config
return get_env_config().data.tushare_token.strip() not in TUSHARE_TOKEN_PLACEHOLDERS
def __init__(self) -> None:
"""Initialize Tushare pro API."""
import tushare as ts
token = os.getenv("TUSHARE_TOKEN", "")
from src.config.accessor import get_env_config
token = get_env_config().data.tushare_token
self.api = ts.pro_api(token)
def fetch(
@@ -40,8 +141,9 @@ class DataLoader:
codes: List[str],
start_date: str,
end_date: str,
fields: Optional[List[str]] = None,
*,
interval: str = "1D",
fields: Optional[List[str]] = None,
) -> Dict[str, pd.DataFrame]:
"""Fetch A-share bars via Tushare API.
@@ -57,33 +159,110 @@ class DataLoader:
"""
validate_date_range(start_date, end_date)
if interval != "1D":
# Tencent/sina-style daily aliases; bare ``1d`` must not take the minute path.
if str(interval).strip().lower() in {"1d", "d", "day", "daily"}:
interval = "1D"
elif interval != "1D":
return self._fetch_minutes(codes, start_date, end_date, interval)
sd = start_date.replace("-", "")
ed = end_date.replace("-", "")
cache_fields = list(fields or [])
result: Dict[str, pd.DataFrame] = {}
# Every code goes through the opt-in cache helper, which is a direct
# passthrough when the cache is disabled. Fundamentals are merged inside
# the cached unit so a cached entry already carries its extra columns.
for code in codes:
try:
df = self.api.daily(ts_code=code, start_date=sd, end_date=ed)
if df is None or df.empty:
continue
df = df.sort_values("trade_date")
df["trade_date"] = pd.to_datetime(df["trade_date"])
df = df.set_index("trade_date")
df = df.rename(columns={"vol": "volume"})
for col in ["open", "high", "low", "close", "volume"]:
if col in df.columns:
df[col] = pd.to_numeric(df[col], errors="coerce")
ohlcv = df[["open", "high", "low", "close", "volume"]].dropna(
subset=["open", "high", "low", "close"]
)
result[code] = ohlcv
except Exception as exc:
print(f"[WARN] failed to fetch {code}: {exc}")
def _fetch_one(code: str = code) -> Optional[pd.DataFrame]:
try:
df = self._fetch_daily_frame(code, sd, ed)
if df is None:
return None
merged = self._merge_basic_fields(
{code: df}, [code], start_date, end_date, cache_fields
)
return merged.get(code)
except Exception as exc:
logger.warning("failed to fetch %s: %s", code, exc)
return None
return self._merge_basic_fields(result, codes, start_date, end_date, fields)
df = cached_loader_fetch(
source=self.name,
symbol=code,
timeframe="1D",
start_date=start_date,
end_date=end_date,
fields=cache_fields,
fetch=_fetch_one,
)
if df is not None and not df.empty:
result[code] = df
return result
def _fetch_daily_frame(
self,
code: str,
start_date: str,
end_date: str,
) -> Optional[pd.DataFrame]:
"""Fetch and normalize one daily OHLCV frame, routing by symbol type."""
if _is_us_equity(code) or _is_crypto(code):
logger.warning("tushare does not support %s (US/crypto); skipping", code)
return None
if _is_etf_listed(code):
endpoint_name = "fund_daily"
df = self.api.fund_daily(ts_code=code, start_date=start_date, end_date=end_date)
adjust = getattr(self.api, "fund_adj", None)
elif _is_index(code):
endpoint_name = "index_daily"
df = self.api.index_daily(ts_code=code, start_date=start_date, end_date=end_date)
# An index level is already continuous across its members' ex-dates.
adjust = None
elif _is_hk_equity(code):
endpoint_name = "hk_daily"
df = _call_with_backoff(self.api.hk_daily, ts_code=code, start_date=start_date, end_date=end_date)
# Tushare publishes no HK adjustment-factor series.
adjust = None
else:
endpoint_name = "daily"
df = _call_with_backoff(self.api.daily, ts_code=code, start_date=start_date, end_date=end_date)
adjust = getattr(self.api, "adj_factor", None)
if df is None or df.empty:
logger.warning("tushare returned empty for %s via %s", code, endpoint_name)
return None
df = df.sort_values("trade_date")
df["trade_date"] = pd.to_datetime(df["trade_date"])
df = df.set_index("trade_date")
df = df.rename(columns={"vol": "volume"})
for col in ["open", "high", "low", "close", "volume"]:
if col in df.columns:
df[col] = pd.to_numeric(df[col], errors="coerce")
ohlcv = df[["open", "high", "low", "close", "volume"]].dropna(
subset=["open", "high", "low", "close"]
)
if adjust is None:
return ohlcv
try:
factor = _call_with_backoff(adjust, ts_code=code, start_date=start_date, end_date=end_date)
except Exception as exc: # noqa: BLE001 - one bad fetch must not raise
logger.warning("tushare adjustment-factor fetch failed for %s: %s", code, exc)
factor = None
adjusted = apply_qfq(ohlcv, factor)
if adjusted is None:
# Returning the raw frame here is what produced the defect: a
# close-to-close return across an ex-date spans the mechanical gap,
# measured at -47.2%% on 300750.SZ 2023-04-26 against a true +5.4%%.
logger.warning(
"tushare: no usable adjustment factors for %s — dropping the "
"symbol rather than backtesting it on unadjusted prices",
code,
)
return adjusted
def _merge_basic_fields(
self,
@@ -113,6 +292,15 @@ class DataLoader:
active_codes = [c for c in codes if c in result]
for code in active_codes:
if (
_is_etf_listed(code)
or _is_index(code)
or _is_hk_equity(code)
or _is_us_equity(code)
or _is_crypto(code)
):
# daily_basic is stock-only; skip fundamental enrichment for non-stock symbols
continue
try:
basic = self.api.daily_basic(
ts_code=code,
@@ -127,7 +315,7 @@ class DataLoader:
if f in basic.columns:
result[code][f] = basic[f]
except Exception as exc:
print(f"[WARN] daily_basic for {code} failed: {exc}")
logger.warning("daily_basic for %s failed: %s", code, exc)
return result
@@ -152,7 +340,7 @@ class DataLoader:
freq_map = {"1m": "1min", "5m": "5min", "15m": "15min", "30m": "30min", "1H": "60min"}
freq = freq_map.get(interval)
if not freq:
print(f"[ERROR] unsupported Tushare interval: {interval}")
logger.error("unsupported Tushare interval: %s", interval)
return {}
sd = start_date.replace("-", "")
@@ -160,10 +348,23 @@ class DataLoader:
result: Dict[str, pd.DataFrame] = {}
for code in codes:
if _is_etf_listed(code):
# tushare has no fund_mins endpoint; ETF intraday is unavailable
logger.warning("tushare does not support intraday data for %s (ETF); skipping", code)
continue
if _is_index(code) or _is_hk_equity(code) or _is_us_equity(code) or _is_crypto(code):
sym_type = (
"index" if _is_index(code)
else "HK" if _is_hk_equity(code)
else "US" if _is_us_equity(code)
else "crypto"
)
logger.warning("tushare does not support intraday data for %s (%s); skipping", code, sym_type)
continue
try:
df = self.api.stk_mins(ts_code=code, freq=freq, start_date=sd, end_date=ed)
if df is None or df.empty:
print(f"[WARN] empty Tushare minute data: {code} (points >= 2000 required)")
logger.warning("empty Tushare minute data: %s (points >= 2000 required)", code)
continue
df = df.sort_values("trade_time")
df["trade_date"] = pd.to_datetime(df["trade_time"])
@@ -177,5 +378,5 @@ class DataLoader:
)
result[code] = ohlcv
except Exception as exc:
print(f"[WARN] failed to fetch minute data {code}: {exc}")
logger.warning("failed to fetch minute data %s: %s", code, exc)
return result
agent/backtest/loaders/tushare_fundamentals.py
+101 -8
View File
@@ -2,7 +2,6 @@
from __future__ import annotations
import os
from dataclasses import dataclass
from typing import Any, Iterable
@@ -115,7 +114,9 @@ class TushareFundamentalProvider:
if api is None:
import tushare as ts
token = os.getenv("TUSHARE_TOKEN", "").strip()
from src.config.accessor import get_env_config
token = get_env_config().data.tushare_token.strip()
if token in TUSHARE_TOKEN_PLACEHOLDERS:
token = ""
api = ts.pro_api(token)
@@ -141,7 +142,57 @@ class TushareFundamentalProvider:
periods: Iterable[str] | None = None,
fields: Iterable[str] | None = None,
) -> pd.DataFrame:
"""Query a fundamental table and filter out rows unpublished by ``as_of``."""
"""Query a fundamental table and filter out rows unpublished by ``as_of``.
After the PIT cut, rows are deduplicated so that each ``(ts_code,
end_date)`` pair keeps only the entry with the latest effective pit
date (``f_ann_date`` when present and non-null, falling back to
``ann_date``). Ties on the same pit date preserve the last row in
input order, which matches the natural Tushare ordering where
restatements appear after originals.
"""
result = self._query_pit_cut(table, codes, as_of=as_of, periods=periods, fields=fields)
if result.empty:
return result
schema = self.describe_table(table)
pit_column = schema.point_in_time_column
if pit_column not in result.columns or result[pit_column].isna().all():
pit_column = "ann_date"
pit_values = result[pit_column]
if pit_column != "ann_date" and "ann_date" in result.columns:
pit_values = pit_values.where(pit_values.notna(), result["ann_date"])
result = result.copy()
result["_eff_pit_date"] = pit_values.map(_parse_tushare_date)
# Keep the row with the latest effective pit date per (ts_code, end_date).
# stable sort + keep='last' means that within ties the last row in
# input order wins, which is what we want for restatements appended
# after the original.
result = result.sort_values("_eff_pit_date", kind="stable")
result = result.drop_duplicates(subset=["ts_code", "end_date"], keep="last")
result = result.drop(columns=["_eff_pit_date"])
output_columns = self._output_columns(schema, result, fields)
result = result.loc[:, output_columns].sort_values(["ts_code", "end_date"]).reset_index(drop=True)
return result
def _query_pit_cut(
self,
table: str,
codes: Iterable[str],
*,
as_of: str | pd.Timestamp,
periods: Iterable[str] | None = None,
fields: Iterable[str] | None = None,
) -> pd.DataFrame:
"""Return all rows visible by ``as_of`` without deduplication.
This is the raw PIT cut used internally by
:func:`enrich_price_frames_with_fundamentals` so that per-trading-day
visibility can be evaluated before deduplication is applied.
"""
schema = self.describe_table(table)
requested_periods = set(periods or [])
frames: list[pd.DataFrame] = []
@@ -175,7 +226,7 @@ class TushareFundamentalProvider:
result = result[pit_dates <= as_of_date]
output_columns = self._output_columns(schema, result, fields)
result = result.loc[:, output_columns].sort_values(["ts_code", "end_date"]).reset_index(drop=True)
result = result.loc[:, output_columns].reset_index(drop=True)
return result
def _validate_schema(self, schema: TableSchema, frame: pd.DataFrame) -> None:
@@ -223,6 +274,14 @@ def enrich_price_frames_with_fundamentals(
Fundamental columns are prefixed with their table name, for example
``income_total_revenue`` and ``fina_indicator_roe``. Each row becomes
visible only on or after its announcement/disclosure date.
Restatement handling: when a later disclosure covers an *older* reporting
period than the most-recently-seen period, the snapshot does **not**
regress to that older period. Specifically, the effective announcement
timeline is built by scanning rows in ascending pit-date order and only
accepting a row when its ``end_date`` is >= the ``end_date`` that is
currently visible. Same-period restatements (same ``end_date``, later
pit date) do update the visible values.
"""
if not data_map or not fields_by_table:
return data_map
@@ -232,7 +291,9 @@ def enrich_price_frames_with_fundamentals(
for table, fields in fields_by_table.items():
field_list = list(fields or [])
fundamentals = provider.query_fundamentals(
# Use the raw PIT cut (no dedup) so that per-day visibility can be
# evaluated correctly before same-period deduplication is applied.
fundamentals = provider._query_pit_cut(
table,
codes,
as_of=as_of,
@@ -256,12 +317,44 @@ def enrich_price_frames_with_fundamentals(
if pit_column != "ann_date" and "ann_date" in rows.columns:
pit_values = pit_values.where(pit_values.notna(), rows["ann_date"])
rows["_pit_date"] = pit_values.map(_parse_tushare_date)
rows = rows.dropna(subset=["_pit_date"]).sort_values("_pit_date")
rows["_end_date_parsed"] = rows["end_date"].map(_parse_tushare_date)
rows = rows.dropna(subset=["_pit_date", "_end_date_parsed"]).sort_values(
["_pit_date", "_end_date_parsed"], kind="stable"
)
if rows.empty:
continue
value_columns = [column for column in rows.columns if column not in {"ts_code", "_pit_date"}]
right = rows[["_pit_date", *value_columns]].rename(
# Build effective timeline: scan in ascending pit-date order and
# only include a row when its end_date >= the currently-visible
# end_date. This prevents an old-period restatement (announced
# later) from regressing the snapshot to an earlier period.
# Same-period restatements (same end_date, later pit_date) pass
# the check and are appended as additional timeline entries;
# merge_asof will then naturally surface the later restatement for
# trade dates on or after its pit_date while keeping the original
# visible for earlier dates.
timeline_rows: list[pd.Series] = []
current_end_date: pd.Timestamp | None = None
for _, row in rows.iterrows():
row_end_date: pd.Timestamp = row["_end_date_parsed"]
if current_end_date is None or row_end_date >= current_end_date:
timeline_rows.append(row)
current_end_date = row_end_date
# Rows where row_end_date < current_end_date are silently
# dropped — they represent old-period restatements that must
# not regress the visible snapshot.
if not timeline_rows:
continue
timeline = pd.DataFrame(timeline_rows).drop(columns=["_end_date_parsed"])
value_columns = [
column
for column in timeline.columns
if column not in {"ts_code", "_pit_date"}
]
right = timeline[["_pit_date", *value_columns]].rename(
columns={column: f"{table}_{column}" for column in value_columns}
)
agent/backtest/loaders/yahoo_client.py
+414
View File
@@ -0,0 +1,414 @@
"""Shared Yahoo Finance public-API client: chart, quote summary, options, search.
Yahoo Finance exposes several unauthenticated JSON endpoints (the same ones the
finance.yahoo.com site itself calls) that, like other free quote providers,
rate-limit by source IP and must be throttled. Every request here routes
through :mod:`backtest.loaders._http` so calls to ``query*.finance.yahoo.com``
share one process-wide minimum-spacing gate and a reused session.
The ``v10/finance/quoteSummary`` endpoint additionally requires a session
cookie plus a matching ``crumb`` token. This client fetches both lazily on the
first call that needs them and refreshes them once on a 401 (the documented
"crumb expired / unauthorized" signal), so callers never manage that handshake.
Symbol convention (Vibe-Trading -> Yahoo):
* US ``AAPL.US`` -> ``AAPL`` (Yahoo carries US tickers bare)
* HK ``00700.HK`` -> ``0700.HK`` (Yahoo drops the leading zero to 4 digits)
* India ``RELIANCE.NS`` / ``500325.BO`` -> unchanged (Yahoo carries the
``.NS``/``.BO`` suffix verbatim)
* Anything else is passed through unchanged (e.g. ``BTC-USD``, ``^GSPC``).
This module is provider-specific glue only; it returns plain Python
dicts/lists so a downstream loader can map them into the project's OHLCV frame.
"""
from __future__ import annotations
import logging
import threading
from typing import Any, Dict, List, Optional
import requests
from backtest.loaders._http import (
DEFAULT_USER_AGENT,
resolve_min_interval,
throttled_get,
throttled_get_json,
)
logger = logging.getLogger(__name__)
# All Yahoo endpoints share one throttle/session bucket so spacing is enforced
# across chart/quote/options/search regardless of which host alias is used.
HOST_KEY = "yahoo"
# Yahoo serves the same data from query1/query2; query2 is conventionally used
# for the crumb-gated endpoints. We keep the chart on query1 like the website.
_CHART_BASE = "https://query1.finance.yahoo.com/v8/finance/chart"
_QUOTE_SUMMARY_BASE = "https://query2.finance.yahoo.com/v10/finance/quoteSummary"
_OPTIONS_BASE = "https://query2.finance.yahoo.com/v7/finance/options"
_SEARCH_BASE = "https://query2.finance.yahoo.com/v1/finance/search"
_CRUMB_URL = "https://query2.finance.yahoo.com/v1/test/getcrumb"
# Hitting fc.yahoo.com first hands back the consent/session cookie the crumb
# endpoint then validates.
_COOKIE_URL = "https://fc.yahoo.com"
_MIN_INTERVAL_ENV = "VIBE_TRADING_YAHOO_MIN_INTERVAL"
_DEFAULT_MIN_INTERVAL_S = 0.6
# The columns Yahoo packs into indicators.quote[0], in our output field order.
_QUOTE_FIELDS = ("open", "high", "low", "close", "volume")
def _min_interval() -> float:
"""Resolve the per-call minimum spacing, honoring the env override."""
return resolve_min_interval(_MIN_INTERVAL_ENV, _DEFAULT_MIN_INTERVAL_S)
def map_symbol(symbol: str) -> str:
"""Translate a Vibe-Trading symbol into Yahoo's ticker convention.
Args:
symbol: Project-side symbol, e.g. ``AAPL.US``, ``00700.HK``, ``BTC-USD``.
Returns:
The Yahoo ticker: ``.US`` suffix stripped; ``.HK`` codes normalized to
a 4-digit base (``00700.HK`` -> ``0700.HK``); India ``.NS``/``.BO`` and
everything else unchanged (Yahoo carries those suffixes verbatim).
"""
cleaned = symbol.strip()
upper = cleaned.upper()
if upper.endswith(".US"):
return cleaned[: -len(".US")]
if upper.endswith(".HK"):
base = cleaned[: -len(".HK")]
digits = base.lstrip("0") or "0"
return f"{digits.zfill(4)}.HK"
return cleaned
class _CrumbStore:
"""Process-wide cookie jar + crumb token for the quoteSummary handshake.
Holds a single :class:`requests.Session`-independent crumb string and the
cookies returned by Yahoo's consent endpoint. Refreshing is serialized by a
lock so a burst of concurrent callers performs the handshake at most once.
"""
def __init__(self) -> None:
self._lock = threading.Lock()
self._crumb: Optional[str] = None
self._cookies: Dict[str, str] = {}
def get(self, *, force_refresh: bool = False) -> tuple[str, Dict[str, str]]:
"""Return ``(crumb, cookies)``, fetching them on first use or on refresh.
Args:
force_refresh: Discard any cached crumb/cookies and re-handshake.
Returns:
A tuple of the crumb token and the cookie mapping to send with the
crumb-gated request.
Raises:
requests.RequestException: If the cookie or crumb fetch fails.
ValueError: If Yahoo returns an empty crumb.
"""
with self._lock:
if force_refresh or self._crumb is None:
self._crumb, self._cookies = self._handshake()
return self._crumb, dict(self._cookies)
def _handshake(self) -> tuple[str, Dict[str, str]]:
cookie_resp = throttled_get(
_COOKIE_URL,
host_key=HOST_KEY,
min_interval=_min_interval(),
)
cookies = requests.utils.dict_from_cookiejar(cookie_resp.cookies)
crumb_resp = throttled_get(
_CRUMB_URL,
host_key=HOST_KEY,
min_interval=_min_interval(),
headers={"Cookie": _cookie_header(cookies)} if cookies else None,
)
crumb_resp.raise_for_status()
crumb = (crumb_resp.text or "").strip()
if not crumb:
raise ValueError("Yahoo returned an empty crumb token")
return crumb, cookies
def _cookie_header(cookies: Dict[str, str]) -> str:
"""Render a cookie mapping as a single ``Cookie:`` header value."""
return "; ".join(f"{name}={value}" for name, value in cookies.items())
# One shared crumb/cookie store for the whole process.
_CRUMB_STORE = _CrumbStore()
def get_chart(
symbol: str,
*,
interval: str = "1d",
period1: Optional[int] = None,
period2: Optional[int] = None,
range_: Optional[str] = None,
) -> List[Dict[str, Any]]:
"""Fetch OHLCV bars from the v8 chart endpoint as ascending rows.
Either supply a ``range_`` (e.g. ``"1y"``, ``"max"``) or a
``period1``/``period2`` epoch-second window; ``range_`` wins when both are
given. Yahoo returns parallel timestamp/indicator arrays which this function
zips into row dicts, dropping any bar whose OHLC is incomplete (Yahoo emits
``null`` for non-trading slots).
Args:
symbol: Project-side symbol (mapped via :func:`map_symbol`).
interval: Bar size accepted by Yahoo (``1d``, ``1h``, ``5m``, ...).
period1: Inclusive start as epoch seconds, or ``None``.
period2: Exclusive end as epoch seconds, or ``None``.
range_: Relative range string; takes precedence over period1/period2.
Returns:
Ascending list of ``{trade_date, open, high, low, close, volume}`` dicts
(``trade_date`` is the bar's epoch-second timestamp). Empty when Yahoo
reports no data for the symbol/window.
Raises:
requests.RequestException: On a network/HTTP failure.
ValueError: If Yahoo reports an error for the symbol or the payload is
structurally unusable.
"""
yahoo_symbol = map_symbol(symbol)
params: Dict[str, Any] = {"interval": interval}
if range_:
params["range"] = range_
else:
if period1 is not None:
params["period1"] = int(period1)
if period2 is not None:
params["period2"] = int(period2)
payload = throttled_get_json(
f"{_CHART_BASE}/{yahoo_symbol}",
host_key=HOST_KEY,
min_interval=_min_interval(),
params=params,
)
return _parse_chart(payload, yahoo_symbol)
def _parse_chart(payload: Any, yahoo_symbol: str) -> List[Dict[str, Any]]:
"""Convert a v8 chart payload into ascending OHLCV row dicts."""
chart = (payload or {}).get("chart") or {}
error = chart.get("error")
if error:
description = error.get("description") if isinstance(error, dict) else error
raise ValueError(f"Yahoo chart error for {yahoo_symbol}: {description}")
results = chart.get("result") or []
if not results:
return []
result = results[0] or {}
timestamps = result.get("timestamp") or []
quotes = (((result.get("indicators") or {}).get("quote")) or [{}])[0] or {}
rows: List[Dict[str, Any]] = []
for index, ts in enumerate(timestamps):
values = {field: _at(quotes.get(field), index) for field in _QUOTE_FIELDS}
# A non-trading slot leaves OHLC null; skip rather than emit a NaN bar.
if any(values[field] is None for field in ("open", "high", "low", "close")):
continue
row: Dict[str, Any] = {"trade_date": ts}
row.update({field: _to_float(values[field]) for field in _QUOTE_FIELDS})
rows.append(row)
return rows
def _at(series: Any, index: int) -> Any:
"""Return ``series[index]`` when in range, else ``None``."""
if isinstance(series, list) and 0 <= index < len(series):
return series[index]
return None
def _to_float(value: Any) -> Optional[float]:
"""Coerce a Yahoo numeric (or ``None``) to ``float``, ``None`` on failure."""
if value is None:
return None
try:
return float(value)
except (TypeError, ValueError):
return None
def get_quote_summary(symbol: str, modules: List[str]) -> Dict[str, Any]:
"""Fetch v10 quoteSummary modules, handling the cookie+crumb handshake.
On a 401 (expired/invalid crumb) the crumb and cookies are refreshed once
and the request is retried; a second 401 propagates.
Args:
symbol: Project-side symbol (mapped via :func:`map_symbol`).
modules: Yahoo module names, e.g. ``["price", "summaryDetail"]``.
Returns:
The ``quoteSummary.result[0]`` mapping, or ``{}`` when Yahoo returns no
result for the symbol.
Raises:
requests.RequestException: On a non-401 HTTP failure or a second 401.
ValueError: If Yahoo reports an error for the symbol.
"""
yahoo_symbol = map_symbol(symbol)
modules_param = ",".join(modules)
payload = _quote_summary_request(yahoo_symbol, modules_param, force_refresh=False)
return _parse_quote_summary(payload, yahoo_symbol)
def _quote_summary_request(
yahoo_symbol: str, modules_param: str, *, force_refresh: bool
) -> Any:
"""Issue one crumb-gated quoteSummary GET, retrying once on a 401."""
crumb, cookies = _CRUMB_STORE.get(force_refresh=force_refresh)
headers = {"Cookie": _cookie_header(cookies)} if cookies else None
response = throttled_get(
f"{_QUOTE_SUMMARY_BASE}/{yahoo_symbol}",
host_key=HOST_KEY,
min_interval=_min_interval(),
params={"modules": modules_param, "crumb": crumb},
headers=headers,
)
if response.status_code == 401 and not force_refresh:
logger.info("Yahoo quoteSummary 401 for %s; refreshing crumb", yahoo_symbol)
return _quote_summary_request(yahoo_symbol, modules_param, force_refresh=True)
response.raise_for_status()
return response.json()
def _parse_quote_summary(payload: Any, yahoo_symbol: str) -> Dict[str, Any]:
"""Extract the first quoteSummary result, raising on a reported error."""
summary = (payload or {}).get("quoteSummary") or {}
error = summary.get("error")
if error:
description = error.get("description") if isinstance(error, dict) else error
raise ValueError(f"Yahoo quoteSummary error for {yahoo_symbol}: {description}")
results = summary.get("result") or []
if not results:
return {}
return results[0] or {}
def get_options(symbol: str, *, expiration: Optional[int] = None) -> Dict[str, Any]:
"""Fetch the v7 option chain for a symbol, handling the cookie+crumb handshake.
Like ``v10/quoteSummary``, the ``v7/finance/options`` endpoint now rejects
bare requests with HTTP 401 and demands the same session cookie plus a
matching ``crumb`` token. This routes through the shared
:data:`_CRUMB_STORE` handshake and, on a 401 (expired/invalid crumb),
refreshes the crumb once and retries; a second 401 propagates.
Args:
symbol: Project-side symbol (mapped via :func:`map_symbol`).
expiration: Optional expiration as epoch seconds; omit for the nearest.
Returns:
The ``optionChain.result[0]`` mapping (expirationDates, strikes, the
calls/puts arrays for the chosen expiry), or ``{}`` when none.
Raises:
requests.RequestException: On a non-401 HTTP failure or a second 401.
ValueError: If Yahoo reports an error for the symbol.
"""
yahoo_symbol = map_symbol(symbol)
payload = _options_request(yahoo_symbol, expiration, force_refresh=False)
return _parse_options(payload, yahoo_symbol)
def _options_request(
yahoo_symbol: str, expiration: Optional[int], *, force_refresh: bool
) -> Any:
"""Issue one crumb-gated v7 options GET, retrying once on a 401."""
crumb, cookies = _CRUMB_STORE.get(force_refresh=force_refresh)
headers = {"Cookie": _cookie_header(cookies)} if cookies else None
params: Dict[str, Any] = {"crumb": crumb}
if expiration is not None:
params["date"] = int(expiration)
response = throttled_get(
f"{_OPTIONS_BASE}/{yahoo_symbol}",
host_key=HOST_KEY,
min_interval=_min_interval(),
params=params,
headers=headers,
)
if response.status_code == 401 and not force_refresh:
logger.info("Yahoo options 401 for %s; refreshing crumb", yahoo_symbol)
return _options_request(yahoo_symbol, expiration, force_refresh=True)
response.raise_for_status()
return response.json()
def _parse_options(payload: Any, yahoo_symbol: str) -> Dict[str, Any]:
"""Extract the first optionChain result, raising on a reported error."""
chain = (payload or {}).get("optionChain") or {}
error = chain.get("error")
if error:
description = error.get("description") if isinstance(error, dict) else error
raise ValueError(f"Yahoo options error for {yahoo_symbol}: {description}")
results = chain.get("result") or []
if not results:
return {}
return results[0] or {}
def search(query: str) -> List[Dict[str, Any]]:
"""Look up matching instruments via the v1 search endpoint.
Args:
query: Free-text query (ticker fragment or company name).
Returns:
The ``quotes`` list (each a dict with ``symbol``, ``shortname``,
``exchange``, ``quoteType``, ...), or an empty list when none match.
Raises:
requests.RequestException: On a network/HTTP failure.
"""
payload = throttled_get_json(
_SEARCH_BASE,
host_key=HOST_KEY,
min_interval=_min_interval(),
params={"q": query},
)
quotes = (payload or {}).get("quotes") or []
return [quote for quote in quotes if isinstance(quote, dict)]
def search_news(query: str, count: int) -> List[Dict[str, Any]]:
"""Look up matching news via the v1 search endpoint.
Args:
query: Free-text query (ticker fragment or company name).
count: Maximum number of news items requested from Yahoo.
Returns:
The ``news`` list (each a dict with ``title``, ``publisher``, ``link``,
``providerPublishTime``, ``relatedTickers``, ...), or an empty list when
none match.
Raises:
requests.RequestException: On a network/HTTP failure.
"""
payload = throttled_get_json(
_SEARCH_BASE,
host_key=HOST_KEY,
min_interval=_min_interval(),
params={"q": query, "newsCount": count},
)
news = (payload or {}).get("news") or []
return [item for item in news if isinstance(item, dict)]
agent/backtest/loaders/yahoo_loader.py
+264
View File
@@ -0,0 +1,264 @@
"""Yahoo Finance loader: free, no-auth US/HK equity OHLCV via direct HTTP.
Wraps the shared :mod:`backtest.loaders.yahoo_client` (the public v8 chart
endpoint) rather than the ``yfinance`` package, so it pulls in no new
dependency and shares the process-wide throttle/session that keeps Yahoo from
IP-rate-limiting us. Covers US equities (``AAPL.US``) and HK equities
(``00700.HK``); the client maps each project symbol to Yahoo's ticker form.
The chart endpoint returns each bar's ``trade_date`` as an epoch-second
timestamp; this loader converts those to a tz-naive ``DatetimeIndex`` and clips
the ascending series to the requested inclusive date window.
"""
from __future__ import annotations
import datetime as dt
import logging
from typing import Dict, List, Optional
import pandas as pd
from backtest.loaders import yahoo_client
from backtest.loaders.base import cached_loader_fetch, validate_date_range
from backtest.loaders.registry import register
logger = logging.getLogger(__name__)
_OHLCV_COLUMNS = ("open", "high", "low", "close", "volume")
# Project interval -> Yahoo chart interval. Daily is the only granularity this
# loader exposes for equities; anything else falls back to lowercasing so an
# explicit ``5m``/``1h`` still reaches Yahoo unchanged.
_INTERVAL_MAP = {
"1D": "1d",
"1H": "1h",
# Yahoo chart has no 4h bar; match yfinance's 4H → 1h approximation.
"4H": "1h",
"1W": "1wk",
"1M": "1mo",
}
def _is_supported(code: str) -> bool:
"""Return whether *code* is a symbol this loader handles.
Covers US/HK/India/Korea equities plus Yahoo's own futures (``GC=F``) and
forex (``EURUSD=X``) suffix conventions, which the public chart endpoint
serves verbatim (the code is used as-is in the request URL, no
conversion) (#718).
"""
upper = code.strip().upper()
return upper.endswith((".US", ".HK", ".NS", ".BO", ".KS", ".KQ", "=F", "=X"))
def _to_yahoo_interval(interval: str) -> str:
"""Map a project interval string to Yahoo's chart interval string.
Args:
interval: Backtest interval such as ``1D`` or ``1H``.
Returns:
Yahoo-compatible interval string (e.g. ``1d``).
"""
normalized = str(interval or "1D").strip()
if not normalized:
return "1d"
# Bare trailing ``m`` is minutes; do not uppercase into monthly ``1M`` → ``1mo``.
if normalized.endswith("m") and not normalized.endswith("M"):
return normalized.lower()
return _INTERVAL_MAP.get(normalized.upper(), normalized.lower())
def _is_intraday_interval(interval: str) -> bool:
"""Return whether *interval* is finer than one day (minute/hour bars).
Daily-and-coarser intervals (``1D``/``1W``/``1M``) carry no meaningful
intraday time and must be midnight-indexed to align with every other
loader's daily bars; minute/hour intervals keep their real timestamp.
Project convention follows the interval map: a bare trailing ``M`` is a
month (e.g. ``1M``) while a bare trailing lowercase ``m`` is a minute (e.g.
``5m``), so case is significant for that one suffix and is NOT folded.
Args:
interval: Backtest interval such as ``1D``, ``1H``, ``5m`` or ``30min``.
Returns:
``True`` for minute/hour granularities, ``False`` for day/week/month.
"""
raw = str(interval or "1D").strip()
if not raw:
return False
lowered = raw.lower()
# Multi-letter month forms ('mo'/'mth'/'mon') are daily-coarser.
if lowered.endswith(("mo", "mth", "mon")):
return False
# Explicit minute spellings and hours are intraday.
if lowered.endswith(("min", "h")):
return True
# Bare trailing letter: lowercase 'm' = minute (intraday), uppercase
# 'M' = month (coarser), 'd'/'w' = day/week (coarser).
return raw.endswith("m")
def _epoch_seconds(date_str: str) -> int:
"""Convert a ``YYYY-MM-DD`` date to UTC midnight epoch seconds.
Computed from the parsed calendar date (not ``time.time()``) so the window
is reproducible and independent of wall-clock fetch time.
Args:
date_str: Date string in ``YYYY-MM-DD`` form.
Returns:
Epoch seconds for that date's UTC midnight.
"""
day = pd.Timestamp(date_str).normalize().date()
moment = dt.datetime(day.year, day.month, day.day, tzinfo=dt.timezone.utc)
return int(moment.timestamp())
def _rows_to_frame(
rows: List[dict], start_date: str, end_date: str, interval: str = "1D"
) -> pd.DataFrame:
"""Build the OHLCV frame from chart rows, clipped to the inclusive window.
For daily-and-coarser intervals the DatetimeIndex is normalized to midnight
so the dates line up with every other loader's midnight-indexed daily bars
(Yahoo stamps daily bars at the US market-open epoch, e.g. 14:30 UTC, which
would otherwise break fallback merge/join/dedup). Minute/hour intervals keep
their real intraday timestamp.
Args:
rows: Ascending ``{trade_date(epoch s), open, high, low, close, volume}``
dicts from :func:`yahoo_client.get_chart`.
start_date: Inclusive window start (``YYYY-MM-DD``).
end_date: Inclusive window end (``YYYY-MM-DD``).
interval: Backtest interval such as ``1D`` or ``1H``; selects whether the
index is normalized to midnight.
Returns:
DataFrame indexed by a tz-naive ``trade_date`` with float OHLCV columns,
or an empty frame when no row carries usable OHLC.
"""
if not rows:
return pd.DataFrame()
frame = pd.DataFrame(rows)
index = pd.to_datetime(frame["trade_date"], unit="s", utc=True).dt.tz_convert(None)
if not _is_intraday_interval(interval):
index = index.dt.normalize()
frame = frame.drop(columns=["trade_date"])
frame.index = pd.DatetimeIndex(index)
frame.index.name = "trade_date"
for column in _OHLCV_COLUMNS:
if column not in frame.columns:
frame[column] = 0.0 if column == "volume" else pd.NA
frame = frame.loc[:, list(_OHLCV_COLUMNS)].apply(pd.to_numeric, errors="coerce")
frame["volume"] = frame["volume"].fillna(0.0)
frame = frame.dropna(subset=["open", "high", "low", "close"])
lower = pd.Timestamp(start_date).normalize()
upper = pd.Timestamp(end_date).normalize() + pd.Timedelta(days=1)
frame = frame[(frame.index >= lower) & (frame.index < upper)]
return frame.astype(float).sort_index()
@register
class DataLoader:
"""Yahoo Finance US/HK equity OHLCV loader (free, direct HTTP, no auth)."""
name = "yahoo"
markets = {"us_equity", "hk_equity", "india_equity", "kr_equity"}
requires_auth = False
def is_available(self) -> bool:
"""Always available — uses the throttled public HTTP client."""
return True
def __init__(self) -> None:
"""Initialize the loader (no credentials needed for public data)."""
pass
def fetch(
self,
codes: List[str],
start_date: str,
end_date: str,
*,
interval: str = "1D",
fields: Optional[List[str]] = None,
) -> Dict[str, pd.DataFrame]:
"""Fetch OHLCV history keyed by the original project symbols.
Args:
codes: Project symbols such as ``AAPL.US`` and ``00700.HK``.
start_date: Inclusive start date (``YYYY-MM-DD``).
end_date: Inclusive end date (``YYYY-MM-DD``).
interval: Backtest interval such as ``1D`` or ``1H``.
fields: Ignored; included for interface compatibility.
Returns:
Mapping of input symbol to a normalized OHLCV DataFrame. A symbol
that fails or returns no data is omitted; one failure never aborts
the batch.
"""
del fields
if not codes:
return {}
validate_date_range(start_date, end_date)
result: Dict[str, pd.DataFrame] = {}
for code in codes:
try:
df = cached_loader_fetch(
source=self.name,
symbol=code,
timeframe=interval,
start_date=start_date,
end_date=end_date,
fields=None,
fetch=lambda code=code: self._fetch_one(
code, start_date, end_date, interval
),
)
if df is not None and not df.empty:
result[code] = df
except Exception as exc:
logger.warning("yahoo failed for %s: %s", code, exc)
return result
def _fetch_one(
self, code: str, start_date: str, end_date: str, interval: str
) -> Optional[pd.DataFrame]:
"""Fetch and normalize one symbol's chart, or ``None`` when unusable.
Args:
code: A single project symbol.
start_date: Inclusive start date (``YYYY-MM-DD``).
end_date: Inclusive end date (``YYYY-MM-DD``).
interval: Backtest interval string.
Returns:
The OHLCV DataFrame for *code*, ``None`` if it is not a US/HK/India
symbol or Yahoo returns no usable bars.
"""
if not _is_supported(code):
return None
# period2 is exclusive on Yahoo; extend one day past end_date so the
# final inclusive bar is fetched, then clip precisely in _rows_to_frame.
period1 = _epoch_seconds(start_date)
period2 = _epoch_seconds(end_date) + 86400
rows = yahoo_client.get_chart(
code,
interval=_to_yahoo_interval(interval),
period1=period1,
period2=period2,
)
frame = _rows_to_frame(rows, start_date, end_date, interval)
return frame if not frame.empty else None
agent/backtest/loaders/yfinance_loader.py
+72 -10
View File
@@ -2,13 +2,21 @@
from __future__ import annotations
import logging
from collections import defaultdict
from typing import Dict, List, Optional, Union
import pandas as pd
import yfinance as yf
from backtest.loaders.base import validate_date_range
logger = logging.getLogger(__name__)
from backtest.loaders.base import (
loader_cache_get,
loader_cache_put,
validate_date_range,
validate_ohlc,
)
from backtest.loaders.registry import register
_OHLCV_COLUMNS = ["open", "high", "low", "close", "volume"]
@@ -28,6 +36,15 @@ _INTERVAL_MAP = {
"1D": "1d",
"1H": "1h",
"4H": "1h",
"4h": "1h", # yfinance has no 4h; match project ``4H`` → ``1h``
"1W": "1wk",
"1w": "1wk",
"1M": "1mo",
# Minute tokens stay lowercase; do not fold ``1M`` (month) via ``.lower()``.
"1m": "1m",
"5m": "5m",
"15m": "15m",
"30m": "30m",
}
@@ -47,6 +64,13 @@ def _to_yfinance_symbol(code: str) -> str:
digits = upper[:-3]
width = max(4, len(digits))
return f"{digits.zfill(width)}.HK"
# Crypto: BTC-USDT -> BTC-USD, ETH-USDT -> ETH-USD, etc.
if upper.endswith("-USDT"):
return upper[:-5] + "-USD"
if upper.endswith("-USDC"):
return upper[:-5] + "-USD"
# India NSE/BSE (RELIANCE.NS, 500325.BO) and Korea KRX (005930.KS,
# 247540.KQ): yfinance carries these suffixes as-is.
return upper
@@ -63,6 +87,11 @@ def _to_yfinance_interval(interval: str) -> str:
return _INTERVAL_MAP.get(normalized, normalized.lower())
def _to_yfinance_exclusive_end(end_date: str) -> str:
"""Convert the project-inclusive end date to yfinance's exclusive end."""
return (pd.Timestamp(end_date).normalize() + pd.Timedelta(days=1)).strftime("%Y-%m-%d")
def _download_history(
tickers: Union[List[str], str],
start_date: str,
@@ -172,6 +201,7 @@ def _normalize_frame(frame: pd.DataFrame, requested_interval: str) -> pd.DataFra
normalized = normalized.sort_index()
normalized["volume"] = normalized["volume"].fillna(0.0)
normalized = normalized.dropna(subset=["open", "high", "low", "close"])
normalized = validate_ohlc(normalized)
if requested_interval == "4H" and not normalized.empty:
normalized = normalized.resample("4h").agg(
@@ -194,7 +224,7 @@ class DataLoader:
"""Fetch HK/US equity bars from Yahoo Finance via yfinance."""
name = "yfinance"
markets = {"us_equity", "hk_equity"}
markets = {"us_equity", "hk_equity", "india_equity", "kr_equity", "crypto"}
requires_auth = False
def is_available(self) -> bool:
@@ -213,8 +243,9 @@ class DataLoader:
codes: List[str],
start_date: str,
end_date: str,
fields: Optional[List[str]] = None,
*,
interval: str = "1D",
fields: Optional[List[str]] = None,
) -> Dict[str, pd.DataFrame]:
"""Fetch OHLCV history keyed by the original project symbols.
@@ -235,6 +266,7 @@ class DataLoader:
requested_interval = str(interval or "1D").strip()
yf_interval = _to_yfinance_interval(requested_interval)
yf_end_date = _to_yfinance_exclusive_end(end_date)
symbol_groups: Dict[str, List[str]] = defaultdict(list)
for code in codes:
@@ -243,27 +275,57 @@ class DataLoader:
unique_symbols = list(symbol_groups.keys())
results: Dict[str, pd.DataFrame] = {}
# Serve cached symbols first so a fully-cached request skips the bulk
# download entirely; only uncached symbols hit the network.
pending: List[str] = []
for symbol in unique_symbols:
cached = loader_cache_get(
source=self.name,
symbol=symbol,
timeframe=requested_interval,
start_date=start_date,
end_date=end_date,
fields=None,
)
if cached is not None:
for original_code in symbol_groups[symbol]:
results[original_code] = cached.copy()
else:
pending.append(symbol)
if not pending:
return results
try:
bulk_data = _download_history(unique_symbols, start_date, end_date, yf_interval)
bulk_data = _download_history(pending, start_date, yf_end_date, yf_interval)
except Exception as exc:
print(f"[WARN] yfinance bulk download failed for {unique_symbols}: {exc}")
logger.warning("yfinance bulk download failed for %s: %s", pending, exc)
bulk_data = pd.DataFrame()
for symbol in unique_symbols:
for symbol in pending:
try:
symbol_frame = _extract_symbol_frame(bulk_data, symbol, len(unique_symbols))
symbol_frame = _extract_symbol_frame(bulk_data, symbol, len(pending))
if symbol_frame.empty:
symbol_frame = _download_history(symbol, start_date, end_date, yf_interval)
symbol_frame = _download_history(symbol, start_date, yf_end_date, yf_interval)
normalized = _normalize_frame(symbol_frame, requested_interval)
if normalized.empty:
print(f"[WARN] yfinance returned no usable data for {symbol}")
logger.warning("yfinance returned no usable data for %s", symbol)
continue
loader_cache_put(
source=self.name,
symbol=symbol,
timeframe=requested_interval,
start_date=start_date,
end_date=end_date,
fields=None,
frame=normalized,
)
for original_code in symbol_groups[symbol]:
results[original_code] = normalized.copy()
except Exception as exc:
print(f"[WARN] Failed to fetch data for {symbol}: {exc}")
logger.warning("Failed to fetch data for %s: %s", symbol, exc)
continue
return results
agent/backtest/metrics.py
+390 -23
View File
@@ -5,6 +5,7 @@ Provides annualisation helpers, trade statistics, and full metric calculation.
from __future__ import annotations
import logging
from typing import Any, Dict, List, Optional
import numpy as np
@@ -14,33 +15,251 @@ from backtest.models import TradeRecord
# ─── Annualisation factor mapping ───
_TRADING_DAYS = {"tushare": 252, "yfinance": 252, "okx": 365, "akshare": 252, "ccxt": 365}
_BARS_PER_DAY = {
"1m": {"tushare": 240, "okx": 1440, "yfinance": 390, "akshare": 240, "ccxt": 1440},
"5m": {"tushare": 48, "okx": 288, "yfinance": 78, "akshare": 48, "ccxt": 288},
"15m": {"tushare": 16, "okx": 96, "yfinance": 26, "akshare": 16, "ccxt": 96},
"30m": {"tushare": 8, "okx": 48, "yfinance": 13, "akshare": 8, "ccxt": 48},
"1H": {"tushare": 4, "okx": 24, "yfinance": 7, "akshare": 4, "ccxt": 24},
"4H": {"tushare": 1, "okx": 6, "yfinance": 2, "akshare": 1, "ccxt": 6},
"1D": {"tushare": 1, "okx": 1, "yfinance": 1, "akshare": 1, "ccxt": 1},
# mootdx (A-share) and futu (HK + A-share) are equity sources, so they mirror
# the tushare/akshare column: 252 trading days and a 240-minute session. HK
# sessions are marginally longer (~330 min) — an approximation in line with the
# rest of this annualisation table; the key fix is that intraday mootdx/futu no
# longer fall back to the bars_per_day=1 default, which mis-annualised vol/Sharpe.
_TRADING_DAYS = {
# existing
"tushare": 252, "yfinance": 252, "okx": 365, "akshare": 252, "ccxt": 365,
"mootdx": 252, "futu": 252, "mt5": 260,
# crypto
"binance": 365,
# A-share equity
"baostock": 252, "tencent": 252, "eastmoney": 252, "sina": 252,
# US / international equity
"yahoo": 252, "finnhub": 252, "alphavantage": 252, "tiingo": 252,
"fmp": 252, "stooq": 252, "longbridge": 252,
# resampling sources (local files / paid data — interval depends on source
# data granularity; model as US equity session as a conservative default)
"local": 252, "qveris": 252,
# Indian equity
"india_broker": 252,
# Korean equity (KRX)
"pykrx": 252,
}
# mt5 is a forex/CFD feed: 24x5 sessions → 260 trading days, 24h intraday bars.
# US equity (yfinance-style): 6.5h sessions → 390 1m bars/day.
# A-share equity (tushare-style): 4.0h sessions → 240 1m bars/day.
# Crypto (okx/ccxt-style): 24h sessions → 1440 1m bars/day.
# Indian equity: 6.25h sessions → 375 1m bars/day.
# Korean equity (pykrx): 6.5h sessions → 390 1m bars/day. The loader itself
# serves daily bars only, so the intraday rows exist to keep the table complete
# (and correct if KRX intraday ever arrives under this key), not because pykrx
# can return them.
_BARS_PER_DAY = {
# --- US/international equity (6.5h session) ---
"1m": {"yfinance": 390, "yahoo": 390, "finnhub": 390, "alphavantage": 390,
"tiingo": 390, "fmp": 390, "stooq": 390, "longbridge": 390,
"local": 390, "qveris": 390,
# A-share equity (4.0h session)
"tushare": 240, "akshare": 240, "baostock": 240, "tencent": 240,
"eastmoney": 240, "sina": 240, "mootdx": 240, "futu": 240,
# crypto (24h)
"okx": 1440, "ccxt": 1440, "binance": 1440,
# forex/CFD (24h intraday)
"mt5": 1440,
# Indian equity (6.25h session)
"india_broker": 375,
# Korean equity (6.5h session, 09:00-15:30 KST)
"pykrx": 390,
},
"5m": {"yfinance": 78, "yahoo": 78, "finnhub": 78, "alphavantage": 78,
"tiingo": 78, "fmp": 78, "stooq": 78, "longbridge": 78,
"local": 78, "qveris": 78,
"tushare": 48, "akshare": 48, "baostock": 48, "tencent": 48,
"eastmoney": 48, "sina": 48, "mootdx": 48, "futu": 48,
"okx": 288, "ccxt": 288, "binance": 288,
"mt5": 288,
"india_broker": 75,
"pykrx": 78,
},
"15m": {"yfinance": 26, "yahoo": 26, "finnhub": 26, "alphavantage": 26,
"tiingo": 26, "fmp": 26, "stooq": 26, "longbridge": 26,
"local": 26, "qveris": 26,
"tushare": 16, "akshare": 16, "baostock": 16, "tencent": 16,
"eastmoney": 16, "sina": 16, "mootdx": 16, "futu": 16,
"okx": 96, "ccxt": 96, "binance": 96,
"mt5": 96,
"india_broker": 25,
"pykrx": 26,
},
"30m": {"yfinance": 13, "yahoo": 13, "finnhub": 13, "alphavantage": 13,
"tiingo": 13, "fmp": 13, "stooq": 13, "longbridge": 13,
"local": 13, "qveris": 13,
"tushare": 8, "akshare": 8, "baostock": 8, "tencent": 8,
"eastmoney": 8, "sina": 8, "mootdx": 8, "futu": 8,
"okx": 48, "ccxt": 48, "binance": 48,
"mt5": 48,
"india_broker": 13,
"pykrx": 13,
},
"1H": {"yfinance": 7, "yahoo": 7, "finnhub": 7, "alphavantage": 7,
"tiingo": 7, "fmp": 7, "stooq": 7, "longbridge": 7,
"local": 7, "qveris": 7,
"tushare": 4, "akshare": 4, "baostock": 4, "tencent": 4,
"eastmoney": 4, "sina": 4, "mootdx": 4, "futu": 4,
"okx": 24, "ccxt": 24, "binance": 24,
"mt5": 24,
"india_broker": 7,
"pykrx": 7,
},
"4H": {"yfinance": 2, "yahoo": 2, "finnhub": 2, "alphavantage": 2,
"tiingo": 2, "fmp": 2, "stooq": 2, "longbridge": 2,
"local": 2, "qveris": 2,
"tushare": 1, "akshare": 1, "baostock": 1, "tencent": 1,
"eastmoney": 1, "sina": 1, "mootdx": 1, "futu": 1,
"okx": 6, "ccxt": 6, "binance": 6,
"mt5": 6,
"india_broker": 2,
"pykrx": 2,
},
"1D": {"yfinance": 1, "yahoo": 1, "finnhub": 1, "alphavantage": 1,
"tiingo": 1, "fmp": 1, "stooq": 1, "longbridge": 1,
"local": 1, "qveris": 1,
"tushare": 1, "akshare": 1, "baostock": 1, "tencent": 1,
"eastmoney": 1, "sina": 1, "mootdx": 1, "futu": 1,
"okx": 1, "ccxt": 1, "binance": 1,
"mt5": 1,
"india_broker": 1,
"pykrx": 1,
},
}
# Runner/loaders also emit these aliases; map them onto the table keys above.
_SOURCE_ALIASES = {"yahoo": "yfinance", "binance": "ccxt"}
def _normalize_interval(interval: str) -> str:
"""Map project interval tokens onto ``_BARS_PER_DAY`` keys.
Minute bars stay lowercase (``1m``); hour/day use the uppercase keys the
table already stores (``1H`` / ``4H`` / ``1D``). Loaders accept both cases
after the interval-map fixes; annualisation must too.
"""
token = str(interval or "1D").strip()
lower = token.lower()
if lower in ("1m", "5m", "15m", "30m"):
return lower
if lower in ("1h", "4h", "1d"):
return lower.upper()
return token
def calc_bars_per_year(interval: str = "1D", source: str = "tushare") -> int:
"""Number of bars per year for annualisation.
Args:
interval: Bar size (1m / 5m / 15m / 30m / 1H / 4H / 1D).
source: Data source (tushare / yfinance / okx).
interval: Bar size (1m / 5m / 15m / 30m / 1H / 4H / 1D), case-insensitive
like loaders accept (``1h`` → ``1H``, ``4h`` → ``4H``, ``1d`` → ``1D``).
source: Data source (any VALID_SOURCES entry). Defaults to 252 days, 1 bar/day
when source is missing from the table.
Returns:
Bars per year.
"""
trading_days = _TRADING_DAYS.get(source, 252)
bars_per_day = _BARS_PER_DAY.get(interval, {}).get(source, 1)
interval_key = _normalize_interval(interval)
source_key = str(source or "").strip().lower()
source_key = _SOURCE_ALIASES.get(source_key, source_key)
trading_days = _TRADING_DAYS.get(source_key, 252)
bars_per_day = _BARS_PER_DAY.get(interval_key, {}).get(source_key, 1)
return trading_days * bars_per_day
# ─── Sign-safe returns ───
_log = logging.getLogger(__name__)
def bar_returns(close: Any, *, label: str = "") -> Any:
"""Per-bar simple returns, defined only where the prior price is positive.
``close.pct_change()`` silently assumes ``price[t-1] > 0``. That holds for
equities but not for instruments that can print zero or negative prices,
such as European day-ahead power. As the divisor approaches zero an
ordinary absolute move reads as an enormous *percentage* move, so a
compounded aggregate explodes; at exactly zero the next bar is ``inf``,
which ``fillna`` does not neutralise (it fills ``NaN``) and which collapses
``(1 + r).prod()`` to ``nan`` (#872).
A return is therefore defined only when the previous price is finite and
strictly positive. Otherwise it is undefined and reported as ``0.0``,
never ``inf`` or ``nan``. For an all-positive series this is identical to
``pct_change().fillna(0.0)``, so ordinary equity/crypto runs are unchanged.
The prior price is carried forward across gaps. ``pct_change`` did this
implicitly via ``fill_method='pad'`` — its default on pandas < 3, removed
in 3.0 — so *not* forwarding it would silently change every gapped series
(a halt longer than ``_align``'s ``ffill_limit``, a thinly-traded symbol)
and would make the bit-identity claim above false on the pinned pandas.
Doing it here explicitly keeps that promise and, unlike the old default,
gives the same answer on every supported pandas version. Note the effect
is to attribute the whole across-gap move to the resumed bar; that is what
a position held through the halt actually earned. Whether that is the right
convention for *statistics* (it is not, for correlation — see
``correlation.py``, which drops the observation instead) is a separate
question from whether this function may change it as a side effect.
Args:
close: Raw price ``Series`` or per-symbol ``DataFrame``.
label: Optional name used when warning about undefined bars.
Returns:
Returns aligned to ``close``, with the first bar ``0.0``.
"""
# ffill *then* shift: the divisor is the last known price, not NaN.
prev = close.ffill().shift(1)
# ``inf > 0`` is True, so finiteness has to be asserted, not assumed.
usable_prev = np.isfinite(prev) & (prev > 0)
positive_prev = prev.where(usable_prev)
undefined = int((prev.notna() & ~usable_prev).to_numpy().sum())
if undefined:
# Silence is the actual hazard here: a wrong return is a wrong reported
# number, not a crash, so the run has to say it defaulted.
_log.warning(
"%s: %d bar(s) follow a non-positive or non-finite prior price; "
"their return is undefined and reported as 0.0 (issue #872)",
label or "returns",
undefined,
)
# ``close / prev - 1`` is exactly the expression ``pct_change`` uses, kept
# verbatim so results for positive series are bit-identical, not merely
# equal to within floating-point error.
ret = close / positive_prev - 1
return ret.replace([np.inf, -np.inf], np.nan).fillna(0.0)
def buy_and_hold_return(close: Any) -> Optional[float]:
"""Total buy-and-hold return as a price relative, not a compounded product.
``(1 + close.pct_change()).prod() - 1`` telescopes to ``P_end / P_start - 1``
only while every price is positive; once a near-zero price enters the
series the product diverges from what a held position actually earned. On
the reproduction in #872 it reported ``+39,560%`` where the price relative
gives ``-42.7%``. This computes the price relative directly, so the two
agree exactly for ordinary series and it stays honest for the rest.
Args:
close: Raw price series, already ``dropna()``-ed.
Returns:
Total return, or ``None`` when the entry price is not strictly
positive and no honest percentage exists.
"""
if len(close) < 2:
return None
first = float(close.iloc[0])
last = float(close.iloc[-1])
# ``inf > 0`` is True, so an infinite entry price would otherwise yield a
# clean-looking -100% instead of "no honest percentage exists".
if not (np.isfinite(first) and first > 0) or not np.isfinite(last):
return None
return last / first - 1.0
def win_rate_and_stats(trades: List[TradeRecord]) -> Dict[str, float]:
"""Win rate and P&L statistics from completed trades.
@@ -143,12 +362,82 @@ def by_exit_reason_stats(trades: List[TradeRecord]) -> Dict[str, Dict[str, Any]]
return result
def calc_turnover_series(positions: pd.DataFrame) -> pd.Series:
"""Per-bar weight-implied portfolio turnover from a position frame.
Turnover for a bar is ``0.5 * sum_i |w_{t,i} - w_{t-1,i}|``, so a full
rotation from one asset to another counts as 1.0 (matching the
``turnover_aware`` optimizer's convention). The first bar's turnover is
``0.5 * sum_i |w_{0,i}|``, treating the initial allocation as entry from
cash. Turnover is measured on the weight frame the caller supplies. It
does not know whether the execution engine filled, rounded, or rejected
those target positions.
Args:
positions: Position-weight matrix (index=timestamp, columns=codes).
Returns:
Per-bar turnover series indexed like ``positions``; empty when the
input is empty.
"""
if positions is None or positions.empty:
return pd.Series(dtype=float)
filled = positions.fillna(0.0)
prev = filled.shift(1).fillna(0.0)
return 0.5 * (filled - prev).abs().sum(axis=1)
def calc_trade_turnover_series(
trades: List[TradeRecord],
equity_curve: pd.Series,
) -> pd.Series:
"""Per-bar turnover from actual entry and exit allocations.
Each filled leg contributes its margin-equivalent traded value. Dividing
gross traded value by twice the portfolio equity preserves the existing
convention: entering a 100% allocation counts as 0.5 and rotating a 100%
allocation between two assets counts as 1.0.
Args:
trades: Completed trades carrying actual entry/exit margin values.
equity_curve: Portfolio equity used to normalize traded values.
Returns:
Per-bar realized turnover aligned to ``equity_curve``. Bars without
fills are zero and remain part of the average-turnover denominator.
"""
if equity_curve is None or equity_curve.empty:
return pd.Series(dtype=float)
traded_margin = pd.Series(0.0, index=equity_curve.index, dtype=float)
for trade in trades:
for timestamp, margin in (
(trade.entry_time, trade.entry_margin),
(trade.exit_time, trade.exit_margin),
):
try:
margin_value = float(margin)
except (TypeError, ValueError):
continue
if (
timestamp in traded_margin.index
and np.isfinite(margin_value)
and margin_value > 0
):
traded_margin.loc[timestamp] += margin_value
denominator = 2.0 * equity_curve.abs().replace(0.0, np.nan)
return (traded_margin / denominator).replace([np.inf, -np.inf], np.nan).fillna(0.0)
def calc_metrics(
equity_curve: pd.Series,
trades: List[TradeRecord],
initial_cash: float,
bars_per_year: Optional[int] = 252,
bench_ret: Optional[pd.Series] = None,
positions: Optional[pd.DataFrame] = None,
turnover_series: Optional[pd.Series] = None,
) -> Dict[str, Any]:
"""Full set of performance metrics.
@@ -159,6 +448,10 @@ def calc_metrics(
bars_per_year: Bars per year for annualisation. None = auto-detect
from equity curve dates (calendar-day method, for cross-market).
bench_ret: Benchmark per-bar return series (optional).
positions: Position-weight frame used as a backward-compatible
turnover fallback when ``turnover_series`` is not supplied.
turnover_series: Actual per-bar execution turnover (optional). When
supplied, it takes precedence over position-implied turnover.
Returns:
Metrics dictionary (compatible with daily_portfolio format).
@@ -171,18 +464,46 @@ def calc_metrics(
# Calendar-day annualization for cross-market (bars_per_year=None)
if bars_per_year is None:
first, last = equity_curve.index[0], equity_curve.index[-1]
calendar_days = (last - first).days
diff = last - first
calendar_days = diff.days if hasattr(diff, "days") else 0
years = calendar_days / 365.25 if calendar_days > 0 else 1.0
bpy = int(n / years) if years > 0 else 252
else:
bpy = bars_per_year
port_ret = equity_curve.pct_change().fillna(0.0)
# Equity that touches zero then recovers (100 → 0 → 50) yields non-finite
# pct_change values; options metrics already skip risk ratios in that case.
returns_finite = bool(np.isfinite(port_ret.to_numpy(dtype=float, copy=False)).all())
total_ret = float(equity_curve.iloc[-1] / initial_cash - 1)
ann_ret = float((1 + total_ret) ** (bpy / max(n, 1)) - 1)
vol = float(port_ret.std())
sharpe = float(port_ret.mean() / (vol + 1e-10) * np.sqrt(bpy))
# A leveraged/short book can end at or below zero equity (``total_ret <= -1``).
# ``(1 + total_ret) ** fractional`` would then raise a negative base to a
# fractional power, which Python evaluates to a ``complex`` and crashes the
# subsequent ``float(...)``. A total wipeout annualises to -100%.
growth = 1 + total_ret
if growth <= 0:
ann_ret = -1.0
else:
# Explosive equity paths (e.g. 1 → 1e6 in a few bars) overflow
# ``float(growth ** …)`` on CPython; treat as non-finite annualisation.
try:
ann_ret = float(growth ** (bpy / max(n, 1)) - 1)
except OverflowError:
ann_ret = float("inf")
if not np.isfinite(ann_ret):
ann_ret = float("inf")
# ``Series.std()`` uses ddof=1, so a single-observation return series
# (e.g. a one-bar backtest) yields NaN and poisons the Sharpe ratio.
# Guard the small sample the same way ``downside_std`` is guarded below.
vol = float(port_ret.std()) if len(port_ret) > 1 and returns_finite else 0.0
sharpe = (
float(port_ret.mean() / (vol + 1e-10) * np.sqrt(bpy))
if returns_finite
else 0.0
)
if not np.isfinite(sharpe):
sharpe = 0.0
# Drawdown
peak = equity_curve.cummax()
@@ -192,22 +513,62 @@ def calc_metrics(
calmar = ann_ret / abs(max_dd) if abs(max_dd) > 1e-10 else 0.0
# Sortino
downside = port_ret[port_ret < 0]
downside_std = float(downside.std()) if len(downside) > 1 else 1e-10
sortino = float(port_ret.mean() / (downside_std + 1e-10) * np.sqrt(bpy))
if returns_finite:
downside = port_ret[port_ret < 0]
downside_std = float(downside.std()) if len(downside) > 1 else 1e-10
sortino = float(port_ret.mean() / (downside_std + 1e-10) * np.sqrt(bpy))
else:
sortino = 0.0
if not np.isfinite(sortino):
sortino = 0.0
trade_stats = win_rate_and_stats(trades)
# Prefer execution-derived turnover; retain the position-frame fallback
# for external callers of calc_metrics that do not have fill records.
turnover_values = (
turnover_series.reindex(equity_curve.index).fillna(0.0).clip(lower=0.0)
if turnover_series is not None
else calc_turnover_series(positions)
if positions is not None
else pd.Series(dtype=float)
)
avg_turnover = float(turnover_values.mean()) if len(turnover_values) > 0 else 0.0
total_turnover = float(turnover_values.sum()) if len(turnover_values) > 0 else 0.0
# Benchmark comparison
bench_return = 0.0
excess = 0.0
ir = 0.0
tracking_error = 0.0
bench_beta = 0.0
if bench_ret is not None and len(bench_ret) > 0:
bench_return = float((1 + bench_ret).prod() - 1)
excess = total_ret - bench_return
active_ret = port_ret - bench_ret.reindex(port_ret.index).fillna(0.0)
active_std = float(active_ret.std())
ir = float(active_ret.mean() / (active_std + 1e-10) * np.sqrt(bpy))
aligned_bench = bench_ret.reindex(port_ret.index).fillna(0.0)
active_ret = port_ret - aligned_bench
# Same ddof=1 small-sample guard as ``vol`` / ``downside_std`` so the
# information ratio stays finite for a single-observation series.
active_std = float(active_ret.std()) if len(active_ret) > 1 and returns_finite else 0.0
ir = (
float(active_ret.mean() / (active_std + 1e-10) * np.sqrt(bpy))
if returns_finite
else 0.0
)
if not np.isfinite(ir):
ir = 0.0
# The information ratio's own denominator, annualised. A
# benchmark-relative mandate is written around this number, and it was
# being computed and thrown away.
tracking_error = active_std * np.sqrt(bpy) if returns_finite else 0.0
if not np.isfinite(tracking_error):
tracking_error = 0.0
bench_var = float(aligned_bench.var()) if len(aligned_bench) > 1 else 0.0
if returns_finite and bench_var > 0:
covariance = float(port_ret.cov(aligned_bench))
bench_beta = covariance / bench_var
if not np.isfinite(bench_beta):
bench_beta = 0.0
return {
"final_value": float(equity_curve.iloc[-1]),
@@ -226,6 +587,10 @@ def calc_metrics(
"benchmark_return": round(bench_return, 6),
"excess_return": round(excess, 6),
"information_ratio": round(ir, 4),
"tracking_error": round(float(tracking_error), 6),
"benchmark_beta": round(float(bench_beta), 4),
"avg_turnover": round(avg_turnover, 6),
"total_turnover": round(total_turnover, 6),
}
@@ -238,4 +603,6 @@ def _empty_metrics(initial_cash: float) -> Dict[str, Any]:
"win_rate": 0, "profit_loss_ratio": 0, "profit_factor": 0,
"max_consecutive_loss": 0, "avg_holding_days": 0, "trade_count": 0,
"benchmark_return": 0, "excess_return": 0, "information_ratio": 0,
"tracking_error": 0.0, "benchmark_beta": 0.0,
"avg_turnover": 0.0, "total_turnover": 0.0,
}
agent/backtest/models.py
+4
View File
@@ -53,6 +53,8 @@ class TradeRecord:
exit_reason: Why closed (signal / liquidation / end_of_backtest).
holding_bars: Number of bars held.
commission: Total commission (entry + exit).
entry_margin: Actual margin allocated at entry, after size rounding.
exit_margin: Margin-equivalent value traded at the exit price.
"""
symbol: str
@@ -68,6 +70,8 @@ class TradeRecord:
exit_reason: str
holding_bars: int
commission: float
entry_margin: float = 0.0
exit_margin: float = 0.0
@dataclass(frozen=True)
agent/backtest/optimizers/__init__.py
+2 -1
View File
@@ -1,10 +1,11 @@
"""Portfolio optimizer package.
Provides four weighting schemes:
Provides five weighting schemes:
- equal_volatility: inverse-volatility weights
- risk_parity: equal risk contribution (Spinu-style)
- mean_variance: max Sharpe via scipy
- max_diversification: maximize diversification ratio
- turnover_aware: mean-variance utility with an L1 turnover penalty
Select via ``optimizer`` in ``config.json``; default is off (1/N).
Add a new optimizer by dropping a module here that exposes ``optimize()``.
agent/backtest/optimizers/base.py
+10 -3
View File
@@ -16,7 +16,7 @@ class BaseOptimizer(ABC):
Subclasses implement ``_calc_weights``; the base handles:
- active asset selection
- rolling window slicing and sanity checks
- causal rolling window slicing and sanity checks
- covariance matrix + NaN checks
- applying weights while preserving signal sign
@@ -42,7 +42,9 @@ class BaseOptimizer(ABC):
"""Apply optimizer to position weights.
Args:
ret: Return matrix (dates x codes).
ret: Return matrix (dates x codes). For a decision at ``dt``,
only rows strictly earlier than ``dt`` are visible to the
optimizer because execution occurs at the decision bar's open.
pos: Raw signal positions.
dates: Date index aligned with ``pos``.
@@ -59,7 +61,12 @@ class BaseOptimizer(ABC):
if not active or i < self.lookback:
continue
window = ret.loc[:dt, active].tail(self.lookback)
# Signals are executed at the decision bar's open. ``ret[dt]``
# is a close-to-close return that is not observable until that
# bar closes, so including it here would leak future information
# into the weights applied at the open.
history = ret.loc[ret.index < dt, active]
window = history.tail(self.lookback)
if len(window) < max(self.lookback // 2, 5):
continue
agent/backtest/optimizers/risk_parity.py
+25 -17
View File
@@ -1,7 +1,4 @@
"""Risk parity: equalize marginal risk contributions.
Iterative refinement so w_i * MRC_i is approximately equal across assets.
"""
"""Long-only risk parity: equalize marginal risk contributions."""
from typing import Any, Dict
@@ -12,33 +9,44 @@ from backtest.optimizers.base import BaseOptimizer
class RiskParityOptimizer(BaseOptimizer):
"""Spinu (2013)-style inverse-vol seed + Newton-style refinement."""
"""Equal-risk-contribution weights on the long-only simplex."""
def _calc_weights(self, ctx: Dict[str, Any]) -> np.ndarray:
"""Equal risk contribution weights."""
from scipy.optimize import minimize
cov = ctx["cov"]
n = cov.shape[0]
if n == 0:
return self._equal_weight(0)
vols = np.sqrt(np.diag(cov))
if np.any(vols < 1e-12):
if not np.isfinite(cov).all() or np.any(vols < 1e-12):
return self._equal_weight(n)
inv_vol = 1.0 / vols
w = inv_vol / inv_vol.sum()
seed = inv_vol / inv_vol.sum()
for _ in range(5):
port_vol = np.sqrt(w @ cov @ w)
if port_vol < 1e-12:
break
mrc = (cov @ w) / port_vol
rc = w * mrc
target = port_vol / n
w = w * (target / (rc + 1e-12))
w = w / w.sum()
def contribution_error(w: np.ndarray) -> float:
variance = float(w @ cov @ w)
if not np.isfinite(variance) or variance <= 1e-18:
return 1e12
contributions = w * (cov @ w)
target = variance / n
return float(np.sum((contributions - target) ** 2) / variance**2)
return w
result = minimize(
contribution_error,
seed,
method="SLSQP",
bounds=[(0.0, 1.0)] * n,
constraints={"type": "eq", "fun": lambda w: w.sum() - 1.0},
options={"maxiter": 200, "ftol": 1e-12},
)
if result.success and np.isfinite(result.x).all():
return self._normalize(result.x)
return self._normalize(seed)
def optimize(
agent/backtest/optimizers/turnover_aware.py
+250
View File
@@ -0,0 +1,250 @@
"""Turnover-aware optimizer: mean-variance utility with an L1 turnover penalty.
Solves, per rebalance date::
min -w'mu + lambda * w'Sigma w + gamma * ||w - w_prev||_1
s.t. w >= 0, sum(w) = 1
w_i <= max_per_name (per-name cap)
sum_{i in group_g} w_i <= max_per_group[g] (per-group cap)
where ``w_prev`` is the weight vector applied at the previous rebalance,
restricted to the current active set (assets absent last time have prior
weight 0, so entries and exits both count as turnover). With ``gamma == 0``
the objective reduces to the mean-variance utility baseline.
The penalty ``gamma`` is scale-sensitive: it is measured against the return
term ``w'mu``, so an appropriate magnitude depends on the return units of the
input window. For daily returns (~1e-3), even ``gamma`` around 0.5 makes the
optimizer strongly prefer holding still. Callers should tune ``gamma`` relative
to their data frequency.
Realized per-rebalance turnover (``0.5 * ||w_t - w_{t-1}||_1``) is accumulated
on the instance for cost-adjusted analysis. This is a class-API affordance:
the engine's module-level ``optimize`` entry constructs a fresh instance and
returns only the positions frame, so callers who want the turnover series must
instantiate ``TurnoverAwareOptimizer`` directly.
"""
from __future__ import annotations
from typing import Any, Dict, List, Optional
import numpy as np
import pandas as pd
from backtest.optimizers.base import BaseOptimizer
class TurnoverAwareOptimizer(BaseOptimizer):
"""Mean-variance weights penalized for turnover against prior weights.
Attributes:
risk_aversion: Weight on the variance term (lambda).
turnover_penalty: Weight on the L1 turnover term (gamma). 0 reduces to
the mean-variance baseline.
max_per_name: Per-asset weight cap (None = no limit).
groups: Asset-code → group-name mapping for per-group caps.
max_per_group: Group-name → maximum total weight for that group.
realized_turnover: Per-rebalance realized turnover collected during
``optimize`` (``0.5 * ||w_t - w_{t-1}||_1``).
"""
def __init__(
self,
lookback: int = 60,
risk_aversion: float = 1.0,
turnover_penalty: float = 0.0,
max_per_name: Optional[float] = None,
groups: Optional[Dict[str, str]] = None,
max_per_group: Optional[Dict[str, float]] = None,
**kwargs: Any,
) -> None:
super().__init__(lookback=lookback, **kwargs)
self.risk_aversion = float(risk_aversion)
self.turnover_penalty = float(turnover_penalty)
if isinstance(max_per_name, (bool, np.bool_)):
raise ValueError("max_per_name must be numeric, not boolean")
self.max_per_name = float(max_per_name) if max_per_name is not None else None
self.groups: Dict[str, str] = dict(groups) if groups else {}
self.max_per_group: Dict[str, float] = dict(max_per_group) if max_per_group else {}
if self.max_per_name is not None and (
not np.isfinite(self.max_per_name) or not 0.0 < self.max_per_name <= 1.0
):
raise ValueError("max_per_name must be finite and in (0, 1]")
if any(not isinstance(code, str) or not isinstance(group, str) for code, group in self.groups.items()):
raise ValueError("groups must map string asset codes to string group names")
unknown_groups = set(self.max_per_group) - set(self.groups.values())
if unknown_groups:
raise ValueError(
"max_per_group references groups with no mapped assets: "
+ ", ".join(sorted(unknown_groups))
)
for group, cap in self.max_per_group.items():
if isinstance(cap, (bool, np.bool_)):
raise ValueError(f"cap for group {group!r} must be numeric, not boolean")
try:
numeric_cap = float(cap)
except (TypeError, ValueError) as exc:
raise ValueError(f"cap for group {group!r} must be numeric") from exc
if not np.isfinite(numeric_cap) or not 0.0 < numeric_cap <= 1.0:
raise ValueError(f"cap for group {group!r} must be finite and in (0, 1]")
self.max_per_group[group] = numeric_cap
self._prev: Dict[str, float] = {}
self.realized_turnover: List[float] = []
def _build_context(
self, window: pd.DataFrame, active: List[str]
) -> "Dict[str, Any] | None":
"""Mean vector, covariance, and active codes for the current window."""
mu = window.mean().values
cov = window.cov().values
if np.isnan(cov).any() or np.isnan(mu).any():
return None
return {"cov": cov, "mu": mu, "active": list(active)}
def _calc_weights(self, ctx: Dict[str, Any]) -> np.ndarray:
"""SLSQP weights for the penalized objective; updates turnover state."""
from scipy.optimize import minimize
mu = np.asarray(ctx["mu"], dtype=float)
cov = np.asarray(ctx["cov"], dtype=float)
active: List[str] = ctx["active"]
n = len(mu)
if n == 0:
return self._equal_weight(0)
w_prev = np.array([self._prev.get(code, 0.0) for code in active], dtype=float)
lam = self.risk_aversion
gamma = self.turnover_penalty
def objective(w: np.ndarray) -> float:
ret = w @ mu
var = w @ cov @ w
turn = np.abs(w - w_prev).sum()
return -ret + lam * var + gamma * turn
# --- bounds: per-name cap ---
upper = 1.0
if self.max_per_name is not None:
upper = min(1.0, float(self.max_per_name))
bounds = [(0.0, upper)] * n
# --- constraints: simplex + per-group caps ---
constraints: list = [
{"type": "eq", "fun": lambda w: w.sum() - 1.0},
]
group_rows: list[np.ndarray] = []
group_caps: list[float] = []
group_constraint_indices: list[List[int]] = []
group_indices: Dict[str, List[int]] = {}
if self.groups and self.max_per_group:
for i, code in enumerate(active):
g = self.groups.get(code)
if g is not None:
group_indices.setdefault(g, []).append(i)
for g, cap in self.max_per_group.items():
indices = group_indices.get(g, [])
if not indices:
continue
cap = float(cap)
row = np.zeros(n)
row[indices] = 1.0
group_rows.append(row)
group_caps.append(cap)
group_constraint_indices.append(indices)
constraints.append({
"type": "ineq",
"fun": lambda w, idx=indices, c=cap: c - w[np.array(idx)].sum(),
})
has_effective_caps = upper < 1.0 or any(cap < 1.0 for cap in group_caps)
if not has_effective_caps:
x0 = w_prev if w_prev.sum() > 1e-12 else self._equal_weight(n)
elif (
np.isfinite(w_prev).all()
and (w_prev >= 0.0).all()
and abs(w_prev.sum() - 1.0) <= 1e-12
and (w_prev <= upper + 1e-12).all()
and all(
row @ w_prev <= cap + 1e-12
for row, cap in zip(group_rows, group_caps)
)
):
x0 = w_prev
else:
# Groups are a single label per asset, so their caps are disjoint.
# Allocate each bucket by capacity to obtain a feasible simplex point.
buckets: list[tuple[List[int], float]] = []
capped_indices: set[int] = set()
for indices, cap in zip(group_constraint_indices, group_caps):
capacity = min(cap, len(indices) * upper)
buckets.append((indices, capacity))
capped_indices.update(indices)
uncapped = [i for i in range(n) if i not in capped_indices]
if uncapped:
buckets.append((uncapped, len(uncapped) * upper))
total_capacity = sum(capacity for _, capacity in buckets)
if total_capacity < 1.0 - 1e-12:
raise ValueError(
"exposure caps are infeasible for active assets "
f"{active}: total capacity is {total_capacity:.6g}"
)
x0 = np.zeros(n)
for indices, capacity in buckets:
x0[indices] = capacity / total_capacity / len(indices)
result = minimize(
objective,
x0,
method="SLSQP",
bounds=bounds,
constraints=constraints,
options={"maxiter": 200, "ftol": 1e-10},
)
if not result.success:
raise RuntimeError(f"turnover-aware optimization failed: {result.message}")
weights = np.maximum(np.asarray(result.x, dtype=float), 0.0)
weights /= weights.sum()
if (
not np.isfinite(weights).all()
or abs(weights.sum() - 1.0) > 1e-7
or weights.max() > upper + 1e-7
or any(row @ weights > cap + 1e-7 for row, cap in zip(group_rows, group_caps))
):
raise RuntimeError("optimizer returned weights that violate exposure caps")
self._record_turnover(active, weights)
return weights
def _record_turnover(self, active: List[str], weights: np.ndarray) -> None:
"""Accumulate realized turnover and roll prior weights forward."""
codes = set(active) | set(self._prev)
new_map = {code: float(weights[i]) for i, code in enumerate(active)}
turnover = 0.5 * sum(
abs(new_map.get(code, 0.0) - self._prev.get(code, 0.0)) for code in codes
)
self.realized_turnover.append(turnover)
self._prev = new_map
def optimize(
ret: pd.DataFrame,
pos: pd.DataFrame,
dates: pd.DatetimeIndex,
lookback: int = 60,
risk_aversion: float = 1.0,
turnover_penalty: float = 0.0,
max_per_name: Optional[float] = None,
groups: Optional[Dict[str, str]] = None,
max_per_group: Optional[Dict[str, float]] = None,
) -> pd.DataFrame:
"""Module-level entry: turnover-penalized mean-variance positions."""
return TurnoverAwareOptimizer(
lookback=lookback,
risk_aversion=risk_aversion,
turnover_penalty=turnover_penalty,
max_per_name=max_per_name,
groups=groups,
max_per_group=max_per_group,
).optimize(ret, pos, dates)
agent/backtest/options_payoff.py
+496
View File
@@ -0,0 +1,496 @@
"""Deterministic payoff and scenario math for multi-leg option strategies.
The expiry summary is solved from the strategy's piecewise-linear payoff,
independently of the caller's display grid. This keeps breakevens and finite
extrema correct even when a chart grid omits one or more strikes.
"""
from __future__ import annotations
from dataclasses import dataclass
import numpy as np
from backtest.engines.options_portfolio import bs_price
_BREAKEVEN_TOLERANCE = 1e-9
_DEFAULT_GRID_POINTS = 2001
@dataclass(frozen=True)
class OptionLeg:
"""One European option leg in a payoff calculation.
Attributes:
option_type: ``"call"`` or ``"put"``.
strike: Positive strike price.
qty: Signed contract quantity; positive is long and negative is short.
premium: Optional per-share entry premium. When omitted, Black-Scholes
pricing is used at the supplied entry conditions.
"""
option_type: str
strike: float
qty: int
premium: float | None = None
@dataclass(frozen=True)
class PayoffReport:
"""Expiry payoff curve and analytic strategy summary.
Attributes:
spot_grid: Caller-supplied display spots.
payoff: Expiry P&L at each display spot.
net_premium: Signed gross entry premium; positive is a debit.
entry_commission: Entry commission paid in account currency.
entry_cost: Gross premium plus entry commission.
breakevens: Isolated non-negative expiry spots where P&L is zero.
breakeven_intervals: Continuous zero-P&L spot intervals. A ``None``
upper bound represents a flat zero-payoff right tail.
max_profit: Analytic maximum, or positive infinity when unbounded.
max_loss: Analytic minimum P&L, or negative infinity when unbounded.
profit_unbounded: Whether right-tail profit is unbounded.
loss_unbounded: Whether right-tail loss is unbounded.
"""
spot_grid: np.ndarray
payoff: np.ndarray
net_premium: float
entry_commission: float
entry_cost: float
breakevens: list[float]
breakeven_intervals: list[tuple[float, float | None]]
max_profit: float
max_loss: float
profit_unbounded: bool
loss_unbounded: bool
def _intrinsic(option_type: str, spots: np.ndarray, strike: float) -> np.ndarray:
"""Return vectorized intrinsic values for one option leg."""
if option_type == "call":
return np.maximum(spots - strike, 0.0)
return np.maximum(strike - spots, 0.0)
def _validate_legs(legs: list[OptionLeg]) -> None:
"""Validate option-leg fields at the calculation boundary."""
if not legs:
raise ValueError("at least one leg is required")
for index, leg in enumerate(legs):
if leg.option_type not in ("call", "put"):
raise ValueError(f"legs[{index}].option_type must be 'call' or 'put'")
if not np.isfinite(leg.strike) or leg.strike <= 0:
raise ValueError(f"legs[{index}].strike must be positive and finite")
if isinstance(leg.qty, bool) or not isinstance(leg.qty, (int, np.integer)):
raise ValueError(f"legs[{index}].qty must be a non-zero integer")
if leg.qty == 0:
raise ValueError(f"legs[{index}].qty must be non-zero")
if leg.premium is not None and (not np.isfinite(leg.premium) or leg.premium < 0):
raise ValueError(f"legs[{index}].premium must be non-negative and finite")
def _validate_market_inputs(
*,
entry_spot: float,
time_to_expiry: float,
rate: float,
iv: float,
multiplier: float,
commission_rate: float,
) -> None:
"""Validate common market and cost inputs."""
values = {
"entry_spot": entry_spot,
"time_to_expiry": time_to_expiry,
"rate": rate,
"iv": iv,
"multiplier": multiplier,
"commission_rate": commission_rate,
}
for name, value in values.items():
if not np.isfinite(value):
raise ValueError(f"{name} must be finite")
if entry_spot <= 0:
raise ValueError("entry_spot must be positive")
if time_to_expiry < 0:
raise ValueError("time_to_expiry must be non-negative")
if iv <= 0:
raise ValueError("iv must be positive")
if multiplier <= 0:
raise ValueError("multiplier must be positive")
if not 0 <= commission_rate < 1:
raise ValueError("commission_rate must be in [0, 1)")
def _validate_grid(values: np.ndarray, *, name: str, allow_zero: bool) -> np.ndarray:
"""Return a validated one-dimensional numeric grid."""
grid = np.asarray(values, dtype=float)
if grid.ndim != 1 or grid.size == 0:
raise ValueError(f"{name} must be a non-empty one-dimensional array")
if not np.isfinite(grid).all():
raise ValueError(f"{name} must contain only finite values")
if allow_zero:
invalid = grid < 0
else:
invalid = grid <= 0
if invalid.any():
qualifier = "non-negative" if allow_zero else "positive"
raise ValueError(f"{name} values must be {qualifier}")
return grid
def _leg_premiums(
legs: list[OptionLeg],
entry_spot: float,
time_to_expiry: float,
rate: float,
iv: float,
) -> np.ndarray:
"""Resolve explicit or Black-Scholes entry premiums for every leg."""
premiums = []
for leg in legs:
premium = (
leg.premium
if leg.premium is not None
else bs_price(
entry_spot,
leg.strike,
time_to_expiry,
rate,
iv,
leg.option_type,
)
)
premiums.append(float(premium))
return np.asarray(premiums, dtype=float)
def _entry_costs(
legs: list[OptionLeg],
premiums: np.ndarray,
multiplier: float,
commission_rate: float,
) -> tuple[float, float, float]:
"""Return gross premium, commission, and total signed entry cost."""
net_premium = float(sum(leg.qty * premium * multiplier for leg, premium in zip(legs, premiums, strict=True)))
entry_commission = float(
sum(abs(leg.qty) * premium * multiplier * commission_rate for leg, premium in zip(legs, premiums, strict=True))
)
return net_premium, entry_commission, net_premium + entry_commission
def _payoff_at_spots(
legs: list[OptionLeg],
spots: np.ndarray,
*,
multiplier: float,
entry_cost: float,
) -> np.ndarray:
"""Calculate expiry P&L at arbitrary non-negative spots."""
intrinsic = np.zeros(len(spots), dtype=float)
for leg in legs:
intrinsic += leg.qty * _intrinsic(leg.option_type, spots, leg.strike)
return intrinsic * multiplier - entry_cost
def _critical_points(legs: list[OptionLeg]) -> np.ndarray:
"""Return every finite point at which expiry-payoff slope may change."""
return np.asarray(sorted({0.0, *(float(leg.strike) for leg in legs)}))
def _analytic_breakevens(
critical_spots: np.ndarray,
critical_payoff: np.ndarray,
right_slope: float,
) -> tuple[list[float], list[tuple[float, float | None]]]:
"""Solve isolated roots and continuous zero intervals analytically."""
roots: list[float] = []
intervals: list[tuple[float, float | None]] = []
for index, (spot, value) in enumerate(zip(critical_spots, critical_payoff, strict=True)):
if index == len(critical_spots) - 1:
continue
next_spot = float(critical_spots[index + 1])
next_value = float(critical_payoff[index + 1])
value = float(value)
if abs(value) <= _BREAKEVEN_TOLERANCE and abs(next_value) <= _BREAKEVEN_TOLERANCE:
intervals.append((float(spot), next_spot))
elif value * next_value < 0:
fraction = value / (value - next_value)
roots.append(float(spot + fraction * (next_spot - spot)))
right_spot = float(critical_spots[-1])
right_value = float(critical_payoff[-1])
if abs(right_slope) <= _BREAKEVEN_TOLERANCE and abs(right_value) <= _BREAKEVEN_TOLERANCE:
intervals.append((right_spot, None))
elif abs(right_slope) > _BREAKEVEN_TOLERANCE:
right_root = right_spot - right_value / right_slope
if right_root > right_spot + _BREAKEVEN_TOLERANCE:
roots.append(float(right_root))
intervals = _merge_zero_intervals(intervals)
for spot, value in zip(critical_spots, critical_payoff, strict=True):
spot = float(spot)
if abs(float(value)) <= _BREAKEVEN_TOLERANCE and not _in_zero_interval(spot, intervals):
roots.append(spot)
roots.sort()
deduped: list[float] = []
for root in roots:
if not deduped or abs(root - deduped[-1]) > _BREAKEVEN_TOLERANCE:
deduped.append(root)
return deduped, intervals
def _merge_zero_intervals(
intervals: list[tuple[float, float | None]],
) -> list[tuple[float, float | None]]:
"""Merge adjacent continuous zero-payoff intervals."""
merged: list[tuple[float, float | None]] = []
for start, end in intervals:
if not merged:
merged.append((start, end))
continue
previous_start, previous_end = merged[-1]
if previous_end is not None and abs(previous_end - start) <= _BREAKEVEN_TOLERANCE:
merged[-1] = (previous_start, end)
else:
merged.append((start, end))
return merged
def _in_zero_interval(spot: float, intervals: list[tuple[float, float | None]]) -> bool:
"""Return whether a spot belongs to a continuous zero-payoff interval."""
for start, end in intervals:
if spot < start - _BREAKEVEN_TOLERANCE:
continue
if end is None or spot <= end + _BREAKEVEN_TOLERANCE:
return True
return False
def expiry_payoff(
legs: list[OptionLeg],
spot_grid: np.ndarray,
*,
entry_spot: float,
time_to_expiry: float,
rate: float = 0.05,
iv: float = 0.3,
multiplier: float = 1.0,
commission_rate: float = 0.001,
) -> PayoffReport:
"""Calculate an expiry payoff curve and analytic risk summary.
Premiums are pinned at entry. An explicit per-share premium wins over the
Black-Scholes price. Entry commission follows the existing options engine:
long legs pay ``premium * (1 + commission_rate)`` and short legs receive
``premium * (1 - commission_rate)``. Expiry settlement has no exit fee.
Args:
legs: Signed option legs in the strategy.
spot_grid: Non-negative display spots. Summary values do not depend on
this grid containing strikes or breakevens.
entry_spot: Underlying spot at entry.
time_to_expiry: Years until expiry at entry.
rate: Annual continuously compounded risk-free rate.
iv: Annualized volatility used for legs without explicit premiums.
multiplier: Currency multiplier per option price unit.
commission_rate: Entry commission as a fraction of gross premium.
Returns:
Payoff report with the display curve and analytic extrema/breakevens.
Raises:
ValueError: If a leg, grid, market input, or cost input is invalid.
"""
_validate_legs(legs)
_validate_market_inputs(
entry_spot=entry_spot,
time_to_expiry=time_to_expiry,
rate=rate,
iv=iv,
multiplier=multiplier,
commission_rate=commission_rate,
)
display_spots = _validate_grid(spot_grid, name="spot_grid", allow_zero=True)
premiums = _leg_premiums(legs, entry_spot, time_to_expiry, rate, iv)
net_premium, entry_commission, entry_cost = _entry_costs(legs, premiums, multiplier, commission_rate)
payoff = _payoff_at_spots(
legs,
display_spots,
multiplier=multiplier,
entry_cost=entry_cost,
)
critical_spots = _critical_points(legs)
critical_payoff = _payoff_at_spots(
legs,
critical_spots,
multiplier=multiplier,
entry_cost=entry_cost,
)
right_slope = float(sum(leg.qty for leg in legs if leg.option_type == "call") * multiplier)
profit_unbounded = right_slope > _BREAKEVEN_TOLERANCE
loss_unbounded = right_slope < -_BREAKEVEN_TOLERANCE
max_profit = float("inf") if profit_unbounded else float(np.max(critical_payoff))
max_loss = float("-inf") if loss_unbounded else float(np.min(critical_payoff))
breakevens, breakeven_intervals = _analytic_breakevens(critical_spots, critical_payoff, right_slope)
return PayoffReport(
spot_grid=display_spots,
payoff=payoff,
net_premium=net_premium,
entry_commission=entry_commission,
entry_cost=entry_cost,
breakevens=breakevens,
breakeven_intervals=breakeven_intervals,
max_profit=max_profit,
max_loss=max_loss,
profit_unbounded=profit_unbounded,
loss_unbounded=loss_unbounded,
)
def default_spot_grid(
center: float,
half_width_pct: float = 0.5,
points: int = _DEFAULT_GRID_POINTS,
) -> np.ndarray:
"""Build a symmetric non-negative display grid around an entry spot.
Args:
center: Positive central spot.
half_width_pct: Fraction of ``center`` added to and subtracted from the
bounds.
points: Number of points, at least two.
Returns:
Increasing NumPy spot grid.
Raises:
ValueError: If a parameter is non-finite or outside its valid range.
"""
if not np.isfinite(center) or center <= 0:
raise ValueError("center must be positive and finite")
if not np.isfinite(half_width_pct) or half_width_pct <= 0:
raise ValueError("half_width_pct must be positive and finite")
if isinstance(points, bool) or not isinstance(points, (int, np.integer)):
raise ValueError("points must be an integer")
if points < 2:
raise ValueError("points must be at least 2")
lower = max(center * (1.0 - half_width_pct), 0.0)
upper = center * (1.0 + half_width_pct)
return np.linspace(lower, upper, int(points))
def scenario_grid(
legs: list[OptionLeg],
spot_grid: np.ndarray,
iv_values: np.ndarray,
*,
entry_spot: float,
time_to_expiry: float,
rate: float = 0.05,
entry_iv: float = 0.3,
multiplier: float = 1.0,
commission_rate: float = 0.001,
) -> np.ndarray:
"""Calculate pre-expiry mark-to-market P&L over spot and volatility.
The entry cost is pinned once using the entry spot/IV and includes entry
commission. Scenario values are marks, so no hypothetical closing
commission is deducted.
Args:
legs: Signed option legs in the strategy.
spot_grid: Non-negative scenario spots.
iv_values: Positive annualized volatility scenarios.
entry_spot: Underlying spot at entry.
time_to_expiry: Years remaining in every scenario.
rate: Annual continuously compounded risk-free rate.
entry_iv: Annualized volatility used to price the entry.
multiplier: Currency multiplier per option price unit.
commission_rate: Entry commission as a fraction of gross premium.
Returns:
Matrix shaped ``(len(iv_values), len(spot_grid))``.
Raises:
ValueError: If a leg, grid, market input, or cost input is invalid.
"""
_validate_legs(legs)
_validate_market_inputs(
entry_spot=entry_spot,
time_to_expiry=time_to_expiry,
rate=rate,
iv=entry_iv,
multiplier=multiplier,
commission_rate=commission_rate,
)
spots = _validate_grid(spot_grid, name="spot_grid", allow_zero=True)
ivs = _validate_grid(iv_values, name="iv_values", allow_zero=False)
entry_premiums = _leg_premiums(legs, entry_spot, time_to_expiry, rate, entry_iv)
_, _, entry_cost = _entry_costs(legs, entry_premiums, multiplier, commission_rate)
grid = np.zeros((len(ivs), len(spots)), dtype=float)
for iv_row, iv_now in enumerate(ivs):
for spot_col, spot_now in enumerate(spots):
marked_value = 0.0
for leg in legs:
marked_value += (
leg.qty
* bs_price(
float(spot_now),
leg.strike,
time_to_expiry,
rate,
float(iv_now),
leg.option_type,
)
* multiplier
)
grid[iv_row, spot_col] = marked_value - entry_cost
return grid
def bull_call_spread(lower_strike: float, upper_strike: float, qty: int = 1) -> list[OptionLeg]:
"""Build a long lower-strike/short upper-strike call spread."""
if upper_strike <= lower_strike:
raise ValueError("upper strike must sit above lower strike")
if isinstance(qty, bool) or not isinstance(qty, (int, np.integer)) or qty <= 0:
raise ValueError("qty must be positive")
return [
OptionLeg("call", lower_strike, qty),
OptionLeg("call", upper_strike, -qty),
]
def long_straddle(strike: float, qty: int = 1) -> list[OptionLeg]:
"""Build a long call plus long put at the same strike."""
if isinstance(qty, bool) or not isinstance(qty, (int, np.integer)) or qty <= 0:
raise ValueError("qty must be positive")
return [OptionLeg("call", strike, qty), OptionLeg("put", strike, qty)]
def iron_condor(
put_wing: float,
put_body: float,
call_body: float,
call_wing: float,
qty: int = 1,
) -> list[OptionLeg]:
"""Build a defined-risk short iron condor."""
if not (put_wing < put_body < call_body < call_wing):
raise ValueError("strikes must nest as put_wing < put_body < call_body < call_wing")
if isinstance(qty, bool) or not isinstance(qty, (int, np.integer)) or qty <= 0:
raise ValueError("qty must be positive")
return [
OptionLeg("put", put_wing, qty),
OptionLeg("put", put_body, -qty),
OptionLeg("call", call_body, -qty),
OptionLeg("call", call_wing, qty),
]
agent/backtest/perpetual_evidence.py
+84
View File
@@ -0,0 +1,84 @@
"""Strict USD-M audit evidence, never read back into accounting decisions."""
from __future__ import annotations
import json
from pathlib import Path
from typing import Any, Mapping, Sequence
SCHEMA_VERSION = "1"
RESOLUTION_LIMITATION = (
"At 100x, an interval of at most 1 hour is only a resolution boundary; "
"exact liquidation sequence precision is not guaranteed."
)
def build_perpetual_summary(
*,
config: Mapping[str, Any],
events: Sequence[Mapping[str, Any]],
trades: Sequence[Any],
terminal_status: str,
maintenance_bracket_versions: Mapping[str, str],
market_risk_sources: Sequence[str],
) -> dict[str, Any]:
"""Build a strict-JSON-safe summary from append-only audit evidence."""
funding = [event for event in events if event["event_type"] == "funding_settlement"]
liquidation_types = {"position_liquidation", "account_liquidation"}
liquidations = [event for event in events if event["event_type"] in liquidation_types]
flags = sorted({flag for event in events for flag in event.get("fidelity_flags", [])})
liquidated_positions = sum(len(event.get("symbols", [event.get("symbol")])) for event in liquidations)
return {
"schema_version": SCHEMA_VERSION,
"margin_mode": str(config.get("margin_mode", "isolated")),
"funding_mode": str(config.get("funding_mode", "fixed")),
"interval": str(config.get("interval", "1D")),
"leverage": float(config.get("leverage", 1.0)),
"taker_rate": float(config.get("taker_rate", 0.0005)),
"liquidation_fee_rate": float(config.get("liquidation_fee_rate", 0.0)),
"terminal_status": terminal_status,
"event_count": len(events),
"funding_settlement_count": len(funding),
"total_funding_pnl": sum(float(event["funding_pnl"]) for event in funding),
"liquidation_event_count": len(liquidations),
"liquidated_position_count": liquidated_positions,
"total_trading_fee": sum(float(trade.commission) for trade in trades),
"total_liquidation_fee": sum(float(event["liquidation_fee"]) for event in liquidations),
"fee_model": {
"market_fill_rate": "taker_rate",
"maker_rate_used": False,
"funding_separate": True,
"liquidation_separate": True,
},
"maintenance_bracket_versions": dict(sorted(maintenance_bracket_versions.items())),
"market_risk_sources": sorted(set(market_risk_sources)),
"fidelity_flags": flags,
"resolution_limitation": RESOLUTION_LIMITATION,
"assumptions": [
"single_asset_usdt",
"no_open_order_margin",
"one_way_positions",
"bar_extrema_order_unknown",
"not_an_identical_binance_liquidation_engine_reproduction",
],
}
def write_perpetual_evidence(
run_dir: Path,
events: Sequence[Mapping[str, Any]],
summary: Mapping[str, Any],
) -> None:
"""Write ordered lifecycle JSONL and its aggregate summary."""
out = Path(run_dir) / "artifacts"
out.mkdir(parents=True, exist_ok=True)
event_lines = [json.dumps(event, sort_keys=True, allow_nan=False) for event in events]
(out / "perpetual_events.jsonl").write_text(
"\n".join(event_lines) + ("\n" if event_lines else ""),
encoding="utf-8",
)
(out / "perpetual_summary.json").write_text(
json.dumps(summary, indent=2, sort_keys=True, allow_nan=False) + "\n",
encoding="utf-8",
)
agent/backtest/perpetual_risk.py
+409
View File
@@ -0,0 +1,409 @@
"""Pure state and risk contracts for historical USD-M perpetual backtests.
The model is deliberately account-local and deterministic. It does not read
live exchange state, model open orders, or store mutable mark prices on
positions.
Maintenance-bracket data comes from the loader's already-validated artifact
contract (``backtest.loaders.ccxt_loader._validate_bracket_artifact``): a
symbol, a list of tiers, and a single content-hash version. This module does
not re-fetch, re-derive, or re-hash brackets — it only re-validates their
structural invariants (ordering, non-negativity) as a defense-in-depth check
on whatever the caller passes in.
"""
from __future__ import annotations
from dataclasses import dataclass
import json
import math
from typing import Literal
import pandas as pd
MarginMode = Literal["isolated", "cross"]
TerminalStatus = Literal["active", "completed", "account_liquidation"]
def _require_finite(name: str, value: float, *, positive: bool = False) -> None:
if not math.isfinite(value) or (positive and value <= 0):
qualifier = "positive and finite" if positive else "finite"
raise ValueError(f"{name} must be {qualifier}")
def _timestamp(value: pd.Timestamp, name: str) -> pd.Timestamp:
timestamp = pd.Timestamp(value)
if pd.isna(timestamp):
raise ValueError(f"{name} must be a valid timestamp")
return timestamp
@dataclass(frozen=True)
class MaintenanceBracket:
"""One symbol-specific maintenance-margin bracket.
Field names and semantics match the loader's bracket-artifact tier
records exactly (``bracket_tier``, ``notional_cap``, ``maintenance_rate``,
``cumulative_maintenance_amount``, optional ``notional_coefficient``).
"""
bracket_tier: int
notional_cap: float
maintenance_rate: float
cumulative_maintenance_amount: float
notional_coefficient: float | None = None
def __post_init__(self) -> None:
if self.bracket_tier < 0:
raise ValueError("bracket_tier must be non-negative")
_require_finite("notional_cap", self.notional_cap, positive=True)
_require_finite("maintenance_rate", self.maintenance_rate)
_require_finite(
"cumulative_maintenance_amount", self.cumulative_maintenance_amount
)
if self.maintenance_rate < 0:
raise ValueError("maintenance_rate must be non-negative")
if self.cumulative_maintenance_amount < 0:
raise ValueError("cumulative_maintenance_amount must be non-negative")
if self.notional_coefficient is not None:
_require_finite("notional_coefficient", self.notional_coefficient)
@dataclass(frozen=True)
class MaintenanceSchedule:
"""A validated, versioned set of maintenance brackets for one symbol.
``version`` is an opaque identifier supplied by the caller — in practice
the loader's ``maintenance_bracket_version`` content-hash column. This
class does not compute its own hash: the artifact contract
(``_validate_bracket_artifact``) is the single source of truth for
content integrity, checked once, before this object is ever built.
"""
symbol: str
version: str
brackets: tuple[MaintenanceBracket, ...]
def __post_init__(self) -> None:
if not self.symbol:
raise ValueError("symbol must not be empty")
if not self.version:
raise ValueError("version must not be empty")
if not self.brackets:
raise ValueError("brackets must not be empty")
tiers = [bracket.bracket_tier for bracket in self.brackets]
if tiers != sorted(tiers) or len(tiers) != len(set(tiers)):
raise ValueError("bracket_tier values must be strictly increasing")
caps = [bracket.notional_cap for bracket in self.brackets]
if any(current <= previous for previous, current in zip(caps, caps[1:])):
raise ValueError("notional caps must be strictly increasing")
@classmethod
def from_loader_columns(
cls, symbol: str, maintenance_brackets: str, maintenance_bracket_version: str,
) -> MaintenanceSchedule:
"""Build a schedule from the loader's ``maintenance_brackets`` /
``maintenance_bracket_version`` DataFrame columns.
The JSON payload is already-validated tier records (from
``_validate_bracket_artifact``) — this only parses and re-checks
structural invariants, it never re-derives the version hash.
"""
try:
records = json.loads(maintenance_brackets)
except (TypeError, json.JSONDecodeError) as exc:
raise ValueError("maintenance_brackets is not valid JSON") from exc
if not isinstance(records, list) or not records:
raise ValueError("maintenance_brackets must be a non-empty list")
brackets = tuple(
MaintenanceBracket(
bracket_tier=record["bracket_tier"],
notional_cap=record["notional_cap"],
maintenance_rate=record["maintenance_rate"],
cumulative_maintenance_amount=record["cumulative_maintenance_amount"],
notional_coefficient=record.get("notional_coefficient"),
)
for record in records
)
return cls(symbol=symbol, version=maintenance_bracket_version, brackets=brackets)
@dataclass(frozen=True)
class PositionState:
"""Exchange-independent open-position accounting state."""
symbol: str
quantity: float
entry_price: float
leverage: float
accumulated_entry_fee: float
isolated_margin: float | None
def __post_init__(self) -> None:
if not self.symbol:
raise ValueError("symbol must not be empty")
_require_finite("quantity", self.quantity)
if self.quantity == 0:
raise ValueError("quantity must be non-zero")
_require_finite("entry_price", self.entry_price, positive=True)
_require_finite("leverage", self.leverage, positive=True)
_require_finite("accumulated_entry_fee", self.accumulated_entry_fee)
if self.accumulated_entry_fee < 0:
raise ValueError("accumulated_entry_fee must be non-negative")
if self.isolated_margin is not None:
_require_finite("isolated_margin", self.isolated_margin, positive=True)
@dataclass(frozen=True)
class PositionRisk:
symbol: str
mark_price: float
notional: float
unrealized_pnl: float
initial_margin: float
maintenance_margin: float
margin_balance: float | None
@dataclass(frozen=True)
class RiskSnapshot:
margin_balance: float
initial_margin: float
maintenance_margin: float
available_balance: float
per_position: tuple[PositionRisk, ...]
status: Literal["healthy", "position_liquidation", "account_liquidation"]
liquidation_targets: tuple[str, ...]
fidelity_flags: tuple[str, ...]
def maintenance_margin(
position: PositionState, mark_price: float, schedule: MaintenanceSchedule
) -> float:
if position.symbol != schedule.symbol:
raise ValueError("position and schedule symbols must match")
_require_finite("mark_price", mark_price, positive=True)
notional = abs(position.quantity) * mark_price
for bracket in schedule.brackets:
if notional <= bracket.notional_cap:
return notional * bracket.maintenance_rate - bracket.cumulative_maintenance_amount
raise ValueError("notional exceeds the final maintenance bracket cap")
@dataclass(frozen=True)
class AccountState:
"""Single-asset USDT account state without open-order margin."""
wallet_balance: float
positions: tuple[PositionState, ...]
margin_mode: MarginMode
terminal_status: TerminalStatus = "active"
def __post_init__(self) -> None:
_require_finite("wallet_balance", self.wallet_balance)
if self.margin_mode not in {"isolated", "cross"}:
raise ValueError("margin_mode must be 'isolated' or 'cross'")
if self.terminal_status not in {"active", "completed", "account_liquidation"}:
raise ValueError("unsupported terminal_status")
symbols = [position.symbol for position in self.positions]
if len(symbols) != len(set(symbols)):
raise ValueError("duplicate position symbol")
@dataclass(frozen=True)
class ExecutionFrame:
"""Normal market-fill input, separate from exchange mark prices."""
timestamp: pd.Timestamp
execution_open: float
def __post_init__(self) -> None:
object.__setattr__(self, "timestamp", _timestamp(self.timestamp, "timestamp"))
_require_finite("execution_open", self.execution_open, positive=True)
@dataclass(frozen=True)
class MarketRiskFrame:
"""Immutable mark, funding, bracket, provenance, and fidelity input.
``schedule`` is optional: the loader only attaches bracket columns when
a caller supplies a validated artifact (``require_brackets=True`` without
one fails closed at the loader level, before data ever reaches here). A
frame with ``schedule=None`` is valid for execution/mark/funding-only use.
"""
timestamp: pd.Timestamp
mark_open: float
mark_high: float
mark_low: float
mark_close: float
funding_rate: float | None
funding_settlement_time: pd.Timestamp | None
schedule: MaintenanceSchedule | None
source: str
fidelity_flags: tuple[str, ...] = ()
def __post_init__(self) -> None:
timestamp = _timestamp(self.timestamp, "timestamp")
object.__setattr__(self, "timestamp", timestamp)
for name in ("mark_open", "mark_high", "mark_low", "mark_close"):
_require_finite(name, getattr(self, name), positive=True)
if self.mark_high < max(self.mark_open, self.mark_low, self.mark_close):
raise ValueError("mark_high must contain the mark OHLC range")
if self.mark_low > min(self.mark_open, self.mark_high, self.mark_close):
raise ValueError("mark_low must contain the mark OHLC range")
has_rate = self.funding_rate is not None
has_settlement = self.funding_settlement_time is not None
if has_rate != has_settlement:
raise ValueError("funding rate and settlement timestamp must be paired")
if self.funding_rate is not None:
_require_finite("funding_rate", self.funding_rate)
settlement = _timestamp(
self.funding_settlement_time, "funding_settlement_time"
)
if settlement != timestamp:
raise ValueError("funding settlement timestamp must match frame timestamp")
object.__setattr__(self, "funding_settlement_time", settlement)
if not self.source:
raise ValueError("source must not be empty")
if len(self.fidelity_flags) != len(set(self.fidelity_flags)):
raise ValueError("fidelity_flags must not contain duplicates")
def _mark_price(
position: PositionState, frame: MarketRiskFrame, price_field: str
) -> float:
if price_field == "adverse":
return frame.mark_low if position.quantity > 0 else frame.mark_high
if price_field not in {"mark_open", "mark_high", "mark_low", "mark_close"}:
raise ValueError("unsupported price_field")
return getattr(frame, price_field)
def _position_risks(
account: AccountState,
frames: dict[str, MarketRiskFrame],
price_field: str,
*,
isolated: bool,
) -> tuple[tuple[PositionRisk, ...], tuple[str, ...]]:
if price_field not in {"adverse", "mark_open", "mark_high", "mark_low", "mark_close"}:
raise ValueError("unsupported price_field")
risks: list[PositionRisk] = []
timestamps: set[pd.Timestamp] = set()
fidelity_flags: list[str] = []
for position in account.positions:
frame = frames.get(position.symbol)
if frame is None:
raise ValueError("missing market risk frame")
if frame.schedule is None:
raise ValueError("missing maintenance schedule")
if frame.schedule.symbol != position.symbol:
raise ValueError("position and schedule symbols must match")
if isolated and position.isolated_margin is None:
raise ValueError("isolated_margin is required")
timestamps.add(frame.timestamp)
fidelity_flags.extend(frame.fidelity_flags)
mark_price = _mark_price(position, frame, price_field)
unrealized_pnl = position.quantity * (mark_price - position.entry_price)
initial_margin = abs(position.quantity) * mark_price / position.leverage
margin_balance = (
position.isolated_margin + unrealized_pnl if isolated else None
)
risks.append(
PositionRisk(
symbol=position.symbol,
mark_price=mark_price,
notional=abs(position.quantity) * mark_price,
unrealized_pnl=unrealized_pnl,
initial_margin=initial_margin,
maintenance_margin=maintenance_margin(position, mark_price, frame.schedule),
margin_balance=margin_balance,
)
)
if len(timestamps) > 1:
raise ValueError("position frame timestamps must match")
if len(account.positions) > 1 and price_field == "adverse":
fidelity_flags.append("conservative_intrabar_assumption")
return tuple(risks), tuple(dict.fromkeys(fidelity_flags))
def _risk_snapshot(
account: AccountState,
risks: tuple[PositionRisk, ...],
fidelity_flags: tuple[str, ...],
status: Literal["healthy", "position_liquidation", "account_liquidation"],
liquidation_targets: tuple[str, ...],
) -> RiskSnapshot:
margin_balance = account.wallet_balance + sum(risk.unrealized_pnl for risk in risks)
initial_margin = sum(risk.initial_margin for risk in risks)
maintenance = sum(risk.maintenance_margin for risk in risks)
return RiskSnapshot(
margin_balance=margin_balance,
initial_margin=initial_margin,
maintenance_margin=maintenance,
available_balance=margin_balance - initial_margin,
per_position=risks,
status=status,
liquidation_targets=liquidation_targets,
fidelity_flags=fidelity_flags,
)
def evaluate_isolated(
account: AccountState,
frames: dict[str, MarketRiskFrame],
price_field: str = "adverse",
) -> RiskSnapshot:
if account.margin_mode != "isolated":
raise ValueError("account margin_mode must be 'isolated'")
risks, fidelity_flags = _position_risks(account, frames, price_field, isolated=True)
liquidation_targets = tuple(
risk.symbol
for risk in risks
if risk.margin_balance is not None
and risk.margin_balance <= risk.maintenance_margin
)
return _risk_snapshot(
account,
risks,
fidelity_flags,
"position_liquidation" if liquidation_targets else "healthy",
liquidation_targets,
)
class CrossMarginRiskModel:
def evaluate(
self,
account: AccountState,
frames: dict[str, MarketRiskFrame],
price_field: str = "adverse",
) -> RiskSnapshot:
if account.margin_mode != "cross":
raise ValueError("account margin_mode must be 'cross'")
if any(position.isolated_margin is not None for position in account.positions):
raise ValueError("cross positions must not have isolated_margin")
risks, fidelity_flags = _position_risks(
account, frames, price_field, isolated=False
)
margin_balance = account.wallet_balance + sum(
risk.unrealized_pnl for risk in risks
)
maintenance = sum(risk.maintenance_margin for risk in risks)
liquidation_targets = (
tuple(position.symbol for position in account.positions)
if risks and margin_balance <= maintenance
else ()
)
return _risk_snapshot(
account,
risks,
fidelity_flags,
"account_liquidation" if liquidation_targets else "healthy",
liquidation_targets,
)
agent/backtest/rebalance_notes.py
+158
View File
@@ -0,0 +1,158 @@
"""Rebalance notes: per-rebalance turnover and weight-drift detail.
The Portfolio Studio epic (#456) asks for per-rebalance reporting as the
last backend slice, alongside the shipped turnover-aware optimizer and the
risk x-ray. The notes here are computed from the target position frame, so
they work for every optimizer and for the no-optimizer baseline: a rebalance
is any decision date whose target weight vector moved past ``epsilon`` from
the previous one. Trade-derived turnover in ``metrics`` measures what the
execution layer actually exchanged; these notes measure what the signal and
optimizer asked for, which is where churn starts.
"""
from __future__ import annotations
import json
from pathlib import Path
from typing import Any, Dict, List
import numpy as np
import pandas as pd
from backtest.validation import _json_safe
def compute_rebalance_notes(
target_pos: pd.DataFrame,
*,
top_n: int = 5,
epsilon: float = 1e-6,
) -> Dict[str, Any]:
"""Summarize per-date weight changes in a target position frame.
Args:
target_pos: Target weights (dates x codes), e.g. the frame behind
``artifacts/positions.csv``. NaN cells are treated as zero.
top_n: How many largest per-name moves to keep per rebalance.
epsilon: Turnover at or below this counts as "no rebalance".
Returns:
JSON-safe dict with ``rebalances`` (per date: turnover, entries,
exits, top moves by absolute weight change) and ``summary`` (count
plus turnover aggregates).
"""
empty = {
"rebalances": [],
"summary": {
"rebalance_count": 0,
"turnover_total": 0.0,
"turnover_mean": 0.0,
"turnover_max": 0.0,
"largest_rebalance_date": None,
},
}
if target_pos.empty or len(target_pos) < 2:
return empty
codes = target_pos.columns.tolist()
values = target_pos.fillna(0.0).to_numpy(dtype=float)
rebalances: List[Dict[str, Any]] = []
prev = values[0]
for i in range(1, len(values)):
curr = values[i]
delta = curr - prev
turnover = 0.5 * float(np.abs(delta).sum())
if turnover > epsilon:
date = target_pos.index[i]
entries = [
{"code": codes[j], "weight": float(curr[j])}
for j in range(len(codes))
if abs(prev[j]) <= epsilon and abs(curr[j]) > epsilon
]
exits = [
{"code": codes[j], "weight": float(prev[j])}
for j in range(len(codes))
if abs(curr[j]) <= epsilon and abs(prev[j]) > epsilon
]
moves = sorted(
(
{
"code": codes[j],
"from": float(prev[j]),
"to": float(curr[j]),
"delta": float(delta[j]),
}
for j in range(len(codes))
if abs(delta[j]) > epsilon
),
key=lambda move: -abs(move["delta"]),
)[:top_n]
rebalances.append(
{
"date": str(date.date()) if hasattr(date, "date") else str(date),
"turnover": turnover,
"entries": entries,
"exits": exits,
"top_moves": moves,
}
)
prev = curr
turnovers = [r["turnover"] for r in rebalances]
largest = max(range(len(rebalances)), key=lambda k: turnovers[k]) if rebalances else None
return {
"rebalances": rebalances,
"summary": {
"rebalance_count": len(rebalances),
"turnover_total": float(sum(turnovers)),
"turnover_mean": float(np.mean(turnovers)) if turnovers else 0.0,
"turnover_max": float(max(turnovers)) if turnovers else 0.0,
"largest_rebalance_date": rebalances[largest]["date"] if largest is not None else None,
},
}
def render_rebalance_notes_markdown(notes: Dict[str, Any]) -> str:
"""Render notes as a compact Markdown report."""
summary = notes["summary"]
lines = [
"# Rebalance Notes",
"",
f"- rebalances: {summary['rebalance_count']}",
f"- turnover total / mean / max: {summary['turnover_total']:.4f} / "
f"{summary['turnover_mean']:.4f} / {summary['turnover_max']:.4f}",
]
if summary["largest_rebalance_date"] is not None:
lines.append(f"- largest rebalance: {summary['largest_rebalance_date']}")
lines.append("")
for rebalance in notes["rebalances"]:
lines.append(f"## {rebalance['date']} (turnover {rebalance['turnover']:.4f})")
if rebalance["entries"]:
joined = ", ".join(item["code"] for item in rebalance["entries"])
lines.append(f"- entries: {joined}")
if rebalance["exits"]:
joined = ", ".join(item["code"] for item in rebalance["exits"])
lines.append(f"- exits: {joined}")
for move in rebalance["top_moves"]:
lines.append(f"- {move['code']}: {move['from']:.4f} -> {move['to']:.4f} ({move['delta']:+.4f})")
lines.append("")
return "\n".join(lines)
def write_rebalance_notes(path: Path, notes: Dict[str, Any]) -> Dict[str, Any]:
"""Write notes to ``path`` as strict, RFC-8259 JSON.
Mirrors :func:`backtest.validation.write_validation_json`: sanitize with
``_json_safe`` (non-finite -> null) and serialize with ``allow_nan=False``
so every strict parser accepts the artifact. Returns the sanitized
payload that was written.
"""
safe_notes = _json_safe(notes)
path.parent.mkdir(parents=True, exist_ok=True)
path.write_text(
json.dumps(safe_notes, indent=2, ensure_ascii=False, allow_nan=False) + "\n",
encoding="utf-8",
)
return safe_notes
agent/backtest/regime.py
+235
View File
@@ -0,0 +1,235 @@
"""Correlation-regime timeline: edge density + hysteresis over time.
Implements Mode 1 of the ``correlation-regime`` skill on top of the same
price data the /correlation endpoint uses: rolling pairwise correlations are
reduced to an edge-density scalar per bar, the density series is smoothed
with a trailing (causal) window, and a two-threshold hysteresis state
machine labels each bar FUSED or not. Used by the /correlation/regime API
endpoint. Descriptive risk context — not a trading signal.
"""
from __future__ import annotations
import logging
from typing import Dict, Optional
import numpy as np
import pandas as pd
from backtest.correlation import _close_series, _fetch_price_series
logger = logging.getLogger(__name__)
def compute_edge_density(
returns: pd.DataFrame,
corr_window: int = 60,
edge_threshold: float = 0.5,
) -> pd.Series:
"""Reduce rolling correlation matrices to an edge-density series.
Edge density is the fraction of distinct asset pairs whose rolling
|correlation| clears ``edge_threshold`` — a scalar "how fused is the
market" gauge in [0, 1].
Args:
returns: Multi-asset return matrix, columns are symbols
corr_window: Rolling window length (bars) for pairwise correlation
edge_threshold: |ρ| level at which a pair counts as an "edge"
Returns:
Edge-density series aligned to ``returns.index`` (NaN during warmup)
"""
n_assets = returns.shape[1]
n_pairs = n_assets * (n_assets - 1) // 2
if n_pairs == 0:
return pd.Series(np.nan, index=returns.index)
upper_mask = np.triu(np.ones((n_assets, n_assets), dtype=bool), k=1)
density = pd.Series(np.nan, index=returns.index)
for i in range(corr_window, len(returns) + 1):
corr = returns.iloc[i - corr_window:i].corr().abs().to_numpy()
density.iloc[i - 1] = float((corr[upper_mask] >= edge_threshold).sum()) / n_pairs
return density
def detect_regimes(
density: pd.Series,
smooth_window: int = 5,
enter_threshold: float = 0.65,
exit_threshold: float = 0.45,
) -> pd.DataFrame:
"""Hysteresis (Schmitt-trigger) regime state machine on smoothed density.
The market is FUSED once smoothed density reaches ``enter_threshold`` and
stays FUSED until it falls back to ``exit_threshold``. The dead band
between the two thresholds is what suppresses chatter.
Args:
density: Edge-density series from :func:`compute_edge_density`
smooth_window: Trailing smoothing window (causal; never centered)
enter_threshold: Density level that opens a FUSED regime
exit_threshold: Density level that closes it (must be < enter_threshold)
Returns:
DataFrame with columns ``density``, ``smoothed``, ``fused`` (0/1)
"""
if exit_threshold >= enter_threshold:
raise ValueError("exit_threshold must be below enter_threshold")
# Trailing mean = causal. A centered window here silently reads the future.
smoothed = density.rolling(smooth_window, min_periods=1).mean()
fused = False
states = np.zeros(len(smoothed), dtype=int)
for i, value in enumerate(smoothed.to_numpy()):
if np.isnan(value):
states[i] = int(fused)
continue
if not fused and value >= enter_threshold:
fused = True
elif fused and value <= exit_threshold:
fused = False
states[i] = int(fused)
return pd.DataFrame(
{"density": density, "smoothed": smoothed, "fused": states},
index=density.index,
)
def _aligned_returns(price_series: Dict[str, pd.DataFrame]) -> pd.DataFrame:
"""Build a date-aligned daily-returns frame (one column per code).
Mirrors the alignment in ``_rolling_correlation_matrix``: indexes are
normalized to midnight so cross-market assets (e.g. crypto at UTC
midnight vs US equity at EDT midnight) share dates, then inner-joined.
``fill_method=None`` is explicit because under the project's
pandas>=2,<3 pin the ``pct_change`` default forward-fills missing
prices, silently manufacturing 0% returns.
"""
returns_frames = []
for code in sorted(price_series):
ts = _close_series(code, price_series[code])
ts.index = ts.index.normalize()
rets = ts.pct_change(fill_method=None).dropna()
rets.name = code
returns_frames.append(rets)
aligned = pd.concat(returns_frames, axis=1).dropna()
if aligned.empty:
raise ValueError("No overlapping return data between assets")
return aligned
def _fused_episodes(dates: list[str], fused: list[int]) -> list[Dict[str, Optional[str]]]:
"""Contiguous FUSED intervals within the returned window.
``end`` is the last date observed FUSED, or None while the final bar is
still FUSED (episode ongoing).
"""
episodes: list[Dict[str, Optional[str]]] = []
start: Optional[str] = None
last_fused: Optional[str] = None
for date, state in zip(dates, fused):
if state:
if start is None:
start = date
last_fused = date
elif start is not None:
episodes.append({"start": start, "end": last_fused})
start = None
if start is not None:
episodes.append({"start": start, "end": None})
return episodes
def compute_regime_timeline(
codes: list[str],
days: int = 90,
corr_window: int = 60,
edge_threshold: float = 0.5,
smooth_window: int = 5,
enter_threshold: float = 0.65,
exit_threshold: float = 0.45,
) -> Dict[str, object]:
"""Fetch price data and compute the correlation-regime timeline.
Args:
codes: List of asset codes, as for :func:`compute_correlation_matrix`.
days: Number of trailing timeline bars to return.
corr_window: Rolling window (bars) for pairwise correlation.
edge_threshold: |ρ| level at which a pair counts as an "edge".
smooth_window: Trailing smoothing window (bars) for the density series.
enter_threshold: Smoothed density that opens a FUSED regime.
exit_threshold: Smoothed density that closes it (must be below
``enter_threshold``).
Returns:
Dict with keys: labels, dates, density, smoothed, fused, episodes,
params. ``density``/``smoothed`` use None for warmup bars; ``fused``
is 0/1 per bar; ``episodes`` lists FUSED intervals with ``end=None``
while the final bar is still FUSED.
"""
from datetime import datetime, timedelta
if exit_threshold >= enter_threshold:
raise ValueError("exit_threshold must be below enter_threshold")
# Each returned bar needs a full corr_window of history behind it, so the
# warmup falls outside the returned window: extend /correlation's +60
# calendar-day fetch buffer by the correlation window (and a margin for
# non-trading days).
end_date = datetime.now().strftime("%Y-%m-%d")
start_date = (
datetime.now() - timedelta(days=days + corr_window + 90)
).strftime("%Y-%m-%d")
price_series = _fetch_price_series(codes, start_date, end_date)
if len(price_series) < 2:
raise ValueError(
f"Could not fetch price data for at least 2 assets. "
f"Fetched: {list(price_series.keys())}"
)
returns = _aligned_returns(price_series)
density = compute_edge_density(
returns, corr_window=corr_window, edge_threshold=edge_threshold
)
regimes = detect_regimes(
density,
smooth_window=smooth_window,
enter_threshold=enter_threshold,
exit_threshold=exit_threshold,
)
# Trim to the requested window only after the state machine has run, so
# the regime state at the window's first bar reflects the full history.
if len(regimes) > days:
regimes = regimes.iloc[-days:]
dates = [d.strftime("%Y-%m-%d") for d in regimes.index]
density_out = [
None if np.isnan(v) else round(float(v), 4) for v in regimes["density"]
]
smoothed_out = [
None if np.isnan(v) else round(float(v), 4) for v in regimes["smoothed"]
]
fused_out = [int(v) for v in regimes["fused"]]
return {
"labels": list(returns.columns),
"dates": dates,
"density": density_out,
"smoothed": smoothed_out,
"fused": fused_out,
"episodes": _fused_episodes(dates, fused_out),
"params": {
"days": days,
"corr_window": corr_window,
"edge_threshold": edge_threshold,
"smooth_window": smooth_window,
"enter_threshold": enter_threshold,
"exit_threshold": exit_threshold,
},
}
agent/backtest/risk_xray.py
+364
View File
@@ -0,0 +1,364 @@
"""Portfolio risk x-ray: concentration, volatility, drawdown, tail risk, and
co-movement for a weighted basket, computed from close-price panels.
This is the Portfolio Studio risk-x-ray backend (roadmap slice). Everything
here is a pure function of prices and weights — no I/O, no network, no
loader imports — so the same computation serves the agent tool, the API,
and tests. Data fetching lives in the caller.
Conventions:
- Long-only v1: negative weights are rejected up front rather than silently
mis-computing concentration (a short leg is a different feature).
- Non-finite inputs are never trusted and non-finite outputs are never
emitted: every reported float passes through ``_finite`` so the result
survives ``json.dumps(..., allow_nan=False)``.
- Look-ahead is structurally impossible: all statistics are computed over
trailing returns only.
"""
from __future__ import annotations
import json
import math
from pathlib import Path
from typing import Any, Mapping, Sequence
import numpy as np
import pandas as pd
from backtest.validation import _json_safe
MIN_HISTORY_DAYS = 30
PERIODS_PER_YEAR = 252
VAR_LEVELS = (0.95, 0.99)
def _finite(value: float | None) -> float | None:
"""Return ``value`` when finite, else ``None`` (strict-JSON safe)."""
if value is None:
return None
try:
out = float(value)
except (TypeError, ValueError):
return None
return out if math.isfinite(out) else None
def _validate_weights(
closes: pd.DataFrame, weights: Mapping[str, float]
) -> tuple[dict[str, float], list[str]]:
"""Normalize weights to sum 1; reject unknown symbols and bad values.
Returns the cleaned weight map (restricted to available columns) and any
warnings worth surfacing to the caller.
"""
warnings: list[str] = []
if not weights:
raise ValueError("weights must name at least one symbol")
unknown = [sym for sym in weights if sym not in closes.columns]
if unknown:
raise ValueError(f"weights reference symbols with no price data: {sorted(unknown)}")
cleaned: dict[str, float] = {}
for sym, raw in weights.items():
try:
value = float(raw)
except (TypeError, ValueError) as exc:
raise ValueError(f"weight for {sym!r} is not a number: {raw!r}") from exc
if not math.isfinite(value):
raise ValueError(f"weight for {sym!r} is not finite: {raw!r}")
if value < 0:
raise ValueError(
f"weight for {sym!r} is negative ({value}); the risk x-ray is long-only for now"
)
cleaned[sym] = value
total = sum(cleaned.values())
if total <= 0:
raise ValueError("weights must sum to a positive value")
if abs(total - 1.0) > 1e-6:
warnings.append(f"weights summed to {total:.6f}; renormalized to 1.0")
cleaned = {sym: value / total for sym, value in cleaned.items()}
return cleaned, warnings
def compute_risk_xray(
closes: pd.DataFrame,
weights: Mapping[str, float],
*,
periods_per_year: int = PERIODS_PER_YEAR,
var_levels: Sequence[float] = VAR_LEVELS,
min_history: int = MIN_HISTORY_DAYS,
) -> dict[str, Any]:
"""Compute the risk x-ray for a weighted basket.
Args:
closes: Close-price panel, one column per symbol, sorted by date.
weights: Symbol → weight. Renormalized to 1.0 with a warning when the
sum differs; must be long-only and reference existing columns.
periods_per_year: Annualization factor for the bar interval.
var_levels: Tail levels for historical VaR / expected shortfall.
min_history: Minimum valid bars a symbol must have to be included.
Returns:
A strict-JSON-safe dict (``json.dumps(..., allow_nan=False)`` must
never fail on it) with concentration, volatility, drawdown, tail
risk, diversification, and correlation sections, plus ``skipped``
and ``warnings`` bookkeeping.
Raises:
ValueError: On unusable input (empty panel, bad weights, or nothing
left after the history filter / calendar alignment).
"""
if closes is None or closes.empty:
raise ValueError("price panel is empty")
frame = closes.dropna(axis=1, how="all")
if frame.empty:
raise ValueError("price panel has no non-NaN closes")
weights, warnings = _validate_weights(frame, weights)
# History filter: thin symbols are excluded rather than allowed to skew
# the whole x-ray on a handful of bars.
kept: list[str] = []
skipped: list[dict[str, str]] = []
for sym in weights:
valid = int(frame[sym].count())
if valid < min_history:
skipped.append({"symbol": sym, "reason": f"only {valid} valid bars (min {min_history})"})
else:
kept.append(sym)
if not kept:
raise ValueError(f"no symbol has at least {min_history} valid bars")
if skipped:
# Renormalize over the survivors so the x-ray still describes a
# fully invested basket, and say so.
kept_weights = {sym: weights[sym] for sym in kept}
total = sum(kept_weights.values())
if total <= 0:
raise ValueError("surviving symbols have zero total weight")
weights = {sym: value / total for sym, value in kept_weights.items()}
warnings.append("weights renormalized over symbols that survived the history filter")
aligned = frame[kept].dropna(axis=0, how="any")
if len(aligned) < 2:
raise ValueError(
"fewer than 2 shared trading days after aligning calendars across symbols"
)
returns = aligned.pct_change(fill_method=None).dropna(how="any")
if returns.empty:
raise ValueError("no overlapping return observations across symbols")
w = np.array([weights[sym] for sym in kept], dtype=float)
port = returns.to_numpy(dtype=float) @ w
port_returns = pd.Series(port, index=returns.index)
result: dict[str, Any] = {
"inputs": {
"symbols": kept,
"weights": {sym: round(float(weights[sym]), 8) for sym in kept},
"aligned_days": int(len(aligned)),
"return_observations": int(len(returns)),
"first_date": str(aligned.index[0]),
"last_date": str(aligned.index[-1]),
},
"concentration": _concentration(w),
"volatility": _volatility(port_returns, periods_per_year),
"drawdown": _drawdown(port_returns),
"tail_risk": _tail_risk(port_returns, var_levels),
"diversification": _diversification(returns, w, port_returns, periods_per_year),
"correlation": _correlation(returns, port_returns),
"skipped": skipped,
"warnings": warnings,
}
return result
def _concentration(w: np.ndarray) -> dict[str, Any]:
hhi = float(np.sum(w**2))
order = np.argsort(w)[::-1]
return {
"hhi": _finite(hhi),
"effective_n": _finite(1.0 / hhi) if hhi > 0 else None,
"top1_weight": _finite(float(w[order[0]])) if len(order) else None,
"top3_weight": _finite(float(w[order[:3]].sum())) if len(order) else None,
}
def _volatility(port: pd.Series, ppy: int) -> dict[str, Any]:
vol = float(port.std(ddof=1)) if len(port) > 1 else None
downside = port[port < 0]
downside_dev = float(downside.std(ddof=1)) if len(downside) > 1 else None
return {
"daily_vol": _finite(vol),
"annualized_vol": _finite(vol * math.sqrt(ppy)) if vol is not None else None,
"downside_deviation_annualized": (
_finite(downside_dev * math.sqrt(ppy)) if downside_dev is not None else None
),
}
def _drawdown(port: pd.Series) -> dict[str, Any]:
if port.empty:
return {"max_drawdown": None, "max_drawdown_start": None, "max_drawdown_trough": None}
equity = (1.0 + port).cumprod()
peak = equity.cummax()
dd = equity / peak - 1.0
trough_idx = dd.idxmin()
start_idx = equity.loc[:trough_idx].idxmax()
return {
"max_drawdown": _finite(float(dd.loc[trough_idx])),
"max_drawdown_start": str(start_idx),
"max_drawdown_trough": str(trough_idx),
}
def _tail_risk(port: pd.Series, levels: Sequence[float]) -> dict[str, Any]:
out: dict[str, Any] = {}
losses = -port.to_numpy(dtype=float)
for level in levels:
if len(losses) < 2:
var = es = None
else:
var = float(np.quantile(losses, level))
tail = losses[losses >= var]
es = float(tail.mean()) if len(tail) else None
key = f"{int(round(level * 100))}"
out[f"var_{key}"] = _finite(var)
out[f"expected_shortfall_{key}"] = _finite(es)
out["method"] = "historical simulation (non-parametric)"
return out
def _diversification(
returns: pd.DataFrame, w: np.ndarray, port: pd.Series, ppy: int
) -> dict[str, Any]:
if returns.shape[1] < 2:
return {"diversification_ratio": None, "note": "needs at least 2 assets"}
asset_vols = returns.std(ddof=1).to_numpy(dtype=float)
port_vol = float(port.std(ddof=1)) if len(port) > 1 else 0.0
if port_vol <= 0 or not math.isfinite(port_vol):
return {"diversification_ratio": None, "note": "portfolio variance is zero"}
ratio = float(np.dot(w, asset_vols) / port_vol)
return {"diversification_ratio": _finite(ratio)}
def _correlation(returns: pd.DataFrame, port: pd.Series) -> dict[str, Any]:
if returns.shape[1] < 2:
return {
"avg_pairwise_abs": None,
"max_pair": None,
"beta_to_equal_weight": None,
"note": "needs at least 2 assets",
}
corr = returns.corr().to_numpy(dtype=float)
n = corr.shape[0]
off_diag = [abs(corr[i, j]) for i in range(n) for j in range(i + 1, n)]
avg_pairwise = float(np.mean(off_diag)) if off_diag else None
max_pair = None
if off_diag:
i, j = max(
((i, j) for i in range(n) for j in range(i + 1, n)),
key=lambda pair: abs(corr[pair[0], pair[1]]),
)
max_pair = {
"symbols": [str(returns.columns[i]), str(returns.columns[j])],
"corr": _finite(float(corr[i, j])),
}
market = returns.mean(axis=1)
market_var = float(market.var(ddof=1)) if len(market) > 1 else 0.0
if market_var > 0 and math.isfinite(market_var):
beta = float(port.cov(market) / market_var)
else:
beta = None
return {
"avg_pairwise_abs": _finite(avg_pairwise),
"max_pair": max_pair,
"beta_to_equal_weight": _finite(beta),
}
def average_invested_weights(target_pos: pd.DataFrame) -> tuple[dict[str, float], float]:
"""Derive the run's average basket from the target position frame.
Returns ``(weights, avg_invested)``: the mean target weight per symbol
restricted to symbols the strategy actually held on average, and the mean
row sum (how much of the book was invested at all). The final row alone
would be the wrong object here, since many strategies end flat.
Raises:
ValueError: when the frame is empty, the strategy never held
anything on average, or any symbol's average exposure is net
short — ``compute_risk_xray`` is long-only, and silently
x-raying just the long half of a long-short book would present
a partial basket as the whole strategy.
"""
if target_pos is None or target_pos.empty:
raise ValueError("target position frame is empty")
means = target_pos.mean(axis=0)
short_book = [str(sym) for sym, w in means.items() if float(w) < 0]
if short_book:
raise ValueError(
"long-only x-ray cannot describe net short average exposure "
f"in {', '.join(short_book)}"
)
weights = {str(sym): float(w) for sym, w in means.items() if float(w) > 0}
avg_invested = float(target_pos.sum(axis=1).mean())
if not weights:
raise ValueError("strategy held no average exposure")
return weights, avg_invested
def render_risk_xray_markdown(report: dict[str, Any]) -> str:
"""Render an x-ray report as a compact Markdown summary."""
inputs = report["inputs"]
conc = report["concentration"]
vol = report["volatility"]
dd = report["drawdown"]
tail = report["tail_risk"]
lines = [
"# Portfolio Risk X-Ray",
"",
f"- basket: {', '.join(inputs['symbols'])}",
f"- window: {inputs['first_date']} .. {inputs['last_date']} "
f"({inputs['aligned_days']} aligned days)",
]
if conc.get("hhi") is not None:
lines.append(
f"- concentration: hhi {conc['hhi']:.4f}, "
f"effective n {conc['effective_n']:.2f}, "
f"top1 {conc['top1_weight']:.2%}, top3 {conc['top3_weight']:.2%}"
)
if vol.get("annualized_vol") is not None:
lines.append(f"- annualized vol: {vol['annualized_vol']:.2%}")
if dd.get("max_drawdown") is not None:
lines.append(f"- max drawdown: {dd['max_drawdown']:.2%}")
if tail.get("var_95") is not None:
lines.append(
f"- tail (historical): VaR95 {tail['var_95']:.2%}, ES95 {tail['expected_shortfall_95']:.2%}"
)
if report["skipped"]:
joined = ", ".join(item["symbol"] for item in report["skipped"])
lines.append(f"- skipped: {joined}")
lines.append("")
return "\n".join(lines)
def write_risk_xray(path: Path, report: dict[str, Any]) -> dict[str, Any]:
"""Write the report to ``path`` as strict, RFC-8259 JSON.
Same contract as ``write_rebalance_notes``: sanitize with ``_json_safe``
and serialize with ``allow_nan=False``. Returns the sanitized payload.
"""
safe_report = _json_safe(report)
path.parent.mkdir(parents=True, exist_ok=True)
path.write_text(
json.dumps(safe_report, indent=2, ensure_ascii=False, allow_nan=False) + "\n",
encoding="utf-8",
)
return safe_report
agent/backtest/run_card.py
+50 -1
View File
@@ -4,6 +4,7 @@ from __future__ import annotations
import hashlib
import json
import math
from datetime import datetime, timezone
from pathlib import Path
from typing import Any, Mapping, Sequence
@@ -29,6 +30,7 @@ def write_run_card(
data_sources: Sequence[str] | None = None,
strategy_path: Path | None = None,
warnings: Sequence[str] | None = None,
artifact_refs: Sequence[Mapping[str, Any]] | None = None,
) -> dict[str, Any]:
"""Write JSON and Markdown run cards for a backtest run.
@@ -40,6 +42,7 @@ def write_run_card(
data_sources: Data sources used by the run.
strategy_path: Optional strategy source file to hash for reproducibility.
warnings: Optional warnings to include in the card.
artifact_refs: Optional IRR-AGL artifact references.
Returns:
The run card payload written to ``run_card.json``.
@@ -67,13 +70,25 @@ def write_run_card(
"warnings": list(warnings or []),
"artifacts": _list_artifacts(run_dir),
}
normalized_refs = _normalize_artifact_refs(artifact_refs)
if normalized_refs:
card["artifact_refs"] = normalized_refs
if "validation" in metrics:
card["validation"] = metrics["validation"]
card = _json_safe(card)
json_path = run_dir / "run_card.json"
md_path = run_dir / "run_card.md"
json_path.write_text(
json.dumps(card, ensure_ascii=False, indent=2, sort_keys=True, default=str) + "\n",
json.dumps(
card,
ensure_ascii=False,
indent=2,
sort_keys=True,
default=str,
allow_nan=False,
)
+ "\n",
encoding="utf-8",
)
md_path.write_text(_render_markdown(card), encoding="utf-8")
@@ -115,6 +130,16 @@ def _scalar_metrics(metrics: Mapping[str, Any]) -> dict[str, Any]:
}
def _json_safe(value: Any) -> Any:
if isinstance(value, float):
return value if math.isfinite(value) else None
if isinstance(value, Mapping):
return {str(key): _json_safe(item) for key, item in value.items()}
if isinstance(value, (list, tuple)):
return [_json_safe(item) for item in value]
return value
def _is_scalar(value: Any) -> bool:
return value is None or isinstance(value, (str, int, float, bool))
@@ -142,6 +167,17 @@ def _list_artifacts(run_dir: Path) -> list[dict[str, Any]]:
return artifacts
def _normalize_artifact_refs(artifact_refs: Sequence[Mapping[str, Any]] | None) -> list[dict[str, Any]]:
refs: list[dict[str, Any]] = []
for ref in artifact_refs or []:
if hasattr(ref, "model_dump"):
value = ref.model_dump(mode="json") # type: ignore[attr-defined]
else:
value = dict(ref)
refs.append(_json_safe(value))
return refs
def _render_markdown(card: Mapping[str, Any]) -> str:
lines = [
"# Backtest Run Card",
@@ -197,4 +233,17 @@ def _render_markdown(card: Mapping[str, Any]) -> str:
else:
lines.append("- None found.")
artifact_refs = card.get("artifact_refs", [])
if artifact_refs:
lines.extend(["", "## IRR Artifact Refs"])
for ref in artifact_refs:
if isinstance(ref, Mapping):
label = ref.get("artifact_id", "")
artifact_type = ref.get("artifact_type", "")
digest = ref.get("sha256", "")
uri = ref.get("uri", "")
lines.append(f"- `{label}` ({artifact_type}, sha256 `{digest}`, uri `{uri}`)")
else:
lines.append(f"- {ref}")
return "\n".join(lines) + "\n"
agent/backtest/runner.py
+714 -66
View File
@@ -8,16 +8,18 @@ Usage: ``python -m backtest.runner <run_dir>``
"""
import ast
import copy
import importlib.util
import inspect
import json
import logging
import sys
from dataclasses import dataclass
from pathlib import Path
from typing import Dict, List, Optional
from typing import Any, Dict, Iterable, List, Optional
import pandas as pd
from pydantic import BaseModel, ConfigDict, model_validator, field_validator
from pydantic import BaseModel, ConfigDict, Field, model_validator, field_validator
try:
from dotenv import load_dotenv
@@ -28,10 +30,11 @@ except ImportError:
from backtest.loaders.registry import (
FALLBACK_CHAINS,
LOADER_REGISTRY,
VALID_SOURCES,
get_loader_cls_with_fallback,
resolve_loader,
)
from backtest.loaders.base import NoAvailableSourceError
from backtest.loaders.base import NoAvailableSourceError, validate_ohlc
# Symbol classification lives in ``_market_hooks`` so runner.py and
# composite.py share a single source of truth (audit-2026-05-18 B1+C1+C2).
# ``_detect_market`` is also re-exported here for back-compat with
@@ -47,7 +50,19 @@ logger = logging.getLogger(__name__)
_VALID_INTERVALS = {"1m", "5m", "15m", "30m", "1H", "4H", "1D"}
_VALID_ENGINES = {"daily", "options"}
_VALID_SOURCES = {"tushare", "okx", "yfinance", "akshare", "ccxt", "auto"}
_PRICE_PANEL_COLUMNS = ("open", "high", "low", "close", "volume", "vwap", "amount")
_FUND_PREFIX = "fund:"
@dataclass(frozen=True)
class DataFetchResult:
"""Market data plus the routing metadata selected by the central registry."""
data_map: Dict[str, pd.DataFrame]
codes: List[str]
source: str
loader: Any
effective_sources: List[str]
class BacktestConfigSchema(BaseModel):
@@ -61,7 +76,12 @@ class BacktestConfigSchema(BaseModel):
source: str = "tushare"
interval: str = "1D"
engine: str = "daily"
# Returns divide by initial_cash, so a non-positive value yields inf/NaN
# metrics (total_return, annual_return, ...). Reject it at the config
# boundary instead of letting the run produce non-finite results.
initial_cash: float = Field(default=1_000_000, gt=0, allow_inf_nan=False)
fundamental_fields: Optional[Dict[str, List[str]]] = None
event_feeds: Optional[List[Dict[str, Any]]] = None
@field_validator("codes")
@classmethod
@@ -98,8 +118,8 @@ class BacktestConfigSchema(BaseModel):
@field_validator("source")
@classmethod
def valid_source(cls, v: str) -> str:
if v not in _VALID_SOURCES:
raise ValueError(f"unsupported source {v!r}, must be one of {_VALID_SOURCES}")
if v not in VALID_SOURCES:
raise ValueError(f"unsupported source {v!r}, must be one of {VALID_SOURCES}")
return v
@field_validator("fundamental_fields")
@@ -117,6 +137,21 @@ class BacktestConfigSchema(BaseModel):
raise ValueError("fundamental_fields field names must be non-empty strings")
return v
@field_validator("event_feeds")
@classmethod
def valid_event_feeds(cls, v: Optional[List[Dict[str, Any]]]) -> Optional[List[Dict[str, Any]]]:
if v is None:
return v
for entry in v:
if not isinstance(entry, dict):
raise ValueError(
"each event_feeds entry must be an object with name/route_template/event_type"
)
for key in ("name", "route_template", "event_type"):
if not str(entry.get(key, "")).strip():
raise ValueError(f"event_feeds entry missing required field: {key}")
return v
@model_validator(mode="after")
def start_before_end(self) -> "BacktestConfigSchema":
if pd.Timestamp(self.start_date) > pd.Timestamp(self.end_date):
@@ -224,6 +259,214 @@ def _validate_class_body(node: ast.ClassDef) -> None:
)
# --- Runtime-reachable operation scrubber (VT-001 defense-in-depth) ---
#
# The structural checks above only reject *import-time* execution. The real
# exposure is that once ``SignalEngine`` is instantiated and ``.generate()`` is
# called, arbitrary code inside its method bodies runs with no runtime sandbox.
# This scrubber walks the code that actually executes during a backtest — every
# ``SignalEngine`` method plus any module-level helper transitively called from
# one — and rejects network / process-spawn / dynamic-exec / filesystem-write
# operations there.
#
# It is deliberately scoped to *reachable* code, not the whole file: the bundled
# skill examples (agent/src/skills/*/example_signal_engine.py) legitimately carry
# ``import requests`` + ``requests.get`` inside standalone ``_fetch_okx`` helpers
# and ``if __name__ == "__main__"`` demo blocks that the runner never executes
# (it does ``import module; SignalEngine().generate(data_map)``). Blocking those
# imports file-wide would reject strategies generated from ~12 shipped skills, so
# we block the dangerous *use* along the executed path instead of a harmless
# unused top-level import. Direct ``getattr``/``setattr``/``delattr`` indirection
# onto ``os`` / forbidden modules is now rejected (see _reject_forbidden_getattr);
# exotic reach via ``builtins.getattr`` or aliasing remains a documented residual
# (VT-001) — this is defense-in-depth, not a kernel-level guarantee.
_FORBIDDEN_IMPORT_MODULES = frozenset(
{
"socket",
"socketserver",
"subprocess",
"urllib",
"urllib2",
"urllib3",
"http",
"requests",
"httpx",
"aiohttp",
"ftplib",
"smtplib",
"telnetlib",
"multiprocessing",
"ctypes",
}
)
# ``os`` itself is allowed (os.path etc.), but these attributes shell out, spawn,
# or read the process environment — none has a place in a signal engine.
_FORBIDDEN_OS_ATTRS = frozenset(
{
"system",
"popen",
"popen2",
"popen3",
"popen4",
"fork",
"forkpty",
"putenv",
"unsetenv",
"getenv",
"environ",
"environb",
"startfile",
}
)
_FORBIDDEN_BUILTINS = frozenset(
{"eval", "exec", "compile", "__import__", "globals", "locals", "vars", "breakpoint"}
)
# getattr/setattr/delattr can indirect around the attribute scanner
# (``getattr(os, "system")("id")``). We reject them ONLY when the target object
# is ``os`` or a forbidden module — keyed off the target, not the attribute
# string, so ``getattr(os, "sys" + "tem")`` is caught too. Legitimate dynamic
# access on user objects (``getattr(tech, name, None)``, ``getattr(self, x)``)
# is unaffected.
_GETATTR_INDIRECTION = frozenset({"getattr", "setattr", "delattr"})
_OPEN_WRITE_MODE_CHARS = frozenset("wax+")
_SCRUB_MSG = "is not allowed inside generated strategy code"
def _is_forbidden_os_attr(attr: str) -> bool:
"""Return whether ``os.<attr>`` shells out, spawns, execs, or reads env."""
return attr in _FORBIDDEN_OS_ATTRS or attr.startswith(("spawn", "exec"))
def _attribute_root_name(node: ast.Attribute) -> str | None:
"""Return the leftmost ``Name`` id of an attribute chain (``a.b.c`` -> ``a``)."""
current: ast.AST = node
while isinstance(current, ast.Attribute):
current = current.value
return current.id if isinstance(current, ast.Name) else None
def _reject_forbidden_open(node: ast.Call) -> None:
"""Reject ``open()`` used to write files or read a non-relative-literal path."""
func = node.func
is_builtin_open = isinstance(func, ast.Name) and func.id == "open"
is_io_os_open = (
isinstance(func, ast.Attribute)
and func.attr == "open"
and isinstance(func.value, ast.Name)
and func.value.id in {"io", "os"}
)
if not (is_builtin_open or is_io_os_open):
return
mode_node: ast.AST | None = node.args[1] if len(node.args) >= 2 else None
for kw in node.keywords:
if kw.arg == "mode":
mode_node = kw.value
if mode_node is not None:
if not (isinstance(mode_node, ast.Constant) and isinstance(mode_node.value, str)):
raise ValueError(f"open() with a non-literal mode {_SCRUB_MSG}")
if any(ch in _OPEN_WRITE_MODE_CHARS for ch in mode_node.value):
raise ValueError(f"Writing files via open(mode={mode_node.value!r}) {_SCRUB_MSG}")
path_node = node.args[0] if node.args else None
if not (isinstance(path_node, ast.Constant) and isinstance(path_node.value, str)):
raise ValueError(f"open() with a non-literal path {_SCRUB_MSG}")
path = path_node.value
if path.startswith(("/", "~", "\\")) or ".." in path or (len(path) > 1 and path[1] == ":"):
raise ValueError(f"open() with a non-relative path {path!r} {_SCRUB_MSG}")
def _reject_forbidden_getattr(node: ast.Call) -> None:
"""Reject getattr/setattr/delattr indirection onto ``os`` / forbidden modules.
Closes the documented bypass ``getattr(os, "system")("id")`` — and computed
variants like ``getattr(os, "sys" + "tem")`` — by keying off the *target*
object (first positional arg), not the attribute string. Dynamic access on
ordinary user objects (``getattr(tech, name, None)``) is left untouched.
"""
func = node.func
if not (isinstance(func, ast.Name) and func.id in _GETATTR_INDIRECTION):
return
if not node.args:
return
target = node.args[0]
if isinstance(target, ast.Name):
root: str | None = target.id
elif isinstance(target, ast.Attribute):
root = _attribute_root_name(target)
else:
root = None
if root == "os" or root in _FORBIDDEN_IMPORT_MODULES:
raise ValueError(f"{func.id}() indirection onto {root!r} {_SCRUB_MSG}")
def _reject_forbidden_node(node: ast.AST) -> None:
"""Raise ``ValueError`` if a single AST node performs a forbidden operation."""
if isinstance(node, ast.Import):
for alias in node.names:
if alias.name.split(".")[0] in _FORBIDDEN_IMPORT_MODULES:
raise ValueError(f"Import of {alias.name!r} {_SCRUB_MSG}")
elif isinstance(node, ast.ImportFrom):
root = (node.module or "").split(".")[0]
if root in _FORBIDDEN_IMPORT_MODULES:
raise ValueError(f"Import from {node.module!r} {_SCRUB_MSG}")
if root == "os":
for alias in node.names:
if _is_forbidden_os_attr(alias.name):
raise ValueError(f"Import of os.{alias.name} {_SCRUB_MSG}")
elif isinstance(node, ast.Attribute):
root = _attribute_root_name(node)
if root in _FORBIDDEN_IMPORT_MODULES:
raise ValueError(f"Use of {root}.{node.attr} {_SCRUB_MSG}")
if root == "os" and _is_forbidden_os_attr(node.attr):
raise ValueError(f"Use of os.{node.attr} {_SCRUB_MSG}")
elif isinstance(node, ast.Name):
if node.id in _FORBIDDEN_BUILTINS:
raise ValueError(f"Use of {node.id!r} {_SCRUB_MSG}")
elif isinstance(node, ast.Call):
_reject_forbidden_open(node)
_reject_forbidden_getattr(node)
def _scan_runtime_reachable(tree: ast.Module) -> None:
"""Reject forbidden ops in ``SignalEngine`` methods + their transitive callees.
Entry points are every method defined directly on the ``SignalEngine`` class.
From there, any bare-name call that resolves to a module-level function is
followed and scanned too, so a payload hidden in a helper that ``generate()``
calls is still caught. Module-level functions never reached from a
``SignalEngine`` method (standalone data-fetch helpers, ``__main__`` demos)
are intentionally left unscanned.
"""
engine_cls = next(
(n for n in tree.body if isinstance(n, ast.ClassDef) and n.name == "SignalEngine"),
None,
)
if engine_cls is None:
return
module_funcs = {
n.name: n
for n in tree.body
if isinstance(n, (ast.FunctionDef, ast.AsyncFunctionDef))
}
worklist: list[ast.AST] = [
m for m in engine_cls.body if isinstance(m, (ast.FunctionDef, ast.AsyncFunctionDef))
]
visited: set[int] = set()
while worklist:
fn = worklist.pop()
if id(fn) in visited:
continue
visited.add(id(fn))
for node in ast.walk(fn):
_reject_forbidden_node(node)
if isinstance(node, ast.Call) and isinstance(node.func, ast.Name):
target = module_funcs.get(node.func.id)
if target is not None:
worklist.append(target)
def _validate_signal_engine_source(file_path: Path) -> None:
"""Reject import-time executable statements before loading signal_engine.py."""
try:
@@ -253,6 +496,10 @@ def _validate_signal_engine_source(file_path: Path) -> None:
f"Executable top-level statement {type(node).__name__} is not allowed"
)
# Deep pass: the structural loop above only guards import-time execution;
# this walks the code that runs on SignalEngine().generate() (VT-001).
_scan_runtime_reachable(tree)
def _validate_signal_engine_class(engine_cls) -> None:
"""Pre-flight check: SignalEngine can be instantiated with no args and has generate()."""
@@ -284,6 +531,8 @@ _MARKET_TO_SOURCE = {
"a_share": "tushare",
"us_equity": "yfinance",
"hk_equity": "yfinance",
"india_equity": "yahoo",
"kr_equity": "pykrx",
"crypto": "okx",
"futures": "tushare",
"fund": "tushare",
@@ -370,6 +619,252 @@ def _normalize_codes(codes: List[str], source: str) -> List[str]:
return codes
def _restore_original_codes(
data_map: dict,
original_codes: List[str],
normalized_codes: List[str],
) -> dict:
"""Map provider-normalized result keys back to requested symbols."""
aliases = dict(zip(normalized_codes, original_codes))
return {aliases.get(code, code): frame for code, frame in data_map.items()}
def _columns_required_from_factor_spec(spec: Any) -> list[str]:
"""Extract ``columns_required`` from supported factor spec shapes.
Args:
spec: Factor metadata as a dict, an Alpha-like object with ``meta``, an
object with ``columns_required``, or a raw string alpha id.
Returns:
Declared panel columns, or an empty list when the shape has none.
"""
if isinstance(spec, dict):
meta = spec.get("meta")
if isinstance(meta, dict):
return [str(c) for c in meta.get("columns_required", [])]
return [str(c) for c in spec.get("columns_required", [])]
meta = getattr(spec, "meta", None)
if isinstance(meta, dict):
return [str(c) for c in meta.get("columns_required", [])]
columns = getattr(spec, "columns_required", None)
if columns is not None:
return [str(c) for c in columns]
return []
def _selected_factor_specs(config: dict) -> list[Any]:
"""Return selected factor metadata configured for the run.
The current runner has no dedicated factor-zoo execution path, so this
accepts the shapes used by callers that already know factor metadata
(``selected_factors``/``factors``/``alpha_metas``) and alpha-id lists that
can be resolved through the registry.
Args:
config: Backtest config.
Returns:
Factor specs or metadata dictionaries.
"""
specs: list[Any] = []
for key in ("selected_factors", "factors", "alpha_metas"):
raw = config.get(key)
if isinstance(raw, list):
specs.extend(raw)
elif raw:
specs.append(raw)
alpha_ids: list[str] = []
for key in ("alpha_ids", "factor_ids", "alphas"):
raw = config.get(key)
if raw is None:
continue
if isinstance(raw, str):
alpha_ids.append(raw)
else:
alpha_ids.extend(str(item) for item in raw)
if alpha_ids:
from src.factors.registry import get_default_registry
registry = get_default_registry()
for alpha_id in alpha_ids:
try:
specs.append(registry.get(alpha_id).meta)
except KeyError:
logger.warning("selected alpha_id %r is not registered; skipping", alpha_id)
return specs
def _fund_columns_required(selected_factors: Iterable[Any]) -> list[str]:
"""Collect requested ``fund:*`` panel columns from selected factors.
Args:
selected_factors: Factor metadata/spec objects.
Returns:
Stable, de-duplicated ``fund:*`` column names.
"""
seen: set[str] = set()
out: list[str] = []
for spec in selected_factors:
for column in _columns_required_from_factor_spec(spec):
if not column.startswith(_FUND_PREFIX) or column in seen:
continue
seen.add(column)
out.append(column)
return out
def _build_price_panel(data_map: dict[str, pd.DataFrame]) -> dict[str, pd.DataFrame]:
"""Convert ``code -> OHLCV frame`` rows into the factor panel shape.
Args:
data_map: Backtest loader output.
Returns:
``column -> dates x symbols`` panel for price columns present in the
data map.
"""
panel: dict[str, pd.DataFrame] = {}
for column in _PRICE_PANEL_COLUMNS:
series_by_symbol = {
symbol: frame[column]
for symbol, frame in data_map.items()
if column in frame.columns
}
if series_by_symbol:
panel[column] = pd.DataFrame(series_by_symbol)
return panel
def _nan_fundamental_frame(
index: pd.DatetimeIndex,
symbols: list[str],
) -> pd.DataFrame:
"""Build an all-NaN fundamental panel frame."""
return pd.DataFrame(float("nan"), index=index, columns=symbols)
def _inject_fundamental_panel(
panel: dict[str, pd.DataFrame],
*,
symbols: list[str],
fund_columns: Iterable[str],
start: str,
end: str,
freq: str = "ttm",
pit: bool = True,
source: str = "auto",
index: pd.DatetimeIndex | None = None,
) -> dict[str, pd.DataFrame]:
"""Inject required fundamental fields into a factor panel.
Args:
panel: Existing factor panel keyed by column name.
symbols: Symbols to load.
fund_columns: Requested ``fund:*`` columns.
start: Start date string.
end: End date string.
freq: Fundamental frequency. Defaults to ``ttm``.
pit: Whether point-in-time loading is enforced.
source: Fundamental data source route.
index: Optional price index to align to. Defaults to ``panel["close"]``.
Returns:
The same panel dictionary with ``fund:<field>`` frames added.
"""
fields = [column[len(_FUND_PREFIX):] for column in fund_columns if column.startswith(_FUND_PREFIX)]
fields = list(dict.fromkeys(fields))
if not fields:
return panel
price_index = index
if price_index is None:
close = panel.get("close")
price_index = close.index if close is not None else pd.DatetimeIndex([])
try:
from backtest.loaders.fundamentals_loader import load_fundamental_panel
loaded = load_fundamental_panel(
symbols=symbols,
fields=fields,
start=start,
end=end,
freq=freq,
pit=pit,
source=source,
index=price_index,
)
except Exception as exc: # noqa: BLE001 - data-source failure must not kill backtest
logger.warning(
"fundamental panel load failed for fields=%s symbols=%s: %s; injecting NaN frames",
fields,
symbols,
exc,
exc_info=True,
)
loaded = {}
for field in fields:
frame = loaded.get(field)
if not isinstance(frame, pd.DataFrame):
frame = _nan_fundamental_frame(price_index, symbols)
else:
frame = frame.reindex(index=price_index, columns=symbols)
panel[f"{_FUND_PREFIX}{field}"] = frame
return panel
def _project_panel_fields_to_data_map(
data_map: dict[str, pd.DataFrame],
panel: dict[str, pd.DataFrame],
fund_columns: Iterable[str],
) -> dict[str, pd.DataFrame]:
"""Copy injected panel fields back to per-symbol backtest frames."""
out = {symbol: frame.copy() for symbol, frame in data_map.items()}
for column in fund_columns:
frame = panel.get(column)
if frame is None:
continue
for symbol, symbol_frame in out.items():
if symbol in frame.columns:
symbol_frame[column] = frame[symbol].reindex(symbol_frame.index)
else:
symbol_frame[column] = float("nan")
return out
def _maybe_inject_fundamentals_for_factor_panel(
data_map: dict[str, pd.DataFrame],
config: dict,
) -> dict[str, pd.DataFrame]:
"""Inject ``fund:*`` factor dependencies when selected factors request them."""
selected_factors = _selected_factor_specs(config)
fund_columns = _fund_columns_required(selected_factors)
if not fund_columns:
return data_map
panel = _build_price_panel(data_map)
close = panel.get("close")
price_index = close.index if close is not None else pd.DatetimeIndex([])
symbols = list(data_map)
_inject_fundamental_panel(
panel,
symbols=symbols,
fund_columns=fund_columns,
start=config.get("start_date", ""),
end=config.get("end_date", ""),
freq="ttm",
pit=True,
source="auto",
index=price_index,
)
return _project_panel_fields_to_data_map(data_map, panel, fund_columns)
# --- Main entry ---
def main(run_dir: Path) -> None:
@@ -442,50 +937,18 @@ def main(run_dir: Path) -> None:
print(json.dumps({"error": f"SignalEngine interface error: {exc}"}))
sys.exit(1)
# Data: auto split vs single loader
fetch_result = fetch_data_map(config)
data_map = fetch_result.data_map
codes = fetch_result.codes
source = fetch_result.source
loader = fetch_result.loader
config["codes"] = codes
config["_run_card_effective_sources"] = fetch_result.effective_sources
interval = config.get("interval", "1D")
if source == "auto":
data_map = _fetch_auto(codes, config, interval)
else:
codes = _normalize_codes(codes, source)
config["codes"] = codes
LoaderCls = _get_loader(source)
loader = LoaderCls()
data_map = loader.fetch(
codes,
config.get("start_date", ""),
config.get("end_date", ""),
fields=config.get("extra_fields") or None,
interval=interval,
)
# Runtime fallback: try next sources in chain when primary returns empty
if not data_map and codes:
market = _detect_market(codes[0])
for fb_name in FALLBACK_CHAINS.get(market, []):
if fb_name == source or fb_name not in LOADER_REGISTRY:
continue
fb_loader = LOADER_REGISTRY[fb_name]()
if not fb_loader.is_available():
continue
fb_codes = _normalize_codes(codes, fb_name)
data_map = fb_loader.fetch(
fb_codes, config.get("start_date", ""),
config.get("end_date", ""), interval=interval,
)
if data_map:
logger.info("Runtime fallback: %s -> %s", source, fb_name)
source = fb_name
loader = fb_loader
break
if not data_map:
print(json.dumps({"error": "No data fetched"}))
sys.exit(1)
if source == "auto":
config["_run_card_effective_sources"] = sorted(_group_codes_by_source(codes))
else:
config["_run_card_effective_sources"] = [source]
data_map = _maybe_inject_fundamentals_for_factor_panel(data_map, config)
# Engine
engine_type = config.get("engine", "daily")
@@ -501,9 +964,10 @@ def main(run_dir: Path) -> None:
else:
bars_per_year = calc_bars_per_year(interval, effective_source)
# Auto mode: wrap preloaded data in a dummy loader
if source == "auto":
loader = _AutoLoader(data_map)
# Every source has already been fetched, sanitized, and enriched above.
# Reuse that exact snapshot so provider costs and run-card provenance stay
# aligned with the data consumed by the engine.
loader = _AutoLoader(data_map)
if engine_type == "options":
from backtest.engines.options_portfolio import run_options_backtest
@@ -550,6 +1014,19 @@ def _create_market_engine(source: str, config: dict, codes: List[str]):
from backtest.engines.forex import ForexEngine
return ForexEngine(config)
# India equity routing — must precede source-based routing because India's
# effective source is ``yahoo``, which has no Wave-1 branch and would
# otherwise fall through to the crypto default.
if "india_equity" in markets:
from backtest.engines.india_equity import IndiaEquityEngine
return IndiaEquityEngine(config)
# Korea equity routing — same reason as India: its effective source
# (``pykrx``) has no Wave-1 branch and would fall through to the default.
if "kr_equity" in markets:
from backtest.engines.korea_equity import KoreaEquityEngine
return KoreaEquityEngine(config)
# Original routing (Wave 1)
if source in ("okx", "ccxt"):
from backtest.engines.crypto import CryptoEngine
@@ -566,6 +1043,13 @@ def _create_market_engine(source: str, config: dict, codes: List[str]):
market = _detect_submarket(codes)
return GlobalEquityEngine(config, market=market)
else:
# Sources without a dedicated branch (local, stooq, ...): follow the
# instrument market rather than the loader name, so e.g. a local
# AAPL.US dataset gets US-equity execution rules instead of crypto.
if markets & {"us_equity", "hk_equity"}:
from backtest.engines.global_equity import GlobalEquityEngine
market = _detect_submarket(codes)
return GlobalEquityEngine(config, market=market)
from backtest.engines.crypto import CryptoEngine
return CryptoEngine(config)
@@ -594,7 +1078,12 @@ def _fetch_auto(codes: List[str], config: dict, interval: str = "1D") -> dict:
Args:
codes: All symbols.
config: Backtest config dict.
config: Backtest config dict. On success the loaders that actually
returned rows are recorded under the private key
``_actual_sources`` so the caller can report true provenance
instead of the symbol-pattern guess (``_group_codes_by_source``
names the *head* of each chain, which lies whenever that head is an
optional package that is not installed).
interval: Bar interval string.
Returns:
@@ -602,6 +1091,7 @@ def _fetch_auto(codes: List[str], config: dict, interval: str = "1D") -> dict:
"""
market_groups = _group_codes_by_market(codes)
merged = {}
served_by: set[str] = set()
start_date = config.get("start_date", "")
end_date = config.get("end_date", "")
@@ -618,29 +1108,187 @@ def _fetch_auto(codes: List[str], config: dict, interval: str = "1D") -> dict:
src_name = getattr(loader, "name", "unknown")
normalized_codes = _normalize_codes(market_codes, src_name)
fields = config.get("extra_fields") if src_name == "tushare" else None
result = loader.fetch(normalized_codes, start_date, end_date, fields=fields, interval=interval)
result = loader.fetch(
normalized_codes,
start_date,
end_date,
fields=fields,
interval=interval,
)
market_result = _restore_original_codes(
result, market_codes, normalized_codes
)
if market_result:
served_by.add(src_name)
missing = [code for code in market_codes if code not in market_result]
# Runtime fallback: try remaining sources when primary returns empty
if not result:
for fb_name in FALLBACK_CHAINS.get(market, []):
if fb_name == src_name or fb_name not in LOADER_REGISTRY:
continue
fb_loader = LOADER_REGISTRY[fb_name]()
if not fb_loader.is_available():
continue
fb_codes = _normalize_codes(market_codes, fb_name)
result = fb_loader.fetch(fb_codes, start_date, end_date, interval=interval)
if result:
logger.info("Runtime fallback: %s -> %s for %s", src_name, fb_name, market)
break
# Retry only missing symbols so a partial primary response does not
# silently shrink the requested universe or refetch successful data.
for fb_name in FALLBACK_CHAINS.get(market, []):
if not missing:
break
if fb_name == src_name or fb_name not in LOADER_REGISTRY:
continue
fb_loader = LOADER_REGISTRY[fb_name]()
if not fb_loader.is_available():
continue
fb_codes = _normalize_codes(missing, fb_name)
fallback_result = fb_loader.fetch(
fb_codes, start_date, end_date, interval=interval
)
mapped = _restore_original_codes(fallback_result, missing, fb_codes)
if mapped:
market_result.update(mapped)
missing = [code for code in missing if code not in mapped]
served_by.add(str(getattr(fb_loader, "name", fb_name) or fb_name))
logger.info(
"Runtime fallback: %s -> %s for %s", src_name, fb_name, market
)
merged.update(result)
if missing:
raise NoAvailableSourceError(
f"incomplete data for {market}; missing symbols: {missing}"
)
merged.update(market_result)
config["_actual_sources"] = sorted(served_by)
return merged
def fetch_data_map(config: dict) -> DataFetchResult:
"""Fetch and sanitize bars through the canonical loader registry.
This is the shared entry point for backtest execution and reconstruction
consumers. It preserves auto routing and the runtime fallback chain.
Args:
config: Backtest configuration containing codes, dates, source, and interval.
Returns:
Data and effective routing metadata. The input config is not mutated.
"""
config = copy.deepcopy(config)
source = str(config.get("source") or "tushare")
codes = list(config.get("codes") or [])
interval = str(config.get("interval") or "1D")
if source == "auto":
data_map = _fetch_auto(codes, config, interval)
loader: Any = _AutoLoader(data_map)
# Prefer the loaders that actually served rows; the symbol-pattern guess
# is only a fallback for a stubbed/patched fetcher that recorded nothing.
recorded = config.pop("_actual_sources", None)
used_sources: list[str] = [
str(name) for name in recorded or [] if str(name).strip()
] or sorted(_group_codes_by_source(codes))
else:
codes = _normalize_codes(codes, source)
primary_source = source
loader = _get_loader(source)()
# ``_get_loader`` may hand back a *different* loader when the requested
# one is unavailable (e.g. an optional package like pykrx is missing, so
# the kr_equity chain resolves to yahoo). Record who actually served the
# bars, never the name that was asked for — a run card that claims
# ``pykrx`` while Yahoo supplied the data is a provenance lie.
served_by = str(getattr(loader, "name", source) or source)
if served_by != source:
logger.warning(
"source=%s is unavailable; %s served this request",
source,
served_by,
)
data_map = loader.fetch(
codes,
config.get("start_date", ""),
config.get("end_date", ""),
fields=config.get("extra_fields") or None,
interval=interval,
)
used_sources = [served_by] if data_map else []
missing = [code for code in codes if code not in data_map]
if missing:
logger.warning(
"source=%s returned data for %d/%d symbols; missing: %s",
source,
len(data_map),
len(codes),
missing,
)
if missing:
market = _detect_market(codes[0])
for fallback_source in FALLBACK_CHAINS.get(market, []):
if not missing:
break
if (
fallback_source == primary_source
or fallback_source not in LOADER_REGISTRY
):
continue
fallback_loader = LOADER_REGISTRY[fallback_source]()
if not fallback_loader.is_available():
continue
fallback_codes = _normalize_codes(missing, fallback_source)
fallback_result = fallback_loader.fetch(
fallback_codes,
config.get("start_date", ""),
config.get("end_date", ""),
interval=interval,
)
mapped = _restore_original_codes(
fallback_result, missing, fallback_codes
)
if mapped:
data_map.update(mapped)
missing = [code for code in missing if code not in mapped]
fb_served_by = str(
getattr(fallback_loader, "name", fallback_source)
or fallback_source
)
if not used_sources:
source = fb_served_by
loader = fallback_loader
used_sources.append(fb_served_by)
logger.info(
"Runtime fallback: %s -> %s", primary_source, fb_served_by
)
if missing:
raise NoAvailableSourceError(
f"incomplete data for source={primary_source}; missing symbols: {missing}"
)
data_map = _sanitize_data_map(data_map)
return DataFetchResult(
data_map=data_map,
codes=codes,
source=source,
loader=loader,
effective_sources=used_sources,
)
def _sanitize_data_map(data_map: dict) -> dict:
"""Drop structurally-invalid OHLC bars from every fetched frame.
Each loader only drops NaN rows, so a bar that violates the OHLC
invariants (``high < low``, a non-positive price, or a high/low that fails
to bracket open/close) can still reach the backtest and surface as NaN/inf
metrics. Applying :func:`validate_ohlc` here — the single point every
fetched map converges through — guards every source uniformly (``auto``,
single-source, runtime fallback, and any future loader), so the per-loader
checks no longer have to be added one at a time.
Args:
data_map: ``code -> DataFrame`` map as returned by a loader fetch.
Returns:
The same mapping with each frame's invalid bars removed.
"""
return {code: validate_ohlc(frame) for code, frame in data_map.items()}
class _AutoLoader:
"""Dummy loader for auto mode: returns pre-fetched data maps."""
"""Loader adapter that returns a pre-fetched data map."""
def __init__(self, data_map: dict):
self._data = data_map
agent/backtest/validation.py
+137 -46
View File
@@ -11,6 +11,9 @@ is present, or invoked directly on backtest outputs.
from __future__ import annotations
import json
import math
from numbers import Integral, Real
from pathlib import Path
from typing import Any, Dict, List
@@ -44,6 +47,13 @@ def monte_carlo_test(
Dict with actual_sharpe, p_value_sharpe, actual_max_dd,
p_value_max_dd, simulated_sharpes (percentiles).
"""
if isinstance(n_simulations, bool) or not isinstance(n_simulations, Integral) or n_simulations < 1:
return {
"error": f"n_simulations must be >= 1, got {n_simulations}",
"p_value_sharpe": 1.0,
}
if isinstance(seed, bool) or not isinstance(seed, Integral) or seed < 0:
return {"error": f"seed must be >= 0, got {seed}", "p_value_sharpe": 1.0}
if len(trades) < 3:
return {"error": "need at least 3 trades", "p_value_sharpe": 1.0}
@@ -54,9 +64,15 @@ def monte_carlo_test(
sharpe_count = 0
dd_count = 0
sim_sharpes = []
# The full path matrix feeds the fan-chart payload; skip it for
# pathological sizes so a huge run cannot balloon memory or the JSON.
keep_paths = n_simulations * len(pnls) <= 2_000_000
sim_equities = np.empty((n_simulations, len(pnls))) if keep_paths else None
for _ in range(n_simulations):
for i in range(n_simulations):
shuffled = rng.permutation(pnls)
if sim_equities is not None:
sim_equities[i] = initial_capital + np.cumsum(shuffled)
sim = _path_metrics(shuffled, initial_capital)
sim_sharpes.append(sim["sharpe"])
if sim["sharpe"] >= actual["sharpe"]:
@@ -65,7 +81,7 @@ def monte_carlo_test(
dd_count += 1
sim_arr = np.array(sim_sharpes)
return {
result = {
"actual_sharpe": round(actual["sharpe"], 4),
"actual_max_dd": round(actual["max_dd"], 4),
"p_value_sharpe": round(sharpe_count / n_simulations, 4),
@@ -76,7 +92,25 @@ def monte_carlo_test(
"simulated_sharpe_p95": round(float(np.percentile(sim_arr, 95)), 4),
"n_simulations": n_simulations,
"n_trades": len(trades),
"sharpe_samples": [round(float(s), 4) for s in sim_sharpes],
}
if sim_equities is not None:
idx = np.unique(np.linspace(0, len(pnls) - 1, min(len(pnls), 400)).astype(int))
sample_rows = np.unique(
np.linspace(0, n_simulations - 1, min(30, n_simulations)).astype(int)
)
result["equity_paths"] = {
"steps": (idx + 1).tolist(),
"initial_capital": round(float(initial_capital), 2),
"actual": np.round((initial_capital + np.cumsum(pnls))[idx], 2).tolist(),
"band_p5": np.round(np.percentile(sim_equities[:, idx], 5, axis=0), 2).tolist(),
"band_p25": np.round(np.percentile(sim_equities[:, idx], 25, axis=0), 2).tolist(),
"band_p50": np.round(np.percentile(sim_equities[:, idx], 50, axis=0), 2).tolist(),
"band_p75": np.round(np.percentile(sim_equities[:, idx], 75, axis=0), 2).tolist(),
"band_p95": np.round(np.percentile(sim_equities[:, idx], 95, axis=0), 2).tolist(),
"samples": np.round(sim_equities[np.ix_(sample_rows, idx)], 2).tolist(),
}
return result
def _path_metrics(pnls: np.ndarray, initial_capital: float) -> Dict[str, float]:
@@ -114,6 +148,18 @@ def bootstrap_sharpe_ci(
Dict with observed_sharpe, ci_lower, ci_upper, median_sharpe,
prob_positive (fraction of samples with Sharpe > 0).
"""
if isinstance(n_bootstrap, bool) or not isinstance(n_bootstrap, Integral) or n_bootstrap < 1:
return {"error": f"n_bootstrap must be >= 1, got {n_bootstrap}"}
if (
isinstance(confidence, bool)
or not isinstance(confidence, Real)
or not math.isfinite(float(confidence))
or not 0.0 < confidence < 1.0
):
return {"error": f"confidence must be in (0, 1), got {confidence}"}
if isinstance(seed, bool) or not isinstance(seed, Integral) or seed < 0:
return {"error": f"seed must be >= 0, got {seed}"}
returns = equity_curve.pct_change().dropna().values
if len(returns) < 5:
return {"error": "need at least 5 return observations"}
@@ -132,7 +178,7 @@ def bootstrap_sharpe_ci(
upper = float(np.percentile(arr, (1 - alpha) * 100))
prob_pos = float(np.mean(arr > 0))
return {
result = {
"observed_sharpe": round(observed, 4),
"ci_lower": round(lower, 4),
"ci_upper": round(upper, 4),
@@ -141,6 +187,9 @@ def bootstrap_sharpe_ci(
"confidence": confidence,
"n_bootstrap": n_bootstrap,
}
if n_bootstrap <= 20_000:
result["sharpe_samples"] = [round(float(s), 4) for s in boot_sharpes]
return result
def _sharpe(returns: np.ndarray, bars_per_year: int = 252) -> float:
@@ -170,6 +219,8 @@ def walk_forward_analysis(
Returns:
Dict with per_window stats, consistency metrics.
"""
if isinstance(n_windows, bool) or not isinstance(n_windows, Integral) or n_windows < 1:
return {"error": f"n_windows must be >= 1, got {n_windows}"}
if len(equity_curve) < n_windows * 2:
return {"error": f"need at least {n_windows * 2} bars for {n_windows} windows"}
@@ -185,10 +236,7 @@ def walk_forward_analysis(
win_end = indices[end_idx - 1]
# Per-window trades
win_trades = [
t for t in trades
if win_start <= t.entry_time <= win_end
]
win_trades = [t for t in trades if win_start <= t.entry_time <= win_end]
# Per-window metrics
ret = float(win_eq.iloc[-1] / win_eq.iloc[0] - 1) if win_eq.iloc[0] > 0 else 0.0
@@ -200,21 +248,20 @@ def walk_forward_analysis(
max_dd = float(dd.min())
win_pnls = [t.pnl for t in win_trades]
win_rate = (
len([p for p in win_pnls if p > 0]) / len(win_pnls)
if win_pnls else 0.0
)
win_rate = len([p for p in win_pnls if p > 0]) / len(win_pnls) if win_pnls else 0.0
windows.append({
"window": i + 1,
"start": str(win_start.date()) if hasattr(win_start, "date") else str(win_start),
"end": str(win_end.date()) if hasattr(win_end, "date") else str(win_end),
"return": round(ret, 6),
"sharpe": round(sharpe, 4),
"max_dd": round(max_dd, 6),
"trades": len(win_trades),
"win_rate": round(win_rate, 4),
})
windows.append(
{
"window": i + 1,
"start": str(win_start.date()) if hasattr(win_start, "date") else str(win_start),
"end": str(win_end.date()) if hasattr(win_end, "date") else str(win_end),
"return": round(ret, 6),
"sharpe": round(sharpe, 4),
"max_dd": round(max_dd, 6),
"trades": len(win_trades),
"win_rate": round(win_rate, 4),
}
)
# Consistency metrics
returns_list = [w["return"] for w in windows]
@@ -266,7 +313,8 @@ def run_validation(
if "monte_carlo" in v_cfg:
mc_cfg = v_cfg["monte_carlo"] if isinstance(v_cfg["monte_carlo"], dict) else {}
results["monte_carlo"] = monte_carlo_test(
trades, initial_capital,
trades,
initial_capital,
n_simulations=mc_cfg.get("n_simulations", 1000),
seed=mc_cfg.get("seed", 42),
)
@@ -274,7 +322,8 @@ def run_validation(
if "bootstrap" in v_cfg:
bs_cfg = v_cfg["bootstrap"] if isinstance(v_cfg["bootstrap"], dict) else {}
results["bootstrap"] = bootstrap_sharpe_ci(
equity_curve, bars_per_year=bars_per_year,
equity_curve,
bars_per_year=bars_per_year,
n_bootstrap=bs_cfg.get("n_bootstrap", 1000),
confidence=bs_cfg.get("confidence", 0.95),
seed=bs_cfg.get("seed", 42),
@@ -283,7 +332,8 @@ def run_validation(
if "walk_forward" in v_cfg:
wf_cfg = v_cfg["walk_forward"] if isinstance(v_cfg["walk_forward"], dict) else {}
results["walk_forward"] = walk_forward_analysis(
equity_curve, trades,
equity_curve,
trades,
n_windows=wf_cfg.get("n_windows", 5),
bars_per_year=bars_per_year,
)
@@ -298,7 +348,12 @@ def _load_equity(run_dir: Path) -> pd.Series:
"""Load equity curve from artifacts/equity.csv."""
path = run_dir / "artifacts" / "equity.csv"
df = pd.read_csv(path, index_col=0, parse_dates=True)
return df["equity"]
for col in ("equity", "nav", "value"):
if col in df.columns:
return df[col]
raise ValueError(
f"equity.csv must contain an equity/nav/value column; got {list(df.columns)}"
)
def _load_trades(run_dir: Path) -> List[TradeRecord]:
@@ -312,21 +367,25 @@ def _load_trades(run_dir: Path) -> List[TradeRecord]:
trades = []
exit_rows = df[df["pnl"] != 0].reset_index(drop=True)
for _, row in exit_rows.iterrows():
trades.append(TradeRecord(
symbol=str(row.get("code", "")),
direction=1 if row.get("side") == "sell" else -1,
entry_price=0.0,
exit_price=float(row.get("price", 0)),
entry_time=pd.Timestamp(row.get("timestamp", "2000-01-01")),
exit_time=pd.Timestamp(row.get("timestamp", "2000-01-01")),
size=float(row.get("qty", 0)),
leverage=1.0,
pnl=float(row.get("pnl", 0)),
pnl_pct=float(row.get("return_pct", 0)),
exit_reason=str(row.get("reason", "signal")),
holding_bars=int(row.get("holding_days", 0)),
commission=0.0,
))
hold = pd.to_numeric(row.get("holding_days", 0), errors="coerce")
holding_bars = 0 if pd.isna(hold) else int(hold)
trades.append(
TradeRecord(
symbol=str(row.get("code", "")),
direction=1 if row.get("side") == "sell" else -1,
entry_price=0.0,
exit_price=float(row.get("price", 0)),
entry_time=pd.Timestamp(row.get("timestamp", "2000-01-01")),
exit_time=pd.Timestamp(row.get("timestamp", "2000-01-01")),
size=float(row.get("qty", 0)),
leverage=1.0,
pnl=float(row.get("pnl", 0)),
pnl_pct=float(row.get("return_pct", 0)),
exit_reason=str(row.get("reason", "signal")),
holding_bars=holding_bars,
commission=0.0,
)
)
return trades
@@ -355,6 +414,41 @@ def _parse_run_dir(argv: List[str]) -> Path:
return run_dir
def _json_safe(value: Any) -> Any:
"""Return a JSON-strict copy of validation results."""
if isinstance(value, np.ndarray):
return [_json_safe(item) for item in value.tolist()]
if isinstance(value, np.generic):
return _json_safe(value.item())
if isinstance(value, float):
return value if math.isfinite(value) else None
if isinstance(value, dict):
return {str(key): _json_safe(item) for key, item in value.items()}
if isinstance(value, (list, tuple)):
return [_json_safe(item) for item in value]
return value
def write_validation_json(path: Path, results: Dict[str, Any]) -> Dict[str, Any]:
"""Write validation results to ``path`` as strict, RFC-8259 JSON.
A validation metric can be non-finite (e.g. a Sharpe computed from a path
whose equity touches zero), and ``json.dumps`` emits bare ``NaN`` /
``Infinity`` tokens for those by default (``allow_nan=True``) — tokens that
strict parsers reject. Sanitize with :func:`_json_safe` (non-finite → null)
and serialize with ``allow_nan=False`` so every writer of
``artifacts/validation.json`` produces the same valid JSON. Returns the
sanitized payload that was written.
"""
safe_results = _json_safe(results)
path.parent.mkdir(parents=True, exist_ok=True)
path.write_text(
json.dumps(safe_results, indent=2, ensure_ascii=False, allow_nan=False) + "\n",
encoding="utf-8",
)
return safe_results
def main(run_dir: Path) -> Dict[str, Any]:
"""Run all three validations on existing backtest artifacts.
@@ -366,8 +460,6 @@ def main(run_dir: Path) -> Dict[str, Any]:
Returns:
Validation results dict.
"""
import json
# Load config for initial_cash
config_path = run_dir / "config.json"
if config_path.exists():
@@ -385,12 +477,11 @@ def main(run_dir: Path) -> Dict[str, Any]:
"walk_forward": walk_forward_analysis(equity, trades),
}
# Write results
out = run_dir / "artifacts" / "validation.json"
out.write_text(json.dumps(results, indent=2, ensure_ascii=False), encoding="utf-8")
safe_results = write_validation_json(out, results)
print(json.dumps(results, indent=2))
return results
print(json.dumps(safe_results, indent=2, allow_nan=False))
return safe_results
if __name__ == "__main__":
agent/cli/_legacy.py
+1276 -69
View File
File diff suppressed because it is too large. Load diff
agent/cli/_version.py
+27 -7
View File
@@ -1,26 +1,46 @@
"""Single source of truth for the CLI version string.
Reads ``vibe-trading-ai``'s installed package metadata when available
(``pip install -e .`` is enough). Falls back to the static constant
below for un-installed checkouts (e.g. running straight from a clone
with ``PYTHONPATH=agent``).
Keep the fallback in sync with ``pyproject.toml`` ``[project] version``.
(``pip install -e .`` is enough). For an un-installed checkout (e.g. running
straight from a clone with ``PYTHONPATH=agent``) it falls back to reading the
version straight out of ``pyproject.toml`` — so ``pyproject.toml`` is the one
and only place the version is ever written. There is deliberately no hardcoded
version constant to drift out of sync on release (issue #156).
"""
from __future__ import annotations
from typing import Final
def _version_from_pyproject() -> str:
"""Read ``[project] version`` from the repo's ``pyproject.toml``.
Returns:
The declared version, or ``"unknown"`` if the file cannot be located
or parsed (only reachable for an un-installed checkout whose tree has
been moved away from its ``pyproject.toml``).
"""
import tomllib
from pathlib import Path
# agent/cli/_version.py -> parents[2] is the repo root holding pyproject.toml.
pyproject = Path(__file__).resolve().parents[2] / "pyproject.toml"
try:
return tomllib.loads(pyproject.read_text(encoding="utf-8"))["project"]["version"]
except (OSError, KeyError, tomllib.TOMLDecodeError):
return "unknown"
try:
from importlib.metadata import PackageNotFoundError, version as _pkg_version
try:
__version__: Final[str] = _pkg_version("vibe-trading-ai")
except PackageNotFoundError:
__version__ = "0.1.8"
__version__ = _version_from_pyproject()
except ImportError: # pragma: no cover — importlib.metadata is stdlib on 3.8+
__version__ = "0.1.8"
__version__ = _version_from_pyproject()
__all__ = ["__version__"]
agent/cli/commands/chat.py
+5 -3
View File
@@ -11,7 +11,6 @@ fallback workflows.
from __future__ import annotations
import os
from typing import Any, Optional
from rich.console import Console
@@ -50,8 +49,11 @@ def cmd_model(ctx: Any = None, *args: str) -> int: # noqa: ARG001 — ctx unuse
_show_settings()
except Exception as exc: # noqa: BLE001 — legacy may be absent on partial install
provider = os.getenv("LANGCHAIN_PROVIDER", "(not set)")
model = os.getenv("LANGCHAIN_MODEL_NAME", "(not set)")
from src.config.accessor import get_env_config
cfg = get_env_config()
provider = cfg.llm.langchain_provider or "(not set)"
model = cfg.llm.langchain_model_name or "(not set)"
console.print(Text(f"Provider: {provider}", style="bold"))
console.print(Text(f"Model: {model}", style="bold"))
console.print(Text(f"(legacy _show_settings unavailable: {exc})", style="dim"))
agent/cli/commands/goal.py
+3 -1
View File
@@ -73,7 +73,9 @@ def _session_id(ctx: Any, *, title: str = "Goal research", create: bool) -> str
existing = getattr(ctx, "session_id", None)
if existing:
return str(existing)
env_session_id = os.getenv("VIBE_GOAL_SESSION_ID")
from src.config.accessor import get_env_config
env_session_id = get_env_config().paths.vibe_goal_session_id
if env_session_id:
return env_session_id
if not create:
agent/cli/commands/institutional/__init__.py
+29
View File
@@ -0,0 +1,29 @@
"""Institutional research workflow slash commands.
This package turns five recurring sell-side / buy-side workflows into explicit,
discoverable slash commands instead of prose buried in a swarm preset:
``/comps`` comparable company analysis
``/dcf`` discounted cash flow
``/attrib`` Brinson-Fachler performance attribution
``/memo`` investment memo
``/earnings`` earnings surprise bridge
``/screen`` systematic idea screen
Layout — deliberately import-cycle free:
``playbooks.py``
Pure data. Imports nothing from :mod:`cli`, so
:mod:`cli.commands.slash_router` can read the command specs at its own
import time without a circular import.
``runner.py``
Rendering + prompt construction + the shared ``run_playbook`` entry.
``comps.py`` / ``dcf.py`` / ``attrib.py`` / ``memo.py`` / ``earnings.py`` /
``screen.py``
One thin ``run(ctx, *args)`` module per command. The REPL dispatcher
(``cli.main._dispatch_slash``) passes the command keyword only for the
modules listed in ``_MULTI_COMMAND_MODULES``; every other handler module
owns exactly one command, so each command gets its own module here.
This ``__init__`` intentionally stays import-free.
"""
agent/cli/commands/institutional/attrib.py
+29
View File
@@ -0,0 +1,29 @@
"""``/attrib`` — Brinson-Fachler performance attribution.
Thin handler: the execution skeleton, the worked numeric example and the
missing-data policy all live in :mod:`.playbooks`, and the render/dispatch
logic in :mod:`.runner`. Running it produces allocation / selection /
interaction per sector, reconciled to the total active return.
"""
from __future__ import annotations
from typing import Any
from .runner import run_playbook
def run(ctx: Any = None, *args: str) -> int:
"""Render or dispatch the ``/attrib`` playbook.
Args:
ctx: Interactive context supplying ``pending_prompt``.
*args: Raw slash-command arguments.
Returns:
Process-style exit code; ``0`` on every user-facing path.
"""
return run_playbook("attrib", ctx, *args)
__all__ = ["run"]
agent/cli/commands/institutional/comps.py
+30
View File
@@ -0,0 +1,30 @@
"""``/comps`` — Comparable company analysis.
Thin handler: the execution skeleton, the worked numeric example and the
missing-data policy all live in :mod:`.playbooks`, and the render/dispatch
logic in :mod:`.runner`. Running it produces peer multiple table, the
subject's premium or discount versus the peer median, and an implied value
range.
"""
from __future__ import annotations
from typing import Any
from .runner import run_playbook
def run(ctx: Any = None, *args: str) -> int:
"""Render or dispatch the ``/comps`` playbook.
Args:
ctx: Interactive context supplying ``pending_prompt``.
*args: Raw slash-command arguments.
Returns:
Process-style exit code; ``0`` on every user-facing path.
"""
return run_playbook("comps", ctx, *args)
__all__ = ["run"]
agent/cli/commands/institutional/dcf.py
+30
View File
@@ -0,0 +1,30 @@
"""``/dcf`` — Discounted cash flow valuation.
Thin handler: the execution skeleton, the worked numeric example and the
missing-data policy all live in :mod:`.playbooks`, and the render/dispatch
logic in :mod:`.runner`. Running it produces value per share, the terminal-
value share of enterprise value, an exit-multiple cross-check and a WACC x g
sensitivity grid.
"""
from __future__ import annotations
from typing import Any
from .runner import run_playbook
def run(ctx: Any = None, *args: str) -> int:
"""Render or dispatch the ``/dcf`` playbook.
Args:
ctx: Interactive context supplying ``pending_prompt``.
*args: Raw slash-command arguments.
Returns:
Process-style exit code; ``0`` on every user-facing path.
"""
return run_playbook("dcf", ctx, *args)
__all__ = ["run"]
agent/cli/commands/institutional/earnings.py
+29
View File
@@ -0,0 +1,29 @@
"""``/earnings`` — Earnings review.
Thin handler: the execution skeleton, the worked numeric example and the
missing-data policy all live in :mod:`.playbooks`, and the render/dispatch
logic in :mod:`.runner`. Running it produces a line-by-line surprise bridge
from revenue to EPS, splitting the recurring beat from the non-recurring one.
"""
from __future__ import annotations
from typing import Any
from .runner import run_playbook
def run(ctx: Any = None, *args: str) -> int:
"""Render or dispatch the ``/earnings`` playbook.
Args:
ctx: Interactive context supplying ``pending_prompt``.
*args: Raw slash-command arguments.
Returns:
Process-style exit code; ``0`` on every user-facing path.
"""
return run_playbook("earnings", ctx, *args)
__all__ = ["run"]
agent/cli/commands/institutional/memo.py
+30
View File
@@ -0,0 +1,30 @@
"""``/memo`` — Investment memo.
Thin handler: the execution skeleton, the worked numeric example and the
missing-data policy all live in :mod:`.playbooks`, and the render/dispatch
logic in :mod:`.runner`. Running it produces thesis, variant view,
probability-weighted scenarios, falsifiable kill criteria and a monitoring
set.
"""
from __future__ import annotations
from typing import Any
from .runner import run_playbook
def run(ctx: Any = None, *args: str) -> int:
"""Render or dispatch the ``/memo`` playbook.
Args:
ctx: Interactive context supplying ``pending_prompt``.
*args: Raw slash-command arguments.
Returns:
Process-style exit code; ``0`` on every user-facing path.
"""
return run_playbook("memo", ctx, *args)
__all__ = ["run"]
agent/cli/commands/institutional/playbooks.py
+654
View File
@@ -0,0 +1,654 @@
"""Playbook definitions for the institutional research slash commands.
Pure data — this module imports nothing from :mod:`cli`, so
:mod:`cli.commands.slash_router` can pull the command specs at its own import
time without a circular import.
Each :class:`Playbook` carries four things that make the command more than a
prompt forwarder:
1. ``steps`` — an ordered execution skeleton where every step names its
inputs, the computation, and the artifact it must produce.
2. ``worked_example`` — a fully arithmetic-consistent numeric walkthrough.
Every number below was computed and reconciled by hand; the totals tie out
(the Brinson decomposition sums exactly to the active return, the earnings
bridge sums exactly to the EPS delta). Both the user and the agent can
reproduce them, which is what makes the skeleton checkable.
3. ``tools`` — the real registered tool names the agent should reach for.
Verified against ``src/tools/*`` (``get_financial_statements``,
``get_stock_profile``, ``get_market_data``, ``get_sec_filings``,
``get_macro_series``, ``screen_market``, ``get_stock_news``,
``get_research_reports``, ``portfolio_risk_xray``).
4. ``gap_policy`` — what to do when a number cannot be fetched. The shared
default forbids filling gaps from model memory.
Numbers in the worked examples are illustrative teaching figures, not
quotes for any real security.
"""
from __future__ import annotations
from dataclasses import dataclass
@dataclass(frozen=True)
class Step:
"""One step of a playbook's execution skeleton.
Attributes:
title: Short imperative label, e.g. ``"Fix the peer set"``.
inputs: What the step consumes (data, prior-step artifacts).
compute: The transformation to perform.
output: The artifact the step must produce before moving on.
"""
title: str
inputs: str
compute: str
output: str
@dataclass(frozen=True)
class Playbook:
"""A single institutional workflow exposed as a slash command.
Attributes:
slug: Slash command keyword, without the leading ``/``.
summary: One-line description shown in ``/help`` and the typeahead.
usage: Usage line, e.g. ``"/dcf <ticker> [horizon=5y] [wacc=9%]"``.
examples: Concrete invocations to show when arguments are missing.
ask: Friendly follow-up question printed when arguments are missing.
objective: One sentence stating what a finished run delivers.
steps: Ordered execution skeleton.
worked_example: Pre-rendered lines of the numeric walkthrough.
tools: Registered tool names the agent should prefer.
aliases: Extra keywords resolving to this command.
"""
slug: str
summary: str
usage: str
examples: tuple[str, ...]
ask: str
objective: str
steps: tuple[Step, ...]
worked_example: tuple[str, ...]
tools: tuple[str, ...]
aliases: tuple[str, ...] = ()
@property
def handler_module(self) -> str:
"""Dotted path of the module exposing ``run(ctx, *args)``."""
return f"cli.commands.institutional.{self.slug}"
# Shared policy text. Every playbook ships both blocks so the rule travels with
# the prompt rather than living only in a system prompt the user cannot see.
GAP_POLICY: tuple[str, ...] = (
"Every figure must carry its source: the tool call and the fiscal period it",
"came from. If a figure cannot be fetched, do NOT substitute a remembered or",
"typical value. Instead: (a) name the missing item explicitly, (b) mark it",
"MISSING in the table, (c) continue the analysis with the evidence actually",
"retrieved, and (d) state which conclusions are now unsupported. A partial",
"analysis with named holes is the correct deliverable; a complete-looking",
"analysis with invented inputs is not.",
)
NOT_ADVICE: str = (
"This is research tooling, not investment advice, and not a recommendation "
"to buy or sell any security. Outputs are analytical artifacts to be "
"reviewed by the user."
)
_COMPS = Playbook(
slug="comps",
summary="Comparable company analysis (peer multiples -> implied range)",
usage="/comps <ticker> [peer ...] [--asof YYYY-MM-DD]",
examples=("/comps AAPL", "/comps 0700.HK 9988.HK BABA", "/comps NVDA AMD AVGO TSM"),
ask="Which company should I run comps on? Peers are optional — I will build the peer set if you omit them.",
objective=(
"Produce a peer multiple table, the subject's premium/discount versus the "
"peer median, and an implied value RANGE with the operating reason for the gap."
),
steps=(
Step(
title="Fix the peer set",
inputs="Subject ticker; user-supplied peers, or screen_market by sector and size when none are given.",
compute=(
"Keep names that share the subject's revenue driver and sit inside a "
"0.25x-4x market-cap band. Reject everything else."
),
output="Peer table listing each INCLUDED name with its inclusion reason, and each REJECTED name with its rejection reason.",
),
Step(
title="Pull the raw inputs",
inputs="get_stock_profile + get_financial_statements (latest annual and quarterly) for the subject and every peer.",
compute=(
"Enterprise value = market cap + total debt - cash. Then EV/Sales, "
"EV/EBITDA and P/E on trailing-twelve-month figures."
),
output="One row per name, every cell tagged with the fiscal period it was taken from.",
),
Step(
title="Normalize",
inputs="The raw multiple table from step 2.",
compute=(
"Mark negative or non-meaningful EBITDA / earnings as 'nm' — never as "
"zero, and never dropped silently. Then compute the median and the "
"25th/75th percentile per multiple."
),
output="Peer median + interquartile range, plus the COUNT of names actually used in each column.",
),
Step(
title="Position the subject",
inputs="Subject multiples versus the peer median.",
compute="Premium/discount in percent for each multiple.",
output=(
"One verdict line per multiple, plus the two operating metrics "
"(revenue growth, margin) that explain the largest gap. A discount is "
"not a conclusion until it is explained."
),
),
Step(
title="Implied value range",
inputs="Peer 25th / median / 75th percentile multiple x the subject's own metric.",
compute="Implied EV -> minus net debt -> implied equity -> divide by diluted shares.",
output="A RANGE with the spot price marked inside or outside it. Never a single point estimate.",
),
),
worked_example=(
"Subject: TTM revenue 4,000 | TTM EBITDA 800 (20.0% margin) | debt 900 |",
"cash 400 (net debt 500) | 250m diluted shares | price 22.00",
"",
" market cap = 250 x 22.00 = 5,500",
" EV = 5,500 + 900 - 400 = 6,000",
" EV/EBITDA = 6,000 / 800 = 7.50x",
"",
"Peer EV/EBITDA sorted: 8.00, 9.20, 10.40, 11.60 (n=4, 0 marked nm)",
" P25 = 8.00 + 0.75 x (9.20 - 8.00) = 8.90x",
" median = 9.20 + 0.50 x (10.40 - 9.20) = 9.80x",
" P75 = 10.40 + 0.25 x (11.60 - 10.40) = 10.70x",
"",
"Position: 7.50 / 9.80 - 1 = -23.5% (discount)",
"",
"Implied equity value per share (EV - net debt 500, / 250 shares):",
" at P25 8.90 x 800 = 7,120 -> 6,620 = 26.48",
" at median 9.80 x 800 = 7,840 -> 7,340 = 29.36",
" at P75 10.70 x 800 = 8,560 -> 8,060 = 32.24",
" spot 22.00 sits BELOW the P25 case; median implies +33.5%",
"",
"Step 4 verdict (the part that is actually analysis):",
" subject revenue growth 4% vs peer median 11%. A -23.5% EV/EBITDA",
" discount on 7pp less growth is roughly the fair penalty, so the",
" gap is EXPLAINED, not an opportunity. The comp set only becomes a",
" buy case if the growth gap is closing — check the last 4 quarters",
" of year-over-year revenue before concluding anything.",
),
tools=("get_stock_profile", "get_financial_statements", "screen_market", "get_market_data"),
aliases=("peers",),
)
_DCF = Playbook(
slug="dcf",
summary="Discounted cash flow valuation with sensitivity grid",
usage="/dcf <ticker> [horizon=5] [wacc=9%] [g=2.5%]",
examples=("/dcf MSFT", "/dcf 600519.SH horizon=10 g=3%", "/dcf TSLA wacc=11%"),
ask="Which company should I value? Optionally pass horizon, wacc and terminal growth, e.g. /dcf MSFT horizon=5 wacc=9% g=2.5%.",
objective=(
"Produce an unlevered DCF: value per share, the terminal-value share of "
"enterprise value, an exit-multiple cross-check, and a WACC x g sensitivity grid."
),
steps=(
Step(
title="Anchor the base year",
inputs="get_financial_statements — last 3 fiscal years plus trailing twelve months. get_sec_filings for the 10-K when a line item is ambiguous.",
compute="Revenue, EBIT, effective tax rate, D&A, capex, change in net working capital. Then unlevered FCF = EBIT x (1 - tax) + D&A - capex - dNWC.",
output="Base-year FCF with every line traced to a specific filing period.",
),
Step(
title="Set the drivers",
inputs="History from step 1, plus management guidance if get_stock_news or get_sec_filings surfaces it.",
compute="Revenue growth path, EBIT margin path, capex and D&A as a percent of revenue converging by the final year, dNWC as a percent of the revenue increment.",
output="A driver table where every cell is labelled FILED, GUIDED, or ASSUMPTION. No unlabelled numbers.",
),
Step(
title="Discount rate",
inputs="get_macro_series for the risk-free rate (e.g. DGS10); beta, equity risk premium, cost of debt, target debt weight.",
compute="WACC = E/V x (rf + beta x ERP) + D/V x kd x (1 - tax).",
output="WACC with each input tagged sourced or assumed. The equity risk premium is ALWAYS an assumption — say so.",
),
Step(
title="Terminal value, computed two ways",
inputs="Final-year FCF and final-year EBITDA.",
compute="Gordon growth TV = FCF_N x (1+g) / (WACC - g), with g capped at long-run nominal GDP. Separately, exit-multiple TV = EV/EBITDA x EBITDA_N.",
output="Both terminal values, the implied exit multiple from the Gordon method, and whether it falls inside the peer range from /comps.",
),
Step(
title="Bridge to value per share",
inputs="Discounted explicit-horizon FCF plus discounted terminal value.",
compute="Enterprise value - net debt + non-operating assets - minority interest = equity value; divide by diluted shares.",
output="Value per share, percent versus spot, and the terminal-value share of EV.",
),
Step(
title="Sensitivity, then the honest read",
inputs="Base case from step 5.",
compute="3x3 grid: WACC +/-100bp and terminal g +/-50bp.",
output="The grid, the single driver that moves value most, and the WACC/g pair the CURRENT market price implies. That last number is the useful one.",
),
),
worked_example=(
"Base year: revenue 1,000 | EBIT margin 15% | tax 25% | D&A 60 | capex 80 | dNWC 10",
" FCF0 = 150 x 0.75 + 60 - 80 - 10 = 82.5",
"Drivers: revenue +8%/yr for 5y | margin flat 15% | D&A 6% of revenue",
" capex 8% of revenue | dNWC 10% of the revenue increment",
"",
" yr revenue EBIT NOPAT D&A capex dNWC FCF DF@9% PV",
" 1 1,080 162.0 121.5 64.8 86.4 8.0 91.9 0.9174 84.31",
" 2 1,166 175.0 131.2 70.0 93.3 8.6 99.3 0.8417 83.54",
" 3 1,260 189.0 141.7 75.6 100.8 9.3 107.2 0.7722 82.77",
" 4 1,360 204.1 153.1 81.6 108.8 10.1 115.8 0.7084 82.02",
" 5 1,469 220.4 165.3 88.2 117.6 10.9 125.0 0.6499 81.26",
" sum PV(FCF) = 413.9",
"",
"Terminal value, g = 2.5%: (carry the unrounded FCF5 = 125.03, not the",
"displayed 125.0 — rounding the terminal driver moves EV by ~0.4)",
" TV = 125.03 x 1.025 / (0.09 - 0.025) = 1,971.6",
" PV(TV) = 1,971.6 x 0.6499 = 1,281.4",
" EV = 413.9 + 1,281.4 = 1,695.3",
" equity = 1,695.3 - net debt 200 = 1,495.3",
" per share, 100m shares = 14.95 (spot 12.50, +19.6%)",
"",
"Two mandatory cross-checks:",
" TV share of EV = 1,281.4 / 1,695.3 = 75.6%",
" -> above ~75% the answer IS the terminal assumption, so the exit",
" multiple check below carries the conclusion, not the 5y forecast.",
" implied exit EV/EBITDA = 1,971.6 / (220.4 + 88.2 = 308.6) = 6.39x",
" -> inside a 6-9x peer range, so the Gordon g is not smuggling in",
" a re-rating. If it landed at 14x, the model would be broken.",
"",
"Sensitivity corners (value per share):",
" WACC 8.0%, g 2.5%: PV(FCF) 425.5 + PV(TV) 1,586.0 -> equity 1,811.5 = 18.12",
" WACC 9.0%, g 2.5%: base case = 14.95",
" WACC 10.0%, g 2.0%: PV(FCF) 402.8 + PV(TV) 989.8 -> equity 1,192.6 = 11.93",
"",
"The deliverable is NOT 'worth 14.95'. It is: the market's 12.50 sits",
"just above the WACC 10% / g 2.0% corner, so the price is already",
"discounting a ~10% cost of capital and near-zero real terminal growth.",
"The question to research next is whether that is too harsh.",
),
tools=("get_financial_statements", "get_sec_filings", "get_macro_series", "get_stock_profile"),
)
_ATTRIB = Playbook(
slug="attrib",
summary="Brinson-Fachler attribution (allocation vs selection)",
usage="/attrib <portfolio-or-holdings> [--benchmark SPY] [--from YYYY-MM-DD] [--to YYYY-MM-DD]",
examples=(
"/attrib my portfolio --benchmark SPY --from 2026-01-01",
"/attrib AAPL:0.3 MSFT:0.4 XOM:0.3 --benchmark SPY",
"/attrib holdings.csv --benchmark 000300.SH",
),
ask="Which portfolio should I attribute, and against which benchmark? e.g. /attrib holdings.csv --benchmark SPY --from 2026-01-01",
objective=(
"Decompose active return into allocation, selection and interaction per "
"sector, reconcile the decomposition to the total active return, and state "
"whether the alpha came from the source the mandate promised."
),
steps=(
Step(
title="Fix the weights",
inputs="Portfolio holdings and weights at the START of the window; benchmark weights; a sector map for both.",
compute="Normalize both weight vectors to 100%. Assign every name a sector.",
output="Two weight vectors that each sum to 100%, plus the unclassified residual named explicitly rather than dumped into 'Other'.",
),
Step(
title="Get returns",
inputs="get_market_data over the window for every holding, every benchmark constituent (or the benchmark's own sector series), and the benchmark itself.",
compute="Sector return = weight-averaged constituent return, computed separately for the portfolio (Rp_i) and the benchmark (Rb_i).",
output="Rp_i and Rb_i per sector, plus total Rp and Rb.",
),
Step(
title="Decompose",
inputs="Weights from step 1 and returns from step 2.",
compute=(
"Per sector i: allocation = (wp_i - wb_i) x (Rb_i - Rb); "
"selection = wb_i x (Rp_i - Rb_i); "
"interaction = (wp_i - wb_i) x (Rp_i - Rb_i)."
),
output="A per-sector table with the three effects and their column totals.",
),
Step(
title="Reconcile — this step is a gate",
inputs="Column totals from step 3 and the total active return Rp - Rb.",
compute="sum(allocation + selection + interaction) versus (Rp - Rb).",
output=(
"The residual. If it exceeds 1bp, STOP and report the gap and its "
"likely cause (intra-period trades, cash, corporate actions, FX). Do "
"not present an attribution table that does not tie out."
),
),
Step(
title="Attribute the residual",
inputs="The gap from step 4 plus the trade log if available.",
compute="Split into trading/timing, cash drag, and FX translation.",
output="Each named with a number. Whatever remains is reported as unexplained — never absorbed into selection.",
),
Step(
title="Mandate verdict",
inputs="The allocation-versus-selection split.",
compute="Share of active return from each source.",
output="A one-line verdict on whether the return came from the skill the mandate is paid for, and whether the top contributor is repeatable or a single lucky tilt.",
),
),
worked_example=(
"Benchmark: Tech wb 50%, Rb 10.0% | Health wb 30%, Rb 4.0% | Energy wb 20%, Rb -2.0%",
"Portfolio: Tech wp 60%, Rp 12.0% | Health wp 30%, Rp 3.0% | Energy wp 10%, Rp -1.0%",
"",
" Rb = 0.50x10.0 + 0.30x4.0 + 0.20x(-2.0) = 5.80%",
" Rp = 0.60x12.0 + 0.30x3.0 + 0.10x(-1.0) = 8.00%",
" active return = +2.20%",
"",
" sector alloc (wp-wb)(Rb_i-Rb) selection wb(Rp_i-Rb_i) interaction",
" Tech (0.10)(10.0-5.80)=+0.420 (0.50)(12.0-10.0)=+1.000 +0.200",
" Health (0.00)(4.0-5.80) = 0.000 (0.30)(3.0- 4.0)=-0.300 0.000",
" Energy (-0.10)(-2.0-5.80)=+0.780 (0.20)(-1.0-(-2.0))=+0.200 -0.100",
" total +1.200 +0.900 +0.100",
"",
" reconciliation: 1.200 + 0.900 + 0.100 = 2.200 = active PASS",
"",
"Step 6 verdict (the part that is actually analysis):",
" 55% of the +220bp is ALLOCATION, and +78bp of that is a single",
" Energy underweight in a sector that fell 2%. Selection contributed",
" +90bp, of which +100bp is Tech stock picking partly given back by",
" -30bp in Health. For a stock-selection mandate this is an OFF-MANDATE",
" quarter: the headline beat is mostly one sector tilt, which is one",
" decision, not a repeatable process. Report it that way.",
),
tools=("get_market_data", "portfolio_risk_xray", "get_sector_info", "get_stock_profile"),
aliases=("attribution",),
)
_MEMO = Playbook(
slug="memo",
summary="Investment memo — thesis, variant view, scenarios, kill criteria",
usage="/memo <ticker> [long|short] [horizon=12m]",
examples=("/memo NVDA long", "/memo BABA short horizon=6m", "/memo 005930.KS"),
ask="Which company should the memo cover? Optionally say long or short and a horizon, e.g. /memo NVDA long horizon=12m.",
objective=(
"Produce a decision-ready memo: a one-sentence thesis, the variant view, "
"probability-weighted scenarios, falsifiable kill criteria, and the "
"quarterly monitoring set."
),
steps=(
Step(
title="Thesis and variant view",
inputs="get_research_reports and get_stock_news for what consensus currently believes.",
compute="State consensus in one sentence, your view in one sentence, and the specific fact or mechanism that separates them.",
output="If you cannot name what you believe that consensus does not, there is no memo — say so and stop.",
),
Step(
title="Business and unit economics",
inputs="get_stock_profile, get_financial_statements, get_sec_filings (segment disclosure).",
compute="Revenue drivers (volume x price x mix), contribution margin, what one marginal unit earns and how that trends.",
output="A driver tree where the top line decomposes into things that can be independently observed.",
),
Step(
title="The numbers",
inputs="Three years of history plus your forecast.",
compute="Growth, margins, returns on capital, cash conversion, leverage.",
output="A table where each historical cell cites a filing period and each forecast cell cites the step-2 driver it comes from.",
),
Step(
title="Valuation with two independent anchors",
inputs="A /dcf run and a /comps run.",
compute="Bear / base / bull value per share, with an explicit probability on each. Expected value = sum(probability x value).",
output="The three cases, the expected value, the upside/downside skew, and the break-even probability of the base case.",
),
Step(
title="What kills it",
inputs="The driver tree from step 2.",
compute="Three risks, each expressed as an observable metric with a numeric threshold.",
output="'Thesis is wrong if X falls below Y for two consecutive quarters' — falsifiable. 'Competition may intensify' is not a risk, it is filler.",
),
Step(
title="Monitoring set",
inputs="Step 5 thresholds.",
compute="Pick the three datapoints that move first when the thesis breaks.",
output="Three metrics, their release cadence, their current value, and the exit trigger.",
),
),
worked_example=(
"Step 4 scenario math, spot 12.50:",
"",
" case prob value/share source",
" bear 25% 9.00 comps at peer P25 on trough EBITDA",
" base 50% 15.00 DCF, WACC 9.0%, g 2.5%",
" bull 25% 22.00 DCF, margin path to 19% by year 5",
"",
" expected value = 0.25x9.00 + 0.50x15.00 + 0.25x22.00",
" = 2.25 + 7.50 + 5.50 = 15.25 (+22.0% vs spot)",
" downside = 9.00/12.50 - 1 = -28.0%",
" upside = 22.00/12.50 - 1 = +76.0%",
" skew = 76.0 / 28.0 = 2.7 : 1",
"",
" break-even test — hold bear at 25% and solve for the bull weight p",
" that makes expected value equal the 12.50 spot:",
" 2.25 + (0.75 - p)x15.00 + p x 22.00 = 12.50",
" 13.50 + 7.00p = 12.50 -> p = -0.14",
"",
" A NEGATIVE break-even weight means even with zero probability on the",
" bull case the expected value (2.25 + 0.75x15.00 = 13.50) still clears",
" the spot price. So the position does not depend on the bull case at",
" all — it depends entirely on the base case being right. That redirects",
" the remaining work: stop arguing the upside, stress-test the 15.00.",
" Concretely, the base case fails if the 15% EBIT margin is not held;",
" at 12% margin the same DCF gives roughly 11.60, below spot.",
),
tools=(
"get_financial_statements",
"get_sec_filings",
"get_stock_profile",
"get_research_reports",
"get_stock_news",
),
)
_EARNINGS = Playbook(
slug="earnings",
summary="Earnings review — surprise bridge from revenue to EPS",
usage="/earnings <ticker> [quarter=Q3-2026]",
examples=("/earnings AAPL", "/earnings MSFT quarter=Q2-2026", "/earnings 0700.HK"),
ask="Whose earnings should I review? e.g. /earnings AAPL quarter=Q3-2026 (quarter defaults to the most recent report).",
objective=(
"Turn a headline beat or miss into a line-by-line bridge, separate the "
"recurring operating surprise from the non-recurring one, and state what "
"actually changed in the model."
),
steps=(
Step(
title="Pull the print",
inputs="get_financial_statements for the latest quarter; get_sec_filings for the 8-K / 10-Q; get_stock_news or get_research_reports for the consensus figure.",
compute="Line up actual versus consensus for revenue, gross margin, opex, tax rate, share count, EPS.",
output="A two-column table. The consensus column is tagged SOURCED or ASSUMPTION — a remembered consensus number is an ASSUMPTION and must be labelled one.",
),
Step(
title="Build the bridge",
inputs="The step-1 table.",
compute=(
"Walk from consensus EPS to actual EPS ONE variable at a time, holding "
"the others at consensus, then re-baselining after each step. Order: "
"revenue, gross margin, opex, below-the-line, tax rate, share count."
),
output="A bridge whose steps sum EXACTLY to actual EPS minus consensus EPS. If it does not sum, a variable is missing — find it before writing anything.",
),
Step(
title="Grade the quality",
inputs="The bridge.",
compute="Split each step into recurring operating (volume, price, mix, structural cost) and non-recurring (one-off tax items, buybacks, FX, asset sales, legal reversals).",
output="Operating-only surprise versus headline surprise, as a number and as a percentage of the headline.",
),
Step(
title="Guidance delta",
inputs="New guidance versus prior guidance versus consensus for the year.",
compute="Implied remainder of the year = full-year guide minus reported year-to-date.",
output="Whether the implied remainder is an acceleration or a quiet cut hidden by a strong quarter already banked.",
),
Step(
title="What changed in the model",
inputs="Steps 3 and 4.",
compute="The three inputs whose forward path you now revise, and the resulting change in fair value.",
output="Old fair value, new fair value, and the single line item responsible for most of the change.",
),
),
worked_example=(
"Consensus: revenue 1,000 | GM 60.0% | opex 400 | interest -20 | tax 25%",
" | 100.0m shares -> EPS 1.35",
"Actual: revenue 1,050 | GM 58.5% | opex 405 | interest -20 | tax 22%",
" | 99.0m shares -> EPS 1.49",
"",
"Bridge, one variable at a time, re-baselining after each step:",
" start (consensus) EPS 1.3500",
" + revenue 1,000 -> 1,050 at 60% GM GP 630, NI 157.50 EPS 1.5750 +0.2250",
" + GM 60.0% -> 58.5% GP 614.25, NI 145.69 EPS 1.4569 -0.1181",
" + opex 400 -> 405 NI 141.94 EPS 1.4194 -0.0375",
" + tax 25% -> 22% NI 147.62 EPS 1.4762 +0.0568",
" + shares 100.0m -> 99.0m NI 147.62 EPS 1.4911 +0.0149",
" sum of steps = 0.2250-0.1181-0.0375+0.0568+0.0149 = +0.1411",
" check: 1.4911 - 1.3500 = +0.1411 PASS",
"",
"Step 3 quality grade (the part that is actually analysis):",
" recurring operating = +0.2250 - 0.1181 - 0.0375 = +0.0694",
" non-recurring = +0.0568 (one-off R&D tax credit)",
" + 0.0149 (buyback) = +0.0717",
" non-recurring share of the headline beat = 0.0717/0.1411 = 50.8%",
"",
" So 'beat by 14 cents on a 5% revenue beat' is really a 7-cent",
" operating beat. The 5% volume upside was half given back by a 150bp",
" mix-driven gross margin decline — that mix shift is the datapoint",
" that changes the model, not the EPS headline. Check whether the",
" lower-margin revenue is a one-quarter mix accident or the new run rate.",
),
tools=(
"get_financial_statements",
"get_sec_filings",
"get_stock_news",
"get_research_reports",
"get_market_data",
),
)
_SCREEN = Playbook(
slug="screen",
summary="Systematic idea screen — hypothesis, funnel, survivor queue",
usage="/screen <hypothesis or criteria> [--universe sp500|csi300|...]",
examples=(
"/screen quality compounders trading below 15x FCF --universe sp500",
"/screen ROIC > 12% and net debt/EBITDA < 2 --universe csi300",
"/screen oversold industrials with rising order books",
),
ask="What should I screen for? Give me the economic hypothesis or the hard criteria, e.g. /screen quality compounders below 15x FCF --universe sp500",
objective=(
"Turn a hypothesis into hard thresholds, run the screen, publish the funnel "
"count at every filter, and hand back a de-duplicated research queue with an "
"honest false-positive caveat."
),
steps=(
Step(
title="State the hypothesis first",
inputs="The user's words.",
compute="Write the economic mechanism in one sentence BEFORE choosing any threshold, so the filters are derived from the idea rather than reverse-engineered from a pretty result.",
output="One sentence. If the hypothesis cannot be stated without referring to the metrics, it is a data-mining exercise — label it as one.",
),
Step(
title="Translate to hard criteria",
inputs="The hypothesis.",
compute="Each clause becomes a metric with a numeric threshold and a direction. Every threshold gets a one-line justification.",
output="A criteria table: metric, operator, threshold, why this number.",
),
Step(
title="Run the funnel",
inputs="screen_market over the stated universe, then get_financial_statements for any criterion the screener cannot express.",
compute="Apply filters in order, from cheapest and most eliminating to most expensive.",
output="The FUNNEL: the surviving count after every single filter. A screen that only reports the final list is unreviewable.",
),
Step(
title="Data-completeness pass",
inputs="The survivor list.",
compute="Drop names missing an input for a binding criterion.",
output="Names dropped for missing data are listed SEPARATELY from names dropped for failing a test. Conflating the two silently biases the result toward well-covered large caps.",
),
Step(
title="De-duplicate by driver",
inputs="Survivors.",
compute="Group names sharing one underlying driver (same commodity, same customer, same rate sensitivity).",
output="One representative per cluster plus the cluster members, so the queue is not five bets on one macro variable.",
),
Step(
title="Hand back a queue, not a portfolio",
inputs="The de-duplicated survivors.",
compute="For each name: the single metric it is weakest on, and the one question to answer first.",
output=(
"A ranked research queue plus a threshold-sensitivity caveat stated in "
"numbers: re-run the funnel with each threshold loosened one notch and "
"report how much of the survivor set is an artifact of where the lines "
"were drawn. Never quote a per-name false-positive rate — a deterministic "
"screen is not a hypothesis test and has none."
),
),
),
worked_example=(
"Hypothesis: 'Businesses that earn well above their cost of capital and",
"are not levered can reinvest through a downturn; the market underpays",
"for that when the free cash flow yield is already high.'",
"",
"Criteria and funnel (universe: 3,200 listed names):",
" filter surviving removed",
" start 3,200",
" market cap >= 2.0bn (liquidity) 812 -2,388",
" ROIC >= 12% (above cost of capital) 245 -567",
" net debt/EBITDA <= 2.0 (survives a downturn) 96 -149",
" FCF yield >= 5% (the underpayment) 31 -65",
" data-completeness pass (step 4) 19 -12 MISSING DATA",
" de-duplicate by driver (step 5) 6 -13 same driver",
"",
" the 12 dropped for missing data are listed by name and by which",
" field was absent; they are NOT counted as failures.",
" the 13 collapsed names are 6 semis-cycle, 4 US housing, 3 freight.",
"",
"Step 6 caveat, stated in numbers rather than hedged in prose:",
" A screen is a deterministic filter on observed data, so there is no",
" per-name 'false positive rate' to quote — the selection risk lives in",
" the CHOICE of the 4 thresholds, not in the 3,200 rows. Size it that way:",
" the funnel keeps 31/3,200 = 0.97% of the universe, so moving any one",
" threshold by a notch (ROIC 12% -> 10%, FCF yield 5% -> 4%) is what to",
" report. Here that widens 31 -> 74, i.e. 58% of the survivor set is an",
" artifact of where the lines were drawn rather than a stable property.",
" So the screen cannot establish that these 6 names are good; it can only",
" establish that they are worth the next 6 hours. Treat the output as a",
" research queue and require an out-of-sample or fundamental confirmation",
" before any of it becomes a position.",
),
tools=("screen_market", "get_financial_statements", "get_stock_profile", "iwencai_search"),
aliases=("screener",),
)
PLAYBOOKS: tuple[Playbook, ...] = (_COMPS, _DCF, _ATTRIB, _MEMO, _EARNINGS, _SCREEN)
PLAYBOOKS_BY_SLUG: dict[str, Playbook] = {pb.slug: pb for pb in PLAYBOOKS}
__all__ = [
"GAP_POLICY",
"NOT_ADVICE",
"PLAYBOOKS",
"PLAYBOOKS_BY_SLUG",
"Playbook",
"Step",
]
agent/cli/commands/institutional/runner.py
+209
View File
@@ -0,0 +1,209 @@
"""Shared rendering and dispatch for the institutional workflow commands.
Every ``/comps``-style command funnels through :func:`run_playbook`:
* no arguments (or ``help`` / ``-h`` / ``--help`` / ``?``) renders the full
playbook card — objective, numbered execution skeleton, the worked numeric
example, the data-gap policy and the not-advice notice — followed by a
friendly follow-up question. This is a normal, successful outcome, so it
returns ``0``; missing arguments are not an error here.
* with arguments the same skeleton is rendered into a prompt and queued on
``ctx.pending_prompt``, which ``cli.main._interactive_loop`` drains into a
real agent turn (the mechanism ``/journal`` and ``/shadow`` already use).
When the context cannot hold a queued prompt — a bare test stub, or a
non-interactive caller — the prompt is printed instead so it can be pasted.
"""
from __future__ import annotations
from typing import Any
from rich.console import Console, Group
from rich.panel import Panel
from rich.table import Table
from rich.text import Text
from src.config.accessor import get_env_config
from .playbooks import GAP_POLICY, NOT_ADVICE, PLAYBOOKS_BY_SLUG, Playbook
# Defensive ceiling on the free-text argument so a paste bomb cannot blow up
# the queued prompt (and, downstream, the model context).
#
# A malformed override must not be fatal: this module is imported by
# ``cli.main._dispatch_slash``, whose import guard only catches ``ImportError``,
# so an exception raised here would escape the dispatcher and take down the
# whole REPL. The config layer drops an unparseable value and hands back the
# declared default, so this cannot raise.
def _arg_max_chars() -> int:
"""Resolve the argument length cap, ignoring a malformed override."""
return max(32, get_env_config().agent_tuning.vibe_trading_slash_arg_max)
_ARG_MAX_CHARS = _arg_max_chars()
_HELP_TOKENS = frozenset({"help", "-h", "--help", "?"})
def _console() -> Console:
"""Return the shared CLI console."""
from cli.theme import get_console
return get_console()
def _render_card(playbook: Playbook) -> None:
"""Print the full playbook card for ``playbook``.
Steps go into a two-column ``Table.grid`` rather than a flat ``Text`` so
the ``in`` / ``compute`` / ``out`` labels keep a hanging indent when the
body wraps on a narrow terminal.
"""
header = Text()
header.append(playbook.objective + "\n\n")
header.append("Usage ", style="dim")
header.append(playbook.usage + "\n", style="bold")
for example in playbook.examples:
header.append(" ")
header.append(example + "\n", style="cyan")
header.append("\nExecution skeleton", style="bold")
steps = Table.grid(padding=(0, 1))
steps.add_column(style="dim", no_wrap=True, justify="right", width=9)
steps.add_column(ratio=1, overflow="fold")
for index, step in enumerate(playbook.steps, start=1):
steps.add_row(f"{index}.", Text(step.title, style="bold"))
steps.add_row("in", Text(step.inputs, style="dim"))
steps.add_row("compute", Text(step.compute, style="dim"))
steps.add_row("out", Text(step.output, style="dim"))
steps.add_row("", "")
example_block = Text()
example_block.append("Worked example\n", style="bold")
for line in playbook.worked_example:
example_block.append(f" {line}\n", style="dim")
example_block.append("\nTools ", style="dim")
example_block.append(", ".join(playbook.tools) + "\n\n", style="cyan")
example_block.append("Missing data\n", style="bold")
for line in GAP_POLICY:
example_block.append(f" {line}\n", style="dim")
example_block.append("\n")
example_block.append(NOT_ADVICE + "\n\n", style="yellow")
example_block.append(playbook.ask, style="bold")
_console().print(
Panel(
Group(header, steps, example_block),
title=f"/{playbook.slug}",
border_style="cyan",
padding=(1, 2),
)
)
def build_prompt(playbook: Playbook, subject: str) -> str:
"""Render ``playbook`` plus the user's arguments into an agent prompt.
The full skeleton, the worked example, the gap policy and the not-advice
notice all travel with the prompt so the behaviour does not depend on any
particular system prompt being loaded.
Args:
playbook: The workflow to run.
subject: The user's raw arguments, already trimmed and length-capped.
Returns:
The prompt text to hand to the agent loop.
"""
lines = [
f"Run the {playbook.slug.upper()} research playbook.",
"",
f"Request: {subject}",
"",
f"Deliverable: {playbook.objective}",
"",
"Work through these steps in order. Do not skip a step, do not reorder them,",
"and show the intermediate artifact each step is required to produce.",
"",
]
for index, step in enumerate(playbook.steps, start=1):
lines.append(f"Step {index} — {step.title}")
lines.append(f" Inputs: {step.inputs}")
lines.append(f" Compute: {step.compute}")
lines.append(f" Output: {step.output}")
lines.append("")
lines.append("Worked example — reproduce this level of arithmetic explicitness with")
lines.append("the real retrieved data. The figures below are illustrative teaching")
lines.append("numbers, not data about any real security, and must not be reused as facts.")
lines.extend(f" {line}" for line in playbook.worked_example)
lines.append("")
lines.append(f"Preferred tools: {', '.join(playbook.tools)}.")
# Some of these are key-gated (``get_macro_series`` needs FRED_API_KEY,
# ``iwencai_search`` needs VIBE_TRADING_IWENCAI_KEY) and are absent from the
# registry unless configured. Say so explicitly: an unavailable tool must
# become a declared gap, never a remembered number.
lines.append(
"Any preferred tool that is not registered in this session is simply "
"unavailable — skip it, record what it would have supplied as a gap under "
"the missing-data policy below, and never fill that gap from memory."
)
lines.append("")
lines.append("Missing data:")
lines.extend(f" {line}" for line in GAP_POLICY)
lines.append("")
lines.append(NOT_ADVICE)
return "\n".join(lines)
def run_playbook(slug: str, ctx: Any = None, *args: str) -> int:
"""Render or dispatch the playbook registered under ``slug``.
Args:
slug: Playbook key, matching the slash command keyword.
ctx: The interactive context. Needs a writable ``pending_prompt``
attribute for the queued-turn path; anything else falls back to
printing the prompt.
*args: Raw slash-command arguments.
Returns:
``0`` in every user-facing path — a missing argument is answered with
the playbook card and a follow-up question, not an error. ``1`` only
when ``slug`` is not a registered playbook, which is a programming
error rather than a user mistake.
"""
playbook = PLAYBOOKS_BY_SLUG.get(slug)
if playbook is None:
_console().print(Text(f"Unknown research playbook: {slug}", style="bold red"))
return 1
if args and args[0].lower() in _HELP_TOKENS:
_render_card(playbook)
return 0
subject = " ".join(args).strip()
if not subject:
_render_card(playbook)
return 0
if len(subject) > _ARG_MAX_CHARS:
subject = subject[:_ARG_MAX_CHARS].rstrip() + " …(truncated)"
prompt = build_prompt(playbook, subject)
console = _console()
if ctx is not None and hasattr(ctx, "pending_prompt"):
try:
ctx.pending_prompt = prompt
console.print(
Text(f"→ Running the /{playbook.slug} playbook on: {subject}", style="dim")
)
return 0
except Exception: # noqa: BLE001 — never let a slash handler kill the REPL
pass
console.print(Text(f"/{playbook.slug} — paste this prompt to run it:", style="bold"))
console.print(Text(prompt, style="dim"))
return 0
__all__ = ["build_prompt", "run_playbook"]
Loaded 100 of 1592 files, more files were not shown because too many files have changed in this diff. Show more