diff --git a/CHANGELOG.md b/CHANGELOG.md index af8809f3..aa24204e 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -5,6 +5,48 @@ This project adheres to [Semantic Versioning](https://semver.org/). ## [Unreleased] +### Maintenance — 2026-10-12 + +- Preserve the original opening fill after scale-ins (#1780), use the book + immediately before terminal liquidation for Brinson attribution (#1782), + and start drawdown duration at the latest visit to the peak (#1784). + Three-statement projections honor the caller's balance tolerance throughout + the model (#1786); existing balance checks remain enforced. +- Add ten SEC fundamental factors and three A-share valuation factors + (`val:pb`, `val:ps_ttm`, `val:dv_ttm`) via Tushare daily_basic (#1787, #1788, + #1789). The registry now has 475 factors, 17 in the fundamental family. + Date-only SEC filings become usable one bar later on the IC path (#1790, + #1794). Prior IC measurements using same-day filings need to be rerun. + XOM keeps its current SEC identity while pre-reorganization filings extend + its history. Basic shares never substitute for missing diluted shares. + Price-times-diluted-shares ratios are valuation proxies, not reported market + capitalization. Missing input data remains missing. +- Add Brazil `.SA`/BRL routing, an equity engine, local B3/CVM/BCB/Tesouro + ingestion and the `get_br_market_data` tool, plus market selection and + labels in all nine UI languages (#1783). COTAHIST bars remain raw; + CVM statements contain latest revisions, not point-in-time evidence. + ZimaOS Compose and a fork publishing template require explicit setup; + the upstream repository gains no automatic publishing schedule. +- Forward active attempt status, tool hints and opt-in reasoning to IM + channels, including Feishu CardKit (#1792). Final answers still pass + grounding before delivery; draft text and tool-result payloads stay private. +- Complete write-target helper extraction with actual loop delegation (#1791, + #1624), preserving runtime model provenance. Clarify that connector profiles, + not the legacy config.readonly field, govern order access (#1793). + +### Maintenance — 2026-10-11 + +- Reject unsupported live options before broker dispatch (#1773). This is a + guard fix; live options execution is not enabled. +- Require evidence for every data operand in derived figures (#1622). + Chosen multipliers/divisors need an explicit parameter declaration and + matching visible label; arithmetic checks remain enforced. +- Keep explicit offline/source-bound benchmarks on their selected loader and + report unresolved benchmarks (#1781). Distinguish entry/exit events so + break-even closes remain in trade artifacts and downstream analysis (#1779). +- Preserve optional Codex tool arguments with explicit non-strict schemas + (#1785). + ### Maintenance — 2026-10-10 - Slack and Discord gain localized connection forms and read-only credential diff --git a/README.md b/README.md index b94dcc08..a5efb252 100644 --- a/README.md +++ b/README.md @@ -52,16 +52,19 @@ > ⚠️ **Security warning:** The X account `VibeTrading_HKU`, Virtuals project `101845`, and token contract `0x640BDBF77b6447E8b7DB7894cED84BD1c40571f4` are not official Vibe-Trading assets. We have never launched or endorsed any token or memecoin. Do not buy, connect a wallet, or sign anything. [Details](SECURITY.md#official-channels--impersonation). +- **2026-10-12** 🧮 **Research and backtest updates**: Correct opening fills, liquidation-time attribution, drawdown duration, valuation tolerances and SEC filing timing in factor IC ([#1780](https://github.com/HKUDS/Vibe-Trading/pull/1780)–[#1794](https://github.com/HKUDS/Vibe-Trading/pull/1794)). Add 13 fundamental/valuation factors, Brazil data and backtests, opt-in ZimaOS templates, and Feishu run progress. See [CHANGELOG](CHANGELOG.md) for scope and data limitations. + +- **2026-10-11** 🛠️ **Five priority fixes**: Derived figures require evidence for every data operand; unsupported live options stay blocked. Offline benchmarks retain their selected source, trade artifacts distinguish break-even exits, and Codex tools preserve optional arguments ([#1622](https://github.com/HKUDS/Vibe-Trading/issues/1622), [#1779](https://github.com/HKUDS/Vibe-Trading/pull/1779), [#1781](https://github.com/HKUDS/Vibe-Trading/pull/1781), [#1785](https://github.com/HKUDS/Vibe-Trading/pull/1785)). + - **2026-10-10** 🛠️ **Slack/Discord setup and reproducible research**: Localized channel forms verify bot credentials without saving changes ([#1764](https://github.com/HKUDS/Vibe-Trading/pull/1764), [#1777](https://github.com/HKUDS/Vibe-Trading/pull/1777)). Portfolio research can reuse an immutable snapshot, and Web exports its sanitized JSON ([#1769](https://github.com/HKUDS/Vibe-Trading/pull/1769)). Indicators report their date boundary and exact input fingerprint; corrupted session records recover without rewriting files. Memory recall, credential redaction and trade-journal detection are corrected. See [CHANGELOG](CHANGELOG.md). +
+Earlier news + - **2026-10-09** 🛠️ **Channel updates and backtest correctness**: Channel settings preserve embedded URL credentials and authenticated WeChat endpoints during in-place updates, with save/reset feedback in all nine languages ([#1737](https://github.com/HKUDS/Vibe-Trading/pull/1737), [#1742](https://github.com/HKUDS/Vibe-Trading/pull/1742), [#1744](https://github.com/HKUDS/Vibe-Trading/pull/1744)). Cross-market option volatility uses only timestamps known at each bar; drawdown includes starting capital. Crypto/precious-metal routing, HK duplicate detection, cache recovery, scientific-notation amounts and missing-equity validation are corrected. mootdx leaves the default A-share chain. See [CHANGELOG](CHANGELOG.md) for all thirteen PRs. - **2026-10-08** 🔐 **Protected PDF reports and clearer broker reads**: Scheduled Email PDFs support optional AES-256 password protection across Web, CLI playbooks and agent confirmations; the password stays in private channel configuration ([#1709](https://github.com/HKUDS/Vibe-Trading/pull/1709)). KIS queries all three venue buckets, refreshes an explicitly expired token once, and reports incomplete reads as errors ([#1726](https://github.com/HKUDS/Vibe-Trading/pull/1726)). Derived formulas keep their arithmetic after descriptive labels while evidence checks remain enforced ([#1728](https://github.com/HKUDS/Vibe-Trading/pull/1728)). - -
-Earlier news - - **2026-10-07** 🛠️ **Research continuity and reliable risk metrics**: Volatile workspace state now travels outside the system prompt, preserving its stable prefix ([#1708](https://github.com/HKUDS/Vibe-Trading/pull/1708)). Cross-market dates align correctly, missing prices no longer bridge return observations, and options Sortino uses full-sample downside deviation ([#1710](https://github.com/HKUDS/Vibe-Trading/pull/1710), [#1717](https://github.com/HKUDS/Vibe-Trading/pull/1717), [#1727](https://github.com/HKUDS/Vibe-Trading/pull/1727)). Loader fixtures share one OHLCV contract; credit valuation, style exposures, walk-forward folds and audit diagnostics are corrected. - **2026-10-06** 🛠️ **Live controls and report corrections**: Stopping a live runner cancels its current analysis and waits for scheduler cleanup, including startup cancellation and API shutdown; status timestamps use the correct units ([#1704](https://github.com/HKUDS/Vibe-Trading/pull/1704)). Emergency cancel/flatten scans skip malformed records without abandoning the remaining book ([#1703](https://github.com/HKUDS/Vibe-Trading/pull/1703)); broker schedules are stored separately and writes handle concurrency and partial writes. US equity half-days respect the early closing bell ([#1706](https://github.com/HKUDS/Vibe-Trading/pull/1706)). Report corrections preserve figures that passed validation, and bounded feedback no longer implies that unlisted figures passed ([#1702](https://github.com/HKUDS/Vibe-Trading/pull/1702)). @@ -376,7 +379,7 @@ It is designed for research, simulation, and backtesting — and, when you choos | **Run analyst teams** | Multi-agent research reviews for investment, quant, crypto, macro, and risk workflows. | | **Put research into IM channels** | Run the same session runtime through WebSocket, Telegram, Slack, Discord, Matrix, WhatsApp, Signal, QQ/NapCat, WeChat/WeCom, Feishu/Lark, DingTalk, Teams, email, and Mochat with CLI, REST, and Web UI controls. | | **Ship usable artifacts** | Reports, TradingView Pine Script, TDX, MetaTrader 5, MCP tools, and later research sessions. | -| **Bench a pre-built alpha zoo** | One-line IC + alive/reversed/dead categorisation across 462 alphas (Qlib 158 + Kakushadze 101 + GTJA 191 + academic + PIT-safe fundamental) on your universe. | +| **Bench a pre-built alpha zoo** | One-line IC + alive/reversed/dead categorisation across 475 alphas (Qlib 158 + Kakushadze 101 + GTJA 191 + academic + PIT-safe fundamental) on your universe. | | **Spot correlation regimes** | An edge-density + hysteresis timeline on the `/correlation` surface showing when markets fuse into one bloc — descriptive risk context, not a signal. Undefined pair correlations keep edge density unknown, so constant-price histories cannot claim a measured zero density. | Daily correlation and regime requests align each source by its local calendar date, including mixed timezone-aware and timezone-naive histories. @@ -580,7 +583,7 @@ the wall-clock activity watchdog. ## 📡 Data Sources & Smart Fallback -One `get_market_data` call, **28 market-data sources**, one of them the optional **QVeris** premium marketplace. Set `source: "auto"` — the loader picks by symbol, then walks a per-market chain ordered by **IP-ban risk**: never-banned public sources first, throttled / key-gated ones last. Zero config, no single point of failure. +One `get_market_data` call, **29 market-data sources**, one of them the optional **QVeris** premium marketplace. Set `source: "auto"` — the loader picks by symbol, then walks a per-market chain ordered by **IP-ban risk**: never-banned public sources first, throttled / key-gated ones last. Zero config, no single point of failure. | Source | Markets | Auth | Role | |--------|---------|------|------| @@ -603,6 +606,7 @@ One `get_market_data` call, **28 market-data sources**, one of them the optional | `tickerall` | forex / metals | key + account (read-only) | same broker MT5 feed, **hosted** — no local terminal, any OS (explicit-only, never in auto fallback) | | `pykrx` | Korea (KRX: KOSPI/KOSDAQ) | none | daily KOSPI / KOSDAQ bars for `.KS` / `.KQ` (optional `krx` extra) | | `india_broker` | India (NSE/BSE) | broker login | read-only Zerodha / Shoonya / Dhan bars for `.NS` / `.BO` (fallback-chain tail) | +| `b3_cotahist` | Brazil (B3) | none | official daily B3 cash/odd-lot bars (raw, BRL) from the local store populated by `python -m src.br_data sync`; offline fallback for `.SA` | | `local` | any | none | your own CSV / Parquet / DuckDB via `local:` prefix | **Fallback chains (by IP-ban risk):** @@ -611,6 +615,7 @@ One `get_market_data` call, **28 market-data sources**, one of them the optional - **US** → `yahoo` · `stooq` · `sina` · `eastmoney` · `yfinance` · `tiingo` · `fmp` · `finnhub` · `alphavantage` · `longbridge` · `akshare` · `local` - **HK** → `tencent` · `eastmoney` · `yahoo` · `futu` · `akshare` · `yfinance` · `tushare` · `longbridge` · `local` - **India (NSE/BSE)** → `yahoo` · `yfinance` · `india_broker` · `local` +- **Brazil (B3 `.SA`)** → `yahoo` · `yfinance` · `b3_cotahist` · `local` - **Korea (KOSPI/KOSDAQ)** → `pykrx` · `yahoo` · `yfinance` · `local` - **UK (LSE)** → `yahoo` · `yfinance` · `local` *(declared GBP/GBp quotes only)* - **Crypto** → `okx` · `binance` · `ccxt` · `yfinance` · `local` @@ -793,9 +798,9 @@ alongside the bundled roster (same-name files override it, like user skills) and
-Alpha Zoo 462 pre-built quant alphas across 5 families +Alpha Zoo 475 pre-built quant alphas across 5 families -- 🧬 462 cross-sectional alphas, lookahead-banned at the operator layer +- 🧬 475 cross-sectional alphas, lookahead-banned at the operator layer - 📈 IC + IR + alive/reversed/dead categorisation in one CLI command - 🔬 AST purity gate + 300-row lookahead sentinel test + `pytest-socket` network kill-switch - 📦 Apache-2 attribution for Qlib; per-zoo `LICENSE.md` declaring formulas as mathematical content @@ -814,7 +819,7 @@ Run `vibe-trading alpha list` to browse, `vibe-trading alpha show ` for form
-Backtest Engines 10 engines + options portfolio, cross-market composite +Backtest Engines 11 engines + options portfolio, cross-market composite | Engine | Market | Notes | |--------|--------|-------| @@ -823,6 +828,7 @@ Run `vibe-trading alpha list` to browse, `vibe-trading alpha show ` for form | **IndiaEquity** | India (NSE/BSE) | T+1, circuit bands, config-driven STT / stamp / SEBI / GST cost stack | | **KoreaEquity** | Korea (KRX: KOSPI/KOSDAQ) | long-only, ±30% band judged at execution time on the unified tick grid, 2026 0.20% transaction tax | | **VietnamEquity** | Vietnam (HOSE) | long-only, T+2 settlement hold, ±7% band on the 10/50/100-VND tick grid, 100-share lots, 0.1% sell-side tax | +| **BrazilEquity** | Brazil (B3) | BRL, B3 trading + settlement fees, configurable broker fee, 1- or 100-share lots, R$0.01 tick, shorting allowed | | **Crypto** | crypto spot / USD-M perps | funding settlements, execution/mark split | | **ChinaFutures** · **GlobalFutures** | futures | margin, contract multipliers | | **Forex** | FX / metals | via the `mt5` loader (local terminal) or the hosted `tickerall` loader (no terminal, any OS) | @@ -1195,7 +1201,7 @@ The interactive TUI (`vibe-trading`) now uses a terminal-native transcript: a st vibe-trading # interactive TUI vibe-trading run -p "..." # single run vibe-trading serve # API server -vibe-trading alpha list # browse 462 pre-built alphas; show / bench / compare / export-manifest sub-commands available +vibe-trading alpha list # browse 475 pre-built alphas; show / bench / compare / export-manifest sub-commands available vibe-trading playbook list # five scheduled-research templates; show / create sub-commands available vibe-trading channels status --local # inspect IM channel config and install hints vibe-trading provider doctor # print redacted provider/proxy/package diagnostics @@ -1339,6 +1345,12 @@ Saves write the `channels.` section of `~/.vibe-trading/agent.json` atomic ### Strategy & Backtesting +New BaseEngine `artifacts/trades.csv` files include an `event` column: +`entry` marks an opening row and `exit` marks a completed trade, including +zero-PnL exits. Validation and strategy-discovery readers use this marker. +Files without it retain legacy loading; their zero-PnL exits remain ambiguous. +The separate options-engine trade format is unchanged. + ```bash # Moving average crossover on US equities vibe-trading run -p "Backtest a 20/50-day moving average crossover on AAPL for the past year, show Sharpe ratio and max drawdown" @@ -1701,6 +1713,8 @@ with `--host` / `--port`. `quantlib_call` attribution requires finite sector weights and returns, finite Carino return inputs, and a finite non-negative weight-sum tolerance. Invalid data is rejected explicitly rather than returned as an attribution result containing `NaN`. Missing returns for a zero-weight sector may still be inferred from the other side. +`quantlib_call` maximum-drawdown analysis measures underwater duration from the latest visit to the peak before the worst trough. An earlier drawdown that already recovered is excluded from that duration. + `quantlib_call` VaR backtests count finite return/forecast pairs for coverage. Missing pairs break the consecutive-observation chain for independence testing; they never create a transition between the observations on either side of a gap. `quantlib_call` option analytics use an absolute **price** tolerance for implied-volatility inversion. A finite result must reprice the quote within that tolerance; the underlying analytics function returns `NaN` for an unconverged or volatility-unidentifiable quote rather than an asserted volatility. @@ -2103,7 +2117,7 @@ Vibe-Trading/ │ │ ├── memory/ # Cross-session persistent memory │ │ │ └── persistent.py # file-based memory (~/.vibe-trading/memory/) │ │ │ -│ │ ├── tools/ # 109 auto-discovered agent tools +│ │ ├── tools/ # 110 auto-discovered agent tools │ │ │ ├── backtest_tool.py # run backtests │ │ │ ├── remember_tool.py # cross-session memory (save/recall/forget) │ │ │ ├── skill_writer_tool.py # skill CRUD (save/patch/delete/file) @@ -2112,7 +2126,7 @@ Vibe-Trading/ │ │ │ ├── web_search_tool.py # DuckDuckGo web search │ │ │ └── ... # bash, file I/O, factor analysis, options, alpha browser + bench, etc. │ │ │ -│ │ ├── factors/ # Alpha Zoo — 462 alphas across 5 families +│ │ ├── factors/ # Alpha Zoo — 475 alphas across 5 families │ │ │ ├── base.py # 19 operators (rank/scale/ts_*/delta/decay_linear/safe_div/vwap) │ │ │ ├── registry.py # AST-only metadata load + lazy compute + sanity gates │ │ │ ├── bench_runner.py # IC + alive/reversed/dead categorisation @@ -2128,8 +2142,8 @@ Vibe-Trading/ │ │ └── providers/ # LLM provider abstraction │ │ │ └── backtest/ # Backtest engines -│ ├── engines/ # 9 engines + composite cross-market engine + options_portfolio -│ ├── loaders/ # 28 sources: tushare, okx, nobitex, wallex, binance, yfinance, akshare, baostock, tencent, mootdx, ccxt, futu, pykrx, local, eastmoney, sina, stooq, yahoo, finnhub, alphavantage, tiingo, fmp, longbridge, mt5, qveris, india_broker, tickerall, gildata +│ ├── engines/ # 10 engines + composite cross-market engine + options_portfolio +│ ├── loaders/ # 29 sources: tushare, okx, nobitex, wallex, binance, yfinance, akshare, baostock, tencent, mootdx, ccxt, futu, pykrx, local, eastmoney, sina, stooq, yahoo, finnhub, alphavantage, tiingo, fmp, longbridge, mt5, qveris, india_broker, tickerall, gildata, b3_cotahist │ │ ├── base.py # DataLoader Protocol │ │ └── registry.py # Registry + auto-fallback chains │ └── optimizers/ # MVO, equal vol, max div, risk parity @@ -2195,7 +2209,7 @@ Vibe-Trading is part of the **[HKUDS](https://github.com/HKUDS)** agent ecosyste | **Data Bridge** | Bring-your-own data: local CSV/Parquet/SQL connectors with schema mapping | Local loader Shipped | | **Options Lab** | Vol surface, Greeks dashboard, payoff/scenario explorer | Analytic payoff/scenario tool **Shipped**; surface/dashboard Planned | | **Portfolio Studio** | Risk x-ray, constraints, turnover-aware optimizer, rebalance notes | Turnover-aware optimizer **Shipped 0.1.11**; rest Planned | -| **Alpha Zoo** | 462 pre-built alphas (Qlib 158 + Kakushadze 101 + GTJA 191 + academic + fundamental) with one-line bench, agent integration, and Web UI | **Shipped 0.1.8**, extended through 0.1.12 | +| **Alpha Zoo** | 475 pre-built alphas (Qlib 158 + Kakushadze 101 + GTJA 191 + academic + fundamental) with one-line bench, agent integration, and Web UI | **Shipped 0.1.8**, extended through 0.1.12 | | **Strategy Development Manager** | Register papers / broker research as factors & strategies with a persistent store + automated IC/Sharpe decay lifecycle | **Shipped 0.1.11** | | **Correlation Regime** | Edge-density + hysteresis regime timeline layered on `/correlation` — spot when markets fuse into one bloc | **Shipped 0.1.12** | | **Research Delivery** | Scheduled briefs and live research sessions through Slack / Telegram / email-style IM channels | Scheduler + IM Runtime Shipped | diff --git a/README_ar.md b/README_ar.md index ef43dc4d..50c79cf3 100644 --- a/README_ar.md +++ b/README_ar.md @@ -52,16 +52,19 @@ > ⚠️ **تحذير أمني:** حساب X باسم `VibeTrading_HKU`، ومشروع Virtuals رقم `101845`، وعقد التوكن `0x640BDBF77b6447E8b7DB7894cED84BD1c40571f4` كلّها غير رسمية ولا تتبع Vibe-Trading. لم نُطلق أو نؤيد مطلقًا أي توكن أو عملة ميم. لا تشترِ هذا التوكن، ولا تربط محفظتك، ولا توقّع أي شيء. [التفاصيل](SECURITY.md#official-channels--impersonation). +- **2026-10-12** 🧮 **تحديثات البحث والاختبار التاريخي**: تصحيح تنفيذ الافتتاح وإسناد الأداء عند التصفية ومدة التراجع وتسامح التقييم وتوقيت إفصاحات SEC في IC للعوامل ([#1780](https://github.com/HKUDS/Vibe-Trading/pull/1780)–[#1794](https://github.com/HKUDS/Vibe-Trading/pull/1794)). إضافة 13 عاملًا أساسيًا وللتقييم، وبيانات البرازيل واختباراتها، وقوالب ZimaOS بتفعيل اختياري، وتقدم التشغيل في Feishu. النطاق والقيود في [CHANGELOG](CHANGELOG.md). + +- **2026-10-11** 🛠️ **خمسة إصلاحات ذات أولوية**: تتطلب الأرقام المشتقة دليلًا لكل قيمة مستخدمة، وتبقى أوامر الخيارات الحية غير المدعومة محظورة. تحافظ المعايير دون اتصال على المصدر المحدد، وتميز سجلات التداول الإغلاق بلا ربح أو خسارة، وتحفظ أدوات Codex المعاملات الاختيارية ([#1622](https://github.com/HKUDS/Vibe-Trading/issues/1622)، [#1779](https://github.com/HKUDS/Vibe-Trading/pull/1779)، [#1781](https://github.com/HKUDS/Vibe-Trading/pull/1781)، [#1785](https://github.com/HKUDS/Vibe-Trading/pull/1785)). + - **2026-10-10** 🛠️ **إعداد Slack وDiscord وبحث قابل لإعادة الإنتاج**: تتحقق النماذج المترجمة من بيانات اعتماد البوت دون حفظ التغييرات ([#1764](https://github.com/HKUDS/Vibe-Trading/pull/1764)، [#1777](https://github.com/HKUDS/Vibe-Trading/pull/1777)). يعيد بحث المحفظة استخدام لقطات ثابتة، وتصدّر واجهة الويب JSON دون بيانات اعتماد أو أرقام حسابات ([#1769](https://github.com/HKUDS/Vibe-Trading/pull/1769)). تذكر المؤشرات حد التاريخ وبصمة المدخلات؛ تُستعاد سجلات الجلسة التالفة دون إعادة كتابة الملفات الأصلية. أُصلح استرجاع الذاكرة وإخفاء بيانات الاعتماد والتعرف على سجل التداول. راجع [CHANGELOG](CHANGELOG.md). +
+أخبار سابقة + - **2026-10-09** 🛠️ **إعدادات القنوات ودقة الاختبارات التاريخية**: تحافظ تحديثات الإعدادات دون إعادة الاتصال على بيانات اعتماد الروابط وعنوان خادم WeChat الناتج عن المصادقة، وتعرض الواجهة نتائج الحفظ وإعادة التشغيل بتسع لغات ([#1737](https://github.com/HKUDS/Vibe-Trading/pull/1737)، [#1742](https://github.com/HKUDS/Vibe-Trading/pull/1742)، [#1744](https://github.com/HKUDS/Vibe-Trading/pull/1744)). يعتمد تقلب الخيارات عبر الأسواق على التوقيتات المعروفة عند كل شمعة فقط، ويشمل التراجع رأس المال الأولي. صُححت مسارات بيانات العملات الرقمية والمعادن الثمينة، وكشف تكرار أسهم هونغ كونغ، واستعادة الذاكرة المؤقتة، والمبالغ بالصيغة العلمية، والتحقق من قيم حقوق الملكية المفقودة. أُزيل mootdx من سلسلة أسهم A الافتراضية. تفاصيل طلبات الدمج الثلاثة عشر في [CHANGELOG](CHANGELOG.md). - **2026-10-08** 🔐 **حماية تقارير PDF وتصحيح قراءات الوسيط**: تدعم تقارير البريد المجدولة حماية اختيارية بكلمة مرور عبر AES-256 في الويب وCLI وتأكيدات الوكيل؛ تُحفظ كلمة المرور في إعدادات القناة الخاصة فقط ([#1709](https://github.com/HKUDS/Vibe-Trading/pull/1709)). يستعلم KIS عن مسارات التداول الثلاثة ويجدد الرمز المنتهي صراحةً مرة واحدة ويرفض القراءة غير المكتملة ([#1726](https://github.com/HKUDS/Vibe-Trading/pull/1726)). تُحفظ الصيغة الحسابية بعد الوصف مع استمرار التحقق من الأدلة ([#1728](https://github.com/HKUDS/Vibe-Trading/pull/1728)). - -
-أخبار سابقة - - **2026-10-07** 🛠️ **استمرارية البحث وتصحيح مقاييس المخاطر**: نُقلت حالة التشغيل خارج تعليمات النظام للحفاظ على بادئة ثابتة ([#1708](https://github.com/HKUDS/Vibe-Trading/pull/1708)). صُححت محاذاة التواريخ بين الأسواق وعوائد الأسعار المفقودة وحساب Sortino للخيارات باستخدام العينة كاملةً ([#1710](https://github.com/HKUDS/Vibe-Trading/pull/1710)، [#1717](https://github.com/HKUDS/Vibe-Trading/pull/1717)، [#1727](https://github.com/HKUDS/Vibe-Trading/pull/1727)). تشترك اختبارات مصادر البيانات في عقد OHLCV واحد، مع إصلاحات لتقييم الائتمان والتعرضات والتحقق الزمني وتشخيص التدقيق. - **2026-10-06** 🛠️ **إصلاح التحكم بالتداول وتصحيح التقارير**: يلغي إيقاف مشغّل التداول الفعلي التحليل الجاري وينتظر انتهاء المجدول، بما يشمل الإلغاء أثناء بدء التشغيل وإغلاق API، مع تصحيح وحدات الوقت في الحالة ([#1704](https://github.com/HKUDS/Vibe-Trading/pull/1704)). يتجاوز إلغاء الأوامر وإغلاق المراكز في الطوارئ السجلات غير الصالحة ويواصل معالجة الباقي ([#1703](https://github.com/HKUDS/Vibe-Trading/pull/1703)). تُحفظ جداول الوسطاء بصورة منفصلة مع معالجة الكتابات المتزامنة والجزئية. تُراعى ساعات الإغلاق المبكر للأسهم الأمريكية ([#1706](https://github.com/HKUDS/Vibe-Trading/pull/1706)). تحافظ تصحيحات التقارير على الأرقام التي اجتازت التحقق، ولا توحي الملاحظات المحدودة بأن الأرقام غير المذكورة قد اجتازته ([#1702](https://github.com/HKUDS/Vibe-Trading/pull/1702)). @@ -377,7 +380,7 @@ Vibe-Trading مساحة عمل بحثية مفتوحة المصدر تحول ا | **تشغيل فرق محللين** | مراجعات بحث متعددة الوكلاء لتدفقات الاستثمار والكم والكريبتو والماكرو والمخاطر. | | **وصل الأبحاث بقنوات IM** | إدارة بيئة جلسة واحدة عبر WebSocket وTelegram وSlack وDiscord وMatrix وWhatsApp وSignal وQQ/NapCat وWeChat/WeCom وFeishu/Lark وDingTalk وTeams وemail وMochat من CLI وREST وWeb UI. | | **إنتاج مخرجات قابلة للاستخدام** | تقارير، وTradingView Pine Script، وTDX، وMetaTrader 5، وأدوات MCP، وجلسات بحث لاحقة. | -| **bench ألفا zoo جاهزة** | تشغيل IC + IR + تصنيف alive/reversed/dead عبر 462 ألفا (Qlib 158 + Kakushadze 101 + GTJA 191 + academic + PIT-safe fundamental) بسطر أوامر واحد على universe الخاص بك. | +| **bench ألفا zoo جاهزة** | تشغيل IC + IR + تصنيف alive/reversed/dead عبر 475 ألفا (Qlib 158 + Kakushadze 101 + GTJA 191 + academic + PIT-safe fundamental) بسطر أوامر واحد على universe الخاص بك. | | **رصد أنظمة الارتباط** | جدول زمني قائم على كثافة الحواف + التباطؤ (hysteresis) على واجهة `/correlation` يُظهر متى تندمج الأسواق في كتلة واحدة — سياق مخاطر وصفي، لا إشارة تداول. | --- @@ -461,7 +464,7 @@ vibe-trading connector install /tmp/my-broker ## 📡 مصادر البيانات والتراجع الذكي -استدعاء واحد لـ `get_market_data`، **28 مصدر بيانات سوقية**، أحدها سوق **QVeris** المدفوع الاختياري (إضافة إلى سوق مدفوع اختياري **QVeris**). اضبط `source: "auto"` — يختار المُحمّل حسب الرمز، ثم يسير عبر سلسلة لكل سوق مرتبة بحسب **خطر حظر عنوان IP**: المصادر العامة التي لا تُحظر أبداً أولاً، والمصادر المُقيّدة أو المحمية بمفتاح أخيراً. بلا أي إعداد، ولا نقطة فشل واحدة. +استدعاء واحد لـ `get_market_data`، **29 مصدر بيانات سوقية**، أحدها سوق **QVeris** المدفوع الاختياري (إضافة إلى سوق مدفوع اختياري **QVeris**). اضبط `source: "auto"` — يختار المُحمّل حسب الرمز، ثم يسير عبر سلسلة لكل سوق مرتبة بحسب **خطر حظر عنوان IP**: المصادر العامة التي لا تُحظر أبداً أولاً، والمصادر المُقيّدة أو المحمية بمفتاح أخيراً. بلا أي إعداد، ولا نقطة فشل واحدة. | Source | Markets | Auth | Role | |--------|---------|------|------| @@ -484,6 +487,7 @@ vibe-trading connector install /tmp/my-broker | `tickerall` | الفوركس / المعادن | مفتاح + حساب (للقراءة فقط) | تغذية MT5 لنفس الوسيط لكن **مستضافة** — دون طرفية محلية وعلى أي نظام تشغيل (بالاختيار الصريح فقط، ولا تدخل سلسلة auto أبدًا) | | `pykrx` | كوريا (KRX: KOSPI/KOSDAQ) | لا شيء | أشرطة يومية لـ KOSPI / KOSDAQ لرموز `.KS` / `.KQ` (إضافة `krx` اختيارية) | | `india_broker` | الهند (NSE/BSE) | تسجيل دخول الوسيط | قراءة فقط لأشرطة Zerodha / Shoonya / Dhan لرموز `.NS` / `.BO` (ذيل سلسلة التراجع) | +| `b3_cotahist` | Brazil (B3) | none | شموع B3 اليومية المحلية للسوق النقدي والكسور (خام، BRL)، عبر `python -m src.br_data sync`؛ مصدر بديل دون اتصال لـ `.SA` | | `local` | any | none | your own CSV / Parquet / DuckDB via `local:` prefix | **سلاسل التراجع (بحسب خطر حظر عنوان IP):** @@ -492,6 +496,7 @@ vibe-trading connector install /tmp/my-broker - **أسهم US** → `yahoo` · `stooq` · `sina` · `eastmoney` · `yfinance` · `tiingo` · `fmp` · `finnhub` · `alphavantage` · `longbridge` · `akshare` · `local` - **أسهم HK** → `tencent` · `eastmoney` · `yahoo` · `futu` · `akshare` · `yfinance` · `tushare` · `longbridge` · `local` - **أسهم الهند (NSE/BSE)** → `yahoo` · `yfinance` · `india_broker` · `local` +- **Brazil (B3 `.SA`)** → `yahoo` · `yfinance` · `b3_cotahist` · `local` - **كوريا (KOSPI/KOSDAQ)** → `pykrx` · `yahoo` · `yfinance` · `local` - **المملكة المتحدة (LSE)** → `yahoo` · `yfinance` · `local` *(الأسعار المعلنة بـ GBP/GBp فقط)* - **الكريبتو** → `okx` · `binance` · `ccxt` · `yfinance` · `local` @@ -662,9 +667,9 @@ LONGBRIDGE_ACCESS_TOKEN=...
-Alpha Zoo 462 ألفا كمّي جاهز عبر 5 families +Alpha Zoo 475 ألفا كمّي جاهز عبر 5 families -- 🧬 462 ألفا cross-sectional، مع منع lookahead على طبقة العوامل (operators) +- 🧬 475 ألفا cross-sectional، مع منع lookahead على طبقة العوامل (operators) - 📈 IC + IR + تصنيف alive/reversed/dead بأمر CLI واحد - 🔬 بوابة نقاء AST + اختبار حماية lookahead بـ 300 صف + قاطع شبكة عبر `pytest-socket` - 📦 إسناد Apache-2 لـ Qlib؛ ملف `LICENSE.md` لكل zoo يصرّح بأن الصيغ محتوى رياضي @@ -683,7 +688,7 @@ LONGBRIDGE_ACCESS_TOKEN=...
-محرّكات الاختبار الرجعي 10 محرّكات + محفظة خيارات، ومركّب عبر الأسواق +محرّكات الاختبار الرجعي 11 محرّكات + محفظة خيارات، ومركّب عبر الأسواق | المحرّك | السوق | ملاحظات | |--------|-------|---------| @@ -692,6 +697,7 @@ LONGBRIDGE_ACCESS_TOKEN=... | **IndiaEquity** | الهند (NSE/BSE) | T+1، ونطاقات القاطع (circuit)، وحزمة تكاليف STT / الدمغة / SEBI / GST قابلة للتهيئة | | **KoreaEquity** | كوريا (KRX: KOSPI/KOSDAQ) | شراء فقط، ونطاق ±30% يُحكم عليه لحظة التنفيذ على شبكة الخطوة السعرية الموحّدة، وضريبة تداول 0.20% لعام 2026 | | **VietnamEquity** | فيتنام (HOSE) | شراء فقط، واحتجاز تسوية T+2، ونطاق ±7% على شبكة الخطوة السعرية 10/50/100 دونغ، ولوت 100 سهم، وضريبة 0.1% على البيع | +| **BrazilEquity** | Brazil (B3) | BRL, B3 trading + settlement fees, configurable broker fee, 1- or 100-share lots, R$0.01 tick, shorting allowed | | **Crypto** | crypto فوري / عقود USD-M الدائمة | تسويات التمويل، وفصل سعر التنفيذ عن سعر العلامة | | **ChinaFutures** · **GlobalFutures** | العقود الآجلة | الهامش، ومضاعِفات العقد | | **Forex** | FX / المعادن | عبر مُحمّل `mt5` | @@ -971,7 +977,7 @@ Vibe-Trading وكيل كثيف الأدوات؛ المهارات والاختب vibe-trading # interactive TUI vibe-trading run -p "..." # single run vibe-trading serve # API server -vibe-trading alpha list # استعرض 462 ألفا جاهز؛ متاح show / bench / compare / export-manifest +vibe-trading alpha list # استعرض 475 ألفا جاهز؛ متاح show / bench / compare / export-manifest vibe-trading playbook list # خمسة قوالب بحث مجدولة؛ متاح show / create vibe-trading channels status --local # فحص إعدادات قنوات IM وتلميحات التثبيت vibe-trading provider doctor # طباعة تشخيصات المزود/الوكيل/الحزم بعد إخفاء الأسرار @@ -1716,7 +1722,7 @@ Vibe-Trading/ │ │ ├── memory/ # Cross-session persistent memory │ │ │ └── persistent.py # file-based memory (~/.vibe-trading/memory/) │ │ │ -│ │ ├── tools/ # 109 auto-discovered agent tools +│ │ ├── tools/ # 110 auto-discovered agent tools │ │ │ ├── backtest_tool.py # run backtests │ │ │ ├── remember_tool.py # cross-session memory (save/recall/forget) │ │ │ ├── skill_writer_tool.py # skill CRUD (save/patch/delete/file) @@ -1725,7 +1731,7 @@ Vibe-Trading/ │ │ │ ├── web_search_tool.py # DuckDuckGo web search │ │ │ └── ... # bash, file I/O, factor analysis, options, alpha browser + bench, etc. │ │ │ -│ │ ├── factors/ # Alpha Zoo — 462 ألفا عبر 5 families +│ │ ├── factors/ # Alpha Zoo — 475 ألفا عبر 5 families │ │ │ ├── base.py # 19 عاملاً (rank/scale/ts_*/delta/decay_linear/safe_div/vwap) │ │ │ ├── registry.py # تحميل بيانات وصفية AST فقط + حساب كسول + بوابات سلامة │ │ │ ├── bench_runner.py # IC + تصنيف alive/reversed/dead @@ -1741,8 +1747,8 @@ Vibe-Trading/ │ │ └── providers/ # LLM provider abstraction │ │ │ └── backtest/ # Backtest engines -│ ├── engines/ # 9 engines + composite cross-market engine + options_portfolio -│ ├── loaders/ # 28 sources: tushare, okx, nobitex, wallex, binance, yfinance, akshare, baostock, tencent, mootdx, ccxt, futu, pykrx, local, eastmoney, sina, stooq, yahoo, finnhub, alphavantage, tiingo, fmp, longbridge, mt5, qveris, india_broker, tickerall, gildata +│ ├── engines/ # 10 engines + composite cross-market engine + options_portfolio +│ ├── loaders/ # 29 sources: tushare, okx, nobitex, wallex, binance, yfinance, akshare, baostock, tencent, mootdx, ccxt, futu, pykrx, local, eastmoney, sina, stooq, yahoo, finnhub, alphavantage, tiingo, fmp, longbridge, mt5, qveris, india_broker, tickerall, gildata, b3_cotahist │ │ ├── base.py # DataLoader Protocol │ │ └── registry.py # Registry + auto-fallback chains │ └── optimizers/ # MVO, equal vol, max div, risk parity @@ -1808,7 +1814,7 @@ Vibe-Trading جزء من نظام وكلاء **[HKUDS](https://github.com/HKUDS) | **Data Bridge** | أحضر بياناتك: موصلات CSV/Parquet/SQL محلية مع schema mapping | المُحمِّل المحلي شُحن | | **Options Lab** | سطح تقلب، ولوحة Greeks، ومستكشف payoff/scenario | مخطط | | **Portfolio Studio** | أشعة مخاطر، وقيود، ومحسن يراعي الدوران، وملاحظات إعادة توازن | محسن يراعي الدوران **تم الإطلاق 0.1.11**؛ الباقي مخطط | -| **Alpha Zoo** | 462 ألفا كمّي جاهز (Qlib 158 + Kakushadze 101 + GTJA 191 + academic + fundamental)، سطر أوامر واحد للـ bench، تكامل agent، وواجهة Web | **تم الإطلاق 0.1.8**، موسّع حتى 0.1.12 | +| **Alpha Zoo** | 475 ألفا كمّي جاهز (Qlib 158 + Kakushadze 101 + GTJA 191 + academic + fundamental)، سطر أوامر واحد للـ bench، تكامل agent، وواجهة Web | **تم الإطلاق 0.1.8**، موسّع حتى 0.1.12 | | **Strategy Development Manager** | تسجيل الأوراق البحثية / أبحاث الوسطاء كعوامل واستراتيجيات مع مخزن دائم + دورة حياة آلية لاضمحلال IC/Sharpe | **تم الإطلاق 0.1.11** | | **Correlation Regime** | جدول زمني لنظام الارتباط قائم على كثافة الحواف + التباطؤ (hysteresis) فوق `/correlation` — رصد متى تندمج الأسواق في كتلة واحدة | **تم الإطلاق 0.1.12** | | **Research Delivery** | موجزات مجدولة وجلسات بحث حي عبر Slack / Telegram / قنوات IM شبيهة بالبريد | المُجدوِل + IM Runtime شُحنا | diff --git a/README_es.md b/README_es.md index c1eb5fdd..260f3bec 100644 --- a/README_es.md +++ b/README_es.md @@ -52,16 +52,19 @@ > ⚠️ **Advertencia de seguridad:** la cuenta de X `VibeTrading_HKU`, el proyecto de Virtuals `101845` y el contrato de token `0x640BDBF77b6447E8b7DB7894cED84BD1c40571f4` no son activos oficiales de Vibe-Trading. Nunca hemos lanzado ni respaldado ningún token o memecoin. No compres, conectes una wallet ni firmes nada. [Detalles](SECURITY.md#official-channels--impersonation). +- **2026-10-12** 🧮 **Actualizaciones de investigación y backtesting**: corregidos la ejecución inicial, la atribución al liquidar, la duración de caídas, la tolerancia de valoración y el momento de los informes SEC en el IC ([#1780](https://github.com/HKUDS/Vibe-Trading/pull/1780)–[#1794](https://github.com/HKUDS/Vibe-Trading/pull/1794)). Se añaden 13 factores fundamentales y de valoración, datos y backtests de Brasil, plantillas ZimaOS de activación explícita y progreso en Feishu. Alcance y límites en [CHANGELOG](CHANGELOG.md). + +- **2026-10-11** 🛠️ **Cinco correcciones prioritarias**: cada dato de un cálculo necesita evidencia; se bloquean las opciones reales aún no soportadas. Los índices de referencia sin conexión conservan su fuente, los registros distinguen cierres sin ganancias ni pérdidas y Codex conserva argumentos opcionales ([#1622](https://github.com/HKUDS/Vibe-Trading/issues/1622), [#1779](https://github.com/HKUDS/Vibe-Trading/pull/1779), [#1781](https://github.com/HKUDS/Vibe-Trading/pull/1781), [#1785](https://github.com/HKUDS/Vibe-Trading/pull/1785)). + - **2026-10-10** 🛠️ **Configuración de Slack/Discord y análisis reproducible**: Los formularios traducidos verifican credenciales del bot sin guardar cambios ([#1764](https://github.com/HKUDS/Vibe-Trading/pull/1764), [#1777](https://github.com/HKUDS/Vibe-Trading/pull/1777)). El análisis de cartera reutiliza instantáneas inmutables y la Web exporta JSON sin datos sensibles ([#1769](https://github.com/HKUDS/Vibe-Trading/pull/1769)). Los indicadores informan la fecha límite y la huella de los datos. Los registros de sesión dañados se recuperan sin reescribir los originales. También se corrigen búsqueda de memoria, filtrado de credenciales e importación de operaciones. Consulta [CHANGELOG](CHANGELOG.md). +
+Noticias anteriores + - **2026-10-09** 🛠️ **Configuración de canales y correcciones de backtests**: Las actualizaciones sin reconexión conservan las credenciales de las URL y la dirección del servidor autenticado de WeChat; la web explica guardados y reinicios en nueve idiomas ([#1737](https://github.com/HKUDS/Vibe-Trading/pull/1737), [#1742](https://github.com/HKUDS/Vibe-Trading/pull/1742), [#1744](https://github.com/HKUDS/Vibe-Trading/pull/1744)). La volatilidad de opciones entre mercados usa solo marcas temporales conocidas en cada barra; la caída máxima incluye el capital inicial. Corregimos las rutas de criptoactivos y metales preciosos, los duplicados de Hong Kong, la recuperación de caché, los importes en notación científica y la validación de valores ausentes. mootdx sale de la cadena predeterminada de acciones A. Los trece PR están en [CHANGELOG](CHANGELOG.md). - **2026-10-08** 🔐 **Informes PDF protegidos y lecturas del bróker corregidas**: Los PDF de correo programado admiten protección AES-256 opcional en Web, CLI y confirmaciones del agente; la contraseña permanece en la configuración privada del canal ([#1709](https://github.com/HKUDS/Vibe-Trading/pull/1709)). KIS consulta las tres rutas, renueva una sola vez un token explícitamente caducado y rechaza lecturas incompletas ([#1726](https://github.com/HKUDS/Vibe-Trading/pull/1726)). Las fórmulas conservan la expresión tras las etiquetas descriptivas y mantienen la verificación de evidencias ([#1728](https://github.com/HKUDS/Vibe-Trading/pull/1728)). - -
-Noticias anteriores - - **2026-10-07** 🛠️ **Continuidad de investigación y métricas de riesgo corregidas**: El estado de ejecución se separa del mensaje del sistema para preservar su prefijo estable ([#1708](https://github.com/HKUDS/Vibe-Trading/pull/1708)). Se corrigen las fechas entre mercados, los retornos alrededor de precios ausentes y el Sortino de opciones con toda la muestra ([#1710](https://github.com/HKUDS/Vibe-Trading/pull/1710), [#1717](https://github.com/HKUDS/Vibe-Trading/pull/1717), [#1727](https://github.com/HKUDS/Vibe-Trading/pull/1727)). Los cargadores comparten un contrato OHLCV; también se corrigen valoración crediticia, exposiciones, validación temporal y diagnóstico de auditoría. - **2026-10-06** 🛠️ **Control de operaciones reales y corrección de informes**: Detener el agente de operaciones reales cancela el análisis en curso y espera a que termine el planificador, incluso al cancelar durante el inicio o cerrar la API; se corrigen también las unidades de tiempo del estado ([#1704](https://github.com/HKUDS/Vibe-Trading/pull/1704)). La cancelación y el cierre de posiciones de emergencia omiten registros inválidos y continúan con los restantes ([#1703](https://github.com/HKUDS/Vibe-Trading/pull/1703)); cada bróker conserva su propia programación y las escrituras manejan concurrencia y escrituras parciales. Las sesiones reducidas de acciones estadounidenses respetan el cierre anticipado ([#1706](https://github.com/HKUDS/Vibe-Trading/pull/1706)). Las correcciones conservan las cifras verificadas y no presentan las cifras omitidas de los comentarios limitados como aprobadas ([#1702](https://github.com/HKUDS/Vibe-Trading/pull/1702)). @@ -376,7 +379,7 @@ Está diseñado para investigación, simulación y backtesting, y, cuando tú lo | **Ejecutar equipos de analistas** | Revisiones de investigación multiagente para flujos de trabajo de inversión, cuantitativo, cripto, macro y riesgo. | | **Llevar la investigación a canales de mensajería** | Ejecuta el mismo runtime de sesión a través de WebSocket, Telegram, Slack, Discord, Matrix, WhatsApp, Signal, QQ/NapCat, WeChat/WeCom, Feishu/Lark, DingTalk, Teams, correo electrónico y Mochat, con controles de CLI, REST y Web UI. | | **Entregar artefactos utilizables** | Informes, TradingView Pine Script, TDX, MetaTrader 5, herramientas MCP y sesiones de investigación posteriores. | -| **Evaluar un zoológico de alfas preconstruido** | Clasificación IC + viva/invertida/muerta en una línea, en 462 alfas (Qlib 158 + Kakushadze 101 + GTJA 191 + académicas + fundamentales PIT-safe) sobre tu universo. | +| **Evaluar un zoológico de alfas preconstruido** | Clasificación IC + viva/invertida/muerta en una línea, en 475 alfas (Qlib 158 + Kakushadze 101 + GTJA 191 + académicas + fundamentales PIT-safe) sobre tu universo. | | **Detectar regímenes de correlación** | Una línea de tiempo de densidad de aristas + histéresis en la superficie `/correlation` que muestra cuándo los mercados se fusionan en un solo bloque: contexto de riesgo descriptivo, no una señal. | --- @@ -460,7 +463,7 @@ La mayoría de las ejecuciones siguen la misma ruta de evidencia: enrutar la sol ## 📡 Fuentes de Datos y Fallback Inteligente -Una sola llamada `get_market_data`, **28 fuentes de datos de mercado**, una de ellas el mercado premium opcional **QVeris** (además del mercado premium opcional **QVeris**). Establece `source: "auto"`: el cargador elige según el símbolo y luego recorre una cadena por mercado ordenada por **riesgo de bloqueo de IP**: primero las fuentes públicas que nunca se bloquean, al final las limitadas o que requieren clave. Cero configuración, sin punto único de fallo. +Una sola llamada `get_market_data`, **29 fuentes de datos de mercado**, una de ellas el mercado premium opcional **QVeris** (además del mercado premium opcional **QVeris**). Establece `source: "auto"`: el cargador elige según el símbolo y luego recorre una cadena por mercado ordenada por **riesgo de bloqueo de IP**: primero las fuentes públicas que nunca se bloquean, al final las limitadas o que requieren clave. Cero configuración, sin punto único de fallo. | Fuente | Mercados | Autenticación | Rol | |--------|---------|------|------| @@ -483,6 +486,7 @@ Una sola llamada `get_market_data`, **28 fuentes de datos de mercado**, una de e | `tickerall` | forex / metales | clave + cuenta (solo lectura) | el mismo feed MT5 del bróker, **alojado**: sin terminal local y en cualquier sistema operativo (solo por selección explícita, nunca en el respaldo automático) | | `pykrx` | Corea (KRX: KOSPI/KOSDAQ) | ninguna | barras diarias de KOSPI / KOSDAQ para `.KS` / `.KQ` (extra opcional `krx`) | | `india_broker` | India (NSE/BSE) | login de broker | barras de solo lectura de Zerodha / Shoonya / Dhan para `.NS` / `.BO` (al final de la cadena de fallback) | +| `b3_cotahist` | Brazil (B3) | none | barras diarias oficiales del mercado al contado y de lotes fraccionarios (sin ajustar, BRL), importadas con `python -m src.br_data sync`; alternativa local para `.SA` | | `local` | cualquiera | ninguna | tu propio CSV / Parquet / DuckDB mediante el prefijo `local:` | **Cadenas de fallback (por riesgo de bloqueo de IP):** @@ -491,6 +495,7 @@ Una sola llamada `get_market_data`, **28 fuentes de datos de mercado**, una de e - **EE. UU.** → `yahoo` · `stooq` · `sina` · `eastmoney` · `yfinance` · `tiingo` · `fmp` · `finnhub` · `alphavantage` · `longbridge` · `akshare` · `local` - **HK** → `tencent` · `eastmoney` · `yahoo` · `futu` · `akshare` · `yfinance` · `tushare` · `longbridge` · `local` - **India (NSE/BSE)** → `yahoo` · `yfinance` · `india_broker` · `local` +- **Brazil (B3 `.SA`)** → `yahoo` · `yfinance` · `b3_cotahist` · `local` - **Corea (KOSPI/KOSDAQ)** → `pykrx` · `yahoo` · `yfinance` · `local` - **Reino Unido (LSE)** → `yahoo` · `yfinance` · `local` *(solo cotizaciones declaradas en GBP/GBp)* - **Cripto** → `okx` · `binance` · `ccxt` · `yfinance` · `local` @@ -668,9 +673,9 @@ junto al catálogo incluido (los archivos con el mismo nombre lo sobrescriben, i
-Alpha Zoo 462 alphas cuantitativos preconstruidos en 5 familias +Alpha Zoo 475 alphas cuantitativos preconstruidos en 5 familias -- 🧬 462 alphas cross-sectional, con prohibición de lookahead a nivel de la capa de operadores +- 🧬 475 alphas cross-sectional, con prohibición de lookahead a nivel de la capa de operadores - 📈 Categorización de IC + IR + vivo/invertido/muerto en un solo comando de la CLI - 🔬 Puerta de pureza AST + test centinela de lookahead de 300 filas + interruptor de apagado de red `pytest-socket` - 📦 Atribución Apache-2 para Qlib; `LICENSE.md` por zoo que declara las fórmulas como contenido matemático @@ -689,7 +694,7 @@ Ejecuta `vibe-trading alpha list` para explorar, `vibe-trading alpha show `
-Motores de Backtest 10 motores + cartera de opciones, composite cross-market +Motores de Backtest 11 motores + cartera de opciones, composite cross-market | Motor | Mercado | Notas | |--------|--------|-------| @@ -697,6 +702,7 @@ Ejecuta `vibe-trading alpha list` para explorar, `vibe-trading alpha show ` | **GlobalEquity** | EE. UU. / HK / Canadá / Reino Unido | trading en la misma sesión; lotes, ticks, divisas de liquidación y costos específicos de cada mercado | | **IndiaEquity** | India (NSE/BSE) | T+1, bandas de circuito, pila de costos STT / stamp / SEBI / GST basada en configuración | | **KoreaEquity** | Corea (KRX: KOSPI/KOSDAQ) | solo largo, banda de ±30% evaluada en el momento de ejecución sobre la malla de ticks unificada, impuesto de transacción del 0.20% en 2026 | +| **BrazilEquity** | Brazil (B3) | BRL, B3 trading + settlement fees, configurable broker fee, 1- or 100-share lots, R$0.01 tick, shorting allowed | | **Crypto** | spot cripto / perps USD-M | liquidaciones de funding, división ejecución/mark | | **ChinaFutures** · **GlobalFutures** | futuros | margen, multiplicadores de contrato | | **Forex** | FX / metales | vía el loader `mt5` | @@ -1009,7 +1015,7 @@ La TUI interactiva (`vibe-trading`) ahora usa una transcripción nativa de termi vibe-trading # interactive TUI vibe-trading run -p "..." # single run vibe-trading serve # API server -vibe-trading alpha list # explora 462 alphas preconstruidos; subcomandos show / bench / compare / export-manifest disponibles +vibe-trading alpha list # explora 475 alphas preconstruidos; subcomandos show / bench / compare / export-manifest disponibles vibe-trading playbook list # cinco plantillas de investigación programada; subcomandos show / create disponibles vibe-trading channels status --local # inspecciona la configuración de canales IM y sugerencias de instalación vibe-trading provider doctor # imprime diagnósticos redactados de provider/proxy/paquetes @@ -1804,7 +1810,7 @@ Vibe-Trading/ │ │ ├── memory/ # Memoria persistente entre sesiones │ │ │ └── persistent.py # memoria basada en archivos (~/.vibe-trading/memory/) │ │ │ -│ │ ├── tools/ # 109 herramientas de agente autodescubiertas +│ │ ├── tools/ # 110 herramientas de agente autodescubiertas │ │ │ ├── backtest_tool.py # ejecuta backtests │ │ │ ├── remember_tool.py # memoria entre sesiones (save/recall/forget) │ │ │ ├── skill_writer_tool.py # CRUD de skills (save/patch/delete/file) @@ -1813,7 +1819,7 @@ Vibe-Trading/ │ │ │ ├── web_search_tool.py # búsqueda web con DuckDuckGo │ │ │ └── ... # bash, E/S de archivos, análisis de factores, opciones, navegador de alpha + bench, etc. │ │ │ -│ │ ├── factors/ # Alpha Zoo — 462 alphas en 5 familias +│ │ ├── factors/ # Alpha Zoo — 475 alphas en 5 familias │ │ │ ├── base.py # 19 operadores (rank/scale/ts_*/delta/decay_linear/safe_div/vwap) │ │ │ ├── registry.py # carga de metadatos solo por AST + cómputo diferido + puertas de sanidad │ │ │ ├── bench_runner.py # IC + categorización alive/reversed/dead @@ -1829,8 +1835,8 @@ Vibe-Trading/ │ │ └── providers/ # Abstracción de proveedores LLM │ │ │ └── backtest/ # Motores de backtest -│ ├── engines/ # 9 motores + motor compuesto multi-mercado + options_portfolio -│ ├── loaders/ # 28 fuentes: tushare, okx, nobitex, wallex, binance, yfinance, akshare, baostock, tencent, mootdx, ccxt, futu, pykrx, local, eastmoney, sina, stooq, yahoo, finnhub, alphavantage, tiingo, fmp, longbridge, mt5, qveris, india_broker, tickerall, gildata +│ ├── engines/ # 10 motores + motor compuesto multi-mercado + options_portfolio +│ ├── loaders/ # 29 fuentes: tushare, okx, nobitex, wallex, binance, yfinance, akshare, baostock, tencent, mootdx, ccxt, futu, pykrx, local, eastmoney, sina, stooq, yahoo, finnhub, alphavantage, tiingo, fmp, longbridge, mt5, qveris, india_broker, tickerall, gildata, b3_cotahist │ │ ├── base.py # Protocolo DataLoader │ │ └── registry.py # Registro + cadenas de fallback automáticas │ └── optimizers/ # MVO, equal vol, max div, risk parity @@ -1896,7 +1902,7 @@ Vibe-Trading forma parte del ecosistema de agentes de **[HKUDS](https://github.c | **Data Bridge** | Trae tus propios datos: conectores locales CSV/Parquet/SQL con mapeo de esquema | Cargador local publicado | | **Options Lab** | Superficie de volatilidad, panel de Greeks, explorador de payoff/escenarios | Herramienta analítica de payoff/escenarios **publicada**; superficie/panel planificados | | **Portfolio Studio** | Risk x-ray, restricciones, optimizador con reducción de turnover, notas de rebalanceo | Optimizador con reducción de turnover **publicado en 0.1.11**; el resto planificado | -| **Alpha Zoo** | 462 alphas preconstruidas (Qlib 158 + Kakushadze 101 + GTJA 191 + academic + fundamental) con benchmark en una línea, integración con el agente y Web UI | **Publicado en 0.1.8**, ampliado hasta 0.1.12 | +| **Alpha Zoo** | 475 alphas preconstruidas (Qlib 158 + Kakushadze 101 + GTJA 191 + academic + fundamental) con benchmark en una línea, integración con el agente y Web UI | **Publicado en 0.1.8**, ampliado hasta 0.1.12 | | **Strategy Development Manager** | Registra papers / research de brokers como factores y estrategias con un almacén persistente + lifecycle automatizado de decaimiento de IC/Sharpe | **Publicado en 0.1.11** | | **Correlation Regime** | Línea temporal de régimen por edge-density + histéresis, superpuesta en `/correlation` — detecta cuándo los mercados se fusionan en un solo bloque | **Publicado en 0.1.12** | | **Research Delivery** | Briefs programados y sesiones de investigación en vivo a través de canales IM tipo Slack / Telegram / correo | Scheduler + runtime de IM publicados | diff --git a/README_id.md b/README_id.md index 91e4cf6b..295b5ee6 100644 --- a/README_id.md +++ b/README_id.md @@ -53,16 +53,19 @@ > ⚠️ **Peringatan keamanan:** Akun X `VibeTrading_HKU`, proyek Virtuals `101845`, dan kontrak token `0x640BDBF77b6447E8b7DB7894cED84BD1c40571f4` bukan aset resmi Vibe-Trading. Kami tidak pernah meluncurkan atau mendukung token maupun memecoin apa pun. Jangan membeli, menghubungkan wallet, atau menandatangani apa pun. [Detail](SECURITY.md#official-channels--impersonation). +- **2026-10-12** 🧮 **Pembaruan riset dan backtest**: memperbaiki eksekusi pembukaan, atribusi saat likuidasi, durasi drawdown, toleransi valuasi, dan waktu laporan SEC dalam IC faktor ([#1780](https://github.com/HKUDS/Vibe-Trading/pull/1780)–[#1794](https://github.com/HKUDS/Vibe-Trading/pull/1794)). Menambahkan 13 faktor fundamental/valuasi, data dan backtest Brasil, templat ZimaOS dengan aktivasi eksplisit, serta progres Feishu. Cakupan dan batasan ada di [CHANGELOG](CHANGELOG.md). + +- **2026-10-11** 🛠️ **Lima perbaikan prioritas**: setiap nilai data dalam perhitungan turunan memerlukan bukti; opsi perdagangan riil yang belum didukung tetap diblokir. Benchmark luring mempertahankan sumber pilihan, catatan membedakan penutupan impas, dan alat Codex mempertahankan argumen opsional ([#1622](https://github.com/HKUDS/Vibe-Trading/issues/1622), [#1779](https://github.com/HKUDS/Vibe-Trading/pull/1779), [#1781](https://github.com/HKUDS/Vibe-Trading/pull/1781), [#1785](https://github.com/HKUDS/Vibe-Trading/pull/1785)). + - **2026-10-10** 🛠️ **Pengaturan Slack/Discord dan riset yang dapat direproduksi**: Formulir terjemahan memverifikasi kredensial bot tanpa menyimpan perubahan ([#1764](https://github.com/HKUDS/Vibe-Trading/pull/1764), [#1777](https://github.com/HKUDS/Vibe-Trading/pull/1777)). Riset portofolio dapat memakai ulang snapshot yang tidak berubah; Web mengekspor JSON tanpa data sensitif ([#1769](https://github.com/HKUDS/Vibe-Trading/pull/1769)). Indikator mencatat batas tanggal dan sidik data masukan. Catatan sesi rusak dipulihkan tanpa menulis ulang berkas asli. Pencarian memori, penyamaran kredensial, dan impor transaksi juga diperbaiki. Lihat [CHANGELOG](CHANGELOG.md). +
+Berita sebelumnya + - **2026-10-09** 🛠️ **Pengaturan kanal dan perbaikan backtest**: Pembaruan tanpa menyambungkan ulang mempertahankan kredensial URL dan alamat server WeChat hasil autentikasi; web menjelaskan hasil penyimpanan dan mulai ulang dalam sembilan bahasa ([#1737](https://github.com/HKUDS/Vibe-Trading/pull/1737), [#1742](https://github.com/HKUDS/Vibe-Trading/pull/1742), [#1744](https://github.com/HKUDS/Vibe-Trading/pull/1744)). Volatilitas opsi lintas pasar hanya memakai waktu yang sudah diketahui pada setiap bar; drawdown mencakup modal awal. Perbaikan meliputi rute data kripto/logam mulia, duplikat saham Hong Kong, pemulihan cache, nominal notasi ilmiah, dan validasi nilai ekuitas yang hilang. mootdx dikeluarkan dari rantai saham A bawaan. Ketiga belas PR tercantum dalam [CHANGELOG](CHANGELOG.md). - **2026-10-08** 🔐 **Laporan PDF terlindungi dan pembacaan broker diperbaiki**: PDF Email terjadwal mendukung perlindungan AES-256 opsional melalui Web, CLI, dan konfirmasi agen; kata sandi hanya disimpan dalam konfigurasi privat kanal ([#1709](https://github.com/HKUDS/Vibe-Trading/pull/1709)). KIS menanyakan ketiga jalur perdagangan, memperbarui token yang secara eksplisit kedaluwarsa satu kali, dan menolak pembacaan yang tidak lengkap ([#1726](https://github.com/HKUDS/Vibe-Trading/pull/1726)). Rumus turunan tetap utuh setelah label deskriptif dengan pemeriksaan bukti tetap berlaku ([#1728](https://github.com/HKUDS/Vibe-Trading/pull/1728)). - -
-Berita sebelumnya - - **2026-10-07** 🛠️ **Kesinambungan riset dan koreksi metrik risiko**: Status eksekusi dipindahkan dari prompt sistem agar prefiksnya tetap stabil ([#1708](https://github.com/HKUDS/Vibe-Trading/pull/1708)). Tanggal lintas pasar, imbal hasil di sekitar harga yang hilang, dan Sortino opsi berdasarkan seluruh sampel diperbaiki ([#1710](https://github.com/HKUDS/Vibe-Trading/pull/1710), [#1717](https://github.com/HKUDS/Vibe-Trading/pull/1717), [#1727](https://github.com/HKUDS/Vibe-Trading/pull/1727)). Pengujian loader memakai satu kontrak OHLCV, disertai koreksi valuasi kredit, eksposur, validasi berjalan, dan diagnosis audit. - **2026-10-06** 🛠️ **Kontrol trading langsung dan koreksi laporan**: Menghentikan runner trading langsung membatalkan analisis aktif dan menunggu penjadwal selesai, termasuk pembatalan saat mulai dan penutupan API; satuan waktu pada status juga diperbaiki ([#1704](https://github.com/HKUDS/Vibe-Trading/pull/1704)). Pembatalan order dan penutupan posisi darurat melewati catatan tidak valid lalu melanjutkan catatan lainnya ([#1703](https://github.com/HKUDS/Vibe-Trading/pull/1703)); jadwal setiap broker disimpan terpisah dengan penanganan penulisan bersamaan dan parsial. Sesi saham AS yang dipersingkat mengikuti waktu tutup lebih awal ([#1706](https://github.com/HKUDS/Vibe-Trading/pull/1706)). Koreksi laporan mempertahankan angka yang lolos verifikasi, dan umpan balik terbatas tidak menyiratkan bahwa angka yang tidak tercantum sudah lolos ([#1702](https://github.com/HKUDS/Vibe-Trading/pull/1702)). @@ -499,7 +502,7 @@ Vibe-Trading dirancang untuk riset, simulasi, dan backtest — serta, jika Anda | **Jalankan tim analis** | Review riset multi-agent untuk workflow investasi, quant, crypto, makro, dan risiko. | | **Bawa riset ke channel IM** | Jalankan runtime sesi yang sama melalui WebSocket, Telegram, Slack, Discord, Matrix, WhatsApp, Signal, QQ/NapCat, WeChat/WeCom, Feishu/Lark, DingTalk, Teams, email, dan Mochat dengan kontrol CLI, REST, dan Web UI. | | **Hasilkan artefak yang siap digunakan** | Laporan, TradingView Pine Script, TDX, MetaTrader 5, tool MCP, dan sesi riset lanjutan. | -| **Benchmark alpha zoo siap pakai** | Satu baris untuk IC + kategorisasi alive/reversed/dead pada 462 alpha (Qlib 158 + Kakushadze 101 + GTJA 191 + akademik + fundamental PIT-safe) di universe Anda. | +| **Benchmark alpha zoo siap pakai** | Satu baris untuk IC + kategorisasi alive/reversed/dead pada 475 alpha (Qlib 158 + Kakushadze 101 + GTJA 191 + akademik + fundamental PIT-safe) di universe Anda. | | **Deteksi regime korelasi** | Timeline edge-density + hysteresis pada surface `/correlation` yang menunjukkan kapan market menyatu menjadi satu blok — konteks risiko deskriptif, bukan sinyal. | --- @@ -622,7 +625,7 @@ the wall-clock activity watchdog. ## 📡 Sumber Data & Smart Fallback -Satu call `get_market_data`, **28 sumber data market**, salah satunya marketplace premium opsional **QVeris**. Atur `source: "auto"` — loader memilih berdasarkan simbol lalu mengikuti chain per market yang diurutkan berdasarkan **risiko IP-ban**: sumber publik yang tidak pernah diblokir lebih dulu, sumber throttled / membutuhkan key belakangan. Zero-config, tanpa single point of failure. +Satu call `get_market_data`, **29 sumber data market**, salah satunya marketplace premium opsional **QVeris**. Atur `source: "auto"` — loader memilih berdasarkan simbol lalu mengikuti chain per market yang diurutkan berdasarkan **risiko IP-ban**: sumber publik yang tidak pernah diblokir lebih dulu, sumber throttled / membutuhkan key belakangan. Zero-config, tanpa single point of failure. | Sumber | Market | Auth | Peran | |--------|---------|------|------| @@ -645,6 +648,7 @@ Satu call `get_market_data`, **28 sumber data market**, salah satunya marketplac | `tickerall` | forex / metals | key + akun (read-only) | feed broker MT5 yang sama, **hosted** — tanpa terminal lokal, OS apa pun (hanya eksplisit, tidak pernah auto fallback) | | `pykrx` | Korea (KRX: KOSPI/KOSDAQ) | tidak ada | bar harian KOSPI / KOSDAQ untuk `.KS` / `.KQ` (extra `krx` opsional) | | `india_broker` | India (NSE/BSE) | login broker | bar Zerodha / Shoonya / Dhan read-only untuk `.NS` / `.BO` (ujung fallback chain) | +| `b3_cotahist` | Brazil (B3) | none | bar harian resmi B3 untuk pasar tunai/odd-lot (mentah, BRL), diimpor dengan `python -m src.br_data sync`; cadangan luring untuk `.SA` | | `local` | apa pun | tidak ada | CSV / Parquet / DuckDB Anda sendiri melalui prefix `local:` | **Fallback chain (berdasarkan risiko IP-ban):** @@ -653,6 +657,7 @@ Satu call `get_market_data`, **28 sumber data market**, salah satunya marketplac - **US** → `yahoo` · `stooq` · `sina` · `eastmoney` · `yfinance` · `tiingo` · `fmp` · `finnhub` · `alphavantage` · `longbridge` · `akshare` · `local` - **HK** → `tencent` · `eastmoney` · `yahoo` · `futu` · `akshare` · `yfinance` · `tushare` · `longbridge` · `local` - **India (NSE/BSE)** → `yahoo` · `yfinance` · `india_broker` · `local` +- **Brazil (B3 `.SA`)** → `yahoo` · `yfinance` · `b3_cotahist` · `local` - **Korea (KOSPI/KOSDAQ)** → `pykrx` · `yahoo` · `yfinance` · `local` - **UK (LSE)** → `yahoo` · `yfinance` · `local` *(declared GBP/GBp quotes only)* - **Crypto** → `okx` · `binance` · `ccxt` · `yfinance` · `local` @@ -828,9 +833,9 @@ bersama roster bawaan (file dengan nama sama akan override, seperti user skill)
-Alpha Zoo 462 alpha quant siap pakai dalam 5 keluarga +Alpha Zoo 475 alpha quant siap pakai dalam 5 keluarga -- 🧬 462 alpha cross-sectional, lookahead dilarang pada layer operator +- 🧬 475 alpha cross-sectional, lookahead dilarang pada layer operator - 📈 IC + IR + kategorisasi alive/reversed/dead dalam satu perintah CLI - 🔬 AST purity gate + test sentinel lookahead 300 baris + network kill-switch `pytest-socket` - 📦 Atribusi Apache-2 untuk Qlib; `LICENSE.md` per-zoo menyatakan formula sebagai konten matematis @@ -849,7 +854,7 @@ Jalankan `vibe-trading alpha list` untuk menjelajah, `vibe-trading alpha show
-Engine Backtest 10 engine + portofolio opsi, komposit lintas market +Engine Backtest 11 engine + portofolio opsi, komposit lintas market | Engine | Market | Catatan | |--------|--------|-------| @@ -858,6 +863,7 @@ Jalankan `vibe-trading alpha list` untuk menjelajah, `vibe-trading alpha show ⚠️ **セキュリティ警告：** Xアカウント `VibeTrading_HKU`、Virtualsプロジェクト `101845`、およびトークンコントラクト `0x640BDBF77b6447E8b7DB7894cED84BD1c40571f4` は、いずれもVibe-Trading公式のものではありません。Vibe-Tradingはこれまで、いかなるトークンやミームコインも発行・公認していません。購入、ウォレットの接続、署名は行わないでください。[詳細](SECURITY.md#official-channels--impersonation) +- **2026-10-12** 🧮 **調査とバックテストの更新**：初回約定、清算時の寄与分析、ドローダウン期間、評価許容誤差、因子 IC の SEC 開示タイミングを修正（[#1780](https://github.com/HKUDS/Vibe-Trading/pull/1780)–[#1794](https://github.com/HKUDS/Vibe-Trading/pull/1794)）。基本面・評価因子 13 種、ブラジル市場データとバックテスト、明示的に有効化する ZimaOS テンプレート、Feishu の実行進捗を追加。範囲と制約は [CHANGELOG](CHANGELOG.md) を参照。 + +- **2026-10-11** 🛠️ **優先度の高い 5 件を修正**：導出値の各データ項に根拠を要求し、未対応の実取引オプション注文を拒否。オフライン基準は指定ソースを維持し、取引記録は損益ゼロの決済を識別、Codex ツールは任意引数を保持します（[#1622](https://github.com/HKUDS/Vibe-Trading/issues/1622)、[#1779](https://github.com/HKUDS/Vibe-Trading/pull/1779)、[#1781](https://github.com/HKUDS/Vibe-Trading/pull/1781)、[#1785](https://github.com/HKUDS/Vibe-Trading/pull/1785)）。 + - **2026-10-10** 🛠️ **Slack/Discord 設定と再現可能な調査**：翻訳済みフォームでボット認証情報を検証できます。テストは変更を保存しません（[#1764](https://github.com/HKUDS/Vibe-Trading/pull/1764)、[#1777](https://github.com/HKUDS/Vibe-Trading/pull/1777)）。ポートフォリオ調査は不変のスナップショットを再利用でき、Web から認証情報や口座番号を除いた JSON を出力できます（[#1769](https://github.com/HKUDS/Vibe-Trading/pull/1769)）。指標は取得日と入力の指紋を報告し、破損したセッション記録は元のファイルを書き換えずに復旧します。メモリ検索、認証情報の秘匿、取引履歴の識別も修正しました。[CHANGELOG](CHANGELOG.md) を参照。 +
+過去のニュース + - **2026-10-09** 🛠️ **チャンネル設定とバックテストの修正**：再接続なしの設定更新で URL 内の認証情報と WeChat 認証後のサーバーアドレスを保持し、保存・再起動の結果を Web で9言語表示します（[#1737](https://github.com/HKUDS/Vibe-Trading/pull/1737)、[#1742](https://github.com/HKUDS/Vibe-Trading/pull/1742)、[#1744](https://github.com/HKUDS/Vibe-Trading/pull/1744)）。複数市場のオプション価格に使うボラティリティは各時点までのタイムスタンプだけで計算し、ドローダウンには初期資金を反映。暗号資産・貴金属のデータ取得経路、香港株の重複判定、キャッシュ復旧、科学表記の金額、欠損資産額の検証を修正しました。mootdx は標準の A 株取得経路から外れます。全13件は [CHANGELOG](CHANGELOG.md)。 - **2026-10-08** 🔐 **PDF レポートのパスワード保護とブローカー読み取り修正**：定期メール PDF に AES-256 保護を選択でき、Web、CLI、エージェント確認画面に反映されます。パスワードは非公開のチャネル設定だけに保存します（[#1709](https://github.com/HKUDS/Vibe-Trading/pull/1709)）。KIS は三つの取引経路を照会し、明示的なトークン失効を一度だけ再取得、不完全な読み取りはエラーにします（[#1726](https://github.com/HKUDS/Vibe-Trading/pull/1726)）。説明ラベルの後の導出式を保持し、証拠検証を維持します（[#1728](https://github.com/HKUDS/Vibe-Trading/pull/1728)）。 - -
-過去のニュース - - **2026-10-07** 🛠️ **リサーチの継続性とリスク指標の修正**：実行状態をシステムプロンプトから分離し、安定したプレフィックスを保ちます（[#1708](https://github.com/HKUDS/Vibe-Trading/pull/1708)）。市場間の日付整列、欠損価格前後のリターン、全標本によるオプションのソルティノ比率を修正しました（[#1710](https://github.com/HKUDS/Vibe-Trading/pull/1710)、[#1717](https://github.com/HKUDS/Vibe-Trading/pull/1717)、[#1727](https://github.com/HKUDS/Vibe-Trading/pull/1727)）。データ取得テストは共通の OHLCV 契約を使い、信用評価、スタイル・エクスポージャー、時系列検証、監査診断も改善しました。 - **2026-10-06** 🛠️ **ライブ運用制御とレポート修正**：ライブランナーの停止は実行中の分析をキャンセルしてスケジューラーの終了を待ち、起動中のキャンセルと API 終了にも対応します。状態の時刻単位も修正しました（[#1704](https://github.com/HKUDS/Vibe-Trading/pull/1704)）。緊急キャンセル・決済処理は不正なレコードを除外して残りの注文・保有を処理します（[#1703](https://github.com/HKUDS/Vibe-Trading/pull/1703)）。ブローカーごとにスケジュールを保存し、並行書き込みと部分書き込みに対応。米国株の短縮取引日は早い閉場時刻を守ります（[#1706](https://github.com/HKUDS/Vibe-Trading/pull/1706)）。レポート修正は検証済みの数値を保持し、省略された指摘を検証済みと誤認させません（[#1702](https://github.com/HKUDS/Vibe-Trading/pull/1702)）。 @@ -377,7 +380,7 @@ Vibe-Trading は、金融に関する問いを実行可能な分析へ変換す | **アナリストチームを走らせる** | 投資、クオンツ、暗号資産、マクロ、リスクのワークフロー向けマルチエージェント・リサーチレビュー。 | | **リサーチを IM チャンネルへ接続する** | WebSocket、Telegram、Slack、Discord、Matrix、WhatsApp、Signal、QQ/NapCat、WeChat/WeCom、Feishu/Lark、DingTalk、Teams、email、Mochat から同じ session runtime を CLI、REST、Web UI で管理。 | | **使える artifacts を出力する** | レポート、TradingView Pine Script、TDX、MetaTrader 5、MCP tools、後続リサーチセッション。 | -| **事前構築 alpha zoo をベンチ** | 462 個の alpha 因子（Qlib 158 + Kakushadze 101 + GTJA 191 + academic + PIT-safe fundamental）に対し、1 行 CLI で IC + IR + alive/reversed/dead 分類を実行 | +| **事前構築 alpha zoo をベンチ** | 475 個の alpha 因子（Qlib 158 + Kakushadze 101 + GTJA 191 + academic + PIT-safe fundamental）に対し、1 行 CLI で IC + IR + alive/reversed/dead 分類を実行 | | **相関レジームを見抜く** | `/correlation` 面上のエッジ密度 + ヒステリシスのタイムライン。市場が 1 つのブロックに融合するタイミングを示す —— シグナルではなく記述的なリスクコンテキスト。 | --- @@ -461,7 +464,7 @@ vibe-trading connector install /tmp/my-broker ## 📡 データソースとスマートフォールバック -1 回の `get_market_data` 呼び出しで **28 のマーケットデータソース**（うち **QVeris** はオプションの有料マーケットプレイス）にアクセスできます。`source: "auto"` を指定すれば、loader が銘柄に応じてソースを選び、**IP 規制リスク**の順に並んだ市場別チェーンをたどります。規制を受けない公開ソースを先に、スロットリングや key を要するソースを最後に試します。設定不要、単一障害点なし。 +1 回の `get_market_data` 呼び出しで **29 のマーケットデータソース**（うち **QVeris** はオプションの有料マーケットプレイス）にアクセスできます。`source: "auto"` を指定すれば、loader が銘柄に応じてソースを選び、**IP 規制リスク**の順に並んだ市場別チェーンをたどります。規制を受けない公開ソースを先に、スロットリングや key を要するソースを最後に試します。設定不要、単一障害点なし。 | Source | Markets | Auth | Role | |--------|---------|------|------| @@ -484,6 +487,7 @@ vibe-trading connector install /tmp/my-broker | `tickerall` | forex / metals | key + アカウント（読み取り専用） | 同じブローカーの MT5 フィードを**ホスト型**で — ローカル端末不要、OS を問わない（明示指定のみ、auto フォールバックには決して入らない） | | `pykrx` | 韓国（KRX：KOSPI/KOSDAQ） | 不要 | `.KS` / `.KQ` の KOSPI / KOSDAQ 日足（任意の `krx` extra） | | `india_broker` | インド（NSE/BSE） | ブローカーログイン | `.NS` / `.BO` 向けの読み取り専用 Zerodha / Shoonya / Dhan bars（フォールバックチェーン末尾） | +| `b3_cotahist` | Brazil (B3) | none | ローカル B3 公式現物・単元未満株の日足（未調整、BRL）。`python -m src.br_data sync` で取得する `.SA` のオフライン代替ソース | | `local` | any | none | your own CSV / Parquet / DuckDB via `local:` prefix | **フォールバックチェーン（IP 規制リスク順）：** @@ -492,6 +496,7 @@ vibe-trading connector install /tmp/my-broker - **米国株** → `yahoo` · `stooq` · `sina` · `eastmoney` · `yfinance` · `tiingo` · `fmp` · `finnhub` · `alphavantage` · `longbridge` · `akshare` · `local` - **香港株** → `tencent` · `eastmoney` · `yahoo` · `futu` · `akshare` · `yfinance` · `tushare` · `longbridge` · `local` - **インド株（NSE/BSE）** → `yahoo` · `yfinance` · `india_broker` · `local` +- **Brazil (B3 `.SA`)** → `yahoo` · `yfinance` · `b3_cotahist` · `local` - **韓国（KOSPI/KOSDAQ）** → `pykrx` · `yahoo` · `yfinance` · `local` - **英国（LSE）** → `yahoo` · `yfinance` · `local` *（GBP/GBp と明示されたクォートのみ）* - **暗号資産** → `okx` · `binance` · `ccxt` · `yfinance` · `local` @@ -662,9 +667,9 @@ Paper-vs-live is a **structural per-broker runtime guard** (account-id format, h
-Alpha Zoo 5 つのファミリーに渡る 462 個の事前構築 quant alpha +Alpha Zoo 5 つのファミリーに渡る 475 個の事前構築 quant alpha -- 🧬 462 個のクロスセクショナル alpha、オペレーター層でルックアヘッドを禁止 +- 🧬 475 個のクロスセクショナル alpha、オペレーター層でルックアヘッドを禁止 - 📈 IC + IR + alive/reversed/dead 分類を 1 つの CLI コマンドで - 🔬 AST 純関数ゲート + 300 行のルックアヘッド sentinel テスト + `pytest-socket` によるネットワーク遮断 - 📦 Qlib には Apache-2 帰属表示、各 zoo ごとに `LICENSE.md` で formula を数学的内容として宣言 @@ -683,7 +688,7 @@ Paper-vs-live is a **structural per-broker runtime guard** (account-id format, h
-Backtest Engines 10 engines + options portfolio, cross-market composite +Backtest Engines 11 engines + options portfolio, cross-market composite | Engine | Market | Notes | |--------|--------|-------| @@ -692,6 +697,7 @@ Paper-vs-live is a **structural per-broker runtime guard** (account-id format, h | **IndiaEquity** | India (NSE/BSE) | T+1, circuit bands, config-driven STT / stamp / SEBI / GST cost stack | | **KoreaEquity** | 韓国（KRX：KOSPI/KOSDAQ） | ロングオンリー、統一呼値グリッド上で ±30% 制限値幅を約定時点に判定、2026 年 0.20% の証券取引税 | | **VietnamEquity** | ベトナム（HOSE） | ロングオンリー、T+2 決済ホールド、10/50/100 ドンの呼値グリッド上で ±7% 制限値幅、100 株単元、売却側 0.1% 課税 | +| **BrazilEquity** | Brazil (B3) | BRL, B3 trading + settlement fees, configurable broker fee, 1- or 100-share lots, R$0.01 tick, shorting allowed | | **Crypto** | crypto spot / USD-M perps | funding settlements, execution/mark split | | **ChinaFutures** · **GlobalFutures** | futures | margin, contract multipliers | | **Forex** | FX / metals | via the `mt5` loader | @@ -976,7 +982,7 @@ Vibe-Trading は tool-heavy agent です。skills、backtests、memory、swarms vibe-trading # interactive TUI vibe-trading run -p "..." # single run vibe-trading serve # API server -vibe-trading alpha list # 462 個の事前構築 alpha を閲覧；show / bench / compare / export-manifest サブコマンド利用可 +vibe-trading alpha list # 475 個の事前構築 alpha を閲覧；show / bench / compare / export-manifest サブコマンド利用可 vibe-trading playbook list # 定期リサーチのテンプレート 5 本；show / create サブコマンド利用可 vibe-trading channels status --local # IM チャンネル設定と install hints を確認 vibe-trading provider doctor # 秘匿処理済みの provider/proxy/package 診断を出力 @@ -1724,7 +1730,7 @@ Vibe-Trading/ │ │ ├── memory/ # クロスセッション永続メモリ │ │ │ └── persistent.py # ファイルベースメモリ (~/.vibe-trading/memory/) │ │ │ -│ │ ├── tools/ # 109 個の自動検出エージェントツール +│ │ ├── tools/ # 110 個の自動検出エージェントツール │ │ │ ├── backtest_tool.py # バックテスト実行 │ │ │ ├── remember_tool.py # クロスセッションメモリ (save/recall/forget) │ │ │ ├── skill_writer_tool.py # skill CRUD (save/patch/delete/file) @@ -1733,7 +1739,7 @@ Vibe-Trading/ │ │ │ ├── web_search_tool.py # DuckDuckGo Web 検索 │ │ │ └── ... # bash、file I/O、factor analysis、options、alpha browser + bench など │ │ │ -│ │ ├── factors/ # Alpha Zoo — 5 つのファミリーにまたがる 462 個の alpha +│ │ ├── factors/ # Alpha Zoo — 5 つのファミリーにまたがる 475 個の alpha │ │ │ ├── base.py # 19 個のオペレーター (rank/scale/ts_*/delta/decay_linear/safe_div/vwap) │ │ │ ├── registry.py # AST 限定のメタデータ読み込み + 遅延計算 + sanity gate │ │ │ ├── bench_runner.py # IC + alive/reversed/dead 分類 @@ -1749,8 +1755,8 @@ Vibe-Trading/ │ │ └── providers/ # LLM プロバイダー抽象化 │ │ │ └── backtest/ # バックテストエンジン -│ ├── engines/ # 9 エンジン + クロスマーケット composite engine + options_portfolio -│ ├── loaders/ # 28 ソース: tushare、okx、nobitex、wallex、binance、yfinance、akshare、baostock、tencent、mootdx、ccxt、futu、pykrx、local、eastmoney、sina、stooq、yahoo、finnhub、alphavantage、tiingo、fmp、longbridge、mt5、qveris、india_broker、tickerall、gildata +│ ├── engines/ # 10 エンジン + クロスマーケット composite engine + options_portfolio +│ ├── loaders/ # 29 ソース: tushare、okx、nobitex、wallex、binance、yfinance、akshare、baostock、tencent、mootdx、ccxt、futu、pykrx、local、eastmoney、sina、stooq、yahoo、finnhub、alphavantage、tiingo、fmp、longbridge、mt5、qveris、india_broker、tickerall、gildata, b3_cotahist │ │ ├── base.py # DataLoader Protocol │ │ └── registry.py # Registry + 自動フォールバックチェーン │ └── optimizers/ # MVO、equal vol、max div、risk parity @@ -1816,7 +1822,7 @@ Vibe-Trading は **[HKUDS](https://github.com/HKUDS)** agent ecosystem の一部 | **Data Bridge** | Bring-your-own data: local CSV/Parquet/SQL connectors with schema mapping | ローカルローダー出荷済み | | **Options Lab** | Vol surface, Greeks dashboard, payoff/scenario explorer | Planned | | **Portfolio Studio** | Risk x-ray, constraints, turnover-aware optimizer, rebalance notes | Turnover を考慮したオプティマイザは **0.1.11 でリリース済み**；残りは Planned | -| **Alpha Zoo** | 462 個の事前構築 alpha 因子（Qlib 158 + Kakushadze 101 + GTJA 191 + academic + fundamental）、1 行 CLI でベンチ、agent 統合、Web UI | **0.1.8 でリリース済み**、0.1.12 まで拡張 | +| **Alpha Zoo** | 475 個の事前構築 alpha 因子（Qlib 158 + Kakushadze 101 + GTJA 191 + academic + fundamental）、1 行 CLI でベンチ、agent 統合、Web UI | **0.1.8 でリリース済み**、0.1.12 まで拡張 | | **Strategy Development Manager** | Register papers / broker research as factors & strategies with a persistent store + automated IC/Sharpe decay lifecycle | **0.1.11 でリリース済み** | | **Correlation Regime** | Edge-density + hysteresis regime timeline layered on `/correlation` — spot when markets fuse into one bloc | **0.1.12 でリリース済み** | | **Research Delivery** | Slack / Telegram / email-style IM channels 経由の scheduled briefs と live research sessions | スケジューラ + IM Runtime 出荷済み | diff --git a/README_ko.md b/README_ko.md index acfcf129..c7814264 100644 --- a/README_ko.md +++ b/README_ko.md @@ -52,16 +52,19 @@ > ⚠️ **보안 경고:** X 계정 `VibeTrading_HKU`, Virtuals 프로젝트 `101845`, 토큰 컨트랙트 `0x640BDBF77b6447E8b7DB7894cED84BD1c40571f4`는 모두 Vibe-Trading 공식과 무관합니다. Vibe-Trading은 어떠한 토큰이나 밈코인도 발행하거나 공식적으로 지지한 적이 없습니다. 해당 토큰을 구매하거나 지갑을 연결하거나 어떠한 서명도 하지 마세요. [자세히 보기](SECURITY.md#official-channels--impersonation). +- **2026-10-12** 🧮 **리서치와 백테스트 업데이트**: 최초 체결, 청산 시 성과 귀속, 낙폭 지속 기간, 가치평가 허용 오차, 팩터 IC의 SEC 공시 반영 시점을 수정했습니다([#1780](https://github.com/HKUDS/Vibe-Trading/pull/1780)–[#1794](https://github.com/HKUDS/Vibe-Trading/pull/1794)). 기본적·가치평가 팩터 13개, 브라질 데이터와 백테스트, 선택적으로 활성화하는 ZimaOS 템플릿, Feishu 실행 진행 표시를 추가했습니다. 범위와 제약은 [CHANGELOG](CHANGELOG.md)를 참고하세요. + +- **2026-10-11** 🛠️ **우선순위 수정 5건**: 계산에 쓰인 각 데이터 값에 근거를 요구하고 미지원 실거래 옵션 주문을 차단합니다. 오프라인 벤치마크는 지정 소스를 유지하며, 거래 기록은 손익 0 청산을 구분하고 Codex 도구는 선택 인수를 보존합니다([#1622](https://github.com/HKUDS/Vibe-Trading/issues/1622), [#1779](https://github.com/HKUDS/Vibe-Trading/pull/1779), [#1781](https://github.com/HKUDS/Vibe-Trading/pull/1781), [#1785](https://github.com/HKUDS/Vibe-Trading/pull/1785)). + - **2026-10-10** 🛠️ **Slack/Discord 설정과 재현 가능한 연구**: 번역된 양식에서 봇 인증 정보를 검증하며 테스트는 변경 사항을 저장하지 않습니다([#1764](https://github.com/HKUDS/Vibe-Trading/pull/1764), [#1777](https://github.com/HKUDS/Vibe-Trading/pull/1777)). 포트폴리오 연구는 불변 스냅샷을 재사용하고 Web에서 인증 정보와 계좌 번호를 제거한 JSON을 내보냅니다([#1769](https://github.com/HKUDS/Vibe-Trading/pull/1769)). 지표는 조회 날짜와 입력 지문을 보고하며 손상된 세션 기록은 원본을 다시 쓰지 않고 복구합니다. 메모리 검색, 인증 정보 마스킹, 거래 내역 식별도 수정했습니다. [CHANGELOG](CHANGELOG.md)를 참조하세요. +
+이전 뉴스 + - **2026-10-09** 🛠️ **채널 설정 및 백테스트 수정**: 재연결 없는 설정 업데이트에서 URL 인증 정보와 WeChat 인증 후 서버 주소를 보존하며, 저장·재시작 결과를 웹에서 9개 언어로 표시합니다 ([#1737](https://github.com/HKUDS/Vibe-Trading/pull/1737), [#1742](https://github.com/HKUDS/Vibe-Trading/pull/1742), [#1744](https://github.com/HKUDS/Vibe-Trading/pull/1744)). 여러 시장의 옵션 변동성은 각 시점까지 알려진 타임스탬프만 사용하고, 낙폭은 초기 자금을 반영합니다. 암호화폐·귀금속 데이터 경로, 홍콩 주식 중복 판별, 캐시 복구, 과학적 표기 금액 및 누락된 자산 값 검증을 수정했습니다. 기본 A주 경로에서 mootdx를 제외했습니다. 13개 PR은 [CHANGELOG](CHANGELOG.md)를 참조하세요. - **2026-10-08** 🔐 **PDF 보고서 비밀번호 보호와 증권사 조회 수정**: 예약 이메일 PDF에 AES-256 보호를 선택할 수 있으며 Web, CLI 및 에이전트 확인에 반영됩니다. 비밀번호는 비공개 채널 설정에만 저장합니다([#1709](https://github.com/HKUDS/Vibe-Trading/pull/1709)). KIS는 세 거래 경로를 조회하고 명시적으로 만료된 토큰을 한 번만 갱신하며 불완전한 조회는 오류로 처리합니다([#1726](https://github.com/HKUDS/Vibe-Trading/pull/1726)). 설명 뒤의 전체 파생 수식을 보존하고 증거 검증을 유지합니다([#1728](https://github.com/HKUDS/Vibe-Trading/pull/1728)). - -
-이전 뉴스 - - **2026-10-07** 🛠️ **리서치 연속성과 위험 지표 수정**: 실행 상태를 시스템 프롬프트 밖으로 옮겨 안정적인 접두부를 유지합니다([#1708](https://github.com/HKUDS/Vibe-Trading/pull/1708)). 시장 간 날짜 정렬, 누락 가격 주변의 수익률, 전체 표본을 사용하는 옵션 Sortino 계산을 수정했습니다([#1710](https://github.com/HKUDS/Vibe-Trading/pull/1710), [#1717](https://github.com/HKUDS/Vibe-Trading/pull/1717), [#1727](https://github.com/HKUDS/Vibe-Trading/pull/1727)). 데이터 로더 테스트는 공통 OHLCV 계약을 사용하며 신용 평가, 스타일 노출, 순차 검증 및 감사 진단도 개선했습니다. - **2026-10-06** 🛠️ **실거래 제어와 보고서 수정**: 실거래 러너 중지는 현재 분석을 취소하고 스케줄러 종료를 기다리며 시작 중 취소와 API 종료도 처리합니다. 상태 시간의 단위도 수정했습니다([#1704](https://github.com/HKUDS/Vibe-Trading/pull/1704)). 긴급 취소·청산은 잘못된 레코드를 건너뛰고 나머지 주문과 포지션을 계속 처리합니다([#1703](https://github.com/HKUDS/Vibe-Trading/pull/1703)). 증권사별 일정을 따로 저장하고 동시 쓰기와 부분 쓰기를 처리합니다. 미국 주식 단축 거래일의 조기 폐장 시간을 반영합니다([#1706](https://github.com/HKUDS/Vibe-Trading/pull/1706)). 보고서 수정은 검증된 수치를 유지하며 생략된 오류가 검증을 통과한 것처럼 안내하지 않습니다([#1702](https://github.com/HKUDS/Vibe-Trading/pull/1702)). @@ -377,7 +380,7 @@ Vibe-Trading은 금융 질문을 실행 가능한 분석으로 바꾸는 오픈 | **애널리스트 팀 실행하기** | 투자, 퀀트, 크립토, 매크로, 리스크 워크플로를 위한 멀티 에이전트 리서치 리뷰. | | **리서치를 IM 채널에 연결하기** | WebSocket, Telegram, Slack, Discord, Matrix, WhatsApp, Signal, QQ/NapCat, WeChat/WeCom, Feishu/Lark, DingTalk, Teams, email, Mochat에서 같은 session runtime을 CLI, REST, Web UI로 관리. | | **사용 가능한 artifacts 만들기** | 리포트, TradingView Pine Script, TDX, MetaTrader 5, MCP tools, 이후 리서치 세션. | -| **사전 빌드된 alpha zoo 벤치** | 462개의 alpha 인자(Qlib 158 + Kakushadze 101 + GTJA 191 + academic + PIT-safe fundamental)에 대해 한 줄 CLI로 IC + IR + alive/reversed/dead 분류 수행 | +| **사전 빌드된 alpha zoo 벤치** | 475개의 alpha 인자(Qlib 158 + Kakushadze 101 + GTJA 191 + academic + PIT-safe fundamental)에 대해 한 줄 CLI로 IC + IR + alive/reversed/dead 분류 수행 | | **상관관계 국면 포착하기** | `/correlation` 화면의 엣지 밀도 + 히스테리시스 타임라인으로 시장이 한 덩어리로 움직이기 시작하는 시점을 보여줍니다 — 시그널이 아니라 서술적 리스크 컨텍스트입니다. | --- @@ -461,7 +464,7 @@ vibe-trading connector install /tmp/my-broker ## 📡 데이터 소스 & 스마트 폴백 -`get_market_data` 한 번의 호출, **28개 시장 데이터 소스**(그중 **QVeris**는 선택형 유료 마켓플레이스). `source: "auto"`로 설정하면 로더가 심볼에 따라 소스를 고르고, 시장별 체인을 **IP 차단 위험** 순으로 따라갑니다: 절대 차단되지 않는 공개 소스를 먼저, 속도 제한 / 키 기반 소스를 마지막에 둡니다. 설정 불필요, 단일 장애 지점 없음. +`get_market_data` 한 번의 호출, **29개 시장 데이터 소스**(그중 **QVeris**는 선택형 유료 마켓플레이스). `source: "auto"`로 설정하면 로더가 심볼에 따라 소스를 고르고, 시장별 체인을 **IP 차단 위험** 순으로 따라갑니다: 절대 차단되지 않는 공개 소스를 먼저, 속도 제한 / 키 기반 소스를 마지막에 둡니다. 설정 불필요, 단일 장애 지점 없음. | Source | Markets | Auth | Role | |--------|---------|------|------| @@ -484,6 +487,7 @@ vibe-trading connector install /tmp/my-broker | `tickerall` | forex / metals | key + 계정(읽기 전용) | 동일 브로커의 MT5 피드를 **호스팅형**으로 — 로컬 터미널 불필요, OS 무관(명시 지정 전용, auto 폴백에는 절대 들어가지 않음) | | `pykrx` | 한국 (KRX: KOSPI/KOSDAQ) | 없음 | `.KS` / `.KQ`용 KOSPI / KOSDAQ 일봉 (선택적 `krx` extra) | | `india_broker` | 인도 (NSE/BSE) | 브로커 로그인 | `.NS` / `.BO`용 읽기 전용 Zerodha / Shoonya / Dhan 봉 (폴백 체인 말단) | +| `b3_cotahist` | Brazil (B3) | none | 로컬 B3 공식 현물·단주 일봉(미조정, BRL). `python -m src.br_data sync`로 가져오는 `.SA` 오프라인 대체 소스 | | `local` | any | none | your own CSV / Parquet / DuckDB via `local:` prefix | **폴백 체인 (IP 차단 위험 순):** @@ -492,6 +496,7 @@ vibe-trading connector install /tmp/my-broker - **미국** → `yahoo` · `stooq` · `sina` · `eastmoney` · `yfinance` · `tiingo` · `fmp` · `finnhub` · `alphavantage` · `longbridge` · `akshare` · `local` - **홍콩** → `tencent` · `eastmoney` · `yahoo` · `futu` · `akshare` · `yfinance` · `tushare` · `longbridge` · `local` - **인도 (NSE/BSE)** → `yahoo` · `yfinance` · `india_broker` · `local` +- **Brazil (B3 `.SA`)** → `yahoo` · `yfinance` · `b3_cotahist` · `local` - **한국 (KOSPI/KOSDAQ)** → `pykrx` · `yahoo` · `yfinance` · `local` - **영국 (LSE)** → `yahoo` · `yfinance` · `local` *(명시된 GBP/GBp 호가만 허용)* - **크립토** → `okx` · `binance` · `ccxt` · `yfinance` · `local` @@ -662,9 +667,9 @@ Paper-vs-live는 **구조적 브로커별 런타임 가드**(account-id 형식,
-Alpha Zoo 5개 패밀리에 걸친 462개 사전 빌드된 quant alpha +Alpha Zoo 5개 패밀리에 걸친 475개 사전 빌드된 quant alpha -- 🧬 operator 계층에서 lookahead가 금지된 462개 cross-sectional alpha +- 🧬 operator 계층에서 lookahead가 금지된 475개 cross-sectional alpha - 📈 한 줄 CLI로 IC + IR + alive/reversed/dead 분류 수행 - 🔬 AST 순수성 게이트 + 300-row lookahead sentinel 테스트 + `pytest-socket` 네트워크 kill-switch - 📦 Qlib에 대한 Apache-2 출처 표기, zoo별 `LICENSE.md`에서 수식을 수학적 콘텐츠로 명시 @@ -683,7 +688,7 @@ Paper-vs-live는 **구조적 브로커별 런타임 가드**(account-id 형식,
-백테스트 엔진 10개 엔진 + 옵션 포트폴리오, 크로스마켓 composite +백테스트 엔진 11개 엔진 + 옵션 포트폴리오, 크로스마켓 composite | Engine | Market | Notes | |--------|--------|-------| @@ -692,6 +697,7 @@ Paper-vs-live는 **구조적 브로커별 런타임 가드**(account-id 형식, | **IndiaEquity** | India (NSE/BSE) | T+1, circuit bands, config-driven STT / stamp / SEBI / GST cost stack | | **KoreaEquity** | 한국 (KRX: KOSPI/KOSDAQ) | 롱 온리, 통합 호가 단위에서 ±30% 가격제한폭을 체결 시점에 판정, 2026년 0.20% 증권거래세 | | **VietnamEquity** | 베트남 (HOSE) | 롱 온리, T+2 결제 보유, 10/50/100 VND 호가 단위에서 ±7% 가격제한폭, 100주 단위, 매도측 0.1% 세금 | +| **BrazilEquity** | Brazil (B3) | BRL, B3 trading + settlement fees, configurable broker fee, 1- or 100-share lots, R$0.01 tick, shorting allowed | | **Crypto** | crypto spot / USD-M perps | funding settlements, execution/mark split | | **ChinaFutures** · **GlobalFutures** | futures | margin, contract multipliers | | **Forex** | FX / metals | via the `mt5` loader | @@ -974,7 +980,7 @@ Vibe-Trading은 tool-heavy agent입니다. skills, backtests, memory, swarms가 vibe-trading # interactive TUI vibe-trading run -p "..." # single run vibe-trading serve # API server -vibe-trading alpha list # 사전 빌드된 462개 alpha 탐색; show / bench / compare / export-manifest 서브커맨드 사용 가능 +vibe-trading alpha list # 사전 빌드된 475개 alpha 탐색; show / bench / compare / export-manifest 서브커맨드 사용 가능 vibe-trading playbook list # 예약 리서치 템플릿 5개; show / create 서브커맨드 사용 가능 vibe-trading channels status --local # IM 채널 설정과 설치 힌트 확인 vibe-trading provider doctor # 마스킹된 provider/proxy/패키지 진단 출력 @@ -1721,7 +1727,7 @@ Vibe-Trading/ │ │ ├── memory/ # Cross-session persistent memory │ │ │ └── persistent.py # file-based memory (~/.vibe-trading/memory/) │ │ │ -│ │ ├── tools/ # 109 auto-discovered agent tools +│ │ ├── tools/ # 110 auto-discovered agent tools │ │ │ ├── backtest_tool.py # run backtests │ │ │ ├── remember_tool.py # cross-session memory (save/recall/forget) │ │ │ ├── skill_writer_tool.py # skill CRUD (save/patch/delete/file) @@ -1730,7 +1736,7 @@ Vibe-Trading/ │ │ │ ├── web_search_tool.py # DuckDuckGo web search │ │ │ └── ... # bash, file I/O, factor analysis, options, alpha browser + bench, etc. │ │ │ -│ │ ├── factors/ # Alpha Zoo — 5개 패밀리에 걸친 462개 alpha +│ │ ├── factors/ # Alpha Zoo — 5개 패밀리에 걸친 475개 alpha │ │ │ ├── base.py # 19개 operator (rank/scale/ts_*/delta/decay_linear/safe_div/vwap) │ │ │ ├── registry.py # AST-only 메타데이터 로딩 + lazy compute + sanity gate │ │ │ ├── bench_runner.py # IC + alive/reversed/dead 분류 @@ -1746,8 +1752,8 @@ Vibe-Trading/ │ │ └── providers/ # LLM provider abstraction │ │ │ └── backtest/ # Backtest engines -│ ├── engines/ # 9 engines + composite cross-market engine + options_portfolio -│ ├── loaders/ # 28 sources: tushare, okx, nobitex, wallex, binance, yfinance, akshare, baostock, tencent, mootdx, ccxt, futu, pykrx, local, eastmoney, sina, stooq, yahoo, finnhub, alphavantage, tiingo, fmp, longbridge, mt5, qveris, india_broker, tickerall, gildata +│ ├── engines/ # 10 engines + composite cross-market engine + options_portfolio +│ ├── loaders/ # 29 sources: tushare, okx, nobitex, wallex, binance, yfinance, akshare, baostock, tencent, mootdx, ccxt, futu, pykrx, local, eastmoney, sina, stooq, yahoo, finnhub, alphavantage, tiingo, fmp, longbridge, mt5, qveris, india_broker, tickerall, gildata, b3_cotahist │ │ ├── base.py # DataLoader Protocol │ │ └── registry.py # Registry + auto-fallback chains │ └── optimizers/ # MVO, equal vol, max div, risk parity @@ -1813,7 +1819,7 @@ Vibe-Trading은 **[HKUDS](https://github.com/HKUDS)** agent ecosystem의 일부 | **Data Bridge** | Bring-your-own data: local CSV/Parquet/SQL connectors with schema mapping | 로컬 로더 출시 | | **Options Lab** | Vol surface, Greeks dashboard, payoff/scenario explorer | Planned | | **Portfolio Studio** | Risk x-ray, constraints, turnover-aware optimizer, rebalance notes | Turnover-aware optimizer **0.1.11 출시 완료**; 나머지 Planned | -| **Alpha Zoo** | 5개 패밀리에 걸친 462개의 사전 빌드된 alpha 인자(Qlib 158 + Kakushadze 101 + GTJA 191 + academic + fundamental), 한 줄 CLI 벤치, agent 통합, Web UI | **0.1.8 출시 완료**, 0.1.12까지 확장 | +| **Alpha Zoo** | 5개 패밀리에 걸친 475개의 사전 빌드된 alpha 인자(Qlib 158 + Kakushadze 101 + GTJA 191 + academic + fundamental), 한 줄 CLI 벤치, agent 통합, Web UI | **0.1.8 출시 완료**, 0.1.12까지 확장 | | **Strategy Development Manager** | 논문 / 브로커 리서치를 영속 store + 자동 IC/Sharpe decay 라이프사이클과 함께 팩터 & 전략으로 등록 | **0.1.11 출시 완료** | | **Correlation Regime** | `/correlation` 위에 얹은 엣지 밀도 + 히스테리시스 레짐 타임라인 — 시장이 하나의 블록으로 융합되는 시점 포착 | **0.1.12 출시 완료** | | **Research Delivery** | Slack / Telegram / email-style IM channels를 통한 예약 brief와 live research sessions | 스케줄러 + IM Runtime 출시 | diff --git a/README_zh.md b/README_zh.md index 138721ce..dfc27ab4 100644 --- a/README_zh.md +++ b/README_zh.md @@ -52,16 +52,19 @@ > ⚠️ **安全警告：** X 账号 `VibeTrading_HKU`、Virtuals 项目 `101845` 及代币合约 `0x640BDBF77b6447E8b7DB7894cED84BD1c40571f4` 均非 Vibe-Trading 官方。我们从未发行或背书任何代币或 meme 币。请勿购买、连接钱包或签名。[详细说明](SECURITY.md#official-channels--impersonation)。 +- **2026-10-12** 🧮 **研究与回测更新**：修正开仓成交、清仓时归因、回撤时长、估值容差及因子 IC 的 SEC 财报可见时点（[#1780](https://github.com/HKUDS/Vibe-Trading/pull/1780)–[#1794](https://github.com/HKUDS/Vibe-Trading/pull/1794)）。新增 13 个基本面／估值因子、巴西数据与回测、需显式启用的 ZimaOS 模板，以及飞书运行进度。范围与数据限制见 [CHANGELOG](CHANGELOG.md)。 + +- **2026-10-11** 🛠️ **五项优先修复**：派生数字的每个数据操作数都须有证据，不支持的实盘期权继续拦截。离线基准保持指定数据源，交易记录区分零盈亏平仓，Codex 工具保留可选参数（[#1622](https://github.com/HKUDS/Vibe-Trading/issues/1622)、[#1779](https://github.com/HKUDS/Vibe-Trading/pull/1779)、[#1781](https://github.com/HKUDS/Vibe-Trading/pull/1781)、[#1785](https://github.com/HKUDS/Vibe-Trading/pull/1785)）。 + - **2026-10-10** 🛠️ **Slack/Discord 配置与可复现研究**：本地化配置表单可测试机器人凭证，测试不会保存改动（[#1764](https://github.com/HKUDS/Vibe-Trading/pull/1764)、[#1777](https://github.com/HKUDS/Vibe-Trading/pull/1777)）。组合研究可复用不可变快照，Web 可导出脱敏 JSON（[#1769](https://github.com/HKUDS/Vibe-Trading/pull/1769)）。技术指标报告日期边界和输入指纹；损坏的会话记录可恢复且不重写原文件。另修复记忆召回、凭证脱敏和交易流水格式识别。详见 [CHANGELOG](CHANGELOG.md)。 +
+更早的新闻 + - **2026-10-09** 🛠️ **渠道设置与回测修正**：原地更新渠道设置时保留 URL 内嵌凭证和微信认证后的服务地址，Web 以九种语言说明保存与重启结果（[#1737](https://github.com/HKUDS/Vibe-Trading/pull/1737)、[#1742](https://github.com/HKUDS/Vibe-Trading/pull/1742)、[#1744](https://github.com/HKUDS/Vibe-Trading/pull/1744)）。跨市场期权波动率仅使用截至当前时点的时间戳，回撤纳入初始资金。修正加密货币／贵金属数据源路由、港股重复识别、缓存恢复、科学计数金额解析及缺失净值验证；mootdx 移出默认 A 股链。全部十三项 PR 见 [CHANGELOG](CHANGELOG.md)。 - **2026-10-08** 🔐 **PDF 报告密码保护与券商读取修复**：定时邮件 PDF 可选 AES-256 密码保护，覆盖 Web、CLI 模板和 Agent 确认入口；密码只保存在渠道私有配置中（[#1709](https://github.com/HKUDS/Vibe-Trading/pull/1709)）。KIS 读取三个交易场所的订单，明确过期的 token 只刷新一次，不完整的读取返回错误（[#1726](https://github.com/HKUDS/Vibe-Trading/pull/1726)）。派生公式保留说明标签之后的完整算式，证据核验规则保持不变（[#1728](https://github.com/HKUDS/Vibe-Trading/pull/1728)）。 - -
-更早的新闻 - - **2026-10-07** 🛠️ **研究连续性与风险指标修正**：运行状态移出系统提示词，保留稳定前缀以利于缓存（[#1708](https://github.com/HKUDS/Vibe-Trading/pull/1708)）。跨市场日期正确对齐，缺失价格不再拼接跨期收益，期权 Sortino 按全样本计算下行偏差（[#1710](https://github.com/HKUDS/Vibe-Trading/pull/1710)、[#1717](https://github.com/HKUDS/Vibe-Trading/pull/1717)、[#1727](https://github.com/HKUDS/Vibe-Trading/pull/1727)）。数据源测试共用 OHLCV 契约，并修正信用估值、风格敞口、滚动验证和审计诊断。 - **2026-10-06** 🛠️ **实盘控制与报告纠错修复**：停止实盘运行会取消当前分析并等待调度器完成清理，覆盖启动期间取消及 API 关闭，运行状态时间单位也已修正（[#1704](https://github.com/HKUDS/Vibe-Trading/pull/1704)）。紧急撤单／平仓扫描跳过异常记录，继续处理其余订单和持仓（[#1703](https://github.com/HKUDS/Vibe-Trading/pull/1703)）；不同券商的调度独立保存，持久化正确处理并发写入与短写入。美股半日交易遵循提前收市时间（[#1706](https://github.com/HKUDS/Vibe-Trading/pull/1706)）。报告纠错保留已核验通过的数字，反馈截断也不会暗示未列出的数字已通过（[#1702](https://github.com/HKUDS/Vibe-Trading/pull/1702)）。 @@ -376,7 +379,7 @@ Vibe-Trading 是一个开源研究工作台，用于把金融问题转化为可 | **运行分析师团队** | 面向投资、量化、加密、宏观和风控工作流的多智能体研究评审。 | | **把研究接入 IM 通道** | 通过 WebSocket、Telegram、Slack、Discord、Matrix、WhatsApp、Signal、QQ/NapCat、微信/企业微信、飞书/Lark、钉钉、Teams、email、Mochat，在 CLI、REST 和 Web UI 中管理同一套 session runtime。 | | **交付可用成果** | 报告、TradingView Pine Script、TDX、MetaTrader 5、MCP tools，以及可延续的研究 sessions。 | -| **跑预置 alpha zoo 横评** | 462 个 alpha 因子（Qlib 158 + Kakushadze 101 + GTJA 191 + academic + PIT-safe fundamental），一行 CLI 在你选的 universe 上算 IC + IR + alive/reversed/dead 分类 | +| **跑预置 alpha zoo 横评** | 475 个 alpha 因子（Qlib 158 + Kakushadze 101 + GTJA 191 + academic + PIT-safe fundamental），一行 CLI 在你选的 universe 上算 IC + IR + alive/reversed/dead 分类 | | **识别相关性状态** | `/correlation` 界面上的边密度 + 迟滞时间线，显示市场何时融合为一个板块——属于描述性风险上下文，而非交易信号。 | --- @@ -476,7 +479,7 @@ vibe-trading connector install /tmp/my-broker ## 📡 数据源与智能 Fallback -一次 `get_market_data` 调用，**28 个行情数据源**（其中 **QVeris** 是可选的付费市场）。设 `source: "auto"`——loader 按符号自动选源，再沿按 **被封 IP 风险** 排序的同市场链向下走（永不封的公开源在前，限速 / 需 key 的在后）。零配置，无单点故障。 +一次 `get_market_data` 调用，**29 个行情数据源**（其中 **QVeris** 是可选的付费市场）。设 `source: "auto"`——loader 按符号自动选源，再沿按 **被封 IP 风险** 排序的同市场链向下走（永不封的公开源在前，限速 / 需 key 的在后）。零配置，无单点故障。 | Source | Markets | Auth | Role | |--------|---------|------|------| @@ -499,6 +502,7 @@ vibe-trading connector install /tmp/my-broker | `tickerall` | 外汇 / 贵金属 | key + 账户（只读） | 同一家券商的 MT5 数据源，**托管** —— 无需本地终端，任意操作系统（仅显式选用，绝不进 auto 链） | | `pykrx` | 韩国（KRX：KOSPI/KOSDAQ） | 无 | `.KS` / `.KQ` 的 KOSPI / KOSDAQ 日线（可选 `krx` extra） | | `india_broker` | 印度（NSE/BSE） | 券商登录 | 只读 Zerodha / Shoonya / Dhan bars，服务 `.NS` / `.BO`（fallback 链尾） | +| `b3_cotahist` | Brazil (B3) | none | 本地 B3 官方现货／零股日线（原始价格、BRL），由 `python -m src.br_data sync` 导入；`.SA` 的离线后备源 | | `local` | any | none | your own CSV / Parquet / DuckDB via `local:` prefix | **Fallback 链（按被封 IP 风险排序）：** @@ -507,6 +511,7 @@ vibe-trading connector install /tmp/my-broker - **美股** → `yahoo` · `stooq` · `sina` · `eastmoney` · `yfinance` · `tiingo` · `fmp` · `finnhub` · `alphavantage` · `longbridge` · `akshare` · `local` - **港股** → `tencent` · `eastmoney` · `yahoo` · `futu` · `akshare` · `yfinance` · `tushare` · `longbridge` · `local` - **印度（NSE/BSE）** → `yahoo` · `yfinance` · `india_broker` · `local` +- **Brazil (B3 `.SA`)** → `yahoo` · `yfinance` · `b3_cotahist` · `local` - **韩国（KOSPI/KOSDAQ）** → `pykrx` · `yahoo` · `yfinance` · `local` - **英国（LSE）** → `yahoo` · `yfinance` · `local` *（仅接受明示为 GBP/GBp 的报价）* - **加密** → `okx` · `binance` · `ccxt` · `yfinance` · `local` @@ -675,9 +680,9 @@ LONGBRIDGE_ACCESS_TOKEN=...
-Alpha Zoo 462 个预置 alpha，覆盖 5 个家族 +Alpha Zoo 475 个预置 alpha，覆盖 5 个家族 -- 🧬 462 个横截面 alpha，算子层即禁用 lookahead +- 🧬 475 个横截面 alpha，算子层即禁用 lookahead - 📈 一条 CLI 命令完成 IC + IR + alive/reversed/dead 分类 - 🔬 AST 纯函数门禁 + 300 行 lookahead 哨兵测试 + `pytest-socket` 网络阻断 - 📦 Qlib 部分附 Apache-2 出处声明；每个 zoo 一份 `LICENSE.md`，声明公式属于数学内容 @@ -696,7 +701,7 @@ LONGBRIDGE_ACCESS_TOKEN=...
-Backtest Engines 10 个引擎 + options portfolio，跨市场 composite +Backtest Engines 11 个引擎 + options portfolio，跨市场 composite | 引擎 | 市场 | 说明 | |------|------|------| @@ -705,6 +710,7 @@ LONGBRIDGE_ACCESS_TOKEN=... | **IndiaEquity** | 印度（NSE/BSE） | T+1、熔断带、config 驱动的 STT / 印花税 / SEBI / GST 成本栈 | | **KoreaEquity** | 韩国（KRX：KOSPI/KOSDAQ） | 只做多，统一最小价位网格上于成交时刻判定 ±30% 涨跌停，2026 年 0.20% 证券交易税 | | **VietnamEquity** | 越南（HOSE） | 只做多，T+2 交收锁定，10/50/100 越南盾最小价位网格上 ±7% 涨跌停，100 股整手，0.1% 卖出方税 | +| **BrazilEquity** | Brazil (B3) | BRL, B3 trading + settlement fees, configurable broker fee, 1- or 100-share lots, R$0.01 tick, shorting allowed | | **Crypto** | 加密现货 / USD-M 永续 | 资金费结算、成交价/标记价分离 | | **ChinaFutures** · **GlobalFutures** | 期货 | 保证金、合约乘数 | | **Forex** | 外汇 / 贵金属 | 经 `mt5` loader | @@ -981,7 +987,7 @@ Vibe-Trading 是高度依赖工具的智能体：skills、backtests、memory 和 vibe-trading # interactive TUI vibe-trading run -p "..." # single run vibe-trading serve # API server -vibe-trading alpha list # 浏览 462 个预置 alpha；支持 show / bench / compare / export-manifest 子命令 +vibe-trading alpha list # 浏览 475 个预置 alpha；支持 show / bench / compare / export-manifest 子命令 vibe-trading playbook list # 五个定时研究模板；支持 show / create 子命令 vibe-trading channels status --local # 检查 IM 通道配置和依赖安装提示 vibe-trading provider doctor # 打印脱敏后的 provider/proxy/依赖诊断 @@ -1719,7 +1725,7 @@ Vibe-Trading/ │ │ ├── memory/ # 跨 session 持久记忆 │ │ │ └── persistent.py # 基于文件的记忆（~/.vibe-trading/memory/） │ │ │ -│ │ ├── tools/ # 109 个自动发现的 agent 工具 +│ │ ├── tools/ # 110 个自动发现的 agent 工具 │ │ │ ├── backtest_tool.py # 运行回测 │ │ │ ├── remember_tool.py # 跨 session 记忆（save/recall/forget） │ │ │ ├── skill_writer_tool.py # skill CRUD（save/patch/delete/file） @@ -1728,7 +1734,7 @@ Vibe-Trading/ │ │ │ ├── web_search_tool.py # DuckDuckGo 网络搜索 │ │ │ └── ... # bash、文件 I/O、因子分析、期权、alpha 浏览 + 横评等 │ │ │ -│ │ ├── factors/ # Alpha Zoo —— 5 个家族共 462 个 alpha +│ │ ├── factors/ # Alpha Zoo —— 5 个家族共 475 个 alpha │ │ │ ├── base.py # 19 个算子（rank/scale/ts_*/delta/decay_linear/safe_div/vwap） │ │ │ ├── registry.py # 纯 AST 元数据加载 + 惰性计算 + sanity 校验 │ │ │ ├── bench_runner.py # IC + alive/reversed/dead 分类 @@ -1744,8 +1750,8 @@ Vibe-Trading/ │ │ └── providers/ # LLM provider 抽象层 │ │ │ └── backtest/ # 回测引擎 -│ ├── engines/ # 9 个引擎 + 跨市场 composite 引擎 + options_portfolio -│ ├── loaders/ # 28 个数据源：tushare、okx、nobitex、wallex、binance、yfinance、akshare、baostock、tencent、mootdx、ccxt、futu、pykrx、local、eastmoney、sina、stooq、yahoo、finnhub、alphavantage、tiingo、fmp、longbridge、mt5、qveris、india_broker、tickerall、gildata +│ ├── engines/ # 10 个引擎 + 跨市场 composite 引擎 + options_portfolio +│ ├── loaders/ # 29 个数据源：tushare、okx、nobitex、wallex、binance、yfinance、akshare、baostock、tencent、mootdx、ccxt、futu、pykrx、local、eastmoney、sina、stooq、yahoo、finnhub、alphavantage、tiingo、fmp、longbridge、mt5、qveris、india_broker、tickerall、gildata, b3_cotahist │ │ ├── base.py # DataLoader Protocol │ │ └── registry.py # Registry + 自动 fallback 链路 │ └── optimizers/ # MVO、equal vol、max div、risk parity @@ -1811,7 +1817,7 @@ Vibe-Trading 是 **[HKUDS](https://github.com/HKUDS)** 智能体生态的一部 | **Data Bridge** | 自带数据：本地 CSV/Parquet/SQL connectors 与 schema mapping | 本地加载器已发布 | | **Options Lab** | Vol surface、Greeks dashboard、payoff/scenario explorer | Planned | | **Portfolio Studio** | Risk x-ray、constraints、turnover-aware optimizer、rebalance notes | Turnover-aware optimizer **已发布 0.1.11**；其余 Planned | -| **Alpha Zoo** | 462 个预置 alpha 因子（Qlib 158 + Kakushadze 101 + GTJA 191 + academic + fundamental），一行 CLI 跑横评，agent 集成，Web UI 浏览 | **已发布 0.1.8**，延续至 0.1.12 | +| **Alpha Zoo** | 475 个预置 alpha 因子（Qlib 158 + Kakushadze 101 + GTJA 191 + academic + fundamental），一行 CLI 跑横评，agent 集成，Web UI 浏览 | **已发布 0.1.8**，延续至 0.1.12 | | **Strategy Development Manager** | 把论文 / 券商研报注册为因子与策略，配持久化 store + 自动化 IC/Sharpe 衰减生命周期 | **已发布 0.1.11** | | **Correlation Regime** | 叠加在 `/correlation` 上的边密度 + 迟滞状态时间线——识别市场何时融合为一个板块 | **已发布 0.1.12** | | **Research Delivery** | 通过 Slack / Telegram / email-style IM channels 发送定时 briefs 与实时研究 sessions | 调度器 + IM Runtime 已发布 | diff --git a/agent/SKILL.md b/agent/SKILL.md index 6d47a50f..073862eb 100644 --- a/agent/SKILL.md +++ b/agent/SKILL.md @@ -1,7 +1,7 @@ --- name: vibe-trading version: 0.1.16 -description: Professional finance research toolkit — backtesting (10 engines + benchmark comparison panel), factor analysis, Alpha Zoo (462 pre-built alphas across qlib158/alpha101/gtja191/academic/fundamental), options pricing, 90 finance skills, 30 multi-agent swarm teams, Trade Journal analyzer, and Shadow Account (extract → backtest → render) across 28 market-data sources (tushare, yfinance, okx, binance, akshare, baostock, tencent, mootdx, ccxt, futu, mt5, tickerall, local, eastmoney, sina, stooq, yahoo, pykrx, india_broker, qveris, longbridge, nobitex, wallex, plus optional-key finnhub/alphavantage/tiingo/fmp/gildata). +description: Professional finance research toolkit — backtesting (11 engines + benchmark comparison panel), factor analysis, Alpha Zoo (475 pre-built alphas across qlib158/alpha101/gtja191/academic/fundamental), options pricing, 90 finance skills, 30 multi-agent swarm teams, Trade Journal analyzer, and Shadow Account (extract → backtest → render) across 29 market-data sources (tushare, yfinance, okx, binance, akshare, baostock, tencent, mootdx, ccxt, futu, mt5, tickerall, local, eastmoney, sina, stooq, yahoo, pykrx, india_broker, qveris, longbridge, nobitex, wallex, plus optional-key finnhub/alphavantage/tiingo/fmp/gildata). dependencies: python: ">=3.11" pip: @@ -23,7 +23,7 @@ mcp: # Vibe-Trading -Professional finance research toolkit with AI-powered backtesting (10 engines), multi-agent teams, 90 specialized skills, the **Alpha Zoo** (462 pre-built quantitative alphas across qlib158 / alpha101 / gtja191 / academic / fundamental with one-line CLI benchmarking), and the Shadow Account loop — extract your implicit trading rules from a journal, backtest them across A股/港股/美股/crypto, then see where they would have served you better. +Professional finance research toolkit with AI-powered backtesting (11 engines), multi-agent teams, 90 specialized skills, the **Alpha Zoo** (475 pre-built quantitative alphas across qlib158 / alpha101 / gtja191 / academic / fundamental with one-line CLI benchmarking), and the Shadow Account loop — extract your implicit trading rules from a journal, backtest them across A股/港股/美股/crypto, then see where they would have served you better. ## Setup @@ -53,7 +53,7 @@ Add to your agent's MCP config: ### API Key Requirements -Core research MCP tools work with zero API keys for HK/US/Canada/crypto. After `pip install`, backtesting, market data, factor analysis, options pricing, chart patterns, web search, document reading, trade journal analysis, shadow-account extraction/backtest/report, the Alpha Zoo (462 pre-built alphas), and all 90 skills are ready to use. IBKR tools require a local TWS / IB Gateway session; `run_swarm` requires an LLM key. +Core research MCP tools work with zero API keys for HK/US/Canada/crypto. After `pip install`, backtesting, market data, factor analysis, options pricing, chart patterns, web search, document reading, trade journal analysis, shadow-account extraction/backtest/report, the Alpha Zoo (475 pre-built alphas), and all 90 skills are ready to use. IBKR tools require a local TWS / IB Gateway session; `run_swarm` requires an LLM key. | Feature | Key needed | When | |---------|-----------|------| @@ -74,7 +74,7 @@ Feed a CSV broker export (同花顺 / 东财 / 富途 / generic), and the agent 5. `scan_shadow_signals` — list today's symbols that match your shadow's entry cadence (research only). ### Backtesting -Create and run quantitative strategies across 10 engines (ChinaA, GlobalEquity, IndiaEquity, KoreaEquity, VietnamEquity, Crypto, ChinaFutures, GlobalFutures, Forex + options) with 28 market-data sources (auto-detect + ordered fallback; the hosted forex `tickerall` source is explicit-only): +Create and run quantitative strategies across 11 engines (ChinaA, GlobalEquity, IndiaEquity, KoreaEquity, VietnamEquity, BrazilEquity, Crypto, ChinaFutures, GlobalFutures, Forex + options) with 29 market-data sources (auto-detect + ordered fallback; the hosted forex `tickerall` source is explicit-only): - **HK/US equities** via yfinance / stooq / yahoo (free, no API key); optionally via **Longbridge** historical OHLCV (`longbridge`, requires the optional SDK and `LONGBRIDGE_APP_KEY` / `LONGBRIDGE_APP_SECRET` / `LONGBRIDGE_ACCESS_TOKEN`). To force it for a run, set `"source": "longbridge"` in `config.json`. - **Canada equities (TSX/TSXV)** via yahoo / yfinance using Yahoo's canonical `.TO` (TSX, e.g. `TD.TO`) or `.V` (TSXV, e.g. `PNG.V`) suffixes — free, no API key. The GlobalEquity engine uses CAD identity, whole-share orders, configurable Canadian commission/slippage, and the TSX/TSXV price-increment grid. - **India equities (NSE/BSE)** via yahoo / yfinance using `.NS` (NSE, e.g. `RELIANCE.NS`) or `.BO` (BSE, e.g. `500325.BO`) — free, no API key. The `IndiaEquityEngine` models T+1 delivery, no overnight shorts (set `allow_short` for intraday), configurable circuit bands, 1-share lots, and the STT/stamp-duty/exchange/GST cost stack. Optionally back-fill from your live broker via the `india_broker` source (Shoonya/Dhan; requires broker login). @@ -110,13 +110,13 @@ Example workflow: Use `list_swarm_presets()` to see all teams, then `run_swarm()` to execute. -### Alpha Zoo (462 pre-built alphas) +### Alpha Zoo (475 pre-built alphas) One-line cross-sectional IC / IR / alive-reversed-dead categorisation across five bundled zoos: - **qlib158** (154 alphas) — Microsoft Qlib's `Alpha158` feature handler, Apache-2.0 with pinned commit SHA. - **alpha101** (101 alphas) — Kakushadze (2015) "101 Formulaic Alphas" (arXiv:1601.00991), written from the paper appendix. - **gtja191** (191 alphas) — Guotai Junan 2014 "191 Short-period Trading Alpha Factors" research report. - **academic** (12 factors) — Fama-French 5 + Carhart momentum + Jegadeesh reversal + George-Hwang 52-week-high + Amihud illiquidity + Harvey-Siddique skew + Frazzini-Pedersen betting-against-beta (price-based proxies) + a correlation-rewiring stability score (from the in-repo correlation-regime skill). -- **fundamental** (4 factors) — PIT-safe earnings yield, ROE, gross profitability, and asset growth from daily fundamental panels. +- **fundamental** (17 factors) — PIT-safe earnings yield, ROE, gross profitability and asset growth, plus ten SEC XBRL additions: book-to-market, FCF yield, accruals, ROA, CFO yield, operating margin, cash-to-assets, leverage, revenue growth and earnings growth, from daily fundamental panels; and three CN A-share daily-valuation factors (PB, PS_TTM, dividend yield) fed by the new ``val:*`` panel namespace from tushare daily_basic. Each alpha ships with `__alpha_meta__` (formula LaTeX + theme + universe + warmup + columns required), guarded by an AST purity gate + 300-row lookahead sentinel test. Use the `vibe-trading alpha {list,show,bench,compare,export-manifest}` CLI, the `/alpha/*` REST routes (browser at `/alpha-zoo`), or compose multi-factor signals via `ZooSignalEngine.from_zoo(...)`. @@ -150,7 +150,7 @@ Use `load_skill(name)` to access full methodology docs with code templates. | `analyze_options` | Black-Scholes price + Greeks | None | | `analyze_options_payoff` | Multi-leg expiry payoff + spot/IV scenarios | None | | `pattern_recognition` | Detect chart patterns (H&S, double top, etc.) | None | -| `get_market_data` | Fetch OHLCV data (auto-detect + ordered fallback across 28 sources) | None* | +| `get_market_data` | Fetch OHLCV data (auto-detect + ordered fallback across 29 sources) | None* | | `get_fund_flow` | Capital fund-flow (main/retail net inflow) | None* | | `get_dragon_tiger` | Dragon-tiger list (龙虎榜) top buyer/seller seats | None* | | `get_northbound_flow` | Northbound (Stock Connect) net flow | None* | diff --git a/agent/backtest/benchmark.py b/agent/backtest/benchmark.py index ed591133..da0ea29d 100644 --- a/agent/backtest/benchmark.py +++ b/agent/backtest/benchmark.py @@ -12,6 +12,7 @@ from typing import Any, Optional import pandas as pd from backtest.loaders.base import resample_bars, source_interval +from backtest.loaders.registry import is_no_network_fallback_source from backtest.loaders.yfinance_loader import DataLoader as YfinanceLoader from backtest.metrics import bar_returns, buy_and_hold_return @@ -24,6 +25,7 @@ MARKET_BENCHMARKS: dict[str, Optional[str]] = { "us_equity": "SPY", "hk_equity": "HK.03100", # Hang Seng China Enterprises ETF "ca_equity": "XIC.TO", # S&P/TSX Capped Composite ETF + "br_equity": "BOVA11.SA", # iShares Ibovespa ETF "a_share": "000300.SH", # CSI 300 (China A-share core index) "crypto": "BTC-USDT", "futures": "ES.CME", # E-mini S&P 500 futures @@ -84,8 +86,8 @@ def resolve_benchmark( explicit: Override ticker (e.g. "SPY" passed via config). loader: Loader of the configured data source. When given, the benchmark is fetched through it first, falling back to - yfinance if it yields no data — except ``local``, - which fails closed to keep offline runs offline. + yfinance if it yields no data, except sources whose + contract prohibits a fallback to a different source. Returns: BenchmarkResult with return series and total return, or None if no @@ -95,18 +97,18 @@ def resolve_benchmark( if ticker is None: return None - offline = source == "local" - if offline and getattr(loader, "name", None) != source: - # The runtime fallback chain in fetch_data_map() may have swapped in a - # network loader while config["source"] still says local — never fetch - # the benchmark through it. Fail closed instead. + source_bound = is_no_network_fallback_source(source) + if source_bound and getattr(loader, "name", None) != source: + # Do not let a swapped loader bypass the configured source's boundary: + # local must stay offline, and source-specific quote currencies and + # explicit provider choices must not silently change for a benchmark. loader = None try: bench_df = _fetch_benchmark( ticker, start_date, end_date, interval, loader=loader, - allow_fallback=not offline, + allow_fallback=not source_bound, ) except Exception: return None @@ -176,6 +178,8 @@ def _infer_market(codes: list[str], source: str) -> str: return "ca_equity" if first.endswith((".NS", ".BO")): return "india_equity" + if first.endswith(".SA"): + return "br_equity" if first.endswith((".KS", ".KQ")): return "kr_equity" crypto_quotes = ("-USDT", "-USDC", "-USD", "-BTC", "-ETH") diff --git a/agent/backtest/engines/_market_hooks.py b/agent/backtest/engines/_market_hooks.py index fb1fda63..bde045d6 100644 --- a/agent/backtest/engines/_market_hooks.py +++ b/agent/backtest/engines/_market_hooks.py @@ -110,6 +110,9 @@ _MARKET_PATTERNS = [ # Argentina: BYMA listings use Yahoo's canonical .BA suffix. Keep this # as a distinct market so ARS can never be mixed with USD/CNY accounting. (re.compile(r"^[A-Z0-9&.\-]+\.BA$", re.I), "ar_equity"), + # Brazil: B3 listings (PETR4.SA, BOVA11.SA, HGLG11.SA, AAPL34.SA) use + # Yahoo's canonical .SA suffix; every line settles in BRL. + (re.compile(r"^[A-Z0-9]+\.SA$", re.I), "br_equity"), # UK equities: London Stock Exchange (VOD.L, SHEL.L). Yahoo carries the # suffix verbatim. (re.compile(r"^[A-Z0-9&.\-]+\.L$", re.I), "uk_equity"), @@ -227,6 +230,7 @@ _MARKET_CURRENCY = { "kr_equity": "KRW", "ca_equity": "CAD", "ar_equity": "ARS", + "br_equity": "BRL", "uk_equity": "GBP", "vietnam_equity": "VND", # Every crypto pattern in _MARKET_PATTERNS is USDT-quoted, and USDT is @@ -347,7 +351,7 @@ def _detect_market(code: str) -> str: Returns: Market type (a_share/us_equity/hk_equity/india_equity/kr_equity/ - ca_equity/ar_equity/crypto/futures/forex). + ca_equity/ar_equity/br_equity/crypto/futures/forex). Bare 1-5 letter alphabetic tickers resolve to ``us_equity``; bare 6-letter codes that start with a precious-metal or G10 currency code (whitelist) resolve to ``forex``; concatenated diff --git a/agent/backtest/engines/base.py b/agent/backtest/engines/base.py index 2f8b8a2f..a9d9ee92 100644 --- a/agent/backtest/engines/base.py +++ b/agent/backtest/engines/base.py @@ -971,6 +971,7 @@ class BaseEngine(ABC): ) bench_ret = ret_df.mean(axis=1) if ret_df.shape[1] > 0 else pd.Series(0.0, index=dates) benchmark_metadata = {} + benchmark_warning: str | None = None # ── External benchmark fetch ────────────────────────────────────────── bench_ticker = config.get("benchmark") @@ -1006,6 +1007,16 @@ class BaseEngine(ABC): window_ret if window_ret is not None else bench_result.total_ret ), } + else: + # The equal-weight basket stands in for the declared benchmark, + # so every benchmark-relative metric (excess_return, + # information_ratio, tracking_error, benchmark_beta) is graded + # against something the caller did not ask for. Say so. + benchmark_warning = ( + f"declared benchmark {bench_ticker!r} could not be resolved; " + "benchmark-relative metrics are graded against the " + "equal-weight basket of the traded symbols instead" + ) # ── External benchmark fetch ────────────────────────────────────────── bench_equity = self.initial_capital * (1 + bench_ret).cumprod() @@ -1151,6 +1162,8 @@ class BaseEngine(ABC): card_warnings.append(config["_run_card_caliber_warning"]) if config.get("_run_card_annualisation_warning"): card_warnings.append(config["_run_card_annualisation_warning"]) + if benchmark_warning: + card_warnings.append(benchmark_warning) from backtest.run_card import write_run_card write_run_card( run_dir, @@ -2013,6 +2026,8 @@ class BaseEngine(ABC): commission=entry_commission + order.exit_commission, entry_margin=order.released_margin, exit_margin=exit_margin, + open_price=before.open_price, + open_size=before.open_size, )) self._record_fill( symbol=before.symbol, @@ -2053,6 +2068,8 @@ class BaseEngine(ABC): leverage=order.leverage, entry_bar_idx=self._bar_idx, entry_commission=order.commission, + open_price=order.price, + open_size=order.size, ) self._record_fill( symbol=order.symbol, @@ -2146,6 +2163,8 @@ class BaseEngine(ABC): commission=pos.entry_commission + exit_comm, entry_margin=margin, exit_margin=exit_margin, + open_price=pos.open_price, + open_size=pos.open_size, )) self._record_fill( symbol=symbol, @@ -2273,21 +2292,28 @@ class BaseEngine(ABC): target_out.index.name = "timestamp" target_out.to_csv(out / "target_positions.csv") - # Trades (compatible format) + # Trades (compatible format): two rows per round trip, entry then exit. + # The entry row is the opening fill as executed. A TradeRecord's + # entry_price is the weighted cost after any scale-ins while its + # entry_time stays on the first open, so pairing those two would + # report a fill that never happened. Scale-ins stay in fills.jsonl. trade_rows = [] for t in self.trades: # Entry event + entry_price = t.entry_price if t.open_price is None else t.open_price + entry_qty = t.size if t.open_size is None else t.open_size trade_rows.append({ "timestamp": str(t.entry_time.date()) if hasattr(t.entry_time, "date") else str(t.entry_time), "code": t.symbol, "side": "buy" if t.direction == 1 else "sell", - "price": round(t.entry_price, 4), - "qty": round(t.size, 6), + "price": round(entry_price, 4), + "qty": round(entry_qty, 6), "reason": "signal", "pnl": 0.0, "holding_days": 0, "holding_bars": 0.0, "return_pct": 0.0, + "event": "entry", }) # Exit event try: @@ -2305,11 +2331,12 @@ class BaseEngine(ABC): "holding_days": hold_days, "holding_bars": t.holding_bars, "return_pct": round(t.pnl_pct, 2), + "event": "exit", }) trade_cols = [ "timestamp", "code", "side", "price", "qty", "reason", "pnl", - "holding_days", "holding_bars", "return_pct", + "holding_days", "holding_bars", "return_pct", "event", ] pd.DataFrame(trade_rows or [], columns=trade_cols).to_csv(out / "trades.csv", index=False) diff --git a/agent/backtest/engines/brazil_equity.py b/agent/backtest/engines/brazil_equity.py new file mode 100644 index 00000000..d7dffbaa --- /dev/null +++ b/agent/backtest/engines/brazil_equity.py @@ -0,0 +1,95 @@ +"""Brazil equity (B3) backtest engine. + +Covers everything B3 lists under Yahoo's ``.SA`` suffix: shares (PETR4.SA), +units, ETFs (BOVA11.SA), FIIs (HGLG11.SA) and BDRs (AAPL34.SA). All settle in +BRL. + +Market rules: + - Same-session round trips are allowed (day trade). Settlement is D+2, which + restricts cash, not the ability to sell shares bought today. + - Short selling is allowed (via the B3 securities-lending facility, BTC). + Set ``allow_short=False`` to model a long-only account. + - No hard daily price limits: B3 uses auction tunnels that pause trading + rather than blocking a price, so every bar is executable. + - Whole shares. The standard lot is 100, but the odd-lot (fractional) + market accepts single shares, so the default lot is 1. Set + ``br_lot_size=100`` to trade only round lots. + - Price increment R$0.01, rounded against the trader. + +Cost stack (all config-driven). B3 tariffs change; verify against the current +B3 price list before relying on absolute figures: + - B3 trading fee (emolumentos): 0.005% bilateral [br_trading_fee] + - B3 settlement fee (liquidação): 0.025% bilateral [br_settlement_fee] + - Broker commission: 0% (most retail brokers) [br_brokerage_rate] + - Fixed broker fee per order: R$0 [br_brokerage_fixed] + +Income tax (15% swing / 20% day trade on net gains, monthly R$20k sales +exemption for shares) is assessed on the investor's monthly result, not per +trade, so it is not charged here. +""" + +from __future__ import annotations + +import math + +import pandas as pd + +from backtest.engines.base import BaseEngine + +_TICK = 0.01 + + +class BrazilEquityEngine(BaseEngine): + """B3 cash-equity engine. + + Config keys (all optional; defaults shown in the module docstring): + - allow_short: bool, default True + - slippage_br: default 0.0005 + - br_lot_size: default 1 + - br_trading_fee / br_settlement_fee / br_brokerage_rate / + br_brokerage_fixed + """ + + def __init__(self, config: dict): + config = {**config, "leverage": config.get("leverage", 1.0)} + super().__init__(config) + self.allow_short: bool = bool(config.get("allow_short", True)) + self.slippage_br: float = config.get("slippage_br", 0.0005) + self.br_lot_size: int = max(int(config.get("br_lot_size", 1)), 1) + self.br_trading_fee: float = config.get("br_trading_fee", 0.00005) + self.br_settlement_fee: float = config.get("br_settlement_fee", 0.00025) + self.br_brokerage_rate: float = config.get("br_brokerage_rate", 0.0) + self.br_brokerage_fixed: float = config.get("br_brokerage_fixed", 0.0) + + def can_execute(self, symbol: str, direction: int, bar: pd.Series) -> bool: + """Every bar is executable; only opening a short can be disabled.""" + if direction == -1 and not self.allow_short: + return False + return True + + def round_size(self, raw_size: float, price: float) -> float: + """Floor to whole lots (1 share by default, 100 for round lots).""" + lot = self.br_lot_size + return float(math.floor(max(raw_size, 0.0) / lot) * lot) + + def calc_commission(self, size: float, price: float, direction: int, is_open: bool) -> float: + """B3 fees (bilateral) plus the configured broker commission.""" + notional = size * price + if notional <= 0: + return 0.0 + comm = notional * (self.br_trading_fee + self.br_settlement_fee) + comm += notional * self.br_brokerage_rate + comm += self.br_brokerage_fixed + return comm + + def apply_slippage(self, price: float, direction: int) -> float: + """Apply slippage, then snap to the R$0.01 grid against the trader.""" + slipped = price * (1 + direction * self.slippage_br) + units = slipped / _TICK + if direction > 0: + steps = math.ceil(units - 1e-9) + elif direction < 0: + steps = math.floor(units + 1e-9) + else: + steps = round(units) + return round(steps * _TICK, 2) diff --git a/agent/backtest/engines/composite.py b/agent/backtest/engines/composite.py index 9bb9401e..069300ef 100644 --- a/agent/backtest/engines/composite.py +++ b/agent/backtest/engines/composite.py @@ -55,6 +55,9 @@ def _build_rule_engines(config: dict, codes: List[str]) -> Dict[str, BaseEngine] "Argentina .BA market data is supported, but Argentina backtest " "execution rules are not modeled yet" ) + elif market == "br_equity": + from backtest.engines.brazil_equity import BrazilEquityEngine + engines["br_equity"] = BrazilEquityEngine(config) elif market == "ca_equity": from backtest.engines.global_equity import GlobalEquityEngine engines["ca_equity"] = GlobalEquityEngine(config, market="ca") diff --git a/agent/backtest/loader_health.py b/agent/backtest/loader_health.py index 759b6a28..46e31c20 100644 --- a/agent/backtest/loader_health.py +++ b/agent/backtest/loader_health.py @@ -44,6 +44,9 @@ CANARY_SYMBOLS = { } EXCLUDED_PUBLIC_SOURCES = { "local": "operator files, not a public endpoint", + # Reads the local SQLite store that `python -m src.br_data sync` fills + # from B3's COTAHIST files; it never calls a network endpoint itself. + "b3_cotahist": "local br-data store, synced separately", # Retired 2026-10-08 (HKUDS/Vibe-Trading#1729): TDX servers stopped # answering the mootdx/tdxpy client protocol in 2026-07 # (mootdx/mootdx#157) and upstream is dormant. The loader stays diff --git a/agent/backtest/loaders/_fundamental_schema.py b/agent/backtest/loaders/_fundamental_schema.py index b2c39b4d..c68e70d6 100644 --- a/agent/backtest/loaders/_fundamental_schema.py +++ b/agent/backtest/loaders/_fundamental_schema.py @@ -150,6 +150,8 @@ SEC_CONCEPT_MAP: dict[str, list[str]] = { "Cash", "CashAndDueFromBanks", ], + # Basic shares are not diluted shares, even when rounded EPS values match. + # Missing diluted counts remain unavailable instead of guessing dilution. "shares_diluted": [ "WeightedAverageNumberOfDilutedSharesOutstanding", "WeightedAverageDilutedSharesOutstanding", @@ -194,6 +196,31 @@ DERIVED_FIELDS: dict[str, DerivedFieldSpec] = { ) - 1, }, + # Year-over-year revenue growth, looked up by the same calendar lag as + # asset_growth (see _lag_365d) rather than row position, since the real + # pipeline feeds this a daily-densified panel, not one row per annual + # period_end. + "revenue_growth": { + "dependencies": ["revenue"], + "compute": lambda data: _safe_divide( + data["revenue"].astype("float64"), + _lag_365d(data["revenue"].astype("float64")), + ) + - 1, + }, + # Year-over-year earnings growth, sign-safe: the denominator is the + # ABSOLUTE value of the lagged net_income, so a negative prior year does + # not flip the growth sign (NI -10 -> +5 must read as positive growth). + # A zero or NaN lag yields NaN via _safe_divide's zero-masking. Calendar + # lag for the same daily-densified-panel reason as asset_growth above. + "earnings_growth": { + "dependencies": ["net_income"], + "compute": lambda data: _safe_divide( + data["net_income"].astype("float64") + - _lag_365d(data["net_income"].astype("float64")), + _lag_365d(data["net_income"].astype("float64")).abs(), + ), + }, "accruals": { "dependencies": ["net_income", "cfo", "total_assets"], "compute": lambda data: _safe_divide( diff --git a/agent/backtest/loaders/b3_cotahist_loader.py b/agent/backtest/loaders/b3_cotahist_loader.py new file mode 100644 index 00000000..5556b8ab --- /dev/null +++ b/agent/backtest/loaders/b3_cotahist_loader.py @@ -0,0 +1,90 @@ +"""B3 COTAHIST loader: official daily bars for every B3 instrument. + +Reads the local SQLite store filled by ``python -m src.br_data sync`` (see +:mod:`src.br_data.cotahist`); it never touches the network, so it is the +offline fallback of the ``br_equity`` chain after Yahoo/yfinance. + +Symbol convention: ``PETR4.SA`` -> COTAHIST ``CODNEG`` ``PETR4``. Round-lot +rows (TPMERC 010) are served; ``PETR4F.SA`` reads the odd-lot market (020). +Prices are as traded (caliber ``raw``): no dividend/split adjustment. +""" + +from __future__ import annotations + +import logging +from typing import Dict, List, Optional + +import pandas as pd + +from backtest.loaders.base import normalize_declared_quote_currency, validate_date_range +from backtest.loaders.registry import register + +logger = logging.getLogger(__name__) + +_OUTPUT_COLUMNS = ["open", "high", "low", "close", "volume"] +_CASH_MARKET = 10 +_ODD_LOT_MARKET = 20 + + +def map_symbol(code: str) -> tuple[str, int]: + """``PETR4.SA`` -> (``PETR4``, 10); ``PETR4F.SA`` -> (``PETR4F``, 20).""" + ticker = code.strip().upper() + if ticker.endswith(".SA"): + ticker = ticker[:-3] + market = _ODD_LOT_MARKET if ticker.endswith("F") and ticker[-2:-1].isdigit() else _CASH_MARKET + return ticker, market + + +@register +class DataLoader: + """Offline B3 daily bars from the official COTAHIST files.""" + + name = "b3_cotahist" + markets = {"br_equity"} + # QUATOT is the number of securities traded. + volume_units = {"br_equity": "shares"} + requires_auth = False + + def is_available(self) -> bool: + """Available once a sync has created the store.""" + from src.br_data.store import db_path + + return db_path().exists() + + def fetch( + self, + codes: List[str], + start_date: str, + end_date: str, + *, + interval: str = "1D", + fields: Optional[List[str]] = None, + ) -> Dict[str, pd.DataFrame]: + """Return ``{code: OHLCV frame}`` for codes present in the store.""" + validate_date_range(start_date, end_date) + if str(interval).strip().lower() not in {"1d", "d", "day", "daily"}: + logger.warning("b3_cotahist serves daily bars only; rejecting interval=%r", interval) + return {} + from src.br_data.store import connect + + result: Dict[str, pd.DataFrame] = {} + try: + with connect() as conn: + for code in codes: + ticker, market = map_symbol(code) + rows = conn.execute( + "SELECT trade_date, open, high, low, close, volume FROM b3_quotes " + "WHERE ticker = ? AND market_type = ? AND trade_date BETWEEN ? AND ? " + "ORDER BY trade_date", + (ticker, market, start_date[:10], end_date[:10]), + ).fetchall() + if not rows: + continue + frame = pd.DataFrame([tuple(r) for r in rows], columns=["trade_date", *_OUTPUT_COLUMNS]) + frame["trade_date"] = pd.to_datetime(frame["trade_date"]) + frame = frame.set_index("trade_date").astype(float) + # The importer admits only explicit R$ records, excluding older currencies. + result[code] = normalize_declared_quote_currency(frame, code, "BRL") + except FileNotFoundError as exc: + logger.warning("b3_cotahist: %s", exc) + return result diff --git a/agent/backtest/loaders/base.py b/agent/backtest/loaders/base.py index 7a8816ed..7aab90b0 100644 --- a/agent/backtest/loaders/base.py +++ b/agent/backtest/loaders/base.py @@ -39,6 +39,7 @@ _UK_EQUITY_PATTERN = re.compile(r"^[A-Z0-9&.\-]+\.L$", re.I) # these markets' chains is Yahoo or yfinance, so the declared currency decides. _SINGLE_CURRENCY_VENUES: tuple[tuple[re.Pattern[str], str], ...] = ( (re.compile(r"^[A-Z0-9&.\-]+\.BA$", re.I), "ARS"), + (re.compile(r"^[A-Z0-9]+\.SA$", re.I), "BRL"), (re.compile(r"^[A-Z0-9&.\-]+\.(?:TO|V)$", re.I), "CAD"), ) @@ -477,7 +478,9 @@ _LOADER_CACHE_TRUE_VALUES = {"1", "true", "yes", "on"} # forward walk hold tail-truncated multi-year series and must never be served. # v8: FMP/Tiingo keep one adjusted/raw basis and cached frames retain their # actual adjustment stamp. Older entries may contain artificial return jumps. -_LOADER_CACHE_VERSION = 8 +# v9: SEC successor/predecessor facts extend historical coverage. Old cached +# panels may omit the predecessor's filings and must be rebuilt. +_LOADER_CACHE_VERSION = 9 _LOADER_FRAME_METADATA_ATTRS = ("quote_currency", "currency_conversion", "adjustment") diff --git a/agent/backtest/loaders/registry.py b/agent/backtest/loaders/registry.py index fa8ada01..cc39e321 100644 --- a/agent/backtest/loaders/registry.py +++ b/agent/backtest/loaders/registry.py @@ -61,6 +61,7 @@ VALID_SOURCES: set[str] = { "longbridge", "mt5", "tickerall", + "b3_cotahist", "local", "auto", } @@ -123,6 +124,7 @@ def _ensure_registered() -> None: "backtest.loaders.longbridge", "backtest.loaders.mt5_loader", "backtest.loaders.tickerall_loader", + "backtest.loaders.b3_cotahist_loader", "backtest.loaders.local_loader", ] import importlib @@ -153,7 +155,7 @@ def _ensure_registered() -> None: # as if it were Toman — a caliber error of about six orders of magnitude, not a # missing-data error. An unreachable Iranian endpoint must be visible. _NO_NETWORK_FALLBACK_SOURCES: frozenset[str] = frozenset( - {"local", "qveris", "tickerall", "fmp", "nobitex", "wallex"} + {"local", "qveris", "tickerall", "fmp", "nobitex", "wallex", "b3_cotahist"} ) # QVERIS-INTEGRATION @@ -226,6 +228,10 @@ FALLBACK_CHAINS: dict[str, list[str]] = { "ca_equity": ["yahoo", "yfinance", "local"], # Argentina (BYMA .BA): Yahoo public chart first, yfinance fallback. "ar_equity": ["yahoo", "yfinance", "local"], + # Brazil (B3 .SA): Yahoo public chart (adjusted) first, yfinance second, + # then the official B3 COTAHIST files (unadjusted, daily, every listed + # instrument) synced by ``vibe-trading br-data sync``. + "br_equity": ["yahoo", "yfinance", "b3_cotahist", "local"], # UK (LSE .L): direct Yahoo first, SDK fallback second. "uk_equity": ["yahoo", "yfinance", "local"], # Vietnam (.VN): Yahoo lists HOSE only — HNX and UPCOM are unsupported, @@ -284,6 +290,7 @@ PRICE_CALIBER_BY_SOURCE: dict[str, str] = { "sina": "raw", "alphavantage": "raw", # TIME_SERIES_DAILY, not the _ADJUSTED endpoint "longbridge": "raw", # pins AdjustType.NoAdjust + "b3_cotahist": "raw", # B3 COTAHIST files carry traded prices only } #: Per-(source, market) exceptions to the per-source table. diff --git a/agent/backtest/loaders/sec_edgar_client.py b/agent/backtest/loaders/sec_edgar_client.py index 1073e6eb..f1006217 100644 --- a/agent/backtest/loaders/sec_edgar_client.py +++ b/agent/backtest/loaders/sec_edgar_client.py @@ -25,6 +25,9 @@ shape. The ticker->CIK table is fetched once per process and memoized. from __future__ import annotations import logging +import json +from copy import deepcopy +from pathlib import Path import threading from typing import Any, Dict, Optional @@ -224,7 +227,31 @@ def get_company_facts(cik: str | int) -> Dict[str, Any]: ValueError: If ``cik`` contains no digits. requests.RequestException: On a non-2xx status or undecodable body. """ - return _sec_get_json(_COMPANY_FACTS_URL.format(cik=_pad_cik(cik))) + padded = _pad_cik(cik) + facts = _sec_get_json(_COMPANY_FACTS_URL.format(cik=padded)) + # Resolve current filings through the SEC table; only the facts history is + # extended. Never redirect submissions or discard the successor's filings. + lineage = json.loads(Path(__file__).with_name("sec_filer_lineage.json").read_text()) + predecessors = lineage.get(padded, []) + if not predecessors: + return facts + merged = deepcopy(facts) + for predecessor in predecessors: + old = _sec_get_json(_COMPANY_FACTS_URL.format(cik=predecessor["cik"])) + for namespace, concepts in old.get("facts", {}).items(): + target = merged.setdefault("facts", {}).setdefault(namespace, {}) + for concept, node in concepts.items(): + for unit, rows in node.get("units", {}).items(): + # Post-reorganization subsidiary reports need not describe + # the consolidated public company. Admit pre-change filings + # only; keep their original dates for PIT extraction. + eligible = [row for row in rows if row.get("filed") + and row["filed"] < predecessor["before"]] + if not eligible: + continue + destination = target.setdefault(concept, {"units": {}}) + destination.setdefault("units", {}).setdefault(unit, []).extend(eligible) + return merged def _reset_ticker_cache_for_tests() -> None: diff --git a/agent/backtest/loaders/sec_filer_lineage.json b/agent/backtest/loaders/sec_filer_lineage.json new file mode 100644 index 00000000..e56aa2d2 --- /dev/null +++ b/agent/backtest/loaders/sec_filer_lineage.json @@ -0,0 +1,7 @@ +{ + "0002115436": [{ + "cik": "0000034088", + "before": "2026-07-01", + "source": "https://www.sec.gov/Archives/edgar/data/2115436/000003408826000093/R9.htm" + }] +} diff --git a/agent/backtest/loaders/valuation_loader.py b/agent/backtest/loaders/valuation_loader.py new file mode 100644 index 00000000..5526465a --- /dev/null +++ b/agent/backtest/loaders/valuation_loader.py @@ -0,0 +1,166 @@ +"""CN valuation panel loader from Tushare ``daily_basic``. + +Supplies the ``val:*`` panel namespace (proposal §4 Tier 2): daily valuation +multiples keyed by ``trade_date``. Unlike the ``fund:*`` XBRL/statement panel +these are daily market data — PIT-safe by construction, with no filing lag to +anchor. Missing dates and non-CN symbols stay NaN; nothing is fabricated. + +Field whitelist: only the ``daily_basic`` columns already proven by the +``extra_fields`` merge in ``backtest/loaders/tushare.py`` are served +(``pe``, ``pe_ttm``, ``pb``, ``ps_ttm``, ``dv_ttm``). Any other ``val:`` field +is rejected with a ``ValueError`` at load time rather than silently NaN-ed, so +a factor declaring an unsupported field fails loudly. + +US/HK supply is deferred (proposal §4 Tier 2 note): non-CN symbols simply get +no data here, and factors declaring ``val:`` columns SkipAlpha on universes +without CN supply. +""" + +from __future__ import annotations + +import logging +from collections.abc import Iterable, Sequence +from typing import Any + +import pandas as pd + +from backtest.loaders._symbol_utils import _is_etf_listed +from backtest.loaders.tushare import ( + TUSHARE_TOKEN_PLACEHOLDERS, + _call_with_backoff, + _is_index, +) + +logger = logging.getLogger(__name__) + +#: ``daily_basic`` columns the ``val:`` namespace serves. ``dv_ttm`` is in +#: percent units (股息率TTM, %); consumers z-score it, which is scale-invariant. +VALUATION_FIELDS: tuple[str, ...] = ("pe", "pe_ttm", "pb", "ps_ttm", "dv_ttm") + +#: A-share exchange suffixes covered by ``daily_basic``. +_CN_EQUITY_SUFFIXES = (".SH", ".SZ", ".BJ") + + +def validate_valuation_fields(fields: Iterable[str]) -> list[str]: + """Validate and de-duplicate requested ``val:`` field names. + + Args: + fields: Field names with the ``val:`` prefix already stripped. + + Returns: + Stable, de-duplicated field list. + + Raises: + ValueError: If any field is outside :data:`VALUATION_FIELDS`. + """ + unknown = [field for field in fields if field not in VALUATION_FIELDS] + if unknown: + raise ValueError( + f"unsupported val: field(s) {unknown}; the valuation namespace " + f"serves only tushare daily_basic columns {list(VALUATION_FIELDS)}" + ) + return list(dict.fromkeys(fields)) + + +def _is_cn_equity(code: str) -> bool: + """Detect A-share equity symbols served by ``daily_basic``. + + ``daily_basic`` is stock-only: indices and listed funds are excluded even + though they carry a ``.SH``/``.SZ`` suffix. + """ + if not code.upper().endswith(_CN_EQUITY_SUFFIXES): + return False + return not (_is_etf_listed(code) or _is_index(code)) + + +def _build_api() -> Any: + """Build a Tushare pro API client from the configured token. + + Raises: + RuntimeError: If ``TUSHARE_TOKEN`` is unset or a placeholder. + """ + import tushare as ts + + from src.config.accessor import get_env_config + + token = get_env_config().data.tushare_token.strip() + if token in TUSHARE_TOKEN_PLACEHOLDERS: + raise RuntimeError( + "TUSHARE_TOKEN not configured; required for val:* valuation columns" + ) + return ts.pro_api(token) + + +def load_valuation_panel( + *, + symbols: Sequence[str], + fields: Iterable[str], + start: str, + end: str, + index: pd.DatetimeIndex, + api: Any | None = None, +) -> dict[str, pd.DataFrame]: + """Load whitelisted ``daily_basic`` valuation fields as wide panels. + + Args: + symbols: Panel symbols; only CN A-share equities are fetched, the rest + stay NaN. + fields: Requested field names (``val:`` prefix stripped). + start: Start date (YYYY-MM-DD). + end: End date (YYYY-MM-DD). + index: Price-panel index to align to. + api: Optional pre-built Tushare pro API (tests inject fakes here). + + Returns: + ``field -> dates x symbols`` frames aligned to ``index``/``symbols``. + Every validated field is present; symbols or dates ``daily_basic`` did + not serve are NaN. + + Raises: + ValueError: If a field is outside :data:`VALUATION_FIELDS`. + RuntimeError: If no API is injected and ``TUSHARE_TOKEN`` is missing. + """ + checked = validate_valuation_fields(fields) + if not checked: + return {} + + def _nan_frame(per_code: dict[str, pd.Series] | None = None) -> pd.DataFrame: + if not per_code: + return pd.DataFrame(float("nan"), index=index, columns=list(symbols)) + return pd.DataFrame(per_code).reindex(index=index, columns=list(symbols)) + + cn_symbols = [code for code in symbols if _is_cn_equity(code)] + if not cn_symbols: + return {field: _nan_frame() for field in checked} + if api is None: + api = _build_api() + + sd = start.replace("-", "") + ed = end.replace("-", "") + series_by_field: dict[str, dict[str, pd.Series]] = {field: {} for field in checked} + + for code in cn_symbols: + try: + basic = _call_with_backoff( + api.daily_basic, + ts_code=code, + start_date=sd, + end_date=ed, + fields="ts_code,trade_date," + ",".join(checked), + ) + except Exception as exc: # noqa: BLE001 - one bad fetch must not raise + logger.warning("daily_basic valuation fetch failed for %s: %s", code, exc) + continue + if basic is None or basic.empty: + logger.warning("daily_basic returned empty valuation data for %s", code) + continue + basic = basic.copy() + basic["trade_date"] = pd.to_datetime(basic["trade_date"], errors="coerce") + basic = basic.dropna(subset=["trade_date"]).set_index("trade_date").sort_index() + for field in checked: + if field in basic.columns: + series_by_field[field][code] = pd.to_numeric( + basic[field], errors="coerce" + ) + + return {field: _nan_frame(per_code) for field, per_code in series_by_field.items()} diff --git a/agent/backtest/loaders/yahoo_loader.py b/agent/backtest/loaders/yahoo_loader.py index ffcf7411..ab8d9e2b 100644 --- a/agent/backtest/loaders/yahoo_loader.py +++ b/agent/backtest/loaders/yahoo_loader.py @@ -64,7 +64,7 @@ def _is_supported(code: str) -> bool: return upper.endswith( ( ".US", ".HK", ".NS", ".BO", ".KS", ".KQ", ".TO", ".V", ".VN", ".BA", - ".L", "=F", "=X", + ".SA", ".L", "=F", "=X", ) ) @@ -191,7 +191,7 @@ class DataLoader: name = "yahoo" markets = { "us_equity", "hk_equity", "india_equity", "kr_equity", "ca_equity", - "vietnam_equity", "uk_equity", "ar_equity", + "vietnam_equity", "uk_equity", "ar_equity", "br_equity", } # Yahoo chart volume is single shares for US/HK equities # (HKUDS/Vibe-Trading#1062; HK verified 2026-08-11, 00700.HK ratio 1.00 diff --git a/agent/backtest/loaders/yfinance_loader.py b/agent/backtest/loaders/yfinance_loader.py index 8cbc3d4a..c239a559 100644 --- a/agent/backtest/loaders/yfinance_loader.py +++ b/agent/backtest/loaders/yfinance_loader.py @@ -255,7 +255,7 @@ class DataLoader: name = "yfinance" markets = { "us_equity", "hk_equity", "india_equity", "kr_equity", "ca_equity", - "vietnam_equity", "uk_equity", "ar_equity", "crypto", + "vietnam_equity", "uk_equity", "ar_equity", "br_equity", "crypto", } # yfinance volume is single shares for US/HK equities # (HKUDS/Vibe-Trading#1062; HK verified 2026-08-11, 00700.HK ratio 1.00 diff --git a/agent/backtest/metrics.py b/agent/backtest/metrics.py index 486d2fad..d05b5d64 100644 --- a/agent/backtest/metrics.py +++ b/agent/backtest/metrics.py @@ -44,6 +44,8 @@ _TRADING_DAYS = { "india_broker": 252, # Korean equity (KRX) "pykrx": 252, + # Brazilian equity (B3) + "b3_cotahist": 252, } # mt5 is a forex/CFD feed: 24x5 sessions → 260 trading days, 24h intraday bars. # US equity (yfinance-style): 6.5h sessions → 390 1m bars/day. @@ -71,6 +73,8 @@ _BARS_PER_DAY = { "india_broker": 375, # Korean equity (6.5h session, 09:00-15:30 KST) "pykrx": 390, + # Brazilian equity (B3, 7h session 10:00-17:00 BRT; daily-only loader) + "b3_cotahist": 420, }, "5m": {"yfinance": 78, "yahoo": 78, "finnhub": 78, "alphavantage": 78, "tiingo": 78, "fmp": 78, "stooq": 78, "longbridge": 78, @@ -82,6 +86,8 @@ _BARS_PER_DAY = { "mt5": 288, "tickerall": 288, "india_broker": 75, "pykrx": 78, + # Brazilian equity (B3, 7h session 10:00-17:00 BRT; daily-only loader) + "b3_cotahist": 84, }, "15m": {"yfinance": 26, "yahoo": 26, "finnhub": 26, "alphavantage": 26, "tiingo": 26, "fmp": 26, "stooq": 26, "longbridge": 26, @@ -93,6 +99,8 @@ _BARS_PER_DAY = { "mt5": 96, "tickerall": 96, "india_broker": 25, "pykrx": 26, + # Brazilian equity (B3, 7h session 10:00-17:00 BRT; daily-only loader) + "b3_cotahist": 28, }, "30m": {"yfinance": 13, "yahoo": 13, "finnhub": 13, "alphavantage": 13, "tiingo": 13, "fmp": 13, "stooq": 13, "longbridge": 13, @@ -104,6 +112,8 @@ _BARS_PER_DAY = { "mt5": 48, "tickerall": 48, "india_broker": 13, "pykrx": 13, + # Brazilian equity (B3, 7h session 10:00-17:00 BRT; daily-only loader) + "b3_cotahist": 14, }, "1H": {"yfinance": 7, "yahoo": 7, "finnhub": 7, "alphavantage": 7, "tiingo": 7, "fmp": 7, "stooq": 7, "longbridge": 7, @@ -115,6 +125,8 @@ _BARS_PER_DAY = { "mt5": 24, "tickerall": 24, "india_broker": 7, "pykrx": 7, + # Brazilian equity (B3, 7h session 10:00-17:00 BRT; daily-only loader) + "b3_cotahist": 7, }, "4H": {"yfinance": 2, "yahoo": 2, "finnhub": 2, "alphavantage": 2, "tiingo": 2, "fmp": 2, "stooq": 2, "longbridge": 2, @@ -126,6 +138,8 @@ _BARS_PER_DAY = { "mt5": 6, "tickerall": 6, "india_broker": 2, "pykrx": 2, + # Brazilian equity (B3, 7h session 10:00-17:00 BRT; daily-only loader) + "b3_cotahist": 2, }, "1D": {"yfinance": 1, "yahoo": 1, "finnhub": 1, "alphavantage": 1, "tiingo": 1, "fmp": 1, "stooq": 1, "longbridge": 1, @@ -137,6 +151,8 @@ _BARS_PER_DAY = { "mt5": 1, "tickerall": 1, "india_broker": 1, "pykrx": 1, + # Brazilian equity (B3, 7h session 10:00-17:00 BRT; daily-only loader) + "b3_cotahist": 1, }, } diff --git a/agent/backtest/models.py b/agent/backtest/models.py index 77797ab3..48b0c477 100644 --- a/agent/backtest/models.py +++ b/agent/backtest/models.py @@ -23,6 +23,10 @@ class Position: leverage: Effective leverage (1 for spot/stocks). entry_bar_idx: Index in the dates array at entry (for holding_bars). entry_commission: Commission paid at entry. + open_price: Execution price of the fill that opened the position. + ``entry_price`` becomes the weighted cost once the position is + scaled into; this keeps what was actually paid at the open. + open_size: Quantity of the fill that opened the position. """ symbol: str @@ -33,6 +37,8 @@ class Position: leverage: float = 1.0 entry_bar_idx: int = 0 entry_commission: float = 0.0 + open_price: float | None = None + open_size: float | None = None @dataclass(frozen=True) @@ -79,6 +85,9 @@ class TradeRecord: commission: Total commission (entry + exit). entry_margin: Actual margin allocated at entry, after size rounding. exit_margin: Margin-equivalent value traded at the exit price. + open_price: Execution price of the fill that opened the position, + or ``None`` when the position did not record it. + open_size: Quantity of that opening fill, or ``None``. """ symbol: str @@ -96,6 +105,8 @@ class TradeRecord: commission: float entry_margin: float = 0.0 exit_margin: float = 0.0 + open_price: float | None = None + open_size: float | None = None @dataclass(frozen=True) diff --git a/agent/backtest/runner.py b/agent/backtest/runner.py index f5ab3928..614e2010 100644 --- a/agent/backtest/runner.py +++ b/agent/backtest/runner.py @@ -59,6 +59,7 @@ _VALID_INTERVALS = {"1m", "5m", "15m", "30m", "1H", "4H", "1D", "1W", "1M"} _VALID_ENGINES = {"daily", "options"} _PRICE_PANEL_COLUMNS = ("open", "high", "low", "close", "volume", "vwap", "amount") _FUND_PREFIX = "fund:" +_VAL_PREFIX = "val:" @dataclass(frozen=True) @@ -851,6 +852,7 @@ _MARKET_TO_SOURCE = { "kr_equity": "pykrx", "ca_equity": "yahoo", "ar_equity": "yahoo", + "br_equity": "yahoo", "uk_equity": "yahoo", "vietnam_equity": "yahoo", "crypto": "okx", @@ -1038,6 +1040,26 @@ def _fund_columns_required(selected_factors: Iterable[Any]) -> list[str]: return out +def _val_columns_required(selected_factors: Iterable[Any]) -> list[str]: + """Collect requested ``val:*`` panel columns from selected factors. + + Args: + selected_factors: Factor metadata/spec objects. + + Returns: + Stable, de-duplicated ``val:*`` column names. + """ + seen: set[str] = set() + out: list[str] = [] + for spec in selected_factors: + for column in _columns_required_from_factor_spec(spec): + if not column.startswith(_VAL_PREFIX) or column in seen: + continue + seen.add(column) + out.append(column) + return out + + def _build_price_panel(data_map: dict[str, pd.DataFrame]) -> dict[str, pd.DataFrame]: """Convert ``code -> OHLCV frame`` rows into the factor panel shape. @@ -1139,6 +1161,85 @@ def _inject_fundamental_panel( return panel +def _inject_valuation_panel( + panel: dict[str, pd.DataFrame], + *, + symbols: list[str], + val_columns: Iterable[str], + start: str, + end: str, + index: pd.DatetimeIndex | None = None, +) -> dict[str, pd.DataFrame]: + """Inject required ``val:*`` daily valuation fields into a factor panel. + + Source is tushare ``daily_basic`` (CN A-share only): daily market data + keyed by ``trade_date``, so PIT-safe by construction — no filing lag. + Non-CN symbols and missing dates stay NaN; nothing is fabricated, and + factors declaring ``val:`` columns SkipAlpha or NaN-out naturally where + there is no CN supply. + + Args: + panel: Existing factor panel keyed by column name. + symbols: Symbols to load. + val_columns: Requested ``val:*`` columns. + start: Start date string. + end: End date string. + index: Optional price index to align to. Defaults to ``panel["close"]``. + + Returns: + The same panel dictionary with ``val:`` frames added. + + Raises: + ValueError: If a declared ``val:`` field is outside the daily_basic + whitelist (pe, pe_ttm, pb, ps_ttm, dv_ttm). + """ + fields = [column[len(_VAL_PREFIX):] for column in val_columns if column.startswith(_VAL_PREFIX)] + fields = list(dict.fromkeys(fields)) + if not fields: + return panel + + from backtest.loaders.valuation_loader import ( + load_valuation_panel, + validate_valuation_fields, + ) + + # Whitelist rejection is a caller-fixable contract error, so it must + # propagate rather than degrade into NaN frames like a provider outage. + fields = validate_valuation_fields(fields) + + price_index = index + if price_index is None: + close = panel.get("close") + price_index = close.index if close is not None else pd.DatetimeIndex([]) + + try: + loaded = load_valuation_panel( + symbols=symbols, + fields=fields, + start=start, + end=end, + index=price_index, + ) + except Exception as exc: # noqa: BLE001 - data-source failure must not kill backtest + logger.warning( + "valuation panel load failed for fields=%s symbols=%s: %s; injecting NaN frames", + fields, + symbols, + exc, + exc_info=True, + ) + loaded = {} + + for field in fields: + frame = loaded.get(field) + if not isinstance(frame, pd.DataFrame): + frame = _nan_fundamental_frame(price_index, symbols) + else: + frame = frame.reindex(index=price_index, columns=symbols) + panel[f"{_VAL_PREFIX}{field}"] = frame + return panel + + def _project_panel_fields_to_data_map( data_map: dict[str, pd.DataFrame], panel: dict[str, pd.DataFrame], @@ -1162,28 +1263,41 @@ def _maybe_inject_fundamentals_for_factor_panel( data_map: dict[str, pd.DataFrame], config: dict, ) -> dict[str, pd.DataFrame]: - """Inject ``fund:*`` factor dependencies when selected factors request them.""" + """Inject ``fund:*``/``val:*`` factor dependencies when selected factors request them.""" selected_factors = _selected_factor_specs(config) fund_columns = _fund_columns_required(selected_factors) - if not fund_columns: + val_columns = _val_columns_required(selected_factors) + if not fund_columns and not val_columns: return data_map panel = _build_price_panel(data_map) close = panel.get("close") price_index = close.index if close is not None else pd.DatetimeIndex([]) symbols = list(data_map) - _inject_fundamental_panel( - panel, - symbols=symbols, - fund_columns=fund_columns, - start=config.get("start_date", ""), - end=config.get("end_date", ""), - freq="ttm", - pit=True, - source="auto", - index=price_index, + if fund_columns: + _inject_fundamental_panel( + panel, + symbols=symbols, + fund_columns=fund_columns, + start=config.get("start_date", ""), + end=config.get("end_date", ""), + freq="ttm", + pit=True, + source="auto", + index=price_index, + ) + if val_columns: + _inject_valuation_panel( + panel, + symbols=symbols, + val_columns=val_columns, + start=config.get("start_date", ""), + end=config.get("end_date", ""), + index=price_index, + ) + return _project_panel_fields_to_data_map( + data_map, panel, [*fund_columns, *val_columns] ) - return _project_panel_fields_to_data_map(data_map, panel, fund_columns) # --- Main entry --- @@ -1306,8 +1420,17 @@ def main(run_dir: Path) -> None: # Every source has already been fetched, sanitized, and enriched above. # Reuse that exact snapshot so provider costs and run-card provenance stay - # aligned with the data consumed by the engine. - loader = _AutoLoader(data_map) + # aligned with the data consumed by the engine. A source that prohibits + # fallback keeps its own loader behind the snapshot: a declared benchmark + # is usually outside ``codes``, but must still come from that same source. + if is_no_network_fallback_source(source): + loader = _AutoLoader( + data_map, + inner=loader, + name=str(getattr(loader, "name", None) or source), + ) + else: + loader = _AutoLoader(data_map) if engine_type == "options": from backtest.engines.options_portfolio import run_options_backtest @@ -1548,6 +1671,11 @@ def _create_market_engine(source: str, config: dict, codes: List[str]): if "vietnam_equity" in markets: from backtest.engines.vietnam_equity import VietnamEquityEngine return VietnamEquityEngine(config) + # Brazil (B3) routing — same reason as India: ``yahoo`` has no Wave-1 + # branch and would otherwise fall through to the crypto default. + if "br_equity" in markets: + from backtest.engines.brazil_equity import BrazilEquityEngine + return BrazilEquityEngine(config) # Argentina market-data routing is supported, but BYMA execution rules # are not modeled yet. Fail closed instead of silently applying US/crypto # commissions, lot sizes, settlement, or short-selling assumptions. @@ -1959,14 +2087,68 @@ def _sanitize_data_map(data_map: dict) -> dict: class _AutoLoader: - """Loader adapter that returns a pre-fetched data map.""" + """Loader adapter that returns a pre-fetched data map. - def __init__(self, data_map: dict): + With an ``inner`` loader, symbols outside the snapshot are fetched through + it, and ``name`` reports that loader's source so benchmark resolution can + tell the adapter apart from a network loader swapped in by a fallback. + """ + + def __init__( + self, + data_map: dict, + inner: Any = None, + name: str | None = None, + ) -> None: + """Wrap a fetched snapshot. + + Args: + data_map: ``code -> DataFrame`` snapshot already consumed by the run. + inner: Loader that served the snapshot; consulted only for symbols + the snapshot does not hold. ``None`` serves the snapshot alone. + name: Source name of ``inner``. When ``None`` the adapter exposes no + ``name``: a bare snapshot does not stand for any one source. + """ self._data = data_map + self._inner = inner + if name is not None: + self.name = name - def fetch(self, codes, start_date, end_date, fields=None, interval="1D"): - """Return preloaded rows for requested codes.""" - return {c: df for c, df in self._data.items() if c in codes} + def fetch( + self, + codes: List[str], + start_date: str, + end_date: str, + fields: Optional[List[str]] = None, + interval: str = "1D", + ) -> dict: + """Return snapshot rows, fetching symbols it lacks through ``inner``. + + Frames from ``inner`` go through the same OHLC sanitisation as the + snapshot, so a benchmark is held to the bar quality the strategy was. + + Args: + codes: Requested symbols. + start_date: Window start, passed through to ``inner``. + end_date: Window end, passed through to ``inner``. + fields: Extra fields, passed through to ``inner``. + interval: Bar interval, passed through to ``inner``. + + Returns: + ``code -> DataFrame`` for every requested symbol that could be served. + """ + served = {c: df for c, df in self._data.items() if c in codes} + missing = [c for c in codes if c not in served] + if missing and self._inner is not None: + source = str(getattr(self, "name", "") or "") + requested = _normalize_codes(missing, source) + fetched = self._inner.fetch( + requested, start_date, end_date, fields=fields, interval=interval + ) + if isinstance(fetched, dict): + restored = _restore_original_codes(fetched, missing, requested) + served.update(_sanitize_data_map(restored)) + return served if __name__ == "__main__": diff --git a/agent/backtest/validation.py b/agent/backtest/validation.py index c7ceb0ae..b7ad3815 100644 --- a/agent/backtest/validation.py +++ b/agent/backtest/validation.py @@ -400,9 +400,11 @@ def _load_trades(run_dir: Path) -> List[TradeRecord]: if df.empty: return [] - # trades.csv has entry+exit row pairs; extract exit rows (they have pnl != 0) + # New artifacts identify exits explicitly, including break-even trades. + # Unmarked historical/other-engine files retain their legacy convention. trades = [] - exit_rows = df[df["pnl"] != 0].reset_index(drop=True) + exit_mask = df["event"].eq("exit") if "event" in df.columns else df["pnl"].ne(0) + exit_rows = df[exit_mask].reset_index(drop=True) for _, row in exit_rows.iterrows(): hold = pd.to_numeric(row.get("holding_bars"), errors="coerce") if pd.isna(hold): diff --git a/agent/src/agent/context.py b/agent/src/agent/context.py index f8d83b98..3b18250a 100644 --- a/agent/src/agent/context.py +++ b/agent/src/agent/context.py @@ -98,7 +98,7 @@ Decide which workflow to use based on the request: Override: if the user explicitly requests full analysis regardless of routing, run all 4 layers. **Layer 1 — Trade Attribution** (always, if `artifacts/trades.csv` exists): - - Read trades.csv. Exit rows have `pnl != 0` (entry rows have pnl = 0). Exit rows contain pnl, holding_days, return_pct — use exit rows directly, no pairing needed + - Read trades.csv. With an `event` column, use `event == "exit"` for completed trades, including zero-PnL exits; `entry` marks openings. Only files without `event` use the legacy `pnl != 0` convention, whose zero-PnL exits are ambiguous. Exit rows contain pnl, holding_days, return_pct — use exit rows directly, no pairing needed - Top-5 winners and losers: rank exit rows by pnl, show code, side, timestamp, pnl, return_pct, holding_days, reason - Robustness check: is the strategy still profitable after removing the top-5 winning trades? - Exit-reason breakdown: group by `reason`, show count, total_pnl, avg_pnl, win_rate per group @@ -218,9 +218,15 @@ Decide which workflow to use based on the request: `observed` with the backtest's run directory as `ref`, e.g. `rp`, or the file you read, e.g. `rp/artifacts/target_positions.csv`; two backtests are two directories, so a comparison names each one (`rp::sharpe`, `ew::sharpe`); - `derived` — arithmetic on observed values (`note`: the formula; every number - added or subtracted must itself be an observed value; `ref`: where the - operands came from, e.g. `rp, ew` for a difference between two backtests); + `derived` — arithmetic on observed values (`note`: the formula; ALL data + operands, including numerators and denominators, must be observed; `ref`: + where they came from, e.g. `rp, ew` for a difference between two backtests). + A chosen multiplier/divisor needs a `count` declaration AND its value and + note label on the same visible line of the answer, even for an integer + (e.g. "Discount factor: 97%.", declared `97% | count | discount factor`). + Literal 1 and square/cube exponents are structural constants. Every added + or subtracted term still needs observed data. Never relabel missing data + as a chosen parameter: fetch it or remove the calculation; `proposed` — a price level you suggest, such as an entry, stop or target: inside the observed price range, or with a formula over observed values in `note`; a percentage is not a level, so state the price it implies; @@ -228,10 +234,12 @@ Decide which workflow to use based on the request: same sentence as the figure, and in `note`; `count` — a count, weight, threshold, window, probability or other parameter you chose, never a price or an amount. Plain integers, dates and security - codes need no line. Example (zh): + codes need no line unless used as a chosen coefficient above. Example (zh, + with "折扣系数：97%。" visible in the answer): ```figures 0.666 | observed | 159516.SZ 收盘 2026-09-09 | 159516.SZ 0.646 | derived | 0.666 × 0.97 | 159516.SZ + 97% | count | 折扣系数 37% | derived | (1.053 − 0.666) / 1.053 | 159516.SZ 0.62 | proposed | 买入参考，位于观测区间 0.567–1.053 内 ``` diff --git a/agent/src/agent/grounding/evidence.py b/agent/src/agent/grounding/evidence.py index 9ce60e2f..932294d6 100644 --- a/agent/src/agent/grounding/evidence.py +++ b/agent/src/agent/grounding/evidence.py @@ -1031,10 +1031,12 @@ class _EvidenceMixin: arguments: Mapping[str, Any], payload: dict[str, Any] | None, call_id: str, + *, + tool_name: str = "get_market_data", ) -> None: """Convert full OHLCV payloads into source-linked evidence rows.""" if payload is None: - self._record_tool_failure("get_market_data", call_id, "malformed JSON result") + self._record_tool_failure(tool_name, call_id, "malformed JSON result") return requested_source = str(arguments.get("source") or "auto") provenance = payload.get("_provenance") @@ -1081,7 +1083,7 @@ class _EvidenceMixin: self._evidence.append( EvidenceRecord( call_id=call_id, - tool="get_market_data", + tool=tool_name, symbol=symbol, source=actual_source, timestamp=timestamp, @@ -1100,7 +1102,7 @@ class _EvidenceMixin: self._evidence.append( EvidenceRecord( call_id=call_id, - tool="get_market_data", + tool=tool_name, symbol=symbol, source=requested_source, timestamp=None, diff --git a/agent/src/agent/grounding/identity.py b/agent/src/agent/grounding/identity.py index 57e5932f..f82393a0 100644 --- a/agent/src/agent/grounding/identity.py +++ b/agent/src/agent/grounding/identity.py @@ -125,7 +125,7 @@ _CANONICAL_SYMBOL_RE = re.compile( # Every equity suffix the market-data layer routes (backtest.engines. # _market_hooks._MARKET_PATTERNS); test_market_identity_parity keeps the # two in step, because .L / .VN / .BA each landed there without landing here. - r"[A-Z][A-Z0-9&.-]{0,19}\.(?:US|NS|BO|FX|TO|V|BA|L|VN)|" + r"[A-Z][A-Z0-9&.-]{0,19}\.(?:US|NS|BO|FX|TO|V|BA|SA|L|VN)|" r"[A-Z0-9]{2,15}(?:-|/)(?:USDT|USDC|USD|BTC|ETH)|" r"[A-Z]{2,15}(?:" + "|".join(_JOINED_CRYPTO_QUOTE_SUFFIXES) + r")|" r"\^[A-Z0-9&.\-]{1,20}|" @@ -255,6 +255,7 @@ def _infer_venue(symbol: str) -> str | None: ".TO": "toronto", ".V": "tsx_venture", ".BA": "buenos_aires", + ".SA": "b3", ".L": "lse", ".VN": "hose", } @@ -322,6 +323,7 @@ def _infer_currency(symbol: str) -> str | None: ".TO": "CAD", ".V": "CAD", ".BA": "ARS", + ".SA": "BRL", # The UK loaders admit only a declared GBP / GBp quote and hand back # GBP, the same contract as the backtest's _MARKET_CURRENCY. ".L": "GBP", diff --git a/agent/src/agent/grounding/ledger.py b/agent/src/agent/grounding/ledger.py index 4ad01341..48dd82ef 100644 --- a/agent/src/agent/grounding/ledger.py +++ b/agent/src/agent/grounding/ledger.py @@ -308,6 +308,16 @@ class GroundingLedger( self.persist() return + if tool_name == "get_br_market_data": + # This tool serves B3 identifiers; bare native tickers must not be + # mistaken for US symbols by generic evidence ingestion. + tickers = arguments.get("tickers") + if isinstance(tickers, list): + arguments = {**arguments, "tickers": [ + str(ticker).strip().upper() if str(ticker).strip().upper().endswith(".SA") + else str(ticker).strip().upper() + ".SA" + for ticker in tickers + ]} self._track_session_symbols(arguments, result) self._note_model_write(tool_name, arguments, payload) if tool_name in _ANALYSIS_TOOLS: @@ -316,6 +326,23 @@ class GroundingLedger( self._ingest_resolution(arguments, payload, call_id) elif tool_name == "get_market_data": self._ingest_market_data(arguments, payload, call_id) + elif (tool_name == "get_br_market_data" and payload is not None + and isinstance(payload.get("data"), dict) + and payload["data"].get("action") == "quotes"): + data = payload["data"] + if data.get("currency") == "BRL": + quotes: dict[str, Any] = {"_provenance": {}} + for row in data.get("rows", []): + if not isinstance(row, dict) or not row.get("ticker"): + continue + ticker = str(row["ticker"]).strip().upper() + symbol = ticker if ticker.endswith(".SA") else ticker + ".SA" + quotes.setdefault(symbol, []).append({ + key: row[key] for key in ("trade_date", "open", "high", "low", "close", "volume") + if key in row + }) + quotes["_provenance"][symbol] = {"source": "b3_cotahist", "quote_currency": "BRL"} + self._ingest_market_data(arguments, quotes, call_id, tool_name=tool_name) elif tool_name == "read_file" and payload is not None: self._ingest_engine_table(payload, call_id) elif tool_name == "read_run_artifact" and payload is not None: diff --git a/agent/src/agent/grounding/policies.py b/agent/src/agent/grounding/policies.py index 43222b52..a6b55178 100644 --- a/agent/src/agent/grounding/policies.py +++ b/agent/src/agent/grounding/policies.py @@ -90,6 +90,7 @@ _CURRENCY_ALIASES = { "CAD": ("cad", "c$", "加元", "加拿大元"), "GBP": ("gbp", "£", "英镑"), "VND": ("vnd", "₫", "越南盾"), + "BRL": ("brl", "r$", "real brasileiro", "reais", "巴西雷亚尔"), "ARS": ("ars", "ar$", "阿根廷比索"), } @@ -493,6 +494,36 @@ class _PolicyMixin: if evaluated is not None for operand in evaluated[1] ] + # The hidden declaration alone cannot turn invented data into a free + # coefficient. A chosen parameter needs a count declaration AND its + # value and label in the visible answer. No prose vocabulary decides + # its role; the same label-matching contract already protects citations. + scalar_parameters: list[float] = [] + for parameter in figures: + declared = block.match(parameter.value, parameter.percent, parameter.digits) + if ( + declared is None + or declared.role != "count" + or parameter.shape not in ("measured", "bare") + or parameter.currency + or parameter.column + ): + continue + line = positions[parameter.line][0] if 0 <= parameter.line < len(positions) else "" + # Citation tokens cover labels embedded in ordinary prose. Also + # accept a complete label in any script, including a short symbol + # such as "w"; it must not match a letter inside an unrelated word. + label = declared.note.strip() + label_visible = bool(label) and ( + bool(_note_tokens(label) & _note_tokens(line)) + or re.search( + r"(? tuple[float, list[float]] | str | None: """Evaluate a declaration's note as an observation-anchored formula. Returns the ``(result, operands)`` pair when the note is arithmetic - over at least two operands, at least one of which the run observed and - every added or subtracted term of which holds an observed operand; - otherwise the reason it is not. + over at least two operands. Data operands must be observed; chosen + scalar parameters must be declared and visible. Every added or + subtracted term still needs an observed anchor. Literal 1 is the + multiplicative identity (also used in ratio-minus-one expressions), + and square/cube exponents belong to the operator, not the operands. """ if declaration is None or not declaration.note.strip(): return "no_formula" @@ -1796,6 +1830,16 @@ class _PolicyMixin: return "no_symbol" if unattributed else "formula_not_anchored" if _unanchored_term(tree, observed): return "additive_operand_not_observed" + if any( + not observed(operand) + and operand != 1 + and not any( + math.isclose(abs(operand), abs(parameter), rel_tol=1e-12, abs_tol=0.0) + for parameter in scalar_parameters + ) + for operand in operands + ): + return "operand_not_observed" return result, operands @staticmethod @@ -1826,9 +1870,13 @@ class _PolicyMixin: declaration: Declaration | None, symbol: str | None, records: Sequence[EvidenceRecord], + scalar_parameters: Sequence[float] = (), ) -> list[dict[str, Any]]: """A derived figure must be the arithmetic its note states.""" - derivation = self._derivation(declaration, symbol, records, money=figure.currency) + derivation = self._derivation( + declaration, symbol, records, money=figure.currency, + scalar_parameters=scalar_parameters, + ) if isinstance(derivation, str): return [ self._figure_issue( @@ -1839,7 +1887,12 @@ class _PolicyMixin: "derived", symbol, derivation, - "is declared derived, but its note adds or subtracts an operand " + "uses a data operand this session did not observe. Fetch the missing " + "data; do not relabel a numerator or denominator as a parameter. " + "For a genuinely chosen coefficient, declare its value as count and " + "show that value and its note label in the answer" + if derivation == "operand_not_observed" + else "is declared derived, but its note adds or subtracts an operand " "this session did not observe" if derivation == "additive_operand_not_observed" else "is declared derived, but its note is not arithmetic over at " @@ -1875,6 +1928,7 @@ class _PolicyMixin: declaration: Declaration | None, symbol: str | None, records: Sequence[EvidenceRecord], + scalar_parameters: Sequence[float] = (), ) -> list[dict[str, Any]]: """A proposed level (entry, target, stop) is derived or inside the observed range. @@ -1907,7 +1961,10 @@ class _PolicyMixin: "instrument and nothing attributes it to one", ) ] - derivation = self._derivation(declaration, symbol, records, money=figure.currency) + derivation = self._derivation( + declaration, symbol, records, money=figure.currency, + scalar_parameters=scalar_parameters, + ) if not isinstance(derivation, str) and self._result_matches(figure, derivation[0]): return [] candidates = self._price_candidates(symbol, records) diff --git a/agent/src/agent/grounding/release.py b/agent/src/agent/grounding/release.py index 9cbc0c90..ea90dce0 100644 --- a/agent/src/agent/grounding/release.py +++ b/agent/src/agent/grounding/release.py @@ -88,6 +88,7 @@ _CORRECTION_REASONS = { "unparseable_declaration": "the declaration line could not be read as `value | role | note | ref`", "symbol_never_handled": "no tool call in this session passed in or returned that symbol", "additive_operand_not_observed": "a number its note adds or subtracts is not a value this session observed", + "operand_not_observed": "a data operand is missing from this session's evidence; fetch it, or explicitly declare and show a genuinely chosen coefficient with its note label", "role_in_price_column": "it sits in a price column, which holds only observed prints", "proposed_not_a_price": "a proposed level is a price, not a percentage; state the price it implies", "symbol_mismatch": "it is declared for one instrument but the sentence writes it about another", @@ -242,7 +243,9 @@ class _ReleaseMixin: "cell, one per line as `value | role | note | ref`, where role is one " "of observed / derived / proposed / cited / count.", "observed must appear in the tool results; derived needs a note that is " - "the arithmetic itself, with one operand this session observed; " + "the arithmetic itself, with every data operand observed. A chosen " + "multiplier/divisor needs a count declaration plus its value and note " + "label on the same visible line. Never relabel missing data as a parameter; " "proposed must be derived or lie inside the observed price range; " "cited needs a source in its note; count is not checked.", "Reuse the exact locked symbol and venue.", diff --git a/agent/src/agent/loop.py b/agent/src/agent/loop.py index f79d4348..ae9f65a2 100644 --- a/agent/src/agent/loop.py +++ b/agent/src/agent/loop.py @@ -142,11 +142,11 @@ from src.agent.tool_results import ( # noqa: E402 _FORCED_TEXT_TOOL_CALL_RE as _FORCED_TEXT_TOOL_CALL_RE, _DSML_BAR_TOOL_CALL_RE as _DSML_BAR_TOOL_CALL_RE, _TARGET_PATH_RE as _TARGET_PATH_RE, - _TARGET_ACTION_RE, + _TARGET_ACTION_RE as _TARGET_ACTION_RE, _failure_code, _is_tool_success, _looks_like_tool_call_syntax, - _named_target_paths, + _named_target_paths as _named_target_paths, _normalize_tool_run_dir, _previously_archived as _previously_archived, _archive_backtest_result, @@ -2170,119 +2170,24 @@ class AgentLoop: return True def _record_written_target(self, arguments: Mapping[str, Any]) -> None: - """Remember a file written by write_file/edit_file for completion checks.""" - raw = arguments.get("path") or arguments.get("file_path") - if not raw: - return - p = Path(str(raw)) - if not p.is_absolute() and self.memory.run_dir: - p = Path(self.memory.run_dir) / p - try: - resolved = p.resolve() - self._written_files.add(str(resolved).casefold()) - except (OSError, ValueError): - resolved = p - self._written_files.add(str(p).casefold()) + """Track writes with the active response's model identity.""" + from src.agent.write_targets import record_written_target - if not self.memory.run_dir: - return - run_root = Path(self.memory.run_dir).resolve() - try: - relative = resolved.relative_to(run_root).as_posix() - except ValueError: - return - if relative not in {"config.json", "code/signal_engine.py"}: - return - - metadata_path = run_root / "strategy_provenance.json" - try: - metadata = json.loads(metadata_path.read_text(encoding="utf-8")) - except FileNotFoundError: - files = {} - except (OSError, json.JSONDecodeError) as exc: - logger.warning( - "Could not read strategy model provenance; update skipped: %s", - exc, - ) - return - else: - if not isinstance(metadata, dict): - logger.warning("Invalid strategy model provenance; update skipped.") - return - files = metadata.get("files", {}) - if not isinstance(files, dict): - logger.warning("Invalid strategy model provenance files; update skipped.") - return - files[relative] = { - "provider": self._llm_runtime.provider or None, - "model_id": self._active_model_id or None, - "model_source": self._active_model_source, - } - try: - metadata_path.write_text( - json.dumps({"files": files}, ensure_ascii=False, sort_keys=True) + "\n", - encoding="utf-8", - ) - except OSError as exc: - logger.warning("Could not write strategy model provenance: %s", exc) + record_written_target( + arguments, written_files=self._written_files, + run_dir=self.memory.run_dir, llm_runtime=self._llm_runtime, + active_model_id=self._active_model_id, + active_model_source=self._active_model_source, + ) def _pending_write_directive( self, user_message: str, run_started_wall: float ) -> str: - """Return a directive naming task target files not yet written this run. + """Name task targets not yet written in this run.""" + from src.agent.write_targets import pending_write_directive - Empty when the message names no .md targets, carries no create/update - intent, or every named target has been written (directly via - write_file/edit_file, or by any process whose mtime is newer than the - run start - which covers the bash workaround). - """ - if not _TARGET_ACTION_RE.search(user_message or ""): - return "" - targets = _named_target_paths(user_message) - if not targets: - return "" - pending: list[str] = [] - # A bare target ("create X.md at the same folder") carries no directory; - # resolve it against the folders of the absolute targets named in the - # same message, or the run would never see the file the model wrote to - # the intended folder and would report a false "not written". - base_dirs = [p.parent for p in targets if p.parent != Path(".")] - any_written = False - for p in targets: - candidates = [p] - if not p.is_absolute() and p.parent == Path("."): - candidates += [d / p.name for d in base_dirs] - written = False - for cand in candidates: - try: - key = str(cand.resolve()).casefold() - except (OSError, ValueError): - key = str(cand).casefold() - if key in self._written_files: - written = True - break - if not written: - try: - if cand.exists() and cand.stat().st_mtime >= run_started_wall: - written = True - break - except OSError: - pass - if written: - any_written = True - else: - pending.append(str(p)) - # If any named target was written this run, the create/update task is - # addressed; reference files named alongside (e.g. "refer to A, create B") - # are never "written" and must not trigger the directive. - if any_written or not pending: - return "" - return ( - "[SYSTEM] The following task target file(s) have NOT been written " - "yet in this run: " + ", ".join(pending) + ". Write them NOW using " - "write_file/edit_file (or bash for file operations). If your task " - "was to create or update a file, a plain-text answer without the " - "file write is a failure." + return pending_write_directive( + user_message, run_started_wall, written_files=self._written_files, ) def _context_budget(self, messages: list) -> CompactionBudget: diff --git a/agent/src/agent/write_targets.py b/agent/src/agent/write_targets.py new file mode 100644 index 00000000..ec2cb912 --- /dev/null +++ b/agent/src/agent/write_targets.py @@ -0,0 +1,152 @@ +"""Write-target tracking: written-file ledger + pending-write directive. + +Extracted from ``AgentLoop`` (issue #1624, roadmap item: split ``loop.py`` +along its existing seams). The regex/path helpers this builds on — +``_TARGET_ACTION_RE`` and ``_named_target_paths`` — already moved to +``src.agent.tool_results`` in the first extraction (#1636); this completes +that relocation by moving the two class methods that consumed them, so all +of write-target tracking lives together and ``loop.py`` stays the +orchestrator. + +State stays on the loop: ``written_files`` is passed in and mutated in +place by ``record_written_target``; the provenance identity fields are +read at delegation time in ``loop.py``. Behavior is unchanged — every +branch condition, regex, and message string moved verbatim. +""" + +from __future__ import annotations + +import json +import logging +from pathlib import Path +from typing import Any, Mapping + +from src.agent.tool_results import _TARGET_ACTION_RE, _named_target_paths + +logger = logging.getLogger(__name__) + + +def record_written_target( + arguments: Mapping[str, Any], + *, + written_files: set[str], + run_dir: str | None, + llm_runtime: Any, + active_model_id: str | None, + active_model_source: str, +) -> None: + """Remember a file written by write_file/edit_file for completion checks.""" + raw = arguments.get("path") or arguments.get("file_path") + if not raw: + return + p = Path(str(raw)) + if not p.is_absolute() and run_dir: + p = Path(run_dir) / p + try: + resolved = p.resolve() + written_files.add(str(resolved).casefold()) + except (OSError, ValueError): + resolved = p + written_files.add(str(p).casefold()) + + if not run_dir: + return + run_root = Path(run_dir).resolve() + try: + relative = resolved.relative_to(run_root).as_posix() + except ValueError: + return + if relative not in {"config.json", "code/signal_engine.py"}: + return + + metadata_path = run_root / "strategy_provenance.json" + try: + metadata = json.loads(metadata_path.read_text(encoding="utf-8")) + except FileNotFoundError: + files = {} + except (OSError, json.JSONDecodeError) as exc: + logger.warning( + "Could not read strategy model provenance; update skipped: %s", + exc, + ) + return + else: + if not isinstance(metadata, dict): + logger.warning("Invalid strategy model provenance; update skipped.") + return + files = metadata.get("files", {}) + if not isinstance(files, dict): + logger.warning("Invalid strategy model provenance files; update skipped.") + return + files[relative] = { + "provider": llm_runtime.provider or None, + "model_id": active_model_id or None, + "model_source": active_model_source, + } + try: + metadata_path.write_text( + json.dumps({"files": files}, ensure_ascii=False, sort_keys=True) + "\n", + encoding="utf-8", + ) + except OSError as exc: + logger.warning("Could not write strategy model provenance: %s", exc) + + +def pending_write_directive( + user_message: str, run_started_wall: float, *, written_files: set[str] +) -> str: + """Return a directive naming task target files not yet written this run. + + Empty when the message names no .md targets, carries no create/update + intent, or every named target has been written (directly via + write_file/edit_file, or by any process whose mtime is newer than the + run start - which covers the bash workaround). + """ + if not _TARGET_ACTION_RE.search(user_message or ""): + return "" + targets = _named_target_paths(user_message) + if not targets: + return "" + pending: list[str] = [] + # A bare target ("create X.md at the same folder") carries no directory; + # resolve it against the folders of the absolute targets named in the + # same message, or the run would never see the file the model wrote to + # the intended folder and would report a false "not written". + base_dirs = [p.parent for p in targets if p.parent != Path(".")] + any_written = False + for p in targets: + candidates = [p] + if not p.is_absolute() and p.parent == Path("."): + candidates += [d / p.name for d in base_dirs] + written = False + for cand in candidates: + try: + key = str(cand.resolve()).casefold() + except (OSError, ValueError): + key = str(cand).casefold() + if key in written_files: + written = True + break + if not written: + try: + if cand.exists() and cand.stat().st_mtime >= run_started_wall: + written = True + break + except OSError: + pass + if written: + any_written = True + else: + pending.append(str(p)) + # If any named target was written this run, the create/update task is + # addressed; reference files named alongside (e.g. "refer to A, create B") + # are never "written" and must not trigger the directive. + if any_written or not pending: + return "" + return ( + "[SYSTEM] The following task target file(s) have NOT been written " + "yet in this run: " + ", ".join(pending) + ". Write them NOW using " + "write_file/edit_file (or bash for file operations). If your task " + "was to create or update a file, a plain-text answer without the " + "file write is a failure." + ) diff --git a/agent/src/api/attribution_core.py b/agent/src/api/attribution_core.py index fe2093dc..dd904633 100644 --- a/agent/src/api/attribution_core.py +++ b/agent/src/api/attribution_core.py @@ -514,10 +514,22 @@ def _attribution_run_brinson( def _attribution_load_position_weights(run_dir: Path, notes: List[str]) -> Optional[Dict[str, float]]: - """Read the last row of ``positions.csv`` as per-symbol weights. + """Read the book Brinson should attribute from ``positions.csv``. - Returns ``None`` (with a skip note) when the artifact is missing or cannot - be parsed into at least one weighted symbol column. + A run closes every position on its last bar (``end_of_backtest``), so the + final row is all zeros by construction and describes the liquidation, not + the portfolio. When that row is flat, the row before it is read instead, + and the note names its timestamp. Only one row is stepped back: a book that + was already flat before the close is reported as flat rather than replaced + by some older holding. + + Args: + run_dir: Backtest run directory containing ``artifacts/positions.csv``. + notes: Report notes; skip and fallback reasons are appended here. + + Returns: + Per-symbol weights, or ``None`` (with a skip note) when the artifact is + missing or has no weighted symbol column. """ path = run_dir / "artifacts" / "positions.csv" try: @@ -535,12 +547,27 @@ def _attribution_load_position_weights(run_dir: Path, notes: List[str]) -> Optio if not symbols: notes.append("brinson attribution skipped: positions.csv could not be parsed") return None - last_row = rows[-1] - weights: Dict[str, float] = {} - for symbol in symbols: - weight = _attribution_finite_float(last_row.get(symbol)) - weights[symbol] = weight if weight is not None else 0.0 - return weights + + def _row_weights(row: Dict[str, Any]) -> Dict[str, float]: + out: Dict[str, float] = {} + for symbol in symbols: + weight = _attribution_finite_float(row.get(symbol)) + out[symbol] = weight if weight is not None else 0.0 + return out + + weights = _row_weights(rows[-1]) + if any(weights.values()) or len(rows) < 2: + return weights + held = _row_weights(rows[-2]) + if not any(held.values()): + # Flat before the close as well: the downstream zero-check reports it. + return weights + timestamp = str(rows[-2].get(fieldnames[0]) or "").strip() + notes.append( + "brinson attribution used the positions row before the end-of-backtest " + f"liquidation ({timestamp or 'second-to-last row'})" + ) + return held def _attribution_brinson_section( diff --git a/agent/src/br_data/__init__.py b/agent/src/br_data/__init__.py new file mode 100644 index 00000000..1367e4f8 --- /dev/null +++ b/agent/src/br_data/__init__.py @@ -0,0 +1,17 @@ +"""Free, official Brazilian market data synced into one local SQLite file. + +Sources (all public, no key): + +* B3 COTAHIST — daily quotes of every instrument traded on B3 (shares, units, + FIIs, ETFs, BDRs, options, forward market) since 1986. +* CVM Dados Abertos — daily NAV/quota of every registered investment fund, + the fund register, the listed-company register and annual/quarterly + financial statements (DFP/ITR). +* Banco Central (SGS) — Selic, CDI, IPCA, IGP-M, PTAX and any other series. +* Tesouro Transparente — Tesouro Direto bond prices and yields. + +``python -m src.br_data sync`` downloads/refreshes everything; +``python -m src.br_data daemon`` repeats it once a day. Readers +(:mod:`backtest.loaders.b3_cotahist_loader`, ``get_br_market_data``) only +read the SQLite file. +""" diff --git a/agent/src/br_data/__main__.py b/agent/src/br_data/__main__.py new file mode 100644 index 00000000..290fa1b2 --- /dev/null +++ b/agent/src/br_data/__main__.py @@ -0,0 +1,58 @@ +"""CLI: ``python -m src.br_data {sync,daemon,status}``.""" + +from __future__ import annotations + +import argparse +import json +import logging +import sys + +from src.br_data.sync import SOURCES, run_daemon, run_sync + + +def _status() -> dict: + from src.br_data.store import connect, db_path + + out: dict = {"db": str(db_path())} + try: + with connect() as conn: + for table, col in ( + ("b3_quotes", "trade_date"), ("cvm_fund_daily", "trade_date"), + ("cvm_statements", "period_end"), ("bcb_series", "obs_date"), + ("tesouro_prices", "base_date"), + ): + row = conn.execute(f"SELECT COUNT(*), MIN({col}), MAX({col}) FROM {table}").fetchone() + out[table] = {"rows": row[0], "from": row[1], "to": row[2]} + for table in ("cvm_funds", "cvm_companies", "cvm_company_tickers"): + out[table] = {"rows": conn.execute(f"SELECT COUNT(*) FROM {table}").fetchone()[0]} + except FileNotFoundError as exc: + out["error"] = str(exc) + return out + + +def main(argv: list[str] | None = None) -> int: + parser = argparse.ArgumentParser(prog="python -m src.br_data", description=__doc__) + sub = parser.add_subparsers(dest="command", required=True) + for name in ("sync", "daemon"): + cmd = sub.add_parser(name) + cmd.add_argument("--only", nargs="+", choices=SOURCES, help="sources to sync (default: all)") + if name == "sync": + cmd.add_argument("--cotahist-from", type=int, help="first COTAHIST year (1986+)") + cmd.add_argument("--cvm-from", type=int, help="first CVM year") + sub.add_parser("status") + args = parser.parse_args(argv) + logging.basicConfig(level=logging.INFO, format="%(asctime)s %(levelname)s %(message)s") + + if args.command == "status": + print(json.dumps(_status(), indent=2, ensure_ascii=False)) + return 0 + if args.command == "daemon": + run_daemon(args.only) + return 0 + report = run_sync(args.only, cotahist_from=args.cotahist_from, cvm_from=args.cvm_from) + print(json.dumps(report, indent=2, ensure_ascii=False)) + return 1 if any(isinstance(v, dict) and "error" in v for v in report.values()) else 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/agent/src/br_data/_download.py b/agent/src/br_data/_download.py new file mode 100644 index 00000000..30dede24 --- /dev/null +++ b/agent/src/br_data/_download.py @@ -0,0 +1,78 @@ +"""Download helpers shared by the Brazilian dataset syncs.""" + +from __future__ import annotations + +import io +import logging +import zipfile +import math +from datetime import datetime +from typing import Optional + +from backtest.loaders._http import throttled_get + +logger = logging.getLogger(__name__) + +_MIN_INTERVAL_S = 1.0 +_TIMEOUT_S = 300.0 + + +def fetch_bytes(url: str, host_key: str) -> Optional[bytes]: + """GET ``url``; ``None`` on 404 (file not published), raise on other errors.""" + response = throttled_get(url, host_key=host_key, min_interval=_MIN_INTERVAL_S, timeout=_TIMEOUT_S) + if response.status_code == 404: + return None + response.raise_for_status() + return response.content + + +def zip_members(payload: bytes) -> dict[str, bytes]: + """Return ``{member name: bytes}`` for every file inside a ZIP payload.""" + with zipfile.ZipFile(io.BytesIO(payload)) as archive: + return { + info.filename: archive.read(info) + for info in archive.infolist() + if not info.is_dir() + } + + +def decode_text(raw: bytes) -> str: + """Decode a CVM/B3 text file (UTF-8 when valid, else Latin-1).""" + try: + return raw.decode("utf-8-sig") + except UnicodeDecodeError: + return raw.decode("latin-1") + + +def to_float(value: str | None) -> Optional[float]: + """Parse ``1234.56`` or Brazilian ``1.234,56``; blank -> ``None``.""" + if value is None: + return None + text = str(value).strip() + if not text: + return None + if "," in text: + text = text.replace(".", "").replace(",", ".") + try: + number = float(text) + return number if math.isfinite(number) else None + except ValueError: + return None + + +def br_date(value: str | None) -> Optional[str]: + """``dd/mm/yyyy`` or ``yyyy-mm-dd`` -> ``yyyy-mm-dd``; anything else -> ``None``.""" + if not value: + return None + text = str(value).strip()[:10] + for fmt in ("%d/%m/%Y", "%Y-%m-%d"): + try: + return datetime.strptime(text, fmt).date().isoformat() + except ValueError: + continue + return None + + +def digits(value: str | None) -> str: + """Keep only the digits (CNPJ normalization).""" + return "".join(ch for ch in str(value or "") if ch.isdigit()) diff --git a/agent/src/br_data/bcb.py b/agent/src/br_data/bcb.py new file mode 100644 index 00000000..6b552514 --- /dev/null +++ b/agent/src/br_data/bcb.py @@ -0,0 +1,97 @@ +"""Banco Central do Brasil — SGS time series (Selic, CDI, IPCA, PTAX ...). + +API: ``https://api.bcb.gov.br/dados/serie/bcdata.sgs.{code}/dados`` +``?formato=json&dataInicial=dd/mm/yyyy&dataFinal=dd/mm/yyyy`` returns +``[{"data": "dd/mm/yyyy", "valor": "1.23"}, ...]``. Daily series are served in +windows of at most 10 years, so requests are chunked. +""" + +from __future__ import annotations + +import logging +import sqlite3 +from datetime import date, timedelta +from typing import Optional + +from backtest.loaders._http import throttled_get +from src.br_data._download import br_date, to_float + +logger = logging.getLogger(__name__) + +HOST_KEY = "bcb_sgs" +BASE_URL = "https://api.bcb.gov.br/dados/serie/bcdata.sgs.{code}/dados" +_WINDOW_DAYS = 365 * 5 + +# Series synced by default (code -> description). +SERIES: dict[int, str] = { + 11: "Selic diária (% a.d.)", + 432: "Meta Selic (% a.a.)", + 12: "CDI diário (% a.d.)", + 4389: "CDI anualizado base 252 (% a.a.)", + 433: "IPCA mensal (%)", + 13522: "IPCA acumulado 12 meses (%)", + 189: "IGP-M mensal (%)", + 188: "INPC mensal (%)", + 1: "Dólar comercial venda - PTAX (R$/US$)", + 21619: "Euro venda - PTAX (R$/€)", + 4380: "PIB mensal - valores correntes (R$ milhões)", + 24369: "Taxa de desocupação - PNAD (%)", +} + + +def fetch_series(code: int, start: date, end: date) -> list[tuple[str, Optional[float]]]: + """Download ``code`` between ``start`` and ``end`` (inclusive).""" + out: list[tuple[str, Optional[float]]] = [] + window_start = start + while window_start <= end: + window_end = min(window_start + timedelta(days=_WINDOW_DAYS), end) + response = throttled_get( + BASE_URL.format(code=code), + host_key=HOST_KEY, + min_interval=0.5, + params={ + "formato": "json", + "dataInicial": f"{window_start:%d/%m/%Y}", + "dataFinal": f"{window_end:%d/%m/%Y}", + }, + timeout=60, + ) + # SGS answers 404 for a window with no observations (e.g. a monthly + # series queried over a few days); that is "no data", not an error. + if response.status_code == 404: + window_start = window_end + timedelta(days=1) + continue + response.raise_for_status() + for item in response.json() or []: + obs = br_date(item.get("data")) + if obs: + out.append((obs, to_float(item.get("valor")))) + window_start = window_end + timedelta(days=1) + return out + + +def store_series(conn: sqlite3.Connection, code: int, rows: list[tuple[str, Optional[float]]]) -> int: + conn.executemany( + "INSERT OR REPLACE INTO bcb_series (code, obs_date, value) VALUES (?, ?, ?)", + [(code, obs, value) for obs, value in rows], + ) + return len(rows) + + +def sync(conn: sqlite3.Connection, start_year: int, today: date, codes: Optional[list[int]] = None) -> int: + """Top up each series from its last stored observation (or ``start_year``).""" + total = 0 + failures: list[str] = [] + codes = codes or list(SERIES) + for code in codes: + row = conn.execute("SELECT MAX(obs_date) FROM bcb_series WHERE code = ?", (code,)).fetchone() + start = date.fromisoformat(row[0]) - timedelta(days=45) if row and row[0] else date(start_year, 1, 1) + try: + total += store_series(conn, code, fetch_series(code, start, today)) + conn.commit() + except Exception as exc: # noqa: BLE001 - one series must not stop the rest + logger.warning("bcb: series %s failed: %s", code, exc) + failures.append(f"{code}: {type(exc).__name__}") + if failures and len(failures) == len(codes): + raise RuntimeError(f"every BCB series failed ({'; '.join(failures)})") + return total diff --git a/agent/src/br_data/cotahist.py b/agent/src/br_data/cotahist.py new file mode 100644 index 00000000..231a6f93 --- /dev/null +++ b/agent/src/br_data/cotahist.py @@ -0,0 +1,197 @@ +"""B3 COTAHIST: daily quotes of every instrument traded on B3. + +B3 publishes the "Séries Históricas" as fixed-width text files (245 chars per +record, Latin-1) inside ZIPs: + +* yearly ``COTAHIST_A{YYYY}.ZIP`` (1986 onwards; the current year is updated + every trading day) +* daily ``COTAHIST_D{DDMMYYYY}.ZIP`` + +Record layout (type 01, 1-based positions from B3's published layout): +DATA 3-10, CODBDI 11-12, CODNEG 13-24, TPMERC 25-27, NOMRES 28-39, +ESPECI 40-49, PREABE 57-69, PREMAX 70-82, PREMIN 83-95, PREMED 96-108, +PREULT 109-121, PREOFC 122-134, PREOFV 135-147, TOTNEG 148-152, +QUATOT 153-170, VOLTOT 171-188, PREEXE 189-201, DATVEN 203-210, +FATCOT 211-217, CODISI 231-242. Prices carry two implied decimals and are +quoted per FATCOT units. + +Prices are as traded (not adjusted for dividends or splits). +""" + +from __future__ import annotations + +import io +import logging +import sqlite3 +import zipfile +from datetime import date, timedelta +from typing import Iterable, Iterator, Optional + +from src.br_data._download import fetch_bytes +from src.br_data.store import is_done, mark_done + +logger = logging.getLogger(__name__) + +HOST_KEY = "b3_cotahist" +BASE_URL = "https://bvmf.bmfbovespa.com.br/InstDados/SerHist" +FIRST_YEAR = 1986 + +_COLUMNS = ( + "trade_date", "ticker", "bdi", "market_type", "short_name", "spec", + "open", "high", "low", "avg", "close", "best_bid", "best_ask", + "trades", "volume", "turnover", "strike", "expiry", "quote_factor", "isin", +) +_INSERT = ( + f"INSERT OR REPLACE INTO b3_quotes ({', '.join(_COLUMNS)}) " + f"VALUES ({', '.join('?' for _ in _COLUMNS)})" +) + + +def _num(field: str) -> int: + field = field.strip() + return int(field) if field.isdigit() else 0 + + +def _iso(field: str) -> Optional[str]: + field = field.strip() + if len(field) != 8 or not field.isdigit() or field == "99991231": + return None + try: + return date(int(field[:4]), int(field[4:6]), int(field[6:8])).isoformat() + except ValueError: + return None + + +def parse_record(line: str) -> Optional[tuple]: + """Parse one COTAHIST line; ``None`` for header/trailer/malformed lines.""" + if len(line) < 245 or not line.startswith("01"): + return None + trade_date = _iso(line[2:10]) + if trade_date is None: + return None + # Earlier COTAHIST history contains pre-real currencies. Never stamp those + # prices as BRL, nor fabricate missing quotation factors or OHLC values. + if line[52:56].strip() != "R$": + return None + factor = _num(line[210:217]) + if factor <= 0 or any(not line[a:b].strip().isdigit() + for a, b in ((56, 69), (69, 82), (82, 95), (108, 121))): + return None + + def price(start: int, end: int) -> float: + return _num(line[start:end]) / 100.0 / factor + + return ( + trade_date, + line[12:24].strip(), + line[10:12].strip(), + _num(line[24:27]), + line[27:39].strip(), + line[39:49].strip(), + price(56, 69), + price(69, 82), + price(82, 95), + price(95, 108), + price(108, 121), + price(121, 134), + price(134, 147), + _num(line[147:152]), + float(_num(line[152:170])), + _num(line[170:188]) / 100.0, + _num(line[188:201]) / 100.0 / factor, + _iso(line[202:210]), + factor, + line[230:242].strip(), + ) + + +def parse_text(text: str) -> Iterator[tuple]: + """Yield parsed records from a whole COTAHIST text body.""" + for line in text.splitlines(): + record = parse_record(line) + if record is not None: + yield record + + +def ingest_zip(conn: sqlite3.Connection, payload: bytes) -> int: + """Insert every record of a COTAHIST ZIP; return the row count. + + The yearly text is several hundred MB once unzipped, so it is streamed + line by line instead of being decoded in one piece. + """ + total = 0 + with zipfile.ZipFile(io.BytesIO(payload)) as archive: + for info in archive.infolist(): + if info.is_dir(): + continue + batch: list[tuple] = [] + with archive.open(info) as raw: + for line in io.TextIOWrapper(raw, encoding="latin-1", newline=""): + record = parse_record(line.rstrip("\r\n")) + if record is None: + continue + batch.append(record) + if len(batch) >= 50_000: + conn.executemany(_INSERT, batch) + total += len(batch) + batch.clear() + if batch: + conn.executemany(_INSERT, batch) + total += len(batch) + return total + + +def _last_date(conn: sqlite3.Connection) -> Optional[date]: + row = conn.execute("SELECT MAX(trade_date) FROM b3_quotes").fetchone() + return date.fromisoformat(row[0]) if row and row[0] else None + + +def _weekdays(start: date, end: date) -> Iterable[date]: + day = start + while day <= end: + if day.weekday() < 5: + yield day + day += timedelta(days=1) + + +def sync(conn: sqlite3.Connection, start_year: int, today: date) -> int: + """Download missing years in full, then the current year day by day. + + Past years are fetched once (tracked in ``sync_log``). The current year is + fetched as one yearly file on first run, then topped up with the small + daily files for every weekday after the last stored date (holidays answer + 404 and are skipped). + """ + start_year = max(start_year, FIRST_YEAR) + total = 0 + for year in range(start_year, today.year): + item = f"A{year}" + if is_done(conn, "cotahist", item): + continue + logger.info("cotahist: downloading %s", item) + payload = fetch_bytes(f"{BASE_URL}/COTAHIST_A{year}.ZIP", HOST_KEY) + if payload is None: + logger.warning("cotahist: %s not published", item) + continue + rows = ingest_zip(conn, payload) + mark_done(conn, "cotahist", item, rows) + conn.commit() + total += rows + + last = _last_date(conn) + if last is None or last.year < today.year: + logger.info("cotahist: downloading A%s (year to date)", today.year) + payload = fetch_bytes(f"{BASE_URL}/COTAHIST_A{today.year}.ZIP", HOST_KEY) + if payload is not None: + total += ingest_zip(conn, payload) + conn.commit() + last = _last_date(conn) + + if last is not None and last.year == today.year: + for day in _weekdays(last + timedelta(days=1), today): + payload = fetch_bytes(f"{BASE_URL}/COTAHIST_D{day:%d%m%Y}.ZIP", HOST_KEY) + if payload is None: + continue + total += ingest_zip(conn, payload) + conn.commit() + return total diff --git a/agent/src/br_data/cvm_companies.py b/agent/src/br_data/cvm_companies.py new file mode 100644 index 00000000..cc043dfe --- /dev/null +++ b/agent/src/br_data/cvm_companies.py @@ -0,0 +1,169 @@ +"""CVM listed companies: register, ticker map and financial statements. + +* ``CIA_ABERTA/CAD/DADOS/cad_cia_aberta.csv`` — company register. +* ``CIA_ABERTA/DOC/FCA/DADOS/fca_cia_aberta_{YYYY}.zip`` — the + ``valor_mobiliario`` file maps B3 tickers (``Codigo_Negociacao``) to CNPJ. +* ``CIA_ABERTA/DOC/DFP/DADOS/dfp_cia_aberta_{YYYY}.zip`` (annual) and + ``.../ITR/DADOS/itr_cia_aberta_{YYYY}.zip`` (quarterly) — one CSV per + statement, e.g. ``dfp_cia_aberta_DRE_con_2025.csv``. + +Only the current-period column (``ORDEM_EXERC = ÚLTIMO``) is kept and the +highest ``VERSAO`` wins. Consolidated statements are stored as ``DRE``, +``BPA`` ...; individual (parent-only) ones as ``DRE_ind`` ... +``ESCALA_MOEDA = MIL`` values are multiplied by 1000 so every value is BRL. +""" + +from __future__ import annotations + +import csv +import io +import logging +import re +import sqlite3 +from datetime import date +from typing import Iterator, Optional + +from src.br_data._download import br_date, decode_text, digits, fetch_bytes, to_float, zip_members +from src.br_data.store import is_done, mark_done + +logger = logging.getLogger(__name__) + +HOST_KEY = "cvm" +BASE_URL = "https://dados.cvm.gov.br/dados/CIA_ABERTA" + +_STATEMENT_FILE = re.compile( + r"(?:dfp|itr)_cia_aberta_(?P[A-Z_]+?)_(?Pcon|ind)_\d{4}\.csv$", re.I +) +_UPSERT_STATEMENT = ( + "INSERT INTO cvm_statements (cnpj, doc, statement, period_end, period_start, account, " + "description, value, version) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?) " + "ON CONFLICT(cnpj, doc, statement, period_end, period_start, account) DO UPDATE SET " + "description = excluded.description, value = excluded.value, version = excluded.version " + "WHERE excluded.version >= cvm_statements.version" +) + + +def _rows(text: str) -> Iterator[dict[str, str]]: + reader = csv.DictReader(io.StringIO(text), delimiter=";") + for row in reader: + yield {str(k or "").strip().upper(): (v or "").strip() for k, v in row.items()} + + +def _opt(value: Optional[str]) -> Optional[str]: + return value or None + + +def parse_register(text: str) -> Iterator[tuple]: + """Yield ``cvm_companies`` tuples from cad_cia_aberta.csv.""" + for row in _rows(text): + cnpj = digits(row.get("CNPJ_CIA")) + if not cnpj: + continue + yield ( + cnpj, + _opt(row.get("CD_CVM")), + _opt(row.get("DENOM_SOCIAL")), + _opt(row.get("DENOM_COMERC")), + _opt(row.get("SIT")), + _opt(row.get("SETOR_ATIV")), + ) + + +def parse_tickers(text: str) -> Iterator[tuple[str, str]]: + """Yield ``(ticker, cnpj)`` from an FCA valor_mobiliario CSV.""" + for row in _rows(text): + ticker = (row.get("CODIGO_NEGOCIACAO") or "").upper() + cnpj = digits(row.get("CNPJ_COMPANHIA") or row.get("CNPJ_CIA")) + if ticker and cnpj and re.fullmatch(r"[A-Z0-9]{4,12}", ticker): + yield ticker, cnpj + + +def parse_statement(text: str, doc: str, statement: str) -> Iterator[tuple]: + """Yield ``cvm_statements`` tuples from one DFP/ITR statement CSV.""" + for row in _rows(text): + order = row.get("ORDEM_EXERC", "").upper() + if order and not order.startswith(("ÚLTIMO", "ULTIMO")): + continue + cnpj = digits(row.get("CNPJ_CIA")) + period_end = br_date(row.get("DT_FIM_EXERC") or row.get("DT_REFER")) + account = row.get("CD_CONTA", "") + if not cnpj or not period_end or not account: + continue + currency = row.get("MOEDA", "").upper() + if currency and currency not in {"REAL", "REAIS", "BRL", "R$"}: + raise ValueError(f"Unsupported CVM statement currency: {currency}") + value = to_float(row.get("VL_CONTA")) + if value is not None and row.get("ESCALA_MOEDA", "").upper() == "MIL": + value *= 1000.0 + version = to_float(row.get("VERSAO")) + yield ( + cnpj, + doc, + statement, + period_end, + br_date(row.get("DT_INI_EXERC")) or "", + account, + _opt(row.get("DS_CONTA")), + value, + int(version) if version is not None else 0, + ) + + +def ingest_statements_zip(conn: sqlite3.Connection, payload: bytes, doc: str) -> int: + """Insert every statement CSV of a DFP/ITR ZIP; return the row count.""" + total = 0 + for name, raw in zip_members(payload).items(): + match = _STATEMENT_FILE.search(name.rsplit("/", 1)[-1]) + if not match: + continue + statement = match.group("stmt").upper() + if match.group("scope").lower() == "ind": + statement += "_ind" + batch = list(parse_statement(decode_text(raw), doc, statement)) + conn.executemany(_UPSERT_STATEMENT, batch) + total += len(batch) + return total + + +def sync(conn: sqlite3.Connection, start_year: int, today: date) -> int: + """Refresh register + tickers, then DFP/ITR from ``start_year``. + + The current and previous year are always refreshed (filings and + restatements keep arriving); older years are fetched once. + """ + total = 0 + register = fetch_bytes(f"{BASE_URL}/CAD/DADOS/cad_cia_aberta.csv", HOST_KEY) + if register is not None: + batch = list(parse_register(decode_text(register))) + conn.executemany("INSERT OR REPLACE INTO cvm_companies VALUES (?, ?, ?, ?, ?, ?)", batch) + total += len(batch) + conn.commit() + + for year in (today.year, today.year - 1): + fca = fetch_bytes(f"{BASE_URL}/DOC/FCA/DADOS/fca_cia_aberta_{year}.zip", HOST_KEY) + if fca is None: + continue + for name, raw in zip_members(fca).items(): + if "valor_mobiliario" in name.lower(): + pairs = list(parse_tickers(decode_text(raw))) + conn.executemany("INSERT OR REPLACE INTO cvm_company_tickers VALUES (?, ?)", pairs) + total += len(pairs) + conn.commit() + break + + for doc in ("DFP", "ITR"): + for year in range(start_year, today.year + 1): + item = f"{doc}{year}" + recent = year >= today.year - 1 + if not recent and is_done(conn, "cvm_statements", item): + continue + url = f"{BASE_URL}/DOC/{doc}/DADOS/{doc.lower()}_cia_aberta_{year}.zip" + payload = fetch_bytes(url, HOST_KEY) + if payload is None: + continue + rows = ingest_statements_zip(conn, payload, doc) + if not recent: + mark_done(conn, "cvm_statements", item, rows) + conn.commit() + total += rows + return total diff --git a/agent/src/br_data/cvm_funds.py b/agent/src/br_data/cvm_funds.py new file mode 100644 index 00000000..b5adc4b5 --- /dev/null +++ b/agent/src/br_data/cvm_funds.py @@ -0,0 +1,183 @@ +"""CVM investment funds: register and daily report (informe diário). + +* Daily report — quota value, net assets, inflows/outflows and holders of + every registered fund/class, one ZIP per month: + ``FI/DOC/INF_DIARIO/DADOS/inf_diario_fi_{YYYYMM}.zip`` (older years as + ``.../DADOS/HIST/inf_diario_fi_{YYYY}.zip``). Since CVM Resolution 175 the + key columns are ``CNPJ_FUNDO_CLASSE``/``TP_FUNDO_CLASSE``/``ID_SUBCLASSE``; + older files use ``CNPJ_FUNDO``/``TP_FUNDO``. Both are accepted. +* Register — legacy ``FI/CAD/DADOS/cad_fi.csv`` plus the post-175 + ``FI/CAD/DADOS/registro_fundo_classe.zip`` (fund and class registers). + +Column lookups are case-insensitive and tolerate either naming, because CVM +renames columns between layouts. +""" + +from __future__ import annotations + +import csv +import io +import logging +import sqlite3 +from datetime import date +from typing import Iterable, Iterator, Optional + +from src.br_data._download import br_date, decode_text, digits, fetch_bytes, to_float, zip_members +from src.br_data.store import is_done, mark_done + +logger = logging.getLogger(__name__) + +HOST_KEY = "cvm" +BASE_URL = "https://dados.cvm.gov.br/dados/FI" + +_INSERT_DAILY = ( + "INSERT OR REPLACE INTO cvm_fund_daily (cnpj, trade_date, fund_type, subclass, " + "total_value, quota, net_assets, inflows, outflows, holders) " + "VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?)" +) +_UPSERT_FUND = ( + "INSERT INTO cvm_funds (cnpj, name, status, fund_class, fund_type, manager, admin, " + "benchmark, start_date) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?) " + "ON CONFLICT(cnpj) DO UPDATE SET " + "name = COALESCE(excluded.name, name), status = COALESCE(excluded.status, status), " + "fund_class = COALESCE(excluded.fund_class, fund_class), " + "fund_type = COALESCE(excluded.fund_type, fund_type), " + "manager = COALESCE(excluded.manager, manager), admin = COALESCE(excluded.admin, admin), " + "benchmark = COALESCE(excluded.benchmark, benchmark), " + "start_date = COALESCE(excluded.start_date, start_date)" +) + + +def _rows(text: str) -> Iterator[dict[str, str]]: + """Iterate a ``;``-separated CVM CSV with upper-cased header keys.""" + reader = csv.DictReader(io.StringIO(text), delimiter=";") + for row in reader: + yield {str(k or "").strip().upper(): (v or "").strip() for k, v in row.items()} + + +def _pick(row: dict[str, str], *names: str) -> str: + for name in names: + value = row.get(name.upper()) + if value: + return value + return "" + + +def _opt(value: str) -> Optional[str]: + return value or None + + +def parse_daily(text: str) -> Iterator[tuple]: + """Yield ``cvm_fund_daily`` tuples from one informe-diário CSV.""" + for row in _rows(text): + cnpj = digits(_pick(row, "CNPJ_FUNDO_CLASSE", "CNPJ_FUNDO")) + trade_date = br_date(_pick(row, "DT_COMPTC")) + if not cnpj or not trade_date: + continue + holders = to_float(_pick(row, "NR_COTST")) + yield ( + cnpj, + trade_date, + _opt(_pick(row, "TP_FUNDO_CLASSE", "TP_FUNDO")), + _pick(row, "ID_SUBCLASSE"), + to_float(_pick(row, "VL_TOTAL")), + to_float(_pick(row, "VL_QUOTA")), + to_float(_pick(row, "VL_PATRIM_LIQ")), + to_float(_pick(row, "CAPTC_DIA")), + to_float(_pick(row, "RESG_DIA")), + int(holders) if holders is not None else None, + ) + + +def ingest_daily_zip(conn: sqlite3.Connection, payload: bytes) -> int: + """Insert every CSV inside an informe-diário ZIP; return the row count.""" + total = 0 + for name, raw in zip_members(payload).items(): + if not name.lower().endswith(".csv"): + continue + batch = list(parse_daily(decode_text(raw))) + conn.executemany(_INSERT_DAILY, batch) + total += len(batch) + return total + + +def parse_register(text: str) -> Iterator[tuple]: + """Yield ``cvm_funds`` tuples from cad_fi.csv or registro_*.csv.""" + for row in _rows(text): + cnpj = digits(_pick(row, "CNPJ_FUNDO", "CNPJ_CLASSE", "CNPJ_FUNDO_CLASSE")) + if not cnpj: + continue + yield ( + cnpj, + _opt(_pick(row, "DENOM_SOCIAL", "DENOMINACAO_SOCIAL")), + _opt(_pick(row, "SIT", "SITUACAO")), + _opt(_pick(row, "CLASSE", "CLASSIFICACAO", "CLASSE_ANBIMA")), + _opt(_pick(row, "TP_FUNDO", "TIPO_CLASSE", "TIPO_FUNDO")), + _opt(_pick(row, "GESTOR")), + _opt(_pick(row, "ADMIN", "ADMINISTRADOR")), + _opt(_pick(row, "RENTAB_FUNDO", "INDICADOR_DESEMPENHO")), + br_date(_pick(row, "DT_INI_ATIV", "DATA_INICIO", "DT_REG", "DATA_REGISTRO")), + ) + + +def _months(start: date, end: date) -> Iterable[tuple[int, int]]: + year, month = start.year, start.month + while (year, month) <= (end.year, end.month): + yield year, month + month += 1 + if month == 13: + year, month = year + 1, 1 + + +def sync_register(conn: sqlite3.Connection) -> int: + """Refresh the fund register (both layouts, best effort each).""" + total = 0 + legacy = fetch_bytes(f"{BASE_URL}/CAD/DADOS/cad_fi.csv", HOST_KEY) + if legacy is not None: + batch = list(parse_register(decode_text(legacy))) + conn.executemany(_UPSERT_FUND, batch) + total += len(batch) + current = fetch_bytes(f"{BASE_URL}/CAD/DADOS/registro_fundo_classe.zip", HOST_KEY) + if current is not None: + for name, raw in zip_members(current).items(): + if name.lower().endswith(".csv") and "subclasse" not in name.lower(): + batch = list(parse_register(decode_text(raw))) + conn.executemany(_UPSERT_FUND, batch) + total += len(batch) + conn.commit() + return total + + +def sync_daily(conn: sqlite3.Connection, start_year: int, today: date) -> int: + """Fetch monthly reports; the last two months are always refreshed. + + CVM revises recent months, so only months older than that are recorded as + done. A month that 404s falls back to the yearly HIST archive once. + """ + total = 0 + recent = {(today.year, today.month)} + prev = (today.year - 1, 12) if today.month == 1 else (today.year, today.month - 1) + recent.add(prev) + hist_years_done: set[int] = set() + for year, month in _months(date(start_year, 1, 1), today): + item = f"{year}{month:02d}" + if (year, month) not in recent and is_done(conn, "cvm_fund_daily", item): + continue + payload = fetch_bytes(f"{BASE_URL}/DOC/INF_DIARIO/DADOS/inf_diario_fi_{item}.zip", HOST_KEY) + if payload is None: + if year in hist_years_done or is_done(conn, "cvm_fund_daily", f"HIST{year}"): + continue + hist = fetch_bytes(f"{BASE_URL}/DOC/INF_DIARIO/DADOS/HIST/inf_diario_fi_{year}.zip", HOST_KEY) + hist_years_done.add(year) + if hist is not None: + rows = ingest_daily_zip(conn, hist) + mark_done(conn, "cvm_fund_daily", f"HIST{year}", rows) + conn.commit() + total += rows + continue + rows = ingest_daily_zip(conn, payload) + if (year, month) not in recent: + mark_done(conn, "cvm_fund_daily", item, rows) + conn.commit() + total += rows + return total diff --git a/agent/src/br_data/store.py b/agent/src/br_data/store.py new file mode 100644 index 00000000..57c38f51 --- /dev/null +++ b/agent/src/br_data/store.py @@ -0,0 +1,180 @@ +"""SQLite store for the Brazilian official datasets.""" + +from __future__ import annotations + +import sqlite3 +from contextlib import contextmanager +from pathlib import Path +from typing import Iterator + +from src.config.accessor import get_env_config + +DEFAULT_DB_PATH = Path.home() / ".vibe-trading" / "br-data" / "br_data.db" + +_SCHEMA = """ +CREATE TABLE IF NOT EXISTS b3_quotes ( + trade_date TEXT NOT NULL, -- YYYY-MM-DD + ticker TEXT NOT NULL, -- CODNEG, e.g. PETR4, PETR4F, PETRJ300 + bdi TEXT, -- CODBDI + market_type INTEGER, -- TPMERC: 10 cash, 20 odd lot, 70/80 options ... + short_name TEXT, -- NOMRES + spec TEXT, -- ESPECI (ON, PN, CI, DRN ...) + open REAL, + high REAL, + low REAL, + avg REAL, + close REAL, + best_bid REAL, + best_ask REAL, + trades INTEGER, + volume REAL, -- QUATOT, number of securities traded + turnover REAL, -- VOLTOT, BRL + strike REAL, + expiry TEXT, + quote_factor INTEGER, + isin TEXT, + PRIMARY KEY (ticker, trade_date, market_type) +) WITHOUT ROWID; +CREATE INDEX IF NOT EXISTS ix_b3_quotes_date ON b3_quotes(trade_date); + +CREATE TABLE IF NOT EXISTS cvm_fund_daily ( + cnpj TEXT NOT NULL, -- digits only + trade_date TEXT NOT NULL, + fund_type TEXT, + subclass TEXT NOT NULL DEFAULT '', + total_value REAL, + quota REAL, + net_assets REAL, + inflows REAL, + outflows REAL, + holders INTEGER, + PRIMARY KEY (cnpj, subclass, trade_date) +) WITHOUT ROWID; + +CREATE TABLE IF NOT EXISTS cvm_funds ( + cnpj TEXT PRIMARY KEY, + name TEXT, + status TEXT, + fund_class TEXT, + fund_type TEXT, + manager TEXT, + admin TEXT, + benchmark TEXT, + start_date TEXT +); + +CREATE TABLE IF NOT EXISTS cvm_companies ( + cnpj TEXT PRIMARY KEY, + cvm_code TEXT, + name TEXT, + trade_name TEXT, + status TEXT, + sector TEXT +); + +CREATE TABLE IF NOT EXISTS cvm_company_tickers ( + ticker TEXT PRIMARY KEY, + cnpj TEXT NOT NULL +); + +CREATE TABLE IF NOT EXISTS cvm_statements ( + cnpj TEXT NOT NULL, + doc TEXT NOT NULL, -- DFP (annual) or ITR (quarterly) + statement TEXT NOT NULL, -- BPA, BPP, DRE, DFC_MD, DFC_MI, DVA, DRA, DMPL + period_end TEXT NOT NULL, + period_start TEXT NOT NULL DEFAULT '', + account TEXT NOT NULL, -- CD_CONTA, e.g. 3.01 + description TEXT, + value REAL, -- in BRL units (ESCALA_MOEDA applied) + version INTEGER, + PRIMARY KEY (cnpj, doc, statement, period_end, period_start, account) +) WITHOUT ROWID; + +CREATE TABLE IF NOT EXISTS bcb_series ( + code INTEGER NOT NULL, + obs_date TEXT NOT NULL, + value REAL, + PRIMARY KEY (code, obs_date) +) WITHOUT ROWID; + +CREATE TABLE IF NOT EXISTS tesouro_prices ( + bond TEXT NOT NULL, -- e.g. "Tesouro IPCA+" + maturity TEXT NOT NULL, + base_date TEXT NOT NULL, + buy_rate REAL, + sell_rate REAL, + buy_price REAL, + sell_price REAL, + base_price REAL, + PRIMARY KEY (bond, maturity, base_date) +) WITHOUT ROWID; + +CREATE TABLE IF NOT EXISTS sync_log ( + source TEXT NOT NULL, + item TEXT NOT NULL, + done_at TEXT NOT NULL, + rows INTEGER, + PRIMARY KEY (source, item) +); +""" + + +def db_path() -> Path: + """Return the configured SQLite path (``VIBE_BR_DATA_DB`` or the default).""" + configured = get_env_config().data.vibe_br_data_db.strip() + return Path(configured).expanduser() if configured else DEFAULT_DB_PATH + + +def init_db(path: Path | None = None) -> Path: + """Create the database file and tables when missing; return its path.""" + target = path or db_path() + target.parent.mkdir(parents=True, exist_ok=True) + with sqlite3.connect(target) as conn: + conn.execute("PRAGMA journal_mode=WAL") + conn.executescript(_SCHEMA) + return target + + +@contextmanager +def connect(path: Path | None = None, *, write: bool = False) -> Iterator[sqlite3.Connection]: + """Open the store. Read-only connections never create the file. + + Raises: + FileNotFoundError: On a read-only open when no sync has run yet. + """ + target = path or db_path() + if write: + init_db(target) + conn = sqlite3.connect(target, timeout=60) + else: + if not target.exists(): + raise FileNotFoundError( + f"Brazilian data store not found at {target}; run " + "`python -m src.br_data sync` first" + ) + conn = sqlite3.connect(f"file:{target}?mode=ro", uri=True, timeout=60) + conn.row_factory = sqlite3.Row + try: + yield conn + if write: + conn.commit() + finally: + conn.close() + + +def mark_done(conn: sqlite3.Connection, source: str, item: str, rows: int) -> None: + """Record that ``source/item`` was fully ingested.""" + from datetime import datetime, timezone + + conn.execute( + "INSERT OR REPLACE INTO sync_log(source, item, done_at, rows) VALUES (?, ?, ?, ?)", + (source, item, datetime.now(timezone.utc).isoformat(timespec="seconds"), rows), + ) + + +def is_done(conn: sqlite3.Connection, source: str, item: str) -> bool: + """Whether ``source/item`` was already ingested.""" + row = conn.execute( + "SELECT 1 FROM sync_log WHERE source = ? AND item = ?", (source, item) + ).fetchone() + return row is not None diff --git a/agent/src/br_data/sync.py b/agent/src/br_data/sync.py new file mode 100644 index 00000000..8a0a61ec --- /dev/null +++ b/agent/src/br_data/sync.py @@ -0,0 +1,70 @@ +"""Run every Brazilian dataset sync, once or as a daily daemon.""" + +from __future__ import annotations + +import logging +import time +from datetime import date, datetime, timedelta +from typing import Callable, Iterable +from zoneinfo import ZoneInfo + +from src.br_data import bcb, cotahist, cvm_companies, cvm_funds, tesouro +from src.br_data.store import connect, init_db +from src.config.accessor import get_env_config + +logger = logging.getLogger(__name__) + +TZ = ZoneInfo("America/Sao_Paulo") +SOURCES = ("cotahist", "cvm_funds", "cvm_companies", "bcb", "tesouro") + + +def _today() -> date: + return datetime.now(TZ).date() + + +def run_sync(only: Iterable[str] | None = None, *, cotahist_from: int | None = None, + cvm_from: int | None = None) -> dict[str, object]: + """Sync the selected sources; a failing source is reported, not fatal.""" + data_cfg = get_env_config().data + selected = [s for s in SOURCES if not only or s in set(only)] + today = _today() + cot_year = cotahist_from or data_cfg.vibe_br_cotahist_start_year + cvm_year = cvm_from or data_cfg.vibe_br_cvm_start_year + jobs: dict[str, Callable] = { + "cotahist": lambda c: cotahist.sync(c, cot_year, today), + "cvm_funds": lambda c: cvm_funds.sync_register(c) + cvm_funds.sync_daily(c, cvm_year, today), + "cvm_companies": lambda c: cvm_companies.sync(c, cvm_year, today), + "bcb": lambda c: bcb.sync(c, min(cot_year, cvm_year), today), + "tesouro": lambda c: tesouro.sync(c), + } + path = init_db() + report: dict[str, object] = {"db": str(path)} + for name in selected: + started = time.monotonic() + try: + with connect(path, write=True) as conn: + rows = jobs[name](conn) + report[name] = {"rows": rows, "seconds": round(time.monotonic() - started, 1)} + logger.info("br-data: %s ok (%s rows)", name, rows) + except Exception as exc: # noqa: BLE001 - keep the other sources going + report[name] = {"error": f"{type(exc).__name__}: {exc}"} + logger.warning("br-data: %s failed: %s", name, exc) + return report + + +def _next_run(now: datetime, hhmm: str) -> datetime: + hour, minute = (int(part) for part in hhmm.split(":", 1)) + target = now.replace(hour=hour, minute=minute, second=0, microsecond=0) + if target <= now: + target += timedelta(days=1) + return target + + +def run_daemon(only: Iterable[str] | None = None) -> None: + """Sync now, then every day at ``VIBE_BR_SYNC_TIME`` (São Paulo time).""" + only = list(only or []) + while True: + logger.info("br-data: sync report %s", run_sync(only)) + target = _next_run(datetime.now(TZ), get_env_config().data.vibe_br_sync_time) + logger.info("br-data: next sync at %s", target.isoformat()) + time.sleep(max((target - datetime.now(TZ)).total_seconds(), 60)) diff --git a/agent/src/br_data/tesouro.py b/agent/src/br_data/tesouro.py new file mode 100644 index 00000000..2ab4758a --- /dev/null +++ b/agent/src/br_data/tesouro.py @@ -0,0 +1,55 @@ +"""Tesouro Direto prices and yields (Tesouro Transparente open data). + +One ``;``-separated CSV with the whole history, Brazilian number format: +``Tipo Titulo;Data Vencimento;Data Base;Taxa Compra Manha;Taxa Venda Manha; +PU Compra Manha;PU Venda Manha;PU Base Manha``. +""" + +from __future__ import annotations + +import csv +import io +import sqlite3 +from typing import Iterator + +from src.br_data._download import br_date, decode_text, fetch_bytes, to_float + +HOST_KEY = "tesouro" +URL = ( + "https://www.tesourotransparente.gov.br/ckan/dataset/" + "df56aa42-484a-4a59-8184-7676580c81e3/resource/" + "796d2059-14e9-44e3-80c9-2d9e30b405c1/download/PrecoTaxaTesouroDireto.csv" +) + + +def parse(text: str) -> Iterator[tuple]: + """Yield ``tesouro_prices`` tuples from the published CSV.""" + reader = csv.DictReader(io.StringIO(text), delimiter=";") + for row in reader: + row = {str(k or "").strip().lower(): (v or "").strip() for k, v in row.items()} + bond = row.get("tipo titulo", "") + maturity = br_date(row.get("data vencimento")) + base = br_date(row.get("data base")) + if not bond or not maturity or not base: + continue + yield ( + bond, + maturity, + base, + to_float(row.get("taxa compra manha")), + to_float(row.get("taxa venda manha")), + to_float(row.get("pu compra manha")), + to_float(row.get("pu venda manha")), + to_float(row.get("pu base manha")), + ) + + +def sync(conn: sqlite3.Connection) -> int: + """Download the full history (a few MB) and upsert it.""" + payload = fetch_bytes(URL, HOST_KEY) + if payload is None: + return 0 + rows = list(parse(decode_text(payload))) + conn.executemany("INSERT OR REPLACE INTO tesouro_prices VALUES (?, ?, ?, ?, ?, ?, ?, ?)", rows) + conn.commit() + return len(rows) diff --git a/agent/src/channels/attempt_progress.py b/agent/src/channels/attempt_progress.py new file mode 100644 index 00000000..dc9daf2c --- /dev/null +++ b/agent/src/channels/attempt_progress.py @@ -0,0 +1,78 @@ +"""Forward attempt status to IM adapters without releasing unchecked answers.""" +from __future__ import annotations + +from typing import Any + +from src.channels.bus.events import InboundMessage, OutboundMessage + + +class AttemptProgress: + """Poll buffered session events with an attempt-scoped replay cursor.""" + + def __init__(self, bus: Any, event_bus: Any, msg: InboundMessage, + session_id: str, attempt_id: str) -> None: + self.bus = bus + self.events = event_bus + self.msg = msg + self.session_id = session_id + self.attempt_id = attempt_id + self.cursor: str | None = None + self.streaming = bool(msg.metadata.get("_wants_stream")) + self.stream_started = False + self.reasoning_started = False + self.last_status = "" + + async def _send(self, content: str, **flags: Any) -> None: + await self.bus.publish_outbound(OutboundMessage( + channel=self.msg.channel, chat_id=self.msg.chat_id, content=content, + metadata={**self.msg.metadata, "_channel_runtime": True, + "session_id": self.session_id, "attempt_id": self.attempt_id, + "_stream_id": self.attempt_id, **flags}, + )) + + async def poll(self) -> None: + """Forward status and opt-in reasoning, excluding answer drafts/results.""" + for event in self.events.replay(self.session_id, self.cursor, replay_all=True): + self.cursor = event.event_id + data = event.data + if data.get("attempt_id") != self.attempt_id: + continue + kind = event.event_type + if kind == "reasoning_delta": + delta = data.get("delta", "") + if delta: + self.reasoning_started = True + await self._send(str(delta), _reasoning_delta=True) + continue + if kind in {"tool_call", "grounding_status", "attempt.completed", "attempt.failed"}: + await self._end_reasoning() + tool = str(data.get("tool") or "tool") + status = { + "attempt.created": "Working on your request…", + "tool_call": f"Using {tool}…", + "tool_heartbeat": f"Still running {tool}…", + "grounding_status": "Checking the figures…", + }.get(kind) + # Arbitrary progress payloads, tool results and text_delta may + # contain unvalidated figures. Only these status labels go out. + if not status or status == self.last_status: + continue + self.last_status = status + hint = kind in {"tool_call", "tool_heartbeat"} + flags = {"_progress": True, "_tool_hint": hint} + if self.streaming: + flags["_stream_delta"] = True + self.stream_started = True + await self._send(status + "\n", **flags) + + async def _end_reasoning(self) -> None: + if self.reasoning_started: + await self._send("", _reasoning_end=True) + self.reasoning_started = False + + async def close(self) -> None: + """Close transient cards on completion, timeout or cancellation.""" + await self._end_reasoning() + if self.stream_started: + await self._send("", _stream_end=True, _resuming=True) + self.stream_started = False diff --git a/agent/src/channels/registry.py b/agent/src/channels/registry.py index 8bca4e2e..07622b20 100644 --- a/agent/src/channels/registry.py +++ b/agent/src/channels/registry.py @@ -21,6 +21,7 @@ logger = logging.getLogger(__name__) _INTERNAL = frozenset( { + "attempt_progress", "base", "bus", "config", diff --git a/agent/src/channels/runtime.py b/agent/src/channels/runtime.py index 4c4d9041..037a00da 100644 --- a/agent/src/channels/runtime.py +++ b/agent/src/channels/runtime.py @@ -192,7 +192,16 @@ class ChannelRuntime: include_shell_tools=False, ) attempt_id = result.get("attempt_id") if isinstance(result, dict) else None - reply = await self._wait_for_reply(session_id, attempt_id) + from src.channels.attempt_progress import AttemptProgress + + events = getattr(self.session_service, "event_bus", None) + progress = (AttemptProgress(self.bus, events, msg, session_id, attempt_id) + if events is not None and attempt_id else None) + try: + reply = await self._wait_for_reply(session_id, attempt_id, progress=progress) + finally: + if progress is not None: + await progress.close() reply_content = reply.content try: from src.scheduled_research.proposals import latest_pending_for_session @@ -233,6 +242,7 @@ class ChannelRuntime: "_channel_runtime": True, "attempt_id": attempt_id, "session_id": session_id, + **{key: msg.metadata[key] for key in ("chat_type", "root_id") if key in msg.metadata}, # QQ (and other platforms) need the originating message id # to reply as a passive message; without it, replies are # treated as active messages and rejected for @@ -355,10 +365,12 @@ class ChannelRuntime: media.append(str(path)) return media - async def _wait_for_reply(self, session_id: str, attempt_id: str | None) -> Message: + async def _wait_for_reply(self, session_id: str, attempt_id: str | None, *, progress: Any = None) -> Message: deadline = time.monotonic() + self.config.reply_timeout_s last_assistant: Message | None = None while time.monotonic() < deadline: + if progress is not None: + await progress.poll() messages = self.session_service.get_messages(session_id, limit=200) for message in reversed(messages): if message.role != "assistant": diff --git a/agent/src/config/env_schema.py b/agent/src/config/env_schema.py index caaf3798..6a366d2a 100644 --- a/agent/src/config/env_schema.py +++ b/agent/src/config/env_schema.py @@ -232,6 +232,16 @@ class DataConfig(_EnvBase): vibe_trading_sec_ftd_files: int = Field(alias="VIBE_TRADING_SEC_FTD_FILES", default=1) vibe_trading_openalex_mailto: str = Field(alias="VIBE_TRADING_OPENALEX_MAILTO", default="") vibe_tw_stock_db: str = Field(alias="VIBE_TW_STOCK_DB", default="") + # Brazilian official data (B3 COTAHIST, CVM, BCB, Tesouro Direto), synced + # by ``vibe-trading br-data sync`` into one SQLite file. Empty = default + # ``~/.vibe-trading/br-data/br_data.db``. + vibe_br_data_db: str = Field(alias="VIBE_BR_DATA_DB", default="") + # First COTAHIST year a sync downloads (B3 publishes from 1986). + vibe_br_cotahist_start_year: int = Field(alias="VIBE_BR_COTAHIST_START_YEAR", default=2015) + # First year of CVM fund daily reports / company filings a sync downloads. + vibe_br_cvm_start_year: int = Field(alias="VIBE_BR_CVM_START_YEAR", default=2022) + # Local time (HH:MM, America/Sao_Paulo) of the daily sync in daemon mode. + vibe_br_sync_time: str = Field(alias="VIBE_BR_SYNC_TIME", default="21:30") vibe_trading_data_cache: EnvBool = Field(alias="VIBE_TRADING_DATA_CACHE", default=False) vibe_trading_data_cache_root: str = Field(alias="VIBE_TRADING_DATA_CACHE_ROOT", default="") aliyun_iqs_api_key: str = Field(alias="ALIYUN_IQS_API_KEY", default="") @@ -259,6 +269,7 @@ class DataConfig(_EnvBase): market_data_order_kr_equity: str = Field(alias="MARKET_DATA_ORDER_KR_EQUITY", default="") market_data_order_ca_equity: str = Field(alias="MARKET_DATA_ORDER_CA_EQUITY", default="") market_data_order_ar_equity: str = Field(alias="MARKET_DATA_ORDER_AR_EQUITY", default="") + market_data_order_br_equity: str = Field(alias="MARKET_DATA_ORDER_BR_EQUITY", default="") market_data_order_uk_equity: str = Field(alias="MARKET_DATA_ORDER_UK_EQUITY", default="") market_data_order_vietnam_equity: str = Field(alias="MARKET_DATA_ORDER_VIETNAM_EQUITY", default="") market_data_order_crypto: str = Field(alias="MARKET_DATA_ORDER_CRYPTO", default="") diff --git a/agent/src/factors/registry.py b/agent/src/factors/registry.py index 5ac8a6ac..ef02794f 100644 --- a/agent/src/factors/registry.py +++ b/agent/src/factors/registry.py @@ -74,13 +74,16 @@ def validate_columns_required(cols: list[str]) -> None: Raises: ValueError: If a column is neither a known price column nor a - ``fund:``-prefixed fundamental column. + ``fund:``-prefixed fundamental or ``val:``-prefixed valuation + column. """ for column in cols: if column in _PRICE_COLS: continue if column.startswith("fund:"): continue + if column.startswith("val:"): + continue raise ValueError(f"unknown panel column: {column}") @@ -108,8 +111,9 @@ class AlphaMeta(BaseModel): """Validate factor panel dependencies. Price columns are fixed, while fundamental fields are an open namespace - under the ``fund:`` prefix. The runtime loader decides whether a - particular fundamental field can be populated. + under the ``fund:`` prefix and daily valuation fields are an open + namespace under the ``val:`` prefix. The runtime loader decides whether + a particular fundamental or valuation field can be populated. Args: v: Declared panel columns. @@ -119,7 +123,7 @@ class AlphaMeta(BaseModel): Raises: ValueError: If any column is outside the price set and the - ``fund:`` namespace. + ``fund:`` and ``val:`` namespaces. """ validate_columns_required(v) return v diff --git a/agent/src/factors/zoo/fundamental/accruals.py b/agent/src/factors/zoo/fundamental/accruals.py new file mode 100644 index 00000000..ecc5ffcd --- /dev/null +++ b/agent/src/factors/zoo/fundamental/accruals.py @@ -0,0 +1,30 @@ +"""Fundamental accruals quality factor (Sloan 1996).""" + +from __future__ import annotations + +import pandas as pd + +from src.factors.base import zscore + +__alpha_meta__ = { + "id": "fund_accruals", + "nickname": "Accruals - inverse (NI-CFO)/TA, Sloan 1996", + "theme": ["quality"], + "formula_latex": r"-\mathrm{zscore}_{x}\left(\frac{\mathrm{NI}-\mathrm{CFO}}{\mathrm{TA}}\right)", + "columns_required": ["fund:accruals"], + "universe": ["equity_us", "equity_cn", "equity_hk"], + "frequency": ["1d"], + "decay_horizon": 252, + "min_warmup_bars": 1, + "notes": ( + "Balance-sheet accrual proxy: high accruals indicate low earnings " + "quality; sign inverted so high score = low accruals. PIT-safe from " + "the fundamental panel; missing statements remain NaN and are " + "excluded from that date's cross-section." + ), +} + + +def compute(panel: dict[str, pd.DataFrame]) -> pd.DataFrame: + """Return cross-sectional z-scored inverse accruals.""" + return -zscore(panel["fund:accruals"]) diff --git a/agent/src/factors/zoo/fundamental/book_to_market.py b/agent/src/factors/zoo/fundamental/book_to_market.py new file mode 100644 index 00000000..dda109ee --- /dev/null +++ b/agent/src/factors/zoo/fundamental/book_to_market.py @@ -0,0 +1,46 @@ +"""Fundamental book-to-market value factor (real Fama-French HML characteristic).""" + +from __future__ import annotations + +import pandas as pd + +from src.factors.base import safe_div +from src.factors.base import zscore + +__alpha_meta__ = { + "id": "fund_book_to_market", + "nickname": "Book-to-market - equity over market cap (FF HML characteristic)", + "theme": ["value"], + "formula_latex": ( + r"\mathrm{zscore}_{x}\left(" + r"\frac{\mathrm{total\_equity}}{\mathrm{close}\times\mathrm{shares\_diluted}}" + r"\right)" + ), + "columns_required": ["close", "fund:total_equity", "fund:shares_diluted"], + "universe": ["equity_us", "equity_cn", "equity_hk"], + "frequency": ["1d"], + "decay_horizon": 252, + "min_warmup_bars": 1, + "notes": ( + "Valuation proxy: close times weighted-average diluted shares is not " + "reported market capitalization; adjusted closes can change the level. " + "Fama-French (1993) HML-style book-to-market proxy built from " + "PIT-safe total equity and shares diluted aligned to the daily close " + "panel; the academic_hml factor in the academic zoo is only a price " + "proxy. High B/M (cheap relative to book value) scores high. " + "market_cap = close x shares_diluted is computed here in the factor " + "file by design: _fundamental_schema.py lines 210-212 deliberately " + "keep price-dependent value factors such as book_to_market out of " + "DERIVED_FIELDS because they need the aligned price panel. Zero or " + "non-finite market cap becomes NaN via safe_div before cross-sectional " + "z-scoring; missing statements remain NaN and are excluded from that " + "date's cross-section." + ), +} + + +def compute(panel: dict[str, pd.DataFrame]) -> pd.DataFrame: + """Return cross-sectional z-scored book-to-market ratio.""" + market_cap = panel["close"] * panel["fund:shares_diluted"] + book_to_market = safe_div(panel["fund:total_equity"], market_cap) + return zscore(book_to_market) diff --git a/agent/src/factors/zoo/fundamental/cash_to_assets.py b/agent/src/factors/zoo/fundamental/cash_to_assets.py new file mode 100644 index 00000000..706593cc --- /dev/null +++ b/agent/src/factors/zoo/fundamental/cash_to_assets.py @@ -0,0 +1,33 @@ +"""Fundamental cash-to-assets balance-sheet strength factor.""" + +from __future__ import annotations + +import pandas as pd + +from src.factors.base import safe_div +from src.factors.base import zscore + +__alpha_meta__ = { + "id": "fund_cash_to_assets", + "nickname": "Cash to assets - balance-sheet cash ratio", + "theme": ["quality"], + "formula_latex": r"\mathrm{zscore}_{x}\left(\frac{\mathrm{cash}}{\mathrm{total\_assets}}\right)", + "columns_required": ["fund:cash", "fund:total_assets"], + "universe": ["equity_us", "equity_cn", "equity_hk"], + "frequency": ["1d"], + "decay_horizon": 252, + "min_warmup_bars": 1, + "notes": ( + "Balance-sheet cash ratio (cash and equivalents / total assets): " + "high cash holdings signal defensive capacity and financial " + "flexibility. Zero or missing total assets become NaN via safe " + "division before cross-sectional z-scoring; PIT-safe from the " + "filed-date-anchored fundamental panel." + ), +} + + +def compute(panel: dict[str, pd.DataFrame]) -> pd.DataFrame: + """Return cross-sectional z-scored cash-to-assets ratio.""" + cash_ratio = safe_div(panel["fund:cash"], panel["fund:total_assets"]) + return zscore(cash_ratio) diff --git a/agent/src/factors/zoo/fundamental/cfo_yield.py b/agent/src/factors/zoo/fundamental/cfo_yield.py new file mode 100644 index 00000000..4058f973 --- /dev/null +++ b/agent/src/factors/zoo/fundamental/cfo_yield.py @@ -0,0 +1,34 @@ +"""Fundamental cash-flow yield quality factor (CFO / total assets).""" + +from __future__ import annotations + +import pandas as pd + +from src.factors.base import safe_div +from src.factors.base import zscore + +__alpha_meta__ = { + "id": "fund_cfo_yield", + "nickname": "CFO yield - operating cash flow over total assets", + "theme": ["quality"], + "formula_latex": r"\mathrm{zscore}_{x}\left(\frac{\mathrm{CFO}}{\mathrm{total\_assets}}\right)", + "columns_required": ["fund:cfo", "fund:total_assets"], + "universe": ["equity_us", "equity_cn", "equity_hk"], + "frequency": ["1d"], + "decay_horizon": 252, + "min_warmup_bars": 1, + "notes": ( + "Valuation proxy: close times weighted-average diluted shares is not " + "reported market capitalization; adjusted closes can change the level. " + "Cash-flow return on assets: the cash-backed counterpart of ROA, so " + "accrual-heavy earnings score lower here. Zero or missing total " + "assets become NaN via safe division before cross-sectional " + "z-scoring; PIT-safe from the filed-date-anchored fundamental panel." + ), +} + + +def compute(panel: dict[str, pd.DataFrame]) -> pd.DataFrame: + """Return cross-sectional z-scored CFO-to-assets yield.""" + cfo_yield = safe_div(panel["fund:cfo"], panel["fund:total_assets"]) + return zscore(cfo_yield) diff --git a/agent/src/factors/zoo/fundamental/dividend_yield.py b/agent/src/factors/zoo/fundamental/dividend_yield.py new file mode 100644 index 00000000..970134fe --- /dev/null +++ b/agent/src/factors/zoo/fundamental/dividend_yield.py @@ -0,0 +1,34 @@ +"""Fundamental dividend-yield (TTM) value factor (CN daily market data).""" + +from __future__ import annotations + +import pandas as pd + +from src.factors.base import zscore + +__alpha_meta__ = { + "id": "fund_dividend_yield", + "nickname": "Dividend yield TTM - dv_ttm (daily market data)", + "theme": ["value"], + "formula_latex": r"\mathrm{zscore}_{x}(\mathrm{DY_{TTM}})", + "columns_required": ["val:dv_ttm"], + "universe": ["equity_cn"], + "frequency": ["1d"], + "decay_horizon": 252, + "min_warmup_bars": 1, + "notes": ( + "Trailing dividend yield from the val: namespace (tushare " + "daily_basic dv_ttm, CN A-share only), reported in percent units — " + "cross-sectional z-scoring is scale-invariant, so no unit " + "conversion is applied. High dividend yield scores high. Keyed by " + "trade_date, so PIT-safe by construction with no filing lag. " + "Missing dates or non-CN symbols stay NaN and are excluded from " + "that date's cross-section. Lineage: JoinQuant style-pro divyild; " + "akshare/Eastmoney dv_ttm convention." + ), +} + + +def compute(panel: dict[str, pd.DataFrame]) -> pd.DataFrame: + """Return cross-sectional z-scored trailing dividend yield.""" + return zscore(panel["val:dv_ttm"]) diff --git a/agent/src/factors/zoo/fundamental/earnings_growth.py b/agent/src/factors/zoo/fundamental/earnings_growth.py new file mode 100644 index 00000000..b0188db3 --- /dev/null +++ b/agent/src/factors/zoo/fundamental/earnings_growth.py @@ -0,0 +1,40 @@ +"""Fundamental earnings-growth factor (sign-safe YoY net-income growth).""" + +from __future__ import annotations + +import pandas as pd + +from src.factors.base import zscore + +__alpha_meta__ = { + "id": "fund_earnings_growth", + "nickname": "Earnings growth - sign-safe YoY net income expansion", + "theme": ["growth"], + "formula_latex": ( + r"\mathrm{zscore}_{x}\left(" + r"\frac{\Delta\,\mathrm{net\_income}}" + r"{\left|\mathrm{lag}_{365d}(\mathrm{net\_income})\right|}" + r"\right)" + ), + "columns_required": ["fund:earnings_growth"], + "universe": ["equity_us", "equity_cn", "equity_hk"], + "frequency": ["1d"], + "decay_horizon": 252, + "min_warmup_bars": 1, + "notes": ( + "PIT-safe earnings growth from the fundamental panel's derived field " + "``earnings_growth``. The schema-side formula is sign-safe: " + "delta net_income / abs(lag_365d net_income), so a negative year-ago " + "base does not flip the growth sign (the proposal #6 footnote " + "resolution; the alternative NaN-on-negative-base was not chosen). " + "Calendar-day lag per the #1564 lesson. High earnings growth scores " + "high, matching the JoinQuant net_profit_growth_rate convention. " + "Missing statements or a zero/missing year-ago base remain NaN and " + "are excluded from that date's cross-section." + ), +} + + +def compute(panel: dict[str, pd.DataFrame]) -> pd.DataFrame: + """Return cross-sectional z-scored earnings growth.""" + return zscore(panel["fund:earnings_growth"]) diff --git a/agent/src/factors/zoo/fundamental/fcf_yield.py b/agent/src/factors/zoo/fundamental/fcf_yield.py new file mode 100644 index 00000000..a749e083 --- /dev/null +++ b/agent/src/factors/zoo/fundamental/fcf_yield.py @@ -0,0 +1,46 @@ +"""Fundamental free-cash-flow yield value factor (Lakonishok et al. 1994).""" + +from __future__ import annotations + +import pandas as pd + +from src.factors.base import safe_div +from src.factors.base import zscore + +__alpha_meta__ = { + "id": "fund_fcf_yield", + "nickname": "FCF yield - (CFO - capex) over market cap", + "theme": ["value"], + "formula_latex": ( + r"\mathrm{zscore}_{x}\left(" + r"\frac{\mathrm{CFO}-\mathrm{capex}}" + r"{\mathrm{close}\times\mathrm{shares\_diluted}}" + r"\right)" + ), + "columns_required": ["close", "fund:cfo", "fund:capex", "fund:shares_diluted"], + "universe": ["equity_us", "equity_cn", "equity_hk"], + "frequency": ["1d"], + "decay_horizon": 252, + "min_warmup_bars": 1, + "notes": ( + "Valuation proxy: close times weighted-average diluted shares is not " + "reported market capitalization; adjusted closes can change the level. " + "PIT-safe free-cash-flow yield: FCF = CFO - capex, scaled by market " + "cap (close x shares_diluted) computed in the factor file per the " + "_fundamental_schema.py price-dependent convention. Sign assumption: " + "capex is reported as a positive magnitude (SEC concept " + "PaymentsToAcquirePropertyPlantAndEquipment), so subtracting it from " + "CFO yields FCF directly. High FCF yield (cheap on cash generation, " + "cf. Lakonishok-Shleifer-Vishny 1994 cash-flow yield value factors) " + "scores high. Zero/non-finite market cap becomes NaN via safe_div " + "before cross-sectional z-scoring; missing statements remain NaN and " + "are excluded from that date's cross-section." + ), +} + + +def compute(panel: dict[str, pd.DataFrame]) -> pd.DataFrame: + """Return cross-sectional z-scored free-cash-flow yield.""" + fcf = panel["fund:cfo"] - panel["fund:capex"] + market_cap = panel["close"] * panel["fund:shares_diluted"] + return zscore(safe_div(fcf, market_cap)) diff --git a/agent/src/factors/zoo/fundamental/leverage.py b/agent/src/factors/zoo/fundamental/leverage.py new file mode 100644 index 00000000..32c0fb52 --- /dev/null +++ b/agent/src/factors/zoo/fundamental/leverage.py @@ -0,0 +1,31 @@ +"""Fundamental leverage factor (low leverage scores high).""" + +from __future__ import annotations + +import pandas as pd + +from src.factors.base import zscore + +__alpha_meta__ = { + "id": "fund_leverage", + "nickname": "Leverage - inverse debt/equity (low leverage scores high)", + "theme": ["leverage"], + "formula_latex": r"-\mathrm{zscore}_{x}\left(\frac{\mathrm{total\_debt}}{\mathrm{total\_equity}}\right)", + "columns_required": ["fund:leverage"], + "universe": ["equity_us", "equity_cn", "equity_hk"], + "frequency": ["1d"], + "decay_horizon": 252, + "min_warmup_bars": 1, + "notes": ( + "PIT-safe market leverage (total debt / total equity) from the " + "fundamental panel; sign inverted so low leverage scores high — " + "heavily levered firms carry distress risk and lower expected " + "returns. Zero-equity or missing statements already yield NaN in the " + "derived field and are excluded from that date's cross-section." + ), +} + + +def compute(panel: dict[str, pd.DataFrame]) -> pd.DataFrame: + """Return negative cross-sectional z-scored leverage.""" + return -zscore(panel["fund:leverage"]) diff --git a/agent/src/factors/zoo/fundamental/operating_margin.py b/agent/src/factors/zoo/fundamental/operating_margin.py new file mode 100644 index 00000000..9008ab3f --- /dev/null +++ b/agent/src/factors/zoo/fundamental/operating_margin.py @@ -0,0 +1,33 @@ +"""Fundamental operating-margin quality factor.""" + +from __future__ import annotations + +import pandas as pd + +from src.factors.base import safe_div +from src.factors.base import zscore + +__alpha_meta__ = { + "id": "fund_operating_margin", + "nickname": "Operating margin - operating income over revenue", + "theme": ["quality"], + "formula_latex": r"\mathrm{zscore}_{x}\left(\frac{\mathrm{operating\_income}}{\mathrm{revenue}}\right)", + "columns_required": ["fund:operating_income", "fund:revenue"], + "universe": ["equity_us", "equity_cn", "equity_hk"], + "frequency": ["1d"], + "decay_horizon": 252, + "min_warmup_bars": 1, + "notes": ( + "Operating profitability (operating income / revenue), " + "cross-sectionally z-scored per date. Captures pricing power and cost " + "efficiency before financing and tax effects. Zero-revenue or missing " + "statements become NaN via safe division and are excluded from that " + "date's cross-section; PIT-safe from the fundamental panel." + ), +} + + +def compute(panel: dict[str, pd.DataFrame]) -> pd.DataFrame: + """Return cross-sectional z-scored operating margin.""" + margin = safe_div(panel["fund:operating_income"], panel["fund:revenue"]) + return zscore(margin) diff --git a/agent/src/factors/zoo/fundamental/pb.py b/agent/src/factors/zoo/fundamental/pb.py new file mode 100644 index 00000000..ab2405ad --- /dev/null +++ b/agent/src/factors/zoo/fundamental/pb.py @@ -0,0 +1,35 @@ +"""Fundamental price-to-book value factor (CN daily market data).""" + +from __future__ import annotations + +import pandas as pd + +from src.factors.base import zscore + +__alpha_meta__ = { + "id": "fund_pb", + "nickname": "P/B - inverse price-to-book (daily market data)", + "theme": ["value"], + "formula_latex": r"-\mathrm{zscore}_{x}(\mathrm{PB})", + "columns_required": ["val:pb"], + "universe": ["equity_cn"], + "frequency": ["1d"], + "decay_horizon": 252, + "min_warmup_bars": 1, + "notes": ( + "Nonpositive multiples are excluded rather than ranked as cheap. " + "Inverse price-to-book from the val: namespace (tushare daily_basic " + "pb, CN A-share only): low PB (cheap relative to book value) scores " + "high, the daily-market-data counterpart of fund_book_to_market. " + "Keyed by trade_date, so PIT-safe by construction with no filing " + "lag. Missing dates or non-CN symbols stay NaN and are excluded " + "from that date's cross-section. Lineage: JoinQuant style " + "book_to_price_ratio; akshare/Eastmoney pb convention." + ), +} + + +def compute(panel: dict[str, pd.DataFrame]) -> pd.DataFrame: + """Return cross-sectional z-scored inverse price-to-book.""" + values = panel["val:pb"] + return -zscore(values.where(values > 0)) diff --git a/agent/src/factors/zoo/fundamental/ps_ttm.py b/agent/src/factors/zoo/fundamental/ps_ttm.py new file mode 100644 index 00000000..6613ab3c --- /dev/null +++ b/agent/src/factors/zoo/fundamental/ps_ttm.py @@ -0,0 +1,35 @@ +"""Fundamental price-to-sales (TTM) value factor (CN daily market data).""" + +from __future__ import annotations + +import pandas as pd + +from src.factors.base import zscore + +__alpha_meta__ = { + "id": "fund_ps_ttm", + "nickname": "P/S TTM - inverse price-to-sales (daily market data)", + "theme": ["value"], + "formula_latex": r"-\mathrm{zscore}_{x}(\mathrm{PS_{TTM}})", + "columns_required": ["val:ps_ttm"], + "universe": ["equity_cn"], + "frequency": ["1d"], + "decay_horizon": 252, + "min_warmup_bars": 1, + "notes": ( + "Nonpositive multiples are excluded rather than ranked as cheap. " + "Inverse trailing price-to-sales from the val: namespace (tushare " + "daily_basic ps_ttm, CN A-share only): low PS scores high. PS stays " + "informative for loss-making names where earnings yield fails. " + "Keyed by trade_date, so PIT-safe by construction with no filing " + "lag. Missing dates or non-CN symbols stay NaN and are excluded " + "from that date's cross-section. Lineage: JoinQuant style-pro " + "sales-to-price; akshare/Eastmoney ps_ttm convention." + ), +} + + +def compute(panel: dict[str, pd.DataFrame]) -> pd.DataFrame: + """Return cross-sectional z-scored inverse price-to-sales (TTM).""" + values = panel["val:ps_ttm"] + return -zscore(values.where(values > 0)) diff --git a/agent/src/factors/zoo/fundamental/revenue_growth.py b/agent/src/factors/zoo/fundamental/revenue_growth.py new file mode 100644 index 00000000..f47d1e17 --- /dev/null +++ b/agent/src/factors/zoo/fundamental/revenue_growth.py @@ -0,0 +1,37 @@ +"""Fundamental revenue-growth factor (YoY top-line growth).""" + +from __future__ import annotations + +import pandas as pd + +from src.factors.base import zscore + +__alpha_meta__ = { + "id": "fund_revenue_growth", + "nickname": "Revenue growth - YoY top-line expansion", + "theme": ["growth"], + "formula_latex": ( + r"\mathrm{zscore}_{x}\left(" + r"\frac{\mathrm{revenue}}{\mathrm{lag}_{365d}(\mathrm{revenue})}-1" + r"\right)" + ), + "columns_required": ["fund:revenue_growth"], + "universe": ["equity_us", "equity_cn", "equity_hk"], + "frequency": ["1d"], + "decay_horizon": 252, + "min_warmup_bars": 1, + "notes": ( + "PIT-safe revenue growth from the fundamental panel's derived field " + "``revenue_growth`` (revenue / lag_365d(revenue) - 1, calendar-day " + "lag per the #1564 lesson rather than a row-wise shift). High " + "top-line growth scores high, matching the JoinQuant " + "operating_revenue_growth_rate convention. Missing statements or a " + "missing year-ago base remain NaN and are excluded from that date's " + "cross-section." + ), +} + + +def compute(panel: dict[str, pd.DataFrame]) -> pd.DataFrame: + """Return cross-sectional z-scored revenue growth.""" + return zscore(panel["fund:revenue_growth"]) diff --git a/agent/src/factors/zoo/fundamental/roa.py b/agent/src/factors/zoo/fundamental/roa.py new file mode 100644 index 00000000..0cf77071 --- /dev/null +++ b/agent/src/factors/zoo/fundamental/roa.py @@ -0,0 +1,30 @@ +"""Fundamental ROA quality factor.""" + +from __future__ import annotations + +import pandas as pd + +from src.factors.base import zscore + +__alpha_meta__ = { + "id": "fund_roa", + "nickname": "ROA - return on assets (PIT-safe fundamentals)", + "theme": ["quality"], + "formula_latex": r"\mathrm{zscore}_{x}\left(\frac{\mathrm{net\_income}}{\mathrm{total\_assets}}\right)", + "columns_required": ["fund:roa"], + "universe": ["equity_us", "equity_cn", "equity_hk"], + "frequency": ["1d"], + "decay_horizon": 252, + "min_warmup_bars": 1, + "notes": ( + "PIT-safe return on assets (net income / total assets) from the " + "fundamental panel; cross-sectional z-score per date. Higher ROA means " + "a more profitable asset base and scores higher. Missing statements " + "remain NaN and are excluded from that date's cross-section." + ), +} + + +def compute(panel: dict[str, pd.DataFrame]) -> pd.DataFrame: + """Return cross-sectional z-scored ROA.""" + return zscore(panel["fund:roa"]) diff --git a/agent/src/live/enforcement.py b/agent/src/live/enforcement.py index 73624e55..5f1031c8 100644 --- a/agent/src/live/enforcement.py +++ b/agent/src/live/enforcement.py @@ -537,8 +537,18 @@ def check_mandate( detail=f"{intent.instrument_type.value} not in allowed_instruments", ) - # 3. Asset-class allowance (universe bucket). OPTION has no bucket and is - # governed by allowed_instruments alone. An explicit intent.asset_class + # Share-notional checks cannot bound an option's contract multiplier or + # maximum loss. Even an explicit instrument allowance cannot enable this + # unsupported path; both MCP and SDK orders pass through this check. + if intent.instrument_type == InstrumentType.OPTION: + return _breach( + broker=broker, remote_tool=remote_tool, intent=intent, + kind=BREACH_KIND_INSTRUMENT, limit="options_unsupported", + limit_value=0.0, attempted_value=0.0, + detail="options require an options-specific risk gate; order placement is unsupported", + ) + + # 3. Asset-class allowance (universe bucket). An explicit intent.asset_class # (multi-market connectors: US/HK/CN equities) wins over the instrument # default so the gate buckets HK/A-share correctly. asset_class = intent.asset_class or _INSTRUMENT_ASSET_CLASS.get(intent.instrument_type) diff --git a/agent/src/live/mandate/model.py b/agent/src/live/mandate/model.py index 6c598406..dc915c44 100644 --- a/agent/src/live/mandate/model.py +++ b/agent/src/live/mandate/model.py @@ -18,9 +18,10 @@ class InstrumentType(str, Enum): """Instrument classes the broker may report or accept. ``CFD`` covers margin contracts-for-difference that are not spot forex - pairs (MT5 metals like XAUUSD, index/energy/crypto CFDs). Like ``OPTION`` - it has no universe asset-class bucket and is admitted only when the user's - mandate explicitly lists ``"cfd"`` in ``allowed_instruments``. + pairs (MT5 metals like XAUUSD, index/energy/crypto CFDs). It has no universe + asset-class bucket and requires ``"cfd"`` in ``allowed_instruments``. + ``OPTION`` is representable for classification, but order enforcement + refuses it until an options-specific risk gate is available. """ EQUITY = "equity" diff --git a/agent/src/market_data.py b/agent/src/market_data.py index dad25ff7..26875c83 100644 --- a/agent/src/market_data.py +++ b/agent/src/market_data.py @@ -30,6 +30,8 @@ _SOURCE_PATTERNS = [ (re.compile(r"^[A-Z0-9&.\-]+\.(TO|V)$", re.I), "yahoo"), # Argentina: BYMA equities and CEDEARs use Yahoo's canonical .BA suffix. (re.compile(r"^[A-Z0-9&.\-]+\.BA$", re.I), "yahoo"), + # Brazil: B3 equities, FIIs, ETFs and BDRs use Yahoo's canonical .SA suffix. + (re.compile(r"^[A-Z0-9]+\.SA$", re.I), "yahoo"), # UK: London Stock Exchange (VOD.L, SHEL.L). Yahoo serves the suffix # verbatim; without this they fell through to the tushare default and were # routed to China-market loaders that cannot resolve them. diff --git a/agent/src/providers/openai_codex.py b/agent/src/providers/openai_codex.py index aef1e60c..9826a3cf 100644 --- a/agent/src/providers/openai_codex.py +++ b/agent/src/providers/openai_codex.py @@ -558,6 +558,11 @@ def _convert_tools(tools: list[dict[str, Any]] | None) -> list[dict[str, Any]]: "name": name, "description": fn.get("description") or "", "parameters": params if isinstance(params, dict) else {}, + # The Codex Responses endpoint treats a tool without an explicit + # ``strict`` as strict, which makes the model emit every property + # (``""`` for an optional string) instead of omitting it. Tool + # schemas here are plain JSON Schema with ``required`` honoured. + "strict": False, } ) return converted diff --git a/agent/src/quantlib/risk.py b/agent/src/quantlib/risk.py index 620e4620..7b7adba3 100644 --- a/agent/src/quantlib/risk.py +++ b/agent/src/quantlib/risk.py @@ -452,7 +452,8 @@ def max_drawdown_analysis(equity: pd.Series | np.ndarray | Sequence[float]) -> d max_drawdown (float): Worst decline as a **positive** fraction, so 0.25 means the curve fell 25% below its running peak. 0.0 when the curve never declines. - peak_date: Index label of the running peak preceding the trough. + peak_date: Index label of the most recent running peak preceding + the trough; revisiting that peak resets the underwater period. trough_date: Index label of the deepest point. recovery_date: Index label of the first observation at or above the peak value, searched from the trough onward; None if the series @@ -491,7 +492,8 @@ def max_drawdown_analysis(equity: pd.Series | np.ndarray | Sequence[float]) -> d running_peak = np.maximum.accumulate(values) drawdown = values / running_peak - 1.0 # <= 0 everywhere trough_pos = int(np.argmin(drawdown)) - peak_pos = int(np.argmax(values[: trough_pos + 1])) + # A revisit closes the previous drawdown: start at the latest peak. + peak_pos = int(np.flatnonzero(values[: trough_pos + 1] == running_peak[trough_pos])[-1]) # A flat/rising curve has drawdown 0 at the trough, so the trough itself # already satisfies "back at the peak" and recovery is instantaneous. diff --git a/agent/src/quantlib/valuation/threestatement.py b/agent/src/quantlib/valuation/threestatement.py index faf78cd2..686840cb 100644 --- a/agent/src/quantlib/valuation/threestatement.py +++ b/agent/src/quantlib/valuation/threestatement.py @@ -521,6 +521,7 @@ def _project_period( minimum_cash: float, circularity_tolerance: float, max_circularity_iterations: int, + balance_tolerance: float = BALANCE_TOLERANCE_REL, ) -> PeriodResult: """Project one period, solving the interest/revolver circularity by iteration. @@ -650,7 +651,7 @@ def _project_period( paid_in_capital=prior_bs.paid_in_capital, retained_earnings=prior_bs.retained_earnings + net_income - dividends_paid, ) - check_balance_sheet(balance_sheet) + check_balance_sheet(balance_sheet, balance_tolerance) return PeriodResult( period=period, @@ -751,10 +752,10 @@ def project_three_statement( minimum_cash=float(drivers["minimum_cash"][index]), circularity_tolerance=circularity_tolerance, max_circularity_iterations=max_circularity_iterations, + balance_tolerance=balance_tolerance, ) - # Defensive re-check with the caller's tolerance: _project_period already - # checked with the module default, but a caller-widened balance_tolerance - # must not be allowed to let a period through unchecked at its own bar. + # Defensive re-check: the inner check already enforces the + # same tolerance, so this second pass is a safety net for future edits. check_balance_sheet(result.balance_sheet, balance_tolerance) results.append(result) prior_bs = result.balance_sheet diff --git a/agent/src/skills/risk-analysis/SKILL.md b/agent/src/skills/risk-analysis/SKILL.md index 223f1ced..c97801ed 100644 --- a/agent/src/skills/risk-analysis/SKILL.md +++ b/agent/src/skills/risk-analysis/SKILL.md @@ -110,7 +110,7 @@ dd["recovered"] # False when the series ends still underwater Full return keys: `max_drawdown`, `peak_date`, `trough_date`, `recovery_date`, `recovered`, `peak_to_trough_periods`, `trough_to_recovery_periods`, `underwater_days`, `recovery_days`. -- Recovery means reaching the **peak** value again, not merely bouncing off the trough; `recovery_date` is None and `recovered` is False if it never happens. +- Recovery means reaching the **peak** value again, not merely bouncing off the trough; `recovery_date` is None and `recovered` is False if it never happens. Revisiting the peak starts a new episode: its `peak_date` and underwater duration exclude an earlier, already recovered drawdown. - `underwater_days` / `recovery_days` are calendar days and require a `DatetimeIndex`; on any other index they come back None and you should use the `*_periods` counts, which are always populated. - Non-positive equity raises — a drawdown *ratio* is undefined at or below zero. Rebase a signed PnL series to a positive net value first. diff --git a/agent/src/strategy_discovery/evidence_harness.py b/agent/src/strategy_discovery/evidence_harness.py index 5f8bf106..28283154 100644 --- a/agent/src/strategy_discovery/evidence_harness.py +++ b/agent/src/strategy_discovery/evidence_harness.py @@ -4,10 +4,9 @@ Builds on the artifact readers in :mod:`run_artifacts`; this module owns the regime labeling and the per-regime metrics. Conventions: * Trade counting — engine ``trades.csv`` artifacts hold TWO rows per round - trip (entry row ``pnl=0.0`` + exit row with realized pnl); only exit rows - are counted, so a round trip contributes exactly one trade. Legacy - one-row-per-trade artifacts are detected via the marker column and keep - counting every row (see :func:`run_artifacts.read_trade_activity`). + trip (``event=entry`` and ``event=exit``); only exits are counted, including + zero-PnL exits. Unmarked files keep their legacy PnL-based interpretation + (see :func:`run_artifacts.read_trade_activity`). * Regime labeling — each bar is labeled by the rolling cumulative benchmark return over the trailing ``benchmark_window`` bars ending at that bar (``<= bear_threshold`` → bear_market, ``>= bull_threshold`` → bull_market, diff --git a/agent/src/strategy_discovery/run_artifacts.py b/agent/src/strategy_discovery/run_artifacts.py index f5076ed9..962b8e5b 100644 --- a/agent/src/strategy_discovery/run_artifacts.py +++ b/agent/src/strategy_discovery/run_artifacts.py @@ -7,23 +7,18 @@ Real engine schema (``backtest/engines/base.py::_write_artifacts``, the writer every reproducible run goes through): * ``trades.csv`` — columns ``timestamp, code, side, price, qty, reason, - pnl, holding_days, return_pct`` with TWO rows per round trip: an entry - row (``reason="signal"``, ``pnl=0.0``, ``holding_days=0``) followed by + pnl, holding_days, holding_bars, return_pct, event`` with TWO rows per + round trip: an entry row (``reason="signal"``, ``pnl=0.0``, ``holding_days=0``) followed by the exit row carrying the realized ``pnl``. * ``equity.csv`` — date column ``timestamp`` (the engine's index name) plus ``ret, equity, drawdown, benchmark_equity, active_ret``. -Entry/exit convention. The engine writes no dedicated marker column, and -``reason`` is not one either — exits can also carry ``reason="signal"`` -(the base engine closes positions on signal). This reader therefore -mirrors the repo's own artifact-reload convention in -``backtest/validation.py``: a row is a closed-trade (exit) row iff its -``pnl`` parses to a nonzero finite value, while ``pnl == 0.0`` rows are -entry rows. Files with no zero-pnl rows at all are treated as legacy -one-row-per-trade artifacts where every dated row is a meaningful closed -trade. Documented limitation (shared with ``validation.py``): an -exact-break-even exit (``pnl == 0.0``) is indistinguishable from an entry -row and is not counted as a round trip. +Entry/exit convention. New BaseEngine artifacts carry an explicit ``event`` +column (``entry`` / ``exit``), so break-even exits count as completed trades. +Unknown or blank event values are skipped, never inferred from PnL. Unmarked +files retain the legacy convention: finite nonzero PnL marks an exit, zero +marks an entry. Without any zero-PnL rows, every dated row counts as a legacy +closed trade. Break-even exits in unmarked files remain ambiguous. Legacy artifacts keep parsing through column aliases (``date`` for ``timestamp``, ``benchmark`` for ``benchmark_equity``). Unusable files — @@ -136,14 +131,14 @@ def read_trade_activity(path: Path) -> tuple[list[date], int | None] | None: Returns ``(exit_dates, max_concurrent)``: * ``exit_dates`` — one date per closed round trip, sorted. Engine - artifacts hold two rows per round trip (entry row ``pnl=0.0`` plus - exit row with realized pnl); only exit rows count, so a round trip + artifacts hold two rows per round trip (``event=entry`` and + ``event=exit``); only exit rows count, so a round trip is counted once, never twice. Legacy one-row-per-trade artifacts (no zero-pnl marker rows) count every dated row. * ``max_concurrent`` — peak simultaneously open positions derived from the entry/exit pairs (FIFO per code, matching the engine's one-position-per-symbol book), or ``None`` when the file lacks the - zero-pnl entry marker and concurrency is undetectable. + entry marker and concurrency is undetectable. Returns ``None`` when the file is unusable: no date column, decode / OS / CSV errors, or no usable content. Rows with unparseable dates @@ -165,7 +160,8 @@ def read_trade_activity(path: Path) -> tuple[list[date], int | None] | None: pnl_column = _find_column(fieldnames, _PNL_ALIASES) code_column = _find_column(fieldnames, _CODE_ALIASES) - rows: list[tuple[date, str, float | None]] = [] + event_column = _find_column(fieldnames, ("event",)) + rows: list[tuple[date, str, float | None, str]] = [] for record in reader: trade_date = parse_iso_date(record.get(date_column)) if trade_date is None: @@ -180,7 +176,14 @@ def read_trade_activity(path: Path) -> tuple[list[date], int | None] | None: if code_column is not None else "" ) - rows.append((trade_date, code, pnl)) + event = ( + str(record.get(event_column) or "") + if event_column is not None + else "entry" if pnl == 0.0 else "exit" + ) + if event_column is not None and event not in {"entry", "exit"}: + continue + rows.append((trade_date, code, pnl, event)) except _UNREADABLE_ERRORS as exc: logger.warning( "trades.csv at %s is unreadable (%s); treated as unusable", path, exc @@ -190,32 +193,28 @@ def read_trade_activity(path: Path) -> tuple[list[date], int | None] | None: if not rows: return [], None - # The engine writes entry rows with pnl=0.0 and exit rows with realized - # pnl (backtest/engines/base.py::_write_artifacts); backtest/validation.py - # reloads the same artifacts with `pnl != 0` as the exit-row filter. When - # no marker row exists, every dated row is a closed trade (legacy - # one-row-per-trade artifacts) and concurrency is undetectable. - has_entry_marker = any(pnl is not None and pnl == 0.0 for _, _, pnl in rows) - if not has_entry_marker: - return sorted(trade_date for trade_date, _, _ in rows), None + # Without explicit events, preserve the legacy one-row-per-trade rule. + has_entry_marker = any(event == "entry" for _, _, _, event in rows) + if event_column is None and not has_entry_marker: + return sorted(trade_date for trade_date, _, _, _ in rows), None exit_dates: list[date] = [] open_entries: dict[str, list[date]] = {} intervals: list[tuple[date, date]] = [] - for trade_date, code, pnl in rows: - if pnl is None: - continue # unclassifiable without a usable pnl marker - if pnl != 0.0: + for trade_date, code, pnl, event in rows: + if event_column is None and pnl is None: + continue # unclassifiable without a usable legacy pnl marker + if event == "exit": # Exit row: one closed round trip. exit_dates.append(trade_date) pending = open_entries.get(code) if pending: intervals.append((pending.pop(0), trade_date)) else: - # Entry row (engine marker pnl == 0.0). + # Explicit entry or legacy zero-PnL entry marker. open_entries.setdefault(code, []).append(trade_date) exit_dates.sort() - return exit_dates, _max_concurrent_positions(intervals) + return exit_dates, _max_concurrent_positions(intervals) if has_entry_marker else None def read_trade_dates(path: Path) -> list[date] | None: diff --git a/agent/src/tools/alpha_bench_tool.py b/agent/src/tools/alpha_bench_tool.py index 784082c7..0f060687 100644 --- a/agent/src/tools/alpha_bench_tool.py +++ b/agent/src/tools/alpha_bench_tool.py @@ -699,6 +699,64 @@ def _validate_bench_data_requirements(registry: Any, alpha_ids: list[str], unive return columns +def _val_columns_required(registry: Any, alpha_ids: list[str], panel: dict) -> list[str]: + """Collect declared ``val:*`` columns missing from the panel. + + Args: + registry: Registry holding the selected factors. + alpha_ids: Factors to evaluate. + panel: Observed price panel. + + Returns: + Sorted ``val:*`` column names not already present in the panel. + """ + return sorted({column for aid in alpha_ids + for column in (registry.get(aid).meta or {}).get("columns_required", []) + if column.startswith("val:") and column not in panel}) + + +def _attach_valuation_columns( + enriched: dict, panel: dict, val_columns: list[str], start: str, end: str +) -> dict: + """Attach ``val:*`` columns for a CN universe from tushare ``daily_basic``. + + Daily market data keyed by ``trade_date`` — PIT-safe by construction, no + filing lag. Frames are masked to the price panel's own cross-section so a + date whose close was membership-masked carries no valuation either. + + Args: + enriched: Panel copy being extended. + panel: Observed price panel (source of the cross-section mask). + val_columns: Declared ``val:*`` columns to attach. + start: Period start (YYYY-MM-DD). + end: Period end (YYYY-MM-DD). + + Returns: + The enriched panel. + + Raises: + ValueError: If a declared field is outside the daily_basic whitelist. + RuntimeError: If fewer than two stocks have data for a field. + """ + from backtest.loaders.valuation_loader import load_valuation_panel + + close = panel["close"] + fields = [column.removeprefix("val:") for column in val_columns] + loaded = load_valuation_panel( + symbols=list(close.columns), fields=fields, start=start, end=end, index=close.index + ) + for column, field in zip(val_columns, fields): + frame = loaded.get(field) + if frame is None or frame.notna().any().sum() < 2: + raise RuntimeError( + f"tushare daily_basic field {field!r} has data for fewer than two " + "stocks. Check TUSHARE_TOKEN permissions and CN coverage for this period." + ) + enriched[column] = frame.reindex(index=close.index, columns=close.columns).where(close.notna()) + enriched["_meta"] = {**enriched.get("_meta", {}), "valuation_source": "tushare_daily_basic"} + return enriched + + def _prepare_bench_panel(registry: Any, alpha_ids: list[str], panel: dict, universe: str, period: str) -> dict: """Load only the declared PIT fundamental inputs for the selected factors. @@ -706,32 +764,42 @@ def _prepare_bench_panel(registry: Any, alpha_ids: list[str], panel: dict, unive registry: Registry holding the factors being evaluated. alpha_ids: Selected factor identifiers. panel: Observed price panel; never mutated or filled with invented values. - universe: Stock universe; SEC fundamentals currently cover US stocks. + universe: Stock universe; SEC fundamentals currently cover US stocks, + ``val:*`` daily valuation columns cover the CN universe only — + other universes skip them and declaring alphas SkipAlpha. period: Requested historical period. Returns: A panel including the required fundamental columns. Raises: - ValueError: If SEC inputs are requested for a non-US universe. + ValueError: If SEC inputs are requested for a non-US universe, or a + ``val:`` field is outside the daily_basic whitelist. RuntimeError: If fewer than two stocks have the required financial inputs. """ columns = [column for column in _validate_bench_data_requirements(registry, alpha_ids, universe) if column not in panel] - if not columns: + val_columns = _val_columns_required(registry, alpha_ids, panel) if universe == "csi300" else [] + if not columns and not val_columns: return panel - from backtest.loaders.fundamentals_loader import load_fundamental_panel start, end = _parse_period(period) close = panel["close"] - fields = [column.removeprefix("fund:") for column in columns] - loaded = load_fundamental_panel(symbols=list(close.columns), fields=fields, start=start, end=end, pit=True, index=close.index) enriched = dict(panel) - for column, field in zip(columns, fields): - frame = loaded.get(field) - if frame is None or frame.notna().any().sum() < 2: - raise RuntimeError(f"SEC field {field!r} has data for fewer than two stocks. Check SEC connectivity and filing coverage for this period.") - enriched[column] = frame.reindex(index=close.index, columns=close.columns) - enriched["_meta"] = {**panel.get("_meta", {}), "fundamental_source": "sec", "fundamental_pit": True} + if columns: + from backtest.loaders.fundamentals_loader import load_fundamental_panel + + fields = [column.removeprefix("fund:") for column in columns] + loaded = load_fundamental_panel(symbols=list(close.columns), fields=fields, start=start, end=end, pit=True, index=close.index) + for column, field in zip(columns, fields): + frame = loaded.get(field) + if frame is None or frame.notna().any().sum() < 2: + raise RuntimeError(f"SEC field {field!r} has data for fewer than two stocks. Check SEC connectivity and filing coverage for this period.") + # Date-only SEC filings can arrive after close(t). IC uses them + # on t+1; the engine's separate next-bar execution is unchanged. + enriched[column] = frame.reindex(index=close.index, columns=close.columns).shift(1) + enriched["_meta"] = {**enriched.get("_meta", {}), "fundamental_source": "sec", "fundamental_pit": True, "fundamental_pit_lag_bars": 1} + if val_columns: + enriched = _attach_valuation_columns(enriched, panel, val_columns, start, end) return enriched diff --git a/agent/src/tools/br_market_data_tool.py b/agent/src/tools/br_market_data_tool.py new file mode 100644 index 00000000..05ca2b93 --- /dev/null +++ b/agent/src/tools/br_market_data_tool.py @@ -0,0 +1,279 @@ +"""Read-only tool: official Brazilian market data from the local store. + +Serves what ``python -m src.br_data sync`` downloaded from B3 (COTAHIST), +CVM (funds, companies, financial statements), Banco Central (SGS) and +Tesouro Transparente. Everything is local SQLite, so calls are fast and +never rate-limited. BCB series that were not synced are fetched live. +""" + +from __future__ import annotations + +import json +import sqlite3 +from datetime import date, datetime, timedelta +from typing import Any, Callable +from zoneinfo import ZoneInfo + +from src.agent.tools import BaseTool +from src.config.limits import TOOL_RESULT_LIMIT + +_DEFAULT_LIMIT = 250 +_MAX_LIMIT = 5000 + +# Friendly names for common SGS series. +_BCB_ALIASES = { + "selic": 11, "selic_meta": 432, "cdi": 12, "cdi_anual": 4389, + "ipca": 433, "ipca_12m": 13522, "igpm": 189, "inpc": 188, + "dolar": 1, "usd": 1, "ptax": 1, "euro": 21619, "pib": 4380, "desemprego": 24369, +} +_STATEMENTS = ("BPA", "BPP", "DRE", "DRA", "DFC_MD", "DFC_MI", "DVA", "DMPL") + + +def _digits(value: Any) -> str: + return "".join(ch for ch in str(value or "") if ch.isdigit()) + + +def _ticker(value: Any) -> str: + text = str(value or "").strip().upper() + return text[:-3] if text.endswith(".SA") else text + + +def _limit(value: Any) -> int: + try: + return max(1, min(int(value), _MAX_LIMIT)) + except (TypeError, ValueError): + return _DEFAULT_LIMIT + + +def _dates(kwargs: dict) -> tuple[str, str]: + end = str(kwargs.get("end_date") or datetime.now(ZoneInfo("America/Sao_Paulo")).date().isoformat())[:10] + start = str(kwargs.get("start_date") or (date.fromisoformat(end) - timedelta(days=365)).isoformat())[:10] + if date.fromisoformat(start) > date.fromisoformat(end): + raise ValueError("start_date must not be after end_date") + return start, end + + +def _error(message: str) -> str: + return json.dumps({"ok": False, "error": message}, ensure_ascii=False) + + +def _envelope(action: str, rows: list[dict], extra: dict | None = None) -> str: + """Return the newest rows that fit the result budget.""" + count = len(rows) + while True: + data = {"action": action, "count": count, "available": len(rows), + "truncated": count < len(rows), "rows": rows[-count:] if count else []} + if extra: + data.update(extra) + text = json.dumps({"ok": True, "market": "BR", "source": "br_data", "data": data}, + ensure_ascii=False, default=str) + if len(text) <= TOOL_RESULT_LIMIT: + return text + if count <= 1: + return _error("Result metadata exceeds the tool limit; narrow the query.") + count = max(1, min(count - 1, int(count * TOOL_RESULT_LIMIT / len(text)))) + + +class BrMarketDataTool(BaseTool): + """Brazilian official data: B3 quotes, CVM funds/companies, BCB, Tesouro.""" + + name = "get_br_market_data" + description = ( + "Official, free Brazilian market data from the local store synced from " + "B3, CVM, Banco Central and Tesouro Direto. Actions: " + "'quotes' (daily B3 bars for any listed instrument — shares, FIIs, ETFs, " + "BDRs, options — e.g. tickers=['PETR4','HGLG11']; prices are as traded, " + "not adjusted); 'search' (find tickers, funds or companies by name); " + "'fund' (daily quota, net assets, flows and holders of any CVM fund by " + "CNPJ); 'financials' (DFP annual / ITR quarterly statements of a listed " + "company by ticker or CNPJ: DRE, BPA, BPP, DFC_MI ...); 'macro' (BCB " + "SGS series: selic, cdi, ipca, igpm, ptax or a numeric code); " + "'tesouro' (Tesouro Direto prices and yields); 'status' (what is synced). " + 'Example: {"action": "fund", "cnpj": "00.017.024/0001-53", ' + '"start_date": "2025-01-01"}.' + ) + parameters = { + "type": "object", + "properties": { + "action": { + "type": "string", + "enum": ["quotes", "search", "fund", "financials", "macro", "tesouro", "status"], + }, + "tickers": { + "type": "array", "items": {"type": "string"}, + "description": "B3 tickers for 'quotes' (PETR4 or PETR4.SA), or one ticker for 'financials'.", + }, + "query": {"type": "string", "description": "Name fragment for 'search' or bond name for 'tesouro'."}, + "cnpj": {"type": "string", "description": "Fund CNPJ ('fund') or company CNPJ ('financials')."}, + "statement": {"type": "string", "enum": list(_STATEMENTS), "description": "Statement for 'financials' (default DRE)."}, + "doc": {"type": "string", "enum": ["DFP", "ITR"], "description": "DFP annual (default) or ITR quarterly."}, + "series": {"type": "string", "description": "BCB alias (selic, cdi, ipca, igpm, ptax, dolar, euro) or SGS code."}, + "start_date": {"type": "string", "description": "YYYY-MM-DD (default: one year before end_date)."}, + "end_date": {"type": "string", "description": "YYYY-MM-DD (default: today)."}, + "limit": {"type": "integer", "description": f"Max rows (1-{_MAX_LIMIT}, default {_DEFAULT_LIMIT}); newest kept."}, + }, + "required": ["action"], + } + repeatable = True + + def execute(self, **kwargs: Any) -> str: + action = str(kwargs.get("action") or "").strip().lower() + handlers: dict[str, Callable[[sqlite3.Connection, dict], str]] = { + "quotes": self._quotes, "search": self._search, "fund": self._fund, + "financials": self._financials, "macro": self._macro, + "tesouro": self._tesouro, "status": self._status, + } + handler = handlers.get(action) + if handler is None: + return _error(f"unknown action {action!r}; use one of {sorted(handlers)}") + from src.br_data.store import connect + + try: + with connect() as conn: + return handler(conn, kwargs) + except FileNotFoundError as exc: + if action == "macro": + return self._macro(None, kwargs) + return _error(str(exc)) + except (ValueError, TypeError) as exc: + return _error(f"invalid request: {exc}") + except sqlite3.Error as exc: + return _error(f"store query failed: {exc}") + + # -- actions ----------------------------------------------------------- + + def _quotes(self, conn: sqlite3.Connection, kwargs: dict) -> str: + tickers = [_ticker(t) for t in (kwargs.get("tickers") or []) if _ticker(t)] + if not tickers: + return _error("'tickers' is required for action 'quotes'") + start, end = _dates(kwargs) + marks = ",".join("?" for _ in tickers) + rows = conn.execute( + "SELECT trade_date, ticker, market_type, spec, open, high, low, close, avg, " + "volume, turnover, trades, strike, expiry FROM b3_quotes " + f"WHERE ticker IN ({marks}) AND trade_date BETWEEN ? AND ? ORDER BY trade_date DESC, ticker DESC LIMIT ?", + (*tickers, start, end, _limit(kwargs.get("limit"))), + ).fetchall() + out = [dict(r) for r in reversed(rows)] + return _envelope("quotes", out, {"adjustment": "raw (as traded)", "currency": "BRL"}) + + def _search(self, conn: sqlite3.Connection, kwargs: dict) -> str: + query = str(kwargs.get("query") or "").strip() + if len(query) < 2: + return _error("'query' (2+ characters) is required for action 'search'") + like = f"%{query.upper()}%" + limit = min(_limit(kwargs.get("limit")), 50) + tickers = conn.execute( + "SELECT ticker, short_name, spec, MAX(trade_date) AS last_trade FROM b3_quotes " + "WHERE market_type = 10 AND (ticker LIKE ? OR UPPER(short_name) LIKE ?) " + "AND trade_date >= (SELECT date(MAX(trade_date), '-90 day') FROM b3_quotes) " + "GROUP BY ticker ORDER BY ticker LIMIT ?", + (like, like, limit), + ).fetchall() + funds = conn.execute( + "SELECT cnpj, name, fund_type, fund_class, status FROM cvm_funds " + "WHERE UPPER(name) LIKE ? OR cnpj LIKE ? ORDER BY (status LIKE 'EM FUNCIONAMENTO%') DESC, name LIMIT ?", + (like, f"%{_digits(query) or '#'}%", limit), + ).fetchall() + companies = conn.execute( + "SELECT c.cnpj, c.name, c.trade_name, c.sector, c.status, " + "GROUP_CONCAT(t.ticker) AS tickers FROM cvm_companies c " + "LEFT JOIN cvm_company_tickers t ON t.cnpj = c.cnpj " + "WHERE UPPER(c.name) LIKE ? OR UPPER(c.trade_name) LIKE ? OR t.ticker LIKE ? " + "GROUP BY c.cnpj LIMIT ?", + (like, like, like, limit), + ).fetchall() + return _envelope("search", [], { + "tickers": [dict(r) for r in tickers], + "funds": [dict(r) for r in funds], + "companies": [dict(r) for r in companies], + }) + + def _fund(self, conn: sqlite3.Connection, kwargs: dict) -> str: + cnpj = _digits(kwargs.get("cnpj")) + if len(cnpj) != 14: + return _error("'cnpj' (14 digits) is required for action 'fund'; use action 'search' to find it") + start, end = _dates(kwargs) + info = conn.execute("SELECT * FROM cvm_funds WHERE cnpj = ?", (cnpj,)).fetchone() + rows = conn.execute( + "SELECT trade_date, subclass, quota, net_assets, total_value, inflows, outflows, holders " + "FROM cvm_fund_daily WHERE cnpj = ? AND trade_date BETWEEN ? AND ? ORDER BY trade_date DESC, subclass DESC LIMIT ?", + (cnpj, start, end, _limit(kwargs.get("limit"))), + ).fetchall() + out = [dict(r) for r in reversed(rows)] + extra: dict[str, Any] = {"fund": dict(info) if info else None, "currency": "BRL"} + if len({r["subclass"] for r in out}) == 1 and len(out) >= 2 and out[0]["quota"] and out[-1]["quota"]: + extra["quota_return"] = round(out[-1]["quota"] / out[0]["quota"] - 1, 6) + return _envelope("fund", out, extra) + + def _financials(self, conn: sqlite3.Connection, kwargs: dict) -> str: + cnpj = _digits(kwargs.get("cnpj")) + tickers = kwargs.get("tickers") or [] + if not cnpj and tickers: + ticker = _ticker(tickers[0]) + row = conn.execute("SELECT cnpj FROM cvm_company_tickers WHERE ticker = ?", (ticker,)).fetchone() + if row is None: + return _error(f"no CVM company mapped to ticker {ticker}; try action 'search'") + cnpj = row["cnpj"] + if len(cnpj) != 14: + return _error("'cnpj' or one entry in 'tickers' is required for action 'financials'") + statement = str(kwargs.get("statement") or "DRE").upper() + doc = str(kwargs.get("doc") or "DFP").upper() + rows = conn.execute( + "SELECT statement, period_end, period_start, account, description, value, version FROM cvm_statements " + "WHERE cnpj = ? AND doc = ? AND statement IN (?, ?) ORDER BY period_end, account", + (cnpj, doc, statement, f"{statement}_ind"), + ).fetchall() + periods = sorted({r["period_end"] for r in rows}) + keep = set(periods[-min(_limit(kwargs.get("limit")), 8 if doc == "DFP" else 12):]) + out = [dict(r) for r in rows if r["period_end"] in keep] + company = conn.execute("SELECT * FROM cvm_companies WHERE cnpj = ?", (cnpj,)).fetchone() + return _envelope("financials", out, { + "company": dict(company) if company else {"cnpj": cnpj}, + "doc": doc, "statement": statement, "currency": "BRL", "periods": sorted(keep), + "point_in_time": False, "basis": "latest stored restatement; scope is labelled per row", + }) + + def _macro(self, conn: sqlite3.Connection | None, kwargs: dict) -> str: + raw = str(kwargs.get("series") or "").strip().lower() + code = _BCB_ALIASES.get(raw) or (int(raw) if raw.isdigit() else None) + if code is None: + return _error(f"'series' must be a BCB SGS code or one of {sorted(_BCB_ALIASES)}") + start, end = _dates(kwargs) + rows: list[dict] = [] + if conn is not None: + rows = [dict(r) for r in conn.execute( + "SELECT obs_date AS date, value FROM bcb_series WHERE code = ? " + "AND obs_date BETWEEN ? AND ? ORDER BY obs_date", (code, start, end), + ).fetchall()] + if not rows: + from src.br_data.bcb import fetch_series + + try: + rows = [{"date": d, "value": v} for d, v in + fetch_series(code, date.fromisoformat(start), date.fromisoformat(end))] + except Exception as exc: # noqa: BLE001 - surface as envelope + return _error(f"BCB SGS request failed: {exc}") + from src.br_data.bcb import SERIES + + return _envelope("macro", rows[-_limit(kwargs.get("limit")):], + {"series": code, "name": SERIES.get(code)}) + + def _tesouro(self, conn: sqlite3.Connection, kwargs: dict) -> str: + start, end = _dates(kwargs) + query = f"%{str(kwargs.get('query') or '').upper()}%" + rows = conn.execute( + "SELECT base_date, bond, maturity, buy_rate, sell_rate, buy_price, sell_price " + "FROM tesouro_prices WHERE UPPER(bond) LIKE ? AND base_date BETWEEN ? AND ? " + "ORDER BY base_date DESC, bond DESC, maturity DESC LIMIT ?", (query, start, end, _limit(kwargs.get("limit"))), + ).fetchall() + return _envelope("tesouro", [dict(r) for r in reversed(rows)], {"currency": "BRL"}) + + def _status(self, conn: sqlite3.Connection, kwargs: dict) -> str: + out = [] + for table, col in (("b3_quotes", "trade_date"), ("cvm_fund_daily", "trade_date"), + ("cvm_statements", "period_end"), ("bcb_series", "obs_date"), + ("tesouro_prices", "base_date")): + row = conn.execute(f"SELECT COUNT(*), MIN({col}), MAX({col}) FROM {table}").fetchone() + out.append({"table": table, "rows": row[0], "from": row[1], "to": row[2]}) + return _envelope("status", out) diff --git a/agent/src/tools/financial_statements_tool.py b/agent/src/tools/financial_statements_tool.py index 53a87cb0..2031473d 100644 --- a/agent/src/tools/financial_statements_tool.py +++ b/agent/src/tools/financial_statements_tool.py @@ -384,6 +384,80 @@ def _fetch_yahoo_statement( return {"periods": _cap_periods(periods)} +# CVM statement codes per requested statement, consolidated first. CVM files +# carry no ratio/indicator block, so "indicators" always comes from Yahoo. +_CVM_STATEMENTS: dict[str, tuple[str, ...]] = { + "balance": ("BPA", "BPP"), + "income": ("DRE",), + "cashflow": ("DFC_MI", "DFC_MD"), +} + + +def _fetch_cvm_statement(code: str, *, statement: str, period: str) -> dict[str, Any] | None: + """Read a B3 company's filed statement from the local CVM store. + + The store is filled by ``python -m src.br_data sync`` (DFP annual, ITR + quarterly). Consolidated statements are preferred; parent-only (``_ind``) + ones are used when a company files no consolidated set. + + Args: + code: B3 symbol, e.g. ``"PETR4.SA"``. + statement: ``balance``, ``income`` or ``cashflow``. + period: ``"annual"`` (DFP) or ``"quarter"`` (ITR). + + Returns: + ``{"periods": [...]}`` newest first, each period a flat + ``{"period_end", "period_start", " ": value}`` + record in BRL, or ``None`` when the store, the ticker or the rows are + missing, so the caller can fall back to Yahoo. + """ + statements = _CVM_STATEMENTS.get(statement) + if not statements: + return None + try: + from src.br_data.store import connect + + ticker = code.rpartition(".")[0].strip().upper() + doc = "DFP" if period == "annual" else "ITR" + with connect() as conn: + row = conn.execute( + "SELECT cnpj FROM cvm_company_tickers WHERE ticker = ?", (ticker,) + ).fetchone() + if row is None: + return None + rows: list[Any] = [] + for suffix in ("", "_ind"): + names = [name + suffix for name in statements] + marks = ",".join("?" for _ in names) + rows = conn.execute( + "SELECT period_end, period_start, account, description, value " + f"FROM cvm_statements WHERE cnpj = ? AND doc = ? AND statement IN ({marks}) " + "ORDER BY period_end DESC, period_start, account", + (row["cnpj"], doc, *names), + ).fetchall() + if rows: + break + except Exception as exc: # noqa: BLE001 - missing store or schema: fall back to Yahoo + logger.info("cvm statement unavailable for %s: %s", code, exc) + return None + if not rows: + return None + periods: dict[tuple[str, str], dict[str, Any]] = {} + for r in rows: + key = (r["period_end"], r["period_start"]) + record = periods.setdefault( + key, {"period_end": r["period_end"], "period_start": r["period_start"] or None} + ) + record[f"{r['account']} {r['description'] or ''}".strip()] = r["value"] + ordered = sorted( + periods.values(), + key=lambda rec: (rec["period_end"], rec["period_start"] or ""), + reverse=True, + ) + return {"periods": _cap_periods(ordered), "point_in_time": False, + "basis": "latest stored CVM revisions"} + + def _to_number(value: Any) -> float | None: """Coerce a numeric provider cell to float, preserving missing values.""" if value in (None, "", "-"): @@ -681,6 +755,9 @@ def _classify_market(code: str) -> str | None: if suffix == "L": # London Stock Exchange (#1206). return "uk" + if suffix == "SA": + # B3 (Brazil). + return "br" return None @@ -692,9 +769,10 @@ class FinancialStatementsTool(BaseTool): "Fetch a single stock's financial statements: balance sheet, income " "statement, cash-flow statement, or key per-period indicators (margins, " "ROE, EPS, etc.). Markets: A-share (.SH/.SZ/.BJ), US (.US), " - "Hong Kong (.HK) and UK LSE (.L). US uses SEC EDGAR companyfacts; " - "A-share and HK use Eastmoney; UK uses Yahoo quoteSummary (annual " - "history). Reports come back newest-first as flat per-period rows. Use " + "Hong Kong (.HK), UK LSE (.L) and Brazil B3 (.SA). US uses SEC EDGAR " + "companyfacts; A-share and HK use Eastmoney; UK uses Yahoo quoteSummary " + "(annual history); Brazil uses the official CVM DFP/ITR filings from the " + "local br-data store (values in BRL), falling back to Yahoo. Reports come back newest-first as flat per-period rows. Use " 'this to read fundamentals before building a valuation or screen. Example: ' '{"code": "600519.SH", "statement": "income", "period": "annual"}.' ) @@ -705,7 +783,7 @@ class FinancialStatementsTool(BaseTool): "type": "string", "description": ( "Single symbol with a market suffix, e.g. '600519.SH', " - "'000001.SZ', 'AAPL.US', '00700.HK', or 'VOD.L'." + "'000001.SZ', 'AAPL.US', '00700.HK', 'VOD.L', or 'PETR4.SA'." ), }, "statement": { @@ -772,7 +850,7 @@ class FinancialStatementsTool(BaseTool): market = _classify_market(code) if market is None: return _error( - "code must carry a supported suffix: .SH/.SZ/.BJ, .US, .HK, or .L" + "code must carry a supported suffix: .SH/.SZ/.BJ, .US, .HK, .L, or .SA" ) if market == "us": @@ -781,6 +859,15 @@ class FinancialStatementsTool(BaseTool): elif market == "uk": result = _fetch_yahoo_statement(code, statement=statement, period=period) source = "yahoo" + elif market == "br": + # Official CVM filings (local store) first; Yahoo when not synced + # yet and for the indicator block CVM does not publish. + cvm = _fetch_cvm_statement(code, statement=statement, period=period) + if cvm is not None: + result, source = cvm, "cvm" + else: + result = _fetch_yahoo_statement(code, statement=statement, period=period) + source = "yahoo" else: result = _fetch_eastmoney_statement( code, statement=statement, period=period diff --git a/agent/src/tools/stock_news_tool.py b/agent/src/tools/stock_news_tool.py index 3cff0668..24efdcb3 100644 --- a/agent/src/tools/stock_news_tool.py +++ b/agent/src/tools/stock_news_tool.py @@ -42,7 +42,7 @@ _EM_NEWS_URL = "https://search-api-web.eastmoney.com/search/jsonp" # A-share / China-market suffixes that route to the Eastmoney news surface. _EM_SUFFIXES = ("SH", "SZ", "BJ") # Suffixes that route to Yahoo's search-news surface. -_YAHOO_SUFFIXES = ("US", "HK") +_YAHOO_SUFFIXES = ("US", "HK", "SA") # Default broad-market query used when ``scope='global'`` carries no code. _GLOBAL_QUERY = "财经" @@ -249,8 +249,9 @@ class StockNewsTool(BaseTool): description = ( "Fetch recent financial news headlines, read-only and no auth. Markets: " "China A-share (SH/SZ/BJ) returns Eastmoney news ARTICLES " - "(title/url/source/published/snippet) under 'articles'. US (.US) and Hong " - "Kong (.HK) return Yahoo Finance news articles with the same fields. Use " + "(title/url/source/published/snippet) under 'articles'. US (.US), Hong " + "Kong (.HK) and Brazil (B3 .SA) return Yahoo Finance news articles with " + "the same fields. Use " "scope 'stock' with a 'code', or scope 'global' (no code) for broad " "China-market finance articles. " 'Example: {"code": "600519.SH", "scope": "stock", "limit": 10}.' @@ -353,6 +354,10 @@ class StockNewsTool(BaseTool): if suffix in _EM_SUFFIXES: return self._stock_via_eastmoney(code, query, limit) if suffix in _YAHOO_SUFFIXES: + # A bare B3 code (PETR4) is ambiguous on Yahoo search; its own + # spelling (PETR4.SA) is the listing. + if suffix == "SA": + query = code.upper() return self._stock_via_yahoo(code, query, limit) return self._error( f"unsupported market for code {code!r}; expected suffix in " @@ -374,7 +379,7 @@ class StockNewsTool(BaseTool): def _stock_via_yahoo(self, code: str, query: str, limit: int) -> str: """Fetch US/HK news articles from Yahoo for one code.""" - market = "hk" if _suffix_of(code) == "HK" else "us" + market = {"HK": "hk", "SA": "br"}.get(_suffix_of(code), "us") try: articles = _fetch_yahoo_news(query, limit) except Exception as exc: # noqa: BLE001 - surface any fetch failure as envelope diff --git a/agent/src/tools/stock_profile_tool.py b/agent/src/tools/stock_profile_tool.py index c95d0856..8dcc63c8 100644 --- a/agent/src/tools/stock_profile_tool.py +++ b/agent/src/tools/stock_profile_tool.py @@ -275,6 +275,8 @@ def _market_for(ticker: str) -> str: return "uk" if upper.endswith(".BA"): return "ar" + if upper.endswith(".SA"): + return "br" return "us" @@ -283,8 +285,8 @@ class StockProfileTool(BaseTool): name = "get_stock_profile" description = ( - "Fetch a read-only company profile for a US, Hong Kong, UK (LSE .L), or " - "Argentina (BYMA .BA) " + "Fetch a read-only company profile for a US, Hong Kong, UK (LSE .L), " + "Argentina (BYMA .BA), or Brazil (B3 .SA) " "listing from Yahoo Finance: valuation key statistics, analyst price " "targets and earnings/revenue estimates, institutional and insider " "ownership, and the analyst recommendation trend. Always includes " @@ -301,7 +303,7 @@ class StockProfileTool(BaseTool): "ticker": { "type": "string", "description": ( - "US, HK, UK (LSE .L), or Argentina (BYMA .BA) symbol. US uses a bare or .US suffix " + "US, HK, UK (LSE .L), Argentina (BYMA .BA), or Brazil (B3 .SA, e.g. PETR4.SA) symbol. US uses a bare or .US suffix " "(AAPL or AAPL.US); HK uses a zero-padded .HK code " "(00700.HK); UK uses the .L suffix (VOD.L)." ), diff --git a/agent/src/tools/symbol_search_tool.py b/agent/src/tools/symbol_search_tool.py index e3163153..8c250696 100644 --- a/agent/src/tools/symbol_search_tool.py +++ b/agent/src/tools/symbol_search_tool.py @@ -160,6 +160,7 @@ _MARKET_BY_SUFFIX: Dict[str, str] = { "TO": "ca", "V": "ca", "BA": "ar", + "SA": "br", "L": "uk", "VN": "vn", } diff --git a/agent/src/trading/connectors/alpaca/sdk.py b/agent/src/trading/connectors/alpaca/sdk.py index f40b80b6..f3ee8aa7 100644 --- a/agent/src/trading/connectors/alpaca/sdk.py +++ b/agent/src/trading/connectors/alpaca/sdk.py @@ -74,7 +74,9 @@ class AlpacaConfig: profile: ``paper``, ``live-readonly`` or ``live``. feed: Market-data feed, ``iex`` (free) or ``sip`` (paid). timeout: Network timeout in seconds. - readonly: Always true for this layer; order methods are not exposed. + readonly: Not consulted by the order methods. Whether a profile may place or + cancel orders is gated first by its ``TradingProfile.readonly`` in the + trading service. """ api_key: str = "" diff --git a/agent/src/trading/connectors/dhan/sdk.py b/agent/src/trading/connectors/dhan/sdk.py index 71c23457..bdbdb022 100644 --- a/agent/src/trading/connectors/dhan/sdk.py +++ b/agent/src/trading/connectors/dhan/sdk.py @@ -80,7 +80,9 @@ class DhanConfig: access_token: Dhan API access token (generated from dhan.co). profile: ``paper``, ``live-readonly`` or ``live``. timeout: Network timeout in seconds. - readonly: Whether order placement is disabled. + readonly: Not consulted by the order methods. Whether a profile may place or + cancel orders is gated first by its ``TradingProfile.readonly`` in the + trading service. """ client_id: str = "" diff --git a/agent/src/trading/connectors/etoro/client.py b/agent/src/trading/connectors/etoro/client.py index f61414c7..272b4556 100644 --- a/agent/src/trading/connectors/etoro/client.py +++ b/agent/src/trading/connectors/etoro/client.py @@ -64,7 +64,9 @@ class EtoroConfig: user_key: eToro user key (shared across demo and real paths). profile: ``paper``, ``live-readonly`` or ``live``. timeout: Network timeout in seconds. - readonly: Always true for this layer unless trade profiles call writes. + readonly: Not consulted by the order methods. Whether a profile may place or + cancel orders is gated first by its ``TradingProfile.readonly`` in the + trading service. """ api_key: str = "" diff --git a/agent/src/trading/connectors/futu/sdk.py b/agent/src/trading/connectors/futu/sdk.py index 3a362417..7c263c7c 100644 --- a/agent/src/trading/connectors/futu/sdk.py +++ b/agent/src/trading/connectors/futu/sdk.py @@ -80,7 +80,9 @@ class FutuConfig: filter_trdmarket: Trade market filter for the trade context, e.g. ``HK``. acc_id: Account id; ``0`` means resolve by ``trd_env``. timeout: Network/connect timeout in seconds. - readonly: Always true for this layer; order methods are not exposed. + readonly: Not consulted by the order methods. Whether a profile may place or + cancel orders is gated first by its ``TradingProfile.readonly`` in the + trading service. """ host: str = DEFAULT_HOST diff --git a/agent/src/trading/connectors/kis/sdk.py b/agent/src/trading/connectors/kis/sdk.py index 0eef2ec2..716a4a2f 100644 --- a/agent/src/trading/connectors/kis/sdk.py +++ b/agent/src/trading/connectors/kis/sdk.py @@ -121,7 +121,9 @@ class KISConfig: e.g. ``"01"``. profile: ``paper`` or ``live-readonly``. timeout: Network timeout in seconds. - readonly: Whether order placement is disabled. + readonly: Not consulted by the order methods. Whether a profile may place or + cancel orders is gated first by its ``TradingProfile.readonly`` in the + trading service. """ app_key: str = "" diff --git a/agent/src/trading/connectors/longbridge/sdk.py b/agent/src/trading/connectors/longbridge/sdk.py index a4b46dbb..5d2c106d 100644 --- a/agent/src/trading/connectors/longbridge/sdk.py +++ b/agent/src/trading/connectors/longbridge/sdk.py @@ -66,7 +66,9 @@ class LongbridgeConfig: profile: ``paper``, ``live-readonly`` or ``live`` (operator-declared). region: ``global`` or ``cn`` (host region; not a paper/live signal). timeout: Network timeout in seconds. - readonly: Always true for this layer; order methods are not exposed. + readonly: Not consulted by the order methods. Whether a profile may place or + cancel orders is gated first by its ``TradingProfile.readonly`` in the + trading service. """ app_key: str = field(default="", repr=False) diff --git a/agent/src/trading/connectors/okx/sdk.py b/agent/src/trading/connectors/okx/sdk.py index 8067b9a8..723dce7a 100644 --- a/agent/src/trading/connectors/okx/sdk.py +++ b/agent/src/trading/connectors/okx/sdk.py @@ -60,7 +60,9 @@ class OKXConfig: host: REST host (default ``https://www.okx.com``). expected_uid: Optional account UID to pin in ``check_status``. timeout: Network timeout in seconds. - readonly: Always true for this layer; order methods are not exposed. + readonly: Not consulted by the order methods. Whether a profile may place or + cancel orders is gated first by its ``TradingProfile.readonly`` in the + trading service. """ api_key: str = "" diff --git a/agent/src/trading/connectors/tiger/sdk.py b/agent/src/trading/connectors/tiger/sdk.py index 998d36b8..8179c754 100644 --- a/agent/src/trading/connectors/tiger/sdk.py +++ b/agent/src/trading/connectors/tiger/sdk.py @@ -68,7 +68,9 @@ class TigerConfig: account: Tiger account number. A 17-digit number is a paper account. profile: ``paper``, ``live-readonly`` or ``live``. timeout: Network timeout in seconds. - readonly: Always true for this layer; order methods are not exposed. + readonly: Not consulted by the order methods. Whether a profile may place or + cancel orders is gated first by its ``TradingProfile.readonly`` in the + trading service. """ tiger_id: str = "" diff --git a/agent/src/trading/connectors/upbit/sdk.py b/agent/src/trading/connectors/upbit/sdk.py index ae8cf0d2..838a9727 100644 --- a/agent/src/trading/connectors/upbit/sdk.py +++ b/agent/src/trading/connectors/upbit/sdk.py @@ -74,7 +74,9 @@ class UpbitConfig: secret_key: Upbit Open API secret key. profile: ``paper`` or ``live-readonly``. timeout: Network timeout in seconds. - readonly: Whether order placement is disabled. + readonly: Not consulted by the order methods. Whether a profile may place or + cancel orders is gated first by its ``TradingProfile.readonly`` in the + trading service. """ access_key: str = "" diff --git a/agent/src/trading/service.py b/agent/src/trading/service.py index f1f688c9..16e47e36 100644 --- a/agent/src/trading/service.py +++ b/agent/src/trading/service.py @@ -131,9 +131,9 @@ def _sdk_config( # # Per-call overrides must still pass the connector's own allowlist. Every # SDK connector deliberately narrows what a caller may override — OKX and - # Binance both exclude ``readonly`` ("always true for this layer") and - # ``timeout``; Longbridge's overlay is ``profile``/``region`` only, - # precisely so a caller "cannot mix or bypass the shared resolver". + # Binance both exclude ``readonly`` and ``timeout``; Longbridge's overlay + # is ``profile``/``region`` only, precisely so a caller "cannot mix or + # bypass the shared resolver". # ``build_config`` applies that filter; constructing from a raw merged # mapping would not, and ``overrides`` is the one part of this payload that # reaches us from an MCP tool argument or a REST body. diff --git a/agent/tests/factors/test_fundamental_expansion_price_samples.py b/agent/tests/factors/test_fundamental_expansion_price_samples.py new file mode 100644 index 00000000..28c2be6b --- /dev/null +++ b/agent/tests/factors/test_fundamental_expansion_price_samples.py @@ -0,0 +1,211 @@ +"""Sample tests for the Tier-1 fundamental expansion factors. + +Covers the four price-dependent / derived-field factors added per +``research/fundamental_factor_expansion_proposal.md`` §4 Tier 1 (#1, #5, #6, #8): + +- ``fund_book_to_market`` (real FF HML characteristic) +- ``fund_fcf_yield`` (CFO - capex over market cap) +- ``fund_revenue_growth`` (schema derived field ``revenue_growth``) +- ``fund_earnings_growth`` (schema derived field ``earnings_growth``) + +All compute-level tests pass hand-built panel dicts directly to ``compute()`` +— never through the loader/schema — so they are independent of when the +concurrent schema change adding ``revenue_growth``/``earnings_growth`` lands. +""" +from __future__ import annotations + +import numpy as np +import pandas as pd +import pytest + +from src.factors.base import zscore +from src.factors.registry import AlphaMeta +from src.factors.registry import get_default_registry +from src.factors.zoo.fundamental import book_to_market +from src.factors.zoo.fundamental import earnings_growth +from src.factors.zoo.fundamental import fcf_yield +from src.factors.zoo.fundamental import revenue_growth + +_DATES = pd.to_datetime(["2024-01-02", "2024-01-03"]) +_CODES = ["A", "B", "C"] + +_PRICE_FACTORS = [book_to_market, fcf_yield] +_GROWTH_FACTORS = [revenue_growth, earnings_growth] +_ALL_FACTORS = _PRICE_FACTORS + _GROWTH_FACTORS + + +def _frame(rows: list[list[float]]) -> pd.DataFrame: + """Build a 2-date x 3-symbol frame from per-date row values.""" + return pd.DataFrame(rows, index=_DATES, columns=_CODES, dtype=np.float64) + + +def _book_to_market_panel() -> dict[str, pd.DataFrame]: + # market cap: A=1000, B=2000, C=5000; equity 500/250/100 + # -> b/m: A=0.50, B=0.125, C=0.02 (strictly decreasing) + return { + "close": _frame([[10.0, 20.0, 50.0], [10.0, 20.0, 50.0]]), + "fund:total_equity": _frame([[500.0, 250.0, 100.0], [500.0, 250.0, 100.0]]), + "fund:shares_diluted": _frame([[100.0, 100.0, 100.0], [100.0, 100.0, 100.0]]), + } + + +def _fcf_yield_panel() -> dict[str, pd.DataFrame]: + # A: cfo=100, capex=30 -> fcf=70, mcap=10*10=100 -> yield 0.70 + # B: cfo=50, capex=15 -> fcf=35, mcap=20*10=200 -> yield 0.175 + # C: cfo=20, capex=25 -> fcf=-5, mcap=50*10=500 -> yield -0.01 + return { + "close": _frame([[10.0, 20.0, 50.0], [10.0, 20.0, 50.0]]), + "fund:cfo": _frame([[100.0, 50.0, 20.0], [100.0, 50.0, 20.0]]), + "fund:capex": _frame([[30.0, 15.0, 25.0], [30.0, 15.0, 25.0]]), + "fund:shares_diluted": _frame([[10.0, 10.0, 10.0], [10.0, 10.0, 10.0]]), + } + + +def _growth_panel(field: str) -> dict[str, pd.DataFrame]: + return {field: _frame([[0.30, 0.10, -0.20], [0.30, 0.10, -0.20]])} + + +def _valid_panel(module) -> dict[str, pd.DataFrame]: + if module is book_to_market: + return _book_to_market_panel() + if module is fcf_yield: + return _fcf_yield_panel() + if module is revenue_growth: + return _growth_panel("fund:revenue_growth") + return _growth_panel("fund:earnings_growth") + + +# --------------------------------------------------------------------------- +# (a) AlphaMeta gate accepts each factor's declared columns_required +# --------------------------------------------------------------------------- + + +@pytest.mark.parametrize("module", _ALL_FACTORS, ids=lambda m: m.__alpha_meta__["id"]) +def test_alpha_meta_gate_accepts_declared_columns(module) -> None: + meta = AlphaMeta(**module.__alpha_meta__) + + assert meta.columns_required == module.__alpha_meta__["columns_required"] + assert meta.decay_horizon == 252 + assert meta.min_warmup_bars == 1 + assert meta.universe == ["equity_us", "equity_cn", "equity_hk"] + assert meta.frequency == ["1d"] + + +def test_alpha_meta_themes_match_proposal() -> None: + assert book_to_market.__alpha_meta__["theme"] == ["value"] + assert fcf_yield.__alpha_meta__["theme"] == ["value"] + assert revenue_growth.__alpha_meta__["theme"] == ["growth"] + assert earnings_growth.__alpha_meta__["theme"] == ["growth"] + + +# --------------------------------------------------------------------------- +# (b) compute() numeric correctness on hand-built panels +# --------------------------------------------------------------------------- + + +def test_book_to_market_matches_hand_computed_ratios() -> None: + panel = _book_to_market_panel() + expected_ratios = _frame([[0.50, 0.125, 0.02], [0.50, 0.125, 0.02]]) + + result = book_to_market.compute(panel) + + pd.testing.assert_frame_equal(result, zscore(expected_ratios)) + + +def test_book_to_market_direction_higher_equity_to_mcap_scores_higher() -> None: + panel = _book_to_market_panel() + + result = book_to_market.compute(panel) + + row = result.iloc[0] + assert row["A"] > row["B"] > row["C"] + + +def test_fcf_yield_subtracts_capex_from_cfo() -> None: + # A: cfo 100, capex 30 -> fcf 70 over mcap 100 -> yield 0.70 + panel = _fcf_yield_panel() + expected_ratios = _frame([[0.70, 0.175, -0.01], [0.70, 0.175, -0.01]]) + + result = fcf_yield.compute(panel) + + pd.testing.assert_frame_equal(result, zscore(expected_ratios)) + + +@pytest.mark.parametrize( + ("module", "field"), + [(revenue_growth, "fund:revenue_growth"), (earnings_growth, "fund:earnings_growth")], + ids=["fund_revenue_growth", "fund_earnings_growth"], +) +def test_growth_factors_zscore_declared_field(module, field: str) -> None: + panel = _growth_panel(field) + + result = module.compute(panel) + + pd.testing.assert_frame_equal(result, zscore(panel[field])) + row = result.iloc[0] + assert row["A"] > row["B"] > row["C"] + + +# --------------------------------------------------------------------------- +# (c) NaN contract: NaN in any declared dependency -> NaN output at that cell +# --------------------------------------------------------------------------- + + +@pytest.mark.parametrize("module", _ALL_FACTORS, ids=lambda m: m.__alpha_meta__["id"]) +def test_nan_in_any_declared_dependency_produces_nan_output(module) -> None: + for column in module.__alpha_meta__["columns_required"]: + panel = _valid_panel(module) + panel[column] = panel[column].copy() + panel[column].iloc[0, 1] = np.nan # symbol B on date 0 + + result = module.compute(panel) + + assert np.isnan(result.iloc[0, 1]), f"{module.__alpha_meta__['id']}: {column}" + assert np.isfinite(result.iloc[0, 0]), f"{module.__alpha_meta__['id']}: {column}" + assert np.isfinite(result.iloc[0, 2]), f"{module.__alpha_meta__['id']}: {column}" + + +# --------------------------------------------------------------------------- +# (d) Zero market cap / zero denominator -> NaN, never inf +# --------------------------------------------------------------------------- + + +@pytest.mark.parametrize("module", _PRICE_FACTORS, ids=lambda m: m.__alpha_meta__["id"]) +def test_zero_close_produces_nan_not_inf(module) -> None: + panel = _valid_panel(module) + panel["close"] = panel["close"].copy() + panel["close"].iloc[0, 1] = 0.0 # zero market cap for B on date 0 + + result = module.compute(panel) + + assert np.isnan(result.iloc[0, 1]) + assert not np.isinf(result.to_numpy(dtype=np.float64)).any() + + +@pytest.mark.parametrize("module", _PRICE_FACTORS, ids=lambda m: m.__alpha_meta__["id"]) +def test_zero_shares_produces_nan_not_inf(module) -> None: + panel = _valid_panel(module) + panel["fund:shares_diluted"] = panel["fund:shares_diluted"].copy() + panel["fund:shares_diluted"].iloc[1, 2] = 0.0 # zero share count for C on date 1 + + result = module.compute(panel) + + assert np.isnan(result.iloc[1, 2]) + assert not np.isinf(result.to_numpy(dtype=np.float64)).any() + + +# --------------------------------------------------------------------------- +# Registry load: the four ids are discoverable from the default registry +# --------------------------------------------------------------------------- + + +@pytest.mark.parametrize( + "alpha_id", + ["fund_book_to_market", "fund_fcf_yield", "fund_revenue_growth", "fund_earnings_growth"], +) +def test_factor_is_loadable_from_default_registry(alpha_id: str) -> None: + registry = get_default_registry() + + alpha = registry.get(alpha_id) + + assert alpha.id == alpha_id diff --git a/agent/tests/factors/test_fundamental_expansion_samples.py b/agent/tests/factors/test_fundamental_expansion_samples.py new file mode 100644 index 00000000..721c60d8 --- /dev/null +++ b/agent/tests/factors/test_fundamental_expansion_samples.py @@ -0,0 +1,231 @@ +"""Registration + numeric regression for the six Tier-1 fundamental factors. + +Covers ``fund_accruals``, ``fund_roa``, ``fund_leverage``, ``fund_cfo_yield``, +``fund_operating_margin`` and ``fund_cash_to_assets`` in ``zoo/fundamental/`` +(research/fundamental_factor_expansion_proposal.md §4 Tier 1 #2/#3/#4/#7/#9/#10). + +Assertions are membership-based, never an exact registry total: the zoo grows +concurrently. Expected values are recomputed here with an independent numpy +formula (mean / sample-std z-score, plain division) rather than reusing +``src.factors.base`` operators, so a regression in the shared operator does not +silently validate itself. +""" +from __future__ import annotations + +import importlib + +import numpy as np +import pandas as pd +import pytest + +from src.factors.registry import Registry + +_DATES = pd.to_datetime(["2024-01-01", "2024-01-02"]) +_CODES = ["AAA", "BBB", "CCC"] + +# alpha_id -> (theme, columns_required, module stem under zoo/fundamental/) +_EXPECTED_META: dict[str, tuple[list[str], list[str], str]] = { + "fund_accruals": (["quality"], ["fund:accruals"], "accruals"), + "fund_roa": (["quality"], ["fund:roa"], "roa"), + "fund_leverage": (["leverage"], ["fund:leverage"], "leverage"), + "fund_cfo_yield": (["quality"], ["fund:cfo", "fund:total_assets"], "cfo_yield"), + "fund_operating_margin": ( + ["quality"], + ["fund:operating_income", "fund:revenue"], + "operating_margin", + ), + "fund_cash_to_assets": (["quality"], ["fund:cash", "fund:total_assets"], "cash_to_assets"), +} + +# One declared dependency per alpha, blanked wholesale in the NaN-contract test. +_BLANKED_DEPENDENCY: dict[str, str] = { + "fund_accruals": "fund:accruals", + "fund_roa": "fund:roa", + "fund_leverage": "fund:leverage", + "fund_cfo_yield": "fund:cfo", + "fund_operating_margin": "fund:operating_income", + "fund_cash_to_assets": "fund:cash", +} + +# safe_div-based alphas and the denominator column driven to exactly zero. +_ZERO_DENOMINATOR: dict[str, str] = { + "fund_cfo_yield": "fund:total_assets", + "fund_operating_margin": "fund:revenue", + "fund_cash_to_assets": "fund:total_assets", +} + + +@pytest.fixture(scope="module") +def registry() -> Registry: + return Registry() + + +def _frame(rows: list[list[float]]) -> pd.DataFrame: + return pd.DataFrame(rows, index=_DATES, columns=_CODES, dtype="float64") + + +def _build_panel() -> dict[str, pd.DataFrame]: + """Hand-built 2-date x 3-symbol panel; every row has distinct values.""" + return { + "fund:accruals": _frame([[0.10, 0.02, -0.05], [0.08, 0.03, -0.01]]), + "fund:roa": _frame([[0.15, 0.05, -0.10], [0.12, 0.06, 0.00]]), + "fund:leverage": _frame([[2.0, 0.5, 0.1], [1.8, 0.6, 0.2]]), + "fund:cfo": _frame([[300.0, 200.0, 100.0], [320.0, 210.0, 90.0]]), + "fund:total_assets": _frame( + [[1000.0, 1000.0, 1000.0], [1100.0, 1050.0, 900.0]] + ), + "fund:operating_income": _frame([[200.0, 100.0, 50.0], [220.0, 110.0, 40.0]]), + "fund:revenue": _frame([[1000.0, 1000.0, 1000.0], [1000.0, 1100.0, 800.0]]), + "fund:cash": _frame([[400.0, 100.0, 50.0], [420.0, 110.0, 60.0]]), + } + + +def _numpy_zscore(values: np.ndarray) -> np.ndarray: + """Independent per-row z-score: (x - mean) / sample-std, NaN-preserving.""" + mean = np.nanmean(values, axis=1, keepdims=True) + std = np.nanstd(values, axis=1, ddof=1, keepdims=True) + with np.errstate(invalid="ignore", divide="ignore"): + return (values - mean) / std + + +def _z(frame: pd.DataFrame) -> pd.DataFrame: + return pd.DataFrame( + _numpy_zscore(frame.to_numpy(dtype=np.float64)), + index=frame.index, + columns=frame.columns, + ) + + +def _expected_outputs(panel: dict[str, pd.DataFrame]) -> dict[str, pd.DataFrame]: + """Expected output frames per alpha, derived from the panel by numpy math.""" + total_assets = panel["fund:total_assets"] + revenue = panel["fund:revenue"] + return { + "fund_accruals": -_z(panel["fund:accruals"]), + "fund_roa": _z(panel["fund:roa"]), + "fund_leverage": -_z(panel["fund:leverage"]), + "fund_cfo_yield": _z(panel["fund:cfo"] / total_assets), + "fund_operating_margin": _z(panel["fund:operating_income"] / revenue), + "fund_cash_to_assets": _z(panel["fund:cash"] / total_assets), + } + + +def _load_module(stem: str): + return importlib.import_module(f"src.factors.zoo.fundamental.{stem}") + + +# --------------------------------------------------------------- (a) registry + + +@pytest.mark.parametrize("alpha_id", sorted(_EXPECTED_META)) +def test_registry_loads_factor_with_expected_meta( + registry: Registry, alpha_id: str +) -> None: + theme, columns, stem = _EXPECTED_META[alpha_id] + alpha = registry.get(alpha_id) + assert alpha.zoo == "fundamental" + assert alpha.module_path == f"src.factors.zoo.fundamental.{stem}" + assert alpha.meta["theme"] == theme + assert alpha.meta["columns_required"] == columns + assert alpha.meta["universe"] == ["equity_us", "equity_cn", "equity_hk"] + assert alpha.meta["frequency"] == ["1d"] + assert alpha.meta["decay_horizon"] == 252 + assert alpha.meta["min_warmup_bars"] == 1 + assert alpha.meta["notes"].strip() + + +# ------------------------------------------------- (b) numeric correctness + + +@pytest.mark.parametrize("alpha_id", sorted(_EXPECTED_META)) +def test_compute_matches_independent_numpy_math( + registry: Registry, alpha_id: str +) -> None: + panel = _build_panel() + result = registry.compute(alpha_id, panel) + expected = _expected_outputs(panel)[alpha_id] + + assert list(result.columns) == _CODES + assert list(result.index) == list(_DATES) + # atol covers values that are zero up to fp noise: when a cell equals the + # row mean exactly (e.g. cfo/TA = [0.3, 0.2, 0.1]), its z-score is ~1e-16 + # with a sign that depends on summation order (pandas vs numpy). + np.testing.assert_allclose( + result.to_numpy(dtype=np.float64), + expected.to_numpy(dtype=np.float64), + rtol=1e-9, + atol=1e-12, + equal_nan=True, + err_msg=f"{alpha_id} diverged from independent numpy expectation", + ) + + +def test_accruals_sign_direction(registry: Registry) -> None: + """Sloan 1996: higher (NI-CFO)/TA must score LOWER on every date.""" + out = registry.compute("fund_accruals", _build_panel()) + accruals = _build_panel()["fund:accruals"] + for date in _DATES: + # CCC has the lowest accruals, AAA the highest, on both dates. + assert accruals.loc[date, "AAA"] > accruals.loc[date, "BBB"] + assert out.loc[date, "CCC"] > out.loc[date, "BBB"] > out.loc[date, "AAA"] + + +def test_leverage_sign_direction(registry: Registry) -> None: + """Low leverage scores high: sign inverted versus the raw ratio.""" + out = registry.compute("fund_leverage", _build_panel()) + leverage = _build_panel()["fund:leverage"] + for date in _DATES: + assert leverage.loc[date, "AAA"] > leverage.loc[date, "BBB"] + assert out.loc[date, "CCC"] > out.loc[date, "BBB"] > out.loc[date, "AAA"] + + +def test_roa_sign_direction(registry: Registry) -> None: + """Positive factor: higher ROA scores higher (no inversion).""" + out = registry.compute("fund_roa", _build_panel()) + for date in _DATES: + assert out.loc[date, "AAA"] > out.loc[date, "BBB"] > out.loc[date, "CCC"] + + +# ------------------------------------------------------------ (c) NaN contract + + +@pytest.mark.parametrize("alpha_id", sorted(_BLANKED_DEPENDENCY)) +def test_blank_dependency_bar_yields_all_nan( + registry: Registry, alpha_id: str +) -> None: + """Blanking a declared dependency on one bar must NaN that whole bar. + + The registry masks output wherever a declared dependency is missing + (#1377), and zscore/safe_div propagate NaN — the second date must come + back all-NaN while the first date is untouched. + """ + panel = _build_panel() + panel[_BLANKED_DEPENDENCY[alpha_id]].loc[_DATES[1]] = np.nan + + out = registry.compute(alpha_id, panel) + + assert out.loc[_DATES[1]].isna().all(), ( + f"{alpha_id}: emitted a value on a bar whose declared dependency was NaN" + ) + assert out.loc[_DATES[0]].notna().all() + + +# --------------------------------------------------- (d) zero denominator + + +@pytest.mark.parametrize("alpha_id", sorted(_ZERO_DENOMINATOR)) +def test_zero_denominator_is_nan_not_inf(alpha_id: str) -> None: + """safe_div contract: exact-zero denominator -> NaN, never +/-inf.""" + stem = _EXPECTED_META[alpha_id][2] + module = _load_module(stem) + panel = _build_panel() + panel[_ZERO_DENOMINATOR[alpha_id]].loc[_DATES[0], "BBB"] = 0.0 + + out = module.compute(panel) + arr = out.to_numpy(dtype=np.float64, na_value=np.nan) + + assert np.isnan(out.loc[_DATES[0], "BBB"]) + assert not np.isinf(arr).any() + # The other two symbols on that date still score against each other. + assert out.loc[_DATES[0], ["AAA", "CCC"]].notna().all() + assert out.loc[_DATES[1]].notna().all() diff --git a/agent/tests/factors/test_fundamental_valuation_samples.py b/agent/tests/factors/test_fundamental_valuation_samples.py new file mode 100644 index 00000000..41da12e8 --- /dev/null +++ b/agent/tests/factors/test_fundamental_valuation_samples.py @@ -0,0 +1,361 @@ +"""Registration + numeric regression for the three Tier-2 CN valuation factors. + +Covers ``fund_pb``, ``fund_ps_ttm`` and ``fund_dividend_yield`` in +``zoo/fundamental/`` (research/fundamental_factor_expansion_proposal.md §4 +Tier 2 #11/#12/#13), plus the ``val:`` namespace plumbing: the AlphaMeta +gate, the runner injection through a mocked tushare ``daily_basic`` API, the +whitelist rejection and the NaN-on-missing-data contract. + +Assertions are membership-based, never an exact registry total: the zoo grows +concurrently. Expected values are recomputed here with an independent numpy +formula (mean / sample-std z-score) rather than reusing ``src.factors.base`` +operators, so a regression in the shared operator does not silently validate +itself. +""" +from __future__ import annotations + +import importlib +import sys +from types import SimpleNamespace +from typing import Any + +import numpy as np +import pandas as pd +import pytest + +from backtest import runner +from src.factors.registry import AlphaMeta, Registry + +_DATES = pd.to_datetime(["2024-01-01", "2024-01-02"]) +_CODES = ["AAA", "BBB", "CCC"] + +# alpha_id -> (theme, columns_required, module stem under zoo/fundamental/) +_EXPECTED_META: dict[str, tuple[list[str], list[str], str]] = { + "fund_pb": (["value"], ["val:pb"], "pb"), + "fund_ps_ttm": (["value"], ["val:ps_ttm"], "ps_ttm"), + "fund_dividend_yield": (["value"], ["val:dv_ttm"], "dividend_yield"), +} + +# One declared dependency per alpha, blanked wholesale in the NaN-contract test. +_BLANKED_DEPENDENCY: dict[str, str] = { + "fund_pb": "val:pb", + "fund_ps_ttm": "val:ps_ttm", + "fund_dividend_yield": "val:dv_ttm", +} + + +@pytest.fixture(scope="module") +def registry() -> Registry: + return Registry() + + +def _frame(rows: list[list[float]]) -> pd.DataFrame: + return pd.DataFrame(rows, index=_DATES, columns=_CODES, dtype="float64") + + +def _build_panel() -> dict[str, pd.DataFrame]: + """Hand-built 2-date x 3-symbol panel; every row has distinct values.""" + return { + # PB: CCC cheapest, AAA most expensive on both dates. + "val:pb": _frame([[3.0, 1.5, 0.5], [2.8, 1.6, 0.6]]), + "val:ps_ttm": _frame([[4.0, 2.0, 1.0], [3.6, 2.2, 0.8]]), + # dv_ttm in percent units, as tushare daily_basic reports it. + "val:dv_ttm": _frame([[0.5, 2.0, 4.5], [0.6, 2.1, 4.4]]), + } + + +def _numpy_zscore(values: np.ndarray) -> np.ndarray: + """Independent per-row z-score: (x - mean) / sample-std, NaN-preserving.""" + mean = np.nanmean(values, axis=1, keepdims=True) + std = np.nanstd(values, axis=1, ddof=1, keepdims=True) + with np.errstate(invalid="ignore", divide="ignore"): + return (values - mean) / std + + +def _z(frame: pd.DataFrame) -> pd.DataFrame: + return pd.DataFrame( + _numpy_zscore(frame.to_numpy(dtype=np.float64)), + index=frame.index, + columns=frame.columns, + ) + + +def _expected_outputs(panel: dict[str, pd.DataFrame]) -> dict[str, pd.DataFrame]: + """Expected output frames per alpha, derived from the panel by numpy math.""" + return { + "fund_pb": -_z(panel["val:pb"]), + "fund_ps_ttm": -_z(panel["val:ps_ttm"]), + "fund_dividend_yield": _z(panel["val:dv_ttm"]), + } + + +def _load_module(stem: str): + return importlib.import_module(f"src.factors.zoo.fundamental.{stem}") + + +# ------------------------------------------------------- (a) gate + registry + + +def test_alpha_meta_gate_accepts_val_columns() -> None: + meta = AlphaMeta( + id="test_val_alpha", + theme=["value"], + formula_latex="x", + columns_required=["val:pb"], + universe=["equity_cn"], + frequency=["1d"], + decay_horizon=252, + min_warmup_bars=1, + ) + + assert meta.columns_required == ["val:pb"] + + +@pytest.mark.parametrize("alpha_id", sorted(_EXPECTED_META)) +def test_registry_loads_factor_with_expected_meta( + registry: Registry, alpha_id: str +) -> None: + theme, columns, stem = _EXPECTED_META[alpha_id] + alpha = registry.get(alpha_id) + assert alpha.zoo == "fundamental" + assert alpha.module_path == f"src.factors.zoo.fundamental.{stem}" + assert alpha.meta["theme"] == theme + assert alpha.meta["columns_required"] == columns + assert alpha.meta["universe"] == ["equity_cn"] + assert alpha.meta["frequency"] == ["1d"] + assert alpha.meta["decay_horizon"] == 252 + assert alpha.meta["min_warmup_bars"] == 1 + assert alpha.meta["notes"].strip() + + +# ------------------------------------------------- (b) numeric correctness + + +@pytest.mark.parametrize("alpha_id", sorted(_EXPECTED_META)) +def test_compute_matches_independent_numpy_math( + registry: Registry, alpha_id: str +) -> None: + panel = _build_panel() + result = registry.compute(alpha_id, panel) + expected = _expected_outputs(panel)[alpha_id] + + assert list(result.columns) == _CODES + assert list(result.index) == list(_DATES) + np.testing.assert_allclose( + result.to_numpy(dtype=np.float64), + expected.to_numpy(dtype=np.float64), + rtol=1e-9, + atol=1e-12, + equal_nan=True, + err_msg=f"{alpha_id} diverged from independent numpy expectation", + ) + + +def test_pb_sign_direction(registry: Registry) -> None: + """Low PB scores high: cheaper-on-book names rank above expensive ones.""" + out = registry.compute("fund_pb", _build_panel()) + pb = _build_panel()["val:pb"] + for date in _DATES: + assert pb.loc[date, "AAA"] > pb.loc[date, "BBB"] > pb.loc[date, "CCC"] + assert out.loc[date, "CCC"] > out.loc[date, "BBB"] > out.loc[date, "AAA"] + + +def test_dividend_yield_sign_direction(registry: Registry) -> None: + """High dv_ttm scores high (no inversion).""" + out = registry.compute("fund_dividend_yield", _build_panel()) + for date in _DATES: + assert out.loc[date, "CCC"] > out.loc[date, "BBB"] > out.loc[date, "AAA"] + + +# ------------------------------------------------------------ (c) NaN contract + + +@pytest.mark.parametrize("alpha_id", sorted(_BLANKED_DEPENDENCY)) +def test_blank_dependency_bar_yields_all_nan( + registry: Registry, alpha_id: str +) -> None: + """Blanking a declared val: dependency on one bar must NaN that whole bar. + + The registry masks output wherever a declared dependency is missing + (#1377), and zscore propagates NaN — the second date must come back + all-NaN while the first date is untouched. + """ + panel = _build_panel() + panel[_BLANKED_DEPENDENCY[alpha_id]].loc[_DATES[1]] = np.nan + + out = registry.compute(alpha_id, panel) + + assert out.loc[_DATES[1]].isna().all(), ( + f"{alpha_id}: emitted a value on a bar whose declared dependency was NaN" + ) + assert out.loc[_DATES[0]].notna().all() + + +# ------------------------------------------- (d) no inf on degenerate inputs + + +@pytest.mark.parametrize("alpha_id", sorted(_EXPECTED_META)) +def test_extreme_and_zero_values_never_produce_inf( + registry: Registry, alpha_id: str +) -> None: + """Invalid multiples stay NaN; zero dividends remain valid observations.""" + panel = _build_panel() + dep = _BLANKED_DEPENDENCY[alpha_id] + panel[dep].loc[_DATES[0]] = [0.0, 0.0, 0.0] + panel[dep].loc[_DATES[1]] = [0.0, -1.5, 1e12] + + if alpha_id in {"fund_pb", "fund_ps_ttm"}: + from src.factors.registry import RegistryError + with pytest.raises(RegistryError, match=">95% NaN"): + registry.compute(alpha_id, panel) + out = _load_module(_EXPECTED_META[alpha_id][2]).compute(panel) + assert out.isna().all().all() # one valid multiple cannot form a cross-section + else: + out = registry.compute(alpha_id, panel) + assert out.loc[_DATES[1]].notna().all() + assert not np.isinf(out.to_numpy(dtype=np.float64)).any() + assert out.loc[_DATES[0]].isna().all(), "constant row must z-score to NaN" + + +# ------------------------------------------------- runner injection (mocked) + + +class _FakeDailyBasicApi: + """Mimics ``pro.daily_basic``: serves only 000001.SZ, one date short.""" + + def __init__(self) -> None: + self.calls: list[dict[str, Any]] = [] + + def daily_basic(self, **kwargs: Any) -> pd.DataFrame: + self.calls.append(kwargs) + if kwargs["ts_code"] != "000001.SZ": + return pd.DataFrame() + return pd.DataFrame( + { + "ts_code": ["000001.SZ", "000001.SZ"], + "trade_date": ["20240102", "20240103"], + "pb": [1.1, 1.2], + "ps_ttm": [2.1, 2.2], + "dv_ttm": [3.1, 3.2], + } + ) + + +@pytest.fixture +def fake_tushare(monkeypatch: pytest.MonkeyPatch) -> _FakeDailyBasicApi: + api = _FakeDailyBasicApi() + monkeypatch.setenv("TUSHARE_TOKEN", "ts-test-token") + monkeypatch.setitem( + sys.modules, "tushare", SimpleNamespace(pro_api=lambda token="": api) + ) + return api + + +def _cn_data_map() -> dict[str, pd.DataFrame]: + index = pd.to_datetime(["2024-01-02", "2024-01-03", "2024-01-04"]) + frame = lambda: pd.DataFrame( # noqa: E731 + { + "open": [10.0, 10.5, 10.2], + "high": [10.6, 10.7, 10.8], + "low": [9.9, 10.1, 10.0], + "close": [10.1, 10.3, 10.4], + "volume": [1000, 1100, 1200], + }, + index=index, + ) + return {"000001.SZ": frame(), "600000.SH": frame()} + + +def test_runner_injects_declared_val_columns_for_cn_symbols( + fake_tushare: _FakeDailyBasicApi, +) -> None: + index = pd.to_datetime(["2024-01-02", "2024-01-03", "2024-01-04"]) + result = runner._maybe_inject_fundamentals_for_factor_panel( + _cn_data_map(), + { + "start_date": "2024-01-02", + "end_date": "2024-01-04", + "selected_factors": [{"columns_required": ["close", "val:pb"]}], + }, + ) + + injected = result["000001.SZ"]["val:pb"] + assert injected.loc[index[0]] == pytest.approx(1.1) + assert injected.loc[index[1]] == pytest.approx(1.2) + # 2024-01-04 was not served by daily_basic: stays NaN, never fabricated. + assert pd.isna(injected.loc[index[2]]) + # 600000.SH was not served either: whole column NaN, still injected. + assert result["600000.SH"]["val:pb"].isna().all() + assert [call["ts_code"] for call in fake_tushare.calls] == [ + "000001.SZ", + "600000.SH", + ] + assert fake_tushare.calls[0]["fields"] == "ts_code,trade_date,pb" + + +def test_runner_val_injection_skips_non_cn_symbols( + fake_tushare: _FakeDailyBasicApi, +) -> None: + index = pd.to_datetime(["2024-01-02", "2024-01-03", "2024-01-04"]) + data_map = { + "AAPL.US": _cn_data_map()["000001.SZ"], + "00700.HK": _cn_data_map()["000001.SZ"], + } + + result = runner._maybe_inject_fundamentals_for_factor_panel( + data_map, + { + "start_date": "2024-01-02", + "end_date": "2024-01-04", + "selected_factors": [{"columns_required": ["val:dv_ttm"]}], + }, + ) + + assert fake_tushare.calls == [] + assert result["AAPL.US"]["val:dv_ttm"].reindex(index).isna().all() + assert result["00700.HK"]["val:dv_ttm"].reindex(index).isna().all() + + +def test_runner_rejects_val_field_outside_whitelist() -> None: + with pytest.raises(ValueError, match="unsupported val: field"): + runner._maybe_inject_fundamentals_for_factor_panel( + _cn_data_map(), + { + "start_date": "2024-01-02", + "end_date": "2024-01-04", + "selected_factors": [{"columns_required": ["val:total_mv"]}], + }, + ) + + +def test_val_whitelist_covers_exactly_the_daily_basic_fields() -> None: + from backtest.loaders.valuation_loader import VALUATION_FIELDS + + assert VALUATION_FIELDS == ("pe", "pe_ttm", "pb", "ps_ttm", "dv_ttm") + + +def test_missing_token_degrades_to_nan_frames( + monkeypatch: pytest.MonkeyPatch, +) -> None: + """No TUSHARE_TOKEN: the run survives on NaN frames instead of crashing.""" + monkeypatch.setenv("TUSHARE_TOKEN", "") + index = pd.to_datetime(["2024-01-02", "2024-01-03", "2024-01-04"]) + panel = {"close": pd.DataFrame({"000001.SZ": [10.1, 10.3, 10.4]}, index=index)} + + runner._inject_valuation_panel( + panel, + symbols=["000001.SZ"], + val_columns=["val:pb"], + start="2024-01-02", + end="2024-01-04", + ) + + assert panel["val:pb"].isna().all().all() + assert list(panel["val:pb"].index) == list(index) + +@pytest.mark.parametrize('stem,column', [('pb', 'pb'), ('ps_ttm', 'ps_ttm')]) +def test_nonpositive_multiples_are_not_ranked_as_cheap(stem, column): + frame = pd.DataFrame([[-1., 0., 1., 2., 3.]], columns=list('ABCDE')) + actual = _load_module(stem).compute({f'val:{column}': frame}) + assert actual[['A', 'B']].isna().all().all() + np.testing.assert_allclose(actual[['C', 'D', 'E']].to_numpy(), [[1., 0., -1.]]) diff --git a/agent/tests/factors/test_missing_bar_fabrication.py b/agent/tests/factors/test_missing_bar_fabrication.py index 2a3d5242..5ee2c459 100644 --- a/agent/tests/factors/test_missing_bar_fabrication.py +++ b/agent/tests/factors/test_missing_bar_fabrication.py @@ -93,8 +93,35 @@ def _base() -> dict[str, np.ndarray]: volume = rng.integers(1000, 50000, (_T, _N)).astype(float) vwap = (high + low + 2 * close) / 4.0 base = {"close": close, "open": open_, "high": high, "low": low, "volume": volume, "vwap": vwap} - # Drawn last so the price arrays above stay what they were. - for name in ("fund:asset_growth", "fund:net_income", "fund:shares_diluted", "fund:gross_profitability", "fund:roe"): + # Drawn last so the price arrays above stay what they were. The original + # five keep their draw order (identical values); the remaining schema + # fields are appended so every fundamental alpha's declared dependencies + # are present and the oracles actually exercise it instead of SkipAlpha. + for name in ( + "fund:asset_growth", + "fund:net_income", + "fund:shares_diluted", + "fund:gross_profitability", + "fund:roe", + "fund:revenue", + "fund:cogs", + "fund:gross_profit", + "fund:operating_income", + "fund:total_assets", + "fund:total_equity", + "fund:total_debt", + "fund:cash", + "fund:cfo", + "fund:capex", + "fund:roa", + "fund:accruals", + "fund:leverage", + "fund:revenue_growth", + "fund:earnings_growth", + "val:pb", + "val:ps_ttm", + "val:dv_ttm", + ): base[name] = rng.integers(32, 96, (_T, _N)) / 64.0 return base diff --git a/agent/tests/grounding_parameter_helpers.py b/agent/tests/grounding_parameter_helpers.py new file mode 100644 index 00000000..5d76b086 --- /dev/null +++ b/agent/tests/grounding_parameter_helpers.py @@ -0,0 +1,6 @@ +"""Explicit chosen coefficients for derivation fixtures.""" + + +def declared_parameter(value: str, label: str = "Chosen coefficient") -> str: + """Show a parameter to the reader and declare the same value and label.""" + return f"\n\n{label}: {value}.\n\n```figures\n{value} | count | {label}\n```\n" diff --git a/agent/tests/quantlib/test_drawdown_revisited_peak.py b/agent/tests/quantlib/test_drawdown_revisited_peak.py new file mode 100644 index 00000000..446d9b8d --- /dev/null +++ b/agent/tests/quantlib/test_drawdown_revisited_peak.py @@ -0,0 +1,60 @@ +"""A recovered high-water mark starts a new underwater episode.""" + +from __future__ import annotations + +import json + +import pandas as pd +import pytest + +from src.quantlib.risk import max_drawdown_analysis +from src.tools.quantlib_tool import QuantlibCallTool + + +@pytest.mark.parametrize( + "values,peak,trough,recovery", + [ + ([100, 90, 100, 80, 100], 2, 3, 4), + ([100, 100, 80, 100], 1, 2, 3), + ([100, 80, 100, 70], 2, 3, None), + ([90, 100, 80, 110], 1, 2, 3), + ([100, 70, 100, 80], 0, 1, 2), + ([100, 100, 100], 0, 0, 0), + ], +) +def test_episode_starts_at_last_high_water_mark_before_worst_trough( + values, peak, trough, recovery +): + result = max_drawdown_analysis(values) + assert result["peak_date"] == peak + assert result["trough_date"] == trough + assert result["recovery_date"] == recovery + assert result["peak_to_trough_periods"] == trough - peak + assert result["max_drawdown"] == pytest.approx(1 - values[trough] / values[peak]) + + +def test_calendar_underwater_duration_excludes_an_already_recovered_episode(): + equity = pd.Series( + [100, 90, 100, 80, 100], + index=pd.to_datetime( + ["2025-01-01", "2025-01-02", "2025-01-10", "2025-01-12", "2025-01-15"] + ), + ) + result = max_drawdown_analysis(equity) + assert result["peak_date"] == pd.Timestamp("2025-01-10") + assert result["underwater_days"] == 2 + assert result["recovery_days"] == 3 + + +def test_registered_quantlib_tool_reports_the_current_drawdown_episode(): + result = json.loads( + QuantlibCallTool().execute( + action="call", + module="risk", + function="max_drawdown_analysis", + kwargs={"equity": [100, 90, 100, 80, 100]}, + ) + ) + assert result["ok"] is True + assert result["result"]["peak_date"] == 2 + assert result["result"]["peak_to_trough_periods"] == 1 diff --git a/agent/tests/quantlib/valuation/test_threestatement.py b/agent/tests/quantlib/valuation/test_threestatement.py index 410f10cf..d7f7db7a 100644 --- a/agent/tests/quantlib/valuation/test_threestatement.py +++ b/agent/tests/quantlib/valuation/test_threestatement.py @@ -374,6 +374,51 @@ def test_project_three_statement_raises_on_broken_opening_balance_sheet(): assert excinfo.value.residual == pytest.approx(-1.0, abs=TOL) +def test_balance_tolerance_is_honored_for_projected_periods(): + """A caller-widened balance_tolerance must apply to every projected period, not just the opening. + + The rollforwards preserve the residual exactly: expanding assets - liabilities - equity for period t + with cash_end = prior_cash + NI + D&A - ΔNWC - capex + draw - repay - div, ppe_end = prior_ppe + capex - D&A, + RE_end = prior_RE + NI - div, debt_end = prior_debt + draw - repay, every term except nwc_end - ΔNWC = prior_nwc + cancels, leaving residual_t ≡ residual_0. So a residual the caller accepts at period 0 is transported unchanged + into period 1 and then rejected if _project_period ignores the tolerance. + + This test builds an opening sheet with a known 0.5 residual (residual/assets = 9.09e-4), which passes at + tolerance=1e-3 but would fail at the default 1e-9. Without the fix, period 1 raises BalanceSheetError even + though the residual is exactly conserved. + """ + opening_with_residual = { + "revenue": 1000.0, + "cash": 100.0, + "net_working_capital": 50.0, + "ppe": 400.0, + "revolver_balance": 200.0, + "paid_in_capital": 200.0, + "retained_earnings": 149.5, # Makes residual = 100+50+400 - (200+200+149.5) = 0.5 + } + drivers_one_period = { + "revenue_growth": [0.05], + "gross_margin": [0.30], + "opex_pct_revenue": [0.20], + "capex_pct_revenue": [0.06], + "nwc_pct_revenue": [0.10], + "tax_rate": [0.25], + "dividend_payout_ratio": [0.20], + "depreciation_amortization": [30.0], + "interest_rate": [0.08], + "minimum_cash": [50.0], + } + # At tolerance=1e-3, both opening (residual/assets = 0.5/550 ≈ 9.09e-4) and period 1 should pass. + result = project_three_statement(opening_with_residual, drivers_one_period, balance_tolerance=1e-3) + assert len(result.periods) == 1 + # Verify the residual is exactly conserved in period 1. + bs1 = result.periods[0].balance_sheet + assets1 = bs1.cash + bs1.net_working_capital + bs1.ppe + liab_eq1 = bs1.revolver_balance + bs1.paid_in_capital + bs1.retained_earnings + residual1 = assets1 - liab_eq1 + assert residual1 == pytest.approx(0.5, abs=1e-9) + + # -------------------------------------------------------------------------- # Interest/revolver circularity: convergent (multi-iteration) and divergent. # -------------------------------------------------------------------------- diff --git a/agent/tests/test_agent_grounding.py b/agent/tests/test_agent_grounding.py index 9dfb80e6..868b76c8 100644 --- a/agent/tests/test_agent_grounding.py +++ b/agent/tests/test_agent_grounding.py @@ -1522,9 +1522,9 @@ def test_price_validation_ignores_symbol_date_and_quantity_digits(tmp_path: Path for draft in ( "000543.SZ 截至 8 月 3 日收盘价 8.20 CNY（source: tencent）", - "000543.SZ 买入价 8.20 CNY（100 股成本 820 CNY；source: tencent）\n\n" + "000543.SZ 买入价 8.20 CNY（股数100，成本 820 CNY；source: tencent）\n\n" "```figures\n8.20 | observed | close | prices\n" - "820 | derived | 100 × 8.20 | prices\n```", + "820 | derived | 100 × 8.20 | prices\n100 | count | 股数\n```", "000543.SZ 建议持有 1–4 周，买入价 8.20 CNY（source: tencent）", ): result = ledger.validate_final_answer(draft) @@ -1729,9 +1729,9 @@ def test_screening_run_reaches_a_final_answer_through_the_agent_loop( assert market.calls == 1 validation = agent._grounding.validate_final_answer( - "000543.SZ 买入价 8.20 CNY（100 股成本 820 CNY；source: tencent）\n\n" + "000543.SZ 买入价 8.20 CNY（股数100，成本 820 CNY；source: tencent）\n\n" "```figures\n8.20 | observed | close | prices\n" - "820 | derived | 100 × 8.20 | prices\n```" + "820 | derived | 100 × 8.20 | prices\n100 | count | 股数\n```" ) assert validation.valid is True, validation.issues @@ -1825,9 +1825,9 @@ def test_plan_level_alone_reaches_a_valid_answer(tmp_path: Path) -> None: assert inside_range.valid is True, inside_range.issues derived_outside = ledger.validate_final_answer( - "000543.SZ 收盘价 8.20 CNY（source: tencent）。目标位 9.02。\n\n" + "000543.SZ 收盘价 8.20 CNY（source: tencent）。目标位 9.02。选定系数110%。\n\n" "```figures\n8.20 | observed | close | prices\n" - "9.02 | proposed | 8.20 × 1.10 | prices\n```" + "9.02 | proposed | 8.20 × 1.10 | prices\n110% | count | 选定系数\n```" ) assert derived_outside.valid is True, derived_outside.issues diff --git a/agent/tests/test_alpha_bench_pit_lag.py b/agent/tests/test_alpha_bench_pit_lag.py new file mode 100644 index 00000000..1b06d109 --- /dev/null +++ b/agent/tests/test_alpha_bench_pit_lag.py @@ -0,0 +1,120 @@ +"""Regression: after-close SEC filings must not leak into the IC one bar early. + +SEC ``filed`` has no acceptance time. A filing on a trading date may be +published after its close. Same-day IC alignment can therefore include a +market reaction that preceded availability of the factor. + +The known-answer check below plants a factor equal to the next day's return +(the strongest possible leak). Before the fix the bench reports a near-perfect +IC; with the filed-on-t value made first visible on bar t+1 the IC collapses +to ~0. These tests are offline: the loader is mocked, exactly like +``test_fundamental_factor_dependencies_are_loaded_at_filing_time``. +""" + +from __future__ import annotations + +from types import SimpleNamespace +from unittest.mock import Mock + +import numpy as np +import pandas as pd + +from src.factors.factor_analysis_core import compute_ic_series +from src.tools.alpha_bench_tool import _compute_forward_returns, _prepare_bench_panel + +_DATES = pd.to_datetime(["2026-01-05", "2026-01-06", "2026-01-07", "2026-01-08", "2026-01-09", "2026-01-12"]) + + +def _close_panel() -> pd.DataFrame: + """Eight deterministic closes; cross-sectional IC needs >= 5 symbols. + + The series is the seed-73 draw from ``default_rng(73).normal(0, 0.02)`` + compounded from 100 — chosen so consecutive-day cross-sectional return + ranks are uncorrelated (|lag-IC| <= 0.12 every day), keeping the + known-answer test about the filing lag, not about the fixture. + """ + return pd.DataFrame( + { + "AAPL.US": [100.00, 97.81, 95.38, 98.83, 99.43, 96.99], + "MSFT.US": [100.00, 101.56, 103.82, 102.25, 99.69, 98.64], + "NVDA.US": [100.00, 99.24, 96.04, 92.59, 93.58, 91.74], + "AMZN.US": [100.00, 100.59, 97.40, 98.70, 99.25, 98.15], + "GOOG.US": [100.00, 100.67, 98.98, 99.41, 102.24, 104.99], + "META.US": [100.00, 96.11, 98.81, 99.49, 98.70, 100.09], + "TSLA.US": [100.00, 101.00, 102.87, 100.62, 101.13, 101.96], + "AMD.US": [100.00, 98.07, 97.74, 98.84, 100.68, 101.34], + }, + index=_DATES, + ) + + +def _leak_frame(close: pd.DataFrame) -> pd.DataFrame: + """A fundamental whose value on bar t is exactly the t->t+1 return. + + This is what a same-day ``filed`` value looks like when the filing is + accepted after the close: the value encodes information the market has + not priced in yet at close(t). + """ + return close.pct_change(fill_method=None).shift(-1) + + +def _registry(columns: list[str]) -> SimpleNamespace: + return SimpleNamespace(get=lambda _aid: SimpleNamespace(meta={"columns_required": columns})) + + +def _ic(factor: pd.DataFrame, close: pd.DataFrame) -> float: + """Mean cross-sectional Spearman IC between the factor and forward returns.""" + returns = _compute_forward_returns({"close": close}) + ic_series = compute_ic_series(factor, returns) + assert not ic_series.empty + return float(ic_series.mean()) + + +def test_unshifted_pit_panel_would_report_a_perfect_ic() -> None: + """The known-answer sanity check: without the lag the leak is real. + + Guards against someone removing the shift in ``_prepare_bench_panel`` + without noticing it re-opens issue #1790. + """ + close = _close_panel() + + ic = _ic(_leak_frame(close), close) + + assert abs(ic) > 0.9 + + +def test_pit_panel_shifts_filing_one_bar_later(monkeypatch) -> None: + """A filed-on-t value must first appear on bar t+1 in the bench panel.""" + close = _close_panel() + # What the loader hands back for a filing anchored on bar 2: NaN before + # the filing, forward-filled after it (fundamentals_loader L194 ffills). + frame = pd.DataFrame(np.nan, index=close.index, columns=close.columns) + frame.iloc[2] = 5.0 + frame = frame.ffill() + loader = Mock(return_value={"net_income": frame}) + monkeypatch.setattr("backtest.loaders.fundamentals_loader.load_fundamental_panel", loader) + + result = _prepare_bench_panel(_registry(["close", "fund:net_income"]), ["aid"], {"close": close}, "sp500", "2026-2026") + + column = result["fund:net_income"] + # Before the filing bar and on it: NaN. From t+1 on: the filed value. + assert column.iloc[2].isna().all() + assert (column.iloc[3:] == 5.0).all().all() + # The lag is recorded so downstream consumers can see the semantics. + assert result["_meta"]["fundamental_pit_lag_bars"] == 1 + + +def test_after_close_filing_does_not_leak_into_the_ic(monkeypatch) -> None: + """Known-answer: a factor equal to next-day returns must not produce IC. + + This is the reproduce step from the issue, planted offline: the loader + returns a value anchored on the filed date (as for an after-close + filing); after the fix the bench must not report a signal. + """ + close = _close_panel() + loader = Mock(return_value={"net_income": _leak_frame(close)}) + monkeypatch.setattr("backtest.loaders.fundamentals_loader.load_fundamental_panel", loader) + + result = _prepare_bench_panel(_registry(["close", "fund:net_income"]), ["aid"], {"close": close}, "sp500", "2026-2026") + + assert abs(_ic(result["fund:net_income"], close)) < 0.1 diff --git a/agent/tests/test_attribution_liquidated_last_row.py b/agent/tests/test_attribution_liquidated_last_row.py new file mode 100644 index 00000000..78a9c315 --- /dev/null +++ b/agent/tests/test_attribution_liquidated_last_row.py @@ -0,0 +1,113 @@ +"""A run's final positions row is its liquidation, not its book. + +Every run exits on the last bar (``end_of_backtest``), so ``positions.csv`` +ends with an all-zero row. Reading only that row made Brinson report a +portfolio that had been invested for years as having "all position weights +zero" -- a one-row problem that nulled the whole section. +""" + +from __future__ import annotations + +from pathlib import Path +from typing import Any + +from src.api.attribution_core import _attribution_load_position_weights + +SYMBOLS = ["AAA.HK", "BBB.HK"] + + +def _write_positions(run_dir: Path, rows: list[list[Any]]) -> None: + """Write ``timestamp`` + per-symbol weight columns, the real artifact layout.""" + path = run_dir / "artifacts" / "positions.csv" + path.parent.mkdir(parents=True, exist_ok=True) + lines = ["timestamp," + ",".join(SYMBOLS)] + lines += [",".join(str(cell) for cell in row) for row in rows] + path.write_text("\n".join(lines) + "\n", encoding="utf-8") + + +def test_end_of_backtest_flat_row_is_skipped(tmp_path: Path) -> None: + """The held row before the liquidation is the one that describes the book.""" + _write_positions( + tmp_path, + [ + ["2026-09-23", 0.40, 0.35], + ["2026-09-24", 0.42, 0.33], + ["2026-09-25", 0.0, 0.0], # end-of-backtest liquidation + ], + ) + notes: list[str] = [] + + weights = _attribution_load_position_weights(tmp_path, notes) + + assert weights == {"AAA.HK": 0.42, "BBB.HK": 0.33} + assert any("end-of-backtest liquidation (2026-09-24)" in note for note in notes) + + +def test_a_held_final_row_is_used_unchanged(tmp_path: Path) -> None: + """A run still holding on its final bar must read exactly that row.""" + _write_positions( + tmp_path, + [ + ["2026-09-24", 0.42, 0.33], + ["2026-09-25", 0.50, 0.20], + ], + ) + notes: list[str] = [] + + weights = _attribution_load_position_weights(tmp_path, notes) + + assert weights == {"AAA.HK": 0.50, "BBB.HK": 0.20} + assert notes == [] + + +def test_a_book_that_never_held_still_reads_flat(tmp_path: Path) -> None: + """True negative: all-zero throughout must stay all-zero. + + This is the case the downstream zero-check was written for, and the fix + must not paper over it. + """ + _write_positions( + tmp_path, + [ + ["2026-09-23", 0.0, 0.0], + ["2026-09-24", 0.0, 0.0], + ["2026-09-25", 0.0, 0.0], + ], + ) + notes: list[str] = [] + + weights = _attribution_load_position_weights(tmp_path, notes) + + assert weights == {"AAA.HK": 0.0, "BBB.HK": 0.0} + assert not any("end-of-backtest" in note for note in notes) + + +def test_a_book_flat_before_the_close_is_not_replaced_by_an_older_one( + tmp_path: Path, +) -> None: + """Only the liquidation row is stepped over, never an earlier exit.""" + _write_positions( + tmp_path, + [ + ["2026-09-22", 0.40, 0.35], + ["2026-09-23", 0.0, 0.0], # the strategy itself exited here + ["2026-09-24", 0.0, 0.0], + ["2026-09-25", 0.0, 0.0], # end-of-backtest liquidation + ], + ) + notes: list[str] = [] + + weights = _attribution_load_position_weights(tmp_path, notes) + + assert weights == {"AAA.HK": 0.0, "BBB.HK": 0.0} + assert not any("end-of-backtest" in note for note in notes) + + +def test_missing_artifact_still_reports_not_found(tmp_path: Path) -> None: + """The other skip reasons must keep their own wording.""" + notes: list[str] = [] + + weights = _attribution_load_position_weights(tmp_path, notes) + + assert weights is None + assert notes == ["brinson attribution skipped: positions.csv not found"] diff --git a/agent/tests/test_base_engine.py b/agent/tests/test_base_engine.py index f7e3df66..d16ab8be 100644 --- a/agent/tests/test_base_engine.py +++ b/agent/tests/test_base_engine.py @@ -6,7 +6,7 @@ Uses ChinaAEngine as a concrete implementation since BaseEngine is abstract. from __future__ import annotations import json -from dataclasses import FrozenInstanceError +from dataclasses import FrozenInstanceError, replace import numpy as np import pandas as pd @@ -239,6 +239,101 @@ def test_fill_delta_artifact_is_jsonl_and_exposes_weighted_holding(tmp_path): assert exits["holding_bars"].tolist() == pytest.approx([1.5, 1.5]) +def test_trades_csv_entry_row_is_the_opening_fill_after_a_scale_in(tmp_path): + """The entry row must be a trade that happened, not the running average cost. + + A scale-in moves the position's weighted entry price, but the entry row is + stamped with the first open's date, so pairing the two would describe a + fill that never took place. Each round trip still writes exactly two rows. + """ + from src.strategy_discovery.run_artifacts import read_trade_activity + + engine = _AdjustmentEngine() + weights = {"A": [0.25, 0.50, 0.0, 0.25, 0.0]} + execution_prices = {"A": [100.0, 120.0, 130.0, 110.0, 115.0]} + _run_adjustments(engine, weights, execution_prices=execution_prices) + dates = pd.date_range("2026-01-02", periods=5) + prices = pd.DataFrame( + {"open": execution_prices["A"], "close": execution_prices["A"]}, index=dates + ) + equity = pd.Series( + [snapshot.equity for snapshot in engine.equity_snapshots], index=dates + ) + engine._write_artifacts( + tmp_path, + {"A": prices}, + dates, + equity, + pd.Series(1_000.0, index=dates), + pd.Series(0.0, index=dates), + pd.DataFrame(weights, index=dates), + {}, + ["A"], + ) + + assert [fill.action for fill in engine.fill_records] == [ + "open", + "increase", + "close", + "open", + "close", + ] + first_open, _, _, second_open, _ = engine.fill_records + assert engine.trades[0].entry_price != pytest.approx(first_open.execution_price) + + path = tmp_path / "artifacts" / "trades.csv" + trades = pd.read_csv(path) + assert len(trades) == 2 * len(engine.trades) == 4 + entries = trades[trades["event"] == "entry"] + assert entries["timestamp"].tolist() == [ + str(first_open.timestamp.date()), + str(second_open.timestamp.date()), + ] + assert entries["price"].tolist() == pytest.approx( + [first_open.execution_price, second_open.execution_price] + ) + assert entries["qty"].tolist() == pytest.approx( + [abs(first_open.signed_quantity), abs(second_open.signed_quantity)] + ) + assert entries["reason"].tolist() == ["signal", "signal"] + + exit_dates, max_concurrent = read_trade_activity(path) + assert [str(day) for day in exit_dates] == [ + str(trade.exit_time.date()) for trade in engine.trades + ] + assert max_concurrent == 1 + + +def test_trades_csv_entry_row_falls_back_to_the_trade_without_an_opening_fill( + tmp_path, +): + """A position that carries no opening record keeps the trade's own fields.""" + engine = _AdjustmentEngine() + engine.positions["A"] = _position(size=5.0) + engine._close_position("A", 110.0, _ADJUST_TS, "signal") + dates = pd.DatetimeIndex([_ENTRY_TS, _ADJUST_TS]) + engine._write_artifacts( + tmp_path, + {}, + dates, + pd.Series(1_000.0, index=dates), + pd.Series(1_000.0, index=dates), + pd.Series(0.0, index=dates), + pd.DataFrame({"A": [1.0, 0.0]}, index=dates), + {}, + ["A"], + ) + + trades = pd.read_csv(tmp_path / "artifacts" / "trades.csv") + assert len(trades) == 2 + entry = trades.iloc[0] + assert (entry["timestamp"], entry["price"], entry["qty"]) == ( + str(_ENTRY_TS.date()), + 100.0, + 5.0, + ) + + def test_partial_reduction_allocates_nonzero_entry_and_exit_fees(): engine = _AdjustmentEngine(fee_rate=0.01) _run_adjustments(engine, {"A": [0.50, 0.20]}) @@ -269,7 +364,9 @@ def test_rebalance_rejects_same_direction_adjustment_with_changed_leverage(): with pytest.raises(ValueError, match="leverage"): _run_adjustments(engine, {"A": [0.25, 0.50]}) assert engine.capital == 750.0 - assert engine.bar_positions == [{"A": _position(size=2.5)}] + assert engine.bar_positions == [ + {"A": replace(_position(size=2.5), open_price=100.0, open_size=2.5)} + ] def test_rebalance_reduces_short_with_correct_signed_pnl(): diff --git a/agent/tests/test_br_data.py b/agent/tests/test_br_data.py new file mode 100644 index 00000000..35db070b --- /dev/null +++ b/agent/tests/test_br_data.py @@ -0,0 +1,334 @@ +"""Brazilian official datasets: parsers, store, loader and agent tool. + +Fixtures are synthetic files in the published layouts (B3 COTAHIST fixed +width, CVM ``;`` CSVs, BCB JSON, Tesouro CSV). No network. +""" + +from __future__ import annotations + +import io +import json +import zipfile +from datetime import date + +import pytest + +from src.br_data import cotahist, cvm_companies, cvm_funds, tesouro +from src.br_data._download import br_date, to_float +from src.br_data.store import connect, init_db +from tests.loader_contract import assert_loader_contract + + +def _cotahist_line(day: str, ticker: str, tpmerc: int, close_cents: int, *, + bdi: str = "02", factor: int = 1, qty: int = 1000) -> str: + """Build one 245-char COTAHIST type-01 record.""" + fields = [ + "01", day, bdi.ljust(2), ticker.ljust(12), f"{tpmerc:03d}", "PETROBRAS".ljust(12), + "PN N2".ljust(10), " " * 3, "R$".ljust(4), + f"{close_cents - 10:013d}", f"{close_cents + 20:013d}", f"{close_cents - 30:013d}", + f"{close_cents:013d}", f"{close_cents:013d}", f"{close_cents - 1:013d}", f"{close_cents + 1:013d}", + f"{150:05d}", f"{qty:018d}", f"{qty * close_cents:018d}", f"{0:013d}", "0", "99991231", + f"{factor:07d}", f"{0:013d}", "BRPETRACNPR6", "123", + ] + line = "".join(fields) + assert len(line) == 245, len(line) + return line + + +def _zip(name: str, text: str, encoding: str = "latin-1") -> bytes: + buf = io.BytesIO() + with zipfile.ZipFile(buf, "w") as archive: + archive.writestr(name, text.encode(encoding)) + return buf.getvalue() + + +@pytest.fixture() +def store(tmp_path, monkeypatch): + path = tmp_path / "br.db" + monkeypatch.setenv("VIBE_BR_DATA_DB", str(path)) + from src.config import accessor + + if hasattr(accessor, "reset_env_config"): + accessor.reset_env_config() + monkeypatch.setattr("src.br_data.store.db_path", lambda: path) + init_db(path) + return path + + +def _cotahist_body() -> str: + lines = [ + "00COTAHIST.2025BOVESPA 20250103" + " " * 214, + _cotahist_line("20250102", "PETR4", 10, 3712), + _cotahist_line("20250103", "PETR4", 10, 3750), + _cotahist_line("20250103", "PETR4F", 20, 3751, bdi="96"), + _cotahist_line("20250103", "XPTO3", 10, 150000, factor=1000), + "99COTAHIST.2025BOVESPA 20250103" + " " * 214, + ] + return "\n".join(lines) + + +def test_cotahist_record_is_parsed_with_implied_decimals_and_factor() -> None: + records = list(cotahist.parse_text(_cotahist_body())) + assert len(records) == 4 + first = dict(zip(cotahist._COLUMNS, records[0])) + assert first["trade_date"] == "2025-01-02" + assert first["ticker"] == "PETR4" + assert first["market_type"] == 10 + assert first["close"] == pytest.approx(37.12) + assert first["high"] == pytest.approx(37.32) + assert first["volume"] == 1000 + assert first["turnover"] == pytest.approx(37120.0) + assert first["expiry"] is None + assert first["isin"] == "BRPETRACNPR6" + per_thousand = dict(zip(cotahist._COLUMNS, records[3])) + assert per_thousand["close"] == pytest.approx(1.5) + + +def test_cotahist_sync_downloads_years_then_daily_files(store, monkeypatch) -> None: + calls: list[str] = [] + yearly = _zip("COTAHIST_A2025.TXT", _cotahist_body()) + daily = _zip("COTAHIST_D06012025.TXT", _cotahist_line("20250106", "PETR4", 10, 3800)) + + def fake_fetch(url: str, host_key: str): + calls.append(url.rsplit("/", 1)[-1]) + if url.endswith("COTAHIST_A2025.ZIP"): + return yearly + if url.endswith("COTAHIST_D06012025.ZIP"): + return daily + return None + + monkeypatch.setattr(cotahist, "fetch_bytes", fake_fetch) + with connect(store, write=True) as conn: + rows = cotahist.sync(conn, 2024, date(2025, 1, 7)) + assert rows == 5 + assert calls[:2] == ["COTAHIST_A2024.ZIP", "COTAHIST_A2025.ZIP"] + assert "COTAHIST_D06012025.ZIP" in calls and "COTAHIST_D07012025.ZIP" in calls + + from backtest.loaders.b3_cotahist_loader import DataLoader, map_symbol + + assert map_symbol("PETR4F.SA") == ("PETR4F", 20) + loader = DataLoader() + assert loader.is_available() + frames = loader.fetch(["PETR4.SA", "PETR4F.SA", "NONE3.SA"], "2025-01-01", "2025-01-31") + assert set(frames) == {"PETR4.SA", "PETR4F.SA"} + petr = frames["PETR4.SA"] + assert_loader_contract(petr, context="b3_cotahist PETR4.SA") + assert list(petr["close"]) == pytest.approx([37.12, 37.50, 38.00]) + assert petr.attrs["quote_currency"] == "BRL" + + +def test_cvm_fund_daily_accepts_both_layouts(store) -> None: + new_layout = ( + "TP_FUNDO_CLASSE;CNPJ_FUNDO_CLASSE;ID_SUBCLASSE;DT_COMPTC;VL_TOTAL;VL_QUOTA;" + "VL_PATRIM_LIQ;CAPTC_DIA;RESG_DIA;NR_COTST\n" + "CLASSES - FIF;00.017.024/0001-53;;2025-01-02;100.5;1.2345;1000000.10;0;10.5;42\n" + "CLASSES - FIF;00.017.024/0001-53;;2025-01-03;100.5;1.2400;1000100.10;5;0;43\n" + ) + old_layout = ( + "TP_FUNDO;CNPJ_FUNDO;DT_COMPTC;VL_TOTAL;VL_QUOTA;VL_PATRIM_LIQ;CAPTC_DIA;RESG_DIA;NR_COTST\n" + "FI;11.111.111/0001-11;2020-06-30;1;2,5;3;0;0;7\n" + ) + with connect(store, write=True) as conn: + assert cvm_funds.ingest_daily_zip(conn, _zip("inf_diario_fi_202501.csv", new_layout)) == 2 + assert cvm_funds.ingest_daily_zip(conn, _zip("inf_diario_fi_2020.csv", old_layout)) == 1 + register = ( + "TP_FUNDO;CNPJ_FUNDO;DENOM_SOCIAL;SIT;CLASSE;GESTOR;ADMIN;RENTAB_FUNDO;DT_INI_ATIV\n" + "FI;00.017.024/0001-53;FUNDO TESTE RENDA FIXA;EM FUNCIONAMENTO NORMAL;Renda Fixa;" + "GESTORA X;ADMIN Y;CDI;2001-01-01\n" + ) + conn.executemany(cvm_funds._UPSERT_FUND, list(cvm_funds.parse_register(register))) + with connect(store) as conn: + row = conn.execute("SELECT * FROM cvm_fund_daily WHERE cnpj='00017024000153' ORDER BY trade_date").fetchall() + assert [r["quota"] for r in row] == [1.2345, 1.24] + assert row[1]["holders"] == 43 + old = conn.execute("SELECT quota FROM cvm_fund_daily WHERE cnpj='11111111000111'").fetchone() + assert old["quota"] == 2.5 + + +def test_cvm_fund_monthly_sync_marks_only_settled_months(store, monkeypatch) -> None: + body = "CNPJ_FUNDO_CLASSE;DT_COMPTC;VL_QUOTA\n00.000.000/0001-91;2025-01-02;1\n" + seen: list[str] = [] + + def fake_fetch(url: str, host_key: str): + seen.append(url.rsplit("/", 1)[-1]) + return _zip("x.csv", body) if "inf_diario_fi_2025" in url else None + + monkeypatch.setattr(cvm_funds, "fetch_bytes", fake_fetch) + with connect(store, write=True) as conn: + cvm_funds.sync_daily(conn, 2025, date(2025, 3, 10)) + done = {r[0] for r in conn.execute("SELECT item FROM sync_log WHERE source='cvm_fund_daily'")} + assert done == {"202501"} + assert seen == ["inf_diario_fi_202501.zip", "inf_diario_fi_202502.zip", "inf_diario_fi_202503.zip"] + + +def test_cvm_statements_keep_latest_version_and_scale(store) -> None: + header = ("CNPJ_CIA;DT_REFER;VERSAO;DENOM_CIA;CD_CVM;GRUPO_DFP;MOEDA;ESCALA_MOEDA;ORDEM_EXERC;" + "DT_INI_EXERC;DT_FIM_EXERC;CD_CONTA;DS_CONTA;VL_CONTA;ST_CONTA_FIXA\n") + body = header + ( + "33.000.167/0001-01;2024-12-31;1;PETROBRAS;9512;DF Consolidado;REAL;MIL;ÚLTIMO;2024-01-01;2024-12-31;3.01;Receita;490000000;S\n" + "33.000.167/0001-01;2024-12-31;1;PETROBRAS;9512;DF Consolidado;REAL;MIL;PENÚLTIMO;2023-01-01;2023-12-31;3.01;Receita;510000000;S\n" + "33.000.167/0001-01;2024-12-31;2;PETROBRAS;9512;DF Consolidado;REAL;MIL;ÚLTIMO;2024-01-01;2024-12-31;3.01;Receita;491000000;S\n" + ) + with connect(store, write=True) as conn: + rows = cvm_companies.ingest_statements_zip( + conn, _zip("dfp_cia_aberta_DRE_con_2024.csv", body, "latin-1"), "DFP") + assert rows == 2 + conn.executemany("INSERT OR REPLACE INTO cvm_company_tickers VALUES (?, ?)", + list(cvm_companies.parse_tickers( + "CNPJ_Companhia;Codigo_Negociacao\n33.000.167/0001-01;PETR4\n"))) + with connect(store) as conn: + stored = conn.execute("SELECT statement, value, version FROM cvm_statements").fetchall() + assert [(r["statement"], r["value"], r["version"]) for r in stored] == [("DRE", 491000000000.0, 2)] + + +def test_tesouro_and_number_helpers() -> None: + body = ("Tipo Titulo;Data Vencimento;Data Base;Taxa Compra Manha;Taxa Venda Manha;" + "PU Compra Manha;PU Venda Manha;PU Base Manha\n" + "Tesouro IPCA+;15/05/2035;02/01/2025;7,12;7,24;2.345,67;2.300,01;2.299,99\n") + rows = list(tesouro.parse(body)) + assert rows == [("Tesouro IPCA+", "2035-05-15", "2025-01-02", 7.12, 7.24, 2345.67, 2300.01, 2299.99)] + assert to_float("1.234,5") == 1234.5 and to_float("12.5") == 12.5 and to_float("") is None + assert br_date("31/12/2024") == "2024-12-31" and br_date("2024-12-31") == "2024-12-31" + + +def test_agent_tool_reads_the_store(store, monkeypatch) -> None: + monkeypatch.setattr(cotahist, "fetch_bytes", + lambda url, host_key: _zip("a.txt", _cotahist_body()) if "A2025" in url else None) + with connect(store, write=True) as conn: + cotahist.sync(conn, 2025, date(2025, 1, 3)) + conn.execute("INSERT INTO bcb_series VALUES (12, '2025-01-02', 0.045)") + conn.execute("INSERT INTO cvm_companies VALUES ('33000167000101','9512','PETROLEO BRASILEIRO','PETROBRAS','ATIVO','Petróleo')") + conn.execute("INSERT INTO cvm_company_tickers VALUES ('PETR4','33000167000101')") + + from src.tools.br_market_data_tool import BrMarketDataTool + + tool = BrMarketDataTool() + quotes = json.loads(tool.execute(action="quotes", tickers=["PETR4.SA"], start_date="2025-01-01", end_date="2025-01-31")) + assert quotes["ok"] and [r["close"] for r in quotes["data"]["rows"]] == pytest.approx([37.12, 37.50]) + macro = json.loads(tool.execute(action="macro", series="cdi", start_date="2025-01-01", end_date="2025-01-31")) + assert macro["data"]["rows"] == [{"date": "2025-01-02", "value": 0.045}] + found = json.loads(tool.execute(action="search", query="petrobras")) + assert found["data"]["companies"][0]["tickers"] == "PETR4" + assert "PETR4" in {r["ticker"] for r in found["data"]["tickers"]} + assert not json.loads(tool.execute(action="fund", cnpj="123"))["ok"] + assert not json.loads(tool.execute(action="nope"))["ok"] + status = json.loads(tool.execute(action="status")) + assert status["data"]["rows"][0]["rows"] == 4 + + +def test_financial_statements_read_cvm_store_then_fall_back_to_yahoo(store, monkeypatch) -> None: + from src.tools import financial_statements_tool as fst + + with connect(store, write=True) as conn: + conn.execute("INSERT INTO cvm_company_tickers VALUES ('PETR4', '33000167000101')") + conn.executemany( + "INSERT INTO cvm_statements VALUES (?, 'DFP', 'DRE', ?, ?, ?, ?, ?, 1)", + [ + ("33000167000101", "2024-12-31", "2024-01-01", "3.01", "Receita", 491e9), + ("33000167000101", "2023-12-31", "2023-01-01", "3.01", "Receita", 511e9), + ], + ) + calls: list[str] = [] + monkeypatch.setattr(fst, "_fetch_yahoo_statement", + lambda code, **kw: calls.append(code) or {"periods": [{"endDate": "2024"}]}) + + income = json.loads(fst.FinancialStatementsTool().execute(code="PETR4.SA", statement="income")) + assert income["ok"] and income["source"] == "cvm" and income["market"] == "br" + periods = income["data"]["PETR4.SA"]["periods"] + assert [p["period_end"] for p in periods] == ["2024-12-31", "2023-12-31"] + assert periods[0]["3.01 Receita"] == 491e9 + assert calls == [] + + # Not synced for this company / no indicator block in CVM -> Yahoo. + other = json.loads(fst.FinancialStatementsTool().execute(code="VALE3.SA", statement="income")) + indicators = json.loads(fst.FinancialStatementsTool().execute(code="PETR4.SA")) + assert other["source"] == "yahoo" and indicators["source"] == "yahoo" + assert calls == ["VALE3.SA", "PETR4.SA"] + + +def test_stock_news_routes_b3_symbols_to_yahoo_with_full_symbol(monkeypatch) -> None: + from src.tools import stock_news_tool as snt + + queries: list[str] = [] + monkeypatch.setattr(snt, "_fetch_yahoo_news", lambda query, limit: queries.append(query) or []) + out = json.loads(snt.StockNewsTool().execute(code="petr4.sa", scope="stock")) + assert out["ok"] and out["market"] == "br" and out["source"] == "yahoo" + assert queries == ["PETR4.SA"] + +@pytest.mark.parametrize('currency', ['CR$', 'US$', '']) +def test_cotahist_does_not_relabel_other_currencies_as_brl(currency): + line = _cotahist_line('20250103', 'TEST3', 10, 1000) + line = line[:52] + currency.ljust(4) + line[56:] + assert cotahist.parse_record(line) is None + + +def test_cotahist_rejects_missing_prices_and_quotation_factor(): + line = _cotahist_line('20250103', 'TEST3', 10, 1000) + assert cotahist.parse_record(line[:108] + ' ' * 13 + line[121:]) is None + assert cotahist.parse_record(line[:210] + '0000000' + line[217:]) is None + + +def test_bad_numeric_and_calendar_cells_are_not_fabricated(): + assert to_float('NaN') is None + assert to_float('Infinity') is None + assert br_date('2025-02-31') is None + + +def test_fund_subclasses_do_not_form_one_return_series(store): + from src.tools.br_market_data_tool import BrMarketDataTool + with connect(store, write=True) as conn: + conn.executemany( + 'INSERT INTO cvm_fund_daily (cnpj, trade_date, subclass, quota) VALUES (?, ?, ?, ?)', + [('00000000000001', '2025-01-02', 'A', 1), + ('00000000000001', '2025-01-03', 'B', 100)], + ) + result = json.loads(BrMarketDataTool().execute(action='fund', cnpj='00000000000001', + start_date='2025-01-01', end_date='2025-01-04')) + assert result['ok'] + assert len(result['data']['rows']) == 2 + assert 'quota_return' not in result['data'] + + +def test_br_tool_rejects_bad_dates_and_preserves_result_limit(): + from src.tools.br_market_data_tool import _dates, _envelope, TOOL_RESULT_LIMIT + with pytest.raises(ValueError): + _dates({'start_date': '2025-03-01', 'end_date': '2025-02-01'}) + result = _envelope('search', [], {'companies': [{'name': 'x' * (TOOL_RESULT_LIMIT * 2)}]}) + assert len(result) <= TOOL_RESULT_LIMIT + assert json.loads(result)['ok'] is False + + +def test_br_quotes_keep_per_symbol_brl_evidence(store, tmp_path): + from src.agent.grounding import GroundingLedger + from src.tools.br_market_data_tool import BrMarketDataTool + with connect(store, write=True) as conn: + cotahist.ingest_zip(conn, _zip('quotes.txt', '\n'.join([ + _cotahist_line('20250103', 'TEST3', 10, 1000), + _cotahist_line('20250103', 'MOCK4', 10, 2000), + ]))) + args = {'action': 'quotes', 'tickers': ['TEST3', 'MOCK4'], + 'start_date': '2025-01-03', 'end_date': '2025-01-03'} + payload = BrMarketDataTool().execute(**args) + ledger = GroundingLedger(run_dir=tmp_path, user_message='Compare TEST3.SA and MOCK4.SA prices') + ledger.ingest_tool_result(tool_name='get_br_market_data', arguments=args, + result=payload, call_id='b3', success=True) + closes = [r for r in ledger._evidence if r.field == 'close'] + assert {(r.symbol, r.value, r.currency, r.timestamp, r.tool) for r in closes} == { + ('TEST3.SA', 10., 'BRL', '2025-01-03', 'get_br_market_data'), + ('MOCK4.SA', 20., 'BRL', '2025-01-03', 'get_br_market_data'), + } + assert ledger.validate_final_answer('Source: b3_cotahist. TEST3.SA closed at BRL 10.00. MOCK4.SA closed at BRL 20.00.').valid + assert not ledger.validate_final_answer('Source: b3_cotahist. TEST3.SA closed at BRL 20.00.').valid + malformed = json.loads(payload) + malformed['data']['currency'] = 'USD' + refused = GroundingLedger(run_dir=tmp_path/'refused', user_message='TEST3.SA price') + refused.ingest_tool_result(tool_name='get_br_market_data', arguments=args, + result=json.dumps(malformed), call_id='bad', success=True) + assert not refused._evidence + + +def test_brl_alias_is_unambiguous(): + from src.agent.grounding import GroundingLedger + assert GroundingLedger._currency_is_surfaced("BRL", "R$ 10.00") + assert GroundingLedger._currency_is_surfaced("BRL", "10 reais") + assert not GroundingLedger._currency_is_surfaced("BRL", "These are real observed prices") diff --git a/agent/tests/test_brazil_market_routing.py b/agent/tests/test_brazil_market_routing.py new file mode 100644 index 00000000..9b083898 --- /dev/null +++ b/agent/tests/test_brazil_market_routing.py @@ -0,0 +1,90 @@ +"""Brazil (B3 .SA) routing and execution rules. + +Deterministic, no network: pins the routing contract and the B3 cost model. +""" + +from __future__ import annotations + +import pytest + +from backtest.benchmark import MARKET_BENCHMARKS, _infer_market +from backtest.engines._market_hooks import _detect_market, code_currency +from backtest.engines.brazil_equity import BrazilEquityEngine +from backtest.engines.composite import _build_rule_engines +from backtest.loaders.base import normalize_declared_quote_currency +from backtest.loaders.registry import FALLBACK_CHAINS, price_caliber +from backtest.loaders.yahoo_loader import _is_supported as yahoo_is_supported +from backtest.loaders.yfinance_loader import _to_yfinance_symbol +from backtest.runner import _create_market_engine, _detect_source +from src.market_data import detect_source +from src.tools.stock_profile_tool import _market_for + + +@pytest.mark.parametrize("symbol", ["PETR4.SA", "VALE3.SA", "BOVA11.SA", "HGLG11.SA", "AAPL34.SA", "PETR4F.SA"]) +def test_sa_symbols_share_one_market_data_contract(symbol: str) -> None: + assert _detect_market(symbol) == "br_equity" + assert code_currency(symbol) == "BRL" + assert detect_source(symbol) == "yahoo" + assert _detect_source(symbol) == "yahoo" + assert yahoo_is_supported(symbol) + assert _to_yfinance_symbol(symbol) == symbol + assert _market_for(symbol) == "br" + + +def test_brazil_chain_is_yahoo_then_official_files() -> None: + assert FALLBACK_CHAINS["br_equity"] == ["yahoo", "yfinance", "b3_cotahist", "local"] + assert price_caliber("b3_cotahist", "br_equity") == "raw" + + +def test_brazil_benchmark_is_ibovespa_etf() -> None: + assert _infer_market(["PETR4.SA"], "yahoo") == "br_equity" + assert MARKET_BENCHMARKS["br_equity"] == "BOVA11.SA" + + +def test_sa_lines_must_be_declared_in_brl() -> None: + import pandas as pd + + frame = pd.DataFrame({"close": [10.0]}) + assert normalize_declared_quote_currency(frame, "PETR4.SA", "BRL").attrs["quote_currency"] == "BRL" + with pytest.raises(ValueError, match="BRL"): + normalize_declared_quote_currency(frame, "PETR4.SA", "USD") + + +def test_runner_and_composite_use_the_b3_engine() -> None: + assert isinstance(_create_market_engine("yahoo", {}, ["PETR4.SA"]), BrazilEquityEngine) + engines = _build_rule_engines({}, ["PETR4.SA", "VALE3.SA"]) + assert isinstance(engines["br_equity"], BrazilEquityEngine) + + +class TestBrazilEquityEngine: + def test_costs_are_b3_fees_bilateral(self) -> None: + engine = BrazilEquityEngine({}) + # 0.005% trading + 0.025% settlement on R$10,000. + assert engine.calc_commission(100, 100.0, 1, True) == pytest.approx(3.0) + assert engine.calc_commission(100, 100.0, 1, False) == pytest.approx(3.0) + assert engine.calc_commission(0, 100.0, 1, True) == 0.0 + + def test_broker_commission_is_configurable(self) -> None: + engine = BrazilEquityEngine({"br_brokerage_rate": 0.001, "br_brokerage_fixed": 4.9}) + assert engine.calc_commission(100, 100.0, 1, True) == pytest.approx(3.0 + 10.0 + 4.9) + + def test_lots_default_to_one_share_or_round_lots(self) -> None: + assert BrazilEquityEngine({}).round_size(157.9, 30.0) == 157.0 + assert BrazilEquityEngine({"br_lot_size": 100}).round_size(157.9, 30.0) == 100.0 + assert BrazilEquityEngine({}).round_size(-3, 30.0) == 0.0 + + def test_slippage_snaps_to_cent_against_trader(self) -> None: + engine = BrazilEquityEngine({"slippage_br": 0.0}) + assert engine.apply_slippage(10.001, 1) == 10.01 + assert engine.apply_slippage(10.009, -1) == 10.0 + engine = BrazilEquityEngine({"slippage_br": 0.001}) + assert engine.apply_slippage(10.0, 1) == 10.01 + assert engine.apply_slippage(10.0, -1) == 9.99 + + def test_short_selling_can_be_disabled(self) -> None: + import pandas as pd + + bar = pd.Series({"close": 10.0}) + assert BrazilEquityEngine({}).can_execute("PETR4.SA", -1, bar) + assert not BrazilEquityEngine({"allow_short": False}).can_execute("PETR4.SA", -1, bar) + assert BrazilEquityEngine({"allow_short": False}).can_execute("PETR4.SA", 1, bar) diff --git a/agent/tests/test_channels_attempt_progress.py b/agent/tests/test_channels_attempt_progress.py new file mode 100644 index 00000000..ad77cf94 --- /dev/null +++ b/agent/tests/test_channels_attempt_progress.py @@ -0,0 +1,90 @@ +"""Session events reach IM during a run, without leaking unchecked answers.""" +import asyncio +from unittest.mock import Mock + +from src.channels.bus.events import InboundMessage +from src.channels.bus.queue import MessageBus +from src.channels.runtime import ChannelRuntime +from src.channels.manager import ChannelManager +from src.session.events import EventBus +from src.session.models import Message, Session + + +class _Service: + def __init__(self): + self.event_bus = EventBus() + self.messages = [] + + def create_session(self, **kwargs): + return Session(session_id='s', **kwargs) + + async def send_message(self, session_id, content, **kwargs): + for attempt, kind, data in [ + ('old', 'tool_call', {'tool': 'stale_tool'}), + ('a', 'attempt.created', {}), + ('a', 'reasoning_delta', {'delta': 'Let me check the source.'}), + ('a', 'tool_call', {'tool': 'get_market_data'}), + ('a', 'text_delta', {'delta': 'UNVALIDATED 987.65'}), + ('a', 'tool_result', {'result': 'RAW_PAYLOAD'}), + ]: + self.event_bus.emit(session_id, kind, {**data, 'attempt_id': attempt}) + return {'attempt_id': 'a'} + + def get_messages(self, *args, **kwargs): + return self.messages + + +def test_progress_is_delivered_before_reply_and_drafts_stay_private(tmp_path): + async def scenario(): + bus, service = MessageBus(), _Service() + runtime = ChannelRuntime(bus=bus, session_service=service, manager=None, + session_map_path=tmp_path/'sessions.json', poll_interval_s=.001) + msg = InboundMessage(channel='feishu', chat_id='oc_chat', sender_id='u', content='research', + metadata={'message_id': 'm', 'chat_type': 'group', '_wants_stream': True}) + task = asyncio.create_task(runtime._handle_inbound(msg)) + first = await asyncio.wait_for(bus.consume_outbound(), 1) + assert first.metadata['_stream_delta'] + assert not task.done() # status actually arrives while work is in flight + service.messages.append(Message(session_id='s', role='assistant', content='Checked answer', linked_attempt_id='a')) + await task + messages = [first] + while bus.outbound_size: + messages.append(await bus.consume_outbound()) + contents = ''.join(m.content for m in messages) + assert 'Using get_market_data' in contents + assert 'stale_tool' not in contents and 'UNVALIDATED' not in contents and 'RAW_PAYLOAD' not in contents + assert contents.count('Checked answer') == 1 + assert any(m.metadata.get('_reasoning_end') for m in messages) + assert any(m.metadata.get('_stream_end') for m in messages) + assert all(m.metadata.get('message_id') == 'm' for m in messages) + assert messages[-1].metadata['chat_type'] == 'group' + # Exercise real manager dispatch into a Feishu adapter's CardKit path. + from src.channels.feishu import FeishuChannel, FeishuConfig + channel = FeishuChannel(FeishuConfig(), bus) + channel._client = object() + channel._create_streaming_card_sync = Mock(return_value='card-1') + channel._stream_update_text_with_reopen_sync = Mock(return_value=(True, 2)) + channel._close_streaming_mode_sync = Mock(return_value=True) + await ChannelManager._send_once(channel, first) + channel._create_streaming_card_sync.assert_called_once() + assert channel._stream_bufs['m'].text == first.content + end = next(m for m in messages if m.metadata.get('_stream_end')) + await ChannelManager._send_once(channel, end) + channel._close_streaming_mode_sync.assert_called_once() + assert not channel._stream_bufs + asyncio.run(scenario()) + + +def test_timeout_closes_transient_stream(tmp_path): + async def scenario(): + bus, service = MessageBus(), _Service() + runtime = ChannelRuntime(bus=bus, session_service=service, manager=None, + session_map_path=tmp_path/'sessions.json', + poll_interval_s=.001, reply_timeout_s=.01) + await runtime._handle_inbound(InboundMessage(channel='feishu', chat_id='c', sender_id='u', content='q', metadata={'_wants_stream': True})) + messages = [] + while bus.outbound_size: + messages.append(await bus.consume_outbound()) + assert any(m.metadata.get('_stream_end') for m in messages) + assert messages[-1].metadata['error'] + asyncio.run(scenario()) diff --git a/agent/tests/test_factor_gate_fund_prefix.py b/agent/tests/test_factor_gate_fund_prefix.py index 8bddf4f7..79298c9a 100644 --- a/agent/tests/test_factor_gate_fund_prefix.py +++ b/agent/tests/test_factor_gate_fund_prefix.py @@ -40,11 +40,30 @@ def test_panel_column_gate_accepts_unknown_fund_field() -> None: assert meta.columns_required == ["fund:whatever"] +def test_panel_column_gate_accepts_val_prefix() -> None: + meta = AlphaMeta(**_meta(columns_required=["close", "val:pb"])) + + assert meta.columns_required == ["close", "val:pb"] + + +def test_panel_column_gate_accepts_unknown_val_field() -> None: + # The gate is prefix-based like fund:; the daily_basic whitelist is + # enforced at injection time, not at metadata validation. + meta = AlphaMeta(**_meta(columns_required=["val:whatever"])) + + assert meta.columns_required == ["val:whatever"] + + def test_panel_column_gate_rejects_unknown_non_prefixed_column() -> None: with pytest.raises(ValueError, match="unknown panel column: garbage"): AlphaMeta(**_meta(columns_required=["garbage"])) +def test_panel_column_gate_rejects_unknown_prefix() -> None: + with pytest.raises(ValueError, match="unknown panel column: est:eps"): + AlphaMeta(**_meta(columns_required=["est:eps"])) + + def test_decay_horizon_accepts_annual_report_scale_window() -> None: meta = AlphaMeta(**_meta(decay_horizon=400)) diff --git a/agent/tests/test_fundamental_schema.py b/agent/tests/test_fundamental_schema.py index e742a0d3..bbf4a32a 100644 --- a/agent/tests/test_fundamental_schema.py +++ b/agent/tests/test_fundamental_schema.py @@ -64,11 +64,16 @@ def test_sec_revenue_fixture_shapes_cover_new_and_old_standard() -> None: assert "RevenueFromContractWithCustomerExcludingAssessedTax" not in old_concepts +def test_shares_diluted_does_not_alias_basic_shares() -> None: + assert "WeightedAverageNumberOfSharesOutstandingBasic" not in SEC_CONCEPT_MAP["shares_diluted"] + + def test_derived_formulas_are_numerically_correct() -> None: idx = pd.to_datetime(["2022-12-31", "2023-12-31"]) data = { "gross_profit": pd.Series([40.0, 60.0], index=idx), "net_income": pd.Series([10.0, 15.0], index=idx), + "revenue": pd.Series([100.0, 120.0], index=idx), "total_assets": pd.Series([200.0, 250.0], index=idx), "total_equity": pd.Series([50.0, 75.0], index=idx), "total_debt": pd.Series([100.0, 125.0], index=idx), @@ -81,6 +86,8 @@ def test_derived_formulas_are_numerically_correct() -> None: "gross_profitability": pd.Series([0.2, 0.24], index=idx), "accruals": pd.Series([0.015, 0.012], index=idx), "leverage": pd.Series([2.0, 125.0 / 75.0], index=idx), + "revenue_growth": pd.Series([float("nan"), 0.2], index=idx), + "earnings_growth": pd.Series([float("nan"), 0.5], index=idx), } for field, expected_series in expected.items(): @@ -118,6 +125,65 @@ def test_asset_growth_reads_a_year_ahead_on_a_daily_densified_panel() -> None: assert result.loc["2025-06-25", "AAPL"] == pytest.approx(0.10) +def test_revenue_growth_uses_annual_period_over_period_semantics() -> None: + idx = pd.to_datetime(["2021-12-31", "2022-12-31", "2023-12-31"]) + data = {"revenue": pd.Series([100.0, 120.0, 90.0], index=idx)} + + result = DERIVED_FIELDS["revenue_growth"]["compute"](data) + expected = pd.Series([float("nan"), 0.2, -0.25], index=idx) + + pd.testing.assert_series_equal(result, expected, check_names=False) + + +def test_earnings_growth_is_sign_safe_with_negative_prior_year() -> None: + idx = pd.to_datetime(["2022-12-31", "2023-12-31"]) + data = {"net_income": pd.Series([-10.0, 5.0], index=idx)} + + result = DERIVED_FIELDS["earnings_growth"]["compute"](data) + expected = pd.Series([float("nan"), 1.5], index=idx) + + pd.testing.assert_series_equal(result, expected, check_names=False) + + +def test_growth_fields_zero_or_missing_lag_is_nan() -> None: + idx = pd.to_datetime(["2021-12-31", "2022-12-31", "2023-12-31"]) + net_income = pd.Series([0.0, 8.0, 4.0], index=idx) + revenue = pd.Series([float("nan"), 100.0, 110.0], index=idx) + + earnings_growth = DERIVED_FIELDS["earnings_growth"]["compute"]({"net_income": net_income}) + assert pd.isna(earnings_growth.loc["2021-12-31"]) + assert pd.isna(earnings_growth.loc["2022-12-31"]) + assert earnings_growth.loc["2023-12-31"] == pytest.approx(-0.5) + + revenue_growth = DERIVED_FIELDS["revenue_growth"]["compute"]({"revenue": revenue}) + assert pd.isna(revenue_growth.loc["2021-12-31"]) + assert pd.isna(revenue_growth.loc["2022-12-31"]) + assert revenue_growth.loc["2023-12-31"] == pytest.approx(0.10) + + +def test_growth_fields_read_a_year_ahead_on_a_daily_densified_panel() -> None: + # Same daily-densification contract as the asset_growth panel test: a + # row-positional shift(1) would compare yesterday to today, so the YoY + # reading must hold across the whole year following a mid-year step. + dates = pd.date_range("2024-01-01", "2025-12-31", freq="D") + revenue = pd.Series(100.0, index=dates) + revenue.loc["2024-07-01":] = 110.0 + net_income = pd.Series(-10.0, index=dates) + net_income.loc["2024-07-01":] = 5.0 + + revenue_growth = DERIVED_FIELDS["revenue_growth"]["compute"]( + {"revenue": pd.DataFrame({"AAPL": revenue})} + ) + earnings_growth = DERIVED_FIELDS["earnings_growth"]["compute"]( + {"net_income": pd.DataFrame({"AAPL": net_income})} + ) + + assert revenue_growth.loc["2025-01-15", "AAPL"] == pytest.approx(0.10) + assert revenue_growth.loc["2025-06-25", "AAPL"] == pytest.approx(0.10) + assert earnings_growth.loc["2025-01-15", "AAPL"] == pytest.approx(1.5) + assert earnings_growth.loc["2025-06-25", "AAPL"] == pytest.approx(1.5) + + def test_resolve_field_and_list_supported_fields() -> None: kind, raw_spec = resolve_field("revenue") assert kind == "raw" diff --git a/agent/tests/test_fundamentals_shares_diluted_alias.py b/agent/tests/test_fundamentals_shares_diluted_alias.py new file mode 100644 index 00000000..33ebfb07 --- /dev/null +++ b/agent/tests/test_fundamentals_shares_diluted_alias.py @@ -0,0 +1,123 @@ +"""Basic shares cannot silently fill a missing diluted-share observation.""" + +from __future__ import annotations + + +import pandas as pd +import pytest + +from backtest.loaders import fundamentals_loader + +def _facts(concept_rows: dict[str, list[dict[str, object]]]) -> dict[str, object]: + return { + "facts": { + "us-gaap": { + concept: {"units": {"shares": rows}} + for concept, rows in concept_rows.items() + } + } + } + + +def _share_row(end: str, filed: str, value: float, *, form: str = "10-Q") -> dict[str, object]: + start = (pd.Timestamp(end) - pd.Timedelta(days=91)).strftime("%Y-%m-%d") + return {"start": start, "end": end, "filed": filed, "val": value, "form": form} + + +def _patch_sec( + monkeypatch: pytest.MonkeyPatch, + facts_by_symbol: dict[str, dict[str, object]], +) -> None: + def cik_for(symbol: str) -> str | None: + return f"CIK-{symbol}" if symbol in facts_by_symbol else None + + def get_company_facts(cik: str) -> dict[str, object]: + symbol = cik.removeprefix("CIK-") + return facts_by_symbol[symbol] + + monkeypatch.setattr(fundamentals_loader.sec_edgar_client, "cik_for", cik_for) + monkeypatch.setattr( + fundamentals_loader.sec_edgar_client, + "get_company_facts", + get_company_facts, + ) + + +def _load_shares( + monkeypatch: pytest.MonkeyPatch, + facts: dict[str, object], + start: str = "2017-05-01", + end: str = "2017-05-10", +) -> pd.Series: + _patch_sec(monkeypatch, {"AAA": facts}) + index = pd.date_range(start, end, freq="D") + panel = fundamentals_loader.load_fundamental_panel( + ["AAA"], + ["shares_diluted"], + start, + end, + freq="quarterly", + index=index, + ) + return panel["shares_diluted"]["AAA"] + + +def test_basic_count_does_not_fill_missing_diluted_shares(monkeypatch): + facts = _facts({"WeightedAverageNumberOfSharesOutstandingBasic": [ + _share_row("2017-03-31", "2017-05-03", 1000.0) + ]}) + # Even equal rounded EPS is not proof that share counts are identical. + facts["facts"]["us-gaap"].update({ + name: {"units": {"USD/shares": [_share_row("2017-03-31", "2017-05-03", 1.0)]}} + for name in ("EarningsPerShareBasic", "EarningsPerShareDiluted") + }) + assert _load_shares(monkeypatch, facts).isna().all() + + +def test_diluted_concept_wins_over_basic_from_the_same_filing( + monkeypatch: pytest.MonkeyPatch, +) -> None: + facts = _facts( + { + "WeightedAverageNumberOfDilutedSharesOutstanding": [ + _share_row("2017-03-31", "2017-05-03", 1000.0) + ], + "WeightedAverageNumberOfSharesOutstandingBasic": [ + _share_row("2017-03-31", "2017-05-03", 990.0) + ], + } + ) + + shares = _load_shares(monkeypatch, facts) + + assert shares.loc["2017-05-10"] == 1000.0 + + +def test_diluted_shares_only_appear_on_their_own_filing_date(monkeypatch): + facts = _facts( + { + "WeightedAverageNumberOfSharesOutstandingBasic": [ + _share_row("2017-03-31", "2017-05-03", 990.0) + ], + "WeightedAverageNumberOfDilutedSharesOutstanding": [ + _share_row("2017-03-31", "2017-08-03", 1000.0) + ], + } + ) + + shares = _load_shares(monkeypatch, facts, "2017-05-01", "2017-08-10") + + assert pd.isna(shares.loc["2017-05-03"]) + assert pd.isna(shares.loc["2017-08-02"]) + assert shares.loc["2017-08-03"] == 1000.0 + assert shares.loc["2017-08-10"] == 1000.0 + + +def test_no_share_concept_at_all_stays_nan( + monkeypatch: pytest.MonkeyPatch, +) -> None: + facts = _facts({"Assets": [_share_row("2017-03-31", "2017-05-03", 1000.0)]}) + + shares = _load_shares(monkeypatch, facts) + + assert shares.isna().all() diff --git a/agent/tests/test_grounding_figures_contract.py b/agent/tests/test_grounding_figures_contract.py index 55c41889..87680d7f 100644 --- a/agent/tests/test_grounding_figures_contract.py +++ b/agent/tests/test_grounding_figures_contract.py @@ -26,6 +26,7 @@ from pathlib import Path import pytest from src.agent.grounding import GroundingLedger +from tests.grounding_parameter_helpers import declared_parameter from src.agent.grounding.figures import ( ROLES, parse_figures_block, @@ -448,13 +449,13 @@ def test_derived_passes_on_its_own_arithmetic_and_fails_without_it( ledger = _ledger(tmp_path) correct = ledger.validate_final_answer( - HDR + " 第一档 0.646 元。" + _block(HDR_ROW, "0.646 | derived | 0.666 × 0.97 | c1") + HDR + declared_parameter("97%") + " 第一档 0.646 元。" + _block(HDR_ROW, "0.646 | derived | 0.666 × 0.97 | c1") ) wrong_result = ledger.validate_final_answer( - HDR + " 第一档 0.700 元。" + _block(HDR_ROW, "0.700 | derived | 0.666 × 0.97 | c1") + HDR + declared_parameter("97%") + " 第一档 0.700 元。" + _block(HDR_ROW, "0.700 | derived | 0.666 × 0.97 | c1") ) unanchored = ledger.validate_final_answer( - HDR + " 第一档 0.646 元。" + _block(HDR_ROW, "0.646 | derived | 0.500 × 1.292 | c1") + HDR + declared_parameter("97%") + " 第一档 0.646 元。" + _block(HDR_ROW, "0.646 | derived | 0.500 × 1.292 | c1") ) assert correct.valid is True, correct.issues @@ -467,7 +468,7 @@ def test_derived_covers_only_the_value_it_declares(tmp_path: Path) -> None: ledger = _ledger(tmp_path) result = ledger.validate_final_answer( - HDR + " 第一档 0.646 元，第二档 0.400 元。" + HDR + declared_parameter("97%") + " 第一档 0.646 元，第二档 0.400 元。" + _block(HDR_ROW, "0.646 | derived | 0.666 × 0.97 | c1", "0.400 | proposed | 第二档") ) @@ -500,7 +501,7 @@ def test_proposed_outside_the_range_passes_with_a_derivation(tmp_path: Path) -> ledger = _ledger(tmp_path) result = ledger.validate_final_answer( - HDR + " 目标价 1.332 元。" + _block(HDR_ROW, "1.332 | proposed | 0.666 × 2 | c1") + HDR + declared_parameter("2") + " 目标价 1.332 元。" + _block(HDR_ROW, "1.332 | proposed | 0.666 × 2 | c1") ) assert result.valid is True, result.issues @@ -714,7 +715,7 @@ def test_the_chinese_prompt_example_passes_the_gate_it_describes(tmp_path: Path) ledger = _ledger(tmp_path) prose = ( "159516.SZ（akshare，CNY）最新收盘 0.666 元，第一档 0.646 元，" - "较 5 月高点回撤 37%，买入参考 0.62 元。" + "折扣系数97%。较 5 月高点回撤 37%，买入参考 0.62 元。" ) result = ledger.validate_final_answer(prose + "\n\n" + _prompt_blocks()[0]) diff --git a/agent/tests/test_grounding_operand_evidence.py b/agent/tests/test_grounding_operand_evidence.py new file mode 100644 index 00000000..dd8bf192 --- /dev/null +++ b/agent/tests/test_grounding_operand_evidence.py @@ -0,0 +1,93 @@ +"""Data operands cannot masquerade as undeclared multiplicative constants.""" + +from __future__ import annotations + +import json +from pathlib import Path + +import pytest + +from src.agent.grounding import GroundingLedger + + +def _ledger(tmp_path: Path, values: tuple[float, ...]) -> GroundingLedger: + ledger = GroundingLedger(run_dir=tmp_path, user_message="Report the calculation") + ledger.ingest_tool_result( + tool_name="measure", + arguments={}, + result=json.dumps({f"amount_{i}": value for i, value in enumerate(values)}), + call_id="measure", + success=True, + ) + return ledger + + +@pytest.mark.parametrize( + ("values", "formula", "result"), + [ + ((125,), "125 / 10000", "1.25%"), + ((10000,), "125 / 10000", "1.25%"), + ((125, 75), "(125 + 75) / 10000", "2.00%"), + ((125,), "125 * 0.0001", "1.25%"), + ], +) +def test_missing_data_operand_is_refused(tmp_path, values, formula, result): + ledger = _ledger(tmp_path, values) + verdict = ledger.validate_final_answer( + f"Result: {result}.\n```figures\n{result} | derived | {formula} | measure\n```" + ) + assert not verdict.valid + assert verdict.issues[0]["reason"] == "operand_not_observed" + + +@pytest.mark.parametrize( + ("formula", "result", "parameters"), + [ + ("125 * 0.97", "121.25", (("0.97", "discount factor"),)), + ("125 * 25%", "31.25", (("25%", "chosen weight"),)), + ("125 * 25%", "31.25", (("25%", "w"),)), + ("125 * 25%", "31.25", (("25%", "الوزن المختار"),)), + ("125 * (1 - 3%)", "121.25", (("3%", "discount rate"),)), + ("(125 + 75) / 2", "100.00", (("2", "sample count"),)), + ], +) +def test_visible_declared_parameters_preserve_ordinary_arithmetic( + tmp_path, formula, result, parameters +): + ledger = _ledger(tmp_path, (125, 75)) + prose = "\n".join(f"{label}: {value}." for value, label in parameters) + rows = "\n".join(f"{value} | count | {label}" for value, label in parameters) + verdict = ledger.validate_final_answer( + f"{prose}\nResult: {result}.\n```figures\n" + f"{result} | derived | {formula} | measure\n{rows}\n```" + ) + assert verdict.valid, verdict.issues + + +def test_fully_observed_ratio_needs_no_parameter_declaration(tmp_path): + verdict = _ledger(tmp_path, (125, 10000)).validate_final_answer( + "Ratio: 1.25%.\n```figures\n1.25% | derived | 125 / 10000 | measure\n```" + ) + assert verdict.valid, verdict.issues + + +@pytest.mark.parametrize( + "prose", + ["", "Unrelated number: 10000.", "```text\nChosen base: 10000.\n```", "Chosen base: $10000."], +) +def test_hidden_or_mislabelled_parameter_cannot_ground_a_denominator(tmp_path, prose): + verdict = _ledger(tmp_path, (125,)).validate_final_answer( + f"{prose}\nRatio: 1.25%.\n```figures\n" + "1.25% | derived | 125 / 10000 | measure\n10000 | count | chosen base\n```" + ) + assert not verdict.valid + assert any(issue.get("reason") == "operand_not_observed" for issue in verdict.issues) + + +def test_a_short_parameter_label_must_not_match_inside_an_unrelated_word(tmp_path): + verdict = _ledger(tmp_path, (125,)).validate_final_answer( + "Unknown number: 10000.\nRatio: 1.25%.\n```figures\n" + "1.25% | derived | 125 / 10000 | measure\n10000 | count | n\n```" + ) + assert not verdict.valid + assert any(issue.get("reason") == "operand_not_observed" for issue in verdict.issues) diff --git a/agent/tests/test_grounding_release.py b/agent/tests/test_grounding_release.py index 696a7f58..d536adfd 100644 --- a/agent/tests/test_grounding_release.py +++ b/agent/tests/test_grounding_release.py @@ -22,6 +22,7 @@ from typing import Any, Callable import pytest from src.agent.grounding import GroundingLedger, ValidationResult +from tests.grounding_parameter_helpers import declared_parameter from src.agent.grounding.evidence import ( _REGISTERED_PRICE_INDICATORS, _is_registered_price_indicator, @@ -199,7 +200,7 @@ def test_a_declared_derivation_is_checked_by_its_own_arithmetic(tmp_path: Path) ledger = _ledger(tmp_path) result = ledger.validate_final_answer( - HDR + " 建议买入价 1.136 元。" + HDR + declared_parameter("97%") + " 建议买入价 1.136 元。" + _block(HDR_ROW, "1.136 | derived | 1.171 × 0.97 | prices") ) @@ -211,7 +212,7 @@ def test_a_derivation_whose_arithmetic_misses_is_rejected(tmp_path: Path) -> Non ledger = _ledger(tmp_path) result = ledger.validate_final_answer( - HDR + " 建议买入价 1.20 元。" + HDR + declared_parameter("97%") + " 建议买入价 1.20 元。" + _block(HDR_ROW, "1.20 | derived | 1.171 × 0.97 | prices") ) @@ -225,7 +226,7 @@ def test_a_derivation_may_take_a_registered_indicator_as_its_input(tmp_path: Pat ledger = _ledger(tmp_path) result = ledger.validate_final_answer( - HDR + " 第一档 1.093 元。" + HDR + declared_parameter("95%") + " 第一档 1.093 元。" + _block(HDR_ROW, "1.093 | derived | 1.150 × 0.95 | indicators") ) @@ -1091,7 +1092,7 @@ def test_a_declaration_covers_its_own_value_and_nothing_beside_it( ledger = _ledger(tmp_path) riding_along = ledger.validate_final_answer( - HDR + " 建议买入价 0.95 元（收盘 1.171 × 0.97 = 1.136 参考）。" + HDR + declared_parameter("97%") + " 建议买入价 0.95 元（收盘 1.171 × 0.97 = 1.136 参考）。" + _block( HDR_ROW, "0.97 | count | 折扣系数", @@ -1099,7 +1100,7 @@ def test_a_declaration_covers_its_own_value_and_nothing_beside_it( ) ) formula_only = ledger.validate_final_answer( - HDR + " 参考价 1.136 元。" + HDR + declared_parameter("97%") + " 参考价 1.136 元。" + _block(HDR_ROW, "1.136 | derived | 1.171 × 0.97 | prices") ) @@ -1454,7 +1455,7 @@ def test_a_declared_derivation_survives_the_cut_beside_a_rejected_figure( """ ledger = _ledger(tmp_path) draft = ( - HDR + " 第一档 1.093 元。建议买入价 0.95 元。" + HDR + declared_parameter("95%") + " 第一档 1.093 元。建议买入价 0.95 元。" + _block( HDR_ROW, "1.093 | derived | 1.150 × 0.95 | indicators", @@ -1685,10 +1686,10 @@ def test_a_derivation_anchored_to_another_symbol_grounds_nothing(tmp_path: Path) ) unattributed = _two_symbol_ledger(tmp_path).validate_final_answer( - header + "买入价 1377.4。" + block + header + "买入价 1377.4。" + block + declared_parameter("97%") ) attributed = _two_symbol_ledger(tmp_path).validate_final_answer( - header + "600519.SH 买入价 1377.4。" + block + header + "600519.SH 买入价 1377.4。" + block + declared_parameter("97%") ) assert "numeric_claim_conflict" in _codes(unattributed) @@ -1786,7 +1787,7 @@ def test_a_restatement_of_a_rejected_figure_is_cut_where_it_stands( """ ledger = _ledger(tmp_path) draft = ( - HDR + " 第一档 1.093 元。\n\n| 档位 | 建议买入价 |\n|---|---|\n| 第一档 | 0.95 |\n" + HDR + declared_parameter("95%") + " 第一档 1.093 元。\n\n| 档位 | 建议买入价 |\n|---|---|\n| 第一档 | 0.95 |\n" + _block(HDR_ROW, "1.093 | derived | 1.150 × 0.95 | indicators", "0.95 | proposed | 系数") ) validation = ledger.validate_final_answer(draft) @@ -2375,6 +2376,7 @@ def test_the_correction_prompt_names_the_figures_to_keep(tmp_path: Path) -> None HDR + "昨日收盘 1.137 元，20 日均线 1.150 元，50 日均线 1.090 元。" + "参考买入价 1.136 元（1.171 × 0.97）。目标价 2.50 元，传闻报价 9.99 元。" + + declared_parameter("97%") + _block( HDR_ROW, "1.137 | observed | close 2026-06-23 | prices", diff --git a/agent/tests/test_grounding_role_hardening.py b/agent/tests/test_grounding_role_hardening.py index 1767c8c6..b063fb48 100644 --- a/agent/tests/test_grounding_role_hardening.py +++ b/agent/tests/test_grounding_role_hardening.py @@ -17,6 +17,7 @@ from typing import Any import pytest from src.agent.grounding import GroundingLedger +from tests.grounding_parameter_helpers import declared_parameter from src.agent.grounding.evidence import _metric_kind_for_path from src.agent.grounding.policies import _note_tokens @@ -232,16 +233,19 @@ def test_an_added_operand_must_be_observed(tmp_path: Path, prose: str, row: str) @pytest.mark.parametrize( - ("prose", "row"), + ("prose", "row", "parameter"), [ - ("第一档 0.646 元。", "0.646 | derived | 0.666 × 0.97 | c1"), - ("第一档 0.646 元。", "0.646 | derived | 0.666 × (1 − 0.03) | c1"), - ("较高点回撤 36.8%。", "36.8% | derived | (0.666 − 1.053) / 1.053 | c1"), - ("中枢 0.860 元。", "0.860 | derived | (0.666 + 1.053) / 2 | c1"), + ("第一档 0.646 元。", "0.646 | derived | 0.666 × 0.97 | c1", "97%"), + ("第一档 0.646 元。", "0.646 | derived | 0.666 × (1 − 0.03) | c1", "3%"), + ("较高点回撤 36.8%。", "36.8% | derived | (0.666 − 1.053) / 1.053 | c1", None), + ("中枢 0.860 元。", "0.860 | derived | (0.666 + 1.053) / 2 | c1", "2"), ], ) -def test_observed_sums_and_free_multipliers_still_derive(tmp_path: Path, prose: str, row: str) -> None: - result = _ledger(tmp_path, MARKET_A).validate_final_answer(HDR + " " + prose + _block(ROW, row)) +def test_observed_sums_and_declared_multipliers_still_derive( + tmp_path: Path, prose: str, row: str, parameter: str | None +) -> None: + chosen = declared_parameter(parameter) if parameter is not None else "" + result = _ledger(tmp_path, MARKET_A).validate_final_answer(HDR + chosen + " " + prose + _block(ROW, row)) assert result.valid is True, result.issues @@ -256,7 +260,7 @@ def test_observed_sums_and_free_multipliers_still_derive(tmp_path: Path, prose: ) def test_a_percentage_factor_and_a_return_ratio_are_not_offsets(tmp_path: Path, prose: str, row: str) -> None: result = _ledger(tmp_path, MARKET_B, message=f"分析 {B}").validate_final_answer( - HDR_B + " " + prose + _block(ROW_B, row) + HDR_B + declared_parameter("3%") + " " + prose + _block(ROW_B, row) ) assert result.valid is True, result.issues @@ -266,8 +270,8 @@ def test_a_derived_result_is_compared_at_the_prose_precision(tmp_path: Path) -> """A declaration written coarser than the prose cannot widen the band.""" ledger = _ledger(tmp_path, MARKET_A) - coarse = ledger.validate_final_answer(HDR + " 目标价 1.00 元。" + _block(ROW, "1 | derived | 0.666 × 0.9 | c1")) - exact = ledger.validate_final_answer(HDR + " 目标价 0.60 元。" + _block(ROW, "0.6 | derived | 0.666 × 0.9 | c1")) + coarse = ledger.validate_final_answer(HDR + declared_parameter("90%") + " 目标价 1.00 元。" + _block(ROW, "1 | derived | 0.666 × 0.9 | c1")) + exact = ledger.validate_final_answer(HDR + declared_parameter("90%") + " 目标价 0.60 元。" + _block(ROW, "0.6 | derived | 0.666 × 0.9 | c1")) assert _reasons(coarse) == ["derivation_result_mismatch"] assert exact.valid is True, exact.issues @@ -357,7 +361,7 @@ def test_a_percent_is_never_a_proposed_level(tmp_path: Path, row: str) -> None: def test_the_same_percent_declared_derived_passes(tmp_path: Path) -> None: result = _ledger(tmp_path, MARKET_A).validate_final_answer( - HDR + " 建议仓位 30%。" + _block(ROW, "30% | derived | 0.666 × 0.45 | c1") + HDR + declared_parameter("45%") + " 建议仓位 30%。" + _block(ROW, "30% | derived | 0.666 × 0.45 | c1") ) assert result.valid is True, result.issues @@ -986,7 +990,7 @@ def test_a_money_figure_is_derived_from_money( ], ) def test_a_price_discount_and_a_drawdown_still_derive(tmp_path: Path, prose: str, row: str) -> None: - result = _ledger(tmp_path, MARKET_A).validate_final_answer(HDR + prose + _block(ROW, row)) + result = _ledger(tmp_path, MARKET_A).validate_final_answer(HDR + declared_parameter("97%") + prose + _block(ROW, row)) assert result.valid is True, result.issues diff --git a/agent/tests/test_grounding_shape_hardening.py b/agent/tests/test_grounding_shape_hardening.py index 467ae465..20f5341e 100644 --- a/agent/tests/test_grounding_shape_hardening.py +++ b/agent/tests/test_grounding_shape_hardening.py @@ -16,6 +16,7 @@ from pathlib import Path import pytest from src.agent.grounding import GroundingLedger +from tests.grounding_parameter_helpers import declared_parameter from src.agent.grounding.figures import Figure, parse_figures_block, scan_figures, strip_figures_block pytestmark = pytest.mark.unit @@ -479,10 +480,10 @@ def test_every_figures_block_declares_and_every_one_is_stripped(tmp_path: Path) assert stripped == HDR + "\n\n第一档 0.646 元。" ledger = _ledger(tmp_path) - passed = ledger.validate_final_answer(HDR + _block(HDR_ROW) + "\n\n第一档 0.646 元。" + _block("0.646 | derived | 0.666 × 0.97 | c1")) - refused = ledger.validate_final_answer(HDR + _block(HDR_ROW) + "\n\n第一档 0.700 元。" + _block("0.700 | derived | 0.666 × 0.97 | c1")) + passed = ledger.validate_final_answer(HDR + declared_parameter("97%") + _block(HDR_ROW) + "\n\n第一档 0.646 元。" + _block("0.646 | derived | 0.666 × 0.97 | c1")) + refused = ledger.validate_final_answer(HDR + declared_parameter("97%") + _block(HDR_ROW) + "\n\n第一档 0.700 元。" + _block("0.700 | derived | 0.666 × 0.97 | c1")) assert passed.valid, passed.issues - assert passed.released_text == HDR + "\n\n第一档 0.646 元。" + assert passed.released_text == HDR + "\n\nChosen coefficient: 97%." + "\n\n第一档 0.646 元。" assert _values(refused) == ["0.700"] assert "```" not in refused.released_text diff --git a/agent/tests/test_loader_contract.py b/agent/tests/test_loader_contract.py index 1c5ba3a9..64bf0d6b 100644 --- a/agent/tests/test_loader_contract.py +++ b/agent/tests/test_loader_contract.py @@ -17,6 +17,7 @@ from tests.loader_contract import loader_contract_errors CONTRACT_SUITES = { "akshare": "test_akshare_loader.py", "alphavantage": "test_alphavantage_loader.py", + "b3_cotahist": "test_br_data.py", "baostock": "test_baostock_loader.py", "binance": "test_binance_fallback.py", "ccxt": "test_ccxt_loader_bounded.py", diff --git a/agent/tests/test_local_connections.py b/agent/tests/test_local_connections.py index 3a373ad5..c888ea5f 100644 --- a/agent/tests/test_local_connections.py +++ b/agent/tests/test_local_connections.py @@ -246,9 +246,9 @@ def test_connection_scoped_overrides_obey_the_connector_allowlist(tmp_path, monk """A vault-backed call must not widen what a caller may override. Every SDK connector narrows overrides on purpose: OKX and Binance both - exclude ``readonly`` ("always true for this layer") and ``timeout``, and - Longbridge's overlay is ``profile``/``region`` only so a caller "cannot mix - or bypass the shared resolver". ``build_config`` enforces that; a config + exclude ``readonly`` and ``timeout``, and Longbridge's overlay is + ``profile``/``region`` only so a caller "cannot mix or bypass the shared + resolver". ``build_config`` enforces that; a config built from a raw merged mapping would not — and ``overrides`` is the one part of that payload that arrives from an MCP tool argument or REST body. """ diff --git a/agent/tests/test_mandate_enforcement.py b/agent/tests/test_mandate_enforcement.py index 1c17ea05..4180a900 100644 --- a/agent/tests/test_mandate_enforcement.py +++ b/agent/tests/test_mandate_enforcement.py @@ -223,6 +223,18 @@ def test_unreadable_positions_fail_closed() -> None: assert breach.limit == "max_total_exposure_usd" +@pytest.mark.parametrize("side", ["buy", "sell"]) +def test_share_notional_gate_refuses_options_even_when_permitted(side: str) -> None: + """An instrument allowance does not supply an options risk calculation.""" + breach = _check( + _intent(instrument_type=InstrumentType.OPTION, side=side), + _mandate(allowed_instruments=(InstrumentType.OPTION,)), + ) + assert breach is not None + assert breach.kind == BREACH_KIND_INSTRUMENT + assert breach.limit == "options_unsupported" + + @pytest.mark.parametrize( "row", [ @@ -303,6 +315,23 @@ def test_guard_forwards_in_mandate_order(live_runtime: Path) -> None: assert counter["count"] == 1 +@pytest.mark.parametrize("type_key", ["type", "instrument_type", "asset_class", "instrument_class"]) +def test_mcp_equity_entry_cannot_forward_an_option( + live_runtime: Path, type_key: str +) -> None: + """Reject option aliases on the real gate path before any order is sent.""" + _write_mandate(live_runtime, _mandate(allowed_instruments=(InstrumentType.OPTION,))) + adapter = _MockAdapter(positions=[], balance=5000.0) + result = json.loads( + _guard(adapter).execute( + symbol="AAPL", side="buy", notional_usd=100.0, **{type_key: "option"} + ) + ) + assert result["status"] == "blocked" + assert adapter.order_calls == [] + assert not (live_runtime / "live" / "robinhood" / "trade_counter.json").exists() + + def test_mcp_guard_serializes_the_daily_cap_with_submission(live_runtime: Path) -> None: """MCP placements share the same one-permit transaction as SDK orders.""" _write_mandate(live_runtime, _mandate(max_trades_per_day=1)) diff --git a/agent/tests/test_market_identity_parity.py b/agent/tests/test_market_identity_parity.py index 71e6051d..d346b11f 100644 --- a/agent/tests/test_market_identity_parity.py +++ b/agent/tests/test_market_identity_parity.py @@ -25,6 +25,7 @@ _SAMPLES: dict[str, tuple[str, str, str]] = { "kr_equity": ("005930.KS", "005930.KS", "kr"), "ca_equity": ("TD.TO", "TD.TO", "ca"), "ar_equity": ("GGAL.BA", "GGAL.BA", "ar"), + "br_equity": ("PETR4.SA", "PETR4.SA", "br"), "uk_equity": ("VOD.L", "VOD.L", "uk"), "vietnam_equity": ("VIC.VN", "VIC.VN", "vn"), } diff --git a/agent/tests/test_offline_benchmark.py b/agent/tests/test_offline_benchmark.py new file mode 100644 index 00000000..a81c9547 --- /dev/null +++ b/agent/tests/test_offline_benchmark.py @@ -0,0 +1,261 @@ +"""A declared benchmark must be honoured, or its absence reported, on every run. + +``runner.main`` hands the engine a wrapper around the already-fetched +snapshot. A benchmark ticker is normally not one of the traded ``codes``, so a +source that must stay offline needs the wrapper to reach its own loader for +it; every other source keeps the plain snapshot wrapper. When the declared +benchmark still cannot be resolved, the run card has to say so, because the +benchmark-relative metrics are then graded against the equal-weight basket. +""" + +from __future__ import annotations + +import json +from pathlib import Path +from types import SimpleNamespace + +import pandas as pd +import pytest + +import backtest.runner as runner + +_DATES = pd.bdate_range("2026-01-05", periods=5) +_STOCK_CLOSE = [100.0, 110.0, 120.0, 130.0, 140.0] +_BENCH_CLOSE = [200.0, 201.0, 202.0, 203.0, 204.0] +_BENCH = "SPY.US" +_UNRESOLVED_MARKER = "could not be resolved" + + +def _bars(closes: list[float]) -> pd.DataFrame: + return pd.DataFrame( + { + "open": closes, + "high": closes, + "low": closes, + "close": closes, + "volume": [1_000.0] * len(closes), + }, + index=_DATES, + ) + + +class _FlatSignal: + """Never trades, so only the benchmark side of the metrics can move.""" + + def generate(self, data_map): + return {code: pd.Series(0.0, index=df.index) for code, df in data_map.items()} + + +def _loader_class(source: str, frames: dict[str, pd.DataFrame], calls: list[list[str]]): + class _RecordingLoader: + name = source + + def is_available(self) -> bool: + return True + + def fetch(self, codes, start_date, end_date, fields=None, interval="1D"): + del start_date, end_date, fields, interval + calls.append(list(codes)) + return {code: frames[code].copy() for code in codes if code in frames} + + return _RecordingLoader + + +def _run_main( + tmp_path: Path, + monkeypatch: pytest.MonkeyPatch, + *, + source: str, + frames: dict[str, pd.DataFrame], +) -> tuple[dict, list[list[str]]]: + """Run ``runner.main`` end to end with a recording loader for ``source``. + + Args: + tmp_path: Pytest temporary directory, used as the allowed run root. + monkeypatch: Pytest monkeypatch fixture. + source: Configured data source. + frames: Bars the recording loader can serve, keyed by symbol. + + Returns: + The written run card and every symbol list the loader was asked for. + """ + run_dir = tmp_path / "run" + (run_dir / "code").mkdir(parents=True) + (run_dir / "code" / "signal_engine.py").write_text( + "class SignalEngine:\n pass\n", encoding="utf-8" + ) + (run_dir / "config.json").write_text( + json.dumps( + { + "codes": ["AAPL.US"], + "start_date": str(_DATES[0].date()), + "end_date": str(_DATES[-1].date()), + "source": source, + "benchmark": _BENCH, + "initial_cash": 100_000.0, + } + ), + encoding="utf-8", + ) + monkeypatch.setenv("VIBE_TRADING_ALLOWED_RUN_ROOTS", str(tmp_path)) + + calls: list[list[str]] = [] + monkeypatch.setattr( + runner, "_get_loader", lambda name: _loader_class(source, frames, calls) + ) + monkeypatch.setattr( + runner, + "_load_module_from_file", + lambda path, name: SimpleNamespace(SignalEngine=_FlatSignal), + ) + monkeypatch.setattr(runner, "_validate_signal_engine_class", lambda cls: None) + + class _NoNetwork: + def fetch(self, *args, **kwargs): + return {} + + monkeypatch.setattr("backtest.benchmark.YfinanceLoader", _NoNetwork) + + runner.main(run_dir) + card = json.loads((run_dir / "run_card.json").read_text(encoding="utf-8")) + return card, calls + + +def test_offline_run_fetches_the_declared_benchmark_through_its_own_loader( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch +) -> None: + frames = {"AAPL.US": _bars(_STOCK_CLOSE), _BENCH: _bars(_BENCH_CLOSE)} + + card, calls = _run_main(tmp_path, monkeypatch, source="local", frames=frames) + + assert calls == [["AAPL.US"], [_BENCH]] + metrics = card["metrics"] + assert metrics["benchmark_ticker"] == _BENCH + assert metrics["benchmark_return"] == pytest.approx( + _BENCH_CLOSE[-1] / _BENCH_CLOSE[0] - 1, abs=1e-6 + ) + assert metrics["excess_return"] == pytest.approx( + metrics["total_return"] - metrics["benchmark_return"], abs=1e-6 + ) + assert not any(_UNRESOLVED_MARKER in w for w in card.get("warnings") or []) + + +def test_networked_run_keeps_the_snapshot_wrapper_away_from_its_loader( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch +) -> None: + frames = {"AAPL.US": _bars(_STOCK_CLOSE), _BENCH: _bars(_BENCH_CLOSE)} + + card, calls = _run_main(tmp_path, monkeypatch, source="yahoo", frames=frames) + + assert calls == [["AAPL.US"]] + assert "benchmark_ticker" not in card["metrics"] + + +def test_snapshot_wrapper_without_inner_loader_serves_only_the_snapshot() -> None: + frame = _bars(_STOCK_CLOSE) + loader = runner._AutoLoader({"AAPL.US": frame}) + + assert loader.fetch(["AAPL.US", _BENCH], "", "") == {"AAPL.US": frame} + assert getattr(loader, "name", None) is None + + +def test_unresolvable_declared_benchmark_is_reported_on_the_run_card( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch +) -> None: + frames = {"AAPL.US": _bars(_STOCK_CLOSE)} + + card, _ = _run_main(tmp_path, monkeypatch, source="local", frames=frames) + + assert "benchmark_ticker" not in card["metrics"] + warnings = [w for w in card.get("warnings") or [] if _UNRESOLVED_MARKER in w] + assert len(warnings) == 1 + assert repr(_BENCH) in warnings[0] + assert "equal-weight basket" in warnings[0] + + +def test_inner_loader_frames_are_sanitised_like_the_snapshot() -> None: + bench = _bars(_BENCH_CLOSE) + bench.iloc[2, bench.columns.get_loc("high")] = 1.0 # high below low + calls: list[list[str]] = [] + inner = _loader_class("local", {_BENCH: bench}, calls)() + loader = runner._AutoLoader({}, inner=inner, name="local") + + served = loader.fetch([_BENCH], "", "") + + assert calls == [[_BENCH]] + assert list(served[_BENCH].index) == [d for i, d in enumerate(_DATES) if i != 2] + + +@pytest.mark.parametrize("source", ["local", "qveris", "tickerall", "fmp", "nobitex", "wallex"]) +@pytest.mark.parametrize("loader_name", ["same", "yahoo"]) +def test_source_bound_benchmark_never_switches_provider(monkeypatch, source, loader_name): + from backtest.benchmark import resolve_benchmark + + calls = [] + network_calls = [] + name = source if loader_name == "same" else loader_name + loader = _loader_class(name, {}, calls)() + + class NetworkLoader: + def fetch(self, *args, **kwargs): + network_calls.append(args) + return {_BENCH: _bars(_BENCH_CLOSE)} + + monkeypatch.setattr("backtest.benchmark.YfinanceLoader", NetworkLoader) + result = resolve_benchmark( + ["AAPL.US"], source, "2026-01-05", "2026-01-09", explicit=_BENCH, loader=loader + ) + + assert result is None + assert calls == ([[_BENCH]] if loader_name == "same" else []) + assert network_calls == [] + + +@pytest.mark.parametrize("benchmark_available", [True, False]) +def test_real_local_csv_benchmark_reaches_run_metrics(tmp_path, monkeypatch, benchmark_available): + """Exercise CSV -> actual loader -> runner -> run card with no network.""" + from backtest.loaders import local_loader + + bridge = tmp_path / "bridge.json" + symbols = [("AAPL.US", _STOCK_CLOSE)] + if benchmark_available: + symbols.append((_BENCH, _BENCH_CLOSE)) + sources = [] + for symbol, closes in symbols: + path = tmp_path / f"{symbol}.csv" + _bars(closes).rename_axis("date").to_csv(path) + sources.append({"symbol": symbol, "type": "csv", "path": str(path)}) + bridge.write_text(json.dumps({"sources": sources})) + monkeypatch.setattr(local_loader, "_CONFIG_PATH", bridge) + monkeypatch.setenv("VIBE_TRADING_ALLOWED_RUN_ROOTS", str(tmp_path)) + monkeypatch.setenv("VIBE_TRADING_DATA_CACHE", "0") + run = tmp_path / "run" + (run / "code").mkdir(parents=True) + (run / "code/signal_engine.py").write_text( + "class SignalEngine:\n" + " def generate(self, data_map):\n" + " return {code: frame['close'] * 0 for code, frame in data_map.items()}\n" + ) + (run / "config.json").write_text(json.dumps({ + "codes": ["AAPL.US"], "source": "local", "benchmark": _BENCH, + "start_date": "2026-01-05", "end_date": "2026-01-09", "initial_cash": 100_000, + })) + network_calls = [] + + class NetworkLoader: + def fetch(self, *args, **kwargs): + network_calls.append(args) + return {_BENCH: _bars(_BENCH_CLOSE)} + + monkeypatch.setattr("backtest.benchmark.YfinanceLoader", NetworkLoader) + runner.main(run) + card = json.loads((run / "run_card.json").read_text()) + assert network_calls == [] + if benchmark_available: + assert card["metrics"]["benchmark_ticker"] == _BENCH + assert card["metrics"]["benchmark_return"] == pytest.approx(0.02) + assert card["metrics"]["excess_return"] == pytest.approx(-0.02) + assert not any(_UNRESOLVED_MARKER in w for w in card.get("warnings") or []) + else: + assert "benchmark_ticker" not in card["metrics"] + assert any(_UNRESOLVED_MARKER in w for w in card["warnings"]) diff --git a/agent/tests/test_openai_codex_tool_schema.py b/agent/tests/test_openai_codex_tool_schema.py new file mode 100644 index 00000000..24c3a715 --- /dev/null +++ b/agent/tests/test_openai_codex_tool_schema.py @@ -0,0 +1,97 @@ +"""Codex tool schemas are sent non-strict so optional arguments can be omitted.""" + +from __future__ import annotations + +import copy +import json + +from src.providers.openai_codex import ( + OpenAICodexLLM, + _convert_tools, + _events_from_lines, + _message_chunks_from_events, +) + +TOOL = { + "type": "function", + "function": { + "name": "lookup_record", + "description": "Read one record.", + "parameters": { + "type": "object", + "properties": { + "record": {"type": "string", "description": "Required record name."}, + "snapshot_id": { + "type": "string", + "description": "Optional id; omit for the latest.", + }, + "no_cache": {"type": "boolean", "default": False}, + }, + "required": ["record"], + }, + }, +} + + +def _calls(arguments: str) -> list[dict]: + lines = [ + 'data: {"type":"response.output_item.added","item":{"type":"function_call","call_id":"c1","id":"fc_1","name":"lookup_record","arguments":""}}', + "", + "data: " + + '{"type":"response.function_call_arguments.delta","call_id":"c1","delta":' + + json.dumps(arguments) + + "}", + "", + 'data: {"type":"response.output_item.done","item":{"type":"function_call","call_id":"c1"}}', + "", + "data: [DONE]", + "", + ] + return [ + c + for chunk in _message_chunks_from_events(_events_from_lines(lines)) + for c in chunk.tool_calls + ] + + +def test_converted_tool_is_explicitly_non_strict_and_keeps_the_schema(): + original = copy.deepcopy(TOOL) + + [converted] = _convert_tools([TOOL]) + + assert converted["strict"] is False + assert converted["type"] == "function" + assert converted["name"] == "lookup_record" + assert converted["description"] == "Read one record." + assert converted["parameters"] == original["function"]["parameters"] + assert converted["parameters"]["required"] == ["record"] + assert TOOL == original + + +def test_every_tool_in_the_request_body_is_non_strict(): + bare = { + "name": "ping", + "description": "", + "parameters": {"type": "object", "properties": {}}, + } + body = ( + OpenAICodexLLM(model="openai-codex/gpt-6-sol") + .bind_tools([TOOL, bare]) + ._body([{"role": "user", "content": "hi"}], stream=True) + ) + + assert [t["name"] for t in body["tools"]] == ["lookup_record", "ping"] + assert all(t["strict"] is False for t in body["tools"]) + + +def test_optional_arguments_can_be_omitted_and_required_ones_still_arrive(): + [omitted] = _calls('{"record":"orders"}') + [filled] = _calls('{"record":"orders","snapshot_id":"abc","no_cache":true}') + + assert omitted["args"] == {"record": "orders"} + assert "snapshot_id" not in omitted["args"] + assert filled["args"] == { + "record": "orders", + "snapshot_id": "abc", + "no_cache": True, + } diff --git a/agent/tests/test_registry.py b/agent/tests/test_registry.py index 8d1c44fc..71abef60 100644 --- a/agent/tests/test_registry.py +++ b/agent/tests/test_registry.py @@ -205,6 +205,7 @@ class TestFallbackChains: "kr_equity", "ca_equity", "ar_equity", + "br_equity", "uk_equity", "vietnam_equity", "crypto", diff --git a/agent/tests/test_sec_filer_lineage.py b/agent/tests/test_sec_filer_lineage.py new file mode 100644 index 00000000..1eb7e26a --- /dev/null +++ b/agent/tests/test_sec_filer_lineage.py @@ -0,0 +1,27 @@ +"""A successor keeps current filings and inherits dated predecessor facts.""" +from copy import deepcopy + +from backtest.loaders import sec_edgar_client as sec + + +def test_successor_facts_include_only_pre_change_history(monkeypatch): + def facts(cik, rows): + return {"cik": cik, "facts": {"us-gaap": {"Revenues": {"units": {"USD": rows}}}}} + old_row = {"end": "2024-12-31", "filed": "2025-02-01", "val": 100} + subsidiary_row = {"end": "2026-09-30", "filed": "2026-11-01", "val": 1} + new_row = {"end": "2026-09-30", "filed": "2026-11-01", "val": 200} + current = facts(2115436, [new_row]) + original = deepcopy(current) + old = facts(34088, [old_row, subsidiary_row]) + urls = [] + def fetch(url): + urls.append(url) + return old if '0000034088' in url else current + monkeypatch.setattr(sec, '_sec_get_json', fetch) + monkeypatch.setattr(sec, '_ticker_map', lambda: {'XOM': '0002115436'}) + assert sec.cik_for('XOM.US') == '0002115436' + result = sec.get_company_facts(sec.cik_for('XOM')) + assert result['cik'] == 2115436 + assert result['facts']['us-gaap']['Revenues']['units']['USD'] == [new_row, old_row] + assert current == original + assert len(urls) == 2 diff --git a/agent/tests/test_settings_api.py b/agent/tests/test_settings_api.py index 3b0a1f9c..00a59fc0 100644 --- a/agent/tests/test_settings_api.py +++ b/agent/tests/test_settings_api.py @@ -690,7 +690,7 @@ def test_get_data_source_settings_lists_default_source_orders( orders = {entry["market"]: entry for entry in entries} assert set(orders) == { "a_share", "us_equity", "hk_equity", "india_equity", "kr_equity", - "ca_equity", "ar_equity", "vietnam_equity", "uk_equity", "crypto", "futures", + "ca_equity", "ar_equity", "br_equity", "vietnam_equity", "uk_equity", "crypto", "futures", "fund", "macro", "forex", "index", } a_share = orders["a_share"] diff --git a/agent/tests/test_trade_artifact_events.py b/agent/tests/test_trade_artifact_events.py new file mode 100644 index 00000000..05c05dad --- /dev/null +++ b/agent/tests/test_trade_artifact_events.py @@ -0,0 +1,145 @@ +"""Closed-trade identity survives the production CSV round trip.""" + +from datetime import date +from pathlib import Path + +import pandas as pd +import pytest + +from backtest.engines.china_a import ChinaAEngine +from backtest.models import TradeRecord +from backtest.validation import _load_trades, monte_carlo_test +from src.strategy_discovery.run_artifacts import read_trade_activity + + +@pytest.mark.parametrize("direction", [1, -1]) +@pytest.mark.parametrize("same_day", [False, True]) +@pytest.mark.parametrize("middle_pnl", [0.0, 0.00001, 1.0]) +def test_exported_trades_keep_zero_pnl_exits( + tmp_path: Path, direction: int, same_day: bool, middle_pnl: float +) -> None: + engine = ChinaAEngine({"initial_cash": 1000.0}) + dates = pd.date_range("2026-01-01", periods=6) + engine.trades = [ + TradeRecord( + symbol="TEST", + direction=direction, + entry_price=100.0, + exit_price=100.0 + direction * pnl, + entry_time=dates[2 * i], + exit_time=dates[2 * i if same_day else 2 * i + 1], + size=1.0, + leverage=1.0, + pnl=pnl, + pnl_pct=pnl, + exit_reason="signal", + holding_bars=0 if same_day else 1, + commission=0.0, + ) + for i, pnl in enumerate([10.0, middle_pnl, -5.0]) + ] + equity = pd.Series(1000.0, index=dates) + engine._write_artifacts( + tmp_path, + {}, + dates, + equity, + equity, + pd.Series(0.0, index=dates), + pd.DataFrame(index=dates), + {}, + ["TEST"], + ) + + restored = _load_trades(tmp_path) + # The artifact already rounds cash PnL to four decimals; preserve that + # contract while retaining even trades rounded to zero. + assert [t.pnl for t in restored] == [10.0, round(middle_pnl, 4), -5.0] + assert [t.direction for t in restored] == [direction] * 3 + assert read_trade_activity(tmp_path / "artifacts/trades.csv") == ( + [t.exit_time.date() for t in engine.trades], + 1, + ) + frame = pd.read_csv(tmp_path / "artifacts/trades.csv") + assert frame["event"].tolist() == ["entry", "exit"] * 3 + if middle_pnl != 0.00001: + assert monte_carlo_test(restored, 1000.0, n_simulations=30) == ( + monte_carlo_test(engine.trades, 1000.0, n_simulations=30) + ) + + +def test_explicit_events_override_pnl_and_skip_unknown_rows(tmp_path: Path) -> None: + artifacts = tmp_path / "artifacts" + artifacts.mkdir() + path = artifacts / "trades.csv" + path.write_text( + "timestamp,code,side,price,qty,pnl,event\n" + "2026-01-01,A,buy,100,1,7,entry\n" + "2026-01-02,A,sell,100,1,0,exit\n" + "2026-01-03,A,sell,100,1,9,unknown\n" + "2026-01-04,A,sell,100,1,8,\n", + encoding="utf-8", + ) + assert [t.pnl for t in _load_trades(tmp_path)] == [0.0] + assert read_trade_activity(path) == ([date(2026, 1, 2)], 1) + + +@pytest.mark.parametrize("options_format", [False, True]) +def test_unmarked_csv_keeps_legacy_loading( + tmp_path: Path, options_format: bool +) -> None: + artifacts = tmp_path / "artifacts" + artifacts.mkdir() + path = artifacts / "trades.csv" + side = "close" if options_format else "sell" + path.write_text( + "timestamp,code,side,price,qty,pnl\n" + "2026-01-01,A,buy,100,1,0\n" + f"2026-01-02,A,{side},101,1,1\n" + f"2026-01-03,A,{side},100,1,0\n", + encoding="utf-8", + ) + assert [t.pnl for t in _load_trades(tmp_path)] == [1.0] + assert read_trade_activity(path) == ([date(2026, 1, 2)], 1) + + +@pytest.mark.parametrize("overlap", [False, True]) +def test_all_breakeven_events_preserve_activity(tmp_path: Path, overlap: bool) -> None: + artifacts = tmp_path / "artifacts" + artifacts.mkdir() + path = artifacts / "trades.csv" + second_entry = "2026-01-02" if overlap else "2026-01-04" + path.write_text( + "timestamp,code,side,price,qty,pnl,event\n" + "2026-01-01,A,buy,100,1,0,entry\n" + "2026-01-03,A,sell,100,1,0,exit\n" + f"{second_entry},B,sell,100,1,0,entry\n" + "2026-01-05,B,buy,100,1,0,exit\n", + encoding="utf-8", + ) + assert [t.pnl for t in _load_trades(tmp_path)] == [0.0, 0.0] + assert read_trade_activity(path) == ( + [date(2026, 1, 3), date(2026, 1, 5)], + 2 if overlap else 1, + ) + + +def test_empty_export_keeps_event_header(tmp_path: Path) -> None: + engine = ChinaAEngine({"initial_cash": 1000.0}) + dates = pd.date_range("2026-01-01", periods=2) + equity = pd.Series(1000.0, index=dates) + engine._write_artifacts( + tmp_path, + {}, + dates, + equity, + equity, + pd.Series(0.0, index=dates), + pd.DataFrame(index=dates), + {}, + [], + ) + path = tmp_path / "artifacts/trades.csv" + assert "event" in pd.read_csv(path).columns + assert _load_trades(tmp_path) == [] + assert read_trade_activity(path) == ([], None) diff --git a/agent/tests/test_yahoo_loader.py b/agent/tests/test_yahoo_loader.py index 9239b3f7..859dfd2c 100644 --- a/agent/tests/test_yahoo_loader.py +++ b/agent/tests/test_yahoo_loader.py @@ -411,7 +411,7 @@ class TestLoaderMetadata: assert loader.name == "yahoo" assert loader.markets == { "us_equity", "hk_equity", "india_equity", "kr_equity", "ca_equity", - "vietnam_equity", "uk_equity", "ar_equity", + "vietnam_equity", "uk_equity", "ar_equity", "br_equity", } assert loader.requires_auth is False assert loader.is_available() is True diff --git a/frontend/src/components/run/PositionsTab.tsx b/frontend/src/components/run/PositionsTab.tsx index 4707c5fb..f5a1f869 100644 --- a/frontend/src/components/run/PositionsTab.tsx +++ b/frontend/src/components/run/PositionsTab.tsx @@ -28,6 +28,7 @@ const ASSET_CLASS_LABEL_KEYS = { hk_equity: "runDetail.positions.assetClass.hk_equity", ca_equity: "runDetail.positions.assetClass.ca_equity", ar_equity: "runDetail.positions.assetClass.ar_equity", + br_equity: "runDetail.positions.assetClass.br_equity", uk_equity: "runDetail.positions.assetClass.uk_equity", vietnam_equity: "runDetail.positions.assetClass.vietnam_equity", kr_equity: "runDetail.positions.assetClass.kr_equity", diff --git a/frontend/src/components/settings/SourcePrioritySettings.tsx b/frontend/src/components/settings/SourcePrioritySettings.tsx index 5452c160..80aaa473 100644 --- a/frontend/src/components/settings/SourcePrioritySettings.tsx +++ b/frontend/src/components/settings/SourcePrioritySettings.tsx @@ -35,6 +35,7 @@ export function SourcePrioritySettings() { ca_equity: t("settings.sourcePriority.markets.ca_equity"), vietnam_equity: t("settings.sourcePriority.markets.vietnam_equity"), ar_equity: t("settings.sourcePriority.markets.ar_equity"), + br_equity: t("settings.sourcePriority.markets.br_equity"), uk_equity: t("settings.sourcePriority.markets.uk_equity"), index: t("settings.sourcePriority.markets.index"), crypto: t("settings.sourcePriority.markets.crypto"), diff --git a/frontend/src/components/settings/__tests__/SourcePrioritySettings.test.tsx b/frontend/src/components/settings/__tests__/SourcePrioritySettings.test.tsx index 496c89b0..e6afa88d 100644 --- a/frontend/src/components/settings/__tests__/SourcePrioritySettings.test.tsx +++ b/frontend/src/components/settings/__tests__/SourcePrioritySettings.test.tsx @@ -84,7 +84,7 @@ describe("SourcePrioritySettings", () => { // Default badge — nothing customized yet. expect(screen.getByText("Default")).toBeInTheDocument(); // Adjustment-caliber caveat is surfaced next to the setting (see PR review). - expect(screen.getByText(/adjustment basis/)).toBeInTheDocument(); + expect(screen.getByText(/returned adjustment\/provenance/)).toBeInTheDocument(); }); it("sends all markets on save: reordered draft as order, default-equal as null", async () => { diff --git a/frontend/src/i18n/locales/ar.json b/frontend/src/i18n/locales/ar.json index c8000bca..32141ef8 100644 --- a/frontend/src/i18n/locales/ar.json +++ b/frontend/src/i18n/locales/ar.json @@ -1044,7 +1044,7 @@ "noMarkets": "لا توجد مصادر بيانات أسواق مسجلة.", "howItWorksTitle": "كيف يعمل", "howItWorks": "تُجرَّب المصادر من الأعلى إلى الأسفل للرموز المكتشفة تلقائيًا، وأي مصدر يفشل ينتقل إلى التالي. يُطبَّق الحفظ فورًا دون إعادة تشغيل.", - "caliberCaveat": "تنبيه: تختلف مصادر البيانات في أساس التعديل — فمثلًا ياهو يعيد أسعار OHLC غير معدّلة بينما تعيد Tencent/Eastmoney/Tushare أسعارًا معدّلة — لذا قد يغيّر إعادة ترتيب السلسلة أرقام الاختبار الرجعي.", + "caliberCaveat": "قد تختلف تسويات الأسعار بين المصادر. راجع معلومات التسوية والمصدر وتحذيرات بطاقة التشغيل؛ فقد يغيّر ترتيب المصادر نتائج الاختبار التاريخي.", "markets": { "a_share": "أسهم الصين (A)", "us_equity": "الأسهم الأمريكية", @@ -1054,6 +1054,7 @@ "ca_equity": "الأسهم الكندية", "vietnam_equity": "أسهم فيتنام", "ar_equity": "الأسهم الأرجنتينية", + "br_equity": "الأسهم البرازيلية", "uk_equity": "الأسهم البريطانية", "index": "المؤشرات", "crypto": "العملات الرقمية", @@ -1384,6 +1385,7 @@ "futures": "عقود آجلة", "forex": "عملات أجنبية", "ar_equity": "أسهم أرجنتينية", + "br_equity": "أسهم برازيلية", "uk_equity": "أسهم بريطانية", "vietnam_equity": "أسهم فيتنامية", "other": "أخرى" diff --git a/frontend/src/i18n/locales/de.json b/frontend/src/i18n/locales/de.json index fe75b172..ccad464a 100644 --- a/frontend/src/i18n/locales/de.json +++ b/frontend/src/i18n/locales/de.json @@ -1049,7 +1049,7 @@ "noMarkets": "Keine Marktdatenquellen registriert.", "howItWorksTitle": "So funktioniert es", "howItWorks": "Bei automatisch erkannten Symbolen werden die Quellen von oben nach unten versucht; eine fehlgeschlagene Quelle fällt zur nächsten durch. Beim Speichern sofort wirksam — kein Neustart nötig.", - "caliberCaveat": "Hinweis: Die Quellen unterscheiden sich in der Adjustierungsbasis — z. B. liefert Yahoo unadjustierte OHLC-Kurse, während Tencent/Eastmoney/Tushare adjustierte Kurse liefern — das Umordnen der Kette kann daher Backtest-Zahlen verändern.", + "caliberCaveat": "Die Kursbereinigung kann je nach Quelle abweichen. Prüfe die Angaben zu Bereinigung und Herkunft sowie die Warnungen im Laufbericht; eine andere Reihenfolge kann die Backtest-Ergebnisse verändern.", "markets": { "a_share": "China-Aktien (A)", "us_equity": "US-Aktien", @@ -1059,6 +1059,7 @@ "ca_equity": "Kanada-Aktien", "vietnam_equity": "Vietnam-Aktien", "ar_equity": "Argentinien-Aktien", + "br_equity": "Brasilien-Aktien", "uk_equity": "UK-Aktien", "index": "Indizes", "crypto": "Krypto", @@ -1389,6 +1390,7 @@ "futures": "Futures", "forex": "Devisen", "ar_equity": "Argentinien-Aktien", + "br_equity": "Brasilien-Aktien", "uk_equity": "UK-Aktien", "vietnam_equity": "Vietnam-Aktien", "other": "Sonstige" diff --git a/frontend/src/i18n/locales/en.json b/frontend/src/i18n/locales/en.json index 25b0cebd..3e98865d 100644 --- a/frontend/src/i18n/locales/en.json +++ b/frontend/src/i18n/locales/en.json @@ -1049,7 +1049,7 @@ "noMarkets": "No market data sources are registered.", "howItWorksTitle": "How it works", "howItWorks": "For auto-detected symbols, sources are tried top to bottom; a source that fails falls through to the next one. Saving applies immediately — no restart needed.", - "caliberCaveat": "Caliber caveat: sources differ in adjustment basis — e.g. Yahoo serves raw (unadjusted) OHLC while Tencent/Eastmoney/Tushare/BaoStock serve adjusted prices — so a reordered chain can change backtest numbers.", + "caliberCaveat": "Sources can use different price adjustments. Check the returned adjustment/provenance and run-card warnings; changing source order can change backtest results.", "markets": { "a_share": "A-Shares (China)", "us_equity": "US Equities", @@ -1059,6 +1059,7 @@ "ca_equity": "Canada Equities", "vietnam_equity": "Vietnam Equities", "ar_equity": "Argentina Equities", + "br_equity": "Brazil Equities", "uk_equity": "UK Equities", "index": "Indices", "crypto": "Crypto", @@ -1389,6 +1390,7 @@ "futures": "Futures", "forex": "Forex", "ar_equity": "Argentina equity", + "br_equity": "Brazil equity", "uk_equity": "UK equity", "vietnam_equity": "Vietnam equity", "other": "Other" diff --git a/frontend/src/i18n/locales/es.json b/frontend/src/i18n/locales/es.json index 3b5d42f1..db4ea1b0 100644 --- a/frontend/src/i18n/locales/es.json +++ b/frontend/src/i18n/locales/es.json @@ -1049,7 +1049,7 @@ "noMarkets": "No hay fuentes de datos de mercado registradas.", "howItWorksTitle": "Cómo funciona", "howItWorks": "Para los símbolos detectados automáticamente, las fuentes se prueban de arriba abajo; una fuente que falla pasa a la siguiente. Al guardar se aplica de inmediato, sin reiniciar.", - "caliberCaveat": "Advertencia: las fuentes difieren en la base de ajuste — p. ej. Yahoo entrega OHLC sin ajustar mientras Tencent/Eastmoney/Tushare entregan precios ajustados — reordenar la cadena puede cambiar los números del backtest.", + "caliberCaveat": "Los ajustes de precios pueden variar según la fuente. Revisa los ajustes, la procedencia y las advertencias de la ejecución; cambiar el orden puede modificar los resultados del backtest.", "markets": { "a_share": "Acciones de China (A)", "us_equity": "Acciones de EE. UU.", @@ -1059,6 +1059,7 @@ "ca_equity": "Acciones de Canadá", "vietnam_equity": "Acciones de Vietnam", "ar_equity": "Acciones de Argentina", + "br_equity": "Acciones de Brasil", "uk_equity": "Acciones del Reino Unido", "index": "Índices", "crypto": "Cripto", @@ -1389,6 +1390,7 @@ "futures": "Futuros", "forex": "Divisas", "ar_equity": "Renta variable Argentina", + "br_equity": "Renta variable Brasil", "uk_equity": "Renta variable Reino Unido", "vietnam_equity": "Renta variable Vietnam", "other": "Otros" diff --git a/frontend/src/i18n/locales/id.json b/frontend/src/i18n/locales/id.json index 869e823e..f39f876d 100644 --- a/frontend/src/i18n/locales/id.json +++ b/frontend/src/i18n/locales/id.json @@ -1049,7 +1049,7 @@ "noMarkets": "Belum ada sumber data market yang terdaftar.", "howItWorksTitle": "Cara kerjanya", "howItWorks": "Untuk simbol yang terdeteksi otomatis, sumber dicoba dari atas ke bawah; jika satu sumber gagal, sumber berikutnya akan dicoba. Perubahan langsung berlaku setelah disimpan — tidak perlu restart.", - "caliberCaveat": "Catatan Caliber: sumber memiliki basis penyesuaian yang berbeda — misalnya Yahoo menyediakan OHLC mentah (tanpa penyesuaian), sedangkan Tencent/Eastmoney/Tushare/BaoStock menyediakan harga yang sudah disesuaikan — sehingga perubahan urutan sumber dapat mengubah hasil backtest.", + "caliberCaveat": "Penyesuaian harga dapat berbeda antar sumber. Periksa informasi penyesuaian, asal data, dan peringatan laporan; mengubah urutan sumber dapat mengubah hasil backtest.", "markets": { "a_share": "A-Shares (China)", "us_equity": "Saham AS", @@ -1059,6 +1059,7 @@ "ca_equity": "Saham Kanada", "vietnam_equity": "Saham Vietnam", "ar_equity": "Saham Argentina", + "br_equity": "Saham Brasil", "uk_equity": "Saham Inggris", "index": "Indeks", "crypto": "Crypto", @@ -1389,6 +1390,7 @@ "futures": "Futures", "forex": "Forex", "ar_equity": "Saham Argentina", + "br_equity": "Saham Brasil", "uk_equity": "Saham Inggris", "vietnam_equity": "Saham Vietnam", "other": "Lainnya" diff --git a/frontend/src/i18n/locales/ja.json b/frontend/src/i18n/locales/ja.json index 999d98a9..4d577244 100644 --- a/frontend/src/i18n/locales/ja.json +++ b/frontend/src/i18n/locales/ja.json @@ -1044,7 +1044,7 @@ "noMarkets": "登録された市場データソースがありません。", "howItWorksTitle": "動作のしくみ", "howItWorks": "自動検出されたシンボルでは、上から順にソースが試行され、失敗したソースは次へフォールスルーします。保存すると再起動なしで即時反映されます。", - "caliberCaveat": "留意点:データソースごとに調整基準が異なります — 例えば Yahoo は未調整の OHLC を、Tencent/Eastmoney/Tushare は調整後価格を返します — 順序を変えるとバックテスト結果が変わる可能性があります。", + "caliberCaveat": "データソースによって価格調整が異なる場合があります。返却された調整・出典情報と実行カードの警告を確認してください。順序の変更でバックテスト結果が変わる可能性があります。", "markets": { "a_share": "中国A株", "us_equity": "米国株", @@ -1054,6 +1054,7 @@ "ca_equity": "カナダ株", "vietnam_equity": "ベトナム株", "ar_equity": "アルゼンチン株", + "br_equity": "ブラジル株", "uk_equity": "英国株", "index": "指数", "crypto": "暗号資産", @@ -1384,6 +1385,7 @@ "futures": "先物", "forex": "外国為替", "ar_equity": "アルゼンチン株", + "br_equity": "ブラジル株", "uk_equity": "英国株", "vietnam_equity": "ベトナム株", "other": "その他" diff --git a/frontend/src/i18n/locales/ko.json b/frontend/src/i18n/locales/ko.json index 014bb953..6c556b99 100644 --- a/frontend/src/i18n/locales/ko.json +++ b/frontend/src/i18n/locales/ko.json @@ -1044,7 +1044,7 @@ "noMarkets": "등록된 시장 데이터 소스가 없습니다.", "howItWorksTitle": "작동 방식", "howItWorks": "자동 감지된 심볼은 위에서 아래로 소스를 시도하며, 실패한 소스는 다음으로 넘어갑니다. 저장하면 재시작 없이 즉시 적용됩니다.", - "caliberCaveat": "주의: 데이터 소스마다 수정 기준이 다릅니다 — 예를 들어 Yahoo는 미수정 OHLC를, Tencent/Eastmoney/Tushare는 수정 주가를 반환합니다 — 순서를 바꾸면 백테스트 수치가 달라질 수 있습니다.", + "caliberCaveat": "데이터 소스마다 가격 조정 방식이 다를 수 있습니다. 반환된 조정·출처 정보와 실행 카드 경고를 확인하세요. 소스 순서에 따라 백테스트 결과가 달라질 수 있습니다.", "markets": { "a_share": "중국 A주", "us_equity": "미국 주식", @@ -1054,6 +1054,7 @@ "ca_equity": "캐나다 주식", "vietnam_equity": "베트남 주식", "ar_equity": "아르헨티나 주식", + "br_equity": "브라질 주식", "uk_equity": "영국 주식", "index": "지수", "crypto": "암호화폐", @@ -1384,6 +1385,7 @@ "futures": "선물", "forex": "외환", "ar_equity": "아르헨티나 주식", + "br_equity": "브라질 주식", "uk_equity": "영국 주식", "vietnam_equity": "베트남 주식", "other": "기타" diff --git a/frontend/src/i18n/locales/pt-BR.json b/frontend/src/i18n/locales/pt-BR.json index dbcfe26b..c75d5249 100644 --- a/frontend/src/i18n/locales/pt-BR.json +++ b/frontend/src/i18n/locales/pt-BR.json @@ -1049,7 +1049,7 @@ "noMarkets": "Nenhuma fonte de dados de mercado está registrada.", "howItWorksTitle": "Como funciona", "howItWorks": "Para símbolos detectados automaticamente, as fontes são testadas de cima para baixo; uma fonte que falha passa para a próxima. O salvamento é aplicado imediatamente – não é necessário reiniciar.", - "caliberCaveat": "Advertência sobre o calibre: as fontes diferem na base de ajuste – por ex. O Yahoo atende OHLC bruto (não ajustado), enquanto o Tencent/Eastmoney/Tushare/BaoStock atende preços ajustados – portanto, uma cadeia reordenada pode alterar os números de backtest.", + "caliberCaveat": "Os ajustes de preços podem variar entre fontes. Confira os ajustes, a origem dos dados e os avisos do relatório; mudar a ordem pode alterar os resultados do backtest.", "markets": { "a_share": "Ações A (China)", "us_equity": "Ações dos EUA", @@ -1059,6 +1059,7 @@ "ca_equity": "Ações do Canadá", "vietnam_equity": "Ações do Vietnã", "ar_equity": "Ações da Argentina", + "br_equity": "Ações do Brasil (B3)", "uk_equity": "Ações do Reino Unido", "index": "Índices", "crypto": "Criptografia", @@ -1389,6 +1390,7 @@ "futures": "Futuros", "forex": "Forex", "ar_equity": "Ações da Argentina", + "br_equity": "Ações do Brasil", "uk_equity": "Ações do Reino Unido", "vietnam_equity": "Ações do Vietnã", "other": "Outro" diff --git a/frontend/src/i18n/locales/zh-CN.json b/frontend/src/i18n/locales/zh-CN.json index 6c6c87b3..e18abd8b 100644 --- a/frontend/src/i18n/locales/zh-CN.json +++ b/frontend/src/i18n/locales/zh-CN.json @@ -1044,7 +1044,7 @@ "noMarkets": "没有注册任何市场数据源。", "howItWorksTitle": "工作方式", "howItWorks": "自动识别的标的按列表自上而下依次尝试,失败的源会顺延到下一个。保存后立即生效,无需重启。", - "caliberCaveat": "口径提示:各数据源的复权口径不同 — 例如 Yahoo 返回未复权原始 OHLC,腾讯/东方财富/Tushare 返回前复权价 — 重排市场链可能因此改变回测数值。", + "caliberCaveat": "不同数据源的复权口径可能不同。请核对返回的复权与来源信息及回测运行卡警告；调整数据源顺序可能改变回测结果。", "markets": { "a_share": "A股", "us_equity": "美股", @@ -1054,6 +1054,7 @@ "ca_equity": "加拿大股票", "vietnam_equity": "越南股票", "ar_equity": "阿根廷股票", + "br_equity": "巴西股票", "uk_equity": "英国股票", "index": "指数", "crypto": "加密货币", @@ -1384,6 +1385,7 @@ "futures": "期货", "forex": "外汇", "ar_equity": "阿根廷股", + "br_equity": "巴西股", "uk_equity": "英股", "vietnam_equity": "越股", "other": "其他" diff --git a/frontend/src/lib/__tests__/positions.test.ts b/frontend/src/lib/__tests__/positions.test.ts index 5cd17ccf..ec72d17b 100644 --- a/frontend/src/lib/__tests__/positions.test.ts +++ b/frontend/src/lib/__tests__/positions.test.ts @@ -114,6 +114,8 @@ describe("classifyAssetClass parity with backend rules", () => { // earlier and were never classified here, so they read as "other". ["GGAL.BA", "ar_equity"], ["GOOGL.BA", "ar_equity"], + ["PETR4.SA", "br_equity"], + ["HGLG11.SA", "br_equity"], ["VOD.L", "uk_equity"], ["VIC.VN", "vietnam_equity"], ["005930.KS", "kr_equity"], diff --git a/frontend/src/lib/api.ts b/frontend/src/lib/api.ts index 3ca18817..50b1878d 100644 --- a/frontend/src/lib/api.ts +++ b/frontend/src/lib/api.ts @@ -1331,6 +1331,7 @@ export type SectorAssetClass = | "kr_equity" | "ca_equity" | "ar_equity" + | "br_equity" | "uk_equity" | "vietnam_equity" | "index" diff --git a/frontend/src/lib/positions.ts b/frontend/src/lib/positions.ts index a1cf6a7c..e3aa300b 100644 --- a/frontend/src/lib/positions.ts +++ b/frontend/src/lib/positions.ts @@ -10,6 +10,7 @@ export type AssetClass = | "hk_equity" | "ca_equity" | "ar_equity" + | "br_equity" | "uk_equity" | "vietnam_equity" | "kr_equity" @@ -46,6 +47,7 @@ const EQUITY_SUFFIX_MAP: Record = { TO: "ca_equity", V: "ca_equity", BA: "ar_equity", + SA: "br_equity", L: "uk_equity", VN: "vietnam_equity", KS: "kr_equity", diff --git a/pyproject.toml b/pyproject.toml index d7403ea5..2f66b89a 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -120,7 +120,7 @@ include = ["src*", "backtest*", "cli*", "evals*"] "shadow_account/assets/*.md", ] "evals.harness" = ["README.md", "cases/**/*.json"] -"backtest" = ["*.py"] +"backtest" = ["*.py", "loaders/*.json"] "src.factors" = ["zoo/**/*.yaml", "zoo/**/*.md", "zoo/**/NOTICE"] [project.optional-dependencies] diff --git a/wiki/alpha-library/i18n.zh.json b/wiki/alpha-library/i18n.zh.json index f0e7fa1e..4b4c5c11 100644 --- a/wiki/alpha-library/i18n.zh.json +++ b/wiki/alpha-library/i18n.zh.json @@ -156,18 +156,70 @@ "en": "Industry neutralization implemented via per-row sector group demean (panel['sector'] required). When sector tag is absent the registry rejects via SkipAlpha; the compute() also has a degraded global demean fallback. This is a partial approximation of the paper's IndClass.industry/subindustry/sector neutralization.", "zh": "行业中性化通过逐行按 sector 分组去均值实现（需要 panel['sector']）。缺少 sector 标签时，注册表通过 SkipAlpha 予以拒绝；compute() 中也带有降级的全局去均值回退。这是对论文 IndClass.industry/subindustry/sector 中性化的部分近似。" }, + "fund_accruals": { + "en": "Balance-sheet accrual proxy: high accruals indicate low earnings quality; sign inverted so high score = low accruals. PIT-safe from the fundamental panel; missing statements remain NaN and are excluded from that date's cross-section.", + "zh": "资产负债表应计代理指标：高应计表明盈利质量低；符号取反，因此高得分 = 低应计。基于基本面面板的时点正确（PIT）数据；缺失的财务报表保持 NaN，并被排除在当日截面之外。" + }, "fund_asset_growth": { "en": "Inverse PIT-safe asset-growth factor. High asset growth is treated as aggressive investment and receives lower scores; low/negative growth receives higher scores.", "zh": "反向的时点正确（PIT）资产增长因子。高资产增长被视为激进投资，得分较低；低 / 负增长得分较高。" }, + "fund_book_to_market": { + "en": "Valuation proxy: close times weighted-average diluted shares is not reported market capitalization; adjusted closes can change the level. Fama-French (1993) HML-style book-to-market proxy built from PIT-safe total equity and shares diluted aligned to the daily close panel; the academic_hml factor in the academic zoo is only a price proxy. High B/M (cheap relative to book value) scores high. market_cap = close x shares_diluted is computed here in the factor file by design: _fundamental_schema.py lines 210-212 deliberately keep price-dependent value factors such as book_to_market out of DERIVED_FIELDS because they need the aligned price panel. Zero or non-finite market cap becomes NaN via safe_div before cross-sectional z-scoring; missing statements remain NaN and are excluded from that date's cross-section.", + "zh": "此因子是估值代理：收盘价乘加权平均稀释股数不等于披露的市值，复权价格会影响数值水平。Fama-French (1993) HML 风格的账面市值比代理，由时点正确（PIT）的总股东权益与稀释后股数对齐到日度收盘价面板构建；academic zoo 中的 academic_hml 因子只是价格代理。高 B/M（相对账面价值便宜）得分高。market_cap = close x shares_diluted 按设计在因子文件内计算：_fundamental_schema.py 第 210-212 行有意将 book_to_market 等价格依赖型价值因子排除在 DERIVED_FIELDS 之外，因为它们需要对齐后的价格面板。为零或非有限的市值经 safe_div 置为 NaN，随后再做截面 z 打分；缺失的财务报表保持 NaN，并被排除在当日截面之外。" + }, + "fund_cash_to_assets": { + "en": "Balance-sheet cash ratio (cash and equivalents / total assets): high cash holdings signal defensive capacity and financial flexibility. Zero or missing total assets become NaN via safe division before cross-sectional z-scoring; PIT-safe from the filed-date-anchored fundamental panel.", + "zh": "资产负债表现金比率（现金及等价物 / 总资产）：高现金持有量预示防御能力与财务灵活性。在截面 z 打分之前，为零或缺失的总资产经安全除法置为 NaN；基于按申报日锚定的基本面面板，时点正确（PIT）。" + }, + "fund_cfo_yield": { + "en": "Valuation proxy: close times weighted-average diluted shares is not reported market capitalization; adjusted closes can change the level. Cash-flow return on assets: the cash-backed counterpart of ROA, so accrual-heavy earnings score lower here. Zero or missing total assets become NaN via safe division before cross-sectional z-scoring; PIT-safe from the filed-date-anchored fundamental panel.", + "zh": "此因子是估值代理：收盘价乘加权平均稀释股数不等于披露的市值，复权价格会影响数值水平。现金流总资产收益率：ROA 的现金支撑对应指标，因此应计占比高的盈利在此得分较低。在截面 z 打分之前，为零或缺失的总资产经安全除法置为 NaN；基于按申报日锚定的基本面面板，时点正确（PIT）。" + }, + "fund_dividend_yield": { + "en": "Trailing dividend yield from the val: namespace (tushare daily_basic dv_ttm, CN A-share only), reported in percent units — cross-sectional z-scoring is scale-invariant, so no unit conversion is applied. High dividend yield scores high. Keyed by trade_date, so PIT-safe by construction with no filing lag. Missing dates or non-CN symbols stay NaN and are excluded from that date's cross-section. Lineage: JoinQuant style-pro divyild; akshare/Eastmoney dv_ttm convention.", + "zh": "来自 val: 命名空间的滚动股息率（tushare daily_basic dv_ttm，仅 CN A 股），以百分比单位报告——截面 z 打分具有尺度不变性，因此不做单位换算。高股息率得分高。按 trade_date 键控，因此构造上即时点正确（PIT），无财报滞后。缺失日期或非 CN 标的保持 NaN，并被排除在当日截面之外。血缘：JoinQuant style-pro divyild；akshare/东方财富 dv_ttm 口径。" + }, + "fund_earnings_growth": { + "en": "PIT-safe earnings growth from the fundamental panel's derived field ``earnings_growth``. The schema-side formula is sign-safe: delta net_income / abs(lag_365d net_income), so a negative year-ago base does not flip the growth sign (the proposal #6 footnote resolution; the alternative NaN-on-negative-base was not chosen). Calendar-day lag per the #1564 lesson. High earnings growth scores high, matching the JoinQuant net_profit_growth_rate convention. Missing statements or a zero/missing year-ago base remain NaN and are excluded from that date's cross-section.", + "zh": "基于基本面面板派生字段 ``earnings_growth`` 的时点正确（PIT）盈利增长。schema 侧公式符号安全：净利润变化量 / abs(滞后 365 天净利润)，因此上年基数为负不会翻转增长符号（提案 #6 脚注的决议；未选择负基数置 NaN 的替代方案）。按 #1564 教训使用日历日滞后。高盈利增长得分高，与 JoinQuant net_profit_growth_rate 约定一致。缺失的财务报表或为零/缺失的上年基数保持 NaN，并被排除在当日截面之外。" + }, "fund_earnings_yield": { "en": "Hybrid value factor using PIT-safe net income and shares diluted aligned to the daily close panel. Division by zero/non-finite market cap becomes NaN before cross-sectional scoring.", "zh": "混合型价值因子：使用时点正确（PIT）的净利润与稀释后股数，并对齐到日度收盘价面板。截面打分之前，除以零 / 非有限市值的结果置为 NaN。" }, + "fund_fcf_yield": { + "en": "Valuation proxy: close times weighted-average diluted shares is not reported market capitalization; adjusted closes can change the level. PIT-safe free-cash-flow yield: FCF = CFO - capex, scaled by market cap (close x shares_diluted) computed in the factor file per the _fundamental_schema.py price-dependent convention. Sign assumption: capex is reported as a positive magnitude (SEC concept PaymentsToAcquirePropertyPlantAndEquipment), so subtracting it from CFO yields FCF directly. High FCF yield (cheap on cash generation, cf. Lakonishok-Shleifer-Vishny 1994 cash-flow yield value factors) scores high. Zero/non-finite market cap becomes NaN via safe_div before cross-sectional z-scoring; missing statements remain NaN and are excluded from that date's cross-section.", + "zh": "此因子是估值代理：收盘价乘加权平均稀释股数不等于披露的市值，复权价格会影响数值水平。时点正确（PIT）的自由现金流收益率：FCF = 经营现金流 - 资本开支，再除以市值（close x shares_diluted，按 _fundamental_schema.py 价格依赖约定在因子文件内计算）。符号假设：资本开支以正值报告（SEC 概念 PaymentsToAcquirePropertyPlantAndEquipment），因此从经营现金流中减去即得 FCF。高 FCF 收益率（按现金生成能力衡量便宜，参见 Lakonishok-Shleifer-Vishny 1994 现金流收益率类价值因子）得分高。为零/非有限的市值经 safe_div 置为 NaN，随后再做截面 z 打分；缺失的财务报表保持 NaN，并被排除在当日截面之外。" + }, "fund_gross_profitability": { "en": "PIT-safe Novy-Marx style gross profitability from the fundamental panel; cross-sectional z-score per date with missing issuers excluded.", "zh": "基于基本面面板的时点正确（PIT）Novy-Marx 式毛利盈利能力；按日期做截面 z 分数，缺失的发行人被排除在外。" }, + "fund_leverage": { + "en": "PIT-safe market leverage (total debt / total equity) from the fundamental panel; sign inverted so low leverage scores high — heavily levered firms carry distress risk and lower expected returns. Zero-equity or missing statements already yield NaN in the derived field and are excluded from that date's cross-section.", + "zh": "基于基本面面板的时点正确（PIT）市场杠杆率（总债务 / 总权益）；符号取反，因此低杠杆得分高——高杠杆公司带有财务困境风险且预期收益较低。零权益或缺失的财务报表在派生字段中已产生 NaN，并被排除在当日截面之外。" + }, + "fund_operating_margin": { + "en": "Operating profitability (operating income / revenue), cross-sectionally z-scored per date. Captures pricing power and cost efficiency before financing and tax effects. Zero-revenue or missing statements become NaN via safe division and are excluded from that date's cross-section; PIT-safe from the fundamental panel.", + "zh": "营业盈利能力（营业利润 / 营业收入），按日期做截面 z 分数。捕捉融资与税收效应之前的定价能力和成本效率。零营收或缺失的财务报表经安全除法置为 NaN，并被排除在当日截面之外；基于基本面面板，时点正确（PIT）。" + }, + "fund_pb": { + "en": "Nonpositive multiples are excluded rather than ranked as cheap. Inverse price-to-book from the val: namespace (tushare daily_basic pb, CN A-share only): low PB (cheap relative to book value) scores high, the daily-market-data counterpart of fund_book_to_market. Keyed by trade_date, so PIT-safe by construction with no filing lag. Missing dates or non-CN symbols stay NaN and are excluded from that date's cross-section. Lineage: JoinQuant style book_to_price_ratio; akshare/Eastmoney pb convention.", + "zh": "非正估值倍数不参与排名，不能当作低估。来自 val: 命名空间的反向市净率（tushare daily_basic pb，仅 CN A 股）：低 PB（相对账面价值便宜）得分高，是 fund_book_to_market 的日频市场口径对应因子。按 trade_date 键控，因此构造上即时点正确（PIT），无财报滞后。缺失日期或非 CN 标的保持 NaN，并被排除在当日截面之外。血缘：JoinQuant style book_to_price_ratio；akshare/东方财富 pb 口径。" + }, + "fund_ps_ttm": { + "en": "Nonpositive multiples are excluded rather than ranked as cheap. Inverse trailing price-to-sales from the val: namespace (tushare daily_basic ps_ttm, CN A-share only): low PS scores high. PS stays informative for loss-making names where earnings yield fails. Keyed by trade_date, so PIT-safe by construction with no filing lag. Missing dates or non-CN symbols stay NaN and are excluded from that date's cross-section. Lineage: JoinQuant style-pro sales-to-price; akshare/Eastmoney ps_ttm convention.", + "zh": "非正估值倍数不参与排名，不能当作低估。来自 val: 命名空间的反向滚动市销率（tushare daily_basic ps_ttm，仅 CN A 股）：低 PS 得分高。PS 对亏损股依然有效（盈利收益率失效的场景）。按 trade_date 键控，因此构造上即时点正确（PIT），无财报滞后。缺失日期或非 CN 标的保持 NaN，并被排除在当日截面之外。血缘：JoinQuant style-pro sales-to-price；akshare/东方财富 ps_ttm 口径。" + }, + "fund_revenue_growth": { + "en": "PIT-safe revenue growth from the fundamental panel's derived field ``revenue_growth`` (revenue / lag_365d(revenue) - 1, calendar-day lag per the #1564 lesson rather than a row-wise shift). High top-line growth scores high, matching the JoinQuant operating_revenue_growth_rate convention. Missing statements or a missing year-ago base remain NaN and are excluded from that date's cross-section.", + "zh": "基于基本面面板派生字段 ``revenue_growth`` 的时点正确（PIT）营收增长（revenue / lag_365d(revenue) - 1，按 #1564 教训使用日历日滞后而非按行位移）。高顶线增长得分高，与 JoinQuant operating_revenue_growth_rate 约定一致。缺失的财务报表或缺失的上年基数保持 NaN，并被排除在当日截面之外。" + }, + "fund_roa": { + "en": "PIT-safe return on assets (net income / total assets) from the fundamental panel; cross-sectional z-score per date. Higher ROA means a more profitable asset base and scores higher. Missing statements remain NaN and are excluded from that date's cross-section.", + "zh": "基于基本面面板的时点正确（PIT）总资产收益率（净利润 / 总资产）；按日期做截面 z 分数。ROA 越高意味着资产基础盈利能力越强，得分越高。缺失的财务报表保持 NaN，并被排除在当日截面之外。" + }, "fund_roe": { "en": "PIT-safe return on equity from the fundamental panel; cross-sectional z-score per date. Missing statements remain NaN and are excluded from that date's cross-section.", "zh": "基于基本面面板的时点正确（PIT）净资产收益率（ROE）；按日期做截面 z 分数。缺失的财务报表保持 NaN，并被排除在当日截面之外。" @@ -1218,18 +1270,70 @@ "en": "Kakushadze Alpha #101", "zh": "Kakushadze 因子 Alpha #101" }, + "fund_accruals": { + "en": "Accruals - inverse (NI-CFO)/TA, Sloan 1996", + "zh": "应计 - 反向 (NI-CFO)/TA，Sloan 1996" + }, "fund_asset_growth": { "en": "Asset growth - inverse investment factor", "zh": "资产增长 - 反向投资因子" }, + "fund_book_to_market": { + "en": "Book-to-market - equity over market cap (FF HML characteristic)", + "zh": "账面市值比 - 股东权益除以市值（FF HML 特征）" + }, + "fund_cash_to_assets": { + "en": "Cash to assets - balance-sheet cash ratio", + "zh": "现金总资产比 - 资产负债表现金比率" + }, + "fund_cfo_yield": { + "en": "CFO yield - operating cash flow over total assets", + "zh": "经营现金流收益率 - 经营现金流除以总资产" + }, + "fund_dividend_yield": { + "en": "Dividend yield TTM - dv_ttm (daily market data)", + "zh": "股息率 TTM - dv_ttm（日频市场数据）" + }, + "fund_earnings_growth": { + "en": "Earnings growth - sign-safe YoY net income expansion", + "zh": "盈利增长 - 符号安全的同比净利润扩张" + }, "fund_earnings_yield": { "en": "Earnings yield - net income over market cap", "zh": "盈利收益率 - 净利润除以市值" }, + "fund_fcf_yield": { + "en": "FCF yield - (CFO - capex) over market cap", + "zh": "自由现金流收益率 - （经营现金流 - 资本开支）除以市值" + }, "fund_gross_profitability": { "en": "Gross profitability - gross profit over total assets", "zh": "毛利盈利能力 - 毛利除以总资产" }, + "fund_leverage": { + "en": "Leverage - inverse debt/equity (low leverage scores high)", + "zh": "杠杆 - 反向债务/权益比（低杠杆得分高）" + }, + "fund_operating_margin": { + "en": "Operating margin - operating income over revenue", + "zh": "营业利润率 - 营业利润除以营业收入" + }, + "fund_pb": { + "en": "P/B - inverse price-to-book (daily market data)", + "zh": "市净率 P/B - 反向市净率（日频市场数据）" + }, + "fund_ps_ttm": { + "en": "P/S TTM - inverse price-to-sales (daily market data)", + "zh": "市销率 P/S TTM - 反向滚动市销率（日频市场数据）" + }, + "fund_revenue_growth": { + "en": "Revenue growth - YoY top-line expansion", + "zh": "营收增长 - 同比顶线扩张" + }, + "fund_roa": { + "en": "ROA - return on assets (PIT-safe fundamentals)", + "zh": "ROA - 总资产收益率，基于时点正确（PIT）基本面" + }, "fund_roe": { "en": "ROE - return on equity (PIT-safe fundamentals)", "zh": "ROE - 净资产收益率，基于时点正确（PIT）基本面" @@ -1245,6 +1349,7 @@ "liquidity": "流动性", "value": "价值", "growth": "成长", - "sentiment": "情绪" + "sentiment": "情绪", + "leverage": "杠杆" } } diff --git a/zimaos/README.md b/zimaos/README.md new file mode 100644 index 00000000..0c979769 --- /dev/null +++ b/zimaos/README.md @@ -0,0 +1,71 @@ +# Vibe-Trading on ZimaOS + +Import `docker-compose.yml` as a custom app, or run it with Docker Compose. +Set `VIBE_TRADING_IMAGE` to an image and version you have reviewed, +`VIBE_TRADING_APPDATA` to your persistent directory (default +`/DATA/AppData/vibe-trading`), `API_AUTH_KEY` to a strong random key, and +`OPENROUTER_API_KEY` to your provider key. `LANGCHAIN_MODEL_NAME` is optional. +Compose refuses to start without the required values. Set these variables in +ZimaOS before importing, or substitute their values in your private copy. + +```bash +docker compose -f zimaos/docker-compose.yml config --quiet +docker compose -f zimaos/docker-compose.yml up -d +``` + +Open `http://:8899` and enter `API_AUTH_KEY` in Settings → API +authentication key. The initialization container changes ownership only within +the mounted application directories to the image's `vibe` user. It does not +mount the Docker socket. Keep `home`, `runs`, `sessions`, `swarm-runs`, `uploads` +and `br-data` on persistent storage. + +## Optional Brazilian data + +The application accepts B3 symbols such as `PETR4.SA`, `VALE3.SA` and +`BOVA11.SA`. Yahoo/yfinance provide adjusted bars; the optional local COTAHIST +store provides raw bars with their adjustment basis labelled. Backtests use +BRL, configurable fees, share lots and tick rounding. They do not model +personal income tax, stock-borrow availability/cost or settlement cash locks; +review these assumptions before interpreting results. + +The `get_br_market_data` agent tool reads B3 quotes, CVM company/fund data, +Banco Central series and Tesouro Direto prices. CVM statements are the latest +stored revisions, **not** point-in-time backtest inputs. Fund subclasses remain +separate. Data is stored outside the checkout in `~/.vibe-trading/br-data/`, +or at `VIBE_BR_DATA_DB`. + +Run selected sources manually inside the application container: + +```bash +python -m src.br_data status +python -m src.br_data sync --only bcb tesouro +python -m src.br_data sync --only cotahist --cotahist-from 2025 +``` + +The initial full download can be several GB. The daily sync service is off by +default; enable it explicitly with `--profile br-data-sync` when starting +Compose. Its schedule is controlled by `VIBE_BR_SYNC_TIME` in São Paulo time. +Historical coverage starts at `VIBE_BR_COTAHIST_START_YEAR` (default 2015) and +`VIBE_BR_CVM_START_YEAR` (default 2022). No sync is launched by opening the UI. + +## Optional fork publishing + +`sync-and-publish.yml.example` is an inactive template for operators who want +to maintain their own fork and GHCR image. It is outside `.github/workflows` +and does not schedule anything in this repository. Review its write permissions, +upstream merge behavior and publishing destination before copying it to your +fork's workflow directory. Configure `UPSTREAM_REPO`, `UPSTREAM_BRANCH` and +`ZIMAOS_PLATFORMS` there. The template supports a manual run and a 30-minute +schedule after explicit installation; conflicts fail for manual resolution. + +Use your fork's image tag as `VIBE_TRADING_IMAGE`. Watchtower labels are off by +default; opt in only if automatic replacement of that image is desired. To +roll back, select a previous immutable `sha-` tag and restart Compose. + +## Resumo em português + +Configure a imagem, a pasta persistente e as chaves antes de importar o Compose. +O aplicativo não inicia com chaves de exemplo. A sincronização diária dos dados +brasileiros e as atualizações automáticas ficam desativadas por padrão. Ative o +perfil `br-data-sync` somente se quiser baixar a base oficial e mantê-la atualizada. +O workflow é um modelo inativo para seu próprio fork; não publica no upstream. diff --git a/zimaos/docker-compose.yml b/zimaos/docker-compose.yml new file mode 100644 index 00000000..b0bb736b --- /dev/null +++ b/zimaos/docker-compose.yml @@ -0,0 +1,181 @@ +# ZimaOS custom-app template. Configure required variables before importing. +# Daily data sync and Watchtower updates are opt-in; see README.md. +name: vibe-trading + +services: + # One-shot: creates the AppData folders and hands them to the image's + # unprivileged `vibe` user. Docker creates missing bind-mount folders as + # root, which `vibe` could not write to. Only files not already owned by + # `vibe` are touched, so restarts stay fast even with GBs of data. + vibe-trading-init: + image: ${VIBE_TRADING_IMAGE:?Set VIBE_TRADING_IMAGE to a reviewed image tag} + container_name: vibe-trading-init + pull_policy: always + user: "0:0" + restart: "no" + # Plain argv, no shell and no variables: ZimaOS interpolates the compose + # file more than once, so an escaped shell variable reaches the shell as + # an empty string. find also descends into the nested br-data mount. + entrypoint: ["find"] + command: + - /app/agent/runs + - /app/agent/sessions + - /home/vibe/.vibe-trading + - /app/agent/.swarm/runs + - /app/agent/uploads + - "!" + - -user + - vibe + - -exec + - chown + - vibe:vibe + - "{}" + - + + healthcheck: + disable: true + volumes: + - ${VIBE_TRADING_APPDATA:-/DATA/AppData/vibe-trading}/runs:/app/agent/runs + - ${VIBE_TRADING_APPDATA:-/DATA/AppData/vibe-trading}/sessions:/app/agent/sessions + - ${VIBE_TRADING_APPDATA:-/DATA/AppData/vibe-trading}/home:/home/vibe/.vibe-trading + - ${VIBE_TRADING_APPDATA:-/DATA/AppData/vibe-trading}/br-data:/home/vibe/.vibe-trading/br-data + - ${VIBE_TRADING_APPDATA:-/DATA/AppData/vibe-trading}/swarm-runs:/app/agent/.swarm/runs + - ${VIBE_TRADING_APPDATA:-/DATA/AppData/vibe-trading}/uploads:/app/agent/uploads + + vibe-trading: + image: ${VIBE_TRADING_IMAGE:?Set VIBE_TRADING_IMAGE to a reviewed image tag} + container_name: vibe-trading + pull_policy: always + restart: unless-stopped + depends_on: + vibe-trading-init: + condition: service_completed_successfully + ports: + - "8899:8899" + environment: + # REQUIRED: the port is published on the LAN, and the API refuses + # non-loopback clients without a key. Use a long random value + # (e.g. `openssl rand -hex 32`) and enter it in the Web UI when asked. + API_AUTH_KEY: "${API_AUTH_KEY:?Set a strong API_AUTH_KEY}" + TZ: America/Sao_Paulo + # LLM via OpenRouter (https://openrouter.ai/keys). Values set here win + # over the Web UI Settings after every restart, so change the model + # here, not in the UI. + LANGCHAIN_PROVIDER: openrouter + LANGCHAIN_MODEL_NAME: "${LANGCHAIN_MODEL_NAME:-deepseek/deepseek-v3.2}" + OPENROUTER_API_KEY: "${OPENROUTER_API_KEY:?Set OPENROUTER_API_KEY}" + OPENROUTER_BASE_URL: https://openrouter.ai/api/v1 + # Do NOT set VIBE_TRADING_TRUST_DOCKER_LOOPBACK here: with a LAN-published + # port it would treat LAN clients as local. + volumes: + - ${VIBE_TRADING_APPDATA:-/DATA/AppData/vibe-trading}/runs:/app/agent/runs + - ${VIBE_TRADING_APPDATA:-/DATA/AppData/vibe-trading}/sessions:/app/agent/sessions + - ${VIBE_TRADING_APPDATA:-/DATA/AppData/vibe-trading}/home:/home/vibe/.vibe-trading + - ${VIBE_TRADING_APPDATA:-/DATA/AppData/vibe-trading}/br-data:/home/vibe/.vibe-trading/br-data + - ${VIBE_TRADING_APPDATA:-/DATA/AppData/vibe-trading}/swarm-runs:/app/agent/.swarm/runs + - ${VIBE_TRADING_APPDATA:-/DATA/AppData/vibe-trading}/uploads:/app/agent/uploads + cap_drop: + - ALL + cap_add: + - SETUID + - SETGID + security_opt: + - "no-new-privileges:true" + read_only: true + tmpfs: + - /tmp + - /home/vibe/.cache + - /home/vibe/.config + mem_limit: 4g + cpus: 2 + pids_limit: 512 + labels: + com.centurylinklabs.watchtower.enable: "false" + x-casaos: + envs: + - container: API_AUTH_KEY + description: + en_us: Required. Secret key for LAN access to the Web UI / API. + pt_br: Obrigatório. Chave secreta para acesso ao Web UI / API pela rede. + - container: OPENROUTER_API_KEY + description: + en_us: OpenRouter API key (https://openrouter.ai/keys) + pt_br: Chave da API do OpenRouter (https://openrouter.ai/keys) + - container: LANGCHAIN_MODEL_NAME + description: + en_us: OpenRouter model id, e.g. deepseek/deepseek-v4-pro + pt_br: Modelo do OpenRouter, ex. deepseek/deepseek-v4-pro + - container: TZ + description: + en_us: Time zone + ports: + - container: "8899" + description: + en_us: Web UI / API + volumes: + - container: /home/vibe/.vibe-trading + description: + en_us: Settings, memory and skills + pt_br: Configurações, memória e skills + - container: /home/vibe/.vibe-trading/br-data + description: + en_us: Brazilian official datasets (SQLite) + pt_br: Base oficial de dados brasileiros (SQLite) + + # Daily sync of the official Brazilian datasets (B3 COTAHIST, CVM funds and + # companies, Banco Central, Tesouro Direto) into br-data/, where the app's + # get_br_market_data tool and the B3 loader read them. The first run + # downloads several GB and can take hours; later runs only fetch what changed. + vibe-trading-br-data: + profiles: ["br-data-sync"] + image: ${VIBE_TRADING_IMAGE:?Set VIBE_TRADING_IMAGE to a reviewed image tag} + container_name: vibe-trading-br-data + pull_policy: always + restart: unless-stopped + depends_on: + vibe-trading-init: + condition: service_completed_successfully + command: ["python", "-m", "src.br_data", "daemon"] + environment: + TZ: America/Sao_Paulo + # First years to download (B3 publishes from 1986; each year of + # COTAHIST is a few hundred MB in SQLite, options included). + VIBE_BR_COTAHIST_START_YEAR: "2015" + VIBE_BR_CVM_START_YEAR: "2022" + # Daily run time (Sao Paulo), after B3 and CVM publish the day's files. + VIBE_BR_SYNC_TIME: "21:30" + volumes: + - ${VIBE_TRADING_APPDATA:-/DATA/AppData/vibe-trading}/home:/home/vibe/.vibe-trading + - ${VIBE_TRADING_APPDATA:-/DATA/AppData/vibe-trading}/br-data:/home/vibe/.vibe-trading/br-data + # The image's healthcheck probes the web server, which this service does + # not run. + healthcheck: + disable: true + cap_drop: + - ALL + security_opt: + - "no-new-privileges:true" + read_only: true + tmpfs: + - /tmp + mem_limit: 2g + labels: + com.centurylinklabs.watchtower.enable: "false" + +x-casaos: + architectures: + - amd64 + main: vibe-trading + author: incorpore + developer: HKUDS + category: Finance + icon: https://raw.githubusercontent.com/HKUDS/Vibe-Trading/main/assets/icon.png + index: / + port_map: "8899" + scheme: http + title: + en_us: Vibe-Trading + tagline: + en_us: Natural-language finance research agent with backtesting + pt_br: Agente de pesquisa financeira com backtesting + description: + en_us: Vibe-Trading with optional Brazilian dataset sync. diff --git a/zimaos/sync-and-publish.yml.example b/zimaos/sync-and-publish.yml.example new file mode 100644 index 00000000..f41bff05 --- /dev/null +++ b/zimaos/sync-and-publish.yml.example @@ -0,0 +1,170 @@ +name: ZimaOS - Sync upstream and publish image + +# Fork-only workflow (incorpore/Vibe-Trading). It lives in its own file so that +# merging HKUDS/Vibe-Trading never conflicts with it. +# +# 1. sync - every 30 min, merge HKUDS/Vibe-Trading:main into this fork's +# main and push when upstream moved. +# 2. publish - build the Dockerfile and push +# ghcr.io//vibe-trading:latest +# ghcr.io//vibe-trading:sha- +# Watchtower on the ZimaOS box (zimaos/docker-compose.yml) +# pulls :latest and recreates the container. +# +# Token: pushes made with GITHUB_TOKEN cannot touch .github/workflows/** and do +# not trigger other workflow runs. Add a fine-grained PAT as the repository +# secret UPSTREAM_SYNC_TOKEN (this repo only; Contents: read/write, Workflows: +# read/write) so upstream workflow changes can be merged too. Without it the +# sync still works for every upstream commit that does not edit workflows. +# +# Optional repository variables: +# UPSTREAM_REPO default HKUDS/Vibe-Trading +# UPSTREAM_BRANCH default main +# ZIMAOS_PLATFORMS default linux/amd64 (use linux/amd64,linux/arm64 for ARM boxes) + +on: + schedule: + - cron: '*/30 * * * *' + workflow_dispatch: + inputs: + force_publish: + description: 'Build and push :latest even if upstream did not change' + type: boolean + default: true + push: + branches: [main] + +concurrency: + group: zimaos-sync-and-publish + cancel-in-progress: false + +env: + UPSTREAM_REPO: ${{ vars.UPSTREAM_REPO || 'HKUDS/Vibe-Trading' }} + UPSTREAM_BRANCH: ${{ vars.UPSTREAM_BRANCH || 'main' }} + +jobs: + sync: + if: github.event_name != 'push' && github.repository != 'HKUDS/Vibe-Trading' + runs-on: ubuntu-24.04 + permissions: + contents: write + outputs: + changed: ${{ steps.merge.outputs.changed }} + via_pat: ${{ steps.merge.outputs.via_pat }} + steps: + - name: Checkout fork main + uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + with: + ref: main + fetch-depth: 0 + token: ${{ secrets.UPSTREAM_SYNC_TOKEN || github.token }} + + - name: Merge upstream + id: merge + env: + HAS_PAT: ${{ secrets.UPSTREAM_SYNC_TOKEN != '' }} + run: | + set -euo pipefail + echo "via_pat=${HAS_PAT}" >> "$GITHUB_OUTPUT" + git config user.name "github-actions[bot]" + git config user.email "41898282+github-actions[bot]@users.noreply.github.com" + + git remote add upstream "https://github.com/${UPSTREAM_REPO}.git" + git fetch --no-tags upstream "${UPSTREAM_BRANCH}" + + if git merge-base --is-ancestor "upstream/${UPSTREAM_BRANCH}" HEAD; then + echo "Fork already contains ${UPSTREAM_REPO}@${UPSTREAM_BRANCH}." + echo "changed=false" >> "$GITHUB_OUTPUT" + exit 0 + fi + + before="$(git rev-parse HEAD)" + if ! git merge --no-edit "upstream/${UPSTREAM_BRANCH}" \ + -m "chore(sync): merge ${UPSTREAM_REPO}@${UPSTREAM_BRANCH}"; then + git merge --abort || true + echo "::error::Merge conflict with ${UPSTREAM_REPO}@${UPSTREAM_BRANCH}. Merge it manually into main." + exit 1 + fi + + if [ "${HAS_PAT}" != "true" ] && \ + [ -n "$(git diff --name-only "${before}" HEAD -- .github/workflows)" ]; then + echo "::error::Upstream changed .github/workflows; GITHUB_TOKEN cannot push that. Add the UPSTREAM_SYNC_TOKEN secret (see header of this workflow)." + exit 1 + fi + + git push origin HEAD:main + echo "changed=true" >> "$GITHUB_OUTPUT" + + publish: + needs: sync + # - push to main: always (own commits, or a sync pushed with the PAT, + # which fires a push event). + # - manual run: when force_publish, or when the sync merged something. + # - schedule: when the sync merged with GITHUB_TOKEN (that push fires no + # event, so the build has to happen here). + if: >- + always() && github.repository != 'HKUDS/Vibe-Trading' && ( + github.event_name == 'push' || + (needs.sync.result == 'success' && ( + (github.event_name == 'workflow_dispatch' && (inputs.force_publish || needs.sync.outputs.changed == 'true')) || + (needs.sync.outputs.changed == 'true' && needs.sync.outputs.via_pat != 'true') + )) + ) + runs-on: ubuntu-24.04 + permissions: + contents: read + packages: write + steps: + - name: Checkout main + uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + with: + ref: main + + - name: Image name + id: img + run: | + owner="${GITHUB_REPOSITORY_OWNER,,}" + echo "name=ghcr.io/${owner}/vibe-trading" >> "$GITHUB_OUTPUT" + echo "sha=$(git rev-parse --short=12 HEAD)" >> "$GITHUB_OUTPUT" + + - name: Set up QEMU + uses: docker/setup-qemu-action@99012661954931238ded8c8b007157a8430204e1 # v4.4.0 + + - name: Set up Docker Buildx + uses: docker/setup-buildx-action@f87e5991a6d7451dcb8d9637bfbc97413f497069 # v4.4.1 + + - name: Log in to GHCR + uses: docker/login-action@dbcb813823bdd20940b903addbd779551569679f # v4.6.0 + with: + registry: ghcr.io + username: ${{ github.actor }} + password: ${{ secrets.GITHUB_TOKEN }} + + - name: Build and push + uses: docker/build-push-action@c3c9e263c25d99ce0380d002d59b67737d91b0dc # v7.4.0 + with: + context: . + file: Dockerfile + platforms: ${{ vars.ZIMAOS_PLATFORMS || 'linux/amd64' }} + push: true + tags: | + ${{ steps.img.outputs.name }}:latest + ${{ steps.img.outputs.name }}:sha-${{ steps.img.outputs.sha }} + labels: | + org.opencontainers.image.source=https://github.com/${{ github.repository }} + org.opencontainers.image.revision=${{ github.sha }} + cache-from: type=gha,scope=zimaos + cache-to: type=gha,mode=max,scope=zimaos + + - name: Summary + env: + IMAGE: ${{ steps.img.outputs.name }} + SHA: ${{ steps.img.outputs.sha }} + run: | + { + echo "### Published" + echo "- \`${IMAGE}:latest\`" + echo "- \`${IMAGE}:sha-${SHA}\`" + echo "" + echo "Watchtower on ZimaOS will pick up \`:latest\` on its next poll." + } >> "$GITHUB_STEP_SUMMARY"