* feat(loaders): validate OHLC invariants at the loader boundary
Loaders only dropped NaN rows, so structurally invalid bars (high<low, a
non-positive price, or a high/low that fails to bracket open/close) passed
straight into the backtest and surfaced downstream as NaN/inf metrics that
break the strict allow_nan=False JSON serializers.
Add a shared validate_ohlc() helper in loaders/base.py and call it after the
existing dropna in the yfinance and local loaders. The default drops offending
rows and logs a count, matching the loaders' existing silent-clean convention;
"warn" and "raise" strategies are available for stricter callers.
Signed-off-by: Lanre Shittu <136805224+Shizoqua@users.noreply.github.com>
* feat(loaders): apply OHLC validation centrally for every source
Wire validate_ohlc into the runner's single fetch-convergence point so
auto, single-source, runtime-fallback, and any future loader are guarded
uniformly — not just the two loaders wired at source. Adds
_sanitize_data_map(data_map) in runner.py, applied right after the data
map is assembled and before the empty-data check, plus a test that a dirty
bar from an unwired loader is dropped before it reaches the backtest.
---------
Signed-off-by: Lanre Shittu <136805224+Shizoqua@users.noreply.github.com>
Malformed values like CCXT_TIMEOUT_MS=abc / OKX_TIMEOUT_S=abc raised ValueError at import time and could block startup. Add shared positive_env_int/float helpers in loaders/base.py; invalid/blank/non-positive timeout/budget values now warn and fall back to defaults. Valid values still honored.
Closes#227.
Add an opt-in local market-data cache (VIBE_TRADING_DATA_CACHE) for the backtest
loaders. When enabled, every data source — tushare, okx, ccxt, akshare, mootdx,
yfinance, futu — caches settled historical bars under ~/.vibe-trading/cache
(user home, never the repo tree) so repeated and long-horizon / cross-market
backtests skip the network and avoid provider rate limits.
- Generic loader_cache_get / loader_cache_put / cached_loader_fetch helpers in
the loader base; batch/connection loaders (yfinance, futu) skip the bulk
download / FutuOpenD connection entirely on a full cache hit.
- Staleness guard: ranges ending today or in the future are never cached, since
the key is content-addressed on end_date, not fetch time.
- Cached frames round-trip byte-identical to freshly fetched ones (columns name
and per-level index dtype preserved across the duckdb parquet round-trip).
- pid + uuid tmp filenames for concurrency-safe atomic writes; cache version 2.
- Documented in .env.example; staleness, real-duckdb, and batch-loader tests.
Closes#171.
Signed-off-by: Haozhe Wu <haozhe_wu@connect.hku.hk>
ccxt_loader and okx each carried their own copy of the same 25-line
retry+budget loop introduced in #121. Hoist the pattern into two
helpers — check_budget() and retry_with_budget() — so any future loader
(yfinance, akshare, tushare) inherits the same guarantees: per-call
transient retries, a wall-clock budget, non-transient exceptions left
unwrapped, original exception preserved as __cause__.
ccxt_loader / okx now declare only what is loader-specific (the
exception class for transient errors and the budget env var); the
scheduling, sleep math, and TimeoutError wrapping live in base.py.
Adds 11 helper-level tests + the 11 pre-existing integration tests
still pass against the refactored loaders.
Follow-up to #121.
- ffill(limit=5) replaces unlimited ffill to prevent stale prices from
masking long suspensions; all-NaN symbols auto-dropped with warning
- try/except per symbol in _execute_bars so one symbol failure doesn't
crash the entire backtest
- pydantic BacktestConfigSchema validates config.json before execution
(codes non-empty, date format, start<=end, valid interval/engine/source)
- validate_date_range() shared helper called from all 5 loaders
- 25 regression tests covering all four fixes
- Add DataLoader Protocol + Registry with market-level fallback chains
- New data sources: AKShare (A-shares/US/HK/futures, free) and CCXT (100+ crypto exchanges)
- Auto-fallback: tushare unavailable -> akshare; OKX unavailable -> CCXT
- Zero-config: all markets work without any API keys
- New web_search tool (DuckDuckGo) for web information retrieval
- Skill categorization: 68 skills organized into 7 categories (data-source, strategy, analysis, asset-class, crypto, flow, tool)
- New data-routing meta-skill for intelligent data source selection
- Registry self-initializes via _ensure_registered() — no manual imports needed
- Pattern detection returns market type instead of source name (decoupled)
- runner.py loads .env via dotenv for reliable token access
shadowinlife