wiki/research-lab/posts/alpha-191-in-2026.tweet.md was an internal
marketing draft (pre-flight checklist + draft tweet text) that should
not have been committed to a public path. Moved to docs/alpha-zoo/
(gitignored) and removed from main.
Note: the file remains in git history at commit 137689c. If history
scrub is required, a force-push rewrite is needed (separate decision).
wiki/scripts/build_alpha_library.py (475L)
- Reads manifest.json produced by `vibe-trading alpha export-manifest`
- Emits per-alpha pages (content/<zoo>/<id>.html) and per-zoo
overview tables (content/<zoo>/index.html) — 452 + 4 = 456 pages
- Jinja2 with select_autoescape(["html","xml"]) mandatory; no |safe
on user-derived content; strict CSP on every generated page
(default-src 'self'; style-src 'self' 'unsafe-inline';
script-src 'none'; img-src 'self' data:)
- Emits content/index.json for the landing page to hydrate per-zoo
counts via fetch()
- Friendly exit when manifest.json missing
- End-of-main self-test compares manifest formula content against
rendered HTML to verify the escape pipeline; prints
"html escape ok" or "html escape FAIL"
wiki/alpha-library/index.html (rewritten from 78-line shell to 192L)
- Hero + 4 zoo cards (Kakushadze 101 Formulaic Alphas, qlib158,
gtja191, academic) + stats grid + 3-line CLI quickstart + GitHub
footer
- Per-zoo counts hydrated at runtime from content/index.json
- Loose CSP (script-src 'self' 'unsafe-inline') only on this landing
page to keep the site theme-init inline script and the inline
fetch() — generated /content/** pages stay on script-src 'none'
wiki/research-lab/index.html
- Post grid replacing the empty hero stub; links to the new blog
- One-line lede explaining the section
wiki/research-lab/posts/alpha-191-in-2026.html (~675L, ~2400 words)
- 6-section deep-dive: TL;DR / Background / Method / Findings /
Caveats / Reproduce
- Numbers from a real bench run on CSI 300 / 2018-2025: 10 alive,
15 reversed, 165 dead (of 190 tested, 1 skipped). Top 5 by IR:
gtja191_171 / _111 / _163 / _002 / _054. Theme survival rates:
microstructure 22%, reversal 11%, volatility 8%, volume 5%,
momentum 2%, liquidity 0%
- Survivorship-bias caveat for cross-zoo readers (the current
constituent list of any index is a forward-biased proxy for the
point-in-time constituents during the window)
- Strict CSP (script-src 'none'); no MathJax/KaTeX — formula_latex
rendered as <code class="formula">; aggregate stats only (no
per-stock per-date values, Tushare ToS)
wiki/research-lab/posts/alpha-191-in-2026.tweet.md
- HKUDS-style draft tweet (Chao Huang voice): emoji-decorated
project name, real numbers (10 / 15 / 165), 8-year OOS qualifier,
GitHub + pip links, ≤2 hashtags. READY to post once the release
goes live.
.gitignore adds wiki/alpha-library/manifest.json and
wiki/alpha-library/content/ as generated, never-checked-in artifacts.
End-to-end product surfaces over the factor framework: a CLI for
power users, an agent-tool layer for natural-language workflows,
a REST + SSE API for browser clients, and the Web UI itself.
CLI (vibe-trading alpha <subcommand>)
- list / show / bench / compare / export-manifest with security
guards (path-traversal, alpha_id allowlist before any FS read)
- agent/cli.py wired (7-line glue), all logic in cli_handlers.py
- alpha bench: requires --zoo (or --yes); drives bench_runner with
an on_progress callback feeding a Rich Progress bar; prints ETA
banner; emits stderr summary "✓ Bench complete. N tested, M
skipped. Report: <path>"; returns exit 1 on status != ok
- alpha list: --limit N / --json flags; TTY next-step hints
- alpha show: --brief; Syntax(word_wrap=True) so source survives
narrow terminals
- TUSHARE_TOKEN errors append a 3-step fix banner
- ConstantInputWarning silenced at module level
Agent tools (auto-discovered via BaseTool.__subclasses__)
- AlphaZooTool: list_alphas / get_alpha / health
- AlphaBenchTool: orchestrator with Jinja2 autoescape HTML report +
strict CSP meta. Module-level run_alpha_bench() reused by CLI
Universe loader (agent/src/tools/alpha_bench_tool.py)
- csi300 → Tushare index_weight + pro.daily, in-house panel build
because the bundled tushare loader drops `amount`. Parallel fetch
via ThreadPoolExecutor(max_workers=4); cache at ~/.vibe-trading/
cache/<universe>_<start>_<end>.pkl with sha256 sidecar integrity
guard
- sp500 → Wikipedia constituent list (UA-spoofed; bare pd.read_html
returns 403) → yfinance loader; surfaces survivorship_bias=True
in panel._meta because the constituent set is point-in-NOW
- btc-usdt → bundled okx loader; fails fast (ValueError) on single-
asset cross-sectional IC degeneracy
REST API (agent/src/api/alpha_routes.py + agent/api_server.py +6 lines)
- GET /alpha/list?zoo=&theme=&universe=&limit= — paginated catalogue
- GET /alpha/{alpha_id} — meta + source code via Registry.get_source
- POST /alpha/bench — kicks off asyncio.create_task running the
bench in asyncio.to_thread; in-memory job state with threading.Lock
+ 1-hour TTL + asyncio.Semaphore(2) concurrency cap (429 on
saturation)
- GET /alpha/bench/{job_id}/stream — Server-Sent Events emitting
progress / result / done / error events. Heartbeat ": ping\n\n"
every 15s so reverse proxies don't close idle streams. Unexpected
exceptions are sanitised to "internal error; see server logs"
Web UI (frontend/src/pages/AlphaZoo.tsx ~950 LOC, 3 sub-views)
- Browse: 4 gradient zoo cards (qlib158 / Kakushadze 101 / gtja191 /
academic) + filter bar (zoo, theme, universe, id substring) +
paginated table (50 rows + Load more)
- Detail: formula_latex in <code> block (no MathJax under strict
CSP); metadata table; collapsible <details> source code; "Run
bench" CTA prefilled
- Bench: form → SSE-streamed progress bar + currently-computing
alpha id → Alive/Reversed/Dead/Skipped stat cards + Top-5-by-IR +
Most-Reversed tables + by-theme echarts bar chart
- New /alpha-zoo route triplet wired in router.tsx; Layout.tsx
nav entry with the Layers icon
- Raw EventSource pattern (useSSE hard-codes agent-stream event
types and would drop our progress/result/done/error)
- a11y: htmlFor/id pairs on every label, aria-hidden on decorative
lucide icons, sr-only table caption, CategoryBadge component
- BTC universe fail-fast surfaces as inline form error with hint
to try sp500/csi300
Registry singleton via get_default_registry() so API requests don't
each pay the 452-module AST scan. compute_ic_series vectorised via
pandas corrwith(method="spearman"); ~5-10x speedup, numerically
identical to the prior scipy loop within 1.1e-16.
VWAP 千元 fix: empirical Tushare probe against 000001.SZ confirmed
daily.amount is in 千元 (not raw CNY). base.vwap(EQUITY_CN) now does
safe_div(amount * 1000.0, volume * 100.0 + 1.0) — previously
~1/1000th of true vwap. Affects factors mixing vwap with price
additively (alpha101_093 etc.).
969 tests pass; registry health 452 loaded / 0 failed; CI grep
gates green; TypeScript clean.
Implements four production-ready alpha catalogues, each in its own
agent/src/factors/zoo/<zoo>/ subdirectory with a per-zoo LICENSE.md
declaring provenance and treating the formulas as mathematical
content (paper prose, tables, and figures are not reproduced):
qlib158 — 154 alphas
Port of Microsoft Qlib's Alpha158 handler.py feature definitions
(github.com/microsoft/qlib @ d5379c5, Apache-2.0). Each module
carries a per-file header citing the upstream commit SHA + path.
Includes the upstream NOTICE per Apache-2.0 §4(d). 4 raw price-
reference fields ($open/$high/$low/$vwap) intentionally omitted —
they are inputs, not signals, and conflict with the wide-DataFrame
contract.
alpha101 — 101 alphas
Implementation of Kakushadze (2015) "101 Formulaic Alphas"
(arXiv:1601.00991), written from the paper appendix. 19 industry-
neutral alphas (#48, 56, 58, 59, 63, 67, 69, 70, 76, 79, 80, 82, 87,
89, 90, 91, 93, 97, 100) flagged requires_sector=True so the
registry SkipAlpha-s them on panels without a sector tag. Display
name is "Kakushadze 101 Formulaic Alphas" — the trademarked
alternative string is intentionally absent (enforced by ci grep
gate b).
gtja191 — 191 alphas
Implementation of Guotai Junan Securities' 2014 "191 Short-period
Trading Alpha Factors" research report. Ambiguous formulas
(MAX/MIN set vs ts, sign convention, SMA(x,n,m) → x.ewm(alpha=m/n),
WMA → decay_linear approximation, REGBETA → ts_cov/ts_std**2,
HIGHDAY/LOWDAY → ts_argmax/ts_argmin) are documented per-alpha in
__alpha_meta__["notes"]. Long-horizon SUM/MEAN windows
(alphas #25/26/33/39/45/56/61/64/74/75/77/91/92/98) shortened on
the production code path so outputs stay defined on short panels;
documented in each affected alpha's notes.
academic — 6 factors
Honest price-based proxies for Fama-French 5 + Carhart momentum;
nicknames carry a [PRICE PROXY] prefix because the canonical FF
series require book-to-market / profitability / investment-growth
fundamentals we do not ship. Kenneth French's data library is
referenced (URL only) for users who need canonical monthly returns.
Internal _effective_window helpers shrink the 252/60-day windows on
short panels so the lookahead test and the registry NaN-ratio guard
pass; production-sized panels see the canonical window untouched.
Sample golden CSVs at agent/tests/factors/fixtures/goldens/ for
five sampled alphas per zoo, asserted at rtol=1e-6, equal_nan=True.
ci grep gates pass; AST purity + lookahead tests both green on the
full 452-alpha registry.
New module agent/src/factors/ providing the foundation for the Alpha
Zoo:
- base.py: 19 cross-sectional + time-series operators on wide
pd.DataFrame panels (rank/scale/ts_*/delta/decay_linear/signed_power/
safe_div + market-aware vwap). NaN propagates strictly; delta(d>=1)
enforced at the operator layer; no negative shifts allowed.
- registry.py: AST-extracts __alpha_meta__ from each zoo .py module
without importing it; pydantic AlphaMeta (extra="forbid",
frozen=True). Lazy importlib.import_module on compute(); filesystem-
loader fallback for plugin/test trees. Sanity gates reject +/-inf,
NaN ratio > 95%, and shape mismatches.
- factor_analysis_core.py: IC/IR + layered backtest math extracted
from agent/src/tools/factor_analysis_tool.py so it's reusable from
CLI, bench tool, and API.
- bench_runner.py: per-alpha compute → IC vs forward return →
alive/reversed/dead categorisation (mean IC > 0.02, pos_ratio >=
0.55, |t-stat| > 2). Shared by CLI driver and API worker.
Multi-factor skill gains a panel-style ZooSignalEngine.from_zoo()
factory (cross-sectional standardisation + weighting + optional
top-N / bottom-N long-short conversion). The legacy per-symbol
example_signal_engine.py is left untouched. SKILL.md edits in
multi-factor and factor-research cross-reference the new zoo.
Safety floor:
- test_alpha_purity.py: AST allowlist scan of every zoo/**/*.py
module. Allowed imports: pandas, numpy, scipy.*, src.factors.base,
__future__, typing, math, dataclasses. Banned names: os/sys/
subprocess/socket/urllib/requests/httpx/pathlib/Path/open/eval/exec/
compile/__import__, plus dunder-access via getattr(_, "__*"),
breakpoint, input, help, memoryview, globals, locals, vars, and
the __class__/__bases__/__mro__/__globals__/__builtins__ chain.
Module-level statements limited to docstring + imports + functions
+ ALPHA_ID/__alpha_meta__ assignments with literal RHS.
- test_lookahead.py: 300-row synthetic panel; corrupt rows >= 270;
assert factor at probe row 260 unchanged within 1e-9.
- test_base.py + test_registry.py + conftest.py with opt-in
pytest-socket integration that hard-fails outbound network during
the factors test suite.
- tools/ci_grep_gates.sh: rejects yaml.load( without safe_load,
any trademarked-name leak in shipped artifacts, and any per-stock-
code data leak in wiki/**/*.{json,csv,html}.
Governance:
- CONTRIBUTING.md with the Developer Certificate of Origin trailer
for community PRs and the alpha PR reviewer checklist.
- NOTICE at the repo root with Apache-2.0 attribution for the
upstream qlib NOTICE and the formulas-as-mathematical-content
stance for Kakushadze / GTJA / Fama-French sources.
ccxt_loader and okx each carried their own copy of the same 25-line
retry+budget loop introduced in #121. Hoist the pattern into two
helpers — check_budget() and retry_with_budget() — so any future loader
(yfinance, akshare, tushare) inherits the same guarantees: per-call
transient retries, a wall-clock budget, non-transient exceptions left
unwrapped, original exception preserved as __cause__.
ccxt_loader / okx now declare only what is loader-specific (the
exception class for transient errors and the budget env var); the
scheduling, sleep math, and TimeoutError wrapping live in base.py.
Adds 11 helper-level tests + the 11 pre-existing integration tests
still pass against the refactored loaders.
Follow-up to #121.
The substring head-match (a) false-positived on nested status fields and
(b) false-negated when the error envelope sat past the 160-char head.
Parse the result as JSON and check only the top-level status field; fall
back to the original substring scan when the payload is truncated /
unparseable so the classifier never raises on the worker hot path.
Adds 7 unit tests pinning the new behavior — nested-status non-match,
past-head match, truncated fallback, non-error statuses (degenerate /
warning) correctly classified as non-error.
Follow-up to #119.
_internal_roots resolves 8 candidate paths + builds 3 separator variants
+ sorts on every redact_internal_paths call. On the worker hot path this
runs once per task error / event / file-tool failure. Result is process-
stable (paths don't change without a restart), so @cache it.
Follow-up to #119.
* fix(swarm): surface task error at read boundaries
run_swarm / get_swarm_status / get_run_result and the in-process
run_swarm tool each hand-maintained a field allowlist that omitted
SwarmTask.error. A misconfigured provider returned status="failed"
with no diagnosable reason, though the error was captured on disk
the whole time.
Add src/swarm/serialization.py as the single source of truth for the
per-task projection (now includes error + iterations) plus a run-level
error summary; route all three boundaries through it so the allowlist
cannot silently drift again. Additive JSON change; the only shape
normalization is _format_result missing-summary "" -> null.
(cherry picked from commit 95f2717ac32bd02829f650bc971e3cd6dd7fffa6)
* fix(swarm): enforce worker output contract
A worker that emitted only its plan ("Phase 1 — Plan"), mock data,
unparsed tool markup, or (for a data agent) made no tool call and
wrote no report was returned status="completed". runtime.py also
folded timeout/token_limit into completed. The run reported success
with a stub as final_report (P01); degraded workers were invisible
(P03).
Add WorkerStatus.incomplete and a hybrid output contract in worker.py:
content-sanity for every agent, tool-evidence only for data agents so
tool-less synthesis/editor roles are not false-rejected. Fail-fast to
incomplete (no corrective retry — left for a follow-up). runtime.py
only maps a true completed to TaskStatus.completed; anything else
fails the run, so final_report can no longer be a non-deliverable.
(cherry picked from commit 774b26824509678c953cd0200ef7a56c03f23201)
* fix(swarm): thread shell tools + unify runs root
The MCP run_swarm wrapper called start_run() without
include_shell_tools, so stdio swarm workers silently lost bash and
could not run the scripts their own execution rules mandate (P03-B).
build_filtered_registry also dropped a requested-but-unavailable
tool with no log, hiding the contradiction.
Pass the server's shell-tool policy into start_run and warn on a
dropped tool. Add swarm_runs_root() as the single source of truth
for the run store location, shared by mcp_server and the path_utils
run-dir allow-list so the two can no longer drift (P03-A: an
installed-wheel layout had put every worker run_dir outside it).
(cherry picked from commit 2975fda09b5ad44d15580d9330890f72f43c00ce)
* fix(swarm): make store atomic write Windows-safe
SwarmStore._atomic_write did tmp.replace(target) with no retry. On
Windows os.replace raises PermissionError WinError 5/32 when the
target is concurrently open by load_run on the poll path, crashing
the worker thread and leaving the run stuck pending (P13). POSIX
os.replace is atomic and never raises these.
Add a WinError-scoped, bounded retry around the rename and a matching
transient read/parse retry in load_run (the other side of the same
race). Non-transient errors re-raise immediately; off-Windows the
loop runs once, so POSIX behavior is unchanged. Pre-existing, not
introduced by the P01/P03/P04 fixes — surfaced by E2E dogfooding.
(cherry picked from commit 46a779cf436353591e7840879f8c32f763dde826)
* style(swarm): drop unused imports (ruff F401)
Remove an unused `from pathlib import Path` in the PR03 test added by
2975fda, and the pre-existing unused `from src.swarm.mailbox import
Mailbox` in runtime.py (file already in this branch's diff). Net ruff
delta for PR1-PR4 is now -1. The remaining mcp_server.py:41 E402 is
pre-existing on upstream and structural — left out of scope.
(cherry picked from commit e96f56af1ead9b5b7058c90ec0d4df738b45e473)
* fix: redact internal absolute paths in errors
Unknown preset / workspace-escape / missing run-or-task dir errors
embedded the full install path (OS username, .venvs/site-packages
topology) — CWE-209/497, reachable via MCP/API/agent/CLI (P10).
Drop the absolute path while keeping the actionable bits: preset
name + Available list, the workspace-root boundary, and the logical
run/task id (basename only). Two test_path_safety assertions that
matched the old leaking message are updated to the redacted wording
(behavior unchanged — still raises and rejects). Verified-clean
sites (caller's own input / env-var / names list) left untouched.
(cherry picked from commit 69ea45295410cd25ea3079c25b7e3c4c73390b80)
* fix(security): redact internal paths in errors
Add agent/src/tools/redaction.py: anchored prefix redaction (no
regex, idempotent, None-safe) that hides only known-internal root
prefixes while keeping the relative tail for diagnosability.
Wire it through the swarm error read boundaries (serialization,
runtime, store) and the file tools' final broad except so a leaked
absolute path cannot reach a user-facing string (CWE-209/497, P10).
ValueError branches and the protected providers/llm.py are left
untouched.
(cherry picked from commit 21dc9cdfacf17cccd31cacfbe34481f62f1390af)
* fix(security): redact swarm event error payloads
---------
Co-authored-by: Haozhe Wu <haozhe_wu@connect.hku.hk>
* fix(mcp): surface unresolved market-data symbols
A typo / wrong-suffix / delisted / no-data code vanished from
get_market_data's result (the dict only held winners), so it was
indistinguishable from "no data", and a loader exception lost every
already-resolved symbol (P05).
Report any requested code that produced nothing under a reserved
`_unresolved` key (additive — omitted when all codes resolve, so the
all-resolved payload is byte-identical to before) and wrap the
per-group loader.fetch so a blow-up is contained, not opaque. Richer
per-symbol status objects are a deliberate breaking follow-up.
(cherry picked from commit eba2044e982c5d467c808bad05386de3d856d635)
* fix(tools): reject invalid analyze_options inputs
Invalid inputs (volatility<=0, spot/strike<=0, negative expiry,
bad option_type) and T=0 all returned status:"ok" with silently
wrong / zeroed Greeks, and NaN could leak into the JSON (P06).
Validate at the boundary and reject invalid inputs with an error
envelope; flag the legitimate T=0 expiry as status:"degenerate"
with a warning (intrinsic value still returned); serialize with
allow_nan=False. The normal path is numerically unchanged. The
Greeks-convention rework (delta=0.5 at expiry) is a larger
follow-up, out of scope here.
(cherry picked from commit 823dc2382d73b97919807cf14a3644073886247d)
* fix(mcp): cap get_market_data row output
Every row of every symbol was emitted, so the canonical "1 symbol,
1 year, daily" request (~251 rows) breached the MCP token budget and
had to spool to a file, defeating an inline data tool (P07).
Add a max_rows cap (default 250): a symbol exceeding it returns a
head+tail window plus truncation metadata (rows/returned/truncated/
hint); symbols within the cap are unchanged (plain list, so small
queries are byte-identical); max_rows=0 restores the unbounded legacy
behavior. Columnar encoding and envelope de-duplication are broader
follow-ups, out of scope here.
(cherry picked from commit 0e426b1d0523e8c8144c699f89cd4847199e6448)
* fix(options): reject nan/inf + log dropped loader
options_pricing previously accepted NaN/Inf numeric inputs and
silently produced unreliable Greeks. Reject any non-finite spot,
strike, expiry_days, volatility, or risk_free_rate at the
validation boundary before pricing, and treat a missing result
key as non-finite so a dropped field is flagged degenerate
instead of defaulting to a confident 0.0.
mcp_server.get_market_data: the contained loader-failure path
(P05) now logs the swallowed exception with logger.exception
instead of vanishing silently; the _unresolved fallthrough is
unchanged.
* fix(mcp): stride row cap, reject neg max_rows
_cap_rows previously returned a head+tail window with a synthetic
{_gap: N} sentinel, hiding the middle of the series and emitting a
fake non-bar record. Replace with an even-stride downsample (every
step-th bar, last bar pinned) so the capped series spans the full
date range with only real bars.
A negative max_rows now enforces the default cap instead of being
treated as <=0 (which silently disabled the cap and returned an
unbounded payload). max_rows=0 still means unbounded; the in-cap
plain-list path is unchanged.
* fix(loaders): bound ccxt fetch timeout + retry
ccxt_loader set no request timeout and paginated fetch_ohlcv in an
unbounded loop with no retry budget, so a transient disconnect hung
get_market_data for 10+ minutes (P12-b — it stalled a crypto E2E).
Set an explicit per-call timeout on the exchange, retry the transient
ccxt.NetworkError family with bounded backoff, and enforce a hard
wall-clock budget that raises a clear TimeoutError instead of
hanging. Non-network errors (e.g. bad symbol) are not retried; the
happy path still issues one call per page (no behavior change).
okx.py has the same shape — noted as a follow-up, out of scope here.
(cherry picked from commit ef934fbd70209d74a110e81757dbe322a2b717ab)
* fix(loaders): bound okx fetch retry + budget
okx.py set a per-request timeout but had no retry and no wall-clock
budget, so a transient blip dropped the whole symbol via the outer
silent skip, and a slow keyless tier could stall roughly
max_pages * timeout (P12-b parity with the ccxt loader).
Retry the transient requests.RequestException family with bounded
backoff and enforce a hard wall-clock budget that raises a clear
TimeoutError; non-network errors are not retried; the happy path
still issues exactly one request per page (no behavior change).
(cherry picked from commit e5030bf3edc3e44cc27003b3ac23e9e4fefc3c97)
- SKILL.md now explicitly notes that read_url forwards the full URL to the third-party Jina Reader (r.jina.ai), and warns against passing credentials or internal URLs.
- New optional no_cache=True parameter sends x-no-cache to bypass Jina's cache when freshness matters.
- Responses that contain Jina's cached-snapshot marker now surface a cached: true flag.
- Error paths keep the upstream HTTP body and exception text in the returned error string (and also log them via logger.warning) so operators can diagnose remote failures without grepping logs.
Reporter saw 'Execution failed: unknown' when AgentLoop hit
max_iterations. The run's state.json recorded a useful reason but
loop.run()'s returned dict omitted it on the cancelled and failed
branches, so SessionService.mark_failed got 'unknown'.
Populate reason on both branches with actionable phrasing and add
iterations + max_iterations to the result dict.
Fixes#114
Lift the anti-fabrication rule from grounding.py into build_worker_prompt
as a universal Data Citation Discipline section that renders
unconditionally. Free-form prompts without explicit symbols previously
escaped the rule; aggregator / synthesizer agents with no data tools
had no defense against inventing numbers upstream omitted. Applies to
all 29 swarm presets via the central prompt builder.
Fixes#106
* feat(memory): harden PersistentMemory.add() for control bytes, length, empty names
Bundles three follow-up fixes for the same module, requested by
@warren618 in the issue triage of #108, #109, and #110.
#109 (highest priority of the three) — `add()` previously wrote the
full body to disk while `_scan_entries` clipped reads at
`MAX_ENTRY_CHARS`, so any content past the limit was "saved but
invisible" from inside Vibe-Trading. Enforce the limit at write time
and append a visible ``[truncated at N chars]`` marker so anyone
inspecting the file sees the clip explicitly.
#110 — Empty / whitespace-only names now raise ``ValueError``
("memory name must not be empty or whitespace-only") instead of
collapsing to the same ``{type}_.md`` filename. For names whose
sanitized slug normalizes to just underscores (emoji-only,
punctuation-only, etc.), append a 6-char SHA-256 prefix so distinct
inputs produce distinct files. The same input still produces the
same filename, so the "edit memory" workflow is unaffected.
#108 — Strip C0 (``U+0000-U+001F`` except ``\t`` ``\n``) and C1
(``U+0080-U+009F``) control bytes from agent-supplied content before
persisting. These cannot legitimately appear in finance notes and
would otherwise replay back through `memory show` as ANSI sequences
into the user's terminal.
Implementation:
- New module-level helpers `_sanitize_body` and `_truncate_body`
(small, pure, regression-tested in isolation).
- New `_CONTROL_CHAR_RE` regex constant + `_TRUNCATION_MARKER`
template, named for readable error/diff output.
- `add()` now strips control bytes before truncation so the marker is
computed against user-visible content length.
Tests:
- 5 unit tests for `_sanitize_body` (ANSI escape, NUL/BEL, tab/newline
preservation, C1 range, empty input).
- 4 unit tests for `_truncate_body` (short passthrough, at-limit,
over-limit with marker, custom limit).
- 3 tests for empty/whitespace/tab-only name rejection.
- 3 tests for hash-suffix slug disambiguation (two emoji names produce
distinct files; same name is deterministic; punctuation-only also
gets a hash).
- 2 integration tests for `add()` end-to-end (control byte stripped on
disk; long body truncated with marker on disk).
All 108 existing memory/CLI/remember-tool tests still pass.
Closes#108, #109, #110.
* fix(memory): reserve truncation marker space inside MAX_ENTRY_CHARS limit
E2E showed that appending the marker AFTER `body[:MAX_ENTRY_CHARS]`
makes the marker live past byte 8000 — _scan_entries clips reads at
MAX_ENTRY_CHARS, so consumers that inspect `entry.body` (rich.print,
recall scoring) never see the marker, defeating the "explicit
truncation" goal of #109.
Reserve the marker space *inside* the limit instead: head =
content[:limit - len(marker)] + marker. Total on-disk body stays ≤
MAX_ENTRY_CHARS, the marker survives the read-side clip, and
programmatic readers see the marker exactly the same as `cat` does.
Trade-off: when content exceeds the limit, ~28 chars of head content
is dropped to fit the marker. That is intentional — surfacing the
truncation event is more valuable than retaining 28 more bytes of
content that nobody could see was clipped.
Tests updated to assert the new invariant (`len(out) <= MAX`,
`out.endswith("chars]\n")`).
Add a static Cloudflare Pages wiki under wiki/, including the landing page, docs app, redirects/headers, maintainer README, and wiki validation workflow.
The previous CJK tokenizer ranges (#87, #95) only matched ``一-鿿``
and ``㐀-䶿``, so memory entries with Thai, Arabic, Hebrew, or
Cyrillic titles:
- Tokenized to the empty set, making recall always miss (e.g.
``find_relevant("ถัวเฉลี่ย")`` returned nothing even when the body
contained the word).
- Had their slug characters stripped to ``_``, so two distinct Thai
titles of equal length silently overwrote each other on disk.
The new ``_NON_LATIN_SCRIPT_RANGES`` constant covers CJK + Thai +
Arabic + Hebrew + Cyrillic and is reused by:
- ``_TOKEN_RE`` — single alternation pattern, one ``re.findall`` per
``_tokenize`` call (one text scan instead of two; precompiled at
module level so it doesn't go through ``re.compile`` cache lookup on
each invocation).
- ``_SLUG_DISALLOWED_RE`` — negation pattern used by ``add()``.
Arabic and Hebrew are deliberately narrowed to the basic letter blocks
(U+0620-U+064A, U+05D0-U+05EA) to keep bidi-control codepoints like
U+061C ARABIC LETTER MARK and combining marks out of on-disk slugs,
where they would render as invisible-but-distinct filenames.
Tests cover tokenization for each script, slug preservation
(parametrized across the four new scripts), and a Thai collision-
distinction regression.
Out of scope: ``agent/src/session/search.py`` has the same CJK-only
range in its FTS sanitizer; worth a follow-up PR to consume the same
constant.
The frontmatter parser returns lists for ``[a, b]`` syntax and bools
for ``true``/``false``, but MemoryEntry annotates ``title``,
``description``, and ``memory_type`` as ``str``. The mismatch was latent
until the new memory CLI tried to render those fields through
``rich.markup.escape``, which raised ``TypeError: expected string or
bytes-like object, got 'list'`` for any description that looked like a
YAML list (e.g. ``description: [red]inject[/red]``).
Add a small ``_coerce_str`` helper and route the three frontmatter
fields through it in ``_scan_entries`` so the dataclass type contract is
truthful at the storage boundary. Other consumers (recall scoring,
``remember_tool``) also benefit transparently.
Tests:
- Unit coverage of ``_coerce_str`` for str/None/list/bool inputs
- Storage-layer regression: a markdown file with a bracketed description
parses to a string-typed description
- CLI regression: ``cmd_memory_list`` no longer crashes on entries with
a bracketed description
The previous two commits inadvertently captured whole-file output from a
local pre-tool-edit formatter, reformatting ~30 functions
(`_print_status_bar`, `_format_tool_call_args`, `_build_benchmark_table`,
`_print_result`, `_build_welcome_panel`, ...) that have nothing to do
with the memory introspection feature.
Restore those functions to upstream `main` while keeping only the
intentional changes for this PR:
- `from datetime import datetime` hoisted to top imports (so the new
CLI commands can use it without a per-call local import)
- `from rich.markup import escape as rich_escape` for safely rendering
user-supplied memory titles/descriptions
- New `memory` subparser block in `_build_parser`
- New `cmd_memory_list / show / search / forget` plus the local
`_MEMORY_TYPES` constant and `_MEMORY_TYPE_STYLES` dict
- Dispatch for `args.command == "memory"` in `main()`
- Drop the now-redundant local `from datetime import datetime` inside
`cmd_trace`
Net diff for agent/cli.py shrinks from +296/-53 to +170/-1.
Hoisting PersistentMemory to top-level cli.py made every CLI invocation
import `src.agent.frontmatter`, which transitively triggers
`src/agent/__init__.py` and eagerly loads AgentLoop, SkillsLoader (74
skills), and ToolRegistry. `vibe-trading --version` measured 7.6s
because of this chain.
- Move `from src.memory.persistent import PersistentMemory` back into
each `cmd_memory_*` so only memory commands pay the import cost
- Mirror MEMORY_TYPES as a local `_MEMORY_TYPES` tuple inside cli.py so
argparse `choices` and `_MEMORY_TYPE_STYLES` no longer require the
module-level import. The module-load assertion still guards against
drift between the local tuple and the style dict
`vibe-trading --version` drops from 7.6s to 0.66s (11.5x). Memory
commands keep the same latency since they need the module anyway.
Add `vibe-trading memory list/show/search/forget` so users can inspect
and manage persistent memory without opening the file store directly.
Two recent memory bugs (#87 underscore tokenization, #95 CJK slug
collisions) were hard to spot without observability into what is
actually saved and recalled.
Also expand the PersistentMemory public surface to keep CLI lookup
logic out of storage internals:
- MEMORY_TYPES tuple as the single source of truth for type names,
shared by argparse choices and the CLI style table
- list_entries() / find() / remove_entry() back the new commands;
find() resolves by title first, then by on-disk filename stem so
users can paste either form from the index
- remove_entry(entry) skips the re-scan that remove(title) does
Existing remove(title) is unchanged; remember_tool stays green.
Address PR review feedback (HKUDS#84):
1. Read report.md back into WorkerResult.summary:
- Add `_resolve_summary()` that checks for report.md in artifact dir
- Applied at all 5 exit points (timeout, token_limit, llm_failure,
normal_completion, iteration_limit)
- Ensures API/UI shows the full report, not just the brief closing text
2. Sanitize event logging:
- Truncate tool arguments to 200 chars per value
- Rename "result" to "result_preview" and reduce from 5000 to 500 chars
- Avoids persisting/streaming large or sensitive tool payloads
3. Add regression tests for report → summary handoff:
- Test report.md content is returned when present
- Test fallback behavior when report.md missing or empty
- Test error handling for unreadable paths
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Add `agent_root / ".swarm" / "runs"` to `_default_run_roots()` in
path_utils.py. Without this, all `write_file` / `edit_file` / `read_file`
calls from swarm workers fail with "run_dir is outside allowed run roots".
- Increase `timeout_seconds` from 600 to 1800 for all 29 swarm presets.
The aggregator agent in commodity_research_team timed out at 615s, and
complex analysis tasks routinely exceed 600s.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Swarm workers frequently produced no usable output because:
- The Execution Rules prompt asked agents to write summaries "directly in
your response," but the worker only captured `last_assistant_content`
(often a mid-process message like "let me fix the script").
- On timeout/iteration limit, all in-memory messages were discarded.
- Event logging recorded only tool names and elapsed time, not I/O.
Changes to worker.py:
- Force agents to write final reports via `write_file report.md` (Phase 3),
decoupling report persistence from the in-memory message lifecycle.
- Add `_best_summary()` to extract the longest assistant message (>100
chars) instead of relying on the last fragment.
- Add `_persist_messages()` to save full conversation to messages.json on
timeout/iteration-limit for post-mortem debugging.
- Include tool arguments in tool_call events and result content (up to 5000
chars) in tool_result events for better observability.
Changes to equity_research_team.yaml:
- Increase timeout_seconds from 600 to 900 for all four agents, as the
aggregator previously timed out at 615s.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Empirical end-to-end verification on a real run revealed the dispatch
in fetch_grounding_data was wrong:
loader_cls = resolve_loader(code) # resolve_loader takes a market key
loader = loader_cls() # resolve_loader already returns
# a ready instance, not a class
So a real call with code="NVDA.US" produced
"No available data source for market 'NVDA.US'. Tried: []." — the
fallback chain table is keyed by market type ("us_equity",
"hk_equity", "crypto", ...), not by raw symbol shape.
The unit tests passed only because they monkey-patched resolve_loader
to a `lambda code: lambda: _StubLoader(...)` that happened to satisfy
both the wrong call site and the wrong return-type expectation. A
useful reminder that mock-shaped tests can hide real interface bugs.
Fix
---
- Import the same `_detect_market` runner.py uses for source routing
and feed its output to `resolve_loader`. The private import is a
reasonable pragmatic choice — there is no public equivalent today,
and replicating the regex table in two places would invite drift.
- Drop the `loader_cls()` instantiation step; `resolve_loader` already
returns a ready loader instance.
Test
----
Updated `test_fetch_returns_normalized_bars` to assert the call is
routed through `"us_equity"` (captured via a stub `resolve_loader`).
This pins the contract: a regression that passes the raw code as the
market key, or rewrites the helper so dispatch goes elsewhere, will
fire here instead of waiting for someone to actually run a swarm.
Verified with the local pipeline:
NVDA.US latest close = $215.20 on 2026-05-08 (7 bars, yfinance)
i.e. the same number this PR was supposed to make available to the
worker prompt, instead of the empty block the original commit
silently produced.
`pytest agent/tests/test_swarm_grounding.py -v` -> 12/12 pass.
Full sweep at the same baseline (877 passed, 3 pre-existing Windows-
specific failures untouched, separately addressed in #88).
shadowinlife