Add a static Cloudflare Pages wiki under wiki/, including the landing page, docs app, redirects/headers, maintainer README, and wiki validation workflow.
The previous CJK tokenizer ranges (#87, #95) only matched ``一-鿿``
and ``㐀-䶿``, so memory entries with Thai, Arabic, Hebrew, or
Cyrillic titles:
- Tokenized to the empty set, making recall always miss (e.g.
``find_relevant("ถัวเฉลี่ย")`` returned nothing even when the body
contained the word).
- Had their slug characters stripped to ``_``, so two distinct Thai
titles of equal length silently overwrote each other on disk.
The new ``_NON_LATIN_SCRIPT_RANGES`` constant covers CJK + Thai +
Arabic + Hebrew + Cyrillic and is reused by:
- ``_TOKEN_RE`` — single alternation pattern, one ``re.findall`` per
``_tokenize`` call (one text scan instead of two; precompiled at
module level so it doesn't go through ``re.compile`` cache lookup on
each invocation).
- ``_SLUG_DISALLOWED_RE`` — negation pattern used by ``add()``.
Arabic and Hebrew are deliberately narrowed to the basic letter blocks
(U+0620-U+064A, U+05D0-U+05EA) to keep bidi-control codepoints like
U+061C ARABIC LETTER MARK and combining marks out of on-disk slugs,
where they would render as invisible-but-distinct filenames.
Tests cover tokenization for each script, slug preservation
(parametrized across the four new scripts), and a Thai collision-
distinction regression.
Out of scope: ``agent/src/session/search.py`` has the same CJK-only
range in its FTS sanitizer; worth a follow-up PR to consume the same
constant.
The frontmatter parser returns lists for ``[a, b]`` syntax and bools
for ``true``/``false``, but MemoryEntry annotates ``title``,
``description``, and ``memory_type`` as ``str``. The mismatch was latent
until the new memory CLI tried to render those fields through
``rich.markup.escape``, which raised ``TypeError: expected string or
bytes-like object, got 'list'`` for any description that looked like a
YAML list (e.g. ``description: [red]inject[/red]``).
Add a small ``_coerce_str`` helper and route the three frontmatter
fields through it in ``_scan_entries`` so the dataclass type contract is
truthful at the storage boundary. Other consumers (recall scoring,
``remember_tool``) also benefit transparently.
Tests:
- Unit coverage of ``_coerce_str`` for str/None/list/bool inputs
- Storage-layer regression: a markdown file with a bracketed description
parses to a string-typed description
- CLI regression: ``cmd_memory_list`` no longer crashes on entries with
a bracketed description
The previous two commits inadvertently captured whole-file output from a
local pre-tool-edit formatter, reformatting ~30 functions
(`_print_status_bar`, `_format_tool_call_args`, `_build_benchmark_table`,
`_print_result`, `_build_welcome_panel`, ...) that have nothing to do
with the memory introspection feature.
Restore those functions to upstream `main` while keeping only the
intentional changes for this PR:
- `from datetime import datetime` hoisted to top imports (so the new
CLI commands can use it without a per-call local import)
- `from rich.markup import escape as rich_escape` for safely rendering
user-supplied memory titles/descriptions
- New `memory` subparser block in `_build_parser`
- New `cmd_memory_list / show / search / forget` plus the local
`_MEMORY_TYPES` constant and `_MEMORY_TYPE_STYLES` dict
- Dispatch for `args.command == "memory"` in `main()`
- Drop the now-redundant local `from datetime import datetime` inside
`cmd_trace`
Net diff for agent/cli.py shrinks from +296/-53 to +170/-1.
Hoisting PersistentMemory to top-level cli.py made every CLI invocation
import `src.agent.frontmatter`, which transitively triggers
`src/agent/__init__.py` and eagerly loads AgentLoop, SkillsLoader (74
skills), and ToolRegistry. `vibe-trading --version` measured 7.6s
because of this chain.
- Move `from src.memory.persistent import PersistentMemory` back into
each `cmd_memory_*` so only memory commands pay the import cost
- Mirror MEMORY_TYPES as a local `_MEMORY_TYPES` tuple inside cli.py so
argparse `choices` and `_MEMORY_TYPE_STYLES` no longer require the
module-level import. The module-load assertion still guards against
drift between the local tuple and the style dict
`vibe-trading --version` drops from 7.6s to 0.66s (11.5x). Memory
commands keep the same latency since they need the module anyway.
Add `vibe-trading memory list/show/search/forget` so users can inspect
and manage persistent memory without opening the file store directly.
Two recent memory bugs (#87 underscore tokenization, #95 CJK slug
collisions) were hard to spot without observability into what is
actually saved and recalled.
Also expand the PersistentMemory public surface to keep CLI lookup
logic out of storage internals:
- MEMORY_TYPES tuple as the single source of truth for type names,
shared by argparse choices and the CLI style table
- list_entries() / find() / remove_entry() back the new commands;
find() resolves by title first, then by on-disk filename stem so
users can paste either form from the index
- remove_entry(entry) skips the re-scan that remove(title) does
Existing remove(title) is unchanged; remember_tool stays green.
Address PR review feedback (HKUDS#84):
1. Read report.md back into WorkerResult.summary:
- Add `_resolve_summary()` that checks for report.md in artifact dir
- Applied at all 5 exit points (timeout, token_limit, llm_failure,
normal_completion, iteration_limit)
- Ensures API/UI shows the full report, not just the brief closing text
2. Sanitize event logging:
- Truncate tool arguments to 200 chars per value
- Rename "result" to "result_preview" and reduce from 5000 to 500 chars
- Avoids persisting/streaming large or sensitive tool payloads
3. Add regression tests for report → summary handoff:
- Test report.md content is returned when present
- Test fallback behavior when report.md missing or empty
- Test error handling for unreadable paths
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Add `agent_root / ".swarm" / "runs"` to `_default_run_roots()` in
path_utils.py. Without this, all `write_file` / `edit_file` / `read_file`
calls from swarm workers fail with "run_dir is outside allowed run roots".
- Increase `timeout_seconds` from 600 to 1800 for all 29 swarm presets.
The aggregator agent in commodity_research_team timed out at 615s, and
complex analysis tasks routinely exceed 600s.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Swarm workers frequently produced no usable output because:
- The Execution Rules prompt asked agents to write summaries "directly in
your response," but the worker only captured `last_assistant_content`
(often a mid-process message like "let me fix the script").
- On timeout/iteration limit, all in-memory messages were discarded.
- Event logging recorded only tool names and elapsed time, not I/O.
Changes to worker.py:
- Force agents to write final reports via `write_file report.md` (Phase 3),
decoupling report persistence from the in-memory message lifecycle.
- Add `_best_summary()` to extract the longest assistant message (>100
chars) instead of relying on the last fragment.
- Add `_persist_messages()` to save full conversation to messages.json on
timeout/iteration-limit for post-mortem debugging.
- Include tool arguments in tool_call events and result content (up to 5000
chars) in tool_result events for better observability.
Changes to equity_research_team.yaml:
- Increase timeout_seconds from 600 to 900 for all four agents, as the
aggregator previously timed out at 615s.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Empirical end-to-end verification on a real run revealed the dispatch
in fetch_grounding_data was wrong:
loader_cls = resolve_loader(code) # resolve_loader takes a market key
loader = loader_cls() # resolve_loader already returns
# a ready instance, not a class
So a real call with code="NVDA.US" produced
"No available data source for market 'NVDA.US'. Tried: []." — the
fallback chain table is keyed by market type ("us_equity",
"hk_equity", "crypto", ...), not by raw symbol shape.
The unit tests passed only because they monkey-patched resolve_loader
to a `lambda code: lambda: _StubLoader(...)` that happened to satisfy
both the wrong call site and the wrong return-type expectation. A
useful reminder that mock-shaped tests can hide real interface bugs.
Fix
---
- Import the same `_detect_market` runner.py uses for source routing
and feed its output to `resolve_loader`. The private import is a
reasonable pragmatic choice — there is no public equivalent today,
and replicating the regex table in two places would invite drift.
- Drop the `loader_cls()` instantiation step; `resolve_loader` already
returns a ready loader instance.
Test
----
Updated `test_fetch_returns_normalized_bars` to assert the call is
routed through `"us_equity"` (captured via a stub `resolve_loader`).
This pins the contract: a regression that passes the raw code as the
market key, or rewrites the helper so dispatch goes elsewhere, will
fire here instead of waiting for someone to actually run a swarm.
Verified with the local pipeline:
NVDA.US latest close = $215.20 on 2026-05-08 (7 bars, yfinance)
i.e. the same number this PR was supposed to make available to the
worker prompt, instead of the empty block the original commit
silently produced.
`pytest agent/tests/test_swarm_grounding.py -v` -> 12/12 pass.
Full sweep at the same baseline (877 passed, 3 pre-existing Windows-
specific failures untouched, separately addressed in #88).
Symptom
-------
Workers in fundamental_research_team and similar presets routinely quote
prices from their model's training cutoff. A run targeting NVDA.US in
2026-05 produced a research report anchored on $145.89 while the actual
yfinance close that day was $215.20. Subsequent recommendations
("Hold/Accumulate, target $148-160") were therefore wrong by 30%+ before
any analyst even saw them. issue #42-class "weird outputs" land here too
when a worker silently substitutes a remembered number for a missing
data call.
Why prompt rules alone don't fix it
-----------------------------------
worker.py:158-171 already prints generic "use tools when needed"
guidance. LLMs nevertheless prefer the cheap path of recalling a number
over the expensive path of issuing a tool call — especially under the
20-tool-call hard limit imposed two paragraphs later. Adding a stronger
"never quote prices" rule helps but does not eliminate the failure mode.
Fix
---
Make the data unavoidable. SwarmRuntime.start_run now:
1. Scans every value in user_vars for suffixed symbols
(NVDA.US / 700.HK / 600519.SH / BTC-USDT / 000001.SZ / 430090.BJ).
2. Pre-fetches the last 30 days of OHLCV via the existing loader
registry (resolve_loader handles per-market routing). Failures
are non-fatal — the run still proceeds, just without grounding.
3. Pins the result on a new SwarmRun.grounding_data field and
persists it via the existing run.json path.
SwarmRuntime._execute_run renders the data into a markdown "Ground
Truth" block once per run and threads it through
_run_worker_with_retries -> run_worker -> build_worker_prompt, which
splices it ahead of Execution Rules. The block contains the recent
window range, last close, the last 5 daily closes, and an explicit
instruction to prefer these prices over training-data prices and to
call get_market_data for any range outside the window.
Backward compatibility
----------------------
* SwarmRun.grounding_data defaults to None; existing run.json files
parse unchanged (Pydantic deserialises the missing key to the
default). The legacy regression case is covered by
test_swarm_run_metadata.py once that lands; this PR's tests cover the
new module and prompt plumbing in isolation.
* build_worker_prompt and run_worker each gain a grounding_block= kw
with default "" so callers outside the swarm runtime keep working.
* Empty grounding (no symbols detected, or every fetch failed) skips
the entire section — no empty heading rendered.
What's not in scope
-------------------
* Bare tickers without a suffix (just "NVDA") are intentionally not
matched. The false-positive cost on common English words is too high
and every shipped preset already supplies suffixed symbols.
* The block is a one-shot snapshot at run start. Long-running swarms
may see stale data after many minutes — still strictly better than
training-data prices from a year ago. A refresh hook can land later.
Tests
-----
agent/tests/test_swarm_grounding.py covers symbol extraction (all four
loader-supported suffix shapes, ordering, non-string values, word
boundaries), fetch behaviour with a stubbed loader (normalised bars,
empty-frame skip, empty-input no-op), markdown rendering (range / last
close / explicit instruction text), and end-to-end prompt plumbing
(grounding block lands before Execution Rules; absent block omits the
section). All 12 pass; full pytest sweep against main shows the same
pre-existing 3 failures (test_backtest_runner_security x2,
test_loop_helpers normalize) as main and 875 passed (= baseline 863 +
the new 12 grounding tests).
run.json currently has no field describing which LLM provider/model the
swarm was actually launched against — SwarmAgentSpec.model_name carries
per-agent overrides but defaults to None ("use the global config"), and
that resolved config is never captured anywhere. Cost auditing,
side-by-side provider experiments, and post-hoc bug triage all need this
information; today it has to be reconstructed from .env timestamps.
This adds two optional fields to SwarmRun:
- provider: lowercase form of LANGCHAIN_PROVIDER, the same env var
src/providers/llm.py:136,195 reads when constructing ChatLLM.
- model: LANGCHAIN_MODEL_NAME (raw, since model strings are
case-sensitive — "gpt-4o" vs "openai/gpt-5").
Both default to None and are populated in SwarmRuntime.start_run before
the run is persisted (runtime.py:start_run), so every new run.json from
this point forward records the active provider/model. Per-agent
overrides remain visible on SwarmAgentSpec.model_name; this field is the
run-level default.
Backward compat: existing run.json files lack the keys entirely. Pydantic
deserializes missing optional fields to their default (None), so
SwarmStore.get_run / list_runs continue to work on legacy files. A
parametrized regression test covers this in
agent/tests/test_swarm_run_metadata.py along with construction,
serialization round-trip, and acceptance of multiple provider names.
backtest.runner.main() previously accepted any Path it was handed and
went straight to reading config.json + importing code/signal_engine.py
from that location. The MCP backtest tool already routes through
safe_run_dir (src/tools/backtest_tool.py:23), but
"python -m backtest.runner <path>" did not — meaning a contributor or
agent that can drop config.json + code/signal_engine.py anywhere on
disk could have it loaded and executed by the CLI path.
The AST scrubber at runner.py:217-239 blocks executable top-level
statements, but any function or method body still runs on instantiation
or method call (BaseEngine.run_backtest -> SignalEngine().__init__ ->
.generate(...)), so the AST guard is not a substitute for a path
boundary.
Reuse the existing safe_run_dir() (src/tools/path_utils.py:190) at the
top of main() so the CLI honours the same VIBE_TRADING_ALLOWED_RUN_ROOTS
whitelist as the MCP path. Errors are surfaced through the existing
{"error": "..."} JSON-on-stdout + sys.exit(1) convention this entry
point already uses.
Existing test_path_safety.py exercises safe_run_dir directly and still
passes; behaviour for run dirs under the default roots
(agent/runs, $cwd/runs, ~/.vibe-trading/shadow_runs) and any path listed
in VIBE_TRADING_ALLOWED_RUN_ROOTS is unchanged.
Three bare except: clauses in api_server.py silently caught BaseException
(KeyboardInterrupt, SystemExit, GeneratorExit) in addition to ordinary
errors. That makes the API server unresponsive to Ctrl+C inside these
blocks and hides real bugs (NameError, AttributeError) as silent passes.
The blocks themselves are best-effort metadata extraction from per-run
artifact files, where falling through with the default values is the
right behavior — but only for the specific I/O / parse failures these
blocks can actually produce. The new clauses spell those out:
- lines 1015, 1022 (req.json / planner_output.json prompt extraction):
except (json.JSONDecodeError, OSError)
- line 1042 (metrics.csv numeric parse):
except (OSError, ValueError)
Behavior is unchanged for the failures the blocks are designed to swallow;
KeyboardInterrupt, SystemExit, and programming errors now propagate as
they should.
Three tests fail when the suite is run on Windows even though the
underlying behaviour is correct on both platforms — the failures are
test-side platform assumptions, not application bugs. Linux CI is
unaffected; this is purely a quality-of-life fix for contributors who
run pytest on Windows.
1. test_backtest_runner_security.py — rce path interpolation
----------------------------------------------------------
Both ``test_signal_engine_rejects_top_level_execution`` and
``test_signal_engine_rejects_class_level_execution`` write a
``signal_engine.py`` whose source contains the pytest tmp_path
formatted into an ``os.system('touch ...')`` call. On Windows,
tmp_path looks like ``C:\Users\teera\AppData\Local\Temp\...``;
interpolated into a single-quoted Python string literal, the leading
``\U`` is interpreted as a (truncated) unicode escape and ``ast.parse``
raises ``SyntaxError`` before the security scrubber under test ever
runs. Switching to ``Path.as_posix()`` produces forward-slash paths
that survive Python source escaping intact while pointing at the same
filesystem location on every OS.
2. test_loop_helpers.py — absolute-path preservation
--------------------------------------------------
``test_preserves_absolute_run_dir`` passed the literal
``/var/tmp/custom_run`` and expected it back unchanged.
``_normalize_tool_run_dir`` calls ``Path.is_absolute()``, which on
Windows returns False for a bare drive-less path — so the function
correctly treats the input as relative and joins it with
``memory_run_dir``. The test was wrong, not the function: it asserted
POSIX semantics on a cross-platform helper. Resolving the input
through ``os.path.abspath`` before handing it to the function
produces the platform's actual absolute form (``/var/tmp/custom_run``
on POSIX, ``C:\var\tmp\custom_run`` on Windows) and the assertion now
holds on both.
Verified locally: the targeted three now pass; the full pytest sweep
goes from "3 failed, 863 passed" on main to "866 passed, 0 failed".
No production code is touched.
Memory titles saved through RememberTool are slugified to snake_case
(e.g. "mcp_wiring_test"), so the previous regex `[a-zA-Z0-9_]{3,}`
collapsed them into a single token. A natural-language recall query
like "mcp wiring" produces tokens {"mcp", "wiring"} and never
intersects {"mcp_wiring_test"} — recall returned 0 results despite
the memory existing on disk.
Drop `_` from the character class so underscores act as token
boundaries. Verbatim queries still work because the same string
splits into the same tokens on both sides of the comparison.
Adds a regression test for the snake_case case.
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
Spawns vibe-trading-mcp via subprocess and walks through the JSON-RPC
happy path: initialize + tools/list + tools/call analyze_options. The
critical assertion is that tools/call does not hang — without the
registry pre-warm in mcp_server.main() (PR #85), the lazy registry build
runs inside FastMCP's worker thread and deadlocks on the first call.
The test also verifies (a) the tool catalogue has at least 20 entries
including analyze_options/get_market_data/list_skills, and (b) the
analyze_options result has a positive price and a delta in [0, 1] —
covering tool-registration regressions and Black-Scholes formula breaks
in addition to the deadlock.
analyze_options is chosen over network-dependent tools (get_market_data,
web_search) so the test stays deterministic and runs offline in CI.
Verified locally with differential validation:
- with pre-warm: PASS in 3.91s
- pre-warm commented out: FAIL with 15s timeout on tools/call
- pre-warm restored: PASS in 3.81s
_get_registry() lazy-imports src.tools.build_registry on first call. Every
caller is a FastMCP tool handler, so the first tools/call triggers the lazy
init inside FastMCP's asyncio worker thread, where the import chain (notably
src.tools.shell.*) deadlocks. initialize and tools/list succeed because
they don't touch the registry.
Pre-warming _get_registry() in main() — after _include_shell_tools is set
and before mcp.run() — performs the import on the main thread and lets
worker threads only ever read the cached singleton. _get_registry() is
idempotent (if _registry is None guard), so the 14 existing call sites are
unaffected.
Verified deterministically: removing the line reintroduces the hang within
15s; adding it back fixes tools/call within 0.01s. End-to-end through
Claude Code's MCP client returns in <2s.
vite.config.ts hardcoded http://localhost:8899 across 14 proxy entries,
ignoring the VITE_API_URL env that docker-compose.yml passes to the
frontend service (http://vibe-trading:8899). Inside the frontend
container, localhost resolves to the frontend's own network namespace,
so all proxied requests (sessions, runs, skills, etc.) ECONNREFUSED.
Read VITE_API_URL via loadEnv with the original localhost:8899 default,
and DRY the per-route target into a single PROXY_PATHS list driven by
Object.fromEntries.
Closes#78
Endpoints under /runs/{run_id}, /swarm/runs/{run_id} and /sessions/{session_id}
constructed filesystem paths as `RUNS_DIR / run_id / ...` without validating
the path-parameter value. An authenticated caller could submit values like
`..` or `foo/../bar` and reach paths outside the intended directory.
The same file already validates `{shadow_id}` with `_SHADOW_ID_RE`. This
change applies the same hardening to `run_id` and `session_id` via a shared
`_validate_path_param()` helper that enforces a safe character class
(`^[A-Za-z0-9_-]{1,128}$`). The class accepts every ID format produced by the
codebase today (regular run_id from `state.create_run_dir`, swarm
`swarm-{ts}-{short_uuid}`, session_id from `uuid.uuid4().hex[:12]`) and
rejects path separators, dots, NUL, whitespace, and oversize input.
13 endpoints are guarded:
- GET /runs/{run_id}/code, /pine, /
- GET|DELETE|PATCH /sessions/{session_id}
- POST /sessions/{session_id}/messages, /cancel
- GET /sessions/{session_id}/messages, /events
- GET /swarm/runs/{run_id}, /events
- POST /swarm/runs/{run_id}/cancel
Tests added in `test_security_auth_api.py`:
- parametrized validator unit tests (5 valid IDs / 9 invalid inputs)
- per-endpoint integration tests using FastAPI TestClient
Haozhe Wu