* feat(bench): add horizon parameter for monthly-frequency IC revalidation
h-bar forward returns (close[t+h]/close[t]-1, no ffill — a NaN close never
bridges) sampled non-overlapping end-anchored, so the naive-iid t-stat and
the alive/dead thresholds stay honest without new statistics. Threaded
through run_bench / run_bench_strict / the alpha_bench tool + MCP wrapper /
CLI --horizon (argparse type + handler guard, exit 2 on bad input).
horizon=1 is byte-identical to the previous next-bar formula.
Follow-up #1 of issue #1789 (monthly-horizon revalidation of the 13
fundamental factors — the conventional reading for fundamental factors).
* fix(bench): exclude SEC-fundamental alphas off sp500 instead of failing the bench
--zoo fundamental --universe csi300 (the repro command printed in issue
#1789) hard-failed with 'Choose the S&P 500 universe' because the
validator rejects fund:* inputs on any non-sp500 universe, while the
mirror case (CN val:* alphas on sp500) skips cleanly per alpha. The
runners now split the selection up front, disclose the excluded ids in
the result envelope, and error only when nothing is left to bench; the
validator stays as the backstop for direct callers.
The CLI/tool JSON envelopes also state their own horizon,
forward_return_sampling and universe_excluded — a saved summary must
show the configuration its numbers came from.
* fix(bench): close review findings on horizon/exclusion disclosure
- multi-zoo CLI aggregation threads horizon / forward_return_sampling /
universe_excluded from the sub-runs into the aggregate envelope (a saved
all-zoos JSON now states its own horizon and reconciles its exclusions)
- alpha compare surfaces universe-excluded shortlist members as skipped
entries with the exclusion reason instead of silently dropping them
- horizon upper bound (1.._MAX_HORIZON) enforced consistently at the CLI
parser, the handler, and both runner entries BEFORE any panel load —
previously only the tool/MCP capped it and an unbounded CLI run fetched
the full panel before failing late
- non-int horizon can no longer TypeError in the entry schema construction
- categorise docstring notes the daily calibration of the |IC|>0.02 floor
at h>1; the sampling comment no longer overclaims (residual
autocorrelation from persistent factor exposures is not modeled)
- MCP wrapper uses _MAX_HORIZON instead of a hardcoded duplicate
Pinned by new tests: parser/handler/runner cap rejection (fetch not
called), multi-zoo disclosure aggregation, compare exclusion disclosure.
* feat(factors): industry-neutralized CN value alphas + PIT SW2021 sector tag
Three within-industry z-scored variants of the CN value factors
(fund_pb/fund_ps_ttm/fund_dividend_yield -> *_ind_neutral, zoo 475 -> 478):
raw A-share PB/PS/dy are heavily industry-confounded, so each name is
z-scored against its Shenwan L1 peers instead of the whole cross-section.
- industry_loader.py: tushare index_member_all (SW2021) membership with
in_date/out_date -> point-in-time labels per trading day; deterministic
overlap resolution; 2000-row truncation guard refetches per L1; degrades
to an unmapped frame on provider failure, never invents a label.
- csi300 bench attaches panel['sector'] outside the pickled price cache,
masked to the PIT membership cross-section, behind the same 0.9 coverage
gate the sp500 GICS tag uses; meta discloses sector_source/coverage.
- base.zscore_by_group: the z-scoring sibling of alpha101's _ind_neutralize
demean - no global fallback (an unlabelled cell stays NaN rather than
wearing an industry-neutral label it did not earn).
- README x7 + SKILL.md counts (incl. the stale 462 prose sites), i18n zh/en.
Follow-up #2 of issue #1789.
* fix(loaders): index_member_all takes no src param; fetch PIT history via is_new Y+N
Live-verified endpoint contract (pinned by test): index_member_all accepts
only the documented l1/l2/l3_code, ts_code and is_new filters — an
undocumented src param silently returns an empty frame (HTTP 200, 0 rows),
which degraded the first csi300 sector attach to coverage 0.0. src now goes
to index_classify (truncation fallback) only.
The server also defaults is_new to 'Y' (current members, no out_date rows),
so the historical half ('N', ~2000 rows with out_date back to the 1990s) is
fetched separately and concatenated — without it every label would be
today's classification and the loader's PIT claim false.
Live smoke: 350/350 csi300 PIT constituents covered (100%), 0.9 gate passes.
* docs(factors): surface the SW2021 taxonomy-retroactivity caveat; pin cache immutability
Review follow-ups: the loader docstring disclosed that SW2021
classifications are applied retroactively, but no user-facing surface
did — one sentence added to each ind-neutral alpha's notes, the i18n
en/zh entries and SKILL.md (membership is point-in-time; the taxonomy
version itself is not). The sector-attach test now pins that the
original pickled panel dict is never mutated by the attach, and the
sector_coverage meta semantics (labelled-somewhere fraction, not
per-date) are documented at the attach site.
* fix(factors): align ind-neutral PB/PS with the upstream nonpositive-multiple guard
The maintenance batch (d3c76819) added a guard to fund_pb/fund_ps_ttm:
nonpositive multiples are excluded rather than ranked as cheap. The
industry-neutral variants follow suit — the within-group z-score sees
the same values.where(values > 0) pre-filter, the guard sentence leads
the notes/i18n entries, and the upstream parametrized test is mirrored
for the neutral stems.
Hand-written _discord_hints() replaces the derived form: 12 localized
fields (enabled excluded) x 9 locales with labels and help, a required
mark on the bot token (start() refuses to connect without it), and
group_policy as a <select> via the FieldHint.choices contract,
drift-pinned to the adapter's Literal["mention", "open"] by two
contract tests. token/proxy_username/proxy_password stay secrets with no
audited subtraction. A guided-setup entry walks through the Developer
Portal flow including the privileged MESSAGE CONTENT INTENT switch that
the default intents (37377) require - the most common Discord setup
failure, which no REST probe can detect (it surfaces only at gateway
IDENTIFY, close code 4014).
Signed-off-by: shadowinlife <shadowinlife@gmail.com>
New discord_probe.py validates a (possibly unsaved) bot token with a
single GET to Discord's users/@me endpoint carrying the Authorization:
Bot header. token_key="bot" also catches the wrong-token-type failure
(a pasted OAuth user token answers 200 without the bot flag) at probe
time instead of at gateway connect. The shared probe_token_endpoint
gains an optional method parameter (GET sends no JSON body); existing
POST callers are untouched. The enable transition is gated by the probe
(failure blocks the persist with 422), and discord leaves the
default-unsupported candidate list in the dingtalk connection test.
Signed-off-by: shadowinlife <shadowinlife@gmail.com>
allow_channels and group_policy are read live from self.config on every
inbound message (the client-side admission helper and
_should_accept_inbound; _should_respond_in_group) and are never captured
into the discord.py client at start(), so a Web UI edit that only changes
them now applies through refresh_config without a gateway disconnect.
Transport-identity keys (token, intents, proxy, streaming) stay on the
hot-swap path. Mirrors the noop-key classification the other adapters
adopted in the #1625/#1737 lineage.
Signed-off-by: shadowinlife <shadowinlife@gmail.com>
CI installs no channel extras, so the module-level `from src.channels.slack
import SlackChannel` aborted collection with ModuleNotFoundError: slack_sdk
on both Python 3.11 and 3.14 jobs. Install the shared stand-ins from
tests/slack_stubs.py first, matching test_slack_dm_pairing.py and the other
Slack tests (see the #1524 import-order note in slack_stubs.py).
Verified in a slack_sdk-less env: the 10 connection-test cases pass, the
channel-related subset (383 tests) passes, and the full gate
`pytest --ignore=agent/tests/e2e_backtest --ignore=agent/tests/test_e2e_harness_v2.py`
is green (17255 passed, 89 skipped).
Signed-off-by: shadowinlife <shadowinlife@gmail.com>
Derived hints left Slack with raw key labels, unmarked required
credentials, free-text group_policy, and user_token_read_only masked as
a secret by the fail-safe regex; add hand-written hints with the audited
secret=False subtraction (websocket token_ttl_s precedent), extend
FieldHint with optional choices rendered as a select, localize 12 fields
in 9 locales; mode/webhook_path stay file-configured (socket-only
transport).
Signed-off-by: shadowinlife <shadowinlife@gmail.com>
Slack was the only Web-UI channel without a credential probe:
POST /channels/slack/test returned "unsupported" and an enable
transition saved unverified tokens, so a wrong xoxb-/xapp- token
surfaced only when the channel failed to start.
Add a standalone two-leg probe per the token_probe add-a-channel
checklist: leg 1 validates the bot token against auth.test, leg 2
validates the app token and that Socket Mode is enabled via
apps.connections.open (the wss:// URL in its success body is discarded,
never returned, logged, or cached). The shared probe gains an optional
headers parameter because Slack authenticates with an Authorization:
Bearer header instead of a JSON body; every existing caller passes
nothing and is unaffected. Slack's 200-with-ok:false error shape rides
the existing "200 but no token" classification, so no new result codes
appear. Missing tokens short-circuit before any HTTP call, mirroring
the start() guard.
Precedent: Feishu connection test (#1572).
Signed-off-by: shadowinlife <shadowinlife@gmail.com>
A failed per-channel hot swap silently degrades to a full runtime reset
today — every connection drops with only a server log to show why. The
PUT response now carries a sanitized reset_reason (exception type +
message, bounded to 200 chars with URL userinfo stripped), and the
changed channel's last chat gets a best-effort notice published on the
fresh runtime's bus after the restart, mirroring nanobot's /restart
completion notice back to the requesting chat.
Refs #1625.
Signed-off-by: shadowinlife <shadowinlife@gmail.com>
Start, send and reload-stop failures are log-only today, so a channel that
built fine but crashed on start shows running:false in the UI with no reason
anywhere. _record_error merges a scrubbed one-line summary (URL userinfo
stripped, capped at 200 chars), a frozen error context and a UTC timestamp
into the per-channel status that /channels/status and /channels/config
already expose; a clean rebuild clears all three fields. Precedent: AstrBot
PlatformStatus + record_error surfaced in its WebUI. Refs #1625.
Signed-off-by: shadowinlife <shadowinlife@gmail.com>
Cosmetic and ACL keys — allow_from, emoji reactions, and the progress toggles — are consumed by live per-message reads rather than captured into the SDK client at start(), so editing one today needlessly drops the IM connection. Each adapter now declares hot_reload_noop_keys; a PUT that changes only declared keys (or the manager bool overrides) refreshes the config in place via ChannelManager.refresh_channel_config and reports applied: "refreshed". Undeclared keys, the enabled transition, and any refresh failure keep the full hot-swap path as a fail-safe. Precedent: OpenClaw plugin-declared reload.noopPrefixes. Refs #1625.
Signed-off-by: shadowinlife <shadowinlife@gmail.com>
Goal: give MCP clients token-cheap structured access to backtest run
artifacts. read_file returns raw CSV text and truncates mid-file; this
tool parses the artifact once and serves typed JSON pages a client can
page through or chart directly. Implements #1645.
Design: the run_card.json artifacts manifest (path/sha256/size_bytes per
file) is the trust anchor. Friendly aliases (equity, trades, metrics,
positions, target_positions, ohlcv:<CODE>, run_card) resolve by a fixed
mapping and work without a card; any other value is treated as a
run_dir-relative path accepted only when the manifest lists it, so a new
artifact kind becomes readable with zero code change. Manifest entries
are themselves validated — no absolute paths, no drive prefixes, no '..'
segments under either separator style — and every resolution is
re-checked with resolve() + is_relative_to(run_root), so a tampered card
can never point the reader outside the run directory. Three formats:
rows (lossless offset paging with honest truncated/next_offset),
downsample (equal-stride, first+last pinned), meta (columns/total_rows/
size_bytes, plus the manifest sha256 and manifest_size_bytes when
listed). Envelopes stay within a 120K-character budget by shrinking
whole records following _result_paging's proportional-shrink pattern —
never mid-record, never broken JSON.
Contract: purely additive read-only surface (75th MCP tool, 108th
registry tool); errors return {"ok": false, "error", "hint"} and success
envelopes carry no ok/status field.
Tests: 44 cases — manifest acceptance and refusal (unlisted relative
paths, '..' and absolute tampered entries, symlinked manifest entries,
non-served suffixes), aliases with absent/corrupt cards, meta sha256
exposure, 3821-row downsample endpoint pinning, lossless paging walk,
traversal refusals, coercion, projection, budget shrink/re-stride/
floor, clamping, exact envelope key sets. README/SKILL count pins
updated (74->75 MCP, 107->108 registry); test_readme_counts,
test_distribution_skill_manifest and the MCP smoke suite are green.
Signed-off-by: shadowinlife <shadowinlife@gmail.com>
The backtest envelope carried metrics only as a JSON string inside
stdout, which is truncated to its last 2000 characters — with the
metrics JSON already ~1.4KB and growing (rebalance_*, risk_xray_*),
the authoritative numbers could be cut mid-structure, and consumers
had to re-parse stdout and guess run_id from the run_dir basename.
On success the envelope now gains a best-effort summary projected
from artifacts the run already writes (run_card.json + equity.csv):
schema_version, run_id, title, codes, date range, interval,
initial_cash, the full metrics object (non-finite floats sanitized
to null), an equal-stride equity_preview of at most 50 points with
the first and last row pinned, artifact_paths including per-symbol
ohlcv files, and warnings. The artifacts map also exposes the ohlcv
files the engine already writes.
Strictly additive: legacy fields stay byte-identical (golden-tested),
failure/timeout paths are unchanged, and a summary that cannot be
built is omitted rather than failing the run.
Signed-off-by: shadowinlife <shadowinlife@gmail.com>
Mirrors the DingTalk/QQ sentence in the "Configure from the Web UI"
paragraph across README.md and the six translations, in lockstep.
Part of #1569.
Signed-off-by: shadowinlife <shadowinlife@gmail.com>
Extends the #1520 channel-config surface to Feishu: hand-written field
hints (13 fields), a standalone tenant-access-token probe that is
domain-aware across open.feishu.cn / open.larksuite.com, a guided setup
guide localized in all nine locales, and enable-transition credential
verification through the existing frozen ok|invalid_credentials|network
contract.
stop() is now hot-swap safe. lark.ws.Client exposes no stop method and
its receive loop runs with auto_reconnect=True, so merely clearing
_running left a zombie WebSocket after reload_channel: the old and the
replacement adapter both answered events. stop() now best-effort closes
the SDK connection (bounded) and terminates the dedicated event loop,
killing the receive/ping/reconnect tasks; it stays idempotent and safe
when never started. Pinned by test_feishu_stop.py, the DingTalk
zombie-stream fix class from #1520.
The probe follows the token_probe checklist verbatim: feishu_probe.py
module, registry._INTERNAL entry, supports_connection_test plus a thin
delegate, FIELD_HINTS, frontend GUIDE_DEFS and fields/guides i18n keys
in all nine locales, and tests mirroring test_qq_connection_test.py.
Feishu answers HTTP 200 with an error body for bad credentials; the
shared 200-but-no-token branch classifies that as invalid_credentials
(the QQ precedent).
Part of #1569.
Signed-off-by: shadowinlife <shadowinlife@gmail.com>
Two defense-in-depth fixes from an adversarial review of the Email/WebSocket
guided-config surface (#1544). Both harden pre-existing patterns that the new
email probe — the first whose target host is caller-controlled while reusing
stored credentials — made relevant.
Email TLS certificate verification (CERT_NONE -> verified):
- The implicit-SSL paths (IMAP4_SSL/SMTP_SSL) in the connection probe and the
polling/send adapter used Python's default unverified context (CERT_NONE, no
hostname check), so the mailbox password was sent to a server whose
certificate was never checked; an active MITM could harvest it. All four
call sites now build their context through a shared email_tls_context()
that verifies the certificate and hostname against the system CA bundle,
matching the STARTTLS path, which already did.
- New EmailConfig.verify_tls field (default True) is the documented opt-out
for self-signed/internal-CA servers, surfaced in the guided Email setup with
a localized label/help in all nine locales.
Settings-write CSRF guard:
- require_settings_write_auth was missing the cross-site browser guard its
siblings (require_auth, require_event_stream_auth) enforce on unsafe
methods. POST /channels/email/test merges stored credentials with a caller
patch, so a cross-site page could steer the stored mailbox password to an
attacker-chosen host. The guard now runs first on every settings-write
route; same-origin Web UI calls and non-browser clients (no Origin) are
unaffected.
Tests: 9 new email TLS regressions (helper + probe/adapter x verify_tls) and
5 new settings-write cross-site guard regressions; the email field-count
(29 -> 30) and i18n parity assertions were updated. Channel sweep 639 passed,
frontend 656 passed, test_readme_counts 108 passed; new/changed files are
black- and ruff-clean (email.py's 9 ruff errors are pre-existing import-block
debt, unchanged 9 -> 9).
Follow-up to #1519 (extends #1520, hardens #1544).
Signed-off-by: shadowinlife <shadowinlife@gmail.com>
The English README's "Configure from the Web UI" paragraph gained the
Email/WebSocket guided-channel sentence in the previous commit; mirror it
into the six translated READMEs (zh, ja, ko, ar, es, id) at the same slot
— after the QQ sentence, before the Test-connection sentence — keeping
IMAP/SMTP/TLS/WebSocket/Test connection verbatim. Prose-only; no count or
roster change, so test_readme_counts stays green (108 passed).
Follow-up to #1519 (extends #1520).
Signed-off-by: shadowinlife <shadowinlife@gmail.com>
Extend the contract-driven channel config surface from #1520 to the Email
and WebSocket adapters: both can now be configured, connection-tested and
hot-applied from Settings with no restart and no hand-edited JSON — the
"one hints dict entry" path #1519 designed for the long tail of adapters.
Backend
- Email: standalone IMAP + SMTP credential probe (email_probe.py) — login,
mailbox select, SMTP connect/(STARTTLS)/login — classified into the frozen
ok|invalid_credentials|network codes with secret-scrubbed, side-prefixed
details; the adapter's test_connection delegates to it.
- WebSocket: local-server probe (websocket_probe.py) — TLS cert/key material
validation plus a read-only bind of every resolved address (SO_REUSEADDR
gated to POSIX, matching asyncio's create_server default); an address held
by the already-running server reads as the expected state, not a failure.
The config routes now build the WebSocket adapter with the gateway services
it requires, so a WebSocket save or test no longer fails validation.
- config_meta: hand-written field hints for both channels. A hand-written
hint's secret flag is now authoritative for the keys it covers (the regex
fail-safe still masks every unhinted key), so token-shaped non-secrets such
as websocket_requires_token stay visible and editable instead of being
masked — a form save can no longer silently flip the stored token requirement.
- Email polling skips stale-config delete/move post-actions once a stop or hot
swap begins; the already-fetched batch is still delivered.
Frontend
- Localized labels/help and guided setup for both channels across all nine
locales; secrets are masked and never round-trip to the browser.
Adapter fix found during a live end-to-end smoke test against a real 163.com
mailbox: NetEase (163/126/yeah.net) accepts LOGIN but rejects the first SELECT
with "Unsafe Login" until the client sends an RFC 2971 IMAP ID, so the Email
channel could not read the most common Chinese mailboxes. The probe and the
polling loop now send a static, best-effort IMAP ID after login
(utils.send_imap_id) — it carries no user data or secrets and is ignored by
servers that do not support it. Verified end to end: an inbound mail from an
allow-listed sender passed SPF/DKIM, reached the agent, and an auto-reply was
delivered back over SMTP; on the same run the anti-spoofing check rejected the
mailbox's own SPF-failing service mail.
Follow-up to #1519 (extends #1520).
Signed-off-by: shadowinlife <shadowinlife@gmail.com>
Convert 23 loguru-style "{}" format strings passed to self.logger (a
stdlib logging.Logger from BaseChannel) to %-style lazy formatting including the {:.0f} -> %.0f spec,
with args present the old form raised TypeError inside handler
formatting - the message was lost and a logging-error traceback went
to stderr instead.
Verified: placeholder count == argument count at every converted site
(AST-checked), zero "{}" logger calls remain in the file, every diff
line is a pure placeholder conversion, and the full backend suite is
green (13809 passed).
Part of #1531.
Signed-off-by: shadowinlife <shadowinlife@gmail.com>
Convert 17 loguru-style "{}" format strings passed to self.logger (a
stdlib logging.Logger from BaseChannel) to %-style lazy formatting
with args present the old form raised TypeError inside handler
formatting - the message was lost and a logging-error traceback went
to stderr instead.
Verified: placeholder count == argument count at every converted site
(AST-checked), zero "{}" logger calls remain in the file, every diff
line is a pure placeholder conversion, and the full backend suite is
green (13809 passed).
Part of #1531.
Signed-off-by: shadowinlife <shadowinlife@gmail.com>
Convert 12 loguru-style "{}" format strings passed to self.logger (a
stdlib logging.Logger from BaseChannel) to %-style lazy formatting
with args present the old form raised TypeError inside handler
formatting - the message was lost and a logging-error traceback went
to stderr instead.
Verified: placeholder count == argument count at every converted site
(AST-checked), zero "{}" logger calls remain in the file, every diff
line is a pure placeholder conversion, and the full backend suite is
green (13809 passed).
Part of #1531.
Signed-off-by: shadowinlife <shadowinlife@gmail.com>
Convert 31 loguru-style "{}" format strings passed to self.logger (a
stdlib logging.Logger from BaseChannel) to %-style lazy formatting including the {:.1f} -> %.1f spec,
with args present the old form raised TypeError inside handler
formatting - the message was lost and a logging-error traceback went
to stderr instead.
Verified: placeholder count == argument count at every converted site
(AST-checked), zero "{}" logger calls remain in the file, every diff
line is a pure placeholder conversion, and the full backend suite is
green (13809 passed).
Part of #1531.
Signed-off-by: shadowinlife <shadowinlife@gmail.com>
Convert 31 loguru-style "{}" format strings passed to self.logger (a
stdlib logging.Logger from BaseChannel) to %-style lazy formatting including the {:.0f} -> %.0f spec,
with args present the old form raised TypeError inside handler
formatting - the message was lost and a logging-error traceback went
to stderr instead.
Verified: placeholder count == argument count at every converted site
(AST-checked), zero "{}" logger calls remain in the file, every diff
line is a pure placeholder conversion, and the full backend suite is
green (13809 passed).
Part of #1531.
Signed-off-by: shadowinlife <shadowinlife@gmail.com>
Convert 13 loguru-style "{}" format strings passed to self.logger (a
stdlib logging.Logger from BaseChannel) to %-style lazy formatting including the {!r} -> %r spec,
with args present the old form raised TypeError inside handler
formatting - the message was lost and a logging-error traceback went
to stderr instead.
Verified: placeholder count == argument count at every converted site
(AST-checked), zero "{}" logger calls remain in the file, every diff
line is a pure placeholder conversion, and the full backend suite is
green (13809 passed).
Part of #1531.
Signed-off-by: shadowinlife <shadowinlife@gmail.com>
Convert 10 loguru-style "{}" format strings passed to self.logger (a
stdlib logging.Logger from BaseChannel) to %-style lazy formatting
with args present the old form raised TypeError inside handler
formatting - the message was lost and a logging-error traceback went
to stderr instead.
Verified: placeholder count == argument count at every converted site
(AST-checked), zero "{}" logger calls remain in the file, every diff
line is a pure placeholder conversion, and the full backend suite is
green (13809 passed).
Part of #1531.
Signed-off-by: shadowinlife <shadowinlife@gmail.com>
Convert 16 loguru-style "{}" format strings passed to self.logger (a
stdlib logging.Logger from BaseChannel) to %-style lazy formatting
with args present the old form raised TypeError inside handler
formatting - the message was lost and a logging-error traceback went
to stderr instead.
Verified: placeholder count == argument count at every converted site
(AST-checked), zero "{}" logger calls remain in the file, every diff
line is a pure placeholder conversion, and the full backend suite is
green (13809 passed).
Part of #1531.
Signed-off-by: shadowinlife <shadowinlife@gmail.com>
Convert 44 loguru-style "{}" format strings passed to self.logger (a
stdlib logging.Logger from BaseChannel) to %-style lazy formatting
with args present the old form raised TypeError inside handler
formatting - the message was lost and a logging-error traceback went
to stderr instead.
Verified: placeholder count == argument count at every converted site
(AST-checked), zero "{}" logger calls remain in the file, every diff
line is a pure placeholder conversion, and the full backend suite is
green (13809 passed).
Part of #1531.
Signed-off-by: shadowinlife <shadowinlife@gmail.com>
Convert 25 loguru-style "{}" format strings passed to self.logger (a
stdlib logging.Logger from BaseChannel) to %-style lazy formatting
with args present the old form raised TypeError inside handler
formatting - the message was lost and a logging-error traceback went
to stderr instead.
Verified: placeholder count == argument count at every converted site
(AST-checked), zero "{}" logger calls remain in the file, every diff
line is a pure placeholder conversion, and the full backend suite is
green (13809 passed).
Part of #1531.
Signed-off-by: shadowinlife <shadowinlife@gmail.com>
Convert 25 loguru-style "{}" format strings passed to self.logger (a
stdlib logging.Logger from BaseChannel) to %-style lazy formatting
with args present the old form raised TypeError inside handler
formatting - the message was lost and a logging-error traceback went
to stderr instead.
Verified: placeholder count == argument count at every converted site
(AST-checked), zero "{}" logger calls remain in the file, every diff
line is a pure placeholder conversion, and the full backend suite is
green (13809 passed).
Part of #1531.
Signed-off-by: shadowinlife <shadowinlife@gmail.com>
Convert 40 loguru-style "{}" format strings passed to self.logger (a
stdlib logging.Logger from BaseChannel) to %-style lazy formatting
with args present the old form raised TypeError inside handler
formatting - the message was lost and a logging-error traceback went
to stderr instead.
Verified: placeholder count == argument count at every converted site
(AST-checked), zero "{}" logger calls remain in the file, every diff
line is a pure placeholder conversion, and the full backend suite is
green (13809 passed).
Part of #1531.
Signed-off-by: shadowinlife <shadowinlife@gmail.com>
qqEntry held download_chunk_size/download_max_bytes as strings, but
since the GET defaults backfill a pristine channel returns the
adapter's int defaults (262144/209715200) on the wire. The panel's
String(raw) seeding masked the drift; the fixture now matches the real
GET shape, so the suite keeps proving that int defaults seed the text
widgets and round-trip through buildPatch.
Part of #1519.
Signed-off-by: shadowinlife <shadowinlife@gmail.com>
The probe extraction kept two helpers alive that nothing consumes: the
response_detail re-export and the scrub_detail config wrapper (both
superseded by token_probe.response_detail / token_probe.scrub_secrets,
which the shared probe applies internally). An adversarial review of
the QQ increment grepped src/ and tests/ and found zero callers; the
DingTalk contract suite - the extraction's regression pin - never
imported either symbol and stays green (22 passed) after the removal.
Keeping them would advertise a per-channel scrubbing API that the
shared seam deliberately owns in one place: the next probe author
should not have to decide between two scrubbing entry points.
Part of #1519.
Signed-off-by: shadowinlife <shadowinlife@gmail.com>
The same pre-existing bug class the QQ increment fixed in qq.py: both
adapters format self.logger calls (a stdlib logger from BaseChannel)
with loguru-style "{}" placeholders. With args present, stdlib's
msg % args raises TypeError inside handler formatting - the message is
lost and a logging-error traceback prints instead. Runtime-proven on
the converted whatsapp deprecated-bridge warning: it now renders
("Ignoring deprecated WhatsApp bridge config fields: bridge_url.
...") where the same call lost the message before.
- slack.py: all 14 sites converted (incl. the {:.0f} handshake-timeout
spec -> %.0f); AST scan shows zero "{}" logger calls left.
- whatsapp.py: all 5 sites converted (incl. the multi-line unauthorized-
sender pairing notice); zero leftover per the same scan.
- feishu.py: drop the stale "logging_bridge not needed" comment - the
last reference to the ghost module the matrix fix removed.
The scan found the same class in the remaining 12 adapters (~321
sites, feishu/dingtalk/telegram largest); that sweep is out of this
increment's scope and tracked in #1531.
Part of #1519.
Signed-off-by: shadowinlife <shadowinlife@gmail.com>
The web-config Test/Save routes construct an ephemeral channel instance
to validate a section before anything is persisted. QQChannel.__init__
mkdir'd its (user-controlled) media root, so every Test click and every
rejected Save still left a created directory behind - the ephemeral-
construction pattern presumes side-effect-free constructors, and QQ is
the first guided channel whose constructor violated that (adversarial-
review finding; heavier __init__s would join as the guided roster grows).
- _init_media_root keeps the path choice and the INFO log but no longer
touches the filesystem; a new _ensure_media_root performs the mkdir
and is called from start() only. Pinned by a construction-vs-ensure
regression test.
- start() treats an unwritable media dir as a warning, not a startup
failure: a text-capable bot must not die for a directory only inbound
attachments need, and each download already re-attempts the mkdir and
fails on its own (previously the OSError surfaced at construction and
made the channel unloadable).
Part of #1519.
Signed-off-by: shadowinlife <shadowinlife@gmail.com>
Three hardening fixes from the adversarial review of the QQ increment,
all on the shared route/metadata seam every future guided channel rides
on:
- config_meta._derive_hints now skips dict-valued default keys. The
generic form edits text/password/bool/list widgets only; a dict field
derived as "text" rendered as "[object Object]" once the GET defaults
backfill reached pristine forms (signal dm/group, whatsapp
lid_mappings, napcat group_policy_overrides, mochat mention/groups).
Such fields stay file-configured: excluded from hints, still present
in GET values, secret masking unchanged (derived hints only ever
marked what SECRET_KEY_RE catches unconditionally). Pinned by the
updated fallback-derivation test and a signal-shaped regression.
- _effective_section's except branch now logs at debug with exc_info,
matching _derive_hints: an adapter whose default_config() breaks no
longer loses form seeding silently - the same class of invisible
failure the backfill fix exists to eliminate.
- POST /channels/{name}/test shares PUT's unknown-key rejection via the
extracted _unknown_keys helper. Before, PUT refused unrecognized keys
with 422 while /test silently dropped them through pydantic
extra="ignore" into the ephemeral probe instance; a future config
model with extra="allow" would have accepted arbitrary keys through
the unguarded path. Both routes now answer the same 422
validation_error envelope, pinned by a /test regression.
Part of #1519.
Signed-off-by: shadowinlife <shadowinlife@gmail.com>
An adversarial review of the QQ increment reproduced a crash in the
shared probe seam: a token endpoint answering HTTP 200 with valid JSON
that is not an object (null, a list, a string - exactly what middleboxes
and proxied error pages emit) made resp.json().get(...) raise
AttributeError. _apply_update wraps the probe in try/except, but the
/test route does not, so POST /channels/{name}/test - the endpoint whose
purpose is honest ok/invalid_credentials/network envelopes - returned a
bare 500. Every future channel routed through the shared module would
inherit the hole.
- token_probe: parse the body, then guard isinstance(body, dict) before
.get(token_key); non-object bodies classify as invalid_credentials.
Pinned at both layers: 3 probe-level cases (null/array/string) and 3
API-level regressions through POST /channels/dingtalk/test asserting a
200 envelope, never a 500.
- token_probe docstring: the per-channel adoption checklist (probe
module, registry._INTERNAL, adapter delegate, FIELD_HINTS, GUIDE_DEFS
+ i18n, tests + README - the real seam count the review measured) and
the documented contract (bounded 200-char detail echo scrubbed of
declared secrets only; success = truthy token_key value).
- qq_probe/dingtalk_probe: the missing-credentials short-circuit now
carries sdk_available like every other envelope branch, matching both
docstrings' promise; the DingTalk contract suite (untouched) stays
green as the regression pin.
Part of #1519.
Signed-off-by: shadowinlife <shadowinlife@gmail.com>
One sentence per README in the existing "Configure from the Web UI"
paragraph: register a bot on the QQ Open Platform (q.qq.com), copy
AppID/AppSecret into the form, verify with Test connection before
saving, then enable - no public callback URL needed (WebSocket via the
official botpy SDK). The DingTalk "first fully-guided channel"
sentence stays; news entries are historical and untouched.
Part of #1519.
Signed-off-by: shadowinlife <shadowinlife@gmail.com>
The guide useMemo branched on name !== "dingtalk"; adding QQ as the
second guided channel turns that into a GUIDE_DEFS registry keyed by
channel name - adding a future channel is one entry plus its i18n
keys, with no component logic touched. Builders receive t and use
literal keys only (typed i18n rejects interpolated ones), so copy
re-resolves on language change.
- QQ: 8 field label/help entries and a 5-step setup guide (QQ Open
Platform q.qq.com, AppID/AppSecret, no public callback URL -
WebSocket via the official botpy SDK) in all 8 locales.
- Tests: QQ fields render from generic metadata with the masked-secret
placeholder, the guide expands with its steps and docs link, and a
channel without a registry entry renders no guide.
Part of #1519.
Signed-off-by: shadowinlife <shadowinlife@gmail.com>
Found by the live QQ web-config E2E: a never-configured channel has no
on-disk section, so GET /channels/config returned empty values, the
generic form seeded "" into every widget, and Test/Save echoed those
empty strings into ephemeral validation - QQ's typed fields rejected
them (validation_error: msg_format, download_chunk_size,
download_max_bytes) before any network call. DingTalk never exposed
this because its fields are all str/bool/list, where "" is valid.
GET-shape values are now {**adapter default_config(), **stored
section}: the form shows the effective configuration, an untouched
form round-trips through validation, and typed fields of any future
channel are covered by construction. Secret masking is unchanged -
defaults are empty, so secrets still reflect only stored state.
Regression tests: defaults backfill for an unconfigured channel, and a
full pristine-form round-trip (GET values echoed back as a form patch
must pass the test endpoint) mirroring the frontend buildPatch.
Part of #1519.
Signed-off-by: shadowinlife <shadowinlife@gmail.com>
src.utils.logging_bridge has never existed in this repository's history
(no ref contains it), so matrix.py failed at import even with the
[matrix] extra fully installed - and the registry's ImportError handler
misreported the cause as a missing SDK, pointing operators at a pip
install that could never fix it.
The only use was redirect_lib_logging("nio", level="WARNING") in
start(); the stdlib equivalent logging.getLogger("nio").setLevel(
logging.WARNING) keeps the chatty nio SDK loggers suppressed with no
new module. Verified the adapter imports cleanly with stubbed nio/nh3/
mistune dependencies.
Found while wiring the QQ connection test (separately-labeled pre-
existing fix, per the one-line websocket Logger fix precedent in this
branch).
Signed-off-by: shadowinlife <shadowinlife@gmail.com>
QQ becomes the second fully-guided web-config channel, and the shape of
the increment is the extensibility proof #1519 asked for: two new probe
modules plus one hints entry, with zero route, manager or component
changes.
- token_probe.py: DingTalk's OAuth accessToken exchange and QQ's
getAppAccessToken are the same contract (POST credentials to a token
endpoint, classify by status + token presence, scrub secrets from
every detail, discard the token), so the request/classification logic
moves here as the shared building block. dingtalk_probe keeps its
public API and behavior unchanged and delegates; its existing
contract suite is the regression pin for the extraction.
- qq_probe.py: standalone QQ probe (fresh httpx client, works pre-start,
QQConfig only under TYPE_CHECKING so a credential check never pulls
aiohttp/botpy). QQ answers HTTP 200 with an error body for bad
credentials (observed live: {"code":100016,"message":"invalid appid
or secret"}) - the shared 200-without-token branch classifies that
as invalid_credentials, pinned by a QQ-shaped regression test.
- qq.py: supports_connection_test + thin test_connection delegate.
Also converts 20 loguru-style "{}" log calls to stdlib %s lazy
formatting: with stdlib logging the old form raised TypeError inside
handler formatting, losing the message and printing a logging error
on every INFO-level startup (verified fixed in production: the
reconnect warning now renders). Import block cleaned (E702/E402) and
the file brought to black compliance.
- config_meta: hand-written QQ hints (8 fields, secret/required flags,
i18n help keys) with an exact-snapshot test.
- registry._INTERNAL gains qq_probe/token_probe; discovery stays at 16.
Part of #1519.
Signed-off-by: shadowinlife <shadowinlife@gmail.com>
User-facing features of this size land a dated News bullet per repo
convention; the branch only carried the "Configure from the Web UI"
subsection. Add the 2026-09-20 entry (validate-before-save, masked
secrets, connection test, per-channel hot apply, DingTalk guided
setup) to all six READMEs, and note in the subsection that only
built-in adapters are listed — entry-points plugin channels remain
file-configured. Dated news bullets are excluded from the
test_readme_counts anchors, which stay green.
Part of #1519.
Signed-off-by: shadowinlife <shadowinlife@gmail.com>
The 422 diagnostic chain died on the frontend: errorFromResponse
assigned a dict detail straight into the Error message ("[object
Object]"), ApiError could not carry the backend's code, and
describeApiFailure probed properties that never existed — so a
validation failure rendered "Unknown error" and the config_not_writable
branch was unreachable.
- ApiError gains an optional code; errorFromResponse parses structured
{code, message} details (string detail/message/error handling is
unchanged for every other endpoint).
- describeApiFailure reads status/code/message from the real error;
the OPAQUE_DETAIL workaround is gone.
- Save is disabled until the form is dirty — a pristine Save used to
fire a no-op PUT plus a full hot-swap cycle.
- Test connection sends pending secret clears, matching the "values
currently in the form" label.
- Re-mock the enable-anyway test to the production error shape (the
old assertion passed only because the mock stuffed the code into the
message) and pin the new Save-gate and pending-clear behavior.
- Trailing newlines on the new components.
Part of #1519.
Signed-off-by: shadowinlife <shadowinlife@gmail.com>
Adversarial-review fixes for the channel-config write path:
- Hot-reload failure fallback reset the runtime singleton and started
a new one while the old runtime (consume loop, dispatcher, connected
adapters) kept running. The fallback now stops the old runtime
first, bounded at 10s so a wedged adapter cannot stall the rebuild;
the covering test used a no-op reset mock that could not observe the
leak — rewritten against the real reset with stop tracking, plus a
wedged-stop case.
- 422 envelopes carry a scrubbed, field-naming message (names only,
never values); a raising enable-probe becomes 422/network instead of
escaping as a bare 500.
- Unknown config keys are rejected with 422 instead of persisting
unvalidated; documented manager override keys stay accepted, and
merge-patch semantics for stored legacy keys are unchanged.
- GET returns a display-only, home-relative config_path instead of the
absolute server path.
- POST /channels/{name}/test honors pending clear_ flags so the probe
matches the UI's "values currently in the form" promise.
- config writer: a process-wide lock serializes the whole-file
read-modify-write across channels and entry points (web PUT, Feishu
QR login), closing the cross-channel lost-update window.
- Drop the dead _NON_CHANNEL_MODULES duplicate, document the
plugin-channel scope cut, and replace a dangling local-plan
reference in the test docstring with the issue link.
Part of #1519.
Signed-off-by: shadowinlife <shadowinlife@gmail.com>
stop_all() iterated manager.channels while awaiting channel.stop() in
the loop body — and a DingTalk stop can take seconds. A config
hot-swap (reload_channel pops/inserts the same dict synchronously)
interleaved with /channels/stop raised "dictionary changed size during
iteration" mid-loop, leaving every later channel running after the
operator requested a full stop. start_all()'s post-gather status loop
had the same exposure.
- Iterate list(...) snapshots in stop_all and start_all; the tolerated
race (a channel inserted after the snapshot) is documented: reload
only spawns starts while the dispatcher is live, and stop_all
cancels it first, so a late insert sits loaded-but-inert.
- Add the two regression pins #1519 claimed but did not ship:
start_all() with zero enabled channels still creates the outbound
dispatcher (a later reload_channel spawns the channel), and
reload_channel returns promptly while the replacement's start()
blocks forever (no inline await; the spawned task is tracked).
Part of #1519.
Signed-off-by: shadowinlife <shadowinlife@gmail.com>
The connection-test feature pushed dingtalk.py from 779 to 901 lines,
past the repo's 800-line hard cap. Both helper clusters are stateless
apart from the config, so they move cleanly:
- dingtalk_probe.py: the credential probe (test_connection, scrub and
response-detail helpers, IPv4-pinned transport builder, token URL).
DingTalkChannel.test_connection becomes a thin delegate; a
compatibility alias keeps _build_http_transport importable for the
existing transport pins.
- dingtalk_media.py: URL classification and upload packaging helpers
(is_http_url, guess_upload_type, guess_filename, zip_bytes) with
their extension sets; call sites updated, no behavior change.
- registry._INTERNAL gains both module names so channel discovery
stays locked at 16.
dingtalk.py is now 787 lines; the probe and media modules are 124 and
82. All existing DingTalk tests pass unchanged.
Part of #1519.
Signed-off-by: shadowinlife <shadowinlife@gmail.com>
An adversarial review of the web config surface found the read path
violating its own "secrets never cross the wire" invariant: the
secret-key regex (secret|token|password|api_key) missed Feishu's
encrypt_key — a live event-subscription shared secret returned in
cleartext by GET /channels/config — plus Discord's proxy_username and
proxy URLs carrying embedded user:pass@ userinfo. _mask also disclosed
secrets of <=4 characters in full ("****" + last-4 echoed the value).
- Extend SECRET_KEY_RE to a deny-by-default credential family
(encrypt/signing keys, credentials, private values, proxy usernames)
so derived hints and the split both classify them as secrets.
- Strip URL userinfo from non-secret values (malformed URLs pass
through unchanged), keeping proxy hosts editable but credential-free.
- Mask secrets of <=8 characters with a fixed "****" (no suffix).
- Add a sweep pinning every credential field of all 16 adapters into
secrets, an identifier-visibility pin against over-masking, and
boundary tests for the mask and the userinfo stripping.
- Drop the local _NON_CHANNEL_MODULES duplicate: registry._INTERNAL
already excludes helper modules from discovery.
Part of #1519.
Signed-off-by: shadowinlife <shadowinlife@gmail.com>
Add a 'Configure from the Web UI' subsection to the IM channels section of
all six READMEs: expandable per-channel panel, DingTalk Stream Mode setup
guide, credential probing before save, immediate hot-apply enable/disable,
masked secrets, and the JSON-only editing constraint.
Part of #1519.
Signed-off-by: shadowinlife <shadowinlife@gmail.com>
Replace the read-only IM Channels section with ChannelSettings (status
table plus expandable rows) and a generic ChannelConfigPanel driven by the
backend field metadata: text/password/bool/list-tag inputs, masked-secret
placeholders with a clear flag, a Test-connection button that probes dirty
unsaved values, a non-destructive Enable toggle with an explicit
skip_verify override after a failed verification, a collapsible DingTalk
setup guide, YAML read-only and missing-SDK banners. Rendering is
field-driven so a secret smuggled into values cannot reach the DOM
(asserted). New keys land in all 8 locales; existing Settings tests keep
covering the migrated section.
Part of #1519.
Signed-off-by: shadowinlife <shadowinlife@gmail.com>
New sibling module (channels_routes.py untouched, same host-resolution and
local-model conventions):
- GET /channels/config: availability + live status + field metadata over
the on-disk section; secrets only as {set, masked}; reports writable
(JSON-only) and runtime_running without forcing a singleton build.
- PUT /channels/config/{name}: merge-patch (absent secret = keep,
clear_<field> removes, empty-string secret ignored); ephemeral adapter
construction validates before any write (422, file byte-unchanged); the
enable transition auto-verifies credentials unless skip_verify; atomic
write then hot apply: running -> reload_channel (hot_swapped),
built-but-stopped -> singleton reset (reset), never built -> deferred.
- POST /channels/{name}/test: probes unsaved credentials on an ephemeral
instance, never persists, honest codes, detail scrubbed.
Per-channel asyncio locks serialize save/test/toggle; a global lock guards
runtime rebuilds. 23 route tests include a concurrency-serialization case
and a secret-hygiene sweep over every raw response body.
Part of #1519.
Signed-off-by: shadowinlife <shadowinlife@gmail.com>
A generic web form needs machine-readable field metadata. config_meta
provides hand-written hints for DingTalk (i18n help keys owned by the
frontend) and derives hints for every other adapter from default_config()
with type inference. split_values_secrets() masks secrets as ****+last4
and strips every secret-heuristic key from values unconditionally, so a
future adapter cannot leak a credential through an unhinted field; a
parametrized sweep asserts this across all 16 adapters.
Also stop discovery from listing the targets and config_meta helper
modules as channels (they surfaced as bogus unavailable rows in
/channels/status).
Part of #1519.
Signed-off-by: shadowinlife <shadowinlife@gmail.com>
Config edits never applied without a process restart: the manager built
channel instances from a constructor snapshot and the API cached the
runtime singleton in module globals plus host attrs.
- Extract _build_channel() and add ChannelManager.reload_channel(): stop
the old instance (bounded, never wedges the swap), rebuild from the new
section, start it iff the dispatcher is live, and keep _status and the
config snapshot in sync. Other channels' instances are untouched.
- start_all() now always creates the outbound dispatcher; the previous
early return on zero enabled channels left a later-enabled channel
unable to start.
- state.reset_channel_runtime() clears both cache surfaces so the next
access rebuilds from disk.
Part of #1519.
Signed-off-by: shadowinlife <shadowinlife@gmail.com>
BaseChannel gains supports_connection_test plus an async test_connection()
defaulting to an honest 'unsupported' envelope. DingTalk overrides it with a
standalone accessToken probe (fresh httpx client, works before start(),
token discarded, detail scrubbed of credentials) so the web UI can verify
unsaved credentials.
Live end-to-end QA against a real DingTalk org app surfaced three adapter
defects, each pinned by a regression test:
- _on_message never forwarded is_dm, so unapproved DMs hit the group deny
log and the pairing-code flow was dead code for DingTalk.
- stop() never terminated the SDK stream loop (it swallows the first
CancelledError and reconnects), leaving a zombie WebSocket after disable
and hanging graceful process shutdown; stop now double-cancels the
tracked stream task.
- New force_ipv4 option binds outbound API calls to IPv4: the robot-send
API enforces the app egress-IP whitelist and a rotating IPv6 prefix on
dual-stack networks keeps falling out of it (observed as 403
IpNotInWhiteList), while the IPv4 egress is stable.
Part of #1519.
Signed-off-by: shadowinlife <shadowinlife@gmail.com>
Channel credentials could only be persisted by hand-editing agent.json; the
only in-repo writer was the Feishu QR-login persister. Extract a generic
update_channel_section() that merges one channels.<name> section of the raw
JSON dict (no schema round-trip, unknown keys preserved), applies clears,
and writes atomically (tmp + fsync + os.replace, 0600/0700, JSON-only with
ConfigNotWritableError otherwise). Feishu now delegates to it, keeping its
ValueError public contract.
Part of #1519.
Signed-off-by: shadowinlife <shadowinlife@gmail.com>
The quick-action chip row styled its first entry with the primary-color
"selected" look unconditionally, while clicking any chip only filled the
composer without changing the chip. Users read the permanent highlight
as a selection that never moved when they clicked another action (e.g.
the investment-committee debate chip stayed neutral while the first chip
kept glowing).
Selection is now component state keyed by the action title: clicking a
chip marks it aria-pressed and moves the primary styling to it, and no
chip is highlighted before the first click. Adds a test covering the
initial, clicked, and moved selection states; verified in a real browser
against the dev server.
Signed-off-by: shadowinlife <shadowinlife@gmail.com>
get_market_data normalized interval spellings through
_INTERVAL_CANON = {v.upper(): v for v in _VALID_INTERVALS}. Because
_VALID_INTERVALS mixes cases ("1m" minutes vs "1H" hours), the uppercase
key for "1m" is "1M", so a caller passing interval="1M" -- the universal
monthly convention in pandas resample('1M'), ccxt and TradingView --
silently received 1-minute bars labelled as the answer to a monthly
question. The is-None guard never fired because the fold always resolved,
so the wrong-unit substitution was invisible to both the agent loop and a
human MCP client.
Resolve a case-exact match against _VALID_INTERVALS first, then fall back
to the fold for unambiguous case variants only ("1d" -> "1D",
"30M" -> "30m"). The unserved month/week spellings "1M"/"1W" are now
rejected with a pointed message naming the minute collision instead of
being mis-served. The case-exact-first ordering keeps this correct if
monthly bars are ever added to _VALID_INTERVALS (#1479).
Closes#1480
Signed-off-by: shadowinlife <shadowinlife@gmail.com>
Southbound (mainland → HK) is the half of Stock Connect whose daily net
buy is still officially disclosed: HKEX publishes per-channel Buy/Sell
Turnover every trading day, and Eastmoney's datacenter series
cross-checks against it exactly. The northbound net ended 2024-08-30
(#1481), which leaves southbound as the only live net-flow signal — and
its tool surface was missing: the hk-connect-flow skill's southbound
methodology could only point at hand-written tushare code whose
southbound fields (ggt_ss/ggt_sz/south_money) have themselves drifted
to cumulative net-bought stock (#1483).
- southbound_tool.py: get_southbound_flow, mirroring the northbound
envelope. Primary: Eastmoney datacenter RPT_MUTUAL_DEAL_HISTORY
(MUTUAL_TYPE 002/004) via the shared throttled client, NET_DEAL_AMT
in 100M HKD. Fallback: the keyless HKEX official daily-statistics
report (latest trading day, net = Buy − Sell, million → 100M HKD),
walking back a week past 404s. No tushare path; the envelope note
says why.
- mcp_server.py: hand-written wrapper (Surfaces 75) + family docstring.
- agent/SKILL.md: manifest heading 74→75 + tool table row.
- hk-connect-flow SKILL.md: Data Access gains the tool route and a
caveat that tushare's southbound fields are cumulative stock.
- six READMEs: MCP tool list + counts (74→75), repo-tree agent-tool
count (107→108), MCP prose counts — all six locales.
- tests: 12 cases — merged envelope, trim/clamp, one-channel-missing,
HKEX fallback triggers (failure + empty rows), 404 walk-back,
both-down error, a decoy top-10-table regression pinning the live
HKEX payload shape, parser guards, metadata/description anchors.
Live-verified 2026-09-18: Eastmoney and HKEX agree to the cent on
2026-09-15/16/17 (e.g. 09-17 港股通沪 net 25.0239亿 == HKEX Buy
26,387.65M − Sell 23,885.26M HKD); an incidental live EM SSL failure
exercised the HKEX fallback end to end with identical figures.
Closes#1483
Signed-off-by: shadowinlife <shadowinlife@gmail.com>
Eastmoney's kamt northbound net-flow feed died with the 2024-08-30 HKEX
disclosure change (all-zero nets, frozen cumulative, drifted payload
shape), but its HTTP calls still succeed, so get_northbound_flow
returned ok:true with null realtime and empty history and never reached
the tushare fallback (#1481).
- northbound_tool: detect the dead state (realtime AND history both
empty/all-zero) and route it through a shared tushare-fallback
envelope; without a usable fallback the tool now returns an honest
ok:false naming the disclosure change instead of an empty success.
- tushare_fallbacks: drop the blanket x100 rescale. Since 2024-08-30
tushare's hgt/sgt/north_money carry turnover in million CNY (verified
equal to HKEX official Total Turnover to the cent on three anchor
days); earlier rows are net buy in the same unit. Values now pass
through unchanged with unit "CNY million" and a note documenting the
semantics boundary.
- tests: dead-feed fall-through, honest error without tushare, healthy
payload guard, one-sided-dead guard; the x100 pin is replaced by the
passthrough+note pin.
Closes#1481
Signed-off-by: shadowinlife <shadowinlife@gmail.com>
Review follow-up to #1413: Runner.execute propagates
subprocess.TimeoutExpired, so a timed-out engine bypassed the
lifecycle block entirely — the run directory kept partial artifacts
with no state.json, indistinguishable from a never-executed run (the
evidence gate fail-closes either way, but the reason was lost and the
caller got a raw exception instead of an envelope).
Catch the timeout, record {"status": "failed", "reason": "backtest
engine timed out after 300s"} via the same RunStateStore contract, and
return a clean error envelope. A new test pins the state payload and
the envelope.
Part of #1412.
Signed-off-by: shadowinlife <shadowinlife@gmail.com>
Two follow-ups from the review of #1411, both protecting the guarantee
the pagination fix establishes — a served series is the complete window
or the fetch fails loudly:
1. Bump _LOADER_CACHE_VERSION 6 -> 7. The cache key is content-addressed
on (source, symbol, timeframe, start, end) with no TTL, so entries
written by the forward walk during the bug window (#1154, 2026-08-20,
onward) hold tail-truncated 500-bar series that would keep being
served after the fix. Follows the v4/v5/v6 precedent of bumping
whenever cached semantics change.
2. Harden the two paths where a throttled-but-HTTP-200 reply could still
truncate silently: _request_page now raises on an error-shaped body
(non-zero code, no payload) so it enters the page retry budget instead
of reading as an empty window, and _fetch_one re-requests a mid-walk
empty page once before accepting it as the genuine start of history.
Three new tests pin all three behaviors.
Part of #1410.
Signed-off-by: shadowinlife <shadowinlife@gmail.com>
SwarmAgentSpec.skills documents a per-worker skill boundary, but the
boundary was only applied to the skill descriptions injected into the
worker prompt; the load_skill tool itself would load any skill by name,
so a worker could pull in documentation its preset never granted.
LoadSkillTool now accepts an optional allowed_skills set and fails
closed with the allowed names when a skill outside it is requested;
None keeps the historical unrestricted behavior for the main agent.
build_filtered_registry and build_swarm_registry take a matching
skill_allowlist parameter and rebuild load_skill with the boundary
baked in (register() replaces the auto-discovered instance by name),
and run_worker delivers agent_spec.skills through it. An empty skills
list remains unrestricted, matching the prompt-side filter semantics.
Also adapts the four build_swarm_registry test doubles in
test_swarm_m4_e2e.py to the new keyword-only parameter (this
adaptation was carried by the feature commit in #1286; the fix
is not self-contained against the current test suite without it).
Resubmitted standalone as requested in the #1286 review ("please do
re-submit it separately, and it does not need to wait on anything");
gap re-verified on main @ f9cb061b — models.py still documents the
boundary, the worker.py prompt-text filter is still the only reader
of spec.skills, and load_skill_tool.py still has no enforcement.
Signed-off-by: shadowinlife <shadowinlife@gmail.com>
refresh_strategy_evidence fail-closes on run directories without state.json,
and only the runtime loop wrote that file — so every run produced by the
backtest MCP tool was rejected as unknown-provenance despite complete,
hash-manifested artifacts ("state.json missing or unreadable (run status
unknown)"). Observed at scale during a strategy-validation campaign: 11/11
healthy tool-driven runs skipped, recoverable only by hand-deriving
state.json from run cards.
run_backtest already knows the outcome (result.success / result.exit_code),
so record the lifecycle status right after engine execution by reusing
RunStateStore.mark_success / mark_failure — the same contract and fsync'd
writer the runtime uses, keeping one source of truth for the state.json
format. Validation errors return before the engine starts and intentionally
record nothing: no execution happened, and artifact-health gates reject
those runs anyway. The ingestion gate is untouched and stays fail-closed
for runs whose status was never recorded.
Tests pin all three paths (success / engine failure with reason / validation
error writes no state). Manual E2E on this branch: a fresh yfinance run
through run_backtest produces state.json {"status": "success"} and
refresh_strategy_evidence_core accepts it (runs:1, strategies:1, rows:1,
skipped:[]) with no bridging.
Closes#1412.
Signed-off-by: shadowinlife <shadowinlife@gmail.com>
#1154 added forward pagination assuming each fqkline response carries the
first <=500 bars from the cursor. The live API does the opposite: it returns
the last <=500 bars ending at the requested end date (verified 2026-09-11:
param=sh000300,day,2018-01-01,2026-06-30,500,qfq -> 500 bars spanning
2024-06-06..2026-06-30; the start clamp is honoured, so a sub-500-bar
window is served in full). With forward advancement the first page already
ends at end_date, the cursor steps past the window, the next page is empty
and the loop exits cleanly — any multi-year window silently degrades to its
most recent ~2 years with exit code 0 and no warning.
Walk backward instead: start_date stays fixed, the end cursor moves to the
day before each page's oldest bar, and the walk stops on a short page or
when the cursor passes start_date. Page-cap exhaustion and failed pages
still raise rather than truncate.
Tests now mock the real last-500 semantics (the previous mocks encoded the
first-500 assumption, which is why the truncation passed CI) and add a
multi-year regression asserting the served window starts at start_date.
Live smoke after the fix: 000300.SH 2018-01-01..2026-06-30 -> 2058 bars
from 2018-01-02 (was 500 bars from 2024-06-06).
Closes#1410.
Signed-off-by: shadowinlife <shadowinlife@gmail.com>
Implements #969 Phase 2 over the Phase 1 facade (#978). The evidence cache
is no longer empty-by-construction and no longer silent about age:
- refresh_strategy_evidence: agent tool (primary, scheduler-reachable) +
MCP wrapper + 'vibe-trading strategy-evidence refresh --manifest' CLI.
Manifest-driven only — runs carry no attribution, so explicit
{strategy_id, run_dir, position_size} specs are the honest path; entries
outside the runtime runs root / allowed run roots are skipped with a
stable path-outside-allowed-roots reason. Writes only the disposable
facade-owned cache.
- backtest-diagnose hookup: the skill's Hard-Gate Checklist becomes the
evidence ingestion gate — state.json success, metrics.csv present,
trade_count > 0, equity present and NaN-free, each with a stable
hard-gate:* skip token. NaN-equity runs are refused entirely instead of
yielding partial-curve evidence (fixes the Phase-1 silent-skip latent).
- atomic rebuild: compute all rows first, then one transaction swaps the
store — a failed rebuild can no longer leave the cache empty.
- decay status surfaced in query_strategies / get_strategy_evidence:
freshness computed at read time from the evidence-window age
(fresh/aging/stale, 90/180-day thresholds tied to the quarterly
re-backtest cadence; last_verified reported but never gating). Stale rows
fail closed out of default recommendations (include_stale opt-out);
sdm:* rows mirror the SDM lifecycle (decayed/disabled never recommended,
sdm-lifecycle: prefix); unparseable windows fail closed to stale.
Decay is never persisted — a stored decay field would go stale by
definition.
- docs + count sync: strategy-discovery SKILL.md gains the Decay &
Freshness Contract and Populating & Refreshing Evidence sections;
backtest-diagnose SKILL.md documents the gate hookup; phantom-guard test
now asserts every tool named in the SKILL.md recipe is registered;
MCP tools 73 -> 74 across five READMEs + agent/SKILL.md (65 -> 66
decorators).
Visual QA of the P0 panels found three cosmetic rendering defects:
- Attribution cumulative and rolling beta/alpha charts set only
lineStyle.color, so ECharts legend swatches fell back to the default
palette and mismatched the rendered lines. Set itemStyle.color to the
same color on every series (legend swatches are driven by itemStyle).
- Factor research rendered the long-short spread as a bare decimal
(0.4168) while sibling stats used percentages; it now renders as a
signed percent (+41.68%), and signed IC stats (IC mean, IR, spread)
take positive/negative tones via RunCardStat.
- Options payoff x-axis used the default nice domain, extending to zero
and leaving a blank dead zone left of the curve; pin min/max to the
curve's data extents.
Adds ChartOptionConsistency tests asserting legend/line color parity and
the payoff axis extents; updates the factor panel test to the signed
percent format.
Signed-off-by: shadowinlife <shadowinlife@gmail.com>
Post-review hardening for the attribution panel and cross-panel audit:
- attribution: an asset class whose long/short weights net to zero
(market-neutral pair books) divided by zero and 500-ed the whole
endpoint; it now degrades to the invested/cash fallback. Also guard
benchmark_equity starting at a non-positive value, and fix the
misleading skip note when ohlcv artifacts are missing.
- attribution: symbols from positions.csv headers / config codes are
interpolated into ohlcv_<symbol>.csv paths; reject path separators,
.. sequences and NUL bytes so artifact-derived names cannot escape
the artifacts directory.
- positions/sectors: reject a symlinked positions.csv (defense in
depth with the artifacts-dir and cache-file guards) and write the
sector_map.json cache atomically via temp file + os.replace so
concurrent refreshes cannot tear it.
- frontend: migrate the three attribution charts to the shared
useChartLifecycle hook introduced by this series, and escapeHtml the
CandlestickChart trade-marker tooltip fields (trades.csv side/price/
qty/reason), closing the last artifact-derived tooltip sink.
Regression tests: market-neutral pair degrades instead of crashing,
zero-start benchmark_equity skips Brinson with a note, traversal-shaped
symbols never reach the path join, and a symlinked positions.csv is
treated as no positions artifact. All four fail on the pre-fix code.
Signed-off-by: shadowinlife <shadowinlife@gmail.com>
The 200-lookup cap gated on the symbol's index in the full positions
list, so a mixed book with 200+ non-A-share names in front pushed every
A-share past the cap and resolved zero industries while the budget sat
unused. Gate on the A-share lookup count instead, matching the
documented contract (at most 200 network lookups, excess degrade).
The symlink guard covered the artifacts directory but not the
sector_map.json cache file inside it, so a planted cache symlink was
still followed by the cache read and rewrite. Reject a symlinked cache
file the same way as a symlinked artifacts dir.
Regression tests: mixed book resolves A-shares after a 205-symbol
non-A-share prefix, the cap still bounds mixed books at exactly 200
lookups, and a symlinked cache file returns the no-positions note
without touching the symlink target.
Signed-off-by: shadowinlife <shadowinlife@gmail.com>
Second-round changes following a five-agent post-implementation audit,
extending this PR from the Positions panel to a comprehensive
optimization across all five visualization panels.
Security (cross-panel):
- Add escapeHtml and apply it to every chart tooltip formatter that
interpolates artifact-derived strings (position symbols and dates,
factor directory names, equity timestamps). ECharts renders custom
formatter output via innerHTML unescaped, and with the API key in
localStorage a crafted run directory's tooltip was an escalation
path to full API control.
Attribution panel (new, gap-analysis P0 #4):
- GET /runs/{run_id}/attribution: factor decomposition (OLS alpha/beta
with t-statistic, rolling window, cumulative portfolio/benchmark/
active) plus Brinson-Fachler attribution (symbol / asset_class /
invested_cash modes), computed entirely from persisted artifacts with
per-section degradation notes, <=500-point downsampling with the last
point pinned, 6-decimal rounding, and threadpool offload.
- Attribution tab in Run Detail: stat cards, Brinson table, waterfall,
rolling beta/alpha and cumulative charts; i18n in all six locales.
Fixes touching code already on main:
- GET /runs/{run_id}/factor: artifact scan offloaded via
run_in_threadpool (was blocking the event loop with synchronous IO).
- Options payoff route: unexpected tool exceptions now return the 502
error envelope instead of a raw 500.
- CorrelationMatrix and EquityChart tooltips XSS-hardened (affects the
Factor Research panel and the tearsheet equity chart).
- Tearsheet equity chart data memoized; FactorResearchPanel <-> RunDetail
circular import resolved (RunCard components extracted); chart
lifecycle promoted to a shared useChartLifecycle hook.
Positions hardening:
- Sector cache rejects a symlinked artifacts directory.
- Industry resolution reuses one shared bounded executor instead of
spawning a thread pool per request.
Tests: attribution API suite, symlink / shared-executor / threadpool-
offload / payoff-envelope regressions, escapeHtml unit tests,
attribution tab states, resolve-button gating. Frontend 554 passed;
backend contract suites 89 passed.
Signed-off-by: shadowinlife <shadowinlife@gmail.com>
Resolve the short-book correctness issues raised in review:
- latestHoldingDate now detects holdings by gross exposure
(sum |weight|) so market-neutral books render instead of showing
the empty state.
- withCashSlice derives the residual cash slice from the same gross
basis as the pie slices (1 - sum |weight|), omitting it when the
book is fully committed or levered instead of fabricating cash
from the signed sum. Card subtitles state each chart's denominator
(gross |weight| vs net signed weight, margin / NAV).
- Sector and evolution charts use signed auto/symmetric axes instead
of a hard-coded [0, 1] window; evolution ranks symbols by average
absolute exposure and splits the remainder into separate long/short
series so omitted legs never cancel.
- GET /runs/{run_id}/positions/sectors runs its blocking work in the
FastAPI threadpool and is bounded: at most 200 network lookups per
resolve, 4 concurrent workers, shared client 15s per-request
timeout; the envelope reports total_symbols/symbol_limit.
Adds regression tests for market-neutral, short-only, levered,
net-short axes, non-cancelling remainder, and capped large books.
Signed-off-by: shadowinlife <shadowinlife@gmail.com>
New Positions tab renders the portfolio book from the existing
positions.csv artifact: per-symbol weight pie/treemap with a date
slider, sector distribution, and a weight-evolution stacked area
chart. Weights arrive as filled fractions per date; the pie shows
gross composition (short legs rendered as labelled slices, uninvested
remainder as a cash slice) while the sector bars show net exposure.
Sector distribution starts from an offline asset-class grouping and
upgrades to Eastmoney Shenwan industries through a new read-only
GET /runs/{run_id}/positions/sectors endpoint, resolved on demand,
cached in artifacts/sector_map.json, and degraded per-symbol so one
failed lookup never aborts the batch. Non-A-share symbols never reach
Eastmoney.
Registers PieChart/TreemapChart in the shared ECharts bundle, adds
the positions i18n keys to all six locales, and covers the transforms
and the tab with vitest plus the endpoint with pytest.
Signed-off-by: shadowinlife <shadowinlife@gmail.com>
Close the P0 gap where factor_analysis artifacts (ic_series.csv / ic_summary.json /
group_equity.csv) were computed but never rendered. Run Detail gains a Factor
Research tab, gated on a new has_factor_artifacts flag, that lazily fetches
GET /runs/{run_id}/factor: a recursive scan of the run's artifacts directory
for factor bundles (one directory per factor, standardized via the
factor-research skill), serving IC series, IC summary stats, quantile group
equity curves, and a pairwise IC time-series correlation matrix.
The tab renders the standard factor-research figure set: IC statistics cards,
daily IC bars with a 20-period mean line, quantile group equity curves with
the extreme quantiles emphasized, and the IC correlation heatmap reusing
CorrelationMatrix. CorrelationMatrix cell labels now pick their text color
from the interpolated RdBu cell luminance so values stay readable on
saturated cells in both themes.
Signed-off-by: shadowinlife <shadowinlife@gmail.com>
The options route registration pushed api_server.py to 403 lines, over
the <400 guard in test_api_server_is_thin_assembler. Normalize the
scheduled-routes banner to the file's one-line section-marker
convention to make room.
Signed-off-by: shadowinlife <shadowinlife@gmail.com>
Backend: new /options/payoff and /options/chain HTTP routes wrapping the
existing options_payoff and get_options_chain tools (parity with MCP),
plus portfolio Greeks (delta/gamma/theta/vega/rho, multiplier-scaled)
computed from src.quantlib.options.bs_greeks.
Frontend: new Options Lab page — strategy builder with 12 presets,
expiry payoff chart with breakeven/spot markers and profit/loss fills,
spot x IV scenario heatmap, Greeks cards with plain-language
interpretations, and a live US options chain table (bid/ask spread,
volume, OI, IV, ITM/ATM highlighting). Full six-locale i18n.
Research only; does not place orders.
Signed-off-by: shadowinlife <shadowinlife@gmail.com>
The Run Detail page showed the equity curve and scalar metrics but none
of the tearsheet views every comparable platform (QuantConnect, pyfolio,
QuantStats) ships: a monthly returns heatmap, annual returns bars, and a
top-N drawdown breakdown with peak/trough/recovery dates.
Everything is derived client-side from artifacts_equity_csv — the full
equity.csv rows the run response already delivers — so there is no
backend or API change and no new dependency. The derivation lives in
lib/tearsheet.ts as pure functions (normalize, monthly/annual returns,
running-peak drawdown episodes) with 21 unit tests covering the
hand-computed cases, gap months, non-positive bases, unrecovered tails,
and unparseable timestamps.
The new Tearsheet tab (hidden when a run has no equity data) renders the
annotated equity curve (top-5 drawdown zones as ranked markArea regions),
the monthly heatmap with a full-year column and locale-aware up/down
colors, the annual bar chart, and the drawdown table with duration spans
and a not-recovered state. EquityChart gains one additive optional prop
(drawdownZones); existing call sites are unchanged.
Keys added to all six locales; month axis labels come from
Intl.DateTimeFormat so no per-locale month keys were needed.
Signed-off-by: shadowinlife <shadowinlife@gmail.com>
Phase 2 of #1062. BaoStock was the single A-share source reporting
volume in single shares while tencent/eastmoney/akshare/mootdx/tushare
report board lots (1 lot = 100 shares) — a silent 100x discrepancy
whose value depended on which fallback source served the request.
- baostock_loader: divide native share volume by 100 at the loader
boundary; fractional lots preserved (odd-lot trades are real);
volume_units declaration now "lots" like every other A-share source
- Loader cache version 3 -> 4 so entries cached under the
pre-normalization unit are never served again
- New empirical cross-source consistency test: the same settled
trading day fetched from every reachable A-share loader must agree
within 1% (network-guarded — skips when fewer than two sources are
reachable); this is the runtime lock against future unit drift
- Pin tests updated to the post-normalization matrix
Verified end-to-end: 600519.SH 2026-07-31 — tencent 55,128 lots vs
baostock 55,127.52 lots (ratio 0.999991; the 0.48-lot gap is baostock's
exact share count including odd lots, within the 1% tolerance).
Signed-off-by: shadowinlife <shadowinlife@gmail.com>
A-share loaders return volume in each source's native unit with no
metadata: tencent/mootdx/eastmoney/akshare/tushare report board lots
(1 lot = 100 shares) while baostock reports single shares — a silent
100x discrepancy whose value depends on which fallback source served
the request (see #1062).
Phase 1 (non-breaking; no data values change):
- Loaders may declare volume_units keyed by market; tencent/eastmoney
are market-dependent (A-share lots vs HK shares, empirically
verified 2026-08-11); baostock pinned to shares, the single outlier
- fetch_market_data surfaces _provenance.volume_unit per symbol from
the serving loader ("lots" / "shares"; null when undeclared)
- get_market_data tool descriptions (agent + MCP) state the contract
so LLM consumers read the unit instead of assuming one
Undeclared markets (crypto, futures, US via unverified sources) stay
null rather than guessed. Pin tests lock the audit matrix from #1062.
Signed-off-by: shadowinlife <shadowinlife@gmail.com>
Split Shadow backtests into CNY, HKD, and USD settlement pools, preserve source-currency headline attribution, isolate pool failures, and report per-pool metrics without FX aggregation.
FastMCP publishes optional list/dict parameters as ``anyOf`` schemas, and some
MCP clients (observed with Claude Desktop / Claude Code) do not surface
``anyOf`` to the model as a concrete type — the model then serializes the
argument as a JSON string (``'["us"]'``). Strict pydantic validation rejected
those strings before the tool body ran, so parameters such as
``run_shadow_backtest.markets`` could not be passed over MCP at all.
Attach a pydantic ``BeforeValidator`` (``_coerce_json_string``) to every
list/dict MCP parameter so a JSON-encoded string is decoded before type
checking. The published JSON schema is unchanged, real arrays/objects keep
working, and a non-JSON string still fails with pydantic's normal type error.
Adds a regression suite covering the schema guard and both coercion paths.
Closes#987
Signed-off-by: shadowinlife <shadowinlife@gmail.com>
Every run_shadow_backtest call failed with "backtest failed" in all markets
(issue #985). The generated signal engine could never clear the runner's AST
sandbox validation, for two independent reasons:
1. The codegen template emitted @staticmethod decorators on _compute_rsi,
_compute_prior_return and _conditional_entry, while the runner's
_validate_function_def rejects every decorator. codegen's own looser
validate_generated passed, then the runner's stricter validator rejected
the same file with "Decorators are not allowed on function '_compute_rsi'".
2. Mined negative thresholds (e.g. prior_5d_return_min = -0.08) land in the
top-level RULES literal, where "-0.08" parses as UnaryOp(USub, Constant).
_is_literal_node only accepted bare constants, so the RULES assignment was
rejected as "Executable top-level statement Assign is not allowed" — the
same user-visible failure through a second path, hit whenever a rule carries
a negative bound.
Fix both at the source instead of loosening the sandbox:
- Drop the three @staticmethod decorators and make the helpers instance
methods (add self). All call sites already used self.-style calls, so
behavior is unchanged and the validator's decorator ban stays intact.
- Teach _is_literal_node that a USub/UAdd UnaryOp over a numeric constant is
still a compile-time literal: no name lookup, call, or attribute access runs
at import, so the import-time-execution threat model is unaffected.
Tests:
- test_shadow_codegen_security.py: render a profile with RSI + negative
prior-return bounds through the real runner loader
(_load_module_from_file) and assert generate() fires entries — pins #985
end to end.
- test_backtest_runner_security.py: signed numeric literals are accepted at
top level, and @staticmethod/@classmethod/custom decorators are still
rejected, pinning the unchanged security posture.
Closes#985
Signed-off-by: shadowinlife <shadowinlife@gmail.com>
_MARKET_PATTERNS only matched US equities in the suffixed "XXX.US"
form, so bare tickers (AAPL, MSFT, ...) fell through to the a_share
default. The Shadow Account US candidate basket uses bare tickers, so
its backtest walked the A-share loader chain and raised
NoAvailableSourceError before reaching yfinance.
- Append a lowest-priority ^[A-Z]{1,5}$ catch-all to _MARKET_PATTERNS
so bare US tickers reach the us_equity chain; every suffixed equity /
futures / crypto / forex pattern above still wins first, and 6-char
bare forex (EURUSD) / longer crypto codes keep the historical default
- Normalize the Shadow Account US basket to the suffixed convention
(AAPL.US), in line with the china_a / hk / crypto baskets
- Add regression coverage: bare-ticker routing, priority guards,
boundary cases, code_currency, and a basket-to-market consistency
guard; align shadow scanner price-frame stubs with the new basket
Closes#986
Signed-off-by: shadowinlife <shadowinlife@gmail.com>
* fix(data): route market-data fallback by the symbol's market
get_market_data picked the fallback chain by source name alone, and
"yahoo" matched the us_equity chain first, so .HK symbols exhausted the
3-attempt budget on the US-only stooq/sina loaders and never reached
eastmoney/akshare — HK fetches failed outright where Yahoo is blocked
(#998).
- resolve the fallback chain from the symbol's market, keeping the old
first-match lookup for sources outside that market's chain
- group fetches by (source, market) so mixed-market requests each walk
their own chain
- hk_equity chain: akshare (Eastmoney-backed, never IP-banned) now
precedes the Yahoo-SDK family; tushare hk_daily joins after the free
sources (the tushare loader now declares hk_equity)
- default max_fallback_attempts 3 -> 5 so akshare is reachable within
the budget
Backtest auto routing already walked the full per-market chain and is
unchanged. tencent HK support is tracked in #999.
Signed-off-by: shadowinlife <shadowinlife@gmail.com>
* feat(data): tencent loader HK support, head of the hk_equity chain
Tencent's ifzq.gtimg.cn endpoint serves HK daily bars through the same
fqkline API as A-shares (hk00700 symbol form), and the source has no
observed IP ban — the most reliable mainland-reachable HK source (#999).
- map NNNNN.HK to hkNNNNN (zero-padded) and parse the shared fqkline
response; daily bars only, matching the loader's A-share contract
- declare hk_equity on the tencent loader
- lead the hk_equity chain with tencent and route .HK auto-detection
to it, consistent with the A-share routing
Signed-off-by: shadowinlife <shadowinlife@gmail.com>
---------
Signed-off-by: shadowinlife <shadowinlife@gmail.com>
run_gc(dry_run=True) promises to log actions without modifying files:
Tier-1 archive/delete honors the flag, but the Tier-2 compression pass
ran unconditionally whenever VT_MEMORY_COMPRESSION was enabled. The
pipeline rewrites entry bodies and frontmatter in place (TF-IDF
sentence extraction, compression_level update), so an audit-type
invocation mutated exactly the entries it was inspecting.
Gate the Tier-2 pass on not dry_run and add two regression tests:
dry_run=True leaves an eligible entry byte-identical, while
dry_run=False still compresses it from raw to daily.
Signed-off-by: shadowinlife <shadowinlife@gmail.com>
MetaTrader5 SDK copy_rates_from_pos() returns numpy structured arrays
containing numpy.int64/float64 scalars. These types are not natively
JSON-serializable, causing TypeError when trading_history responses
are serialized downstream.
Add _int_or_none() and _float_or_none() helpers in reads.py and apply
them to all numeric fields in _rate_to_dict(). Add regression tests
covering numpy, plain-dict, and None input scenarios.
Closes#774
Signed-off-by: shadowinlife <shadowinlife@gmail.com>
Add P@5/MRR/NDCG@5 evaluation framework for memory retrieval A/B testing.
Benchmark compares Baseline (pure token overlap) vs Treatment (BM25 + importance
decay) on a 200-entry corpus with 50 annotated queries.
The benchmark suite is NOT intended for CI — it requires corpus files
(tmp/benchmark_corpus/) that are not version-controlled. The suite
automatically skips when corpus files are absent.
Corpus is derived from anonymized production agent sessions.
Usage:
python -m pytest agent/tests/memory/benchmarks/ -v
Refs: #732
Signed-off-by: shadowinlife <shadowinlife@gmail.com>
Evaluation framework (benchmarks, eval tests) moved to separate
contribution scope. Community discussion will determine whether
these are wanted as part of the memory lifecycle work.
Refs: #732
Signed-off-by: shadowinlife <shadowinlife@gmail.com>
Benchmark corpus lives outside version control (tmp/benchmark_corpus/).
CI runners do not have these files. Skip the entire benchmark module at
collection time rather than failing individual tests.
Refs: #732
Signed-off-by: shadowinlife <shadowinlife@gmail.com>
These files are not part of the community contribution scope.
Benchmark results are kept locally for validation only.
Refs: #732
Signed-off-by: shadowinlife <shadowinlife@gmail.com>
shadowinlife