fix(grounding): keep recovery refs within figure kinds
This commit is contained in:
Haozhe Wu committed
1 parent
ecc02039c8
commit
c214c0356d
9 files changed
+216
-16
No files matched your search
@@ -52,7 +52,7 @@
|
||||
|
||||
> ⚠️ **Security warning:** The X account `VibeTrading_HKU`, Virtuals project `101845`, and token contract `0x640BDBF77b6447E8b7DB7894cED84BD1c40571f4` are not official Vibe-Trading assets. We have never launched or endorsed any token or memecoin. Do not buy, connect a wallet, or sign anything. [Details](SECURITY.md#official-channels--impersonation).
|
||||
|
||||
- **2026-10-05** 🛠️ **Research prompts and calculation fixes**: Chat accepts longer research prompts and gives a localized recovery message when input is too large ([#1701](https://github.com/HKUDS/Vibe-Trading/pull/1701)). Backtests use full-sample Sortino downside deviation; grouped validation purges overlapping labels, covariance weights remain finite, shadow RSI uses Wilder seeds, and memory removal accepts filename stems.
|
||||
- **2026-10-05** 🛠️ **Research prompts and calculation fixes**: Chat accepts longer research prompts and gives a localized recovery message when input is too large ([#1701](https://github.com/HKUDS/Vibe-Trading/pull/1701)). Backtests use full-sample Sortino downside deviation; grouped validation purges overlapping labels, covariance weights remain finite, shadow RSI uses Wilder seeds, and memory removal accepts filename stems. Invalid call aliases now receive exact source references for correction ([#1638](https://github.com/HKUDS/Vibe-Trading/pull/1638)); numeric validation stays unchanged.
|
||||
|
||||
- **2026-10-04** 🛠️ **Scheduled reports and research workflows**: Edit scheduled runs and choose a configured destination; Email reports support HTML or PDF attachments ([#1649](https://github.com/HKUDS/Vibe-Trading/pull/1649), [#1680](https://github.com/HKUDS/Vibe-Trading/pull/1680)).
|
||||
Backtests expose structured summaries and paged artifact reads ([#1646](https://github.com/HKUDS/Vibe-Trading/pull/1646), [#1647](https://github.com/HKUDS/Vibe-Trading/pull/1647)). Fixes cover memory-search snippets, export-path guidance, macro truncation, monthly risk, turnover on reversals and cash reentry, IV accuracy, VaR gaps, VCS updates and Robinhood option-order blocking.
|
||||
|
||||
+1
-1
@@ -52,7 +52,7 @@
|
||||
|
||||
> ⚠️ **تحذير أمني:** حساب X باسم `VibeTrading_HKU`، ومشروع Virtuals رقم `101845`، وعقد التوكن `0x640BDBF77b6447E8b7DB7894cED84BD1c40571f4` كلّها غير رسمية ولا تتبع Vibe-Trading. لم نُطلق أو نؤيد مطلقًا أي توكن أو عملة ميم. لا تشترِ هذا التوكن، ولا تربط محفظتك، ولا توقّع أي شيء. [التفاصيل](SECURITY.md#official-channels--impersonation).
|
||||
|
||||
- **2026-10-05** 🛠️ **إدخال البحث وتصحيح الحسابات**: تدعم المحادثة نصوص بحث أطول وتوضح باللغة الحالية كيفية تقصير النص عند تجاوز الحد ([#1701](https://github.com/HKUDS/Vibe-Trading/pull/1701)). يستخدم Sortino انحراف الجانب السلبي عبر جميع الفترات، ويستبعد التحقق الجماعي التسميات المتداخلة. تحسنت أوزان التباين وRSI بمتوسط Wilder الأولي وحذف الذاكرة باسم الملف.
|
||||
- **2026-10-05** 🛠️ **إدخال البحث وتصحيح الحسابات**: تدعم المحادثة نصوص بحث أطول وتوضح باللغة الحالية كيفية تقصير النص عند تجاوز الحد ([#1701](https://github.com/HKUDS/Vibe-Trading/pull/1701)). يستخدم Sortino انحراف الجانب السلبي عبر جميع الفترات، ويستبعد التحقق الجماعي التسميات المتداخلة. تحسنت أوزان التباين وRSI بمتوسط Wilder الأولي وحذف الذاكرة باسم الملف. عند استخدام اسم مستعار غير صحيح للاستدعاء، يقترح التصحيح مراجع الحقول الفعلية مع بقاء التحقق من الأرقام كما هو ([#1638](https://github.com/HKUDS/Vibe-Trading/pull/1638)).
|
||||
|
||||
- **2026-10-04** 🛠️ **تحسين التقارير المجدولة وسير البحث**: يمكن تعديل المهام المجدولة واختيار وجهة مُعدّة مسبقًا، مع إرسال تقارير البريد بصيغة HTML أو كمرفقات PDF ([#1649](https://github.com/HKUDS/Vibe-Trading/pull/1649), [#1680](https://github.com/HKUDS/Vibe-Trading/pull/1680)).
|
||||
تتضمن الاختبارات التاريخية ملخصات منظّمة وقراءة ملفات النتائج على صفحات ([#1646](https://github.com/HKUDS/Vibe-Trading/pull/1646), [#1647](https://github.com/HKUDS/Vibe-Trading/pull/1647)). تشمل الإصلاحات مقتطفات بحث الذاكرة، وإرشادات مسار التصدير، وبيان اقتطاع البيانات الكلية، ومخاطر الشموع الشهرية، ودوران المحفظة عند عكس الاتجاه أو إعادة الاستثمار بعد التحول إلى النقد، ودقة التقلب الضمني، وفجوات VaR، وتحديث التثبيت من VCS، وحظر أوامر خيارات Robinhood غير المدعومة.
|
||||
|
||||
+1
-1
@@ -52,7 +52,7 @@
|
||||
|
||||
> ⚠️ **Advertencia de seguridad:** la cuenta de X `VibeTrading_HKU`, el proyecto de Virtuals `101845` y el contrato de token `0x640BDBF77b6447E8b7DB7894cED84BD1c40571f4` no son activos oficiales de Vibe-Trading. Nunca hemos lanzado ni respaldado ningún token o memecoin. No compres, conectes una wallet ni firmes nada. [Detalles](SECURITY.md#official-channels--impersonation).
|
||||
|
||||
- **2026-10-05** 🛠️ **Consultas de investigación y cálculos corregidos**: El chat admite consultas más largas e indica en el idioma actual cómo acortarlas cuando superan el límite ([#1701](https://github.com/HKUDS/Vibe-Trading/pull/1701)). Sortino usa la desviación a la baja de todos los períodos; la validación por grupos excluye etiquetas solapadas. También mejoran los pesos de covarianza, el RSI con media inicial de Wilder y la eliminación de memorias por nombre de archivo.
|
||||
- **2026-10-05** 🛠️ **Consultas de investigación y cálculos corregidos**: El chat admite consultas más largas e indica en el idioma actual cómo acortarlas cuando superan el límite ([#1701](https://github.com/HKUDS/Vibe-Trading/pull/1701)). Sortino usa la desviación a la baja de todos los períodos; la validación por grupos excluye etiquetas solapadas. También mejoran los pesos de covarianza, el RSI con media inicial de Wilder y la eliminación de memorias por nombre de archivo. Los alias de llamada incorrectos reciben referencias exactas para corregirlos, conservando la validación numérica ([#1638](https://github.com/HKUDS/Vibe-Trading/pull/1638)).
|
||||
|
||||
- **2026-10-04** 🛠️ **Informes programados y flujos de investigación**: Edita tareas programadas y elige un destino configurado; los informes por email admiten HTML o archivos PDF adjuntos ([#1649](https://github.com/HKUDS/Vibe-Trading/pull/1649), [#1680](https://github.com/HKUDS/Vibe-Trading/pull/1680)).
|
||||
Los backtests ofrecen resúmenes estructurados y lectura paginada de resultados ([#1646](https://github.com/HKUDS/Vibe-Trading/pull/1646), [#1647](https://github.com/HKUDS/Vibe-Trading/pull/1647)). Se corrigen fragmentos de búsqueda, indicaciones de exportación, avisos de datos macro truncados, riesgo mensual, rotación al invertir posiciones o reinvertir tras pasar a efectivo, precisión de IV, huecos de VaR, actualizaciones desde VCS y el bloqueo de órdenes de opciones no admitidas en Robinhood.
|
||||
|
||||
+1
-1
@@ -53,7 +53,7 @@
|
||||
|
||||
> ⚠️ **Peringatan keamanan:** Akun X `VibeTrading_HKU`, proyek Virtuals `101845`, dan kontrak token `0x640BDBF77b6447E8b7DB7894cED84BD1c40571f4` bukan aset resmi Vibe-Trading. Kami tidak pernah meluncurkan atau mendukung token maupun memecoin apa pun. Jangan membeli, menghubungkan wallet, atau menandatangani apa pun. [Detail](SECURITY.md#official-channels--impersonation).
|
||||
|
||||
- **2026-10-05** 🛠️ **Input riset dan perbaikan perhitungan**: Chat menerima input riset lebih panjang dan memberi petunjuk dalam bahasa aktif untuk mempersingkat teks yang melampaui batas ([#1701](https://github.com/HKUDS/Vibe-Trading/pull/1701)). Sortino memakai deviasi penurunan dari seluruh periode; validasi grup membuang label yang tumpang tindih. Bobot kovarians, RSI dengan rata-rata awal Wilder, dan penghapusan memori lewat nama file juga diperbaiki.
|
||||
- **2026-10-05** 🛠️ **Input riset dan perbaikan perhitungan**: Chat menerima input riset lebih panjang dan memberi petunjuk dalam bahasa aktif untuk mempersingkat teks yang melampaui batas ([#1701](https://github.com/HKUDS/Vibe-Trading/pull/1701)). Sortino memakai deviasi penurunan dari seluruh periode; validasi grup membuang label yang tumpang tindih. Bobot kovarians, RSI dengan rata-rata awal Wilder, dan penghapusan memori lewat nama file juga diperbaiki. Alias panggilan yang keliru kini mendapat referensi sumber yang tepat untuk koreksi, dengan validasi angka tetap berlaku ([#1638](https://github.com/HKUDS/Vibe-Trading/pull/1638)).
|
||||
|
||||
- **2026-10-04** 🛠️ **Laporan terjadwal dan alur riset**: Edit tugas terjadwal dan pilih tujuan yang sudah dikonfigurasi; laporan email mendukung HTML atau lampiran PDF ([#1649](https://github.com/HKUDS/Vibe-Trading/pull/1649), [#1680](https://github.com/HKUDS/Vibe-Trading/pull/1680)).
|
||||
Backtest menyediakan ringkasan terstruktur dan pembacaan artefak per halaman ([#1646](https://github.com/HKUDS/Vibe-Trading/pull/1646), [#1647](https://github.com/HKUDS/Vibe-Trading/pull/1647)). Perbaikan mencakup cuplikan pencarian memori, panduan lokasi ekspor, pemberitahuan pemangkasan data makro, risiko bulanan, turnover saat membalik posisi atau masuk kembali setelah menjadi kas, akurasi IV, celah VaR, pembaruan instalasi VCS, serta pemblokiran order opsi Robinhood yang belum didukung.
|
||||
|
||||
+1
-1
@@ -52,7 +52,7 @@
|
||||
|
||||
> ⚠️ **セキュリティ警告：** Xアカウント `VibeTrading_HKU`、Virtualsプロジェクト `101845`、およびトークンコントラクト `0x640BDBF77b6447E8b7DB7894cED84BD1c40571f4` は、いずれもVibe-Trading公式のものではありません。Vibe-Tradingはこれまで、いかなるトークンやミームコインも発行・公認していません。購入、ウォレットの接続、署名は行わないでください。[詳細](SECURITY.md#official-channels--impersonation)
|
||||
|
||||
- **2026-10-05** 🛠️ **調査入力と計算の修正**：チャットで長い調査入力に対応し、上限を超えた場合は現在の言語で短縮を案内します（[#1701](https://github.com/HKUDS/Vibe-Trading/pull/1701)）。Sortino は全期間の下方偏差を使用し、グループ検証は重複ラベルを除外します。共分散ウェイト、Wilder 初期平均による RSI、ファイル名によるメモリ削除も改善しました。
|
||||
- **2026-10-05** 🛠️ **調査入力と計算の修正**：チャットで長い調査入力に対応し、上限を超えた場合は現在の言語で短縮を案内します（[#1701](https://github.com/HKUDS/Vibe-Trading/pull/1701)）。Sortino は全期間の下方偏差を使用し、グループ検証は重複ラベルを除外します。共分散ウェイト、Wilder 初期平均による RSI、ファイル名によるメモリ削除も改善しました。 呼び出しの別名を誤って引用した場合、修正用に実際のフィールド参照を提示します（[#1638](https://github.com/HKUDS/Vibe-Trading/pull/1638)）。数値の検証規則は維持されます。
|
||||
|
||||
- **2026-10-04** 🛠️ **定期レポートと調査フローの改善**：定期タスクを編集し、設定済みの送信先を選択できます。メールレポートは HTML または PDF 添付に対応しました（[#1649](https://github.com/HKUDS/Vibe-Trading/pull/1649), [#1680](https://github.com/HKUDS/Vibe-Trading/pull/1680)）。
|
||||
バックテストに構造化サマリーと成果物のページ読み取りを追加（[#1646](https://github.com/HKUDS/Vibe-Trading/pull/1646), [#1647](https://github.com/HKUDS/Vibe-Trading/pull/1647)）。メモリ検索の抜粋、エクスポート先の案内、マクロデータの切り詰め表示、月足リスク、売買方向の反転・現金化後の再投資の回転率、IV 精度、VaR の欠損区間、VCS インストールの更新、Robinhood の未対応オプション注文の遮断も修正しました。
|
||||
|
||||
+1
-1
@@ -52,7 +52,7 @@
|
||||
|
||||
> ⚠️ **보안 경고:** X 계정 `VibeTrading_HKU`, Virtuals 프로젝트 `101845`, 토큰 컨트랙트 `0x640BDBF77b6447E8b7DB7894cED84BD1c40571f4`는 모두 Vibe-Trading 공식과 무관합니다. Vibe-Trading은 어떠한 토큰이나 밈코인도 발행하거나 공식적으로 지지한 적이 없습니다. 해당 토큰을 구매하거나 지갑을 연결하거나 어떠한 서명도 하지 마세요. [자세히 보기](SECURITY.md#official-channels--impersonation).
|
||||
|
||||
- **2026-10-05** 🛠️ **리서치 입력과 계산 수정**: 더 긴 리서치 입력을 지원하며 한도를 넘으면 현재 언어로 줄이도록 안내합니다([#1701](https://github.com/HKUDS/Vibe-Trading/pull/1701)). Sortino는 전체 기간의 하방 편차를 사용하고 그룹 검증은 겹치는 레이블을 제외합니다. 공분산 가중치, Wilder 초기 평균을 사용하는 RSI, 파일명으로 메모리 삭제도 개선했습니다.
|
||||
- **2026-10-05** 🛠️ **리서치 입력과 계산 수정**: 더 긴 리서치 입력을 지원하며 한도를 넘으면 현재 언어로 줄이도록 안내합니다([#1701](https://github.com/HKUDS/Vibe-Trading/pull/1701)). Sortino는 전체 기간의 하방 편차를 사용하고 그룹 검증은 겹치는 레이블을 제외합니다. 공분산 가중치, Wilder 초기 평균을 사용하는 RSI, 파일명으로 메모리 삭제도 개선했습니다. 잘못된 호출 별칭을 인용하면 수정할 수 있도록 실제 필드 참조를 안내하며 수치 검증 규칙은 유지합니다([#1638](https://github.com/HKUDS/Vibe-Trading/pull/1638)).
|
||||
|
||||
- **2026-10-04** 🛠️ **예약 보고서와 리서치 흐름 개선**: 예약 작업을 편집하고 설정된 수신 대상을 선택할 수 있습니다. 이메일 보고서는 HTML 또는 PDF 첨부를 지원합니다([#1649](https://github.com/HKUDS/Vibe-Trading/pull/1649), [#1680](https://github.com/HKUDS/Vibe-Trading/pull/1680)).
|
||||
백테스트에 구조화된 요약과 결과 파일 페이지 읽기를 추가했습니다([#1646](https://github.com/HKUDS/Vibe-Trading/pull/1646), [#1647](https://github.com/HKUDS/Vibe-Trading/pull/1647)). 메모리 검색 미리보기, 내보내기 경로 안내, 매크로 데이터 잘림 표시, 월봉 리스크, 포지션 반전과 현금화 후 재진입의 회전율, IV 정확도, VaR 결측 구간, VCS 설치 업데이트, Robinhood의 미지원 옵션 주문 차단도 수정했습니다.
|
||||
|
||||
+1
-1
@@ -52,7 +52,7 @@
|
||||
|
||||
> ⚠️ **安全警告：** X 账号 `VibeTrading_HKU`、Virtuals 项目 `101845` 及代币合约 `0x640BDBF77b6447E8b7DB7894cED84BD1c40571f4` 均非 Vibe-Trading 官方。我们从未发行或背书任何代币或 meme 币。请勿购买、连接钱包或签名。[详细说明](SECURITY.md#official-channels--impersonation)。
|
||||
|
||||
- **2026-10-05** 🛠️ **研究输入与计算修复**：聊天支持更长的研究输入，超限时以当前语言提示缩短（[#1701](https://github.com/HKUDS/Vibe-Trading/pull/1701)）。回测 Sortino 使用全样本下行偏差；分组验证剔除重叠标签，协方差权重保持有限，影子账户 RSI 使用 Wilder 初始均值，记忆支持按文件名删除。
|
||||
- **2026-10-05** 🛠️ **研究输入与计算修复**：聊天支持更长的研究输入，超限时以当前语言提示缩短（[#1701](https://github.com/HKUDS/Vibe-Trading/pull/1701)）。回测 Sortino 使用全样本下行偏差；分组验证剔除重叠标签，协方差权重保持有限，影子账户 RSI 使用 Wilder 初始均值，记忆支持按文件名删除。 报告误用调用别名时，纠正提示提供真实字段引用（[#1638](https://github.com/HKUDS/Vibe-Trading/pull/1638)），数值校验保持不变。
|
||||
|
||||
- **2026-10-04** 🛠️ **定时报告与研究流程完善**：定时任务支持编辑并选择已配置的交付目标，邮件报告可选 HTML 或 PDF 附件（[#1649](https://github.com/HKUDS/Vibe-Trading/pull/1649), [#1680](https://github.com/HKUDS/Vibe-Trading/pull/1680)）。
|
||||
回测提供结构化摘要与产物分页读取（[#1646](https://github.com/HKUDS/Vibe-Trading/pull/1646), [#1647](https://github.com/HKUDS/Vibe-Trading/pull/1647)）；同时修复记忆搜索片段、导出路径提示、宏观数据截断说明、月线风险、反向持仓及清仓后重新入场的换手率、隐含波动率精度、VaR 缺口、源码安装更新，以及 Robinhood 不受支持的期权下单拦截。
|
||||
|
||||
@@ -99,9 +99,6 @@ _OTHER_CURRENCY_PREFIXES = "港美日欧韩台新加澳"
|
||||
#: Relative band a value must fall in to count as matching evidence.
|
||||
_TOLERANCE = 0.005
|
||||
|
||||
#: Most exact refs a correction lists for a ref whose call id names nothing.
|
||||
_MAX_FIELD_REF_CANDIDATES = 5
|
||||
|
||||
#: A plain integer is read as a price only for an instrument quoted in the
|
||||
#: thousands (600519.SH, an index, BTC). Below that, a prose integer is a window,
|
||||
#: a horizon or a count ("20 日均线", "200-day") and stays unchecked.
|
||||
@@ -1242,8 +1239,9 @@ class _PolicyMixin:
|
||||
no evidence. This only lists where the field really lives, so the next
|
||||
draft can copy an exact ref; it grants nothing, and the figure stays
|
||||
rejected until it is re-declared with one of them. Refs whose value the
|
||||
figure matches come first; at most :data:`_MAX_FIELD_REF_CANDIDATES` are
|
||||
returned.
|
||||
figure matches come first; at most :data:`_MAX_CALL_REF_CANDIDATES` are
|
||||
returned. Candidates keep the same symbol and kind restrictions as a
|
||||
real field ref, so the correction cannot recommend an unusable ref.
|
||||
"""
|
||||
key = (ref or "").strip()
|
||||
if "::" not in key:
|
||||
@@ -1253,22 +1251,22 @@ class _PolicyMixin:
|
||||
return []
|
||||
records, entries = self._field_sources(field, symbol)
|
||||
found: dict[str, list[float]] = {}
|
||||
money = bool(figure.currency and not figure.percent)
|
||||
for record in records:
|
||||
if record.call_id and record.field:
|
||||
if record.call_id and record.field and self._kind_fits(record, figure):
|
||||
label = self._ref_source(record.call_id, record.field, record.scope)[1]
|
||||
found.setdefault(label, []).append(float(record.value))
|
||||
for entry in entries:
|
||||
for entry in entries if not (money or figure.column) else ():
|
||||
if entry.get("call_id") and entry.get("field"):
|
||||
label = self._ref_source(str(entry["call_id"]), str(entry["field"]), None)[1]
|
||||
found.setdefault(label, []).append(float(entry["value"]))
|
||||
money = bool(figure.currency and not figure.percent)
|
||||
compatible = {
|
||||
label
|
||||
for label, values in found.items()
|
||||
if self._matches_evidence(figure, values, [] if money else values)
|
||||
}
|
||||
ranked = sorted(found, key=lambda label: (label not in compatible, label))
|
||||
return ranked[:_MAX_FIELD_REF_CANDIDATES]
|
||||
return ranked[:_MAX_CALL_REF_CANDIDATES]
|
||||
|
||||
def _names_session_source(self, name: str) -> bool:
|
||||
"""Whether ``name`` is a call id, tool name or backtest run of this session."""
|
||||
|
||||
@@ -9,6 +9,10 @@ import pytest
|
||||
|
||||
from src.agent.context import _SYSTEM_PROMPT
|
||||
from src.agent.grounding import GroundingLedger
|
||||
from src.agent.loop import AgentLoop
|
||||
from src.agent.tools import BaseTool, ToolRegistry
|
||||
from src.agent.trace import TraceWriter
|
||||
from src.providers.chat import LLMResponse, ToolCallRequest
|
||||
|
||||
pytestmark = pytest.mark.unit
|
||||
|
||||
@@ -110,3 +114,201 @@ def test_prompt_example_is_a_placeholder_not_a_call_id_shape() -> None:
|
||||
assert "q1::" not in _SYSTEM_PROMPT
|
||||
assert "<call_id>::historical_var" in _SYSTEM_PROMPT
|
||||
assert "never invent a short alias" in _SYSTEM_PROMPT
|
||||
|
||||
|
||||
@pytest.mark.parametrize("written", ["12.6", "12.55", "15.0"])
|
||||
def test_copying_a_real_ref_does_not_authorize_an_invented_value(
|
||||
tmp_path: Path, written: str
|
||||
) -> None:
|
||||
ledger = _ledger(tmp_path, {"call_9f3a": 12.5})
|
||||
rejected = _check(ledger, f"r1::{PATH}", written)
|
||||
assert not rejected.valid
|
||||
ref = rejected.issues[0]["field_ref_candidates"][0]
|
||||
|
||||
assert not _check(ledger, ref, written).valid
|
||||
assert _check(ledger, ref).valid
|
||||
|
||||
|
||||
@pytest.mark.parametrize("written", ["12.5%", "0.125"])
|
||||
def test_corrected_percent_or_decimal_preserves_existing_unit_matching(
|
||||
tmp_path: Path, written: str
|
||||
) -> None:
|
||||
ledger = _ledger(tmp_path, {"call_9f3a": 0.125})
|
||||
rejected = _check(ledger, f"r1::{PATH}", written)
|
||||
assert not rejected.valid
|
||||
ref = rejected.issues[0]["field_ref_candidates"][0]
|
||||
|
||||
assert _check(ledger, ref, written).valid
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
("field", "written"),
|
||||
[
|
||||
("total_return", "$12.5"),
|
||||
("close", "12.5%"),
|
||||
("return_observations", "12.5%"),
|
||||
],
|
||||
)
|
||||
def test_candidates_do_not_offer_a_field_of_the_wrong_kind(
|
||||
tmp_path: Path, field: str, written: str
|
||||
) -> None:
|
||||
ledger = GroundingLedger(run_dir=tmp_path, user_message="Report the value")
|
||||
ledger.ingest_tool_result(
|
||||
tool_name="report_tool",
|
||||
arguments={},
|
||||
result=json.dumps({"data": {field: 12.5}}),
|
||||
call_id="call_9f3a",
|
||||
success=True,
|
||||
)
|
||||
result = _check(ledger, f"r1::data.{field}", written)
|
||||
|
||||
assert not result.valid
|
||||
assert not any(issue.get("field_ref_candidates") for issue in result.issues)
|
||||
assert not _check(ledger, f"call_9f3a::data.{field}", written).valid
|
||||
|
||||
|
||||
def test_candidates_remain_scoped_to_the_declared_symbol(tmp_path: Path) -> None:
|
||||
ledger = GroundingLedger(run_dir=tmp_path, user_message="Report summary metrics")
|
||||
for call_id, symbol, value in (
|
||||
("call_aapl", "AAPL.US", 12.5),
|
||||
("call_msft", "MSFT.US", 12.5),
|
||||
):
|
||||
ledger.ingest_tool_result(
|
||||
tool_name="report_tool",
|
||||
arguments={"symbol": symbol},
|
||||
result=json.dumps({"symbol": symbol, "data": {"summary": {"total_return": value}}}),
|
||||
call_id=call_id,
|
||||
success=True,
|
||||
)
|
||||
result = ledger.validate_final_answer(
|
||||
f"Total return 12.5.\n```figures\n12.5 | observed | AAPL.US total return | r1::{PATH}\n```"
|
||||
)
|
||||
|
||||
assert not result.valid
|
||||
issue = next(issue for issue in result.issues if issue.get("reason") == "unknown_call_id")
|
||||
assert issue["field_ref_candidates"] == [f"call_aapl::{PATH}"]
|
||||
|
||||
|
||||
@pytest.mark.parametrize("path", ["data.positions[0].total_return", "positions.0.total_return"])
|
||||
def test_alias_with_an_explicit_index_never_offers_a_sibling(
|
||||
tmp_path: Path, path: str
|
||||
) -> None:
|
||||
ledger = GroundingLedger(run_dir=tmp_path, user_message="Report the values")
|
||||
ledger.ingest_tool_result(
|
||||
tool_name="report_tool",
|
||||
arguments={},
|
||||
result=json.dumps({"data": {"positions": [{"total_return": 3.0}, {"total_return": 12.5}]}}),
|
||||
call_id="call_9f3a",
|
||||
success=True,
|
||||
)
|
||||
rejected = _check(ledger, f"r1::{path}")
|
||||
assert not rejected.valid
|
||||
candidates = rejected.issues[0]["field_ref_candidates"]
|
||||
assert candidates == ["call_9f3a::data.positions[0].total_return"]
|
||||
assert not _check(ledger, candidates[0]).valid
|
||||
assert _check(ledger, candidates[0], "3.0").valid
|
||||
|
||||
|
||||
def test_a_known_tool_keeps_its_existing_correction_reason(tmp_path: Path) -> None:
|
||||
ledger = _ledger(tmp_path, {"call_9f3a": 12.5})
|
||||
result = _check(ledger, f"report_tool::{PATH}")
|
||||
|
||||
assert not result.valid
|
||||
assert result.issues[0]["reason"] == "field_ref_needs_call_id"
|
||||
assert result.issues[0]["field_ref_candidates"] == [f"call_9f3a::{PATH}"]
|
||||
|
||||
|
||||
@pytest.mark.parametrize("ref", [f"::{PATH}", "r1::", "r1::data.summary"])
|
||||
def test_incomplete_or_container_refs_get_no_scalar_hint(tmp_path: Path, ref: str) -> None:
|
||||
result = _check(_ledger(tmp_path, {"call_9f3a": 12.5}), ref)
|
||||
|
||||
assert not result.valid
|
||||
assert not any(issue.get("field_ref_candidates") for issue in result.issues)
|
||||
|
||||
|
||||
@pytest.mark.parametrize("written", ["12.5%", "$0.125"])
|
||||
def test_analysis_only_candidates_preserve_the_figure_kind(tmp_path: Path, written: str) -> None:
|
||||
ledger = GroundingLedger(run_dir=tmp_path, user_message="Report the tail risk")
|
||||
ledger.ingest_tool_result(
|
||||
tool_name="quantlib_call",
|
||||
arguments={"action": "call", "function": "historical_var"},
|
||||
result=json.dumps({"ok": True, "result": 0.125}),
|
||||
call_id="call_9f3a",
|
||||
success=True,
|
||||
)
|
||||
result = _check(ledger, "r1::historical_var", written)
|
||||
|
||||
assert not result.valid
|
||||
if written.endswith("%"):
|
||||
ref = result.issues[0]["field_ref_candidates"][0]
|
||||
assert ref == "call_9f3a::historical_var"
|
||||
assert _check(ledger, ref, written).valid
|
||||
else:
|
||||
assert not any(issue.get("field_ref_candidates") for issue in result.issues)
|
||||
|
||||
|
||||
class _ReportTool(BaseTool):
|
||||
name = "report_tool"
|
||||
description = "Return summary metrics."
|
||||
parameters = {"type": "object", "properties": {}}
|
||||
|
||||
def __init__(self, result: str) -> None:
|
||||
self.result = result
|
||||
|
||||
def execute(self, **_kwargs) -> str:
|
||||
return self.result
|
||||
|
||||
|
||||
def test_agent_loop_sends_the_hint_and_releases_the_corrected_answer(tmp_path: Path) -> None:
|
||||
"""Replay the provider response locally through the actual tool and release path."""
|
||||
exact_ref = f"call_9f3a::{PATH}"
|
||||
|
||||
def draft(ref: str) -> str:
|
||||
return f"Total return 12.5.\n```figures\n12.5 | observed | total return | {ref}\n```"
|
||||
|
||||
class CopyingLLM:
|
||||
calls = 0
|
||||
|
||||
def stream_chat(self, messages, on_text_chunk=None, **_kwargs):
|
||||
self.calls += 1
|
||||
if self.calls == 1:
|
||||
return LLMResponse(tool_calls=[ToolCallRequest("call_9f3a", "report_tool", {})])
|
||||
if self.calls == 2:
|
||||
content = draft(f"r1::{PATH}")
|
||||
else:
|
||||
assert self.calls == 3
|
||||
correction = messages[-1]["content"]
|
||||
assert exact_ref in correction
|
||||
assert "not an alias" in correction
|
||||
content = draft(exact_ref)
|
||||
if on_text_chunk:
|
||||
on_text_chunk(content)
|
||||
return LLMResponse(content=content)
|
||||
|
||||
def chat(self, _messages, **_kwargs):
|
||||
return LLMResponse(content="")
|
||||
|
||||
registry = ToolRegistry()
|
||||
registry.register(_ReportTool(json.dumps({"data": {"summary": {"total_return": 12.5}}})))
|
||||
llm = CopyingLLM()
|
||||
events = []
|
||||
agent = AgentLoop(
|
||||
registry=registry,
|
||||
llm=llm,
|
||||
max_iterations=5,
|
||||
event_callback=lambda event, data: events.append((event, data)),
|
||||
)
|
||||
run_dir = tmp_path / "run"
|
||||
run_dir.mkdir()
|
||||
agent.memory.run_dir = str(run_dir)
|
||||
result = agent.run("Report the total return")
|
||||
|
||||
assert result["status"] == "success"
|
||||
assert not result.get("degraded")
|
||||
assert result["content"] == "Total return 12.5."
|
||||
assert llm.calls == 3
|
||||
rejected = [event for event in TraceWriter.read(run_dir) if event.get("type") == "answer_rejected"]
|
||||
assert len(rejected) == 1
|
||||
assert rejected[0]["issues"][0]["reason"] == "unknown_call_id"
|
||||
assert rejected[0]["issues"][0]["field_ref_candidates"] == [exact_ref]
|
||||
assert any(event == "grounding_status" and data["stage"] == "revising" for event, data in events)
|
||||
Reference in new issue
Block a user