The previous CJK tokenizer ranges (#87, #95) only matched ``一-鿿``
and ``㐀-䶿``, so memory entries with Thai, Arabic, Hebrew, or
Cyrillic titles:
- Tokenized to the empty set, making recall always miss (e.g.
``find_relevant("ถัวเฉลี่ย")`` returned nothing even when the body
contained the word).
- Had their slug characters stripped to ``_``, so two distinct Thai
titles of equal length silently overwrote each other on disk.
The new ``_NON_LATIN_SCRIPT_RANGES`` constant covers CJK + Thai +
Arabic + Hebrew + Cyrillic and is reused by:
- ``_TOKEN_RE`` — single alternation pattern, one ``re.findall`` per
``_tokenize`` call (one text scan instead of two; precompiled at
module level so it doesn't go through ``re.compile`` cache lookup on
each invocation).
- ``_SLUG_DISALLOWED_RE`` — negation pattern used by ``add()``.
Arabic and Hebrew are deliberately narrowed to the basic letter blocks
(U+0620-U+064A, U+05D0-U+05EA) to keep bidi-control codepoints like
U+061C ARABIC LETTER MARK and combining marks out of on-disk slugs,
where they would render as invisible-but-distinct filenames.
Tests cover tokenization for each script, slug preservation
(parametrized across the four new scripts), and a Thai collision-
distinction regression.
Out of scope: ``agent/src/session/search.py`` has the same CJK-only
range in its FTS sanitizer; worth a follow-up PR to consume the same
constant.
Teerapat Vatpitak
02a16a6422