Pure-CJK (and generally non-ASCII) memories are silently never indexed → unretrievable by any search

Summary
When a memory's content contains no ASCII \w character ([A-Za-z0-9_]), the row is written to the DB but never becomes searchable — no error, no warning, no log line. Affects ~11% of our store, and 100% of pure-CJK content.

Confirmed impact on a real store (4845 memories, zh-centric)

Every single memory whose text is entirely non-ASCII is invisible to search — not "low similarity", simply absent from the candidate set.

Workaround we validated (and what does not work)

• ✅ append an ASCII anchor to the content and re-insert (POST /v4/memories): 253/254 recovered.
  • Works: _, 7, AB, 2026_10_01, _2026-10-01, 20261001, [7]
  • Does not work: 2026-10-01, [2026-10-01], （记录于 2026-10-01）, 2026年10月1日
  • Practical rule: the appended token needs an underscore or a ≥5-char contiguous ASCII run; isolated digits embedded in CJK don't count.
• ❌ PATCH /v4/memories with newContent does NOT re-embed — the store keeps the old vector, so patching the text changes what is displayed but not what is searchable. Repair requires insert + delete, not patch.
• ❌ Changing embedding provider/model does not help (see below).

Two smaller API issues found while debugging

1. POST /v4/search with limit > 100 returns 0 results (silently) instead of clamping or erroring. limit=100 → 100, limit=200 → 0. This silently broke our audit tooling.
2. POST /v4/search uses q as the query parameter; passing query yields a 400 with no hint about the correct name.

• Version: reproduced on 0.0.7 and 0.0.8 (self-hosted single-file binary, linux-x64)
• Deployment: Docker, --network host, local storage, embedded local embeddings
• Embedding model: Xenova/bge-small-zh-v1.5 (512d) — but see "not model-specific" below

Reproduce

# 1) write a pure-CJK memory
POST /v4/memories
{"containerTag":"t","memories":[{"content":"超哥是程序员。"}]}

# 2) write the same sentence with ONE ascii char appended
POST /v4/memories
{"containerTag":"t","memories":[{"content":"超哥是程序员。7"}]}

Wait for ingestion to finish, then query each with its own exact content:

POST /v4/search
{"q":"超哥是程序员。","containerTag":"t","limit":100,"threshold":0}

POST /v4/search
{"q":"超哥是程序员。7","containerTag":"t","limit":100,"threshold":0}

Expected: both are returned, ideally rank 1 with similarity ≈ 1.0.
Actual: the ASCII-containing one is returned (sim 0.99+). The pure-CJK one is never returned — not at rank 1, not anywhere in the top 100, and not for any of 28 other queries we tried, including queries whose keywords appear literally in the text (e.g. searching 台灯, 桂花树, 岳母, 廖丹 never surfaces the memory containing those exact words).

Verified on a brand-new empty store as well as on a migrated store, so it is not state corruption.

Scale in our store

Ran a self-retrieval audit (q = the memory's own exact text, limit:100, threshold:0, hit = exact content present in results):

So the control methodology is sound, and the failure is specific to non-ASCII content, not to the audit.

What we ruled out

1. Tokenizer is fine. Loaded the model's own tokenizer: pure Chinese tokenizes correctly (18 tokens for an 18-char sentence), zero [UNK], full vocab coverage.
2. Query side is fine. Pure-CJK queries work — they retrieve ASCII-containing memories, and 6 mutually unrelated pure-CJK queries return largely disjoint result sets (mean pairwise Jaccard 0.029), i.e. no query-embedding collapse.
3. The embedding IS computed. With SUPERMEMORY_VERBOSE=1 the ingest log shows Embedded 1 text(s) for pure-CJK input. The vector exists — it just never becomes searchable.
4. Not specific to the local ONNX path. We rebuilt against SUPERMEMORY_EMBEDDING_PROVIDER=openai-compatible pointing at a custom deterministic embedding endpoint, bypassing ONNX entirely — the exact same failure reproduces (pure non-ASCII content unfindable, ASCII-containing content findable, in the same batch, through the same endpoint).
5. Not a search-mode or threshold issue. searchMode = memories / hybrid / documents, threshold:0, and ingesting via POST /v3/documents (status done) all fail to surface pure-CJK content.
6. Not a limit artifact. Re-audited with limit:100 (within the working range, see below) — same result.

The discriminating factor is exactly \w

Same sentence, same batch, same write call; only the appended character differs:

. and - are ASCII punctuation and still fail, while _ passes — that is the fingerprint of a JavaScript \w ([A-Za-z0-9_]) test used as an index-admission guard.

Suspected cause: an index/BM25 admission (or content-cleaning) step that drops rows whose text has no \w match, so the row never enters the searchable index. We could not locate the exact line in the bundled binary, but behaviour is 10/10 consistent with \w semantics.

Suggested fix: make the guard Unicode-aware (e.g. \p{L}\p{N} with the u flag, or drop the guard), and add a test covering pure-CJK / pure-Cyrillic / pure-Greek content.

Two smaller issues found while debugging

(a) POST /v4/search with limit > 100 returns 0 results instead of clamping or erroring.

No error, HTTP 200, empty results. This silently broke our audit tooling and produced a bogus "everything is missing" report before we caught it. Should either clamp to the max or return a 400.

(b) POST /v4/search requires q, not query. Passing query returns 400 invalid_type ... path:["q"] while every other endpoint in the OpenAPI spec uses query. Also POST /v4/memories/list requires containerTags (plural) while POST /v4/memories requires containerTag (singular) — easy to trip on.

Impact note

For anyone storing non-English memory (Chinese, Japanese, Korean, Cyrillic, Greek, Arabic…), a meaningful share of entries become silently unsearchable with no signal at write time. In our case that was 512 memories, including core personal facts, discovered only by an audit months later.