Skip to content

fix(qdrant): recover vectors when payload id is missing - #3362

Merged
danielaskdd merged 4 commits into
HKUDS:mainfrom
VectorPeak:qdrant-vector-id-fallback
Jul 17, 2026
Merged

danielaskdd merged 4 commits into
HKUDS:mainfrom
VectorPeak:qdrant-vector-id-fallback

Conversation

@VectorPeak

@VectorPeak VectorPeak commented Jul 6, 2026 •

Copy link
Copy Markdown
Contributor

Description

This fixes a Qdrant-only read-path gap in QdrantVectorDBStorage.get_vectors_by_ids() where a vector could be retrieved from Qdrant but still omitted from LightRAG's returned mapping when the point payload did not contain LightRAG's logical id field.

What Problem This Solves

LightRAG normally writes Qdrant points with two related identifiers:

point.id       = compute_mdhash_id_for_qdrant(requested_id, prefix=effective_workspace)
payload["id"]  = requested_id

get_vectors_by_ids() retrieves persisted vectors by computing the Qdrant point ID for each requested LightRAG ID. Before this change, a retrieved point was only added to the result map when its payload contained LightRAG's logical id field.

That left a narrow compatibility gap for legacy-style or externally migrated Qdrant points: the point could still be found by its computed Qdrant point ID and still contain vector data, but LightRAG would drop it because payload["id"] was absent. In other words, the storage layer could successfully retrieve the physical Qdrant point while failing to recover the caller-facing LightRAG chunk ID.

The sibling get_by_ids() path already handles this shape by falling back from the computed Qdrant point ID to the originally requested LightRAG ID. get_vectors_by_ids() did not have the same fallback, so an otherwise retrievable vector could be treated as missing by LightRAG.

Changes

This PR keeps the normal payload-ID path unchanged and adds a narrow read-time fallback for the missing-payload-ID case:

  • build a qdrant_id -> requested_id map for the IDs being retrieved from Qdrant;
  • prefer payload["id"] when it exists, preserving existing behavior for normal points;
  • when payload["id"] is absent, normalize the returned Qdrant point.id and recover the result key only if that point ID matches one of the IDs requested in the current lookup;
  • return the recovered vector under the original caller-facing LightRAG ID instead of dropping it.

The fallback is intentionally limited to points returned by the current retrieve() call. It does not scan Qdrant, infer IDs from arbitrary payloads, mutate existing payloads, or broaden workspace visibility.

This PR does not change Qdrant writes, point ID generation, payload schema, collection naming, workspace filtering, migration behavior, delete/drop/finalize semantics, ordinary query() vector search, or any non-Qdrant vector backend.

Evidence

The regression test models a legacy-style Qdrant point that can be retrieved by the computed Qdrant point ID but lacks the logical payload ID:

qdrant_id = compute_mdhash_id_for_qdrant("chunk-1", prefix=s.effective_workspace)
s._client.retrieve.return_value = [
    SimpleNamespace(
        id=qdrant_id,
        payload={"content": "legacy chunk"},
        vector=[0.1, 0.2, 0.3],
    )
]

Before this fix, get_vectors_by_ids(["chunk-1"]) skipped that point because payload["id"] was absent and returned {}. After this fix, the same lookup returns the vector under the requested LightRAG ID:

{"chunk-1": [0.1, 0.2, 0.3]}

The tests also cover Qdrant returning the same computed point ID in canonical hyphenated UUID form, so the fallback works when qdrant-client returns point.id differently from LightRAG's simple hex computed ID. The sibling get_by_ids(["chunk-1"]) fallback path is covered as well, keeping payload lookup and vector lookup consistent for this missing-payload-ID shape.

Focused validation passed locally:

uv run --extra test --extra offline-storage pytest tests/kg/qdrant_impl/test_qdrant_deferred_embedding.py -q
uv run --extra test --extra offline-storage pytest tests/kg/qdrant_impl/test_qdrant_migration.py -q
uv run pre-commit run --files lightrag/kg/qdrant_impl.py tests/kg/qdrant_impl/test_qdrant_deferred_embedding.py
git diff --check

Remote PR checks are also green for this branch.

Possible call chain / impact

query / retrieval path
  -> lightrag.utils.pick_by_vector_similarity(...)
  -> chunks_vdb.get_vectors_by_ids(all_chunk_ids)
  -> QdrantVectorDBStorage.get_vectors_by_ids(...)
  -> Qdrant retrieve(ids=[hash(workspace + chunk_id)], with_vectors=True, with_payload=True)
  -> retrieved point has vector data but no payload["id"]
  -> previously: the vector was omitted from the returned mapping
  -> now: point.id is mapped back to the requested chunk ID for this lookup only

The practical impact is limited to Qdrant points where the computed point ID still matches the requested LightRAG ID but the logical payload ID is absent. In that case, downstream vector-similarity chunk selection can now receive a vector that Qdrant already returned instead of silently losing it during result mapping.

This is a read-time compatibility fix, not a data migration. It does not repair points whose Qdrant point.id does not match the computed ID for the requested LightRAG ID, and it does not recover points that have no vector data.

Related Issues

N/A

Changes Made

  • Build a qdrant_id -> requested_id map for the IDs requested from Qdrant.
  • Preserve the existing payload["id"] behavior when that field is present.
  • Normalize returned Qdrant point IDs before fallback matching, including canonical hyphenated UUIDs returned by qdrant-client.
  • Fall back to the requested ID only when the retrieved point has a vector but its payload is missing id.
  • Add a regression test covering a legacy-style Qdrant point with payload={"content": ...} and a valid computed point ID.

Checklist

  • Changes tested locally
  • Code reviewed
  • Documentation updated (if necessary)
  • Unit tests added (if applicable)

Additional Notes

No additional notes beyond the scoped read-time fallback, evidence, call-chain impact, and validation described in ## Description.

@danielaskdd

Copy link
Copy Markdown
Collaborator

@codex review

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 6a964c215b

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread lightrag/kg/qdrant_impl.py Outdated
VectorPeak and others added 2 commits July 6, 2026 21:42
Normalize Qdrant point IDs before using them for the missing-payload-id
fallback in get_vectors_by_ids(). This covers Qdrant clients returning
UUID point IDs in canonical hyphenated form while LightRAG's computed IDs
use the simple hex representation.

Co-authored-by: chatgpt-codex-connector[bot] <199175422+chatgpt-codex-connector[bot]@users.noreply.github.com>
Normalize Qdrant point IDs in get_by_ids() fallback mapping as well, so
canonical hyphenated UUIDs returned by Qdrant match LightRAG's simple hex
computed IDs consistently across both sibling lookup paths.

Co-authored-by: chatgpt-codex-connector[bot] <199175422+chatgpt-codex-connector[bot]@users.noreply.github.com>
@VectorPeak

Copy link
Copy Markdown
Contributor Author

@danielaskdd 已根据 Codex review 的建议修改完毕:

  • 已规范化 Qdrant 返回的 point.id,让 qdrant-client 返回 canonical hyphenated UUID 时也能匹配 LightRAG 的 simple hex computed ID。
  • 已同时覆盖 get_vectors_by_ids() 和 sibling get_by_ids() 的 fallback 路径,避免两个读取路径行为不一致。
  • 已补充对应的 hyphenated UUID 回归测试。
  • 本地验证通过:Qdrant deferred embedding tests、migration tests、pre-commit、git diff --check。
  • 远端 CI 也已经全部通过。

相关修改已推送到当前 PR 分支。

@danielaskdd

Copy link
Copy Markdown
Collaborator

@codex review

@chatgpt-codex-connector

Copy link
Copy Markdown
Contributor

Codex Review: Didn't find any major issues. Delightful!

Reviewed commit: f9d941e7b4

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@danielaskdd
danielaskdd merged commit a64cd8a into HKUDS:main Jul 17, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants