Repository navigation
Hackathon [Feature]: Add incremental sync to the Notion connector #4820
Description
Activity
Submitted PR to \cognee-community\ implementing the watermark-based incremental sync while preserving the full snapshot for deletion detection: topoteretes/cognee-community#157
Reacted by Nandha140904这个增量约束抓得对——不能简单跳过页面，否则
orphan_cleanup会把活页面当删除忘掉。正确形状是每轮仍然全量枚举（廉价 list），只对last_edited_time变更的页调_render_blocks，其余复用已存内容。
下面这段在保持快照权威的同时做水位增量：# Incremental Notion sync that keeps the snapshot authoritative (orphan-safe). # Plug in: replace the full re-render loop in the DLT source's _page_to_row. def incremental_pages(pages, prev_watermarks, render, stored_content): """Enumerate EVERY page (keeps snapshot complete -> safe orphan_cleanup), but only call _render_blocks for pages whose last_edited_time moved. Returns (rows_to_emit, page_ids_seen).""" rows, seen = [], set() for p in pages: # full enumeration, cheap list call pid = p["id"] seen.add(pid) edited = p["last_edited_time"] if prev_watermarks.get(pid) == edited: rows.append(stored_content[pid]) # unchanged -> reuse, skip render else: rows.append(render(p)) # changed -> re-render return rows, seen def orphan_deletes(current_ids, graph_ids): """Forget only pages absent from the authoritative listing.""" return [g for g in graph_ids if g not in current_ids] # rows, seen = incremental_pages(pages, watermarks, _render_blocks, cache) # forget(orphan_deletes(seen, graph_page_ids))
如果你们想把 watermark 持久化到 DLT 的 state，我可以补一段 state 读写样例。
Hi @NMZivkovic
I would like to work on this issue.
Could you please assign this to me.
ThanksHi @NMZivkovic, I'd like to take this one for Mergetober.
I'll start by reading packages/connector/notion/ in cognee-community, in particular the full-snapshot setup (write_disposition="replace"), _page_to_row, _render_blocks, and the orphan_cleanup/_is_gone deletion path, so I understand why the snapshot is load-bearing before I change anything.
My plan for the incremental fetch:
- Keep enumerating every visible page on each run, so the snapshot stays authoritative and forget-on-delete keeps working (trashed and unshared pages still get removed).
- Store a per-page last_edited_time watermark from the previous run and compare against it.
- Only call _render_blocks for pages whose last_edited_time changed, and reuse the stored rendered content for the rest.
- Keep the content-hash data_id as is, and use last_edited_time only as a fetch filter, never as the id.
- Keep the existing failure semantics: a render error aborts the run instead of yielding a partial snapshot.
- I'll test the nested-block case explicitly, to check whether edits to child blocks bump the parent page's last_edited_time, and handle it if they don't.
Tests: a second sync with no changes makes no blocks.children calls; editing one page re-renders only that page; deleting, trashing and unsharing still forget; a mid-run render failure still aborts cleanly. I'll also document the watermark and where it's stored in the README.
I'll test against a real Notion workspace, and I'll treat the OAuth part as a possible follow-up PR once the incremental sync is merged-ready.
Could you assign this to me? Thanks!
Hi @NMZivkovic, I'd like to work on this issue for Mergetober.
Here is my technical plan for the incremental sync:
- Enumerate pages using page search/database.query list calls to keep the full snapshot complete (ensuring orphan_cleanup & deletion mechanics remain intact).
- Store a run watermark (
last_edited_time) in the local DLT state / connector storage. - Compare each page's
last_edited_timeagainst the previous watermark, skipping_render_blockswhen unchanged, while retaining content-hashdata_idmapping. - Maintain error-handling semantics so mid-run render failures abort without emitting partial snapshots.
- Add unit and integration tests covering the watermark logic and page deletion/unsharing scenarios.
Could you please assign this to me?
Hi @NMZivkovic, I’d like to work on this for Mergetober if it’s still available.
I went through the issue and also noticed the earlier watermark PR (#157) was closed without being merged. My approach would be to keep the existing full-snapshot behavior intact so deletion/unsharing continues to work, while avoiding unnecessary block fetches for pages that haven’t changed.
I’d use
last_edited_timeonly to decide whether a page needs_render_blocksagain, reuse the previously rendered content for unchanged pages, and keep the existing content-hash baseddata_id. I’d also make sure a failed render never leaves a partial snapshot or advances the saved state.For testing, I’d cover the no-change second sync, a single edited page, deletion/unsharing, and failure during rendering.
If this is still open for a contributor, could you assign it to me? Thanks!
Hi @NMZivkovic! I'd love to implement incremental sync for the Notion connector.
Implementation Plan:
- State Tracking: Track and persist the
last_synced_attimestamp / Notion cursor per database/page. - Filtered Query: Use Notion's
filter: { timestamp: "last_edited_time", last_edited_time: { after: <timestamp> } }to fetch only updated/new blocks. - Sync Logic: Update only modified records in Cognee's graph/vector memory without re-indexing unchanged entries.
- Tests: Add unit tests verifying incremental vs full sync behavior.
Could you please assign this to me?
- State Tracking: Track and persist the
Hi @NMZivkovic, I'd like to work on this. I read through
notion.pyand agree with the approach in the issue: keep the fullreplacesnapshot and only skip_render_blocksfor pages that haven't changed. A few details decide whether that's actually safe, though.First, Notion rounds
last_edited_timedown to the minute (changelog). If a sync reads a page at 10:05:20 and someone edits it at 10:05:40, the timestamp still says 10:05, so a plain "same timestamp, skip it" check would miss that edit until the page changes again. I'd re-render any page whose timestamp falls in the minute the previous sync started.Second,
_render_blocksfollows every block withhas_children, which can pull in content from child pages and synced blocks. That content can change without the parent page'slast_edited_timemoving. So while rendering I'd note those dependencies and their timestamps, and re-render the parent when one of them changes. Checking them only needs page/block metadata, so a sync with no changes still makes zeroblocks.childrencalls. I'll confirm both cases with fixtures.Third, where the cached content lives. With
replace, every unchanged page still has to be sent again with its text. I see two options:- Keep it in dlt resource state. Simple, and state is saved together with the load, so a failed run can't leave the cache ahead of the snapshot. The downside is that the whole workspace's text sits in one state blob that's rewritten every run.
- Keep only a small
{id: timestamp, hash, deps}map in state and read unchanged content back from the previously stagednotion_pagestable, rendering again if a row is missing or doesn't match.
The first is simpler and the second scales better. Do you have a preference?
Everything else stays as it is: the content-hash
data_id,_is_gonefor unshared pages, skipping archived and trashed pages, and a render error aborting the run. If the cache is missing or broken, it just does a full render.On top of the existing tests I'd add: a no-change sync with zero
blocks.childrencalls, one edit causing exactly one re-render, a same-minute edit, a child page or synced block edit re-rendering its parent, delete/trash/unshare still forgetting, and a failed render leaving the cache alone. I'd document the watermark and the same-minute rule in the README, and leave OAuth for a follow-up PR.Hi @NMZivkovic, I've opened a PR for the incremental sync part: topoteretes/cognee-community#272
Design, against the constraints in the issue
- Snapshot stays authoritative.
write_disposition="replace"is unchanged and every visible page is still enumerated each run, soorphan_cleanupnever mistakes a skipped page for a deleted one. - Only the expensive part is skipped. Each page's
last_edited_timeis compared with a stored watermark, and_render_blocksruns only for pages that changed. Unchanged pages reuse their stored content. data_idis still the content hash, so metadata-only edits cause at worst one extra render, never re-cognification.- Failure semantics preserved. A render error still aborts the run, and the watermark lives in dlt source state, which dlt commits only after a successful run, so a failed run can't advance it.
- No new storage layer. It uses
dlt.current.source_state()instead of a separate file. Diff is limited topackages/connector/notion/.
Trade-offs, stated openly
- State holds rendered content per page, so it grows with workspace size.
- Entries for vanished pages aren't pruned yet.
- Nested child-block edits that don't bump the parent's
last_edited_timearen't handled in code. This is documented in the README instead of adding extra API calls.
Tests cover the first sync, a no-change sync (zero
blocks.childrencalls), a single-page edit re-rendering only that page, and render-failure behaviour. OAuth is not part of this PR.- Snapshot stays authoritative.
Issue assigned to @Anurag13075
The Notion connector in cognee-community (packages/connector/notion/) works via a full snapshot (write_disposition="replace"), but suffers from two main issues:
High API Read Cost: Every sync re-reads all pages by calling _render_blocks on every page, making thousands of blocks.children API calls against Notion's rate limits just to rediscover unchanged text.
Missing Public OAuth: Auth relies solely on an internal integration token (NOTION_API_KEY), which prevents hosted multi-tenant deployment ("Connect your Notion").
Deletion Detection Constraint: The graph relies on orphan_cleanup reading a page's absence as a deletion. Because Notion omits trashed or unshared pages instead of flagging them, skipping unedited pages during search would cause orphan_cleanup to misinterpret live pages as deleted.
Proposed Solution
Implement an incremental fetch while preserving full snapshot enumeration:Full Page Enumeration: Continue listing every visible page on every run using paginated search or database.query calls to keep the snapshot complete and preserve deletion/unsharing logic (_is_gone).
Watermark Comparison: Compare each page’s last_edited_time against the previous run’s stored watermark.
Selective Rendering: Only call _render_blocks for pages whose last_edited_time changed; reuse stored content for unchanged pages.
Preserve Failure Semantics: If rendering fails mid-run, abort the entire process so a partial snapshot is never emitted (preventing accidental memory loss).
Retain Content Hash: Continue using the content hash as data_id (not last_edited_time) so metadata-only edits do not trigger downstream LLM re-cognification.
Public OAuth (Optional/Follow-up): Implement Notion's public OAuth flow alongside the existing integration token path.
@NMZivkovic i like to work on this issue.
As part of the Mergetober. I read through notion.py and agree with the approach in the issue: keep the full replace snapshot and only skip _render_blocks for pages that haven't changed.
Problem Statement
Unlike the other connector tickets in this batch, the Notion connector already exists —
packages/connector/notion/in cognee-community has a working DLT source (~290 lines), tests, an example, and a README. #4541 even points contributors at it as the reference implementation. Read it before you start; this is not a greenfield ticket.Two things are missing.
1. Every sync re-reads the entire workspace. The source is a full snapshot (
write_disposition="replace"), and_page_to_rowcalls_render_blocksfor every page on every run. Unchanged pages are not re-cognified — the content-hashdata_idalready prevents that, so the LLM cost is fine — but the Notion API read cost is paid in full each time. On a large workspace that means thousands of block-children calls per sync, against Notion's rate limit, to rediscover text that did not change.2. There is no OAuth flow. Auth is a single internal integration token (
NOTION_API_KEY), which works for one workspace you control. A hosted "connect your Notion" experience needs Notion's public OAuth.Proposed Solution
Incremental fetch, without breaking deletion. This is the interesting constraint and the reason the ticket is not trivial: forget-on-delete depends on the snapshot being complete. If you skip pages,
orphan_cleanupreads their absence as deletion and forgets live pages from the graph. Any design must keep listing every visible page while skipping only the expensive part.The shape that satisfies both:
search/database.queryare paginated list calls), so the snapshot stays authoritative.last_edited_timeagainst the previous run's watermark._render_blocksfor pages that changed; reuse the stored content for the rest.Note the existing subtlety in
_page_to_row:last_edited_timebumps on metadata-only edits that do not change text. Using it as a fetch filter is fine (worst case you re-render a page unnecessarily); using it as thedata_idwould cause churn, which is why the content hash exists. Do not replace the content hash with it.OAuth can be a second PR, or a second contributor: add the public OAuth flow alongside the token path, keeping the token path as the default for local use.
Watch out for
hard_deletecannot work here — Notion omits trashed pages rather than flagging them._is_gonealready treats that as "forget", which is correct. Keep that behaviour._render_blocksrecurses with a depth guard. Any caching must key on page id and handle child-block edits that do not bump the parent'slast_edited_time.Acceptance Criteria
blocks.childrencallsContext
Part of the cognee hackathon connector push. Difficulty: Intermediate (M) — the code is small, but the snapshot/deletion interaction is the part to get right. Best suited to someone who wants a correctness puzzle rather than a fresh API integration.