Telemetry insights — 2026-09-20

Executive summary

1. v1.6.0's first GA day is the finding: 67.3% of its pipeline runs (8,088 of 12,014) ended with neither a Pipeline Run Completed nor a Pipeline Run Errored event — 4.3× the fleet's 15.8% and 5.9× the fleet median, on a version whose first_seen is inside the window.
2. The gap is not inflated start counts: 1.6.0 emits 32.7 pipeline starts per cognee.remember (1.5.3: 34.8, 1.4.2: 31.5) but only 5.3 completions (1.5.3: 15.4, 1.4.2: 29.6). Same work submitted, one third of the terminal events.
3. Prime suspect is 1.6.0's new keyless default: with no LLM key the first remember() now downloads ~750 MB of GLiNER + fastembed (SDK-754's own commit message: "A first run that takes minutes looked hung"). Self-hosted 1.6.0 — pre-existing installs, warm caches — sits at 5.5%.
4. Yesterday's prediction in Telemetry insights: 2026-09-19 — fleet-wide pipeline error step change around Sept 10 #5140 was wrong and the reason is structural: cognee/modules/cognify/recovery.py contains no send_telemetry call, so SDK-764 fix(pipelines): close abandoned pipeline runs as ERRORED at startup #5090/SDK-765 fix(pipelines): startup recovery keeps the documents an abandoned run completed #5104's startup recovery closes the DB row but emits no event. Telemetry silent gaps can never close by that route.
5. The fleet-wide error step change from Telemetry insights: 2026-09-19 — fleet-wide pipeline error step change around Sept 10 #5140 persists (17.1% on 09-19 vs its W1 16.5% / W2 11.9%). Not re-filed — tracked there.

Data window and method

• Window: 2026-07-12 → 2026-09-20 (70 days), telemetry_aggregates/*.csv (WINDOW.txt: window_days=70).
• 2026-09-20 is a partial day and is excluded from every rate comparison: fleet pipeline starts are 43,973 on 09-20 versus 235,649 on 09-19 (18.7%), consistent with an extract that ran at ~05:37 UTC. 09-20 figures appear only where noted as directional.
• No interpreter is available in this runner (python3, awk, sort with arguments are all refused by the sandbox), so every aggregate below is a hand-summed set of rows from the CSVs. The load-bearing sums (fleet 09-19 started/completed/errored) were computed twice, by different groupings, and cross-check against the per-version splits in daily_event_volumes.csv exactly (see Finding 1).
• Silent-gap ratio = (started − completed − errored) / started, the metric named in the brief. For pipelines it is computed from pipeline_outcomes_daily.csv and cross-checked against daily_event_volumes.csv, which additionally splits by self_hosted.
• A note on what "started" counts. run_tasks_with_telemetry is invoked per data item, not per logical run (run_tasks_data_item.py:213 and :313), so these are per-item counts. PR fix(pipelines): log pipeline run start/complete once per run, not per data item #3922 proposed changing that and was closed unmerged (2026-07-05), so the convention is stable across every version in the window and version-to-version ratios are comparable.

Finding 1 — v1.6.0's GA day: two thirds of pipeline runs emit no terminal event (severity: high)

Computed

pipeline_outcomes_daily.csv, every 1.6.0 row in the window:

Splitting 09-19 by self_hosted in daily_event_volumes.csv (the three components sum to the table above exactly: 11,373+641 = 12,014; 1,858+586 = 2,444; 1,462+20 = 1,482):

Same-day controls. Other versions on 09-19, non-self-hosted — same date, same deployment class, different code:

Launch-day controls. Every 1.x GA launch in the window, at the same point in its life:

Against the threshold in the brief. Fleet 09-19: 235,649 started / 158,130 completed / 40,377 errored → gap 37,142 = 15.76%, error rate 17.13%. Across the 20 versions with >500 started on 09-19, the median gap ratio is 11.4%; 1.6.0 at 67.3% is 5.9× the median, far past the 1.5× bar. 1.6.0 carries 21.8% of the fleet's entire silent gap (8,088 of 37,142) on 5.1% of its starts (12,014 of 235,649). Its error rate, 12.3%, is unremarkable — below the fleet's 17.1%. The problem is specifically runs that end saying nothing.

The alternative hypothesis, tested and rejected. A 1.6.0 change that emitted more Started events per operation would produce the same ratio without anything being wrong. It doesn't hold — normalising by SDK entry point on 09-19, non-self-hosted:

1.6.0's starts-per-remember sits inside the normal range; its completions-per-remember is a third of its peers'. Starts are not inflated — terminal events are missing.

Day-boundary carryover, bounded. Runs begun late on 09-19 would complete on 09-20. Even attributing all of 09-20's 1.6.0 completions (952) to 09-19 closes only 11.8% of the 8,088 gap.

Hypothesis

Two things have to be true: a mechanism that turns an abandoned run into silence, and something in 1.6.0 that makes abandonment common.

The mechanism is in the code and is version-independent. run_tasks_with_telemetry.py sends Pipeline Run Started at line 25, Completed at line 40 only after the generator is fully drained, and Errored at line 55 from an except Exception: at line 50. GeneratorExit, asyncio.CancelledError and KeyboardInterrupt are all BaseException, not Exception — every one of them produces a Started with no terminal event. The same class of bug was already fixed one layer up: run_tasks.py:292 catches (Exception, asyncio.CancelledError) with a comment explaining that a bare except Exception misses cancellation (CLO-365). That fix was never applied to the telemetry wrapper. A second amplifier sits at run_tasks.py:210: asyncio.gather(...) is called without return_exceptions=True, so one item's failure abandons its in-flight siblings — each of which has already emitted Started. That the very next block (run_tasks.py:220-235) tests isinstance(result, BaseException), which is unreachable without return_exceptions=True, suggests the flag was intended and dropped.

The 1.6.0-specific trigger is most likely the new keyless default. 1.6.0 is the first release where a user with no LLM key gets local models instead of an error (54db3c102 "Default to GLiNER + fastembed without LLM key", b98eec29e "Make fastembed a core dependency", both 09-16). SDK-754's own commit message (9947be39e, 09-18) describes the consequence: "with no LLM key the first remember() downloads the GLiNER demo model (about 750 MB) and the fastembed embedder through the hub, and nothing in cognee said so… A first run that takes minutes looked hung." A day-1 cohort of 71 new public deployments running the quickstart — which 663a2dc15 (09-19) put at the top of the README — hits a multi-minute silent download inside its first pipeline run. A user who Ctrl-Cs there produces precisely this signature. It also explains the split: self-hosted 1.6.0 deployments are pre-existing installs with configured keys and warm caches, and they sit at 5.5%.

This is a hypothesis, not a conclusion, and the aggregates cannot close it: provider_stack_daily.csv records only completed runs, so runs abandoned during a first-use download are invisible there by construction. Two weaker candidates were considered and are less likely: 9da9c130b (per-dataset embedding-model check, SDK-756) would raise and be counted as errored, not vanish; and the 1.6.0 sdist that shipped without Ladybug binaries (4294605f2, fixed 09-19 00:44) explicitly "preserve[d] the existing wheel packaging behavior", and nearly all installs take the py3-none-any wheel.

Proposed fix

1. Close the mechanism — cognee/modules/pipelines/operations/run_tasks_with_telemetry.py:50: widen to except (Exception, asyncio.CancelledError) and add a finally/GeneratorExit path so an abandoned generator still emits a terminal event (a distinct Pipeline Run Abandoned is better than mislabelling it errored). Mirror the CLO-365 comment already at run_tasks.py:292.
2. Close the amplifier — cognee/modules/pipelines/operations/run_tasks.py:210: pass return_exceptions=True, which also makes the existing isinstance(result, BaseException) branch at line 221 reachable as written.
3. Confirming observable: after (1), 1.6.0's silent gap should collapse into a named terminal event. If the replacement events are dominated by cancellation on a first remember() against fastembed/gliner_demo, the keyless-download hypothesis is confirmed and the product fix is to make the download an explicit, resumable, pre-pipeline step rather than something that happens inside a pipeline run. If they are spread across configured LLM providers instead, the hypothesis is wrong and the cause is elsewhere in 1.6.0.

Duplicate check

• gh issue list --label telemetry-insights --state all → only Telemetry insights: 2026-09-19 — fleet-wide pipeline error step change around Sept 10 #5140 (2026-09-19, OPEN). Telemetry insights: 2026-09-19 — fleet-wide pipeline error step change around Sept 10 #5140's subject is the fleet-wide error-rate step change; it explicitly listed 1.6.0 under "watching" as "Too small to judge — establish the adoption curve next run" (09-18 was 216 starts, self-hosted only). This is that next run, and the finding is a different metric (silent gap, not error rate) on data that did not exist yesterday. New.
• gh pr list --state all --search "abandoned pipeline runs errored" → SDK-764 fix(pipelines): close abandoned pipeline runs as ERRORED at startup #5090 (merged 09-17) and SDK-765 fix(pipelines): startup recovery keeps the documents an abandoned run completed #5104 (merged 09-17) recover abandoned runs, and SDK-577 fix: Recover interrupted pipeline runs #4966 (open) proposes more. None of them touch telemetry emission — see Finding 2. Not addressed by a merged PR.
• gh pr list --state closed --search "pipeline run completed telemetry" → fix(pipelines): log pipeline run start/complete once per run, not per data item #3922 (closed, unmerged) concerns per-item vs per-run logging, not lost terminal events. Not a duplicate.

Finding 2 — Startup recovery closes the database row but emits no telemetry, so the gap is undiagnosable (severity: medium, observability)

Computed

cognee/modules/cognify/recovery.py — the file #5090 and #5104 rewrote — contains zero matches for send_telemetry or Pipeline Run. Within cognee/modules/pipelines/, send_telemetry appears in exactly two files: run_tasks_with_telemetry.py (lines 25, 40, 55) and run_tasks_base.py (lines 162, 233, 250, which emit Task-level events). log_pipeline_run_error.py, which 30aef3c8f modified for the recovery path, emits none.

The consequence is visible in the data. #5140 predicted, in its watching section, that #5090 would make 1.6.0's "measured error rates rise and silent-gap ratios fall … for mechanical reasons". The first release to ship that code measured the opposite:

Hypothesis

No hypothesis needed for the cause; it is a direct code reading. The recovery path updates pipeline_runs in the relational database and never reaches the telemetry emitter. A run abandoned by process death is therefore correctly closed locally and remains permanently silent in telemetry. The two systems have diverged: operators see a clean pipeline_runs table, this report sees an unexplained third of the fleet's runs. That divergence is why Finding 1 cannot be diagnosed from the aggregates — there is no error_type, no exception class, nothing but an absence.

Proposed fix

Emit Pipeline Run Errored (or a new Pipeline Run Recovered) from the recovery path in cognee/modules/cognify/recovery.py when it closes an abandoned run, carrying the original run's cognee_version and the recovery reason — the version matters, since the recovering process may be a different build from the one that abandoned the run. If a new event name is used, add it to the extractor's event list at .github/scripts/telemetry_aggregate_extract.py:75-77, which currently tracks only the three Pipeline Run * events. Confirming observable: the fleet silent-gap ratio and the count of recovery events should move in opposite directions by roughly the same magnitude in the next extract.

Duplicate check

• Telemetry insights: 2026-09-19 — fleet-wide pipeline error step change around Sept 10 #5140 Finding 3 covers three different telemetry blind spots (search never emits ERRORED; add/cognify carry no version; TELEMETRY_ORIGIN unset). This is a fourth, in a different module, and Telemetry insights: 2026-09-19 — fleet-wide pipeline error step change around Sept 10 #5140 does not mention it — it predicted the opposite behaviour. New.
• SDK-764 fix(pipelines): close abandoned pipeline runs as ERRORED at startup #5090 and SDK-765 fix(pipelines): startup recovery keeps the documents an abandoned run completed #5104 are the merged PRs whose effect is being assessed; the pattern persists after their merge and is visible in the first release to contain them. Per the brief, that makes this "fix did not take" (for telemetry purposes) rather than a suppressed duplicate. To be fair to those PRs: their stated goal was the database state, which they appear to achieve; they simply do not reach telemetry.

Persisting, already tracked — not re-filed

Fleet-wide pipeline error step change (#5140, Finding 1). On 09-19 the fleet error rate was 17.13% (40,377 of 235,649), against #5140's W1 (Sep 12–18) 16.5% and W2 (Sep 5–11) 11.9%, both ex-1.2.2.dev0. That runaway is now negligible (09-19: 212 started, 1 errored), so 09-19's figure needs no such correction and is directly comparable. The step change persists at its elevated level. Tracked in #5140; not re-filed.

Watching (near-threshold, decaying, or single-deployment)

• 1.0.3's error spike is resolving. Telemetry insights: 2026-09-19 — fleet-wide pipeline error step change around Sept 10 #5140 flagged 09-18 as a one-day anomaly (45,514 started / 22,370 errored, 49.1%, against ~0.5% on 09-14–09-17). It has decayed: 09-19 47,625 / 7,359 = 15.5%, 09-20 (partial) 3,778 / 186 = 4.9%. Still above its own ~0.5% baseline but trending back. No action.
• Two old versions also exceed the 1.5× gap bar on 09-19 and are worth a second look only if they persist: 1.4.0 11,623 started / 5,866 gap = 50.5% (driven by its self-hosted side, 9,106 / 5,350 = 58.8%) and 1.0.5 4,252 / 1,735 = 40.8%. Both are two-month-old builds predating every fix in Finding 1's area; neither is a new release, so neither is a regression candidate.
• 1.5.4rc1 reports more completions than starts: 09-19, 3,046 started / 3,605 completed / 0 errored. Same integrity class Telemetry insights: 2026-09-19 — fleet-wide pipeline error step change around Sept 10 #5140 Finding 3b reported for cognify and 1.5.0 search — started and completed are not emitted from the same scope. Only 2 identities; excluded from the median in Finding 1.
• cognify still reports ~zero completions. Across the whole 70-day window sdk_exec_outcomes_daily.csv shows cognify completing 0 on most days (09-18: 685/0/784, 09-19: 901/0/723, 09-20: 158/0/279 — errored exceeding started on all three). Same root as Telemetry insights: 2026-09-19 — fleet-wide pipeline error step change around Sept 10 #5140 Finding 3b; noted here only because the absolute numbers grew ~30× in the last three days as 1.6.0 landed.
• 1.6.0's search path is elevated but not yet actionable: 09-19, 502 started / 388 completed = 22.7% gap, versus 1.5.4's 9.3%. 1.5.3 is comparable at 20.6%, and the volume is small. Re-check once 1.6.0 has a full week.
• 1.5.4 and 1.5.3 both fell by roughly half on 09-19 (1.5.4: 90,103 → 42,856, −52.4%; 1.5.3: 48,799 → 21,478, −56.0%). Because both moved together while 1.0.3 rose (45,514 → 47,625), this reads as a fleet-level effect plus migration onto 1.6.0 rather than anything version-specific. Not flagged.
• Search-type mix is quiet. No type collapsed or exploded on 09-19. CHUNKS_LEXICAL, CHUNKS and SUMMARIES continue to dominate; 1.6.0's own search volume is negligible (GRAPH_COMPLETION 69, CHUNKS 58, RAG_COMPLETION 1). The undocumented SearchType.HYBRID / HYBRID_RERANK values on 1.0.3 that Telemetry insights: 2026-09-19 — fleet-wide pipeline error step change around Sept 10 #5140 noted are still present (09-19: 8 and 1,332) and still legacy.
• No new or disappeared event types. Remember Entry API Endpoint Invoked was checked specifically because it appears prominently in 1.6.0's rows — it dates to 2026-07-12 and is not new.

Data window and privacy note

This analysis used only the pre-aggregated, anonymized CSVs in telemetry_aggregates/ (window 2026-07-12 → 2026-09-20, 70 days) together with this repository's own git history and public PR/issue metadata via gh. No warehouse or external data source was queried, and none was available to this runner. The CSVs contain no user identifiers, no query text and no dataset names; distinct_identities counts deployments only, and no attempt was made to obtain anything finer. Provider and model values containing identifier-shaped text arrive pre-grouped into a redacted bucket and were treated as a mixed set of private configurations, never as a specific provider. Cross-version identity comparisons are avoided: persistent_id exists only on builds from ~April 2026 onward, so identity counts on older-version rows skew high. Where identity counts appear above (1.6.0's 71 non-self-hosted deployments) they are used only to characterise one recent version against itself and against same-day peers, not to compare old builds with new ones.