Skip to content

test: Check the memory estimation under every multiprocessing start method - #2264

Draft
Pijukatel wants to merge 1 commit into
masterfrom
claude/fervent-curie-cwnadu
Draft

Pijukatel wants to merge 1 commit into
masterfrom
claude/fervent-curie-cwnadu

Conversation

@Pijukatel

Copy link
Copy Markdown
Collaborator

Description

The shared-memory regression test for get_memory_info() ran its children under an explicit fork context only. The start methods differ in how much memory the children share with the rest of the process tree, which is what the PSS-based estimation has to account for, and the default on Linux changed from fork to forkserver in Python 3.14.

  • Parametrize test_memory_estimation_does_not_overestimate_due_to_shared_memory over fork, forkserver and spawn.
  • Move the children to module-level functions so that every start method can pickle them.
  • Warm the context up before measuring. forkserver (and the resource tracker) launch long-lived helper processes on first use, which otherwise land in the first baseline reading and inflate it. Without the warm-up the first round under forkserver on 3.14 misses the tolerance (ratio about 0.34) and only the retry saves it.
  • Filter the DeprecationWarning about forking a multi-threaded process for this test. The fork variant forks the test runner on purpose, and the forked process uses no threads.

Findings from running the estimation under all three methods, on 3.11 and 3.14, in a cgroup-limited container (the code path production takes after #2128): the de-duplication holds in every case, with a difference ratio below 0.05. The cgroup memory charge, which the kernel counts shared pages once for, matched the PSS deltas within a few MB throughout. Playwright browsers do not go through multiprocessing at all, and src/ has no ProcessPoolExecutor, so the 3.14 default change does not affect production estimates. Details in the issue.

Issues

Testing

  • uv run pytest tests/unit/_utils/test_system.py -k shared_memory -W default passes all three variants on Python 3.11 and 3.14, with no warnings and no tracebacks from the children.
  • ruff, ty clean on the file.

Checklist

  • CI passed

🤖 Generated with Claude Code

https://claude.ai/code/session_01TZuPLMf9Yf1NKZMqtGDS9J


Generated by Claude Code

@codecov

codecov Bot commented Oct 1, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 93.89%. Comparing base (df1178c) to head (9260a17).
⚠️ Report is 3 commits behind head on master.

Additional details and impacted files
@@            Coverage Diff             @@
##           master    #2264      +/-   ##
==========================================
+ Coverage   93.87%   93.89%   +0.01%     
==========================================
  Files         182      182              
  Lines       13130    13140      +10     
==========================================
+ Hits        12326    12338      +12     
+ Misses        804      802       -2     
Flag Coverage Δ
unit 93.89% <ø> (+0.01%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

…ethod

The shared-memory regression test for `get_memory_info()` ran its
children under an explicit `fork` context only. The start methods
differ in how much memory the children share with the rest of the
process tree, and the default changed from `fork` to `forkserver` in
Python 3.14, so the test is now parametrized over `fork`, `forkserver`
and `spawn`.

The children are module-level functions so that every start method can
pickle them. Start methods that launch long-lived helper processes on
first use (the fork server, the resource tracker) get a warm-up before
the measurements, since those processes would otherwise land in the
first baseline reading and inflate it. The `fork` variant intentionally
forks the multi-threaded test runner, so the deprecation warning that
Python 3.12+ emits for it is filtered for this test.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TZuPLMf9Yf1NKZMqtGDS9J
@Pijukatel
Pijukatel force-pushed the claude/fervent-curie-cwnadu branch from 74195a7 to 9260a17 Compare October 1, 2026 12:51

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Verify memory-usage estimation across multiprocessing start methods (fork → forkserver in Python 3.14)

2 participants