Leave the versioned docs alone

website/versioned_docs is generated content, synced by its own pass - the
ConcurrencySystem split has no business hand-editing the 4.0 snapshot. Reverts
this branch's edits to version-4.0/guides/scaling_crawlers.mdx and the example
file swap alongside it.

website/versioned_docs/version-4.0/guides/scaling_crawlers.mdx

@@ -10,7 +10,7 @@ import CodeBlock from '@theme/CodeBlock';
 
 import MinMaxConcurrencySource from '!!raw-loader!./scaling_crawlers_minMaxConcurrency.ts';
 import MaxRequestsPerMinuteSource from '!!raw-loader!./scaling_crawlers_maxRequestsPerMinute.ts';
-import ConcurrencySystemSource from '!!raw-loader!./scaling_crawlers_concurrencySystem.ts';
+import AutoscaledPoolOptionsSource from '!!raw-loader!./scaling_crawlers_autoscaledPoolOptions.ts';
 
 As we build our crawler, we might want to control how many requests we do to the website at a time. Crawlee provides several options to fine tune how many parallel requests should be made at any time, how many requests should be done per minute, and how should scaling work based on the available system resources.
 
@@ -45,51 +45,55 @@ It's recommended to leave it at the default value that is provided and letting t
 
 ## Advanced options
 
-While the options above should be enough for most users, if we wanted to get super deep into the configuration of autoscaling (the internal machinery in Crawlee that helps us allow crawlers to scale up and down), we can do so by injecting a pre-configured <ApiLink to="core/class/ConcurrencySystem">`ConcurrencySystem`</ApiLink> through the <ApiLink to="cheerio-crawler/interface/CheerioCrawlerOptions#concurrencySystem">`concurrencySystem`</ApiLink> crawler option. All the fine-grained scaling configuration lives on that instance (see <ApiLink to="core/interface/ConcurrencySystemOptions">`ConcurrencySystemOptions`</ApiLink>).
+While the options above should be enough for most users, if we wanted to get super deep into the configuration of the autoscaling pool (the internal utility in Crawlee that helps us allow crawlers to scale up and down), we can do so through the <ApiLink to="cheerio-crawler/interface/CheerioCrawlerOptions#autoscaledPoolOptions">`autoscaledPoolOptions`</ApiLink> object available on crawler options.
 
 :::danger Complex options up ahead!
 
 This section is super advanced and, unless you test the changes extensively and know what you're doing, it's better to leave these options to their defaults, as they are most likely going to work fine without much fuss.
 
 :::
 
-With that warning aside, if we're feeling adventurous, this is how we would pass these options when using a crawler. Note that when we inject a `ConcurrencySystem`, we own its lifecycle — we `start()` it before running the crawler and `stop()` it once we're done. The `minConcurrency`/`maxConcurrency`/`maxRequestsPerMinute` shortcuts configure the default system, so they cannot be combined with an injected one — set those limits on the `ConcurrencySystem` instead:
+With that warning aside, if we're feeling adventurous, this is how we would pass these options when using a crawler:
 
 <CodeBlock language="js">
-	{ConcurrencySystemSource}
+	{AutoscaledPoolOptionsSource}
 </CodeBlock>
 
-:::tip Capping the combined concurrency of several crawlers
-
-Injecting the *same* `ConcurrencySystem` instance into several crawlers makes them share a single concurrency budget, capping their combined parallelism instead of letting each crawler scale independently.
-
-:::
-
 ### `desiredConcurrency`
 
 This option specifies the amount of requests that should be running in parallel at the start of the crawler, assuming there are so many available. It defaults to the same value as `minConcurrency`.
 
 ### `desiredConcurrencyRatio`
 
-The minimum ratio of concurrency to reach before more scaling up is allowed (a number between `0` and `1`). By default, it is set to `0.9`.
+The minimum ratio of concurrency to reach before more scaling up is allowed (a number between `0` and `1`). By default, it is set to `0.95`.
 
-We can think of this as the point where the concurrency system can attempt to scale up (or down), monitor if there's any changes, and correct them if necessary.
+We can think of this as the point where the autoscaling pool can attempt to scale up (or down), monitor if there's any changes, and correct them if necessary.
 
 ### `scaleUpStepRatio` and `scaleDownStepRatio`
 
-These values define the fractional amount of desired concurrency to be added or subtracted as the concurrency system scales up or down. Both of these values default to `0.05`.
+These values define the fractional amount of desired concurrency to be added or subtracted as the autoscaling pool scales up or down. Both of these values default to `0.05`.
 
-Every time the concurrency system attempts to scale up or down, this value will be added or subtracted from the current concurrency, and, based on the [`desiredConcurrencyRatio`](#desiredconcurrencyratio) and [`maxConcurrency`](#minconcurrency-and-maxconcurrency), determines how many requests can run concurrently.
+Every time the autoscaled pool attempts to scale up or down, this value will be added or subtracted from the current concurrency, and, based on the [`desiredConcurrencyRatio`](#desiredconcurrencyratio) and [`maxConcurrency`](#minconcurrency-and-maxconcurrency), determines how many requests can run concurrently.
+
+### `maybeRunIntervalSecs`
+
+Indicates how often the autoscaling pool should check if more requests can be started and, if that's true, starts a new request if there are any available. This value is represented in seconds, and defaults to `0.5`.
+
+:::info
+
+Changing this has no effect for requests that are fired immediately after the previous ones are finished. However, it will influence how fast new requests will be started after the autoscaled pool scales up.
+
+:::
 
 ### `loggingIntervalSecs`
 
-This option lets us control how often the concurrency system should log its current state (the current concurrency ratio, desired ratios, if the system is overloaded and so on).
+This option lets us control how often the autoscaled pool should log its current state (the current concurrency ratio, desired ratios, if the system is overloaded and so on).
 
 We can disable logging altogether by setting this to `null`. By default, it is set to `60` seconds.
 
 ### `autoscaleIntervalSecs`
 
-This option lets us control how often the concurrency system should check if it can and should scale up or down. This value is represented in seconds, and defaults to `10`.
+This option lets us control how often the autoscaling pool should check if it can and should scale up or down. This value is represented in seconds, and defaults to `10`.
 
 :::tip
 
@@ -114,69 +118,3 @@ This controls how many total requests can be made per minute. It counts the amou
 This option can be set by specifying [`maxRequestsPerMinute`](#maxrequestsperminute) in your crawler options too, as it is a shortcut for visibility and ease of access.
 
 :::
-
-## Load signals
-
-Whether the machine counts as overloaded is decided by *load signals*. Four are built in — memory, event loop, CPU and the storage client's rate-limit errors — and each is configured by its own bag under <ApiLink to="core/interface/ConcurrencySystemOptions#loadSignals">`loadSignals`</ApiLink>, carrying both its limits and the `overloadedRatio` at which it fires. If any signal reports overload, the system is overloaded and concurrency is held down.
-
-A signal we don't want watched at all can be switched **off** with `false`, which stops it being collected as well as evaluated (its entry in the reported status then simply reads as not overloaded):
-
-```ts
-const concurrencySystem = new ConcurrencySystem({
-    loadSignals: {
-        // Our storage backend reports no rate-limit statistics, so stop polling it every second.
-        client: false,
-        eventLoop: { maxBlockedMillis: 100 },
-    },
-});
-```
-
-We can also watch resources of our own by implementing <ApiLink to="core/interface/LoadSignal">`LoadSignal`</ApiLink> — navigation timeouts or proxy health, say — and passing them in `loadSignals.custom`. The <ApiLink to="core/class/SnapshotStore">`SnapshotStore`</ApiLink> helper does the time-windowed bookkeeping for us:
-
-```ts
-const store = new SnapshotStore();
-
-const proxyHealth: LoadSignal = {
-    name: 'proxyHealth',
-    overloadedRatio: 0.3,
-    async start({ maxSampleWindowMillis }) {
-        // Retain exactly the window we will be sampled over.
-        store.useSampleWindow(maxSampleWindowMillis);
-        // ...start measuring, calling store.push() per measurement
-    },
-    async stop() { /* ...stop measuring */ },
-    getSample: (sampleDurationMillis) => store.getSample(sampleDurationMillis),
-};
-
-const concurrencySystem = new ConcurrencySystem({ loadSignals: { custom: [proxyHealth] } });
-```
-
-Each built-in is a public class too — `MemoryLoadSignal`, `EventLoopLoadSignal`, `CpuLoadSignal`, `ClientLoadSignal` — taking exactly the bag its `loadSignals` key accepts. Constructing one directly is how we *wrap* a built-in rather than reimplement it, for instance to stop its verdict from flapping:
-
-```ts
-const cpu = new CpuLoadSignal({ overloadedRatio: 0.4 });
-
-const stickyCpu: LoadSignal = {
-    ...cpu,
-    name: 'cpuSticky',
-    start: (context) => cpu.start(context),
-    stop: () => cpu.stop(),
-    // Hold the overload verdict for a while after the CPU recovers.
-    getSample: (ms) => holdOverloads(cpu.getSample(ms)),
-};
-
-const concurrencySystem = new ConcurrencySystem({
-    // The built-in has to go: signal names are unique, and a duplicate throws.
-    loadSignals: { cpu: false, custom: [stickyCpu] },
-});
-```
-
-:::info Signal names are the keys of the reported status
-
-Two signals cannot share a name, and a duplicate throws at construction time. Taking over one of the built-in names (`memInfo`, `eventLoopInfo`, `cpuInfo`, `clientInfo`) therefore means switching that built-in off first, as above.
-
-:::
-
-### `snapshotHistorySecs` and `currentHistorySecs`
-
-Signals are not read as a single instantaneous measurement but averaged over a window, and there are two: `currentHistorySecs` (default `5`) is the short window that gates whether one more request may start, while `snapshotHistorySecs` (default `30`) is the longer window autoscaling decisions are based on. Dispatch therefore reacts to spikes quickly while scaling stays stable. Both apply to every signal alike, built-in or custom, and signals size their snapshot retention to the wider of the two — so raising `snapshotHistorySecs` is what costs memory.

website/versioned_docs/version-4.0/guides/scaling_crawlers_autoscaledPoolOptions.ts

@@ -0,0 +1,8 @@
+import { CheerioCrawler } from 'crawlee';
+
+const crawler = new CheerioCrawler({
+    // Pass in advanced options by providing them in the autoscaledPoolOptions
+    autoscaledPoolOptions: {
+        // ...
+    },
+});