refactor: confusing timeout handling

[anghel9]

Problem:

Confusing timeout logic that causes a user sees a 60s request handler timeout, but the errors will mention 130s (or 100s for HTTP crawlers where navigation timeout is only 30s).

What this PR changes

• Refactors _handleNavigation in http-crawler.ts and browser-crawler.ts so the navigation timeout covers the entire navigation flow, including navigation hooks.
• Removes the buffer timeout in both crawlers.
• Improves BasicCrawler messaging: instead of “requestHandler timed out after 100s/130s,” the error now states that a safety timeout was reached.
• This safety timeout is disabled by default with the new logic, but can be re-enabled by setting it to any value > 0.

Impact / Compatibility

• Workloads that implicitly relied on the hidden buffer may now time out sooner. We may want to increase the default navigation timeout to compensate.
• No behavioral surprises beyond the tighter, more honest timing.

Closes #2951

Contributors:
@ezequiel38
@anghel9

[barjin]

Thank you for your contribution!

I have a few questions and code quality ideas for now. Other contributors have worked with the timeouts deeper in the past and might have better insight into this part of Crawlee.

[barjin]

Random thought - it would make more sense to me to do this timeout one level higher in _runRequestHandler (where _handleNavigation is called - i.e. here). The syntax would require less nesting and would align with how we deal with timeouts in user defined request handlers - see below:

crawlee/packages/browser-crawler/src/internals/browser-crawler.ts

Lines 506 to 510 in 2b74503

	await addTimeoutToPromise(
	async () => Promise.resolve(this.userProvidedRequestHandler(crawlingContext as LoadedContext<Context>)),
	this.requestHandlerTimeoutInnerMillis,
	`requestHandler timed out after ${this.requestHandlerTimeoutInnerMillis / 1000} seconds.`,
	);

[anghel9]

It would align with how you guys are currently handling the user request handler but the current implementation makes it so _handleNavigation has built in protection if you ever decide to reuse it and _runRequestHandler stays slimmer. If you believe it to be necessary I can make the change. I will flatten _handleNavigation to improve readability.