-
-
Notifications
You must be signed in to change notification settings - Fork 6.5k
[Bug]: AsyncHTTPCrawlerStrategy passes page_timeout (ms) directly to aiohttp ClientTimeout (seconds) #1894
Copy link
Copy link
Open
Labels
⚙️ In-progressIssues, Features requests that are in ProgressIssues, Features requests that are in Progress🐞 BugSomething isn't workingSomething isn't working
Description
crawl4ai version
0.8.0
Expected Behavior
page_timeout=5000 (5 seconds) in CrawlerRunConfig should cause AsyncHTTPCrawlerStrategy to timeout HTTP requests after approximately 5 seconds.
Current Behavior
AsyncHTTPCrawlerStrategy._handle_http passes page_timeout directly to aiohttp.ClientTimeout(total=page_timeout). But page_timeout is in milliseconds (as documented: "Timeout in ms for page operations") while aiohttp.ClientTimeout.total expects seconds. So page_timeout=5000 sets a 5000-second timeout (1.4 hours) instead of 5 seconds. The default page_timeout=60000 becomes 60000 seconds (16.7 hours).
Is this reproducible?
Yes
Inputs Causing the Bug
Steps to Reproduce
Code snippets
OS
Windows
Python version
3.12
Browser
No response
Browser version
No response
Error logs & Screenshots (if applicable)
No response
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
⚙️ In-progressIssues, Features requests that are in ProgressIssues, Features requests that are in Progress🐞 BugSomething isn't workingSomething isn't working