feat: expose token usage in CrawlResult (#1745)

[hafezparast]

Summary

Adds token_usage field to CrawlResult so LLM token consumption is returned in the crawl response — including via the Docker API.

What changed

crawl4ai/models.py:

# Added to CrawlResult
token_usage: Optional[Dict[str, Any]] = None

crawl4ai/async_webcrawler.py:
After extraction completes, reads config.extraction_strategy.total_usage and passes it to CrawlResult:

if hasattr(config.extraction_strategy, 'total_usage'):
    token_usage = {k: v for k, v in _token_usage.__dict__.items() if v is not None and v != 0}

Before / After

Before (Docker API response):

{
  "extracted_content": "...",
  "success": true
  // no token usage
}

After:

{
  "extracted_content": "...",
  "token_usage": {
    "prompt_tokens": 1234,
    "completion_tokens": 567,
    "total_tokens": 1801
  },
  "success": true
}

Python SDK (also works)

result = await crawler.arun(url, config=config)
print(result.token_usage)
# {'prompt_tokens': 1234, 'completion_tokens': 567, 'total_tokens': 1801}

Test plan

• 15/15 unit tests pass
• 290/291 regression tests pass (1 pre-existing transformers issue)
• CrawlResult serializes token_usage correctly
• Default is None when no LLM extraction is used

Closes #1745

🤖 Generated with Claude Code