Skip to content

Added firecrawl as a toolcall to crawl website contents from url #13

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 7 commits into
base: main
Choose a base branch
from

Conversation

Kitenite
Copy link

Description

Added firecrawl with an input box to crawl website contents from url and pass the response data in the LLM prompt

Related Issues

fixes onlook-dev#1624

Type of Change

Created a New Script to crawl website contents using firecrawl
Added an Input box for the users to provide url

  • Bug fix
  • New feature
  • Documentation update
  • Release
  • Refactor
  • Other (please describe):

Testing

Tested by running locally

Screenshots

Screenshot from 2025-04-18 17-43-08


Important

Adds Firecrawl integration for website content crawling with a URL input in the project creation flow.

  • New Feature:
    • Adds CrawlerService in crawler.ts to crawl website content using Firecrawl.
    • Integrates crawling functionality into PromptingCard.tsx with a URL input box.
    • Updates create.ts to handle crawled content in sendPrompt().
  • Dependencies:
    • Adds @mendable/firecrawl-js to package.json.
  • Environment:
    • Adds VITE_FIRECRAWL_API_KEY to .env.example.
  • UI/UX:
    • Updates translation.json and PromptingCard.tsx for new crawl input UI.
  • Error Handling:
    • Validates URL format and handles errors with user feedback in PromptingCard.tsx.

This description was created by Ellipsis for 793342e. It will automatically update as commits are pushed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[feat] Add webcrawling to get content from websites
2 participants