Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

A faster approach for fast crawler! #3

Open
black-fractal opened this issue Jan 17, 2021 · 0 comments
Open

A faster approach for fast crawler! #3

black-fractal opened this issue Jan 17, 2021 · 0 comments
Assignees
@black-fractal
Labels
enhancement New feature or request

Comments

@black-fractal
Copy link
Owner

black-fractal commented Jan 17, 2021 •
edited
Loading

  • Is your feature request related to a problem? Please describe.
    1- Every time fast crawler run, it open all JSON files for gathering historical information about repetitive crawling!
    2- Many JSON files would be merged! only the longest chan should be left and the others would be eliminated!

  • Describe the solution you'd like
    1- A new python script or a new function in fast crawler should be written for more multiple runs!
    2- in function traverse_link or continue_crawl or search_in_files_history new path should be stored in new data structure if:

    • The new fetched link is repetitive
    • The new path has longer chain!
@black-fractal black-fractal self-assigned this Jan 17, 2021
@black-fractal black-fractal added the enhancement New feature or request label Jan 17, 2021
@black-fractal black-fractal mentioned this issue Jan 18, 2021
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@black-fractal black-fractal

Labels
enhancement New feature or request
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@black-fractal