Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

extend the four-component to every request #808

Open
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

Henior
Copy link

@Henior Henior commented Jul 9, 2018

对每一个请求,进行可定制化配置,即每个请求包含自己的downloader,processor,scheduler和pipeline,
如果没有设置则走默认的
举个例子,当你在爬虫的时候,有的请求只需要httpclient进行简单的页面请求,但同时有的请求却要进行phantomjs的本地js操作,这种情形下为了拥有js操作的功能,我们只能选后者的downloader,即使逻辑复杂化又降低了性能(仿浏览器操作比简单的网络请求消耗更多资源),代码耦合性也很高

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant