handle yahoo US collecotr api limit issue (Fix #1953) #1970
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Fix eastmoney API pagination for US stock data collection
Description
Fixed pagination issue in
_get_eastmoney()
function that was causing "requesterror" when collecting US stock symbols. Changed from requesting 10,000
symbols per page to proper pagination with 100 symbols per page, iterating
through all pages until completion.
Changes made:
pz
parameter from10000
to100
(page size)Motivation and Context
Related Issue: eastmoney API now limits page size to 100 symbols maximum,
but the original code was trying to fetch 10,000 symbols in a single request.
Problem: When running
python collector.py download_data --source_dir ~/.qlib/stock_data/source/us_data --start 2020-01-01 --end 2020-12-31 --delay 1 --interval 1d --region US
, it failed with "request error" becauselen(_symbols) < 8000
.Root Cause: The eastmoney API
http://4.push2.eastmoney.com/api/qt/clist/get
was returning only 100 symbolsinstead of the requested 10,000, triggering the validation error.
How Has This Been Tested?
API Endpoint Testing:
Test Commands Used: