Screener

[24thronin]

Describe bug

Hello,
I have been trying to find a way to create a list of all securities on yahoo finance. I thought that screener with a custom query would be able to do this but the screener.response only returns up to 250 results. Is there a way to retrieve all values?
Thank you for the package.

Simple code that reproduces your problem

import yfinance as yf
yf.enable_debug_mode()
r1 = yf.EquityQuery('eq', ['sector', 'Basic Materials'])
r2 = yf.EquityQuery('eq', ['region', 'us'])
crit = yf.EquityQuery('and', [r1,r2])
screener.set_default_body(crit,size=10000)
res=screener.response

Debug log

DEBUG Entering post()
DEBUG Entering _make_request()
DEBUG url=https://query2.finance.yahoo.com/v1/finance/screener
DEBUG params={'corsDomain': 'finance.yahoo.com', 'formatted': 'false', 'lang': 'en-US', 'region': 'US'}
DEBUG Entering _get_cookie_and_crumb()
DEBUG cookie_mode = 'basic'
DEBUG Entering _get_cookie_and_crumb_basic()
DEBUG reusing cookie
DEBUG reusing crumb
DEBUG Exiting _get_cookie_and_crumb_basic()
DEBUG Exiting _get_cookie_and_crumb()
DEBUG response code=400
DEBUG toggling cookie strategy basic -> csrf
DEBUG Entering _get_cookie_and_crumb()
DEBUG cookie_mode = 'csrf'
DEBUG Entering _get_crumb_csrf()
DEBUG Failed to find "csrfToken" in response
DEBUG Exiting _get_crumb_csrf()
DEBUG toggling cookie strategy csrf -> basic
DEBUG Entering _get_cookie_and_crumb_basic()
DEBUG loaded persistent cookie
DEBUG reusing cookie
DEBUG crumb = '5CmrIa/1bJ9'
DEBUG Exiting _get_cookie_and_crumb_basic()
DEBUG Exiting _get_cookie_and_crumb()
DEBUG response code=400
DEBUG Exiting _make_request()
DEBUG Exiting post()
ERROR Failed to get screener data for '{'operator': 'AND', 'operands': [{'operator': 'EQ', 'operands': ['sector', 'Basic Materials']}, {'operator': 'EQ', 'operands': ['region', 'us']}]}' reason: 400 Client Error: Bad Request for url: https://query2.finance.yahoo.com/v1/finance/screener?corsDomain=finance.yahoo.com&formatted=false&lang=en-US&region=US&crumb=5CmrIa%2F1bJ9
DEBUG Got response:
DEBUG -------------
DEBUG None
DEBUG -------------

Bad data proof

No response

yfinance version

0.2.50

Python version

No response

Operating system

No response

[ericpien]

Hi, the 250 is a limit set by Yahoo, over which we have no control.

You could try iterating (~11 times for your query) the full set via "offset" param in the screener body:

import yfinance as yf
# yf.enable_debug_mode()
r1 = yf.EquityQuery('eq', ['sector', 'Basic Materials'])
r2 = yf.EquityQuery('eq', ['region', 'us'])
crit = yf.EquityQuery('and', [r1,r2])

increment = 250
screener = yf.Screener()
screener.set_default_body(crit, size=increment)
screener.response

screener.patch_body({"offset": increment})
screener.response

[24thronin]

Thank you, this is very helpful! Adding a comment about this would be helpful for the documentation.