Index Error in get_total_rows

[Coelodonta]

I'm using finviz 1.4.6 with Python 3.10 and Ubuntu 22.04. If I run the sample code from the github README file I get the error below. The same code worked well until last week, when I started getting this error. Suggestions?

The code is simply:

from finviz.screener import Screener

filters = ['exch_nasd', 'idx_sp500']
stock_list = Screener(filters=filters, table='Performance', order='price')

The error is:

Traceback (most recent call last):
File "/home/fq01/finviz_test.py", line 4, in
stock_list = Screener(filters=filters, table='Performance', order='price')
File "/home/fq01/venv/lib/python3.10/site-packages/finviz/screener.py", line 128, in init
self.data = self.__search_screener()
File "/home/fq01/venv/lib/python3.10/site-packages/finviz/screener.py", line 436, in __search_screener
self._rows = self.__check_rows()
File "/home/fq01/venv/lib/python3.10/site-packages/finviz/screener.py", line 404, in __check_rows
self._total_rows = scrape.get_total_rows(self._page_content)
File "/home/fq01/venv/lib/python3.10/site-packages/finviz/helper_functions/scraper_functions.py", line 46, in get_total_rows
etree.tostring(total_element[0]).decode("utf-8").split("")[1].split()[0]
IndexError: list index out of range

[PBbanana]

I can confirm this error.

IDK if it matters, but I get the error even after applying the fix to whitespace characters #168

[PBbanana]

Here is a temporary workaround that worked for me:

change the get_total_rows function to:

def get_total_rows(page_content):
""" Returns the total number of rows(results). """

total_element = page_content.cssselect('td[width="128"]')
total_number = (total_element[0].text.split()[2])

try:
    return int(total_number)
except ValueError:
    return 0

[Coelodonta]

PBbanana,
Unfortunately, that fix didn't work for me. Still getting errors. Thanks anyway!

[vpereira]

@PBbanana and @Coelodonta using the version from git, it works. However it needs the patch from #168 then I'm able to get the table as expected

[AGG2017]

I don't see 'td[width="128"]' in the html content anymore and get_total_rows() fails.
My workaround at the moment is this one:

def get_total_rows(page_content):
    """ Returns the total number of rows(results). """
    import lxml.html
    total_number = str(lxml.html.tostring(page_content)).split('class="count-text">#1 / ')[1].split(' Total</td>')[0]
    try:
        return int(total_number)
    except ValueError:
        return 0

[AGG2017]

I found a solution that works and I already posted it here

[thor-cognitica]

Thanks! This fix seems to work. (At least for now LOL!)