Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

API issues #2

Open
chasealanbrown opened this issue Apr 29, 2020 · 14 comments
Open

API issues #2

chasealanbrown opened this issue Apr 29, 2020 · 14 comments

Comments

@chasealanbrown
Copy link

After realizing that a simple mkdir db command is required (This is a simple fix that can be added to the package), there is some trouble regarding the API.

The input/api.json file shows ..."morningstar.com/api/v2/search/"... being used for the REST API, which doesn't appear to be functional anymore.

Here is the output when running now:

=================================================================
====================== Welcome to msTables ======================

Available actions:

0 - Change database file name (current name = 'mstables.sqlite')
1 - Create database tables and import latest symbols
2 - Download Morningstar data into database
3 - Erase all records from database tables
4 - Delete all database tables
5 - Erase all downloaded history from 'Fetched_urls' table
6 - Create a database back-up file

=================================================================
Enter action no.:
2

Qty. of records to be updated:
1000000

Qty. of records pending update per API no.:

      Pending
API          
1      88,701
2      88,701
3      88,701
4     138,312
5     138,312
6     138,312
7     138,312
8     138,312
9      34,032
10    138,312
11    138,312
12    138,312
13    138,312
14    138,312
15    138,312
16    138,312

Total URL requests pending =	1,959,879
Total URL requests planned =	1,959,879


Run 1 / 817	(150 requests per API per run = 2400 requests per run)
 - Success rate:	0 out of 2,400 (0.0%)
 - Fetch Duration:	48.57 sec
 - Total Duration:	48.59 sec
 - Speed:		0.00 records/sec

Run 2 / 817
 - Success rate:	0 out of 2,400 (0.0%)
 - Fetch Duration:	41.45 sec
 - Total Duration:	41.47 sec
 - Speed:		0.00 records/sec

Run 3 / 817
 - Success rate:	0 out of 2,400 (0.0%)
 - Fetch Duration:	41.82 sec
 - Total Duration:	41.84 sec
 - Speed:		0.00 records/sec

In order to fix this, it seems that a large re-write of the package would be neccessary. This would likely be fruitful, as it appears that authentication is required now for morningstar API access, and it would also make this code far more simple and readable using something such as https://github.com/aaaccell/morningstar in order to make the calls.

One simple and fast suggestion is to provide a link to the resulting sqlite3 file via dropbox or torrent at specified yearly / monthly intervals, in case some users are incapable of creating the database due to changes in the API or other issues.
@joe-wojniak
Copy link

Web scraping isn't necessary. Here's a repo that provides a locally saved dataset:
https://github.com/joe-wojniak/PythonForFinance

@datatalking
Copy link
Contributor

What if we split the difference and create a function that allows the user loads your data from the file function, then a separate function for scraping. I have one like this I use for yahoo finance I could supply.

@datatalking
Copy link
Contributor

@joe-wojniak That is a great repo, I wonder why it was voted down... it seems to point to an empty folder inside of https://github.com/caiobran/mstables which is deprecated.

@chasealanbrown As well written as mstables is I would like to continue working on it as I like the format it is written in. @joe-wojniak if this repo is Archived would you consider forking this and adding me as a contributor so we can add these functions and other for future PR requests or future functionss

@joe-wojniak
Copy link

joe-wojniak commented Aug 18, 2021 via email

@datatalking
Copy link
Contributor

datatalking commented Aug 21, 2021 •
edited
Loading

@joe-wojniak Perhaps I misspoke.

In your Nov 10, 2020 reply you posted a URL https://github.com/joe-wojniak/PythonForFinance which (when I click it) actually opens https://github.com/caiobran/mstables/issues/url.

Is that a mistype in the HTML link?

markdown is a new language format for me so I didn't know if it was a trick or bug.

Screenshot Below.
Screen Shot 2021-08-21 at 10 51 16 AM

@joe-wojniak
Copy link

joe-wojniak commented Aug 21, 2021 via email

@joe-wojniak
Copy link

joe-wojniak commented Aug 21, 2021 via email

@caiobran
Copy link
Owner

Hi all, my apologies for being MIA here. I haven't worked on this project in a couple of years but I am happy to help continue it if you guys are still interested.

@datatalking
Copy link
Contributor

datatalking commented Aug 23, 2021 •
edited
Loading

@caiobran I am interested and wanted to say again that your repo here is Awesome. The last few months data_overview.ipynb has fun to work with and inspiring me with other ideas.

In general, and in a sort of order I want to:

  1. Debug a few errors that data_overview.ipynb has been giving me.
  2. Document your current code a bit more so I can follow all 1500+ lines of code
  3. Figure out how much functionality we lost with the Morningstar API deprecated
  4. What functionality we can add, for example integrating the open-source backtrader or zipline products
  5. Expand on what you made to accept API, CSV, or scrape data feeds like IEX, Yahoo, Trading economics
  6. Add me as a contributor as I've been working on this all summer and learning so much.
  7. @joe-wojniak seems to have a solid understanding and although I've not reviewed all his code there seems to be some overlap that we can all benefit from.
  8. Add in historical trend regression stuff
  9. Lots of potential

Some of these functions are available elsewhere so we don't need to start from scratch, we could integrate pandas data frames, but your menu is such a simple and helpful tool I want to build on your work.

@caiobran
Copy link
Owner

Hi @datatalking, ill be honest, this was a project i worked on to learn python. There are lot of better ways to have built this. I am thinking it might be worth making it into a class object. I got a new job after that and we only use R so i kinda stopped developing in Python but would like to pick it back up again. I also have never collaborated with others on GitHub outside of work so I think it might be worth having a call to discuss a plan on how we can work together.

With all that said, i totally agree with your points and would be more then happy to add you as a collaborator. Let me know if you are open to having a chat. I am in San Diego so I'm in Pacific time.

@joe-wojniak
Copy link

joe-wojniak commented Aug 23, 2021 via email

@datatalking
Copy link
Contributor

Is morning star worth web scraping though? I've been helping and digging into issues for Pandas datareader and really like that setup. Would it makes sense to keep the scraping feature or change this to accept pandas? I'll dig into options, I need the widest net of detailed data, but intraday numbers is my current goal.

I really like how this is saved to a database, there is a similar database I'm building for practice in postgres for securities so there is lots to build.

I've just upgraded my main machine to a 12 core processor tonight with 64 gig of ram and a GPU that I'm integrating has about 2000 cores so i am excited to work on intensive processing.

Seattle here, usually I'm up at 5am for the market till around 10pm, would this Saturday or Friday work for you? [email protected] is a good email.

@joe-wojniak
Copy link

joe-wojniak commented Aug 25, 2021 via email

@datatalking
Copy link
Contributor

@chasealanbrown I solved (sort of) the api issue as we can now scrape a few of the Morningstar data points. I Will work with @caiobran to issue a PR soon.
@joe-wojniak lets move the hardware thread to a separate issue.

@chrisidso chrisidso mentioned this issue Feb 10, 2022
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@datatalking @joe-wojniak @caiobran @chasealanbrown