-
Notifications
You must be signed in to change notification settings - Fork 5
Aaron Log
Week 1 (8/31 - 9/4): Attempting to schedule an initial meeting with the rest of the team. May get pushed to next week due to conflicting schedules. On the side I have been performing some research on frameworks and APIs.
Week 2 (9/7 - 9/11): Have had 2 meetings so far. One to catch up on the Form Teams Milestone and another to finish the Personas and User Stories Milestone. Personally, I wrote the Marshall Presley Persona, created the wiki pages for the Form Teams Milestone, and have begun testing various potential APIs for stock price grabbing and trend data. For stock price grabbing, I have tested Alpha Vantage (not enough queries per day), pandas-datareader (not live data), and Yahoo Finance. Currently, I’m leaning towards Yahoo Finance but I want to do more research since it has been taken down before (from 2017 - 2019). For trend data I have only tested pytrends.
Week 3 (9/14 - 9/18): None.
Week 4 (9/21 - 9/25): Researched Finnhub API. This seems like our best current option since it is the first free one I have found with a minute limit instead of a daily limit (30 API calls per minute). Collaborated with the rest of the team for the design milestone. Personally, I made the final design using FrameWorks and discussed options for our rough design.
Week 5 (9/28 - 10/2): Read through the guide on how to create a web app in Django that was provided by Professor Vidal. My progress building an app is in this Repo. I have also been working on redesigning the app using FrameWorks to account for our change from an iOS application to a web app.
Week 6 (10/5 - 10/9): Finished my research milestone which can be found at this Repo. I also met with the team to discuss the requirements milestone. Personally, I reformatted the requirements wiki page to better organize the information.
Week 7 (10/12 - 10/16): None.
Week 8 (10/19 - 10/23): Modified README for the Source Control Milestone. Met with team to discuss a plan of action as we move through the rest of the semester. Currently, I am in charge of finding and testing potential APIs. The main one to use currently is Finnhub. I have been working on creating some tests for Finnhub since it is our best bet at having current data with enough API calls per minute.
Week 9 (10/26 - 10/30): Continued research into using Finnhub. During this testing I actually found some really interesting functionality. So far I found out that when using a websocket to obtain current stock data, those pings made to the Finnhub server do not count towards our API call limit. The downside to this however is that it appears that the stock will only update on the websocket when someone else has used an API call to request that specific stock's price. This means that large, highly watched stocks (Apple, Google, Microsoft, etc.) get updated constantly. While smaller stocks do not get updated. This is a very important bit of functionality to know because it means that we do not have to manually update only specific stocks based off of how many people want to view them, it'll just automatically do it for us. This does mean though, that we'll have to add a button to the stocks detailed view which requests a stocks price (and subsequent prediction). Since this would count towards our API calls we'll need a way to regulate the number of calls.
Week 10 (11/2 - 11/6): Have performed more stress tests on Finnhub. I verified that the websocket does not count towards our API call limit by adding more than 30 stocks to one websocket and verify that the number of responses does go past 30 calls / minutes. The next step will be to have a program which performs both simultaneously. I plan to do this by using a library called "django-background-tasks" to have a daemon worker which will constantly run the websocket and another worker which will run a request whenever one is requested. I am performing these tests to ensure that:
- Users do not have to wait when a request is made (or in other words that it can be made asynchronously).
- The websocket and request worker combo does not drastically impede server performance.
I will be doing more tests on these next week, hopefully by the end of it I can have some good functionality concretely setup.
Week 11 (11/9 - 11/13): Have downloaded approximately 180k data-points to use for training our neural network. Currently, the data-points only have very simple daily stock data (high, low, open, close, adjusted-close, volume, and date). So, we'll need to add more data to each of these (YTD%, week high, week low, 52 week high, 52 week low, googletrends data, etc.).
Week 12 (11/16 - 11/20): Created skeleton code for requesting stock prices, polling finnhub, and requesting media buzz. We may decide to remove pytrends from our third-party dependencies. This would be because I believe that pytrends is a web-scraping based API and if we can use less third-party software the better our end result will be. Side note, this project would be made much simpler by purchasing the premium versions of Finnhub. With the $100/month product we could create a websocket which polls stock prices and breaking news from every stock, have 300 API calls/minute, and have access to 25 years of market data. Still need to test that our polling background tasks can be run in concurrence with our server. I feel as though that point is ambiguous in django-background-tasks's documentation.
Week 13 (11/23 - 11/27): None. Happy Thanksgiving.
Week 14 (11/30 - 12/4): Prepped for PoC. Met with team to discuss current state of project. Coordinated to determine work done over winter break.
Week 15 (12/7 - 12/11): Helped Max deploy the application using Heroku. Finalized Pipfile for release.
Week 16 (1/11 - 1/15): Updated group availability sheet for the new semester.
Week 17 (1/18 - 1/22): Coordinated with team to schedule the first meeting of this semester.
Week 18 (1/25 - 1/29): Continued finding resources for showing prediction lines.
Week 19 (2/1 - 2/5): None.
Week 20 (2/8 - 2/12): None.
Week 21 (2/15 - 2/19): Sent a formal request to TradingView for access to their charting library. Have wrote a PineScript program to be able to draw predictions onto TradingView charts. Cannot implement this script for our widgets until I receive access from TradingView. Although, I do have the script privately created and working.
Week 22 (2/22 - 2/26): Finished finding issues with adding TradingView widgets into the react environment. Currently it does not seem like react works well with inline script tags. Hopefully, all visual bugs can be fixed by the release. Integrated @ahmedSC's work into the django webpage, spent ~8 hours rewriting code, finding bugs, fixing bugs, configuring Django, and re-configuring Heroku in an attempt to get a minimal working version of the app deployed.
Week 23 (3/1 - 3/5): Scheduling a meeting with as many team members as possible this week. Will be finding suitable roles for everyone and ensuring that we do not have to do an all-night deployment again.
Week 24 (3/8 - 3/12): Officially received access to TradingView's Charting Library. Began reading their documentation and finding some tutorials.
Week 25 (3/15 - 3/19): None.
Week 26 (3/22 - 3/26): A lot of progress has been made with integrating TradingView's Charting library.
- Added their official chart as opposed to their widget.
- Added the prediction lines to the chart.
- Created a streaming datafeed to supply TradingView's Library with data.
- Added a Finnhub Authentication token to allow our website to not become broken by CORS.
- Forced TradingView's chart to update whenever a prediciton is changed.
- Replaced TradingView's Single Ticker widget with a custom StockList element so we can have full control over routing.
- Added ability to give prediction to StockList component (currently very buggy, predictions get swapped between elements or don't become placed at all).
Week 27 (3/29 - 4/2): Removed TradingView widget from backend (just a remnant from when all work was done on Django), worked on creating a better .gitignore. Created a simple Autocomplete (currently buggy, will be fixed later).
Week 28 (4/5 - 4/9): None.
Week 29 (4/12 - 4/16): None.
Week 30 (4/19 - 4/23): Refined Search bar autocomplete.
Week 31 (4/26 - 4/30): Reverted some changes which may have added potential bugs. Substituted investpy with yfinance. While I do not like the idea of relying on a scraping stock API it has become necessary with the failure of investpy.
..