Major-Project

A central repo for code, sharing resources, workflows and roadmaps.Check the Projects section for project managemet plans.

Roadmap

0. Project proposal (In development)
1. Scrape data from all sites mentioned below. (In development)
2. Clean the data and remove unnecessary variables and noise. (In development)
3. Perfrom EDA and find out the relationships between variables. (Pending)
4. Build a model to predict/give an objective price of the house based on the parameters. (Pending)
5. Deploy the model using flask(Api), HTML, CSS, JS and PHP/Laravel(Backend). (Pending)
6. Documentation (Pending)

INFORMATION AND RESOURCES

Folder heirarchy

- major-project
  - csv_files
    - raw (Raw unclean data after scraping)
    - links (Links to be scrapped)
    - clean (Cleaned data)
- jupyter_notebooks (Notebooks to clean data and develop the model)
- scraping_files

Tutorial resources

• Hosing price prediction from towards data science
• Coursera linear regression Only good for conceptual/theoritcal learning. Uses outdated libraries.
• Vectorizaiton - Important for understanding numpy and pandas.
• Gradient descent - Important for understanding linear regression.
• Real estate prediction - For brief overview of how the project will work.
• First Class functions - How python's functions can be treated as objects.
• For Numpy, Pandas and Matplotlib
• Google's machine learning course
• Pip Python package manager
• Setting up python virtual env in VSCode
• Data Science Handbook
• Mathematics for machine learning Book
• Google Colab - Online GPU for extra power.

Important programming concepts

• creating and importing a python module and using its functions (files)
• namespace of functions in a python module
• how functions from other python modules are called
• Python data types: Lists, Tuples, dictionary and sets
• for loops, map function
• method call from a library vs function call ie how obj.method() works
• lambda function/closures/annonyomous function
• Python virtual environmets
• Basics of Object oriented python. Python class constructors. How a function gets access to the object it is called upon and how it manipulates the object.
• What is anacodna?
• How pip works

Important Mathematics and Statistics Concepts (Add more)

• Co-relation between variables and multicollinearity.
• Linear/Multiple/Logistic reression
• Squared error of regression line
• p-value, level of significance, null and alternate hypothesis.
• contour plot
• Fit a line using least square method.
• Vectors, Partial derivative, gradient of a function and gradient descent.

Important diagrams

• Scatter plot
• Heatmap - To find multicollinearity
• Box plot
• Histograms
• Violin plot

Important Machine learning Comncepts

• One hot encoding
• Dummy variable trap
• Co-relation does not mean causation

Algorithms (listed possible best to worst)

1. Ridge regression - Because the data seems to have multicollinearity
2. Linear regression - Related = Wald's test
3. Logistic regression ??
4. Convolution neural networks
5. Random forest
6. ID3

Use waka as a testing tool and may use external librariess to check efficiency of the model.

Miscellaneous

• Using VSCode commmand pallete
• The Spyder IDE - Optional for now
• Markdown - Very important for editing and adding resourses.
• Land measurement - Details about the different units of measurement of land.
• Creating checkmarks in projects
• How Much Training Data is Required for Machine Learning? (Article)
• Install jupyter lab using conda for notebook dark theme:

Libraries (Must read official docs)

Go through the basic contents of the docs atleast once

• Numpy (Vectorization and broadcasting in Numpy)
• Pandas
• Seaborn
• Matplotlib

Websites to scrape

S.N.	Name	Data Amount	Library required	Status	Remarks
1	99aana	3K	BS4	Completed
2	Nepal Homes	1K	Selenium & BS4	Links fetched	Data to be fetched using BS4
3	Basobaas	2K	Selenium & BS4	Links fetched	Data to be fetched using BS4
4	1Ropani	600	Selenium & BS4	Completed
5	Hamrobazar	3K		Halted	Presents a captcha to check for bots
6	Gharbazar	340	Selenium	Links fetched	Low amount of data. Scan for house keyword in title.
7	Gharghaderi	300	Selenium	Halted	Low amount of data
8	Housing Nepal	less than 300	BS4	Halted	Low amount of data
9	Real Estate In Nepal	less than 300	BS4	Halted	Low amount of data. Restriction for bots
10	Nepal Home Search	140	BS4	Halted	Low amount of data
11	Nepal Realestates	less than 300	BS4	Halted	Low amoun of data.
12	The Realtors	300	Selenium & BS4	Halted	Low amount of data. Scan title or house keyword
13	GharJagga Nepal	330	Selenium	Halted	Low data and has infinite scrolling

Parameters (subject to change)

1. Price
2. Location in District
3. Number of rooms
4. Number of floors
5. Area
6. Time posted
7. Road size and road type*
8. Room size *
9. Bedrooms *
10. Bathrooms *
11. Car parking *
12. kitchen *
13. Living room *
14. Garage
15. Furnished ? *
16. Guestroom *

Naming convention

• Files/Modules and variables = snake_case
• directories = all lowercase preferably without underscores
• classes = PascalCase