CA675-Assignment-1-20211114

Assignment 1: Data Analysis

Documentation files:

1. Assignment Documentation.pdf
2. Complete chronological screenshots.pdf

Source Code files:

Stack Exchange Query:

Data extraction query codes from Stack Exchange.txt

Pig:

Pig ETL Source Code.txt

Hive:

Hive Query Source Code.txt

MapReduce for TF-IDF:

Hadoop source code for TF-IDF.txt

mapper1.py

mapper2.py

mapper3.py

mapper4.py

reducer1.py

reducer2.py

reducer3.py

Source for mapper and reducer scripts : https://github.com/devangpatel01/TF-IDF-implementation-using-map-reduce-Hadoop-python-.git

HDFS Commands:

General HDFS Commands Source Code.txt

Results:

Extracted data from Stack Exchange.zip

ETL Transformations using Pig.zip

Data queried from Hive.zip

TF-IDF calculated using MapReduce.zip