This repository contains the homework assignments from the "Big Data Computing" course as part of the Master's degree in Data Science at the University of Padova. These assignments were completed as a teamwork collaboration.
The main focus of these homework assignments was to explore and utilize parallelism using Apache Spark.
Implement and test in Spark two MapReduce algorithms to count the number of distinct triangles in an undirected graph.
- Algorithm 1 (MR_ApproxTCwithNodeColors) returns an approximation of the number of triangles, and uses a hash function for dividing the edges deterministically into partitions.
- Algorithm 2 (MR_ApproxTCwithSparkPartitions) returns the precise number of triangles, and utilizes the partitions provided by Spark.
Run a Spark program on the CloudVeneto cluster. As for HW 1, the objective is to estimate (approximately or exactly) the number of triangles in an undirected graph, using two new algorithms.
2_MapReduce_CloudVenetoCluster.py
Use the Spark Streaming API to devise a program which processes a stream of items and assesses experimentally the space-accuracy tradeoffs featured by the count sketch to estimate the individual frequencies of the items and the second moment F2.
| Dejan Dichoski | Marija Cveevska |
Feel free to explore, learn, and adapt these assignments for your own projects. Happy coding and data crunching!