CHANGELOG

BDEv Changelog
-------------------------------------
This file briefly describes the main changes for each version of
the Big Data Evaluator (BDEv) software package.

BDEv 3.9 - 2024-04-11
---------------------
	A point release for the BDEv 3.x line

	NEW FEATURES
		- Updated versions of the following frameworks
			- Hadoop 3.3.6
			- Spark 3.2.4
			- Spark 3.3.4
			- Flink 1.14.6
			- Flink 1.15.4

		- Support for the Hadoop 3.4.x line

		- Support for the Spark 3.4.x and 3.5.x lines
				
		- Support for the Flink 1.16.x, 1.17.x, 1.18.x and 1.19.x lines

		- Add HDFS parameter to configure DataNode heartbeat (hdfs-default.sh)

		- Add YARN parameter to configure the scheduler class (yarn-default.sh)

		- Add several options to customize the YARN fair scheduler (yarn-default.sh)
		
		- Add YARN parameter to configure NodeManager heartbeat (yarn-default.sh)

		- Configuration of each experiment is now copied into the output report directory

		- HDFS formatting and data deletion can be configured (hdfs-default.sh)

		- HDFS replication factor is automatically set when there are not enough DataNodes

		- HDFS input data is reused if already exist

		- Update RAPL-based power monitoring tool to PAPI v7.1.0

		- Upate RGen data generator to include minor improvements and bug fixes

		- Update resource monitoring tool to dool v1.3.1
		
		- SSH optional parameters can now be set (bdev-default.sh)
		
		- Fix bug when running Spark and Flink on YARN
				
		- Remove support for Flink versions 1.12.x and 1.13.x

		- Minor changes and bug fixes


BDEv 3.8 - 2022-07-29
---------------------
	A point release for the BDEv 3.x line

	NEW FEATURES
		- Updated versions of the following frameworks
			- Hadoop 2.10.2
			- Hadoop 3.2.4
			- Hadoop 3.3.3
			- Spark 3.2.2
			- Flink 1.14.5
		
		- Support for the Spark 3.3.x line
				
		- Support for the Flink 1.15.x line
		
		- Add new sorting benchmark for Hadoop, Spark and Flink: TPCx-HS

		- Update RGen data generator to support TPCx-HS (HSGen) and fix TeraGen bug with Hadoop 3.X
				
		- Hadoop classpath is now automatically configured for Spark and Flink
	
		- Add option to use hostnames or IPs for cluster nodes (bdev-default.sh)

		- Add Hadoop parameter for setting timeout on waiting response from server (core-default.sh)
				
		- Add HDFS parameter to configure write packet size (hdfs-default.sh)

		- Add HDFS parameter to configure retries when writing blocks to DataNodes (hdfs-default.sh)
		 
		- Add HDFS parameter to configure handler count for DataNodes (hdfs-default.sh)

		- Add option to set an IP for loopback interface (system-default.sh)

		- Improve frameworks cleanup
		
		- Add sanity checks for networking setup
				
		- Remove support for Flink versions 1.11.x
		
		- Minor changes and bug fixes


BDEv 3.7 - 2022-03-18
---------------------
	A point release for the BDEv 3.x line

	NEW FEATURES
		- Updated versions of the following frameworks
			- Flink 1.11.6
			- Flink 1.12.7
			- Flink 1.13.6
			- Spark 3.1.3
			- Hadoop 3.3.2

		- Support for the Spark 3.2.x line

		- Support for the Flink 1.14.x line

		- Add Flink configuration parameters for several shuffle settings

		- Remove support for Flink versions 1.10.x
		
		- Add sanity check for Flink configuration file

		- Update resource monitoring tool to dool v1.0.0

		- Minor changes and bug fixes


BDEv 3.6 - 2021-07-16
---------------------
	A point release for the BDEv 3.x line

	NEW FEATURES
		- Updated versions of the following frameworks
			- Hadoop 2.10.1
			- Hadoop 3.1.4
			- Hadoop 3.2.2
			- Hadoop 3.3.1
			- Spark 2.4.8
			- Spark 3.0.3
			- Flink 1.10.3
			- Flink 1.11.3

		- Support for the Spark 3.1.x line

		- Support for the Flink 1.12.x and 1.13.x lines

		- Add Spark configuration parameters for:
			- spark.memory.fraction
			- spark.memory.storageFraction
			- spark.kryoserializer.buffer.max
			- spark.executor.heartbeatInterval

		- Adaptative Query Execution (AQE) can be enabled and configured for Spark 3.x

		- Add Flink configuration parameters for several network memory settings and timeouts

		- Add HDFS configuration parameters for setting socket-related timeouts

		- Command batch mode now supports running multiple scripts stored in a directory

		- Remove support for Flink versions <= 1.9

		- All input datasets are now created using an internal data generator tool (RGen)

		- Set Hadoop classpath properly

		- Add sanity checks for some dependencies

		- Fix missing dool files

		- Minor changes and bug fixes


BDEv 3.5 - 2020-09-04
---------------------
	A point release for the BDEv 3.x line

	NEW FEATURES
		- Updated versions of the following frameworks
			- Spark 2.4.6
			- Flink 1.9.3
			- Flink 1.10.2

		- Support for the Hadoop 3.3.x line

		- Support for the Spark 3.0.x line

		- Support for the Flink 1.11.x line

		- Update resource monitoring tool to dool v0.9.9, a Python 3 compatible clone of dstat

		- Update RAPL-based power monitoring tool to use PAPI v6.0.0

		- Update iLO scripts to version 5.30

		- RAPL-based monitoring tool now can report readings for uncore devices if PP0 is available

		- Fix bug when setting iLO credentials

		- Minor bug fixes


BDEv 3.4 - 2020-03-25
---------------------
	A point release for the BDEv 3.x line

	NEW FEATURES
		- Updated versions of the following frameworks
			- Spark 2.4.5
			- Flink 1.8.3
			- Flink 1.9.2

		- Support for Flink 1.10.x line

		- Allow to start the job history server for Flink

		- Simplified and improved cluster deployments for Spark & Flink

		- Add support to configure the YARN Timeline server (yarn-default.sh)

		- Add support to configure the disk health checker service for YARN (yarn-default.sh)
		
		- Add support to configure last access time precision for HDFS (hdfs-default.sh)

		- Add support to enable short-circuit local reads for HDFS (hdfs-default.sh)

		- Memory settings for some daemons (eg, DataNode) can now be set individually

		- Fix SQL and ConnComp workloads for Spark

		- Spark & Flink benchmarking suites are now downloaded on demand (removed from git repo)

		- Scala version for Spark & Flink can be set in solutions-default.sh (2.11 by default)

 		- Set LD_LIBRARY_PATH in Spark & Flink to include the path to Hadoop native libraries

		- Hadoop existence is checked when deployed together with other frameworks

		- Minor bug fixes


BDEv 3.3 - 2019-12-17
---------------------
	A point release for the BDEv 3.x line

	NEW FEATURES
		- Updated versions of the following frameworks
			- Hadoop 2.10.0
			- Hadoop 3.1.3
			- Hadoop 3.2.1
			- Spark 2.3.4
			- Spark 2.4.4
			- Flink 1.8.2

		- Support for the Flink 1.9.x line (tested with Flink 1.9.1)

		- Add support for BDWatchdog tool, which allows per-process resource monitoring and JVM profiling
			- More info at http://bdwatchdog.dec.udc.es

		- BDEv now runs by default in command mode when no framework is selected in solutions.lst

		- Update PAPI library to version 5.7.0

		- Update RAPL power monitoring tool to use PAPI 5.7.0

		- Add support to configure the binary names for Python2 and Python3 (system-default.sh)

		- Fix bug when running Flink in standalone mode

		- Minor bug fixes


BDEv 3.2 - 2019-06-06
---------------------
	A point release for the BDEv 3.x line

	NEW FEATURES
		- Updated versions of the following frameworks
			- Hadoop 2.7.7
			- Hadoop 2.8.5
			- Hadoop 2.9.2
			- Hadoop 3.1.2
			- Hadoop 3.2.0
			- RDMA-Hadoop-3 0.9.1
			- Flame-MR 1.2
			- Spark 2.3.3
			- Spark 2.4.3
			- Flink 1.7.2
			- Flink 1.8.0

		- Add support to configure handler count for Hadoop NameNode (hdfs-default.sh)

		- Add support to use unsafe based Kryo serializer for Spark (solutions-default.sh)

		- Fix bug when running multiple Executors per worker/node in Spark configuration

		- Minor bug fixes


BDEv 3.1 - 2018-05-16
---------------------
	A point release for the BDEv 3.x line

	NEW FEATURES
		- Updated versions of some frameworks
			- Hadoop 2.7.6
			- Hadoop 2.8.3
			- Hadoop 2.9.0
			- Flame-MR 1.1
			- RDMA-Hadoop-2 1.3.5
			- Spark 2.3.0
			- RDMA-Spark 0.9.5
			- Flink 1.4.2

		- Support for the Hadoop 3.1.x line (tested with Hadoop 3.1.0)

		- Improved customization of experiments with setup and clean up commands

		- Extended configuration parameters
		

BDEv 3.0 - 2017-11-07
--------------------
	This is a major release which contains a number of significant 
	enhancements, along with new characteristics previously unreleased.

	NEW FEATURES

		- New monitoring tools
			- Power monitoring via RAPL
			- Microarchitecture-level event counting via Oprofile

		- Enhanced framework configuration parameters

		- Support for the Flink 1.3.x line (tested with Flink 1.3.2)

		- Support for the Spark 2.2.x line (tested with Spark 2.2.0)

		- Support for the Flame-MR 1.x line (tested with Flame-MR 1.0)

		- Updated versions of some frameworks
			- Hadoop 2.7.4
			- Hadoop 2.8.1
			- RDMA-Hadoop 1.2.0
			- RDMA-Spark 0.9.4

		- Updated version of resource monitoring tool to dstat 0.7.3

		- Improved graph generation portability


BDEv 2.3 - 2016-11-25
---------------------
	A point release for the BDEv 2.x line

	NEW FEATURES

		- Added support for Flame-MR version 0.10.0

		- Added support for the Flink 1.1.x line (tested with version 1.1.3)

		- Added support for the SLURM job scheduler

		- Updated versions of some frameworks
			- Hadoop 2.6.5
			- Hadoop 2.7.3
			- RDMA-Hadoop-2 1.1.0
			- Spark 1.6.3
			- Flink 1.0.3

		- Memory settings for mappers and reducers can now be set separately

		- Added configuration variable in etc/core-default.sh to control the 
		Hadoop parameter "io.file.buffer.size"

		- Virtual and physical memory limits in YARN containers as well as the 
		ratio between them can now be configured in etc/yarn.default.sh

		- Fix critical bug when running Flink under YARN related with JobManager
		memory

		- Minor bug fixes


BDEv 2.2 - 2016-07-04
---------------------
	A point release for the BDEv 2.x line

	NEW FEATURES

		- Add support for multiple Workers/TaskManagers per node in Spark and Flink

		- Enhancements in several Flink workloads
			- Grep
			- PageRank (with delta iterations)
			- Connected Components (with Gelly)
			- K-Means

		- Enhancements in several Spark workloads
			- TeraSort
			- K-Means (with MLlib)

		- Bug fixes


BDEv 2.1 - 2016-05-05
---------------------
	A point release for the BDEv 2.x line

	NEW FEATURES

		- Allow to start the job history server for Spark-based frameworks

		- Added support for some new frameworks
			- Hadoop 2.6.4
			- RDMA-Hadoop-2 0.9.9
			- Spark 1.6.1
			- Flink 1.0.2

		- Added support for Mahout 0.11.2 and 0.12.0

		- Added new benchmarks for Spark
			- PageRank
			- Connected Components
			- K-Means
			- Bayes
			- Hive SQL queries:
				- Aggregation
				- Join
				- Scan

		- Added new benchmarks for Flink
			- PageRank
			- Connected Components
			- K-Means

		- Multiple bug fixes


BDEv 2.0 - 2016-03-02
---------------------
	This is a major release which contains a number of significant 
	enhancements. Starting with this release, the MapReduce Evaluator
	(MREv) tool has been renamed to Big Data Evaluator (BDEv).

	NEW FEATURES

		- Added support for Flink version 0.10.2 on YARN and standalone modes

		- Added support for RDMA-Spark version 0.9.1 on YARN and standalone modes

		- Added support for Spark on standalone mode

		- Updated versions of some frameworks
			- Hadoop 2.7.2
			- RDMA-Hadoop-2 0.9.8
			- Spark 1.5.2
			- Spark 1.6.0

		- New data set generation using the HiBench DataGen tool

		- Added new benchmarks for Hadoop-based frameworks
			- Grep
			- K-Means
			- Connected Components
			- Hive SQL queries:
				- Aggregation
				- Join
				- Scan

		- Added new benchmarks for Spark and Flink
			- WordCount
			- Sort
			- Grep
			- TeraSort

		- Added new benchmark for DataMPI
                        - Grep

		- Allow to start the job history server for Hadoop-based frameworks

		- Input datasize for WordCount, Sort and TeraSort can now be different

		- Enhanced configuration of parameters 
			- Separated configuration files for HDFS, MapReduce and YARN
			- Solution-specific parameters in a separated file
			- Many new configuration parameters

		- Support for multiple disks

		- Network interfaces are now optional parameters. If not specified, IPs
		are determined using the hostfile

		- Updated iLO scripts to version 4.70

		- Allow to configure IP, user name and password for iLO interfaces

		- Automatically download Apache Mahout and Apache Hive on demand

		- Optimized code refactorization and simplified internal configuration

		- Multiple bug fixes


MREv 1.1 - 2015-08-10
---------------------
	A point release for the MREv 1.x line

	NEW FEATURES

		- Updated versions of some frameworks
			- Hadoop-2.7.1-GbE
			- Hadoop-2.7.1-IPoIB
			- Hadoop-2.7.1-UDA
			- RDMA-Hadoop-2-0.9.7-GbE
			- RDMA-Hadoop-2-0.9.7-IPoIB
			- Spark-1.4.1-YARN-GbE
			- Spark-1.4.1-YARN-IPoIB

		- Configurable timeout for the workloads

		- User-defined batch commands

		- Separate configuration and log directories for each evaluation

		- Enhanced resource configuration


MREv 1.0 - 2014-12-09
---------------------
	The first public release of MREv (MREv 1.0)