-
Notifications
You must be signed in to change notification settings - Fork 0
/
CHANGELOG
529 lines (341 loc) · 13 KB
/
CHANGELOG
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
BDEv Changelog
-------------------------------------
This file briefly describes the main changes for each version of
the Big Data Evaluator (BDEv) software package.
BDEv 3.9 - 2024-04-11
---------------------
A point release for the BDEv 3.x line
NEW FEATURES
- Updated versions of the following frameworks
- Hadoop 3.3.6
- Spark 3.2.4
- Spark 3.3.4
- Flink 1.14.6
- Flink 1.15.4
- Support for the Hadoop 3.4.x line
- Support for the Spark 3.4.x and 3.5.x lines
- Support for the Flink 1.16.x, 1.17.x, 1.18.x and 1.19.x lines
- Add HDFS parameter to configure DataNode heartbeat (hdfs-default.sh)
- Add YARN parameter to configure the scheduler class (yarn-default.sh)
- Add several options to customize the YARN fair scheduler (yarn-default.sh)
- Add YARN parameter to configure NodeManager heartbeat (yarn-default.sh)
- Configuration of each experiment is now copied into the output report directory
- HDFS formatting and data deletion can be configured (hdfs-default.sh)
- HDFS replication factor is automatically set when there are not enough DataNodes
- HDFS input data is reused if already exist
- Update RAPL-based power monitoring tool to PAPI v7.1.0
- Upate RGen data generator to include minor improvements and bug fixes
- Update resource monitoring tool to dool v1.3.1
- SSH optional parameters can now be set (bdev-default.sh)
- Fix bug when running Spark and Flink on YARN
- Remove support for Flink versions 1.12.x and 1.13.x
- Minor changes and bug fixes
BDEv 3.8 - 2022-07-29
---------------------
A point release for the BDEv 3.x line
NEW FEATURES
- Updated versions of the following frameworks
- Hadoop 2.10.2
- Hadoop 3.2.4
- Hadoop 3.3.3
- Spark 3.2.2
- Flink 1.14.5
- Support for the Spark 3.3.x line
- Support for the Flink 1.15.x line
- Add new sorting benchmark for Hadoop, Spark and Flink: TPCx-HS
- Update RGen data generator to support TPCx-HS (HSGen) and fix TeraGen bug with Hadoop 3.X
- Hadoop classpath is now automatically configured for Spark and Flink
- Add option to use hostnames or IPs for cluster nodes (bdev-default.sh)
- Add Hadoop parameter for setting timeout on waiting response from server (core-default.sh)
- Add HDFS parameter to configure write packet size (hdfs-default.sh)
- Add HDFS parameter to configure retries when writing blocks to DataNodes (hdfs-default.sh)
- Add HDFS parameter to configure handler count for DataNodes (hdfs-default.sh)
- Add option to set an IP for loopback interface (system-default.sh)
- Improve frameworks cleanup
- Add sanity checks for networking setup
- Remove support for Flink versions 1.11.x
- Minor changes and bug fixes
BDEv 3.7 - 2022-03-18
---------------------
A point release for the BDEv 3.x line
NEW FEATURES
- Updated versions of the following frameworks
- Flink 1.11.6
- Flink 1.12.7
- Flink 1.13.6
- Spark 3.1.3
- Hadoop 3.3.2
- Support for the Spark 3.2.x line
- Support for the Flink 1.14.x line
- Add Flink configuration parameters for several shuffle settings
- Remove support for Flink versions 1.10.x
- Add sanity check for Flink configuration file
- Update resource monitoring tool to dool v1.0.0
- Minor changes and bug fixes
BDEv 3.6 - 2021-07-16
---------------------
A point release for the BDEv 3.x line
NEW FEATURES
- Updated versions of the following frameworks
- Hadoop 2.10.1
- Hadoop 3.1.4
- Hadoop 3.2.2
- Hadoop 3.3.1
- Spark 2.4.8
- Spark 3.0.3
- Flink 1.10.3
- Flink 1.11.3
- Support for the Spark 3.1.x line
- Support for the Flink 1.12.x and 1.13.x lines
- Add Spark configuration parameters for:
- spark.memory.fraction
- spark.memory.storageFraction
- spark.kryoserializer.buffer.max
- spark.executor.heartbeatInterval
- Adaptative Query Execution (AQE) can be enabled and configured for Spark 3.x
- Add Flink configuration parameters for several network memory settings and timeouts
- Add HDFS configuration parameters for setting socket-related timeouts
- Command batch mode now supports running multiple scripts stored in a directory
- Remove support for Flink versions <= 1.9
- All input datasets are now created using an internal data generator tool (RGen)
- Set Hadoop classpath properly
- Add sanity checks for some dependencies
- Fix missing dool files
- Minor changes and bug fixes
BDEv 3.5 - 2020-09-04
---------------------
A point release for the BDEv 3.x line
NEW FEATURES
- Updated versions of the following frameworks
- Spark 2.4.6
- Flink 1.9.3
- Flink 1.10.2
- Support for the Hadoop 3.3.x line
- Support for the Spark 3.0.x line
- Support for the Flink 1.11.x line
- Update resource monitoring tool to dool v0.9.9, a Python 3 compatible clone of dstat
- Update RAPL-based power monitoring tool to use PAPI v6.0.0
- Update iLO scripts to version 5.30
- RAPL-based monitoring tool now can report readings for uncore devices if PP0 is available
- Fix bug when setting iLO credentials
- Minor bug fixes
BDEv 3.4 - 2020-03-25
---------------------
A point release for the BDEv 3.x line
NEW FEATURES
- Updated versions of the following frameworks
- Spark 2.4.5
- Flink 1.8.3
- Flink 1.9.2
- Support for Flink 1.10.x line
- Allow to start the job history server for Flink
- Simplified and improved cluster deployments for Spark & Flink
- Add support to configure the YARN Timeline server (yarn-default.sh)
- Add support to configure the disk health checker service for YARN (yarn-default.sh)
- Add support to configure last access time precision for HDFS (hdfs-default.sh)
- Add support to enable short-circuit local reads for HDFS (hdfs-default.sh)
- Memory settings for some daemons (eg, DataNode) can now be set individually
- Fix SQL and ConnComp workloads for Spark
- Spark & Flink benchmarking suites are now downloaded on demand (removed from git repo)
- Scala version for Spark & Flink can be set in solutions-default.sh (2.11 by default)
- Set LD_LIBRARY_PATH in Spark & Flink to include the path to Hadoop native libraries
- Hadoop existence is checked when deployed together with other frameworks
- Minor bug fixes
BDEv 3.3 - 2019-12-17
---------------------
A point release for the BDEv 3.x line
NEW FEATURES
- Updated versions of the following frameworks
- Hadoop 2.10.0
- Hadoop 3.1.3
- Hadoop 3.2.1
- Spark 2.3.4
- Spark 2.4.4
- Flink 1.8.2
- Support for the Flink 1.9.x line (tested with Flink 1.9.1)
- Add support for BDWatchdog tool, which allows per-process resource monitoring and JVM profiling
- More info at http://bdwatchdog.dec.udc.es
- BDEv now runs by default in command mode when no framework is selected in solutions.lst
- Update PAPI library to version 5.7.0
- Update RAPL power monitoring tool to use PAPI 5.7.0
- Add support to configure the binary names for Python2 and Python3 (system-default.sh)
- Fix bug when running Flink in standalone mode
- Minor bug fixes
BDEv 3.2 - 2019-06-06
---------------------
A point release for the BDEv 3.x line
NEW FEATURES
- Updated versions of the following frameworks
- Hadoop 2.7.7
- Hadoop 2.8.5
- Hadoop 2.9.2
- Hadoop 3.1.2
- Hadoop 3.2.0
- RDMA-Hadoop-3 0.9.1
- Flame-MR 1.2
- Spark 2.3.3
- Spark 2.4.3
- Flink 1.7.2
- Flink 1.8.0
- Add support to configure handler count for Hadoop NameNode (hdfs-default.sh)
- Add support to use unsafe based Kryo serializer for Spark (solutions-default.sh)
- Fix bug when running multiple Executors per worker/node in Spark configuration
- Minor bug fixes
BDEv 3.1 - 2018-05-16
---------------------
A point release for the BDEv 3.x line
NEW FEATURES
- Updated versions of some frameworks
- Hadoop 2.7.6
- Hadoop 2.8.3
- Hadoop 2.9.0
- Flame-MR 1.1
- RDMA-Hadoop-2 1.3.5
- Spark 2.3.0
- RDMA-Spark 0.9.5
- Flink 1.4.2
- Support for the Hadoop 3.1.x line (tested with Hadoop 3.1.0)
- Improved customization of experiments with setup and clean up commands
- Extended configuration parameters
BDEv 3.0 - 2017-11-07
--------------------
This is a major release which contains a number of significant
enhancements, along with new characteristics previously unreleased.
NEW FEATURES
- New monitoring tools
- Power monitoring via RAPL
- Microarchitecture-level event counting via Oprofile
- Enhanced framework configuration parameters
- Support for the Flink 1.3.x line (tested with Flink 1.3.2)
- Support for the Spark 2.2.x line (tested with Spark 2.2.0)
- Support for the Flame-MR 1.x line (tested with Flame-MR 1.0)
- Updated versions of some frameworks
- Hadoop 2.7.4
- Hadoop 2.8.1
- RDMA-Hadoop 1.2.0
- RDMA-Spark 0.9.4
- Updated version of resource monitoring tool to dstat 0.7.3
- Improved graph generation portability
BDEv 2.3 - 2016-11-25
---------------------
A point release for the BDEv 2.x line
NEW FEATURES
- Added support for Flame-MR version 0.10.0
- Added support for the Flink 1.1.x line (tested with version 1.1.3)
- Added support for the SLURM job scheduler
- Updated versions of some frameworks
- Hadoop 2.6.5
- Hadoop 2.7.3
- RDMA-Hadoop-2 1.1.0
- Spark 1.6.3
- Flink 1.0.3
- Memory settings for mappers and reducers can now be set separately
- Added configuration variable in etc/core-default.sh to control the
Hadoop parameter "io.file.buffer.size"
- Virtual and physical memory limits in YARN containers as well as the
ratio between them can now be configured in etc/yarn.default.sh
- Fix critical bug when running Flink under YARN related with JobManager
memory
- Minor bug fixes
BDEv 2.2 - 2016-07-04
---------------------
A point release for the BDEv 2.x line
NEW FEATURES
- Add support for multiple Workers/TaskManagers per node in Spark and Flink
- Enhancements in several Flink workloads
- Grep
- PageRank (with delta iterations)
- Connected Components (with Gelly)
- K-Means
- Enhancements in several Spark workloads
- TeraSort
- K-Means (with MLlib)
- Bug fixes
BDEv 2.1 - 2016-05-05
---------------------
A point release for the BDEv 2.x line
NEW FEATURES
- Allow to start the job history server for Spark-based frameworks
- Added support for some new frameworks
- Hadoop 2.6.4
- RDMA-Hadoop-2 0.9.9
- Spark 1.6.1
- Flink 1.0.2
- Added support for Mahout 0.11.2 and 0.12.0
- Added new benchmarks for Spark
- PageRank
- Connected Components
- K-Means
- Bayes
- Hive SQL queries:
- Aggregation
- Join
- Scan
- Added new benchmarks for Flink
- PageRank
- Connected Components
- K-Means
- Multiple bug fixes
BDEv 2.0 - 2016-03-02
---------------------
This is a major release which contains a number of significant
enhancements. Starting with this release, the MapReduce Evaluator
(MREv) tool has been renamed to Big Data Evaluator (BDEv).
NEW FEATURES
- Added support for Flink version 0.10.2 on YARN and standalone modes
- Added support for RDMA-Spark version 0.9.1 on YARN and standalone modes
- Added support for Spark on standalone mode
- Updated versions of some frameworks
- Hadoop 2.7.2
- RDMA-Hadoop-2 0.9.8
- Spark 1.5.2
- Spark 1.6.0
- New data set generation using the HiBench DataGen tool
- Added new benchmarks for Hadoop-based frameworks
- Grep
- K-Means
- Connected Components
- Hive SQL queries:
- Aggregation
- Join
- Scan
- Added new benchmarks for Spark and Flink
- WordCount
- Sort
- Grep
- TeraSort
- Added new benchmark for DataMPI
- Grep
- Allow to start the job history server for Hadoop-based frameworks
- Input datasize for WordCount, Sort and TeraSort can now be different
- Enhanced configuration of parameters
- Separated configuration files for HDFS, MapReduce and YARN
- Solution-specific parameters in a separated file
- Many new configuration parameters
- Support for multiple disks
- Network interfaces are now optional parameters. If not specified, IPs
are determined using the hostfile
- Updated iLO scripts to version 4.70
- Allow to configure IP, user name and password for iLO interfaces
- Automatically download Apache Mahout and Apache Hive on demand
- Optimized code refactorization and simplified internal configuration
- Multiple bug fixes
MREv 1.1 - 2015-08-10
---------------------
A point release for the MREv 1.x line
NEW FEATURES
- Updated versions of some frameworks
- Hadoop-2.7.1-GbE
- Hadoop-2.7.1-IPoIB
- Hadoop-2.7.1-UDA
- RDMA-Hadoop-2-0.9.7-GbE
- RDMA-Hadoop-2-0.9.7-IPoIB
- Spark-1.4.1-YARN-GbE
- Spark-1.4.1-YARN-IPoIB
- Configurable timeout for the workloads
- User-defined batch commands
- Separate configuration and log directories for each evaluation
- Enhanced resource configuration
MREv 1.0 - 2014-12-09
---------------------
The first public release of MREv (MREv 1.0)