Skip to content

Commit ffd0aa9

Browse files
ibuenrosibuenros
committed
Merge pull request #653 from ibuenros/v0.6.2
V0.6.2
2 parents feec99e + 116c79b commit ffd0aa9

File tree

3 files changed

+133
-81
lines changed

3 files changed

+133
-81
lines changed

‎CHANGELOG

Lines changed: 51 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,56 @@
1-
GOBBLIN 0.6.1
1+
GOBBLIN 0.6.2
22
=============
33

4+
## NEW FEATURES
5+
* [Admin Dashboard] Added a web based GUI for exploring running and finished jobs in a running Gobblin daemon (thanks Eric Ogren).
6+
* [Admin Dashboard] Added a CLI for finding jobs in the job history store and seeing their run details (thanks Eric Ogren).
7+
* [Configuration Management] WIP: Configuration management library. Will enable Gobblin to be dataset aware, ie. to dynamically load and apply different configurations to each dataset in a single Gobblin job.
8+
** APIs: APIs for configuration stores and configuration client.
9+
** Configuration Library: loads low level configurations from a configuration store, resolves configuration dependencies / imports, and performs value interpolation.
10+
* [Distcp] Allow using *.ready files as markers for files that should be copied, and deletion of *.ready files once the file has been copied.
11+
* [Distcp] Added file filters to recursive copyable dataset for distcp. Allows to only copy files satisfying a filter under a base directory.
12+
* [Distcp] Copied files that fail to be published are persisted for future runs. Future runs can recover the already copied file instead of re-doing the byte transfer.
13+
* [JDBC] Can use password encryption for JDBC sources.
14+
* [YARN] Added email notifications on YARN application shutdown.
15+
* [YARN] Added event notifications on YARN container status changes.
16+
* [Metrics] Added metric filters based on name and type of the metrics.
17+
* [Dataset Management] POC embedded sql for config-driven retention management.
18+
* [Exactly Once] POC for Gobblin managed exactly once semantics on publisher.
19+
20+
## BUG FIXES
21+
* **Core** File based source includes previously failed WorkUnits event if there are no new files in the source (thanks Joel Baranick).
22+
* **Core** Ensure that output file list does not contain duplicates due to task retries (thanks Joel Baranick).
23+
* **Core** Fix NPE in CliOptions.
24+
* **Core/YARN** Limit Props -> Typesafe Config conversion to a few keys to prevent overwriting of certain properties.
25+
* **Utility** Fixed writer mkdirs for S3.
26+
* **Metrics** Made Scheduled Reporter threads into daemon threads to prevent hanging application.
27+
* **Metrics** Fixed enqueuing of events on event reporters that was causing job failure if event frequency was too high.
28+
* **Build** Fix POM dependencies on gobblin-rest-api.
29+
* **Build** Added conjars and cloudera repository to all projects (fixes builds for certain users).
30+
* **Build** Fix the distribution tarball creation (thanks Joel Baranick).
31+
* **Build** Added option to exclude Hadoop and Hive jars from distribution tarball.
32+
* **Build** Removed log4j.properties from runtime resources.
33+
* **Compaction** Fixed main class in compaction manifest file (thanks Lorand Bendig).
34+
* **JDBC** Correctly close JDBC connections.
35+
36+
## IMPROVEMENTS
37+
* [Build] Add support for publishing libraries to maven local (thanks Joel Baranick).
38+
* [Build] In preparation to Gradle 2 migration, added ext. prefix to custom gradle properties.
39+
* [Build] Can generate project dependencies graph in dot format.
40+
* [Metrics] Migrated Kafka reporter and Output stream reporter to Root Metrics Reporter managed reporting.
41+
* [Metrics] The last metric emission in the application has a "final" tag for easier Hive identification.
42+
* [Metrics] Metrics for Gobblin on YARN include cluster tags.
43+
* [Hive] Upgraded Hive to version 1.0.1.
44+
* [Distcp] Add file size to distcp success notifications.
45+
* [Distcp] Each work unit in distcp contains exactly one Copyable File.
46+
* [Distcp] Copy source can set upstream timestamps for SLA events emitted on publish time.
47+
* [Scheduling] Added Gobblin Oozie config files.
48+
* [Documentation] Improved javadocs.
49+
50+
51+
GOBBLIN 0.6.1
52+
-------------
53+
454
## BUG FIXES
555

656
- **Build/release** Adding build instrumentation for generation of rest-api-* artifacts

‎build.gradle

Lines changed: 74 additions & 72 deletions
Original file line numberDiff line numberDiff line change
@@ -256,25 +256,93 @@ if (ext.javadocVersion.indexOf('-') > 0) {
256256
}
257257

258258
ext.javadocPackages = new HashSet<String>()
259-
subprojects.each{Project pr ->
259+
subprojects.each{Project pr ->
260260
if (file(pr.projectDir.absolutePath + "/src/main/java").exists()) {
261261
rootProject.ext.javadocPackages += pr.name
262262
}
263263
}
264264

265265
subprojects {
266266
plugins.withType(JavaPlugin) {
267-
267+
268268
// Sometimes generating javadocs can lead to OOM. This may needs to be increased.
269269
// Also force javadocs to pick up system proxy settings if available
270270
javadoc {
271271
options.jFlags('-Xmx256m', '-Djava.net.useSystemProxies=true');
272272
}
273-
273+
274274
rootProject.tasks.javadocTarball.dependsOn project.tasks.javadoc
275275
if ( rootProject.ext.javadocPackages.contains(project.name)) {
276276
rootProject.tasks.javadocTarball.into(project.name){from(fileTree(dir: "${project.buildDir}/docs/javadoc/"))}
277-
}
277+
}
278+
}
279+
}
280+
281+
ext.pomAttributes = {
282+
name "${project.name}"
283+
packaging 'jar'
284+
// optionally artifactId can be defined here
285+
description 'Gobblin Ingestion Framework'
286+
url 'https://github.com/linkedin/gobblin/'
287+
288+
scm {
289+
connection 'scm:git:[email protected]:linkedin/gobblin.git'
290+
developerConnection 'scm:git:[email protected]:linkedin/gobblin.git'
291+
url '[email protected]:linkedin/gobblin.git'
292+
}
293+
294+
licenses {
295+
license {
296+
name 'The Apache License, Version 2.0'
297+
url 'http://www.apache.org/licenses/LICENSE-2.0.txt'
298+
}
299+
}
300+
301+
developers {
302+
developer {
303+
name 'Abhishek Tiwari'
304+
organization 'LinkedIn'
305+
}
306+
developer {
307+
name 'Chavdar Botev'
308+
organization 'LinkedIn'
309+
}
310+
developer {
311+
name 'Issac Buenrostro'
312+
organization 'LinkedIn'
313+
}
314+
developer {
315+
name 'Min Tu'
316+
organization 'LinkedIn'
317+
}
318+
developer {
319+
name 'Narasimha Veeramreddy'
320+
organization 'LinkedIn'
321+
}
322+
developer {
323+
name 'Pradhan Cadabam'
324+
organization 'LinkedIn'
325+
}
326+
developer {
327+
name 'Sahil Takiar'
328+
organization 'LinkedIn'
329+
}
330+
developer {
331+
name 'Shirshanka Das'
332+
organization 'LinkedIn'
333+
}
334+
developer {
335+
name 'Yinan Li'
336+
organization 'LinkedIn'
337+
}
338+
developer {
339+
name 'Ying Dai'
340+
organization 'LinkedIn'
341+
}
342+
developer {
343+
name 'Ziyang Liu'
344+
organization 'LinkedIn'
345+
}
278346
}
279347
}
280348

@@ -380,73 +448,7 @@ subprojects {
380448
authentication(userName: ossrhUsername, password: ossrhPassword)
381449
}
382450

383-
pom.project {
384-
name "${project.name}"
385-
packaging 'jar'
386-
// optionally artifactId can be defined here
387-
description 'Gobblin Ingestion Framework'
388-
url 'https://github.com/linkedin/gobblin/'
389-
390-
scm {
391-
connection 'scm:git:[email protected]:linkedin/gobblin.git'
392-
developerConnection 'scm:git:[email protected]:linkedin/gobblin.git'
393-
url '[email protected]:linkedin/gobblin.git'
394-
}
395-
396-
licenses {
397-
license {
398-
name 'The Apache License, Version 2.0'
399-
url 'http://www.apache.org/licenses/LICENSE-2.0.txt'
400-
}
401-
}
402-
403-
developers {
404-
developer {
405-
name 'Abhishek Tiwari'
406-
organization 'LinkedIn'
407-
}
408-
developer {
409-
name 'Chavdar Botev'
410-
organization 'LinkedIn'
411-
}
412-
developer {
413-
name 'Issac Buenrostro'
414-
organization 'LinkedIn'
415-
}
416-
developer {
417-
name 'Min Tu'
418-
organization 'LinkedIn'
419-
}
420-
developer {
421-
name 'Narasimha Veeramreddy'
422-
organization 'LinkedIn'
423-
}
424-
developer {
425-
name 'Pradhan Cadabam'
426-
organization 'LinkedIn'
427-
}
428-
developer {
429-
name 'Sahil Takiar'
430-
organization 'LinkedIn'
431-
}
432-
developer {
433-
name 'Shirshanka Das'
434-
organization 'LinkedIn'
435-
}
436-
developer {
437-
name 'Yinan Li'
438-
organization 'LinkedIn'
439-
}
440-
developer {
441-
name 'Ying Dai'
442-
organization 'LinkedIn'
443-
}
444-
developer {
445-
name 'Ziyang Liu'
446-
organization 'LinkedIn'
447-
}
448-
}
449-
}
451+
pom.project pomAttributes
450452
}
451453
}
452454
}
@@ -483,7 +485,7 @@ subprojects {
483485
"https://dropwizard.github.io/metrics/${dropwizardMetricsVersion}/apidocs/"
484486
rootProject.ext.javadocPackages.each {
485487
tasks.javadoc.options.linksOffline "http://linkedin.github.io/gobblin/javadoc/${javadocVersion}/${it}/",
486-
"${rootProject.buildDir}/${it}/docs/javadoc/"
488+
"${rootProject.buildDir}/${it}/docs/javadoc/"
487489
}
488490

489491
afterEvaluate {

‎gobblin-rest-service/gobblin-rest-api/build.gradle

Lines changed: 8 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -28,31 +28,31 @@ if (rootProject.ext.publishToMaven) {
2828
configure(uploadArchives.repositories.mavenDeployer) {
2929
addFilter('gobblin-rest-api-rest-client') {artifact, file ->
3030
artifact.name == 'gobblin-rest-api-rest-client'
31-
}.artifactId = 'gobblin-rest-api-rest-client'
31+
}.project(rootProject.pomAttributes).artifactId = 'gobblin-rest-api-rest-client'
3232

3333
addFilter('gobblin-rest-api-data-template') {artifact, file ->
3434
artifact.name == 'gobblin-rest-api-data-template'
35-
}.artifactId = 'gobblin-rest-api-data-template'
35+
}.project(rootProject.pomAttributes).artifactId = 'gobblin-rest-api-data-template'
3636

3737
addFilter('gobblin-rest-api') {artifact, file ->
3838
artifact.name == 'gobblin-rest-api'
39-
}.artifactId = 'gobblin-rest-api'
39+
}.project(rootProject.pomAttributes).artifactId = 'gobblin-rest-api'
4040
// artifact names for 'data-model', 'avro-schema' and 'rest-model' may be added as well if needed
4141
}
4242
}
4343

4444
configure(install.repositories.mavenInstaller) {
4545
addFilter('gobblin-rest-api-rest-client') {artifact, file ->
4646
artifact.name == 'gobblin-rest-api-rest-client'
47-
}.artifactId = 'gobblin-rest-api-rest-client'
47+
}.project(rootProject.pomAttributes).artifactId = 'gobblin-rest-api-rest-client'
4848

4949
addFilter('gobblin-rest-api-data-template') {artifact, file ->
5050
artifact.name == 'gobblin-rest-api-data-template'
51-
}.artifactId = 'gobblin-rest-api-data-template'
51+
}.project(rootProject.pomAttributes).artifactId = 'gobblin-rest-api-data-template'
5252

5353
addFilter('gobblin-rest-api') {artifact, file ->
5454
artifact.name == 'gobblin-rest-api'
55-
}.artifactId = 'gobblin-rest-api'
55+
}.project(rootProject.pomAttributes).artifactId = 'gobblin-rest-api'
5656
// artifact names for 'data-model', 'avro-schema' and 'rest-model' may be added as well if needed
5757
}
5858

@@ -101,8 +101,8 @@ task restClientJavadocJar(type: Jar, dependsOn: 'restClientJavadoc') {
101101
classifier = 'javadoc'
102102
}
103103

104-
artifacts {
105-
archives dataTemplateSourcesJar, dataTemplateJavadocJar, restClientSourcesJar, restClientJavadocJar
104+
artifacts {
105+
archives dataTemplateSourcesJar, dataTemplateJavadocJar, restClientSourcesJar, restClientJavadocJar
106106
}
107107

108108
buildscript {

0 commit comments

Comments
 (0)