Skip to content

Commit

Permalink
Update defaultHadoopCoordinates in documentation. (apache#5720)
Browse files Browse the repository at this point in the history
* Update defaultHadoopCoordinates in documentation.

To match changes applied in apache#5382.

* Remove a parameter with defaults from example configuration file.

If it has reasonable defaults, then why would it be in an example config file?

Also, it is yet another place that has been forgotten to be updated and will be forgotten in the future.

Also, if someone is running different hadoop version, then there's much more work to be done than just changing this property, so why give users false hopes?

* Fix typo in documentation.
jkukul authored and b-slim committed May 1, 2018

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature.
1 parent a0c2ae7 commit e2431ae
Showing 6 changed files with 4 additions and 6 deletions.
2 changes: 1 addition & 1 deletion docs/content/configuration/indexing-service.md
Original file line number Diff line number Diff line change
@@ -340,7 +340,7 @@ Additional peon configs include:
|`druid.peon.mode`|Choices are "local" and "remote". Setting this to local means you intend to run the peon as a standalone node (Not recommended).|remote|
|`druid.indexer.task.baseDir`|Base temporary working directory.|`System.getProperty("java.io.tmpdir")`|
|`druid.indexer.task.baseTaskDir`|Base temporary working directory for tasks.|`${druid.indexer.task.baseDir}/persistent/tasks`|
|`druid.indexer.task.defaultHadoopCoordinates`|Hadoop version to use with HadoopIndexTasks that do not request a particular version.|org.apache.hadoop:hadoop-client:2.3.0|
|`druid.indexer.task.defaultHadoopCoordinates`|Hadoop version to use with HadoopIndexTasks that do not request a particular version.|org.apache.hadoop:hadoop-client:2.8.3|
|`druid.indexer.task.defaultRowFlushBoundary`|Highest row count before persisting to disk. Used for indexing generating tasks.|75000|
|`druid.indexer.task.directoryLockTimeout`|Wait this long for zombie peons to exit before giving up on their replacements.|PT10M|
|`druid.indexer.task.gracefulShutdownTimeout`|Wait this long on middleManager restart for restorable tasks to gracefully exit.|PT5M|
2 changes: 1 addition & 1 deletion docs/content/operations/other-hadoop.md
Original file line number Diff line number Diff line change
@@ -69,7 +69,7 @@ classloader.
2. Batch ingestion uses jars from `hadoop-dependencies/` to submit Map/Reduce jobs (location customizable via the
`druid.extensions.hadoopDependenciesDir` runtime property; see [Configuration](../configuration/index.html)).

`hadoop-client:2.3.0` is the default version of the Hadoop client bundled with Druid for both purposes. This works with
`hadoop-client:2.8.3` is the default version of the Hadoop client bundled with Druid for both purposes. This works with
many Hadoop distributions (the version does not necessarily need to match), but if you run into issues, you can instead
have Druid load libraries that exactly match your distribution. To do this, either copy the jars from your Hadoop
cluster, or use the `pull-deps` tool to download the jars from a Maven repository.
2 changes: 1 addition & 1 deletion docs/content/querying/lookups.md
Original file line number Diff line number Diff line change
@@ -332,7 +332,7 @@ The return value will be the json representation of the factory.
```

# Configuration
See the [coordinator configuration guilde](../configuration/coordinator.html) for coordinator configuration
See the [coordinator configuration guide](../configuration/coordinator.html) for coordinator configuration.

To configure a Broker / Router / Historical / Peon to announce itself as part of a lookup tier, use the `druid.zk.paths.lookupTier` property.

Original file line number Diff line number Diff line change
@@ -17,4 +17,3 @@ druid.indexer.fork.property.druid.processing.numThreads=2

# Hadoop indexing
druid.indexer.task.hadoopWorkingPath=var/druid/hadoop-tmp
druid.indexer.task.defaultHadoopCoordinates=["org.apache.hadoop:hadoop-client:2.7.3"]
1 change: 0 additions & 1 deletion examples/conf/druid/middleManager/runtime.properties
Original file line number Diff line number Diff line change
@@ -17,4 +17,3 @@ druid.indexer.fork.property.druid.processing.numThreads=2

# Hadoop indexing
druid.indexer.task.hadoopWorkingPath=var/druid/hadoop-tmp
druid.indexer.task.defaultHadoopCoordinates=["org.apache.hadoop:hadoop-client:2.7.3"]
2 changes: 1 addition & 1 deletion services/src/main/java/io/druid/cli/PullDependencies.java
Original file line number Diff line number Diff line change
@@ -167,7 +167,7 @@ public class PullDependencies implements Runnable

@Option(
name = "--no-default-hadoop",
description = "Don't pull down the default hadoop coordinate, i.e., org.apache.hadoop:hadoop-client:2.3.0. If `-h` option is supplied, then default hadoop coordinate will not be downloaded.",
description = "Don't pull down the default hadoop coordinate, i.e., org.apache.hadoop:hadoop-client:2.8.3. If `-h` option is supplied, then default hadoop coordinate will not be downloaded.",
required = false)
public boolean noDefaultHadoop = false;

0 comments on commit e2431ae

Please sign in to comment.