Skip to content

PiFlow V0.6 Release
Compare
Choose a tag to compare
@yg000 yg000 released this 28 Nov 12:54
· 568 commits to master since this release
v0.6
6446a40

Requirements

  • JDK 1.8 or newer
  • Spark-2.1.0 （piflow jar of other spark version should be build by the code）
  • Hadoop-2.6.0 （piflow jar of other hadoop version should be build by the code）
  • Hive-1.2.1（if you need to use hive）

Configure

  #server Ip and Port
  server.ip=10.0.88.70
  server.port=8002
  
  #Spark master and deploy mode
  spark.master=yarn
  spark.deploy.mode=cluster
  
  #yarn related configurations
  yarn.resourcemanager.hostname=10.0.88.70
  yarn.resourcemanager.address=10.0.88.70:8032
  yarn.access.namenode=hdfs://10.0.88.70:9000
  yarn.stagingDir=hdfs://10.0.88.70:9000/tmp/
  yarn.jars=hdfs://10.0.88.70:9000/user/spark/share/lib/*.jar
  yarn.url=http://10.0.88.70:8088/ws/v1/cluster/apps/
  
  #hive metaStore uris
  hive.metastore.uris=thrift://10.0.88.71:9083

  #piflow server jar folder, please change this parameter to your path
  piflow.bundle=/data/piflow/piflow-server-v0.6/lib/piflow-server-0.9.jar
  
  #hdfs path for checkpoint、debug、increment，please create these folders first 
  checkpoint.path=hdfs://10.0.88.70:9000/user/piflow/checkpoints/
  debug.path=hdfs://10.0.88.70:9000/user/piflow/debug/
  increment.path=hdfs://10.0.88.70:9000/user/piflow/increment/

  #set 0 if you don not want to show data in log
  data.show=10

  #h2 db port
  h2.port=50002

Command

  ./start.sh
  ./stop.sh
  ./restart.sh
  ./status.sh
Assets 3
Loading