GoSpark: An In-Memory Distributed Computation Platform in Go

Kuan-Ting Yu, Jiasi Shen, Bolei Zhou

Course project for MIT EECS 6.824: Distributed Systems

Spring 2014, course page: http://css.csail.mit.edu/6.824/2014/

Install Go

mkdir ~/software  # -> this is where I install go
cd ~/software
hg clone -u release https://code.google.com/p/go
cd go/src
./all.bash

Environment Setup

Add these in your .bashrc

export HADOOP_HOME=/usr/local/hadoop   #-> where you install the hadoop
export DSPROJECT_HOME=$HOME/824project   #-> where you put our project
export JAVA_HOME=/usr/lib/jvm/java-6-openjdk/   #-> where is the jdk installed

lzohead () {
	hadoop dfs -cat $1 | lzop -dc | head -1000 | less
}

unalias dfs &> /dev/null
alias dfs="hdfs dfs"
unalias hls &> /dev/null
alias hls="dfs -ls"
export PATH=$PATH:$HADOOP_HOME/bin
# Add Hadoop bin/ directory to PATH
export CLASSPATH=${DSPROJECT_HOME}/HDFSSplitUtil/bin:`hadoop classpath`:${CLASSPATH}

# For Go
export GOROOT=$HOME/software/go
export GOPATH=${DSPROJECT_HOME}/Spark
export PATH=$PATH:$HOME/software/go/bin

Testing

Update master host name (and/or port number) in config.txt.

On scheduler machine, run:

go test -run KMeans -timeout 1h

go test -run MasterMRLineCount -debug

On worker machines, run:

go test -run BasicWorker -timeout 1h

go test -run WorkerSuperUnrel -timeout 1h -debug

Options:

• -run (required) chooses which application to run.
  • Scheduler: -run KMeans, -run MasterMRLineCount, etc.
  • Workers: -run BasicWorker for reliable, -run WorkerRPCUnrel for unreliable network connection, or -run WorkerSuperUnrel for unreliable network & may crash.
• -timeout sets the timeout. For example, -timeout 1h for 1 hour.
• -debug prints verbose log.

Utility Shell

install tmux: sudo apt-get install tmux

get into tmux, then run the sh

tmux
cd shells
./start-all.sh