-
Notifications
You must be signed in to change notification settings - Fork 48
Using Spark Scala Graphx to analyze the Bitcoin transaction graph
This is a Spark/Graphx application written in Scala demonstrating some of the capabilities of the hadoopcryptoledger library. It takes as input a set of files on HDFS containing Bitcoin Blockchain data. It generates out of the data a Bitcoin transaction graph, i.e. a graph consisting of the vertices describing Bitcoin addresses and the edges represent transactions between these addresses.
It returns as a result from the graph the top 5 Bitcoin addresses in terms of number of transfers to them. Most of the times these addresses belong to so called Mixing services that try to obfuscate transactions in the Bitcoin blockchain.
It has successfully been tested with the Cloudera Quickstart VM 5.5 and HDP Sandbox 2.5, but other Hadoop distributions should work equally well. Spark 1.5 was used for testing.
See here how to fetch Bitcoin blockchain data.
After it has been copied you are ready to use the example.
Execute
git clone https://github.com/ZuInnoTe/hadoopcryptoledger.git hadoopcryptoledger
You can build the application by changing to the directory hadoopcryptoledger/examples/scala-spark-graphx-bitcointransaction and using the following command:
sbt clean assembly test it:test
This will also execute the integration tests
You will find the jar "example-hcl-spark-scala-graphx-bitcointransaction.jar" in ./target/scala-2.10
Make sure that the output directory is clean:
hadoop fs -rm -R /user/bitcoin/output
Execute the following command (to execute it using a local master)
spark-submit --class org.zuinnote.spark.bitcoin.example.SparkScalaBitcoinTransactionGraph --master local[8] ./target/scala-2.10/example-hcl-spark-scala-graphx-bitcointransaction.jar /user/bitcoin/input /user/bitcoin/output
After the Spark job has completed, you find the result in /user/spark/bitcoin/output. You can display it using the following command:
hadoop fs -cat /user/bitcoin/output/part-00000
An example for such an output can be found here (vertexId,(receiving Bitcoin address, number of inputs)):
(26038,(bitcoinaddress_F4B004C3CA2E7F96F9FC5BCA767708967AF67A44,119539))
(138462,(bitcoinaddress_8ED0DCFF2D3D8F6F6EDC244885EEAE895469D7BB,7938))
(29231,(bitcoinaddress_2D6FE761506A0FBDCED503FDA7FB85C26ED1E76E,7784))
(260013,(bitcoinaddress_77E84EAC6A7132B1E9267BA926EEA90F7FED9C74,5565))
(385502,(bitcoinaddress_A92BA6F9C91233B6A4EC84DC04CDB7AF08246184,5136))
Hint: You can put the addresses (without the prefix bitcoinaddress_) in popular Bitcoin explorers (e.g. https://blockchain.info) to get more details about the address.
Of course, you can integrate other data together with the blockchain data or add to the edges in the Graph the amount for each transaction.
Understanding the structure of Bitcoin data:
Understanding the destination of Bitcoin addresses and why they cannot be compared with transactions occurring in the current financial system: https://en.bitcoin.it/wiki/From_address
Blocks: https://en.bitcoin.it/wiki/Block
Transactions: https://en.bitcoin.it/wiki/Transactions
Heuristics for joining Bitcoin Transactions: Meiklejohn, Sarah; Pomarole, Marjori; McKoy, Damon; Jordan, Grant; Voelker, Geoffrey: "A Fistful of Bitcoins: Characterizing Payments Among Men with No Names" (Extended Abstract), Proceedings of the 2013 conference on Internet Measurement, Barcelona, Spain, 2013