GCS Datastore Implementation for IPFS

An implementation of the go-datastore interface backed by Google Cloud Storage, GCS. The implementation is based on go-ds-s3

Building binary

Build this plugin as an included plugin with kubo.

Clone kubo.

git clone https://github.com/ipfs/kubo

Get the GCS datastore plugin and add it to the preload list.

(cd kubo; go get github.com/ipfs-shipyard/go-ds-gcs/plugin)
echo "gcsds github.com/ipfs-shipyard/go-ds-gcs/plugin 0" >> kubo/plugin/loader/preload_list

Build kubo with the plugin

(cd kubo; make build)

(Optionally) install kubo onto the local system.

(cd kubo; make install)

Building Docker container

Follow the "Building binary" section first. Clone the GCS datastore plugin and build the Docker entrypoint.

git clone https://github.com/ipfs-shipyard/go-ds-gcs
(cd go-ds-gcs; go build docker/entrypoint.go)

Build docker and container.

docker build -f go-ds-gcs/docker/Dockerfile -t ipfs .

(Optional) tag and push image to repository. Replace repository with your GCR repository.

docker tag ipfs gcr.io/repository/ipfs
docker push gcr.io/repository/ipfs

Deployment: Docker

ipfs is the image name from the previous section, "Building docker container"

docker run -d -p 4001:4001 -p 5001:5001 -p 8080:8080 --name=ipfs ipfs

Or you can use the tagged image from the previous section.

docker run -d -p 4001:4001 -p 5001:5001 -p 8080:8080 --name=ipfs gcr.io/repository/ipfs

Deployment: Kubernetes (GKE)

1. Create a GKE cluster. The node pool must have write permission to the GCS bucket.
2. Replace repository with your repository name in kubernetes/ipfs.yml
3. Replace mynamespace with your namespace name in kubernetes/ipfs.yml
4. Create namespace in GKE.

kubectl create namespace mynamespace

5. Deploy to GKE.

kubectl apply -f kubernetes/ipfs.yml

6. IPFS can now be accessed at ipfs.mynamespace from other pods in the GKE cluster:

curl http://ipfs.mynamespace:8080/...

Deployment: Local

For a brand new ipfs instance (no data stored yet):

1. Install kubo/cmd/ipfs/ipfs
2. Initialize IPFS. Replace mybucket with your bucket name. Note that this depends on this Pull Request.

KUBO_GCS_BUCKET=mybucket ipfs init --profile gcsds

4. To run as a server, also run

ipfs config profile apply server
ipfs config Addresses.Gateway /ip4/0.0.0.0/tcp/8080
ipfs daemon

Configuration

The config file should include the following. Replace mybucket with your bucket name.

{
	"Datastore": {
		"Spec": {
			"mounts": [{
				"child": {
					"bucket": "mybucket",
					"cachesize": 40000,
					"prefix": "ipfs",
					"type": "gcsds",
					"workers": 100
				},
				"mountpoint": "/blocks",
				"prefix": "flatfs.datastore",
				"type": "measure"
			}]
		}
	}
}

Google Cloud credentials

Google Cloud credentials should automatically be provided when running in Google Compute Engine (GCE) or Google Kubernetes Engine (GKE). Note that for both GCE and GKE, the (node) VM needs to have write permission (scope) to GCS. For GKE, this is achieved by creating the node pool with the "Storage read/write" scope, which is "devstorage.read_write".

In other environments, you may have to provide credentials. One way is to use the GOOGLE_APPLICATION_CREDENTIALS environment variable. See this document for more details.

Contribute

Feel free to join in. All welcome. Open an issue!

This repository falls under the IPFS Code of Conduct and the Google Code of Conduct

License

Apache 2.0