Skip to content
/ moulin Public

A reliable queue for batch processing written in Go and backed by Redis.

License

Notifications You must be signed in to change notification settings

dhrp/moulin

Repository files navigation

Moulin

'Moulin' is 'Mill' in French. A Dutch saying goes: wie het eerst komt, wie het eerst maalt. Who brings their wheat to the mill first, is the first to get his/her wheat milled, and is the first to eat.

For information how it works, read here

Running the service

Run the server

First make sure you have Redis running. For example like so:

docker run -p 6379:6379 --name redis -d redis

Then start the moulin server. In this example we link to the redis container that we just started.

docker run --link redis:redis -e REDIS_ADDRESS=redis:6379 --name moulin -d dhrp/moulin

Specify the REDIS_ADDRESS, to point to a Redis server (defaults to localhost)

Run the client

The client is also built into the docker image. You can run it with

docker run --entrypoint moulin-cli --link moulin -e MOULIN_SERVER=moulin:8042 dhrp/moulin work queue1 forever

Here we tell use moulin-cli to listen to a queue 'queue1' and execute each command that comes from the queue.

Using Moulin

How to use it

Moulin primarily meant to simply execute a given task on the shell; and as such it is language agnostic. Let's say you have a python script "download_files.py" which takes one argument, a URL to download. You can create a task like:

moulin-cli create queue1 "./download_files.py https://myserver.org/file2.bin"

You can then also create a worker like so:

moulin-cli work queue1 forever

It will listen on a socket from the moulin server, pop a task from the queue and processes it. When no more tasks are available it will wait for the next to show up.

Reliability

Once a Moulin worker takes a task from the queue it moves to a 'in process' queue, and only moves to "succeeded" if the child process exits with error code 0. The worker is also expected to send a regular heartbeat to the server to make sure the process didn't die unexpectedly. After 5 minutes of no heartbeat the task is considered failed.

See how it works

Functions available:

  • create: Create a task to be executed.
  • progress: Check the progress of a certain queue
  • peek: Have a look at the latest N items in the queue

Building

Creating and starting the server

You can build this image with Docker, docker build -t dhrp/moulin . and then just run it like so:

make image
# docker build -t dhrp/moulin .
make build
make cli

GRPC

When the communication protocol changes; GRPC definitions need to be updated. Generate a new protocol buffer definition with a version of the following command:

$ protoc -I helloworld/ helloworld/helloworld.proto --go_out=plugins=grpc:helloworld

Uploading a file for task batch:

Currently we're supporting multipart file uploads; this is what you'd get when you'd upload a file through a form. Probably it would make more sense to (just) support receiving a text/plain body which has only the content of the file (lines with instructions).

http --verify no --form POST https://localhost:8042/v1/task_list/batch/ file@./kafkaproducer/test/testtextfile.txt -v

further reading: