Plan of supernode's refactoring plan

[allencloud]

Background

This is one part of Golang Refactoring Plan of the Whole Dragonfly Project

Supernode is the brain of Dragonfly. It takes over the image CDN from remote registry, and schedules peer node according the actual situation of themselves. In addition, flow control, security guarantee are also what it is responsible for. And supernode is also supposed to de designed with HA(High Availability).

Refactoring Details

APIs

Golang SDK

Dragonfly's supernode needs a Golang SDK which is stored in https://github.com/dragonflyoss/Dragonfly/tree/master/client. Then we achieve the following goals:

• every caller of supernode can take advantages of this SDK by integrating it
• golang sdk encapsulates of the details of HTTP communication
• it could provide scalability and portability for other system to integrate supernode.

In project Dragonfly, it will be used in component dfget and the dashboard. The refactoring plan of golang sdk is described in dragonflyoss/dragonfly#366.

• Golang SDK refactoring

Peer Releated

• POST /peers: register a peer node in supernode.
• GET /peers/{id}: Get information for one single peer node;

Task Related

• POST /tasks: register a file/image downloading task;
• GET /tasks/{id}/pieces: client side requests to get the pieces list of the downloading task;
• PUT /tasks/{id}/pieces/{pieceRange}: Update the status of each piece in the downloading task;
• GET /tasks/{id}: Get information for one single downloading task;
• PUT /tasks/{id}: Update the status of downloading task;
• DELETE /tasks/{id}: Delete a downloading task and its local cache on supernode.

Preheat related

• POST /preheats: Creates preheat task in supernode to cache a file/image first before starting distribution
• GET /preheats/{id}: Get the status of preheat task
• GET /preheats: Get preheat task list

Features

Peer/Task Registration Logistic

• peer node creation and peer storage interface definition and implementation;
• task creation and task storage interface definition and implementation;

Download Task and peer node management

• mapping relationship between downloading task and peer node and downloading tasks' status implemetation;
• mapping relationship and status of peer node and its downloading pieces
• support blacklist，and dynamically adjust the available peer nodes of requested pieces

Scheduling

• definition of scheduling interface
• scheduling logistic implementation which is based on minimal work load

CDN source file downloading logistic

• CDN cache management：cache hit and evict rule
• definition of pieces' data format and rule of source file's splitting
• downloading of source file and support to resume from break-point
• definition of file meta data and storage replacement policy

Underlying Storage

• definition of storage drive interface
• default storage implementation which is based on local filesystem

flow control

• bandwidth limit for file downloading from file remote source and CDN cache
• speed limit globally, and support to dynamically adjust client downloading speed
• support to dynamically adjust uploading task number of each peer

Security

• data encryption
• blacklist blocking by callers
• API authentication

HA(High Availability)

• meta data sharing, issue: Create a file server system for CDN dragonfly#232, pr:

CLI ( Command Line )

• Use cobra to construct supernode's CLI.
• detailed description of every essential cobra command's fields