Skip to content

Commit 800411a

Browse files
xmonaderxmonader
authored
Merge pull request #108 from threefoldtech/development_docs
Add architecture, concepts, tutorials, user-guides docs and update th…
2 parents 0eaf96a + 27cc407 commit 800411a

24 files changed

+6347
-184
lines changed

‎README.md‎

Lines changed: 157 additions & 137 deletions
Large diffs are not rendered by default.

‎docs/README.md‎

Lines changed: 88 additions & 47 deletions
Original file line numberDiff line numberDiff line change
@@ -1,73 +1,114 @@
1-
# FungiList specifications
1+
# RFS Documentation
22

3-
## Introduction
3+
Welcome to the comprehensive documentation for the RFS (Remote File System) command-line tool. This documentation covers all aspects of using the `rfs` command, including its various subcommands, concepts, tutorials, and user guides.
44

5-
The idea behind the FL format is to build a full filesystem description that is compact and also easy to use from almost ANY language. The format need to be easy to edit by tools like `rfs` or any other tool.
5+
## Table of Contents
66

7-
We decided to eventually use `sqlite`! Yes the `FL` file is just a `sqlite` database that has the following [schema](../rfs/schema/schema.sql)
7+
### Architecture
88

9-
## Tables
9+
The architecture documentation provides a high-level overview of the RFS system and how it works.
1010

11-
### Inode
11+
- [Architecture Overview](./architecture/overview.md) - High-level architecture of the RFS system
12+
- [Components](./architecture/components.md) - How the different components work together
13+
- [Data Flow](./architecture/data-flow.md) - How data flows through the system
14+
- [Storage Backends](./architecture/storage-backends.md) - Details about the supported storage backends
1215

13-
Inode table describe each entry on the filesystem. It matches really closely the same `inode` structure on the linux operating system. Each inode has a unique id called `ino`, a parent `ino`, name, and other parameters (user, group, etc...).
16+
### Concepts
1417

15-
The type of the `inode` is defined by its `mode` which is a `1:1` mapping from the linux `mode`
18+
The concepts documentation explains the core concepts and principles behind RFS.
1619

17-
> from the [inode manual](https://man7.org/linux/man-pages/man7/inode.7.html)
20+
- [Understanding Flists](./concepts/flists.md) - What are flists and how they work
21+
- [Understanding Storage Backends](./concepts/stores.md) - How storage backends work
22+
- [Understanding Caching](./concepts/caching.md) - How caching works in RFS
23+
- [Understanding Sharding and Replication](./concepts/sharding.md) - How sharding and replication work
1824

19-
```
25+
### Tutorials
2026

21-
POSIX refers to the stat.st_mode bits corresponding to the mask
22-
S_IFMT (see below) as the file type, the 12 bits corresponding to
23-
the mask 07777 as the file mode bits and the least significant 9
24-
bits (0777) as the file permission bits.
27+
The tutorials provide step-by-step guides for common tasks using the `rfs` command.
2528

26-
The following mask values are defined for the file type:
29+
### Core Functionality
2730

28-
S_IFMT 0170000 bit mask for the file type bit field
31+
- [Getting Started](./tutorials/getting-started.md) - Installation and basic usage
32+
- [Creating Flists](./tutorials/creating-flists.md) - How to create flists from directories
33+
- [Mounting Flists](./tutorials/mounting-flists.md) - How to mount and use flists
34+
- [End-to-End Flist Workflow](./tutorials/end-to-end-flist-workflow.md) - Complete workflow for creating and using flists
2935

30-
S_IFSOCK 0140000 socket
31-
S_IFLNK 0120000 symbolic link
32-
S_IFREG 0100000 regular file
33-
S_IFBLK 0060000 block device
34-
S_IFDIR 0040000 directory
35-
S_IFCHR 0020000 character device
36-
S_IFIFO 0010000 FIFO
37-
```
36+
### Docker Integration
37+
38+
- [Docker Conversion](./tutorials/docker-conversion.md) - How to convert Docker images to flists
39+
- [End-to-End Docker Workflow](./tutorials/end-to-end-docker-workflow.md) - Complete workflow for Docker conversion
40+
41+
### Server and Distribution
42+
43+
- [Server Setup](./tutorials/server-setup.md) - How to set up the server functionality
44+
- [Website Publishing](./tutorials/website-publishing.md) - How to publish websites using RFS
45+
- [Syncing Files](./tutorials/syncing-files.md) - How to sync files between RFS servers
46+
47+
### User Guides
48+
49+
The user guides provide detailed information about using specific features of the `rfs` command.
3850

39-
## Extra
51+
- [RFS Command Reference](./user-guides/rfs-cli.md) - Comprehensive command reference
52+
- [RFS Server Guide](./user-guides/fl-server.md) - Server configuration and management
53+
- [Web Interface Guide](./user-guides/frontend.md) - Using the web interface
54+
- [Performance Tuning](./user-guides/performance-tuning.md) - Optimizing performance
55+
- [Troubleshooting](./user-guides/troubleshooting.md) - Common issues and solutions
4056

41-
the `extra` table holds any **optional** data associated to the inode based on its type. For now it holds the `link target` for symlink inodes.
57+
## Command Overview
4258

43-
## Tag
59+
The `rfs` command provides all the functionality you need to work with flists:
4460

45-
tag is key value for some user defined data associated with the FL. The standard keys are:
61+
```bash
62+
# Core functionality
63+
rfs pack # Create flists from directories
64+
rfs mount # Mount flists as filesystems
65+
rfs unpack # Extract flist contents to a directory
66+
rfs clone # Copy data between stores
67+
rfs config # Manage flist metadata and stores
68+
69+
# Docker conversion
70+
rfs docker # Convert Docker images to flists
71+
72+
# Server functionality
73+
rfs server # Run the RFS server for web-based management
74+
75+
# Server API interaction
76+
rfs upload # Upload a file to a server
77+
rfs upload-dir # Upload a directory to a server
78+
rfs download # Download a file from a server
79+
rfs download-dir # Download a directory from a server
80+
rfs exists # Check if a file exists on a server
81+
rfs flist create # Create an flist on a server
82+
rfs website-publish # Publish a website to a server
83+
```
84+
85+
For detailed information about each command, use the `--help` flag:
86+
87+
```bash
88+
rfs --help
89+
rfs pack --help
90+
rfs mount --help
91+
```
4692

47-
- `version`
48-
- `description`
49-
- `author`
93+
## Getting Help
5094

51-
But an FL author can add other custom keys there
95+
If you encounter issues or have questions that aren't addressed in the documentation:
5296

53-
## Block
97+
1. Check the [Troubleshooting Guide](./user-guides/troubleshooting.md) for common issues and solutions.
98+
2. Use the `--help` flag with any command for detailed usage information.
99+
3. Search for similar issues on the [RFS GitHub repository](https://github.com/threefoldtech/rfs/issues).
100+
4. Open a new issue if needed.
54101

55-
the `block` table is used to associate data file blocks with files. An `id` field is the blob `id` in the `store`, the `key` is the key used to decrypt the blob. The current implementation of `rfs` does the following:
102+
## Contributing to Documentation
56103

57-
- For each blob (512k) the `sha256`. This becomes the encryption key of the block. We call it `key`
58-
- The block is then `snap` compressed
59-
- Then encrypted with `aes_gcm` using the `key`, and the first 12 bytes of the key as `nonce`
60-
- The final encrypted blocked is hashed again with `sha256` this becomes the `id` of the block
61-
- The final encrypted blob is then sent to the store using the `id` as a key.
104+
We welcome contributions to improve this documentation. If you find errors, omissions, or areas that could be clarified:
62105

63-
## Route
106+
1. Fork the repository
107+
2. Make your changes
108+
3. Submit a pull request
64109

65-
the route table holds routing information for the blobs. It basically describe where to find `blobs` with certain `ids`. The routing is done as following:
110+
Please follow the existing style and structure when making changes.
66111

67-
> Note routing table is loaded one time when `rfs` is started.
112+
## License
68113

69-
- We use the first byte of the blob `id` as the `route key`
70-
- The `route key` is then consulted against the routing table
71-
- While building an `FL` all matching stores are updated with the new blob. This is how the system does replication
72-
- On `getting` an object, the list of matching routes are tried in random order the first one to return a value is used
73-
- Note that same range and overlapping ranges are allowed, this is how shards and replications are done.
114+
This documentation is licensed under the [Apache License 2.0](../LICENSE).

‎docs/architecture/components.md‎

Lines changed: 139 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,139 @@
1+
# RFS Command Components
2+
3+
This document provides detailed information about the main components of the RFS command and how they work together.
4+
5+
## The RFS Command Structure
6+
7+
The `rfs` command is a comprehensive tool that integrates multiple functionalities into a single command-line interface. It's organized into several logical components, all accessible through subcommands:
8+
9+
```
10+
rfs
11+
├── Core Functionality
12+
│ ├── pack
13+
│ ├── mount
14+
│ ├── unpack
15+
│ ├── clone
16+
│ └── config
17+
├── Docker Functionality
18+
│ └── docker
19+
├── Server Functionality
20+
│ └── server
21+
└── Client Functionality
22+
├── upload
23+
├── upload-dir
24+
├── download
25+
├── download-dir
26+
├── exists
27+
├── flist create
28+
└── website-publish
29+
```
30+
31+
## 1. Core Functionality
32+
33+
The core functionality of the `rfs` command provides the fundamental operations for working with flists.
34+
35+
### Key Features
36+
37+
- **Flist Creation** (`rfs pack`): Processes directories and creates compact metadata representations (flists)
38+
- **Mounting** (`rfs mount`): Implements FUSE-based mounting of flists as read-only filesystems
39+
- **Extraction** (`rfs unpack`): Allows downloading and extracting flist contents to a local directory
40+
- **Store Management** (`rfs config`): Manages storage backends and flist metadata
41+
- **Data Cloning** (`rfs clone`): Copies data between storage backends
42+
43+
### Core Modules
44+
45+
- **fungi**: Handles the flist metadata format (reading/writing)
46+
- **store**: Manages different storage backends (dir, zdb, s3, http)
47+
- **cache**: Implements caching of downloaded content
48+
- **fs**: Provides filesystem operations and FUSE integration
49+
50+
## 2. Docker Functionality
51+
52+
The Docker functionality (`rfs docker`) allows converting Docker images into flists.
53+
54+
### Key Features
55+
56+
- **Docker Image Extraction**: Pulls and extracts Docker images
57+
- **Layer Processing**: Handles Docker image layers and filesystem differences
58+
- **Metadata Preservation**: Preserves Docker image metadata in the resulting flist
59+
- **Multi-Store Support**: Supports uploading to multiple storage backends
60+
61+
### Implementation
62+
63+
The Docker functionality leverages Docker's API to extract image contents and then processes them using the core modules to create flists. This integration allows users to easily convert Docker images to flists using the same command-line tool they use for other flist operations.
64+
65+
## 3. Server Functionality
66+
67+
The server functionality (`rfs server`) provides a REST API for managing flists.
68+
69+
### Key Features
70+
71+
- **User Authentication**: Supports JWT-based authentication
72+
- **Flist Management**: APIs for creating, listing, and downloading flists
73+
- **Docker Integration**: Supports creating flists from Docker images
74+
- **Multi-User Support**: Manages flists for multiple users
75+
76+
### Components
77+
78+
- **Authentication Module**: Handles user authentication and authorization
79+
- **Block Handlers**: Manages content blocks (chunks of file data)
80+
- **File Handlers**: Manages file operations
81+
- **Website Handlers**: Serves the web interface
82+
- **Database Integration**: Stores user and flist metadata
83+
84+
## 4. Client Functionality
85+
86+
The client functionality provides commands for interacting with the server API.
87+
88+
### Key Features
89+
90+
- **File Upload** (`rfs upload`, `rfs upload-dir`): Upload files and directories to the server
91+
- **File Download** (`rfs download`, `rfs download-dir`): Download files and directories from the server
92+
- **File Verification** (`rfs exists`): Check if files exist on the server
93+
- **Flist Creation** (`rfs flist create`): Create flists on the server
94+
- **Website Publishing** (`rfs website-publish`): Publish websites to the server
95+
96+
### Implementation
97+
98+
The client functionality communicates with the server API to perform operations. It provides a command-line interface for operations that would otherwise require direct API calls.
99+
100+
## 5. Web Interface
101+
102+
When the server functionality is running, it serves a web interface that provides a graphical user interface for interacting with the server API.
103+
104+
### Key Features
105+
106+
- **User Authentication**: Login interface for accessing the system
107+
- **Flist Creation**: Interface for creating flists from Docker images
108+
- **Flist Management**: Listing, previewing, and downloading flists
109+
- **User-Friendly Design**: Intuitive interface for managing flists
110+
111+
## Integration Between Components
112+
113+
The components of the `rfs` command work together to provide a complete system:
114+
115+
1. **Core + Docker**: The Docker functionality uses the core modules to create flists from Docker images.
116+
117+
2. **Server + Core**: The server functionality uses the core modules to handle flist operations.
118+
119+
3. **Client + Server**: The client functionality communicates with the server API to perform operations.
120+
121+
4. **Web Interface + Server**: The web interface communicates with the server API through HTTP requests.
122+
123+
5. **All Components + Storage Backends**: All components interact with various storage backends to store and retrieve content.
124+
125+
## Deployment Scenarios
126+
127+
The `rfs` command can be used in various deployment scenarios:
128+
129+
1. **Local Usage**: Using the core and Docker functionality locally for creating and mounting flists.
130+
131+
2. **Server Deployment**: Running the server functionality to provide a central service for managing flists.
132+
133+
3. **Client-Server Usage**: Using the client functionality to interact with a remote server.
134+
135+
4. **Web Interface Usage**: Using the web interface to interact with the server through a browser.
136+
137+
## Next Steps
138+
139+
For more information about how data flows through these components, see the [Data Flow Documentation](./data-flow.md).

0 commit comments

Comments
 (0)