Add architecture, concepts, tutorials, user-guides docs and update th…

docs/README.md

@@ -1,73 +1,114 @@
-# FungiList specifications
+# RFS Documentation
 
-## Introduction
+Welcome to the comprehensive documentation for the RFS (Remote File System) command-line tool. This documentation covers all aspects of using the `rfs` command, including its various subcommands, concepts, tutorials, and user guides.
 
-The idea behind the FL format is to build a full filesystem description that is compact and also easy to use from almost ANY language. The format need to be easy to edit by tools like `rfs` or any other tool.
+## Table of Contents
 
-We decided to eventually use `sqlite`! Yes the `FL` file is just a `sqlite` database that has the following [schema](../rfs/schema/schema.sql)
+### Architecture
 
-## Tables
+The architecture documentation provides a high-level overview of the RFS system and how it works.
 
-### Inode
+- [Architecture Overview](./architecture/overview.md) - High-level architecture of the RFS system
+- [Components](./architecture/components.md) - How the different components work together
+- [Data Flow](./architecture/data-flow.md) - How data flows through the system
+- [Storage Backends](./architecture/storage-backends.md) - Details about the supported storage backends
 
-Inode table describe each entry on the filesystem. It matches really closely the same `inode` structure on the linux operating system. Each inode has a unique id called `ino`, a parent `ino`, name, and other parameters (user, group, etc...).
+### Concepts
 
-The type of the `inode` is defined by its `mode` which is a `1:1` mapping from the linux `mode`
+The concepts documentation explains the core concepts and principles behind RFS.
 
-> from the [inode manual](https://man7.org/linux/man-pages/man7/inode.7.html)
+- [Understanding Flists](./concepts/flists.md) - What are flists and how they work
+- [Understanding Storage Backends](./concepts/stores.md) - How storage backends work
+- [Understanding Caching](./concepts/caching.md) - How caching works in RFS
+- [Understanding Sharding and Replication](./concepts/sharding.md) - How sharding and replication work
 
-```
+### Tutorials
 
-POSIX refers to the stat.st_mode bits corresponding to the mask
-S_IFMT (see below) as the file type, the 12 bits corresponding to
-the mask 07777 as the file mode bits and the least significant 9
-bits (0777) as the file permission bits.
+The tutorials provide step-by-step guides for common tasks using the `rfs` command.
 
-The following mask values are defined for the file type:
+### Core Functionality
 
-    S_IFMT     0170000   bit mask for the file type bit field
+- [Getting Started](./tutorials/getting-started.md) - Installation and basic usage
+- [Creating Flists](./tutorials/creating-flists.md) - How to create flists from directories
+- [Mounting Flists](./tutorials/mounting-flists.md) - How to mount and use flists
+- [End-to-End Flist Workflow](./tutorials/end-to-end-flist-workflow.md) - Complete workflow for creating and using flists
 
-    S_IFSOCK   0140000   socket
-    S_IFLNK    0120000   symbolic link
-    S_IFREG    0100000   regular file
-    S_IFBLK    0060000   block device
-    S_IFDIR    0040000   directory
-    S_IFCHR    0020000   character device
-    S_IFIFO    0010000   FIFO
-```
+### Docker Integration
+
+- [Docker Conversion](./tutorials/docker-conversion.md) - How to convert Docker images to flists
+- [End-to-End Docker Workflow](./tutorials/end-to-end-docker-workflow.md) - Complete workflow for Docker conversion
+
+### Server and Distribution
+
+- [Server Setup](./tutorials/server-setup.md) - How to set up the server functionality
+- [Website Publishing](./tutorials/website-publishing.md) - How to publish websites using RFS
+- [Syncing Files](./tutorials/syncing-files.md) - How to sync files between RFS servers
+
+### User Guides
+
+The user guides provide detailed information about using specific features of the `rfs` command.
 
-## Extra
+- [RFS Command Reference](./user-guides/rfs-cli.md) - Comprehensive command reference
+- [RFS Server Guide](./user-guides/fl-server.md) - Server configuration and management
+- [Web Interface Guide](./user-guides/frontend.md) - Using the web interface
+- [Performance Tuning](./user-guides/performance-tuning.md) - Optimizing performance
+- [Troubleshooting](./user-guides/troubleshooting.md) - Common issues and solutions
 
-the `extra` table holds any **optional** data associated to the inode based on its type. For now it holds the `link target` for symlink inodes.
+## Command Overview
 
-## Tag
+The `rfs` command provides all the functionality you need to work with flists:
 
-tag is key value for some user defined data associated with the FL. The standard keys are:
+```bash
+# Core functionality
+rfs pack        # Create flists from directories
+rfs mount       # Mount flists as filesystems
+rfs unpack      # Extract flist contents to a directory
+rfs clone       # Copy data between stores
+rfs config      # Manage flist metadata and stores
+
+# Docker conversion
+rfs docker      # Convert Docker images to flists
+
+# Server functionality
+rfs server      # Run the RFS server for web-based management
+
+# Server API interaction
+rfs upload      # Upload a file to a server
+rfs upload-dir  # Upload a directory to a server
+rfs download    # Download a file from a server
+rfs download-dir # Download a directory from a server
+rfs exists      # Check if a file exists on a server
+rfs flist create # Create an flist on a server
+rfs website-publish # Publish a website to a server
+```
+
+For detailed information about each command, use the `--help` flag:
+
+```bash
+rfs --help
+rfs pack --help
+rfs mount --help
+```
 
-- `version`
-- `description`
-- `author`
+## Getting Help
 
-But an FL author can add other custom keys there
+If you encounter issues or have questions that aren't addressed in the documentation:
 
-## Block
+1. Check the [Troubleshooting Guide](./user-guides/troubleshooting.md) for common issues and solutions.
+2. Use the `--help` flag with any command for detailed usage information.
+3. Search for similar issues on the [RFS GitHub repository](https://github.com/threefoldtech/rfs/issues).
+4. Open a new issue if needed.
 
-the `block` table is used to associate data file blocks with files. An `id` field is the blob `id` in the `store`, the `key` is the key used to decrypt the blob. The current implementation of `rfs` does the following:
+## Contributing to Documentation
 
-- For each blob (512k) the `sha256`. This becomes the encryption key of the block. We call it `key`
-- The block is then `snap` compressed
-- Then encrypted with `aes_gcm` using the `key`, and the first 12 bytes of the key as `nonce`
-- The final encrypted blocked is hashed again with `sha256` this becomes the `id` of the block
-- The final encrypted blob is then sent to the store using the `id` as a key.
+We welcome contributions to improve this documentation. If you find errors, omissions, or areas that could be clarified:
 
-## Route
+1. Fork the repository
+2. Make your changes
+3. Submit a pull request
 
-the route table holds routing information for the blobs. It basically describe where to find `blobs` with certain `ids`. The routing is done as following:
+Please follow the existing style and structure when making changes.
 
-> Note routing table is loaded one time when `rfs` is started.
+## License
 
-- We use the first byte of the blob `id` as the `route key`
-- The `route key` is then consulted against the routing table
-- While building an `FL` all matching stores are updated with the new blob. This is how the system does replication
-- On `getting` an object, the list of matching routes are tried in random order the first one to return a value is used
-- Note that same range and overlapping ranges are allowed, this is how shards and replications are done.
+This documentation is licensed under the [Apache License 2.0](../LICENSE).

docs/architecture/components.md

@@ -0,0 +1,139 @@
+# RFS Command Components
+
+This document provides detailed information about the main components of the RFS command and how they work together.
+
+## The RFS Command Structure
+
+The `rfs` command is a comprehensive tool that integrates multiple functionalities into a single command-line interface. It's organized into several logical components, all accessible through subcommands:
+
+```
+rfs
+├── Core Functionality
+│   ├── pack
+│   ├── mount
+│   ├── unpack
+│   ├── clone
+│   └── config
+├── Docker Functionality
+│   └── docker
+├── Server Functionality
+│   └── server
+└── Client Functionality
+    ├── upload
+    ├── upload-dir
+    ├── download
+    ├── download-dir
+    ├── exists
+    ├── flist create
+    └── website-publish
+```
+
+## 1. Core Functionality
+
+The core functionality of the `rfs` command provides the fundamental operations for working with flists.
+
+### Key Features
+
+- **Flist Creation** (`rfs pack`): Processes directories and creates compact metadata representations (flists)
+- **Mounting** (`rfs mount`): Implements FUSE-based mounting of flists as read-only filesystems
+- **Extraction** (`rfs unpack`): Allows downloading and extracting flist contents to a local directory
+- **Store Management** (`rfs config`): Manages storage backends and flist metadata
+- **Data Cloning** (`rfs clone`): Copies data between storage backends
+
+### Core Modules
+
+- **fungi**: Handles the flist metadata format (reading/writing)
+- **store**: Manages different storage backends (dir, zdb, s3, http)
+- **cache**: Implements caching of downloaded content
+- **fs**: Provides filesystem operations and FUSE integration
+
+## 2. Docker Functionality
+
+The Docker functionality (`rfs docker`) allows converting Docker images into flists.
+
+### Key Features
+
+- **Docker Image Extraction**: Pulls and extracts Docker images
+- **Layer Processing**: Handles Docker image layers and filesystem differences
+- **Metadata Preservation**: Preserves Docker image metadata in the resulting flist
+- **Multi-Store Support**: Supports uploading to multiple storage backends
+
+### Implementation
+
+The Docker functionality leverages Docker's API to extract image contents and then processes them using the core modules to create flists. This integration allows users to easily convert Docker images to flists using the same command-line tool they use for other flist operations.
+
+## 3. Server Functionality
+
+The server functionality (`rfs server`) provides a REST API for managing flists.
+
+### Key Features
+
+- **User Authentication**: Supports JWT-based authentication
+- **Flist Management**: APIs for creating, listing, and downloading flists
+- **Docker Integration**: Supports creating flists from Docker images
+- **Multi-User Support**: Manages flists for multiple users
+
+### Components
+
+- **Authentication Module**: Handles user authentication and authorization
+- **Block Handlers**: Manages content blocks (chunks of file data)
+- **File Handlers**: Manages file operations
+- **Website Handlers**: Serves the web interface
+- **Database Integration**: Stores user and flist metadata
+
+## 4. Client Functionality
+
+The client functionality provides commands for interacting with the server API.
+
+### Key Features
+
+- **File Upload** (`rfs upload`, `rfs upload-dir`): Upload files and directories to the server
+- **File Download** (`rfs download`, `rfs download-dir`): Download files and directories from the server
+- **File Verification** (`rfs exists`): Check if files exist on the server
+- **Flist Creation** (`rfs flist create`): Create flists on the server
+- **Website Publishing** (`rfs website-publish`): Publish websites to the server
+
+### Implementation
+
+The client functionality communicates with the server API to perform operations. It provides a command-line interface for operations that would otherwise require direct API calls.
+
+## 5. Web Interface
+
+When the server functionality is running, it serves a web interface that provides a graphical user interface for interacting with the server API.
+
+### Key Features
+
+- **User Authentication**: Login interface for accessing the system
+- **Flist Creation**: Interface for creating flists from Docker images
+- **Flist Management**: Listing, previewing, and downloading flists
+- **User-Friendly Design**: Intuitive interface for managing flists
+
+## Integration Between Components
+
+The components of the `rfs` command work together to provide a complete system:
+
+1. **Core + Docker**: The Docker functionality uses the core modules to create flists from Docker images.
+
+2. **Server + Core**: The server functionality uses the core modules to handle flist operations.
+
+3. **Client + Server**: The client functionality communicates with the server API to perform operations.
+
+4. **Web Interface + Server**: The web interface communicates with the server API through HTTP requests.
+
+5. **All Components + Storage Backends**: All components interact with various storage backends to store and retrieve content.
+
+## Deployment Scenarios
+
+The `rfs` command can be used in various deployment scenarios:
+
+1. **Local Usage**: Using the core and Docker functionality locally for creating and mounting flists.
+
+2. **Server Deployment**: Running the server functionality to provide a central service for managing flists.
+
+3. **Client-Server Usage**: Using the client functionality to interact with a remote server.
+
+4. **Web Interface Usage**: Using the web interface to interact with the server through a browser.
+
+## Next Steps
+
+For more information about how data flows through these components, see the [Data Flow Documentation](./data-flow.md).