GCFS is a metadata management tool for file servers.
GCFS is a tool for building cloud file systems. It provides default Couchbase utilities to store data about your files, located on a distant server.
GCFS is the link between your UI and the distant file. It will provide client adequate information for retrieving any files stored on your custom server.
You can use GCFS in two modes : api mode and methods mode. The first gives you more easy setups, while the second aims for deeper integration in your server application.
You can use both modes together, depending on your configuration.
Before using GCFS, you need a minimal setup.
You need a running Couchbase instance. If you don't want to provide a cluster configuration, you need to setup a local Couchbase instance with the following defaults :
- a server running on 127.0.0.1
- empty username and password to access cluster
- one bucket named "metadata"
To provide custom cluster configuration, please refer to the database section below.
For Api mode, you need your go app to run in server mode on a port. By default and for local development, this port is set to 8080.
package my_package
import "github.com/Alvarios/gcfs"
func main() {
gcfs.Setup(gcfs.Configuration{})
}Configuration interface can be left empty. However, it is required to fill it if you want to use the package in api mode.
coming soon
package my_package
import "github.com/Alvarios/gcfs"
func main() {
gcfs.Setup(gcfs.Configuration{
Database: gcfs.DbConfig{
BucketName: "metadata",
Address: "couchbase://127.0.0.1",
Username: "",
Password: "",
Bucket: nil,
},
Global: gcfs.GlobalConfig{
AutoProvide: false,
Strict: false,
},
Server: gcfs.ServerConfig{
Port: "8080",
},
Metadata: nil,
})
}Provide information to connect to your Couchbase cluster. See Couchbase official documentation for more information.
Alternatively, you can do the setup on your own and pass a Bucket pointer into the Database configuration. You can then ignore other configuration arguments :
package my_package
import (
"github.com/Alvarios/gcfs"
"github.com/couchbase/gocb/v2"
)
func main() {
var myClust *gocb.Cluster
var myBucket *gocb.Bucket
myClust = gocb.Connect(...) // Enter your cluster configuration here.
myBucket = myClust.Bucket(bucketName)
gcfs.Setup(gcfs.Configuration{
Database: gcfs.Database{
Bucket: myBucket,
},
})
}Configuration of flags to set the package default behaviors.
Provides a default behavior for the insert method. Default is false.
AutoProvide mode allows insert method to automatically fill up some metadata when not specified in the newly created document.
Only 2 fields currently support the autofill mode : general.creation_time and
general.modification_time. Both are automatically set to current date.
For more detail about metadata, please refer to this section.
Strict mode only enables required metadata for insertion and update of a document. It will reject any attempt to add unauthorized metadata.
Provide server configuration for Api mode.
The port to run the server on. Running a server on a port from any domain
makes it accessible via domain:port url. Domain is localhost for local
server, or 127.x.x.x (range from 127.0.0.1 to 127.255.255.254).
GCFS is a metadata utility, which stores textual information about a distant file. Metadata is used to represent a file without having to load it.
As a service, GCFS provides some pre-configured metadata. This metadata is present on every file handled by GCFS, and serves as a standard representation.
Some pre-configured metadata is required, while the rest can be left blank. They are listed below.
| Metadata | Required | AutoFillable | Type | Description |
|---|---|---|---|---|
| url | true | - | string | url pointing to the actual file. |
| general.name | true | - | string | name of the file. |
| general.format | true | - | string | format of the file. |
| general.size | - | - | int | size of the file in bytes. |
| general.creation_time | true | true | uint64 | date of the file creation. (1) |
| general.modification_time | - | true | uint64 | date of the file last modification. (1) |
(1) In milliseconds since January 1st, 1970, 00:00:00 UTC. Be careful as this format is different from the one used by Go and UNIX systems, calculated from the same date but using nanoseconds. The reason we use milliseconds is to keep consistent with javascript and other web standards (since Couchbase is a JSON document database system).
As Couchbase works with the very permissive JSON format, you are totally free to add any metadata to your file, as long as you leave the default provided ones.
You can even add some metadata under the general key. For example, a general.author to keep track of the file owner. GCFS treats a document like a basic interface{}, so you have total freedom.
Now you may want to work with your own standards. Your system will grow up in a specific direction, and maybe you'd like to require some more metadata. You can do a check on your own in methods mode, but that's time and processor costly. Instead, GCFS provides you a prebuilt and efficient solution, that also works in Api mode.
When adding a new document, GCFS will perform an integrity check to ensure every required field is present with a correctly typed value.
You can add some required fields to check, with the Metadata field inside the Configuration interface.
package my_package
import "github.com/Alvarios/gcfs"
func main() {
gcfs.Setup(gcfs.Configuration{
Metadata: map[string]interface{}{
"general": map[string]interface{}{
"author": "string",
},
"version": "string",
},
})
}For example, the above configuration will force every document to have a
general.author and a version field, both of type string. If a document
miss one of the above fields, it will be refused and the insertion will
return an error.
Metadata consist of key-value pairs : key points to a field, and value is a
string representing a go type. To access nested fields, declare the parent
field as a hardcoded map[string]interface{}, then every key declared
inside will be considered as a child of its parent.
GCFS provides methods for interacting with your metadata. They wrap the gocb methods and add some useful checks and syntax.
fileId, err := gcfs.Insert(fileMetadata, fileId)
Insert metadata and returns the id of the generated document.
arguments
| Name | Type | Description |
|---|---|---|
| fileMetadata | map[string]interface{} (1) | metadata document. |
| fileId | string | optional file id. |
(1) The map has to contain the required metadata, if not autofilled.
package my_package
import (
"github.com/Alvarios/gcfs"
"log"
)
func main() {
// Works with AutoFill = true.
data := map[string]interface{}{
"url": "/path/to/my/file",
"general": gcfs.GeneralMetadata{ // You can also pass a map[string]interface{}, as long as it contains every required field.
Name: "my awesome file",
Format: "txt",
},
"another key": 123456,
}
// Leave id blank will generate a unique id.
fileId, err := gcfs.Insert(data, "")
log.Println(err == (*gcfs.Error)(nil)) // true
// Fails with missing url error.
data := map[string]interface{}{
"general": map[string]interface{}{
"name": "my awesome file",
"format": "txt",
},
"another key": 123456,
}
fileId, err = gcfs.Insert(data, "")
log.Println(err == (*gcfs.Error)(nil)) // false
}return value
| Name | Type | Description |
|---|---|---|
| fileId | string | id pointing to the newly created document. |
| err | Error object | - |
Shortcut for InsertFlagged. Allows to set custom flag and ignore Global configuration for a specific action.
fileId, err := gcfs.InsertF(fileMetadata, fileId, flags)
Insert metadata and returns the id of the generated tuple.
arguments
| Name | Type | Description |
|---|---|---|
| fileMetadata | map[string]interface{} | file metadata object. |
| fileId | string | optional file id. |
| flags | gcfs.InsertFlags | override default configuration. |
return value
| Name | Type | Description |
|---|---|---|
| fileId | string | id pointing to the newly created document. |
| err | Error object | - |
package my_package
import (
"github.com/Alvarios/gcfs"
"log"
)
func main() {
// Works with AutoFill = true.
data := map[string]interface{}{
"url": "/path/to/my/file",
"general": gcfs.GeneralMetadata{ // You can also pass a map[string]interface{}, as long as it contains every required field.
Name: "my awesome file",
Format: "txt",
},
"another key": 123456,
}
// Leave id blank will generate a unique id.
fileId, err := gcfs.InsertF(data, "", gcfs.InsertFlags{
AutoProvide: false,
Strict: false,
Force: false,
})
log.Println(err == (*gcfs.Error)(nil)) // true
}Autofill the missing metadata when possible.
Toggle strict mode for the current request.
Force document to be inserted in the database by skipping integrity checks. The only way to fail in Force mode is to try an update with a non JSON marshable interface.
fileMetadata, err := gcfs.Get(fileId)
Retrieve metadata from database.
arguments
| Name | Type | Description |
|---|---|---|
| fileId | string | id pointing to the document. |
return value
| Name | Type | Description |
|---|---|---|
| fileMetadata | map[string]interface{} | file metadata object. |
| err | Error object | - |
timestamp, err := gcfs.Update(fileId, updateSpecs)
Perform a partial update of a document.
arguments
| Name | Type | Description |
|---|---|---|
| fileId | string | id pointing to the document. |
| fileMetadata | Update specs | - |
return value
| Name | Type | Description |
|---|---|---|
| timestamp | uint64 | timestamp of the update. |
| err | Error object | - |
GCFS provides an easy, declarative way to perform a partial update of your metadata, using gocb MutateIn function.
updateSpecs := gcfs.UpdateSpec{
Remove: []string{"key1", "key2", "key4.key4-2"},
Upsert: map[string]interface{}{
"key3": "new value",
"key4": map[string]interface{}{
"key4-1": "new value",
},
},
Append: map[string]interface{}{
"array-key": ["newValue1", "newValue2"],
},
Force: false,
}A list of keys to remove from the document. Support the short dot syntax for nested keys.
💡 Tip : remove specs are forbidden in strict mode, when Force flag is set to false.
Update a list of values. Each value in the given map will replace the equivalent one in the original document. If a path doesn't exist, it will be created.
You can also use short dot syntax to access nested keys.
updateSpecs := gcfs.UpdateSpec{
Upsert: map[string]interface{}{
"key3": "new value",
"key4.key4-1": "new value",
},
}Append a list of values to an array key.
💡 Tip : you can declare a non array value to append a single value. An array value will always be treaten with the
HasMultipleflag set to true (see gocb specs). Add a nested array if you want to append an array as an array, and not a list of values.
err := gcfs.Delete(fileId)
Delete metadata from database. Only returns an error.
arguments
| Name | Type | Description |
|---|---|---|
| fileId | string | Id pointing to the document. |
return value
| Name | Type | Description |
|---|---|---|
| err | Error object | - |
autoProvidedData, err = gcfs.AutoProvide(fileMetadata)
Auto fill metadata with default values. More details in the Metadata section.
coreMetadata, err := gcfs.CheckIntegrity(fileMetadata)
Check integrity of a dataset.
💡 Tip : coreMetadata represents the default provided metadata. More details in the Metadata section.
GCFS returns a pointer to an error object adapted to web servers. This error object is retrieved from kushuh-go-utils package.
| Key | Type | Description |
|---|---|---|
| Code | int | The http status of the error. |
| Message | string | Describes the nature of the error. |
💡 Tip : as a custom error object,
err == nilwont work to check for errors. Instead, use the following guard clause :err == (*gcfsresponses.Error)(nil).
This section provides you information if you want to participate in the project.
Once you pulled the project, navigate through your terminal to the root folder
(usually gcfs/). To launch the tests, run :
go test -v ./...
Test command will fail if you don't have a little setup environment.
First of all, you need a Couchbase instance running. You can use one of your existing instances. If so, you can skip the following section and go here.
If you don't have any running instance, the easiest way is to download the Couchbase Server from source.
You can go for the free community edition. Select your platform and make sure you go for the last version (marked as current). Then follow the instruction.
At the end, you should have a Couchbase service running on your computer.
Connect to http://127.0.0.1:8091/ui/index.html from any web browser.
Again, follow the instruction to setup your local Cluster. You can leave
Username and Password blank, since we don't need any security here. Setup
your Cluster on localhost (127.0.0.1).
Finally, create a single bucket named 'metadata'. You can leave every parameter to default.
If you want to use a pre-configured Couchbase instance, or just don't want to comply to default configuration above, then you'll need to set some environment variables before running any test, to tell GCFS where to look for.
You don't need to provide each environment variables : only set one when its value is different in your configuration, from the default configuration below.
| ENV | Default | Description |
|---|---|---|
| GCFS_TEST_ADDRESS | "couchbase://127.0.0.1" | address of your server. |
| GCFS_TEST_USERNAME | "" | username to access the cluster. |
| GCFS_TEST_PASSWORD | "" | password to access the cluster. |
| GCFS_TEST_BUCKETNAME | "metadata" | bucket for running tests. |
For example, if you have a local instance with some credentials, but you run on localhost with a metadata bucket, then you only need to export the following before running any test :
export GCFS_TEST_USERNAME="admin" GCFS_TEST_PASSWORD="123456"
- Configuration > Metadata
- Add a short syntax for nested fields "general.author"
- Ignore spaces and case for type declaration
- Methods
- Search method and api with nefts package
2020 Alvarios - MIT license