Skip to content

Commit

Permalink
Merge pull request #32 from jxsl13/feat/remove-afero-only-support-os-…
Browse files Browse the repository at this point in the history
…filesystems

feat: remove afero dependency, only support os filesystems
  • Loading branch information
jxsl13 committed Jul 9, 2024
2 parents 3c3e646 + 627b7aa commit 82c0e49
Show file tree
Hide file tree
Showing 41 changed files with 2,766 additions and 2,305 deletions.
42 changes: 23 additions & 19 deletions .github/workflows/test.yaml
Original file line number Diff line number Diff line change
@@ -1,5 +1,11 @@
on: [push, pull_request]
name: Test

concurrency:
# prevent multiple workflows from running at the same time for the same pr/branch/tag etc.
group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }}
cancel-in-progress: true

jobs:
test:
strategy:
Expand All @@ -8,25 +14,23 @@ jobs:
platform: [ubuntu-latest, macos-latest, windows-latest]
runs-on: ${{ matrix.platform }}
steps:
- name: Install Go
uses: actions/setup-go@v5
with:
go-version: ${{ matrix.go-version }}
- name: Checkout code
uses: actions/checkout@v4

- name: Vet
run: go vet ./...

- name: Code Coverage
run: go test ./... -timeout 30s -race -count=1 -covermode=atomic -coverprofile=coverage.txt
- name: Install Go
uses: actions/setup-go@v5
with:
go-version: ${{ matrix.go-version }}
- name: Checkout code
uses: actions/checkout@v4

- name: Upload coverage to Codecov
uses: codecov/codecov-action@v4
with:
token: ${{ secrets.CODECOV_TOKEN }}
files: ./coverage.txt
fail_ci_if_error: false
verbose: false
- name: Vet
run: go vet ./...

- name: Code Coverage
run: go test ./... -timeout 30s -race -count=1 -covermode=atomic -coverprofile=coverage.txt

- name: Upload coverage to Codecov
uses: codecov/codecov-action@v4
with:
token: ${{ secrets.CODECOV_TOKEN }}
files: ./coverage.txt
fail_ci_if_error: false
verbose: false
2 changes: 2 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -1,2 +1,4 @@
coverage
.vscode/*
tmp/
main/
9 changes: 6 additions & 3 deletions Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -10,13 +10,16 @@ coverage:
rm ./coverage.txt

fuzz_prefixfs:
gotip clean -testcache && gotip test -fuzz=FuzzPrefixFs -race -fuzztime=300s
go clean -testcache && go test -fuzz=FuzzPrefixFS -race -fuzztime=300s

fuzz_hiddenfs_create:
gotip clean -testcache && gotip test -fuzz=FuzzHiddenFsCreate -race -fuzztime=300s
go clean -testcache && go test -fuzz=FuzzHiddenFSCreate -race -fuzztime=300s

fuzz_hiddenfs_remove_all:
gotip clean -testcache && gotip test -fuzz=FuzzHiddenFsRemoveAll -race -fuzztime=300s
go clean -testcache && go test -fuzz=FuzzHiddenFSRemoveAll -race -fuzztime=300s

fmt:
go fmt ./...

gen_mock:
go generate ./...
90 changes: 46 additions & 44 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,8 +1,7 @@

# BackupFs
# BackupFS

Multiple filesystem abstraction layers working together to create a straight forward rollback mechanism for filesystem modifications with OS-independent file paths.
This package provides multiple filesystem abstractions which implement the spf13/afero.Fs interface as well as the optional interfaces.
This package provides multiple filesystem abstractions which implement the spf13/afero.FS interface as well as the optional interfaces.

They require the filesystem modifications to happen via the provided structs of this package.

Expand All @@ -17,14 +16,16 @@ type Command interface {
Undo() error
}
```

A multitude of such commands allows to provision software packages (archives) and configuration files to target systems running some kind of agent software.
Upon detection of invalid configurations or incorrect software, it is possible to rollback the last transaction.

A transaction is also a command containing a list of non-transaction commands embedding and providing a `BackupFs` to its subcommands requiring to execute filesystem operations.
A transaction is also a command containing a list of non-transaction commands embedding and providing a `BackupFS` to its subcommands requiring to execute filesystem operations.

For all commands solely operating on the filesystem the `Undo()` mechanism consists of simply calling `BackupFs.Rollback()`
For all commands solely operating on the filesystem the `Undo()` mechanism consists of simply calling `BackupFS.Rollback()`

Further commands might tackle the topics of:

- un/tar
- creation of files, directories & symlinks
- removal of files, directories & symlinks
Expand All @@ -34,50 +35,53 @@ Further commands might tackle the topics of:
If you try to tackle the rollback/undo problem yourself you will see pretty fast that the rollback mechanism is a pretty complex implementation with lots of pitfalls where this approach might help you out.

If you follow the rule that **filesystem modifying commands**

- creation,
- deletion
- or modification of files, directories and symlinks
- creation of systemd unit files (writing service configuration)

are to be strictly separated from **side effects causing commands**

- creation of linux system users and groups
- start of linux systemd services configured with the above file in the filesystem

then you will have a much easier time!

## VolumeFs
## VolumeFS

`VolumeFs` is a filesystem abstraction layer that hides Windows volumes from file system operations.
`VolumeFS` is a filesystem abstraction layer that hides Windows volumes from file system operations.
It allows to define a volume of operation like `c:` or `C:` which is then the only volume that can be accessed.
This abstraction layer allows to operate on filesystems with operating system independent paths.

## PrefixFs
## PrefixFS

`PrefixFs` forces a filesystem to have a specific prefix.
`PrefixFS` forces a filesystem to have a specific prefix.
Any attempt to escape the prefix path by directory traversal is prevented, forcing the application to stay within the designated prefix directory.
This prefix makes the directory basically the application's root directory.

## BackupFs
## BackupFS

The most important part of this library is `BackupFs`.
The most important part of this library is `BackupFS`.
It is a filesystem abstraction that consists of two parts.
A base filesystem and a backup filesystem.
Any attempt to modify a file, directory or symlink in the base filesystem leads to the file being backed up to the backup filesystem.

Consecutive file modifications are ignored, as the initial file state has already been backed up.

## HiddenFs
## HiddenFS

HiddenFs has a single purpose, that is to hide your backup location and prevent your application from seeing or modifying it.
In case you use BackupFs to backup files that are overwritten on your operating system filesystem (OsFs), you want to define multiple filesystem layers that work together to prevent you from creating a non-terminating recursion of file backups.
HiddenFS has a single purpose, that is to hide your backup location and prevent your application from seeing or modifying it.
In case you use BackupFS to backup files that are overwritten on your operating system filesystem (OsFS), you want to define multiple filesystem layers that work together to prevent you from creating a non-terminating recursion of file backups.

- The zero'th layer is the underlying real filesystem, be it the OsFs, MemMapFs, etc.
- The first layer is a VolumeFs filesystem abstraction that removes the need to provide a volume prefix for absolute file paths when accessing files on the underlying filesystem (Windows)
- The second layer is a PrefixFs that is provided a prefix path (backup directory location) and the above instantiated filesystem (e.g. OsFs)
- The third layer is HiddenFs which takes the backup location as path that needs hiding and wraps the first layer in itself.
- The fourth layer is the BackupFs layer which takes the third layer as underlying filesystem to operate on (backup location is not accessible nor viewable) and the second PrefixFs layer to backup your files to.
- The zero'th layer is the underlying real filesystem, be it the OsFS, MemMapFS, etc.
- The first layer is a VolumeFS filesystem abstraction that removes the need to provide a volume prefix for absolute file paths when accessing files on the underlying filesystem (Windows)
- The second layer is a PrefixFS that is provided a prefix path (backup directory location) and the above instantiated filesystem (e.g. OsFS)
- The third layer is HiddenFS which takes the backup location as path that needs hiding and wraps the first layer in itself.
- The fourth layer is the BackupFS layer which takes the third layer as underlying filesystem to operate on (backup location is not accessible nor viewable) and the second PrefixFS layer to backup your files to.

At the end you will create something along the lines of:

```go
package main

Expand All @@ -86,39 +90,37 @@ import (
"path/filepath"

"github.com/jxsl13/backupfs"
"github.com/spf13/afero"
)

func main() {

var (
// first layer: abstracts away the volume prefix (on Unix the it is an empty string)
volume = filepath.VolumeName(os.Args[0]) // determined from application path
base = backupfs.NewVolumeFs(volume, afero.NewMemMapFs())
base = backupfs.NewVolumeFS(volume, backupfs.NewOSFS())
backupPath = "/var/opt/app/backups"

// second layer: abstracts away a path prefix
backup = backupfs.NewPrefixFs(backupPath, base)
backup = backupfs.NewPrefixFS(base, backupPath)

// third layer: hides the backup location in order to prevent recursion
masked = backupfs.NewHiddenFs(backupPath, base)
masked = backupfs.NewHiddenFS(base, backupPath)

// fourth layer: backup on write filesystem with rollback
backupFs = backupfs.NewBackupFs(masked, backup)
backupFS = backupfs.NewBackupFS(masked, backup)
)
// you may use backupFs at this point like the os package
// except for the backupFs.Rollback() machanism which
// you may use backupFS at this point like the os package
// except for the backupFS.Rollback() machanism which
// allows you to rollback filesystem modifications.
}

```

## Example

We create a base filesystem with an initial file in it.
Then we define a backup filesystem as subdirectory of the base filesystem.

Then we do wrap the base filesystem and the backup filesystem in the `BackupFs` wrapper and try modifying the file through the `BackupFs` file system layer which has initiall ybeen created in the base filesystem. So `BackupFs` tries to modify an already existing file leading to it being backedup. A call to `BackupFs.Rollback()` allows to rollback the filesystem modifications done with `BackupFs` back to its original state while also deleting the backup.
Then we do wrap the base filesystem and the backup filesystem in the `BackupFS` wrapper and try modifying the file through the `BackupFS` file system layer which has initiall ybeen created in the base filesystem. So `BackupFS` tries to modify an already existing file leading to it being backedup. A call to `BackupFS.Rollback()` allows to rollback the filesystem modifications done with `BackupFS` back to its original state while also deleting the backup.

```go
package main
Expand All @@ -129,7 +131,6 @@ import (
"os"

"github.com/jxsl13/backupfs"
"github.com/spf13/afero"
)

func checkErr(err error) {
Expand All @@ -143,29 +144,30 @@ func main() {

var (
// base filesystem
baseFs = afero.NewMemMapFs()
baseFS = backupfs.NewPrefixFS(backupfs.NewOSFS(), os.TempDir())
filePath = "/var/opt/test.txt"
)

// create an already existing file in base filesystem
f, err := baseFs.Create(filePath)
f, err := baseFS.Create(filePath)
checkErr(err)

f.WriteString("original text")
f.Close()

// at this point we have the base filesystem ready to be ovwerwritten with new files
// at this point we have the base filesystem ready to be overwritten with new files
var (
// sub directory in base filesystem as backup directory
// where the backups should be stored
backup = backupfs.NewPrefixFs("/var/opt/application/backup", baseFs)
backup = backupfs.NewPrefixFS(baseFS, "/var/opt/application/backup")

// backup on write filesystem
backupFs = backupfs.NewBackupFs(baseFs, backup)
backupFS = backupfs.NewBackupFS(baseFS, backup)
)

// we try to override a file in the base filesystem
f, err = backupFs.Create(filePath)
// but in this case we use the backup on write filesystem
// on top of the base filesystem.
f, err = backupFS.Create(filePath)
checkErr(err)
f.WriteString("new file content")
f.Close()
Expand All @@ -180,11 +182,11 @@ func main() {

b, err := io.ReadAll(f)
checkErr(err)
f.Close()
_ = f.Close()

backedupContent := string(b)

f, err = baseFs.Open(filePath)
f, err = baseFS.Open(filePath)
checkErr(err)
b, err = io.ReadAll(f)
checkErr(err)
Expand All @@ -194,18 +196,18 @@ func main() {
fmt.Println("Overwritten file: ", overwrittenFileContent)
fmt.Println("Backed up file : ", backedupContent)

afs := afero.Afero{Fs: backupFs}
fi, err := afs.ReadDir("/var/opt/")
dir, err := backupFS.Open("/var/opt/")
checkErr(err)
defer dir.Close()

for _, f := range fi {
fmt.Println("Found name: ", f.Name())
fis, err := dir.Readdir(-1)
checkErr(err)
for _, fi := range fis {
fmt.Println("Found name: ", fi.Name())
}

}
```


## TODO

- Add symlink fuzz tests on os filesystem that deletes the symlink after each test.
Loading

0 comments on commit 82c0e49

Please sign in to comment.