Skip to content

Commit

Permalink
feat: improve storage schema descriptions and defaults
Browse files Browse the repository at this point in the history
  • Loading branch information
msudgh committed Aug 18, 2024
1 parent 740c827 commit 258e3d8
Show file tree
Hide file tree
Showing 2 changed files with 97 additions and 121 deletions.
158 changes: 53 additions & 105 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,45 +5,30 @@
[![Codecov Status](https://codecov.io/gh/msudgh/sync-cloud-storage/branch/main/graph/badge.svg?token=2BY6063VOY)](https://codecov.io/gh/msudgh/sync-cloud-storage)
[![License](https://img.shields.io/github/license/msudgh/sync-cloud-storage)](LICENSE)

Synchronize files and directories between a remote computer and a cloud storage through cloud formation.

Supported cloud formation providers:

- [x] Serverless
- [x] CDK

Supported cloud storages:

- [x] AWS S3
Synchronize files and directories between a remote machine and a cloud storage via cloud frameworks and stacks consisting of [AWS SAM (Serverless)](https://www.serverless.com/) and [AWS Cloud Development Kit (CDK)](https://aws.amazon.com/cdk/). This package supports the following cloud storage providers: [AWS S3](https://aws.amazon.com/s3/).

## Features

- Sync multiple storages at once and flexible file matching (single or multiple file/dir sync) by defining patterns of [`glob`](<https://en.wikipedia.org/wiki/Glob_(programming)>) to include or exclude
- Supports a set of options as following for each file based on storage: `Prefix`, `Access Control List (ACL)`, `Tags`, `Metadata`
- Select a list of specific sync actions for each storage: `uploading`, `deleting`
- Modern and uses the latest official cloud provider's SDK
- AWS S3: [`[email protected]`](https://www.npmjs.com/package/@aws-sdk/client-s3)
- Sync multiple storages at once and flexible file matching (single or multiple file/dir sync) by just defining patterns of [`glob`](<https://en.wikipedia.org/wiki/Glob_(programming)>) to include or exclude
- Set the following configs for each synced file:
- prefix
- access control list (ACL)
- tags
- metadata
- Select a list of specific sync actions for each storage:
- upload
- delete
- Written in **TypeScript** with strong type checking
- Integration test with cloud providers

## Installation

1. [**ni**](https://github.com/antfu/ni): `ni sync-cloud-storage -D`
2. [**npm**](https://npmjs.com/): `npm i sync-cloud-storage -D`
3. [**yarn**](https://yarnpkg.com/): `yarn add sync-cloud-storage -D`
4. [**pnpm**](https://pnpm.io/): `pnpm add sync-cloud-storage -D`
- [**npm**](https://npmjs.com/): `npm i sync-cloud-storage`
- [**yarn**](https://yarnpkg.com/): `yarn add sync-cloud-storage`
- [**pnpm**](https://pnpm.io/): `pnpm add sync-cloud-storage`
- [**ni**](https://github.com/antfu/ni): `ni sync-cloud-storage`

## Usage

### Serverless
### AWS S3

#### AWS S3
#### Serverless

Sync storages action as a pre-deploy hook in the `serverless.yml`:

```yaml
plugins:
Expand All @@ -64,9 +49,9 @@ custom:
bar: foo
```
### CDK
#### CDK
#### AWS S3
Call sync storages action after setting up a CDK App:
```typescript
import { Stack, App } from '@aws-cdk/core'
Expand All @@ -89,81 +74,44 @@ const syncCloudStorage = new SyncCloudStorage(stack, {
},
],
})

// Sync storages
syncCloudStorage.storages()

// Sync tags
syncCloudStorage.tags()

// Sync metadata
syncCloudStorage.metadata()
```

## Configuration Reference

This section provides a detailed reference for all configuration options.

- `name`:

- Type: `string`
- Required: `true`
- Minimum length: 1
- Example: `name: assets`

- `patterns`: File patterns to include or exclude during synchronization.

- Type: `array` of `string`
- Required: `true`
- Minimum items: 1
- Example:
```yaml
patterns:
- 'assets/*'
- '!assets/temp/*'
```
- `actions`: List of actions to perform during synchronization.

- Type: `array` of `string`
- Required: `true`
- Valid Values: `upload`, `delete`
- Example:
```yaml
actions:
- upload
- delete
```

- `prefix`: The prefix to apply to all synced files in the bucket.

- Type: `string`
- Required: `false`
- Default: `""`
- Example: `prefix: assets`

- `enabled`: Whether to enable the sync for the storage.

- Type: `boolean`
- Required: `false`
- Default: `true`

- `acl`: Access control list setting for the synced files.

- Type: `string`
- Required: `false`
- Valid Values: `private`, `public-read`, `public-read-write`, `authenticated-read`
- Reference: [AWS S3 Canned ACL](https://docs.aws.amazon.com/AmazonS3/latest/userguide/acl-overview.html#canned-acl)
- Example: `acl: public-read`

- `metadata`: Custom metadata to apply to the synced files.

- Type: `object`
- Required: `false`
- Example:
```yaml
metadata:
foo: bar
bar: foo
```

- `tags`: Custom tags to apply to the synced files.
- Type: `object`
- Required: `false`
- Default: `{}`
- Example:
```yaml
tags:
environment: production
```
## Options

### General

| Option | Notes | Type | Required | Default |
| -------- | ---------------------------------- | -------------------------------- | -------- | ---------------------------------------------------- |
| storages | List of storages, Minimum items: 1 | `array` of [`storage`](#storage) | true | undefined |
| region | Cloud (AWS) region | `string` | false | undefined or `AWS_REGION` environment variable |
| endpoint | Cloud (AWS) Endpoint URL | `string` | false | undefined or `AWS_ENDPOINT_URL` environment variable |
| offline | Offline mode | `boolean` | false | false or `IS_OFFLINE` environment variable |
| disabled | Disable sync | `boolean` | false | false |
| silent | Silent output logs | `boolean` | false | false |

### Storage

| Option | Notes | Type | Required | Default |
| ----------- | ----------------------------------------------------------------------------------------------------------- | ------------------- | -------- | ------------------ |
| name | Name of storage (AWS S3 Bucket), Minimum length: 1 | `string` | true | undefined |
| patterns | Patterns of [`glob`][glob] paths to include or exclude on sync action, Minimum items: 1 | `array` of `string` | true | undefined |
| actions | Sync actions, Valid values: `upload`, `delete` | `array` of `string` | false | `upload`, `delete` |
| prefix | Prefix for the storage files and folders | `string` | false | `''` |
| enabled | Enable or disable the storage on sync action | `boolean` | false | `true` |
| acl | [AWS S3 Canned ACL][acl], Valid values: `private`, `public-read`, `public-read-write`, `authenticated-read` | `string` | false | undefined |
| metadata | A set of metadata key/value pair to be set or unset on the object | `object` | false | undefined |
| tags | A set of tag key/value pair to be set or unset on the object | `object` | false | `{}` |
| gitignore | Use .gitignore file to exclude files and directories | `boolean` | false | false |
| ignoreFiles | Ignore files and directories to exclude from sync action | `array` of `string` | false | undefined |

[glob]: https://en.wikipedia.org/wiki/Glob_(programming)
[acl]: https://docs.aws.amazon.com/AmazonS3/latest/userguide/acl-overview.html#canned-acl
60 changes: 44 additions & 16 deletions src/schemas/input.ts
Original file line number Diff line number Diff line change
Expand Up @@ -12,34 +12,62 @@ const objectCannedACLs = Object.values(ObjectCannedACL).map(
const tags = z.record(z.string(), z.string())

const storage = z.object({
name: z.string().min(1),
patterns: z.array(z.string()).min(1),
actions: z.array(z.string()).default(['upload', 'delete']),
prefix: z.string().default(''),
enabled: z.boolean().default(true),
acl: z.enum(objectCannedACLs).optional(),
metadata: z.record(z.string(), z.string()).optional(),
tags: z.record(z.string(), z.string()).default({}),
gitignore: z.boolean().default(false),
ignoreFiles: z.array(z.string()).optional(),
name: z.string().min(1).describe('Storage name'),
patterns: z
.array(z.string())
.min(1)
.describe('Patterns of glob paths to include or exclude on sync action'),
actions: z
.array(z.enum(['upload', 'delete']))
.default(['upload', 'delete'])
.describe('Sync actions'),
prefix: z
.string()
.default('')
.describe('Prefix for the storage files and folders'),
enabled: z
.boolean()
.default(true)
.describe('Enable or disable the storage on sync action'),
acl: z.enum(objectCannedACLs).optional().describe('Access control list'),
metadata: z
.record(z.string(), z.string())
.optional()
.describe(
'A set of metadata key/value pair to be set or unset on the object'
),
tags: z
.record(z.string(), z.string())
.default({})
.describe('A set of tag key/value pair to be set or unset on the object'),
gitignore: z
.boolean()
.default(false)
.describe('Use .gitignore file to exclude files and directories'),
ignoreFiles: z
.array(z.string())
.optional()
.describe('Ignore files and directories to exclude from sync action'),
})

const storages = z.array(storage).min(1)
const storages = z.array(storage).min(1).describe('List of storages')

const customOptions = z.object({
syncCloudStorage: z.object({
disabled: z.boolean().optional().default(false),
disabled: z.boolean().optional().default(false).describe('Disable sync'),
storages: storages,
endpoint: z
.string()
.optional()
.default(process.env.AWS_ENDPOINT_URL ?? ''),
.default(process.env.AWS_ENDPOINT_URL ?? '')
.describe('Cloud (AWS) Endpoint URL'),
offline: z
.boolean()
.optional()
.default(process.env.IS_OFFLINE === 'true'),
region: z.string().optional(),
silent: z.boolean().optional(),
.default(process.env.IS_OFFLINE === 'true')
.describe('Offline mode'),
region: z.string().optional().describe('Cloud (AWS) region'),
silent: z.boolean().optional().describe('Silent output logs'),
}),
})

Expand Down

0 comments on commit 258e3d8

Please sign in to comment.