Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: user selectable csv delimiter #1975

Open
wants to merge 181 commits into
base: data-tables-v1
Choose a base branch
from
Open
Changes from 1 commit
Commits
Show all changes
181 commits
Select commit Hold shift + click to select a range
5af7552
Create a storage trait and implement for s3 and filesystem
ddimaria Jul 30, 2024
0a0ba6b
Implement storage when reading/writing in the file service
ddimaria Jul 30, 2024
e0f760c
Serve files from the file service
ddimaria Jul 30, 2024
26a5789
Abstract jwt and error functionality
ddimaria Jul 30, 2024
e7a0eea
Ignore storage contents
ddimaria Jul 30, 2024
b8c1fe4
Rename aws dir to storage
ddimaria Jul 30, 2024
550b484
Handle thumbnail uploads, start presigned URL work
ddimaria Aug 1, 2024
4d613db
Presigned URLs, cleanup, documentation, tests
ddimaria Aug 2, 2024
310c8ab
Cleanup
ddimaria Aug 2, 2024
afec88e
Merge remote-tracking branch 'origin/qa' into simple-file-storage
ddimaria Aug 5, 2024
8c93d43
Merge remote-tracking branch 'origin/qa' into simple-file-storage
ddimaria Aug 5, 2024
159e42a
Merge remote-tracking branch 'origin/qa' into simple-file-storage
ddimaria Aug 5, 2024
c384653
Fix jest test mock
ddimaria Aug 5, 2024
443ab19
Fix docker compose down command in quadratic-connection
ddimaria Aug 5, 2024
f6a3e6b
Fix quadratic-api test mocks
ddimaria Aug 5, 2024
82df198
Merge branch 'qa' into simple-file-storage
AyushAgrawal-A2 Aug 5, 2024
35ad668
Abstract auth functionality in API, switch using env var AUTH_TYPE
ddimaria Aug 6, 2024
9ba22f8
Full kratos integration into docker compose
ddimaria Aug 7, 2024
8b5ac52
Abstract auth0Client into a generic AuthClient implementation
ddimaria Aug 8, 2024
b7aed3d
add env var to example files
davidkircos Aug 12, 2024
19f7d63
Full login/logout flow with server integration
ddimaria Aug 13, 2024
1102a07
Add getUsersFromOryByEmail() in api + cleanup
ddimaria Aug 13, 2024
e33cc35
Remove unused
ddimaria Aug 13, 2024
ad1b2b2
Merge branch 'simple-file-storage' into self-hosting-auth-alternative
ddimaria Aug 13, 2024
a4c270a
Merge remote-tracking branch 'origin/qa' into simple-file-storage
ddimaria Aug 14, 2024
2297262
Merge remote-tracking branch 'origin/simple-file-storage' into self-h…
ddimaria Aug 14, 2024
30e7412
iMVP of self-hosting
ddimaria Aug 16, 2024
5f340fe
Merge branch 'qa' into simple-file-storage
AyushAgrawal-A2 Aug 18, 2024
770de01
Checkpoint after full setup, plus non-working TLS
ddimaria Aug 20, 2024
3f8976a
Route the client in self-hosting through caddy
ddimaria Aug 21, 2024
1d1d43c
Embed env vars in docker-compose
ddimaria Aug 21, 2024
298bc07
Create script that downloads self-hosting directory and starts docker
ddimaria Aug 21, 2024
8584061
Enforce self-hosting license
ddimaria Aug 23, 2024
73909aa
Handle licenses
ddimaria Aug 26, 2024
d8c11bf
Update copy of revoked license on the file view
ddimaria Aug 26, 2024
4599956
Ask user for license key in self-hosting and add value to docker-compose
ddimaria Aug 26, 2024
b00d61b
Merge remote-tracking branch 'origin/simple-file-storage' into self-h…
ddimaria Aug 26, 2024
a84179b
Treat invalid licenses like revoked licenses
ddimaria Aug 26, 2024
3e7c743
Don't cache errors or non-active licenses
ddimaria Aug 26, 2024
71457ed
Update copy on share dialog
ddimaria Aug 26, 2024
8bd7c9d
Replace docker compose license key before starting docker-compose
ddimaria Aug 26, 2024
2b53087
Separate apart init.sh and start.sh
ddimaria Aug 26, 2024
f1ef337
Include the url to the self-hosting admin in inti.sh
ddimaria Aug 26, 2024
bddbe98
Re-enable the cache
ddimaria Aug 26, 2024
4dc570a
test image push
davidkircos Aug 26, 2024
8b6d8ed
point towards dev
davidkircos Aug 26, 2024
3d19a2b
public
davidkircos Aug 26, 2024
d442f4e
create in us-east-1
davidkircos Aug 26, 2024
d695e85
use correct URL
davidkircos Aug 27, 2024
9a078dc
use public
davidkircos Aug 27, 2024
016874f
always tag latest as well
davidkircos Aug 27, 2024
03d8f40
change repo to staging
davidkircos Aug 27, 2024
661bc4e
create repo then describe it
davidkircos Aug 27, 2024
85b5f5e
add second tag
davidkircos Aug 27, 2024
2771463
Use ecr images in self-hosting docker-compose
ddimaria Aug 28, 2024
19454ac
Update client Dockerfile
ddimaria Aug 28, 2024
e48f821
Add self-hosting documentation, fix quadratic-api to conditionally load
ddimaria Aug 28, 2024
18e2d12
Revert docker-compose
ddimaria Aug 28, 2024
11e73b6
Add prisma target debian-openssl-3.0.x and add empty passwords
ddimaria Aug 28, 2024
22a8e9e
Run postinstall during the api build, better workaround for auth0/s3 …
ddimaria Aug 28, 2024
9a3b537
Hard code LICENSE_API_URI
ddimaria Aug 28, 2024
1688d1a
Fix axios headers
ddimaria Aug 28, 2024
2f4bb84
Fix typo in init.sh
ddimaria Aug 29, 2024
81f4594
Add more info to self-hosting README
ddimaria Aug 29, 2024
fb3ef81
Merge remote-tracking branch 'origin/qa' into self-hosting-setup
ddimaria Aug 29, 2024
c848618
Fix all broken things from the merge and beyond
ddimaria Aug 29, 2024
24067c8
Remove container_name from postgres and caddy
ddimaria Aug 29, 2024
e20a821
Improve init experience for self-hosting
ddimaria Aug 30, 2024
2fdb1ed
Update init.sh link
ddimaria Aug 30, 2024
a857530
Update init.sh link
ddimaria Aug 30, 2024
e8d4d5d
Migrate database on startup
ddimaria Aug 30, 2024
067b083
Migrate database on startup v2
ddimaria Aug 30, 2024
065f531
Redirect to login-result after kratos login
ddimaria Aug 30, 2024
b31bec3
Redirect to login-result after kratos registration
ddimaria Aug 30, 2024
6c4c62b
Use bash over sh in the init.sh script
ddimaria Sep 3, 2024
86e1cf6
Update new init.sh link
ddimaria Sep 3, 2024
c3e2461
Add Installing on Ubuntu instructions
ddimaria Sep 3, 2024
5ca28b9
Prefer 127.0.0.1 over host.docker.internal
ddimaria Sep 3, 2024
c6ebfd2
Prefer 0.0.0.0 over 127.0.0.1 for self hosting
ddimaria Sep 3, 2024
e425d5a
Add extra_hosts in docker compose for ubuntu
ddimaria Sep 3, 2024
50b8ed5
Remove trailing slash from LICENSE_API_URI
ddimaria Sep 3, 2024
c4b8f44
File perms logging
ddimaria Sep 3, 2024
fbf4b28
File perms logging v2
ddimaria Sep 3, 2024
de599cf
Improve the parseDomain() function
ddimaria Sep 4, 2024
3cc9298
Add CORS headers to nginx
ddimaria Sep 4, 2024
c0212b4
Add tests for parseDomain
ddimaria Sep 4, 2024
2cddc14
Hard-code kratos login and registration URLs
ddimaria Sep 4, 2024
e93082d
Update to mac friendly sed
ddimaria Sep 4, 2024
2b9f2ea
Prompt user for host name during initialization
ddimaria Sep 4, 2024
7a8b99c
Remove .bak files
ddimaria Sep 4, 2024
e208253
Update init.sh link in README
ddimaria Sep 4, 2024
bf35b7a
Update instructions to include Creating an EC2 Instance
ddimaria Sep 4, 2024
3951a55
Ensure login-redirect after ory registration
ddimaria Sep 4, 2024
21057a2
Allow command line arguments to be used in init.sh
ddimaria Sep 4, 2024
443179d
Update readme with new init link and ports to open
ddimaria Sep 4, 2024
d90f78c
Update readme requirement to include all ports
ddimaria Sep 4, 2024
0f44de8
Merge remote-tracking branch 'origin/qa' into self-hosting-setup
ddimaria Sep 13, 2024
b818ecc
Remove self-hosting directory (in favor of new self-hosting repo
ddimaria Sep 13, 2024
9b59b3f
Update URLs to self-hosting portal production domain
ddimaria Sep 13, 2024
8c9c9de
Fix clippy lints
ddimaria Sep 13, 2024
414a74e
Fix some api test failures
ddimaria Sep 13, 2024
ad9bcda
Mock licenseClient
ddimaria Sep 13, 2024
5e822a2
Fix quadratic-client lints
ddimaria Sep 13, 2024
22df915
Fix quadratic-api tests
ddimaria Sep 13, 2024
fa43dac
Create .env.test for quadratic-client to fix unit tests
ddimaria Sep 16, 2024
e3f8b8c
Prefer npm run compile --workspace=quadratic-shared in the client Doc…
ddimaria Sep 16, 2024
917b77e
Merge remote-tracking branch 'origin/qa' into self-hosting-setup
ddimaria Oct 8, 2024
36e49f4
try auto detection
AyushAgrawal-A2 Oct 14, 2024
7bf6edd
feat: user selectable csv delimiter
AyushAgrawal-A2 Oct 16, 2024
dbc4e70
Merge branch 'qa' into ayush/csv_delimiter
AyushAgrawal-A2 Oct 16, 2024
f8ef800
use existing components
jimniels Oct 16, 2024
5d90d30
Merge pull request #1977 from quadratichq/jim/csv_delimiter
jimniels Oct 16, 2024
2cda27e
show preview and first row heading checkbox
AyushAgrawal-A2 Oct 17, 2024
b295ced
tests and cleanup
AyushAgrawal-A2 Oct 17, 2024
fd8eccf
GridControllerWasm
AyushAgrawal-A2 Oct 17, 2024
17fe743
cleanup
AyushAgrawal-A2 Oct 17, 2024
3873046
initial commit
jimniels Oct 17, 2024
bc4426e
UI tweaks
jimniels Oct 17, 2024
ab61bb9
reject csv import setting promise
AyushAgrawal-A2 Oct 17, 2024
7d73229
text select-none
AyushAgrawal-A2 Oct 17, 2024
f83e21a
reset file imports states
AyushAgrawal-A2 Oct 17, 2024
7b0a046
fix bugs
AyushAgrawal-A2 Oct 17, 2024
7b37058
Update CSVImportSettings.tsx
jimniels Oct 18, 2024
a1a0cc0
Update FilesListItemCore.tsx
jimniels Oct 18, 2024
49c52f3
Merge branch 'qa' of github.com:quadratichq/quadratic into ayush/csv_…
AyushAgrawal-A2 Oct 19, 2024
712f293
clear preview
AyushAgrawal-A2 Oct 19, 2024
b0d5076
5 rows
AyushAgrawal-A2 Oct 19, 2024
ab2a22b
fix Enter
AyushAgrawal-A2 Oct 20, 2024
5eff635
Merge branch 'qa' into ayush/csv_delimiter
AyushAgrawal-A2 Oct 22, 2024
073f776
fix: dirty_hashes for change_spill
AyushAgrawal-A2 Oct 22, 2024
9ac106d
fix test
AyushAgrawal-A2 Oct 22, 2024
d0bcfdd
test: try aws s3 transfer accelerate
AyushAgrawal-A2 Oct 22, 2024
edd5cf8
fix localstack
AyushAgrawal-A2 Oct 22, 2024
d26564f
updates to filtering by name
jimniels Oct 22, 2024
8151a40
Add in bypass for quadratic license key
ddimaria Oct 23, 2024
6d80dbf
Merge branch 'qa' into jim-filter-files-by-creator
jimniels Oct 23, 2024
e8cdd29
productionize image creation
davidkircos Oct 23, 2024
9e6fc6b
Address PR feedback
ddimaria Oct 23, 2024
1b45c8f
Merge remote-tracking branch 'origin/qa' into self-hosting-setup
ddimaria Oct 23, 2024
ef2d28c
Refresh user count when added or removed from a team
ddimaria Oct 24, 2024
061bcb2
search triggers when manually changing sheets
davidfig Oct 24, 2024
d13358d
Fix lint issue related to different event emitter library
ddimaria Oct 24, 2024
398ee36
productionize
davidkircos Oct 24, 2024
353a2d7
Merge branch 'self-hosting-setup' of https://github.com/quadratichq/q…
davidkircos Oct 24, 2024
881a59a
Merge branch 'qa' into jim-filter-files-by-creator
jimniels Oct 29, 2024
7658d4a
initial commit
jimniels Oct 29, 2024
151dd04
Merge branch 'qa' into ayush/aws_s3_ta
AyushAgrawal-A2 Oct 30, 2024
ab44c0f
Merge branch 'qa' into self-hosting-setup
ddimaria Oct 30, 2024
764324b
Update _dashboard.tsx, remove useTheme
ddimaria Oct 30, 2024
6fdcf3e
Fix import issue
ddimaria Oct 30, 2024
5e16fac
tweaks
jimniels Oct 30, 2024
38ba7af
Try export-table option when building rust-client
ddimaria Oct 30, 2024
33cd271
initial commit
jimniels Oct 30, 2024
717ec59
Merge remote-tracking branch 'origin/qa' into self-hosting-setup
ddimaria Oct 30, 2024
b435723
Remove wasm target
ddimaria Oct 30, 2024
1ac02f1
Merge pull request #2000 from quadratichq/jim-dashboard-sidebar
davidkircos Oct 31, 2024
ccca1fa
Merge pull request #2005 from quadratichq/jim-fix-theme
davidkircos Oct 31, 2024
130dda6
Merge remote-tracking branch 'origin/qa' into self-hosting-setup
ddimaria Oct 31, 2024
19ac842
Remove export-tables option
ddimaria Oct 31, 2024
e65c3e6
Fix kratos creation bug
ddimaria Oct 31, 2024
12adda9
Merge branch 'qa' into jim-filter-files-by-creator
jimniels Nov 1, 2024
d9dc4ec
ensure 1902-01 doesn't parse to date
davidfig Nov 1, 2024
c30902a
Merge pull request #2012 from quadratichq/fix-weird-date-parsing
davidkircos Nov 1, 2024
f43d051
echo versions
davidkircos Nov 4, 2024
249e0c9
try
davidkircos Nov 4, 2024
bea8d4f
Respect the S3 storage type on the client
ddimaria Nov 4, 2024
3fa9673
Hide examples where VITE_STORAGE_TYPE === 'file-system'
ddimaria Nov 4, 2024
9062aef
Only show revoked message when the license is revoked, not exceeded
ddimaria Nov 4, 2024
7991c6b
Merge branch 'self-hosting-setup' of github.com:quadratichq/quadratic…
ddimaria Nov 4, 2024
7653c07
Move JWT initialization
ddimaria Nov 5, 2024
17ebf8d
Merge pull request #1718 from quadratichq/self-hosting-setup
ddimaria Nov 5, 2024
85fd084
Merge pull request #1979 from quadratichq/jim-filter-files-by-creator
davidkircos Nov 5, 2024
3fb7db0
Merge pull request #1990 from quadratichq/find-dialog-sheets
davidkircos Nov 5, 2024
853c8f9
Update DEVELOPMENT.md
davidkircos Nov 6, 2024
a313d0a
Merge pull request #1985 from quadratichq/ayush/fix_spill_render
davidkircos Nov 6, 2024
db40c46
Merge branch 'qa' of github.com:quadratichq/quadratic into ayush/aws_…
AyushAgrawal-A2 Nov 6, 2024
2e0cc6f
Merge branch 'ayush/aws_s3_ta' of github.com:quadratichq/quadratic in…
AyushAgrawal-A2 Nov 6, 2024
a3a6f4b
Merge branch 'qa' of github.com:quadratichq/quadratic into ayush/csv_…
AyushAgrawal-A2 Nov 6, 2024
f9e251b
Merge branch 'ayush/csv_delimiter' of github.com:quadratichq/quadrati…
AyushAgrawal-A2 Nov 6, 2024
7036773
Merge pull request #1986 from quadratichq/ayush/aws_s3_ta
AyushAgrawal-A2 Nov 6, 2024
39b9798
Merge branch 'qa' into ayush/csv_delimiter
AyushAgrawal-A2 Nov 6, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Implement storage when reading/writing in the file service
ddimaria committed Jul 30, 2024
commit 0a0ba6be654749be4532e17dbe35de5b6ade1800
5 changes: 3 additions & 2 deletions Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

9 changes: 8 additions & 1 deletion quadratic-files/.env.docker
Original file line number Diff line number Diff line change
@@ -15,7 +15,14 @@ PUBSUB_PASSWORD=
PUBSUB_ACTIVE_CHANNELS=active_channels
PUBSUB_PROCESSED_TRANSACTIONS_CHANNEL=processed_transactions

# Storage
STORAGE_TYPE=s3 # s3 or filesystem

# Storage: s3
AWS_S3_REGION=
AWS_S3_BUCKET_NAME=quadratic-api-docker
AWS_S3_ACCESS_KEY_ID=
AWS_S3_SECRET_ACCESS_KEY=
AWS_S3_SECRET_ACCESS_KEY=

# Storage: filesystem
FILE_DIR=/storage
9 changes: 8 additions & 1 deletion quadratic-files/.env.example
Original file line number Diff line number Diff line change
@@ -15,7 +15,14 @@ PUBSUB_PASSWORD=
PUBSUB_ACTIVE_CHANNELS=active_channels
PUBSUB_PROCESSED_TRANSACTIONS_CHANNEL=processed_transactions

# Storage
STORAGE_TYPE=s3 # s3 or filesystem

# Storage: s3
AWS_S3_REGION=us-east-2
AWS_S3_BUCKET_NAME=quadratic-api-docker
AWS_S3_ACCESS_KEY_ID=test
AWS_S3_SECRET_ACCESS_KEY=test
AWS_S3_SECRET_ACCESS_KEY=test

# Storage: filesystem
FILE_DIR=/storage
11 changes: 9 additions & 2 deletions quadratic-files/.env.test
Original file line number Diff line number Diff line change
@@ -15,7 +15,14 @@ PUBSUB_PASSWORD=
PUBSUB_ACTIVE_CHANNELS=active_channels
PUBSUB_PROCESSED_TRANSACTIONS_CHANNEL=processed_transactions

# Storage
STORAGE_TYPE=s3 # s3 or filesystem

# Storage: s3
AWS_S3_REGION=
AWS_S3_BUCKET_NAME=
AWS_S3_BUCKET_NAME=quadratic-api-docker
AWS_S3_ACCESS_KEY_ID=
AWS_S3_SECRET_ACCESS_KEY=
AWS_S3_SECRET_ACCESS_KEY=

# Storage: filesystem
FILE_DIR=/storage
1 change: 1 addition & 0 deletions quadratic-files/Cargo.toml
Original file line number Diff line number Diff line change
@@ -7,6 +7,7 @@ authors = ["David DiMaria <[email protected]>"]
[dependencies]
axum = { version = "0.7.1", features = ["ws"] }
axum-extra = { version = "0.9.0", features = ["typed-header"] }
bytes = "1.6.1"
chrono = { version= "0.4.31", features = ["serde"] }
dotenv = "0.15.0"
envy = "0.4.2"
23 changes: 19 additions & 4 deletions quadratic-files/src/config.rs
Original file line number Diff line number Diff line change
@@ -9,6 +9,14 @@ use crate::error::{FilesError, Result};
use dotenv::dotenv;
use quadratic_rust_shared::environment::Environment;
use serde::Deserialize;
use strum_macros::Display;

#[derive(Deserialize, Debug, Display)]
#[serde(rename_all = "kebab-case")]
pub(crate) enum StorageType {
S3,
FileSystem,
}

#[allow(dead_code)]
#[derive(Deserialize, Debug)]
@@ -30,10 +38,17 @@ pub(crate) struct Config {
pub(crate) quadratic_api_uri: String,
pub(crate) m2m_auth_token: String,

pub(crate) aws_s3_region: String,
pub(crate) aws_s3_bucket_name: String,
pub(crate) aws_s3_access_key_id: String,
pub(crate) aws_s3_secret_access_key: String,
// Storage Type: s3 or filesystem
pub(crate) storage_type: StorageType,

// StorageConfig::S3
pub(crate) aws_s3_region: Option<String>,
pub(crate) aws_s3_bucket_name: Option<String>,
pub(crate) aws_s3_access_key_id: Option<String>,
pub(crate) aws_s3_secret_access_key: Option<String>,

// StorageConfig::FileSystem
pub(crate) path: Option<String>,
}

/// Load the global configuration from the environment into Config.
4 changes: 2 additions & 2 deletions quadratic-files/src/error.rs
Original file line number Diff line number Diff line change
@@ -34,8 +34,8 @@ pub(crate) enum FilesError {
#[error("Internal server error: {0}")]
InternalServer(String),

#[error("Unable to load file {0} from bucket {1}: {2}")]
LoadFile(String, String, String),
#[error("Unable to load file {0}: {1}")]
LoadFile(String, String),

#[error("PubSub error: {0}")]
PubSub(String),
40 changes: 16 additions & 24 deletions quadratic-files/src/file.rs
Original file line number Diff line number Diff line change
@@ -12,12 +12,9 @@ use quadratic_core::{
},
};
use quadratic_rust_shared::{
aws::{
s3::{download_object, upload_object},
Client,
},
pubsub::PubSub as PubSubTrait,
quadratic_api::{get_file_checkpoint, set_file_checkpoint},
storage::{Storage, StorageContainer},
};

use crate::{
@@ -45,20 +42,18 @@ pub(crate) fn apply_transaction(grid: &mut GridController, operations: Vec<Opera

/// Exports a .grid file
pub(crate) async fn get_and_load_object(
client: &Client,
bucket: &str,
storage: &StorageContainer,
key: &str,
sequence_num: u64,
) -> Result<GridController> {
let file = download_object(client, bucket, key).await?;
let body = file
.body
.collect()
let body = storage
.read(key)
.await
.map_err(|e| FilesError::LoadFile(key.into(), bucket.to_string(), e.to_string()))?
.into_bytes();
let body = std::str::from_utf8(&body)
.map_err(|e| FilesError::LoadFile(key.into(), bucket.to_string(), e.to_string()))?;
.map_err(|e| FilesError::LoadFile(key.into(), e.to_string()))?;

// TODO(ddimaria): remove this line when the binary file format PR is merged
let body =
std::str::from_utf8(&body).map_err(|e| FilesError::LoadFile(key.into(), e.to_string()))?;
let grid = load_file(key, body)?;

Ok(GridController::from_grid(grid, sequence_num))
@@ -70,16 +65,14 @@ pub(crate) fn key(file_id: Uuid, sequence: u64) -> String {

/// Load a file from S3, add it to memory, process transactions and upload it back to S3
pub(crate) async fn process_transactions(
client: &Client,
bucket: &str,
storage: &StorageContainer,
file_id: Uuid,
checkpoint_sequence_num: u64,
final_sequence_num: u64,
operations: Vec<Operation>,
) -> Result<u64> {
let mut grid = get_and_load_object(
client,
bucket,
storage,
&key(file_id, checkpoint_sequence_num),
checkpoint_sequence_num,
)
@@ -90,7 +83,7 @@ pub(crate) async fn process_transactions(
apply_transaction(&mut grid, operations);
let body = export_file(&key, grid.grid_mut())?;

upload_object(client, bucket, &key, &body).await?;
storage.write(&key, &body.into()).await?;

Ok(final_sequence_num)
}
@@ -105,8 +98,7 @@ pub(crate) async fn process_queue_for_room(
let channel = &file_id.to_string();

let Settings {
aws_client,
aws_s3_bucket_name,
storage,
quadratic_api_uri,
m2m_auth_token,
..
@@ -173,8 +165,7 @@ pub(crate) async fn process_queue_for_room(

// process the transactions and save the file to S3
let last_sequence_num = process_transactions(
aws_client,
aws_s3_bucket_name,
storage,
*file_id,
checkpoint_sequence_num,
last_sequence_num,
@@ -199,14 +190,15 @@ pub(crate) async fn process_queue_for_room(

// update the checkpoint in quadratic-api
let key = &key(*file_id, last_sequence_num);

set_file_checkpoint(
quadratic_api_uri,
m2m_auth_token,
file_id,
last_sequence_num,
CURRENT_VERSION.into(),
key.to_owned(),
aws_s3_bucket_name.to_owned(),
storage.path().to_owned(),
)
.await?;

46 changes: 33 additions & 13 deletions quadratic-files/src/state/settings.rs
Original file line number Diff line number Diff line change
@@ -1,33 +1,53 @@
use quadratic_rust_shared::aws::{client, Client};
use quadratic_rust_shared::aws::client;
use quadratic_rust_shared::environment::Environment;
use quadratic_rust_shared::storage::file_system::{FileSystem, FileSystemConfig};
use quadratic_rust_shared::storage::s3::{S3Config, S3};
use quadratic_rust_shared::storage::StorageContainer;

use crate::config::Config;
use crate::config::{Config, StorageType};

#[derive(Debug)]
pub(crate) struct Settings {
pub(crate) quadratic_api_uri: String,
pub(crate) m2m_auth_token: String,
pub(crate) aws_client: Client,
pub(crate) aws_s3_bucket_name: String,
pub(crate) storage: StorageContainer,
pub(crate) pubsub_processed_transactions_channel: String,
}

impl Settings {
// Create a new Settings struct from the provided Config.
// Panics are OK here since this is set at startup and we want to fail fast.
pub(crate) async fn new(config: &Config) -> Self {
let is_local =
config.environment == Environment::Docker || config.environment == Environment::Local;
let expected = |val: &Option<String>, var: &str| {
val.to_owned()
.expect(&format!("Expected {} to have a value", var))
};

let storage = match config.storage_type {
StorageType::S3 => StorageContainer::S3(S3::new(S3Config {
client: client(
&expected(&config.aws_s3_access_key_id, "AWS_S3_ACCESS_KEY_ID"),
&expected(&config.aws_s3_secret_access_key, "AWS_S3_SECRET_ACCESS_KEY"),
&expected(&config.aws_s3_region, "AWS_S3_REGION"),
"Quadratic File Service",
is_local,
)
.await,
bucket: expected(&config.aws_s3_bucket_name, "AWS_S3_BUCKET_NAME"),
})),
StorageType::FileSystem => {
StorageContainer::FileSystem(FileSystem::new(FileSystemConfig {
path: expected(&config.path, "FILE_DIR"),
}))
}
};

Settings {
quadratic_api_uri: config.quadratic_api_uri.to_owned(),
m2m_auth_token: config.m2m_auth_token.to_owned(),
aws_client: client(
&config.aws_s3_access_key_id,
&config.aws_s3_secret_access_key,
&config.aws_s3_region,
"Quadratic File Service",
is_local,
)
.await,
aws_s3_bucket_name: config.aws_s3_bucket_name.to_owned(),
storage,
pubsub_processed_transactions_channel: config
.pubsub_processed_transactions_channel
.to_owned(),
15 changes: 11 additions & 4 deletions quadratic-rust-shared/src/storage/file_system.rs
Original file line number Diff line number Diff line change
@@ -14,14 +14,13 @@ pub struct FileSystemConfig {
pub path: String,
}

#[derive(Debug)]
pub struct FileSystem {
pub config: FileSystemConfig,
}

#[async_trait]
impl<'a> Storage<'a> for FileSystem {
type Config = FileSystemConfig;

impl Storage for FileSystem {
async fn read(&self, key: &str) -> Result<Bytes> {
let file_path = self.full_path(key, false).await?.0;
let mut bytes = vec![];
@@ -36,7 +35,7 @@ impl<'a> Storage<'a> for FileSystem {
Ok(bytes.into())
}

async fn write(&self, key: &'a str, data: &'a Bytes) -> Result<()> {
async fn write<'a>(&self, key: &'a str, data: &'a Bytes) -> Result<()> {
let file_path = self.full_path(key, true).await?.0;
let mut file = File::create(file_path)
.await
@@ -47,9 +46,17 @@ impl<'a> Storage<'a> for FileSystem {

Ok(())
}

fn path(&self) -> &str {
&self.config.path
}
}

impl FileSystem {
pub fn new(config: FileSystemConfig) -> Self {
Self { config }
}

pub async fn full_path(&self, key: &str, create_dir: bool) -> Result<(PathBuf, PathBuf)> {
let FileSystemConfig { path } = &self.config;
let parts = key.split('-').collect::<Vec<&str>>();
40 changes: 35 additions & 5 deletions quadratic-rust-shared/src/storage/mod.rs
Original file line number Diff line number Diff line change
@@ -9,17 +9,47 @@ pub mod file_system;
pub mod s3;

#[derive(Debug)]
pub enum Config<'a> {
S3(S3Config<'a>),
pub enum Config {
S3(S3Config),
FileSystem(FileSystemConfig),
}

#[derive(Debug)]
pub enum StorageContainer {
S3(s3::S3),
FileSystem(file_system::FileSystem),
}

// TODO(ddimaria): this is a temp hack to get around some trait issues, do something better
#[async_trait]
pub trait Storage<'a> {
type Config;
impl Storage for StorageContainer {
async fn read(&self, key: &str) -> Result<Bytes> {
match self {
Self::S3(s3) => s3.read(key).await,
Self::FileSystem(fs) => fs.read(key).await,
}
}

async fn write<'a>(&self, key: &'a str, data: &'a Bytes) -> Result<()> {
match self {
Self::S3(s3) => s3.write(key, data).await,
Self::FileSystem(fs) => fs.write(key, data).await,
}
}

fn path(&self) -> &str {
match self {
Self::S3(s3) => s3.path(),
Self::FileSystem(fs) => fs.path(),
}
}
}

#[async_trait]
pub trait Storage {
async fn read(&self, key: &str) -> Result<Bytes>;
async fn write(&self, key: &'a str, data: &'a Bytes) -> Result<()>;
async fn write<'a>(&self, key: &'a str, data: &'a Bytes) -> Result<()>;
fn path(&self) -> &str;

fn read_error(key: &str, e: impl ToString) -> SharedError {
SharedError::Storage(StorageError::Read(key.into(), e.to_string()))
25 changes: 17 additions & 8 deletions quadratic-rust-shared/src/storage/s3.rs
Original file line number Diff line number Diff line change
@@ -9,19 +9,18 @@ use crate::{
};

#[derive(Debug)]
pub struct S3Config<'a> {
pub struct S3Config {
pub client: Client,
pub bucket: &'a str,
pub bucket: String,
}

pub struct S3<'a> {
pub config: S3Config<'a>,
#[derive(Debug)]
pub struct S3 {
pub config: S3Config,
}

#[async_trait]
impl<'a> Storage<'a> for S3<'a> {
type Config = S3Config<'a>;

impl Storage for S3 {
async fn read(&self, key: &str) -> Result<Bytes> {
let S3Config { client, bucket } = &self.config;

@@ -39,7 +38,7 @@ impl<'a> Storage<'a> for S3<'a> {
Ok(bytes)
}

async fn write(&self, key: &'a str, data: &'a Bytes) -> Result<()> {
async fn write<'a>(&self, key: &'a str, data: &'a Bytes) -> Result<()> {
let S3Config { client, bucket } = &self.config;

upload_object(client, bucket, key, data)
@@ -48,6 +47,16 @@ impl<'a> Storage<'a> for S3<'a> {

Ok(())
}

fn path(&self) -> &str {
&self.config.bucket
}
}

impl S3 {
pub fn new(config: S3Config) -> Self {
Self { config }
}
}

#[cfg(test)]