Six bucket upload

[sckott]

fixes #67

@seankross please review, thanks.

Note that six_file_upload function mentioned in the related issue is already in the package on main branch

I'm still not sure about what the best behavior should be with respect to directories of files locally. Should a dir of files be uploaded into a dir of the same name, or just exploded into the bucket name? When I used the s3fs fxn for uploading to a bucket it exploded the files in a local folder into the aws bucket at the root rather than into a folder in the bucket. Perhaps only if the user specifies a path in the remote arg in addition to the bucket name, e.g., bucketasdfasdfasdf/somedir, then somedir would be the dir name - BUT that somedir could also just be the file name, so this seems precarious.

We could just wrap the s3fs function, but since it's been a while since we were focusing on this package, I know I tried to move away from s3fs functions but I'm not remembering why now.

[seankross]

the magic for this function would be:

• a path to a file appears at the top level of a bucket
• a path to a dir reconstructs the dir structure within the bucket

let's say I pass in this vector into the function:

c(
  "/Users/sean/science/microscope/img1.png",
  "/Users/sean/science/microscope/img2.png",
  "/Users/sean/science/experiment1/",
)

and let's say this is the structure of /Users/sean/science/experiment1/:

experiment1/
  README.md
  data/
    results.csv
  code/
    analysis.py

the following "keys" (to use AWS parlance) would be created in the bucket:

/img1.png
/img2.png
/experiment1/README.md
/experiment1/data/results.csv
/experiment1/code/analysis.py

It would be good to be able to change the level too, so you didn't always have to use the top level, but I think you've built that functionality already here.

If this does what I've described let me know and I will take it for a spin.

[seankross]

See above.

[sckott]

that's not the current behavior right now wrt dir's. right now all the files in the dir get put into the root bucket path. i'll work on that and ping you when it's done

[seankross]

It would be good to be able to change the level too, so you didn't always have to use the top level, but I think you've built that functionality already here.

@sckott do you understand what I'm saying here?

[sckott]

By level I assume you mean how far down do you walk through the local directory? If that's what you're referring to, probably the thing people are most familiar with is just recursive or not, yeah? That is, just the top level, which in this example would be just the readme.md file, or recursive, which would be the example you gave where you go recursively down through dirs until there's no more dirs to go into

[seankross]

@sckott

By level I assume you mean how far down do you walk through the local directory?

nah lemme clarify, I'm thinking of functionality like:

six_bucket_upload(
  c(
    "/Users/sean/science/microscope/img1.png",
    "/Users/sean/science/microscope/img2.png",
    "/Users/sean/science/experiment1/",
  ),
  destination_path = "nih_grant/aim1" # probably a bad choice for an arg name
)

which would create these keys:

/nih_grant/aim1/img1.png
/nih_grant/aim1/img2.png
/nih_grant/aim1/experiment1/README.md
/nih_grant/aim1/experiment1/data/results.csv
/nih_grant/aim1/experiment1/code/analysis.py

But if you don't specify destination_path it just defaults to use the top level of the bucket.

[sckott]

Okay, so I think what i'm hearing with respect to dirs is:

• for the local side of things we just recursively go through the dir and extract files all the way down the tree and mirror that structure on the bucket
• for the aws bucket we use the root of the bucket for all the uploaded stuff UNLESS the user specifies some other dir within the bucket

Let's assume for simplicity that the arg name is destination_path ... If the user wants different local dirs in different destination_path dirs on the bucket, then they just need to call the function once for each of the destination_path values. I think it would be too complex to allow >1 destination_path value

[seankross]

Sounds great! Thank you. And yes I agree.

[seankross]

You need an @importFrom fs path wherever you think is appropriate.

[seankross]

Looks and works great except for the one comment above.