Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add s3 service and resource plugin #40

Closed
scaleoutsean opened this issue Nov 2, 2022 · 11 comments
Closed

add s3 service and resource plugin #40

scaleoutsean opened this issue Nov 2, 2022 · 11 comments
Assignees
Labels
enhancement New feature or request

Comments

@scaleoutsean
Copy link
Contributor

Would it be possible to add to README.md advice on DIY for adding S3 resource plugin?

In the plugin's repo they say:

This plugin can work as a standalone "cacheless" resource or as an archive resource under the iRODS compound resource

In this (containerized) scenario, would it be better to edit a Dockerfile (which one?) to deploy the plugin and which resource mode would be recommended?

I've tried building and installing inside a live container, but there's a lot of packages and deps that need to be taken care of. Perhaps editing a Dockerfile would be a better idea?

@trel
Copy link
Member

trel commented Nov 2, 2022

Two things if you want to play with the S3 plugin...

  1. An S3 service to talk to (could be Amazon, or minio, or zenko, etc.) -- this is what you would probably stand up with a new service in the top-level docker-compose.yml (we've considered this, but haven't done it yet).

  2. The s3 resource plugin itself. The resource plugin can be installed from https://packages.irods.org. You would install it on the provider (see its README), and then configure it to talk to the S3 service (above).

Happy to leave this issue open as a feature request for the irods_demo itself.

@scaleoutsean
Copy link
Contributor Author

Right, 1) is easy but it could be made a part of the demo Compose file to minimize surprises and make it easy to follow a handful commands that don't fail. Appreciate the willingness to consider this as feature request.

@trel
Copy link
Member

trel commented Nov 2, 2022

No problem.

How did you find this project? You're the first one in the door...

@scaleoutsean
Copy link
Contributor Author

scaleoutsean commented Nov 2, 2022

I've been working with parallel filesystems for a while, so I heard of iRODS a long time ago.
But now I'm looking to create a simple PoC with BeeGFS-to-$something tiering with iRODS (and S3 seems easier to deploy than NFS v4 as destination tier :-)).

@trel
Copy link
Member

trel commented Nov 2, 2022

Fair enough - thanks!

Yes, NetApp is a consortium member, so please send us email directly if in-public isn't the right forum.

@trel trel changed the title S3 resource plugin advice for demo users add s3 service and resource plugin Nov 2, 2022
@trel trel added the enhancement New feature or request label Nov 2, 2022
@scaleoutsean
Copy link
Contributor Author

Cool! That's probably some tech or product group, but I don't know who exactly.

Anyway, I've added a MinIO container and S3 resource plugin (#42) - not sure if that's how it supposed to be done but it may save you some time. As I wrote in the comment, feel free to use whatever parts (or none at all) in own commit.

Maybe it'd be nice to have the MinIO client for easy verification (mc, below) added to icommand container's Dockerfile, but I didn't want to bloat that image so in my case I installed it manually after docker-compose up. I mention this because for some reason when I tried S3 resource plugin yesterday, iCommand container was unable to connect to existing S3 service endpoint on my LAN - it took me a while to realize it's not a config problem but a container networking problem of some sort, and then I thought it's probably better to deploy S3 together with the demo to avoid these situations.

irods@3cc28af82a1e:~$ date > sean-irods.txt 
irods@3cc28af82a1e:~$ cat sean-irods.txt 
Thu Nov  3 10:57:51 UTC 2022
irods@3cc28af82a1e:~$ iput -R s3resc sean-irods.txt --force
irods@3cc28af82a1e:~$ mc ls s3resc/irods/home/rods
[2022-11-03 10:58:00 UTC]    29B STANDARD sean-irods.txt
[2022-11-03 10:43:00 UTC]     0B STANDARD testfile.txt
irods@3cc28af82a1e:~$ rm sean-irods.txt
irods@3cc28af82a1e:~$ iget -R s3resc sean-irods.txt
irods@3cc28af82a1e:~$ date >> sean-irods.txt
irods@3cc28af82a1e:~$ cat sean-irods.txt 
Thu Nov  3 10:57:51 UTC 2022
Thu Nov  3 10:58:24 UTC 2022

@trel
Copy link
Member

trel commented Nov 3, 2022

nice - thanks for the success copy/paste - great that it 'just worked'.

yes, adding a minio container is the right answer.

and we'll definitely consider if/where to install mc.

@alanking
Copy link
Contributor

alanking commented Nov 9, 2022

The MinIO service and S3 plugin have been added. Getting a resource set up automatically is not done yet, so I will leave this open.

@trel
Copy link
Member

trel commented Nov 10, 2022

resource set up, and mc installed ... somewhere? is it already available in the minio container?

@scaleoutsean
Copy link
Contributor Author

It's not yet available - I think the idea was to make un invasive changes for phase v1, and then decide whether and how to automate the creation of bucket and client-side S3 config.

@trel
Copy link
Member

trel commented Nov 12, 2024

demobucket is now created by default in minio, and the s3 plugin is now configured to talk to that bucket.

closing!

@trel trel closed this as completed Nov 12, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Development

No branches or pull requests

3 participants