Can froster be set up to only use on-prem S3 (Minio in my case)

[wdennis]

I see by the config instructions that it looks like AWS must be configured? In our case (PoC), we'd like to only use on-prem S3 (Minio.) Can Froster be configured to only use non-AWS S3? If so, how?

[dirkpetersen]

That should be possible but let me confirm and get back to you. We have also tested froster with GCS, Wasabi, IDrive Ceph and Minio and it just uses rclone underneath.

[macheitor]

During development, we focused on the main Froster features, using AWS as the main S3 storage provider.
Other providers rather than AWS will be available next week.
I will keep you updated, and I will post when the release is available.

[wdennis]

Awesome, thanks!

…

On Fri, Jun 14, 2024 at 1:13 PM victormachadoperez ***@***.***> wrote: During development, we focused on the main Froster features, using AWS as the main S3 storage provider. Other providers rather than AWS will be available next week. I will keep you updated, and I will post when the release is available. — Reply to this email directly, view it on GitHub <#71 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABNDWNNVTHEL7C3LGVTHMITZHMQEZAVCNFSM6AAAAABJHQCOLOVHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM4TONZXGA3DQ> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[macheitor]

Froster v0.14.1 is already deployed.
This version supports the following S3 providers: AWS, Google Cloud Storage (GCS), Wasabi, IDrive, and Minio.
Ceph is also supported but is still pending to be tested.

The command "froster update" will give you the command for update froster to the newer version.
If you find any issue do not hesitate to open a ticket at: https://github.com/dirkpetersen/froster/issues

[wdennis]

So, the config dialog will not require one to set up an AWS S3 account/etc?

…

On Fri, Jun 21, 2024 at 11:53 AM victormachadoperez < ***@***.***> wrote: Froster v0.14.1 is already deployed. This version supports the following S3 providers: AWS, Google Cloud Storage (GCS), Wasabi, IDrive, and Minio. Ceph is also supported but is still pending to be tested. The command "froster update" will give you the command for update froster to the newer version. If you find any issue do not hesitate to open a ticket at: https://github.com/dirkpetersen/froster/issues — Reply to this email directly, view it on GitHub <#71 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABNDWNIRQLGK3JCNFKYKLLDZIRD6BAVCNFSM6AAAAABJHQCOLOVHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM4TQNBRGUYTI> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[dirkpetersen]

Yes, that is exactly the idea. There are still some issues that you may stumble over (I just added a few where I ran into problems) . I could provide you with a config ini that should work if you want to test it now before we ironed out the config issues.

[wdennis]

@dirkpetersen Yes, I will take your .ini - see below, the froster config process is erroring out for me...

[wdennis]

So, not sure what's happening here when I'm trying to run froster config --

root@my-test-pc:~# froster config

Error
  File: froster.py
  Function: __init__
  Line: 1483
  Error code: 'ConfigManager' object has no attribute 'profile'

If you thing this is a bug, please report this to froster developers at: https://github.com/dirkpetersen/froster/issues


*****************************
*** FROSTER CONFIGURATION ***
*****************************


*** SET USER ***

[?] Enter your full name: Will Dennis


*** SET EMAIL ***


Error
  File: froster.py
  Function: match
  Line: 917
  Error code: expected string or bytes-like object

If you thing this is a bug, please report this to froster developers at: https://github.com/dirkpetersen/froster/issues

Any ideas?

[dirkpetersen]

Sorry for the confusion Will, I will provide you with an ini shortly. However, if you could try again with 'froster update' to the latest version and then another foster config that would be great. For me it now works with my ceph system to a point but I am getting an error as our ceph config does not allow me to see buckets that were already created and I cannot create new buckets either..... this may not be the case for you with minio ?

[wdennis]

Getting this on an update attempt...

root@my-test-pc:~# curl -s https://raw.githubusercontent.com/dirkpetersen/froster/main/install.sh?$(date +%s) | bash

Backing up older froster installation...
    Config back up at /root/.config/froster_20240626152517.bak
...older froster installation backed up

Uninstalling existing froster installation...
...froster uninstalled

Installing pipx...
...pipx already installed

Upgrading pipx...
...pipx upgraded

Installing latest version of froster...
  Installing from PyPi package repository
...froster installed

Installing third-party dependency: pwalk...
    Downloading pwalk files
    Compiling pwalk
    Moving pwalk to froster's binaries folder
    Cleaning up pwalk installation files
...pwalk installed

Installing third-party dependency: rclone...
    Downloading rclone files
    Extracting rclone files
    Moving rclone to froster's binaries folder

Rolling back installation...
main: line 378: =/usr/bin/pipx: No such file or directory

But, pipx does indeed exist there...

root@my-test-pc:~# which pipx
/usr/bin/pipx

[dirkpetersen]

Thanks for reporting, i think our test cases have only included installing pipx as non-root user, i am adding an issue

[macheitor]

I just updated the installation script to make it more robust. It should work fine now.
Current release version is froster v0.15.0.

Could you execute the update command again?

[dirkpetersen]

also this is my ~/.config/froster/config.ini

[USER]
name = Dirk Petersen
email = [email protected]

[SHARED]
is_shared = False
shared_dir =
shared_config_file =

[NIH]
is_nih = False

[S3]
provider = Minio
endpoint = https://cluster.domain.edu
bucket_name = archive
archive_dir = froster
storage_class = STANDARD

[CREDENTIALS]
credentials_dir = /home/users/yourusername/.aws
profile = aws

[SLURM]
slurm_walltime_days = 7
slurm_walltime_hours = 7
slurm_partition = exacloud
slurm_qos = long_jobs
slurm_lscratch = --gres disk:1024
lscratch_mkdir = /usr/local/bin/mkdir-scratch.sh
lscratch_rmdir = /usr/local/bin/rmdir-scratch.sh
lscratch_root = /mnt/scratch

[UPDATE]
timestamp = 1718983568

[macheitor]

New froster v0.16.0 has been released.

Now the configuration is a bit different: it configures profiles instead of providers.

Please update froster to the latest version and run "froster config" again.

(preferably run "froster config --reset" to have a clean configuration file).

[wdennis]

Sorry, got busy with other things for a while there.

I tried a froster update, but - now getting another kind of error, see below:

root@my-test-pc:~# froster update

A froster update is available!
  Current version: froster v0.14.16
  Latest version: froster v0.18.1

You can update froster using the command:
    curl -s https://raw.githubusercontent.com/dirkpetersen/froster/main/install.sh?$(date +%s) | bash


A froster update is available!
  Current version: froster v0.14.16
  Latest version: froster v0.18.1

You can update froster using the command:
    curl -s https://raw.githubusercontent.com/dirkpetersen/froster/main/install.sh?$(date +%s) | bash

root@my-test-pc:~# curl -s https://raw.githubusercontent.com/dirkpetersen/froster/main/install.sh?$(date +%s) | bash

Backing Froster config folder ...
    source: /root/.config/froster
    destination: /root/.config/froster_backups/froster_20240723202042.bak
...config backed up

Installing pipx...
...pipx already installed

Upgrading pipx...
...pipx upgraded

Ensuring path for pipx...
...path ensured

Adding /root/.local/bin to PATH for this installation session

Removing old froster files...
...old froster files removed

Uninstalling old Froster...
...old Froster uninstalled

Installing Froster from PyPi package repository
...Froster installed

Installing third-party dependency: pwalk...
    Downloading pwalk files
    Compiling pwalk
    Moving pwalk to froster's binaries folder
    Installed pwalk at /root/.local/pipx/venvs/froster/bin/pwalk
    Cleaning up pwalk installation files
...pwalk installed

Installing third-party dependency: rclone...
    Downloading rclone files
    Extracting rclone files
    Moving rclone to froster's binaries folder

Error: mv: Installation failed!

Any ideas on why?

[wdennis]

I went ahead and configured my froster install, but getting an (expected as per the above) error when I run froster -i:

root@my-test-pc:~# froster -v
froster v0.18.1
root@my-test-pc:~# froster -D

[?] Set your default profile:
 > profile nj-archiving

root@my-test-pc:~# froster -i

FILES

    Configuration: /root/.config/froster/config.ini
    DataBase: /root/.local/share/froster/froster-archives.json

TOOLS

  froster
    version: v0.18.1
    path: /root/.local/pipx/venvs/froster/bin/froster

  python
    version: v3.10.12
    path: /root/.local/pipx/venvs/froster/bin/python

  pwalk
    version: v3.0.0
    path: /root/.local/pipx/venvs/froster/bin/pwalk

  rclone

Error
  File: froster.py
  Function: _execute_child
  Line: 7608
  Error code: 2
  Exception type: <class 'FileNotFoundError'>
  Exception value: [Errno 2] No such file or directory: '/root/.local/pipx/venvs/froster/bin/rclone'

If you thing this is a bug, please report this to froster developers at: https://github.com/dirkpetersen/froster/issues

[dirkpetersen]

@wdennis, I am still working on trying to reproduce this. When I install as root in a clean (never installed froster) debian machine , I am getting this. Could you manually download rclone and put the rclone binary into /root/.local/pipx/venvs/froster/bin ? or remove all froster files in /root/.local/pipx/venvs/froster and /root/.config/froster and restart install ?

root@grammy:~# curl -s https://raw.githubusercontent.com/dirkpetersen/froster/main/install.sh?$(date +%s) | bash

Installing pipx...
  Installing pipx via pip
  Ensuring path for pipx
...pipx installed

Adding /root/.local/bin to PATH for this installation session

Removing old froster files...
...old froster files removed
nothing has been installed with pipx 😴

Installing Froster from PyPi package repository
...Froster installed

Installing third-party dependency: pwalk...
    Downloading pwalk files
    Compiling pwalk
    Moving pwalk to froster's binaries folder
    Installed pwalk at /root/.local/share/pipx/venvs/froster/bin/pwalk
    Cleaning up pwalk installation files
...pwalk installed

Installing third-party dependency: rclone...
    Downloading rclone files
    Extracting rclone files
    Moving rclone to froster's binaries folder
    Installed rclone at /root/.local/share/pipx/venvs/froster/bin/rclone
    Cleaning up rclone installation files
...rclone installed


SUCCESS!

Froster version: v0.18.1
Installation path: /root/.local/bin/froster


You will need to open a new terminal or refresh your current terminal session by running command:
  source ~/.bashrc

[wdennis]

Found out it was our darn corporate firewall blocking the rclone site… Tried on another connection with a more lenient firewall rule set, and the install did complete with no issue. Now on to configuration!

…

On Mon, Aug 5, 2024 at 5:50 PM Dirk Petersen ***@***.***> wrote: @wdennis <https://github.com/wdennis>, I am still working on trying to reproduce this. When I install as root in a clean (never installed froster) I am getting this. Could you manaully download rclone and put it into /root/.local/pipx/venvs/froster/bin ? or remove all froster files in /root/.local/pipx/venvs/froster and /root/.config/froster and restart install ? ***@***.***:~# curl -s https://raw.githubusercontent.com/dirkpetersen/froster/main/install.sh?$(date +%s) | bash Installing pipx... Installing pipx via pip Ensuring path for pipx ...pipx installed Adding /root/.local/bin to PATH for this installation session Removing old froster files... ...old froster files removed nothing has been installed with pipx 😴 Installing Froster from PyPi package repository ...Froster installed Installing third-party dependency: pwalk... Downloading pwalk files Compiling pwalk Moving pwalk to froster's binaries folder Installed pwalk at /root/.local/share/pipx/venvs/froster/bin/pwalk Cleaning up pwalk installation files ...pwalk installed Installing third-party dependency: rclone... Downloading rclone files Extracting rclone files Moving rclone to froster's binaries folder Installed rclone at /root/.local/share/pipx/venvs/froster/bin/rclone Cleaning up rclone installation files ...rclone installed SUCCESS! Froster version: v0.18.1 Installation path: /root/.local/bin/froster You will need to open a new terminal or refresh your current terminal session by running command: source ~/.bashrc — Reply to this email directly, view it on GitHub <#71 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABNDWNPGF3ZUHL7VSKLNPULZP7XTTAVCNFSM6AAAAABJHQCOLOVHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTAMRUG4ZTOMQ> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[wdennis]

Got this after setting up the Minio target...

Error
  File: froster.py
  Function: _make_api_call
  Line: 1699
  Error code:
  Exception type: <class 'botocore.exceptions.ClientError'>
  Exception value: An error occurred (InvalidBucketName) when calling the ListBuckets operation: The specified bucket is not valid.

If you thing this is a bug, please report this to froster developers at: https://github.com/dirkpetersen/froster/issues

Two things:

1. where are the credentials stored for the Minio access? (Mine is named "froster_creds")
2. There is a typo in the post-exception sentence: "you thing" s/b "you think"

[wdennis]

I see via the use of find with -exec grep -l froster_creds that you are using /root/.aws/credentials and /root/.aws/config which I guess makes sense :) Had to fix a problem in the .aws/config file, and now is working :)