-
Notifications
You must be signed in to change notification settings - Fork 21
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
added additional information on using buckets (#193)
- Loading branch information
Showing
1 changed file
with
27 additions
and
8 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -15,13 +15,15 @@ This guide is for managing objects in GCP buckets available on 2i2c CIROH Jupyte | |
|
||
### 1. Overview | ||
CIROH JupyterHub uses object Google Cloud Storage to store data in buckets (containers for objects). Currently, there are two buckets available to use on CIROH JupyterHub. | ||
- **Scratch Buckets**: It is intended for storing temporary files since any files in scratch buckets get deleted after seven days. Open a terminal in CIROH JupyterHub and run this command to display your scratch bucket name: | ||
- **Scratch Bucket**: It is intended for storing temporary files since any files in scratch bucket get deleted after seven days. Open a terminal in CIROH JupyterHub and run this command to display your scratch bucket name: | ||
|
||
``` | ||
echo $SCRATCH_BUCKET | ||
gs://awi-ciroh-scratch/<username> | ||
``` | ||
- **Persistent Buckets**: It is recommended to use for storing files that you will be using for a longer period of time. Open a terminal in CIROH JupyterHub and run this command to display your persistent bucket name: | ||
**Note:** In the above command output, the name of the bucket is 'awi-ciroh-scratch' and `<username>` is the folder in the bucket. | ||
|
||
- **Persistent Bucket**: It is recommended to use for storing files that you will be using for a longer period of time. Open a terminal in CIROH JupyterHub and run this command to display your persistent bucket name: | ||
|
||
``` | ||
echo $PERSISTENT_BUCKET | ||
|
@@ -32,30 +34,47 @@ gs://awi-ciroh-persistent/<username> | |
You can copy files on your CIROH JupyterHub to an available bucket using the following command. | ||
|
||
``` | ||
gcloud storage ls cp <filepath> $PERSISTENT_BUCKET/<filepath> | ||
gcloud storage cp <filepath> $PERSISTENT_BUCKET/<filepath> | ||
``` | ||
|
||
### 3. Copying file from a bucket to CIROH JupyterHub | ||
You can copy files from an accessible bucket to your CIROH JupyterHub using the following command. | ||
|
||
``` | ||
gcloud storage ls cp $PERSISTENT_BUCKET/<filepath> <destination-filepath> | ||
gcloud storage cp $PERSISTENT_BUCKET/<filepath> <destination-filepath> | ||
``` | ||
|
||
### 4. Listing files in a bucket | ||
You can list all files/folder in a bucket using the following command. | ||
|
||
``` | ||
gscloud storage ls $PERSISTENT_BUCKET | ||
``` | ||
**Note:** The above command will list all files/folders in the folder `<username>`. It won't list files in the sub-folders of folder `<username>`. To list all files including the files in the sub-folders of the root folder `<username>`, use the following command. | ||
|
||
### 4. Deleting file from a bucket | ||
``` | ||
gscloud storage ls --recursive $PERSISTENT_BUCKET | ||
``` | ||
|
||
### 5. Deleting file from a bucket | ||
You can delete a file in a bucket with the following command: | ||
|
||
``` | ||
gcloud storage ls rm $PERSISTENT_BUCKET/<filepath> | ||
gcloud storage rm $PERSISTENT_BUCKET/<filepath> | ||
``` | ||
|
||
### 6. User permssions on buckets | ||
All users have read/write permissions on both the scratch and persistent buckets. | ||
|
||
:::note | ||
Anyone can access each other's files in persistent buckets on the hub. Please be careful not to delete other user's files. Your actions impact the entire organization's storage. If unsure, consult with the team lead or ciroh IT support. | ||
Anyone can access each other's files in buckets on the hub. Please be careful not to delete other user's files. Using the enviornment variables ($SCRATCH_BUCKET & $PERSISTENT_BUCKET) to access buckets in commands would prevent accidententally deleting any other user's files. Your actions impact the entire organization's storage. If unsure, consult with the team lead or ciroh IT support. | ||
::: | ||
|
||
### 7. Accessing buckets in Python | ||
You can find information on how to access buckets in Python code, [here.](https://docs.2i2c.org/user/topics/data/object-storage/working-with-object-storage/) | ||
|
||
## Where to go for help: | ||
|
||
- Email [email protected] UA CIROH Cloud Team | ||
- CIROH Cloud Slack Channel - #ciroh-ua-it-admin | ||
- CIROH Infrastructure Support Slack Channel - #ciroh-infrastructure-support | ||
- CIROH Infrastructure Support Slack Channel - #ciroh-infrastructure-support |