Skip to content

Commit

Permalink
added additional information on using buckets (#193)
Browse files Browse the repository at this point in the history
  • Loading branch information
arpita0911patel committed Aug 22, 2024
2 parents eb7a294 + ecc5233 commit e463527
Showing 1 changed file with 27 additions and 8 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -15,13 +15,15 @@ This guide is for managing objects in GCP buckets available on 2i2c CIROH Jupyte

### 1. Overview
CIROH JupyterHub uses object Google Cloud Storage to store data in buckets (containers for objects). Currently, there are two buckets available to use on CIROH JupyterHub.
- **Scratch Buckets**: It is intended for storing temporary files since any files in scratch buckets get deleted after seven days. Open a terminal in CIROH JupyterHub and run this command to display your scratch bucket name:
- **Scratch Bucket**: It is intended for storing temporary files since any files in scratch bucket get deleted after seven days. Open a terminal in CIROH JupyterHub and run this command to display your scratch bucket name:

```
echo $SCRATCH_BUCKET
gs://awi-ciroh-scratch/<username>
```
- **Persistent Buckets**: It is recommended to use for storing files that you will be using for a longer period of time. Open a terminal in CIROH JupyterHub and run this command to display your persistent bucket name:
**Note:** In the above command output, the name of the bucket is 'awi-ciroh-scratch' and `<username>` is the folder in the bucket.

- **Persistent Bucket**: It is recommended to use for storing files that you will be using for a longer period of time. Open a terminal in CIROH JupyterHub and run this command to display your persistent bucket name:

```
echo $PERSISTENT_BUCKET
Expand All @@ -32,30 +34,47 @@ gs://awi-ciroh-persistent/<username>
You can copy files on your CIROH JupyterHub to an available bucket using the following command.

```
gcloud storage ls cp <filepath> $PERSISTENT_BUCKET/<filepath>
gcloud storage cp <filepath> $PERSISTENT_BUCKET/<filepath>
```

### 3. Copying file from a bucket to CIROH JupyterHub
You can copy files from an accessible bucket to your CIROH JupyterHub using the following command.

```
gcloud storage ls cp $PERSISTENT_BUCKET/<filepath> <destination-filepath>
gcloud storage cp $PERSISTENT_BUCKET/<filepath> <destination-filepath>
```

### 4. Listing files in a bucket
You can list all files/folder in a bucket using the following command.

```
gscloud storage ls $PERSISTENT_BUCKET
```
**Note:** The above command will list all files/folders in the folder `<username>`. It won't list files in the sub-folders of folder `<username>`. To list all files including the files in the sub-folders of the root folder `<username>`, use the following command.

### 4. Deleting file from a bucket
```
gscloud storage ls --recursive $PERSISTENT_BUCKET
```

### 5. Deleting file from a bucket
You can delete a file in a bucket with the following command:

```
gcloud storage ls rm $PERSISTENT_BUCKET/<filepath>
gcloud storage rm $PERSISTENT_BUCKET/<filepath>
```

### 6. User permssions on buckets
All users have read/write permissions on both the scratch and persistent buckets.

:::note
Anyone can access each other's files in persistent buckets on the hub. Please be careful not to delete other user's files. Your actions impact the entire organization's storage. If unsure, consult with the team lead or ciroh IT support.
Anyone can access each other's files in buckets on the hub. Please be careful not to delete other user's files. Using the enviornment variables ($SCRATCH_BUCKET & $PERSISTENT_BUCKET) to access buckets in commands would prevent accidententally deleting any other user's files. Your actions impact the entire organization's storage. If unsure, consult with the team lead or ciroh IT support.
:::

### 7. Accessing buckets in Python
You can find information on how to access buckets in Python code, [here.](https://docs.2i2c.org/user/topics/data/object-storage/working-with-object-storage/)

## Where to go for help:

- Email [email protected] UA CIROH Cloud Team
- CIROH Cloud Slack Channel - #ciroh-ua-it-admin
- CIROH Infrastructure Support Slack Channel - #ciroh-infrastructure-support
- CIROH Infrastructure Support Slack Channel - #ciroh-infrastructure-support

0 comments on commit e463527

Please sign in to comment.