Skip to content

Commit

Permalink
for #22 adding bucket logs and data and codes
Browse files Browse the repository at this point in the history
  • Loading branch information
subash-taranga committed Jul 17, 2024
1 parent c8f5a38 commit f9a8e8b
Show file tree
Hide file tree
Showing 15 changed files with 1,230 additions and 0 deletions.
83 changes: 83 additions & 0 deletions logs/bucket/cpmpare-gsutil-vs-gcloud/copy_gcloud_storage.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,83 @@
subash@MSI:/mnt/c/SUBASH/G-CLOUD/BUCKET$ ./download_script.sh
STARTING COPYING USING GCLOUD STORAGE
Copying gs://bucket-europe-west9-paris/200.txt to file://./200.txt
Completed files 1/1 | 200.0MiB/200.0MiB | 10.7MiB/s

Average throughput: 10.8MiB/s

real 0m23.711s
user 0m2.306s
sys 0m0.998s
Copying gs://bucket-europe-west9-paris/300.txt to file://./300.txt
Completed files 1/1 | 300.0MiB/300.0MiB | 11.1MiB/s

Average throughput: 11.1MiB/s

real 0m33.574s
user 0m2.958s
sys 0m1.431s
Copying gs://bucket-europe-west9-paris/512.txt to file://./512.txt
Completed files 1/1 | 512.0MiB/512.0MiB | 11.5MiB/s

Average throughput: 10.9MiB/s

real 0m51.689s
user 0m3.863s
sys 0m2.208s
Copying gs://bucket-europe-west9-paris/700.txt to file://./700.txt
Completed files 1/1 | 700.0MiB/700.0MiB | 11.0MiB/s

Average throughput: 10.8MiB/s

real 1m10.569s
user 0m5.736s
sys 0m3.201s
Copying gs://bucket-europe-west9-paris/1GB.zip to file://./1GB.zip
Completed files 1/1 | 1.0GiB/1.0GiB | 11.1MiB/s

Average throughput: 8.2MiB/s

real 2m10.469s
user 0m9.193s
sys 0m4.870s
Copying gs://bucket-europe-west9-paris/1_5GB/1GB.zip to file://./1_5GB/1GB.zip
Copying gs://bucket-europe-west9-paris/1_5GB/512MB.zip.crdownload to file://./1_5GB/512MB.zip.crdownload
Completed files 2/2 | 1.5GiB/1.5GiB | 9.8MiB/s

Average throughput: 10.5MiB/s

real 2m34.729s
user 0m11.141s
sys 0m6.369s
Copying gs://bucket-europe-west9-paris/2GB/1GB.zip to file://./2GB/1GB.zip
Copying gs://bucket-europe-west9-paris/2GB/1GB/1GB.zip to file://./2GB/1GB/1GB.zip
Completed files 2/2 | 2.0GiB/2.0GiB | 10.4MiB/s

Average throughput: 10.8MiB/s

real 3m15.154s
user 0m12.736s
sys 0m7.642s
Copying gs://bucket-europe-west9-paris/2_5GB/2GB/1GB.zip to file://./2_5GB/2GB/1GB.zip
Copying gs://bucket-europe-west9-paris/2_5GB/2GB/1GB/1GB.zip to file://./2_5GB/2GB/1GB/1GB.zip
Copying gs://bucket-europe-west9-paris/2_5GB/512MB.zip.crdownload to file://./2_5GB/512MB.zip.crdownload
Completed files 3/3 | 2.5GiB/2.5GiB | 10.9MiB/s

Average throughput: 10.9MiB/s

real 4m4.881s
user 0m16.969s
sys 0m10.345s
Copying gs://bucket-europe-west9-paris/5GB/1GB/1GB.zip to file://./5GB/1GB/1GB.zip
Copying gs://bucket-europe-west9-paris/5GB/2GB/1GB.zip to file://./5GB/2GB/1GB.zip
Copying gs://bucket-europe-west9-paris/5GB/2GB/1GB/1GB.zip to file://./5GB/2GB/1GB/1GB.zip
Copying gs://bucket-europe-west9-paris/5GB/2GB/2GB/1GB.zip to file://./5GB/2GB/2GB/1GB.zip
Copying gs://bucket-europe-west9-paris/5GB/2GB/2GB/1GB/1GB.zip to file://./5GB/2GB/2GB/1GB/1GB.zip
Completed files 5/5 | 5.0GiB/5.0GiB | 11.1MiB/s

Average throughput: 10.9MiB/s

real 7m55.030s
user 0m43.599s
sys 0m21.106s
subash@MSI:/mnt/c/SUBASH/G-CLOUD/BUCKET$
73 changes: 73 additions & 0 deletions logs/bucket/cpmpare-gsutil-vs-gcloud/copy_gsutil.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,73 @@
subash@MSI:/mnt/c/SUBASH/G-CLOUD/BUCKET$ ./download_script.sh
STARTING COPYING USING GSUTIL
Copying gs://bucket-europe-west9-paris/200.txt...
/ [1/1 files][200.0 MiB/200.0 MiB] 100% Done 7.4 MiB/s ETA 00:00:00
Operation completed over 1 objects/200.0 MiB.

real 0m22.302s
user 0m3.247s
sys 0m1.602s
Copying gs://bucket-europe-west9-paris/300.txt...
\ [1/1 files][300.0 MiB/300.0 MiB] 100% Done 10.2 MiB/s ETA 00:00:00
Operation completed over 1 objects/300.0 MiB.

real 0m36.587s
user 0m4.473s
sys 0m1.915s
Copying gs://bucket-europe-west9-paris/512.txt...
- [1/1 files][512.0 MiB/512.0 MiB] 100% Done 8.9 MiB/s ETA 00:00:00
Operation completed over 1 objects/512.0 MiB.

real 1m15.568s
user 0m7.294s
sys 0m3.593s
Copying gs://bucket-europe-west9-paris/700.txt...
/ [1/1 files][700.0 MiB/700.0 MiB] 100% Done 10.7 MiB/s ETA 00:00:00
Operation completed over 1 objects/700.0 MiB.

real 1m6.875s
user 0m8.331s
sys 0m4.514s
Copying gs://bucket-europe-west9-paris/1GB.zip...
/ [1/1 files][ 1.0 GiB/ 1.0 GiB] 100% Done 7.9 MiB/s ETA 00:00:00
Operation completed over 1 objects/1.0 GiB.

real 1m36.536s
user 0m11.866s
sys 0m5.940s
Copying gs://bucket-europe-west9-paris/1_5GB/1GB.zip...
Copying gs://bucket-europe-west9-paris/1_5GB/512MB.zip.crdownload...
\ [2/2 files][ 1.5 GiB/ 1.5 GiB] 100% Done 8.7 MiB/s ETA 00:00:00
Operation completed over 2 objects/1.5 GiB.

real 2m25.500s
user 0m20.299s
sys 0m9.994s
Copying gs://bucket-europe-west9-paris/2GB/1GB.zip...
Copying gs://bucket-europe-west9-paris/2GB/1GB/1GB.zip...
- [2/2 files][ 2.0 GiB/ 2.0 GiB] 100% Done 10.0 MiB/s ETA 00:00:00
Operation completed over 2 objects/2.0 GiB.

real 3m15.222s
user 0m28.931s
sys 0m14.161s
Copying gs://bucket-europe-west9-paris/2_5GB/2GB/1GB.zip...
Copying gs://bucket-europe-west9-paris/2_5GB/2GB/1GB/1GB.zip...
Copying gs://bucket-europe-west9-paris/2_5GB/512MB.zip.crdownload...
- [3/3 files][ 2.5 GiB/ 2.5 GiB] 100% Done 8.4 MiB/s ETA 00:00:00
Operation completed over 3 objects/2.5 GiB.

real 4m14.677s
user 0m38.821s
sys 0m19.219s
Copying gs://bucket-europe-west9-paris/5GB/1GB/1GB.zip...
Copying gs://bucket-europe-west9-paris/5GB/2GB/1GB.zip...
Copying gs://bucket-europe-west9-paris/5GB/2GB/1GB/1GB.zip...
Copying gs://bucket-europe-west9-paris/5GB/2GB/2GB/1GB.zip...
Copying gs://bucket-europe-west9-paris/5GB/2GB/2GB/1GB/1GB.zip...
- [5/5 files][ 5.0 GiB/ 5.0 GiB] 100% Done 10.6 MiB/s ETA 00:00:00
Operation completed over 5 objects/5.0 GiB.

real 7m52.375s
user 1m11.503s
sys 0m31.322s
90 changes: 90 additions & 0 deletions logs/bucket/download-logs/asia-southeast-1-singapore.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,90 @@
subash@MSI:/mnt/c/SUBASH/G-CLOUD/BUCKET$ time gsutil -m cp -r gs://nano-data-output-asia-southeast-1-singapore/512MB.zip.crdownload .
Copying gs://nano-data-output-asia-southeast-1-singapore/512MB.zip.crdownload...
- [1/1 files][512.0 MiB/512.0 MiB] 100% Done 10.5 MiB/s ETA 00:00:00
Operation completed over 1 objects/512.0 MiB.

real 0m53.731s
user 0m7.157s
sys 0m4.560s
subash@MSI:/mnt/c/SUBASH/G-CLOUD/BUCKET$ time gsutil -m cp -r gs://nano-data-output-asia-southeast-1-singapore/1_5GB/ .
Copying gs://nano-data-output-asia-southeast-1-singapore/1_5GB/1GB.zip...
Copying gs://nano-data-output-asia-southeast-1-singapore/1_5GB/512MB.zip.crdownload...
- [2/2 files][ 1.5 GiB/ 1.5 GiB] 100% Done 10.2 MiB/s ETA 00:00:00
Operation completed over 2 objects/1.5 GiB.

real 2m27.596s
user 0m20.662s
sys 0m11.477s
subash@MSI:/mnt/c/SUBASH/G-CLOUD/BUCKET$ time gsutil -m cp -r gs://nano-data-output-asia-southeast-1-singapore/1GB.zip .
Copying gs://nano-data-output-asia-southeast-1-singapore/1GB.zip...
\ [1/1 files][ 1.0 GiB/ 1.0 GiB] 100% Done 10.0 MiB/s ETA 00:00:00
Operation completed over 1 objects/1.0 GiB.

real 1m40.784s
user 0m12.795s
sys 0m8.000s
subash@MSI:/mnt/c/SUBASH/G-CLOUD/BUCKET$ time gsutil -m cp -r gs://nano-data-output-asia-southeast-1-singapore/2GB/ .

Copying gs://nano-data-output-asia-southeast-1-singapore/2GB/1GB.zip...
Copying gs://nano-data-output-asia-southeast-1-singapore/2GB/1GB/1GB.zip...
- [2/2 files][ 2.0 GiB/ 2.0 GiB] 100% Done 11.6 MiB/s ETA 00:00:00
Operation completed over 2 objects/2.0 GiB.

real 3m15.546s
user 0m25.954s
sys 0m14.656s
subash@MSI:/mnt/c/SUBASH/G-CLOUD/BUCKET$ time gsutil -m cp -r gs://nano-data-output-asia-southeast-1-singapore/2_5GB/ .
Copying gs://nano-data-output-asia-southeast-1-singapore/2_5GB/2GB/1GB.zip...
Copying gs://nano-data-output-asia-southeast-1-singapore/2_5GB/2GB/1GB/1GB.zip...
Copying gs://nano-data-output-asia-southeast-1-singapore/2_5GB/512MB.zip.crdownload...
- [3/3 files][ 2.5 GiB/ 2.5 GiB] 100% Done 10.4 MiB/s ETA 00:00:00
Operation completed over 3 objects/2.5 GiB.

real 4m3.119s
user 0m33.807s
sys 0m18.005s
subash@MSI:/mnt/c/SUBASH/G-CLOUD/BUCKET$ time gsutil -m cp -r gs://nano-data-output-asia-southeast-1-singapore/5GB/ .
Copying gs://nano-data-output-asia-southeast-1-singapore/5GB/1GB/1GB.zip...
Copying gs://nano-data-output-asia-southeast-1-singapore/5GB/2GB/1GB.zip...
Copying gs://nano-data-output-asia-southeast-1-singapore/5GB/2GB/1GB/1GB.zip...
Copying gs://nano-data-output-asia-southeast-1-singapore/5GB/2GB/2GB/1GB.zip...
Copying gs://nano-data-output-asia-southeast-1-singapore/5GB/2GB/2GB/1GB/1GB.zip...
\ [5/5 files][ 5.0 GiB/ 5.0 GiB] 100% Done 11.0 MiB/s ETA 00:00:00
Operation completed over 5 objects/5.0 GiB.

real 7m57.676s
user 1m12.466s
sys 0m33.232s
subash@MSI:/mnt/c/SUBASH/G-CLOUD/BUCKET$ time gsutil -m cp -r gs://nano-data-output-asia-southeast-1-singapore/7_5GB/ .
Copying gs://nano-data-output-asia-southeast-1-singapore/7_5GB/2.5GB/2GB/1GB/1GB.zip...
Copying gs://nano-data-output-asia-southeast-1-singapore/7_5GB/2.5GB/2GB/1GB.zip...
Copying gs://nano-data-output-asia-southeast-1-singapore/7_5GB/2.5GB/512MB.zip.crdownload...
Copying gs://nano-data-output-asia-southeast-1-singapore/7_5GB/5GB/1GB/1GB.zip...
Copying gs://nano-data-output-asia-southeast-1-singapore/7_5GB/5GB/2GB/1GB/1GB.zip...
Copying gs://nano-data-output-asia-southeast-1-singapore/7_5GB/5GB/2GB/2GB/1GB.zip...
Copying gs://nano-data-output-asia-southeast-1-singapore/7_5GB/5GB/2GB/2GB/1GB/1GB.zip...
Copying gs://nano-data-output-asia-southeast-1-singapore/7_5GB/5GB/2GB/1GB.zip...
| [8/8 files][ 7.5 GiB/ 7.5 GiB] 100% Done 11.0 MiB/s ETA 00:00:00
Operation completed over 8 objects/7.5 GiB.

real 12m16.677s
user 2m12.451s
sys 0m54.926s
subash@MSI:/mnt/c/SUBASH/G-CLOUD/BUCKET$ time gsutil -m cp -r gs://nano-data-output-asia-southeast-1-singapore/10GB/ .
Copying gs://nano-data-output-asia-southeast-1-singapore/10GB/5GB/1GB/1GB.zip...
Copying gs://nano-data-output-asia-southeast-1-singapore/10GB/5GB/2GB/1GB.zip...
Copying gs://nano-data-output-asia-southeast-1-singapore/10GB/5GB/2GB/1GB/1GB.zip...
Copying gs://nano-data-output-asia-southeast-1-singapore/10GB/5GB/2GB/2GB/1GB.zip...
Copying gs://nano-data-output-asia-southeast-1-singapore/10GB/5GB/2GB/2GB/1GB/1GB.zip...
Copying gs://nano-data-output-asia-southeast-1-singapore/10GB/5GB/5GB/1GB/1GB.zip...
Copying gs://nano-data-output-asia-southeast-1-singapore/10GB/5GB/5GB/2GB/1GB.zip...
Copying gs://nano-data-output-asia-southeast-1-singapore/10GB/5GB/5GB/2GB/1GB/1GB.zip...
Copying gs://nano-data-output-asia-southeast-1-singapore/10GB/5GB/5GB/2GB/2GB/1GB.zip...
Copying gs://nano-data-output-asia-southeast-1-singapore/10GB/5GB/5GB/2GB/2GB/1GB/1GB.zip...
| [10/10 files][ 10.0 GiB/ 10.0 GiB] 100% Done 10.9 MiB/s ETA 00:00:00
Operation completed over 10 objects/10.0 GiB.

real 16m8.564s
user 3m8.747s
sys 1m17.473s
subash@MSI:/mnt/c/SUBASH/G-CLOUD/BUCKET$
102 changes: 102 additions & 0 deletions logs/bucket/download-logs/bucket-europe-central2-warsaw.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,102 @@
Copying gs://bucket-europe-central2-warsaw/200.txt...
\ [1/1 files][200.0 MiB/200.0 MiB] 100% Done 9.0 MiB/s ETA 00:00:00
Operation completed over 1 objects/200.0 MiB.

real 0m21.573s
user 0m3.791s
sys 0m2.011s
Copying gs://bucket-europe-central2-warsaw/300.txt...
\ [1/1 files][300.0 MiB/300.0 MiB] 100% Done 7.9 MiB/s ETA 00:00:00
Operation completed over 1 objects/300.0 MiB.

real 0m31.690s
user 0m4.839s
sys 0m2.442s
Copying gs://bucket-europe-central2-warsaw/512.txt...
- [1/1 files][512.0 MiB/512.0 MiB] 100% Done 7.7 MiB/s ETA 00:00:00
Operation completed over 1 objects/512.0 MiB.

real 0m51.643s
user 0m8.841s
sys 0m4.930s
Copying gs://bucket-europe-central2-warsaw/700.txt...
| [1/1 files][700.0 MiB/700.0 MiB] 100% Done 8.8 MiB/s ETA 00:00:00
Operation completed over 1 objects/700.0 MiB.

real 1m6.898s
user 0m9.864s
sys 0m4.382s
Copying gs://bucket-europe-central2-warsaw/1GB.zip...
- [1/1 files][ 1.0 GiB/ 1.0 GiB] 100% Done 10.1 MiB/s ETA 00:00:00
Operation completed over 1 objects/1.0 GiB.

real 1m38.031s
user 0m13.176s
sys 0m7.409s
Copying gs://bucket-europe-central2-warsaw/1_5GB/1GB.zip...
Copying gs://bucket-europe-central2-warsaw/1_5GB/512MB.zip.crdownload...
/ [2/2 files][ 1.5 GiB/ 1.5 GiB] 100% Done 7.5 MiB/s ETA 00:00:00
Operation completed over 2 objects/1.5 GiB.

real 2m25.607s
user 0m18.388s
sys 0m9.188s
Copying gs://bucket-europe-central2-warsaw/2GB/1GB.zip...
Copying gs://bucket-europe-central2-warsaw/2GB/1GB/1GB.zip...
\ [2/2 files][ 2.0 GiB/ 2.0 GiB] 100% Done 11.0 MiB/s ETA 00:00:00
Operation completed over 2 objects/2.0 GiB.

real 3m8.967s
user 0m26.759s
sys 0m13.706s
Copying gs://bucket-europe-central2-warsaw/2_5GB/2GB/1GB.zip...
Copying gs://bucket-europe-central2-warsaw/2_5GB/512MB.zip.crdownload...
Copying gs://bucket-europe-central2-warsaw/2_5GB/2GB/1GB/1GB.zip...
- [3/3 files][ 2.5 GiB/ 2.5 GiB] 100% Done 8.6 MiB/s ETA 00:00:00
Operation completed over 3 objects/2.5 GiB.

real 4m0.371s
user 0m32.946s
sys 0m17.103s
Copying gs://bucket-europe-central2-warsaw/5GB/1GB/1GB.zip...
Copying gs://bucket-europe-central2-warsaw/5GB/2GB/2GB/1GB.zip...
Copying gs://bucket-europe-central2-warsaw/5GB/2GB/1GB.zip...
Copying gs://bucket-europe-central2-warsaw/5GB/2GB/1GB/1GB.zip...
Copying gs://bucket-europe-central2-warsaw/5GB/2GB/2GB/1GB/1GB.zip...
\ [5/5 files][ 5.0 GiB/ 5.0 GiB] 100% Done 9.6 MiB/s ETA 00:00:00
Operation completed over 5 objects/5.0 GiB.

real 7m56.219s
user 1m11.586s
sys 0m31.992s
Copying gs://bucket-europe-central2-warsaw/7_5GB/2.5GB/2GB/1GB.zip...
Copying gs://bucket-europe-central2-warsaw/7_5GB/2.5GB/2GB/1GB/1GB.zip...
Copying gs://bucket-europe-central2-warsaw/7_5GB/2.5GB/512MB.zip.crdownload...
Copying gs://bucket-europe-central2-warsaw/7_5GB/5GB/1GB/1GB.zip...
Copying gs://bucket-europe-central2-warsaw/7_5GB/5GB/2GB/1GB.zip...
Copying gs://bucket-europe-central2-warsaw/7_5GB/5GB/2GB/1GB/1GB.zip...
Copying gs://bucket-europe-central2-warsaw/7_5GB/5GB/2GB/2GB/1GB/1GB.zip...
Copying gs://bucket-europe-central2-warsaw/7_5GB/5GB/2GB/2GB/1GB.zip...
| [8/8 files][ 7.5 GiB/ 7.5 GiB] 100% Done 10.4 MiB/s ETA 00:00:00
Operation completed over 8 objects/7.5 GiB.

real 11m50.116s
user 1m54.695s
sys 0m49.072s
Copying gs://bucket-europe-central2-warsaw/10GB/5GB/1GB/1GB.zip...
Copying gs://bucket-europe-central2-warsaw/10GB/5GB/2GB/1GB.zip...
Copying gs://bucket-europe-central2-warsaw/10GB/5GB/2GB/1GB/1GB.zip...
Copying gs://bucket-europe-central2-warsaw/10GB/5GB/2GB/2GB/1GB.zip...
Copying gs://bucket-europe-central2-warsaw/10GB/5GB/2GB/2GB/1GB/1GB.zip...
Copying gs://bucket-europe-central2-warsaw/10GB/5GB/5GB/1GB/1GB.zip...
Copying gs://bucket-europe-central2-warsaw/10GB/5GB/5GB/2GB/1GB.zip...
Copying gs://bucket-europe-central2-warsaw/10GB/5GB/5GB/2GB/2GB/1GB.zip...
Copying gs://bucket-europe-central2-warsaw/10GB/5GB/5GB/2GB/2GB/1GB/1GB.zip...
Copying gs://bucket-europe-central2-warsaw/10GB/5GB/5GB/2GB/1GB/1GB.zip...
| [10/10 files][ 10.0 GiB/ 10.0 GiB] 100% Done 10.6 MiB/s ETA 00:00:00
Operation completed over 10 objects/10.0 GiB.

real 15m43.523s
user 2m36.899s
sys 1m5.150s
subash@MSI:/mnt/c/SUBASH/G-CLOUD/BUCKET$
Loading

0 comments on commit f9a8e8b

Please sign in to comment.