Skip to content

Commit

Permalink
adding comparision log files gsutil vs gcloud storage & bucket downlo…
Browse files Browse the repository at this point in the history
…ad time logs for #22
  • Loading branch information
subash-taranga committed Jul 11, 2024
1 parent d14f1b1 commit 13d8812
Show file tree
Hide file tree
Showing 13 changed files with 991 additions and 156 deletions.
83 changes: 0 additions & 83 deletions compare_gsutil_Vs_gcloud-storage/copy_gcloud_storage.txt

This file was deleted.

73 changes: 0 additions & 73 deletions compare_gsutil_Vs_gcloud-storage/copy_gsutil.txt

This file was deleted.

90 changes: 90 additions & 0 deletions log-bkt-download-time/asia-southeast-1-singapore.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,90 @@
subash@MSI:/mnt/c/SUBASH/G-CLOUD/BUCKET$ time gsutil -m cp -r gs://nano-data-output-asia-southeast-1-singapore/512MB.zip.crdownload .
Copying gs://nano-data-output-asia-southeast-1-singapore/512MB.zip.crdownload...
- [1/1 files][512.0 MiB/512.0 MiB] 100% Done 10.5 MiB/s ETA 00:00:00
Operation completed over 1 objects/512.0 MiB.

real 0m53.731s
user 0m7.157s
sys 0m4.560s
subash@MSI:/mnt/c/SUBASH/G-CLOUD/BUCKET$ time gsutil -m cp -r gs://nano-data-output-asia-southeast-1-singapore/1_5GB/ .
Copying gs://nano-data-output-asia-southeast-1-singapore/1_5GB/1GB.zip...
Copying gs://nano-data-output-asia-southeast-1-singapore/1_5GB/512MB.zip.crdownload...
- [2/2 files][ 1.5 GiB/ 1.5 GiB] 100% Done 10.2 MiB/s ETA 00:00:00
Operation completed over 2 objects/1.5 GiB.

real 2m27.596s
user 0m20.662s
sys 0m11.477s
subash@MSI:/mnt/c/SUBASH/G-CLOUD/BUCKET$ time gsutil -m cp -r gs://nano-data-output-asia-southeast-1-singapore/1GB.zip .
Copying gs://nano-data-output-asia-southeast-1-singapore/1GB.zip...
\ [1/1 files][ 1.0 GiB/ 1.0 GiB] 100% Done 10.0 MiB/s ETA 00:00:00
Operation completed over 1 objects/1.0 GiB.

real 1m40.784s
user 0m12.795s
sys 0m8.000s
subash@MSI:/mnt/c/SUBASH/G-CLOUD/BUCKET$ time gsutil -m cp -r gs://nano-data-output-asia-southeast-1-singapore/2GB/ .

Copying gs://nano-data-output-asia-southeast-1-singapore/2GB/1GB.zip...
Copying gs://nano-data-output-asia-southeast-1-singapore/2GB/1GB/1GB.zip...
- [2/2 files][ 2.0 GiB/ 2.0 GiB] 100% Done 11.6 MiB/s ETA 00:00:00
Operation completed over 2 objects/2.0 GiB.

real 3m15.546s
user 0m25.954s
sys 0m14.656s
subash@MSI:/mnt/c/SUBASH/G-CLOUD/BUCKET$ time gsutil -m cp -r gs://nano-data-output-asia-southeast-1-singapore/2_5GB/ .
Copying gs://nano-data-output-asia-southeast-1-singapore/2_5GB/2GB/1GB.zip...
Copying gs://nano-data-output-asia-southeast-1-singapore/2_5GB/2GB/1GB/1GB.zip...
Copying gs://nano-data-output-asia-southeast-1-singapore/2_5GB/512MB.zip.crdownload...
- [3/3 files][ 2.5 GiB/ 2.5 GiB] 100% Done 10.4 MiB/s ETA 00:00:00
Operation completed over 3 objects/2.5 GiB.

real 4m3.119s
user 0m33.807s
sys 0m18.005s
subash@MSI:/mnt/c/SUBASH/G-CLOUD/BUCKET$ time gsutil -m cp -r gs://nano-data-output-asia-southeast-1-singapore/5GB/ .
Copying gs://nano-data-output-asia-southeast-1-singapore/5GB/1GB/1GB.zip...
Copying gs://nano-data-output-asia-southeast-1-singapore/5GB/2GB/1GB.zip...
Copying gs://nano-data-output-asia-southeast-1-singapore/5GB/2GB/1GB/1GB.zip...
Copying gs://nano-data-output-asia-southeast-1-singapore/5GB/2GB/2GB/1GB.zip...
Copying gs://nano-data-output-asia-southeast-1-singapore/5GB/2GB/2GB/1GB/1GB.zip...
\ [5/5 files][ 5.0 GiB/ 5.0 GiB] 100% Done 11.0 MiB/s ETA 00:00:00
Operation completed over 5 objects/5.0 GiB.

real 7m57.676s
user 1m12.466s
sys 0m33.232s
subash@MSI:/mnt/c/SUBASH/G-CLOUD/BUCKET$ time gsutil -m cp -r gs://nano-data-output-asia-southeast-1-singapore/7_5GB/ .
Copying gs://nano-data-output-asia-southeast-1-singapore/7_5GB/2.5GB/2GB/1GB/1GB.zip...
Copying gs://nano-data-output-asia-southeast-1-singapore/7_5GB/2.5GB/2GB/1GB.zip...
Copying gs://nano-data-output-asia-southeast-1-singapore/7_5GB/2.5GB/512MB.zip.crdownload...
Copying gs://nano-data-output-asia-southeast-1-singapore/7_5GB/5GB/1GB/1GB.zip...
Copying gs://nano-data-output-asia-southeast-1-singapore/7_5GB/5GB/2GB/1GB/1GB.zip...
Copying gs://nano-data-output-asia-southeast-1-singapore/7_5GB/5GB/2GB/2GB/1GB.zip...
Copying gs://nano-data-output-asia-southeast-1-singapore/7_5GB/5GB/2GB/2GB/1GB/1GB.zip...
Copying gs://nano-data-output-asia-southeast-1-singapore/7_5GB/5GB/2GB/1GB.zip...
| [8/8 files][ 7.5 GiB/ 7.5 GiB] 100% Done 11.0 MiB/s ETA 00:00:00
Operation completed over 8 objects/7.5 GiB.

real 12m16.677s
user 2m12.451s
sys 0m54.926s
subash@MSI:/mnt/c/SUBASH/G-CLOUD/BUCKET$ time gsutil -m cp -r gs://nano-data-output-asia-southeast-1-singapore/10GB/ .
Copying gs://nano-data-output-asia-southeast-1-singapore/10GB/5GB/1GB/1GB.zip...
Copying gs://nano-data-output-asia-southeast-1-singapore/10GB/5GB/2GB/1GB.zip...
Copying gs://nano-data-output-asia-southeast-1-singapore/10GB/5GB/2GB/1GB/1GB.zip...
Copying gs://nano-data-output-asia-southeast-1-singapore/10GB/5GB/2GB/2GB/1GB.zip...
Copying gs://nano-data-output-asia-southeast-1-singapore/10GB/5GB/2GB/2GB/1GB/1GB.zip...
Copying gs://nano-data-output-asia-southeast-1-singapore/10GB/5GB/5GB/1GB/1GB.zip...
Copying gs://nano-data-output-asia-southeast-1-singapore/10GB/5GB/5GB/2GB/1GB.zip...
Copying gs://nano-data-output-asia-southeast-1-singapore/10GB/5GB/5GB/2GB/1GB/1GB.zip...
Copying gs://nano-data-output-asia-southeast-1-singapore/10GB/5GB/5GB/2GB/2GB/1GB.zip...
Copying gs://nano-data-output-asia-southeast-1-singapore/10GB/5GB/5GB/2GB/2GB/1GB/1GB.zip...
| [10/10 files][ 10.0 GiB/ 10.0 GiB] 100% Done 10.9 MiB/s ETA 00:00:00
Operation completed over 10 objects/10.0 GiB.

real 16m8.564s
user 3m8.747s
sys 1m17.473s
subash@MSI:/mnt/c/SUBASH/G-CLOUD/BUCKET$
102 changes: 102 additions & 0 deletions log-bkt-download-time/bucket-europe-central2-warsaw.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,102 @@
Copying gs://bucket-europe-central2-warsaw/200.txt...
\ [1/1 files][200.0 MiB/200.0 MiB] 100% Done 9.0 MiB/s ETA 00:00:00
Operation completed over 1 objects/200.0 MiB.

real 0m21.573s
user 0m3.791s
sys 0m2.011s
Copying gs://bucket-europe-central2-warsaw/300.txt...
\ [1/1 files][300.0 MiB/300.0 MiB] 100% Done 7.9 MiB/s ETA 00:00:00
Operation completed over 1 objects/300.0 MiB.

real 0m31.690s
user 0m4.839s
sys 0m2.442s
Copying gs://bucket-europe-central2-warsaw/512.txt...
- [1/1 files][512.0 MiB/512.0 MiB] 100% Done 7.7 MiB/s ETA 00:00:00
Operation completed over 1 objects/512.0 MiB.

real 0m51.643s
user 0m8.841s
sys 0m4.930s
Copying gs://bucket-europe-central2-warsaw/700.txt...
| [1/1 files][700.0 MiB/700.0 MiB] 100% Done 8.8 MiB/s ETA 00:00:00
Operation completed over 1 objects/700.0 MiB.

real 1m6.898s
user 0m9.864s
sys 0m4.382s
Copying gs://bucket-europe-central2-warsaw/1GB.zip...
- [1/1 files][ 1.0 GiB/ 1.0 GiB] 100% Done 10.1 MiB/s ETA 00:00:00
Operation completed over 1 objects/1.0 GiB.

real 1m38.031s
user 0m13.176s
sys 0m7.409s
Copying gs://bucket-europe-central2-warsaw/1_5GB/1GB.zip...
Copying gs://bucket-europe-central2-warsaw/1_5GB/512MB.zip.crdownload...
/ [2/2 files][ 1.5 GiB/ 1.5 GiB] 100% Done 7.5 MiB/s ETA 00:00:00
Operation completed over 2 objects/1.5 GiB.

real 2m25.607s
user 0m18.388s
sys 0m9.188s
Copying gs://bucket-europe-central2-warsaw/2GB/1GB.zip...
Copying gs://bucket-europe-central2-warsaw/2GB/1GB/1GB.zip...
\ [2/2 files][ 2.0 GiB/ 2.0 GiB] 100% Done 11.0 MiB/s ETA 00:00:00
Operation completed over 2 objects/2.0 GiB.

real 3m8.967s
user 0m26.759s
sys 0m13.706s
Copying gs://bucket-europe-central2-warsaw/2_5GB/2GB/1GB.zip...
Copying gs://bucket-europe-central2-warsaw/2_5GB/512MB.zip.crdownload...
Copying gs://bucket-europe-central2-warsaw/2_5GB/2GB/1GB/1GB.zip...
- [3/3 files][ 2.5 GiB/ 2.5 GiB] 100% Done 8.6 MiB/s ETA 00:00:00
Operation completed over 3 objects/2.5 GiB.

real 4m0.371s
user 0m32.946s
sys 0m17.103s
Copying gs://bucket-europe-central2-warsaw/5GB/1GB/1GB.zip...
Copying gs://bucket-europe-central2-warsaw/5GB/2GB/2GB/1GB.zip...
Copying gs://bucket-europe-central2-warsaw/5GB/2GB/1GB.zip...
Copying gs://bucket-europe-central2-warsaw/5GB/2GB/1GB/1GB.zip...
Copying gs://bucket-europe-central2-warsaw/5GB/2GB/2GB/1GB/1GB.zip...
\ [5/5 files][ 5.0 GiB/ 5.0 GiB] 100% Done 9.6 MiB/s ETA 00:00:00
Operation completed over 5 objects/5.0 GiB.

real 7m56.219s
user 1m11.586s
sys 0m31.992s
Copying gs://bucket-europe-central2-warsaw/7_5GB/2.5GB/2GB/1GB.zip...
Copying gs://bucket-europe-central2-warsaw/7_5GB/2.5GB/2GB/1GB/1GB.zip...
Copying gs://bucket-europe-central2-warsaw/7_5GB/2.5GB/512MB.zip.crdownload...
Copying gs://bucket-europe-central2-warsaw/7_5GB/5GB/1GB/1GB.zip...
Copying gs://bucket-europe-central2-warsaw/7_5GB/5GB/2GB/1GB.zip...
Copying gs://bucket-europe-central2-warsaw/7_5GB/5GB/2GB/1GB/1GB.zip...
Copying gs://bucket-europe-central2-warsaw/7_5GB/5GB/2GB/2GB/1GB/1GB.zip...
Copying gs://bucket-europe-central2-warsaw/7_5GB/5GB/2GB/2GB/1GB.zip...
| [8/8 files][ 7.5 GiB/ 7.5 GiB] 100% Done 10.4 MiB/s ETA 00:00:00
Operation completed over 8 objects/7.5 GiB.

real 11m50.116s
user 1m54.695s
sys 0m49.072s
Copying gs://bucket-europe-central2-warsaw/10GB/5GB/1GB/1GB.zip...
Copying gs://bucket-europe-central2-warsaw/10GB/5GB/2GB/1GB.zip...
Copying gs://bucket-europe-central2-warsaw/10GB/5GB/2GB/1GB/1GB.zip...
Copying gs://bucket-europe-central2-warsaw/10GB/5GB/2GB/2GB/1GB.zip...
Copying gs://bucket-europe-central2-warsaw/10GB/5GB/2GB/2GB/1GB/1GB.zip...
Copying gs://bucket-europe-central2-warsaw/10GB/5GB/5GB/1GB/1GB.zip...
Copying gs://bucket-europe-central2-warsaw/10GB/5GB/5GB/2GB/1GB.zip...
Copying gs://bucket-europe-central2-warsaw/10GB/5GB/5GB/2GB/2GB/1GB.zip...
Copying gs://bucket-europe-central2-warsaw/10GB/5GB/5GB/2GB/2GB/1GB/1GB.zip...
Copying gs://bucket-europe-central2-warsaw/10GB/5GB/5GB/2GB/1GB/1GB.zip...
| [10/10 files][ 10.0 GiB/ 10.0 GiB] 100% Done 10.6 MiB/s ETA 00:00:00
Operation completed over 10 objects/10.0 GiB.

real 15m43.523s
user 2m36.899s
sys 1m5.150s
subash@MSI:/mnt/c/SUBASH/G-CLOUD/BUCKET$
Loading

0 comments on commit 13d8812

Please sign in to comment.