feat(fetcher): implement vaksinasi.id db fetcher #806

togetherwithasteria · 2021-10-09T06:54:40Z

Fetches vaksinasi.id's DB and store it in wbw-vaccination-database.json

Closes #803

Description

The issue didn't give a lot of information, but I assume the goal is to have all the data from the API, as the endgoal is to write it all to wbw-vaccination-database. So this will fetch for the /regions endpoint of the vaksinasi.id API first, then iterate over the provinces and fetch its vaccination locations from the /locations/:region endpoint, dump all of the Promise'd JSON response into the promisedLocations array and then use Promise.all to await all of the promises in the locations array. Lastly, stringify locations and write it into wbw-vaccination-database.json

Just a quick question. I wrote the mockups for the responses in the vaksinid-${endpoint} format, but as I think this project uses the wbw prefix a lot, should I rename them to use the wbw prefix or not?

Also this is my first time contributing to big projects. Please forgive me for my mistakes.

Current Tasks

Typings (should I?)
Main code
Tests

Fetches vaksinasi.id's DB and store it in wbw-vaccination-database.json Refs: #803

netlify · 2021-10-09T06:54:46Z

✔️ Deploy Preview for wargabantuwarga ready!

🔨 Explore the source changes: d5de61b

🔍 Inspect the deploy log: https://app.netlify.com/sites/wargabantuwarga/deploys/616aa402ea93030007ec85f0

😎 Browse the preview: https://deploy-preview-806--wargabantuwarga.netlify.app

codecov · 2021-10-09T06:58:36Z

Codecov Report

Merging #806 (d5de61b) into main (e373b9f) will increase coverage by 0.13%.
The diff coverage is 100.00%.

@@            Coverage Diff             @@
##             main     #806      +/-   ##
==========================================
+ Coverage   86.92%   87.06%   +0.13%     
==========================================
  Files         133      134       +1     
  Lines        1423     1438      +15     
  Branches      454      455       +1     
==========================================
+ Hits         1237     1252      +15     
  Misses        181      181              
  Partials        5        5

Impacted Files	Coverage Δ
etc/fetchers/fetch-vaccination-database.ts	`100.00% <100.00%> (ø)`

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 1487177...d5de61b. Read the comment docs.

togetherwithasteria · 2021-10-09T07:04:59Z

Mm, should I put mark this as "Ready for Review" or not?

There is an ESLint warning in c84e78c, this commit addresses that warn

zainfathoni · 2021-10-09T11:53:09Z

Just a quick question. I wrote the mockups for the responses in the vaksinid-${endpoint} format, but as I think this project uses the wbw prefix a lot, should I rename them to use the wbw prefix or not?

Since your vaksinid-${endpoint} is for your internal fetchers usage, I think it's okay to keep the vaksinid prefix.
We just used the wbw prefix because those data are provided by WBW, but in this case, the data is provided by another party, so I think putting the data source as the prefix makes sense.

Also this is my first time contributing to big projects. Forgive me for my n00b mistakes.

For a first-time contribution, your code seems sufficiently clean. You also write good tests as well. Thanks!

togetherwithasteria · 2021-10-09T12:13:28Z

Also this is my first time contributing to big projects. Forgive me for my n00b mistakes.

For a first-time contribution, your code seems sufficiently clean. You also write good tests as well. Thanks!

Oh, well. Maybe I'm just too shy. Your welcome!

zainfathoni

Overall, your implementation looks good and is going in the right direction.

However, we need you to perform further transformation into the data so that it's ready to be used downstream in the components.

etc/fetchers/fetch-vaccination-database.ts

mazipan

LGTM, wait for approval from others

mazipan · 2021-10-10T04:04:29Z

You can contact me on Twitter @FortressValkyrie after this PR merged.

--> https://twitter.com/Maz_Ipan

etc/fetchers/fetch-wbw.ts

etc/fetchers/fetch-vaccination-database.ts

fetchVaccinationDatabase() is so fast that a spinner stucks the program. So, move it to the top and remove line 37.

…tible Transforms the output of the vaccination db fetcher to VaccinationContact[]. This will make it ready to be used downstream in the components. The VaccinationContact interface has a 'map' property. This is unecessary as WBW's "Buka Peta" button uses the address instead. Though, I still add it in case we want to use that for the "Buka Peta" button.

etc/fetchers/fetch-wbw.ts

…nce[] Make it easy for the DB to be merged with the other contact DBs. Using VaccinationContact[] might force us into unnecessarily iterating the array just to merge it.

togetherwithasteria

Wait, will d4d8379 really help reducing unnecessary iterations? Idk, I'm dumb. If you want to revert this, just tell me.

togetherwithasteria · 2021-10-11T10:59:49Z

Is ccb05a7 good?

zainfathoni · 2021-10-11T11:07:34Z

I'll review it later tonight.

zainfathoni · 2021-10-11T13:51:26Z

Wait, will d4d8379 really help reducing unnecessary iterations? Idk, I'm dumb. If you want to revert this, just tell me.

Yes, I think so. But I think it would be better if you can structure it into this format instead, so when we're iterating the provinces, we can easily get the data by accessing the property using computed values from the province name.
e.g. vaccinationDatabase[provinceName].

{
  "Aceh": [
    {
      "alamat": "Jalan Inong Balee No.38 Darussalam Banda Aceh",
      "lokasi": "Kota Banda Aceh",
      "buka_waktu": "08:00:00",
      "id": "0",
      "informasi_2": "-Senin dan Sabtu (bisa berubah sewaktu-waktu)\n-Dibeikan bagi warga dengan kriteria:\n1.petugas pelayanan publik\n2. Pra lansia &lansia\n3. Masyarakat Banda Aceh di utamakan yang berdomisili di wilayah Puskesmas Kopelma Darussalam,Rukoh,ie masen kayee adang,Lamgugob & Deah raya\n-Membawa KTP\n-informasi tambahan bisa dilihat di akun Instagram https://instagram.com/puskesmaskopelmadarussalam?utm_medium=copy_link",
      "keterangan": "Lokasi Vaksinasi COVID-19",
      "link": "https://www.instagram.com/p/CQgFAepnZkQ/?utm_medium=copy_link",
      "map": "https://goo.gl/maps/9hViESZJ9rYwGypb9",
      "mulai_tanggal": "2021-06-24",
      "penyedia": "UPTD PUSKESMAS KOPELMA DARUSSALAM",
      "rentang_umur": ["Dewasa (18-59 Tahun)", "Lansia (60-)"],
      "selesai_tanggal": "2022-03-31",
      "slug": "uptd-puskesmas-kopelma-darussalam",
      "terakhir_update": "2021-07-30",
      "tutup_waktu": "12:00:00",
      "verifikasi": 1
    }
  ]
}

zainfathoni

Nice, thanks for making the changes! ✅

But I still have another minor request here, could you please address it? Thanks! 🙏

etc/fetchers/fetch-vaccination-database.ts

togetherwithasteria · 2021-10-12T11:51:43Z

f35f99b is done. Is this ready now?

zainfathoni

The code seems good enough so far.

However, when I tried running it locally while logging each step of the API calls, it seems that it only managed to fetch the data for four provinces.

Could you please try doing the same thing here in your local machine and share your results in screenshots? Thanks! 🙏

togetherwithasteria · 2021-10-13T07:23:32Z

Could you please try doing the same thing here in your local machine and share your results in screenshots? Thanks! 🙏

Umm, seems negative.

zainfathoni · 2021-10-13T15:10:32Z

Okay, then. Let's try seeing it in action at Netlify.
Please change mirror-box usage into fetch-wbw in this line and push your changes to this branch.

We will be able to see how it works in Netlify. If it breaks nothing, we can proceed with merging this PR. Thanks!

togetherwithasteria · 2021-10-14T06:57:52Z

Wait, with the console.logs you put or not?

zainfathoni · 2021-10-14T12:40:25Z

Wait, with the console.logs you put or not?

Without the console logging, we couldn't see anything here.

Let's try adding the console logging and see how it goes.

togetherwithasteria · 2021-10-15T11:10:32Z

6:05:09 PM: - Fetching next data...
6:05:09 PM: Aceh
6:05:09 PM: Sumatera Utara
6:05:09 PM: Sumatera Barat
6:05:09 PM: Riau
6:05:09 PM: Jambi
6:05:09 PM: Sumatera Selatan
6:05:09 PM: Bengkulu
6:05:09 PM: Lampung
6:05:09 PM: Kepulauan Bangka Belitung
6:05:09 PM: Kepulauan Riau
6:05:09 PM: DKI Jakarta
6:05:09 PM: Jawa Barat
6:05:09 PM: Jawa Tengah
6:05:09 PM: DI Yogyakarta
6:05:09 PM: Jawa Timur
6:05:09 PM: Banten
6:05:09 PM: Bali
6:05:09 PM: Nusa Tenggara Barat
6:05:09 PM: Nusa Tenggara Timur
6:05:09 PM: Kalimantan Barat
6:05:09 PM: Kalimantan Tengah
6:05:09 PM: Kalimantan Selatan
6:05:09 PM: Kalimantan Timur
6:05:09 PM: Kalimantan Utara
6:05:09 PM: Sulawesi Utara
6:05:09 PM: Sulawesi Tengah
6:05:09 PM: Sulawesi Selatan
6:05:09 PM: Sulawesi Tenggara
6:05:09 PM: Gorontalo
6:05:09 PM: Sulawesi Barat
6:05:09 PM: Maluku
6:05:09 PM: Maluku Utara
6:05:09 PM: Papua
6:05:09 PM: Papua Barat
6:05:12 PM: Aceh 186
6:05:12 PM: Sumatera Utara 1185
6:05:12 PM: Sumatera Barat 32
6:05:12 PM: Riau 67
6:05:32 PM: ✔ Fetching Database done in 23.768 seconds

Deployed it to a personal Netlify site. Yep, could reproduce your issue. Idk why though.

…a.com into feat/fetch-vaccination-database

togetherwithasteria · 2021-10-15T14:11:56Z

LET'S GET THIS MERGED!!!

9:08:29 PM: Papua Barat
9:08:47 PM: ✔ Fetching Database done in 18.907 seconds
9:08:47 PM: - Fetching next data...
9:09:53 PM: ✔ Fetching Sheets done in 84.806 seconds
9:09:53 PM: Kepulauan Bangka Belitung 26
9:09:53 PM: Lampung 18
9:09:53 PM: Riau 67
9:09:53 PM: DI Yogyakarta 41
9:09:53 PM: Sumatera Barat 32
9:09:53 PM: Jawa Timur 75
9:09:53 PM: Kepulauan Riau 85
9:09:53 PM: Nusa Tenggara Timur 5
9:09:53 PM: Kalimantan Utara 17
9:09:53 PM: Maluku 8
9:09:54 PM: Bengkulu 123
9:09:54 PM: Jawa Barat 128
9:09:54 PM: Sulawesi Tengah 43
9:09:54 PM: Maluku Utara 33
9:09:54 PM: Gorontalo 26
9:09:54 PM: Sulawesi Barat 32
9:09:54 PM: Papua 41
9:09:54 PM: Sulawesi Selatan 52
9:09:54 PM: Sulawesi Utara 17
9:09:54 PM: Kalimantan Tengah 17
9:09:54 PM: Banten 49
9:09:54 PM: Nusa Tenggara Barat 43
9:09:54 PM: Papua Barat 24
9:09:54 PM: Kalimantan Timur 28
9:09:54 PM: Aceh 186
9:09:54 PM: Kalimantan Barat 61
9:09:54 PM: DKI Jakarta 53
9:09:54 PM: Bali 86
9:09:54 PM: Sulawesi Tenggara 67
9:09:54 PM: Kalimantan Selatan 145
9:09:54 PM: Sumatera Selatan 330
9:09:54 PM: Jawa Tengah 370
9:09:55 PM: Sumatera Utara 1185
9:09:55 PM: $ next build
9:09:58 PM: info  - Using webpack 5. Reason: Enabled by default

https://gist.github.com/FortressValkyrie/209da5c86d48e18f0bf825d01803a289

togetherwithasteria · 2021-10-16T10:05:08Z

@zainfathoni

Is e94dbfb good yet? I didn't expect this PR to take so much time for this simple fetcher.

togetherwithasteria · 2021-10-24T07:04:30Z

@zainfathoni Hey, I have fixed the bug already. Can you review it?
The fix is in e94dbfb.

zainfathoni · 2021-10-25T02:51:16Z

Sorry for the slow response. Okay, I'll try to make some time later today to review it. Thanks! 🙏

togetherwithasteria · 2021-10-25T13:25:37Z

Sorry for the slow response. Okay, I'll try to make some time later today to review it. Thanks! pray

Okay. Waiting for it.

zainfathoni

LGTM 💯

Thanks for tirelessly working on this issue. Sorry for my slow response, I was pretty occupied in the past few weeks. 😅 🙏

zainfathoni · 2021-10-26T13:34:41Z

@all-contributors please add @FortressValkyrie for code, test.

allcontributors · 2021-10-26T13:34:49Z

@zainfathoni

I've put up a pull request to add @FortressValkyrie! 🎉

togetherwithasteria · 2021-10-27T04:57:52Z

LGTM 100

Thanks for tirelessly working on this issue. Sorry for my slow response, I was pretty occupied in the past few weeks. sweat_smile pray

Your welcome. Haha, that's fine.

mazipan · 2021-10-27T13:08:38Z

Can you contact me on the Twitter @FortressValkyrie ?

feat(fetcher): implement vaksinasi.id db fetcher

c84e78c

Fetches vaksinasi.id's DB and store it in wbw-vaccination-database.json Refs: #803

style(fetcher): sort union type members in fetch-vaccine-database.ts

5984d61

There is an ESLint warning in c84e78c, this commit addresses that warn

togetherwithasteria marked this pull request as ready for review October 9, 2021 08:37

zainfathoni requested review from adityapurwa, resir014 and mazipan October 9, 2021 12:10

zainfathoni requested changes Oct 9, 2021

View reviewed changes

etc/fetchers/fetch-vaccination-database.ts Show resolved Hide resolved

etc/fetchers/fetch-vaccination-database.ts Outdated Show resolved Hide resolved

etc/fetchers/fetch-vaccination-database.ts Outdated Show resolved Hide resolved

refactor(fetcher): move types of vaksinid fetcher to lib

ebbbaa0

mazipan reviewed Oct 9, 2021

View reviewed changes

etc/fetchers/fetch-vaccination-database.ts Outdated Show resolved Hide resolved

refactor(fetcher): move the vaksin.id url to the top

03c7baf

mazipan approved these changes Oct 10, 2021

View reviewed changes

feat(fetcher): fetch vaksinasi.id db in fetch-wbw.ts

fa2c55f

zainfathoni requested changes Oct 10, 2021

View reviewed changes

etc/fetchers/fetch-wbw.ts Outdated Show resolved Hide resolved

etc/fetchers/fetch-vaccination-database.ts Outdated Show resolved Hide resolved

togetherwithasteria added 3 commits October 10, 2021 20:22

refactor(fetcher): reduce repetition in vaccination types

d0c21b7

fix(fetcher): fix program stuck caused by spinner

8b80b67

fetchVaccinationDatabase() is so fast that a spinner stucks the program. So, move it to the top and remove line 37.

togetherwithasteria commented Oct 11, 2021

View reviewed changes

etc/fetchers/fetch-wbw.ts Show resolved Hide resolved

refactor(fetcher): change vaksin.id output to PartialVaccinationProvi…

d4d8379

…nce[] Make it easy for the DB to be merged with the other contact DBs. Using VaccinationContact[] might force us into unnecessarily iterating the array just to merge it.

togetherwithasteria commented Oct 11, 2021

View reviewed changes

togetherwithasteria requested a review from zainfathoni October 11, 2021 10:59

zainfathoni requested changes Oct 11, 2021

View reviewed changes

etc/fetchers/fetch-vaccination-database.ts Outdated Show resolved Hide resolved

zainfathoni reviewed Oct 12, 2021

View reviewed changes

togetherwithasteria added 2 commits October 14, 2021 09:50

chore: switch netlify export to yarn fetch-wbw

a7532dd

Merge branch 'main' into feat/fetch-vaccination-database

b56e628

feat(fetcher): log some info from vaccination db

6620521

togetherwithasteria added 3 commits October 15, 2021 18:12

feat(fetcher): log the output text of vaccine.db fetcher

3f43140

Merge branch 'main' of https://github.com/kawalcovid19/wargabantuwarg…

67c64d4

…a.com into feat/fetch-vaccination-database

fix(fetcher): fix vaksinasi.id fetcher only fetching 4 provinces

e94dbfb

togetherwithasteria requested a review from zainfathoni October 15, 2021 14:23

refactor(fetcher): remove console.log()s from vaccination fetcher

d5de61b

zainfathoni approved these changes Oct 26, 2021

View reviewed changes

zainfathoni merged commit 93b1f76 into kawalcovid19:main Oct 26, 2021

allcontributors bot mentioned this pull request Oct 26, 2021

docs: add FortressValkyrie as a contributor for code, test #816

Merged

zainfathoni added the hacktoberfest-accepted Accepted for Hacktoberfest participation label Oct 26, 2021

togetherwithasteria mentioned this pull request Nov 27, 2021

feat: use vaccination database data for Tempat vaksin #817

Draft

4 tasks

togetherwithasteria mentioned this pull request Dec 21, 2021

chore: revert fetching data every Netlify export #832

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(fetcher): implement vaksinasi.id db fetcher #806

feat(fetcher): implement vaksinasi.id db fetcher #806

togetherwithasteria commented Oct 9, 2021 •

edited

Loading

netlify bot commented Oct 9, 2021 •

edited

Loading

codecov bot commented Oct 9, 2021 •

edited

Loading

togetherwithasteria commented Oct 9, 2021

zainfathoni commented Oct 9, 2021

togetherwithasteria commented Oct 9, 2021

zainfathoni left a comment

mazipan left a comment

mazipan commented Oct 10, 2021

togetherwithasteria left a comment •

edited

Loading

togetherwithasteria commented Oct 11, 2021

zainfathoni commented Oct 11, 2021

zainfathoni commented Oct 11, 2021

zainfathoni left a comment

togetherwithasteria commented Oct 12, 2021 •

edited

Loading

zainfathoni left a comment

togetherwithasteria commented Oct 13, 2021

zainfathoni commented Oct 13, 2021

togetherwithasteria commented Oct 14, 2021

zainfathoni commented Oct 14, 2021

togetherwithasteria commented Oct 15, 2021 •

edited

Loading

togetherwithasteria commented Oct 15, 2021 •

edited

Loading

togetherwithasteria commented Oct 16, 2021 •

edited

Loading

togetherwithasteria commented Oct 24, 2021 •

edited

Loading

zainfathoni commented Oct 25, 2021

togetherwithasteria commented Oct 25, 2021

zainfathoni left a comment

zainfathoni commented Oct 26, 2021

allcontributors bot commented Oct 26, 2021

togetherwithasteria commented Oct 27, 2021

mazipan commented Oct 27, 2021

feat(fetcher): implement vaksinasi.id db fetcher #806

feat(fetcher): implement vaksinasi.id db fetcher #806

Conversation

togetherwithasteria commented Oct 9, 2021 • edited Loading

Description

Current Tasks

netlify bot commented Oct 9, 2021 • edited Loading

codecov bot commented Oct 9, 2021 • edited Loading

Codecov Report

togetherwithasteria commented Oct 9, 2021

zainfathoni commented Oct 9, 2021

togetherwithasteria commented Oct 9, 2021

zainfathoni left a comment

Choose a reason for hiding this comment

mazipan left a comment

Choose a reason for hiding this comment

mazipan commented Oct 10, 2021

togetherwithasteria left a comment • edited Loading

Choose a reason for hiding this comment

togetherwithasteria commented Oct 11, 2021

zainfathoni commented Oct 11, 2021

zainfathoni commented Oct 11, 2021

zainfathoni left a comment

Choose a reason for hiding this comment

togetherwithasteria commented Oct 12, 2021 • edited Loading

zainfathoni left a comment

Choose a reason for hiding this comment

togetherwithasteria commented Oct 13, 2021

zainfathoni commented Oct 13, 2021

togetherwithasteria commented Oct 14, 2021

zainfathoni commented Oct 14, 2021

togetherwithasteria commented Oct 15, 2021 • edited Loading

togetherwithasteria commented Oct 15, 2021 • edited Loading

togetherwithasteria commented Oct 16, 2021 • edited Loading

togetherwithasteria commented Oct 24, 2021 • edited Loading

zainfathoni commented Oct 25, 2021

togetherwithasteria commented Oct 25, 2021

zainfathoni left a comment

Choose a reason for hiding this comment

zainfathoni commented Oct 26, 2021

allcontributors bot commented Oct 26, 2021

togetherwithasteria commented Oct 27, 2021

mazipan commented Oct 27, 2021

togetherwithasteria commented Oct 9, 2021 •

edited

Loading

netlify bot commented Oct 9, 2021 •

edited

Loading

codecov bot commented Oct 9, 2021 •

edited

Loading

togetherwithasteria left a comment •

edited

Loading

togetherwithasteria commented Oct 12, 2021 •

edited

Loading

togetherwithasteria commented Oct 15, 2021 •

edited

Loading

togetherwithasteria commented Oct 15, 2021 •

edited

Loading

togetherwithasteria commented Oct 16, 2021 •

edited

Loading

togetherwithasteria commented Oct 24, 2021 •

edited

Loading