Data found under https://github.com/Santosh-Gupta/datasets is not 700,000 question/answer pairs

[EphChem]

Hello!
There are 2 links under https://github.com/Santosh-Gupta/datasets:

1 - https://drive.google.com/drive/folders/1PymmjbrgfOIs-HJ7oBmjZKH8j4rYsGZj
2 - https://drive.google.com/drive/folders/1kYD57uStDd4kXyb3JOYCTQd92Al6Il4K

However, there are duplicates in both links; for example each of the AskDocs.csv and icliniqQAs.csv is found in both links. Therefore, when I import all the non-duplicate data, I only see about 200,000 QA pairs, not the 700,000 which your repo mentions. Is the rest of the data somewhere else. Please kindly let me know how to import the entire 700,000 QA Dataset.

Thank you!

[Santosh-Gupta]

How much data points are in the askdocs , healthtap , and webmd csvs? Each of those should contain several hundred thousand data points each. @ash3n @JayYip Are there updated versions of those files?

[Santosh-Gupta]

In the meanwhile, here's a link to the binary tfrecords files

https://drive.google.com/drive/u/8/folders/1wRc1jtl5Q0objpfualNFwpg4H575tmks

[EphChem]

Number of rows in healthtap, webmd, AskDocs is about 137000, 46200, 78000, respectively. I do not see a few hundred thousand data points per dataset as you mention.
Thanks for the tfrecords but not sure how helpful this will be for my work.

[Santosh-Gupta]

@ash3n @JayYip I'm going through the chats regarding the data. Unless I missed something, I believe that is the total amount of data we have, ~300,000 question answer pairs, not 700,000. Health tap, askdocs, and webmd are the most significant datasets we have. All the other datasets have very low numbers compared to those three.

[JayYip]

I'm not very familiar with the size of data since I'm not the one who collected and parsed them.

[Santosh-Gupta]

I'll wait until @ash3n comments. If my analysis is correct, we should update the Read.me

[Better-Boy]

#15 (comment) - link expired
https://drive.google.com/drive/folders/1PymmjbrgfOIs-HJ7oBmjZKH8j4rYsGZj - link expired

Please provide working links

[Santosh-Gupta]

@JayYip that's your folder

[JayYip]

Link restored. Please check.

[Better-Boy]

Still this link is not working - https://drive.google.com/drive/u/8/folders/1wRc1jtl5Q0objpfualNFwpg4H575tmks

[Santosh-Gupta]

also @JayYip

[JayYip]

@Better-Boy restored. I'm running out of google drive space. @Santosh-Gupta @Better-Boy I wonder if you guys can help to store the data and start a PR to update links in README? Thanks.

[Santosh-Gupta]

Sure, lets chat over the slack