![](\"https://www.cvh.ac.cn/public/uploaded/images/20240124vRP3KLaz.png\")
"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "b25d9f42-2dc2-45f2-aecd-62d989de9900",
+ "metadata": {},
+ "source": [
+ "# CVH Dataset Scraping"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "ebaba42b-31a5-4871-9b93-e1bf76290c82",
+ "metadata": {},
+ "source": [
+ "## Why Scrapping?\n",
+ "\n",
+ "As discussed, the CVH Dataset does not offer API access and thus a scrapper is needed. We will show the PoC of Scrapping when needed. "
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "babe7180-f6dc-4c6c-89cb-e27a9651e66d",
+ "metadata": {},
+ "source": [
+ "## Use of Selenium\n",
+ "The CVH ddatabase has dynamic element (JS), thus need the use of selenium.\n",
+ "The SCC is a virutal machine and lacks the normal browser support, so we could\n",
+ "* Use SCC's virtual machine or local machines build pipeline from there\n",
+ "* Figure out another way."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 15,
+ "id": "1f8cf5fa-9408-452e-8da4-cb9973814495",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "!pip install selenium webdriver_manager -q"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "920bb3cb-66b4-4b0e-a5b1-b7f068977ea8",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "from selenium import webdriver\n",
+ "from selenium.webdriver.chrome.service import Service\n",
+ "from selenium.webdriver.common.by import By\n",
+ "from webdriver_manager.chrome import ChromeDriverManager\n",
+ "import pandas as pd\n",
+ "import time\n",
+ "\n",
+ "# Set up Selenium WebDriver\n",
+ "service = Service(ChromeDriverManager().install())\n",
+ "driver = webdriver.Chrome(service=service)\n",
+ "\n",
+ "# URL to scrape\n",
+ "url = \"https://www.cvh.ac.cn/spms/list.php?&offset=0\"\n",
+ "\n",
+ "# Open the URL\n",
+ "driver.get(url)\n",
+ "\n",
+ "# Wait for the dynamic content to load\n",
+ "time.sleep(5) # Adjust the sleep time according to your internet speed and website response\n",
+ "\n",
+ "# Find all elements with the specified class\n",
+ "rows = driver.find_elements(By.CLASS_NAME, 'spms-row')\n",
+ "\n",
+ "# Extract data-collection-id from each row\n",
+ "data_collection_ids = [row.get_attribute('data-collection-id') for row in rows]\n",
+ "\n",
+ "# Close the WebDriver\n",
+ "driver.quit()\n",
+ "\n",
+ "# Create a DataFrame\n",
+ "data_collection_ids = pd.DataFrame(data_collection_ids, columns=['Data Collection ID'])\n",
+ "\n",
+ "# Display the DataFrame\n",
+ "data_collection_ids"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "bc84faf8-1c50-41c7-b71c-9e84410164b6",
+ "metadata": {},
+ "source": [
+ "## Sample Dataframe"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 22,
+ "id": "7588a2ee-29b3-48ab-bd33-8e0415e0a637",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "data = {\n",
+ " \"Data Collection ID\": [\n",
+ " \"e6e73365\", \"e6e73489\", \"e6e73514\", \"e6e7359d\", \"e6e73625\",\n",
+ " \"e6e736ae\", \"e6e73738\", \"e6e737c0\", \"e6e73847\", \"e6e738ce\",\n",
+ " \"e6e7395a\", \"e6e739e8\", \"e6e73a6f\", \"e6e73af6\", \"e6e73b7e\",\n",
+ " \"e6e73c07\", \"e6e73c8f\", \"e6e73d18\", \"e6e73da0\", \"e6e73e2b\",\n",
+ " \"e6e73eb9\", \"e6e73f42\", \"e6e73fc9\", \"e6e74052\", \"e6e740da\",\n",
+ " \"e6e74162\", \"e6e741eb\", \"e6e74274\", \"e6e742fc\", \"e6e74389\"\n",
+ " ]\n",
+ "}\n",
+ "\n",
+ "# Convert the dictionary to a DataFrame\n",
+ "data_collection_ids = pd.DataFrame(data)"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 18,
+ "id": "7a4ca53c-66da-499d-9c92-457ba72dd005",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "from selenium import webdriver\n",
+ "from selenium.webdriver.chrome.options import Options\n",
+ "from selenium.webdriver.common.by import By\n",
+ "from selenium.webdriver.chrome.service import Service\n",
+ "from selenium.webdriver.support.ui import WebDriverWait\n",
+ "from selenium.webdriver.support import expected_conditions as EC\n",
+ "from selenium.common.exceptions import TimeoutException\n",
+ "from webdriver_manager.chrome import ChromeDriverManager\n",
+ "from concurrent.futures import ThreadPoolExecutor, as_completed\n",
+ "from tqdm.auto import tqdm\n",
+ "import pandas as pd\n",
+ "import time\n",
+ "\n",
+ "def setup_driver(headless=True):\n",
+ " chrome_options = Options()\n",
+ " if headless:\n",
+ " chrome_options.add_argument(\"--headless\")\n",
+ " service = Service(ChromeDriverManager().install())\n",
+ " driver = webdriver.Chrome(service=service, options=chrome_options)\n",
+ " return driver\n",
+ "\n",
+ "def wait_for_element(driver, by_method, value, timeout=10, retry_interval=5, max_retries=3):\n",
+ " retries = 0\n",
+ " while retries < max_retries:\n",
+ " try:\n",
+ " return WebDriverWait(driver, timeout).until(EC.presence_of_element_located((by_method, value)))\n",
+ " except TimeoutException:\n",
+ " print(f\"Retry {retries+1}/{max_retries} for element {value} after timeout.\")\n",
+ " time.sleep(retry_interval)\n",
+ " retries += 1\n",
+ " raise TimeoutException(f\"Element {value} not found after {max_retries} retries.\")\n",
+ "\n",
+ "def fetch_data(collection_id):\n",
+ " driver = setup_driver()\n",
+ " try:\n",
+ " driver.get(f\"https://www.cvh.ac.cn/spms/detail.php?id={collection_id}\")\n",
+ " \n",
+ " # Fetch specific details as per your logic\n",
+ " data = {\n",
+ " 'Collection ID': collection_id,\n",
+ " 'Image Link': wait_for_element(driver, By.ID, \"spm_image\", max_retries=5).get_attribute('src'),\n",
+ " 'Phylum (门)': wait_for_element(driver, By.ID, \"taxon_phy_c\", max_retries=5).text,\n",
+ " 'Order (目)': wait_for_element(driver, By.ID, \"taxon_ord_c\", max_retries=5).text,\n",
+ " 'Family (科)': wait_for_element(driver, By.ID, \"taxon_fam_c\", max_retries=5).text,\n",
+ " 'Genus (属)': wait_for_element(driver, By.ID, \"taxon_gen_c\", max_retries=5).text,\n",
+ " 'Scientific Name': wait_for_element(driver, By.ID, \"formattedName\", max_retries=5).text,\n",
+ " 'Chinese Name': wait_for_element(driver, By.ID, \"chineseName\", max_retries=5).text,\n",
+ " 'Identified By': wait_for_element(driver, By.ID, \"identifiedBy\", max_retries=5).text,\n",
+ " 'Date Identified': wait_for_element(driver, By.ID, \"dateIdentified\", max_retries=5).text,\n",
+ " 'Collector': wait_for_element(driver, By.ID, \"recordedBy\", max_retries=5).text,\n",
+ " 'Collection Number': wait_for_element(driver, By.ID, \"recordNumber\", max_retries=5).text,\n",
+ " 'Collection Date': wait_for_element(driver, By.ID, \"verbatimEventDate\", max_retries=5).text,\n",
+ " 'Collection Location': wait_for_element(driver, By.ID, \"locality\", max_retries=5).text,\n",
+ " 'Altitude': wait_for_element(driver, By.ID, \"elevation\", max_retries=5).text,\n",
+ " 'Habitat': wait_for_element(driver, By.ID, \"habitat\", max_retries=5).text,\n",
+ " 'Phenology': wait_for_element(driver, By.ID, \"reproductiveCondition\", max_retries=5).text\n",
+ " }\n",
+ " return data\n",
+ " finally:\n",
+ " driver.quit()\n",
+ "\n",
+ "def sample_run(collection_ids, max_workers=5):\n",
+ " results = []\n",
+ " with ThreadPoolExecutor(max_workers=max_workers) as executor:\n",
+ " futures = [executor.submit(fetch_data, cid) for cid in collection_ids]\n",
+ " for future in tqdm(as_completed(futures), total=len(collection_ids)):\n",
+ " results.append(future.result())\n",
+ "\n",
+ " df = pd.DataFrame(results)\n",
+ " return df"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "bcd8b481-4f12-4b4a-a572-3122d6129ac7",
+ "metadata": {},
+ "source": [
+ "## Sample Scrapper Run with first 5 Collection_ids"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 23,
+ "id": "8baa4d68-a18b-4e70-9f52-81ecdd8a3748",
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "/projectnb/sparkgrp/ml-herbarium-grp/spring2024\n"
+ ]
+ }
+ ],
+ "source": [
+ "!pwd"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 21,
+ "id": "e37cc10d-38ee-4cc2-b6cb-119fd88892f6",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "pd.set_option('display.max_columns', None)\n",
+ "\n",
+ "sample_collections = list(data_collection_ids['Data Collection ID'].head(5))\n",
+ "resutls = sample_run(sample_collections, 5) \n",
+ "results"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "b6e14bb4-4d3a-41c5-bab9-7414d11a1c36",
+ "metadata": {},
+ "source": [
+ "### Scrapper Results\n",
+ "\n",
+ "Here is the screenshot from my local environment:\n",
+ "\n",
+ "![](\"cvh_images/cvh_scrapper_sample_run.jpg\")
\n",
+ "\n",
+ "\n",
+ "### Scrapped Info (In red)\n",
+ "* Image link\n",
+ "* Taxon data on right\n",
+ "\n",
+ "![](\"cvh_images/cvh_e6e73365_screenshot.png\")
"
+ ]
+ }
+ ],
+ "metadata": {
+ "kernelspec": {
+ "display_name": "Python 3 (ipykernel)",
+ "language": "python",
+ "name": "python3"
+ },
+ "language_info": {
+ "codemirror_mode": {
+ "name": "ipython",
+ "version": 3
+ },
+ "file_extension": ".py",
+ "mimetype": "text/x-python",
+ "name": "python",
+ "nbconvert_exporter": "python",
+ "pygments_lexer": "ipython3",
+ "version": "3.11.6"
+ }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}
diff --git a/Spring2024/cvh_images/cvh_3ba3608e_screenshot.png b/Spring2024/cvh_images/cvh_3ba3608e_screenshot.png
new file mode 100644
index 0000000..1bad866
Binary files /dev/null and b/Spring2024/cvh_images/cvh_3ba3608e_screenshot.png differ
diff --git a/Spring2024/cvh_images/cvh_e6e73365_screenshot.png b/Spring2024/cvh_images/cvh_e6e73365_screenshot.png
new file mode 100644
index 0000000..a964b1d
Binary files /dev/null and b/Spring2024/cvh_images/cvh_e6e73365_screenshot.png differ
diff --git a/Spring2024/cvh_images/cvh_sample_geo_distribution.png b/Spring2024/cvh_images/cvh_sample_geo_distribution.png
new file mode 100644
index 0000000..db91bf8
Binary files /dev/null and b/Spring2024/cvh_images/cvh_sample_geo_distribution.png differ
diff --git a/Spring2024/cvh_images/cvh_scrapper_sample_run.jpg b/Spring2024/cvh_images/cvh_scrapper_sample_run.jpg
new file mode 100644
index 0000000..2bf136f
Binary files /dev/null and b/Spring2024/cvh_images/cvh_scrapper_sample_run.jpg differ
diff --git a/Spring2024/cvh_images/image1.jpeg b/Spring2024/cvh_images/image1.jpeg
new file mode 100644
index 0000000..b974b87
Binary files /dev/null and b/Spring2024/cvh_images/image1.jpeg differ
diff --git a/Spring2024/cvh_images/image2.png b/Spring2024/cvh_images/image2.png
new file mode 100644
index 0000000..39f43f1
Binary files /dev/null and b/Spring2024/cvh_images/image2.png differ
diff --git a/Spring2024/cvh_images/image3.png b/Spring2024/cvh_images/image3.png
new file mode 100644
index 0000000..f386710
Binary files /dev/null and b/Spring2024/cvh_images/image3.png differ
diff --git a/Spring2024/cvh_images/image4.png b/Spring2024/cvh_images/image4.png
new file mode 100644
index 0000000..657f600
Binary files /dev/null and b/Spring2024/cvh_images/image4.png differ