Automated Web Scraper For Amazon.com.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "64d40855-a4a6-4d5d-894e-95c5c92818ab",
   "metadata": {},
   "source": [
    "# **Automated Web Scraper For Amazon.com**"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6b9da544-bd78-4b40-aedc-8965a2b14866",
   "metadata": {},
   "source": [
    "In this tutorial we will build an **Automated Web Scraper** to extract data from **amazon.com** that we can use for any data analysis,data science or machine learning project.\n",
    "\n",
    "Before we get started let me make it clear that Amazon has a tight security for their platform and some of the things you can easily do on other webpages wont work on Amozon platform.\n",
    "\n",
    "Previously, we could have used **Beautiful Soup and Request** to easily get titles from the page, but things have changed little bit. We will still use **Beautiful Soup** but in a different way.\n",
    "\n",
    "Let's see how we can do things differently now."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d59e1f2d-a471-401e-96a4-1135bbcdc11e",
   "metadata": {},
   "source": [
    "# Installation"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9ecd8549-c889-427b-a2f0-68f90bf935a3",
   "metadata": {},
   "source": [
    "We will be using:\n",
    "* **Selenium**\n",
    "* **BeautifulSoup**\n",
    "* **Webdrivers**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "id": "ae4d28fe-8bf1-4cc1-9683-fb3192972833",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: selenium in c:\\users\\hp\\anaconda3\\lib\\site-packages (3.141.0)\n",
      "Requirement already satisfied: urllib3 in c:\\users\\hp\\anaconda3\\lib\\site-packages (from selenium) (1.26.4)\n",
      "Requirement already satisfied: msedge-selenium-tools in c:\\users\\hp\\anaconda3\\lib\\site-packages (3.141.4)\n",
      "Requirement already satisfied: selenium==3.141 in c:\\users\\hp\\anaconda3\\lib\\site-packages (from msedge-selenium-tools) (3.141.0)\n",
      "Requirement already satisfied: urllib3 in c:\\users\\hp\\anaconda3\\lib\\site-packages (from selenium==3.141->msedge-selenium-tools) (1.26.4)\n",
      "Requirement already satisfied: bs4 in c:\\users\\hp\\anaconda3\\lib\\site-packages (0.0.1)\n",
      "Requirement already satisfied: beautifulsoup4 in c:\\users\\hp\\anaconda3\\lib\\site-packages (from bs4) (4.9.3)\n",
      "Requirement already satisfied: soupsieve>1.2 in c:\\users\\hp\\anaconda3\\lib\\site-packages (from beautifulsoup4->bs4) (2.2.1)\n"
     ]
    }
   ],
   "source": [
    "!pip install selenium\n",
    "!pip install msedge-selenium-tools\n",
    "!pip install bs4"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "05ea8baa-1c6a-47d9-8adc-77990eeccca2",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Collecting chromedriver_binary==112.0.5615.49.0\n",
      "  Using cached chromedriver-binary-112.0.5615.49.0.tar.gz (5.1 kB)\n",
      "Building wheels for collected packages: chromedriver-binary\n",
      "  Building wheel for chromedriver-binary (setup.py): started\n",
      "  Building wheel for chromedriver-binary (setup.py): finished with status 'done'\n",
      "  Created wheel for chromedriver-binary: filename=chromedriver_binary-112.0.5615.49.0-py3-none-any.whl size=7091665 sha256=95a00ef60fa8807db11d27b0cd744f9e328f6b31c6aa09d6bcbe24c23a8014ca\n",
      "  Stored in directory: c:\\users\\hp\\appdata\\local\\pip\\cache\\wheels\\46\\59\\c2\\be916df54da4ba517a74b756d7bc11d1018fa56f86c00daf10\n",
      "Successfully built chromedriver-binary\n",
      "Installing collected packages: chromedriver-binary\n",
      "Successfully installed chromedriver-binary-112.0.5615.49.0\n"
     ]
    }
   ],
   "source": [
    "!pip install chromedriver_binary==112.0.5615.49.0"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "947281ba-9230-4d8c-b241-2079dcb0da85",
   "metadata": {},
   "source": [
    "**Import the necessary libraries**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "b89554cf-3139-4468-8e1e-2585228ca775",
   "metadata": {},
   "outputs": [],
   "source": [
    "from selenium import webdriver\n",
    "import chromedriver_binary\n",
    "\n",
    "#for microsoft edge\n",
    "from msedge.selenium_tools import Edge, EdgeOptions\n",
    "import csv"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d06c3a54-2bc6-42ef-ae44-df413c521263",
   "metadata": {},
   "source": [
    "**Setup the web driver**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "7899f94c-e992-42e3-96c0-611713e6a83e",
   "metadata": {},
   "outputs": [],
   "source": [
    "driver = webdriver.Chrome()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "6aacdb06-1f43-48b5-8724-a3f7f7ff5761",
   "metadata": {},
   "outputs": [],
   "source": [
    "url= 'https://www.amazon.com/'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "bdde3b9b-5ba7-4592-ae77-c96c5a238bbb",
   "metadata": {},
   "outputs": [],
   "source": [
    "driver.get(url)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "f498beed-0df3-4c66-b3f5-4b0e33be4c88",
   "metadata": {},
   "outputs": [],
   "source": [
    "def my_url(keyword):\n",
    "    \n",
    "#     temp = 'https://www.amazon.com/s?k=phone+case&ref=nb_sb_noss_1' #let's get rid of the 'phone+case' and replace it with {} to make the url generic.\n",
    "      temp = 'https://www.amazon.com/s?k={}&ref=nb_sb_noss_1' # a template url\n",
    "      keyword = keyword.replace(' ', '+')\n",
    "      return temp.format(keyword)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "784cc42a-3453-43f1-b09d-964f7fe5e655",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'https://www.amazon.com/s?k=laptop&ref=nb_sb_noss_1'"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "url=my_url('laptop') \n",
    "url"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "2ba08fa0-398b-4faf-81cd-eebbb6d8e2e0",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'https://www.amazon.com/s?k=laptop+charger&ref=nb_sb_noss_1'"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "url=my_url('laptop charger')\n",
    "url"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "36e3fafc-5967-4ee9-814c-48f9124e9fe5",
   "metadata": {},
   "outputs": [],
   "source": [
    "driver.get(url) #this will open in your browser and return the page for your keyword"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "6b3036ae-581c-4d3b-8b4f-192eb9e1b46a",
   "metadata": {},
   "outputs": [],
   "source": [
    "from bs4 import BeautifulSoup\n",
    "\n",
    "soup = BeautifulSoup(driver.page_source,'html.parser')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "56fbec27-37f7-4d9c-bd38-1ed5c0c577fe",
   "metadata": {},
   "outputs": [],
   "source": [
    "soup_results=soup.find_all('div',{'data-component-type':'s-search-result'})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "b1388417-72af-4085-aaac-f5946c560fd8",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "22"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(soup_results)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "45277c8f-39df-4dc5-aae7-26ca8ac80328",
   "metadata": {},
   "outputs": [],
   "source": [
    "obj=soup_results[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "8cfd99ea-42cb-417a-9fe9-151d1011f94d",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<div class=\"sg-col-20-of-24 s-result-item s-asin sg-col-0-of-12 sg-col-16-of-20 AdHolder sg-col s-widget-spacing-small sg-col-12-of-16\" data-asin=\"B093V1XBJS\" data-cel-widget=\"search_result_1\" data-component-id=\"12\" data-component-type=\"s-search-result\" data-index=\"1\" data-uuid=\"12352664-edf5-4687-9f3f-543b2419ba87\"><div class=\"sg-col-inner\"><div cel_widget_id=\"MAIN-SEARCH_RESULTS-1\" class=\"s-widget-container s-spacing-small s-widget-container-height-small celwidget slot=MAIN template=SEARCH_RESULTS widgetId=search-results_1\" data-cel-widget=\"MAIN-SEARCH_RESULTS-1\" data-csa-c-id=\"y40s0q-f4yc4u-w1y0zl-i6eeq2\" data-csa-c-item-id=\"amzn1.asin.1.B093V1XBJS\" data-csa-c-pos=\"1\" data-csa-c-type=\"item\" data-csa-op-log-render=\"\">\n",
       "<div class=\"rush-component\" data-component-id=\"13\" data-component-props='{\"percentageShownToFire\":\"50\",\"batchable\":true,\"requiredElementSelector\":\".s-image:visible\",\"url\":\"https://unagi-na.amazon.com/1/events/com.amazon.eel.SponsoredProductsEventTracking.prod?qualifier=1681569216&amp;id=21906916870758&amp;widgetName=sp_atf&amp;adId=200152267693298&amp;eventType=1&amp;adIndex=0\"}' data-component-type=\"s-impression-logger\">\n",
       "<div class=\"rush-component s-featured-result-item\" data-component-id=\"2\" data-component-props='{\"presenceCounterName\":\"sp_delivered\",\"testElementSelector\":\".s-image\",\"hiddenCounterName\":\"sp_hidden\"}' data-component-type=\"s-impression-counter\">\n",
       "<div class=\"s-card-container s-overflow-hidden aok-relative puis-wide-grid-style puis-wide-grid-style-t2 puis-include-content-margin puis s-latency-cf-section s-card-border\"><div class=\"a-section\"><div class=\"sg-row\"><div class=\"sg-col sg-col-4-of-12 sg-col-4-of-16 sg-col-4-of-20 sg-col-4-of-24 s-list-col-left\"><div class=\"sg-col-inner\"><div class=\"a-section a-spacing-none aok-relative puis-status-badge-container s-list-status-badge-container\"></div><div class=\"s-product-image-container aok-relative s-text-center s-image-overlay-grey s-padding-left-small s-padding-right-small s-flex-expand-height\"><div class=\"aok-relative\"><span class=\"rush-component\" data-component-id=\"14\" data-component-type=\"s-product-image\"><a class=\"a-link-normal s-no-outline\" href=\"/gp/slredirect/picassoRedirect.html/ref=pa_sp_atf_aps_sr_pg1_1?ie=UTF8&amp;adId=A04772791KWX0QN54XAOW&amp;qualifier=1681569216&amp;id=21906916870758&amp;widgetName=sp_atf&amp;url=%2FAdapter-Charger-Chromebook-IdeaPad-ThinkPad%2Fdp%2FB093V1XBJS%2Fref%3Dsr_1_1_sspa%3Fkeywords%3Dlaptop%2Bcharger%26qid%3D1681569216%26sr%3D8-1-spons%26psc%3D1%26smid%3DA6R8HPJED4DRN\"><div class=\"a-section aok-relative s-image-fixed-height\"><img alt=\"Sponsored Ad - 65W 45W USB C Laptop Power Replacement Adapter Charger for Lenovo Chromebook/Yoga/ThinkPad L580 L590 E580 E...\" class=\"s-image\" data-image-index=\"1\" data-image-latency=\"s-product-image\" data-image-load=\"\" data-image-source-density=\"1\" src=\"https://m.media-amazon.com/images/I/71dPHTRusTS._AC_UY218_.jpg\" srcset=\"https://m.media-amazon.com/images/I/71dPHTRusTS._AC_UY218_.jpg 1x, https://m.media-amazon.com/images/I/71dPHTRusTS._AC_UY327_FMwebp_QL65_.jpg 1.5x, https://m.media-amazon.com/images/I/71dPHTRusTS._AC_UY436_FMwebp_QL65_.jpg 2x, https://m.media-amazon.com/images/I/71dPHTRusTS._AC_UY545_FMwebp_QL65_.jpg 2.5x, https://m.media-amazon.com/images/I/71dPHTRusTS._AC_UY654_FMwebp_QL65_.jpg 3x\"/></div></a></span></div></div></div></div><div class=\"sg-col sg-col-4-of-12 sg-col-8-of-16 sg-col-12-of-20 sg-col-12-of-24 s-list-col-right\"><div class=\"sg-col-inner\"><div class=\"a-section a-spacing-small a-spacing-top-small\"><div class=\"a-section a-spacing-none puis-padding-right-small s-title-instructions-style\"><div class=\"a-row a-spacing-micro\"><span class=\"a-declarative\" data-a-popover='{\"closeButton\":\"true\",\"dataStrategy\":\"preload\",\"name\":\"sp-info-popover-B093V1XBJS\",\"position\":\"triggerVertical\"}' data-action=\"a-popover\" data-csa-c-func-deps=\"aui-da-a-popover\" data-csa-c-id=\"eelui-hr10yz-4jfj50-9eyj8l\" data-csa-c-type=\"widget\"><a aria-label=\"View Sponsored information or leave ad feedback\" class=\"puis-label-popover puis-sponsored-label-text\" href=\"javascript:void(0)\" role=\"button\" style=\"text-decoration: none;\"><span class=\"puis-label-popover-default\"><span class=\"a-color-secondary\">Sponsored</span></span><span class=\"puis-label-popover-hover\"><span class=\"a-color-base\">Sponsored</span></span> <span class=\"aok-inline-block puis-sponsored-label-info-icon\"></span></a></span><div class=\"a-popover-preload\" id=\"a-popover-sp-info-popover-B093V1XBJS\"><div class=\"puis\"><span>You’re seeing this ad based on the product’s relevance to your search query.</span><div class=\"a-row\"><span class=\"a-declarative\" data-action=\"s-safe-ajax-modal-trigger\" data-csa-c-func-deps=\"aui-da-s-safe-ajax-modal-trigger\" data-csa-c-id=\"otvro3-f7vycc-37xr93-nvp68o\" data-csa-c-type=\"widget\" data-s-safe-ajax-modal-trigger='{\"ajaxUrl\":\"/af/sp-loom/feedback-form?pl=%7B%22adPlacementMetaData%22%3A%7B%22searchTerms%22%3A%22bGFwdG9wIGNoYXJnZXI%3D%22%2C%22pageType%22%3A%22Search%22%2C%22feedbackType%22%3A%22sponsoredProductsLoom%22%2C%22slotName%22%3A%22TOP%22%7D%2C%22adCreativeMetaData%22%3A%7B%22adProgramId%22%3A1024%2C%22adCreativeDetails%22%3A%5B%7B%22asin%22%3A%22B093V1XBJS%22%2C%22title%22%3A%2265W+45W+USB+C+Laptop+Power+Replacement+Adapter+Charger+for+Lenovo+Chromebook%2FYoga%2FThinkPad+L580+L590%22%2C%22priceInfo%22%3A%7B%22amount%22%3A17.99%2C%22currencyCode%22%3A%22USD%22%7D%2C%22sku%22%3A%22JT-DOH010-1-FBA%22%2C%22adId%22%3A%22A04772791KWX0QN54XAOW%22%2C%22campaignId%22%3A%22A00109221KHIZ28VX9DKB%22%2C%22advertiserIdNS%22%3Anull%2C%22selectionSignals%22%3Anull%7D%5D%7D%7D\",\"dataStrategy\":\"ajax\",\"header\":\"Share your feedback\"}'><a class=\"a-link-normal s-underline-text s-underline-link-text s-link-style\" href=\"#\"><span>Leave ad feedback</span> </a> </span></div></div></div></div><h2 class=\"a-size-mini a-spacing-none a-color-base s-line-clamp-2\"><a class=\"a-link-normal s-underline-text s-underline-link-text s-link-style a-text-normal\" href=\"/gp/slredirect/picassoRedirect.html/ref=pa_sp_atf_aps_sr_pg1_1?ie=UTF8&amp;adId=A04772791KWX0QN54XAOW&amp;qualifier=1681569216&amp;id=21906916870758&amp;widgetName=sp_atf&amp;url=%2FAdapter-Charger-Chromebook-IdeaPad-ThinkPad%2Fdp%2FB093V1XBJS%2Fref%3Dsr_1_1_sspa%3Fkeywords%3Dlaptop%2Bcharger%26qid%3D1681569216%26sr%3D8-1-spons%26psc%3D1%26smid%3DA6R8HPJED4DRN\"><span class=\"a-size-medium a-color-base a-text-normal\">65W 45W USB C Laptop Power Replacement Adapter Charger for Lenovo Chromebook/Yoga/ThinkPad L580 L590 E580 E585 P43s P53s with Power Cord</span> </a> </h2></div><div class=\"a-section a-spacing-none a-spacing-top-micro\"><div class=\"a-row a-size-small\"><span aria-label=\"4.3 out of 5 stars\"><span class=\"a-declarative\" data-a-popover='{\"closeButton\":false,\"closeButtonLabel\":\"\",\"position\":\"triggerBottom\",\"popoverLabel\":\"\",\"url\":\"/review/widgets/average-customer-review/popover/ref=acr_search__popover?ie=UTF8&amp;asin=B093V1XBJS&amp;ref=acr_search__popover&amp;contextId=search\"}' data-action=\"a-popover\" data-csa-c-func-deps=\"aui-da-a-popover\" data-csa-c-id=\"gvpanx-caqxrm-kfbu7l-d69zmd\" data-csa-c-type=\"widget\"><a class=\"a-popover-trigger a-declarative\" href=\"javascript:void(0)\" role=\"button\"><i class=\"a-icon a-icon-star-small a-star-small-4-5 aok-align-bottom\"><span class=\"a-icon-alt\">4.3 out of 5 stars</span></i><i class=\"a-icon a-icon-popover\"></i></a></span> </span><span aria-label=\"1,129\"><a class=\"a-link-normal s-underline-text s-underline-link-text s-link-style\" href=\"/gp/slredirect/picassoRedirect.html/ref=pa_sp_atf_aps_sr_pg1_1?ie=UTF8&amp;adId=A04772791KWX0QN54XAOW&amp;qualifier=1681569216&amp;id=21906916870758&amp;widgetName=sp_atf&amp;url=%2FAdapter-Charger-Chromebook-IdeaPad-ThinkPad%2Fdp%2FB093V1XBJS%2Fref%3Dsr_1_1_sspa%3Fkeywords%3Dlaptop%2Bcharger%26qid%3D1681569216%26sr%3D8-1-spons%26psc%3D1%26smid%3DA6R8HPJED4DRN#customerReviews\"><span class=\"a-size-base s-underline-text\">1,129</span> </a> </span></div></div><div class=\"sg-row\"><div class=\"sg-col sg-col-4-of-12 sg-col-4-of-16 sg-col-4-of-20 sg-col-4-of-24\"><div class=\"sg-col-inner\"><div class=\"a-section a-spacing-none a-spacing-top-micro s-price-instructions-style\"><div class=\"a-row a-size-base a-color-base\"><a class=\"a-size-base a-link-normal s-underline-text s-underline-link-text s-link-style a-text-normal\" href=\"/gp/slredirect/picassoRedirect.html/ref=pa_sp_atf_aps_sr_pg1_1?ie=UTF8&amp;adId=A04772791KWX0QN54XAOW&amp;qualifier=1681569216&amp;id=21906916870758&amp;widgetName=sp_atf&amp;url=%2FAdapter-Charger-Chromebook-IdeaPad-ThinkPad%2Fdp%2FB093V1XBJS%2Fref%3Dsr_1_1_sspa%3Fkeywords%3Dlaptop%2Bcharger%26qid%3D1681569216%26sr%3D8-1-spons%26psc%3D1%26smid%3DA6R8HPJED4DRN\"><span class=\"a-price\" data-a-color=\"base\" data-a-size=\"xl\"><span class=\"a-offscreen\">$17.99</span><span aria-hidden=\"true\"><span class=\"a-price-symbol\">$</span><span class=\"a-price-whole\">17<span class=\"a-price-decimal\">.</span></span><span class=\"a-price-fraction\">99</span></span></span> </a> </div><div class=\"a-row a-size-small a-color-secondary\"><span class=\"rush-component\" data-component-id=\"15\" data-component-props='{\"asin\":\"B093V1XBJS\"}' data-component-type=\"s-coupon-component\"><span class=\"s-coupon-clipped aok-hidden\"><span class=\"a-color-base\">5% coupon applied at checkout</span></span><span class=\"s-coupon-unclipped\"><span class=\"a-size-base s-highlighted-text-padding aok-inline-block s-coupon-highlight-color\">Save 5%</span> <span class=\"a-color-base\"> with coupon</span></span></span> </div></div><div class=\"a-section a-spacing-none a-spacing-top-micro\"><div class=\"a-row a-size-base a-color-secondary s-align-children-center\"><span class=\"a-size-small a-color-base\">Ships to India</span></div></div></div></div><div class=\"sg-col sg-col-4-of-12 sg-col-4-of-16 sg-col-8-of-20 sg-col-8-of-24\"><div class=\"sg-col-inner\"></div></div></div></div></div></div></div></div></div>\n",
       "</div>\n",
       "</div>\n",
       "</div></div></div>"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "obj"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "id": "57acefdc-8c9d-4503-a9ac-fecfdb899701",
   "metadata": {},
   "outputs": [],
   "source": [
    "atag = obj.h2.a #create the h2 tag variable"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "8ce34033-6908-40d5-b9f7-341708b0abb8",
   "metadata": {},
   "outputs": [],
   "source": [
    "des = atag.text.strip()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "a8bd64ad-12a6-45cd-89c4-c9923e9b1cb7",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'65W 45W USB C Laptop Power Replacement Adapter Charger for Lenovo Chromebook/Yoga/ThinkPad L580 L590 E580 E585 P43s P53s with Power Cord'"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "des #we can see below that we have the title correctly scraped"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "618d53b8-f55b-4e0a-a58e-0eebcf6eb4fc",
   "metadata": {},
   "outputs": [],
   "source": [
    "#let's now create a generic url\n",
    "\n",
    "url='https://www.amazon.com/'+atag.get('href')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "63786c32-3776-4c65-aa91-54de460e49b2",
   "metadata": {},
   "source": [
    "# **Get the Price**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "id": "25a4fe88-faba-45ef-9cda-5f5a53f643a2",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'$17.99'"
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#let's get the price same way we searched for the title by looking for the div tag, in this case, we will look for the tag that contains the price of the item.\n",
    "\n",
    "#we will get this from the 'span' which contains the a-price and then use the 'span' which contains 'a-offscreen' to obtain the actual price.\n",
    "\n",
    "parent=obj.find('span','a-price')\n",
    "\n",
    "price=parent.find('span','a-offscreen').text\n",
    "\n",
    "price"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5410eb62-f071-46f9-9284-83410f050323",
   "metadata": {},
   "source": [
    "# **Get the Reviews**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "id": "3c7fe910-00e0-4998-bea3-c3e6e83ffc33",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'4.3 out of 5 stars'"
      ]
     },
     "execution_count": 45,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#We will do the same thing for the Reviews\n",
    "\n",
    "rate=obj.find('span','a-icon-alt').text\n",
    "rate"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "id": "290cc034-e7ff-4802-9525-6818eab3ef04",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'4.3 out of 5 stars'"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "obj.i.text"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c2d0141b-2ee0-4a8e-8ce2-3322409017ae",
   "metadata": {},
   "source": [
    "# **Get the review Counts**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "id": "e16de348-4c4b-4d99-97b9-51e6993cf01f",
   "metadata": {},
   "outputs": [
    {
     "ename": "AttributeError",
     "evalue": "'NoneType' object has no attribute 'text'",
     "output_type": "error",
     "traceback": [
      "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[1;31mAttributeError\u001b[0m                            Traceback (most recent call last)",
      "\u001b[1;32m<ipython-input-46-7e6ffd01226b>\u001b[0m in \u001b[0;36m<module>\u001b[1;34m\u001b[0m\n\u001b[0;32m      1\u001b[0m \u001b[1;31m#we can get the number of customers who have reviewed the item as well\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m      2\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m----> 3\u001b[1;33m \u001b[0mcounts_review\u001b[0m\u001b[1;33m=\u001b[0m\u001b[0mobj\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mfind\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;34m'span'\u001b[0m\u001b[1;33m,\u001b[0m\u001b[1;33m{\u001b[0m\u001b[1;34m'class'\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;34m'a-size-base'\u001b[0m\u001b[1;33m,\u001b[0m\u001b[1;34m'dir'\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;34m'auto'\u001b[0m\u001b[1;33m}\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mtext\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m      4\u001b[0m \u001b[0mcounts_review\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;31mAttributeError\u001b[0m: 'NoneType' object has no attribute 'text'"
     ]
    }
   ],
   "source": [
    "#we can get the number of customers who have reviewed the item as well\n",
    "\n",
    "counts_review=obj.find('span',{'class':'a-size-base','dir':'auto'}).text\n",
    "counts_review"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "id": "4733b7b3-924a-4a13-9fce-635534d5245a",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<img alt=\"Sponsored Ad - 65W 45W USB C Laptop Power Replacement Adapter Charger for Lenovo Chromebook/Yoga/ThinkPad L580 L590 E580 E...\" class=\"s-image\" data-image-index=\"1\" data-image-latency=\"s-product-image\" data-image-load=\"\" data-image-source-density=\"1\" src=\"https://m.media-amazon.com/images/I/71dPHTRusTS._AC_UY218_.jpg\" srcset=\"https://m.media-amazon.com/images/I/71dPHTRusTS._AC_UY218_.jpg 1x, https://m.media-amazon.com/images/I/71dPHTRusTS._AC_UY327_FMwebp_QL65_.jpg 1.5x, https://m.media-amazon.com/images/I/71dPHTRusTS._AC_UY436_FMwebp_QL65_.jpg 2x, https://m.media-amazon.com/images/I/71dPHTRusTS._AC_UY545_FMwebp_QL65_.jpg 2.5x, https://m.media-amazon.com/images/I/71dPHTRusTS._AC_UY654_FMwebp_QL65_.jpg 3x\"/>"
      ]
     },
     "execution_count": 47,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "obj.img"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9187d364-5377-433c-a685-e2a7c306af64",
   "metadata": {},
   "source": [
    "# **Generic Fuction**"
   ]
  },
  {
   "cell_type": "raw",
   "id": "87876415-8142-48f2-bd94-9f6348f976e7",
   "metadata": {},
   "source": [
    "We will now create a generic fuction for the sub codes to extract all the data at once."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "id": "a7e3dbf5-17da-47ad-94f3-11753d36fa95",
   "metadata": {},
   "outputs": [],
   "source": [
    "from selenium import webdriver\n",
    "import chromedriver_binary\n",
    "from bs4 import BeautifulSoup\n",
    "#for moicrosoft edge\n",
    "from msedge.selenium_tools import Edge, EdgeOptions\n",
    "import csv\n",
    "\n",
    "#We will be using functions to achieve this\n",
    "\n",
    "def my_url(keyword):\n",
    "    temp = 'https://www.amazon.com/s?k={}&ref=nb_sb_noss_1'\n",
    "    keyword = keyword.replace(' ', '+')\n",
    "    \n",
    "    # Add Term Query To URL\n",
    "    url = temp.format(keyword)\n",
    "    \n",
    "    # Add Page Query Placeholder\n",
    "    url += '&page{}'\n",
    "    \n",
    "    return url\n",
    "\n",
    "def extract_record(obj):\n",
    "    atag = obj.h2.a\n",
    "    description = atag.text.strip()\n",
    "    url = 'https://www.amazon.com' + atag.get('href')\n",
    "    \n",
    "    #it is possible that some items on amazom.com might not be having one of the items we are looking for(e.g. some items might not be having ratings or price), we will be getting error if we dont take care of that. We will therefore add some error handlers\n",
    "    #if there are no price,probably the item is out of stock or not available, then we will ignore the item, but if there are no reviews yet, it's fine, we will still want to extract the item.\n",
    "    try:\n",
    "        parent=obj.find('span','a-price')\n",
    "        price=parent.find('span','a-offscreen').text\n",
    "    except AttributeError: #we are excepting the error if it occurs so that we can move to extract the next item, else the program will stop running and gives error\n",
    "        return\n",
    "    \n",
    "    try:\n",
    "        rate=obj.i.text\n",
    "        counts_review = obj.find('span', {'class': 'a-size-base', 'dir': 'auto'}).text\n",
    "    except AttributeError:\n",
    "        #assigning empty string to ratings and \n",
    "        rate = ''\n",
    "        counts_review = ''\n",
    "    \n",
    "    image = obj.find('img', {'class': 's-image'}).get('src') \n",
    "    \n",
    "    #let's create a tuple that will contain all these items and assign it to a result variable\n",
    "    result = (description, price, rate, counts_review, url,image)\n",
    "    return result\n",
    "\n",
    "'''Run Main Program Routine'''\n",
    "def main(keyword):\n",
    "    # Startup The Webdriver\n",
    "    driver = webdriver.Chrome()\n",
    "#     options = EdgeOptions()\n",
    "#     options.use_chromium =True\n",
    "#     driver = Edge(options=options)\n",
    "    \n",
    "    records = []  #an empty records list to contain all of our extracted records\n",
    "    url =my_url(keyword)\n",
    "    \n",
    "    for page in range(1, 50):\n",
    "        driver.get(url.format(page))\n",
    "        soup =BeautifulSoup(driver.page_source, 'html.parser')\n",
    "        results=soup.find_all('div',{'data-component-type':'s-search-result'})\n",
    "#         results=soup.find_all('div',{'data-component-type': 's-search-result'}) #same as we did above\n",
    "\n",
    "        \n",
    "#we will like to check if what we have return from the extract_record function is empty or not\n",
    "        for item in results:\n",
    "            record = extract_record(item) \n",
    "            if record: #if the record has something in it append to records list\n",
    "                records.append(record) \n",
    "                \n",
    "#         driver.quit()\n",
    "    \n",
    "#     # Save Results To CSV File\n",
    "        with open('Results.csv', 'w', newline='', encoding='utf-8') as f:\n",
    "            writer = csv.writer(f)\n",
    "            writer.writerow(['Description', 'Price', 'Rating', 'Reviews Count', 'URL','Image link'])\n",
    "            writer.writerows(records)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "id": "2e05519f-72de-4ba9-8d8d-d24d54d0db81",
   "metadata": {},
   "outputs": [
    {
     "ename": "NoSuchWindowException",
     "evalue": "Message: no such window: target window already closed\nfrom unknown error: web view not found\n  (Session info: chrome=112.0.5615.121)\n",
     "output_type": "error",
     "traceback": [
      "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[1;31mNoSuchWindowException\u001b[0m                     Traceback (most recent call last)",
      "\u001b[1;32m<ipython-input-40-c5fe2de71d89>\u001b[0m in \u001b[0;36m<module>\u001b[1;34m\u001b[0m\n\u001b[1;32m----> 1\u001b[1;33m \u001b[0mmain\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;34m'cloths'\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[1;32m<ipython-input-37-a9025c33f8db>\u001b[0m in \u001b[0;36mmain\u001b[1;34m(keyword)\u001b[0m\n\u001b[0;32m     60\u001b[0m     \u001b[1;32mfor\u001b[0m \u001b[0mpage\u001b[0m \u001b[1;32min\u001b[0m \u001b[0mrange\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;36m1\u001b[0m\u001b[1;33m,\u001b[0m \u001b[1;36m50\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m     61\u001b[0m         \u001b[0mdriver\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mget\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0murl\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mformat\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mpage\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m---> 62\u001b[1;33m         \u001b[0msoup\u001b[0m \u001b[1;33m=\u001b[0m\u001b[0mBeautifulSoup\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mdriver\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mpage_source\u001b[0m\u001b[1;33m,\u001b[0m \u001b[1;34m'html.parser'\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m     63\u001b[0m         \u001b[0mresults\u001b[0m\u001b[1;33m=\u001b[0m\u001b[0msoup\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mfind_all\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;34m'div'\u001b[0m\u001b[1;33m,\u001b[0m\u001b[1;33m{\u001b[0m\u001b[1;34m'data-component-type'\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;34m's-search-result'\u001b[0m\u001b[1;33m}\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m     64\u001b[0m \u001b[1;31m#         results=soup.find_all('div',{'data-component-type': 's-search-result'}) #same as we did above\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32m~\\anaconda3\\lib\\site-packages\\selenium\\webdriver\\remote\\webdriver.py\u001b[0m in \u001b[0;36mpage_source\u001b[1;34m(self)\u001b[0m\n\u001b[0;32m    677\u001b[0m             \u001b[0mdriver\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mpage_source\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    678\u001b[0m         \"\"\"\n\u001b[1;32m--> 679\u001b[1;33m         \u001b[1;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mexecute\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mCommand\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mGET_PAGE_SOURCE\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m[\u001b[0m\u001b[1;34m'value'\u001b[0m\u001b[1;33m]\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m    680\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    681\u001b[0m     \u001b[1;32mdef\u001b[0m \u001b[0mclose\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mself\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32m~\\anaconda3\\lib\\site-packages\\selenium\\webdriver\\remote\\webdriver.py\u001b[0m in \u001b[0;36mexecute\u001b[1;34m(self, driver_command, params)\u001b[0m\n\u001b[0;32m    319\u001b[0m         \u001b[0mresponse\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mcommand_executor\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mexecute\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mdriver_command\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mparams\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    320\u001b[0m         \u001b[1;32mif\u001b[0m \u001b[0mresponse\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m--> 321\u001b[1;33m             \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0merror_handler\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mcheck_response\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mresponse\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m    322\u001b[0m             response['value'] = self._unwrap_value(\n\u001b[0;32m    323\u001b[0m                 response.get('value', None))\n",
      "\u001b[1;32m~\\anaconda3\\lib\\site-packages\\selenium\\webdriver\\remote\\errorhandler.py\u001b[0m in \u001b[0;36mcheck_response\u001b[1;34m(self, response)\u001b[0m\n\u001b[0;32m    240\u001b[0m                 \u001b[0malert_text\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mvalue\u001b[0m\u001b[1;33m[\u001b[0m\u001b[1;34m'alert'\u001b[0m\u001b[1;33m]\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mget\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;34m'text'\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    241\u001b[0m             \u001b[1;32mraise\u001b[0m \u001b[0mexception_class\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mmessage\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mscreen\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mstacktrace\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0malert_text\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m--> 242\u001b[1;33m         \u001b[1;32mraise\u001b[0m \u001b[0mexception_class\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mmessage\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mscreen\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mstacktrace\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m    243\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    244\u001b[0m     \u001b[1;32mdef\u001b[0m \u001b[0m_value_or_default\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mself\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mobj\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mkey\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mdefault\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;31mNoSuchWindowException\u001b[0m: Message: no such window: target window already closed\nfrom unknown error: web view not found\n  (Session info: chrome=112.0.5615.121)\n"
     ]
    }
   ],
   "source": [
    "main('cloths')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "id": "c8862157-8ff6-4232-a619-5578d78c3e4e",
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "id": "a7180427-20e5-4650-9793-ea4af2be5802",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Description</th>\n",
       "      <th>Price</th>\n",
       "      <th>Rating</th>\n",
       "      <th>Reviews Count</th>\n",
       "      <th>URL</th>\n",
       "      <th>Image link</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Amazon Basics Microfiber Cleaning Cloths, Non-...</td>\n",
       "      <td>$13.22</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>https://www.amazon.com/gp/slredirect/picassoRe...</td>\n",
       "      <td>https://m.media-amazon.com/images/I/91srWSvgA3...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>USANOOKS Microfiber Cleaning Cloth Grey - 12Pc...</td>\n",
       "      <td>$13.99</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>https://www.amazon.com/Microfiber-Cleaning-Clo...</td>\n",
       "      <td>https://m.media-amazon.com/images/I/A1U4ZA-OJm...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>AIDEA Microfiber Cleaning Cloths-8PK, All-Purp...</td>\n",
       "      <td>$5.99</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>https://www.amazon.com/AIDEA-Microfiber-Cleani...</td>\n",
       "      <td>https://m.media-amazon.com/images/I/71QbuUZzMX...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>AIDEA Microfiber Cleaning Cloths White-50PK, S...</td>\n",
       "      <td>$18.99</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>https://www.amazon.com/AIDEA-Microfiber-Absorp...</td>\n",
       "      <td>https://m.media-amazon.com/images/I/71xVouSe2E...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>AIDEA Microfiber Cleaning Cloths-100PK, Softer...</td>\n",
       "      <td>$28.99</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>https://www.amazon.com/AIDEA-Microfiber-Absorb...</td>\n",
       "      <td>https://m.media-amazon.com/images/I/81zpetuiJz...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1411</th>\n",
       "      <td>Simple Joys by Carter's Unisex Babies' 6-Piece...</td>\n",
       "      <td>$22.80</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>https://www.amazon.com/Simple-Joys-Carters-6-P...</td>\n",
       "      <td>https://m.media-amazon.com/images/I/81D7d6cfKw...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1412</th>\n",
       "      <td>Aunti Em's Kitchen Wrinkle Resistant Dinner Na...</td>\n",
       "      <td>$22.88</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>https://www.amazon.com/Aunti-Ems-Kitchen-Wrink...</td>\n",
       "      <td>https://m.media-amazon.com/images/I/61tM2TfPwM...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1413</th>\n",
       "      <td>The Children's Place Baby Girls' and Toddler P...</td>\n",
       "      <td>$12.25</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>https://www.amazon.com/Childrens-Place-Toddler...</td>\n",
       "      <td>https://m.media-amazon.com/images/I/81bgz1vT+0...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1414</th>\n",
       "      <td>KAFIREN Baby Boy Clothes Toddler Boy Summer Ou...</td>\n",
       "      <td>$18.95</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>https://www.amazon.com/KAFIREN-Short-Light-Gre...</td>\n",
       "      <td>https://m.media-amazon.com/images/I/61Sm7M-x16...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1415</th>\n",
       "      <td>Simple Joys by Carter's Baby Boys' Cotton Pant...</td>\n",
       "      <td>$15.20</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>https://www.amazon.com/Simple-Joys-Carters-4-P...</td>\n",
       "      <td>https://m.media-amazon.com/images/I/91uqhGcxqm...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>1416 rows × 6 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                                            Description   Price  Rating  \\\n",
       "0     Amazon Basics Microfiber Cleaning Cloths, Non-...  $13.22     NaN   \n",
       "1     USANOOKS Microfiber Cleaning Cloth Grey - 12Pc...  $13.99     NaN   \n",
       "2     AIDEA Microfiber Cleaning Cloths-8PK, All-Purp...   $5.99     NaN   \n",
       "3     AIDEA Microfiber Cleaning Cloths White-50PK, S...  $18.99     NaN   \n",
       "4     AIDEA Microfiber Cleaning Cloths-100PK, Softer...  $28.99     NaN   \n",
       "...                                                 ...     ...     ...   \n",
       "1411  Simple Joys by Carter's Unisex Babies' 6-Piece...  $22.80     NaN   \n",
       "1412  Aunti Em's Kitchen Wrinkle Resistant Dinner Na...  $22.88     NaN   \n",
       "1413  The Children's Place Baby Girls' and Toddler P...  $12.25     NaN   \n",
       "1414  KAFIREN Baby Boy Clothes Toddler Boy Summer Ou...  $18.95     NaN   \n",
       "1415  Simple Joys by Carter's Baby Boys' Cotton Pant...  $15.20     NaN   \n",
       "\n",
       "      Reviews Count                                                URL  \\\n",
       "0               NaN  https://www.amazon.com/gp/slredirect/picassoRe...   \n",
       "1               NaN  https://www.amazon.com/Microfiber-Cleaning-Clo...   \n",
       "2               NaN  https://www.amazon.com/AIDEA-Microfiber-Cleani...   \n",
       "3               NaN  https://www.amazon.com/AIDEA-Microfiber-Absorp...   \n",
       "4               NaN  https://www.amazon.com/AIDEA-Microfiber-Absorb...   \n",
       "...             ...                                                ...   \n",
       "1411            NaN  https://www.amazon.com/Simple-Joys-Carters-6-P...   \n",
       "1412            NaN  https://www.amazon.com/Aunti-Ems-Kitchen-Wrink...   \n",
       "1413            NaN  https://www.amazon.com/Childrens-Place-Toddler...   \n",
       "1414            NaN  https://www.amazon.com/KAFIREN-Short-Light-Gre...   \n",
       "1415            NaN  https://www.amazon.com/Simple-Joys-Carters-4-P...   \n",
       "\n",
       "                                             Image link  \n",
       "0     https://m.media-amazon.com/images/I/91srWSvgA3...  \n",
       "1     https://m.media-amazon.com/images/I/A1U4ZA-OJm...  \n",
       "2     https://m.media-amazon.com/images/I/71QbuUZzMX...  \n",
       "3     https://m.media-amazon.com/images/I/71xVouSe2E...  \n",
       "4     https://m.media-amazon.com/images/I/81zpetuiJz...  \n",
       "...                                                 ...  \n",
       "1411  https://m.media-amazon.com/images/I/81D7d6cfKw...  \n",
       "1412  https://m.media-amazon.com/images/I/61tM2TfPwM...  \n",
       "1413  https://m.media-amazon.com/images/I/81bgz1vT+0...  \n",
       "1414  https://m.media-amazon.com/images/I/61Sm7M-x16...  \n",
       "1415  https://m.media-amazon.com/images/I/91uqhGcxqm...  \n",
       "\n",
       "[1416 rows x 6 columns]"
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df=pd.read_csv('Results.csv')\n",
    "df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2367e254-4bb2-4d14-b646-e30eefe2ed63",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}