09.Web_Scraping_2z2.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Warsztaty Python w Data Science\n",
    "\n",
    "---\n",
    "## Web Scraping - część 2 z 2  \n",
    "\n",
    "- ### Automatyzacja autentykacji \n",
    " - #### Anatomia nowoczesnej strony\n",
    " - #### browser cookies\n",
    " - #### wykorzystanie API\n",
    "- ### Iteratory, Generatory i `yield` \n",
    "- ### Zaawansowane scrapowanie przy użyciu biblioteki `Scrapy`\n",
    " - #### _\"Grzeczne\"_ pająki w Scrapy\n",
    " - #### Rozbudowany pająk do różnych stron\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "https://drive.google.com/drive/folders/1HR8VCledCwD7BRMO1AucUM3x7cYC-AVT?usp=sharing"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "https://github.com/MichalKorzycki/PythonDataScience"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "# Zadanie 1.\n",
    "Wyciągnąć z dowolnego ogłoszenia cenę mieszkania"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Jeśli Ci logi przeskadzają, odkomentuj to poniżej:\n",
    "\n",
    "# import logging\n",
    "# logger = logging.getLogger()\n",
    "# logger.setLevel(logging.CRITICAL)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import requests\n",
    "from bs4 import BeautifulSoup\n",
    "\n",
    "\n",
    "url='https://www.gumtree.pl/a-mieszkania-i-domy-do-wynajecia/srodmiescie/pokoj-z-kuchnia-36m2-marszalkowska-nr-140-na-ii-pietrze-wszedzie-blisko-dwie-linie-metra/1008881683130910468533409'\n",
    "page = requests.get(url)\n",
    "soup = BeautifulSoup(page.content, 'html.parser')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "values = [flat.next_element for flat in soup.find_all('span', class_ = \"amount\")] "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "cena = float(''.join(c for c in values[0] if c.isdigit()))\n",
    "cena"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "# Zadanie 2.\n",
    "Wyciągnąć z dowolnego ogłoszenia cenę ZA METR\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "values = [flat.next_element for flat in soup.find_all('span', class_ = \"value\")] \n",
    "values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "cells =  soup.find_all('div', class_ = \"attribute\")\n",
    "for cell in cells:\n",
    "    for desc in cell.find_all('span', class_ = \"name\"):\n",
    "        if desc.next_element.find('m2') > -1:\n",
    "            for value in cell.find_all('span', class_ = \"value\"):\n",
    "                print(value.next_element)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "# Automatyzacja autentykacji \n",
    "\n",
    "https://github.com/techtanic/Udemy-Course-Grabber\n",
    "\n",
    "- ## Anatomia nowoczesnej strony\n",
    "\n",
    "\n",
    "### JAM Stack\n",
    "- #### JavaScript\n",
    "- #### API\n",
    "- #### Markup"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "- ### browser cookies"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### `!pip install browser-cookie3`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import browser_cookie3"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "- ### *6. Gdzie to możliwe,  korzystaj z API*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import requests \n",
    "\n",
    "cookies = browser_cookie3.load(domain_name='www.udemy.com')\n",
    "requests.utils.dict_from_cookiejar(cookies)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "   head = {\n",
    "        'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.138 Safari/537.36',\n",
    "        'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9'\n",
    "    }\n",
    "    \n",
    "requests.get('https://www.udemy.com/api-2.0/users/me/subscribed-courses/',headers=head).json()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "cookies = browser_cookie3.load(domain_name='www.udemy.com')\n",
    "my_cookies = requests.utils.dict_from_cookiejar(cookies)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import random\n",
    "\n",
    "access_token = my_cookies['access_token']\n",
    "csrftoken = my_cookies['csrftoken']\n",
    "ip = \".\".join(map(str, (random.randint(0, 255) for _ in range(4))))\n",
    "head = {\n",
    "        'authorization': 'Bearer ' + access_token,\n",
    "        'accept': 'application/json, text/plain, */*',\n",
    "        'x-requested-with': 'XMLHttpRequest',\n",
    "        'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.141 Safari/537.36 Edg/87.0.664.75',\n",
    "        'x-forwarded-for': str(ip),\n",
    "        'x-udemy-authorization': 'Bearer ' + access_token,\n",
    "        'content-type': 'application/json;charset=UTF-8',\n",
    "        'origin': 'https://www.udemy.com',\n",
    "        'referer': 'https://www.udemy.com/',\n",
    "        'dnt': '1',\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "courses = requests.get('https://www.udemy.com/api-2.0/users/me/subscribed-courses/',headers=head).json()\n",
    "[ x[\"title\"] for x in courses['results']]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Iteratory, Generatory i `yield` "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print (range(5))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for i in range(5):\n",
    "    print (i)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "mystr = \"banana\"\n",
    "myit = iter(mystr)\n",
    "\n",
    "print(next(myit))\n",
    "print(next(myit))\n",
    "print(next(myit))\n",
    "print(next(myit))\n",
    "print(next(myit))\n",
    "print(next(myit))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "class Counter:\n",
    "    def __init__(self, low, high):\n",
    "        self.current = low - 1\n",
    "        self.high = high\n",
    "\n",
    "    def __iter__(self):\n",
    "        return self\n",
    "\n",
    "    def __next__(self): # Python 2: def next(self)\n",
    "        self.current += 1\n",
    "        if self.current < self.high:\n",
    "            return self.current\n",
    "        raise StopIteration\n",
    "\n",
    "\n",
    "for c in Counter(3, 9):\n",
    "    print(c)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### *Generatory* są mechanizmem\n",
    "* tworzenia iteratorów\n",
    "* Zwraca dane przez *yield*\n",
    "* Każde wywołanie _next()_ zaczyna od miejsca gdzie skończył poprzedni krok\n",
    "* _next()_ tworzona jest automatycznie\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def reverse(data):\n",
    "    for index in range(len(data)-1, -1, -1):\n",
    "        print(index)\n",
    "        yield data[index]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for c in reverse('Python'):\n",
    "    print (c)\n",
    "    print()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Generatory, Yield"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "mylist = [0, 1, 4]\n",
    "for i in mylist:\n",
    "    print(i)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "mylist = [x*x for x in range(3)]\n",
    "for i in mylist:\n",
    "    print(i)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "mylist = (x*x for x in range(3))\n",
    "for i in mylist:\n",
    "    print(i)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def create_generator():\n",
    "    mylist = range(3)\n",
    "    for i in mylist:\n",
    "        yield i*i\n",
    "        \n",
    "for i in create_generator():\n",
    "    print(i)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def fib(n):\n",
    "    if n == 0 or n == 1:\n",
    "        return n\n",
    "    else:\n",
    "        return fib(n-1) + fib(n-2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for i in range(36):\n",
    "    print (\"n=%d => %d\" % (i, fib(i)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def fib(n):\n",
    "    a, b = 0, 1\n",
    "    i=0\n",
    "    while i < n:\n",
    "        yield (i, a)\n",
    "        a, b = b, a + b\n",
    "        i += 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for i, f in fib(36):\n",
    "    print (\"n=%d => %d\" % (i, f))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "## Zaawansowane scrapowanie przy użyciu biblioteki `Scrapy`\n",
    "\n",
    "https://scrapy.org/\n",
    "\n",
    " - ### _\"Grzeczne\"_ pająki w Scrapy\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- 1. Po pierwsze - nie szkodzić! Nie obciążaj niepotrzebnie strony scrapowanej\n",
    "- 2. Przestrzegaj `robots.txt` i warunków korzystania z usługi\n",
    "- 5. Nie ukrywaj się"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Scrapy doesn’t wait a fixed amount of time between requests, but uses a random interval between `0.5 * DOWNLOAD_DELAY` and `1.5 * DOWNLOAD_DELAY`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### `FAIL2BAN` - typowe zabezpieczenia\n",
    "\n",
    "Przykład z dokumentacji:\n",
    "\n",
    "*As you can see in my example, I have set up 300 maxretry and 300 for findtime, so, we need to have 300 GETs from the same IP in a time window of 300 seconds to have the originating IP blocked.*\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import scrapy\n",
    "import scrapy.crawler as crawler\n",
    "from bs4 import BeautifulSoup\n",
    "\n",
    "from scrapy.crawler import CrawlerProcess\n",
    "\n",
    "class MySpider(scrapy.Spider):\n",
    "    name = 'myspider'\n",
    "    start_urls = [\n",
    "        'https://www.gumtree.pl/s-mieszkania-i-domy-do-wynajecia/warszawa/v1c9008l3200008p1'\n",
    "        ]\n",
    "    \n",
    "    item_urls2 = ['https://www.gumtree.pl/s-mieszkania-i-domy-do-wynajecia/warszawa/page-2/v1c9008l3200008p2']    \n",
    "           \n",
    "    custom_settings = {\n",
    "        'DOWNLOAD_DELAY': '2.0',\n",
    "        'ROBOTSTXT_OBEY': True,\n",
    "        'AUTOTHROTTLE_ENABLED': True,\n",
    "        'USER_AGENT': 'My Bot (email@myemail.com)'\n",
    "    }\n",
    "\n",
    "    top_url = 'https://www.gumtree.pl'\n",
    "    \n",
    "    \n",
    "    def parse(self, response):\n",
    "        self.logger.info('1. Got successful response from {}'.format(response.url))\n",
    "\n",
    "        for item_url in self.item_urls2:\n",
    "                yield scrapy.Request(item_url, self.parse)\n",
    "\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "process = CrawlerProcess()\n",
    "process.crawl(MySpider)\n",
    "process.start()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import scrapy\n",
    "import scrapy.crawler as crawler\n",
    "from bs4 import BeautifulSoup\n",
    "\n",
    "from scrapy.crawler import CrawlerProcess\n",
    "\n",
    "class SimpleGumtreeApartmentsSpider(scrapy.Spider):\n",
    "    name = 'simplegumtreeapartmentsspider'\n",
    "    start_urls = []\n",
    "    start_urls.append(\n",
    "        'https://www.gumtree.pl/s-mieszkania-i-domy-sprzedam-i-kupie/mazowieckie/v1c9073l3200001p1'\n",
    "    )\n",
    "    found_apartments = []\n",
    "   \n",
    "    custom_settings = {\n",
    "        'DOWNLOAD_DELAY': '2.0',\n",
    "        'ROBOTSTXT_OBEY': True,\n",
    "        'AUTOTHROTTLE_ENABLED': True,\n",
    "        'USER_AGENT': 'My Bot (email@myemail.com)'\n",
    "    }\n",
    "\n",
    "    top_url = 'https://www.gumtree.pl'\n",
    "    def parse(self, response):\n",
    "        self.logger.info('Got successful response from {}'.format(response.url))\n",
    "        soup = BeautifulSoup(response.body, 'lxml')\n",
    "        titles = [flat.next_element for flat in soup.find_all('a', class_ = \"href-link tile-title-text\")] \n",
    "        links = ['https://www.gumtree.pl' + link.get('href')\n",
    "                for link in soup.find_all('a', class_ =\"href-link tile-title-text\")]\n",
    "            \n",
    "        for item_url in links:\n",
    "            yield scrapy.Request(item_url, self.parse_item)\n",
    "        \n",
    "    def parse_item(self, response): #item_url - odwiedzanie strony, #self.parse_item - przetworzenie przy pomocy funkcji\n",
    "        self.logger.info('Got successful response from {}'.format(response.url))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "process = CrawlerProcess()\n",
    "process.crawl(SimpleGumtreeApartmentsSpider)\n",
    "process.start()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import scrapy\n",
    "import scrapy.crawler as crawler\n",
    "from bs4 import BeautifulSoup\n",
    "\n",
    "from scrapy.crawler import CrawlerProcess\n",
    "\n",
    "class GumtreeApartmentsSpider(scrapy.Spider):\n",
    "    name = 'gumtreeapartmentsspider'\n",
    "    start_urls = [\n",
    "        'https://www.gumtree.pl/s-mieszkania-i-domy-sprzedam-i-kupie/mazowieckie/page-'+str(i)+'/v1c9073l3200001p'+str(i)  for i in range(2,4)\n",
    "        ]\n",
    "    start_urls.append(\n",
    "        'https://www.gumtree.pl/s-mieszkania-i-domy-sprzedam-i-kupie/mazowieckie/v1c9073l3200001p1'\n",
    "    )\n",
    "    found_apartments = []\n",
    "   \n",
    "    custom_settings = {\n",
    "        'DOWNLOAD_DELAY': '2.0',\n",
    "        'ROBOTSTXT_OBEY': True,\n",
    "        'AUTOTHROTTLE_ENABLED': True,\n",
    "        'USER_AGENT': 'My Bot (email@myemail.com)'\n",
    "    }\n",
    "\n",
    "    top_url = 'https://www.gumtree.pl'\n",
    "    def parse(self, response):\n",
    "        self.logger.info('Got successful response from {}'.format(response.url))\n",
    "        soup = BeautifulSoup(response.body, 'lxml')\n",
    "        titles = [flat.next_element for flat in soup.find_all('a', class_ = \"href-link tile-title-text\")] \n",
    "        links = ['https://www.gumtree.pl' + link.get('href')\n",
    "                for link in soup.find_all('a', class_ =\"href-link tile-title-text\")]\n",
    "            \n",
    "        for item_url in links:\n",
    "            yield scrapy.Request(item_url, self.parse_item)\n",
    "        \n",
    "    def parse_item(self, response): \n",
    "        self.logger.info('Got successful response from {}'.format(response.url))\n",
    "        # I tu uzupełniamy logiką"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "process = CrawlerProcess()\n",
    "process.crawl(GumtreeApartmentsSpider)\n",
    "process.start()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "# Zadanie 1.\n",
    "Wyciągnąć z _*kilku*_ ogłoszeń ich tytuły i treści"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}