08.Web_Scraping_1z2.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Warsztaty Python w Data Science\n",
    "\n",
    "---\n",
    "## Web Scraping - część 1 z 2  \n",
    "\n",
    "- ### Budowa web crawlera\n",
    " - #### Anatomia pająka\n",
    " - #### Zarządzanie granicą (*Crawling Frontier*)\n",
    " - #### Jak scrapować etycznie i bezpiecznie\n",
    "- ### Anatomia strony WWW\n",
    " - #### HTML jaki jest - każdy widzi\n",
    " - #### Parsowanie pobranych danych\n",
    "- ### Prosty, praktyczny scraper\n",
    "---\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Budowa (web) crawlera\n",
    "\n",
    "*Web crawler, webbot, pająk, spiderm pełzacz, web crawler, web wanderer, scraper* - program zbierający informacje o strukturze i treściach stron WWW. Używany najczęściej do \n",
    "\n",
    "### Anatomia pająka\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*pająk* - to program który:\n",
    "- odwiedzają linki ze listy określanej jako granica (*the frontier*)\n",
    "- z odwiedzonych stron wyciąga i zapisuje informacje\n",
    "  - w szczególności dalsze linki (web indexing)\n",
    "  - odpowiednie linki uzupełniają *granicę* crawlingu\n",
    "  - informację w sposób trwały zapisuje web scraper \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Zarządzanie granicą (*Crawling Frontier*)\n",
    "\n",
    "- Zakres granicy powinien być ZAWSZE na początku określony\n",
    "    - najlepiej z góry zawężony do określonej liczby i typu linków\n",
    "- Granica winna być mocno ograniczana\n",
    "- Granica rozbudowywana powinna być BARDZO selektywnie\n",
    "- Duża granica powoduje problemy skali"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "### Jak scrapować etycznie i bezpiecznie"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "1. Po pierwsze - nie szkodzić! Nie obciążaj niepotrzebnie strony scrapowanej\n",
    "2. Przestrzegaj `robots.txt` i warunków korzystania z usługi\n",
    "3. Miej na uwadze, że bazy danych są chronione na podstawie przepisów ustawy o ochronie baz danych \n",
    "4. Przestrzegaj RODO\n",
    "5. Nie ukrywaj się\n",
    "6. Gdzie to możliwe,  korzystaj z API\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# art. 8 ustawy z dnia 27 lipca 2001 r. o ochronie baz danych (Dz.U. Nr 28, poz. 1402 ze zm.) \n",
    "\n",
    "1. Wolno korzystać z istotnej, co do jakości lub ilości, części rozpowszechnionej bazy danych:\n",
    "\n",
    "  1)   do własnego użytku osobistego, ale tylko z zawartości nieelektronicznej bazy danych,\n",
    "\n",
    "  2)   w charakterze ilustracji, w celach _**dydaktycznych lub badawczych**_ (podkreslenie moje), ze wskazaniem źródła, jeżeli takie korzystanie jest uzasadnione niekomercyjnym celem, dla którego wykorzystano bazę,\n",
    "  \n",
    "  3)   do celów bezpieczeństwa wewnętrznego, postępowania sądowego lub administracyjnego.\n",
    "\n",
    "2. Nie jest dozwolone powtarzające się i systematyczne pobieranie lub wtórne wykorzystanie sprzeczne z normalnym korzystaniem i powodujące nieusprawiedliwione naruszenie słusznych interesów producenta.\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Anatomia strony WWW\n",
    "#### HTML jaki jest - każdy widzi\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "html_doc = \"\"\"\n",
    "<html>\n",
    "<head><title>The Dormouse's story</title></head>\n",
    "<body>\n",
    "<p class=\"title\"><b>The Dormouse's story</b></p>\n",
    "\n",
    "<p class=\"story\">Once upon a time there were three little sisters; and their names were\n",
    "    <a href=\"http://example.com/elsie\" class=\"sister\" id=\"link1\">Elsie</a>,\n",
    "    <a href=\"http://example.com/lacie\" class=\"sister\" id=\"link2\">Lacie</a> and\n",
    "    <a href=\"http://example.com/tillie\" class=\"sister\" id=\"link3\">Tillie</a>;\n",
    "    and they lived at the bottom of a well.</p>\n",
    "\n",
    "<p class=\"story\">...</p>\n",
    "</body></html>\n",
    "\"\"\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "#### Parsowanie pobranych danych"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from bs4 import BeautifulSoup\n",
    "soup = BeautifulSoup(html_doc, 'html.parser')\n",
    "\n",
    "print(soup.prettify())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "## HTML ma\n",
    "- strukturę drzewiastą\n",
    "- znaczniki się zagnieżdzają i mają mieć (w teorii):\n",
    "  - `<a>` - początek\n",
    "  - `</a>` - koniec\n",
    "- pomiędzy początkiem a końcem są tzw. dzieci\n",
    "- znaczniki nie mogą się \"zazębiac\" np. `<a><b></a></b>` (w teorii ...)\n",
    "- znaczniki mają atrybuty - np. `<a href=\"linkdostrony\">Tu jest link</a>\n",
    "---"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from bs4 import BeautifulSoup\n",
    "import lxml\n",
    "soup = BeautifulSoup(html_doc, 'lxml')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "soup.p"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "soup.p['class']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "soup.a"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Szukanie po znaczniku"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "soup.find_all('a')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Szukanie po id"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "soup.find(id=\"link3\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Wyciąganie atrybutów"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "[ link.get('href') for link in soup.find_all('a')]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Wędrowanie po drzewie"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "soup.a"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "soup.a.find_next_sibling(\"a\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "soup.p"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "soup.p.find_next_sibling(\"p\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pn=soup.p.find_next_sibling(\"p\")\n",
    "children = pn.children"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "children"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "lista = [ x for x in children ]\n",
    "lista"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "lista[1].get('href')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "head_tag = soup.head\n",
    "head_tag"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for child in head_tag.children:\n",
    "    print(child)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for child in head_tag.descendants:\n",
    "    print(child)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "last_a_tag = soup.find(\"a\", id=\"link3\")\n",
    "last_a_tag\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "last_a_tag.next_sibling"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "last_a_tag.next_element"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "last_a_tag.parent"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Szukanie przy użyciu predykatu (funckji logicznej)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def has_class_but_no_id(tag):\n",
    "    return tag.has_attr('class') and not tag.has_attr('id')\n",
    "\n",
    "soup.find_all(has_class_but_no_id)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "soup.find_all(id='link2')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "soup.find_all(\"a\", class_=\"sister\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "soup.find_all(\"a\")\n",
    "soup(\"a\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "## Prosty, praktyczny scraper"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import requests\n",
    "from bs4 import BeautifulSoup\n",
    "import pandas as pd\n",
    "import time\n",
    "\n",
    "\n",
    "frontier = [ f'https://www.gumtree.pl/s-mieszkania-i-domy-do-wynajecia/warszawa/page-{n}/v1c9008l3200008p{n}' for n in range(3) ]\n",
    "data = {'title': [], 'link':[]}\n",
    "    \n",
    "for url in frontier:\n",
    "    time.sleep(1)\n",
    "    page = requests.get(url)\n",
    "    print (page)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import requests\n",
    "from bs4 import BeautifulSoup\n",
    "import pandas as pd\n",
    "import time\n",
    "\n",
    "\n",
    "frontier = [ f'https://www.gumtree.pl/s-mieszkania-i-domy-do-wynajecia/warszawa/page-{n}/v1c9008l3200008p{n}' for n in range(3) ]\n",
    "data = {'title': [], 'link':[]}\n",
    "    \n",
    "for url in frontier:\n",
    "    time.sleep(1)\n",
    "    page = requests.get(url)\n",
    "    soup = BeautifulSoup(page.content, 'html.parser')\n",
    "    titles = [flat.next_element for flat in soup.find_all('a', class_ = \"href-link tile-title-text\")] \n",
    "    print(titles)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import requests\n",
    "from bs4 import BeautifulSoup\n",
    "import pandas as pd\n",
    "import time\n",
    "\n",
    "frontier = [ f'https://www.gumtree.pl/s-mieszkania-i-domy-do-wynajecia/warszawa/page-{n}/v1c9008l3200008p{n}' for n in range(3) ]\n",
    "data = {'title': [], 'link':[]}\n",
    "    \n",
    "for url in frontier:\n",
    "    time.sleep(1)\n",
    "    page = requests.get(url)\n",
    "    soup = BeautifulSoup(page.content, 'html.parser')\n",
    "    titles = [flat.next_element for flat in soup.find_all('a', class_ = \"href-link tile-title-text\")] \n",
    "    links = ['https://www.gumtree.pl' + link.get('href')\n",
    "                for link in soup.find_all('a', class_ =\"href-link tile-title-text\")]\n",
    "    print(links)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import requests\n",
    "from bs4 import BeautifulSoup\n",
    "import pandas as pd\n",
    "import time\n",
    "\n",
    "frontier = [ f'https://www.gumtree.pl/s-mieszkania-i-domy-do-wynajecia/warszawa/page-{n}/v1c9008l3200008p{n}' for n in range(3) ]\n",
    "data = {'title': [], 'link':[]}\n",
    "    \n",
    "for url in frontier:\n",
    "    time.sleep(1)\n",
    "    page = requests.get(url)\n",
    "    soup = BeautifulSoup(page.content, 'html.parser')\n",
    "    titles = [flat.next_element for flat in soup.find_all('a', class_ = \"href-link tile-title-text\")] \n",
    "    links = ['https://www.gumtree.pl' + link.get('href')\n",
    "                for link in soup.find_all('a', class_ =\"href-link tile-title-text\")]\n",
    "    data['link'].extend(links)\n",
    "    data['title'].extend(titles)\n",
    "print(data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "df = pd.DataFrame(data).drop_duplicates()\n",
    "df.head(100)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.to_csv(\"./gumtree_all_pages.csv\", sep=';',index=False, encoding = 'utf-8')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "# Zadanie 1.\n",
    "Wyciągnąć z dowolnego ogłoszenia cenę mieszkania\n",
    "\n",
    "# Zadanie 2.\n",
    "Wyciągnąć z dowolnego ogłoszenia cenę ZA METR\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}