searchengine

Search engine using Scrapy Framework to get web data, Elasticsearch to store it, Flask as backend and Vuejs Framework to visualizate it.

Windows 10 Dev Setup

Python environment

py -3 -m venv virtualenv

.\virtualenv\Scripts\Activate.ps1

pip install -r requirements.txt

Run Scrapy

See how to install Scrapy. After installed, with Anaconda/Miniconda opened go to project root and run:

cd crawler\wikipedia

scrapy crawl wikipedia -O data\wikipedia.json

Elasticsearch

Install elasticsearch 7.10.1 and in another terminal run it from .bat file. For example:

C:\elasticsearch-7.10.1\bin\elasticsearch.bat

Run index_data.py

Be careful to not run it twice.

cd backend

py index_data.py

Run Flask

$env:FLASK_APP="serve.py"

$env:FLASK_ENV="development"

flask run --host=0.0.0.0 --port=8088

Vuejs environment

With Node.js installed, in another terminal, run:

cd vuesearch

npm install

npm run serve