Scrape tutorial - lbxml2 related errors installing Scrapy and unresolved external errors installing psycopg2 #199
Description
Systeminfo: MS Windows 7, Python 2.7
Running a command pip install -r requirements.txt from within virtualenv I kept getting bunch of errors, once running the requirements one by one it turned out that issue was with scrapy and psycopg2.
Solving issue with Scrapy
The installation would fail with the following error
c:\users\<user>\appdata\local\temp\xmlXPathInitjfnyng.c(1) : fatal error C1083: Cannot open include file: 'libxml/xpat
h.h': No such file or directory
*********************************************************************************
Could not find function xmlCheckVersion in library libxml2. Is libxml2 installed?
*********************************************************************************
error: command 'C:\\Users\\<user>\\AppData\\Local\\Programs\\Common\\Microsoft\\Visual C++ for Python\\9.0\\VC\\Bin\\c
l.exe' failed with exit status 2
----------------------------------------
Command "c:\users\<user>\.virtualenvs\scrapeproj\scripts\python.exe -u -c "import setuptools, tokenize;__file__='c:\\users
\\<user>\\appdata\\local\\temp\\pip-build-8nda_z\\lxml\\setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).r
ead().replace('\r\n', '\n'), __file__, 'exec'))" install --record c:\users\<user>\appdata\local\temp\pip-nkczkk-record\ins
tall-record.txt --single-version-externally-managed --compile --install-headers c:\users\<user>\.virtualenvs\scrapeproj\in
clude\site\python2.7\lxml" failed with error code 1 in c:\users\<user>\appdata\local\temp\pip-build-8nda_z\lxml
Simply running pip install lxml within virtualenv did not help (suggestion from here),
but running
easy_install.exe lxml
as indicated here helped
Solving issue with psycopg2
While installing the psycopg2==2.5.4 I kept getting the following errors:
build\lib.win32-2.7\psycopg2\_psycopg.pyd : fatal error LNK1120: 62 unresolved externals
error: Setup script exited with error: command 'C:\\Users\\mita\\AppData\\Local\\Programs\\Common\\Microsoft\\Visual C++
for Python\\9.0\\VC\\Bin\\link.exe' failed with exit status 1120
and managed to solve them by installing the binaries (see more details here) by running
pip install git+https://github.com/nwcell/psycopg2-windows.git@win32-py27#egg=psycopg2
within virtualenv