Python and scrapy deployment in centos environment

1 operating system##

Centos 6.5 64bit or above

2 Software Environment##

Tips: (1) what software to use; (2) operating environment (3) development kit.

1. Python：Python-2.7.6.tgz
2. Scrapy：Scrapy-1.0.3.tar.gz
3. Selenium：selenium (2.48.0)
4. Phantomjs：phantomjs-1.9.8-linux-x86_64.tar.bz2
5. MySQL-python：MySQL-python-1.2.5.zip
6. MySQL：MySQL-5.6.27-1.linux_glibc2.5.x86_64.rpm-bundle.tar

3 installation steps##

1. Install dependent libraries###

yum install zlib-devel bzip2-devel ncurses-devel sqlite-devel readline-devel tk-devel mysql-devel gcc libxslt-devel libxml2-devel libffi-devel  openssl-devel python-devel

# yum install zlib-devel
# yum install bzip2-devel
# yum install openssl-devel
# yum install ncurses-devel
# yum install sqlite-devel
# yum install readline-devel
# yum install tk-devel
# yum install mysql-devel
# yum install gcc
# yum install make(centos may have its own, no need to install)

Scrapy must be installed:

# yum install libxslt-devel 
# yum install libxml2-devel 
# yum install libffi-devel

2. Upgrade Python

(Download address: https://www.python.org/ftp/python/2.7.6/Python-2.7.6.tgz)
Centos6.5 system comes with python2.6.6 and needs to be upgraded to Python2.7
Install the library file: yum install zlib-devel openssl-devel sqlite-devel
Download: wget https://www.python.org/ftp/python/2.7.6/Python-2.7.6.tgz
Unzip: tar zxvf Python-2.7.6.tgz
installation:

cd Python-2.7.6./configure --prefix=/usr/local/python
make
make install

Create a soft link: ln -sf /usr/local/python/bin/python2.7 /usr/bin/python
Fix yum: modify the first line of the file /usr/bin/yum (#!/usr/bin/python changed to #!/usr/bin/python2.6)

3. Install setuptools and pip

An error occurs when using pip to install the python module. It is recommended to update setuptools and pip to eliminate it.

Download link: https://pypi.python.org/packages/source/s/setuptools/setuptools-18.4.tar.gz
Install after decompression: python setup.py install (or pip install –upgrade setuptools)

Download link: https://pypi.python.org/packages/source/p/pip/pip-7.1.2.tar.gz
Install after decompression: python setup.py install (or pip install -upgrade pip)

4. Install scrapy+selenium+phantomjs

Download link: https://pypi.python.org/packages/source/S/Scrapy/Scrapy-1.0.3.tar.gz
Install after decompression: python setup.py install (you can also use the pip install scrapy command to install)
Note: When installing with pip install scrapy, there may be an error when the network is abnormal or other dependent libraries are downloaded and installed. You can download the dependent libraries separately for installation.

selenium+phantomjs is installed on demand, if you use a phantomjs browser to parse js, install it:
Download link: https://pypi.python.org/packages/source/s/selenium/selenium-2.48.0.tar.gz
Install after decompression: python setup.py install (you can also use the pip install selenium command to install)
download link:
https://bitbucket.org/ariya/phantomjs/downloads/phantomjs-1.9.8-linux-i686.tar.bz2
wget https://bitbucket.org/ariya/phantomjs/downloads/phantomjs-1.9.8-linux-i686.tar.bz2

tar jxvf phantomjs-1.9.8-linux-i686.tar.bz2
mv phantomjs-1.9.8-linux-i686 /usr/local/src/phantomjs
ln -sf /usr/local/src/phantomjs/bin/phantomjs /usr/local/bin/phantomjs
chmod a+x phantomjs 

5. Install MySQL-python

Python uses MySQL database, and this module needs to be installed.
download link:
https://pypi.python.org/packages/source/M/MySQL-python/MySQL-python-1.2.5.zip

Install after decompression: python setup.py install

pip install mysql-python

6. Install MySQL

download link:
http://dev.mysql.com/get/Downloads/MySQL-5.6/MySQL-5.6.27-1.linux_glibc2.5.x86_64.rpm-bundle.tar
Install after decompression and install using rpm command

rpm MySQL-server-5.6.27-1.linux_glibc2.5.x86_64.rpm
rpm MySQL-client-5.6.27-1.linux_glibc2.5.x86_64.rpm
rpm MySQL-devel-5.6.27-1.linux_glibc2.5.x86_64.rpm
rpm MySQL-shared-5.6.27-1.linux_glibc2.5.x86_64.rpm

4 Deployment notes##

1、 Don't forget to modify environment variables

2、 After installing MySQL, you need to modify the default character set to utf8