Monday, May 25, 2009

Nginx + PHPFPM + Memcache + Mysql + XCache + Monit + Logrotate

Step by Step
============
1)install debian (in case of EC2 - grab a public debian image and create a instance using it)
2)apt-get update
3)apt-get install openssh-server
4)apt-get install vim screen rsync rcconf telnet emacs locate unzip
5)set up .bashrc and  .vimrc
6)apt-get install sysstat subversion sudo
7)add new users and configure sudo
8)aptitude install postfix
9)apt-get install mysql-server mysql-client
10) mysql_secure_installation 
11)aptitude install build-essential libtool libltdl3-dev libgd-dev libmcrypt-dev libxml2-dev libmysqlclient15-dev flex m4 awk automake autoconf bison make libbz2-dev libpcre3-dev libssl-dev zlib1g-dev re2c
12)install php with fpm, curl, mysql

  Install PHP with FPM, CURL, Mysql etc
  =============
  1)sudo apt-get install libcurl4-openssl-dev(if 4 is not found, intall libcurl3-openssol-dev)
  2)sudo apt-get install libjpeg62
  3)sudo apt-get install libjpeg62-dev
  4)sudo apt-get install libpng12-dev
  5)cd /usr/local/src/
  6)wget http://in.php.net/distributions/php-5.2.8.tar.bz2
  7)wget http://php-fpm.anight.org/downloads/head/php-5.2.8-fpm-0.5.10.diff.gz
  8)gunzip php-5.2.8-fpm-0.5.10.diff.gz 
  9)tar -jxvpf php-5.2.8.tar.bz2
  10)patch -d php-5.2.8 -p1 <>
  11)./configure  --enable-fastcgi --enable-fpm --with-mcrypt --with-zlib --enable-mbstring --with-openssl --with-mysql --with-mysql-sock --with-gd --with-jpeg-dir=/usr/lib --enable-gd-native-ttf --without-sqlite --disable-pdo --disable-reflection --with-curl --with-mysqli=/usr/bin/mysql_config
  9)make
  9)make test
Test whether both mysql and mysqli are supported
  10)sudo make install


13)vi /usr/local/etc/php-fpm.conf 
:63
www-data
:66
www-data
You may want to add: /usr/local/lib/php to your php.ini include_path

14)install  xcache

   Install XCahce
   =============
   1)wget http://xcache.lighttpd.net/pub/Releases/1.2.2/xcache-1.2.2.tar.gz
   2)tar -zxvf xcache-1.2.2.tar.gz 
   3)cd xcache-1.2.2
   4)phpize
   5)./configure --with-php-config=/usr/local/bin/php-config --enable-xcache
   6)make install


15)install nginx

  Install Nginx
  ==============
  1)wget http://sysoev.ru/nginx/nginx-0.6.36.tar.gz
  2)tar xzvf nginx
  3)cd nginx
  4)./configure --sbin-path=/usr/local/sbin --with-http_ssl_module --without-mail_pop3_module --without-mail_imap_module --without-mail_smtp_module
  5)make
  6)make install


16)configure nginx

    Configure Nginx
    ==============
    1)cd /usr/local/nginx/
    2)vi conf/nginx.conf
    read for more details
    http://www.yawn.it/2008/04/30/nginx-php-php-fpm-on-debian-etch-40/
    Enable forwarding php requests to php-fpm by uncommenting the handling of .php files
    edit fastcgi_param as follows
    fastcgi_param SCRIPT_FILENAME /usr/local/nginx/html$fastcgi_script_name;
    https://calomel.org/nginx.html
    http://articles.slicehost.com/2008/5/15/ubuntu-hardy-nginx-configuration
    http://articles.slicehost.com/2008/5/16/ubuntu-hardy-nginx-virtual-hosts
    3)change log format to include response time 
    http://wiki.nginx.org/NginxHttpUpstreamModule#.24upstream_response_time



17)configure xcache

    Configure XCache
    ================
    1)vi /usr/local/lib/php.ini
    add the following contents 

    magic_quotes_gpc=0
    [xcache-common]
    zend_extension = /usr/local/lib/php/extensions/no-debug-non-zts-20060613/xcache.so
    [xcache]
    xcache.shm_scheme = "mmap"
    xcache.size = 64M
    date.timezone = “Europe/Rome”
    default_charset = “utf-8″


18)install memcache extension for php

  Install Memcache extension for php
  ===================================
  compile php with memcache support
  http://www.mail-archive.com/php-install@lists.php.net/msg15223.html
  1)wget http://pecl.php.net/get/memcache-2.2.5.tgz
  2)tar -zxvf memcache-2.2.5.tgz
  3)cd memcache-2.2.5
  4)phpize 
  5)./configure 
  6)make && make test
  7)$ sudo make install
  Installing shared extensions:     /usr/local/lib/php/extensions/no-debug-non-zts-20060613/

  8)vi /usr/local/lib/php.ini
  Add the following to php.ini
  extension=memcache.so
  display_startup_errors = On
  extension_dir = "/usr/local/lib/php/extensions/no-debug-non-zts-20060613/"

  9)run memcache test case in genlib/testcases/basic_memcache.php 


19)test nginx - add init scripts 

Test nginx add  init.d scripts
=======================================
1)nginx
2)php-fpm start
3)create a info.php file and test it.
4)sudo kill `cat /usr/local/nginx/logs/nginx.pid`
5)add init.d scripts - follow instructions in 
http://articles.slicehost.com/2007/10/19/debian-etch-adding-an-nginx-init-script/

20)test php-fpm - add init script 

Test php-fpm add init.d scripts
===============================
1)cd /etc/init.d/
2)ln -s /usr/local/sbin/php-fpm php-fpm
3)/usr/sbin/update-rc.d -f php-fpm defaults

21)optimize

optimize
=======
Find out the number of connections to phpfpm from nginx
$netstat -np | grep 127.0.0.1:9000 | grep -v TIME_WAIT
set appropraite value for variables in php-fpm.conf 
set request_terminate_timeout to 10s
set request_slowlog_timeout to 6s
set user and group to  www-data 
set max_children to 20
perform ab testing and optimize  worker_processes, worker_connections in nginx and max_children in php-fpm.conf

22)add logrotate scripts for nginx,php-fpm

       logrotate scripts
       ================
       1)create logroate conf for nginx and php-fpm - copy logroate files from conf_files dir 
       2)add entries for nginx and php-fpm logrotation at /etc/cron.daily/logrotate (not needed if /etc/logrotate.d is included in the logrotate.conf files, just make sure logrotate is called daily)

23)fine tune php-fpm.conf - enable slow query - set max execution time
24)fine tune mysql - enable slow query - log queries that don't use index
25)check if query cache is turned on in mysql - optimize it

Optimize Query Cache in mysql
============================
1) show variables like 'query%';
2) show status like 'qc%';
http://dev.mysql.com/tech-resources/articles/mysql-query-cache.html

26)install monit

  Monit
  =====
  1)apt-get install monit
  2)vim /etc/monit/monitrc  - configure it
  3)create /etc/monit/monit.d directory - add sshd,nginx monit config files here
  4)add include /etc/monit/monit.d/* option in /etc/monit/monitrc
  3)monit -t
  4)vim /etc/default/monit  set startup=1
  5)test monit




2 comments:

Anonymous said...

Beautifully done! Wish I'd found this much sooner. :)

--Chris Cortese

Anonymous said...

For high load sites you can also add:

MUST DO Kernal modifications
nano /etc/sysctl.conf

add the following lines

# These ensure that TIME_WAIT ports either get reused or closed fast.
net.ipv4.tcp_fin_timeout = 1
net.ipv4.tcp_tw_recycle = 1
# TCP memory
net.core.rmem_max = 16777216
net.core.rmem_default = 16777216
net.core.netdev_max_backlog = 262144
net.core.somaxconn = 262144
net.ipv4.tcp_syncookies = 1
net.ipv4.tcp_max_orphans = 262144
net.ipv4.tcp_max_syn_backlog = 262144
net.ipv4.tcp_synack_retries = 2
net.ipv4.tcp_syn_retries = 2
# For Large File Hosting Servers
#net.core.wmem_max = 1048576
#net.ipv4.tcp_wmem = 4096 87380 524288