Monday, May 25, 2009

Nginx + PHPFPM + Memcache + Mysql + XCache + Monit + Logrotate

Step by Step
1)install debian (in case of EC2 - grab a public debian image and create a instance using it)
2)apt-get update
3)apt-get install openssh-server
4)apt-get install vim screen rsync rcconf telnet emacs locate unzip
5)set up .bashrc and  .vimrc
6)apt-get install sysstat subversion sudo
7)add new users and configure sudo
8)aptitude install postfix
9)apt-get install mysql-server mysql-client
10) mysql_secure_installation 
11)aptitude install build-essential libtool libltdl3-dev libgd-dev libmcrypt-dev libxml2-dev libmysqlclient15-dev flex m4 awk automake autoconf bison make libbz2-dev libpcre3-dev libssl-dev zlib1g-dev re2c
12)install php with fpm, curl, mysql

  Install PHP with FPM, CURL, Mysql etc
  1)sudo apt-get install libcurl4-openssl-dev(if 4 is not found, intall libcurl3-openssol-dev)
  2)sudo apt-get install libjpeg62
  3)sudo apt-get install libjpeg62-dev
  4)sudo apt-get install libpng12-dev
  5)cd /usr/local/src/
  8)gunzip php-5.2.8-fpm-0.5.10.diff.gz 
  9)tar -jxvpf php-5.2.8.tar.bz2
  10)patch -d php-5.2.8 -p1 <>
  11)./configure  --enable-fastcgi --enable-fpm --with-mcrypt --with-zlib --enable-mbstring --with-openssl --with-mysql --with-mysql-sock --with-gd --with-jpeg-dir=/usr/lib --enable-gd-native-ttf --without-sqlite --disable-pdo --disable-reflection --with-curl --with-mysqli=/usr/bin/mysql_config
  9)make test
Test whether both mysql and mysqli are supported
  10)sudo make install

13)vi /usr/local/etc/php-fpm.conf 
You may want to add: /usr/local/lib/php to your php.ini include_path

14)install  xcache

   Install XCahce
   2)tar -zxvf xcache-1.2.2.tar.gz 
   3)cd xcache-1.2.2
   5)./configure --with-php-config=/usr/local/bin/php-config --enable-xcache
   6)make install

15)install nginx

  Install Nginx
  2)tar xzvf nginx
  3)cd nginx
  4)./configure --sbin-path=/usr/local/sbin --with-http_ssl_module --without-mail_pop3_module --without-mail_imap_module --without-mail_smtp_module
  6)make install

16)configure nginx

    Configure Nginx
    1)cd /usr/local/nginx/
    2)vi conf/nginx.conf
    read for more details
    Enable forwarding php requests to php-fpm by uncommenting the handling of .php files
    edit fastcgi_param as follows
    fastcgi_param SCRIPT_FILENAME /usr/local/nginx/html$fastcgi_script_name;
    3)change log format to include response time

17)configure xcache

    Configure XCache
    1)vi /usr/local/lib/php.ini
    add the following contents 

    zend_extension = /usr/local/lib/php/extensions/no-debug-non-zts-20060613/
    xcache.shm_scheme = "mmap"
    xcache.size = 64M
    date.timezone = “Europe/Rome”
    default_charset = “utf-8″

18)install memcache extension for php

  Install Memcache extension for php
  compile php with memcache support
  2)tar -zxvf memcache-2.2.5.tgz
  3)cd memcache-2.2.5
  6)make && make test
  7)$ sudo make install
  Installing shared extensions:     /usr/local/lib/php/extensions/no-debug-non-zts-20060613/

  8)vi /usr/local/lib/php.ini
  Add the following to php.ini
  display_startup_errors = On
  extension_dir = "/usr/local/lib/php/extensions/no-debug-non-zts-20060613/"

  9)run memcache test case in genlib/testcases/basic_memcache.php 

19)test nginx - add init scripts 

Test nginx add  init.d scripts
2)php-fpm start
3)create a info.php file and test it.
4)sudo kill `cat /usr/local/nginx/logs/`
5)add init.d scripts - follow instructions in

20)test php-fpm - add init script 

Test php-fpm add init.d scripts
1)cd /etc/init.d/
2)ln -s /usr/local/sbin/php-fpm php-fpm
3)/usr/sbin/update-rc.d -f php-fpm defaults


Find out the number of connections to phpfpm from nginx
$netstat -np | grep | grep -v TIME_WAIT
set appropraite value for variables in php-fpm.conf 
set request_terminate_timeout to 10s
set request_slowlog_timeout to 6s
set user and group to  www-data 
set max_children to 20
perform ab testing and optimize  worker_processes, worker_connections in nginx and max_children in php-fpm.conf

22)add logrotate scripts for nginx,php-fpm

       logrotate scripts
       1)create logroate conf for nginx and php-fpm - copy logroate files from conf_files dir 
       2)add entries for nginx and php-fpm logrotation at /etc/cron.daily/logrotate (not needed if /etc/logrotate.d is included in the logrotate.conf files, just make sure logrotate is called daily)

23)fine tune php-fpm.conf - enable slow query - set max execution time
24)fine tune mysql - enable slow query - log queries that don't use index
25)check if query cache is turned on in mysql - optimize it

Optimize Query Cache in mysql
1) show variables like 'query%';
2) show status like 'qc%';

26)install monit

  1)apt-get install monit
  2)vim /etc/monit/monitrc  - configure it
  3)create /etc/monit/monit.d directory - add sshd,nginx monit config files here
  4)add include /etc/monit/monit.d/* option in /etc/monit/monitrc
  3)monit -t
  4)vim /etc/default/monit  set startup=1
  5)test monit


Anonymous said...

Beautifully done! Wish I'd found this much sooner. :)

--Chris Cortese

Anonymous said...

For high load sites you can also add:

MUST DO Kernal modifications
nano /etc/sysctl.conf

add the following lines

# These ensure that TIME_WAIT ports either get reused or closed fast.
net.ipv4.tcp_fin_timeout = 1
net.ipv4.tcp_tw_recycle = 1
# TCP memory
net.core.rmem_max = 16777216
net.core.rmem_default = 16777216
net.core.netdev_max_backlog = 262144
net.core.somaxconn = 262144
net.ipv4.tcp_syncookies = 1
net.ipv4.tcp_max_orphans = 262144
net.ipv4.tcp_max_syn_backlog = 262144
net.ipv4.tcp_synack_retries = 2
net.ipv4.tcp_syn_retries = 2
# For Large File Hosting Servers
#net.core.wmem_max = 1048576
#net.ipv4.tcp_wmem = 4096 87380 524288