Tuesday, May 26, 2009

Auto Mounting EBS on EC2

I assume that you have already created a EC2 instance (On top of your custom AMI),  EBS volume (both in the same availability zone) . Below are some instructions to automount EBS on EC2. 


AutoMount Init Script
===========================
http://developer.amazonwebservices.com/connect/thread.jspa?threadID=25071&start=0&tstart=0

1)apt-get install sun-java6-jre sun-java6-bin sun-java6-source
2)cd /usr/local/src/
3)wget http://developer.amazonwebservices.com/connect/servlet/JiveServlet/download/30-25071-102530-2022/mountec2vol
4)wget http://s3.amazonaws.com/ec2-downloads/ec2-api-tools.zip
unzip ec2-api-tools.zip
5)edit mountec2vol
set JAVA_HOME=/usr/lib/jvm/java-6-sun-1.6.0.13/  or equivalent
set EC2_PRIVATE_KEY EC2_CERT
set EC2_HOME=/usr/local/src/ec2-api-tools-1.3-36506/
set VOL and DEV
6)cp mountec2vol /etc/init.d/
7)chmod +x /etc/init.d/mountec2vol
8)test  mountec2vol
/etc/init.d/mountec2vol start
/etc/init.d/mountec2vol stop
9)update-rc.d mountec2vol start 15 2 3 4 5 . start 47 S . start 35 0 1 6 .
10)Bundle the new image
ec2-bundle-vol -d /mnt -k /mnt/pk-balbal.pem -c /mnt/cert-balhbalh.pem -u userid  -r i386  -p automountingebs
11)upload the new image
ec2-upload-bundle  --url https://s3.amazonaws.com/subdir/ -b yourbucket  -m /mnt/automountingebs.xml -a accessid -s yoursecret
12)register image
ec2-register yourbucket /subdir/automountingebs.xml



Some additional info to create EBS 
=========================
On local machine
================
ec2-create-volume -z us-east-1c -s 10
ec2-describe-volumes vol-VVVV1111
ec2-attach-volume -d /dev/sdh -i i-IIII1111 vol-VVVV1111

On EC2 Instance
===============
apt-get install -y xfsprogs
modprobe xfs
mkfs.xfs /dev/sdh

echo "/dev/sdh /vol xfs noatime 0 0" >> /etc/fstab

mkdir /vol
mount /vol

http://developer.amazonwebservices.com/connect/entry.jspa?externalID=1663


AutoMount
=========
http://www.ioncannon.net/system-administration/199/automounting-amazon-ebs-volumes-on-ec2-instances/

Monday, May 25, 2009

Nginx + PHPFPM + Memcache + Mysql + XCache + Monit + Logrotate

Step by Step
============
1)install debian (in case of EC2 - grab a public debian image and create a instance using it)
2)apt-get update
3)apt-get install openssh-server
4)apt-get install vim screen rsync rcconf telnet emacs locate unzip
5)set up .bashrc and  .vimrc
6)apt-get install sysstat subversion sudo
7)add new users and configure sudo
8)aptitude install postfix
9)apt-get install mysql-server mysql-client
10) mysql_secure_installation 
11)aptitude install build-essential libtool libltdl3-dev libgd-dev libmcrypt-dev libxml2-dev libmysqlclient15-dev flex m4 awk automake autoconf bison make libbz2-dev libpcre3-dev libssl-dev zlib1g-dev re2c
12)install php with fpm, curl, mysql

  Install PHP with FPM, CURL, Mysql etc
  =============
  1)sudo apt-get install libcurl4-openssl-dev(if 4 is not found, intall libcurl3-openssol-dev)
  2)sudo apt-get install libjpeg62
  3)sudo apt-get install libjpeg62-dev
  4)sudo apt-get install libpng12-dev
  5)cd /usr/local/src/
  6)wget http://in.php.net/distributions/php-5.2.8.tar.bz2
  7)wget http://php-fpm.anight.org/downloads/head/php-5.2.8-fpm-0.5.10.diff.gz
  8)gunzip php-5.2.8-fpm-0.5.10.diff.gz 
  9)tar -jxvpf php-5.2.8.tar.bz2
  10)patch -d php-5.2.8 -p1 <>
  11)./configure  --enable-fastcgi --enable-fpm --with-mcrypt --with-zlib --enable-mbstring --with-openssl --with-mysql --with-mysql-sock --with-gd --with-jpeg-dir=/usr/lib --enable-gd-native-ttf --without-sqlite --disable-pdo --disable-reflection --with-curl --with-mysqli=/usr/bin/mysql_config
  9)make
  9)make test
Test whether both mysql and mysqli are supported
  10)sudo make install


13)vi /usr/local/etc/php-fpm.conf 
:63
www-data
:66
www-data
You may want to add: /usr/local/lib/php to your php.ini include_path

14)install  xcache

   Install XCahce
   =============
   1)wget http://xcache.lighttpd.net/pub/Releases/1.2.2/xcache-1.2.2.tar.gz
   2)tar -zxvf xcache-1.2.2.tar.gz 
   3)cd xcache-1.2.2
   4)phpize
   5)./configure --with-php-config=/usr/local/bin/php-config --enable-xcache
   6)make install


15)install nginx

  Install Nginx
  ==============
  1)wget http://sysoev.ru/nginx/nginx-0.6.36.tar.gz
  2)tar xzvf nginx
  3)cd nginx
  4)./configure --sbin-path=/usr/local/sbin --with-http_ssl_module --without-mail_pop3_module --without-mail_imap_module --without-mail_smtp_module
  5)make
  6)make install


16)configure nginx

    Configure Nginx
    ==============
    1)cd /usr/local/nginx/
    2)vi conf/nginx.conf
    read for more details
    http://www.yawn.it/2008/04/30/nginx-php-php-fpm-on-debian-etch-40/
    Enable forwarding php requests to php-fpm by uncommenting the handling of .php files
    edit fastcgi_param as follows
    fastcgi_param SCRIPT_FILENAME /usr/local/nginx/html$fastcgi_script_name;
    https://calomel.org/nginx.html
    http://articles.slicehost.com/2008/5/15/ubuntu-hardy-nginx-configuration
    http://articles.slicehost.com/2008/5/16/ubuntu-hardy-nginx-virtual-hosts
    3)change log format to include response time 
    http://wiki.nginx.org/NginxHttpUpstreamModule#.24upstream_response_time



17)configure xcache

    Configure XCache
    ================
    1)vi /usr/local/lib/php.ini
    add the following contents 

    magic_quotes_gpc=0
    [xcache-common]
    zend_extension = /usr/local/lib/php/extensions/no-debug-non-zts-20060613/xcache.so
    [xcache]
    xcache.shm_scheme = "mmap"
    xcache.size = 64M
    date.timezone = “Europe/Rome”
    default_charset = “utf-8″


18)install memcache extension for php

  Install Memcache extension for php
  ===================================
  compile php with memcache support
  http://www.mail-archive.com/php-install@lists.php.net/msg15223.html
  1)wget http://pecl.php.net/get/memcache-2.2.5.tgz
  2)tar -zxvf memcache-2.2.5.tgz
  3)cd memcache-2.2.5
  4)phpize 
  5)./configure 
  6)make && make test
  7)$ sudo make install
  Installing shared extensions:     /usr/local/lib/php/extensions/no-debug-non-zts-20060613/

  8)vi /usr/local/lib/php.ini
  Add the following to php.ini
  extension=memcache.so
  display_startup_errors = On
  extension_dir = "/usr/local/lib/php/extensions/no-debug-non-zts-20060613/"

  9)run memcache test case in genlib/testcases/basic_memcache.php 


19)test nginx - add init scripts 

Test nginx add  init.d scripts
=======================================
1)nginx
2)php-fpm start
3)create a info.php file and test it.
4)sudo kill `cat /usr/local/nginx/logs/nginx.pid`
5)add init.d scripts - follow instructions in 
http://articles.slicehost.com/2007/10/19/debian-etch-adding-an-nginx-init-script/

20)test php-fpm - add init script 

Test php-fpm add init.d scripts
===============================
1)cd /etc/init.d/
2)ln -s /usr/local/sbin/php-fpm php-fpm
3)/usr/sbin/update-rc.d -f php-fpm defaults

21)optimize

optimize
=======
Find out the number of connections to phpfpm from nginx
$netstat -np | grep 127.0.0.1:9000 | grep -v TIME_WAIT
set appropraite value for variables in php-fpm.conf 
set request_terminate_timeout to 10s
set request_slowlog_timeout to 6s
set user and group to  www-data 
set max_children to 20
perform ab testing and optimize  worker_processes, worker_connections in nginx and max_children in php-fpm.conf

22)add logrotate scripts for nginx,php-fpm

       logrotate scripts
       ================
       1)create logroate conf for nginx and php-fpm - copy logroate files from conf_files dir 
       2)add entries for nginx and php-fpm logrotation at /etc/cron.daily/logrotate (not needed if /etc/logrotate.d is included in the logrotate.conf files, just make sure logrotate is called daily)

23)fine tune php-fpm.conf - enable slow query - set max execution time
24)fine tune mysql - enable slow query - log queries that don't use index
25)check if query cache is turned on in mysql - optimize it

Optimize Query Cache in mysql
============================
1) show variables like 'query%';
2) show status like 'qc%';
http://dev.mysql.com/tech-resources/articles/mysql-query-cache.html

26)install monit

  Monit
  =====
  1)apt-get install monit
  2)vim /etc/monit/monitrc  - configure it
  3)create /etc/monit/monit.d directory - add sshd,nginx monit config files here
  4)add include /etc/monit/monit.d/* option in /etc/monit/monitrc
  3)monit -t
  4)vim /etc/default/monit  set startup=1
  5)test monit