Testing your webapp using Selenium and PHPUnit

For a long time we have been testing our websites manually. Finally, we have decided to put in some effort to automate part of this testing. After a little bit of research, we found that Selenium could test any website in any set of browser-operating system combinations. There are some other equivalent tools, however selenium seems to be the most robust.

Selenium has many sub-projects. Selenium IDE is the best place for novices to start. Selenium IDE is a Firefox plugin that lets you record events in Firefox and replay them. Events can be like entering text in an input box. You can record a bunch of events like entering username, entering password, clicking on submit, etc. and replay them. Selenium has client libraries in almost all prevailing languages. You could record set of events in Selenium IDE and export the scripts to any of the supported languages. Java, Python, PHP, Ruby are some of the supported languages.

Replaying a script in Selenium IDE will execute the script inside Firefox. If you wish the execute the script in other browsers, then you will have to use a tool called Selenium RC. The acronym RC stands for remote client. Selenium RC is a jar file which can invoke any browser of your choice and execute the commands that you pass to it.

I have several instances of Selenium RC running, one in Windows, one in Mac and one in Linux. When I execute the PHP test scripts, they connect to Selenium RC running in various machines over the network and execute the scripts on a specific browser. This way you could test your website on all major browser and operating systems.

Forking in PHP on Mac OSX using pcntl_fork

1)sudo port selfupdate

2)sudo port update outdated

3)sudo port install php5-pcntl

4)Try the following example

$pid = pcntl_fork();

if($pid) {

// parent process runs what is here

print "parent\n";

}

else {

// child process runs what is here

print "child\n";

}

?>

Init script to mount EBS and attach elastic IPs to EC2 instances

Modified init script to mount EBS automatically to your EC2 instances.  It also dynamically attaches your instances to an elastic ip.  You have to provide both EBS volume id and elastic ip as userdata separated by pipe symbol for this script to work.  Sample userdata 

vol-12345678|171.213.234.123

You have to use 

update-rc.d mountec2vol start 47 S . stop 35 0 6 . start 37 6 .

to add this script to startup. This script also additional code to mount mysql data directory and nginx directories from EBS. You can skip that part.



#! /bin/sh
#
# /etc/init.d/mountec2vol
#
# chkconfig: 234 20 50
# description: Assigns an EC2 EBS Volume to a device and mounts the device
#
# To add this as a service run:
# /sbin/chkconfig --add mountec2vol
#
# VARS
#
DEV="/dev/sdh"
MOUNT_POINT="/vol"
EC2_LOG='/var/log/ec2.log';
SELF=$(cd $(dirname $0); pwd -P)/$(basename $0)

export JAVA_HOME=/usr/lib/jvm/java-6-sun-1.6.0.13/
export EC2_PRIVATE_KEY=/root/.ec2/pk.pem
export EC2_CERT=/root/.ec2/cert.pem
export EC2_HOME=/usr/local/src/ec2-api-tools-1.3-36506/
PATH=$PATH:$HOME/bin:$EC2_HOME/bin
MAX_TRIES=60

attach_ebs()
{
/bin/echo "=====================================" | tee -a $EC2_LOG
/bin/echo "Date: "`date` | tee -a $EC2_LOG;
/bin/echo "Attaching Elastic Block Store Volumes." | tee -a $EC2_LOG
ec2-attach-volume $VOL -i $INSTANCE -d $DEV 1>&2 2>>$EC2_LOG
CTR=0
while [ ! -e "$DEV" ]; do
/bin/sleep 1
CTR=`expr $CTR + 1`
if [ $CTR -eq $MAX_TRIES ]
then
/bin/echo "WARNING: Cannot attach volume $VOL to $DEV -- Giving up after $MAX_TRIES attempts" | tee -a $EC2_LOG
exit 1
fi
done
}
mount_ebs()
{
/bin/echo "Checking if $MOUNT_POINT is present" | tee -a $EC2_LOG
if [ ! -d $MOUNT_POINT ]; then
mkdir $MOUNT_POINT 1>&2 2>>$EC2_LOG
fi
/bin/echo "Mounting EBS to /vol " | tee -a $EC2_LOG
/bin/mount $DEV $MOUNT_POINT 1>&2 2>>$EC2_LOG
}
mount_mysql_dirs()
{
/bin/echo "Mounting mysql dirs" | tee -a $EC2_LOG
/bin/mount /etc/mysql/ 1>&2 2>>$EC2_LOG
/bin/mount /var/lib/mysql/ 1>&2 2>>$EC2_LOG
/bin/mount /var/log/mysql/ 1>&2 2>>$EC2_LOG
/bin/mount /usr/local/nginx/ 1>&2 2>>$EC2_LOG
}
unmount_mysql_dirs()
{
set +e;
/bin/echo "stopping mysql" | tee -a $EC2_LOG
/etc/init.d/mysql stop;
set -e;
/bin/echo "Unmounting mysql dirs" | tee -a $EC2_LOG
/bin/umount /etc/mysql/ 1>&2 2>>$EC2_LOG
/bin/umount /var/lib/mysql/ 1>&2 2>>$EC2_LOG
/bin/umount /var/log/mysql/ 1>&2 2>>$EC2_LOG
}
unmount_nginx_dirs()
{
set +e;
/bin/echo "stopping nginx" | tee -a $EC2_LOG
/etc/init.d/nginx stop;
set -e;
/bin/echo "Unmounting nginx dirs" | tee -a $EC2_LOG
/bin/umount /usr/local/nginx/ 1>&2 2>>$EC2_LOG
}
unmount_ebs()
{
/bin/echo "Unmounting Elastic Block Store Volumes." | tee -a $EC2_LOG
/bin/umount $MOUNT_POINT 1>&2 2>>$EC2_LOG
}
detach_ebs()
{
/bin/echo "Detaching Elastic Block Store Volumes." | tee -a $EC2_LOG
ec2-detach-volume $VOL 1>&2 2>>$EC2_LOG
}
populate_aws_data()
{
/bin/echo "Populating AWS data" | tee -a $EC2_LOG
INSTANCE=`curl http://169.254.169.254/latest/meta-data/instance-id 2> /dev/null`
USERDATA=`curl http://169.254.169.254/1.0/user-data 2>/dev/null`;
}
parse_user_args()
{
/bin/echo "Parsing user_args " | tee -a $EC2_LOG
VOL=${USERDATA%|*};
IP=${USERDATA#*|};
echo "usedata vol ip";
echo $USERDATA;
echo $VOL;
echo $IP;
}
bind_ip()
{
/bin/echo "Associating ip address: $IP to $INSTANCE" | tee -a $EC2_LOG
ec2-associate-address -i $INSTANCE $IP;
}

# start/stop functions for OS

case "$1" in

start)
populate_aws_data;
parse_user_args;
bind_ip;
attach_ebs;
mount_ebs;
mount_mysql_dirs;
sh -x /vol/ec2_stuff/everytime_by_mountec2.sh;
;;
stop)
unmount_mysql_dirs;
unmount_nginx_dirs;
unmount_ebs;
detach_ebs;
;;
restart)
set +e;
$SELF stop;
set -e;
sleep 60;
$SELF start
;;
*)
echo "Usage: $0 {start|stop|restart}" | tee -a $EC2_LOG
exit 1

esac

exit 0




Auto Mounting EBS on EC2

I assume that you have already created a EC2 instance (On top of your custom AMI),  EBS volume (both in the same availability zone) . Below are some instructions to automount EBS on EC2. 

AutoMount Init Script

===========================

http://developer.amazonwebservices.com/connect/thread.jspa?threadID=25071&start=0&tstart=0

1)apt-get install sun-java6-jre sun-java6-bin sun-java6-source

2)cd /usr/local/src/

3)wget http://developer.amazonwebservices.com/connect/servlet/JiveServlet/download/30-25071-102530-2022/mountec2vol

4)wget http://s3.amazonaws.com/ec2-downloads/ec2-api-tools.zip

unzip ec2-api-tools.zip

5)edit mountec2vol

set JAVA_HOME=/usr/lib/jvm/java-6-sun-1.6.0.13/  or equivalent

set EC2_PRIVATE_KEY EC2_CERT

set EC2_HOME=/usr/local/src/ec2-api-tools-1.3-36506/

set VOL and DEV

6)cp mountec2vol /etc/init.d/

7)chmod +x /etc/init.d/mountec2vol

8)test  mountec2vol

/etc/init.d/mountec2vol start

/etc/init.d/mountec2vol stop

9)update-rc.d mountec2vol start 15 2 3 4 5 . start 47 S . start 35 0 1 6 .

10)Bundle the new image

ec2-bundle-vol -d /mnt -k /mnt/pk-balbal.pem -c /mnt/cert-balhbalh.pem -u userid  -r i386  -p automountingebs

11)upload the new image

ec2-upload-bundle  --url https://s3.amazonaws.com/subdir/ -b yourbucket  -m /mnt/automountingebs.xml -a accessid -s yoursecret

12)register image

ec2-register yourbucket /subdir/automountingebs.xml

Some additional info to create EBS 

=========================

On local machine

================

ec2-create-volume -z us-east-1c -s 10

ec2-describe-volumes vol-VVVV1111

ec2-attach-volume -d /dev/sdh -i i-IIII1111 vol-VVVV1111

On EC2 Instance

===============

apt-get install -y xfsprogs

modprobe xfs

mkfs.xfs /dev/sdh

echo "/dev/sdh /vol xfs noatime 0 0" >> /etc/fstab

mkdir /vol

mount /vol

http://developer.amazonwebservices.com/connect/entry.jspa?externalID=1663

AutoMount

=========

http://www.ioncannon.net/system-administration/199/automounting-amazon-ebs-volumes-on-ec2-instances/

Nginx + PHPFPM + Memcache + Mysql + XCache + Monit + Logrotate

Step by Step

============

1)install debian (in case of EC2 - grab a public debian image and create a instance using it)

2)apt-get update

3)apt-get install openssh-server

4)apt-get install vim screen rsync rcconf telnet emacs locate unzip

5)set up .bashrc and  .vimrc

6)apt-get install sysstat subversion sudo

7)add new users and configure sudo

8)aptitude install postfix

9)apt-get install mysql-server mysql-client

10) mysql_secure_installation 

11)aptitude install build-essential libtool libltdl3-dev libgd-dev libmcrypt-dev libxml2-dev libmysqlclient15-dev flex m4 awk automake autoconf bison make libbz2-dev libpcre3-dev libssl-dev zlib1g-dev re2c

12)install php with fpm, curl, mysql

  Install PHP with FPM, CURL, Mysql etc

  =============

  1)sudo apt-get install libcurl4-openssl-dev(if 4 is not found, intall libcurl3-openssol-dev)

  2)sudo apt-get install libjpeg62

  3)sudo apt-get install libjpeg62-dev

  4)sudo apt-get install libpng12-dev

  5)cd /usr/local/src/

  6)wget http://in.php.net/distributions/php-5.2.8.tar.bz2

  7)wget http://php-fpm.anight.org/downloads/head/php-5.2.8-fpm-0.5.10.diff.gz

  8)gunzip php-5.2.8-fpm-0.5.10.diff.gz 

  9)tar -jxvpf php-5.2.8.tar.bz2

  10)patch -d php-5.2.8 -p1 <>

  11)./configure  --enable-fastcgi --enable-fpm --with-mcrypt --with-zlib --enable-mbstring --with-openssl --with-mysql --with-mysql-sock --with-gd --with-jpeg-dir=/usr/lib --enable-gd-native-ttf --without-sqlite --disable-pdo --disable-reflection --with-curl --with-mysqli=/usr/bin/mysql_config

  9)make

  9)make test

Test whether both mysql and mysqli are supported

  10)sudo make install

13)vi /usr/local/etc/php-fpm.conf 

:63

www-data

:66

www-data

You may want to add: /usr/local/lib/php to your php.ini include_path

14)install  xcache

   Install XCahce

   =============

   1)wget http://xcache.lighttpd.net/pub/Releases/1.2.2/xcache-1.2.2.tar.gz

   2)tar -zxvf xcache-1.2.2.tar.gz 

   3)cd xcache-1.2.2

   4)phpize

   5)./configure --with-php-config=/usr/local/bin/php-config --enable-xcache

   6)make install

15)install nginx

  Install Nginx

  ==============

  1)wget http://sysoev.ru/nginx/nginx-0.6.36.tar.gz

  2)tar xzvf nginx

  3)cd nginx

  4)./configure --sbin-path=/usr/local/sbin --with-http_ssl_module --without-mail_pop3_module --without-mail_imap_module --without-mail_smtp_module

  5)make

  6)make install

16)configure nginx

    Configure Nginx

    ==============

    1)cd /usr/local/nginx/

    2)vi conf/nginx.conf

    read for more details

    http://www.yawn.it/2008/04/30/nginx-php-php-fpm-on-debian-etch-40/

    Enable forwarding php requests to php-fpm by uncommenting the handling of .php files

    edit fastcgi_param as follows

    fastcgi_param SCRIPT_FILENAME /usr/local/nginx/html$fastcgi_script_name;

    https://calomel.org/nginx.html

    http://articles.slicehost.com/2008/5/15/ubuntu-hardy-nginx-configuration

    http://articles.slicehost.com/2008/5/16/ubuntu-hardy-nginx-virtual-hosts

    3)change log format to include response time 

    http://wiki.nginx.org/NginxHttpUpstreamModule#.24upstream_response_time

17)configure xcache

    Configure XCache

    ================

    1)vi /usr/local/lib/php.ini

    add the following contents 

    magic_quotes_gpc=0

    [xcache-common]

    zend_extension = /usr/local/lib/php/extensions/no-debug-non-zts-20060613/xcache.so

    [xcache]

    xcache.shm_scheme = "mmap"

    xcache.size = 64M

    date.timezone = “Europe/Rome”

    default_charset = “utf-8″

18)install memcache extension for php

  Install Memcache extension for php

  ===================================

  compile php with memcache support

  http://www.mail-archive.com/php-install@lists.php.net/msg15223.html

  1)wget http://pecl.php.net/get/memcache-2.2.5.tgz

  2)tar -zxvf memcache-2.2.5.tgz

  3)cd memcache-2.2.5

  4)phpize 

  5)./configure 

  6)make && make test

  7)$ sudo make install

  Installing shared extensions:     /usr/local/lib/php/extensions/no-debug-non-zts-20060613/

  8)vi /usr/local/lib/php.ini

  Add the following to php.ini

  extension=memcache.so

  display_startup_errors = On

  extension_dir = "/usr/local/lib/php/extensions/no-debug-non-zts-20060613/"

  9)run memcache test case in genlib/testcases/basic_memcache.php 

19)test nginx - add init scripts 

Test nginx add  init.d scripts

=======================================

1)nginx

2)php-fpm start

3)create a info.php file and test it.

4)sudo kill `cat /usr/local/nginx/logs/nginx.pid`

5)add init.d scripts - follow instructions in 

http://articles.slicehost.com/2007/10/19/debian-etch-adding-an-nginx-init-script/

20)test php-fpm - add init script 

Test php-fpm add init.d scripts

===============================

1)cd /etc/init.d/

2)ln -s /usr/local/sbin/php-fpm php-fpm

3)/usr/sbin/update-rc.d -f php-fpm defaults

21)optimize

optimize

=======

Find out the number of connections to phpfpm from nginx

$netstat -np | grep 127.0.0.1:9000 | grep -v TIME_WAIT

set appropraite value for variables in php-fpm.conf 

set request_terminate_timeout to 10s

set request_slowlog_timeout to 6s

set user and group to  www-data 

set max_children to 20

perform ab testing and optimize  worker_processes, worker_connections in nginx and max_children in php-fpm.conf

22)add logrotate scripts for nginx,php-fpm

       logrotate scripts

       ================

       1)create logroate conf for nginx and php-fpm - copy logroate files from conf_files dir 

       2)add entries for nginx and php-fpm logrotation at /etc/cron.daily/logrotate (not needed if /etc/logrotate.d is included in the logrotate.conf files, just make sure logrotate is called daily)

23)fine tune php-fpm.conf - enable slow query - set max execution time

24)fine tune mysql - enable slow query - log queries that don't use index

25)check if query cache is turned on in mysql - optimize it

Optimize Query Cache in mysql

============================

1) show variables like 'query%';

2) show status like 'qc%';

http://dev.mysql.com/tech-resources/articles/mysql-query-cache.html

26)install monit

  Monit

  =====

  1)apt-get install monit

  2)vim /etc/monit/monitrc  - configure it

  3)create /etc/monit/monit.d directory - add sshd,nginx monit config files here

  4)add include /etc/monit/monit.d/* option in /etc/monit/monitrc

  3)monit -t

  4)vim /etc/default/monit  set startup=1

  5)test monit

vim tip for copying code from webpage

Sometimes copying code from webpage to vim can be painful. You might end up loosing all the newlines. So I prefer to copy (from webpages) using html source. If you use html source, you will have to replace &gt, &lt .. and lot more to <,> and stuff.

The following will do it automatically for you.
:%!perl -pne 'use HTML::Entities; $_=decode_entities($_)'

Thanks to godlygeek from #vim .

Wifi at Changi Airport

I just checked out the free internet terminals in Changi Airport. Few of them run secured version of Internet Explorer. I was also able to use "Wireless@SG", the free island wide wireless at Changi. The speed was amazing. I had to download JavaDoc for JDK 1.5. It completed in 12 minutes. If you are not aware, JavaDoc for JDK 1.5 is 150MB.

IT Security Awareness Day 2007 at Singapore Poly

I gave a talk on WebSecurity at Singapore Poly at the IT Security Awareness Day Event. Around 300 students attended the event, and most of them were school kids. I mostly spoke about Johnny's Google Hacking Database, and little bit about XSS. Justin Lister from InterSect Singapore also spoke at the event. He organized a simple game to explain Phishing. Thanks to Samson, and Cecil for organizing the event.

Lunch with David Axmark, Co-founder of MySQL

David Axmark, the Co-founder of MySQL was in Singapore today. The local MySQL interest group meet along with David at GeekTerminal for lunch. David patiently answered variety of questions relating to BitKeeper, GIT, SpikeSource's testing framework , Enterprise version of MySQL, SQLite, India, SQL Injection, etc. Thanks to Wong for organizing the event.

UandMe

Social networking has grown a big way. FaceBook has taken social networking to a different level. UandMe is a mobile social networking application build by Locatrix. UandMe goes one step further, by integrating with GPS. With UandMe, you can know where your friend is, find a near by restaurant to meet with your friend, create alerts for various events, etc. Location based advertisement is also possible when you integrate with GPS. I guess UandMe will change the way people meet, dine, work etc. I was privileged to meet with Mark White, the founder of Locatrix. He was also kind enough to demonstrate UandMe.

Security Related Groups in Singapore

1) LinkedIn Group for OWASP
2) OWASP-Singapore mailing list
3) Security Meetup - Singapore == Next meeting on 13th Dec 2007.

OWASP Meeting at GeekTerminal Singapore

Two days back, I attended the second OWASP-Singapore (Open Web Application Security Project) meeting at Geek Terminal. Six people turned out and we had three different presentations. Michael Boman spoke about "Overcoming USB (In) Security". He also demonstrated Lock Picking. Wong from Resolve spoke about ILDP Strategies. Later we discussed about MMORPG hacking. We discussed about SecondLife hacking, gaming client hacking, gaming logic exploitation, etc . We also discussed about developing a framework to test and prevent Game Hacking. Later we discussed about websecurity. The whole meeting went for around 3 hours. I wish more people join.

Singleton Vs Static

Singleton, MVC and Factory patterns are the design patterns I use very often. I also use static classes, more often than Singleton, but static classes are not design patterns. For every implementation of Singleton, one can come out with an alternate implementation with static classes. Lot of critics argue that Singleton is not the right way to program, and it is an anti-pattern. Singleton patterns can be considered as substitutions for global variables, and they are most useful when you want to use the same instance all over your program. Using one object ( say of type Class A) to manage instances of a different class (say Class B) , and providing all functionalities using the objects of Class B is obviously a better coding practice. This is a better replacement of Singleton pattern, compared with static classes. Here is an interesting discussing I'd with Cybereal at #java@irc.freenode.net. Cybereal explains why Singleton is a bad wrt to dependency injection.


(12:03:11 AM) codeshepherd: cybereal .. Aren't Singleton the best suitable implementations for Database connection pooling ?
(12:03:43 AM) codeshepherd: In a more general way... Singletons are best suitable to manager global variables..
(12:03:53 AM) cybereal: singletons *ARE* global variables, basically
(12:03:56 AM) cybereal: and that's why they are bad
(12:04:24 AM) cybereal: codeshepherd: you shouldn't need to use a singleton to use the same resource across a program
(12:04:50 AM) cybereal: but because people are lazy about typing and passing on dependency information to instantiated objects, you find they will lean towards singletons to do little more than save some keystrokes
(12:05:01 AM) cybereal: It's really hard to justify it for any other reason
(12:05:41 AM) cybereal: codeshepherd: despite being an anti-pattern, it does get used frequently so you'll still need to know how to deal with it, and sometimes to work around them
(12:06:02 AM) cybereal: at least in java if you really need to you can bury a singleton in a separate classloader instance
(12:06:13 AM) codeshepherd: So how will you implement a Database connection pool? Just a static class ?
(12:06:30 AM) cybereal: no, a regular class that you pass around to all the new instances of other classes that need to know about it
(12:06:44 AM) cybereal: like handing out invitations to a party
(12:06:58 AM) cybereal: you don't hand out houses with those invitations, and people aren't expected to figure out the address themselves
(12:07:06 AM) codeshepherd: then.. some dumb guy who comes tomorrow will start creating instances of the regular class..
(12:07:13 AM) cybereal: it won't matter
(12:07:24 AM) cybereal: but even so, if he's dumb he's gonna do dumb things :)
(12:07:34 AM) cybereal: you shouldn't write bad code just to try and protect from dumb coders
(12:07:54 AM) cybereal: but the point is you should be able to have more than one in the same vm, and not have them hurt each other
(12:07:56 AM) codeshepherd: It is really a debatable issue..
(12:08:54 AM) cybereal: codeshepherd: you probably won't really see it this way until it eventually comes back to bite you in the ass as it has me :)
(12:09:08 AM) codeshepherd: I still agree with the fact that.. you let one class (A) provide all functions.. and another class manage instances of Class A.. .. that looks more clean..
(12:09:33 AM) cybereal: you're making a mistake of thinking a class should ever be in charge of anything
(12:09:37 AM) cybereal: INSTANCES of classes should
(12:09:43 AM) cybereal: classes are blue prints
(12:09:54 AM) codeshepherd: hmm.. ok....
cybereal cybereal
(12:10:19 AM) codeshepherd: interesting argument.. thanks a lot cybereal
(12:10:28 AM) cybereal: nothing stops you, as I already described, from sharing a single instance of a class across your whole app
(12:10:50 AM) cybereal: you get what you want from a singleton without the limiting factors, and all it takes is a little longer parameter list in constructors :)
(12:11:18 AM) codeshepherd: ok.. I guess I will realize it better when it bites me back..
(12:11:24 AM) cybereal: hehe ok

GMail's POP and SMTP service.

GMail's POP and SMTP service have been giving lot of troubles lately. Very often, I'm not able to send out mails via SMTP, and downloading new mails occurs at snails pace. Many others have reported the same problem, and I hope Google fixes it very soon. Email is an essential service, and use web interface for checking emails can be tedious if you have dozens of email accounts with a variety of service providers.

Red Black Tree

TreeSet is an implementation of Red-Black Tree in Java. It implements the Set and Collection interface. Here is a short yet excellent video on adding new nodes to TreeSet.

Barcamp - Brisbane, Australia.

I'm planning to fly a bit early for OSDC, so that I can also attend Brisbane Barcamp. It is on Saturday, 24/11/2007. I'm planning to talk there on websecurity. I'm also looking forward to meet David Novakovic, a Computational Linguistics researcher and geek from Gold Coast. He will also speak about his new technology incubator. Click here for complete list of talk at Brisbane Barcamp. Let me know if anyone else is also planning to attend.

Design Patterns by Codingkriggs

Codingkriggs has excellent set of videos on Design Patterns. I wish he continues making videos :)

Linus on GIT

The concept of distributed source code management system is very interesting. IMHO GIT is a much better idea compared with Subversion. GIT takes care of the social factors involved in running a open source project. Here is a talk by Linus on GIT.

Interactive Shell

Interactive Shells for some popular languages.

IPython ==> Python (supports readline)
IRB ==> Ruby (supports readline)
php -a ==> PHP
BeanShell ==> Java (interpretor for compiled language?? )
SquareFree Shell ==> JavaScript

Installing readline library should help you auto complete and browse through history.

Yahoo! Pipes

I've been playing with Yahoo Pipes for a while now. Here is a Yahoo Pipe to read orkut scraps. You can choose to receive the scraps in RSS form.