Thursday, June 04, 2009

XHProf 0.9.2 is out and ready for Mac OS X

Version 0.9.2 of Facebook's XHProf PHP module is out!

Thanks to Kannan from Facebook's dev team the patch for OS X is now a part of the official release.

Go get it Mac users! It compiles on OS X perfectly.

Two weeks ago I wrote that I've made a patch for version 0.9.1 to build under OS X.
Then I decided to post a request on XHProf's bug track and supply the patch there.
Kannan (leader of the project) answered and we collaborated in the next week for applying the patch in the next official release of XHProf.

You can read more about the whole story on PECL.

Thank you, Kannan, for your time and efforts!

I am glad that version 0.9.2 has a port for Mac OS X and I was able to help for that.

Sunday, May 17, 2009

Facebook's XHProf PHP module patch for Mac OS X Leopard Build

*) note (05 Jun 2009): Thanks to Kannan from Facebook dev team the patch is now available as part of the official 0.9.2 release. You can download it from PECL.

Download full patched XHProf 0.9.1 for MAC OS X Leopard
Bug submitted on PECL.

Full story
-----------

Yesterday (15 May 2009) I ran into XHProf module made by Facebook and published open source in March 2009.
Right after I read about its great features I was sure I had to try it and eventually put it on production servers. I was looking for such kind of PHP profiler with stats since 2005 when I found XDebug, which is really cool while developing but I wouldn't put it on a live server with high load like Tupalka.com's.

Eventually later that day our server hosting Tupalka.com was hit by a huge traffic and it was very loaded from time to time when the eAccelerator caches expires and most of the heavy mysql queries have to be ran once per 15 mins. That force me to try the XHProf.
Yup, I am sure that the problem was in the queries then why I was going for the XHProf? Good question. Curiosity.

These very moments are the once that push me to find and try different and new tools for optimization of web application/web sites.
So I was desparate to put XHProf on the production server at that moment, but I wanted just to take a look at it on my laptop for 30minutes before going live.

And here starts my real story with XHProf.

I downloaded latest PHP 5.2.9 and copied XHProf/extension folder to php_source/ext, ran:
#rm ./configure; ./buildconf --force; ./configure --prefix=/usr/local/php --with-xhprof; make

and ... it did not compile. The make exited with:

php_source/ext/xhprof/xhprof.c:202: error: syntax error before 'cpu_set_t'
php_source/ext/xhprof/xhprof.c:202: warning: no semicolon at end of struct or union
php_source/ext/xhprof/xhprof.c:213: error: syntax error before '}' token
php_source/ext/xhprof/xhprof.c:213: warning: data definition has no type or storage class
php_source/ext/xhprof/xhprof.c:222: error: syntax error before 'hp_globals'
php_source/ext/xhprof/xhprof.c:222: warning: data definition has no type or storage class

"Ooohh, gush" I thought and I opened xhprof.c
It's kinda miss-reading the docs on my side, cause I opened then and saw:

"Note: A windows port hasn't been implemented yet. We have tested xhprof on Linux/FreeBSD so far.

Note: XHProf uses the RDTSC instruction (time stamp counter) to implement a really low overhead timer for elapsed time. So at the moment xhprof only works on x86 architecture. Also, since RDTSC values may not be synchronized across CPUs, xhprof binds the program to a single CPU during the profiling period."

Ahh, they haven't tested the module on MAC OS X. And why should they? All of their servers run on Linux or FreeBSD. It's clear.

Ok, now I have to look and see if I can patch it.
(Oh, I forgot to tell that meanwhile I decided to change eAccelerator's cache expire on the loaded server to 2 hours, so load problem was resolved for a day or two and I had time to look at the XHProf)

Clearly XHProf documentation says that the module has to be bind to a single CPU and I knew that the "Thread Affinity" is implemented in a little bit different way in MAC OS X than in Linux or FreeBSD. Maybe that was the problem?
Yup, it is. Facebook team have made a patch for the affinity function in FreeBSD, but that's it. Now I have to add a similar solution for OS X.

I started to read more about thread affinity in Leopard and I found this document, which explains Apple's affinity API. Then after reading again and again, looking for some code snippets over the net I got the solution:

on line 30 to 49 replace with this:

#ifdef __FreeBSD__
# if __FreeBSD_version >= 700110
#   include
#   include
#   define cpu_set_t cpuset_t
# define SET_AFFINITY(pid, size, mask) \
cpuset_setaffinity(CPU_LEVEL_WHICH, CPU_WHICH_TID, -1, size, mask)
# define GET_AFFINITY(pid, size, mask) \
cpuset_getaffinity(CPU_LEVEL_WHICH, CPU_WHICH_TID, -1, size, mask)
# else
#   error "This version of FreeBSD does not support cpusets"
# endif /* __FreeBSD_version */
#elif __APPLE__
# include
# define cpu_set_t thread_affinity_policy_data_t
# define CPU_SET(cpu_id, new_mask) \
*new_mask.affinity_tag = (cpu_id + 1)
# define CPU_ZERO(new_mask) \
*new_mask.affinity_tag == THREAD_AFFINITY_TAG_NULL
#   define SET_AFFINITY(pid, size, mask) \
thread_policy_set(mach_thread_self(), THREAD_AFFINITY_POLICY, mask, THREAD_AFFINITY_POLICY_COUNT)
#   define GET_AFFINITY(pid, size, mask) \
thread_policy_get(mach_thread_self(), THREAD_AFFINITY_POLICY, mask, THREAD_AFFINITY_POLICY_COUNT)
#else
/* To enable CPU_ZERO and CPU_SET, etc.     */
# define __USE_GNU
/* For sched_getaffinity, sched_setaffinity */
# include
# define SET_AFFINITY(pid, size, mask) sched_setaffinity(0, size, mask)
# define GET_AFFINITY(pid, size, mask) sched_getaffinity(0, size, mask)
#endif /* __FreeBSD__ */


and on line 375 to 378 replace with:
#ifndef __APPLE__
  if (GET_AFFINITY(0, sizeof(cpu_set_t), &hp_globals.prev_mask) <>
    perror("getaffinity");
    return FAILURE;
  }
#endif

You can download whole the patched version of XHProf 0.9.1 from here.
Or get only the diff patch.

After successful compile and run I think it works - sharing the same L2 cache for a given thread when running on multiple cores, but NOT on different CPUs!
At least you can compile and see what's this module and implement it into your project. Then on the live server running Linux/FreeBSD you will definitely have correct data comming from XHProf.

If you find some errors or mistakes please let me know.
I also submitted a bug on the topic on PECL.

Thanks in advance for your oppinion and suggestions.

Wednesday, May 13, 2009

Tupalka.com (aka databg.svilen.com)

Few weeks ago we launched new design of Tupalka.com along with new features and the whole core rewritten and moved to SFrameWork.

SFrameWork is my own PHP framework born in 2006 (some internal classes date from 2005). It is fully MVC and I am thinking of publishing a open source project at SorceFourge. Some of you may find it useful.

Some of the new key features of Tupalka.com

- Implemented Sphinx search engine (it's very very fast and efficient compared to MySQL build-in fulltext index)
- New design
- New statistics: most popular searches, most recent searches, random seaches
- Added new data sources dir.bg, youtube.com along with the old rock solid one - data.bg
- Truly enabled cyrillic support for search queries
- Sort by list columns (just click on column name, e.g. filename, filesize, date)
- Categories trasferred to tabs

I hope these will help you for better experience while using Tupalka.com

And I would like to thank to all the people involved in some way in this new release of Tupalka.com - Georgi, Evgeni, Nikolay, Marian, Dimitar (Co-founder and owner of the domain name)

Wednesday, May 06, 2009

Lighttpd the Book

Following my post "Using mod_negotiation with Lighttpd" from April 23, 2009
I received a nice email from Packt Publishing who has published a new book about Lighttpd ("Lighttpd" written by Andre Bogus) asking me to review it.

I said - "ok, I will be glad to read your book."

And the story goes on...

Fortunately, I was nicely surprised that it deeply describes not only the Lighty 1.4.x, but the author has focused on 1.5.0, too. As some of you may know there are a lot of improvements in Lighttpd v.1.5.0, a lot of re-coded stuff, my beloved mod_uploadprogress was added again, etc., which is very essential in production servers.

Good. Now let's read on.

What's the first question come into your mind?
Maybe - Why Lighttpd? Huh?

Simple - cause it's fast, light, flexible and FREE. :)
Just look at the sample chapter of the book and you will see what I am talking about.

First few chapters are standard:
Introduction, Configuring & Compiling (I must say that it's very very well explained even for a beginner in UNIX systems), Virtual Hosts, CGI ...
oh wait - CGI? Isn't it slow?
Heh, yup CGI is a bit slow, but FastCGI is really fast! That's the key thing here - a whole chapter about FastCGI & SCGI support and how you should use it and configure in production environment.

Next is - "Optimizing Lighttpd".
Believe or not this is one of the most important things Andre (the author) has written about in his book.
How we can test our Lighttpd setup under stress with http_load (which I use, too) and how we can make some optimizations.
For example a simple mod_magnet caching script can deal easily with lots of load. And of course - mesuring system load is very handy here.

Are you Migrating from Apache?
Ok, then look at chapter 10 when you have a chance or here.

And now my favourite - "Using Lua with Lighttpd" (of course it means "using mod_magnet in Lighttpd").
This is one of the powers of Lighty where it leaves in the dust the good old Apache.
Lua is lightweight, embeddable language nevertheless it is very powerful. Combined with a lightweight and poweful webserver makes the unstoppable "killing" machine for huge amounts of web traffic.
Learn how to manipulate URLs, requested file names, caches, how to simulate Apache'с modules, etc.
This is a MUST READ topic!

And finally "Writing custom modules/plugins for the Lighttpd API" is for you C lovers.
Yes, that's right. A simple explanation how you can write a strong, stable module for Lighty in C.
It's pretty simple, although keep in mind this quote taken from the book:

"Do not write a module where a mod_magnetscript will do
A mod_magnetscript is usually smaller and much easier to write. The
higher level of abstraction leaves less chances of making mistakes. C,
unlike Lua, leaves the memory handling to the programmer. Also the
standard string-handling functions open security holes when used
improperly. Even if we are hell-bent on writing a module, it makes sense
to write a prototype mod_magnetscript first, presuming, of course, that
the function is possible to implement in Lua."


I cannot say anything else but Congratulations to Andre Bogus and Packt Publishing for this great book!

I recommend you to read it if you're going to deal with heavy load and big websites with lots of users. Also newbies can learn how to test their websites/web application and configure/compile a webserver by themselves.

Thursday, April 23, 2009

Using mod_negotiation with Lighttpd

Yesterday we finally decided to switch our web servers to Lighttpd 1.5.0
Everything went fine on the test environments until I ran into a problem with Lighty and Apache's mod_negotiation that we are using on some websites.
It turned out that there is no easy solution for a technique like mod_negotiation in Lighttpd, but fortunately Lighty has one very powerful module called Mod_Magnet.

With Mod_Magnet you actually can write your our LUA scripts for use with Lighttpd.
I was eager to try it. And here it comes my first LUA script emulating mod_negotiation and rewriting to a predefined file if nothing was found while "negotiating".

Use it like this:

# set document root for domains
$HTTP["host"] =~ "^(.*)(domain\.com)$" {
magnet.attract-physical-path-to = ( "/usr/local/lighttpd/etc/lighttpd-multiviews-to-php+rewrite.lua" ) # multiviews module
}

To compile Lighty with LUA support use:
./configure --prefix=/usr/local/lighttpd --with-lua

but before that you should install LUA 5.1

Saturday, April 18, 2009

Bulding Lighttpd 1.5.0 r2914 on MAC OS X Leopard

I haven't blogger for almost a year and it's time to start sharing my experience from the past months again.

Last week I went into trouble trying to build/compile Lighttpd 1.5.0 (latest prerelease r2914) on Mac OS X for test purposes.
One of the major features of Lighty 1.5.0 is the mod_uploadprogress, but unfortunately it is avaialable only in 1.5.0 version for now. So I was desparate to build it. :) (see mod_uploadprogress in action here)

What happened. After downloading the r2914 I ran the configure script with:
./configure --with-zlib --with-bzip2 --prefix=/usr/local/lighttpd --with-lua --sysconfdir=/usr/local/lighttpd/etc

(oh, yeah I wanted mod_magnet, too, to emulate Apache's mod_negotiation, so I added --with-lua)

All went ok with the configure, but then the make exits with:

Undefined symbols:
  "_network_get_backends", referenced from:
      _mod_status_handle_server_config in mod_status.o
  "_network_get_backend_info_by_type", referenced from:
      _mod_status_handle_server_config in mod_status.o
ld: symbol(s) not found
collect2: ld returned 1 exit status
make[4]: *** [mod_status.la] Error 1
make[3]: *** [all-recursive] Error 1
make[2]: *** [all] Error 2
make[1]: *** [all-recursive] Error 1
make: *** [all] Error 2


Ok, so now what? :) Of course, googling, googling and finally I got it! Hurray! :)
A patch was released on 07/08/2008 (download it directly from here), but why it is not applied in the official prerelease? Hm.. I hope the guys from Lighttpd's team will fix this in future.

Building Lighttpd 1.5.0 r2419 and later with applied patch for Mac OS X platform:
(directions for users that are not so much into command line tools)

1) Download and install XCode (if you already have it skip this step)
3) Extract Lighty
4) Download patch and copy it to Lighty's folder
5) Run following commands inside Lighty's folder to apply the patch:

patch -p1 < 0001-fix-build-on-mac-os-x.patch
aclocal
autoheader
glibtoolize --automake --force
automake --add-missing --force
autoconf

6) Configure Lighty with:
./configure --with-zlib --with-bzip2 --prefix=/usr/local/lighttpd

if you need mod_magnet install LUA 5.1 with fink (command: sudo fink install lua51) and configure with:
./configure --with-zlib --with-bzip2 --prefix=/usr/local/lighttpd --with-lua --sysconfdir=/usr/local/lighttpd/etc

7) Build Lighty:
make

8) Install Lighty:
sudo make install

9) Now you have Lighttpd 1.5.0 installed into /usr/local/lighttpd
10) Enjoy!