XMPP Web Chat

Recently I had to develop a scalable web chat which should be compatable with normal chat software like Aduim, Miranda, etc.

I decided to use a XMPP server and I liked Brad Fritzpatriks Djabberd because it's easy scalable, memory optimized (300k+ connection in only 1GB RAM), written in Perl (one of the languages I am familiar with) and easily customizable.

All went okay during the tests with Adium and PSI and I managed to complete the custom modules in just two days.

But then there was the last part - web chat system integration. I stoped my attention to JwChat looks nice and has a Facebook alike bar aka iJabBar.

And here begins the Hell ...

All web based XMPP chat clients use so called HTPP-Binding ot Polling for communication (aka BOSH) to connect to Jabber server and Djabberd has no plugin for that. Sure at first I thought I can write one but ... isn't there anything already done? ... and no, there is nothing.

I started searching for a compromise solution and I went over JabberHTTPBind and Punjab. Both unreliable and buggy. Both tested on Mac OS X and Linux. At first I tried to fix JabberHTTPBind which is written in Java, but unfortunately I am not so experienced in Java programming to fix everything and XML parsing bugs left.

Punjab on the other hand is written in python which I am familiar with but after 4 hours of debugging and trying to fix it's XML parser, too, finally I gave up.

Now, after posting this I will go and write a working XMPP BOSH script using libevent and based on phpDaemon (asynchronous PHP framework). I am starting to think like Brad (formerly SixApart's guy and now working for Google) - if there is no solution or everything out there just sux, do write your own and do it well.

Performance

Everyone in IT field who know me well is aware that I am a performance maniac.

I love making software run faster with the same resources, I love saving machines/hardware, etc.

On this topic I would recommend you a short paper about optimizations which focuses on how you should think about it to do it in the right way.

One we interesting part of the paper is:

"Won’t software with extra code path to measure timings be slower than the same software without that extra code path?

I like an answer that I heard Tom Kyte give once in response to this question.15 He estimated that the measurement intrusion effect of Oracle’s extensive performance instrumentation is negative 10% or less. 16 He went on to explain to a now-­‐vexed questioner that the product is at least 10% faster now because of the knowledge that Oracle Corporation has gained from its performance instrumentation code, more than making up for any “overhead” the instrumentation might have caused."

I admire the Tom Kyte's answer. I want to point a finger at the face of all system administrators, CTO's, etc. - people turn on logs and your profile systems. They really help developers to optimize their code. Do not just blame them that the application sucks.

Go read "Thinking Clearly about Performance" by Cary Millsap.

And if you want to read more check Baron Schwartz's bookshelf.

Fast & efficient Smarty view renderer for Yii Framework

Working with online games and especially PenguinFarmer and using Yii Framework I've been driven to solve a problem with Smarty view renderers available already on the internet.

Smarty was a required template engine by the web designers, so I should have found a way to optimize the plugin or just write a new one.

And here it is. Fast, efficient and tested on a huge traffic.

One of the first problems I faced wa that the regular Smarty renderers do failer when there were huge amount of traffic as they render the template then saves it into a temp file ready for the PHP renderer mechanism which Yii has.

There are major issues here which cause unnecessary overhead.

1) Why do we need to save the rendered view template? This is a unnecessary disk I/O

2) Why do we need to read the smarty parsed temp file again to parse it as a PHP template? Again unnecessary disk I/O

3) When there is a lot of traffik there is a gap between writing a smarty template and reading it again. So many times there were a blank pages or wrong smarty templates. Even with fast HDDs there we enought time between reads and writes so we can get problems.

After going throught the code, optimizing it and testing I am glad to present you a good and reliable Smarty view renderer for Yii Framework that you can use as a drop in replacement for your current one.

Download

GitHub repository

Git - Linus Torvalds

I finally tried Torvald's Git version system and I can tell you - it's AWESOME!

Really. The way you can commit in just a second is so comfortable.

Watch his tech talk here.

More comming soon ...

XHProf 0.9.2 is out and ready for Mac OS X

Version 0.9.2 of Facebook's XHProf PHP module is out!

Thanks to Kannan from Facebook's dev team the patch for OS X is now a part of the official release.

Go get it Mac users! It compiles on OS X perfectly.

Two weeks ago I wrote that I've made a patch for version 0.9.1 to build under OS X.

Then I decided to post a request on XHProf's bug track and supply the patch there.

Kannan (leader of the project) answered and we collaborated in the next week for applying the patch in the next official release of XHProf.

You can read more about the whole story on PECL.

Thank you, Kannan, for your time and efforts!

I am glad that version 0.9.2 has a port for Mac OS X and I was able to help for that.

Facebook's XHProf PHP module patch for Mac OS X Leopard Build

*) note (05 Jun 2009): Thanks to Kannan from Facebook dev team the patch is now available as part of the official 0.9.2 release. You can download it from PECL.

Download full patched XHProf 0.9.1 for MAC OS X Leopard

or get the patch file only.

Bug submitted on PECL.

Full story

-----------

Yesterday (15 May 2009) I ran into XHProf module made by Facebook and published open source in March 2009.

Right after I read about its great features I was sure I had to try it and eventually put it on production servers. I was looking for such kind of PHP profiler with stats since 2005 when I found XDebug, which is really cool while developing but I wouldn't put it on a live server with high load like Tupalka.com's.

Eventually later that day our server hosting Tupalka.com was hit by a huge traffic and it was very loaded from time to time when the eAccelerator caches expires and most of the heavy mysql queries have to be ran once per 15 mins. That force me to try the XHProf.

Yup, I am sure that the problem was in the queries then why I was going for the XHProf? Good question. Curiosity.

These very moments are the once that push me to find and try different and new tools for optimization of web application/web sites.

So I was desparate to put XHProf on the production server at that moment, but I wanted just to take a look at it on my laptop for 30minutes before going live.

And here starts my real story with XHProf.

I downloaded latest PHP 5.2.9 and copied XHProf/extension folder to php_source/ext, ran:

#rm ./configure; ./buildconf --force; ./configure --prefix=/usr/local/php --with-xhprof; make

and ... it did not compile. The make exited with:

php_source/ext/xhprof/xhprof.c:202: error: syntax error before 'cpu_set_t'

php_source/ext/xhprof/xhprof.c:202: warning: no semicolon at end of struct or union

php_source/ext/xhprof/xhprof.c:213: error: syntax error before '}' token

php_source/ext/xhprof/xhprof.c:213: warning: data definition has no type or storage class

php_source/ext/xhprof/xhprof.c:222: error: syntax error before 'hp_globals'

php_source/ext/xhprof/xhprof.c:222: warning: data definition has no type or storage class

"Ooohh, gush" I thought and I opened xhprof.c

It's kinda miss-reading the docs on my side, cause I opened then and saw:

"Note: A windows port hasn't been implemented yet. We have tested xhprof on Linux/FreeBSD so far.

Note: XHProf uses the RDTSC instruction (time stamp counter) to implement a really low overhead timer for elapsed time. So at the moment xhprof only works on x86 architecture. Also, since RDTSC values may not be synchronized across CPUs, xhprof binds the program to a single CPU during the profiling period."

Ahh, they haven't tested the module on MAC OS X. And why should they? All of their servers run on Linux or FreeBSD. It's clear.

Ok, now I have to look and see if I can patch it.

(Oh, I forgot to tell that meanwhile I decided to change eAccelerator's cache expire on the loaded server to 2 hours, so load problem was resolved for a day or two and I had time to look at the XHProf)

Clearly XHProf documentation says that the module has to be bind to a single CPU and I knew that the "Thread Affinity" is implemented in a little bit different way in MAC OS X than in Linux or FreeBSD. Maybe that was the problem?

Yup, it is. Facebook team have made a patch for the affinity function in FreeBSD, but that's it. Now I have to add a similar solution for OS X.

I started to read more about thread affinity in Leopard and I found this document, which explains Apple's affinity API. Then after reading again and again, looking for some code snippets over the net I got the solution:

on line 30 to 49 replace with this:

#ifdef __FreeBSD__

# if __FreeBSD_version >= 700110

#   include

#   include

#   define cpu_set_t cpuset_t

# define SET_AFFINITY(pid, size, mask) \

cpuset_setaffinity(CPU_LEVEL_WHICH, CPU_WHICH_TID, -1, size, mask)

# define GET_AFFINITY(pid, size, mask) \

cpuset_getaffinity(CPU_LEVEL_WHICH, CPU_WHICH_TID, -1, size, mask)

# else

#   error "This version of FreeBSD does not support cpusets"

# endif /* __FreeBSD_version */

#elif __APPLE__

# include

# define cpu_set_t thread_affinity_policy_data_t

# define CPU_SET(cpu_id, new_mask) \

*new_mask.affinity_tag = (cpu_id + 1)

# define CPU_ZERO(new_mask) \

*new_mask.affinity_tag == THREAD_AFFINITY_TAG_NULL

#   define SET_AFFINITY(pid, size, mask) \

thread_policy_set(mach_thread_self(), THREAD_AFFINITY_POLICY, mask, THREAD_AFFINITY_POLICY_COUNT)

#   define GET_AFFINITY(pid, size, mask) \

thread_policy_get(mach_thread_self(), THREAD_AFFINITY_POLICY, mask, THREAD_AFFINITY_POLICY_COUNT)

#else

/* To enable CPU_ZERO and CPU_SET, etc.     */

# define __USE_GNU

/* For sched_getaffinity, sched_setaffinity */

# include

# define SET_AFFINITY(pid, size, mask) sched_setaffinity(0, size, mask)

# define GET_AFFINITY(pid, size, mask) sched_getaffinity(0, size, mask)

#endif /* __FreeBSD__ */

and on line 375 to 378 replace with:

#ifndef __APPLE__

  if (GET_AFFINITY(0, sizeof(cpu_set_t), &hp_globals.prev_mask) <>

    perror("getaffinity");

    return FAILURE;

  }

#endif

You can download whole the patched version of XHProf 0.9.1 from here.

Or get only the diff patch.

After successful compile and run I think it works - sharing the same L2 cache for a given thread when running on multiple cores, but NOT on different CPUs!

At least you can compile and see what's this module and implement it into your project. Then on the live server running Linux/FreeBSD you will definitely have correct data comming from XHProf.

If you find some errors or mistakes please let me know.

I also submitted a bug on the topic on PECL.

Thanks in advance for your oppinion and suggestions.

Tupalka.com (aka databg.svilen.com)

Few weeks ago we launched new design of Tupalka.com along with new features and the whole core rewritten and moved to SFrameWork.

SFrameWork is my own PHP framework born in 2006 (some internal classes date from 2005). It is fully MVC and I am thinking of publishing a open source project at SorceFourge. Some of you may find it useful.

Some of the new key features of Tupalka.com

- Implemented Sphinx search engine (it's very very fast and efficient compared to MySQL build-in fulltext index)
- New design
- New statistics: most popular searches, most recent searches, random seaches
- Added new data sources dir.bg, youtube.com along with the old rock solid one - data.bg
- Truly enabled cyrillic support for search queries
- Sort by list columns (just click on column name, e.g. filename, filesize, date)
- Categories trasferred to tabs

I hope these will help you for better experience while using Tupalka.com

And I would like to thank to all the people involved in some way in this new release of Tupalka.com - Georgi, Evgeni, Nikolay, Marian, Dimitar (Co-founder and owner of the domain name)