Google App Engine Twitter Bot

After I became familiar with Python and Google's hosting platform (called App Engine) I decided to write my first app for the platform.

And here it is - a simple Twitter bot which get's shared posts with comments "#twitter" from your Google Reader Atom feed and posts them to your Twitter feed.

Download it, open Python's App Engine docs, Service APIs docs and try yourself.

Take a look and change the following lines in twitter.py file accordingly in order to run the app successfully:

# twitter configs

twitter_id = "username"

twitter_pwd = "password"

# Google reader configs

# change to your GReader atom url

feed = "http://www.google.com/reader/public/atom/user/07723902623423188282/state/com.google/broadcast"

If you have questions leave your comments below. I will try to help you.

Python 2.6 & Google App Engine

Some of you may know that I am mainly developing on PHP (as a backend language) and MySQL database with different storage engines (XtraDB, InnoDb and PBXT).

A week ago I decided to get involved more deeply into Python and Google App Engine platform. I was already looking into Python as a generic type language for my system administration needs which arised in the last two months.

I know that all (most of) sys-admins use Perl as it is their old-school tool-language, but I wanted something better with easier and cleaner syntax and flexible enough.

So here came Python and it's brief tutorial. I must say it's fabulous and easy to understand.

I recommend you take a look at it (please read it carefully, there is nothing unnecessary written) and also watch some of the Google I/O talks of the Google Python gurus - Guido van Rossum (Pythons creator) and Alex Martelli.

Painless Python part 1 by Alex Martelli

Painless Python part 2 by Alex Martelli

Google App Engine is a new (to me) scalable platform for building web apps and I must say it's very easy for start. You don't have to think about software tuning, optimization, etc. Except of course the optimization algorithms and hints given from Google on how to build a successful app, which are easy to take in account. After you've become familiar with Python you may want to take a closer look at App Engine and the provided Python APIs.

Yes instead of software and a dedicated server you get an application storage space and a bunch of APIs. Pretty interesting approach for hosting from Google, but I must admit it is very conveninent. That way Google takes care about all software and service tuning, configuration, etc. and as a Developet you only have to ... DEVELOP.

Here are some interesting talks about Google's App Engine:

Python, Django and App Engine by Guido van Rossum

Building Scalable Web Apps with App Engine by Brett Slatkin (Google's App Engine main engineers)

App Engine Datastore Under the Covers by Ryan Barrett (lead App Engine datastore)

Enjoy!

MySQL Tuner

For those of you which does not have a clue how to optimize your MySQL machines, try MySQL Tuner script.

I recommend it even to experienced guys.

Hope Major Hayden will find a good follower who will continue the support of the script.

NoSQL fashion

I am sure some of you have already heard about NoSQL and available solutions like MongoDB, Cassandra, etc.

Yes, everything came back from Google's Bigtable and the ideas standing behind it.

And if you want to learn more go watch the Guide to NoSQL from Brian Aker and part 2.

Performance tool

Today I went over Poor man's profiler.

A simple & great tool to see what's going on your server.

Go try it.

XMPP Web Chat

Recently I had to develop a scalable web chat which should be compatable with normal chat software like Aduim, Miranda, etc.

I decided to use a XMPP server and I liked Brad Fritzpatriks Djabberd because it's easy scalable, memory optimized (300k+ connection in only 1GB RAM), written in Perl (one of the languages I am familiar with) and easily customizable.

All went okay during the tests with Adium and PSI and I managed to complete the custom modules in just two days.

But then there was the last part - web chat system integration. I stoped my attention to JwChat looks nice and has a Facebook alike bar aka iJabBar.

And here begins the Hell ...

All web based XMPP chat clients use so called HTPP-Binding ot Polling for communication (aka BOSH) to connect to Jabber server and Djabberd has no plugin for that. Sure at first I thought I can write one but ... isn't there anything already done? ... and no, there is nothing.

I started searching for a compromise solution and I went over JabberHTTPBind and Punjab. Both unreliable and buggy. Both tested on Mac OS X and Linux. At first I tried to fix JabberHTTPBind which is written in Java, but unfortunately I am not so experienced in Java programming to fix everything and XML parsing bugs left.

Punjab on the other hand is written in python which I am familiar with but after 4 hours of debugging and trying to fix it's XML parser, too, finally I gave up.

Now, after posting this I will go and write a working XMPP BOSH script using libevent and based on phpDaemon (asynchronous PHP framework). I am starting to think like Brad (formerly SixApart's guy and now working for Google) - if there is no solution or everything out there just sux, do write your own and do it well.

Performance

Everyone in IT field who know me well is aware that I am a performance maniac.

I love making software run faster with the same resources, I love saving machines/hardware, etc.

On this topic I would recommend you a short paper about optimizations which focuses on how you should think about it to do it in the right way.

One we interesting part of the paper is:

"Won’t software with extra code path to measure timings be slower than the same software without that extra code path?

I like an answer that I heard Tom Kyte give once in response to this question.15 He estimated that the measurement intrusion effect of Oracle’s extensive performance instrumentation is negative 10% or less. 16 He went on to explain to a now-­‐vexed questioner that the product is at least 10% faster now because of the knowledge that Oracle Corporation has gained from its performance instrumentation code, more than making up for any “overhead” the instrumentation might have caused."

I admire the Tom Kyte's answer. I want to point a finger at the face of all system administrators, CTO's, etc. - people turn on logs and your profile systems. They really help developers to optimize their code. Do not just blame them that the application sucks.

Go read "Thinking Clearly about Performance" by Cary Millsap.

And if you want to read more check Baron Schwartz's bookshelf.