Recently in Debian Category

The Call for Papers for the Distro Summit 2010 is still open!

While developing web applications using Zope3, my Python web framework of choice, I always have the same issue: where should I store the UGC data? I usually used an NFS filesystem, using a Zope utility to manage the storage and retrieval of the files.

The most obvious disadvantage of this technique is that the NFS server is a single point of failure: if it disappears, there is no way to access the files from the application servers.

Another problem with this implementation is that all the UGC data files have to be served by the application servers, using resources which could be used to serve more clients instead of transmitting a static file over the network.

MogileFS is an application-level distributed filesystem which solves these problems, ensuring data integrity and redundancy, written by Danga. The only problem is that it is written in Perl, it works only with MySQL and requires WebDAV servers for the storage.

To solve my problem, I decided to borrow some of the ideas behind MogileFS and to develop my own distributed application-level filesystem. KoboldFS is written in Python, uses a PostgreSQL database to store the filesystem status and it is released under the GPL license.

I deployed it a few days ago on one of our production clusters, and it reduced the load of the application servers allowing us to serve the UGC files directly from nginx. You can find the source code along with a rough description here.

I'll take part to PyCon2, the second italian Python Conference which will be held in Firenze (Florence) from the 9th to the 11th of May. I'll also give a talk about one of our last projects built on Zope 3 and all the issues we faced with while deploying and scaling it.

I've never been in Florence before, so I decided to bring my family with me to spend a few days before the conference visiting the city.

I'm looking forward to taking part to PyCon2 and meeting in person a lot of people I only know from the web, although I'll probably follow just a few talks and spend the rest of my time with my wife and the babies enjoying the city.

I started reading planet debian a few years ago when it became obvious for me that it was an important communication channel for the Debian community, but never managed to start my own blog. I think it is time to do it. After reading Ganneff's experiences with Movable Type, I finally installed it on my webserver.

Now, let's see how often I'll update it.