The world (and word) of jaymz

Two things I use a lot at work – Django and Active Collab. One is the python framework we build everything on and the other is a project management tool that we use – think a local install of Basecamp roughly.

It can get annoying when we create development versions of things we’re working on and have to go and create test users for people; so I thought since writing an auth backend for django is so easy why not just use that to allow any user with a valid Active Collab account to login to the dev admin.

The code is on github and will automatically create a superuser in django’s auth table. Users can then login with their active collab emails & passwords without having to pass around test accounts. It also means users wont suddenly lose access when you nuke the database for imports saving you some earache. Just add your base active collab url to the AC_URL variable in settings.py and add ‘acollabauth.backends.ActiveCollabBackend‘ to your AUTHENTICATION_BACKENDS tuple.

I have also blogged about this over at udox.

We use active collab at work to manage our various projects and track issues on sites we’re building. It comes complete with a REST API which returns results in XML. So I’ve written some code that abstracts out the process of making a request and displaying the returned data. It’s, predictably, on github.

Here’s an example based on the 0.2.0 code which is simply outputting data for now allowing me to check on open tickets easily from code. Remember, to enable write methods to work (setting a ticket to complete for example) you need to have Write Access enabled via the config.php file.

Code block

In [1]: from activecollab.library import ACRequest
 
In [2]: req = ACRequest('projects')
 
In [3]: req.execute()
34: AC 101: http://my.ac-site.com/projects/34
# More results trimmed
 
In [4]: req = ACRequest('projects', item_id=34, subcommand='tickets')
 
In [5]: req.execute()
2208: ie6 error when zooming on map: http://my.ac-site.com/projects/34/tickets/1: 1
2216: new user accounts for testing: http://my.ac-site.com/projects/34/tickets/3: 3

I’ve also blogged about this over at our udox company site.

geoip-redirect is available on my github page.

Something that I’ve had need of more than once the past while is placing lots of address data on google maps. For a few places it’s easy to manually handle that but I’ve been having to work with databases with thousands and thousands of samples – many in an unclean form with fields missing.

I’ve ended up with a small django based library over at github. For the data I had to work with much of the complications where the mis-matching of city/town level names – for example as well as London many items had the actual area listed instead, e.g. Camden or Westminster. When doing queries we wanted to be able to return items within certain areas but without having to manually clean everything up, so the model code provides a within_radius method which will return matching enteries that fall within the radial bounds given.

Within the scripts folder there’s an example of how I decided to work with an existing django based site that I then installed the geosearch code to.

Code block

#!/usr/bin/python2.7
# -*- coding: utf-8 -*-
 
# To run this you'll need to set two environment variables. Could probably do this
# as a management command but seems a bit messy since it's a very specific thing...
#
# export PYTHONPATH=/home/jaymz/development/www/vansemea
# export DJANGO_SETTINGS_MODULE=settings
#
# Obviously you'll need to change that path if you want to run it yourself
 
from locator.models import Dealer, City
from geosearch.models import GeoEntry
from base.models import Country
from django.db.models import Count
 
MAX_TOP_CITIES = 30 # how many of our "has the most dealers" to bin the others into
BOUNDRY_RADIUS = 25 # number of *miles* from our source point to include
CTYPES = {
    'dealer': 32,
    'city': 29,
}
 
for country in Country.objects.all():
    try:
        cities = Dealer.objects.values('city__pk', 'city__name') \
            .filter(country=country).annotate(Count('city')) \
            .order_by('-city__count')[:MAX_TOP_CITIES]
    except IndexError:
        print "No data for %s" % country.name
 
    print "Working out cities for %s\n" % country
 
    for city in cities[::-1]:
        # Look up the GeoEntry for this city. If we have one then we'll get
        # back a list of other enteries which are within BOUNDRY_RADIUS miles
        print "\nGetting db places within %dmiles of %s...\n" % (BOUNDRY_RADIUS, city['city__name'])
 
        try:
            primary_city = City.objects.get(pk=city['city__pk'], country=country)
            primary_city.primary_city = primary_city
            primary_city.save()
        except:
            "Could not find city matching %d and country id %d" % (city['city__pk'], country.pk)
            continue
 
        try:
            city_geoentry = GeoEntry.objects.get(content_type__pk=CTYPES['city'],
                object_id= city['city__pk'])
        except GeoEntry.DoesNotExist:
            print "No geoentry for %s" % city['city__name']
            continue
 
        radial_results = GeoEntry.within_radius(
            [city_geoentry.latitude, city_geoentry.longitude],
            BOUNDRY_RADIUS
        )
 
        # Trim off our source point, reverse and update our source cities
        # (we need to reverse it to have it assign by most populated last)
        for point in radial_results[1:][::-1]:
            if point['content_type'] == CTYPES['city']:
                source_city = City.objects.get(pk=point['object_id'])
                source_city.primary_city = primary_city
                source_city.save()
                print "Set %s as primary city for %s" % (primary_city, source_city)

We’ve recently had to do some automatic posting of content to facebook from a wordpress blog. I did find this wordbook plugin but it apparently has broke recently and people have been complaining it no longer works (27 out of 27!). So I’ve rolled my own. It’s in a very early stage but the code seems to work ok.

Instructions (you’ll need to create a new application and then authorize that for your user) and code are all over at github. I can’t really tell if I’m “doing it right” but it’s working and is a start. Fork and be happy…

Updated I’ve now added in support to push the title to twitter. I’m currently working on hooking in Bit.ly support to add the short URL to the tweet. Code is pushed to GH.

More updates The twitter posting now uses the post tile + a bit.ly link to your blog permalink. I’ve decided it’s now worthy of v0.2!

Ooyala is a feature rich video hosting platform. Something I needed recently was some way to link the data over at ooyala into a django site I’m building. So I wrote myself a library. It’s available over at github and it’s called (predictably) django-ooyala. Currently there is a management command syncooyala to pull in all the data using the Backlot Query API. These imported items are then linked to a specific URL. Finally in your templates there is a ooyala_video tag which when given the current path (via request.path) returns the <script> tags needed for it to render.

Expect some updates as I flesh it out into the front end over the next few days.

Updated (11th August): I have added in analytics support. You can now make requests for video stat’s for a given account or video. The facebook SDK has also been added with a new template tag to output the headers in your template for a given video. Remember to request whitelisting from facebook for SWF embeds to work.

Code block

from django.http import HttpResponse
from ooyala.library import OoyalaAnalytics
from ooyala.constants import OoyalaConstants as O
from ooyala.models import OoyalaItem
 
def backlot_query(request):
    req = OoyalaAnalytics(video=OoyalaItem.objects.all()[0].embed_code, \
        method=O.ANALYTIC_METHODS.VIDEO)
    ooyala_response = req.process()
    return HttpResponse(ooyala_response.toprettyxml(), mimetype="text/xml")

I’ve been working a lot on Vans recently and they have a large number of blogs that are currently hosted on the blogger platform. This works really well for them, they have a straightforward & easy to use blog platform that does what they need it to do.  The sites themselves work great as they are but the integration into the main site isn’t quite as nice.

If you go to the blogs section on vans as it stands it will pull in an XML feed of all the current blogs and display them on one giagantic page. It can take a fair bit of time to load and its hard to see how each different blog gets displayed as its really one big list. For now it’s not so easy to pull in bits of content from the various blogs without someone having to mess with files or copy & paste, with the build in django I wanted to create a much cleaner & easier way to work with this content.

For this I introduce django-blogger, a django application which will integrate Google Blogger blogs via their RSS feeds. As it is it will import the blogs for a given profile id and then sync up with the latest data via their RSS feeds. These can be enabled easily for your given blogs. When you first install it comes with some admin actions which will all ow you to sync up the blogs manually all at once. There is also a management command, syncblogs, which is more suited to scheduling an update via cron (if you’re going to use cron you might be interested in django-crontab).

This works on the feeds and not an archive, so it doesn’t require authentication, just access to the feed URL. A basic template is included to show how to render out the blog posts & blogs as a menu, I override these myself for the format I need. The app itself now lets me pull content from any of the blogs and use it within the rest of the django based site cleanly & easily. Rather than directly reading and displaying via the feed URL I’m creating actual objects for each post and blog so it’s easily extendible also, say to return posts in various formats or pulling images from each blog post to create blog galleries automatically.

The code is available from GitHub.

I was quite happy with the random scatter effect as different pools of colour made their way through the canvas space. I thought it would be a nice addition to add in some basic boundary conditions so I added a quick polygon class and added in an array of those. A few functions later and I could click around the screen defining new polygons. Using the well known Jordan Curve theorem we can tell if the walker is currently within an arbitrary polygon (without holes of course). Here’s an example video of it in action. Heart’s seem pretty easy to draw compared to anything crazy complex and give a good idea of the effect:

I have been coding in java for about a week now (when I’m not coding in Python of course) and I fancied a break away from it. I’m not entirely sure why but I decided to take a look at coding chrome extensions. These are in fact very, very easy to make. The tube status thing below took me just over an hour to get how I wanted. If you’re running chrome you can get the installer here. I think I’ll do some more work on it to add customization – that was my initial motivation to do this, the first one I installed from the current extension gallery just showed the little panel of status’ from the site and I wanted to just pick a few lines & tweak the output.

In the end it’s easier to just code my own. It makes use of the excellent REST api provided by Ben Dodson. Creating such an extension is trivial. You can make your life a bit easier if you just include jquery from the google ajax api’s. Then its really just a collection of a few html files (in this case just the one popup.html). All my HTML is doing is making an AJAX request for the JSON data from Ben’s api. That comes back and it simply renders out a few div’s with the colors set and based on the status a unicode entity to add some fun to it.

The method of making extensions on chrome really is a breeze and the documentation is plentiful, so hats off to google yet again for making me happy to be an open source, linux loving  developer. The extension is now in the chrome gallery and available here.

I’ve been rebuilding the Crooked Tongues forum and one of the things we’ve been mindful of is trying to stick to some good practises with regards optimzing the site for fast loading. One of the easy ways to decrease your page load times is to use CSS sprites. This is just stiching together all your common images into one big one and then using background positions & width/heights in CSS to offset and “crop” the big image. This way you only get the one HTTP request made to request it.

If you’ve got hundreds of small images it can really speed up your page. A prime candidate for sprites are typically your buttons, header images and in the case of a forum, emoticons. Crooked uses lots of custom designed icons and when writing them out as individual images for the post toolbar really slowed down the page. There was no chance I was going to do it manually so I turned to a stable of my web development toolset – Imagemagicks convert.

Convert can do a lot. I’ll show you two such ways it made my life easier and cut down on the time I had to spend doing grunt work. First, here’s how the toolbar looks:

I wrote some bash to basically loop over a folder of images and then using convert to pull the width & height info, I use that to write the CSS file. Finally I loop over the images once more using the append command to output one big, long image. (gist)

Code block

#!/bin/bash
 
# uses imagemagick to stich together all images in a folder and
# then writes a css file with the correct offsets along with a
# test html page for verification that its all good
 
if [ $# -gt 0 ]
then
 
    if [ $3 ]
    then
        ext="."$3; # the extension to iterate over for input files
    else
        ext=".gif"; # the extension to iterate over for input files
    fi
 
    name=$1; # output will be placed in a folder named this
 
    if [ $2 ]
    then
        classname=$2"-sprite";
    else
        classname=$1"-sprite";
    fi
    css="$name/$classname.css";
    html="$name/test.html";
 
    rm -fr $name;
    mkdir $name;
    touch $css $html;
 
    echo "Generating sprite file...";
    convert *$ext -append $name/$classname$ext;
    echo "Sprite complete! - Creating css & test output...";
 
    echo -e "<html>\n<head>\n\t<link rel=\"stylesheet\" href=\"`basename $css`\" />\n</head>\n<body>\n\t<h1>Sprite test page</h1>\n" >> $html
 
    echo -e ".$classname {\n\tbackground:url('$classname$ext') no-repeat top left; display:inline-block;\n}" >> $css;
    counter=0;
    offset=0;
    for file in *$ext
    do
        width=`identify -format "%[fx:w]" "$file"`;
        height=`identify -format "%[fx:h]" "$file"`;
        idname=`basename "$file" $ext`;
        clean=${idname// /-}
        echo ".$classname#$clean {" >> $css;
        echo -e "\tbackground-position:0 -${offset}px;" >> $css;
        echo -e "\twidth: ${width}px;" >> $css;
        echo -e "\theight: ${height}px;\n}" >> $css;
 
        echo -e "<a href=\"#\" class=\"$classname\" id=\"$clean\"></a>\n" >> $html;
 
        let offset+=$height;
        let counter+=1;
        echo -e "\t#$counter done";
    done
 
    echo -e "<h2>Full sprite:</h2>\n<img src=\"$classname$ext\" />" >> $html;
    echo -e "</body>\n</html>" >> $html;
 
    echo -e "\nComplete! - $counter sprites created, css written & test page output. ~jaymz";
 
else
 
    echo -e "There should be at least 1 argument!\n\tbuildSprite.sh output_folder classname input_extension"
 
fi

The script outputs the names incrementing by one, I’ve gone over myself and turned them into “nice” names. All in all its taken maybe 10 minutes to convert it all into a sprite and nice CSS file. If you were using a primary key field like 1, 2, 3… etc then you could probably fudge it to not even have to name the id’s “nicely”.

Imagemagicks convert can also do some modification of your files. It can for example convert your image to a black & white copy. It’s not a great deal of work to then loop over a set of images, create b&w versions in a temporary folder and then stitch the two versions together to get a color→b/w sprite for hovers. I’ve actually seen people do this manually and spend an age at it, with this method you output them once then sit back for the 30 seconds or so whilst the computer does the work.

Code block

for file in *.jpg; do convert -colorspace Gray $file bw/$file; done
for file in *.jpg; do convert $file bw/$file -append sprites/$file; done

It’s times like this that I am reminded of just how much time you can save with a little shell script and the right command line tools. Not everthing needs point & click.