Integrating ooyala in django (or just python)

Ooyala is a feature rich video hosting platform. Something I needed recently was some way to link the data over at ooyala into a django site I’m building. So I wrote myself a library. It’s available over at github and it’s called (predictably) django-ooyala. Currently there is a management command syncooyala to pull in all the data using the Backlot Query API. These imported items are then linked to a specific URL. Finally in your templates there is a ooyala_video tag which when given the current path (via request.path) returns the <script> tags needed for it to render.

Expect some updates as I flesh it out into the front end over the next few days.

Updated (11th August): I have added in analytics support. You can now make requests for video stat’s for a given account or video. The facebook SDK has also been added with a new template tag to output the headers in your template for a given video. Remember to request whitelisting from facebook for SWF embeds to work.

Code block

from django.http import HttpResponse
from ooyala.library import OoyalaAnalytics
from ooyala.constants import OoyalaConstants as O
from ooyala.models import OoyalaItem
 
def backlot_query(request):
    req = OoyalaAnalytics(video=OoyalaItem.objects.all()[0].embed_code, \
        method=O.ANALYTIC_METHODS.VIDEO)
    ooyala_response = req.process()
    return HttpResponse(ooyala_response.toprettyxml(), mimetype="text/xml")

Google Blogger to Django integration

I’ve been working a lot on Vans recently and they have a large number of blogs that are currently hosted on the blogger platform. This works really well for them, they have a straightforward & easy to use blog platform that does what they need it to do.  The sites themselves work great as they are but the integration into the main site isn’t quite as nice.

If you go to the blogs section on vans as it stands it will pull in an XML feed of all the current blogs and display them on one giagantic page. It can take a fair bit of time to load and its hard to see how each different blog gets displayed as its really one big list. For now it’s not so easy to pull in bits of content from the various blogs without someone having to mess with files or copy & paste, with the build in django I wanted to create a much cleaner & easier way to work with this content.

For this I introduce django-blogger, a django application which will integrate Google Blogger blogs via their RSS feeds. As it is it will import the blogs for a given profile id and then sync up with the latest data via their RSS feeds. These can be enabled easily for your given blogs. When you first install it comes with some admin actions which will all ow you to sync up the blogs manually all at once. There is also a management command, syncblogs, which is more suited to scheduling an update via cron (if you’re going to use cron you might be interested in django-crontab).

This works on the feeds and not an archive, so it doesn’t require authentication, just access to the feed URL. A basic template is included to show how to render out the blog posts & blogs as a menu, I override these myself for the format I need. The app itself now lets me pull content from any of the blogs and use it within the rest of the django based site cleanly & easily. Rather than directly reading and displaying via the feed URL I’m creating actual objects for each post and blog so it’s easily extendible also, say to return posts in various formats or pulling images from each blog post to create blog galleries automatically.

The code is available from GitHub.

Image color analysis for your ecom site

Recently we’ve been working rather hard on a new look and complete re-launch of the underlying code for the whole of crookedtongues.com We’re a python shop these days so its running on django with a host of apps and the best part of over 100,000 lines of code.

One of the things that I was tasked with was the data migration and import and I’ll write about that post launch, what I fancy getting out now is how I’ve worked on adding color codes automagically to the store’s entire product database. More or less, its a little rough round the edges but gives fairly good results on a lot of our products. First off you can find the source code in a raw (read that as ‘I’m still working on it personally so take it as it comes’) way over at github. As a taste of what it does, here’s a couple of existing product shots:

It works quite well for colors that fall into the center areas of the color wheel. For my analysis I’ve segmented it by 30° so there are 12 bands of color. I’ve found that yellow & light greens are the most tricky to pull out accurately from an instinctive perception view of the result. By that I mean what looks very much yellow can get labelled as a bright orange instead as values close to the segment boundary (45°) cause an imbalance towards the orange side.

The basic process is working well for products with one dominating color, for multi-color shots it can pull out the relative levels of each tones hue band but blindly chooses the largest, however slight and uses that for its naming output. A better way would be to do some further analysis on the collected results for each image – one way to enforce some confidence that the main hue really is the main color would be to compare its standard deviation from the average across the sample set. A small σ would indicate a more uniform mix of colors – as we have already sampled out low saturation points this would indicate an image of strong multicolor. Taking the set of hue/tones within the 90th percentile and then comparing their relative deviations of count would allow identifying (and quantifying) the number of colors – so it would be possible to say mainly dark blue with a little light red.

If two bands had similar counts it is also more likley that the color in question is not two separate closely aligned colors but in fact a single color at the midpoint (or thereabouts) of them. A scheme to compare these before the final decision would no doubt improve the detection of yellow which seems to constantly tip into the orange side of the wheel. All in all working on this has been a decidedly pleasant break from data imports, javascript and enough MVC style code to do me a lifetime.

Since I wrote the above I’ve tweaked it somewhat to pull out gray (and black/white) counts also. It works out the rough percentage that corresponds to and if its over a certain limit for black or white will output a value for that also (with a lot of the product shots they’re on a white background so I need to be more granular with that than just regular colors).

image_scan.py

#!/usr/bin/python
# -*- coding: utf-8 -*-
 
# A color analysis script to help you label your store's products with color data
# automagically. It will take either a single file or scour an entire folder for
# folders of images and do each one individually printing a summary of what it
# thinks is the correct color value. A work in progress...
#
# ~jaymz | @jaymzcampbell | jaymz.eu
#
#
# MIT Licesnsed for what its worth, copy: http://www.opensource.org/licenses/mit-license.php
 
import Image
import ImageFilter
import os
import glob
import sys
import colorsys
import re
from copy import copy
from operator import itemgetter
from decimal import Decimal
 
output = open('colors.csv', 'w')
 
# Pixels will be first compared to these values before being
# added to the data list of color information on the first pass
LBOUND = 0
UBOUND = 255
 
MIN_SATURATION = 30 # avoid washed out pixels influencing counts
 
# Base folder for the processFolder function, it'll iterate over here on subfolders
FOLDER = '/home/jaymz/documents/crooked-docs/data-export/store-migration/product-images/'
 
# Meh, i need to flip between these two, you can probably tweak this 
SUMMARY_FORMAT, SQL_FORMAT = True, True
 
# Names based off: http://bluelobsterart.com/wordpress/wp-content/uploads/2009/03/rgb-color-wheel-lg.jpg
COLOR = ['RED', 'ORANGE', 'YELLOW',
    'LIME', 'GREEN', 'TURQUOISE',
    'CYAN', 'OCEAN', 'BLUE',
    'VIOLET', 'MAGENTA', 'RASPBERRY',
    ]
TONE = ['DARK', '', 'BRIGHT']
 
# via the createColorSQL.py file , addition added in GRAY/BLACK/WHITE to after this
SQL_IDS = {'DARK YELLOW': 7, 'DARK ORANGE': 4, 'BRIGHT GREEN': 15, 'BRIGHT ORANGE': 6, 'DARK RED': 1, 'BRIGHT OCEAN': 24, 'BRIGHT RED': 3, 'DARK OCEAN': 22, 'YELLOW': 8, 'OCEAN': 23, 'BRIGHT YELLOW': 9, 'RASPBERRY': 35, 'GREEN': 14, 'BRIGHT TURQUOISE': 18, 'CYAN': 20, 'MAGENTA': 32, 'RED': 2, 'ORANGE': 5, 'BLUE': 26, 'TURQUOISE': 17, 'LIME': 11, 'BRIGHT LIME': 12, 'DARK MAGENTA': 31, 'DARK LIME': 10, 'BRIGHT MAGENTA': 33, 'BRIGHT VIOLET': 30, 'DARK VIOLET': 28, 'DARK BLUE': 25, 'BRIGHT BLUE': 27, 'VIOLET': 29, 'BRIGHT RASPBERRY': 36, 'DARK TURQUOISE': 16, 'DARK CYAN': 19, 'BRIGHT CYAN': 21, 'DARK GREEN': 13, 'DARK RASPBERRY': 34}
 
pcnt = 0
 
def trimFloat(val, places=2):
    return float(repr(val)[0:places+2])
 
def withinBounds(allowance, _rgb):
    rgb = copy(_rgb)
    diff = 0
    allowance = Decimal(repr(allowance))
    for c in rgb:
        for d in rgb:
            dec_d = Decimal(repr(d)).quantize(allowance)
            dec_c = Decimal(repr(c)).quantize(allowance)
 
            diff = abs(dec_d-dec_c)
 
            if (d != c) and diff>allowance:
                return False
    return True
 
def processImage(i, name=None):
  """ Scales down the image, blurs it to ease the blending of the color values
and reduce spikes from anomolies. It then samples pixels creating a list of
colors. This list is then looped over to build counts which are placed into
bins of 30° hue's seperated into three based on their value. Pixels less than
a certain saturation are discarded. """
 
  global pcnt
 
  i = i.resize((200,200))
  i = i.convert("RGB")
  i = i.filter(ImageFilter.BLUR)
  d = i.getdata()
  cnt = 0
 
  h = [] #holds the hsv info
  grays = [] #holds just gray content
  black_count = 0
  white_count = 0
  total_samples = 0
 
  for p in d:
      cnt = cnt + 1
      if cnt == 8: #take every 4th pixel
        if p[0]>LBOUND and p[1]>LBOUND and p[2]>LBOUND and p[0]<UBOUND and p[1]<UBOUND and p[2]<UBOUND:
            r = trimFloat(float(p[0])/255)
            g = trimFloat(float(p[1])/255)
            b = trimFloat(float(p[2])/255)
 
            if not withinBounds(0.02, (r,g,b)):
                h.append(colorsys.rgb_to_hsv(r,g,b))
            else:
                if (r+g+b)/3>0.94:
                    white_count += 1
                elif (r+g+b)/3<0.3:
                    black_count += 1
                else:
                    grays.append(colorsys.rgb_to_hsv(r,g,b))
            total_samples += 1
        cnt = 0 #reset sample counter
 
  h.sort()
  grays.sort()
  bin_width = 30 # size of hue slices (degress)
  max_bin = 360
 
  darks = [0] * int(max_bin/bin_width)
  mids = [0] * int(max_bin/bin_width)
  lites = [0] * int(max_bin/bin_width)
 
  for p in h[::]:
      hue = p[0]*360
      sat = p[1]*100
      val = p[2]*100
      if sat >= MIN_SATURATION:
        bin_number = ((int(hue)+15)/bin_width)%(max_bin/bin_width)
        if val<33:
            darks[bin_number] += 1
        elif val>33 and val < 66:
            mids[bin_number] += 1
        else:
            lites[bin_number] += 1
        #print "HUE BIN: %s VALUE : %d" % (int(hue)/bin_width, int(hue))
 
  c = 0
  data = zip(darks, mids, lites)
 
  if SUMMARY_FORMAT:
    for x in data:
        print '%d %s : %s %d°' % (c, COLOR[c], x, c*bin_width)
        c += 1
 
  # the following area needs a rework. the index technique works alright as long
  # as counts and values dont all match up, then it starts picking the first one
  # so this needs re-writing to better order the list data
 
  darks_sort, mids_sort, lites_sort = darks[::], mids[::], lites[::]
  darks_sort.sort()
  mids_sort.sort()
  lites_sort.sort()
 
  sorted_counts = (darks_sort, mids_sort, lites_sort)
 
  primary_idx = (darks.index(sorted_counts[0][-1]), mids.index(sorted_counts[1][-1]), lites.index(sorted_counts[2][-1]))
  primary_cnts = (darks[primary_idx[0]], mids[primary_idx[1]], lites[primary_idx[2]])
  tone = primary_cnts.index(max(primary_cnts))
  max_hbin = primary_idx[tone]
 
  pcnt += 1
 
  if SUMMARY_FORMAT:
    print "\nDominant Hue: %s %s" % (TONE[tone], COLOR[max_hbin])
 
  if SQL_FORMAT and name and max(primary_cnts) > 30:
    output.write('%d, %s, %s\n' % (pcnt, name, SQL_IDS[' '.join([TONE[tone], COLOR[max_hbin]]).strip()]))
 
  sorted_counts[0][-1], sorted_counts[1][-1], sorted_counts[2][-1] = (0, 0, 0) # kind of reset the primary to null
  for l in sorted_counts:
      l.sort()
 
  primary_idx = (darks.index(sorted_counts[0][-1]), mids.index(sorted_counts[1][-1]), lites.index(sorted_counts[2][-1]))
  primary_cnts = (darks[primary_idx[0]], mids[primary_idx[1]], lites[primary_idx[2]])
  tone = primary_cnts.index(max(primary_cnts))
  max_hbin = primary_idx[tone]
 
  if SUMMARY_FORMAT:
    print "Secondary Hue: %s %s" % (TONE[tone], COLOR[max_hbin])
 
  if SQL_FORMAT and name and max(primary_cnts) > 30:
    pcnt += 1
    output.write('%d, %s, %s\n' % (pcnt, name, SQL_IDS[' '.join([TONE[tone], COLOR[max_hbin]]).strip()]))
 
  # area to rewrite ends...
 
  gray_total = [(g[0]+g[1]+g[2])/3 for g in grays]
  gray_average = reduce(lambda x,y : x+y, gray_total)/len(gray_total)
 
  black_percent = black_count/float(total_samples)*100
  gray_percent = len(gray_total)/float(total_samples)*100
  white_percent = white_count/float(total_samples)*100
 
  if SUMMARY_FORMAT:
    print "\nAverage Gray: %s (samples: %0.1f%%), White count: %s (%0.1f%%), Black count: %s (%0.1f%%)" % (gray_average, gray_percent, white_count, white_percent, black_count, black_percent)
    print "Total samples taken: %s\n\n" % total_samples
 
  if SQL_FORMAT:
    if black_percent > 10:
        pcnt += 1
        output.write('%d, %s, %d\n' % (pcnt, name, 38))
    if gray_percent > 10:
        pcnt += 1
        output.write('%d, %s, %d\n' % (pcnt, name, 37))
    if white_percent > 30:
        pcnt += 1
        output.write('%d, %s, %d\n' % (pcnt, name, 39))
 
# Helper functions follow along with __main__ def
 
def processFolder(folder):
    for image_folder in glob.glob(folder+'*'):
        try:
            folder_images = []
            for image in os.listdir(image_folder):
                if "jpg" in image and "._" not in image:
                    folder_images.append(image)
            folder_images.sort()
            j = os.path.join(image_folder, folder_images[1])
            if SUMMARY_FORMAT:
                print "working: "+j
            i = Image.open(j)
            processImage(i, image_folder.split('/')[-1])
        except:
            pass
 
def processFile(_file):
    i = Image.open(_file)
    processImage(i)
 
if __name__ == "__main__":
    try:
        if 'product-images' not in sys.argv[1]:
            processFile('product-images/'+sys.argv[1])
        else:
            processFile(sys.argv[1])
    except IndexError:
        processFolder(FOLDER)
    output.close()

Google’s results plotted for repeated character strings

Don’t ask why but out of interest I googled for the string “AAAAAAAA” earlier and after looking at the millions of pages that came back and thinking “wtf”, I searched again only making the string much longer. I was expecting it to just keep going down but at around 20 characters there was a significant jump in returned results. I scratched my beard and proclaimed this interesting (as you can probably tell I have no distractions on valentines day). To skip over further bullshit, this is the graph of 3,328 searches – that is, the number of results for every character (A-Z) repeated one to 128 times. Some of the peaks are interesting.

Why 128 and not something higher? Google wont let you, at least via the query string. The raw data for this graph was generated by a simple python script. If you aren’t coding in python already, please do look into it, its jolly nice.

Code block

#!/usr/bin/python
 
from urllib import FancyURLopener
from BeautifulSoup import BeautifulSoup
import csv
import string 
 
BASE = "http://www.google.com/search?q="
 
class MozOpener(FancyURLopener):
    version = 'Mozilla/5.0 (Windows; U; Windows NT 5.1;' +
' it; rv:1.8.1.11) Gecko/20071127 Firefox/2.0.0.11'
ff = MozOpener()
 
out = csv.writer(open('out.csv', 'w'))
 
headers = ['count',]
for l in string.ascii_uppercase:
    headers.append(l)
out.writerow(headers)
 
for x in range(1, 128): # 128 is the max length of chars allowed
    results = [x,]
    for l in string.ascii_uppercase:
        qry = l*x
        raw_data = ff.open(BASE+qry).read()
        soup = BeautifulSoup(raw_data)
        result = soup.find(id='resultStats').findAll('b')[2]
        result = int(result.contents[0].replace(',', ''))
        results.append(result)
        print [result, qry]
    out.writerow(results)

When that had finished I simply loaded it up into OpenOffice calc (3.2 is out by the way) and plotted it with the result count set to a logarithmic scale so result #1 doesn’t just skew the thing into one boring L shape. The first thing I noticed was a very visible spiking at length 100.

This isn’t so hard to imagine, 100 is a “nice” number. It’s not hard to imagine someone using 100x a character as a test input or just a “long” string. Every character string of length 100 exhibits this spike. The much bigger blue curve is for the letter x. This is used by children and adults alike to mark kisses, and everyone knows the more x’s the more someone loves you. If you look at many of the results for 100 or 101 character X searches it seems to be when people are using it in this context. Could it be that the much bigger spike for the 101 character string X is simply because its 100 + 1 kisses?? Towards the start of the graph there are a number of interesting spikes, I’ve marked some of them along with the length.

Some of these spikes are easy to explain, the biggest number of results returned for a single repeated character phrase is a by product of DNS, yep, its “WWW”. This accounts for the slightly higher result count than the simple “A” with 17,090,000,000 pages returned versus 12,260,000,000. Another easy one is 6 F’s – the hex code for white. I am totally stumped by F’s latter behaviour though, there is a spike at F-31 and F-33 but not F-32. There is a big jump for X-12 and X-34. International mobiles have 12 digits, as do UPC codes but I feel like I’m clutching at straws trying to explain that. Down the other end of the graph, between 115 and 128 the characters P, H, A, M and O all have significant spikes for specific counts. For M & O when you browse many of the first few pages of google’s results many of the pages are using them as part of exaggerated speech. It’s almost like the collective conciousness of the world has decided that 120 characters is just right to describe a particularly tasty dinner.

A spreadsheet with the data I gathered is available via google docs if you’d like to investigate yourself. I should note that the python script above gets its data from .com, if you are using the site to look up some searches it will more than likely switch to your local domain. Depending on your cookies google may also perform extra filtering (e.g. safe searches) so you’re numbers may not be the exact same as mine. I’d be interested in any theories as to some of the more prominent spikes. The OpenOffice spreadsheet with the charts done is also available for download here.

If you are going to play around then save your fingers & sanity and use python to create you’re test strings, just drop into a python shell and use “X”*120 etc or perl -e ‘print “X”x100′ from the command line etc.

Importing existing visitor stats from Google Analytics to Piwik

Recently at work we had to aggregate a lot of google analytics accounts and do some tracking and custom reporting. We found that it wasn’t quite as straightforward as we thought, one of the problems we had was getting multiple tracking codes to work on the same page. It was no real surprise that this wouldn’t work easily because google themselves have this to say:

Installing multiple instances of the Google Analytics Tracking code on a single web page is not a supported implementation. We suggest you remove all but one instance, and make sure you have the code from the correct profile installed on every page you would like to track.

With a lot of searching and reading I did find a number of scattered blog posts that suggested that it would work and was possible. But no matter what I tried I couldn’t get it tracking data into multiple accounts from the same page. I got fed up and was tasked with looking at alternative tracking solutions. That’s when I came across the very handy list at wikipedia and from there Piwik. Piwik is a GPL license PHP application which aims to mimic the functionality of google analytics. You install it on your server and then add sites to it much like GA. It then gives you a tracking code to install on your site(s).

As it was so similar to GA but locally ran, I played about with it and decided to go with that, working on the assumption that I could at least get at the db tables and source in the future. I setup a number of sites to track and then installed the codes on each page. One acted as a “master” code which was on all pages whilst others only appeared when the domain matched a certain string. I left it for an hour and came back to find the data all populated in each account as I expected. I was jolly pleased.

When it was shown to the intended user they really loved it. The only thing they wanted to sweeten the whole experience even more was to have the visitor count data for the prior month loaded in from Google Analytic’s.  I wasn’t over the moon with this as exporting you’re data from GA isn’t very straightfoward and certainly not in a simple “just dump and drop it in” way either. I had a quick look at what I could export from GA and said I could import in the unique hits per day fairly easily but tying that to specific browsers or page titles etc would be quite a lot of work.

I sort of half expected that that would result in a “ah, ok, lets forget about it” conversation with the end agreement being use GA for data prior to the switchover and Piwik for reports since then but nope, they still wanted to have just the hit counts loaded in regardless of if they were not attached to page titles or a users tech setup. Looking at the piwik developers zone the post about an API to push data in was initially promising but it was focusing on apache logs and most recently a user said the timestamps weren’t coming through. So I looked at another way to get it done. What follows is how I personally went about loading data in. You may find it useful if you end up migrating yourself.

To begin with I installed piwik afresh and dumped out the database. Then I set up just one site and let it record a single hit. Then I went through and compared each table with the previous state. This told me there were 2 places I needed to put data in if I wanted it logged and being used in piwik.

_log_visit & _log_link_visit_action are the two key tables that receive data on each click. The link_visit_action table ties a particular visit to a particular page. I wasn’t going to b doing this so in log_action I added 2 new rows – 1 for the url of my “Google Analytics Dummy page” and another for its title. Then I noted the id’s for each of those rows.

Confident that this would be all I required I went over to GA to export out my data. I clicked through to visitors and set the date range for that required. Then clicked export and chose CSV. You will note that the actual data is not by time but instead aggregated for each day. This means that at most this is just going to allow someone to see the total hits per day but no further drilldown by hour etc. I made that clear to the client and they where again (worse luck!) happy with that level of reporting.

The first thing I needed to do was clean up this data. GA exports it out with day & month names along with some other cruft that wasn’t required. There would be a myriad of ways to sort this out and I chose to use the unix stalwart, sed. The code that follows I saved and chmod’d and then ran on my GA csv file.

Code block

#!/bin/sed -nrf
s/.*day, //
s/January/01,/
s/February/02,/
s/March/03,/
s/April/04,/
s/May/05,/
s/June/06,/
s/July/07,/
s/August/08,/
s/September/09,/
s/October/10,/
s/November/11,/
s/December/12,/
1,10d
s/\"//
s/([0-9]{1,2}), ([0-9]{1,2}), ([0-9]{4}),(.+)/\3-\1-\2 00:00:00,"\4"/
/^[0-9]/p

You can download that here.

That should take

and turn it into:

Now its still not the most ISO-formatted csv in the world but it will do for what we need. I obviously can’t just load that into piwik so I then used this as input to a python script that simply magics up the rest of the needed info for the two piwik table’s. As the visitor stats from GA are the uniques (again I made that clear before starting all this to the client) I just create an md5 hash via the current time as I make my way through the counts. That way piwik records them as unique visitors. If the cookies hashes are the same in the db piwik will consider it a returning visitor even if the return flag is 0.

Code block

import csv
import os
from datetime import datetime
from md5 import md5
import sys
 
def main(argv):
    if argv[1]:
        COL_ID = int(argv[1])
    else:
        COL_ID = 1
    if argv[2]:
        LVA_ID = int(argv[2])
    else:
        LVA_ID = COL_ID
    SITE_ID = 1
    LOCAL_TIME = "00:00:00"
    VISITOR_RETURNING = 0
    GA_ACTION_URL = 1
    GA_ACTION_NAME = 1
    TOTAL_ACTIONS = 1
    VISIT_TOTAL_TIME = 10
    GOAL_CONVERT = 0
    REFERER_TYPE = 1
    CONFIG_OS = "GA"
    CONFIG_BROWSER = "GA"
    CONFIG_B_VER = "0.1"
    CONFIG_RES = "1600x1200"
    CONFIG_MD5 = md5(CONFIG_BROWSER+CONFIG_OS+CONFIG_RES).hexdigest()
    LOCATION_IP = "168450866"
    BROWSER_LANG = "en-gb"
    LOCATION_COUNTRY = "gb"
    LOCATION_CONTINENT = "eur"
    LOCATION_PROVIDER = "GA Import"
    data = csv.reader(open(argv[0]))
    piwik_output = [csv.writer(open('log_visit-'+argv[0], 'w')),
        csv.writer(open('log_vaction-'+argv[0], 'w'))]
    for row in data:
        hits = int(row[1].replace(',', ''))
        for i in range(0, hits):
            cookie = md5(datetime.now().__str__()).hexdigest()
            action_time = row[0] + LOCAL_TIME
            output = [COL_ID, SITE_ID, action_time, cookie,
                VISITOR_RETURNING, action_time, action_time, row[0],
                GA_ACTION_URL, GA_ACTION_NAME, TOTAL_ACTIONS,
                VISIT_TOTAL_TIME, GOAL_CONVERT, REFERER_TYPE, None,
                None, None, CONFIG_MD5, CONFIG_OS, CONFIG_BROWSER,
                CONFIG_B_VER, CONFIG_RES, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1,
                LOCATION_IP, BROWSER_LANG, LOCATION_COUNTRY,
                LOCATION_CONTINENT, LOCATION_PROVIDER
            ]
            (piwik_output[0]).writerow(output)
            output = [LVA_ID, COL_ID, GA_ACTION_URL, 0, GA_ACTION_NAME, 0]
            (piwik_output[1]).writerow(output)
            COL_ID += 1
            LVA_ID += 1
 
if __name__ == "__main__":
    main(sys.argv[1:])

You can download the script here.

The above script outputs 2 csv files which you can then load straight into the piwik tables. You can do that via the console:

Code block

LOAD DATA LOCAL INFILE 'output-log_visit.csv' INTO
TABLE piwik_log_visit FIELDS terminated BY ','
ENCLOSED BY '"' LINES terminated BY '\n';
 
LOAD DATA LOCAL INFILE 'output-log_vaction.csv' INTO
TABLE piwik_log_link_visit_action FIELDS terminated BY ','
ENCLOSED BY '"' LINES terminated BY '\n';

Now I did think that I was all done but there are a couple of caveats before you’ll finally see that visitor graph take shape. When you add a site to piwik it gives it a creation date in the backend database. This is not editable from the front end and piwik will only examine row data which is greater than that date. So change the  ts_created field for your site to the earliest date of the data you have imported. Finally, drop the archive_blob_* tables, these are caches of calculations piwik has done and when missing will be recreated when the dashboard is loaded.

With all that done when you refresh your dashboard you should see your visitor graph with actual data! Huzzah! In the below image I’ve loaded in a csv containing hit data for January into a fresh install of piwik locally.

Google do provide an export API for GA data but I’ve not had the time to become familiar with it. In any case it will only export the same level of data that you get on the website, so even connecting via the API you’ll not get hourly hits. However that could be a starting point for dumping out a list of page data which could be converted into a table of log_actions which is where piwik stores page names and urls for binding to a visit. I’m open to work with someone on that if anyone’s interested. For now this should save me a giant ballache of time on Monday morning.

Blender generative art with halos

Some time ago when I first started to play with blender, particles where about the only half decent thing I could manage. Whilst working with a little script to draw me some cubes at along a grid (yeah, think city generator), I added a particle system to each one and gave them a random color. When I rendered it I thought it made a pretty cool background and played about with adding more variation.

The key to nice particles in blender is often to have more of them and bump up the alpha. Then enable the mysterious “x-alpha” setting in the halo pipeline. To get that feeling of energy or glowing you normally want to have the alpha quite low and the add setting fairly high.

An example blend file is here: halos.

Reddit comment/submission vote tracker

Some time ago I noticed that the up/down count on comments & submissions on reddit would vary wildly under the hood whilst the aggregate displayed would stay fairly steady quite often. By that I mean a comment that had 20 points might be flicking wildy as redditors turn arrows blue & orange.

I thought it might be interesting to graph such activity and I had at first thought of using google charts’ fairly decent api but quickly discovered Flot and was completely sold. If you have ever needed to create some small chart in a web app and started to roll your own solution (very often people will do some graph drawing server side then send over an image) then stop! and go look at Flot. Its an amazing bit of javascript.

In hindsight the “battle of your comment” isn’t as interesting as I thought it would be but it is quite nice to watch how a submission progresses on the site over time. Its only really useful for semi-popular ones as otherwise there just isn’t the data. The whole thing works by appending .json to the reddit url given. That’s the unoffical/offical reddit API. That is then parsed to grab the up/down vote count and then its simply sent back to Flot as JSON itself.

I have noticed that whilst the comment count on the site will vary much more rapidly the JSON file itself doesn’t and seems to lag behind somewhat. I guess that makes sense from a load/abuse point of view. In real terms this means there’s not a lot of point setting the timeout on this page to hammer reddit for “by the second” data.

Some may find it useful so here it is. I have it currently sitting at jaymzcd.webfactional.com. This makes use of cherrypy and jquery. If you’d like the code its here.

Django vs Magento

sigh… Recently I have had to (more than once) explain why we’re choosing Django to develop a e-commerce & web app platform rather than using something off the shelf like Magento. I’ve heard lots of negative things regarding Magento and here’s what I’ve outlined in an email to PM’s etc which others might find useful if they have a similiar “justify it” experience.

The number one reason though I’d rather not work with Magento is that time and again developers express difficulty in making magento work for them and not their magento consultant.

Anywayz, I’m sure someone will take offense to this and I’m well aware I’m comparing a framework to a specific application but whatever…

Actually come to think of it, I’m not even comparing it really django, just listing some gripes and hate I’ve read/heard elsewhere

Code extension & extendability

June 2009 : pickledshark.com

This guy is basically critical of the codebase being complex and difficult to work with. He ends with this:

Due to the Zend/OOP/MVC influence on Magento it is impossible to follow the code. Classes are referenced dynamically, various aspects are contained in XML files and there is no clear flow that you can just debug through. The sheer volume of files and folders makes finding something unbelievably tedious.

Even the database is a minefield. In every other system I have used finding data is easy. In Magento, the use of EAV means that data is split amongst hundreds of abstract tables. Again, it doesn’t flow and it doesn’t make sense without a great deal of time developing a solid understanding of what they have done.

A final Cynical Note… Many people claim that the complexity of Magento is somewhat intentional. The profitability of Magento relys on consulting, technical support and installations. Making the codebase complex could mean that many developers will start out, get stuck and pay for help. If this is the intention of Varien then perhaps they have been very goal focused

Hmm…… Another post here says that for most bits that arent in the free or enterprise version you can probably find a plugin to do it for you at magento-connect. Here’s something you’d imagine would be useful “get the lowest price free” and there is an extension for it. Problem being its $150.
Lowest Price Free plugin

The comments that I find over & over are that the system itself is a nightmare to work from from a developer viewpoint unless you are some magento expert consulting for a fee (big surprise). Eg, here: (april 2009)

The big thing for me right now is the documentation(specifically around customization) – it is nearly non-existent. The forums have TONS more questions that(sic.) answers, most of the Wiki articles focus on simple store administration or VERY simple theme level customizations (via CMS pages or edits to layouts).

Multiple products with similiar data & attributes (eg sizes)

Last post 1 HOUR AGO!

A long discussion about how much of a ball-ache it is to manage multiple products that share a common base (i.e. different sizes of the same product). There is some talk about configurable products, grouped products etc but it seems that even all these still require duplication/entry of the same data per SKU:

This problem is driving me nuts! I have a shop to build with over 100,000 possibilities of one single configurable product. Now… Could anyone tell me, how exactly I am supposed to manage this? I just can’t seemto find anyone dumb enough to add those products manually and I’m certainly not doing it myself! This is magento’s great weakness!

AJAXifing Templates and core features

Last post 3 Weeks ago

Lots of talk here trying to work out how to add ajax functionality to the magento site and surprise surprise, its not supported out of the box and rather complex for most people to try and work out. Instead, well:
Vivendo has a great extension and it’s very affordable!

Whoop-ee! Who’s Vivendo? Oh yeah, the people that write the magento core.

PCI DSS Compliance / Gift Certs / Logging of admin data

For processing cards onsite your cart system should comply to the PCI-DSS regulations. Magento community does not and should be used in situations where you hand payment to another system like paypal or do not store card data on site if using a gateway. The Enterprise (the $8000/yr) system is scheduled to be certified to comply to PCI-DSS soon. Partly because the Enterprise version includes data encryption which the community one does not.

While a Django system would not be “out of the box” PCI-DSS compliant we would be writing the code to store attributes etc and would be able to add an encryption layer (such as 3DES or blowfish etc) to that data. It should also be noted that only the Enterprise edition supports admin logging, gift certs and some nice “extras” such as store credit.

Also the enterprise version is the only one to allow for a proper “walled garden” of catalogues so for example having a section of the store for dealers and not general public orders is not possible with the stock-standard magento. Its also not possible to have sub-admins (could only add new products for example) either something which the django admin site has out of the box (and is easily customizable/extended).

Compare magento versions

Licensing

This is valid at the moment Magento Licesnsing:
As such, you must disclose any changes you make to the OSL 3.0-licensed copyrighted works whenever you distribute Magento or make your Magento store or software available over the web to a third party.

We would probably require a commercial license for magento (i.e. enterprise) if we did not want to re-release any changes to core. Judging from the way people talk about working with magento you will probably end up having to modify core at some point. A Django solution would be completely our own code and even hacking django core, its licensed under BSD meaning commercial works need not release derivative works back to the original source branch.

BSD License Terms
Due to the extremely minimal restrictions of BSD-style licenses, software released under such licenses can be freely modified and used in proprietary (i.e., commercial) software for which the source code is kept secret.

Payment Systems

Magento may or may not support the gateway that we want to use. They have support for a lot (most with an additional fee for the module, seeing a pattern here?):

Supported magento payment systems

Paypal, google checkout, sagepay (protx) are supported with free/paid extensions. The protx one (updated for sage pay) is in beta at the minute and there are 3 pages worth of reviews pointing out problems people are having with it. This is an extra reason for concern for me personally – having to rely on a load of extensions written all by different people, some charged, some not just to get the core functioning. What happens if there’s a bug in one of them?

This is a great extension and have Just upgraded to latest version and now it dont work? Using vewrsion 1.3.1.1 beta.

Fin

Ok, thats probably enough for now. Main point being that this system is customisable – for a fee. If they are mainly concerned about linking systems together then they really need to understand that the data can come as and how they want it. If they want XML of all their orders based on a SKU then we can provide that. If they have custom systems or expect data in such a way then we will fit output to that schema! It still feels like they think magento is this software like microsoft office that is going to somehow solve a load of problems off the bat.

One of the main things for me is that Magento is a dual licensed, essentially paid for system with a big consultant/community that work to push magento. Django is an open source BSD licensed framework built for developers to develop web applications on. Django does not offer paid-for licensed versions, its in django’s interest to have as much and as transparent documentation as possible. For Magento its not quite in their interest to have that same level of documentation, something which seems to be reflected when you read about developers experiences trying to extend it. Invariably it ends with a “magento expert” doing the work or the person struggling to fix it and ending up hacking core or finding some extension that just about does what they want and they live with it.

I know I’m going to end up having some “out of the ordinary” use cases when it comes to discounts, cart functioning and products themselves and the idea of working with magento does not thrill me. When you see posts on stackoverflowasking if theres any other documentation but the source code for the ORM then alarm bells ring. Especially when you’re used to the sort of quality & thorough documentation django provides

End rant! So yeah, I’m not the worlds biggest fan of magento. And I’ve not even touched on the actual quantity & scale of the db tables it installs on a fresh version. Magento fanatics please keep your “its brilliant & you are a knob” mail to yourself.