The world (and word) of jaymz

It’s widely known amongst my friends that I absolutely love the Eurovision song contest. Growing up with a father that was obsessed with war movies my early formative years tended to view europe like this:

Europe for me as a young child was world war Europe. I’d always thought of the many different countries as totally separate and thought the huge amount of cultural difference a difficult thing to placate, especially growing up in Northern Ireland where even people in the same country hated each other.

Then one evening my parents put on BBC1 and bemoaned that “Eurovision” was coming on. I’d no idea what it was and after being told “it’s a singing competition for countries in Europe” I was significantly intrigued enough to demand to watch it. After a few hours I’d decided that this programme alone was responsible for the prevention of world war 3 and began to see Europe more like this:

The reason I believe what many consider a giant waste of time to be the reason we’re not all living under the fourth Reich is simple. If you actually watch the show it’s hard to not get a sense of communal euro-love throughout. Read the rest of this article »

As a coder I tend to not eat as well as I could. I grew up on a diet of home made stews, casseroles, soups and lots more irishy “proper” food. Then I moved to London and got a job. At one point, when I actually lived with other people, I would occasionally cook, but since I moved in on my own I end up eating out a hell of a lot more than I probably should.

Anyway, for sunday style coding sessions I tend to like just zoning out with a pizza and a bottle of pepsi max. I don’t rate pizza hut and papa johns don’t deliver here so dominos is my preferred major pizza chain retailer. Anyone that’s ever bought a pizza from them will know that they’re fucking expensive! It can easily run you £16+ for a “large” pizza. But only a total internet noob would order straight up without checking out the many voucher sites. I say many, I only ever check retailmenot.com (or voucherhub as its UK site is known). This gem just got me the pizza + more for less than the actual pizza SGBLFENR.

It can seem a bit of a waste but you might as well have the money than dominos. And if you’re a social type that has someone else in the house you can be all sharing and get a warm glow inside.

Don’t ask why but out of interest I googled for the string “AAAAAAAA” earlier and after looking at the millions of pages that came back and thinking “wtf”, I searched again only making the string much longer. I was expecting it to just keep going down but at around 20 characters there was a significant jump in returned results. I scratched my beard and proclaimed this interesting (as you can probably tell I have no distractions on valentines day). To skip over further bullshit, this is the graph of 3,328 searches – that is, the number of results for every character (A-Z) repeated one to 128 times. Some of the peaks are interesting.

Why 128 and not something higher? Google wont let you, at least via the query string. The raw data for this graph was generated by a simple python script. If you aren’t coding in python already, please do look into it, its jolly nice.

Code block

#!/usr/bin/python
 
from urllib import FancyURLopener
from BeautifulSoup import BeautifulSoup
import csv
import string 
 
BASE = "http://www.google.com/search?q="
 
class MozOpener(FancyURLopener):
    version = 'Mozilla/5.0 (Windows; U; Windows NT 5.1;' +
' it; rv:1.8.1.11) Gecko/20071127 Firefox/2.0.0.11'
ff = MozOpener()
 
out = csv.writer(open('out.csv', 'w'))
 
headers = ['count',]
for l in string.ascii_uppercase:
    headers.append(l)
out.writerow(headers)
 
for x in range(1, 128): # 128 is the max length of chars allowed
    results = [x,]
    for l in string.ascii_uppercase:
        qry = l*x
        raw_data = ff.open(BASE+qry).read()
        soup = BeautifulSoup(raw_data)
        result = soup.find(id='resultStats').findAll('b')[2]
        result = int(result.contents[0].replace(',', ''))
        results.append(result)
        print [result, qry]
    out.writerow(results)

When that had finished I simply loaded it up into OpenOffice calc (3.2 is out by the way) and plotted it with the result count set to a logarithmic scale so result #1 doesn’t just skew the thing into one boring L shape. The first thing I noticed was a very visible spiking at length 100.

This isn’t so hard to imagine, 100 is a “nice” number. It’s not hard to imagine someone using 100x a character as a test input or just a “long” string. Every character string of length 100 exhibits this spike. The much bigger blue curve is for the letter x. This is used by children and adults alike to mark kisses, and everyone knows the more x’s the more someone loves you. If you look at many of the results for 100 or 101 character X searches it seems to be when people are using it in this context. Could it be that the much bigger spike for the 101 character string X is simply because its 100 + 1 kisses?? Towards the start of the graph there are a number of interesting spikes, I’ve marked some of them along with the length.

Some of these spikes are easy to explain, the biggest number of results returned for a single repeated character phrase is a by product of DNS, yep, its “WWW”. This accounts for the slightly higher result count than the simple “A” with 17,090,000,000 pages returned versus 12,260,000,000. Another easy one is 6 F’s – the hex code for white. I am totally stumped by F’s latter behaviour though, there is a spike at F-31 and F-33 but not F-32. There is a big jump for X-12 and X-34. International mobiles have 12 digits, as do UPC codes but I feel like I’m clutching at straws trying to explain that. Down the other end of the graph, between 115 and 128 the characters P, H, A, M and O all have significant spikes for specific counts. For M & O when you browse many of the first few pages of google’s results many of the pages are using them as part of exaggerated speech. It’s almost like the collective conciousness of the world has decided that 120 characters is just right to describe a particularly tasty dinner.

A spreadsheet with the data I gathered is available via google docs if you’d like to investigate yourself. I should note that the python script above gets its data from .com, if you are using the site to look up some searches it will more than likely switch to your local domain. Depending on your cookies google may also perform extra filtering (e.g. safe searches) so you’re numbers may not be the exact same as mine. I’d be interested in any theories as to some of the more prominent spikes. The OpenOffice spreadsheet with the charts done is also available for download here.

If you are going to play around then save your fingers & sanity and use python to create you’re test strings, just drop into a python shell and use “X”*120 etc or perl -e ‘print “X”x100′ from the command line etc.

I really like GIMP, I use it all the time at work along with Photoshop. The great thing about computers is that you can install whatever the fuck you want. If you fancy driving yourself crazy go google for “photoshop vs gimp” and try and read more than a page of banal arguments bouncing back and forth with very few people piping up to say “use whatever the fuck you want”. A lot of people seem to think that if you don’t use photoshop you’re just an amatuer. Very often these are the same people that are using PS simply to open a photo, click “auto-levels” and then resize it.

If you’ve never used GIMP and you are one of these “pro’s” then I suggest you install it anyway and at very least checkout the selection tools. I know you can transform a selection with Photoshop but I find the marquee tools vastly superior in GIMP and its something I hope finds its way into Adobe’s offering. I’ve gone off on a tangent here, the real purpose of this post is all down to the number fucking one reason people using GIMP complain about it – the multi window interface. This is so much of a PR problem for the GIMP team that they’ve finally started coding a single window mode for the upcoming 2.8.x branch. I have never understood the trouble this causes people. I understand it even less because many people then say Photoshop’s UI is so much better. This winds me up further because whenever I think of PS I think of something like this:

When I think of GIMP I see this:

Do you know what I see here? I see a floating toolbox, a floating image and another fucking set of floating panels. I see the same fucking thing. Ok, Photoshops menu is right at the top since its a mac but I really have never understood what the hell is so wrong with this. Very often even the linux users complain about this, when really what they should do is maximise the image and then mark the two floating panels as “always on top”. I get that’s not ideal but hey, you’re using a free OS, with a free desktop with a free graphics app. Work with us 1 fucking minute!

No doubt I am simply delusional but I really have never found the GIMP UI, especially when compared to the PS UI on a mac, to be such a problem that you literally couldn’t fucking use it.

This is my typical desktop (well one of them) when I’m doing work at home. I really like kate as my code editor. It does pretty much everything I need and I like the built in terminal and the extensions & scripts you can use with it. But I’m not here to proclaim another holy war begun, instead I want to outline some of the tools I’m using in the background to avoid ripping out my hair and generally being more productive as a *nix based developer. For the most part, half of this shit works fine on a mac too out of the box. Something my good friend ServeBlunted will happily remind me of.

I say “some” tools. I really only mean one. SSH. Thats it. As a linux user both at home & at work I spend a lot of my time in X. It can be handy to have a GUI app I’ve got on my work machine running at home if I’m working so when I ssh into work I always tunnel through X. You can do this by just adding “-X” as an option.

ssh -X jaymz@work.company.com

X tunelling is handy, it sure beats trying to work via the likes of a VNC session, but the best thing about SSH for me is sshfs. Using that you can create a folder on your local machine and then mount a remote folder over SSH to it. Then you fire up your editor and work away. If you’re like “yeah, whatever, I can edit my files via FTP in my editor and it syncs” then shut up, go give it a go and get back to me. It’s a lot more flexible, you almost forget its a remote filesystem. To mount a remote folder as if it’s sitting on your home machine its typically done like this:

sshfs /tmp/local jaymz@work.company.com:/var/www/dev-code

Where you’d replace the names of the folders with whatever works for you. This lets me mount my code and work on it from within any application as if it was totally local. In the above screenshot I’m actually working on code on my work computer a few miles away. I’ve also ssh’d in via the built in terminal so for all intents and purposes its just like I’m working at the office. The final thing that ssh makes easy for me is port tunneling. I don’t use this as much but it can be handy. The basic command is:

ssh -L 8080:10.10.91.1:445 jaymz@bowser.worknetwork.com

The red number is the local port. This is what you are going to be using in any config or browser screen on your workstation. The green IP address is the machine you’re actually interested in connecting to and finally the orange port is the port in question that you want to connect to. Then you give it your login credentials as you would normally. This is really useful for when you have dev servers that are locked down to only local network connections (like a mysql setup to only connect via localhost for example). The above is what I would use if I wanted to connect to the firewall’s web interface. This is only accesible to my workstation in the office but using tunnelling I can bring it up from home without tunnelling X and loading firefox (which would also work ).

There are a whole lot of ways to set tunnels up with SSH, the -L means you are wanting the connection to go from your local machine to the SSH box and then to the destination. You can also use -R which goes from the SSH box to your local machine and then to the destination. You might need to read that a few times There’s a whole lot more info on tunneling here.

Ok, that might seem an semi-inflammatory title if your a little house publisher but its not so much because they make your life “unlocking” some software infinitely easier. No, I love keygens because they tend to take me way back to the early nineties when my only exposure to computing was the venerable commodore 64 at home.

That may seem odd to most people who haven’t come across these tiny wonders of software engineering but keygens really are some of the most creative and user friendly programs you’ll ever encounter. Really… And to prove that I’m going to look at a few of my favourites.

First off, my new flavour of the month, its the amazingly complete Adobe CS4 creator from CORE. CORE, or Challenge of Reverse Engineering to give them their full title, really epitomise what I love about keygens. First off its got a nice look to it. These things tend to not bother with a standard interface, instead opting to have something graphically out-there. This is pretty tame compared to the other I’ll be mentioning but it does have that “underground” look to it. You dont look at an app like that and think “ooh! corporate multinational”.

This thing is jam packed with keygen goodness

It is hard to understate just how feature-complete this one is. It has an alog to create serials for applications I didnt even know adobe made. Robohelp? What the fuck is Robohelp? Some designers of these things go all out for it and really do make something nice to look at. For such an example I’ll point to (again adobe, they seem to really go for it on these releases) Paradox’s Photoshop CS2 generator:

I would pirate CS2 today just to use this

Aesthetics aside the reason I’m forever enamoured with these things is the tendency for the authors to include a 4 or 5 minute long 8-bit style music accompaniment to start blasting out the minute you open them. As a child of the late eighties/early nineties I found my niche in the commodore 64 and the unique sound of the SID chip is now chemically associated with happiness in my brain. As such once these things start up and begin dropping synth beats I’m instantly taken back to the very dawn of my computing experience. When I was just getting started on what has been the bulk of my life for 18 years. Paradox have gone for a slightly haunting tune that keeps building up and up with a real trippy sound to it. I’ve even had it just loop over & over in the background it really is that good. Sinergy has this upbeat party tune that puts me in mind ofFlimbos Quest whilst CORE went for something that wouldnt be out of place in the likes of an 80′s TV show.

Now, you probably have to appreciate the nuances of the SID/8-bit/synth style though to listen to these and not go wtf when you hear it. There are samples a plenty on youtube, if you do happen to enjoy it then check out some of Martin Gallway’s stuff, a pioneer and legend on the sid scene.

Putting on my ‘I can appreciate a good UI’ hat the other thing you’ll find with keygens is that they are totally aware of their purpose. You wont find menu’s and dialogs or settings to tick or files to open. No you tend to get an astonishingly complete list of products & versions that it’ll handle and a simple “generate” button and job done. They give you exactly what you want straight up, there is little in the way of bullshit (if you’re not counting the creative reign with the look & feel). I’m always amazed at the slickness of some of these things. The CORE & Paradox gen’s mentioned earlier are great examples as is this rather old offering from sinergy for Flash MX:

Not much to look at but spot on functionality and a pick me up tune

I think the phrase I’m looking for is “does what it says on the tin”.

Paradox’s serial/auth-request/auth-code flow is an especially pleasant total breath of fresh air when you’ve become accustomed to jumping through hoops following wizards and other ways to hand-hold your way through user land. Its no lie to say that I now look forward just as much to that years new CS-whatever keygen than the actual suite itself.

As a developer, I should also mention just how amazing these things are as an example of pure software engineering. The CORE CS4 ‘gen weighs in at under 95kb. Yeah, K. When you consider just how big applications are these days it’s a refreshing reminder of just how much you can get out of a tiny amount of code. Most word docs are bigger than that. And they don’t come playing 8-bit style tunes. The paradox one is a more meaty (but understandable given its more polished UI and auditory experience) 186kb, with sinergy an anorexic 75kb. I am regularly amazed/shocked at the size of applications these days. Games I can understand, thats down to textures, audio, video sequences etc. But 4Gb applications!? Still, when compared to the rate of change of hard drive capacity things have amazingly “gotten better”. Kind of. Anyway, all that shit is off topic…

Very often these things are programmed straight up in assembly. It’s not hard to imagine given the guys that manage to reverse out the serial checking algorithm are normally staring at screen after screen of hex & rather basic assembler from a debugger. I have a major respect for guys that practise the dark art of reversing. If you’re curious about such things then you might want to do some googling for crackme’s. These are little apps written often by other reversers as learning exercises. My friend Sarid (I will forever know him asSurreal from when I was a kid but he’s moved on from that moniker) just so happens to be a rather good practitioner of this skill so I’d recommend checking out his blog where you can find some crackme’s too.

Even if you have no interest in the actual program its pumping out numbers for do yourself a favour and check out these awesome bits of software just to marvel at the audio feast you’ll encounter. Of course most people will (should?) question the idea of running random exe’s of a few kb’s but if you check out the trusted torrents and well commented ones on sites like tpb or mininova then you’re probably all right. Anyway, we can have virtual machines now…

I have no doubt that there is likely legislation that would make it silly of me to directly offer such things for download so I’ll just point you somewhere that *might* just know.

On a totally unrelated note, I’ve written this whilst waiting for that fucking ISP of mine (tiscali) to come back online (about 3 hours now). Do yourself a favour and avoid this piece of fucking shit joke of a carrier.

sigh… Recently I have had to (more than once) explain why we’re choosing Django to develop a e-commerce & web app platform rather than using something off the shelf like Magento. I’ve heard lots of negative things regarding Magento and here’s what I’ve outlined in an email to PM’s etc which others might find useful if they have a similiar “justify it” experience.

The number one reason though I’d rather not work with Magento is that time and again developers express difficulty in making magento work for them and not their magento consultant.

Anywayz, I’m sure someone will take offense to this and I’m well aware I’m comparing a framework to a specific application but whatever…

Actually come to think of it, I’m not even comparing it really django, just listing some gripes and hate I’ve read/heard elsewhere

Code extension & extendability

June 2009 : pickledshark.com

This guy is basically critical of the codebase being complex and difficult to work with. He ends with this:

Due to the Zend/OOP/MVC influence on Magento it is impossible to follow the code. Classes are referenced dynamically, various aspects are contained in XML files and there is no clear flow that you can just debug through. The sheer volume of files and folders makes finding something unbelievably tedious.

Even the database is a minefield. In every other system I have used finding data is easy. In Magento, the use of EAV means that data is split amongst hundreds of abstract tables. Again, it doesn’t flow and it doesn’t make sense without a great deal of time developing a solid understanding of what they have done.

A final Cynical Note… Many people claim that the complexity of Magento is somewhat intentional. The profitability of Magento relys on consulting, technical support and installations. Making the codebase complex could mean that many developers will start out, get stuck and pay for help. If this is the intention of Varien then perhaps they have been very goal focused

Hmm…… Another post here says that for most bits that arent in the free or enterprise version you can probably find a plugin to do it for you at magento-connect. Here’s something you’d imagine would be useful “get the lowest price free” and there is an extension for it. Problem being its $150.
Lowest Price Free plugin

The comments that I find over & over are that the system itself is a nightmare to work from from a developer viewpoint unless you are some magento expert consulting for a fee (big surprise). Eg, here: (april 2009)

The big thing for me right now is the documentation(specifically around customization) – it is nearly non-existent. The forums have TONS more questions that(sic.) answers, most of the Wiki articles focus on simple store administration or VERY simple theme level customizations (via CMS pages or edits to layouts).

Multiple products with similiar data & attributes (eg sizes)

Last post 1 HOUR AGO!

A long discussion about how much of a ball-ache it is to manage multiple products that share a common base (i.e. different sizes of the same product). There is some talk about configurable products, grouped products etc but it seems that even all these still require duplication/entry of the same data per SKU:

This problem is driving me nuts! I have a shop to build with over 100,000 possibilities of one single configurable product. Now… Could anyone tell me, how exactly I am supposed to manage this? I just can’t seemto find anyone dumb enough to add those products manually and I’m certainly not doing it myself! This is magento’s great weakness!

AJAXifing Templates and core features

Last post 3 Weeks ago

Lots of talk here trying to work out how to add ajax functionality to the magento site and surprise surprise, its not supported out of the box and rather complex for most people to try and work out. Instead, well:
Vivendo has a great extension and it’s very affordable!

Whoop-ee! Who’s Vivendo? Oh yeah, the people that write the magento core.

PCI DSS Compliance / Gift Certs / Logging of admin data

For processing cards onsite your cart system should comply to the PCI-DSS regulations. Magento community does not and should be used in situations where you hand payment to another system like paypal or do not store card data on site if using a gateway. The Enterprise (the $8000/yr) system is scheduled to be certified to comply to PCI-DSS soon. Partly because the Enterprise version includes data encryption which the community one does not.

While a Django system would not be “out of the box” PCI-DSS compliant we would be writing the code to store attributes etc and would be able to add an encryption layer (such as 3DES or blowfish etc) to that data. It should also be noted that only the Enterprise edition supports admin logging, gift certs and some nice “extras” such as store credit.

Also the enterprise version is the only one to allow for a proper “walled garden” of catalogues so for example having a section of the store for dealers and not general public orders is not possible with the stock-standard magento. Its also not possible to have sub-admins (could only add new products for example) either something which the django admin site has out of the box (and is easily customizable/extended).

Compare magento versions

Licensing

This is valid at the moment Magento Licesnsing:
As such, you must disclose any changes you make to the OSL 3.0-licensed copyrighted works whenever you distribute Magento or make your Magento store or software available over the web to a third party.

We would probably require a commercial license for magento (i.e. enterprise) if we did not want to re-release any changes to core. Judging from the way people talk about working with magento you will probably end up having to modify core at some point. A Django solution would be completely our own code and even hacking django core, its licensed under BSD meaning commercial works need not release derivative works back to the original source branch.

BSD License Terms
Due to the extremely minimal restrictions of BSD-style licenses, software released under such licenses can be freely modified and used in proprietary (i.e., commercial) software for which the source code is kept secret.

Payment Systems

Magento may or may not support the gateway that we want to use. They have support for a lot (most with an additional fee for the module, seeing a pattern here?):

Supported magento payment systems

Paypal, google checkout, sagepay (protx) are supported with free/paid extensions. The protx one (updated for sage pay) is in beta at the minute and there are 3 pages worth of reviews pointing out problems people are having with it. This is an extra reason for concern for me personally – having to rely on a load of extensions written all by different people, some charged, some not just to get the core functioning. What happens if there’s a bug in one of them?

This is a great extension and have Just upgraded to latest version and now it dont work? Using vewrsion 1.3.1.1 beta.

Fin

Ok, thats probably enough for now. Main point being that this system is customisable – for a fee. If they are mainly concerned about linking systems together then they really need to understand that the data can come as and how they want it. If they want XML of all their orders based on a SKU then we can provide that. If they have custom systems or expect data in such a way then we will fit output to that schema! It still feels like they think magento is this software like microsoft office that is going to somehow solve a load of problems off the bat.

One of the main things for me is that Magento is a dual licensed, essentially paid for system with a big consultant/community that work to push magento. Django is an open source BSD licensed framework built for developers to develop web applications on. Django does not offer paid-for licensed versions, its in django’s interest to have as much and as transparent documentation as possible. For Magento its not quite in their interest to have that same level of documentation, something which seems to be reflected when you read about developers experiences trying to extend it. Invariably it ends with a “magento expert” doing the work or the person struggling to fix it and ending up hacking core or finding some extension that just about does what they want and they live with it.

I know I’m going to end up having some “out of the ordinary” use cases when it comes to discounts, cart functioning and products themselves and the idea of working with magento does not thrill me. When you see posts on stackoverflowasking if theres any other documentation but the source code for the ORM then alarm bells ring. Especially when you’re used to the sort of quality & thorough documentation django provides

End rant! So yeah, I’m not the worlds biggest fan of magento. And I’ve not even touched on the actual quantity & scale of the db tables it installs on a fresh version. Magento fanatics please keep your “its brilliant & you are a knob” mail to yourself.

I am happy to hear that finally that stuffy cow with no respect for peoples liberty or freedoms is finally going. Yes, Jacqui Smith is out. As is Hazel Blears. Both have shown complete ignorance for the populous. Smith has trampled on civil liberties in this country for too long and Blears is one of those people you could slap. Both have been home secretaries and both seem to think that they know best.

It was Smiths refusal to listen to the very board the government have to investigate drugs that led to the purely political decision to re-classify cannabis. Hazel Blears has effectively just thrown her toys out of her pram after Gordon Brown criticized her abuse of the expense system. I feel a little sorry for him surrounded by assholes like this (but a general election wouldn’t hurt either).

Cunts like this totally ruin politics and lead to disillusion. I’m glad the bitch(es) are gone but, like someone on reddit mentioned, it is a shame they will be remembered for the abuse of expenses rather than the abuse of civil liberties the pair have inflicted on this nation.

Jacqui Smith / HAZEL BLEARS – FUCK OFF YOU LOATHSOME PRICKS

I have spent the past 3 working days bug fixing google checkout code and it has been, to put it mildly a total headfuck. Fixing bugs & code refactoring is of course the mainstay of the working programmer. It’s fun when you’re working through something, you can see progress, but bug fixing is like travelling through London. You feel like you’ve gone miles when really you’ve only made it a couple of stops and it would’ve been quicker to walk.

First off it took the best part of an afternoon to realise that the code I was modifying wasn’t actually being called. Now I’ve never worked with googlecheckout before so you can probably chalk a few hours up to that. For googlecheckout you set the callback code in the account preferences for the store’s account. Why on fucking earth that can’t be passed in the calling code I don’t know. After working that out and seeing that the myriad of tests I’d been doing had actually screwed up a live database the majority of the blood drained from my fingers.

The next two hours where spent looking through log files in a terminal prompt. That was fairly thrilling. At the end of the day I felt like I’d done ten minutes work. Thats sort of the problem with fixing bugs, you can spend hours looking at code, logs and status numbers and when you finally find the error, which is probably something easy, the sheer simplicity of the fix and the few lines of code you need to modify make you feel like you’ve not done much.

So in that way you can kind of compare fixing bugs in source code (especially source code that involves 4 separate processes to actually run) to sex. To get sex you do hours if not days of monotonous, boring work for what boils down to 15 (or maybe 30 if you’ve had a good meal beforehand) minutes of payoff. Obviously this metaphor breaks down rather quickly since fixing source code bugs is a productive pursuit…

Thankfully, on the third day and several hundred tests later (at one point I charged my card over £600 when I got so used to mindlessly clicking through purchase screens) I started making actual progress. In the end I’ve probably changed or moved around about 15 lines of code. I’m now a lot more familiar with code igniter and google checkout but I can’t help but feel the past few days have been slow going and unproductive. Tomorrow I’m bugfixing yet again but now that I have a setup thats working these last few bugs shouldn’t be anywhere near as bad.

One last thing, the documentation for the PHP api of googlecheckout is fucking abysmal.