Tom Morris

Hackdiary: Wikihacking in Brighton

This weekend, I went down to Brighton for the MediaWiki Hackathon. I’m not a MediaWiki developer, and I only really do PHP when I need to. I do have access to the Toolserver which is a live, read-only mirror of the databases that power the Wikimedia projects. This is very useful: if I need to, I can log in via SSH, type in sql -r enwiki_p and run SQL queries against Wikipedia.

So, what did I build at the Hackathon?

Not as much as I’d like. The four regular MediaWiki developers there smashed lots of bugs¹. Me? I worked on a few different things.

Open Plaques

I helped Jez from Open Plaques with the MediaWiki API, specifically Open Plaques can start using Wikimedia Commons to host images of plaques. A while back, I started pushing at Open Plaques to have Commons as an alternative file host to Flickr. Currently, Open Plaques recommends that you take a photo, CC license it, then post it on Flickr with a machine tag, and then Open Plaques pulls it in, and Flickr links to Open Plaques.

But Open Plaques could also support Commons in three ways:

1. By making it easy to add images from Commons.
2. We might be able to export all the license compatible files from Flickr to Commons, but with added metadata from Open Plaques (OP data is public domain although the photos are under various CC licenses, not all of which are Commons compatible).
3. Using Commons as a file hosting back-end: when people come to Open Plaques, they could upload photos directly to Open Plaques, and we’d then push them straight on to Commons and use that as a file host. They’d obviously have to agree to the relevant license and so on.

The first thing is a fairly straightforward one: being able to simply provide a Commons URL and then extract image metadata and the path to the image. That’s relatively easy, and it looks like that might be something we can add.

Scala

After Jez left, I tried to work out what to do next. I decided it might be useful to have an API library in my trendier-than-thou functional programming language of choice, Scala. On the bus home, I made a few notes about the general design of such a library. After a bit of noodling with Maven, I actually got to the point where i could type mvn compile.

That I have to play these silly games every time I start a new Scala project is profoundly depressing. I basically either need to stop being a wimp and fully embrace SBT 0.10/0.11, or I need to write myself a new Maven archetype that does Scala properly and has all the libraries I want. And to wrap said archetype in a command line alias called something like “scala-new-project”. Scala is supposed to be a fun, pragmatic and functional (in the sense of not-dysfunctional) language. Choosing between SBT and Maven is basically a choice between choosing a build system designed by hipsters and a build system designed by enterprise people. Whatever choice you make you will regret.

My build still has one major issue: mvn scala:console gives me errors, and JLine still buggers up my shell after I exit the damn console.

Databinder Dispatch is a breath of fresh air. It is intimidating at first to use a library where 70% of it seems to be punctuation, but the design decisions make sense. For using the MediaWiki API, it’s actually very easy: you can basically do each different thing as a series of layered objects. Firstly, you construct a Request object that points to the API endpoint (MediaWiki isn’t a RESTful API), then you use the <:< pseudo-operator to add all the relevant headers (specifically User-Agent), and then you can do it again to add in the authentication token (cookies rather than OAuth, but the principle is the same). Then, finally, you can simply provide a Map of the query you are sending, and then you supply an inline response handler.

I haven’t yet gotten anywhere particularly interesting with the Scala library, but I’ve got some good ideas. It’s currently in a closed repo. When it sucks less, I’ll release it. That might not be for some time, sadly.

VMs, vagrants and maintenance/dev/

Over the weekend, I mentioned to one of the experienced MediaWiki developers about one of the things that puts me off MediaWiki hacking: dependencies. Hacking on MediaWiki generally means having a working MySQL install and a working Apache install, and so on. This can be a giant pain-in-the-ass on OS X, as the bazillions of tutorials on how to set them all up in the right way has shown. There’s a reason why people are using things like Vagrant and other VM based systems. The same sort of quasi-VM type strategy seems to be happening with RVM, the Ruby Version Manager, which the Ruby community nicked from Python’s virtualenv. Obviously, if you are in Javaland, you are running in a VM… the JVM.

But PHP is a bit more of a pain: this is the downside of being built around the very reasonable use case of “I want to be able to FTP it onto my server and have it work”. What you lose in the development stage, you more than make up for in deployment.

At this point, someone pointed me to this discussion on wikitech-l.

The current bleeding edge MediaWiki basically has a script/server, and a built-in SQLite. It’s almost like… Rails! You simply run maintenance/dev/install.php and it downloads PHP 5.4, and sets you up a development version of MediaWiki that uses PHP 5.4’s built in web server. You can then just run maintenance/dev/start.sh and it’ll boot up on port 4881.

There was one slight hiccup: the PHP 5.4 build script didn’t like the fact that i had a space in one of the names of a parent directory to the SVN checkout of MediaWiki and borked. I quickly renamed it to remove the space and it compiled fine. Once the initial compile was done, I can now boot up a new, clean MediaWiki install in a few seconds. This makes it dramatically easier to start hacking on MediaWiki.

And now.

Although I didn’t do any MediaWiki hacking over the weekend, I’ve just assigned to myself an issue. Lemme see if I can get some of my code running on Wikipedia…

Folding in Ruby and Scala

I’d like to think I’m no slouch at functional programming, but I came across a nice little example today of where Ruby’s functional programming features weird me out: inject.

Consider a perfectly simple example of where functional programming would be useful: summing an array of objects of the same type that all have a property or method that returns an integer value.

Here is how one might do it in Java (or C, PHP or another language without the benefit of higher order functions):

That’s easy enough. You might need to deal with what happens the length value is nil, and you might need to handle exceptions. Whatever.

But, of course, functional programming is here to make your life better. You shouldn’t need to write the same foreach-accumulator code every time for every different property. So, imagine if we had an array of objects that had five different properties that are all integers: length, height, weight, smelliness. We want to sum them all, or average them all or whatever. This is where higher-order functions become so useful.

In Scala:

Okay, now imagine we’ve got something similar in Ruby. The code transliterates reasonably well.

Two nitpicky things here about syntax:

1. The map syntax is simpler in Scala: I’ve always found “and-colon” in Ruby to be a rather ugly and unreadable construction. I understand it alright, but I’m not confident that Ruby newbies understand it. (But, then, lotsa Ruby newbies don’t grok functional programming: I certainly didn’t.)
2. The word “inject” is a bloody strange word for a well-understood concept: every other language with fold-left calls it fold-left (or foldLeft or foldl or fold or something similar involving the word “fold”). “inject” is eminently forgettable.

I could, of course, monkey patch in “fold_left” as an alias to inject.

Here is where it gets more interesting for Scala though. The type system lets you do type-dependent monkey patching.

Consider the generic concept of folding or reducing values: it is simply a process where you turn numerous values of the same type into a unified result of a particular type. So, summing is a folding operation, as is producing a mean, a mode, a median etc.

Enter Scala implicits.

Now one can do something like this…

How does this differ from just monkey patching ‘sum’ into Ruby’s Array class? It only applies to lists made exclusively of one particular type: Ints. Implicits let you target specific types in a way that Ruby monkey patching doesn’t. This is pretty powerful, and yet another reason to learn to love static typing.

You can call sum on List[Int] but not on List[String] (unless you define a new class that implements a sum method and an implicit conversion function for List[String]). Without one of those, you’ll get a type error if you call something like the following:

Think about the power of this: programming often boils down to doing some repetitive action to a list of values of the same type. In Ruby, you can monkey patch the individual class, but think about how you can make code more readable by adding methods to both the objects and the collections of objects, without those modifications spilling out and affecting collections of objects of other types.

I think this is a real example of where Scala’s static type system combined with implicits and functional programming allows for a far more expressive way of coding.

Trying IronPython

For reasons I can’t talk about but which may or may not become clear soon, I’m brushing up on my Python. I decided today to give IronPython a try, since of the combinations of zip (Iron, J) (Ruby, Python) it was the only one I hadn’t really tried.

As I don’t really do Windows or Mono, I didn’t really know where to start. With most JVM-but-not-Java languages I try, the first thing I do to test the interop is to try and create a collections object from the JCF. This seems like a pretty good test of a .NET language like IronPython.

Doing so is easy enough.

from System.Collections.Generic import Dictionary
foo = Dictionary[str, float]()
foo.Add("bar", 5)
foo.TryGetValue("bar") #=> (True, 5.0)

But what if I want to iterate through the .NET dictionary? Well, if it were a Python dictionary, I’d do something like the following. So let’s try that.

for k, v in foo:
    print k, v

TypeError: ‘KeyValuePair[str, float]’ object not iterable

Okay, that makes sense. It’s basically a list with a load of KeyValuePair’s in the list, and a convenience method which finds the one which matches the key.

So I have to do this instead:

for i in foo:
    print i.Key, i.Value

This will take some getting used to.

Interestingly, this is where my lack of Python skills gets annoying. If this were Ruby and I didn’t like how it was behaving, I’d reopen that class and patch it into shape (and then curse myself later on). With Scala, I could do the same, albeit with some magic and indirection.

Actually, I figured it out.

So, you define some function that does it using yield.

def dotnet_iter(coll):
    for p in coll:
        yield (p.Key, p.Value)

Now you call that to iterate the .NET collections:

for k, v in dotnet_iter(foo):
    print k, v

Obviously, you can wrap that into objects for abstraction.

One other slight annoyance with IronPython that slightly got me coming from Ruby and JRuby is that JRuby does some clever stuff with function/method names. So, imagine you’ve got a Java class that uses the standard Java way of naming things, camelCaseWithNoLeadingUppercaseCharacter(). In JRuby, you can call this as either that or as camel_case_with_no_leading_uppercase_character - the Rubyish way of naming methods.

In IronPython, no such luck. The .NET thing of giving you upper case property/method names is a bit jarring in Python (or Ruby).

The other irritation I find with trying out anything Iron- is Microsoft’s bloody abysmal terminal. It truly is the worst bit about using a Windows machine. Everything else you can learn to tolerate, but the console is absolutely soul-destroying. The crappy font, mark mode, no readline. And it’s everywhere. Everything you use brings up a horrible blood-curdling DOS-era terminal. Even PowerShell: Microsoft’s “next-gen” shell for leet sysadmin-fu. I do not know how the Windows gang get any hacking done. (Don’t tell me: Visual Studio.)

Things I’m currently quite excited about

There’s a few tech things I’m pretty excited about at the moment.

Firstly, Scala is now at the point where it is actually productive. Scala 2.9.0 RC1 has just come out and it’s got freaking parallel collections in the standard library. Finally, I can stop looking so longingly at Clojure. As far as I can tell, you’ll be able to do something like this:

Array(1,2,3,4,5) par map (_ * 2) seq

And it’ll make an array, turn it into a ParArray, map it, and turn it back into a sequential Array. That’s pretty cool.

Then there’s Scalatra. Scalatra is Sinatra but for Scala. And the setup is very simple. Once you’ve written the app, you get a WAR file, which you can upload to…

Elastic Beanstalk. Finally, Amazon have delivered a Heroku-like experience. This is really exciting for hacking stuff together. For 2¢ an hour, you can host JVM apps on an EC2 micro-slice, but without having to touch the operating system: you literally go to the website, pick a subdomain and upload a JAR file. The Hello World I wrote in Scalatra was about 5Mb.

Completely unrelated: WebGL. Holy moley is that exciting. If you are running Chrome or Firefox 4, go and ogle this. It’s beautiful. I’m not nearly clever enough to do 3D programming, but just imagine a world of composable 3D objects with URIs and a universe of RDF-esque ties between them. Virtual objects roaming the one Web intermingled with thoughts and data in some kind of bizarre interlinked Platonic ether. That was kind of the dream of VRML, right? And is sort of the reality that Second Life has. But we could go further.

Imagine Wikipedia with WebGL: you click a little link in Wikipedia, and the article fades slightly only for the objects the article is representing to appear in front of you. You can rotate them, download them, throw them around in physics demonstrations, attach them to your avatar. But then each part of the WebGL model could have RDF metadata attached: this particular set of polygons is supposed to represent a particular type of material. You click and all the other virtual objects made of iron or copper or polyurethane or semi-transparent blue plastic or plutonium appear before you. As you are exploring Wikipedia, you copy a few items you find into your personal inventory, a sort of 3D version of your web bookmarks. Then you come across the Wikipedia article on nuclear reactors. And you are a curious sort, so you start seeing what happens if you chuck all sorts of strange things into the nuclear reactor. What happens if you drop this 3D model of Thomas Jefferson in to the reactor? Suddenly, our mashups have stopped being dumping crime figures on Google Maps and they’ve become shockingly real world and Grand Theft Auto-ish.

What else? Open Election Data. This one’s a simple one: a new link in the Web of Data, joining data.gov.uk up with dbpedia. Haters gonna hate, right, but making it so that when one dataset is talking about “the Monster Raving Looney Party”, they use an identifier that is linked to all the other representations of the same thing is the absolute start of the Web of Data. We may have to have these proxy projects to sit as intermediaries between governments and more direct community-created resources like Wikipedia/Dbpedia. This is an interesting start.

One final thing: Wikimedia. I’ve only got back into Wikipedia recently, and there’s loads of fun stuff we can do with the data. I’ve got a list of hacks as long as my arm. I may work on one tomorrow at Rewired State.

HackDiary: Start actually doing something

Tonight I reaffirmed a simple truth often discovered by programmers: keep it simple and just get on with it.

I’ve got a project I’ve been working on for a month or so now. It’s a web project in Scala. The actual web front end is extremely minimal: in terms of interface, we’re talking old-school Google here. A box where you enter a string, and a web page you get in response. No sessions or other complexity.

But once you enter said string, a lot of parsing and magic has to go on behind the scenes. And more importantly than that, the project is going to be open source and have an API that other programmers will have to write to.

I’ve been basically fucking around with the project for a month or so. I’ve been modelling the API and drawing fancy diagrams and doing way too much design up front. But unless this actually works, it’s mental masturbation. And mental masturbation is a lot less enjoyable than physical masturbation.

Yesterday, I got Scalate to run on Elastic Beanstalk in about ten minutes. Inspired by this, I decided to “throw one away” with my existing code and start afresh.

I took the same code I’d used to produce the Scalate HelloWorld, changed all the project names and class/package names, added some more HTML to the get("/") method, and rewrote the basic logic of the application.

Getting the web front-end working took about half an hour, and it’s taken about another hour to get the basic logic of the back-end plugin architecture working. But now I actually have some working code. This is good. In total, I’ve written about 72 lines of code including HTML, but excluding the web.xml file.¹ In total, I’ve created four separate compilation units (basically source files), a total of four classes/singletons (one of which is just an abstract class to define the plugin interface).

Now, the previous incarnation of the same project had about ten compilation units, but that probably meant about 15 or 20 classes/singletons. I couldn’t quite decide on the API. But because there was no actual working code, it was all masturbation.

There’ll be bits I can pull into the new version. There’s at least two classes I can just copy straight in because I’m happy with them. But a lot of it I can throw away.

There is such satisfaction in getting things to work, and it provides such a good base to work on. Once the code actually does something, you have something to build on. You can build one or two plugin modules and show some real-world results. You can then abstract it all away a bit, refactor and start adding new features.

That I can sbt package and put it up on Beanstalk in a few minutes means I can show someone a prototype if, you know, hypothetically, I were trying to get funding (cough).

Here is my prescription if you have a project where the code compiles just fine but doesn’t actually do anything useful.

mv project/ project-old/

mkdir project/

cd project

Now get on with it by building the simplest prototype you can. Now your mental masturbation can be turned into actually building features, and you can transfer your masturbation energy back to physical masturbation.²

I know, I know. I’m probably the fifth person to give this lecture in the last hour. But O(1) bird in the hand is worth O(2ⁿ) in the bush, right?³

If you want to go to things like Hack Days and HackCamps, this important skill of actually doing something is a profoundly important necessity. One very good way of learning to do that is to try something new. You’ll naturally want to get it working in order to shortcut your own impatience.

I tried Visual Studio 2010 with .NET 4 on Windows 7 the other day because of the thought of building Windows apps. I don’t know C# very well, but the curiosity about the platform kind of nudges you into actually doing something small to see if it works. And, what do you know, in about half an hour, it’s actually possible to build a simple Windows app from scratch. That’s pretty cool. It’s such an easy win, and it prompts you into a sort of change in perspective: if you can build a Windows app in half an hour, or a Mac app with MacRuby too, or put up a Java web app in ten minutes, you raise your consciousness of what is actually possible. Doing things makes you aware that you can do things.

Yeah, yeah, insert appropriate cynicism if you want. Actually, don’t. Fuck you. This is my HackDiary. I can be non-cynical and point out the bleeding obvious. Fuck the cynicism. I’m gonna end on a nice cute puppy picture.

Actually achieving tangible things in programming makes me happy. As do puppies.

WikiLovesGeo: We have explosive

The project I worked on last weekend at History Hack Day is now in private alpha, by which I mean it’s finally running on my laptop. Here’s proof of it working in the form of some screenshots.

The current version uses the following technology stack: a simple Ruby script that hooks up to Twitter using the User Streams API and parses the stream of JSON. It finds any geotagged tweet from anyone the wikilovesgeo user is following (just me at the moment), then does a lookup on the co-ordinates and if it finds anywhere suitably close to you sends back a tweet. Simple enough.

The data storage is done with MongoDB. I was going to use PostgreSQL for this for two reasons: firstly, because it would be an excuse to play with PostGIS, and secondly because if I want to host it on another machine, I think Heroku do Postgres with PostGIS and so do Stax.net. But I couldn’t find documentation for a mere mortal to work out how to do PostGIS (I since have), so ended up using MongoDB in anger, thinking that it was fast enough that if I needed to write my own geo queries, it probably would be fine.

MongoDB is surprisingly good for what I’m doing. It’s just one giant collection of locations at the moment with three fields: name, co-ordinates and ID. MongoDB can be used for doing simple geo queries and it’s not at all bad. I mean, they are using it for Foursquare, so obviously the geo functionality has to be good enough for that.

Where’s it going to go then? Well, I’m planning to try and do some experiments in London over the weekend with it just running on my laptop and me accessing it using Twitter over 3G. If that all goes well, I’ll move it over to one of two always-on machines in my house and then open it up for private beta.

The way it’ll work will be like this: I’ll start picking people who are following the account. Once the account follows you, it’ll just tell you there’s something nearby when you post something. So beyond following the account and it following you back, there’s no signup or anything.

If that all works out, then we can make it public, and I’ll launch it properly with big announcements and stuff. There’s some more code I need to write before it can go properly public: something to handle follows and unfollows. And I need to think about spam.

There’s probably a scalability issue: if a few hundred thousand people followed it and started geo-tweeting heavily, I’m not sure what would happen with API limits and just the processing power needed on the server to handle all that. Well, bridge that when we come to it. If there are hundreds of thousands of people who want to go and take photos to illustrate Wikipedia, that’s a problem I can probably live with.

I may also build a web front end sometime: basically, I’d need to build a little web application that lets you find things that need photographing. That will probably happen later rather than sooner, because for that to happen, the database needs to be somewhere other than my laptop.

And finally, I need to write a periodical scraper that will go and poke away at the relevant categories on Wikipedia. My current script is an ugly hack: it basically does a recursive search of all the categories, but there’s a problem with the queues getting too long once it hits five or six levels of tree structure. And there’s no way of delisting a page yet. Coverage is sporadic because I’ve only pulled in stuff that’s interested me or other people at the hack day - so I’ve got most of England, most of Texas (for SXSW, of course), San Francisco Bay Area and Antarctica (there’s lots of, well, bits of cold stuff in Antarctica that need photographing for Wikipedia, and it’s marginally colder than the train I take to London).

I did think about using Scala, but I went with the absolute simplest and least dogmatic way of picking programming languages: what had the libraries available and required me to do least work. After a bit of digging, I found the Ruby library. For Scala, I was going to have to extend a Java library. And the thought of dealing with Ant or Maven or something put me right off.

Post title is obviously a reference to the Future Sound of London track I’m listening to now. It was used fantastically on the soundtrack to the PlayStation game Wipeout 2097 (in the US, they called it Wipeout XL for some reason).

HackDiary: Scala now more comfortable than Ruby?

A few years back, Python was the first programming language I’d pull out of the toolbox. It was the second programming language I learned that is actually still used (BBC BASIC, AMSBASIC, PHP, then Python). Then I learned how to use Ruby, and while it has some shortcomings compared to Python, it has some real advantages as a hacking language. The Perl regex heritage was one, as was the functional programming constructs like map and filter, and the REPL.

But tonight, I think I may have reached the point where the balance has tipped towards Scala.

The problem was simple. I had a text file consisting of times in the format HH:MM. I wanted to get an idea of the distribution of those times across the day. I opened a Ruby REPL:

a = "17:05
16:53
16:29
16:13
14:19
14:14
14:09
14:08
14:02
13:39
13:38
13:36
13:28
13:26
13:26
13:25
13:11
13:08
12:57
12:44
12:40
12:37
12:32
12:21
12:11
12:02
11:45
11:28
11:25"

Err, now what. I guess I need to reverse it. Okay.

a.reverse!

No, no, I shouldn’t be using bang methods. Stateful is hateful or some other Haskelly slogan. Naughty me.

a = a.reverse

I know. I’m fussy.

Err, now what. I want to group ‘em.

a.methods.sort

What? Ruby doesn’t have a grouping method? It must have. I’m not writing a for loop. It’s 2011. Fuck that.

Tab over to Google Chrome.

Wait, no Internet connection.

Open up a Scala shell. Paste the same lot in with some Python-style triple-quotes around them. Yeah, Scala has that. That’s pretty much one enormous reason to use it as opposed to Java, right? REPL assigns them to res0

var times = res0.split("\n").reverse.toList

There we go. A List[java.lang.String]. Scala’s immutable lists rock.

And then doing the data mangling is pretty easy. Type times., press tab, look down the method list. The Scala REPL puts all the methods in an alphabetical, pretty-printed list—like ls does in the bash shell.

“groupBy”. That looks right. I haven’t used this before. Better check the Scaladoc. Oh, wait, offline.

cp /usr/local/Cellar/scala/2.8.0/src/scala-library-src.jar ~/tmp/scaladocs/
cd ~/tmp/scaladocs
unzip scala-library-src.jar
scaladoc scala/collection/immutable/List.scala
open index.html

(That’s for Scala on OS X installed using Homebrew. The location of scala-library-src.jar will vary on other systems.)

Click around. Look at List. Here we go “groupBy”. What’s it tell me?

def groupBy[K](f: (A) ⇒ K): Map[K, Repr]

That’s pretty cryptic, right? It makes sense eventually but it takes time to learn how to read it. It says that it takes a function (f) which takes something of any type and returns a key (K), then gives you a back a Map[K, Repr]. Basically you hand it your list, and a function which takes each member of that list and produces a key from it. Then it gives you back a map with the key and all the members that represent that key.

times.groupBy(x => x.split(":").first)

Now I got a map of like “12” with a list of all the stuff that begins with “12” and it’s in, you know, res1 or something

res1 toList

Now we’ve got a List[(java.lang.String, List[java.lang.String])] in res2. For Scala-newbs, that’s a List of a 2-tuple made up of a String and a list of Strings.

So, where were we?

res2 sortBy(_._1) foreach {x => println(x._1 + ": " + x._2.length + x._2.map(x => "#").mkString)}

And here’s the result:

11: 3 ###
12: 8 ########
13: 9 #########
14: 5 #####
16: 3 ###
17: 1 #

Not the prettiest graph, but, you know, it’s a good start. It wouldn’t be very hard to turn this kind of data into a Google Charts API graph and open it in your web browser. Yeah, this is the JVM: there are ways. Horrible, evil ways.

The sort of code I’ve written isn’t pretty. It won’t win any awards from those clean code people. But that’s not the point. Nobody’s REPL code is pretty: that’s why it’s REPL code, not bloody production code.

The point is quick hacks and data processing. How does Scala do for that? Let’s see: type safety, Java libraries, a ridiculously awesome collections library, a permissive and enjoyable syntax (especially for higher-order functions—filter { _ > 5 }, anyone?). The only thing I really miss is some of the Perl inheritance in Ruby ($_, regex syntax). There’ll be tasks that Ruby will probably be better for, and I’ll probably still write a lot of scripts in Ruby (a large chunk of my life consists of crappy little Ruby scripts talking to each other, and typically filling up my inbox with a bunch of shit when they don’t). Scala may have tipped over into being the first thing I type into my terminal when starting to hack.

Or you could take another reading: all you need to do to get me to use your programming language is build a better collections library and a better REPL. I’m just fickle, a higher-order function whore (as opposed to a functioning whore of a higher order). If that’s so, great. Get on with building better REPLs and better collections libraries for your language and you’ll get my total, unerring loyalty until someone builds something better.

jptp 1.0 now available!

I’ve finally gotten around to putting out the first release of jPushThePage (jptp).¹

It’s a command-line client for Push The Page, a service produced by Simon Maddox and Sam Machin to push web pages between devices. It was based on Chrome-to-Phone, a feature in Google Chrome which lets you push the current open page to your Android phone - but Push The Page implements the same functionality for the iOS platform, so you can push pages from any browser that supports bookmarklets to any iOS device.

With jptp, you can now go back in the other direction: push pages to your computer. You simply run the jptp client on any machine and it’ll open up pages that get pushed to it.

• jptp.jar
• Source code

You can run jptp from anywhere:

java -jar jptp.jar

You need to create a file in your home directory with your Google Talk credentials. On Windows, that needs to called _jpushthepage.properties - on Linux/OS X/etc. ~/.jpushthepage.properties

Create a simple file like this:

username = exampleusername@gmail.com
password = examplepassword

Obviously, change exampleusername and examplepassword to your Gmail username and password.

Then run the jptp client.

It is designed to work in the background, so you should be able to run it as a daemon or whatnot. It’ll remain logged into Google Talk in the background, but if you aren’t signed in with another chat client, it won’t appear as online in your friend’s contacts.

I hope people find this useful. Ideally, this will prompt people into building better, native desktop clients, or building plugins for existing chat applications like Pidgin, Empathy and Adium rather than having a separate app running.

Hack diary: streams of XMPP, how nicely you flow

Recently, I’ve been doing some XMPP hacking. It’s really jolly good fun.

What prompted this was the release of Push the Page, which I knew was in the offing a while back, but I finally found on the App Store today. It is Sam Machin and Simon Maddox’s attempt to create a version of Chrome to Phone but for iOS rather than Android and for any browser rather than just Chrome.

I have actually got an experimental command line desktop client for Push the Page written in Scala that lets you set your desktop computer up as a Push the Page device. So you can push from your iOS device (or indeed, any computer that has a JavaScript-capable browser) to your desktop machine. I wrote it in Scala because it is (a) sexy, (b) cross-platform and (c) JVM-based which means it is supposedly good for long-running processes.

Writing a Push the Page client is fairly trivial to do: basically, all you need to do is connect to Google Talk and listen for messages from the relevant XMPP account (Push the Page is hosted on App Engine, which can be used for XMPP bots pretty well).

I’m having a little bit of a problem with my Scala Push the Page client: namely, I haven’t managed to work out how to coax simple-build-tool to produce a standalone distributable JAR with all the dependencies bundled in and minified. I want to be able to produce a binary, but I can’t. And when I wrote it, the Push the Page service wasn’t public and the app wasn’t available in the Apple App Store. Now both of those have changed, I’ll publish the source openly soon and let many eyes fix this annoyance.

Of course, it is written in Scala, and Scala is Java, and Apple’s JVM is being discontinued and Java as a language on the Mac is “deprecated” by Apple along with Flash. So it might be useful if it was rewritten in another language for all the future Macsters out there who don’t have a JVM.

Ideally, it could be written as an IM plugin for the various IM applications: Adium, Empathy, Pidgin etc.

Or it could be a standalone application. I am interested in building things for Linux partly because it is, of course, the OS all geeks will end up using - some of us just take long Windows or OS X detours first. GTK always struck me as a useful thing to know: it has a good set of language bindings and if you can produce GTK apps, you can get them to run on Linux and Windows, and even OS X with a bit of coaxing.

I have a Mac app that I use, and it would be pretty useful to have that running on Linux as well or instead of on the Mac. It is one of the few things on the Mac which I have difficulty living without, and it is a closed-source shareware application that isn’t really maintained. It’d be good if I could have an open source, free, cross-platform application so not only I can benefit, but other people who use this app can also have the freedom to switch (to Linux, to Windows, or even to have it running on a server and accessing it in a browser).

But that is a big effort. I thought Push the Page would be a nice little GTK app I could build with a lot less work. I never really thought of how one might create a UI for it until I saw it on the iPad, where it keeps a history of URIs you’ve sent to it. It’d be neat to have it as a background service but also have the ability to pop it open with a history. And, you know, if I store this in a dotfile in an open format, if someone wants to make an OS X version or a Windows version or whatever, that’d be easy.

First, though, I’ve gotta learn GTK. And I’ve gotta choose a language to do that. Boom. Python. That was easy. I’m not as familiar with Python as I am with Ruby, so why choose Python? Well, Python seems pretty popular with the Linux community specifically for writing GTK apps, and there are plenty of existing GTK apps written in Python, which means things like the interface drag’n’drop builders (in GTK land, that’s Glade) should be pretty easy. The differences between Python and Ruby are pretty minimal and I’ve done more Python programming than anything else. And I know the Python libraries reasonably well and how to get packages.

PyGTK is refreshingly good. The documentation is straight-forward, and there are lots of code samples out there. It all mostly just worked. The few times it didn’t, it was mostly because I fat-fingered the keys or did something awkward like mixed tabs and spaces in my Python source file (:set expandtab wasn’t on) or forgot Pythonisms like the self.foo thing.¹

To see how to use PyGTK, I downloaded the source from Gajim, an Jabber client. I figured that since I’d basically be building a simple, app-specific Jabber client, I may as well read how one works, plus I can steal bits that work (yes, yes, and properly open source my modifications bla bla, I know). I’ve read the source code of three different Linux GUI applications, in three different languages: the Xfce Terminal application (in C), Banshee (an iTunes-style media player written in C#) and now Gajim (in Python). I’m not sure that the Gajim code felt much less verbose than the C# or C code I’d read for the other apps. I thought it was going to be a little rinky-dinky student project, but it is pretty fully-featured, while still being noticeably less bloated than some of the other IM clients I’ve used (Jabber-only too: thankfully, I’ve almost reached the point where my contact list is almost exclusively Jabber only).

Trying out Gajim, it reminded me: I remember using Twitter back in the day when it had IM support. That was actually damn useful. And if I am thinking of moving over to Linux any time in the near future, I’d lose my native Twitter client. I haven’t really found any on Linux that I liked. Why bother though? We have IM clients. Why not just reuse those? The nice thing with XMPP is that it can follow you around.

Then it struck me that I could use the User Streams API to get my own personal Twitter XMPP bot. I quickly set up a new handle on a public XMPP server, and got down to it. In Ruby this time. It was pretty simple to wire the User Streams API in using the twitter/json_stream library. Basically, it is as simple as making an EventMachine loop that hooks up to the Jabber server, then hooks up to Twitter, formats the messages and posts them. There’s more than this though: if I’m getting a 24x7 live stream of everything that comes into my account, there’s so much more I can do with it.

I’m going to store all the tweets for one. I’m still waiting for Twitter to let us access our old tweets like they’ve been promising to do for fucking ages! In the meantime, I may as well archive all my tweets and my friends tweets and retweets and whatnot. Tacking on a database onto my user stream is easy enough, and these days there are plenty to choose from. I’ve already got MongoDB installed on my Linux box, so I will probably pipe the tweets into there, and post my stuff into the Talis N2 triplestore I have access to.

I can also use it for filtering. Foursquare tweets? Ignored. Tweets from otherwise nice people who like to add seventeen social-media-related hashtags? BALEETED. Tweets from crappy conferences you don’t care about. Begone.

When I log off on one IM client, my server will eventually be able to save my place, and then drip-feed me the backlog (or just those with ‘device delivery’ turned on? or just those on a private ‘people I give a shit about’ list?) when I log on again. It can monitor when I read tweets.

Being able to hack with XMPP means you can hack the data-streams that fill our life both with joy and infuriation. They need hacking so that we can make them filled with more joy and less infuriation.

vlc-control

The other day, I asked a question on SuperUser called How do I add files to a VLC playlist from the command line on OS X?.

I wanted to use the RC interface, and the HTTP interface. But for what I need to do, the AppleScript interface is more than enough. I’ve made a little script using RubyOSA called vlc-control which you may find useful if you have, say, VLC on a Mac and like lounging around using your netbook to hack with.

As part of my blog-annotations project, I needed to get a list of all the tags I’ve used on my Tumblr blog. The Tumblr API was pretty easy: just XML over HTTP. I wrote a little script to do this. Here it is.

As an aside, you can use JSON with the Tumblr API, but XML and XPath was just as easy for this. But then I am an XDork…

As part of my Blog Annotations project, I’ve put together a pretty nice build script. I’ve just changed it so that it is version-control aware. It won’t bother rebuilding files if they haven’t changed in the version control system (but if the rendering script has changed, it re-renders everything). Currently, it is Mercurial only, but it will be very easy to add Git support. It’s all written in bash script. Have a look.

Hack diary: Readability golf

Day of the week is a simple little problem but allows for some really good code-golfing (at worst) and sensible code improvements. I’m not wild about the Scala solution though. It uses for-loop comprehension which hurts my head. I’m okay with list comprehension in Python in some circumstances, but I find that stringing together higher order functions is much more readable than list/loop comprehensions.

Here’s a cleaned up version of the for-loop version that uses an anonymous function instead of being in a main method:

(start: Int, end: Int) => 
  for (year <- start to end;
       date <- Some(new GregorianCalendar(year, DECEMBER, 25));
       if date.get(DAY_OF_WEEK) == SUNDAY)
  yield(year)

The logic here feels smushed together. Creating the array feels like a separate step from filtering them on the basis of the conditional. This is very efficient to write if you only have to do it once and there is no plan to ever refactor it.

Here is how it looks if you swap the for-loop comprehension for higher-order functions:

(start: Int, end: Int) => (start to end) filter {
  (x: Int) => new GregorianCalendar(x, DECEMBER, 25).get(DAY_OF_WEEK) == SUNDAY
}

For both, you have to import the Calendar and GregorianCalendar libraries from the Java class library:

import java.util.{Calendar, GregorianCalendar}
import Calendar._

The reason I prefer the higher-order function version is simple. It clearly separates the construction of the array (technically, the immutable inclusive range – scala.collection.immutable.Range.Inclusive with scala.collection.immutable.Range.ByOne) from the filtering of that range. One could abstract this out into two separate functions: one which constructs the range and the other which checks whether in a particular year December 25 falls on a Sunday.

Once you’ve finished refactoring it into separate functions and wrapped it all in a singleton object, you get something like this:

object YuletideRule {
  import java.util.{Calendar, GregorianCalendar}
  import Calendar._

  def christmasFallsOnSunday(year: Int) = {
    new GregorianCalendar(year, DECEMBER, 25).get(DAY_OF_WEEK) == SUNDAY
  }

  def checkYearRange(start: Int, end: Int) = {
    (start to end) filter christmasFallsOnSunday
  }

  def printYearRangeReport(start: Int, end: Int) = {
    checkYearRange(start, end) foreach println
  }
}

I think most people can understand this, even if they aren’t Scala or Java programmers. You could make it more readable for non-Scala programmers by explicitly adding the return type annotation on the christmasFallsOnSunday method so they know it returns a boolean.

Similarly, those not used to functional programming might find the infix-with-no-punctuation syntax a bit strange. You can change checkYearRange(start, end) foreach println to something more explicit like checkYearRange(start, end).foreach(println), but I find that the more Scala I do, the more this Haskell-style syntax is appealing. That said, I find I can only tolerate it in small amounts, and it certainly looks strange when I see lots and lots of it piled up.

Hack diary: Vim, Command-T and the Linux escape hatch

So, Apple announced today that Mac OS is going to get an ‘app store’. It already has one, of course: homebrew. This is one of those signs that Apple are going to start fiddling with the Mac platform, and so I thought I’d check out how well I can get on with my Linux setup (netbook: running Xubuntu 10.04).

I managed to get the Fn+F3 command working so that I can type without the trackpad jiggling around, following the instructions found here.

The one thing I’ve been missing from Vim is TextMate’s Cmd+T functionality. I looked at PeepOpen which isn’t open source and is Mac only, but very very cool (except, gah, it uses menu bar icons, my personal bugbear with OS X). That doesn’t help me if I want to run away from OS X if Steve Jobs decides to become the tyrant he seems always on the borderline of doing.

Instead, there’s Command-T which isn’t quite as pretty, but is open source and is cross-platform.

There’s some interesting discussion about Command-T/PeepOpen/FuzzyFinder Vim plugins and Mac-to-Linux cross-platform issues in this Hacker News thread.

I also came across an interesting post called Configuring Vim right. I haven’t done all of the things in it, nor do I endorse all the advice. But there’s some good suggestions in there that’ll probably make its way into my vimrc file.

This is all getting a bit complicated now: I’ve got a Mac laptop, a Linux desktop machine, an account on a shared Hackintosh, a Linux netbook and so on. So, I’ve started working on a little project called ‘buildup’. It is a set of post-OS-install setup scripts that are there to be run on either OS X or Ubuntu/Debian Linux. Using apt-get and/or homebrew, it installs all the packages I need, and has all my dotfiles which can be installed by running a script called something imaginative like “dotfiles”. It has been a fun little project because it has allowed me to offload a lot of tacit knowledge about the crapness of all build/packaging systems into a trusted, version controlled script.

It has also been fun because it has let me experience two tools I haven’t really used much: Python and Mercurial. I’m usually a Git user and I generally prefer using Ruby over Python (despite a number of significant shortcomings I think Ruby has). The reason I’ve done the project in Python is because Python has become a sort of lowest common denominator scripting language on all computers. Both OS X and most sane Linux distros come with some version of Python pre-installed. It may be a crappy out-of-date one with a completely arse-over-backwards relationship between the system package manager and the language package manager, but it is there nonetheless. Ruby is there on OS X, but Python is more universal in Linuxland (partly because a lot of the default stuff that comes in GNOME and Ubuntu is written in Python, for instnace).

It is slightly annoying because the sort of thing I’m doing would really match what Rake does. But if Ruby isn’t installed by default on the machine, you can bet RubyGems and Rake aren’t. So, Python. I might look into Waf but I’m actually quite happy with using just plain Python.

The other thing I’m doing with Buildup is using Mercurial. As I said, I’m a Git user mostly, but I’m trying Mercurial on this project because it is a private project and since Atlassian bought BitBucket, they are now offering unlimited free private Mercurial repositories while GitHub charge for hosting private repos. Mercurial is okay. It is doing what I need, but my mind has been so warped into the ways of Git that I’m finding it a bit hard to think in the way Mercurial is making me. This is not a problem because Buildup is not very complicated, nor am I collaborating with anyone. The things which annoy me about Mercurial are really the lack of an equivalent thing to git commit --amend and the lack of an index. I really like the index. I’m told you can get something like an index back by installing some plugin or the other, but I can’t be bothered. I’m using Git for public projects and for work, and I’m just using Mercurial and BitBucket for a few small private projects. I’m not dissing Mercurial at all though: it is an excellent piece of software, but I’m just a bit too familiar with Git and I haven’t learned Mercurial in any depth.

One of the other things I may be using Mercurial for is for dissertation work if and when that starts happening. Private repositories plus LaTeX, BibTeX and possibly LyX make a happy philosopher. Much happier than some nonsense with Word or OpenOffice or whatnot. I really need to learn LaTeX properly: LyX is okay, but it’d be nice to churn it out in Vim (or Emacs, I hear it is good for writing LaTeX) rather than be reliant on an often rather temperamental GUI app.

Anyway, I’m learning lots in Linux using it on the netbook. As I said, I’m using Xubuntu, and Xfce is really the least unpleasant desktop environment I’ve found. I tried Awesome a while back, and Xfce is about halfway between Awesome and GNOME. It reminds me of Windows 95: it is pretty lightweight and uncluttered. I’ve already learned about a whole stash of keyboard commands. Alt+F10 (maximize) and Alt+F11 (full screen) are worth knowing. I’m using GNOME Do, but I seem to have two ‘competing’ terminal emulators (the Xfce Terminal and GNOME Terminal) installed both called ‘Terminal’ in GNOME Do, one of which isn’t actually set up properly. That has been a bit confusing. It’d be nice if I could set it to just always load Xfce Terminal. (I got help from some Xfce developers on IRC a while back on how to make Xfce Terminal have really short tabs, which is kind of useful on the netbook.)

As for the other stuff on Linux: I can’t stand Empathy. I’ll probably replace it with something that runs in the shell. The annoying thing is that the open source community can be proud that the best IM app ever made is in fact open source. It just happens to run on OS X and be called Adium. Heh. And Transmission too: that was on OS X before other platforms if I recall correctly.

Will I end up moving to Linux? Maybe. Maybe not. But having it there as an option is important to me. It is an escape hatch if Mac OS X veers too far off into App Store, DRM land. If being an Apple customer becomes more of a burden than it is being a joy, it is very, very useful to have an escape hatch. For me, it is like assisted dying. I once read that a lot of people who seek the services of euthanasia clinics like Dignitas never use them, but it serves as a very useful psychological tool to know that if they want to commit suicide, they have the right to do so. A slightly less gloomy example perhaps: in the argument over social contracts, I think Hume said that only when you can emigrate can you claim the possibility of a tacit social contract. If you can’t leave, you can’t really be said to be consenting (tacitly or otherwise) to being there.

I’m planning to work over the next six months or so to see where I am tied into OS X and then either find or build escape hatches. Not that I necessarily plan to stop using OS X, but just in case. The first of these is an application I use called CDFinder which is a “digital asset manager”. I’ve looked and tried to find a good open source alternative to CDFinder but I just can’t. I’ve looked into building my own, but I couldn’t really find a good way of storing the data. These days, I think the answer may be here in all these NoSQL stores. I can see something like MongoDB or CouchDB being a good example of a reasonably lightweight way of storing an index of CD and DVD volumes. The key to something like this is making it nice and flexible: an open source backend on a platform like MongoDB and then provide a simple web front-end with something like Sinatra, and on Linux maybe a GTK app either written in C/C++, Python or C#. It may be possible to separate out the client side and the server side: the client side simply needs to run the indexing process on the disk and produce a data file representing the disk’s contents. Then it pushes it up to the server which could be running on another machine. You could then query it over the web, or from a terminal or anywhere.

I’m obviously tied into iTunes, which is a bit of an annoyance. Everyone bitches and moans about iTunes, but I think it is actually one of the better things Apple does. The people who complain about user experience don’t seem to actually provide many reasons why iTunes is such a terrible piece of software, they just sort of expect me to know by osmosis that it sucks. The lack of iTunes-style resume support is one of the reasons I’m still stuck into iTunes. It looks like a new bug has been raised for Banshee requesting this. One day it’ll happen: probably the time when I learn C# properly and code it myself damnit.

There’s other stuff I’ve been hacking on: some of it is behind the company firewall, some of it is turning up in the Open Plaques SVN. All good fun.