Do your own plumbing
I finally got around last night to replacing a load of Rube Goldberg code with some clean code.
And I just want to say: screw The Cloud. Write your own code. And if you can’t write code, the cloud won’t save you - you’ll still need to learn to code because visual programming tools just suck.
A while back I wanted to take an RSS feed and put it into Twitter. That’s easy, right? There are a bunch of services that do that with names like “twitterfeed” and so on. You take the RSS feed and give it to that service and let it do its thing. One less piece of code that you don’t have to maintain, one less box to keep running.
Trust me. It doesn’t work. Because the Twitterfeed-type services don’t let you do much in the way of customisation. They just take the title and the description, shorten the link and post it. So, what if the feed doesn’t quite fit that? Well, then, in the Cloud Computing magical scenario, you feed it into another service that does some kind of transformation on it. Like, oh, Yahoo! Pipes. That’d do the job.
Yahoo! Pipes is a wonderful service for many use cases, but if you are a programmer, you’ll hit the limits really quickly. You’ll hit the inherent clumsiness of the visual programming environment first - because you are working with very high-level functions that have been designed in a way that doesn’t exactly gel with how you think. Yes, you get a filtering function, and you get some other stuff. But you don’t really get a proper map function, meaning you can’t really transform the feed into something else. You don’t get low-level conditionals. You can emulate some of the things you want to do in Yahoo! Pipes by creating all sorts of bizarre and byzantine pipe structures. You end up with a screenful of blobs with wiring that would make a network engineer blush.
So you hack together the thing you are working on, and push it over to the Twitterfeed service. Only they handle RSS feeds strangely - they do fancy stuff with GUIDs. (Side-note: I don’t understand why we need GUIDs on the web. We have a globally unique identifier - it is called a damn URI. Use that, damnit.)
So it is late, and you hack it together by having a Ruby script that pulls in the feed from Yahoo! Pipes, runs a damn regex over it to pull the element you don’t want out and pushes it back out. Well done, you have avoided writing code by creating a Rube Goldberg machine.
The data goes like this:
Original site to Yahoo! Pipes to your machine to your server to Twitterfeed to Twitter.
This is barmy. And why did I do this? Because I wanted to avoid writing some code.
Like all Rube Goldberg machines, entropy took over and the thing broke a few weeks later and stopped tweeting.
I finally got around to rewriting it. It was really painful to do. It ended up being 61 lines of Ruby. That wasn’t very hard, was it? Now the data flow is this:
Original site to my machine to Twitter.
Really, what was the point of this silliness? To avoid having to write some code? That’s not a very good reason, is it?
I mean, seriously, we have these tools to avoid a bit of programming for non-programmers. But the end result is one where you are relying on so many servers to keep doing what you expect them to do without problem. Well done Web community, we now have ways for non-programmers to make shockingly brittle pseudo-code. You also don’t get abstraction easily: you can’t really write functions or classes to abstract common patterns. You just get to reimplement them over and over again.
What else don’t you get with the “stick the code on all these interlocking services” approach? Logging. Debugging. Unit testing. Open source libraries. In the above linked example, I can see myself wanting to add logging (so I can just chuck in Log4R). I also will probably add a thing to verify that the XML has actually downloaded properly and retry if it didn’t work, and stick in a better time handling routine - so that it immediately logs the last successful download and then uses that, so we don’t get any missed updates. I do a lot of this kind of feed mangling, and it’d be pretty simple to abstract out some of the common patterns. This is basically what Pipes has done. But the problem is that even if you abstract out the common patterns, you still need everything else in the language!
I haven’t got a problem with services like Yahoo! Pipes, but don’t use it for anything beyond prototyping and very simple hacks. Once you’ve got more than about four action blobs on screen, close the tab and write some code. If you don’t know how to code, learn. It is now easier to learn to code than at any time in the past. And there are more than enough places to host your code: on your own computer, on a VPS or on services like App Engine or Heroku. Even a cheap PHP hosting account will do the job. And, damnit, publish your source code. Go through and pull the passwords and API keys out and replace them with “SECRET” and put them up. That way others can take a look and make suggestions on how to improve the code.
Visual programming tools have been tried so often. If you want to remix data, code is the most viable way of doing it. Learn to code. If you want to mangle data, you’ll be less stressed if you have full control over what is going on, rather than blind faith in a bunch of web services to do the right thing.
Station Approach, Tunbridge Wells, TN1 1
