mikewilliamson

The fight over HTML5

There has been plenty of virtual ink spilled lately on the whole HTML5 video element.

In case you need a recap; talks between the W3C and browser vendors about a single universally supported codec embedded in the HTML5 standard have broken down and the issue has spilled out into the tech media.

The situation can be summed up pretty simply:

Organisation:	Makes browser:	Owns IP in AVC/H.264:	Owns video service:	Will support:
Google	Chrome	No	Youtube	H.264, Theora
Microsoft	Internet Explorer	Yes	MS Showcase and MSN video andMSNBC	H.264
Apple	Safari	Yes	Itunes, AppleTV, Apple Trailers	H.264
Mozilla	Firefox	No	None	Theora
Opera	Opera	No	None	Theora

A big part of the purpose of supporting open codecs like Theora is to ensure that the cost of creating video sites and services that compete with Youtube, MSN video and iTunes stays as low as possible. The result:

3 of 3 browser makers that own a video service come out in support of H.264.

Both of the companies that contribute IP to the AVC/H.264 patent portfolio voted that H.264 become the codec built into the standard.

Subsequently the Web Hypertext Application Technology Working Group (WHATWG) dropped Theora from the standard since the major players can’t/won’t agree on.

With a fad driven business like video serving, and another 75% of the worlds population yet to come online, Google, Apple and Microsoft did what they needed to do to make sure they remain the top dogs in video; they raised the price for potential competitors.

Mozilla is pretty explicit about ensuring the web still has some disruptive potential left in it by the time the rest of the world starts using it but for some reason tech bloggers seem unable to see forest behind the trees (with one or two excellent exceptions):

Brian Crescimanno writes to complain that the open source “idealists” at Mozilla are ruining the opportunity to have an “encode once, deploy anywhere format”. He rekons they should support it because its convienient to have a single format.

John Gruber seconds Brian’s supporting argument that “Theora sucks”, and goes on to chalk up these companies support of H.264 to strictly technical reasons like the needs of the hardware in the mobile computing market.

I think there are better questions to be asked. Questions about what we expect from standards and how to tell the difference between a standard and a marketing ploy. Questions about whether we truly believe that businesses will always (or ever) act in the public interest, especially when the public interest is served by making it easier for people to compete against them. Those answers will not be found discussing bitrates.

I, for one am very grateful for is that Mozilla is a big enough player to even make this much of an impact. Without them in the room, the future of the internet would be shaped by a room full of patent holders bent on monetizing your every click and keystroke. Having nowhere near the kind of business entanglements and conflicts of interest the others do, they are free to advocate for the users. They are able to do it without worrying about producing a profit this quarter or answering to an angry board about why they didn’t wring every penny out of their client base.

Its a position afforded them by funding from Google who desperately need someone to stave off Microsoft from dominating the web. It may not last forever but, for now they are seizing the moment and anything other than H.264 becoming part of the standard should be viewed as a huge win. Not just for them but for us.

Rails integer handling

Well it certainly is the case that you learn something new every day. After digging into some confusion around how Rails chooses its data types I ended up turning to the API docs. From the API docs:

:limit – Requests a maximum column length. This is number of characters for :string and :text columns and number of bytes for :binary and :integer columns.

:limit => 11 would give us 11 characters in a string column, but I was trying to apply it to an integer. As they said in the docs, when you use :limit with an integer it means the number of bytes. So, for the sake of my own mental clarity here is what you should get when you use the :limit option:

:limit => 1	`TINYINT`	1 byte	-128 to 127
:limit => 2	`SMALLINT`	2 bytes	-32768 to +32767
:limit => 3	`MEDIUMINT`	3 bytes	-8388608 to 8388607
:limit => 4	`INT`	4 bytes	-2147483648 to +2147483647
:limit => 8	`BIGINT`	8 bytes	-9223372036854775808 to 9223372036854775807

I say “should” above because Rails will take the value you give it and map that as best it can to whatever the actual database you are using can provide. So for MySQL it will give you a bigint for any :limit value between 5 and 8. One strange thing is that :limit => 11 maps to a 4 byte int(11) in MySQL. Not sure what thats about. For values that fall outside those, well, I guess I’ll figure that out another day.

Bundle mania

I ended up using Heroku’s bundles feature a fair bit today. For those who have not played with this yet, a bundle is essentially a zip/tar file containing all the code from your application along with a dump of your database. From the help file:

bundles                      # list bundles for the app
bundles:capture [<bundle>]   # capture a bundle of the app’s code and data
bundles:download             # download most recent app bundle as a tarball
bundles:download <bundle>    # download the named bundle
bundles:animate <bundle>     # animate a bundle into a new app
bundles:destroy <bundle>     # destroy the named bundle

In general its pretty intuitive. The commands do what you expect; “bundles:capture” captures a bundle, while “bundles:download mybundle” downloads the bundle named mybundle. With my expectations set by using the other commands I tried out bundles:animate

mike@sleepycat:~/projects/myapp$ heroku bundles
2010-03-16 complete 03/17/2010 14:25
mike@sleepycat:~/projects/myapp$ heroku bundles:animate 2010-03-16
Animated myapp 2010-03-16 into http://myapp-2010-03-16.heroku.com/ | git@heroku.com:myapp-2010-03-16.git

Given the description “animate a bundle into a new app” I was expecting a fully functional copy of my application waiting for me at that new address, based on the code and the database dump contained in the bundle. Instead I get:

mike@sleepycat:~/projects/myapp$ heroku console –app myapp-2010-03-16
! myapp-2010-03-16 has an empty code repository. Push and try again.

When I log in and look at my account, sure enough the repository is empty and even more surprising:

Data size

0 Bytes in 0 tables

So if it does not load any of the code from my bundle, and doesn’t use the database dump to create a database, how does this differ from creating a new app?

Lost in translation: the down side of taps

UPDATE: Both Taps and Sequel have been updated and the problems that caused my issues have been resolved. I am now happily db:pushing and db:pulling. It took a while, but it got fixed in the end. :)

I have been using Heroku for a while now and its definitely great. One of the things that really blew me away when I started using it was being able to send my my entire database data up to the server with a single command:

heroku db:push

Could it be any easier? Thanks to the magic of taps, Schema AND data, transferred between my local MySQL database and Heroku’s Postgres database. At the time I might of stopped for a second to marvel at how big a deal that is, but only for a second. It turns out that its really worth stopping and thinking about what goes on when you run that command; especially when you are developing on MySQL. MySQL and Postgres have different datatypes to store data in and making an finding an exact equivalent from one database to the next is much easier said than done.

I think I will let the data do the talking for me on this. Keep you eye on the terminal_id column:

mysql> desc terminals;
+————-+—————+——+—–+————-+—————-+
| Field       | Type          | Null | Key | Default     | Extra          |
+————-+—————+——+—–+————-+—————-+
| id          | int(11)       | NO   | PRI | NULL        | auto_increment |
| location_id | int(11)       | YES |     | NULL        |                |
| merchant_id | int(11)       | YES |     | NULL        |                |
| terminal_id | decimal(11,0) | YES |     | NULL        |                |
| reference   | varchar(255) | YES |     | NULL        |                |
…
8 rows in set (0.00 sec)

The data:

mysql> select terminal_id from terminals;
+————-+
| terminal_id |
+————-+
| 10792146001 |
| 10152407001 |
| 10392407002 |
| 10152617003 |
| 10184619001 |
…
16 rows in set (0.00 sec)

Pushing it up to the server this suddenly becomes:

CREATE TABLE terminals (
id integer NOT NULL,
location_id integer,
merchant_id integer,
terminal_id integer,
reference character varying(255),
…

After a Heroku db:pull command:

mysql> desc terminals;
+————-+————–+——+—–+————-+—————-+
| Field       | Type         | Null | Key | Default     | Extra          |
+————-+————–+——+—–+————-+—————-+
| id          | int(11)      | NO   | PRI | NULL        | auto_increment |
| location_id | int(11)      | YES |     | NULL        |                |
| merchant_id | int(11)      | YES |     | NULL        |                |
| terminal_id | int(11)      | YES |     | NULL        |                |
| reference   | varchar(255) | YES |     | NULL        |                |
…
8 rows in set (0.00 sec)

The data:

mysql> select terminal_id from terminals;
+————-+
| terminal_id |
+————-+
| 2147483647 |
| 2147483647 |
| 2147483647 |
| 2147483647 |
| 2147483647 |
| 2147483647 |
| 2147483647 |
| 2147483647 |
…

How about that. Now I have a column of maxint values and a lot of very confusing bugs. Nothing like a little integer overflow to make the day go quickly. Worse still is pushing back up to the staging server before noticing, duplicating the corrupted data up there. Good thing it didn’t effect production. I am guessing that had I been developing using Postgres there would be no problem, because taps would not have to convert from one datatype to another. I am going to have to think about doing that even though I am much more familiar with MySQL. Either way the ease of that heroku db:push/pull command definitely belies the magnitude of what that command is doing.

So the takeaway from this? For me:

Backups = good.

Staging server = good.

Taps = use with caution.

The downward spiral

There are few things that get computer users as riled up as the user interface. Everyone is effected by it and everyone has an opinion about it. What works for one person is likely to induce tourette’s in another. Battle lines have been drawn for years between the Mac world and the PC world and their differing implementations of effectively the same “desktop metaphor”.

Though most of my computer use has been on Windows, I have a few years of solid use of both Mac and Linux. In the last few years I have ended up switching between them at pretty regular intervals: Windows at work, Linux at home and Mac at my Girlfriends. Several months ago, I set up my laptop as a triple boot, (Windows 7, Ubuntu 9.10, and OS X) further accelerating the rate at which I switch between them.

All this jumping from one User Interface to another has slowly created a pretty generic sense of the desktop metaphor in my head along with a notion of trajectory. While there are things I like and things I don’t about each, my overall impression of the trajectory is this: UI is getting worse.

Learning object oriented programming in school we were warned against creating a “God object” that does every possible task in the program. Though that particular anti-pattern is studiously avoided by programmers, it seems increasingly popular for UI designers. So what do the Windows “Start” menu, the KDE “kickoff application launcher”, Windows 7 Start menu and the soon to be released Gnome 3 Activities menu have to do with the God object?

I think they are all part of a UI anti-pattern which I’ll call the “God box”.

To me, its an insane idea that everything that happens on a computer should start by clicking on a single widgit (be it button, menu, launcher or whatever). Many have pointed out how twisted the logic is to have to click a button labelled “Start” to be able to turn off your computer, but this is exactly the kind of stuff you get with a God box. While programs and system settings are proliferating, UI designers are trying to cover up the complexity rather than reduce it.

As Jensen Harris (the guy who designed the Microsoft office ribbon) said of the Word 2002 task pane: “We did what every user interface designer does when they run out of ideas; we invented a new rectangle”. The “Start” menu or the “kickoff application launcher” aren’t new panes but they are essentially a “new rectangle” placed above the mess of submenus in hopes managing the presentation of what lies below.

To me the KDE kickoff application launcher is the most over the top example of this anti-pattern, but it seems that UI nerds everywhere are following it. In Window 7 Microsoft finally ditched the classic start menu and forces everyone to use their version of the God box type Windows menu.

While God boxes allow others stay above the issue, offering ways of filtering the mess below, Gnome waded in and broke the entirety of the operating systems capabilities into enough separate menus that they no longer required filtering or deeply nested sub menus. Currently I have a menu for each of the most common things I am going to do: find a program to run (Applications), look for something (Places) or change something about my computer (System).

With rare exceptions I never need to go below a single sub menu. To me, that’s good UI design. The path to what I want should be as direct as possible. Why should I have to take a trip down the UI rabbithole, hunting for something in a sub sub sub menu when I just want to run one of the applications I have installed? Gnome also kept the best parts of both Windows (like the taskbar) and the Mac (like having a CD/DVD appear on the desktop). In general I think its highly underrated in terms of UI.

Unfortunately, with Gnome 3 around the corner, the UI team at Gnome seem to have succumbed to the same thinking that influenced Microsoft and KDE. It looks like they have created their own God box, known as the “activities menu” to replace all the current menus in Ubuntu 10.10.

Apple seems to avoid most of these issues by saying: “Fuck you, build your own menu” and leaves the users to root through theirs applications folders to set up their equivalent of the applications/programs menu (the dock) themselves. The “apple menu” is really the only other place to look for things that aren’t somewhere in a folder. I suppose simple is a reasonable substitute for friendly, but to me it doesn’t seem like good UI any more that if Microsoft added every program to a quicklaunch bar.

So as far as I can tell the trend seems to be towards more clicking and more sub sub sub menus. Add to that the recent decision by Canonical to purposelessly move the min, max and close buttons to the right (what conceivable benefit could there be that would make the aggravation worth it?), and news about things like the “task pooper”, the path that current UI design is going down is inducing more eye rolling than anticipation in me. I hope I’m wrong about the Gnome activity menu, but I’m not holding my breath.

Rails 3 on Ubuntu Lucid.

I just got finished updating my Rails setup script and testing it on the Alpha 3 version of Lucid. Now that the packages seem to be settling down the the repositories for Lucid suddenly I can find the ones I need for Rails. The script now gets Rails 3 running on either Karmic or Lucid. If you need older versions of Rails just uncomment the lines for the version you need.

You can get the latest version of the script here: http://github.com/sleepycat/wrong-side-of-the-tracks

Lucid Alpha 3 in Virtualbox.

Everything I am hearing about the upcoming Lucid Lynx release of Ubuntu sounds really great. I just grabbed Alpha 3 from the Ubuntu site and was a little dissappointed to see it hang as soon as I selected “Install Ubuntu”. The fix is to turn off the ACPI option in the system settings for your VM.

The side-effect of this is that the virtual machine will no longer close the window and disappear when you shut it down. C’est la vie.

From there the installer started up fine, but it hung again when I tried to update the installer. When I skipped the update it installed without any other problems. Thats what worked for me on Virtualbox 3.08 OSE.

UPDATE: This issue still seems to be effecting the new Lucid Beta 1…

Just a copy?

There is an argument that is trotted out pretty regularly by people critical of “open source”. It takes various forms but boils down to roughly this:

“Open source is just copying an existing program”.

Whether its “Open Source is Not Innovative” or “Open-source companies absolutely can’t have a new, innovative technology.” or “a herd simply cannot innovate”, at base, this argument is an argument glorifying individualism and warning against collectivism. The reality however is that many “open source” projects started as the work of a single person. Ruby on Rails and the underlying Ruby language are both examples of this. They are also maintained and extended by individuals, each of whom is more empowered to make any improvement they want than the average corporate employee. There is no collectivist bogeyman here. There is plenty of brilliant and original work being done in the “open source” world. In fact the majority of the internet is build on it. And yet because they are licensed in such a way that people can read the source code, somehow in the minds of these people these projects are incapable of “innovation”.

Those arguments still resonate with people however flawed they may be. While I strongly disagree with the assumptions that are being made, for the sake of argument lets assume its true. There is more going on than meets the eye even when it seems that Programme B is just an open source copy of Programme A.

The difference is probably illustrated nicely by a website update I did a while ago. Wanting to spruce up my site a little and try out a the CSS @font-face command. @font-face allows me, the developer, to point to an actual font file that the person viewing my website will then automatically download a copy of so it can be used by their browser when viewing my site. That’s pretty exciting since I am pretty bored of Arial. The problem is that the licence for most fonts forbid that sort of thing, since most font designers are looking to be paid for each copy that is used.

For me any snazzy looking font was fine for my site but I needed an additional feature beyond serifs. I needed a feature that is not inherent in the font itself, but one that comes from the licence it is released under. I needed freedom to distribute the font. I looked around and eventually found a copy of some standard font that had been released under a liberal licence. The reality was that it was not “just a copy”. Even though it was nearly indistinguishable from a commercial font, the licence alone had added the killer feature that I needed.

The same applies for any other software. Assuming two programmes, identical in all ways but one is closed source while the other is GPL. The licence alone has added to the feature set.

That’s right, freedom is a feature.

So at least in my mind, even if I pretend that its true that “open source” is just copying, I would still say that it’s not “just” copying. Its improving.

Just my two cents.

Mysql’s explain and indexes in ActiveRecord.

After doing some work on optimising a query in Rails app I am working on I got to thinking about how useful MySQL’s explain statement is. It happens to be really useful for figuring out what, if any, indexes are being used by a given select statement. So I decided to see what I could do about adding that functionality to Rails myself. Projects like this really give you a great tour through a lot of Rails internals, so I figured it would be good for me whether I could figure it out or not. After a little hacking I came up with a patch to ActiveRecord::Base.connection that runs an explain statement before every select. I wrapped it in an unless statement to make sure it does not run if the mode is production.

Explain (0.000000) | select_type: SIMPLE | key_len: 263 | table: taggings | id: 1 | possible_keys: index_taggings_on_tag_id,index_taggings_on_taggable_id_and_taggable_type | type: ref | Extra: Using where | rows: 1 | ref: const,const | key: index_taggings_on_taggable_id_and_taggable_type
Tag Load (0.000910) SELECT `tags`.* FROM `tags` INNER JOIN taggings ON tags.id = taggings.tag_id WHERE ((`taggings`.taggable_id = 5) AND (`taggings`.taggable_type = ‘Member’))

The “possible keys” portion is where it lists the indexes for that apply to that query. If there are none there and this is a query that runs a fair bit you probably want to think about adding an index. For me this is a great way to make indexes, or a lack thereof more visible to me during development because they are really easy to forget. So in the spirit of “If you want to truly understand something, try to change it”, here is my first bit of tinkering with ActiveRecord:

unless RAILS_ENV == ‘production’
module ActiveRecord
module ConnectionAdapters
class MysqlAdapter < AbstractAdapter

def select_with_explain(sql, name = nil)

explanation = execute_with_disable_logging(‘EXPLAIN ‘ + sql)

e = explanation.all_hashes.first
exp = e.collect{|k,v| ” | #{k}: #{v} “}.join

log(exp, ‘Explain’)

select_without_explain(sql, name)
end

def execute_with_disable_logging(sql, name = nil) #:nodoc:
#Run a query without logging
@connection.query(sql)
rescue ActiveRecord::StatementInvalid => exception
if exception.message.split(“:”).first =~ /Packets out of order/
raise ActiveRecord::StatementInvalid, “‘Packets out of order’ error was received from the database. Please update your mysql bindings (gem install mysql) and read http://dev.mysql.com/doc/mysql/en/password-hashing.html for more information. If you’re on Windows, use the Instant Rails installer to get the updated mysql bindings.”
else
raise
end
end

alias_method_chain :select, :explain

end
end
end
end

There it is, quick and dirty. If it ends up being something that I, or other people find really useful I will look at cleaning it up, adding some tests and maybe making a gem out of it. In the mean time you can just paste it into a file in your initializers directory. Happy indexing!

Caveats: I have got this working with Rails 2.1 with MySQL. Its only intended for helping out a little during development, and I have no idea what might happen if you run it elsewhere. If you have some suggested improvements I would love to hear them.

Getting started with Nokogiri on Ubuntu

I am working on a project at the moment that requires that I pick specified elements out of an HTML page. This was the first time that had come up for me and initially I thought I might be able to do this with REXML, but after I tried it in IRB and quickly realised that for parsing potentially dirty HTML this was not the tool for the job. It turns out that Nokogiri is designed for exactly this sort of thing. Getting started with it turned out to be easy as well.

First thing is installing the gem:

mike@sleepycat:~/Desktop$ sudo gem install nokogiri

And then the dependencies:

mike@sleepycat:~$ sudo aptitude install libxml2-dev libxslt-dev

And then the fun of forgetting that I need to require rubygems BEFORE trying to run this in IRB (that part is optional for everyone except me):

mike@sleepycat:~/Desktop$ irb
irb(main):001:0> require ‘nokogiri’
LoadError: no such file to load — nokogiri
from (irb):1:in `require’
from (irb):1
irb(main):002:0> require ‘rubygems’
=> true
irb(main):003:0> require ‘nokogiri’
=> true
irb(main):004:0> require ‘open-uri’
=> true
irb(main):005:0> test = Nokogiri::HTML(open(‘test.html’))

And out comes parsed xml goodness. A little further twiddling I had gotten my XPaths to work had it doing what I needed. I’m pretty impressed how much I could do with Nokogiri in just a few minutes fooling around for the first time. I have a feeling I am going to end up using this a lot.