tecznotes

Michal Migurski's notebook, listening post, and soapbox. Subscribe to this blog. Check out the rest of my site as well.

Nov 1, 2009 11:31pm

the future is staring us in the face

That's the line we use around the office whenever the subject of HTML and canvas comes up - we use Adobe Flash for most everything now, but we don't expect that situation to last forever. The work done by Mozilla on Gecko and Apple on WebKit is one possible future for online design and visualization, and it's turning slowly to face us right about now.

A few developments during recent years have brought us here.

One of the first widespread demonstrations of canvas viability as an interactive medium came from legendary Javascript developer John Resig, who ported the popular educational / artistic Processing environment to Javascript and released Processing.js to the world in May 2008 or so, just about 1 1/2 years ago. At the time, the library was lauded as an "amazing hack" (Andy Baio). Christopher Blizzard said:

The web is going to win, filling the market niche where Flash and other similar technologies found their hold. And John's little project can hopefully act as a great catalyst to take us there. Thanks, John!

Earlier this year, Mike Bostock and Jeffrey Heer at Stanford released Protovis, a visualization toolkit focused on the use of "marks" to build up graphical charts and visualizations from raw Javascript function chains. The library is currently implemented using SVG, but when they first launched everything rendered smartly to HTML canvas. The language Mike and Jeff developed to express these marks is succinct, and beautifully flips the logic of normal charting on its head: instead of decided on (say) a pie chart at the outset and filling in the constituent wedges, you build up a complete chart from patterns of basic geometry and data. You might end up with a pie chart in the end, but you might also end up with something much more subtle and expressive, like this reinterpretation of Burtin's antibiotics chart. I dare you to try this with Google Chart API.

Finally, as we work on increasing amounts of public, government, and civic work, we're mindful of the relative opacity of Adobe's various transmission formats, as noted by Sunlight Labs in Adobe Is Bad For Government. They're talking about PDF files in particular, but I liked this excerpt from an Adobe open government white paper: "Since the advent of the web, an entire infrastructure has evolved to enable public access to information. Such technologies include HTML, Adobe PDF, and Adobe Flash technology." Not wrong, necessarily, but then again Froot Loops is also part of a balanced breakfast so there you go.

Anyway, interesting times.

Usually when I experiment with a new technology like this one, it's nice to revisit an old problem or two, to see if some new light can be shed on it with a new tool. Coincidentally, while John Resig was releasing Processing.js in May 2008, I was busy working on some ideas for flea market mapping. I had recently found GDAL, a toolkit for manipulating digital raster maps, and rubbersheeted a collection of old Oakland road and real estate maps to match the one true cartographic projection. The annoying bit in the middle of this process is a three-way dance between a graphics program, a web browser, and a text editor for the collection of ground control points used to warp the image:

One of the simple things that canvas can do really well is transform and clip bitmaps, so it's possible to take an image file, turn it into a set of triangles, twist and bend them, and rapidly render the result. Cassidy Curtis clued me in to the linear math behind these operations many years ago, it really does just turn out to be a bunch of arithmetic.

This will be important later.

A thing about triangles is that there are some good algorithms for taking a bag of points and making sets of triangles out of them. Delaunay triangulation in particular is a simple, aesthetically pleasing mesh useful for dividing up a space defined by known points. Fred Scharmen has been writing about manual performance of graphics algorithms, and he has an excellent recent post on drawing the Delaunay triangulation and the Voronoi diagram.

Amsterdam designer Jonathan Puckey has been exploring a similar vein with his triangulated portraits, though sticking to the computer all the same.

The algorithm is quite simple, trivially easy to implement in Javascript. Look, here's mine:

Put these things together, and you get a way to perform image deformation and rubbersheeting in the browser without the use of Flash. I wrote a demo that shows a few pieces in action. First, it uses the Javascript port of Modest Maps, and introduces a map provider based on Zoomify for the manipulation of potentially very large tiled images. Second, it overlays each interactive map with a canvas element. One is used to display the triangulated mesh of control points, the other shows a preview of the image deformed to match the underlying geography. You can see here how five points nicely handle the area around downtown Oakland:

Both maps can be panned around, and zoomed with the mousewheel. Move the circular control points around on each side to see how they change the visual display. I've tested this with Firefox and Safari on a Mac, and the resulting interaction is just a little bit snappier in the latter. There's a distracting visual flickr in both, which I'm sure could be dealt with by double-buffering the canvas or some similar optimization.

There are a few things here I really like about the canvas interface. It seems to use a stack model for the drawing state (with save() and restore() instead of the more traditional, cryptic push() and pop()) to save things like clipping paths and coordinate transformations. This makes it possible to load the image pyramid tiles into the warped view asynchronously, throwing a transformation and clipping triangle into an array with a bunch of closures and callbacks to blit out pixels when images are loaded. It's not particularly encapsulated or object-oriented, but it does keep the pixel manipulation model working consistently. Something that canvas definitively lacks compared to an environment like Flash is any concept of object identity - once you draw out a shape, it's promptly forgotten and unavailable for things like mouse hit tests. This is what SVG is for I gather, and I hear that the original Processing environment has some of the same limitations.

One thing I've also found is that the current implementations of this technology are complete CPU hogs, as measured by the readiness of my Macbook to kick its cooling fan into high gear - canvas-only interfaces like Ask Ken devour cycles.

I certainly believe it's a possible improvement over the interaction model of the current state-of-the-art in online map warping, Schuyler Erle and Tim Waters's New York Public Library Map Rectifier. God bless those guys but the modal interface with server round-trip for retrieving a preview can get a bit clumsy.

I'm fervently hoping that an approach like this triangulated mesh can bump the interactiveness and usability of the NYPL map project, whose output could be so incredibly useful to an old map collection.

More broadly, I'm finding that the plugin-free development route might be a good one to pursue for real, mature interactive work. The rendering engines behind Safari and Firefox and becoming ubiquitous on desktop and mobile devices, and if you're willing to ignore the increasingly-irrelevant Internet Explorer and the occasional cooling fan overdrive you should be golden. Tom uses an informal measurement of the number of animated objects an environment can simultaneously throw at a screen, and canvas is definitively catching up to Flash quickly, while retaining the critical benefit of a fast, seamless start, something that Java has always lacked.

Comments (13)

  1. http://blog.klokan.cz/2009/10/raster-map-reprojection-warping-with.html seems to be heading in an interesting direction too. Nice work!

    Posted by Howard Butler on Monday, November 2 2009 7:35am PST

  2. Thanks Mike. I love starting the week with the future staring me in the face!

    Posted by Amenity on Monday, November 2 2009 9:42am PST

  3. Nice writeup! I think of Canvas as just a bag of bits; its basically just a scriptable image tag -- you have to keep track of everything yourself, which is great when you don't _want_ to keep track of things so you can get raw performance without overhead. SVG makes more sense with what Protovis is doing, since they need hit tracking and object tracking to do their graphs in general. Nice demos with the maps BTW! You should put the source up for folks to download and play with. Best, Brad

    Posted by Brad Neuberg on Monday, November 2 2009 12:04pm PST

  4. Thanks guys! Howard, I've been following Klokan's work a bit, I think it's interesting that he's using proj4js and a pixel-by-pixel model. It's not what I'd call fast but the results are beautiful. Brad, I agree that Protovis really needed to move to SVG. I do like that they've hit both sides of the aisle and settled on an optimal target. The source code for all this stuff is now on Github: http://github.com/migurski/canvas-warp Amenity, it's nice to see you too. =)

    Posted by Michal Migurski on Monday, November 2 2009 1:48pm PST

  5. This is a really excellent, smart post! I have a question, since I'm such a beginner. I don't quite understand how widely supported these technologies are (or will be). In particular, how much of this works on common versions of Internet Explorer? I've done some reading on this, but it's obviously complicated (like it's not supported unless you download something from Google? but will people do that?) I'd appreciate hearing your take on this.

    Posted by Mr. Beginner on Tuesday, November 3 2009 6:13am PST

  6. Oh, I should add... on Internet Explorer 7 on Windows, I get errors on Protovis and on the canvas warp example. I desperately want to use this technology myself, but if it's not in a relatively recent IE, I'm worried.

    Posted by Mr. Beginner on Tuesday, November 3 2009 6:24am PST

  7. Mr. Beginner, I'm pretty certain that none of this will work on common versions Internet Explorer. According to http://en.wikipedia.org/wiki/Canvas_(HTML_element) there is a plugin for IE that supports canvas, which is at least a similar situation to Flash, minus the install base of Adobe's plugin. To be totally honest, IE has been such a consistent headache for so many years that I simply ignore it when working on this stuff. If MS feels the need to get on this particular boat then I will applaud them, but it's definitely not something I'm willing to chase down.

    Posted by Michal Migurski on Tuesday, November 3 2009 9:11am PST

  8. Oops, that link should have a closing paren. Here: http://teczno.com/s/664

    Posted by Michal Migurski on Tuesday, November 3 2009 9:12am PST

  9. Thanks! This is very helpful. And alas, newbie though I am, I've already experienced a few Explorer-induced migraines. How do your clients react to non-IE-compatible work? Can we perhaps hope for change in attitude that lets us ignore Explorer?

    Posted by Mr. Beginner on Tuesday, November 3 2009 11:50am PST

  10. Last week I had similar problem. We got a SVG map that wasn't in Mercator Projection. We want to add place marks based on geo position so we need to transform it somehow. Looks like Delaunay triangulation might be a way to go.

    Posted by Marcin Ignac on Monday, November 16 2009 7:06am PST

  11. Marcin, hope it works out for you! There are a wide variety of tools in the GDAL suite that you might find useful as well, especially if the SVG map is in another known projection like Lambert or Albers.

    Posted by Michal Migurski on Monday, November 16 2009 9:01am PST

  12. hy in surfing on the web i find your blog and also can see that you are on FFFFound. So i research also an invitation. Thus if you want ... nico

    Posted by nico on Wednesday, November 18 2009 9:33am PST

  13. Sorry Nico, I haven't seen a new Ffffound invite in more than two years.

    Posted by Michal Migurski on Wednesday, November 18 2009 9:40am PST

Sorry, no new comments on old posts.

December 2014
Su M Tu W Th F Sa
 
   

Recent Entries

  1. bike ten: schwinn touring
  2. more open address machine
  3. open address machine
  4. making the right job for the tool
  5. the hard part
  6. end the age of gotham-everywhere
  7. on this day
  8. write code
  9. managers are awesome / managers are cool when they’re part of your team
  10. bike eight: french parts
  11. being a client
  12. bike seven: building a cargo bike
  13. blog all video timecodes: how buildings learn, part 3
  14. talk notes, urban airship speaker series
  15. john mcphee on structure
  16. blog all oft-played tracks V
  17. tiled vectors update, with math
  18. disposable development boxes: linux containers on virtualbox
  19. week 1,851: week one
  20. tilestache 0.7% better

Archives