Michal Migurski's notebook, listening post, and soapbox. Subscribe to this blog. Check out the rest of my site as well.

Oct 9, 2008 5:48am

design engaged 2008

The third Design Engaged happened in Montreal, just a few days ago.

Number two, held in Berlin three years ago in November, was something like a debut party for Stamen. Many of the people whose circles we're in now, we met there and then for the first time. DE is a high water mark for conference-style events, borrowing much from the DIY ethos. In Berlin, attending meant speaking, and most of the event was occupied by rapid-fire descriptions of what each participant was up to. Adam beat the drum for peak oil and copper theft, while Jack delivered a fairly stream-of-consciousness rundown of interesting comic books and metals with low eutectic melting points. This time, my personal favorite was Russell's talk on advertising and design, the upshot of which was that "it's worth not being shit."

I adapted some of my favorite bits from the UX Week talk and described a tiling metaphor for the web, why it was interesting, and how we're about to hit something of an inflection point. The title of the talk is "Tiles Then, Flows Now", changed from "Tiles Now, Flows Soon" when I decided that too many people were saying too many things about the future for me to join in as well.

Three years ago, we first showed Cabspotting at Design Engaged 2005. It was one of our first map projects to combine a tiled base layer and constantly-updating data.

The tiled-map idea was not exactly new at the time, but it had just been immensely popularized by Google Maps, released earlier in 2005. The scale and coverage of GMaps was a wholly new thing.

I'd already spent plenty of time futzing with Terraserver and other satellite image sources looking down on familiar places, but those services always forced viewers to understand image resolution and source satellites. It was hard to pan around, the UI metaphor was ultimately form-based ("click here to go east").

GMaps showed how it was possible to fake the appearance of continuous flow by assembling mipmapped images on the client, and serving up simple 256x256 tile images. When it first came out, I spent an hour panning slowly around the Bay Area from San Francisco to San Jose.

This all coincided with the appearance of REST as a guiding metaphor for the web. REST gives you a lot of mileage; we faked the continuous flow of user activity on Digg by punctuating snapshots over time. Like film, you can string a number of these together to present a convincing illusion of continuity. Digg Swarm makes requests every 30-60 seconds, yet visually dribbles out activity continuously. Worries about certain events being dropped are basically swept under the rug here, the picture is lossy.

All of this is motivated by a desire to pump dynamic, flowing data through the "thin straw" of the internet. Here's another experiment that's starting to come back around, for SOM's San Francisco City Model. The monochrome renders are rich, heavy and detailed. The false-color renders below encode information about tax assessor's parcels. Combined, the maps give you a convincing form of interaction including per-building highlights and links to parcel information faked with simple images and a bit of code.

Oakland Crimespotting is a similar bit of foolishness. Initially data was harvested by scraping vintage late 1990's CrimeView map products, now it comes from Excel spreadsheets updated nightly. Yet the view we try to present is a continuous, unbroken surface for exploration. We routinely get asked about same-day crimes by people lulled into a sense of immediacy. It doesn't help that we promise to tell people what the sirens in their neighborhood are in realtime of course.

This project directly led to Modest Maps, a condensation of thoughts on good, flexible, online cartography for designers.

It also connects to this years-long meditation on Oakland's historical geography. Motivated by a project for a sustainable urban design class, my girlfriend Gem researched century-old maps of the east bay to understand why certain city streets were freakishly wide like this view of E. 21st St and 14th Ave. The thinking was that all this seemingly wasted space might be more effectively used with a permeable road surface and native vegetation to revive the natural water filtration systems disrupted by culverted creeks and concrete construction.

Here's a whole series of downtown Oakland maps. It's great fun to run through them, showing where construction has taken place, where rail used to run, what the oil companies considered relevant, and where the freeways were eventually dropped in. In the 1912 map you can see exactly where the old Key Route streetcars used to run, and therefore why certain roads are now uselessly wide. I've just won an EBay auction for a 1967 road map that will show the drastically modified, post 1950s landscape of raised highways.

This last Oakland map is actually sourced from OpenStreetMap, an attempt to introduce Wikipedia's crowdsourcing model to geographical data. I like that it shows a lot more pedestrian-relevant information than the Microsoft Virtual Earth one immediately before. The really interesting thing about this whole series of Oakland Maps is that they're tightly matched to the specific cartographic projection used by Google and Microsoft and everybody else: it's a standard that provides strong footing for a comparison.

This is something similar we've been working on for the London Organizing Committee for the Olympic Games. Again you can see the contrast between a map that draws attention to what's on the ground vs. one that helps your satnav get you through town unscathed.

Being able to take certain things for granted, like projections, publishing mechanisms, and display libraries, allows for enormous variations in expression. There's also a bit of Google cargo cult mentality here, looking at the technological leavings of the biggest technology company out there. For better or worse, they've set the tone for geographic publishing by using the web more effectively and deferring complexity to the client browser.

All of this is a way of showing how Roy T. Fielding's 2000 PhD thesis on Representational State Transfer has utterly dominated the discourse of publishing through the web. It's an incredibly productive architectural idea, and it's been the primary area of experimentation and development for geeks like us since the implosion of the previous web bubble.

The new thing on the horizon, though, is a promise to stop faking it with really-realtime event-based notifications. Sudden interest in Jabber / XMPP, an instant messaging protocol from ~2001, is bubbling up from the kinds of system design and operations geeks building sites like Flickr and Twitter.

There's a connection to game development here, illustrated by a favorite paper of mine called The Continuous World of Dungeon Siege. In it, author Scott Bilas describes the technical challenges inherent in presenting an illusion of seamlessness. Some of it, the way in which the environment is loaded and constructed around the ever-shifting vantage point of the player, sounds a lot like the RESTful tile-based methodology driving the maps above. A lot of it also sounds an awful lot like the pipelines and data flows I'm hearing from friends working on massive, distributed services like Flickr.

These are the cartograms published in the wake of the 2004 U.S. presidential election, they were a major influence on our geographic thinking when they were first released: imagine, using color and spatial distortion to communicate the underlying complexity of a contentious national election.

Meanwhile, these are graphics from Nate Silver's 538, a very different current approach to electoral politics from the world of continuous statistical analysis in baseball. This is a lot more like the current environment, showing constant change from multiple daily polls. Nate taps a constant stream of contradictory, voluminous data to create simulations and ultimately predictions of the election outcome in November.

They also do maps. Right now they're predicting a 90% chance of an Obama win of some form, so yay.

I'm going to close with a slide about capacity planning from Flickr's John Allspaw. I love how on the right, they're translating their capacity metrics into days-of-Flickr-remaining. REST is going to stick around for as long as it continues to be a productive metaphor that can guide action and make predictions, but we're going to see a lot of this stream-tapping behavior bubble up over the next few years.

Extra thanks to Andrew, Boris, Jenn, and Mouna for organizing the festivities. It was nice to visit Laika in person!

Comments (7)

  1. Regarding REST vs. event streams, these are worth reading (if you haven't already): http://roy.gbiv.com/untangled/2008/paper-tigers-and-hidden-dragons http://roy.gbiv.com/untangled/2008/economies-of-scale

    Posted by Ryan on Thursday, October 9 2008 3:35am UTC

  2. Hi Ryan, Thanks for the links - I've seen the first but not the second. I think Roy is sadly way off the mark in the Paper Tigers post, particularly with regard to Flickr. I've heard from inside that he's too low by well over an order of magnitude in his estimation of Flickr's userbase ("let's be generous and estimate it at one million") so I think his sparse bit array is going to be quite a bit larger than he estimates. The mapping from bit array to user ID and is the real issue, and he sort of waves his hands through it. I can only see it working with sequential user IDs where new accounts are appended to the end of a strictly-growing state representation, bad for two reasons: it's poor form to expose internal identifiers like numeric primary keys, and there's going to be a lot of dead air at the start as old accounts go dark due to disuse or cancellation. It probably doesn't matter, though - the places where proper experimentation with streams and XMPP happens will almost certainly be on the margins, perhaps in smaller services that don't have as much technical debt sunk into their current infrastructure. I'm most of all interested in the sorts of new behaviors that proper experimentation with XMPP will lead to. It seems best adapted to places where mutual obligations and flows are important, which is different from the statelessness and choppiness of REST. Roy's thesis has been one of the most consistently productive design ideas for a medium I've ever seen, so I'm certainly not arguing that it's going to "die" in any way; I still routinely deal with clients who are *just barely* getting on the bus and I encourage them to do so with no doubts about the longevity of the idea. I do believe that there are new forms of behavior being explored by some people on the margins, and those new behaviors are popping up in unexpected places. Just today I had a look at the API calls driving http://flickr.com/explore/panda, and found a "method=flickr.streams.getStream&stream_id=1" - interesting!

    Posted by Michal Migurski on Thursday, October 9 2008 5:01am UTC

  3. Hi Michal, I think you are misreading the Paper Tigers post. Be sure to read the comments. I wish we were around a table so I could draw on a napkin what is REST *and* HTTP. You seem to put XMPP into play which is a protocol in opposition to REST. REST, as you mentionned it, is an architectural style. HTTP is the protocol. Flickr has /some/ REST style but it is not a RESTful API. And you can hammer it in many ways, it will not become it, just because say it ;) That doesn't make Flickr bad, but that misleads people about some of the understanding of HTTP and REST. And I know where you get the information about Flickr ;) Next time you come to Montreal, we sit at Laika with Boris and if Aaron is around, it means that we will have a funny ol' time discussion ;) love it!

    Posted by karl on Thursday, October 9 2008 10:20am UTC

  4. Hi Karl, I admit I'm being a bit fuzzy in my terms! I'm using XMPP as an example of a style based on agreements and flows, modeled as persistent TCP connections. No one's really come up with as cohesive and useful description of what these things get you as Roy has with REST, so it's all a bit fumbling in the dark for a while. Probably fine, definitely interesting! Are you one of the people who gives Aaron a hard time for making all Flickr API responses 200? ;)

    Posted by Michal Migurski on Thursday, October 9 2008 4:01pm UTC

  5. naaaah I'm a gentle guy. I'm cuddling Aaron when he (sometimes we) cook!

    Posted by karl on Friday, October 10 2008 5:29am UTC

  6. check this site out re: elections: http://www.perspctv.com interesting to note the venn diagrams depicting the media coverage of the candidates.

    Posted by David on Sunday, October 12 2008 4:14pm UTC

  7. David, the venn diagrams are excellent, it seems like the precise are matters there.

    Posted by Michal Migurski on Sunday, October 12 2008 5:30pm UTC

Sorry, no new comments on old posts.

October 2023
Su M Tu W Th F Sa

Recent Entries

  1. Mapping Remote Roads with OpenStreetMap, RapiD, and QGIS
  2. How It’s Made: A PlanScore Predictive Model for Partisan Elections
  3. Micromobility Data Policies: A Survey of City Needs
  4. Open Precinct Data
  5. Scoring Pennsylvania
  6. Coming To A Street Near You: Help Remix Create a New Tool for Street Designers
  7. planscore: a project to score gerrymandered district plans
  8. blog all dog-eared pages: human transit
  9. the levity of serverlessness
  10. three open data projects: openstreetmap, openaddresses, and who’s on first
  11. building up redistricting data for North Carolina
  12. district plans by the hundredweight
  13. baby steps towards measuring the efficiency gap
  14. things I’ve recently learned about legislative redistricting
  15. oh no
  16. landsat satellite imagery is easy to use
  17. openstreetmap: robots, crisis, and craft mappers
  18. quoted in the news
  19. dockering address data
  20. blog all dog-eared pages: the best and the brightest