...so take it easy.

Thought experiment for the day, spurred by jets flying overhead for fleet week. If something out of ordinary is going on, and you need to ask people around you, what do you do?

Imagine a chat system, similar to IRC or AOL - a big room where you log in and talk to lots of other people simultaneously. When you log in, your Fire Eagle account gets a ping, so that the server can know your geographic location to some level of precision. You are then dumped into a big room, with the 150 people physically closest to you in the real world. As long as there are fewer than 150 total people in the system, everyone gets to talk to one another. As more people join, you begin to see overlapping conversation bubbles. You might be in San Francisco, talking to someone in Kansas. That person in Kansas can talk to someone else in New York, but you can't. Your conversational circle is strictly limited to the nearest 150 people, some of whom drop off occasionally as they are bumped out by more proximate newcomers.

As the population of the system grows, everyone's personal horizons begin to shrink. With enough people, eventually you're talking to the people right in your neighborhood. To get a message to someone across the country, you might lie about your location, or ask that it be passed on, Milgram-style.

Would this feel like a natural way to interact with people around you, self-limited to a reasonable number of participants but always those around you? Would people lie about their location? Would it be like Usenet, dense with microscopic subcultures? Would certain people emerge as hubs, offering to post messages around between bubbles in the absence of other means of communication?

The third Design Engaged happened in Montreal, just a few days ago.

Number two, held in Berlin three years ago in November, was something like a debut party for Stamen. Many of the people whose circles we're in now, we met there and then for the first time. DE is a high water mark for conference-style events, borrowing much from the DIY ethos. In Berlin, attending meant speaking, and most of the event was occupied by rapid-fire descriptions of what each participant was up to. Adam beat the drum for peak oil and copper theft, while Jack delivered a fairly stream-of-consciousness rundown of interesting comic books and metals with low eutectic melting points. This time, my personal favorite was Russell's talk on advertising and design, the upshot of which was that "it's worth not being shit."

I adapted some of my favorite bits from the UX Week talk and described a tiling metaphor for the web, why it was interesting, and how we're about to hit something of an inflection point. The title of the talk is "Tiles Then, Flows Now", changed from "Tiles Now, Flows Soon" when I decided that too many people were saying too many things about the future for me to join in as well.

Three years ago, we first showed Cabspotting at Design Engaged 2005. It was one of our first map projects to combine a tiled base layer and constantly-updating data.

The tiled-map idea was not exactly new at the time, but it had just been immensely popularized by Google Maps, released earlier in 2005. The scale and coverage of GMaps was a wholly new thing.

I'd already spent plenty of time futzing with Terraserver and other satellite image sources looking down on familiar places, but those services always forced viewers to understand image resolution and source satellites. It was hard to pan around, the UI metaphor was ultimately form-based ("click here to go east").

GMaps showed how it was possible to fake the appearance of continuous flow by assembling mipmapped images on the client, and serving up simple 256x256 tile images. When it first came out, I spent an hour panning slowly around the Bay Area from San Francisco to San Jose.

This all coincided with the appearance of REST as a guiding metaphor for the web. REST gives you a lot of mileage; we faked the continuous flow of user activity on Digg by punctuating snapshots over time. Like film, you can string a number of these together to present a convincing illusion of continuity. Digg Swarm makes requests every 30-60 seconds, yet visually dribbles out activity continuously. Worries about certain events being dropped are basically swept under the rug here, the picture is lossy.

All of this is motivated by a desire to pump dynamic, flowing data through the "thin straw" of the internet. Here's another experiment that's starting to come back around, for SOM's San Francisco City Model. The monochrome renders are rich, heavy and detailed. The false-color renders below encode information about tax assessor's parcels. Combined, the maps give you a convincing form of interaction including per-building highlights and links to parcel information faked with simple images and a bit of code.

Oakland Crimespotting is a similar bit of foolishness. Initially data was harvested by scraping vintage late 1990's CrimeView map products, now it comes from Excel spreadsheets updated nightly. Yet the view we try to present is a continuous, unbroken surface for exploration. We routinely get asked about same-day crimes by people lulled into a sense of immediacy. It doesn't help that we promise to tell people what the sirens in their neighborhood are in realtime of course.

This project directly led to Modest Maps, a condensation of thoughts on good, flexible, online cartography for designers.

It also connects to this years-long meditation on Oakland's historical geography. Motivated by a project for a sustainable urban design class, my girlfriend Gem researched century-old maps of the east bay to understand why certain city streets were freakishly wide like this view of E. 21st St and 14th Ave. The thinking was that all this seemingly wasted space might be more effectively used with a permeable road surface and native vegetation to revive the natural water filtration systems disrupted by culverted creeks and concrete construction.

Here's a whole series of downtown Oakland maps. It's great fun to run through them, showing where construction has taken place, where rail used to run, what the oil companies considered relevant, and where the freeways were eventually dropped in. In the 1912 map you can see exactly where the old Key Route streetcars used to run, and therefore why certain roads are now uselessly wide. I've just won an EBay auction for a 1967 road map that will show the drastically modified, post 1950s landscape of raised highways.

This last Oakland map is actually sourced from OpenStreetMap, an attempt to introduce Wikipedia's crowdsourcing model to geographical data. I like that it shows a lot more pedestrian-relevant information than the Microsoft Virtual Earth one immediately before. The really interesting thing about this whole series of Oakland Maps is that they're tightly matched to the specific cartographic projection used by Google and Microsoft and everybody else: it's a standard that provides strong footing for a comparison.

This is something similar we've been working on for the London Organizing Committee for the Olympic Games. Again you can see the contrast between a map that draws attention to what's on the ground vs. one that helps your satnav get you through town unscathed.

Being able to take certain things for granted, like projections, publishing mechanisms, and display libraries, allows for enormous variations in expression. There's also a bit of Google cargo cult mentality here, looking at the technological leavings of the biggest technology company out there. For better or worse, they've set the tone for geographic publishing by using the web more effectively and deferring complexity to the client browser.

All of this is a way of showing how Roy T. Fielding's 2000 PhD thesis on Representational State Transfer has utterly dominated the discourse of publishing through the web. It's an incredibly productive architectural idea, and it's been the primary area of experimentation and development for geeks like us since the implosion of the previous web bubble.

The new thing on the horizon, though, is a promise to stop faking it with really-realtime event-based notifications. Sudden interest in Jabber / XMPP, an instant messaging protocol from ~2001, is bubbling up from the kinds of system design and operations geeks building sites like Flickr and Twitter.

There's a connection to game development here, illustrated by a favorite paper of mine called The Continuous World of Dungeon Siege. In it, author Scott Bilas describes the technical challenges inherent in presenting an illusion of seamlessness. Some of it, the way in which the environment is loaded and constructed around the ever-shifting vantage point of the player, sounds a lot like the RESTful tile-based methodology driving the maps above. A lot of it also sounds an awful lot like the pipelines and data flows I'm hearing from friends working on massive, distributed services like Flickr.

These are the cartograms published in the wake of the 2004 U.S. presidential election, they were a major influence on our geographic thinking when they were first released: imagine, using color and spatial distortion to communicate the underlying complexity of a contentious national election.

Meanwhile, these are graphics from Nate Silver's 538, a very different current approach to electoral politics from the world of continuous statistical analysis in baseball. This is a lot more like the current environment, showing constant change from multiple daily polls. Nate taps a constant stream of contradictory, voluminous data to create simulations and ultimately predictions of the election outcome in November.

They also do maps. Right now they're predicting a 90% chance of an Obama win of some form, so yay.

I'm going to close with a slide about capacity planning from Flickr's John Allspaw. I love how on the right, they're translating their capacity metrics into days-of-Flickr-remaining. REST is going to stick around for as long as it continues to be a productive metaphor that can guide action and make predictions, but we're going to see a lot of this stream-tapping behavior bubble up over the next few years.

Extra thanks to Andrew, Boris, Jenn, and Mouna for organizing the festivities. It was nice to visit Laika in person!

A few weekends ago, I had the opportunity to participate in the 2008 Online News Association conference in Washington DC. Laura Cochran of the Washington Post invited me to join a panel on mapping crime, along with USAToday datacruncher Paul Overberg and LA Times power couple Sean Connelly and Katy Newton.

The conference got off to an inauspicious start when Tina Brown capped off a terrible keynote Q&A by called a journalism student an "easy lay". Fortunately, the How We Built It track featuring MSNBC, New York Times, Las Vegas Sun and others was a perfect way to spend a conference Friday. News organizations producing interactive pieces for the web described the various challenges they encountered, and it was fascinating to hear about the sausage-making process from the inside.

One of the most important things I learned in this series of talks is that no one likes their IT department, not even at the New York Times. Presenters repeatedly described ways in which they had to circumvent or overrule their own IT infrastructures to get anything interesting done. Two stood out. I asked the designers and developers at the Las Vegas Sun about the political/technical environment in their organization that allowed them to explore and refine iterative, agile production methods, and they said that it was necessary for them to go straight to the top for a mandate from the editor to give the group decision-making power over their development and deployment environment. Matt Ericson and Aron Pilhofer of the NYT described a more circuitous approach. Apparently, the NYTimes.com online election coverage is hosted entirely on Amazon's pay-as-you-go EC2 service, and totally detached from the content management and other server infrastructure at the Times. They use Ruby on Rails and other open-source software components to develop and deploy their work, and their seven-person team is wholly responsible for the care and feeding of these servers. This was a shocking thing to learn, and it raised my opinion of the NYT team by a solid order of magnitude.

Despite such a high-level of problem solving ingenuity, the majority of people in the business are journalists first and programmers last. The technical proficiency and funding available to publishers less blessed or lucky than the major dailies is substantially lower, and forces them into products like Caspio. This company had a substantial percentage of ONA attendees by the short hairs with their hosted solutions for data-driven web pages and mashups. I'm convinced that this is bad news, but I'm already predisposed to suspicion of turnkey software for this kind of work. I've also read plenty about the product in particular from journalist/technologist Derek Willis, who offers six reasons to look past Caspio in his blog archives.

The silver lining on this particular cloud is Django, the Python web framework developed by Simon Willison, Adrian Holovaty, and others. Django is finding a solid niche in the journalism world as a thoughtful, educated, D.I.Y. response to hosted rentware, and a kind of software Schelling point for journalists looking to really understand data-driven reporting.

The end-of-conference Online Journalism Awards ceremony was a parade of excellent interactive and data-driven work. The impression I got here was of deadline-motivated ingenuity on a tight budget. My co-panelists Sean and Katy especially illustrate the point with their 2007 winner Not Just A Number, a look at homicide in Oakland. Despite focusing on the same geographical area and the same topic, it's such a wildly different project from our Oakland Crimespotting. By narrowing their sights to the year's killings and entering the community itself to talk with those affected, Not Just A Number shows how narrative rigor can color statistical data with a backstory.

Overall, the conference had a distinctly different feel than the tech-oriented events I generally participate in. For one, there's an undercurrent of a siege mentality in journalism right now, with newsrooms cutting staff and print operations frozen stiff in the headlights of the internet. The focus on narrative and story gives a softer edge and an escape valve, though - this group is not primarily a tech-driven community, but they catch on to new developments quickly and bend them into the service of storytelling.

I'm continuing my months-long meditation on city cartography with a jump into OpenStreetMap, the "editable map of the whole world ... being built largely from scratch ... released with an open content license."

A few weeks back, I released Cascadenik, an application of cascading stylesheets to the Mapnik rendering library. The rationale for writing it in the first place was to replace the base map we're using for Oakland Crimespotting. I love the look of Microsoft's VEarth cartography, but it's missing data crucial to an understanding of urban crime: parks, schools, businesses, and transit. OpenStreetMap is the only free-as-in-speech way to create a beautiful, useful, and complete city map that can incorporate such ground truths. The NavTeqs and Teleatlases of the world where the online mapping services get their data are primarily interested in and funded by navigation, so it's not going to be in their interest to go neighborhood-deep to track locations of playgrounds or liquor licenses.

It's going to take a substantial outlay of cognitive surplus to get all this information into the map, but I've started by working on the visual appearance to get a feel for OSM's data:

(In-progress stylesheets can be found bundled with Cascadenik in mapnik-utils)

There are more than a few social decisions encoded in those styles:

• I'm trying to foreground modes of public transportation, especially rail. BART plays such a huge role in the Bay Area, and an understanding of where stations lie in relation to homes and businesses is crucial to understanding the local streetscape. For an historical view of this, check out my old flea market mapping experiment, and pay attention to the difference in appearance between the 1912 map, made to show rail coverage, and the 1936 map, made by Shell Oil to hide it.
• Taking a cue from the 1936 map and VEarth's road rendering, there's a much sharper distinction between major and minor roads, with minor roads dropping back to form a spidery matrix of connectivity between major roads and transit stops. This seems to help with the legibility of parks and other features at zoomed-out views, showing how they anchor neighborhoods and provide textural variety. It also makes room for labels on schools & parks that would otherwise be crowded out by street names.
• A lot of excess detail is being intentionally omitted. Parking lots and ATMs exist in the standard OpenStreetMap tileset, but I'm leaving those out here because I don't feel that they're helpful. I'm also omitting underground rail, it's just not relevant to surface use.
• The color of freeways is red, a fairly standard decision seen on most U.S. maps. Major roads are all fairly pale, with small variations in color around yellow and orange to make them visible but less overpowering than the blues and reds used by OpenStreetMap's own tiles.

Working with Potlatch, the Flash-based OSM editor, has been interesting. Although it does the job exceedingly well, I'd welcome an editing interface derived more from KidPix and SimCity than AutoCAD or ArcGIS. My dream is a UI that dispenses with tagging in favor of tools like "road", "school", "park", or "bulldozer".

The new tiles are being updated from fresh OSM data on an almost-daily basis, and hosting on S3 means you can hit it pretty much all you want for your nine-county Bay Area mapping needs.

Now, to get all these schools included.

This just went out yesterday, our new Hurricane Tracker for MSNBC:

I'm so impressed with the work, co-created by Tom and Geraldine with raw data licensed from Hurricane Mapping. They're evacuating New Orleans right now, I hope the reaction to this storm isn't as tragically bungled as the last one.