tecznotes

There's something really nice going on in this series of zooms around San Jose Ave and 280. I like the introduction of layering as the details move into focus:

Two short videos from my morning commute.

1. I started recording this one right at the moment I swiped my credit card to pay for my TransLink pass (it's like a Bay Area Oyster card). From swipe to "transaction completed" takes about 45 seconds on these machines, with a disturbing amount of detail in between. "Dialing host" alone took most of that time. I thought credit card transactions were a solved problem?

2. This guy was riding up Market St at 10am on his tall bike. When he came to red lights, he'd have to circle a bit to avoid a tortuous dismount/remout cycle. He looked pretty badass up there.

Last week, Stamen was involved in the MTV Hope For Haiti Now telethon. It was apparently the largest such fundraising effort in history, pulling down over $60 million in what the New York Times called "a study in carefully muted star power".

So it was, you know, a pretty intense event to be a part of. With about a week or so of lead time, we put together an interactive map showing moderated Twitter messages connected to the live event and relief for Haiti generally. It was also part of the post-show on television.

It looks like this:

Sha and I traveled to Los Angeles to work out last minute kinks and watch over the project. Aaron was up here, carefully seeing to the smooth operation of the engine driving the Twitter collection process for the duration. Much of the office pulled together to crank out this project on pretty much no notice, and it was an inspiring and energetic effort.

I want to talk about that engine, though - it's occupied most of my headspace since we all got back from the holidays, headspace ordinarily full of geography and cartography. After dabbling in the consumption and development of real-time API's for a few years, we've started working with the high-volume Twitter stream this year. Back in 2007, Kellan Elliot-McCrea and Blaine Cook opened our eyes to the possibilities of streaming APIs with their series of Social Software For Robots talks. We poked at XMPP and other persistent technologies a few times, but what really whet the appetite was a project we did with MTV for the 2009 Video Music Awards, our first large-scale public project connected to the Twitter streaming API. The VMA project was a collaboration with social media monitoring company Radian6, who handled the backend bits. Given Stamen's development and design focus it was only going to be a matter of time before we started pawing at that backend technology ourselves.

The consumption and moderation system we have developed was christened "Hose Drawer" by Aaron, a name that neatly encapsulates two strands in our recent interests:

• "Hose" draws on Blaine's joking references to "drinking from a fire hose" about live streams of Twitter's real-time database, and now graces the official name of the service itself. The complete feed is called the fire hose, while a limited feed for casual users is the garden hose. I've heard that the names persist internally, with names like "hose bird cluster", referring to Twitter's avian mascot, among others.
• "Drawer" is a nod to last year's Tile Drawer, an EC2 virtual machine image for rendering OpenStreetMap cartography. We're thinking about a future where this kind of functionality is as much a piece of furniture as a PC, albeit one that can be created out of thin air and destroyed just as quickly. With a growing selection of infrastructure products, Amazon's Web Services are making it possible for us to develop services that act like products, materialized by small python scripts that bootstrap themselves into multi-machine clusters.

This is the naming convention you get from living the pun-driven life*.

The firehose offers a continuous flow of data, yet requires us to break that flow down into discrete chunks. I've been thinking a bit about this process in our work since Tiles Then, Flows Now, my 2008 Design Engaged talk about map tiles and continuity, so much so that I see the process of breaking and reforming continuity everywhere around me. I flew several times last month, and each time I passed through security I thought about the check-in process as an elaborate map/reduce implementation, atomizing a stream of passengers into packets of shoes, laptops, jackets, bags and bodies. Numerous fascinating patents cover the splitting up of a steady flow, from t-shirts cut from unending tubular knit fabrics, to continuously-cast steel to simulated egg yolks sliced from unbroken cylinders.

Scott Bilas, whose Continuous World Of Dungeon Siege paper I drew on heavily for DE 2008, describes the challenges of writing a streaming system in the world of video game design:

The core problem that we had so much trouble with is that, with our smoothly continuous world, there are no fixed places in the world to periodically destroy everything in memory and start fresh. There are no standard entry/exit points from which to hang scripted events to initialize or shut down various logical systems for plot advancement, flag checking, etc. There is no single load screen to fill memory with all the objects required for the entire map, or save off the previous map's state for reload on a later visit to that area. In short, not only is the world continuous, but the logic must be continuous as well!

The pattern we see here is to keep crises small and frequent, as Ed Catmull of Pixar says in an excellent recent talk. When describing the difficulty Pixar's artists had with reviews ("it's not ready for you to look at"), he realized that the only way to break through resistance to reviews was to increase the frequency until no one could reasonably expect to be finished in time for theirs. The point was to gauge work in motion, not work at rest. "So often that you won't even notice it," said Elwood Blues.

I'm interested to see where we can take this product-that-isn't-a-product. We're going to be using it for an upcoming high-profile sports broadcasting client (you'll hear more about this in another two weeks), and the stress tests administered by the Haiti telethon have shown exactly where we need to do more work. Oddly the overall performance was great, but we found ourselves occasionally needing to go tweet diving through the database, looking for specific messages that were good for television. This ability to reach in a meddle with the guts, place yourself on a calm island in the middle of the stream, rewind the tape and alter the flow, is the next type of control we're experimenting with.

I had kind of a bonkers trip to New York this past week. My proximate reason for being there was an invitation to Microsoft Research's annual Social Computing Symposium (SCS), an invitational held this year at NYU's ITP program. Another reason was to visit the U.S. Geological Survey's Volunteered Geographic Information workshop. A third was to pay visits to a bunch of east coast superheroes. The thing I love most about these whirlwind adventures is that you have an excuse to compress a full schedule of visiting into a very short span of time. I had a chance to meet and reconnect with a few people I like and respect quite a bit. I felt warmly embraced.

The SCS theme this year was "City As Platform", and the structure of the event was brief talks punctuated by puppet shows. About half the people gave presentations, though I wasn't one of them because I was thinking more about the USGS thing for later in the week (more on that below). Steven Johnson gave an opening talk that made me think I had accidentally bumbled into a TED or Davos, though the situation did rapidly improve.

Kevin Slavin gave my favorite kind of talk, a barnstormer that I found simultaneously fascinating and provided a lot of hooks for debate. Liz Goodman took better notes than I, and the basic gist was like this:

1. Concealment technologies, like the radar-defeating B2 Bomber, work by breaking up a shape rather than hiding it entirely.
2. High frequency, algorithmic trades work the same way, tearing larger financial movements down into microscopic dust.
3. This modern computerized finance stands in contrast to people-driven, proximity-reliant trading off the past that gave rise to today's financial district.
4. Physical proximity required for algorithmic trading is no longer a result of human distance but of microsecond ping times necessary to exploit fast, small differences in price.
5. Why can't the whole thing just float off and go away if it doesn't need people anymore?

Kevin connected to the city theme with 60 Hudson as a pivot. It's a massive telecommunications node, and physical distances along the network from this building translate to communication delays, which subsequently translate to money lost or money gained. We used to think about time in terms of distance ("The furlong was the distance a team of oxen could plough without resting"), which we've mostly forgotten since communications speeds got so fast, but the high rents directly around this building show that on some scales, velocity of communication can still be measured.

Now I don't quite agree with Kevin's conclusion. Where he sees an inhuman system that's begging to be pushed off into space where it will cease to bother its once creators, I'm looking at a natural consequence of technological speed in the service of technology. Someone's always going to want to exploit the seams with science. What we need is not to set the thing adrift off-world but to tame it with a new Glass-Steagal. Still, the core humanism of where he goes with this is touching, beautiful and deserving of some attention. Most "city as platform" talk comes in two varieties: information technology and space syntax. There was plenty of both on offer at ITP, lightly brushing the testicular, amoral war-metaphoring that views urban fabric and shared space as just scenery to play out control scenarios. I don't want to call out anything in particular, so I'll just link to an unfortunate recent BLDGBLOG post about Die Hard and the IDF walking through walls after reading Deleuze and Guattari. One of the commenters, Jim Meredith, has this response:

"However, as you note early in you piece by quoting those who maintain, live in, and trust the concept of private space, the Nakatomi/Nablus/DieHard concept is, in fact, a shocking and gross violation of a core concept of civil society."

(apologies to Marc Ngui)

Not everyone views the city as a battle suit or a maze of twisty passages all alike, and it's worth thinking about provided functions (sewage, macadamizing) in terms of what they do, who they do it for, and for what price. Anil Dash in particular described his Expert Labs concept, a project to consider and modify the motivations for geek work and the federal government. Did you know that you can't, technically, "give" things to the U.S. government? It's true and Anil's trying to change that. Which is an interesting segue to the USGS event in DC, a workshop intended to explore successful volunteer geographic data programs (like OpenStreetMap) to see how that can be applied to the forthcoming update to the National Map.

Thanks to a door malfunction on my NYC/DC flight, I arrived late to the party and unwittingly delivered a closing keynote. The full content of the talk and my slides immediately follow these other exciting things that happened:

• I visited the New York Public Library map people twice and was allowed to flip through a 17th century Dutch naval atlas.
• I got to meet the amazing New York Times graphics department, including Matt Ericson, Matthew Bloch, Amanda Cox, Shan Carter, and others. These are the people turning our ideas about interaction and mapping online utterly upside-down.
• Grand Central Oyster Bar.
• Veselka for 2:00am pierogi.
• Mark Hansen's genius installation at the NYT lobby.
• Poking my head into the Saturday Night Live studio during filming of Laser Cats.
• Played the new(ish) Mario Brothers Wii game and discovered that my muscle memory of the old game's standard controls has remained fully intact for 20 years despite no external reinforcement.

Now, my slides for the USGS thing.

Readiness.

For many years, OpenStreetMap was understood to be a project about the future, meant for productive use someday but clearly unready. Potential users still had to be convinced, and we passed over OSM as a data source for projects in favor of commercial data on a number of occasions. This phase lasted from 2004 to about 2008 for us at Stamen.

In 2008, it started to look more done than not-done.

The social structure and set of motivations that OSM thrives under can be seen as part of the broader trend towards shared development practices and better communication enabled by the open internet. We're seeing a trend toward a larger number of smaller operations unified by an ethos of participation and local scale, something that Brian Marick has jokingly summarized as "artisinal retro-futurism" and "team-scale anarcho-syndicalism". I love this description.

In our work, we're generally on the receiving end of map data, and I've got some examples showing cartography I particularly enjoy that's benefited from OSM as a resource.

Stamen's "Fresh" map style for Cloudmade, showing legible web cartography for inner London, was immediately applicable to the rest of the world upon launch. It's used by OffMaps, Noticings, and others as a default, web-native basic road map without some of the distracting recognizeability of Google Maps.

UK designer Matt Jones of BERG used Cloudmade's style editor product to apply Kevin Lynch's ideas about the city as a network of paths, edges, districts, nodes, and landmarks for heavy traveler social network Dopplr. Particular ideas about the role of information on the map can be rapidly experimented with and published to a broad audience.

Stamen also did the Cloudmade Midnight Commander map style, answering a creative brief suggesting a design appropriate to Jason Bourne's in-car GPS. Again, something that starts as a small idea or even a joke can very quickly go into full production when simple, well-understood data is available.

I've also been exploring the adaptation of paper maps like this 1996 Rand McNally SF street atlas popular among bike messengers. The result has been OSM-derived, print-ready cartography that additionally incorporates civic parcel and contour data.

Finally, Yahoo photo-sharing web service Flickr has used specific portions of OpenStreetMap data in places where Yahoo Maps has poor or non-existent coverage. The transient desert city of Burning Man is one such example, surveyed and laid-out fresh each year. OSM Foundation board member Mikel Maron contributes his mapping expertise to the Burning Man Earth project to provide visual context for the thousands of photogrpahs that come from the art festival each year.

When anti-government protests broke out in Iran last year, Flickr was able to rapidly pull in high-quality maps of Tehran to provide immediate geographic context to the sudden flood of news and imagery. I'm told that high quality road data for cities like Tehran is traditionally quite hard to come by via normal channels, since there's very little road navigation market for western companies there.

Services such as Google's and the iPhone are bringing more different kinds of people in close contact with cartographic design through their daily lives than ever before. We know that use and familiarity breeds discerning taste, and cartography has become popular, decorative, desirable, and functional.

What specific technical characteristics of OpenStreeMap motivate the creation of this broadly useful data set? I think that a few critical decisions made in the early days of the project have endowed it with generative properties, the "capacity to produce unanticipated change through unfiltered contributions from broad and varied audiences" as Jonathan Zittrain has said.

Three specific features of OpenStreetMap have this effect.

First, each object in the system has a unique and durable identifier. Objects basically stay put, and are composed of very simple primitives. There are no curvy lines in OSM. The identity of each object in the OpenStreetMap database is exposed to the outside world, so that Flickr can let you refer to specific OSM features in your photography. Here we have a photograph of the Queen's Chambers in Manchester that's explicitly connected to the building footprint in OSM because the two entities can be tagged together. The system is useful even when not explicitly mapped.

Second, the tagging structure is free-form. You can apply your own arbitrary descriptions to features, generally conforming to the basic expectations of the community with tags like "highway", but often departing from them entirely. Andy Allan created a rendering for his OpenCycleMap project that specifically called out features with tags relevant to bicyclists, like these portions of the bike network in Oakland. New tag conventions are decided through use rather than committee. Most theoretical arguments about the appropriateness of one approach over another are moot until actual use by a number of people over time can be shown.

Finally, there is an open API. OSM's core feature is a complete, well-understood way to move data into and out of the service. This has made it possible to create numerous ways of recording and editing OSM data. JOSM is the editor for the precise, obsessive, and German. Potlatch is the web-based editor that lives under the default "edit" tab on the site, and includes simple tools for pulling different kinds of points of interest into the aerial map.

Walking Papers is my own research project, and provides a way for paper data collection to help OSM. Each of these applications takes advantage of OSM's well-documented and simple protocol to moved information through the system, with varying amounts of complexity available to different user populations.

I want to close with this napkin sketch by Mikel Maron. Many years ago, Mikel showed a possible "common operating procedure" around the project, incorporating numerous technologies not normally associated with desktop geography. Much of the methodology drawn here did not yet exist at the time, but Mikel could be sure that with healthy usage patterns it would be possible to draw double heads on each of the arrows with some reasonable expectation of future possibility.

Next week, from the 10th to the 15th, I'll be in New York, with a very quick side trip to DC. Anything interesting I should be doing with a smallish amount of free time later in the week?