tecznotes

The full title of this 1973 U.C. Berkeley public planning book (recommended by A Better Oakland) is formidable: Implementation: How Great Expectations In Washington Are Dashed In Oakland; Or, Why It's Amazing That Federal Programs Work At All, Economic Development Administration As Told By Two Sympathetic Observers Who Seek To Build Morals On a Foundation Of Ruined Hopes. It seems significant that all the illustrations are excerpts of Rube Goldberg machines.

I bought this book because of the Oakland connection, but there's a lot in here that's relevant to any form of project planning and completion, especially for software developers and designers (like me) trying to figure out why it's so easy to start, and so hard to finish, a project. Other writers in the development world have touched on this before, and there's an entire discipline called Agile that seeks to cut through impediments to completion like a Gordian Knot.

There's an undercurrent of misery and pathos to the book - nothing ruins dinner like 200+ pages on fucked-out-of-the-gate early 1970's social welfare programs in a city you love. The historical framing is an EDA jobs program for the hardcore unemployed that sought to deliver funding to projects and businesses which would in turn employ economically disadvantaged Oakland residents. The late-1960's urgency behind the project stemmed from a desire to nip in the bud further urban race riots like those that had taken place in Watts and elsewhere. Oakland, home of the Black Panthers, was viewed as a potential trouble spot. Rapid flows of federal money aimed at helping the unemployed was identified as the solution. As you might expect from the title, things didn't turn out as planned: money and time were generally wasted, and few people received the promised help. The program fit the general pattern of the past 50 years: splashy introduction, front page news, energy and excitement at the outset, slow leakage of enthusiasm, and an eventual page 10 notice of cancellation several years later.

This is going to get wildly relevant in the coming years, especially in light of Obama's recently-alluded-to New New Deal: "We'll be working out the details in the weeks ahead, but it will be a two-year, nationwide effort to jumpstart job creation in America and lay the foundation for a strong and growing economy." Part of me is saying "uh oh", but a bigger, louder part of me is saying "hellz yeah, where do I sign up to help?"

The core question that authors Aaron Wildavsky and Aaron Pressman hope to answer is: why is the road to hell paved with good intentions? Is there a difference between policy and implementation that can be somehow bridged, or at least described more precisely? "Implementation" refers to that often-overlooked part of the project that happens after the ideas, funding, and excitement, but before any tangible results.

Page 87, on the complexity of joint action:

When we say that programs have failed, this suggests we are surprised. If we thought from the beginning that they were unlikely to be successful, their failure to achieve stated goals or to work at all would not cry out for any special explanation. If we believed that intense conflicts of interests were involved, if people who had to cooperate were expected to be at loggerheads, if necessary resources were far beyond those available, we might wonder rather more why the programs were attempted instead of expressing amazement at their shortcomings. The problem would dissolve, so to speak, in the statement of it.

I love this idea, and it slots in neatly with the commercial world's wisdom that successful companies create room for mistakes, if those mistakes can be used to gain experience and learn. From a project point of view, no one wants to be on the job that fails. From a societal point of view, failed experiments that are adequately described point the way toward eventual success.

Page 98, on mismatched means and ends:

When programs are not being implemented, it is tempting to conclude that the participants disagreed about the special ends they sought rather than the ordinary means for attaining them. Thinking about means and ends in isolation, however, imposes an artificial distinction, especially when more than one program is involved. One participant's ends, such as a training facility, may be another actor's means. Once innumerable programs are in operation, the stream of transactions among people who are simultaneously involved in them may evidence neither clear beginning nor end but only an ebb and flow. As the managers of each program try to impose their preferred sequence of events on the others, their priorities for the next step, which differs for each one and cannot be equally important to all, may conflict. The means loom larger all the time because they are what the action is about. Actually, it is easier to disagree about means because they are there to provoke quarrels, while ends are always around the corner.

The difference between success and failure seems to be the difference between turbulent and laminar flow. Each participant has the same end in mind, but one is relaxed while another is in a hurry, one wants to start here and another there. Even with all actors ostensibly moving in the same direction, turbulence and chaotic flow result from these seemingly-small differences in chosen velocity.

Page 113, on delay:

What had looked like a relatively simple, urgent, and direct program - involving one federal agency, one city, and a substantial and immediate funding commitment - eventually involved numerous diverse participants and a much longer series of decisions that was planned. None of the participants actually disagreed with the goal of providing jobs for minority unemployed, but their differing perspectives and senses of urgency made it difficult to translate broad substantive agreement into effective policy implementation. It was not merely the direction of their decisions - favorable or unfavorable - but the time orientation of the participants - fast or slow, urgent or indolent - that determined the prospects of completion. When so many future decisions depend on past actions, delay in time may be equivalent to defeat in substance.

Much of the process methodology behind Agile seems to recognize that priority-setting is the critical point for most friction: people must agree on what the next most important task is, and this is where most negotiation is designed to take place.

Page 133, on the need for bureaucracy:

If one wishes to assure a reasonable prospect of program implementation, he had better begin with a high probability that each every actor will cooperate. The purpose of bureaucracy is precisely to secure this degree of predictability. Many of its most criticized features, such as the requirement for multiple and advance clearances and standard operating procedures, serve to increase the ability of each participant to predict what the others will do and to smooth over differences. The costs of bureaucracy - a preference for procedure over purpose or seeking the lowest common denominator - may emerge in a different light when they are viewed as part of the price paid for predictability of agreement over time among diverse participants. The price may be too high, but the cost of accomplishing little or nothing otherwise must be placed against it.

Big, dumb bureaucracy has a lubricating effect here. Things take a long time because processes are designed to insulate actors from each others' instabilities. The computation metaphor that seems appropriate here is boundedness: CPU or I/O? What exactly are you waiting for at any given time, and how can project management help participants understand that some given task or responsibility is simply going to take a while, and maybe you should find something else to do?

Page 134, on coordination:

When one bureaucrat tells another to coordinate a policy, he means that it should be cleared with other official participants who have some stake in the matter. This is a way of sharing the blame in case things go wrong (each initial on the documents being another hostage against retribution) and of increasing the predictability of securing each agreement needed for further action. Since other actors cannot be coerced, their consent must be obtained. Bargaining must take place to reconcile the differences, with the result that the policy may be modified, even to the point of compromising its original purpose. Coordination in this sense is another word for consent.

Telling another person to coordinate, therefore, does not tell him what to do. He does not know whether to coerce or bargain, to exert power or secure consent. Here we have one aspect of an apparently desirable trait of antibureaucratic administration that covers up the very problems - conflict versus cooperation, coercion versus consent - its invocation is supposed to resolve.

Everyone wants coordination on his own terms.

This is the part where I criticize unilateral approaches like 37 Signals' Getting Real. The core tenets of Getting Real seem to essentially boil down to a pathological aversion to commitment: commitment to people ("small teams"), to goals ("flexible scope"), and to details ("ignore details", "it doesn't matter"). Generally speaking, people who believe this will have already put themselves in a position to live it: it's no accident that Stamen is seven people. The act of externalizing Getting Real makes it a process, one that's spectacularly bad at addressing coordination. Fine for small projects where everyone starts on roughly the same page, but disastrous for any situation where other actors need to give consent: managers, clients, investors, customers. The universe of Getting Real is a cramped, airless one populated by to-do list managers and communication software for tiny teams.

Where someone needs to be convinced, coerced, or seduced into cooperating with you, process gives way to sub-rational animal instinct.

Pages 165-166, on implementation-as-control:

In this view, for instance, implementers must know what they are supposed to do in order to be effective. Yet, "street level" bureaucrats are notorious for being too busy coping with their day-to-day problems to recite to themselves the policies they are supposed to apply. ... Writing about the administrative process in the regulatory commissions of the New Deal era, James Landis recalls how "one of the ablest administrators that it was my good fortune to know, I believe, never read, at least more than casually, the statutes that he translated into reality. He assumed that they gave him power to deal with the broad problems of an industry and, upon that understanding, he sought his own solutions."

The planning model recognizes that implementation may fail because the original plan was infeasible. But it does not recognize the important point that many, perhaps most, constraints remain hidden in the planning stage, and are only discovered in the implementation process.

This is what I think Agile seeks to address: the idea that requirements change because they flex and respond to previous requirements already met.

Pages 167-168, on implementation as interaction:

This view is strangely reminiscent of old syndicalist doctrines summarized in once-popular slogans like "The Railroads to the Railroadmen" and "The Mines to the Miners." The syndicalists' demand for "industrial democracy" actually concealed a view of production as an end in itself rather than a means of satisfying consumers' wants. We feel the emphasis on consensus, bargaining, and political maneuvering can easily lead (and has, in fact, led) to the conception that implementation is its own reward.

The interaction model of implementation carries interesting evolutionary overtones. The results are not predictable, an element of surprise is maintained, and the outcomes are likely to be different from those sought by any single participant.

This is where I think Agile falls apart: the manifesto promises to do away with process, but introduces process of its own. In particular, the process it introduces is fundamentally introspective, a kind of "Programming for the Programmers" frame of mind that seems to focus on the needs of the development team over the needs of the broader project. The outcomes are likely to have been bent or twisted somewhat along the way.

Page 215, on implementation as adaptation:

In a world of flux, it is only through continuous negotiation between administrators, implementers, and decision makers that any "congruence between program design and program implementation" (mentioned as essential in the literature) can ever be achieved.

"Adaptation" is Pressman and Wildavsky's final watchword for a useful view of implementation. It encodes ideas of flexibility, negotiation, while still leaving room for a deeper goal. This is not willy-nilly natural selection, but a process of constant self-evaluation. There's a lot more on this topic in a future post on Arthur Bentley's The Process Of Government.

Page 228, on learning from error:

In reaction to what is widely perceived as a dismal record, students of implementation, like the evaluators before them, have sought to guard themselves against failure. Instead of learning from error as it is occurring, they hope to prevent future failure before it takes place. Since there can be little learning without mistakes to learn from, however, the field of implementation is caught in a double bind: too much error suggests incompetence and too little inhibits learning.

Earlier tonight I briefly met Spencer Salazar from Smule, the makers of the iPhone Ocarina. They have a small suite of like Sonic Boom ("turns your phone into a virtual firecracker"), Sonic Vox ("the real-time voice shifter"), and Sonic Lighter ("Sonic Lighter is a lighter") that are mostly technology gimmicks. Spencer admitted as much but I'm still completely smitten with the fact that 75% of their applications have a simple globe view that uses the network features of the phone to show you what other people, all around the world, are doing with each app right now. You can hear other people's clumsy ocarina playing, watch little explosions when other people use Sonic Boom, and see who's using the lighter app with some sense of how those people are related to you based on flame-passing connections.

We've seen this all before, in Twittervision and other such globetrotting applications. These Smule globes seem strangely different and much more interesting, largely I think because you hold the phone in your hand instead of the laptop or monitor on your desk. It's a more personal, touched engagement with the screen that makes visualizing an earth-spanning army of phone lighters and flute blowers more physically personal. In particular, the Sonic Boom visualization is like watching television: no reading, no place names, just tiny explosions with audio all over the world with the same unmediated appearance as old top-down resource gathering games like War Craft I.

Having just read Teeming Void's Against Information (a critique of "data art"), I'm thinking about direct perception of data as a way of making it more visceral. The Golan Levin and Jonathan Harris pieces referenced in the paper all suffer from various forms of indirection: Levin makes breaking up look like math and physics, while Harris jumps to all sorts of crazy conclusions based on faulty language parsing and excessively abstract visual metaphor. How can a visual representation of data make itself felt right there, in your hand? Pictures help. Sound helps.

I've been expanding the georeferenced collection of Oakland maps that Gem and I started back in May. Recently, I purchased a 1967 Standard Oil map of Oakland for a few bucks from EBay. I was looking for late 1960's / early 1970's, because that's when the freeway structure here really started to take shape. Previously, we looked at a switch from rail to roads. Through the 50's and 60's, the switch was accelerated with the construction of massive highways through what had formerly been residential neighbhorhoods.

Particularly interesting is the Cypress Viaduct, a raised connection between highways 880, 580 and 80 running through West Oakland. When built, it was sharply criticized for splitting the neighborhood and further isolating it from downtown Oakland. The current site of the viaduct was where I made some of my first edits to OpenStreetMap. The structure was destroyed in the 1989 Loma Prieta earthquake shortly after my family moved to California, but on this map it's a fresh addition to the landscape:

The 19th anniversary of the quake was October 17th, one month ago.

The new 1967 map is a striking constrast to the previous 1952 map. The various freeways connected to Interstate 80 are one major difference, but the cartography is also a big contrast. This map is similar to the other Gousha-designed map from 1936 in its choice of bright colors, but it also features topographic shading up in the hills and orange highlights around freeway exits. A significant piece of infrastructure still under construction at this point is the 980 / 24 connector from downtown Oakland up into the hills toward the Caldecott Tunnel. The construction areas for the southern stretch are marked, while the northern route is still a whispy dotted line through miles of backyards.

Are you a talented developer interested in supporting our data visualization and mapping practice from the server end of things? Are you interested in a full-time gig at our San Francisco office?

You'll be working with a small team of designers and engineers who will be looking to you to make their ideas feasible. You're excited by the possibility of cutting and bending data to fit it through the thin straw of the internet. You can look at a source of information and model it as resources, rows and columns, messages and queues.

...

Our technology choices lean towards open source databases, unix-flavored operating systems, and scripting languages like Python and PHP. You'll be expected to know these things, and bring something new and unexpected besides.

Read the rest of the job description and let us know! (watch the spamarrest response on that e-mail address)

Earlier this morning, I returned from a four-day spin through Tokyo, my second visit there, to speak at Web Directions East. This trip was wholly unexpected, as I was pinch-hitting for Jeff Veen who had to cancel at the last minute and suggested me as a replacement. Fortunately it was possible to book a last-minute flight, take over a hotel reservation, and write an hour-long keynote talk in just a few days. It was an surprise honor to even be asked, and the experience was smooth sailing all the way, including the miniature hotel room.

I'm not going to post exact talk notes, but I outlined a general overview of data visualization and then focused on a range of Stamen's projects from the past four years and how they illustrate some deeper trends. Elements of this should be familiar to anyone who's heard me, Eric, Shawn or Tom speak before:

• Mark Newman's 2004 election cartograms. I learned from Karl that they'd been freshly updated to reflect last week's results.
• Examples of flow-management including 538 and John Allspaw's Flickr capacity-planning ideas.
• Our 2004 MoveOn Virtual Town Hall application, for the first time in what seemed like years. It would be amazing to revisit this project again somehow, possibly in a non-political context.
• Digg Labs, in the context of "liveness", from our Live/Vast/Deep iron triangle of visualization.
• Trulia Hindsight and Snapshot, as examples of "vastness".
• The hot, fresh, new SFMOMA ArtScope as an example of "deepness". This also served as a nice digression about map tiles, something I've been thinking and talking about a bunch.
• Oakland Crimespotting to round out some ideas about stable linking and data on the web. I also heavily name-checked Tom Coates's Native To A Web Of Data and Matt Biddulph's Designing Data For Reuse here.

A few people asked after the source of a particular background photo from my slides - it was an image of the Shanghai Urban Planning Museum's city model, possibly gaffled from blog.360dgrs.nl.

In addition to meeting the other speakers and organizers, John, Oli, Andy, Jeremy, Eric, Dan, and Doug, I also had a chance to take an excellent nighttime Tokyo bicycle and beer ride with Craig and Verena that made me wish SF and Oakland maintained the pavements a bit more effectively. I now desperately want to add a third bike to my collection, a mama charion granny bike.

It's been a while since I've done one of these, I'm a bit rusty. I started Cognition In The Wild over a full year ago, put it down for a while, and only recently came back to finish the book. The topic is cognitive activity, and how it plays out in social situations. Sort of a behaviorist tract in a way, which is interesting because the idea that everyone loved to hate (everyone in UC Berkeley CogSci department ten years ago that is) is starting to pop back up in some odd places in my life: this book, another one about politics from 1908, Obama's economic policies, etc.

Edwin Hutchins frames his story in the context of an observational trip aboard a U.S. Navy vessel, the Palau, and its crew of sailors and navigators. Hutchins particularly concerns himself with the way in which practice and instrumentation constitutes a meta-cognitive process above the level of the individual: the observations and computations that enable the crew of the ship to steer it are carried out by a collection of participants, some of them quite inexperienced, all of them performing small pieces of a bigger task. Together, they form a complete computational process, a sort of full adder made of half adders. He's particularly interested in the instrumentation that lets these guys do their jobs: slide rules, sighting scopes, variations on the protractor, and conventions surrounding verbal communication on the bridge and over the ship's intercom.

Hutchins's theory seems to be that these devices and practices act as a form of cognitive jig, embodying complex trigonometric and geometric processes in tangible form the way a slide rule converts multiplication into a simple linear movement. I've been interested in this idea before, via David Pye's Nature and Art of Workmanship. Pye argued that a lot of what we consider to be "hand work" is actually jigged and regulated through external forces. Pye calls unregulated workmanship "the workmanship of risk", and it's an interesting contrast to the kind of cognitive risk minimization that Hutchins is describing here. There's joy in navigation by dead reckoning as in risky, dextrous workmanship, but the U.S. Navy is having none of it and prefers its interpersonal procedures immaculately specified to the finest detail.

The place where I think this touches some of my recent interests in tiling and flows is that the purpose of a jig is to turn the latter into the former, to transform fluid into constrained motion. In particular I'm thinking about by most recent favorite general-purpose example, the use of "remaining days" as a transposed operations metric by Flickr's capacity planning guru John Allspaw.

Flickr takes one kind of motion, the consumption of storage space or saturation of network bandwidth, and transposes it into another kind of motion, the number of days they're free to sit on their hands until everything falls to pieces.

There's an extended example in Hutchins' book that's similar in spirit to this, and it forms the only coherent set of pages I bothered to dog-ear. As a counterpoint to Western-style navigation that places a moving boat in the context of a static ocean, he offers an in-depth analysis of Micronesian navigation practices that proceed along utterly different lines and yet still allow canoe navigators to travel between tiny islands out of sight of land without losing their bearings. The background to this alternate navigation frame is rote memorization of angular relationships among islands, but the surprising bit is the way it recontextualizes the navigator as a static center with respect to the sidereal compass, surrounding islands moving past him on parallel tracks to the left and the right.

These excerpts constitute my first donation to the Analogy Library. That Flickr capacity thing above is my second. Also worth a read is UPenn's Traditional Navigation in the Western Pacific: A Search for Pattern, written by 1989-2003 This Old House host Steve Thomas.

Page 66, a bit of context:

Without recourse to mechanical, electrical, or even magentic devices, the navigators of the Central Caroline Islands of Micronesia routinely embark on ocean voyages that take them several days out of the sight of land. Their technique seems at first glance to be inadequate for the job demanded of it, yet it consistently passes what Lewis has called "the stern test of landfall." ... Western researchers traveling with these people have found that at any time during the voyage the navigators can accurately indicate the bearings of the port of departure, the destination, and other islands off to the side of the course being steered, even though all of these may be over the horizon and out of sight. These navigators are also able tack upwind to an unseen island while keeping mental track of its changing bearing - a feat that is simply impossible for a Western navigator without instruments.

Page 67, on clues from what lies beneath:

The world of the navigator, however, contains more than a ser of tiny islands on an undifferentiated expanse of ocean. Deep below, the presence of submerged reefs changes the apparent color of the water. The surface of the sea undulates with swells born in distant weather systems, and the interaction of the swells with islands produces distinctive swell patterns in the vicinity of land. Above the sea surface are the winds and weather patterns which govern the fate of sailors. Seabirds abound, especially in the vicinity of land. Finally, at night, there are the stars. Here in the Central Pacific, away from pollution and artificial light, the stars shine brightly and in incredible numbers. All these elements in the navigator's world are sources of information.

Page 68, on the sidereal compass:

Seeing the night sky in terms of linear constellations is a simple representational artifice that converts the moving field of stars into a fixed frame of reference.

This seeing is not a passive perceptual process. Rather, it is the projection of external structure (the arrangement of stars in the heaves) and internal structure (the ability to identify the linear constellations) onto a single spatial image. In this superimposition of internal and external, elements of of the external structure are given culturally meaningful relationships to one another. The process is actively constructive.

Page 71, on picturing a frame of reference:

The fundamental conception in Caroline Island navigation is that a canoe on the course between islands is stationary and the islands move by the canoe. This is, of course, unlike our notion of the vessel moving between stationary islands. A passage from Gladwin (1970: 182) amplifies this:

Picture yourself on a Pulawat canoe at night. The weather is clear, the stars are out, but no land is in sight. The canoe is a familiar little world. Men sit about, talk, perhaps move around a little within their microcosm. On either side of the canoe, water streams past, a line of turbulence and bubbles merging into a wake and disappearing into the darkness. Overhead there are star, immovable, immutable. They swing in their paths across and out of the sky but invariably come up again in the same places. ... Everything passes by the little canoe - everything except the stars by night and the sun in the day.

Page 81, intersecting lines:

It is tempting to criticize the Caroline Island navigators for maintaining an egocentric perspective on the voyage when the global perspective of the chart seems so much more powerful. Before concluding that the Western view is superior, consider the following thought experiment: Go at dawn to a high place and point directly at the center of the rising sun. Return to the same high place at noon and point again to the center of the sun. That defines another line in space. I assert that the sun is located in space where those two lines cross. Does that seem wrong? Do you feel that the two lines cross where you stand and nowhere else?

...

Our everyday models of the sun's movement are exactly analogous to the Caroline Island navigator's conception of the location of the reference island. The choice of representations limits the sorts of inferences that make sense.

Page 92, on relative difficulty in frames of reference:

All navigation computations make use of frames of reference. The most prominent aspect of the Micronesian conception is the apparent motion of the etak island against the fixed backdrop of the star points defined by the sidereal compass. Here there are three elements to be related to one another: the vessel, the islands, and the directional frame. In order to preserve the the observed relationship of motion parallax, one can have the vessel and the directional frame move while the islands stay stationary (the Western solution) or one can have the vessel and the directional frame stationary while the islands move (the Micronesian solution). ... Each of these schemes makes some things easy to compute and other difficult.