tecznotes

Noah Brier noticed a piece of research I linked the other day, on the end of online monoculture. Money quote:

While each customer on average experiences more unique products in Internet World, the recommender system generates a correlation among the customers. To use a geographical analogy, in Internet World the customers see further, but they are all looking out from the same tall hilltop. In Offline World individual customers are standing on different, lower, hilltops. They may not see as far individually, but more of the ground is visible to someone. In Internet World, a lot of the ground cannot be seen by anyone because they are all standing on the same big hilltop.

It's worded as a sharp critique of recommender systems, like the "people who bought" feature on Amazon that's everyone's example #1 of this stuff. I have something of a personal history here, having done some super-early, super-basic research on Digg's recommendation engine before Anton Kast arrived to make it real. One of the challenges that Anton observed is that recommenders can often drop you into a rut, and you have to add a lot of extra logic to work against your own algorithm. In Digg's case, they don't want your recommendations to include too many stories from a single person or small group of people, to maintain a degree of input diversity. I'm an avid user of Fffffound!, and they have a terrible recommendation engine - it routinely pulls you off into the weeds and thickets of a single prolific ffffinder's stream, and woe to you if that person who has great taste in architectural renderings also has a thing for soft porn.

Noah suggests that large-scale movements of this sort aren't well-suited for the web's focusing powers:

...everyone always uses the same few examples for every question about successful internet companies (Facebook, Twitter, Amazon, Netflix at the moment). This happens in every industry, actually, just ask someone to name the best brands and I'll give you $1 if Nike and Apple aren't on the list. But let's put brands aside for a second and focus on the web. The number of successful large scale communities out there is quite small. Now there's nothing wrong with that, but it's the reality of the situation. So maybe, just maybe, that's not what the web is built for. Maybe it's built for small scale communities. Maybe we need to reframe the way we think about this stuff and worry less about "scalability."

I'm not sure it's possible to even encourage smallness; people flock to a good thing and communities grow. Good communities have processes in place for handling change over time, clarify their terms and maintain order organically. I'm thinking in particular about Flickr's community manager, Heather Champ, and the specific guidelines she managed to get approved by legal:

Don't be creepy. You know the guy. Don't be that guy.

I think issues of power and governance are going to swiftly rise in importance on internet communities, as they expand to include more different kinds of people. It's interesting that some of the best, most resonant ideas on these topics that I've encountered over the years has come from political writers and may have been produced even before the internet.

Jo Freeman wrote in 1970 on the tyranny of a structureless group:

Elites are nothing more, and nothing less, than groups of friends who also happen to participate in the same political activities. They would probably maintain their friendship whether or not they were involved in political activities; they would probably be involved in political activities whether or not they maintained their friendships. It is the coincidence of these two phenomena which creates elites in any group and makes them so difficult to break.

There's a hint of the same-hilltop issue here: groups of people watch each other, learn from each other, and sometimes making a really, really tall hilltop for other people can be quite an advance. I'm thinking about the Invisible College or Simon Reynolds's hardcore continuum, where interesting and interested people self-organize into groups, communication costs drop through the floor, and a sudden massive spike in creative output is generated to some end. The Manhattan Project is another good example, just so it's clear that the end isn't always strictly peaceful.

Who decides where the hill goes? A recent conversation between Jay Rosen & Glenn Greenwald about Jim Webb talks about the nature of leadership like this:

...leaders, by talking about things, make them legitimate. Parties, by pushing for things, make them part of the sphere of debate. ... if the conversation of democracy is alive and if you make your leaders talk about things, it becomes valid to talk about them.

Leadership is the ability to direct or coalesce attention, to identify Schelling points. Leadership is a form of power, and there are two critical writings on the nature of group power that I think speak to this idea of hilltop choice. A recent Hoover Institute piece by Liam Julian explores the moral beliefs of philosopher Reinhold Niebuhr, who asserted that only the coalescing and wielding of group power could make change in the world: "They do not recognise that when collective power, whether in the form of imperialism or class domination, exploits weakness, it can never be dislodged unless power is raised against it." There are some excellent connections to Obama in that article, the way his organizational experience is based on recognition and routing of power:

As Obama told Lizza, "The key to creating successful organizations was making sure people's self-interest was met and not basing it on pie-in-the-sky idealism. So there were some basic principles that remained powerful then, and in fact I still believe in."

One of the things that really gets under my fingernails about Facebook is the way that the system seems designed to limit the potential for collective action among its participants: effective limits on the right to assemble and communicate mean that there's a sharp power imbalance in the system, which sounds like small potatoes until you remember that quite literally, everyone and their mother is on Facebook, uses the system to mediate their social interactions, and seems generally uninterested in the enclosing Panopticon. It's reassuring that changes to Facebook's terms of service provoke bursts of outrage that force the company to backtrack, but it would be a lot more reassuring if these changes were made in a public, cooperative fashion. Too much more of this and we'll see a real, online storming of the Bastille.

Another, earlier political philosopher relevant here is Arthur F. Bentley, whose 1906 book The Process of Government carefully lays out the dynamics of group interests, in his opinion the only kinds of interest relevant to the study of governance:

We shall find as we go on that even in the most deliberative acts of heads of governments, what is done can be fully stated in terms of the social activity that passes through, or is reflected, or represented, or mediated in those high officials, much more fully than by their alleged mental states as such. Mark Twain tells of a question he put to General Grant: "With whom originated the idea of the march to the sea? Was it Grant's or was it Sherman's idea?" and of Grant's reply: "Neither of us originated the idea of Sherman's march to the sea. The enemy did it;" an answer which points solidly to the social context, always in individuals, but never to be stated adequately in terms of individuals.

It's significant, to me, that President Obama cut his teeth as a community organizer. Partially I think it means that we're on the cusp of a new self-recognition of group power, because we have a laboratory in which to test it. Where previously we might have been exposed just a few forms of government (federal, state, city), now we are daily and simultaneously subjected to literally dozens: Facebook, Metafilter, Flickr, Delicious, Digg, Slashdot, Twitter, Google, etc. Each of these is in something of a formative, benevolent dictator stage, but the more mature ones have developed infrastructure for self-analysis. Metafilter has the gray and a small team of known admins, Flickr has Heather Champ, Slashdot has a reputation-based moderation and meta-moderation system.

Getting back to the recommendation engine business, I think we're going to see two things happen in the not-so-distant future. One, community participants will have a much clearer understanding of the particular hilltop they're standing on, because the language of recommendation and similarity will enter the mainstream discussion. Two, recommendation engines will mature to the point that their features and techniques will be understood and debated by non-technical (though educated) subjects, much as participatory government is today.