Michal Migurski's notebook, listening post, and soapbox. Subscribe to this blog. Check out the rest of my site as well.

Apr 5, 2009 5:33pm

re: diverselessness

Noah Brier noticed a piece of research I linked the other day, on the end of online monoculture. Money quote:

While each customer on average experiences more unique products in Internet World, the recommender system generates a correlation among the customers. To use a geographical analogy, in Internet World the customers see further, but they are all looking out from the same tall hilltop. In Offline World individual customers are standing on different, lower, hilltops. They may not see as far individually, but more of the ground is visible to someone. In Internet World, a lot of the ground cannot be seen by anyone because they are all standing on the same big hilltop.

It's worded as a sharp critique of recommender systems, like the "people who bought" feature on Amazon that's everyone's example #1 of this stuff. I have something of a personal history here, having done some super-early, super-basic research on Digg's recommendation engine before Anton Kast arrived to make it real. One of the challenges that Anton observed is that recommenders can often drop you into a rut, and you have to add a lot of extra logic to work against your own algorithm. In Digg's case, they don't want your recommendations to include too many stories from a single person or small group of people, to maintain a degree of input diversity. I'm an avid user of Fffffound!, and they have a terrible recommendation engine - it routinely pulls you off into the weeds and thickets of a single prolific ffffinder's stream, and woe to you if that person who has great taste in architectural renderings also has a thing for soft porn.

Noah suggests that large-scale movements of this sort aren't well-suited for the web's focusing powers:

...everyone always uses the same few examples for every question about successful internet companies (Facebook, Twitter, Amazon, Netflix at the moment). This happens in every industry, actually, just ask someone to name the best brands and I'll give you $1 if Nike and Apple aren't on the list. But let's put brands aside for a second and focus on the web. The number of successful large scale communities out there is quite small. Now there's nothing wrong with that, but it's the reality of the situation. So maybe, just maybe, that's not what the web is built for. Maybe it's built for small scale communities. Maybe we need to reframe the way we think about this stuff and worry less about "scalability."

I'm not sure it's possible to even encourage smallness; people flock to a good thing and communities grow. Good communities have processes in place for handling change over time, clarify their terms and maintain order organically. I'm thinking in particular about Flickr's community manager, Heather Champ, and the specific guidelines she managed to get approved by legal:

Don't be creepy. You know the guy. Don't be that guy.

I think issues of power and governance are going to swiftly rise in importance on internet communities, as they expand to include more different kinds of people. It's interesting that some of the best, most resonant ideas on these topics that I've encountered over the years has come from political writers and may have been produced even before the internet.

Jo Freeman wrote in 1970 on the tyranny of a structureless group:

Elites are nothing more, and nothing less, than groups of friends who also happen to participate in the same political activities. They would probably maintain their friendship whether or not they were involved in political activities; they would probably be involved in political activities whether or not they maintained their friendships. It is the coincidence of these two phenomena which creates elites in any group and makes them so difficult to break.

There's a hint of the same-hilltop issue here: groups of people watch each other, learn from each other, and sometimes making a really, really tall hilltop for other people can be quite an advance. I'm thinking about the Invisible College or Simon Reynolds's hardcore continuum, where interesting and interested people self-organize into groups, communication costs drop through the floor, and a sudden massive spike in creative output is generated to some end. The Manhattan Project is another good example, just so it's clear that the end isn't always strictly peaceful.

Who decides where the hill goes? A recent conversation between Jay Rosen & Glenn Greenwald about Jim Webb talks about the nature of leadership like this:

...leaders, by talking about things, make them legitimate. Parties, by pushing for things, make them part of the sphere of debate. ... if the conversation of democracy is alive and if you make your leaders talk about things, it becomes valid to talk about them.

Leadership is the ability to direct or coalesce attention, to identify Schelling points. Leadership is a form of power, and there are two critical writings on the nature of group power that I think speak to this idea of hilltop choice. A recent Hoover Institute piece by Liam Julian explores the moral beliefs of philosopher Reinhold Niebuhr, who asserted that only the coalescing and wielding of group power could make change in the world: "They do not recognise that when collective power, whether in the form of imperialism or class domination, exploits weakness, it can never be dislodged unless power is raised against it." There are some excellent connections to Obama in that article, the way his organizational experience is based on recognition and routing of power:

As Obama told Lizza, "The key to creating successful organizations was making sure people's self-interest was met and not basing it on pie-in-the-sky idealism. So there were some basic principles that remained powerful then, and in fact I still believe in."

One of the things that really gets under my fingernails about Facebook is the way that the system seems designed to limit the potential for collective action among its participants: effective limits on the right to assemble and communicate mean that there's a sharp power imbalance in the system, which sounds like small potatoes until you remember that quite literally, everyone and their mother is on Facebook, uses the system to mediate their social interactions, and seems generally uninterested in the enclosing Panopticon. It's reassuring that changes to Facebook's terms of service provoke bursts of outrage that force the company to backtrack, but it would be a lot more reassuring if these changes were made in a public, cooperative fashion. Too much more of this and we'll see a real, online storming of the Bastille.

Another, earlier political philosopher relevant here is Arthur F. Bentley, whose 1906 book The Process of Government carefully lays out the dynamics of group interests, in his opinion the only kinds of interest relevant to the study of governance:

We shall find as we go on that even in the most deliberative acts of heads of governments, what is done can be fully stated in terms of the social activity that passes through, or is reflected, or represented, or mediated in those high officials, much more fully than by their alleged mental states as such. Mark Twain tells of a question he put to General Grant: "With whom originated the idea of the march to the sea? Was it Grant's or was it Sherman's idea?" and of Grant's reply: "Neither of us originated the idea of Sherman's march to the sea. The enemy did it;" an answer which points solidly to the social context, always in individuals, but never to be stated adequately in terms of individuals.

It's significant, to me, that President Obama cut his teeth as a community organizer. Partially I think it means that we're on the cusp of a new self-recognition of group power, because we have a laboratory in which to test it. Where previously we might have been exposed just a few forms of government (federal, state, city), now we are daily and simultaneously subjected to literally dozens: Facebook, Metafilter, Flickr, Delicious, Digg, Slashdot, Twitter, Google, etc. Each of these is in something of a formative, benevolent dictator stage, but the more mature ones have developed infrastructure for self-analysis. Metafilter has the gray and a small team of known admins, Flickr has Heather Champ, Slashdot has a reputation-based moderation and meta-moderation system.

Getting back to the recommendation engine business, I think we're going to see two things happen in the not-so-distant future. One, community participants will have a much clearer understanding of the particular hilltop they're standing on, because the language of recommendation and similarity will enter the mainstream discussion. Two, recommendation engines will mature to the point that their features and techniques will be understood and debated by non-technical (though educated) subjects, much as participatory government is today.

Comments (3)

  1. We I built the recommendation for systems for eMusic, BigPond DVDs and Verizon we totally found the issue you mention of hill topping, whether it was similar users or music by the same artist etc. Over time we developed many approaches to minimise the effect for clients but we always had some tweaking to do with new clients and their data. Especially at eMusic, which had a GREAT set of hardcore users, we would get excellent feedback for tweaks to algorithm or problematic relationships, so I can see that a very small select set of interested and engaged users will contribute and assist with the recs systems. That said, there is still great possibility for both serendipity and more importantly I would argue greater ability for discovery with recs systems rather than customer overwhelm. One great example we had was of linking The Smiths to a band called the New York Dolls. This caused quite a lot of discussion in the forums that it was incorrect until someone mentioned that Morrissey was the president of their fan club.

    Posted by Ben Hosken on Sunday, April 5 2009 9:11pm UTC

  2. First off, great post. Second, just want to clarify the comment about scalability - I totally agree that keeping communities small is a near impossibility. This is a problem everywhere (bars are my personal favorite example). However, I do think it's important to note that the majority of thriving communities on the web aren't the big ones.

    Posted by Noah Brier on Monday, April 6 2009 1:17am UTC

  3. The most missing feature for me of all content Web sites which have a the top 10 most something is precisely a filter that gives me the possibility to filter that out. Show me new stuff, except the stuff which is in the 20% top most recommended (with the percentage adjustable.) A bit like spam where you filter out 90% of the most popular emails ;)

    Posted by karl on Monday, April 6 2009 2:41am UTC

Sorry, no new comments on old posts.

July 2024
Su M Tu W Th F Sa

Recent Entries

  1. Mapping Remote Roads with OpenStreetMap, RapiD, and QGIS
  2. How It’s Made: A PlanScore Predictive Model for Partisan Elections
  3. Micromobility Data Policies: A Survey of City Needs
  4. Open Precinct Data
  5. Scoring Pennsylvania
  6. Coming To A Street Near You: Help Remix Create a New Tool for Street Designers
  7. planscore: a project to score gerrymandered district plans
  8. blog all dog-eared pages: human transit
  9. the levity of serverlessness
  10. three open data projects: openstreetmap, openaddresses, and who’s on first
  11. building up redistricting data for North Carolina
  12. district plans by the hundredweight
  13. baby steps towards measuring the efficiency gap
  14. things I’ve recently learned about legislative redistricting
  15. oh no
  16. landsat satellite imagery is easy to use
  17. openstreetmap: robots, crisis, and craft mappers
  18. quoted in the news
  19. dockering address data
  20. blog all dog-eared pages: the best and the brightest