tecznotes

I went to hear Michael Pollan (author, Omnivore's Dilemma) and John Mackey (CEO, Whole Foods) converse on the Zellerbach stage in Berkeley tonight. Peter has more notes. One thing really struck me. Actually, two things really struck me, but I don't want to talk about the fucked-up video depicting graphic abuse of animals at dairy, poultry, and hog farms that was part of Mackey's talk.

Pollan brought up a service offered in selected Danish supermarkets:

Packages of meat and poultry carry a bar code that, when scanned by a machine in the store, calls up pictures of the farm where the animal was raised, as well as information about its diet, living conditions, the date of its slaughter and so on. Imagine how quickly this sort of transparency would force a revolution in our food chain. (Produce Politics, scroll down)

Two years ago, Patrick told us about a service offered in Japan, where your store-bought bananas come with a code printed on the label that can be used to find a web page featuring the grower of that particular banana, and (I think) some details on when it was grown and where it came from.

It's too late to make an informed buying decision about a banana when you're back near a computer, and installing special reader hardware in supermarkets is expensive. Both ideas could be easily handled by cameraphone-readable QR codes like this one:

They're big in Japan, not so much in Europe or here. It'd be wild if an upcoming generation of mid-range mobile phones supported these dumb things. I'm told they're easy, and Aaron has been doing some smart stuff with codes and little folded-up paper printouts. They'd be an everyday vehicle for the kind of environmental information overlay that would make Pollan and Mackey's desire for food industry transparency remotely possible - where else would you want to make the decision, than right in the store when you're staring at two kinds of stilton trying to guess which cow suffered less?

I returned from Japan yesterday, where I spent a week hanging out with Boris, helping with part of the redesign effort for Global Voices Online.

I didn't take a camera with me, so this is a collection of photos from Flickr that are better than any I would have taken.

My first stop off the train from the airport was Shibuya (渋谷) Station, where I was supposed to meet Boris. I mistakenly expected the European experience of meeting on the platform, so I spent about 40 minutes pacing around as wave after wave of people finishing work boarded their evening trains. The platform was constantly this crowded, with a fresh rotation of passengers every five minutes or so:

(photo by halonfury)

Eventually, I figured out that a ticket was required to get to the platform in the first place, so I used my piddling 50¥ to call and make everything okay. Next time, better arrival preparation.

I stayed at Joilab, in Jiyugaoka, an apparently ritzy part of town with extremely narrow streets.

(photo by shibo)

Bananas often come individually wrapped. I saw my first $4 apple.

(photo by A is for Angie)

Adam warned a bunch of people I was coming, but I got to meet them anyway. This is Craig and Chris, of AQ:

(photo by cpalmieri)

Both were wearing normal shirts when I met them. Chris wrote the excellent guide to mapmaking for Tokyo I linked the other day.

AQ's office is in Co-Lab, a shared office space where designers of various sorts rent tiny cubicles and ethernet connections. It's partially a rent-reduction move, but also a way to squeeze with like-minded people. The space houses AQ's other project, Tokyo Art Beat. Paul showed me their use of QR codes on venue pages, which is interesting. This is probably old hat to anyone with any experience of mobile phones outside the US, but the idea of a commonly-readable 2D barcode for transmitting snippets of information via stickers or posters is just great.

Weekend was all work-work-work and food-food-food, including these oysters:

(photo by bopuc)

Then we took a quick trip up Mori Tower. The view from the top of this beast is just insane, and especially worthwhile when your visit can straddle the sunset. We didn't see Fuji, but it was still quite a panorama.

(photo by Koninho)

(photo by Urban|nexus)

(photo by /\ltus)

While in Tokyo, I kept seeing bikes like this one around town. They're made by Muji, a sort of Japanese general store that sells well-designed stuff ranging from ice cream to stationery to bicycles. The frame design with single horizontal post is great, everyone rides them around town, and I almost bought a new one for $150 but decided it was silly to drag a bike home.

(photo by ihateanarchists)

My other shopping experience while there was Tokyu Hands, a "Creative Life Store" that sells an unclassifiable range of everything. The Shinjiku branch I visited was about eight floors of stuff. I bought some Tabi that ended up not fitting very well.

(photo by antimega)

Now I am home, it's late, and I'm mildly jetlagged.

We're hiring again, this time for a visual designer:

This is a hands-on web design position. You'll be working with a team of designer/engineers who will be looking to you to help make their ideas sing. You should be the kind of person who can rework and iterate an idea until it's perfect.

Let us know if you think you fit the description.

-

Also, I'm typing this from Japan. Woot!

My iMac spent the weekend tearing through NASA's Blue Marble satellite imagery set, projecting the 500m/pixel base map to Mercator, chopping it up into tiles, and uploading it to S3.

Here's the root-level tile:

I took care to ensure that the slices match those in use by Google, Yahoo!, Microsoft, and OSM for cross-compatibility among any code that uses tiles from those providers.

The general procedure took three steps:

1. After downloading the full-resolution imagery from NASA and converting the PNG's to VIPS .v format, I extracted tiles at the highest zoom level to individual 256x256 images. This step had to include the mercator projection, which meant extracting squat rectangles near the poles and stretching them vertically to 256 pixels. I used VIPS for this step because its random-access file format is specially designed to support fast region selection from very large bitmaps (eight 21600x21600 pixel images).
2. For each lower zoom level, groups of four tiles were combined with ImageMagick. This turned out to be a very basic recursive function that spit out a series of ImageMagick commands as text. God bless sh.
3. Finally, every tile was uploaded to S3 and given a standard name: {zoom level}-r{row number}-c{column number}.jpg. The numbering scheme should be familiar to anyone who's worked with tile providers or Zoomify before.

S3's cheap bandwidth and the public domain status of these images means that you should feel free to use them in your own applications. If you plan to do anything unusually demanding with them, please go easy on my wallet and take a moment to sign up for an AWS account and use the following Python script to copy the complete tileset to a fresh S3 bucket of your own:

DuplicateBlueMarble.tar.gz (4KB)

Scott Karp writes:

If I were YouTube, I'd think long and hard about a business model based on cats flushing toilets and flatulence flambe. Anyone with any kind of professional interest in their video content will soon realize that YouTube's platform is increasingly a comodity, and that if your content is 1) really good, and 2) embedable, you're pretty much good to go, regardless of which platform you use.

Amazon's S3 (simple storage service) has been amazing for me, and I think that two of the three things YouTube and other video services provide (embedding, hosting, but not socializing) could be covered by S3 and a cheap conversion and uploading application. Local software would reformat video files to .flv, upload them to the user's specified S3 account along with the necessary .swf file for playing them, and spit out the HTML required to embed the video on your site. This feels like a week's worth of late night programming to anyone even marginally accomplished in Windows or Mac development. Amazon's web services account signup is probably not streamlined enough for this to be a widespread thing, largely because they're still thinking of it as a tool for developers, rather than public storage for the internet.

Shawn is a dedicated user of last.fm, and he's the one usually in control of the speakers here. Last.fm makes it easy to shoulder-surf what he's listening to:

I'm really in love with their leaderboard design, notably the pale gray arrows that show chart movement from week to week. The screenshot above shows that between now and last week, Of Montreal and The Flaming Lips have exhanged #1 and #2 spots, while Broadcast has jumped 14 spots to #4. The charts are just HTML, but every piece of information is linked up, and the display as a whole is quite pretty.

A recent message to the Yahoo JSON user group got me thinking about social norms as programming techniques. The question was about iterations over long sets of data:

I am trying to iterate through a big JSON variable (about 1500 nodes). It works but FF pops up with the message saying the script is not responding (A script on this page may be busy... do you want to stop the script, debug, continue). If I select continue, it works fine. It is just that the iteration takes a bit of time to go through all the nodes. Is there a way to avoid the above? (Carl Harroch)

There's a handy technique I picked up from Twisted Python that I like to call polite loops. Polite, because sharing resources in an environment like a browser takes tact. When you're at a party with an open bar, you don't hog the bartender's attention, downing pint after pint while he's force to wait on you to have your fill. You're polite; you get your Newkie Brown and walk away so someone else can order.

I've found a similar approach especially useful in languages typically used in web browsers, like Actionscript and Javascript. A greedy loop in a typical web browser affects not only the page it's on, but any other page open at the same time. There are usually allowances for this, such as messages that offer you the ability to kill a script in progress. These only appear after a prolonged period of uselessness, so relying on them is a bad idea.

Twisted's built-in event loop, the Reactor, is single-threaded but built to handle asynchronous events such as network traffic or user input. It does this by encouraging finely-grained function calls that hand control back to the Reactor frequently. For example, this recursive function:

def perturbItems(items):
    if len(items):
        item.pop().perturb()
        perturbItems(items)

...might be rewritten like this:

def perturbItemsPolitely(items):
    if len(items):
        item.pop().perturb()
        Reactor.callLater(0, perturbItemsPolitely, items)

It continues to do exactly the same thing: perturbItems() takes the last item, perturbs it (perturbation might be a lengthy and expensive procedure), then passes the remainder of the list back recursively. Instead of calling itself directly, it schedules a call with the Reactor so that other pending or more urgent tasks can be handled in the meantime. The Javascript analogues could look like this:

function perturbItems(items)
{
    if(items.length) {
        items.pop().perturb();
        perturbItems(items);
    }
}

...and the polite version:

function perturbItemsPolitely(items)
{
    var perturbItem = function()
    {
        if(items.length) {
            items.pop().perturb();
            window.setTimeout(perturbItem, 100);
        }
    }
    perturbItem();
}

This function sets up a closure and uses the Javascript window.setTimeout() method to introduce a 100 msec delay between calls. The short delay greatly increases the amount of time the function needs to run, but is also enough of a breather to allow other scripts or user input to have an effect. It prevents the "this script is running slowly..." issue.

Another way to write the same function does it by chunks:

function perturbItemsPolitely(items)
{
    var perturbChunk = function()
    {
        var start = new Date();
        while(items.length) {
            items.pop().perturb();
            if(start.getTime() + 100 < new Date().getTime()) {
                window.setTimeout(perturbChunk, 100);
                break;
            }
        }
    }
    perturbChunk();
}

Same function, same closure. This time, the function perturbs as many items as it can (note the while loop) before reaching 100 msec and introducing a 100 msec rest. Best case, it's no slower than the original recursive loop. Worst case, it takes twice as long with plenty of rest stops. Going back to the original post above, I'd solve the problem by introducing two polite loops: one to request the 1,500 element data set in chunks of a hundred, and a second to iterate over those hundred-element chunks with occasional rest stops.

Ever since visiting BMW's design strategy group in Germany two years ago, I've been thinking about switching to a standing desk (that's a photo of former Secretary of Defense Donald Rumsfeld using his!). I've also more recently been suffering from sciatica, which makes sitting for any length of time kind of an ordeal.

So, my friend Bryan helped me design a new desk that would fit on a typical 30-inch tabletop, let me type at a natural height, and place my computer monitor at eye-height. We decided on maple hardwood legs and a Richlite surface, and I've been using it here for the past two weeks. He's an enormously talented carpenter and cabinetmaker, and the finished piece feels solid and indestructible.

I couldn't be happier with my new non-seating arrangement. It helps my leg feel better, but it's also a more comfortable work surface overall. Two things not visible in these photos make it workable: An anti-fatigue mat on the floor keeps my feet from hurting (though they do, a little), and Synergy on the two computers lets me use the one keyboard & mouse to control the laptop.

Did a little blog-housekeeping this weekend. Pair hasn't been particularly CGI-friendly for several months now (shared environment, limits on memory usage by scripts) so I translated the parts of Blosxom I was interested in to PHP, and moved away from Perl permanently. The only change you should see is that single-digit dates (like today, February 4) are now displayed without the leading zero, thanks to PHP's excellent date support. I cleaned up the RSS and Atom outputs a bit, and switched to Atom 1.0 as the default feed output. I have no plans to ditch the RSS feed, I just think that Atom's syntax is more aesthetically pleasing.

All links should continue to work as before, but obviously please send me a mail if you see any problems or find broken links.

Help me understand why OpenID is worth paying attention to. Smart, respectable alpha-geek Simon Willison has focused on pretty much nothing else for the past few weeks, but I can't picture why this is an interesting or desirable technology. It's being touted as an example of "light-weight identity", an adjective I don't think should be applied to any proposed standard suggesting an entirely new kind of identifier. I'm still having trouble with the difference between URL and URI, so what is an XRI for? The protocol may be "drop dead simple", but this sounds like a deal-breaker for any non-wizard looking to understand how it works:

Now, in practice there will probably be concerns about spoofing, so Consuming sites will have whitelists, and which is why you may need multiple Providers to ensure they have one that works everywhere they need it.

My gut feeling is that OpenID is this year's architecture aeronautics moon shot. This ZDNet article makes the case for OpenID, but it lists only three proposed benefits for the "internet user", the only actor whose opinion actually matters in the long haul. Personally, all of the suggested OpenID uses (starting with: "I don't want to remember a long list of usernames and passwords for every site I visit") I've seen are already handled elegantly by KeyChain, Mac OSX's client-side password and secret storage program. "Identity" is itself a fairly abstract concept - I suspect that most people think of it in concrete terms, using wallet-compatible tokens like their drivers license or gym membership card as stand-ins. For internet stuff, my token is my laptop, under my personal control at all times with copies of all those usernames and passwords. If the iPhone spurs US carriers to open up WiFi or bluetooth on phones, my token might be my cell phone. Either way, it's going to stay a physical object with predictable real-world properties.

I haven't been neglecting my Oakland Crime work, just quietly plugging away at it in my off hours. I'm getting to a point where I can start to think about visual presentation, which I'm hoping will be an improvement on the traditional pins-in-a-map GMaps mashup style. I'm moving on two fronts: designing a possible display metaphor for the crimes, and writing software so I can do this in Flash and continue to take advantage of the highly-available, highly-detailed map tile collections from Google, Yahoo!, and Microsoft.

First, interface display. Crime leaves an imprint on the place and time it is committed, and a visual representation should reflect this. I've been thinking about this in terms of residual hauntings:

A residual haunting is a playback of a past event. The apparitions involved are not spirits, they are "recordings" of the event. Video and audio tapes capture sounds and images on a film of special material that has been oxidized or rusted. Certain building materials, such as slate used in older castles and stone structures and iron nails used in many older buildings, have properties similar to that of the tapes. When a traumatic event occurs or a time of heightened emotions, these materials record the event for future playback.

Any personal experience in a crime-prone neighborhood underscores this characteristic: the feeling of fragile safety, heightened alertness, and sensitivity to rumor or hearsay. There are places in Oakland and San Francisco where few non-residents dare to visit. These tend to be on the wrong side of the freeway, and even the brave or foolhardy artists and warehouse hipsters from "outside" who move there do so in packs, hiding inside secured buildings and staying separate from the local street life. The aura of street crime in these areas is strong, and daily movements feel like travels through hostile territory. Entire cities can take on the coloration of their most dangerous parts, leading to generalizations about Oakland being an inherently dangerous place. Sometimes, gentrification may set in and dull the impact of past crime.

In thinking about how to represent these halos on a map, my first idea was to calculate a linear relationship between space and time, and model them as spheres. For example, a mugging or burglary may impact the neighborhood for a few blocks in each direction and linger in memory for a few days. Car break-ins and petty theft can go entirely unnoticed, while homicides make the evening news in neighboring towns. It should be possible to establish a correspondence like "one block equals one day", and assign various influence radii to classes of crime.

This has a direct usability benefit: when selecting a time range to view on a visual map of crime incidents, Crime Watch shows only the crimes that occurred within that span. With spherical halos around each event sized proportionately to their severity, it should be possible to display pins on a map along with some indication that major events may have happened near the selected range, providing a backplane of additional information. Overlapping spheres would give a dynamic, shifting sense of how dangerous a given area is and help in time navigation by hinting at incidents just beyond the selected parameters.

Crime has a social weight, which should be represented by the effect it has on its spatial and temporal neighborhood. Dan Catt brought up this same concept as it relates to geo-aware mobile phone apps and Flickr's new machine tags feature, suggesting an application that gives your handset a light buzz when you pass over a place where some emotional event took place. I fully expect Aaron to have something like already running on one of his Python-and-GPS-enabled Nokias.

Adam clued me in to a slightly different way of thinking about this problem:

On top is my original, spherical zone of influence. On the bottom is a set of radiating cones suggested by a few of his comments. Each event still has a mass, but in the sense of general relativity rather than campfire ghost stories.

Adam:

Think of the "light cone" of a murder spreading ... it should be additive. Two murders on my block is more than twice as bad as one, because it tips my block into another region of possibility space.

The difference leads to the second picture, where the crime remains a localized event as it happens, but sends ripples through time and space. The forward influence is obvious, the backward influence less so. I think that both directions are necessary for two reasons: just one wouldn't have the interface navigational advantage of seeing around the horizon in each direction, and there are often social preconditions that lead to disturbing events happening in a given place and time. Jane Jacobs likes to talk about "border vacuums" in urban spaces, places such as open plazas that normal people avoid because the emptiness feels intuitively dangerous.

"Violence is a force acting on a place" (Adam again).

Visually, the radiating circles would need to be outlined rather than filled, to suggest the pond ripple connection and minimize clutter. Scrubbing time in one direction or another would expand and contract these ripples, hinting at the direction of their cause. The open question I have is about limits: spheres have an obvious limit, their radius, but cones do not. Do lower-intensity crimes simply have a shorter cone with an identical slope?

The second thing I've been hammering away at is a Flash-based, open source tile display engine for maps or large images. My friend Darren has been plotting a map-based application that would need to run on mobile phones and I want to do sparkly vector graphic overlays, so working with him to target Flash and FlashLite was a no-brainer. Expect to see this sometime in the coming weeks, since we're nearing completion.