tecznotes

Right, so I feel totally swamped right now, mostly by e-mails and a general feeling of not enough time in the day.

Two big interesting things have happened in the past week: Tom and I went to O'Reilly's FOO Camp in Sebastopol, an invitation-only hootenany attended by a variety of nerds. Among other talks and sessions, Kevin Slavin gave an understated, epic rundown of Area/code's relationship to that one meme about how kids don't roam nearly as far from home as they used. Kevin neatly tied up a bunch of threads about location, technology, television, and media, and my life is the richer for it.

The other thing is that Pownce launched. Our own Shawn Allen built the Adobe AIR desktop client for this messaging application, and large chunks of the project were conceived and perfected in our office. I've been close to the work and participated in a number of API design discussions. There's a bunch of noise about how it's like-Twitter-this, and isn't-it-just-email-that, but it's a stake in the ground, fun to use, and has a bright future.

Scott McCloud promised that the web was going to make distribution of lone-creator comics work, and he was right. The way to keep up with these things is via RSS feeds, and the artists generally seem to understand I've never in my life paid attention to comics, but now there are a few that I check up on regularly.

Wondermark is David Malki's bi-weekly (Tuesdays and Fridays) strip with artwork yanked from old-timey expired copyright illustrations, funny captions appended.

Perry Bible Fellowship is sometimes a bit raw, but Nicholas Gurewitch also uses it to experiment with a variety of drawing styles.

Cat And Girl is a long-running series featuring snarky literary references. I own an apparently-no-longer-in-print CnG Rube Goldberg t-shirt.

Penny Arcade is mostly about video games. I don't really play any games, but the strip is consistently awesome. I wish they'd fix their stupid RSS feed.

Wonderella is a comic book hero spoof, a little bit Cathy but generally funny.

I don't fully understand the cast of characters from Achewood, perhaps this is okay.

Also, the New Yorker Cartoon Anti-Caption Contest doesn't fully count as a web comic, but it's close. Each week the real drawings from the New Yorker cartoon caption contest are posted, and readers are encouraged to submit aggressively unfunny captions.

We've been thinking a lot about authentication recently, both as consumers and designers of web API's. Although certain best practices in this area are being solidified, I still think it's a wide-open field for experimentation. This post is a run-down of various patterns we've encountered for authenticating applications and users, and has been greatly helped along by conversations with Shawn, Steve, Matt, and others.

Keys

The simplest application authentication method is the developer key. Flickr has been using these since day one, and they mostly help in monitoring usage. Generally, the idea is that a site issues a unique key to each application consuming the interface, and then requires that this key be passed along with every request. Keys are not expected to remain secret or be subject to rigorous control, but they do help Flickr keep tabs on how applications use the API, and provide a way to find someone to blame when requests with a given key cause problems. We used to routinely get mails from Stewart about Mappr's (ab)use of expensive search parameters.

Flickr's API keys are explicitly connected to Flickr accounts, and are issued via an application form that asks for a description of your intended use and a promise to abide by the terms of use. There's also a monitoring page that displays your own API usage:

When we designed the Digg API, it was decided that key enforcement was not a high-enough priority to warrant the overhead of administration, so we went with a simple form of consensual disclosure. Digg application keys must be provided, must be in the form of a valid absolute URI, and should point to a page that describes the application. The URI isn't checked for normal usage, so it's possible to experiment and play with the API with minimal hassle.

Tracking keys is enough of a hassle that companies like Mashery have popped to provide this as a service.

Usernames, Passwords

Authenticating individual users is more sensitive, especially when an API provides read/write methods for posting new information to a user's account. The easiest way to authenticate this is to require that a user's account name and password be attached to requests.

The original Del.icio.us API required HTTP basic authentication for all methods, including the ones that returned information available on public, anonymous web pages in the application. Basic auth is well-understood and reasonable well-supported, so this made it quite easy to write tools that used the API. The major drawback of this method is that account passwords can be sniffed on every request, making them wildly insecure. At some point last year, Del.icio.us began requiring that all API requests be done over HTTPS. This solves the problem of password exposure, but introduces a new problem: HTTPS is a considerable resource hog, and is expensive to serve. Cal estimates that the cryptographic overhead of HTTPS can cut a web server's performance by 90%. It is useful for HTTPS to keep the contents of an interaction secret where the data is sensitive, as with banks and medical records, but it's total overkill in the case of a typical web API.

A more subtle problem with asking for usernames and passwords is the inherent phishing risk. An API that can be operated with a user's permanent password is a magnet for potential abuse, because something you know might also be something someone else knows. Flickr's early approach to this problem was to ask for the user's e-mail address in the request, not their Flickr username. A sniffed API password would be useless for logging into the main website, and knowing a username and password wouldn't get you into the read/write API.

Digests

One way to deal with the risk of password exposure without touching HTTPS is digest authentication. This is a pattern that uses one-way hashing functions such as MD5 or SHA to hide a password in transit, while still allowing it to be verified by the API server. Generally, an API client will send the server a hashed combination of username, password, and possibly other details. The server can't deconstruct the hash, but it can make one of its own and ensure that the two are identical.

At one point, the Atom Publishing Protocol defined WSSE as its preferred form of authentication. A visitor from the miserable world of SOAP, WSSE defines a simple way to hash up the user's password, the creation date of the message, and a nonce ("number used once") for a bit of randomness. The hashed tokens are difficult to pry apart, and the method helps prevent replay attacks by enforcing recency (via the creation date) and randomness (the client makes up a new nonce on every turn). WSSE has come under a great deal of criticism due to its requirement that the password be part of the hash. No sane application developer stores passwords in cleartext, but WSSE requires that this be the case in order for the server to re-create the hashed token for comparison.

Amazon's web services define their own authentication protocol that borrows a number of advantages from WSSE. First, the value that the client hashes includes HTTP headers, the request body, the URL, and the date, among other details. Second, the instead of asking for an account password, Amazon assigns each API user a secret key for use in such hashes. The secret cannot be used to retrieve API user account details, and it can be invalidated and re-generated if the user thinks it's been leaked. Third, Amazon offers several ways to attach the authorization signature to requests, from packing it into special-purpose HTTP headers to tacking it onto the request CGI parameters. The latter method makes it possible to generate limited-use URL's for private data, allowing an Amazon API user fine-grained control over public access to stored data. Because use of Amazon's API is billed, these features add up to a sane way to ensure that it's difficult to rack up excessive costs on user's account.

Tokens

A useful response to the phishing risk of passwords is a limited-user token, a pattern I'm starting to see used more often in authentication schemes.

Flickr switched to this model some time ago, adding the concept of a secret key to be shared between an application developer and Flickr. The general pattern is that authenticating as a Flickr user to a 3rd party web application involves having that application send you to a page on Flickr.com, which accepts your user credentials and asks whether the requesting 3rd party application should be allowed to read/write data on your behalf. The application and Flickr share a secret key which is checked at this time. If you agree, Flickr will redirect you to the 3rd party application's authentication handling page along with a freshly-minted frob. The 3rd party application can then convert this frob to a token, which can then be used to perform actions on that user's account.

There are a few significant things going on here. First, only Flickr needs to see your username and password, which is great security. Second, the frobs and tokens are tracked by Flickr, so the permissions you've granted to the 3rd party can be revoked at any time. Third, the secret key means that an intercepted frob is not useful to an interloper.

Unfortunately, this also means that Flickr's authentication process is (in my humble opinion) a total fucking hassle (sorry Aaron).

Google's AuthSub is a similar approach that I believe dispenses with some of Flickr's complications. Unlike Flickr, AuthSub does not require a pre-existing arrangement between the 3rd party application and Google, and there is no secret key. Instead, Google displays the authentication handler URL and domain name, and lets users determine whether they trust that application by name. The token sent by Google at this point (what Flickr calls a frob) is valid for a single-use, but can be exchanged for a session token if the user explicitly allowed this to happen. Tokens issued by Google can only be used for a limited subset of their applications, e.g. just gmail or calendars. AuthSub also agreeably allows for experimentation: it's possible to request a valid token without a publicly-viewable web application.

Google's access confirmation page looks like this:

Google rounds out AuthSub by providing a page in each user account that lists the currently-valid tokens and the web applications to which they've been granted. These can be revoked by the user on an individual basis, and offer a granular level of control over how their data is exposed and manipulated.

One potential security weakness in AuthSub is that the token may be intercepted and used. I'm not clear on how Google's web services use these tokens, though - it may be necessary to pair the token with some other piece of information that's harder to intercept, such as the user's Google account name.

An approach to keeping tokens secret that I've not yet seen in practice, but one that looks promising, is Diffie-Hellman key exchange. D-H uses a property of modular arithmetic that allows two parties to agree upon a shared secret over an insecure channel. The algorithm is roughly analogous to two people exchanging a box with two padlocks on it, keeping the box locked while in transit but not requiring either person to give up the key to their own lock. With a few extra round trips, the contents of the box can be exchanged securely.

This means that it should be possible to replace the open token transmission above with a secure exchange, resulting in a temporary secret shared between the API client and server, highly-resistant to sniffing.

Summary

I'm seeing a clear progression in API authentication from a two-party relationship between the application developer and the application user, to a three-party relationship between the application developer, the user, and the 3rd party needing temporary access to the application on the user's behalf, no doubt driven by the way popular applications are starting to treat themselves as platforms to be extended and built upon. One major recent entry that I haven't yet touched at all is Facebook.

Links mentioned above:

• Flickr API
• Flickr API developer key application form
• Flickr API terms of use
• Digg API
• Digg API application key requirements
• Mashery
• Del.icio.us API
• HTTP basic authentication
• Cal Henderson's book Building Scalable Websites
• Cornell CS 153 lecture notes
• Atom Publishing Protocol
• Amazon Web Services REST Authentication
• Google AuthSub
• Diffie-Hellman key exchange

Soul of a New Machine is Tracy Kidder's vicarious account of the design and creation of Data General's Eagle minicomputer in the early 1980's. The project was a classic skunkworks operation, developed in competition against a more prominent, better-funded 32-bit project named Fountainhead ("North Carolina", below). I first heard of this book in Bruno Latour's Science in Action, an account of the codependence between science and technology, "the role of scientific literature, the activities of laboratories, the institutional context of science in the modern world, and the means by which inventions and discoveries become accepted".

"West", below, refers to Eagle project head Tom West, an eccentric manager who brought out the best in his team by creating a barrier between them and the rest of the company, while driving them hard from above.

Update: Jason Kottke says that Tom West is Metafilter moderator Jessamyn West's dad, and has a website. The way Soul ends, it felt a bit like Shane. Closure!

Page 57, on hiring:

North Carolina's leaders had assembled a large crew mainly by luring experienced engineers away from Westborough and other companies. But around this time a videotape was circulating in the basement, and it suggested another approach. In the movies, an engineer named Seymour Cray described how his little company, located in Chippewa Falls, Wisconsin, had come to build what are generally acknowledged to be the fastest computers in the world, the quintessential number-crunchers. Cray was a legend in computers, and in the movie Cray said that he liked to hire inexperienced engineers right out of school, because they do not usually know what's supposed to be impossible. Moreover, using novices might be another way to disguise his team's real intentions. Who could believe that a bunch of completely inexperienced engineers could produce a major CPU to rival North Carolina's?

Pages 119-120, on doing things well:

On the magic marker board in his office, West wrote the following: Not Everything Worth Doing Is Worth Doing Well. Asked for a translation, he smiled and said, "If you can do a quick-and-dirty job and it works, do it." Worry, in other words, about how Eagle will look to a prospective buyer; make it an inexpensive but powerful machine and don't worry about what it'll look like to the technology bigots when they peek inside. ... To some the design reviews seemed harsh and arbitrary and often technically shortsighted. Later on, though, one Hardy Boy would concede that the managers had probably known something he hadn't yet learned: that there's no such thing as a perfect design. Most experienced computer engineers I talked to agreed that absorbing this simple lesson constitutes the first step in learning how to get machines out the door.

Page 177, on dumb jobs:

It did not work out he planned. "I thought I'd get a really dumb job. I found out dumb jobs don't work. You come home too tired to do anything," he said. He remembered a seemingly endless succession of meetings out of which only the dullest, most cautious decisions could emerge.

Pages 208-209, on naming:

The solution takes the form of a circuit called a NAND gate, which reproduces the "not and" function of Boolean algebra. The part costs eight cents, wholesale. The NAND gate produces a signal. Writing up the ECO, Holberger christens the signal "NOT YET." He's very pleased with the name. Schematics he's seen from other companies use formal, technical names for signals. The Eclipse Group, by contrast, looks for something simple that fits and if they can't come up with something appropriate they're apt to use their own names. ... It's the general approach that West has in mind when he says, "No muss, no fuss." It's also a way - a small one, to be sure - of leaving something of yourself inside your creations.

Pages 227-228, on skunkworks:

Alsing came away convinced, however, that West had an important strategy. "We're small potatoes now, but when Eagle is real, he'll have clout and can make nonnegotiable demands for salary, space, equipment and especially future products." Rasala came away with the same idea: "Maybe it's ego. But West has some interesting notions, ahhhhhnd, I kinda believe him. His whole notion is that he doesn't want to fight for petty wins when there's a bigger game in town."

Page 232, on staving off post-delivery depression:

West stubbed out his cigarette, lit another, and went back to looking at whatever it was he saw in the ceiling. "The postpartum depression on this project is going to be phenomenal. These guys don't realize how dependent they are on that thing to create their identities. That's why we gotta get the new things in place."

Page 242, on invisible computers:

Wallach and I retreated from the fair, to a cafe some distance from the Coliseum. Sitting there, observing the more familiar chaos of a New York City street, I was struck by how unnoticeable the computer revolution was. ... Computers were everywhere, of course - in the cafe's beeping cash registers and the microwave oven and the jukebox, in the traffic lights, under the hoods of the honking cars snarled out there on the street, in the airplanes overhead - but the visible differences somehow seemed insignificant.

Page 280-281, on pinball:

Their group, as they saw it, was the most dogged, hardworking, practical, productive and dangerous in the company, a bastion of the old successful ways, a paradigm of the company as it had been when it was small. The believed in the rule of pinball: if you win, you get to play again; but failure is unthinkable, so you'd better let no one get in your way.

Via Ben and Junk Charts, Andrew Kuo's blog emo + beer = busted career:

How awesome is this?

Andrew also gets paid and written up for his visual snarkery, by The New York Times.

Answering Jeff's question about sports logos made me realize how much I like the new London 2012 identity. Seeing it made me recoil at first, but a few things have changed my mind about the identity.

The Saved By The Rave Olympic Remix totally nails the retro aesthetic the brand is tickling. The official brand video even makes some of the same covert references with its sinister electronic soundtrack: new rave is a "thing" and by 2012, we should be just about ready for a 20 year bounce of late 80's/early 90's pop cultural nostalgia. Speak Up calls out two other obvious references: Money For Nothing and MTV.

There's also some incredible stuff going on at the end of that brand video (fast-forward to ~1:50):

The logo defines a basic visual grammar that will survive reproduction in print, video, web, etc., and the use I'm seeing so far crackles with energy. In contrast to Jeff's two other contestants for worst sporting event logo (2006 World Cup and Tour de France), 2012 is the only one that has any sort of life in it. World Cup is flaccid and committee-drenched, while Tour is conservative.

London 2012 is absurd and wants to be shown around, so ridiculous that it spawns a wave of derision for maximum exposure.

He hated that stupid twig-ring.