tecznotes

I picked up Michael Lewis's new book, The Big Short, pretty much as soon as it hit the Kindle a few weeks ago. It's a post-catastrophe account of the subprime mortgage crisis, told through the eyes of a small group of traders who shorted the supposedly unshortable mortgage backed securities that made everyone rich five years ago.

It's partially a financial story, but to me it's also a story about assumptions. I've been thinking a bit about the effects of unspoken, day-to-day dependencies that we all rely on in our lives. Can we live without them? Are we light enough on our feet to adjust when they shift? Do we even know what they are, and can we explain how they fit together? In a small company like mine, these questions can lead to some fairly serious existential crises. A few years ago, our client base seemed disproportionately tied to the galloping Web 2.0 economy. More recently, the brash announcement by Apple that Flash would be unsupported on the iPad confirmed a long-held suspicion that the platform was on rocky ground. In the trenches of my day-to-day as technology director, I've become excessively sensitive to the problems of cross-dependencies among projects, code bases, and servers. This is not so much an issue of identifying single points of failure as it is a matter of understanding which doorknobs you've tied your teeth to and subsequently forgotten.

Three questions:

1. Do you depend on anything outside your control? What is it?
2. Can you repeat past successes with those same externalities?
3. Could you quarantine, isolate, or replace them, if you had to?

The Big Short is the story of one particular set of external dependencies that turned out to be hopelessly intertwined. Specifically, it's about the revelation that an entire class of financial products based on the performance of mortgage payments was more deeply interdependent and market-distorting than anyone had imagined. It's the moment near the end of a Stephen King novel where all the townspeople are revealed to have first names that start with "K" and they're sitting silently in their cars waiting for you up the road. NPR's Planet Money does a better job of explaining the details ("we bought the toxic asset..."), but the underpinning of the story shows how difficult it is to reject a lie when your livelihood depends on believing it.

Anyway.

Loc. 476-82, an opening anecdote showing the matter-of-fact cultural role of Wall Street greed:

When a Wall Street firm helped him to get into a trade that seemed perfect in every way, he asked the salesman, "I appreciate this, but I just want to know one thing: How are you going to fuck me?" Heh-heh-heh, c'mon, we'd never do that, the trader started to say, but Danny, though perfectly polite, was insistent. We both know that unadulterated good things like this trade don't just happen between little hedge funds and big Wall Street firms. I'll do it, but only after you explain to me how you are going to fuck me. And the salesman explained how he was going to fuck him. And Danny did the trade.

Loc. 483-87, Steven Eisman is one of the main characters, a brusque gadfly with odd listening habits:

Working for Eisman, you never felt you were working for Eisman. He'd teach you but he wouldn't supervise you. Eisman also put a fine point on the absurdity they saw everywhere around them. "Steve's fun to take to any Wall Street meeting," said Vinny. "Because he'll say 'explain that to me' thirty different times. Or 'could you explain that more, in English?' Because once you do that, there's a few things you learn. For a start, you figure out if they even know what they're talking about. And a lot of times they don't!"

Loc. 985-89, on the undesireability of defending an idea:

Inadvertently, he'd opened up a debate with his own investors, which he counted among his least favorite activities. "I hated discussing ideas with investors," he said, "because I then become a Defender of the Idea, and that influences your thought process." Once you became an idea's defender you had a harder time changing your mind about it. He had no choice: Among the people who gave him money there was pretty obviously a built-in skepticism of so-called macro thinking.

Loc. 1788-96, on the role of research that seemingly no one else wants to do. This is actually one of the most interesting aspects of The Big Short for me, the relative rarity of legwork compared to the ease of sticking to first appearances:

It wasn't a question two thirty-something would-be professional investors in Berkeley, California, with $110,000 in a Schwab account should feel it was their business to answer. But they did. They went hunting for people who had gone to college with Capital One's CEO, Richard Fairbank, and collected character references. Jamie paged through the Capital One 10-K filing in search of someone inside the company he might plausibly ask to meet. "If we had asked to meet with the CEO, we wouldn't have gotten to see him," explained Charlie. Finally they came upon a lower-ranking guy named Peter Schnall, who happened to be the vice-president in charge of the subprime portfolio. "I got the impression they were like, 'Who calls and asks for Peter Schnall?'" said Charlie. "Because when we asked to talk to him they were like, 'Why not?'" They introduced themselves gravely as Cornwall Capital Management but refrained from mentioning what, exactly, Cornwall Capital Management was. "It's funny," says Jamie. "People don't feel comfortable asking how much money you have, and so you don't have to tell them."

Loc. 1830-34, on arguing convincingly:

Both had trouble generating conviction of their own but no trouble at all reacting to what they viewed as the false conviction of others. Each time they came upon a tantalizing long shot, one of them set to work on making the case for it, in an elaborate presentation, complete with PowerPoint slides. They didn't actually have anyone to whom they might give a presentation. They created them only to hear how plausible they sounded when pitched to each other. They entered markets only because they thought something dramatic might be about to happen in them, on which they could make a small bet with long odds that might pay off in a big way.

Loc. 2206-11, more on Eisman's listening habits:

Eisman had a curious way of listening; he didn't so much listen to what you were saying as subcontract to some remote region of his brain the task of deciding whether whatever you were saying was worth listening to, while his mind went off to play on its own. As a result, he never actually heard what you said to him the first time you said it. If his mental subcontractor detected a level of interest in what you had just said, it radioed a signal to the mother ship, which then wheeled around with the most intense focus. "Say that again," he'd say. And you would! Because now Eisman was so obviously listening to you, and, as he listened so selectively, you felt flattered.

Loc. 3260-64, on $1.2 billion:

In early July, Morgan Stanley received its first wake-up call. It came from Greg Lippmann and his bosses at Deutsche Bank, who, in a conference call, told Howie Hubler and his bosses that the $4 billion in credit default swaps Hubler had sold Deutsche Bank's CDO desk six months earlier had moved in Deutsche Bank's favor. Could Morgan Stanley please wire $1.2 billion to Deutsche Bank by the end of the day? Or, as Lippmann actually put it - according to someone who heard the exchange - Dude, you owe us one point two billion.

Loc. 3413-22, on eight days of chlorine for all of Chicago:

His wife's extended English family of course wondered where he had been, and he tried to explain. He thought what was happening was critically important. The banking system was insolvent, he assumed, and that implied some grave upheaval. When banking stops, credit stops, and when credit stops, trade stops, and when trade stops - well, the city of Chicago had only eight days of chlorine on hand for its water supply. Hospitals ran out of medicine. The entire modern world was premised on the ability to buy now and pay later. "I'd come home at midnight and try to talk to my brother-in-law about our children's future," said Ben. "I asked everyone in the house to make sure their accounts at HSBC were insured. I told them to keep some cash on hand, as we might face some disruptions. But it was hard to explain." How do you explain to an innocent citizen of the free world the importance of a credit default swap on a double-A tranche of a subprime-backed collateralized debt obligation? He tried, but his English in-laws just looked at him strangely. They understood that someone else had just lost a great deal of money and Ben had just made a great deal of money, but never got much past that. "I can't really talk to them about it," he says. "They're English."

Loc. 3747-52, on being dumb and looking for grownups:

The big Wall Street firms, seemingly so shrewd and self-interested, had somehow become the dumb money. The people who ran them did not understand their own businesses, and their regulators obviously knew even less. Charlie and Jamie had always sort of assumed that there was some grown-up in charge of the financial system whom they had never met; now, they saw there was not. "We were never inside the belly of the beast," said Charlie. "We saw the bodies being carried out. But we were never inside." A Bloomberg News headline that caught Jamie's eye, and stuck in his mind: "Senate Majority Leader on Crisis: No One Knows What to Do."

Loc. 3880-82, a last word on dependencies:

The changes were camouflage. They helped to distract outsiders from the truly profane event: the growing misalignment of interests between the people who trafficked in financial risk and the wider culture. The surface rippled, but down below, in the depths, the bonus pool remained undisturbed.

On June 16th 2010, the Bay Area Metropolitan Transportation Commission released Clipper, a rebranding of the original Translink payment card for public transportation. In the ensuing five years, Clipper has overtaken other forms of fare and payment to become the only remaining acceptable method of beeping a ride on Bay Area public transit. Starting with San Francisco's Muni and Alameda County Transit, later expanding to BART, Santa Clara Valley, San Mateo's SamTrans, Golden Gate Transit, Yellow Cab, Veterans Cab Co., and most recently the full complement of toll bridges including the Golden Gate, Bay Bridge, Richmond and San Mateo bridges thanks to Caltrans. Clipper is everywhere around the Bay.

It's difficult to remember now that just half a decade ago, local transit was a confusing jumble of mismatched schedules and fares. Riders no longer restrict themselves to monthly passes from a single agency, choosing instead to hop from one mode to another as their needs emerge. Unified payment makes most of this activity possible: the Clipper card has grown in importance along with the government ID and credit card. Initially envisioned as a purely a multi-agency payment card, Clipper has long since erased the functional distinctions among agencies thanks to the smooth thoughtlessness of synchronizing payments on all sides of the Bay.

Similar to the historical unification of competing streetcar companies under the umbrella of city-operated transportation authorities, we expect that late next year or possibly early 2017, the MTC Commissioners will retire the independent identities of SF Muni, AC Transit, Santa Clara VTA and SamTrans to be replaced by a unified "Bay Area Clipper" name. Most local observers view this as a pure formality, though it's expected that in keeping with its historic reluctance to participate in inter-agency plans, the BART Board will delay participation in the new name until 2018 at the earliest. They'll come around eventually.

Clipper has also managed to tap into a full range of data streams connected to urban transit, making them more interesting and valuable along the way. The fluidity and ease of motion we enjoy now is also made possible through the nearly ubiquitous availability up-to-the-minute locations for participating vehicles, made available from the agencies themselves through a public stream of real-time updates accessible to programmers and normal users alike. Buses and trains are almost universally equipped with location-tracking devices based on GPS and wireless signals, and a suite of applications from commercial, open-source, and philanthropic developers build on the predictive, route-finding data services published for every vehicle in the system. This has helped ease the pain of occasional budget cuts and service disruptions, giving users a way to minimize the amount of time they waste waiting around for their next ride.

Interestingly, the commercial providers of wi-fi, cell tower, and other "RF beacon" geolocation services have transitioned to something more like a regulated utility model. Originally established up to provide location lookup services for smartphones near the end of the 00's, the largest of these providers recently folded their own billing for Bay Area users into the Clipper system. The number of lookups you do for your physical location or finding an arrival time for a bus are simply charged to your card. Many users never even see these charges, since a large number of employers pick up the tab for employee accounts on a pre-tax basis.

More recently, concerns have surfaced around the privacy of data tracked through the Clipper servers. People are understandably jittery, after the numerous social networking data breach debacles of three years ago that seemingly turned a generation off of oversharing. MTC have gone to great pains to assure users of the system that their data is safe from "getting zucked", and they've begun to provide free personal monitoring services to users of Clipper. It's now possible to access to a complete, up-to-the-minute stream of your own card usage (including the geographic location of each beep) along with a record of access requests to that same data by parents, friends, mobile apps, credit reporting firms, or government agencies monitoring transportation use for oil-credit tax breaks. If someone's peeking at your transit history, you're the first one to know.

As Clipper begins its fifth year, we're seeing movement toward expanding the card into other uses around the Bay Area. Loose legal definitions of "transportation spending" have bike and rollerblade shops lobbying to allow repairs and maintenance to be charged to a Clipper account. The financial district congestion zone has announced a feasibility study for ditching its proprietary payment structure for Clipper. Even hardware companies have started to retool their smartchip-based door locks to optionally work with the system, bringing the card right into private homes. It seems too obvious to mention and too pervasive to notice, but seemingly everywhere you turn the one-time bus payment card has turned into a key to complete mobility and total access.

Let's try and not screw this one up.

There's a short series of posts in here, but I'm out of practice with blogging so I'll start with just this.

I've exhumed some of last year's thinking on heat maps, and re-encountered Zach and Andy's excellent series of posts on geographic isolines. Zach Johnson posted some ideas on a quick-and-dirty method for generating isolines from a field of point measurements, using the Delaunay triangulation (which I've mentioned before). Andy Woodruff followed up with ideas about curves, specifically the problem of passing smooth curves through a set of points.

The general idea is to go from this:

...to this:

...using a process somewhat like this:

("S" from Raph Levien)

Anyway. Splines are pretty much a solved problem, in the sense that your typical graphics library is going to support at least cubic splines - Flash, SVG, etc. all have native methods for making smooth curves between two endpoints with two additional control points. If you've used Adobe Illustrator at any time in the past 23 years, you'll know how this works:

Andy writes about the generation of curves that are smooth, yet guaranteed to pass through a full set of points without appearing discontinuous. What makes this difficult is that there are an infinite number of solutions, generally differentiated by the amount of tension or distance along connections between pairs of points. It's also difficult to express these solutions using control points; where do you place them? Often you need to make assumptions about the correct slope through a given point, and often those assumptions lead to some weird-looking results.

With a bit of basic algebra it's possible to deconstruct these smooth curves into a set of parametric equations. The trick is to introduce a new independent variable, t, which represents progress along the curve through a third dimension. T is for time, because it makes sense to think about x and y in the image plane varying through time. You can find a smooth curve through any set of points - a straight line through two points, a quadratic curve through three points, a cubic curve through four points, etc. I'm stopping at four points and the cubic curve, it looks good and is easy to calculate.

Here's one such 2D cubic curve:

It might look familiar if you've ever used a TI-81. The twisty math bit here is that given a set of any four arbitrary points, it's possible to generate one such curve that passes through them all. The second twisty math bit is that you can do this for the x and y components of the curve separately, creating two separate functions over t that move through space and describe a curve. Wikipedia has an excellent article on systems of linear equations, though I avoided most of the tedious algebra by using Sympy, a symbolic math library for Python that does the work. This is the resulting function, and here is an animation showing eight separate cubic functions over a set of eight arbitrary points:

Here's the same image showing all the functions overlaid on top of one another, with the "closest" central one highlighted in turn:

This is the third twisty math bit. To create a single, smooth curve out of all these pieces, you adjust the relative influence of each one over its length using a basis function. They're described in excruciating detail at ibiblio.org, where you can scroll down that page to play with some interactive java applets. I've cheated, and used sine waves that still look something like this over the length of the curve:

Putting it all together, you get this transition from simple cubic curves to a complete, smooth system that closes a full set of points:

The code for all this is a bit of a hairball (also syntax-highighted), but I hope potentially useful.

One reason I find this method interesting is that the end product is actually not a curve at all, but a series of points joined by straight line segments. You provide the illusion of a curve by make lots of them, and very short. This means that the resulting "curves" are trivially compatible with geographic software like PostGIS or Mapnik, and therefore possible to simplify using tools like Mapshaper, distribute in formats like Shapefile, and render with plain old Tilecache. At the cost of additional file size, you are freed from Flash and dumped wide-eyed and blinking in the world of actual geography.