...so take it easy.

On the advice of Tod Kurt, Ben, and others, I bought some shift registers to try a second pass at the Atkinson-dithered 8x8 screen. Success!

The wiring now requires just three data pins instead of the previous 12 data pins, and results in a completely addressable screen:

There are two chips on that board: one controls the rows and one the columns. If I added a third, and a mess of more wires, I could use the full red/green capacity of the LED matrix. Sadly, it's getting tangled. I think if I learned to solder it'd be possible to get all three shift registers squeezed under the matrix for a nice little screen module:

Anyone out there willing to spend an evening showing how to properly connect components to a prototype board? I can't get back to this stuff for a little while anyway, I think I may have fried the controller on the Arduino somehow.

This is the second half of a talk that Tom Carden and I did together at U.C. Berkeley's iSchool, a few weeks ago on March 20. Tom has the first half with slides posted on his own blog. He talks about the general studio/Stamen context for our forays into visual depictions of urban data, while I go deep on Oakland Crimespotting in particular. I returned to Berkeley's Graduate School of Journalism a few weeks later to deliver a slightly-edited version of this talk alone. A video that I'm afraid to watch is on Youtube here and here.

CrimeWatch

First, it's important for us to back up a bit and understand the need that Crimespotting tries to address.

Our initial work was based on the City of Oakland's existing CrimeWatch application, a "wizard-like" form-based website available at http://gismaps.oaklandnet.com/crimewatch/. The first thing a visitor to CrimeWatch sees is a disclaimer form and a few pages of instructions.

Once past the first page, there is a long series of questions that a visitor must answer: what kind of crime to view, where to look, how far back in time to search. The crime types here seem to correspond to broadly-used categories, but there are differences among jurisdictions. For example, Oakland does not publish domestic violence as a separate category. San Francisco, which uses a similar crime mapping application, has removed homicides entirely from its online maps.

Eventually, the resulting map loads in the browser. The presentation is not entirely satisfying, with a range of tiny report icons overlaid on a map and frequently obscured by information detail windows.

Parsing Images

This is the technically difficult part. The second of the two images above shows our original data input: a compressed image covered with small, graphic icons.

We used Python Imaging Library and a collection of scraping tools to search for instances of known icons and convert them to rough GPS positions.

Because we knew what we were looking for (e.g. Vehicle Theft, Prostitution, Narcotics, Robbery, Vandalism shown here) it was possible to search by shape and color to come up with positions for these icons on CrimeWatch maps.

Early this year, Oakland City IT opened up a reliable, text-based feed of crime information for us. This was a huge help, and we no longer need to go through the slow image-scraping process detailed above. Just before IT made a better source of data available to us, we were putting the finishing touches on a Mozilla browser plug-in for distributed scraping, which introduced an element of human sanity-checking after each sweep.

Permanent Links

All of the data we collect is available in a non-mapped, text form. Everything is shown on a map, but it's also possible to slice and dice the data to show reports of a certain type or from a certain date, listed out.

The single most important improvement we think Crimespotting introduced to Oakland is the concept of a permanent link for every reported crime. These permalinks contain isolated maps of the event, connected reports, a place for people to leave comments, and other reports from the same area and time.

Patterns

Street crime in Oakland really only happens between the 880 and 13 freeways. South and west of 880 is industrial, while north and east of 13 is hilly, suburban, and quite affluent. Property crimes (marked in green) happen all over, but violent crime (marked in red) is further constrained to the poorer parts of town below 580.

We've noticed that prostitution arrests seem to come in waves - long stretches of no activity, followed by rapid bursts along San Pablo Boulevard in West Oakland or International Boulevard in East Oakland. Above we can see the result of a string of busts along International in the Fruitvale area shown in blue, the color we use for quality of life crimes like drugs, alcohol, or disturbing the peace.

Although West Oakland is commonly thought to be a dangerous place, the data we've seen shows that violent crime is really spread evenly between West Oakland and downtown. There's no cluster of violence to the West, but there is a sharp cluster of drug activity (in blue) between 580 and San Pablo.

One important lesson we learned from our users: police beats are more important than we thought. This is a map of reports from beat 04X. The beat number system is how citizens interact with the police department, and what we assumed was an arcane administrative detail is really a living, breathing idea. Not so with "police service areas" or city council districts.

Outcomes

Really, the most important thing we've learned is that motivation can come from everywhere. I initially started this project because I had hurt my back in late 2006, and had a bunch of free time on my hands over that winter. What started as a technical curiosity became a studio project and eventually a public website. Motivation comes from all over, give people data and they'll figure out something interesting to do with it.

Just showing crime is relentlessly negative, and seems to really draw out the kind of graffiti-squad neighborhood busybodies who focus solely on little problems. A near-universal reaction from non-residents to this particular project has been relief that they don't live in Oakland, but it's really not that bad here. It just looks bad when all you show is crime. We'd like to map other things: city services (police, fire, emergency), tax parcels, effects of policy, other administrative information that's hugely important.

We're also thinking about how to fill in the tapestry around Oakland - currently we're covering a very small, narrowly-defined area.

On the bright side, we're using our projects to test the assumption that there's something interesting going on with city data. Decreasing costs (money, complexity) of data analysis techniques (e.g. Google's new visualization kit) drive demand for available data.

We think in the very near future, it will become cheaper for cities to publish raw data and let citizens do their own analysis with IT in a support/enabling/superhero role. Our argument is not about "democratizing" anything and the political baggage that rides along with such terminology, it's about responding to changing costs.

Matt Webb sent this to me a while back, it's the Cliff Martinez soundtrack to the George Clooney version of Stanislaw Lem's Solaris. The soundtrack is instrumental, spooky, and calming.

1. Is This What Everybody Wants
2. First Sleep
3. Can I Sit Next To You
4. Will She Come Back
5. Death Shall Have No Dominion
6. Maybe You're My Puppet
7. Don't Blow It
8. High Energy Proton Accelerator
9. Wear Your Seat Belt
10. Wormhole
11. We Don't Have To Think Like That Anymore

Also apropos of nothing, the Alan Parker edition of The Orb's Little Fluffy Clouds.

This is so dumb.

Schweddy Eagle is an application I just wrote that yanks your location from Fire Eagle and tells you what NPR stations are in your area. You could use it with a jesus phone, perhaps in a rental car, to find quality radio wherever you happen to be.

It uses the excellent OAuth protocol to safely ask for your current location.

Last October I published a tiny implementation of (Bill) Atkinson dithering in Python. Aaron ran with it and created a Modest Maps filter to prepare maps for print. Now, I'm finding Atkinson dithering useful when pumping pixels through this LED matrix I bought from Sparkfun and connected to my Arduino.

This is me messing with hardware.

First off, the matrix isn't an easy thing to wire up. The datasheet provides these two diagrams:

A bunch of trial-and-error showed that the pins on the matrix are ordered from #1 in the lower left, to #12 in the lower right, up to #13 in the upper right and #24 in the upper left: counter-clockwise, one pin for each row and two pins (green, red) for each column. The pins are grouped in sets of (green, red, row) across the top and bottom, with the top pins controlling the left-hand columns and upper rows, and the bottom pins controlling the right-hand columns and bottom rows. It's necessary to know which end is up, shown by the (now rubbed-off) product code printed on the bottom of the matrix component. I think a little extra effort on Foryard Optoelectronics' part would have resulted in a matrix that was radially symmetrical, and worked identically regardless of how it was pushed into the breadboard.

You connect a column and a row from power to ground, and the corresponding LED lights up. You can address one entire row or one entire column at a time, but a full 8x8 image requires you to scan from one to the next, illuminating pixels in sets of 8 hundreds of times per second.

I don't have 16 digital outputs on the Arduino, so I can only address six rows and six columns. This is enough to beging to experiment with tiny images, like the fades from all-off to all-on and back shown here:

There are two pieces of code making that happen. The first, matrix.pde, is running on the Arduino, set up to accept incoming bytes and display them on the little screen. It's super dumb, but it's got scan lines and an off-screen buffer to reduce flicker, so I'm happy with how it works. The second, fade.py, is running on my laptop and pumping strings of binary data over a serial connection to make the pretty pictures. At the moment, it's fading from white to black and back and sending dithered versions of those images over the wire. If my math is right, it should be able to do this at least 100 times per second without breaking a sweat, so I'm thrilled with the time resolution.

I'm not yet sure where this goes next, but I'm going to try running some simple video over the wire to see whether it's even remotely recognizeable as an image.