Michal Migurski's notebook, listening post, and soapbox. Subscribe to this blog. Check out the rest of my site as well.

Feb 20, 2017 5:30pm

things I’ve recently learned about legislative redistricting

Interesting things are afoot in legislative redistricting! Over the past ten years, Republicans have enacted partisan gerrymanders in a number of state houses in order to establish and maintain control of U.S. politics despite their unpopular policies. I’ve been learning what I can about redistricting and I’m curious if there’s something useful I could offer as a geospatial open data person.

This post is a summary of things I’ve been learning. If any of this is wrong or incomplete, please say so in the comments below. Also, here’s an interactive map of the three overlapping districts you’re probably in right now:

Three exciting things are happening now.

First, Wisconsin is in court trying to defend its legislative plan, and not doing well. It’s a rare case of a district plan being challenged on explicitly partisan grounds; in the past we’ve seen racial and other measures used in laws like the Voting Rights Act, but partisan outcomes have not typically been considered grounds for action. It might be headed to the Supreme Court.

Second, a new measure of partisan gerrymandering, the Efficiency Gap, is providing a court-friendly measure for partisan effects. Defined by two scholars, Nicholas Stephanopoulos and Eric McGhee, the measure defines two kinds of wasted votes: “lost votes” cast in favor of a defeated candidate, and “surplus votes” cast in favor of a winning candidate that weren’t actually necessary for the candidate’s victory. Stephanopoulos sums it up as “the difference between the parties’ respective wasted votes in an election, divided by the total number of votes cast.”

Wisconsin happens to be one of the biggest bullies on this particular block:

This New Republic article provides a friendly explanation.

Third, former U.S. Attorney General Eric Holder has created the National Democratic Redistricting Committee (NDRC), a “targeted, state-by-state strategy that ensures Democrats can fight back and produce fairer maps in the 2021 redistricting process.” Right now, I’m hearing that NDRC is in early fundraising mode.

So that’s a lot.

I sent some fan mail to Eric McGhee and he graciously helped me understand a bunch of the basic concepts over coffee.

One thing I learned is the significance and use of political geography. As Marco Rogers has pointed out, liberals and democrats clump together in urban areas: “Look at the electoral maps. Drill into the states. What we see is singular blue counties, clustered around cities, in an endless sea of red.” At Code for America, we worked with a number of cities that fell into this pattern, and frequently they were looking to CfA for help dealing with blue town vs. red county issues.

Jowei Chen, associate professor of political science in Michigan, has an extensive bibliography of writing about legislative districts. In his 2013 paper Unintentional Gerrymandering, Chen demonstrates how a sampling of possible redistricting proposals can maintain partisan bias:

In contemporary Florida and several other urbanized states, voters are arranged in geographic space in such a way that traditional districting principles of contiguity and compactness will generate substantial electoral bias in favor of the Republican Party.

Geometry is a red herring. Over the years I’ve encountered a few geometry optimizations for proposed districts, including this one from Brian Olson, written up in the Washington Post:

Olson’s proposed district plans

While compactness is desirable in a district, Olson’s method prioritizes visual aesthetics above political geography, and he notes some of the factors he ignores on his site, such as travel time: “it might be the right kind of thing to measure, but it would take too long.” Olson’s method selects aesthetically pleasing shapes that are fast to calculate on his home computer. I think that’s a terrible basis for a redistricting plan, but the goofy shapes that exist in many current plans are a popular butt of jokes:

Gerrymandering T-Shirts by BorderlineStyle

Chen particularly calls out how cartographic concerns can be a dead-end:

Our simulations suggest that reducing the partisan bias observed in such states would require reformers to give up on what Dixon (1968) referred to as the “myth of non-partisan cartography,” focusing not on the intentions of mapmakers, but instead on an empirical standard that assesses whether a districting plan is likely to treat both parties equally (e.g., King et al., 2006; Hirsch, 2009).

However, geography is not insurmountable.

In a later 2015 paper, Cutting Through the Thicket, Chen argues through statistical simulations that legislative outcomes can be predicted for a given redistricting plan, and plots the potential results of many plans to show that a given outcome can be intentionally selected:

A straightforward redistricting algorithm can be used to generate a benchmark against which to contrast a plan that has been called into constitutional question, thus laying bare any partisan ad- vantage that cannot be attributed to legitimate legislative objectives.

Here’s Florida’s controversial 2012 plan shown as a dotted line to the right of 1,000 simulated plans, demonstrating a “clearer sense of how this extreme partisan advantage was created:”

A graph from Chen’s 2015 paper showing simulated partisan outcomes for Florida district plans

Chen concludes that the position of the dotted line relative to the modal outcomes shows partisan intent, if you agree that such an outcome is unlikely to be random.

In 2010, Republicans systematically generated skewed partisan outcomes in numerous state houses, as documented in this NPR interview with the author of Ratf**ked:

There was a huge Republican wave election in 2010, and that is an important piece of this. But the other important piece of Redmap is what they did to lock in those lines the following year. And it's the mapping efforts that were made and the precise strategies that were launched in 2011 to sustain those gains, even in Democratic years, which is what makes RedMap so effective and successful.

“RedMap” was a GOP program led by Republican strategist Chris Jankowski to turn the map red by targeting state legislative races:

The idea was that you could take a state like Ohio, for example. In 2008, the Democrats held a majority in the statehouse of 53-46. What RedMap does is they identify and target six specific statehouse seats. They spend $1 million on these races, which is an unheard of amount of money coming into a statehouse race. Republicans win five of these. They take control of the Statehouse in Ohio - also, the state Senate that year. And it gives them, essentially, a veto-proof run of the entire re-districting in the state.

Holder’s NDRC effort is a counter-effort to RedMap. They’re planning electoral, ballot, and legal initiatives to undo the damage of RedMap. Chen’s simulation method could allow a legislature to overcome geographic determinism and decide on an outcome that better represents the distribution of voters. Chen again:

We do not envision that a plaintiff would use our approach in isolation. On the contrary, it would be most effective in combination with evidence of partisan asymmetry and perhaps more traditional evidence including direct testimony about intent and critiques of individual districts. As with Justice Stevens’ description of partisan symmetry, we view it as a “helpful (though certainly not talismanic) tool.”

So, back to the efficiency gap.

McGhee and Stephanopoulos’s measure counts actual votes in real elections. That’s helpful to courts trying to determine whether a given plan is fair, because it does not rely on guessing about possible outcome from public opinion. Chen’s approach provides a statistical expectation for what a normal plan could do, as well as ways to adjust plans based on desired outcomes. Calculating the efficiency gap for a proposed district plan is complicated, because you need to account for cases where simple red/blue data is missing, such as a uncontested races. You have to impute the potential vote in each proposed new district.

To do this, you need precinct-level election data. Jon Schleuss, Joe Fox, and others working with Ben Welsh at the LA Times Data Desk recently created the most detailed election result map ever made for California. In other states, the data is often not available online, and must be specially requested from sometimes-unhelpful officials. Eric McGhee told me that many experts working on redistricting use a dataset maintained by DailyKos, an independent liberal news website.

LA Times maps of California’s 2016 election results

There’s a big opportunity here for a carefully-vetted online tool that could calculate measures like the efficiency gap for a variety of districting plans. For my part, I’m getting started understanding the sources and types of data that might help pull district plans in a fairer direction. If you’re curious about your own district, find yourself on this map:

Jan 26, 2017 1:44am

oh no

A dumb thing I made:

A response from Alex Norris!

Jacob takes it to politics:

This was unexpected:

Oh no.

Dec 14, 2016 1:16am

landsat satellite imagery is easy to use

Helping Bobbie Johnson with the Medium Ghost Boat series about a boat of migrants that’s been missing since 2014, I had need of satellite imagery for context to illustrate a dataset of boat sightings in the Mediterranean Sea off the coast of Libya.

(near Zurawah, Libya)

Fortunately, Landsat 8 public domain U.S. government imagery exists and is easy to consume and use if you’re familiar with the GDAL collection of raster data tools. Last year, Development Seed worked with Astro Digital to create Libra, a simple browser of Landsat imagery over time and space:

At libra.developmentseed.org, you can browse images spatially and sort by cloud cover to pick the best recent Landsat products by clicking a circle and using the dated images in the right side of your browser window. Imagery is not provided as simple JPEG files, and is instead divided into spectral bands described at landsat.usgs.gov. Four of them can be used to generate output that looks like what a person might see looking down on the earth: Blue (2), Green (3), Red (4), and Panchromatic (8). Download and extract these four bands using curl and tar; a bundle of bands will be approximately 760MB:

curl -O https://storage.googleapis.com/earthengine-public/landsat/L8/038/032/LC80380322016233LGN00.tar.bz
tar -xjvf LC80380322016233LGN00.tar.bz LC80380322016233LGN00_B{2,3,4,8}.TIF

Combining these bands is possible with a processing script I’ve adapted from Andy Mason, which corrects each band before merging and pansharpening them into a single RGB output like this:

So that’s a good start, but it’s only a corner of Utah’s Great Salt Lake when you might want an image of the whole lake. Each file has large, useless areas of black around the central square. Download four images overlapping the lake and extract the four bands from each:

Use gdalwarp to convert each of the separate band raster files into a single geographic projection, in this case the common web spherical mercator:

gdalwarp -t_srs EPSG:900913 LC80380322016233LGN00_B2.TIF LC80380322016233LGN00_B2-merc.TIF
gdalwarp -t_srs EPSG:900913 LC80390312016240LGN00_B2.TIF LC80390312016240LGN00_B2-merc.TIF
# repeat for 14 more bands

Then, use gdal_merge to combine them into individual band mosaics that cover the whole area:

gdal_merge.py -o mosaic_B2.TIF -n 0 LC*_B2-merc.TIF
gdal_merge.py -o mosaic_B3.TIF -n 0 LC*_B3-merc.TIF
gdal_merge.py -o mosaic_B4.TIF -n 0 LC*_B4-merc.TIF
gdal_merge.py -o mosaic_B8.TIF -n 0 LC*_B8-merc.TIF

Now, Andy’s processing script can combine a larger area and color-correct all four images together:

Jul 28, 2016 7:07pm

openstreetmap: robots, crisis, and craft mappers

The OpenStreetMap community is at a crossroads, with some important choices on where it might choose to head next. I spent last weekend in Seattle at the annual U.S. State Of The Map conference, and observed a few forces acting on OSM’s center of gravity.

I see three different movements within OpenStreetMap: mapping by robots, intensive crisis mapping in remote areas, and local craft mapping where technologists live. The first two represent an exciting future for OSM, while the third could doom it to irrelevance.

The OpenStreetMap Foundation should make two changes that will help crisis responders and robot mappers guide OSM into the future: improve diversity and inclusion efforts, and clarify the intent of OSM’s license, the ODbL.

Robot Mappers

Engineers from Facebook showed up to talk about how machine-learning and artificial intelligence (“robot”) techniques might help them produce better maps. Facebook has been collaborating for the past year with another SOTMUS attendee, DigitalGlobe, to consume and analyze high-resolution satellite imagery searching for settled areas as part of its effort to expand internet connectivity.

However, there are still parts of the world in which the map quality varies. Frequent road development and changes can also make mapping challenging, even for developed countries. … In partnership with DigitalGlobe, we are currently researching how to solve this problem by using a high resolution satellite imagery (up to 30cm per pixel). … For small geographical areas, this technique has allowed our team to contribute additional secondary and residential roads to OSM, offering a noticeable improvement in the level of details of the map.

OSM has long had a difficult relationship with so-called “armchair mapping,” and Facebook’s efforts here are a quantum leap in seeing from a distance. This form of mapping typically requires the use of third party data. For sources such as non-public-domain satellite imagery, robot mappers must be sure that licenses are compatible with derived data and OSM’s own ODbL license. Copyright concerns can make or break any such effort. Fortunately, OSM has a sufficiently high profile that contributors rarely attempt to undermine the ODbL directly. Instead the choose to cooperate with its terms, to the extent they understand the license.

Crisis Mappers

Meanwhile, representatives of numerous crisis mapping organizations showed to talk about the use of OSM for mapping vulnerable, typically remote populations. Dale Kunce of the Red Cross Missing Maps project gave the second day keynote, while Lindsey Jacks gave an update on her work with Field Papers (Stamen’s ongoing product I originally called Walking Papers) and numerous members of organizations like Humanitarian OpenStreetMap Team (HOT), Digital Democracy, and others attending workshops on collecting data.

Disaster response and crisis mapping organizations take a more direct, on-the-ground approach to map features that can’t be seen remotely or require local knowledge to interpret. While these efforts have often used remote data, as in the satellite-aided Haiti earthquake response in 2010, that data has always been paired with ground efforts in the area concerned.

When major disaster strikes anywhere in the world, HOT rallies a huge network of volunteers to create, online, the maps that enable responders to reach those in need. … HOT supports community mapping projects around the world and assists people to create their own maps for socio-economic development and disaster preparedness.

The populations most in need of crisis-response mapping efforts are typically furthest from OSM’s W.E.I.R.D. founding core. Such efforts will succeed or fail based on their participation. OSM should do a better job of welcoming them.

Craft Mappers

Historically, OpenStreetMap activity took place in and around the home areas of OSM project members, as a kind of weekend craft gathering winding up in a local pub. OSM originated from a mapping party model. Western European countries like England, Germany, and France achieved high coverage density early in the project’s history due to active local mappers carrying GPS units, riding bikes to collect data, and getting together at a pub afterward. Mapping pubs and similar amenities was a cultural touchstone for the project’s founding participants. The project has always featured better-quality data in areas where these craft mappers lived, for example near universities with computer science or information technology programs.

Many historic OSM tendencies, such as aversion to large-scale imports and a distinctly individualist technical worldview, are the result of this origin. They mirror the histories of other open source and open data projects, which often start as itch-scratching projects by enthusiastic nerds. Former OSM Foundation board member Henk Hoff’s 2009 “My Map” keynote is a great example of OSM’s early focus on areas local to individual mappers. The founding work of this branch of the community treats the project as a kind of large-scale hobby, like craft brewing or model railroading.

At its current stage of development, OSM’s public communications channels seem to be divided amongst these communities. I heard much frustration from crisis mappers about the craft-style focus of the international State Of The Map conference in Brussels later this year, while the hostility of the public OSM-Talk mailing list to newcomers of any kind has been a running joke for a decade. The robot mappers show up for conferences but engage in a limited way dictated by the demands of their jobs. Craft mapping remains the heart of the project, potentially due to a passive Foundation board who’ve let outdated behaviors go unexamined.

It’s a big downer to see a fascinating community mutely sitting out important discussions and decisions about the community’s future. Left to the craft wing, OSM will slide into weekend irrelevance within 5-10 years.

Two Modest Proposals

Two linked efforts would help address the needs of the crisis response and robot mapping communities.

First, U.S. technical gatherings like PyCon have been invigorated over the past years by codes of conduct and other mechanisms intended to welcome new participants from under-represented communities. State Of The Map US has done a great job at this, but the international conference and foundation seem to be engaging only in passive efforts, if any. A prominent code of conduct for in-person and online events would bring OSM in-line with advances in other technology forums, who have chosen to value the contributions of diverse participants. With the appearance of crisis and disaster mapping on the OSM landscape over the past five years, we need to appeal to people of color disproportionately affected by crises and disasters, so they understand they are welcome within OSM’s core. Ignoring this or settling for “#FFFFFF Diversity” is a copout.

Ashe Dryden says this about CoC’s at events with longtime friends running the show:

We focus specifically on what isn't allowed and what violating those rules would mean so there is no gray area, no guessing, no pushing boundaries to see what will happen. "Be nice" or "Be an adult" doesn't inform well enough about what is expected if one attendee's idea of niceness or professionalism are vastly different than another's. On top of that, "be excellent to each other" has a poor track record. You may have been running an event for a long time and many of the attendees feel they are "like family", but it actually makes the idea of an incident happening at the event even scarier.

Second, the license needs to be publicly and visibly explained and defended for the benefit of large-scale and robot participants. ODbL FUD (“fear, uncertainty, and doubt”) is a grand tradition since the community switched licenses from CC-BY-SA to ODbL in 2012. I support the share-alike and attribution goals of the license. But passive communication about its intent and use has left the door wide open for unhelpful criticisms. OSM Foundation publishes community guidelines on a separate wiki rather than a proper website. It’s not enough: the license needs to be promoted, defended, and consistently reaffirmed. Putting it under active discussion may even make it possible to adapt to new needs via a mechanism like Steve Coast’s license ascent, where “work starts out under a restrictive and painful license and over time makes its way into the public domain.”

I’ve struggled to write this post without overusing the word “actively,” but it’s the heart of what I’m suggesting. OSMF Board has been at best a background observer of project progress, while OSM itself has slowly moved along Simon Wardley’s “evolution axis” from a curiosity to a utility:

A utility like today’s OSM requires a different form of leadership than the uncharted and transitional OSM’s of 2005-2010. There are substantial businesses and international efforts awkwardly balanced on a project still being run like a community garden, without visible strategy or leadership.

Thanks to Nelson, Mike, Kate, and Randy for their input on earlier drafts of this post. I am an employee of Mapzen, a Samsung-owned company that features OSM and other open data in our maps, search, mobility, and data products.

Jul 13, 2016 11:52pm

quoted in the news

I’ve been quoted in a few recent news stories, which happens from time to time but this is unusually a lot.

If You Can’t Follow Directions, You’ll End Up on Null Island (Wall Street Journal)

Science reporter Lee Hotz got in touch after talking to my coworker Nathaniel Kelso about the Null Island geo-meme. It’s a short piece, but I hear it’s supposed to run on A-1 tomorrow and Lee did an impressive amount of interviewing to make sure the story was right. Nathaniel included Null Island in the wildly popular Natural Earth dataset around the time he worked at Stamen. We had just included Null Island in a basemap style for GeoIQ where Kate Chapman, another interviewee, had worked at the time. I was happy to lead Lee to Steve Pellegrin from Tableau, who was the initial creator of the meme that birthed a thousand t-shirts.

Stronger Together: Could Data Standards Help Build Better Transportation Systems? (Govtech)

Ben Miller digs into data availability for transportation systems. At Mapzen, we host Transitland for transit data based on Bibiana McHugh’s pioneering work on the General Transit Feed Specification (GTFS) at Portland TriMet. It’s the reason you can find a bus ride in both Google Maps and Mapzen Turn-By-Turn!

The App That Wants to Simplify Postal Addresses (The Atlantic)

Robinson Meyer writes one of the few non-sycophantic, critical articles about What3Words and addresses generally. I’ve got a few friends who work there so I hate to be mean, but in my eyes What3Words is an anti-institutional bet whose success, like the price of gold, will correlate strongly with the unraveling of basic public infrastructure. I can’t believe Meyer got the part-owner of privatized Mongol Post to cop to having heard about W3W at the Davos cartoon supervillian convention. Thinking about it makes me mad enough that I’ll change topics and leave you with this amazing quote from another Robinson Meyer article on word processors and history, from English professor Matthew Kirschenbaum:

Another interesting story that’s in the book is about John Updike, who gets a Wang word processor at about the time Stephen King does, in the early 1980s. I was able to inspect the last typewriter ribbon that he used in the last typewriter he owned. A collector who had the original typewriter was kind enough to lend it to me. And you can read the text back off that typewriter ribbon—and you can’t make this stuff up, this is why it’s so wonderful to be able to write history—the last thing that Updike writes with the typewriter is a note to his secretary telling her that he won’t need her typing services because he now has a word processor.
February 2017
Su M Tu W Th F Sa

Recent Entries

  1. things I’ve recently learned about legislative redistricting
  2. oh no
  3. landsat satellite imagery is easy to use
  4. openstreetmap: robots, crisis, and craft mappers
  5. quoted in the news
  6. dockering address data
  7. blog all dog-eared pages: the best and the brightest
  8. five-minute geocoder for openaddresses
  9. notes on debian packaging for ubuntu
  10. guyana trip report
  11. openaddresses population comparison
  12. blog all oft-played tracks VII
  13. week 1,984: back to the map
  14. bike eleven: trek roadie
  15. code like you don’t have the time
  16. projecting elevation data
  17. the bike rack burrito n’ beer box
  18. a historical map for moving bodies, moving culture
  19. the other openstreetmap churches post
  20. platforminess, smartness, and meaningfulness