tecznotes

Michal Migurski's notebook, listening post, and soapbox. Subscribe to this blog. Check out the rest of my site as well.

May 27, 2016 12:20pm

five-minute geocoder for openaddresses

The OpenAddresses project recently crossed 250 million worldwide address points with the addition of countrywide data for Australia. Data from OA is used by Mapbox, Consumer Finance Protection Bureau, and my company, Mapzen.

Now, you can use OpenAddresses in a high-quality geocoder yourself. “Geocoding” is the process of transforming input text, such as an address, or a name of a place to a geographic location on the earth's surface. Every time you search for a destination on your phone, you’re geocoding. Mapzen’s Search service uses an open source server we call Pelias, and if you’re using the popular Ubuntu Linux operating system, you can get it set up and serving addresses in just a few minutes.

Start with a clean server running a current version of Ubuntu LTS (long-term support); either 14.04 or 16.04 will work. Amazon has readymade Ubuntu images available on EC2, or a local copy running under Virtualbox will do for testing. Both the address import process and the Elasticsearch index are hungry for lots of memory, so pick a server with 4-8GB of memory to prevent failures.

Next, install the Pelias software using instructions from OpenAddresses:

# Tell Ubuntu where to find packages:
add-apt-repository ppa:openaddresses/geocoder -y
wget -qO - https://packages.elastic.co/GPG-KEY-elasticsearch | apt-key add -
echo "deb http://packages.elastic.co/elasticsearch/1.7/debian stable main" | tee -a /etc/apt/sources.list.d/elasticsearch-1.7.list

# Install Pelias and dependencies:
apt-get update && apt-get install pelias

This installs the Pelias geocoder, the OpenAddresses importer, a simple web-based map search interface, and the underlying Elasticsearch index.

After installation, you will need to import data. Visit results.openaddresses.io and pick a processed zip file to download. Start small with a city like Berkeley, CA to test the process. Download and unzip it in the directory `/var/tmp/openaddresses` where Pelias expects to find CSV files, then run `pelias-openaddresses-import` to index the data.

# Get a sample file of address data:
cd /var/tmp/openaddresses
curl -OL https://results.openaddresses.io/latest/run/us/ca/berkeley.zip
apt-get install unzip && unzip berkeley.zip

# Index the addresses:
pelias-openaddresses-import

That’s it!

Pelias includes many neat features out of the box, such as reverse geocoding and autocomplete. Read the docs on Github.

The Mapzen Search service includes some additional features that aren’t yet covered here. For example, to include administrative areas like cities or states in searches, it’s necessary to do an admin lookup while importing, and to include data from Who’s On First. I’m also interested to learn more about tuning Elasticsearch for smaller-sized servers with less system RAM. It should be possible to run a geocoder with 1-2GB of memory, and Elasticsearch may require adjustments to make this possible.

Links to more information about geocoding with OpenAddresses:

Comments (3)

  1. You should probably point out that many if the data sources in OpenAddresses have usage terms that need to be adhered to (for example the Australian GNAF address data, that is not actually "open") and that you will have to check them for licence compatibility with whatever you want to do with Pelias (and the same goes for WOF).

    Posted by Simon Poole on Saturday, May 28 2016 6:22am EDT

  2. Thanks, Simon!

    Posted by Michal Migurski on Saturday, May 28 2016 12:13pm EDT

  3. @Simon Poole- I'm interested in the restrictions you've been hitting with the Australian G-NAF address data. What part of them has been closed? I'm also using the dataset. My reading of the license is that it falls under Creative Commons 4, which is quite open, except with a clause preventing the data being used for sending spam snail mail: https://data.gov.au/dataset/geocoded-national-address-file-g-naf

    Posted by Timothy Asquith on Thursday, July 28 2016 8:37pm EDT

Sorry, no new comments on old posts.

February 2017
Su M Tu W Th F Sa
   
    

Recent Entries

  1. things I’ve recently learned about legislative redistricting
  2. oh no
  3. landsat satellite imagery is easy to use
  4. openstreetmap: robots, crisis, and craft mappers
  5. quoted in the news
  6. dockering address data
  7. blog all dog-eared pages: the best and the brightest
  8. five-minute geocoder for openaddresses
  9. notes on debian packaging for ubuntu
  10. guyana trip report
  11. openaddresses population comparison
  12. blog all oft-played tracks VII
  13. week 1,984: back to the map
  14. bike eleven: trek roadie
  15. code like you don’t have the time
  16. projecting elevation data
  17. the bike rack burrito n’ beer box
  18. a historical map for moving bodies, moving culture
  19. the other openstreetmap churches post
  20. platforminess, smartness, and meaningfulness

Archives