tecznotes
Michal Migurski's notebook, listening post, and soapbox. Subscribe to 
this blog.
Check out the rest of my site as well.
Dec 9, 2008 7:09am
making friends with hill shading
Living in a city that's quite hilly in places, street patterns make a lot more sense if you can see how they interact with the landscape. The inclusion of elevation data adds legibility to a map, and in the case of the Bay Area it's also interesting to see how overall urban development hugs the flatlands in most places. My goal here is still a beautiful map of Oakland for use with Oakland Crimespotting, with street-level details like schools, hospitals, and major buildings included.
I've just pushed a major update to the Bay Area cartography I've been working on. When I last posted about it in September, I had just added the Cascadenik CSS preprocessor to Dane's mapnik-utils repository. I was inspired to investigate elevation data by Andy Allan's addition of hill coloring to his award-winning OpenCycleMap project, and spurred on by finding the USGS BARD 10-meter elevation data for the San Francisco Bay Area.
Data
Turning a bag of digital elevation model (*.dem) files into shaded hills integrated with OSM map data is a multi-step process. Each file covers a rectangular area, and contains elevation in feet or meters for each included point. This is the northern part of San Francisco with Angel Island and a small bit of Marin showing. I exaggerated the colors somewhat to make it more obvious what the data contains:
Shading
OpenCycleMap doesn't actually use elevation data to simulate shadows; instead it's used to color the ground shades of green or brown, and to provide isolines. They look like this:
Andy told me that he used PerryGeo's DEM utilities to do his coloring, so I started there. It was a bit of a hassle to get hillshade.cpp compiled (see my comment on that page from Nov. 18), but eventually I was able to convert elevation files to GeoTIFFs with shading like this:
Now I had two problems. One was that the shading algorithm trims a single pixel off the edges of its input, because it can't correctly figure out the slope on the border of an area without data. The other was that the BARD *.dem files are published in a mix of meters and feet, so some sections appeared to have an exaggerated height compared to others. Happily, the heavy lifting of dealing with geographic raster data turns out to be mostly handled by the amazing GDAL library, so it was easy to write a Python script to stitch adjoining elevation files together into larger, overlapping, normalized panels and adjust for the feet-vs.-meters problem (stitch.py, 8K). It was also easy to port the C++ hillshading program to Python, which let me fine-tune some other annoying problems around the edges (hillshade.py, 4K).
Tiling
The library I use to generate map tiles, Mapnik, has a way to get raster images into a map, but it doesn't yet support niceties like warping or smooth interpolation. I still have a giant bag of multi-purpose tiling code sitting around from all my flea market mapping experimentation, so this turned out to be an easy step. I warped and tiled all the overlapping bits of shaded hill into a smooth, grayscale tile set that covers the entire SF Bay Area up to zoom level 15.
I've posted all of these hill shaded tiles to their own S3 bucket, so they can be used in slippy maps by anyone. The URL format for these is http://hills-bayarea.s3.amazonaws.com/{zoom}-r{row}-c{column}.png, e.g. Mt. Tamalpais and Mt. Diablo seen here:
I've also included a permissive crossdomain policy file, so these can be used in Flash unencumbered.
Compositing
The other thing lacking in Mapnik's RasterSymbolizer is a way to choose how a raster image visually combines with other cartography, so this ended up being a somewhat custom operation as well. I started with the OpenStreetMap style.mml style file I included as part of Cascadenik example data. I moved some roads up and down in the layering order, and made it split cleanly into two separate styles: ground.mml for ground cover, parks, and roads at very low zoom levels, and figure.mml for labels, buildings, bridges, symbols, and so on. The idea is that figure.mml and ground.mml together should look identical to style.mml, but that the split provides a convenient place to slip in a grayscale set of hills to lighten or darken the ground as necessary.
I implemented a version of Photoshop's Hard Light transfer mode because it seemed to look best in this situation. I also added a feature request to Mapnik in the hopes that this sort of thing will be a built-in feature of the library sometime.
Ta-Da
Check out the current version of the map for the results. OpenStreetMap and OpenCycleMap's own tiles are included on that page for comparison. If you see a mistake, you can correct it yourself or just mark it as a bug.
subscribe to this site.
|
contact Michal Migurski
Comments (10)
Excellent work - the map is beautiful and oh-so-handy for a cyclist. I'll also throw in belated admiration for Cascadenik which is an awesome idea. Aside: That PS1 installation in your opening image was fantastic - though I did have a run-in with a docent after sneaking a camera into the "head space".
Posted by amenity on Tuesday, December 9 2008 12:58pm UTC
This is beautiful. That is all.
Posted by Blaine Cook on Tuesday, December 9 2008 2:05pm UTC
What an amazing weaving of GDAL, python, and good looks - I'm floored. And thanks for doing the 'good work' of open source and filing solid tickets. One I'd add to the list in case you've not bumped into it is: http://trac.osgeo.org/gdal/ticket/2640 (for integrating Matt's dem tools into the gdal utilities). Thanks!
Posted by Dane on Tuesday, December 9 2008 5:19pm UTC
stunning. the bay area makes sense. thanks for documenting it all. wonder about the effect of exaggerating the slope. san francisco in particular seems less steep that the reality because the height is not great.
Posted by Mikel on Tuesday, December 9 2008 5:40pm UTC
Beautiful stuff, well done! The tech details are over my head but the end results are excellent. As I do much of my OSM mapping by bike, I often think a map like this would be a much better representation than the standard mapnik or osmarender styles.
Posted by Ben on Tuesday, December 9 2008 5:43pm UTC
Great work Mike!!! Can you explain how You "implemented a version of Photoshop's Hard Light transfer mode"?
Posted by Jaromil on Friday, December 12 2008 3:42pm UTC
Thanks everyone! @Mikel, I had some surprising results along the way when I was still confused about meters vs. feet: SF was coming through much much steeper than I had expected, and the dividing line was across the Bay so it wasn't obvious why. @Ben, check out http://opencyclemap.org if you haven't already. @Jaromil, I posted my source code for the hard light filter here: http://dpaste.com/hold/98507/ It requires NumPy http://numpy.scipy.org/ and PIL http://www.pythonware.com/library/pil/ to work. Basically it takes two images and sandwiches them together by multiplying the dark and light parts. The "flat" argument is important, because hard light has no effect at gray=128, but the colors I was getting for flat ground out of the hillshading step were gray=180. So the function gets called like this: hard_light(bgimg, hillimg, 0.3, 180) ...with a hillimg like this one: http://hills-bayarea.s3.amazonaws.com/11-r790-c326.png
Posted by Michal Migurski on Friday, December 12 2008 5:10pm UTC
Tnx Mike. At the weekend I find more poverful solution to manipulate with srtm dems. I try povray raytracer tool. Try out this .pov script http://dpaste.com/hold/99231/ ... it requires povray instaled on your machine. Before tracing you must convert tiff to png (gdal_translate recommended) and just start command povray +W6000 +H6000 test.pov -D +P +A +dem.tif (where W=with H=height of your png). You can play with colors and camera and it seems very poverful tool for visualize dems. What You think?
Posted by Jaromil on Monday, December 15 2008 11:15am UTC
this is amazing.
Posted by david on Thursday, December 25 2008 3:10am UTC
i noticed you were a member of the website ffffound, and i was wondering if you had an invite :D or if you knew anyone willing to share please contact me at my_name_is_emma77@yahoo.com
Posted by Emma on Monday, December 29 2008 8:19am UTC
Sorry, no new comments on old posts.