Toronto elevation model in Minecraft

Minecraft is a fascinating video game that remains popular with the pre-teen, teen, and post-teen crowds. You build and/or exploit a 3D world by manipulating blocks of various materials such as “stone”, “dirt”, or “sand”. In the footsteps of my colleague Pamela Robinson in the School of Urban and Regional Planning, and her student Lisa Ward Mather, I became interested in ‘serious’ applications of Minecraft. Lisa studied the use of the game as a civic engagement tool. Apparently, the blocky 3D nature of Minecraft worlds can be useful in planning to give viewers an idea of planned building volumes while making it clear that preliminary display are not architectural plans.

Taking a geographic perspective, I am interested in the potential of Minecraft to educate kids about larger areas, say the City of Toronto. In this post, I outline the conversion of a digital elevation model (DEM) into a Minecraft terrain. I imagine the output as a novel way for ‘gamers’ to explore and interact with the city’s topography. Some pointers to related, but not Toronto-specific work include:

  • GIS StackExchange discussion on “Bringing GIS data into Minecraft“, including links to the UK and Denmark modeled in Minecraft
  • A video conversation about “Professional Minecraft GIS“, where Ulf Mansson combined OpenStreetMap and open government data
  • Workflow instructions for converting “Historical Maps into Minecraft” using WorldPainter, which automatically converts DEMs into Minecraft terrain (if I had seen this before I started implementing the Python script outlined below…)
  • An extensive webinar on “Geospatial and Minecraft” by FME vendor Safe Software, touching on creating Minecraft worlds from DEMs, GPS, LiDAR, building information management, and the rule-based CityEngine software

The source data for my modest pilot project came from the Canadian Digital Elevation Model (CDEM) by Natural Resources Canada, accessed using the GeoGratis Geospatial Data Extraction tool at http://geogratis.gc.ca/site/eng/extraction. In QGIS, I converted the GeoTIFF file to ASCII Grid format, which has the advantage of being human-readable. I also experimented with clipping parts from the full DEM and/or reducing the raster resolution, since the first attempts at processing would have taken several hours. The QGIS 2.2 raster translate or clip operations ran a GDAL function along the following lines (see http://www.gdal.org/gdal_translate.html and http://www.gdal.org/formats_list.html for details):

gdal_translate -projwin [xmin ymin xmax ymax] -outsize 25% 25% -of AAIGrid [input_file.tif] [output_file.asc]

On the Minecraft side, you need an account (for a small cost), a working copy of the game, and an installation of MCEdit. Player accounts are sold and managed by the game’s developer company, Mojang, see https://minecraft.net/store/minecraft. The Minecraft software itself is launched from the Web – don’t ask about the details but note that I am using version 1.8.7 at the time of writing. MCEdit is a free tool for editing saved Minecraft worlds. It has an option to add functionality through so-called ‘filters’.

The MCEdit filter I wrote is “dem_gen.py”, a Python script that collects a few input parameters from the user and then reads an ASCII GRID file (currently hard-coded to the above-mentioned Toronto area DEM), iterates through its rows (x direction) and columns (y direction in GIS, z in Minecraft), and recreates the DEM in Minecraft as a collection of ‘columns’ (z direction in GIS, y in Minecraft). Each terrain column is made of stone at the base and dirt as the top-most layer(s), or of other user-defined materials.

I have freshly uploaded the very first version 0.1 to GitHub, see https://github.com/crinner/mc_dem_gen. (This also serves as my first developer experience with GitHub!) The general framework for an MCEdit filter and the loop creating the new blocks were modified from the “mountain_gen.py” (Mountain Generator) filter found at http://www.mediafire.com/download.php?asfkqo3hk0lkv1f. The filter is ‘installed’ by placing it in the filter subfolder in the MCEdit installation. The process then simply involves creating an empty world (I used a superflat world with only a bedrock layer) and running the DEM Generator filter. To run any filter in MCEdit, select an area of the world, press ‘5’, and select the filter from the list.

QGIS-screenshot_minecraft-DEM-project

Converting the 2,400 by 1,600 pixel CDEM dataset shown in the above screenshot of my QGIS project took about half a day on a middle-aged Dell Latitude E6410 laptop.  The screenshot below shows that many data “chunks” are missing from this preliminary result, perhaps an issue when saving the terrain in MCEdit.

toronto-dem1_chunky

With a coarser DEM resolution of 600 by 400 pixels and using a newer Dell XPS 12 tablet (!), the processing time was reduced to 10 or so minutes and the result is promising. In the following screenshots, we are – I believe – looking at the outlets of the Humber River and Don River into Lake Ontario. Note the large vertical exaggeration that results from the horizontal dimensions being shrunk from around 1 block = 20m to 1 block = 80m, while vertically 1 block corresponds to 5m.

2015-06-08_21.24.32

2015-06-08_21.25.35

There remain a number of challenges, including a problem translating the geographic x/y/z coordinate system into the game’s x/-z/y coordinate system – the terrain currently is not oriented properly. More thought also has to be put into the scaling of the horizontal dimensions vis-a-vis the vertical dimension, adding the Lake Ontario water level, and creating signs with geographic names or other means of orientation. Therefore, your contributions to the GitHub project are more than welcome!

Update, 10 June 2015: I was made aware of the #MinecraftNiagara project, which Geospatial Niagara commissioned to students in the Niagara College GIS program. They aim to create “a 1:1 scale representation of Niagara’s elevation, roads, hydrology and wooded areas” to engage students at local schools with the area’s geography. It looks like they used ArcGIS and the FME converter, as described in a section of this blog post: http://geospatialniagara.com/backlog-of-updates/. Two screenshots of the Lower Balls Falls near St. Catharines were provided by @geoniagara’s Darren Platakis (before and after conversion):

minecraftNiagara-screenshot1    minecraftNiagara-screenshot2

 

My takeaways from AAG 2015

The 2015 Annual Meeting of the Association of American Geographers (AAG) in Chicago is long gone – time for a summary of key lessons and notable ideas taken home from three high-energy conference days.

Choosing which sessions to attend, was the first major challenge, as there were over ninety (90!) parallel sessions scheduled in many time slots. I put my program together based on presentations by Ryerson colleagues and students (https://gis.blog.ryerson.ca/2015/04/17/ryerson-geographers-at-aag-2015/) and those given by colleagues and students of the Geothink project (http://geothink.ca/american-associaton-of-geographers-aag-2015-annual-meeting-geothink-program-guide/), as well as by looking through the presenter list and finding sessions sponsored by select AAG specialty groups (notably GIScience and Cartography). Abstracts for the presentations mentioned in this blog can be found via the “preliminary” conference program at http://meridian.aag.org/callforpapers/program/index.cfm?mtgID=60.

Upon arrival, I was impressed by the size and wealth of the industrial and transportation infrastructure in Chicago as well as the volume of the central business district, as seen from the airport train and when walking around in the downtown core.

aag-photo1 aag-photo3

My conference started on Wednesday, 22 April 2015, with Session 2186 “Cartography in and out of the Classroom: Current Educational Practices“. In a diverse set of presentations, Pontus Hennerdal from Stockholm University presented an experiment with a golf-like computer game played on a Mercator-projected world map to help children understand map projections. Pontus also referred to the issue of “world map continuity” using an animated film that is available on his homepage at http://www.su.se/profiles/poer5337-1.188256. In the second presentation, Jeff Howarth from Middlebury College assessed the relationship between spatial thinking skills of students and their ability to learn GIS. This research was motivated by an anonymous student comment about a perceived split of GIS classes into those students who “get it” vs. those who don’t. Jeff notes that spatial thinking along with skills in orientation, visualization, and a sense of direction sets students up for success in STEM (science, technology, engineering, math) courses, including GIS. Next was Cindy Brewer, Head of the Department of Geography at Penn State University, with an overview of additions and changes to the 2nd edition of her Esri Press book “Designing Better Maps”. The fourth presentation was given by David Fairbairn of Newcastle, Chair of the Commission on Education and Training of the International Cartographic Association. David examined the accreditation of cartography-related programs of study globally, and somewhat surprisingly, reported his conclusion that cartography may not be considered a profession and accreditation would bring more disadvantages (incl. management, liability, barriers to progress) than benefits to the discipline. Finally, Kenneth Field of Esri took the stage to discuss perceptions and misconceptions of cartography and the cartographer. These include the rejection of the “map police” when trained cartographers dare to criticize the “exploratory playful” maps created by some of today’s map-makers (see my post at http://gis.blog.ryerson.ca/2015/04/04/about-quick-service-mapping-and-lines-in-the-sand/).

A large part of the remainder of Wednesday was spent in a series of sessions on “Looking Backwards and Forwards in Participatory GIS“. Of particular note the presentations by Renee Sieber, professor of many things at McGill and leader of the Geothink SSHRC Partnership Grant (http://www.geothink.ca), and Mike McCall, senior researcher at Universidad Nacional Autonoma de Mexico. Renee spoke thought-provokingly, as usual, about “frictionless civic participation”. She observes how ever easier-to-use crowdsourcing tools are reducing government-citizen interactions to customer relationships, and participation is becoming a product being delivered efficiently, rather than a democratic process that engages citizens in a meaningful way. Mike spoke about the development of Participatory GIS (PGIS) in times of volunteered geographic information (VGI) and crowdsourcing, arguing to operationalize VGI within PGIS. The session also included a brief discussion among members of the audience and presenters about the need for base maps or imagery as a backdrop for PGIS – an interesting question, as my students and I are arguing that “seed contents” will help generate meaningful discussion, thus going even beyond including just a base map. Finally, two thoughts brought forward by Muki Haklay of University College London: Given the “GIS chauffeurs” of early-day PGIS projects, he asked whether we continue to need such facilitators in times of Renee Sieber’s frictionless participation? And, he observed that the power of a printed map brought to a community development meeting is still uncontestable. Muki’s extensive raw notes from the AAG conference can be found on his blog at https://povesham.wordpress.com/.

In the afternoon, I dropped in to Session 2478, which celebrated David Huff’s contribution to applied geography and business. My colleague Tony Hernandez chaired and co-organized the session, in which Tony Lea, Senior VP Research of Toronto-based Environics Analytics and instructor in our Master of Spatial Analysis (MSA) program, and other business geographers paid tribute to the Huff model for predicting consumers’ spatial behaviour (such as the probability of patronizing specific store locations). Members of the Huff family were also present to remember the man behind the model, who passed away in Summer 2014. A written tribute by Tony Lea can be found at http://www.environicsanalytics.ca/footer/news/2014/09/04/a-tribute-to-david-huff-the-man-and-the-model.

Also on my agenda was a trip to the AAG vendor expo, where I was pleased to see my book – “Multicriteria Decision Analysis in Geographic Information Science” – in the Springer booth!

aag-springer-books

Thursday, 23 April 2015, began with an 8am session on “Spatial Big Data and Everyday Life“. In a mixed bag of presentations, Till Straube of Goethe University in Frankfurt asked “Where is Big Data?”; Birmingham’s Agnieszka Leszczynski argued that online users are more concerned with controlling their personal location data than with how they are ultimately used; Kentucky’s Matt Wilson showed select examples from half a century of animated maps that span the boundary between data visualization and art; Monica Stephens of the University at Buffalo discussed the rural exclusions of crowdsourced big data and characterized Wikipedia articles about rural towns in the US as Mad Libs based on Census information; and finally, Edinburgh’s Chris Speed conducted an IoT self test, in which he examined the impact of an Internet-connected toilet paper holder on family dynamics…

The remainder of Thursday was devoted to CyberGIS and new directions in mapping. The panel on “Frontiers in CyberGIS Education” was very interesting in that many of the challenges reported in teaching CyberGIS really are persistent challenges in teaching plain-old GIS. For example, panelists Tim Nyerges, Wenwen Li, Patricia Carbajalas, Dan Goldberg, and Britta Ricker noted the difficulty of getting undergraduate students to take more than one or two consecutive GIS courses; the challenge of teaching advanced GIS concepts such as enterprise GIS and CyberGIS (which I understand to mean GIS-as-a-service); and the nature of Geography as a “discovery major”, i.e. a program that attracts advanced students who are struggling in their original subjects. One of the concluding comments from the CyberGIS panel was a call to develop interdisciplinary, data-centred program – ASU’s GIScience program was named as an example.

Next, I caught the first of two panels on “New Directions in Mapping“, organized by Stamen’s Alan McConchie, Britta Ricker of U Washington at Tacoma, and Kentucky’s Matt Zook. A panel consisting of representative of what I call the “quick-service mapping” industry (Google, Mapbox, MapZen, Stamen) talked about job qualifications and their firms’ relation to academic teaching and research. We heard that “Geography” has an antiquated connotation and sounds old-fashioned, that the firms use “geo” to avoid the complexities of “geography”, and that geography is considered a “niche” field. My hunch is that geography is perhaps rather too broad (and “geo” even broader), but along with Peter Johnson’s (U Waterloo) comment from the audience, I must also admit that you don’t need to be a geographer to make maps, just like you don’t have to be a mathematician to do some calculations. Tips for students interested in working for the quick-service mapping industry included to develop a portfolio, practice their problem-solving and other soft skills, and know how to use platforms such as GitHub (before learning to program). A telltale tweet summarizing the panel discussion:

Thursday evening provided an opportunity to practice some burger cartography. It was time for the “Iron Sheep” hackathon organized by the FloatingSheep collective of academic geographers. Teams of five were given a wild dataset of geolocated tweets and a short 90-or-so minute time frame to produce some cool & funny map(s) and win a trophy for the best or worst or inbetween product. It was interesting to see how a group of strangers new to the competition and with no clue about how to get started, would end up producing a wonderful map such as this :-)

aag-sheep-map2

My last day at AAG 2015, Friday, April 24, took off with a half-day technical workshop on “Let’s Talk About Your Geostack”. The four active participants got a tremendous amount of attention from instructor-consultant @EricTheise. Basically, I went from zero to 100 in terms of having PostgreSQL, PostGIS, Python, NodeJS, and TileMill installed and running on my laptop – catching up within four hours with the tools that some of my students have been talking about, and using, in the last couple of years!

In the afternoon, attention turned to OpenStreetMap (OSM), with a series of sessions organized by Muki Haklay, who argues that OSM warrants its own branch of research, OpenStreetMap Studies. I caught the second session which started with Salzburg’s Martin Loidl showing an approach in development to detect and correct attribute (tag) inconsistencies in OSM based on information contained in the OSM data set (intrinsic approach). Geothink co-investigator Peter Johnson of UWaterloo presented preliminary results of his study of OSM adoption (or lack thereof) by municipal government staff. In eight interviews with Canadian city staff, Peter did not find a single official use of OSM. Extensive discussions followed the set of four presentations, making for a highly informative session. One of the fundamental questions raised was whether OSM is distinct enough from other VGI and citizen science projects that it merits its own research approach. While typically considered one of the largest crowdmapping projects, it was noted that participation is “shallow” (Muki Haklay) with only 10k active users among 2 million registered users. Martin Loidl had noted that OSM is focused on geometry data, yet with a flat structure and no standards other than those agreed-upon via the OSM wiki. Alan McConchie added the caution that OSM contributions only make it onto the map if they are included in the “style” files used to render OSM data. Other issues raised by Alan included the privacy of contributors and questions about authority. For example, contributors should be aware of the visualization and statistics tools developed by Pascal Neis at http://neis-one.org/! We were reminded that Muki Haklay has developed a code of engagement for researchers studying OSM (read the documentation, experience actively contributing, explore the data, talk to the OSM community, publish open access, commit to knowledge transfer). Muki summarized the debate by suggesting that academics should act as “critical friends” vis-à-vis the OSM community and project. To reconcile “OSM Studies” with VGI, citizen science, and the participatory Geoweb, I’d refer to the typology of user contributions developed by Rinner & Fast (2014). In that paper, we do in fact single out OSM (along with Wikimapia) as a “crowd-mapping” application, yet within a continuum of related Geoweb applications.

Notes from #NepalQuake Mapping Sessions @RyersonU Geography

This is a brief account of two “Mapping for Nepal” sessions at Ryerson University’s Department of Geography and Environmental Studies. In an earlier post found at http://gis.blog.ryerson.ca/2015/04/27/notes-for-nepalquake-mapping-sessions-ryersonu-geography/, I collected information on humanitarian mapping for these same sessions.

Mapathon @RyersonU, Geography & Spatial on Monday, 27 April 2015, 10am-2pm. 1(+1) prof, 2 undergrads, 3 MSAs, 1 PhD, 1 alumnus came together two days after the devastating earthquake to put missing roads, buildings, and villages in Nepal on the map using the Humanitarian OpenStreetMap Team’s (HOT) task manager. Thank you to MSA alumnus Kamal Paudel for initiating and co-organizing this and the following meetings.

hotosm-for-nepal_msa-lab_27april2015

Mapathon @RyersonU, Geography & Spatial on Sunday, 3 May 2015, 4pm-8pm. Our second Nepal mapathon brought together a total of 15 volunteers, including undergraduate BA in Geographic Analysis and graduate Master of Spatial Analysis (MSA) students along with MSA alumni, profs, and members of the Toronto-area GIS community. On this Sunday afternoon we focused on completing and correcting the road/track/path network and adding missing buildings to the map of Nepal’s most affected disaster zones. Photos via our tweets:

 

My observations and thoughts from co-organizing and leading these sessions, and participating in the HOT/OSM editing:

  • In addition to supporting the #EqResponseNp in a small way, the situation provided an invaluable learning opportunity for everyone involved. Most participants of our sessions had never contributed to OSM, and some did not even know of its existence, despite being Geography students or GIS professionals. After creating OSM accounts and reading up on the available OSM and Nepal-specific documentation, participants got to map hundreds of points, lines, or polygons within just a couple of hours.
  • The flat OSM data model – conflating all geometries and all feature types in the same file – together with unclear or inconsistent tagging instructions for features such as roads, tracks, and paths challenged our prior experience with GIS and geographic data. Students in particular were concerned about the fact that their edits would go live without “someone checking”.
  • While the HOT task manager and general workflow of choosing, locking, editing, and saving an area was a bit confusing at first, the ID editor used by most participants was found to be intuitive and was praised by GIS industry staff as “slick”.
  • The most recent HOT tasks were marked as not suitable for beginners after discussions among the OSM community about poor-quality contributions, leaving few options for (self-identified) beginners. It was most interesting to skim over the preceding discussion on the HOT chat and mailing list, e.g. reading a question about “who we let in”. I am not sure how the proponent would define “we” in a crowd-mapping project such as OSM.
  • There was a related Twitter #geowebchat on humanitarian mapping for Nepal: “How can we make sure newbies contribute productively?”, on Tuesday, 5 May 2015 (see transcript at http://mappingmashups.net/2015/05/05/geowebchat-transcript-5-may-2015-how-can-newbies-contribute-productively-to-humanitarian-mapping/).
  • The HOT tasks designated for more experienced contributors allowed to add post-disaster imagery as a custom background. I was not able to discern whether buildings were destroyed or where helicopters could land to reach remote villages, but I noticed numerous buildings (roofs) that were not included in the standard Bing imagery and therefore missing from OSM.
  • The GIS professionals mentioned above included two analysts with a major GIS vendor, two GIS analysts with different regional conservation authorities, a GIS analyst with a major retail chain, and at least one GIS analyst with a municipal planning department (apologies for lack of exact job titles here). The fact that these, along with our Geography students, had mostly not been exposed to OSM is a concern, which however can be easily addressed by small changes in our curricula or extra-curricular initiatives. I am however a bit concerned as to whether the OSM community will be open to collaborating with the #GIStribe.
  • With reference to the #geowebchat, I’d posit that newbie != newbie. Geographers can contribute a host of expertise around interpreting features on the ground, even if they have “never mapped” (in the OSM sense of “mapping”). Trained GIS experts understand how feature on the ground translate into data items and cannot be considered newbies either. In addition, face-to-face instructions by, and discussion with, experienced OSM contributors would certainly help to achieve a higher efficiency and quality of OSM contributions. In this sense, I am hoping that we will have more crowd-mapping sessions @RyersonU Geography, for Nepal and beyond.

Notes for #NepalQuake Mapping Sessions @RyersonU Geography

This is an impromptu collection of information to support a series of meetings of Ryerson students, faculty, and alumni of the Department of Geography and Environmental Studies with getting started with OpenStreetMap (OSM) improvements for Nepal. As part of the international OSM community’s response, contributions may help rescuers and first-responders to locate victims of the devastating earthquake.

Note that I moved the reports on our mapping sessions out into a separate post at http://gis.blog.ryerson.ca/2015/05/04/notes-from-nepalquake-mapping-sessions-ryersonu-geography/.

Information from local mappers: Kathmandu Living Labs (KLL), https://www.facebook.com/kathmandulivinglabs. KLL’s crowdmap for reports on the situation on the ground: http://kathmandulivinglabs.org/earthquake/

Humanitarian OpenStreetMap Team (HOT): http://hotosm.org/, http://wiki.openstreetmap.org/wiki/2015_Nepal_earthquake

Guides on how to get started with mapping for Nepal:

Communications among HOT contributors worldwide: https://kiwiirc.com/client/irc.oftc.net/?nick=mapper?#hot. Also check @hotosm and #hotosm on Twitter.

Things to consider when mapping:

  • When you start editing, you are locking “your” area (tile) – make sure you tag along, save your edits when you are done, provide a comment on the status of the map for the area, and unlock the tile.
  • Please focus on “white” tiles – see a discussion among HOT members on the benefits and drawbacks of including inexperienced mappers in the emergency situation, http://thread.gmane.org/gmane.comp.gis.openstreetmap.hot/7540/focus=7615 (via @clkao)
  • In the meantime (May 3rd), some HOT tasks have been designated for “more experienced mappers” and few unmapped areas are left in other tasks; you can however also verify completed tiles or participate in tasks marked as “2nd pass” in order to improve on previous mapping.
  • Don’t use any non-OSM/non-HOT online or offline datasets or services (e.g. Google Maps), since their information cannot be redistributed under the OSM license
  • Don’t over-estimate highway width and capacity, consider all options (including unknown road, track, path) described at http://wiki.openstreetmap.org/wiki/Nepal/Roads. Here is a discussion of the options, extracted from the above-linked IRC (check for newer discussions on IRC or HOT email list):

11:23:18 <ivansanchez> CGI958: If you don’t know the classification, it’s OK to tag them as highway=track for dirt roads, and highway=road for paved roads

11:26:06 <SK53> ivansanchez: highway=road is not that useful as it will not be used for routers, so I would chose unclassified or track

12:31:12 <cfbolz> So track is always preferable, if you don’t have precise info?
12:32:11 <cfbolz> Note that the task instructions directly contradict this at the moment: “highway=road Roads traced from satellite imagery for which a classification has not been determined yet. This is a temporary tag indicating further ground survey work is required.”

Another example of a discussion of this issue: http://www.openstreetmap.org/changeset/30490243

  • Map only things that are there, not those that may/could be there. Example: Don’t map a helipad object if you spot an open area that could be used for helicopter landing, create a polygon with landuse=grass instead (thanks to IRC posters SK53 and AndrewBuck).
  • Buildings as point features vs. residential areas (polygons): To expedite mapping, use landuse=residential, see IRC discussion below.
    hotosm_how-to-map-remote-buildings
    More about mapping buildings: http://wiki.openstreetmap.org/wiki/Nepal_remote_mapping_guide
  • Be aware that your edits on OSM are immediately “live” (after saving) and become part of the one and only OSM dataset. In addition, your work can be seen by anyone and may be analyzed in conjunction with your user name and locations (and thus potentially with your personal identity)

Note that I am a geographer (sort of) and GIScientist, but not an OpenStreetMap expert (yet). If you have additions or corrections to the above, let me know!

About Quick-Service Mapping and Lines in the Sand

A walk on the beach along the still-frozen Georgian Bay has helped me sort some thoughts regarding fast food cartography, quick-service mapping, and naturally occurring vs. artificial lines in the sand … but first things first: This post refers to a debate about Twitter mapping and neo-cartography that is raging on blogs across the planet and will flare up in the Geoweb chat on Twitter this Tuesday, https://twitter.com/hashtag/geowebchat. Update: #geowebchat transcript prepared by Alan McConchie available at http://mappingmashups.net/2015/04/07/geowebchat-transcript-7-april-2015-burger-cartography/.

Lines in the sand (Photos: Claus Rinner)
Lines in the sand (Photos: Claus Rinner)

A few days ago, The Atlantic’s CityLab published an article entitled “Why Most Twitter Maps Can’t Be Trusted”, http://www.citylab.com/housing/2015/03/why-most-twitter-maps-cant-be-trusted/388586/. There have been other cautions that Twitter maps often just show where people live or work – and thus where they tweet. Along similar lines, a comic at xkcd illustrates how heatmaps of anything often just show population concentrations – “The business implications are clear!”, https://xkcd.com/1138/.

The CityLab article incited Andrew Hill, senior scientist at CartoDB and mapping instructor at New York University, to respond with a polemic “In defense of burger cartography”, http://andrewxhill.com/blog/2015/03/28/in-defense-of-burger-cartography/. In it, Hill replies to critics of novel map types by stating “The dogma of cartography is certain to be overturned by new discoveries, preferences, and norms from now until forever.” He likens the good people at CartoDB (an online map service) with some action movie characters who will move cartography beyond its “local optima [sic]”. Hill offers his personal label for the supposedly-new “exploratory playfulness with maps”: burger cartography.

Examples of CartoDB-based tweet maps in the media (Source: Taylor Shelton)
Examples of CartoDB-based tweet maps in the media (Source: Taylor Shelton)

The core portion of Hill’s post argues that CartoDB’s Twitter maps make big numbers such as 32 million tweets understandable, as in the example of an animated map of tweets during the 2014 soccer world cup final. I find nothing wrong with this point, as it does not contradict the cautions against wrong conclusions from Twitter maps. However, the rest of Hill’s post is written in such a derogatory tone that it has drawn a number of well-thought responses from other cartographers:

  • Kenneth Field, Senior Cartographic Product Engineer at Esri and an avid blogger and tweeter of all things cartography, provides a sharp, point-by-point rebuttal of Hill’s post – lamenting the “Needless lines in the sand”, http://cartonerd.blogspot.co.uk/2015/03/needless-lines-in-sand.html. The only point I disagree with is the title, since I think we actually do need some lines in the sand (see below).
  • James Cheshire, Lecturer and geospatial visualization expert at University College London, Department of Geography, supports “Burger Cartography”, http://spatial.ly/2015/03/burger-cartography/, but shows that “Hill’s characterisation of cartography … is just wrong”.
  • Taylor Shelton, “pseudopositivist geographer”, PhD candidate at Clark University, and co-author of the study that triggered this debate, writes “In defense of map critique”, https://medium.com/@kyjts/in-defense-of-map-critique-ddef3d5e87d5. Shelton reveals Hill’s oversimplification by pointing to the need to consider context when interpreting maps, and to the “plenty of other ways that we can make maps of geotagged tweets without just ‘letting the data speak for themselves’.”

Extending the fast food metaphor, CartoDB can be described as a quick-service mapping platform – an amazing one at that, which is very popular with our students (more on that in a future post). I am pretty sure that CartoDB’s designers and developers generally respect cartographic design guidelines, and in fact have benefited commercially from implementing them. However, most of us do not live from fast food (= CartoDB, MapBox, Google Maps) alone. We either cook at home (e.g., R with ggplot2, QGIS; see my previous post on recent Twitter mapping projects by students) or treat ourselves to higher-end cuisine (e.g., ArcMap, MapInfo, MAPublisher), if we can afford it.

I fully expect that new mapping pathways, such as online public access to data and maps, crowdmapping, and cloud-based software-as-a-service, entail novel map uses, to which some existing cartographic principles will not apply. But dear Andrew Hill, this is a natural evolution of cartography, not a “goodbye old world”! Where the established guidelines are not applicable, we will need new ones – surely CartoDB developers and CartoDB users will be at the forefront of making these welcome contributions to cartography.

MacEachren's Some Truth with Maps (Source: Amazon.com)
MacEachren’s Some Truth with Maps (Source: Amazon.com)

While I did not find many naturally occurring lines in the Georgian Bay sand this afternoon, I certainly think society needs to draw lines, including those that distinguish professional expertise from do-it-yourselfism. I trust trained map-makers (such as our Geographic Analysis and Spatial Analysis graduates!) to make maps that work and are as truthful as possible. We have a professional interest in critically assessing developments in GIS and mapping technologies and taking them up where suitable. The lines in the sand will be shifting, but to me they will continue to exist: separating professional and DIY cartographers, mapping for presentation of analysis results vs. exploratory playing with maps, quantitative maps vis-a-vis the map as a story … Of course, lines in the sand are pretty easy to cross, too!

Twitter Analytics Experiments in Geography and Spatial Analysis at Ryerson

In my Master of Spatial Analysis (MSA) course “Cartography and Geographic Visualization” in the Fall 2014 semester, three MSA students experimented with geospatial analysis of tweets. This post provides a brief account of the three student projects and ends with a caution about mapping and spatially analyzing tweets.

Yishi Zhao wrote her “mini research paper” assignment about “Exploring the Thematic Patterns of Twitter Feeds in Toronto: A Spatio-Temporal Approach”. Yishi’s goal was to identify the spatial and thematic patterns of geolocated tweets in Toronto at different times of day, as well as to explore the use of R for spatio-temporal analysis of the Twitter stream. Within the R platform, Yishi used the streamR package to collect geolocated tweets for the City of Toronto and mapped them by ward using a combination of MapTools, GISTools, and QGIS. Additionally, the tm package was used for text mining and to generate word clouds of the most frequent words tweeted at different times of the day.

Toronto tweets per population at different times of day - standard-deviation classification (Source: Yishi Zhao)
Toronto tweets per population at different times of day – standard-deviation classification (Source: Yishi Zhao)
Frequent words in Toronto tweets at different times of day (Source: Yishi Zhao)
Frequent words in Toronto tweets at different times of day (Source: Yishi Zhao)

One general observation is that the spatial distribution of tweets (normalized by residential population) becomes increasingly concentrated in downtown throughout the day, while the set of most frequent words expands (along with the actual volume of tweets, which peaked in the 7pm-9pm period).

MSA student Alexa Hinves pursued a more focused objective indicated in her paper’s title, “Twitter Data Mining with R for Business Analysts”. Her project aimed to examine the potential of geolocated Twitter data towards branding research using the example of singer Taylor Swift’s new album “1989”. Alexa explored the use of both, the streamR and twitteR packages in R. The ggplot2, maps, and wordcloud packages were used for presentation of results.

Distribution of geolocated tweets and word cloud referring to Taylor Swift (Source: Alexa Hinves)
Distribution of geolocated tweets and word cloud referring to Taylor Swift (Source: Alexa Hinves)

Alexa’s map of 1,000 Taylor Swift-related tweets suffers from a challenge that is common to many Twitter maps – they basically show population distribution rather than spatial patterns that are specific to tweet topics or general Twitter use. In this instance, we see the major cities in the United States lighting up. The corresponding word cloud (which I pasted onto the map) led Alexa to speculate that businesses can use location-specific sentiment analysis for targeted advertising, for example in the context of product releases.

The third project was an analysis and map poster on “#TOpoli – Geovisualization of Political Twitter Data in Toronto, Ontario”, completed by MSA cand. Richard Wen. With this project, we turn our interest back to the City of Toronto and to the topic of the October 2014 municipal election. Richard used similar techniques as the other two students to collect geolocated tweets, the number of which he mapped by the 140 City neighbourhoods (normalized by neighbourhood area – “bubble map” at top of poster). Richard then created separate word clouds for the six former municipalities in Toronto and mapped them within those boundaries (map at bottom of poster).

#TOpoli map poster - spatial pattern and contents of tweets in Toronto's mayoral election 2015 (Source: Richard Wen)
#TOpoli map poster – spatial pattern and contents of tweets in Toronto’s mayoral election 2015 (Source: Richard Wen)

Despite the different approach to normalization (normalization by area compared to Yishi’s normalization by population), Richard also finds a concentration of Twitter activity in downtown Toronto. The word clouds contain similar terms, notably the names of the leading candidates, now-mayor John Tory and candidate Doug Ford. An interesting challenge arose in that we cannot tell just from the word count whether tweets with a candidate’s name were written in support or opposition to this candidate.

The three MSA students used the open-ended cartography assignment to acquire expertise in a topic that is “trending” among neo-cartographers. They have already been asked for advice by a graduate student of an environmental studies program contemplating a Twitter sentiment analysis for her Master’s thesis. Richard’s project also led to an ongoing collaboration with journalism and communication researchers. However, the most valuable lesson for the students and myself was an increased awareness of the pitfalls of analyzing and mapping tweets. These pitfalls stem from the selective use of Twitter among population subgroups (e.g., young professionals; globally the English-speaking countries), the small proportion of tweets that have a location attached (less than 1% of all tweets by some accounts), and the limitations imposed by Twitter on the collection of free samples from the Twitter stream.

I have previously discussed some of these data-related issues in a post on “Big Data – Déjà Vu in Geographic Information Science”. An additional discussion of the cartography-related pitfalls of mapping tweets will be the subject of another blog post.

A Raster-Based Game of Life Using Python in QGIS

Blog post authored by Richard Wen and Claus Rinner

A great way to demonstrate the manipulation of geospatial raster data is Conway’s Game of Life [1]. The “game” starts with a grid (“board”) of binary cells, which represent either alive (populated) or dead (empty) states. Each cell interacts with its eight adjacent neighbours to determine its next state. At each iteration of the game clock, the following rules are applied [1]:

  • A live cell with less than two or more than three live neighbours dies (under-population, overcrowding).
  • A live cell with two or three live neighbours continues to live.
  • A dead cell with three live neighbours becomes alive (reproduction).

The free and open-source Geographic Information System (GIS) software package QGIS [2] offers support for scripting with the Python programming language (pyQGIS module), which enables the use of powerful libraries such as NumPy and GDAL for dealing with raster data. Numerical Python (NumPy) [3] is a package developed for Python that is geared towards scientific computation with support for multi-dimensional arrays and matrices. The Geospatial Data Abstraction Library (GDAL) [4] is a library for translating raster and vector geospatial data formats available as a binding for Python.

Using NumPy, GDAL, and pyQGIS, we implemented the Game of Life, where NumPy manipulates the arrays, GDAL handles reading and writing of the raster data, and pyQGIS visualizes the rasters and their relative changes. The source code was written by Master of Spatial Analysis student Richard Wen with input from Dr. Claus Rinner and is available at https://github.com/rwenite/QGIS_RasterArray. The project was inspired by Anita Graser’s visit to Ryerson’s Lab for Geocomputation in October 2014, during which Anita developed a vector-based version of the Game of Life in QGIS (see http://anitagraser.com/2014/11/16/more-experiments-with-game-of-life/).

Our implementation takes an object-oriented approach, in which an object of a Game of Life class is instantiated and the gaming board is updated with the cycle() method using the QGIS python console. The core function is the manipulation of individual raster cells based on a coded algorithm – in this case, the rules defined by the Game of Life.

Let’s start by initializing and cycling a gaming board using default parameters:

# Instantiate a starting board
x = GameofLife()

game-of-life_fig1a

# Cycle the board twice
x.cycle(2)

game-of-life_fig1

The gaming board may be initialized with a random raster, a filled raster, a custom raster, or from a pre-defined raster file:

# The default is a random raster, we can set the width and height as well
x = GameofLife(width=3,height=5)
# Cycle the board
x.cycle()

game-of-life_fig2

# Fill a cells object with 1s
y = Cells(inRaster=1)
# Create a raster with the filled cells object in the directory
y.toRaster("path\\to\\filledraster\\file.tif")
# Instantiate a starting board with the filled raster
x = GameofLife(raster="path\\to\\filledraster\\file.tif")
# Cycle the board
x.cycle()

game-of-life_fig3

# Generate a raster from a list of tuples
y=Cells(inRaster=[
(0,0,0,0,0,0,0,0),
(0,0,0,0,0,0,0,0),
(0,0,1,0,0,1,0,0),
(0,0,0,0,0,0,0,0),
(0,0,1,0,0,1,0,0),
(0,0,0,1,1,0,0,0),
(0,0,0,0,0,0,0,0),
(0,0,0,0,0,0,0,0)])
# Create a raster with the custom cells object in the directory
y.toRaster("path\\to\\customraster\\file.tif")
# Instantiate a starting board with the custom raster
x = GameofLife(raster="path\\to\\customraster\\file.tif")

game-of-life_fig3b

# Instantiate a starting board with a raster
x = GameofLife(raster="path\\to\\raster\\file.tif")

game-of-life_fig4a

Date source: City of Toronto Open Data [5]

Some other interesting features include changing animation speed, jumping cycles, and applying customized layer styles:

# Adjust delay to 3 seconds
x.speed=3
# Cycle 10 times normally
x.cycle(10)
# Cycle 5 times and display every 2nd cycle
x.cycle(5,2)
# Set the style to the defined qml file
x.style = “path\\to\\qml\\style\\file.qml”

This post focuses on the functionality of the program, while its inner workings can be grasped from comments in the Python source code posted at https://github.com/rwenite/QGIS_RasterArray. The code was written and tested for QGIS 2.6; feedback on any issues is most welcome. The use of a NumPy array to iterate through the grid cells was found in an answer by user “gene” on GIS StackExchange [6]. Reading and processing raster data does have its challenges. When dealing with large grids, reading raster data in blocks rather than as a whole is advisable, because there may not be enough RAM to store an entire file at once [7].

The aim of implementing the Game of Life with Python and QGIS is to demonstrate some fundamental concepts of raster data analysis and cellular automata modeling, both of which have important applications in Geography and GIS. Existing QGIS functionality and scripts for raster processing seem to focus more on low-level input/output operations than higher-level analysis functions. For example, we did not find advanced local and focal raster operations in QGIS’ raster calculator. Thus, we envision that the RasterArray code could serve as a basis for expanding raster analysis in QGIS. The code will also be used in a yet-to-be-written lab assignment in GEO641 “GIS and Decision Support” in Ryerson’s BA in Geographic Analysis program.

 

References:

[1] Wikipedia, Conway’s Game of Life
http://en.wikipedia.org/wiki/Conway%27s_Game_of_Life

[2] QGIS
http://www2.qgis.org/en/site/

[3] NumPy, Numerical Python
http://www.numpy.org/

[4] GDAL, Geospatial Data Abstraction Library
http://trac.osgeo.org/gdal/wiki/GdalOgrInPython

[5] Toronto Open Data, Regional Municipal Boundary
http://www.toronto.ca/open

[6] How to do loops on raster cells with python console in QGIS?
http://gis.stackexchange.com/questions/107996/how-to-do-loops-on-raster-cells-with-python-console-in-qgis

[7] Chris Garrard, Utah State University, Reading Raster Data with GDAL
http://www.gis.usu.edu/~chrisg/python/2008/os5_slides.pdf

 

Thought Spot – Crowdmapping of Mental Health and Wellness Resources

Thought Spot is a project designed by post-secondary students to support mental health and wellbeing among Toronto-area youth. The main feature is the online map at http://mythoughtspot.ca/, which is based on the Ushahidi crowdsourced mapping platform. The Thought Spot project was initiated at the Centre for Addiction and Mental Health (CAMH), in collaboration with the University of Toronto, OCAD, and Ryerson. The map allows students to find mental health and wellness resources in ­their geographic area, without the need for an intermediary (parent, teacher, physician). The mapped information originates from ConnexOntario and Kids Help Phone data as well as data that were crowdsourced from members of the target audience.

thoughspot-screenshot

Ryerson Master of Spatial Analysis (MSA) candidate Heather Hart took a lead role in designing the Thought Spot map (shown above), bringing unique geospatial expertise to the table of the project’s student advisory board. Through her MSA practicum placement with a different research group at CAMH, Heather got in contact with the Thought Spot team and brought the funding for her own summer position to Ryerson, to devote half of her time to ensuring that the project’s crowdmapping would be successful. Heather’s involvement culminated in co-organizing a Thought Spot hackathon at Ryerson’s Digital Media Zone in October 2014, which led to the ongoing development of a mobile version of the Thought Spot map.

photo-thoughtspot-heather

This photo shows Heather at GIS Day at Ryerson on November 19th, 2014, presenting the Thought Spot project to an interested University audience. In collaboration with Environmental Applied Science and Management PhD candidate Victoria Fast, Heather has now also submitted a conference abstract about “Crowd mapping mental health promotion through the Thought Spot project”. The abstract brings together Victoria’s extensive expertise in volunteered geographic information systems and Heather’s on-the-ground experience with the Thought Spot project. Their presentation at the annual meeting of the Association of American Geographers in April 2015 is part of the “International Geospatial Health Research” theme.

It is wonderful to see two enterprising Geography graduate students contribute to supporting mental health and wellbeing on campus, a goal that the University is committed to. At the same time, the Thought Spot project informs Heather’s thesis research on the role of maps in evidence-based health care decision-making and Victoria’s dissertation on crowdmapping of local food resources.

Thirty-Two Thousand One Hundred Eighty-Nine Points and Counting

In another little mapping experiment with QGIS and open data from the City of Toronto, I visualized the 32,189 locations of [type-of-incident-withheld] that were recorded in Toronto from 1990 to 2013. I put out a little quiz about this map on Twitter, so I will only reveal what the points represent towards the end of this post. However, the dataset is readily available from Toronto’s open data catalog, both in tabular and GIS-ready Shapefile format.

According to a report by Global News, City crews on occasion have to deal with 20-25 of these incidents a day. As part of their data journalism, Global News created a hexagonal heatmap of the 1990-2013 data, see their article [type of incident will be disclosed].

In contrast, I mapped each point individually using lighter shades of blue for more recent years. While it is often recommended to use the darker and/or more saturated end of a colour scheme for the more important values (arguably the more recent incidents), with the ever more popular black map background, this approach is inverted: the lighter symbols will create the greater contrast, and thus appropriately represent the more important, often the larger, values. The boundaries shown in the background are City wards.

blue-dots-across-toronto_96dpi

As I finish teaching GEO241, our 2nd-year Cartography course in the BA in Geographic Analysis program, I am still having trouble identifying the thematic map type implemented here. It is not a dot density map, as a dot density map uses a unit value (could be seen as 1 dot = 1 incident) and places dots within the area for which the data were collected, but not at the exact location of occurrence. The same reasoning applies to Dr. John Snow’s map of cholera death in London 1854, which is not a dot (density) map either.

Instead, I think this map can be considered a proportional symbol map, where the point symbols at real point locations — not conceptual points such as Census tract centroids — are defined in proportion to a variable (BREAK_YEAR), yet not in terms of their size but in terms of their lightness. Clicking on the above teaser will open the full map with the title Water Main Breaks, City of Toronto, 1990-2013. So yes, there were a whopping 32,189 water main breaks in the City of Toronto during those 24 years! This situation is expected to worsen with the aging municipal infrastructure, see for example the Toronto Star’s 2010 article with a map showing downtown water mains built pre-1900. And it is not a new phenomenon either, as shown by this lovely photograph from the City of Toronto Archives (Fonds 200, Series 372, Subseries 72, Item 31), dated May 3, 1911:

Fonds 200, Series 372, Subseries 72 - Toronto Water Works photog

Toronto’s Traffic Lights Re-Visited and Animated

My map of Toronto’s traffic signals described in a post on April 4th, 2014, was recently published on the title page of Cartouche, the newsletter of the Canadian Cartographic Association (CCA). This is my first-ever published map that is stand-alone, not included in an article or other text document! Here is a screenshot of the newsletter title:

screenshot-cartouche-title-spring2014

Motivated by this unexpected outcome and using the occasion of the launch event of Maptime Toronto on May 29th, 2014, I wanted to try animating the dots representing the traffic signals. More precisely, each traffic light should iterate through a green-yellow-red sequence, and each mid-block pedestrian crossing should go through an off-blinking-off sequence. I was aiming for an animated GIF image with ten frames displayed in a continuous loop.

To create the colour sequences for each dot in QGIS, I copied the last digit of an existing  feature ID from the City of Toronto traffic signals data table into a new field to act as a random group assignment. Using a suggestion by Michael Markieta, I then created nine additional integer fields and cycled through the group numbers by adding 1. To keep these numbers in the 0…9 range, I used QGIS’ “modulo” function, e.g. Cycle1 = (“Cycle2” + 1) % 10. I then assigned the green, yellow, and red dot symbols from the static traffic lights map as a categorized “style” to different group numbers. Finally, I manually iterated the symbology through the ten group columns and took a screenshot each time. I put these together in the animated GIF shown below.

animation_25

I must admit that I am not super convinced of the outcome. Maybe, ten frames are not enough to overcome the clocked appearance of the traffic signal system. But at least, things are moving now :)

It is important to note that this animation does not show the real-time status of the traffic lights! In fact, there is only one dot for an entire intersection that would include two to four sets of vehicle traffic lights, plus pedestrian lights, etc. – all represented by the same green-yellow-red cycle on the map. I also made the assumption that the green and red phases are the same length (4 out of 10 ticks each, with the remaining 2 ticks used for the yellow phase). You will note that the mid-block crossings have an active phase with three on-off cycles followed by a longer off phase. In this case, it would be fancier to individually control each crossing and have it come on randomly.