So finally we are nearing the end of the project (ok technically at the end but its both complex and exciting so we might squeeze in a few more days yet) we would like to introduce the world to GEMMA, the Geospatial Engine for Mass Mapping Applications.

Gemma Map shown 3 layers of data

GEMMA was designed to allow anyone to make complex map mashups with a minimum of geographic knowledge. So easy in fact that even the most modest of users will be able to quickly collect data and visualise it in a few clicks, no worrisome logins, data downloads or nasty terms and conditions . Hopefully, GEMMA will get everyone excited about making their own maps, from a map of all their local Tesco locations through to a map of the number of cars that pass by their house. Building maps online takes time and often basic programming knowledge. GEMMA solves this problem by integrating many of the popular CASA products, along with some new features and iPhone app, combined into one easy to use platform allowing you make an interactive map that can be shared with the world.

Gemma Map shown 3 layers of data

So what are GEMMA’s main features?

Basically it’s a structured mashing application. A little bit from MapTube, a little bit from Google MyMaps, a little bit of opening up the wealth of data from OpenStreetMap. Add your own measured data with the accompanying iPhone app that is super simple to use – choose what you want to count, start tapping the screen to count, and then upload. Back in the GEMMA website, login and add.

With GEMMA you can stand on street corners and easily measure the demographics of, for example, Boris Bike users, create a shiny PDF of the data, and stick it up on a wall or embed it into your homework. We provide a unqiue shortcode for sharing with the world, or GEMMA will remember it in a list for future access.

Gemma Data Collector showing values 4, 8, 11, 4 and users position

How would you use GEMMA?

If you’re a Geography student who needs to prepare some maps for your course work on local population groups, you may need to collect counts of people for various locations around your local town and  compare that with the Output Area Classifications (OAC). What are you going to do? In the past you would have had to go and find an OAC map of your area and annotate it with the various bits of data you collect but you have left it to the last minute, again, and don’t know what to do! That’s where GEMMA steps in. Take a look at the video to see how easy using GEMMA actually is:

You can view the PDF created in the video here and take a preview look at  GEMMA.

The iPhone Application will be available to download from the iTunes App Store and we will let you know in this blog when it’s available. The GEMMA website is still in the BETA stages and we are just cleaning up the interface to make it easier for you to use as soon as it’s ready you’ll be able to use the web app from this link.

So who built GEMMA?

A few of the people at CASA, namely  Steven Gray, Ollie O’Brien our Principle Investigator Andy Hudson-Smith and finally Richard Milton who built some of the backend features that GEMMA requires to work so a big thank you to him. We will try and get a team photo together soon as we can.

As Ollie notes he has already blogged about the different APIs and libraries used. With any significant web project these days, good use of the rich tapestry of libraries out there is invaluable. Why write a polished Javascript-based UI from scratch when there is the powerful, beautiful and simple JQuery UI? Why spend ages creating sortable, resizable tables when JQuery Datatables will do that for you? Mapnik 2 has a whole host of new features, let’s use them. Let’s use authentication APIs from Twitter and Google rather than writing our own – the last thing people on the web need is yet another logon and password to remember (although, as Steve who did this part can attest, it’s not that easy – or smooth – for developers to take incorporate third-party identity management.) Let’s use custom-styled maps in the Google Maps API, and let Google take the strain of rendering them (it’s dead easy to create a grayscale layer for the Google Maps API).

If you sat Ollie down and asked the work he was most personally proud of, the answer would be GEMMAs PDF creation. During the project he has blogged about a couple of fairly cool things he has been able to do with WMS and SVG inputs for Mapnik and Cairo respectively, which looked to be a challenge. Steve on the other hand has been central to the GEMMA iPhone app, an app that allows anyone or any group to count anything, anywhere and have it mapped in realtime. From pedestrians and traffic through to flora and fauna, if it can be counted it can be mapped. The creation of a whole new mapping service with surveys, open data, custom markers and a bespoke iPhone app within a single interface has honestly been a challenge and one that has led to intense project meetings in the GEMMA lab.

Through it all we feel we have honestly come up with something unique, we hope you are as excited about GEMMA as we are and it doesn’t stop here. GEMMA has so far been a short, focused project. It’s tough to stop once you’ve started going – there’s so more we could do. We have more features to add to GEMMA in the pipeline so if you are interested in or have a feature that you would like to see in GEMMA then do get in contact.

Thanks as ever go to JISCGeo for allowing the creation of a Geographic Engine for Mass Mapping Applications.

, , , , , , , , ,

Just one more techy post (from me anyway) before this blog gets formal and we submit our Final Project Post for GEMMA, summing the project and applications up. Why? I wanted to share with you two rather specialist bits of GEMMA that required quite a bit of head-scratching and trial-and-error, in the hope that, for some developer out there, they will be of use in their own work.

One of the core features of GEMMA that I have always desired is the ability to output the created map as a PDF. Not just any PDF, but a vector-based one – the idea that it will be razor-sharp when you print it out, rather than just looking like a screen grab. I had written a basic PDF creator, using Mapnik and Cairo, for OpenOrienteeringMap (OOM), an earlier side-project, and because the GEMMA project is about embracing and extending our existing technologies and knowledge at CASA, rather than reinventing the wheel, I was keen to utilise this code in GEMMA.

Most of the layers were quite straightforward – the two OpenStreetMap layers (background and feature) are very similar indeed to OOM, while the Markers and Data Collector layers were also reasonably easy to do – once I had imported (for the former) and hand-crafted (for the latter) suitable SVG images, so that they would stay looking nice when on the PDF. The trickiest layer was the MapTube layer. For the terms of this project, the MapTube imagery is not a vector layer, i.e. we are not using WFS. However I was still keen to include this layer in the PDF, so I turned to Richard Milton (the creator of MapTube) and discovered there is an WMS service that will stitch together the tiled images and serve them across the net. I could combine this requesting the WMS images on the GEMMA server (not the client!), converting them to temporary files, and then using a RasterSymbolizer in Mapnik 2, and an associated GDAL filetype.

The trickiest part was setting georeferencing information for the WMS image. Georeferencing is used to request the image, but it is also needed to position the image above or below the other Mapnik layers. Initially it looked like I would have to manually create a “worldfile”, but eventually I found a possibly undocumented Mapnik feature which allows manual specification of the bounding box.

I’ve not seen this done anywhere else before, although I presume people have just done it and not written it down on the web, so here’s my take, in Python.

First we get our WMS image. MAP_W and MAP_H are the size of the map area on the “sheet” in metres. We request it with a resolution of 5000 pixels per metre, which should produce a crisp looking image without stressing the server too much.

mb = map.envelope()
url = maptube_wms_path + "/?request=GetMap&service=WMS&version=1.3.0"
url = url + &format=image/png&crs=EPSG:900913"
url = url + "&width=" + str(int(MAP_W*5000))
url = url + "&height=" + str(int(MAP_H*5000))
url = url + "&bbox=" + str(mb.minx) + "," + str(mb.miny) + ","
url = url + str(mb.maxx) + "," + str(mb.maxy)
url = url + "&layers=MAPID" + str(maptubeid)
furl = urllib2.urlopen(url, timeout=30)

Mapnik doesn’t work directly with images, but files, so we create a temporary file:

ftmp = tempfile.NamedTemporaryFile(suffix = '.png')
filename = ftmp.name
ftmp.write(furl.read())

Next we set up the layer and style. It’s nice that we can pass the opacity, set on the GEMMA website, straight into the layer in Mapnik.

style = mapnik.Style()
rule = mapnik.Rule()
rs = mapnik.RasterSymbolizer()
rs.opacity = opacity
rule.symbols.append(rs)
style.rules.append(rule)
map.append_style(filename,style)
lyr = mapnik.Layer(filename)

Here’s the key step, where we manually provide georeferencing information. epsg900913 is the usual Proj4 string for this coordinate reference system.

lyr.datasource = mapnik.Gdal(base='',file=filename, [contd...]
[contd...] bbox=(mb.minx, mb.miny, mb.maxx, mb.maxy)) #Override GDAL
lyr.srs = epsg900913

Finally:

lyr.styles.append(filename)
map.layers.append(lyr)

I’m excited about one other piece of code in the PDF generation process, as again it involves jumping through some hoops, that are only lightly documented – adding an SVG “logo” – the SVG in this case being the GEMMA gerbil logo, that Steve (co-developer) created from Illustrator. Cairo does not allow native SVG import (only export) but you can use the RSVG Python package to pull this in. I’m being a bit lazy in hard-coding widths and scales here, because the logo never changes. There are more sophisticated calls, e.g. svg.props.width, that could be useful.

svg = rsvg.Handle(gemma_path + "/images/logo.svg")
ctx = cairo.Context(surface)
ctx.translate((MAP_WM+MAP_W)*S2P-ADORN_LOGO_SIZE, CONTENT_NM*S2P)
ctx.scale(0.062, 0.062)
svg.render_cairo(ctx)

Note that we are calling render_cairo, a function in RSVG, rather than a native Cairo function that we do for all the other layers in the PDF.

A few words about the main technologies and APIs used in GEMMA. Using these has helped make the project achievable within the timeframes we have had.

JQuery and JQuery UI

A rich and comprehensive extension to the JQuery Javascript framework, JQuery UI allows easy and uniform skinning of user interface elements (buttons, tabs, dialog boxes). JQuery itself comes with a large number of functions that are more powerful and simpler to use than their vanilla Javascript equivalents.

JQuery Datatables

JQuery Datatables builds on JQuery, incorporating the JQuery UI look and providing, with a minimum of fuss, powerful pageable, sortable, searchable and flexible tables of data.

Mapnik 2

Mapnik, the map renderer used for the “default” OpenStreetMap. It is a C++ library but comes with mostly complete Python bindings, which I use. Mapnik received a major version upgrade very recently, jumping from version 0.7 to 2. The new version allows “expressions” – certain values (e.g. text captions) that can be specified during the creation of each feature, rather than hard-coding them in to the code or stylesheet. These are used to create the “captioned markers” that are a feature of GEMMA.

Python

I’m mentioning Python here because it forces me to write neat code! The tile renderer is a number of Python scripts calling Mapnik. Mapnik itself uses the (Py)Cairo graphics renderer to create the images and PDF maps, but I am also extending its use for creating full-blown PDF “printed sheets”, by including adornments on the area outside of the map. I’m using RSVG to include the GEMMA logo (imported in as an SVG) in all GEMMA’s PDF maps.

The two processes – producing images for the website (and iPhone), and producing PDFs, are quite different. In the former, Mapnik is called repeatedly for each layer, and the order and opacities of the layers are controlled with OpenLayers. In the latter. Mapnik itself needs to handle the ordering and opacities, as well as producing a high quality (scalable) PDF.

Google Maps Javascript API v3

The original mass-application mapping API and still the most popular. It’s best feature is the simple way that custom colours can be applied – in only a few lines of Javascript, we are able to pull in a nice greyscale layer for use as a background map in GEMMA. Interestingly, Google very recently allowed that extremely heavy users of the API will start to be charged. For custom coloured maps, the threshold is much lower than for “normal” Google Maps, although we don’t anticipate it would be an issue for GEMMA.

OpenLayers

OpenLayers is also being used in GEMMA, although limited to the “preview” map that appears when you are adding captioned markers to the map. It also can pull in the Google Maps tiles with custom colours used in the main interface.

Google Fusion Tables

Google’s powerful and extensible cloud-based tables are used to store data collected for use in GEMMA.

Google and Twitter Authentication Libraries

We use both Google and Twitter authentication in GEMMA, rather than creating our own standalone user data storage and authentication structure. The world has enough logins.

iPhone App development (Objective-C coding)

In order to create a native iPhone app, you need to learn Objective-C. Thankfully, Steve is an experienced Objective-C person.

Gemma’s had a bit of a wash-and-blow-dry:

We liked Gemma’s strong branding, and the brown/orange colour theme we used during internal development (a taste of which is on the “coming soon” page) was certainly striking. But perhaps we have a few too many animals at CASA – there’s Rodger the giraffe for SurveyMapper, the goldfish in Tales of Things, and the woolly mammoth in QRator. So Gemma the gerbil has a lower profile now – you’ll still see her when you first visit the website, and get a pop-up welcoming you to the site. The bar at the top is now aquamarine blue, with orange highlighting – this blends with the look and feel of the JQueryUI-powered dialog boxes on the website, and is similar to the look of CASA’s well-designed and inviting SurveyMapper website – the clouds are from there too.

In terms of the core development, the last few features are being polished and the code and design is being tided up, ready for the feature-freeze at the end of the month. We are moving into the bug-squashing and foolproof-testing stage – not so glamorous as the blue-sky coding but just as important for a finished product. We hope to have a public beta available soon.

A problem which I think GEMMA could solve:

I need to have a map showing the locations of the five main “London” airports – and Eurostar’s London terminal at St Pancras for good measure. The idea is to show the general direction from those locations to the centre of London, to help international visitors to the capital understand where they would be arriving in relation to the city itself. Ideally, I would be able to produce the map in a couple of minutes, without needing to open up a graphics program or do anything fancy.

First I tried Google Maps:

No sign of any airports there (& similarly for the satellite view or terrain view).

Then I tried OpenStreetMap‘s default “Mapnik” layer:

A little better, in that the airports are shown – but they are rather hard to see in the sea of other detail that you tend to get with this style. Also, there are other airports shown, with the same symbologoy, that are really not important and don’t really deserve to be in the same league as Heathrow, e.g. Redhill Aerodrome and Elstree Airfield.

Then I tried Cloudmade. The Cloudmade maps are based on OpenStreetMap, but have many styles available. Crucially, they allow editing of the styles, to change colours and line widths and toggle on and off features at different scales:

Unfortunately at certain zoom levels, some features are just not allowed. Airports thankfully are allowed at the zoomed-out level I’m using, but there’s no option to increase the size of the “airport” logo, or force its name to be shown. Slightly confusingly, the airports appear under the “aerodrome” section rather than the “airport” section. The generalisation has also obscured London City Airport completely. (Using other initial styles may produce better results.)

Finally I went back to Google Maps and used the “My Place” functionality, which I think used to be called MyMaps:

There’s a lovely custom airport icon, but no way to add a caption below the icon, to help distinguish which airport is which.

I also tried the OpenStreetMap Static Maps API which is a similar idea, but has a similar shortcoming in that you can’t label the pins.

So, perhaps GEMMA will be able to fill the gap of providing an easy way to produce a simple map with captioned pins. I think this will be achievable with four things:

1. An uncluttered base-map. The GEMMA team is still deciding about OpenStreetMap vs Google Maps for background mapping. Currently I’ve swung quite a bit in favour of Google Maps (v3 really is quite nice.) It’s easy, almost trival, to produce a greyscale basemap in Google Maps:

The above example is, incidentally, also highlighting the features in OpenStreetMap that include “airport” in the name, in turquoise, using GEMMA’s OpenStreetMap Feature Highlighter functionality. Not very useful in this stage as it’s not captioned and includes those unimportant airport locations.

The relevant code for this, using the Google Maps v3 API, looks like this:


var style = [
{ featureType: 'road.highway', elementType: "labels", stylers: [ {visibility: "off"} ] },
{ featureType: 'all', stylers: [ {saturation: -100}, {lightness: 50} ] }
];

2. Use of the shiny new Mapnik MemoryDatasource, which appears in Mapnik2, the unreleased next-generation version of Mapnik that we just happen to already be using for GEMMA.

3. Use of the really quite sophisticated ShieldSymbolizer, which guarantees the appearance of both an icon and an accompanying caption. You can change the text colour, font and relative positioning, and even the size and colour of the “halo” around the text.

4. Mapnik-based on-the-fly map tiling, as used for the already-built OpenStreetMap Feature Highlighter. Instead of using a PostGIS datasource and a copy of the OpenStreetMap database, we use a user-specified point-and-click datasource, and a Mapnik2 MemoryDatasource.

…now we just need to build it…

The GEMMA team (Steven Gray and myself) will be in Vienna at the end of this week, for the State of the Map EU conference. On Saturday morning we have a short presentation on the GEMMA project, with hopefully (the code is a little unstable at the moment!) a live demo of the OpenStreetMap Feature Highlighter* which forms one of the data sources to GEMMA. We’ll also mention SurveyMapper and MapTube, two other data sources for the project. A short paper, on the same theme as the presentation, should also make it into the conference’s electronic proceedings.

Here’s one of the maps from the OSM Feature Highlighter – can you guess what it is highlighting?

* Yes, I know we will have to come up with a much snappier name when we launch for real.

The GEMMA team were at Wherecamp EU which took place in Berlin at the end of May. There were some interesting sessions, on topics relevant to building modern “neo-geo” websites such as what will form part of GEMMA.

One big debate within the team we have had is whether to go with Google Maps v3 or OpenLayers, as the “API” that underpins the “slippy map” that people will see, that forms the fundamental part of the website. It was interesting to read that Patrick Weber, of JISCG3, another JISC Geospatial project also being implemented at UCL, has been having a similar debate. As the project moves from the design and prototyping stage to some core coding, the Google vs OpenLayers question is also coming to the fore.

Our decision is complicated by:

  • having a mobile as well as web application stream
  • the need to do some fairly low-level integration of OpenStreetMap and OS OpenData datasets (for one of our streams)
  • making good use of existing CASA technologies, such as MapTube* (built deeply into the Google Maps v2 API which is quite different to v3) and SurveyMapper (on Google Maps v3) as well as my own CensusProfiler, EducationProfiler and Bike Share Map work (all three on OpenLayers)
  • the relatively short timeframe and team-size for this multi-faceted project, meaning that pragmatic choices on speed of development are important
  • an increased focus on using fully open technologies for our core work as an academic research group
  • …and one team member having experience mainly with the Google Maps v2/v3 API and one member having experience mainly with OpenLayers.

At the moment we are focusing on Google Maps for mobile development, and a mix of the two for “desktop” web applications. However OpenLayers 2.11, currently in Release Candidate phase and so soon to be released, has some promising developments, particularly in the field of mobile support. We’ll be designing our work as generically as possible, to allow maximum flexibility as to using the underlying API.

*MapTube also uses OpenLayers as an alternative, although this is fairly well hidden.

…is one that makes possible something you’ve always wished *you* could do.

It’s a philosophy that I’m using when planning what GEMMA will do. I want it to things I want to do, and if it does those, and other people enjoy doing those things too, then it’s a win all around.

Three seemingly disparate statements:

  • MapTube and SurveyMapper are great CASA products, made by Richard Milton and Steven Gray (the former who’s partly involved, and the latter fully involved, in GEMMA, by the way.) I’d love to use these visualisations, in reports and presentations, more.
  • OpenStreetMap’s a fascinating dataset, collection of APIs and associated services. We are only starting to explore the amazing things we can do with it. But sometimes I just want a simple map, showing my points of interest and a suitable background, just the way I like it. For example, where are all the Tesco Expresses in east London?
  • I love seeing all those Boris Bikes whizzing through the streets of London. I wonder how many there are compared to those regular cyclists. Are they taking over London? I would love to have a map showing, for several key locations within and without the scheme in central London, the % of bikes passing that are Boris Bikes, and I would love a dead easy way to collect the data, using my iPhone and its built in geolocation, while sitting in a coffee shop or a nice park.

I’d love GEMMA to work with these three ideas. I’d love to be able to produce bespoke maps showing just the pubs surrounding the office. I’d love to drop my Olympics tickets SurveyMapper survey – maybe focused on just the London area – in a PDF report. And I’d love to find out just how many Boris Bikes zoom past my office, or across Blackfriars Bridge.

These are three “use cases” of what we hope GEMMA will allow to happen. A service where I can access the OpenStreetMap data quickly and produce a PDF showing a particular local feature or the results of a recent survey. And wouldn’t be great if I could even build up the data for that survey, with a mobile app, there and then?

I want to advance all three of these ideas, and so will be building GEMMA with them in mind.

Sneak Preview

GEMMA is still very much in stealth mode, however we will be presenting a very early glimpse of one of the application’s workflows at the forthcoming Wherecamp EU unconference, in Berlin next week. We’ll also be showing more functionality at State of the Map EU, which is in Vienna in mid-July. The OpenStreetMap dataset will be a key part of the GEMMA system, and it will be great to present at the project’s main European conference this year.

We have set up an announcement list so you can be the first to hear about GEMMA developments. We envisage this will be a very low volume email list. We’ll email you when GEMMA launches, and we may also occasionally email about developments in GEMMA leading up to the launch.

You can sign up for the announcement list by following the link on the main GEMMA page.