GSoC 2008 Project - Situation Awareness / GIS enhancements

This page is for documenting the work done in GSoC 2008 on the Situation Awareness Module.

This project intends to build up the existing Sahana Situation Mapping Module by providing a number of geo-spatial type libraries that will represent objects, areas, and routes in a disaster situation. On top of this, an interface for users to create and save their own objects will be added. The viewing system will have enhanced filtering options to give each team an uncluttered view of any situation. Extra features for this would be added to the developing analysis GIS module. Lastly the project will look at ways in which other modules in the Sahana Management System can use this data to best effect, including the ability to create new geo-spatial features in addition to improving the currently implemented nodes, these will represent new disaster types such as:

1. Disaster Routes – might represent supply routes and new escape paths (lines)
2. Disaster Areas – might represent an area affected by fire or flood (polygons)

Each of these new types will have a number of user defined variable attributes associated with them, for example: population of area, supplies assigned, population density, and images etc. If no one else is assigned I would also like to work on the GIS analysis section to incorporate new features involved with the above. In doing this I would like to improve the HCI (human-computer interaction) processes involved in creating these disaster types to make the processes more straightforward. After completing this I would like to improve the ability to filter this data so that a comprehensive map could be built up of the whole effected area which could be then tailored to each of the response team's needs.

• This project intends to build up the existing Sahana Situation Mapping Module by providing a number of geo-spatial type libraries that will represent objects, areas, and routes in an disaster situation.

This aim has been met by the redesign and implementation of the internal GIS DB, with the inclusion of the “GIS_feature_class” table. Users can now add their own types, pre install by customising the “mysql-config.sql” file or alternatively through the user interface in the Administration→GIS Configuration→GIS Feature Class Management section of Sahana. Through this user interface Feature Classes can be created, deleted and edited. The underlying functions to this are all available through the new GIS API which I have built (LINK) so that other modules can easily take advantage of these features if they wish.

Currently there are 65 pre installed features to choose from in the system, 33 of which are directly linked into other modules.

• On top of this, an interface for users to create and save their own objects will be added.

See section above on adding your own features Administration→GIS Configuration→GIS Feature Class Management

• The viewing system will have enhanced filtering options to give each team a uncluttered view of any situation. Extra features for this would be added to the developing analysis GIS module.

Currently users can drill down using only “features within radius”. Although all the underlying code exits to drill down by Feature Class and Layer (see below for explanation) the GUI code is not yet complete and has been left out of the 11/09/08 submission. This functionality should be made available sometime during the following week.

The Mapping GUI has had significant improvements in usability, as users can now create, delete, edit position and metadata of points, lines and polygons within the map itself with no need for refreshes. This is a great improvement on the previous ability which was only to view points only.

• Lastly the project will look at ways in which other modules in the Sahana Management System can use this data to best affect.

Including the ability to create new geo-spatial features in addition to improving the currently implemented nodes, these will represent new disaster types such as:

1. Disaster Routes – might represent supply routes and new escape paths (lines)
2. Disaster Areas – might represent an area affected by fire or flood (polygons)

Every item in Sahana that added its location to the GIS (Pre GSoC) has a feature class associated with it in the feature class table. Also there are 3 fields (url_view, url_edit and url_del) available to every item that is added by another module to the GIS. These can be used to give a reference back to the module should the user wish to manipulate them while using the analysis map. Each feature provides a popup link to each of these locations, if a module has created a feature in the GIS but not registered one of these links a popup warning is presented to the user informing them that they cannot delete a feature in the GIS map view but instead must navigate to the appropriate module instead. Thus there should be no way a user can introduce concurrency issues between items in a module and features, representing them in the GIS. (The module owner could perhaps forget to delete them from the GIS when they are deleted from their module, but there is little the GIS can do about that… although I have had the idea of creating a GIS sync function that checks every feature in its tables and checks there is an equivalent feature with that uuid somewhere else in Sahana).

• Each of these new types will have a number of user defined variable attributes associated with them, for example: population of area, supplies assigned, population density, and images etc. If no one else is assigned I would also like to work on the GIS analysis section to incorporate new features involved with the above.

In the redesign of the internal GIS database it became apparent that it would be unfeasible to have user determined fields. As it would be too hard to display every possible variation in an uncomplicated and user friendly manor. Instead the GIS API has been written in such a way that adding extra fields (or keys) will not affect how other modules already access the GIS. Thus it would be relatively easy for an on site modification to add extra fields in the database and then update the API as necessary (with no affect to other modules). For the time being a number of generic fields are made available. Such as name, description, date, address, author, url. Also each item in the database can be linked to another item in another table. Therefore any extra data about a feature can be added to another table (eg module specific) and then linked though uuid references in the GIS Database.

• In doing this I would like to improve the HCI (human-computer interaction) processes involved in creating these disaster types to make the processes more straightforward.

The Situation Awareness modules underlying openlayers code has been largely re-written to use feature-vector layers instead of marker layers. This is due to the fact that feature vector layers allow other geometry types to be used on them such as lines and polygons where as marker layers do not. Also features layers allow more controls to be added for things like moving, adding and manipulating geographies displayed on the screen. As such a user can now add, reposition, (edit details where applicable) and delete features all within one map. Every action is synced with the internal database using ajax so that users need not refresh pages. Users can also use a revamped step by step page sequence to add features that allows more fine grained options to be selected (feature class, layers).

• After completing this I would like to improve the ability to filter this data so that a comprehensive map could be built up of the whole effected area which could be then tailored to each of the response team's needs.

Also in addition to the Feature Class assigned to every object in Sahana users can now add features to “Layers” these (pending next weeks addition) will can be displaying in the Situation Awareness maps, feeds and file downloads. Both individual Features and Feature Classes can be added to a layer to make selecting large numbers of Features as easy as possible.

Terms - internal GIS = gis mySQL database

Above I have explained how i have fulfilled the brief that was outlined for me at the begging of the project. Below i will go through every addition to the Sahana codebase i have made with the aim of providing a more comprehensive overview of my work.

Links:
GIS API

Files:

/inc/lib_gis/gis_fns.inc [2.2kloc]

After beginning work on the high levels of Sahana it became aparent that a lower level API for both inputing and retrieving data from the internal GIS would be very useful for Sahana for the following reasons:

1. As many modules where accessing the GIS though raw SQL query's,this was subsequently making it almost impossible to change the databases structure without need for heavy re-factoring.
2. To reduce complexity for external module developers, who should not need to have to worry about understanding the setup of the database.
3. To reduce the number of errors and concurrency issues from data being added, changed, removed when it shouldn't by external module developers. (who don't fully understand the setup of the database ;)

When I begin work on this work had already begun and a number of methods existed for entering data into the database, but none to retrieve it. After a redesign of the main GIS database (see below) I set about providing methods to retrieve, input and edit the new table design. Currently i have created roughly 24 functions for directly accessing the database and a further 10 or 12 supporting functions for doing other tasks not directly related to enter/retrieving data. This can be seen in the /inc/lib_gis/gis_fns.inc file. Further documentation can be seen on the GIS API page.

Links:
GIS Structure

After deciding to implement a full API for the GIS database i begin to analyse the current tables that made up the GIS. After some deliberation i decided this would be the best time to redesign the layout and contents of the tables. I choose to do this because up to that point the GIS table structure had expanded on an as needed basis and as a result there was some ambiguity and repetition within the tables. As i was writing a new API mainly from scratch it also made little difference if we where using the old setup or a new one save the time taken to design it. The new design for the database can be seen on the GIS Structure page.

This design made it far easier for me to implement the idea of “type libraries” (Feature Types) that i laid out in my brief. On top of the idea of Feature Types i have also introduced the notion of layers within the GIS database. (not to be confused with Openlayer layers). The idea being that a user can create a layer and then add any number of feature types and/or features to it. This can then be used to narrow down search results and map reviews, as well as being made available for download (in various file formats) and/or publishing to geoRSS, KML or GML feeds. As this was not a primary aim of GSoC i have not completed all GUIs for the layers functionality. This is something i aim to add to the project in the near future. This should not be too difficult as the underlying API functions have been implemented and are working.

In summery the new GIS Database now supports:

• Points
• Lines
• Polygons
• 3D Coordinate system.
• Projections (see Proposals)
• Feature Types & Associated Icons / Display Styles
• Layers
• Meta data Associated with each Feature (name, desc etc)
• Links to associated external module items for viewing, editing and deleting.

Files:

/inc/lib_gis/plugins/openlayers/handler_openlayers.inc [0.3kloc] 
/inc/lib_gis/plugins/openlayers/openlayers_fns.inc [2.1kloc (only about 2/3 mine)]
/mod/sm/wm_fns.inc [0.8kloc]

I have redesigned the openlayers mapping js code to use the vector layers instead of marker layers. The there were a number of reasons for this recode:

1. Vector Layers use geomotrys and as such suport points, lines and polygons, whereas markers layers only support the marker icons themselfs.
2. Openlayers “Controls” can be added to vector layers to allow the implementation of selecting, draging and manipulating of features within a vector layer. Markers layers do not suport this.
3. Features in a vector layer can be styled with icons and a number of other features whereas markers layers can only be styled with markers.

As a result of this refactor the situation main map now supports the Addition, Editing (both location and meta-details) and Deletion of “Default Feature Class”, Feature (Points, Lines and Polygons). This can all be done within the map using popups and is all handled asynchronously with the database using AJAX so that no refreshes are needed between manipulated features and the GIS database being updated. Also a number of features have been been added to stop users from editing/deleting Features belonging to other modules. If a user trys to delete a feature that belongs to another module they are informed and if available directed to the url at which they can perform their desired action. If the feature has been added in the Situation Awareness module it can be deleted right there.

The add_marker_map that other modules use for selecting the location of feature to add to the database has also been updated so that it now supports points, lines and polgons. Currently no modules useing this map (save the page by page add feature in Situation Awareness) have the ability to save features as lines and polygons but it is a relativity simple matter to re-factor them to do so.

Currently the Situation Awareness module does not have the ability to drill down features by Feature Class Or Layer but as all other functions are in place it should only take a couple of hours coding to add this functionality (I didn't want to break anything by adding new features too close to the deadline).

Files

/inc/lib_gis/admin_feature_class.inc [1kloc]

I have implemented an interface to create, delete and edit Feature Classes within the system, it can be found at “Administration→GIS Configuration→GIS Feature Class Management”. Here users can choose the icon, name and description of a class as well as which layers it belongs to.

Files

inc/lib_gis/admin_internal_layer.inc [0.3kloc]

An interface for creating, deleting and editing layers. This is something i have not had time to finish. Currently layers can be defined on install through the “mysql-config.sql” file

Files:

/inc/lib_gis/lib_gis_file_factory_download.inc [1kloc]
/inc/lib_gis/lib_gis_file_factory_upload.inc [0.4kloc]
/inc/lib_gis/admin_files.inc [0.8kloc]

Although not an initial aim of my brief it soon became apparent that the ability to selectively upload/download features from the GIS to an external machine would be very useful. At the beginning of my project the GPS module provided support for this though a custom gps table. As the new GIS design saw the assimilation of this into its main features table i decided to add and improve the functionality of this feature. On inspection the GPS module provided the ability to up/download features in GPX format. The actual parsing of data from the database to GPX format occurred within the same section of code that was printing the data to the web. As i wished to Add to the number of supported file types i refactored this into 2 new files:

• /inc/lib_gis/lib_gis_file_factory_download.inc
• /inc/lib_gis/lib_gis_file_factory_upload.inc

Firstly i upgraded the original GPX parsing code to support the new fields from the GIS. Secondly i added support for KML, GML and GeoRSS file types. By providing this new functionality in separate abstracted object files i have allowed other areas of Sahana to take advantage of its feature, rather than just the GUI that existed in the GPS module.

After finishing the lower level functions i redesigned and refactored the interface to the “Administration→GIS Configuration→GIS File Management” Section of Sahana. (This can be seen in /inc/lib_gis/admin_files.inc). Thus making the GPS module now redundant and ready for removal.

As This was not a original aim of my brief there as some improvements (mainly testing) that this section could do with before being ready to release.

Files

inc/lib_gis/admin_feeds.inc [0.5kloc]

As the completion of lib_gis_file_factory_download.inc saw the ability to create GeoRSS feeds from any internal features i decided to create a custom interface for publishing feeds This can be seen at inc/lib_gis/admin_feeds.inc .

This interface provides the options to create feeds by narrowing down features in the database then selecting the appropriate features and finally selecting the type of feed they wish.

Future expansions to this section will be the ability to:

• Publishing of Feature Types by selection a checkbox.
• Publishing of Layers by selection a checkbox.
• Feeds that update when the GIS db updates.

Note: Publishing of a specific Feature Type or Layer is already possible using the standard Feed Publish search but adding the option to do it from a single check box would improve usability. Some work on this has already been implemented but is unfinished.

Files

inc/lib_gis/lib_gis_forms.inc [0.7kloc]
mod/xst/handler_gis.inc [0.2kloc]

I have created a file (“inc/lib_gis/lib_gis_forms.inc”) for storing command forms used within the gis. I hope to move all methords for creating maps and any other display related methods from gis_fns.inc to this file in the near future.

I have also created number of AJAX functions for which responce is generated in the “mod/xst/handler_gis.inc” file. This will all need refactoring to use the new AJAX module once it is finished!

key HCI - Human Computer Interaction.

All Graphical interfaces i have designed during my GSoC contribution can be navigated in both directions. This means that should a user enter incorrect data in a form and only realise later on they can now retrace their steps through the back button instead of having to restart the form from scratch. Each form that is entered is stored in a SESSION var and is uploaded to the form on revising that page reducing the need to refill forms.

All pages have the option to cancel progress which returns users sink state that removes any entered form values from the SESSION and also adds the ability to navigate to their original position(see below).

Each graphical interface i have added is controlled by one function eg shn_gis_internal_class_add_seq($return_action, $return_module) to activate this form sequence all another module has to do is call this function once. Once called all processing through this functionality is self contained and if the $return_action and $return_module parameters are entered the system will return the user to wherever the developer wishes on completing the form process.

To make the use of Sahana more intuitive i have created a new form in lib_form.inc

shn_form_progress_bar($steps, $current_step)

This takes an array of steps and a current step number:

$steps = array('Feature Search', 'Feature Selection', 'Download & Extra Options');
$current_step = 2;
shn_form_progress_bar($steps, $current_step);

Produces

Green indicates a step that has been completed.
Dark Gray indicates the step the user is currently on.
Light Gay indicates a step the user has yet to complete.

To give the user interface a more Standardized feel i have created an extra function in lib_form.inc

shn_form_submit_set($button_set)

This function prints button sets at bottom of a page in the order of cancel(0..1), back(0..1), next(0..1) | finish(0..1) | confirm(0..1), custom_button(0..*)

Example Input:
  Normal: 
    $button_set = array('cancel' => true, 'back' => true, 'next' => true);

  Advanced:
    $button_set = array('cancel' => true, 'back' => true, 'next' => true, 
                   'custom' => array(
                      array('value' => 'button text1', 'submit_opts' => 'name = "post value1"'),
                      array('value' => 'button text2', 'submit_opts' => 'name = "post value2"')
                                    )
                       );

Links:
Module Integration Guide

Roughly half way through GSoC 2008 I refactored all the modules using GIS to use the new API. Since then there have been a number of small changes and additions that will mean extra refactoring will be needed to complete the integration. I have not chosen to do this in the GSoC branch as it would much better to make the changes if/when the GSoC branch is merged with the Main trunk.

To help future refactoring I have written a Module Integration Guide.

Note: While refactoring I noticed a number of modules do not have delete or edit functionality for even their own items!!! This is something that should be taken care of asap.

As gis_fns is getting a bit big for its boots (well over 2k now) I suggest it be split into 3 separate files:

1 - lib_gis_db_api.inc

Functions just related to accessing the internal Sahana mysql db (currently over 1.5k lines)

2 - lib_gis_forms.inc

Forms for generating maps, query, drop downs solely related to the GIS.

3 - lib_gis_gen_fns.inc

Other general functions that do not come under the above 2 cats but are related to the GIS.

Currently the GIS database supports a projection field. This is currently largely unused for reasons identified below:

1. If any other projection is used all GIS coord form vaildataion will fail (or need to be updated to take into account the new projection)
2. If any other projection is used all openlayers map projections will currently fail

I propose that the GIS db use only one standard projection type. Data that is coming into the database in another projection should be transformed before it is entered into the tables.

Ontop of this data being uploaded from GML, KML, GPX files currently needs to be in the standard form otherwise it will probably be displayed incorrectly in maps…

In short projections are a particular thorny area that need a lot more time and thought than i had available on GSoC.

• an ajax popup for selecting icons for points on map. Below are some concept designs i have made (using msn icons as they are about the right size)

• straight popup

{{wiki:popup.jpg}}

• helper popup

{{wiki:helppopup.jpg}}

• combined (clicking + on description would open slide down a box)

{{wiki:both.png}}

having thought about it a bit more i think that instead of having a side open description box the description could be overlayed on top of recent icons (if we even think a description is necessary)