This release has the following new features

• New Menu Bar
• Export of media files
• Improvements to Task Management interface
• User management styled with Bootstrap
• Form Builder Interface Modified
• Ability to add media at an organisation level so that it can be attached to questions or choices in any form
• Performance improvements
• Added support for select multiple questions where the choices are sourced from an external CSV file.
• Forwarding, which was broken in release 14.12, is now working again
• The “from” address and “subject” in a notification email can be specified.
• The form name will default to the file name when you upload a form
• Addition of Spanish Translation
• Updates to French Translation

Menus

The Smap server has been separated into 4 modules:

• Home
• Edit
• Admin
• Web Forms
• Analysis

“Forms” page of “Admin” module with modules menu expanded

These modules can be accessed from any page by selecting the “Modules” menu.  To the right of the modules menu will be the menu items to select a page within the current module.

Note there are still some pages that have not been updated with these new menus.

Export of Media Files 

Simply select Export from the Analysis page then select your survey and the export type of “Media”.  You then need to select the question that contains you media.  By default each file will be named with the record number in which it was submitted.  However you optional choice one or more other questions, the results of which will be combined to create the name of the media file.

Options for exporting images to a zip file

The media will be downloaded in a zip file.

Form Builder Interface Modified

The form builder now allows you to edit question text and media for any language in the form.

Adding media to choices using the new form builder

More details can be found in the training documentation: https://docs.google.com/file/d/0B5_SmpWlQYxvaDVYUGdtNU9MZmc/edit

Task Management

The user interface for the task management page was changed to use “Bootstrap”.  Hence it looks different.  However the behaviour of the page also changed quite significantly.

Task Management Page

These changes are:

• Tasks located at location 0,0 are no longer shown on the map. They are assumed to be tasks without location.
• You can add ad-hoc tasks to existing task groups.
• Ad-hoc tasks can be created without specifying a location
• Tasks selected on the map are also selected in the tables and vice versa

For more information on using the updated tasks interface refer to the training material in https://docs.google.com/file/d/0B5_SmpWlQYxvbHBCLW0ySkw2Z3M/edit.

Automatic Deleting of Submitted Results

The server can be configured to request that the phone automatically delete successfully submitted surveys.  This improves security on the phone as data is much less likely to fall into the wrong hands if, for instance, the phone is lost.

It is strongly recommended that automatic deletion be enabled.  Currently it is off by default as I don’t want to surprise anyone with the changed behaviour.  However in a future release automatic deletion may be made the default.

To enable automatic deletion edit your organisation and select “Delete submitted results from the phone”.  Only an organisational administrator can do this. Automatic deletion currently will only happen when you use “Refresh” to submit your data.  

Also if you configure the settings in fieldTask to send results automatically then the “refresh” method will be invoked, new form versions downloaded and submitted results deleted.

Recording GPS Trails

Recording of GPS trails for each user can be enabled by selecting a checkbox in the organisation dialog. Its the same dialog that can be used to specify automatic deletion of submitted results.

The path taken by the user will then be visible on the phone in the Maps tab, along with each location where they completed a survey.  These survey locations are recorded even if there were no location questions in the survey form.

Map on Phone – One task has been rejected, one has been completed

When the fieldTask user presses the refresh button the GPS trail will be sent to the server along with any completed survey results.

The following video shows how the analyst or administrator can review the paths taken by the data collectors.

Privacy Issues

There are potential privacy issues with recording the path taken by the phone users.  For this reason this function is off by default and has to be enabled on the server.  It is advised that you contact the developers to discuss how you might use this feature and what additional privacy measures might need to be included prior to enabling.  Current privacy controls are limited to:

• The user can switch off GPS
• The user can switch off recording by going to settings in fieldTask. However it will be re-enabled each time they do a refresh (assuming the server has be set to record the GPS trail)
• Recording of GPS set to “on” is shown in fieldTask settings
• The fieldTask map will show the trail of GPS points that will be sent to the server

Miscellaneous

1. The obsolete Nutiteq libraries used for showing maps on fieldTask have been replaced by the MapBox API.  This is a lot more pleasant to look at and also will support future functionality such as offline and custom maps.
2. Added translations codes to the Analysis page. Thank you to Baba Sy and Telmo Peixinho for the French and Portugese translations.
3. Reduced each row on the task list in fieldTask to only have two lines. The appearance is now less crowded.  The forms version number has been put on the same line as the form name and abbreviated to (v: NN)
4. Reporting of errors in template upload has been improved. These improvements include, checks for read only questions that are mandatory, checks for constraints that don’t refer to the current question, a reduction in unnecessary repetition when reporting on invalid functions.

This is the second release in October and contains some more bug fixes as well as some new features that allows users to set and reset their own passwords when they are first added to the system or have forgotten their password.

This release is available for download at http://www.smap.com.au/downloads.shtml.

Adding a new user without specifying a password

This feature requires an email server to be specified in the Tomcat configuration.  Refer to the operations manual for details if you are running your own Smap server.  I am using Google smtp server for my hosted servers however if you are running your own Smap server you can set up your own email server.

When you select “Add User” in the user management screen the default setting will be to send an email to the user. The administrator will not need to specify the password.

Send email to user specified by default

The user will then get an email with a link that they can select to set their password.  The email will also inform them of the URL of the server and of the user name they can use to log on with. The link will be valid for 48 hours.  After that the user will need to request a new link by selecting “Forgot Password” on the home screen.

When creating the new user, you can select “Specify password” if you want to set the user’s password as in the case of a data collector.  You should do this if the data collector’s ident is shared by several people.

Forgot Password

If you forget your password you can now reset it yourself, assuming that you have your correct email address recorded against your user account.

1. Select “Forgot Password” menu option.
2. Enter your email address and submit
3. When you get the email, click on the link and enter your new password

You should now be able to logon with your new password.  The link sent in the email will expire after 1 hour.

Uploading multiple large attachments

This release fixes a significant bug whereby if the total size of attachments in a survey exceeded 10MB then some attachments were not processed correctly after being uploaded and could not be seen in the analysis pages.

odk splits large attachments into separate uploads.  Smap has now been updated to process these and merge then into the main upload.  You may see this on the monitor page.

In the above screen shot from the monitor you can see that 4 surveys have been uploaded for media2 and 13 uploads have been merged into these 4 surveys.  In this template there were 4 video questions, the videos were recorded in high definition and each survey had around 60MB of data.

There were also problems with the server configuration which caused “unauthorised” responses to be sent if uploads of large attachments exceeded 5 minutes. This has also been fixed by removing the timeout on the digest authentication for submissions:

               AuthDigestNonceLifetime -1

Miscellaneous

1. Added Translations for Form Management page. Thank you to Baba Sy for providing the French translations.
2. FieldTask has been updated so that it only shows the forms valid for the current server.  If you change the server in the settings screen then the forms valid for that different server will be shown instead.
3. When getting a list of forms to download only those forms from your current organisation will now be shown.  This change only affects users who have access to more than 1 organisation.
4. When editing survey results using web forms, the dialog with the “edit” button has been removed. If you select the button at the left of the survey record the web form will be opened immediately populated with the existing data.
5. Fixed a bug where existing data was not adding to a webform if the survey contained questions with names that are reserved for use by the database such as “end”

Summary of changes

• You can edit question and option text directly in the server.  This includes the ability to edit translations. This change is the first stage of creating a complete editor for Smap.  To access it click on the form name in the template management page.  Then select the “view” and then “translations” menu option to start editing. When you save changes the version of the form will be updated.  This version will be returned with the data in each submission and is included in the export of results.  You can see what changes were applied to each version by selecting “view” and then “changes” on the edit page. This feature should be considered a “beta”.  If you want to record the version of the template with each data submission you will need to get version 3.4 of fieldTask from the server.
• Reporting of errors in surveys defined by xlsform have been made clearer.  More information is now provided to assist in getting help in fixing the errors.
• Support for presenting the server in multiple languages has been added.  Much more needs to be done to complete translations but the basic capability is there.  If you are interested in contributing to a translation then contact neilpenman@gmail.com.
• Exporting of surveys to Stata.  This export creates a zip file with a csv containing the data and a “do” file which can be used to add labels to the data in stata.  This video shows how it works: https://www.youtube.com/watch?v=wnhR9JPw6NY
• Read only questions are no longer included in exports as a default.  There is a chenckbox you can select to include them.
• FieldTask has been upgraded to use ODK1.4.3
• External csv files can now be added to a survey. These can contain data from previous surveys or lists of options.
• You can no longer forward a survey to itself. That prevents the survey forwarding going into an infinite loop.
• Forms can be moved to a different project

I’ve updated the downloads page to point to version 14.05.02 of Smap.  This contains a minor bug fix to address a problem where the individual coordinates of a polygon were not being shown in the whole of survey table view.  The overall polygon was shown.

More importantly I’ve added some instructions in the README file on changing Apache configuration files if you are doing an upgrade to your own server. There have been some changes to these files in the latest release and the deploy script does not apply these changes as system configuration files are only modified on install.

The files affected are:

• /etc/apache2/sites-available/default
• /etc/apache2/sites-available/default-ssl

You can update these by replacing them with the default and default-ssl files from smap/install/config.  However these files need to be updated to set the location of your attachments on the file system.  Everywhere in these files that you see {your_files} replace that with the correct file location, by default this should be /smap.

If you get an error message ‘Invalid command Header’ when starting Apache then you will need to type “sudo a2enmod headers”. That only needs to be done once then Apache will have the module that processes headers.

If you have any problems with this upgrade contact smap support.

Announcing the release of version 14.05. If you are installing or upgrading you can download from http://www.smap.com.au/downloads.shtml.  I’ll be rolling this release out to other servers I manage over the next week.

New Features

Humanitarian Openstreetmap Layer (HOT)

You can make HOT the background map on a map view by selecting the expand button labelled as “<“. You will then see a list of the available map backgrounds including openstreetmaps, google maps, google satellite and now HOT.

HOT Map Background Layer

The data on the map is the same as standard Openstreetmap however it is styled to show features in a way that is more relevant to aid workers.

Webforms

These has been updated to use the latest version of Enketo core which includes new features for collecting lines and shapes.

You can also add an appearance of “likert” to a select_one question to get the options displayed on a Likert scale.

Although webforms are now fairly mature I would not recommend their use in large critical offline surveys just yet.  In recent testing there were issues with slow response to user actions on even quite small surveys and in some browsers on some devices there were crashes when accessing location.  However feel free to try then out and share your experiences.

Results Filter

You can specify a filter on results.  For example as well as say, requesting a graph of problems by region, you can also specify that only results from women are returned.  Currently this where filter only works on select_one, select_multiple and text questions. However it will be extended to other types of questions in a future release.

Settings Popup

If you click on the “Filter By Value” button on the settings popup, then you will be able to select the question to filter by.  Unfortunately at the moment you can select any question even though only the select_one, select_multiple questions and text will work.

Once you have set a filter on an analysis panel / view then the “filter By” Value” button will be green and the filtering will be included in the results description.  To remove the filter select the “Filter By Value” button and press “clear”.

Filter popup

Miscelaneous Changes to Analysis

• Paging of results. The forward and back buttons no longer scroll with the results so you don’t need to scroll all the way back to the beginning in order to select these buttons.
• Total function. Previously if you had a numeric question and you wanted to see results grouped against another question such as region, then you could only get the “average” value per region.  Now you can get the “total”. For example the total area of all the farm plots by region.
• Specifying units in server side calculations. You can now specify units in server side functions. An area function can return results in Hectares or meters squared.  A length function in kilometers or meters. These units are shown on graph labels. Because you can now select the units the number of places after the decimal point has been limited to 2.
• “Bad” records are no longer shown in Map and Media views.  Previously if you looked at the whole of survey results then you would see the records that had been marked “bad” on the map.  If you want to review bad records the only way to do that is to create a table view for the whole of the survey, that is do not select a question.
• The default title of an analysis panel is now set from the survey name if the user does not provide a title.
• Analysis panels shown in expanded view are, nearly, full screen.  The only part of the web page that can still be seen, apart from the expanded panel, is the main menu across the top.
• This button on the analysis page, has been replaced by 4 buttons each with a picture of the type of view / panel that is to be created; Graph, Map, Table and Media.

CSV Exports

CSV Exports have been replaced by a new function that is much faster.  Its faster both to export the data and to download it as the csv file is zipped before downloading. It is highly recommended that you use this instead of the XLS export when you have large numbers of results.  The main restrictions compared to exporting as XLS are

• Some formatting is not performed
• Only a single form can be selected and this form will be exported along with its parent and so on up to the top level form. So you can’t export data from more than 1 form at the same level in the hierarchy at the same time.
• You can’t specify that a form will be exported as pivot or flat.  Effectively all the forms are exported as pivot and the number of records will be equal to the number of records in the selected form.

The above sounds more complex than it is.   If you try it then it should be obvious what is happening.

Exporting Surveys with Polygons and Lines

When exporting results of a polygon or line to excel the individual points are no longer included by default.  You can add them in if you like but this can make the exported spreadsheet large as in a pivot export an extra row is created per point.

Changes to Installation of the Server

• Database upgraded to Postgresql 9.3
• Support for Ubuntu 14.04 added
• The need to set server IP address during install has been removed.  To install Smap on ubuntu you now just need to type ./install.sh into a terminal.  This change is in part to support installation on laptops where IP addresses change with each network that you connect to.

The Smap website has been updated to show what I am calling the “Smap Assessment Cycle”.

Smap Assessment Cycle

The 4 stages are:

1. Collect.
   Collection of data using fieldTask, odkCollect or WebForms
2. Assess.
   The analysis of data on the Smap Server where you can view graphs, maps and tables as well as perform data cleaning.
3. Publish (Internally or Externally).
   The export of data to a spreadsheet, GIS system, report for use in decision making. Publishing could be to another IT system such as program management, asset management, patient record system etc.
4. Act.
   Assign tasks to users. For Smap a task involves completion of a survey. However real world actions can be prompted by that survey. For example:

• Read only question, “Replace water pump”
• Image question, Take a picture of installed pump
• Barcode question, record barcode of pump

Demonstration

These stages can be performed in any order. In this demonstration the published data has come from an external asset management system, and the cycle starts from the Publish stage.

There is also no “Act” stage in the demonstration, refer to this post for more information.  Note this demonstration is using features that will be included in the 30th May Release of Smap including updates to the latest version of Enketo core.

Publish

The data has been published as map tiles using TileMill from MapBox. Tiles are used by google maps and most other online maps as they highly scalable. With TileMill you can style your maps to produce beautiful clear representations of the data that you want to show.

As well as the map images you can attach data to the maps. Hence if you select an item such as a road, or medical clinic, information on that item can be shown.

Collect

The data associated with each item can be used as a key to conduct a survey using Smap’s webforms. By including the data in the survey URL, if there is any existing survey results that match, then they will be downloaded and set as the initial survey values. If there are no matches then a blank form will be returned.

Webform

Analyse

The data can be analysed using the standard Smap views of Maps, Graphs, Tables and Media.

Analysis

History of Changes

If you have updated existing data then that data is replaced and the graphs will only show the latest view. However a complete history of changes is kept and this can be seen by opening a table view and not specifying a specific question.

History of changes

Just a small release this month due to the lateness of the March release.  The following bug fixes are included:

• There is an option to group totals of uploads in the monitor page by device, week or day.  This should have been disabled unless a single survey has been selected.  If the total uploads for the whole project are being shown then its not possible to group these totals.
• Addition of media such as photos to questions is now prevented if the question is not actually answerable by the user.  For example the survey start question and the device question are not shown to the user on the phone hence it is pointless adding a photo, video or audio to the question.
• A security issue that allowed a user from one organisation to view data uploaded by another organisation has been fixed.
• Upload of a template would fail if there were blank labels in some of the options in all of the languages.  This occured during a survey in Uganda when a partially completd survey was loaded for testing while the translations were being prepared.
• If the user is not authorised to submit a survey now they will get an error message and can resubmit the results when they are authorised to do so. Currently the surveys are loaded onto the server successfully but then an error is reported on the monitoring page and the results are not applied to the database.  This can mean that the uploads are missed.

You can download the server application from http://www.smap.com.au/downloads.shtml

This is a major release and if you are upgrading from a previous version I recommend that you backup your database first. Also if upgrading you should add the following directive to /etc/apache2/sites-available/default and /etc/apache2/sites-available/default-ssl.  This will make the fieldManager javascript files available to people accessing reports without requiring them to enter a password:

<Location /fieldManager/js>
allow from all
Satisfy any
</location>

This release has been tested with Ubuntu 12.04 LTS.  There may be some issues with web forms when using Ubuntu 14.04 or any other Ubuntu version that uses Apache 2.4.