Category Archives: Smap Release

Release announcements for Smap

Version 16.05

New Features

A better “download the form” dialog

When you select the download button next to a form name on the form management page you will see a dialog asking what you want to download.   You can often download either of two versions of the form template as an XLS file.

  • The original XLS file uploaded to the server
  • The latest version of the form including modifications that you have made using the editor on the server

These two choices are now a little clearer.

download

The first choice is now the latest version of the form.  Next to the choice is the current version number.  The second choice is now the original XLS file, however if no XLS file was loaded then this choice will not be shown.  The version number for the original XLS file will always be “1”.

Logon button on the home page

You can now login and also logout from the home page.  The menus that you will see on the home page change according to your role and whether or not you are logged in.

Improved Task Management (Hosted Servers Only)

A new improved task management page has been added.  This is only available on hosted Smap servers and won’t be available in the download version.  New features include:

  • Import / Export of tasks to an XLS file
  • Viewing tasks on a calendar

tasks

version 14-12 Notifications

Here is, depending upon your location in the world, the last release of Smap for 2014 or the first release for 2015. The major change in this release is the addition of email notifications when a survey is submitted.  However the menus at the top of each page are also being progressively updated which will probably be the most visible change for most people.

Menu and Style Changes

The “bootstrap” javascript library is being implemented to replace “jquery-ui” and “foundation”.  In this release the following pages have been updated:

  • Form Management
  • Notifications
  • Locations (User Trail)

Pages will look a bit different, here is the form management page:

Menus

Smap has been divided into 4 sections:

  1. Home
  2. Admin
  3. Web Forms
  4. Analysis

These are accessible from a drop down menu labelled “View”.

dd

Drop down “View” menu

In the image above the “Admin” option is greyed out as the view option was selected from “Form Management” which is part of administration.  In a future release another section called “Edit” will be added.

Within each section the menu options will appear to the right of the “view” menu.  For administration these options are:

  • Form Management
  • User Management
  • Notifications
  • Monitoring
  • Locations
  • Tasks

Each section will also have common menu options

  • help 
  • User Settings (an icon that looks like someone’s head and shoulders)

Unfortunately because this new menu layout is being implemented progressively you will find yourself jumping back and forth between the new and old menus as you navigate from page to page.  If you find this unbearable let me know and I will try and make all the pages consistent faster.

Notifications

A notifications page has been added which allows you to request that an email be sent every time a new result is submitted to the server.

Notification Page

The notification page above shows a single notification that will send an email to two email addresses (separated by commas) when a result is submitted from the “modify stress test” survey. The email will contain a link to a web form view of the submission.

Web Form View of a Submission – Accessed by link in email

A log of all the notification events can be seen on the monitor page.

Application of email notifications

These could be used for quality assurance.  A supervisor can set up a notification for each submitted result which they can then review.  A future enhancement may allow filtering so that the emails are sent conditionally dependent on value of the data. Suggestions welcome.

Forwards

Previously you could set up a survey to be forwarded onto another server on the form management page.  Now however, forwarding is considered a type of notification.  Hence you can set up forwarding on the notifications page.

Miscellaneous Changes

  • The email host can be set per organisation.  Previously an IT administrator had to set the address of your email server (smtp_host) in the tomcat web.xml file.  Now it can be set in the users page by editing the organisation details.  Note the value for smtp_host in web.xml is now ignored so you will need to update the organisation.  
  • A default value of smtp_host can be set for every organisation on the server.  This does have to be done by an it administrator in the survey_defintions database using a command like: insert into server (smtp_host) values (‘Your smtp host’);
  • When uploading a new form to the server from an XLS file,  the name of the form will default to the name of the file.  You can optionally change the form name.
  • Increased the width of the line showing the path taken by the user on the locations screen.
  • Previously you could not select a set of survey results for editing if that survey had been marked bad.  This restriction has been lifted. You are now only prevented from editing results if that record has been replaced by another one.
  • Performance improvement to “data cleaning”.  There were some issues in performance last month when multiple people were cleaning data in the same survey simultaneously. Database deadlocks were causing updates to hang which resulted in the server running out of database connections.  These deadlocks have been fixed.

 

 

Version 14.10 – Bug Fixes

This is a minor release with just some fairly minor bug fixes.  These include:

  • French translations have been added replacing the dummy words that I had previously used for testing.  Thank you to Baba Sy for providing these translations.
  • Show an error message if a template is uploaded when the user does not have access to any projects.  (Thank you to Janna Metzler for identifying this issue)
  • Prevent the editing of results that have been marked as “bad”.  For example if they have already been edited and replaced by another record.
  • Fixed several issues with roles.  It now works much better if a user is assigned a role of analyst but not administrator or vice versa.  A user with only enum rights can now also use webForms.
  • Replaced links to the old help pages with links to the google drive holding the documentation
  • Prevent a malformed template being created if you upload a completely empty template.  This now reports an error and does not create the template.  Previously it reported the error but still created a template that could not be run on fieldTask
  • Fix paging through results on the monitor page. Previously the page buttons were hidden.
  • Thumbnails were not being created for pictures with an extension of JPG (in capitals)
  • Support added for select multiple questions when exporting to Neo4J
  • Only allow a single select multiple question to be exported to a Neo4J node or relation
  • Prevent Neo4J labels from having spaces
  • Improve clarity of reporting errors in templates when using template upload.  Multiple redundant messages were previously created when calculate questions with lots of parameters had circular references.

The server can be downloaded from http://www.smap.com.au/downloads.shtml.

 

Version 14.09 – Form Refresh and Export to Neo4j

Version 14.09 of Smap is available for download.

Refresh – Synchronising phones with the server

The refresh button on fieldTask now does more than just clear the submitted results from the screen. It also now refreshes the forms that you have on your phone and submits any finalised results.  Note you will need to download fieldTask version 2.3.5 from your server on to your phone to use this new feature.

Pressing refresh will now:

        • Download new forms from the server that have been assigned to the user of the phone
        • Show available forms with a yellow clipboard icon
        • Update forms on the phone that have a new version
        • Delete forms from the phone for which the user no longer has access
        • Submit finalised results from the phone to the server
        • Notify the server of the list of forms and their version that the user has downloaded to their phone

You can select a form from the task list screen to complete a survey.  The name, version and project containing the form are shown next to the icon.

device-2014-09-20-090412

Phone with two forms

This new feature should simplify the training of the data collectors as the steps they need to download new forms, start a survey and submit results are all simpler.

Checking that data collectors have the correct version of a form

Sometimes it can happen that data collectors go out into the field with the wrong version of a form on  their phone.  At worst this may have an impact on quality of the collected data and at best you will need to do some messy combining of the data from the two forms prior to analysis. This new feature makes it much easier to check that all the data collectors have the correct version of the form.

From the monitor page select forms.  This will show the users, phones, downloaded forms and the versions of those forms that have been downloaded.  Any users / phones that have not downloaded a form or have an older version are highlighted.

monitor

Monitor Downloaded Forms

 Exporting results to Neo4J / Thingsat

You can now export results to the Neo4J graph database. This change was developed in collaboration with thingsat.  The following video shows how this can be done.

Other Changes

  1. You can edit collected data using web forms.  This capability is enabled by an administrator for your organisation.  By default it is off.  If you select a table showing data for the whole of survey then there will be a button next to each data record that will open the survey results in a web form. If the data is updated the old results will be marked as bad, and then the updated results added.
  2. When you attempt to submit results from a phone for a survey that has been deleted on the server then the error message will be “Not Found” rather than “Not Authorised”
  3. The error message shown, if you upload a survey template that has duplicate options, has been made clearer

Smap Version 14.03 Forwarding

Smap version 14.03 has finally been released.  It should have been available from the end of March. The next release should be a small one and hopefully we can get back to the end of month regular release date.

The major new feature is forwarding of surveys.  Other changes are improvements to monitoring, support for larger result sets (paging) and a more laptop friendly deployment.

Monitoring

Totals

When you select “Monitoring” by default you will be shown total counts for success, errors, duplicates and “not uploaded”.

monitor

Totals are only available in Table view.  In order to view locations of uploaded data you need to first select the source as “Instances” then select “show as map”.

Filtering

You can filter the uploads (or forwards) that you view in the monitor page by their status. Success, errors, duplicates and “not uploaded”.  Hence you can uncheck all statuses except for “Error” in order to see which uploads in your project or survey were reported as being in error.  When showing instances rather than totals the last 200 records are still shown.  However if you apply a filter to only show errors then it will be the last 200 errors that are shown.

Paging

Paging is now available in the Monitoring screen and for the survey table view in the dashboard.  200 records are shown by default but you can then select a forward / reverse button to move forward and back through the results. These buttons are located at the top of the table and are labelled “>>>” for forward and “<<<” for back.  They will be disabled unless there are additional pages available to view.

Total Result Records

In the dashboard, when you view all survey results as a table, then the total count of surveys will be shown as well as the count of “bad” records. This wil be a faster way of getting a count of all surveys loaded than having to export to a spreadsheet.

Deployment

Changing URL of the server

Previously it was assumed that the address of the server would not change.  This allowed the server to get away with storing the full url of all survey attachments in the database.  However this assumption is completely untenable if Smap Server is deployed on laptops.  The server URL can now be changed and all the attachments will still be accessible.

Simplified deployment

Almost all the editing of files that had to be done during the installation process is now not needed.  The one exception is that the IP address of the server still has to be set in the install script.   This is only required for webForms and will hopefully be removed soon.  So, assuming you are installing on a clean Ubuntu machine, the new installation process is:

  1. Download and extract the release
  2. change directory to smap/install
  3. Edit the install script to add the IP address (temporary)
  4. Type ./install.sh

This will also now run the deploy script so once it finishes you shoulld be able to log on and use the server.

This release will be added to the downloads page over the coming weekend.  There will be another blog post to announce its availability.

Bug Fixes

  1. Error in retrieving languages on the audit page fixed.  This bug prevented auditing.
  2. If the first record in survey had an invalid geometry then export to shape files would fail. An example of an invalid geometry is recording only one point when a polygon or line was requested.

 

Smap Version 14.01 Task Management

Announcing the release of Smap Evaluation Tools Server version 14.01.  This release includes

  • Task management: the ability to filter the surveys that are used as locations for new tasks
  • Task management: Printing of task allocations across multiple pages
  • Support for Internet Explorer 10+ as the browser
  • Improved reporting of errors when uploading templates

Task Management

Task Filtering

When you create new tasks from an existing survey you can now specify a filter on one of the questions in the existing survey.  Only survey locations that match that filter will be added to the new task group. Select the checkbox called “Filter tasks” to enable filtering.

tm1

Task Filtering

 

In the following example the previous survey recorded the paths of various cable types including copper and fibre.  However for the new task I only wanted the surveyor to go to the cables of type copper so I enabled filtering and selected the option of “copper”

tm2

Tasks filtered by cable type of copper

Printing Task Sheets

When you select the “Print” button to print a task sheet the tasks are now shown on more than one page.  This attached PDF shows this. Note the address parameters that have been added to help locate the location of each task.  In this case these include the age, ethnicity and a picture.

 

 

 

Smap Server version 13.12 with Time Series Analysis

Announcing the release of Smap Evaluation Tools Server version 13.12. The time series analysis functionality in this release has been funded by the Speed Evidence project. This is to assist in the long term response to Typhoon Haiyan.

New functionality

  1. Time Series Analysis.  Showing trends over time using graphs, maps and tables in the analysis module.
  2. Server side calculations.  There are only two functions available at the moment; area of a polygon and length of a line or the perimeter of a polygon.  However other functions could be added such as the calculation of Z-score anthropometric indicators.
  3. Export of attributes from parent forms to shape files.  If you select a child form containing a geometry and export to a shape file then all the attributes of the parent (up to the maximum of 255) will also be exported.
  4. Specification of default language.  The default language to show on the phone can now be specified by selecting “view/edit” link next to the survey name on the template management page.
  5. Support for Internet Explorer has been improved however it is still not supported, that is I haven’t tested this release against IE.

Availability

Hosted servers will be progressively updated over the next week or so.  The download for this release will be available by the 19th of January.

 

 

Release of Smap Shapefiles Export

Announcing the release Smap Evaluation Tools Server version 13.10. New functionality includes:

  1. Exporting surveys to ESRI shapefiles.
  2. Administration and analysis screens are a bit more mobile phone friendly.  With the this release you should be able to create and view analysis panels on your mobile phone now.  However this is work in progress.
  3. Surveys that have been blocked will no longer be available for download.  Submission of results from these surveys will continue to be prevented until the survey is un-blocked.
  4. Incomplete surveys are no longer included in results automatically.  These can be uploaded from the phones but they will be marked as “bad” surveys with a reason of “incomplete” by default.

Exporting Shapefiles

Individual forms within a survey can be exported as shapefiles and then imported into a GIS such as ESRI.

The Smap Server now supports exporting to XLS, CSV, Openstreetmap as well as shapefiles.

Blocking Surveys

Installing

You can download the server tar file from here.

 

Release of Smap HTML / iPhone Evaluation Tools

Announcing Smap Evaluation Tools Server version 13.09.  New functionality includes:

  1. Support for completing surveys on a web browser. Browsers need to support HTML5 and can be on a laptop, iPhone, Android or Windows Mobile .
  2. Automatic dropping of duplicate surveys.  Previously the duplicate would have been marked as “bad” so that it did not appear in exported results or graphs however these duplicates could still be seen in table views.
  3. Calculate questions are now shown in the exported spreadsheet in their correct position relative to other questions in the survey.  So if you add a question to calculate the age in months after a date of birth question, then age in months will appear after DOB in the exported results.
  4. If you explicitly specify a “start time”, “end time” or IMEI question in your survey then these are no longer automatically added by Smap as well

WebForm Evaluation Tool

webform

Screen shot of web form in Chrome browser on a  Mac laptop

Features

  • Uses the open source Enketo forms engine
  • (Should) work in any HTML5 capable browser
  • GPS coordinates
  • Multi- choice and single select questions
  • Integer and text questions
  • Uploading of photographs (on some browsers, those with better HTML5 support)
  • Repeating groups
  • Relevance
  • Works offline and saves results for submission later when there is a network connection

Issues

The web forms feature is really a beta release so you can expect to find some issues.  Report any you find and we will try to get them fixed in the next release.  Known issues include:

  1. Some odk 1.4 functions that depend on javarosa do not work.  For example the new date time functions.
  2. Surveys that have images in their questions or options will not work with webForms
  3. Large surveys may be slow particularly when downloading or expanding a conditional group
  4. Safari on iPhone has some additional restrictions:
  • File uploads are not supported on Safari / iPhone
  • Expansion of conditional groups that are made relevant based on responses to prior questions do not work

Phone Browsers

iphone

Situation Assessment using Safari on an iPhone

android

Situation Assessment using Chrome on Android

 

 

 

 

 

 

 

Applications for the web forms feature

The ability to complete forms on web browsers adds a lot of flexibility to your data collection. For example it is now possible to send out an email or tweet with a link to a survey that the recipient can complete on their browser.

Installation

This is straightforward and very similar to previous installation.  Updated instructions are in the README file.  If you have an existing smap installation you should be able to just run the webformsInstall script to update the platform to support webForms (MySQL and PHP).  You can then deploy the updated application code as before.  Contact me if you have any issues.

If you are installing your own server you can get it from here.

Deployment onto Smap Managed Servers

I will progressively deploy this release onto the servers managed by Smap Consulting over the next few days.  However the ability to complete surveys using WebForms may not suit everyone.  Hence I will disable this functionality by default on each server unless you request that it be switched on.

Acknowledgements

Web Forms uses the excellent Enketo toolkit. This toolkit has been integrated into Smap by the following undergraduate students from RMIT who have done an excellent job.

  • HeeSien Ooi
  • Kar Ming Clement Kam
  • Kiman Sze
  • Mohamad Bahaa Maarouf
  • Shishir Chawla
  • Teck We Kee

 

Release of fieldTask v3.0

Announcing fieldTask version 3.0. This release includes the changes in the odkCollect 1.4 release (thanks to the ODK team) along with the ability to send partially completed forms back to the server. You can download version 3 from google play and from the smap development server. It will be deployed onto other servers as they get upgraded.

ft3

fieldTask v3.0 Home Screen

1) Change Location. [Provided by ODK 1.4] There is a new ‘placement-map’ appearance for location questions.  If you use this “appearance” setting on a question of type “geopoint” then your current location will be shown on a map and you can do a long press on the map to change the position that will be recorded. The meaning of the 4 icons on the map is:

        1. Exit map without saving location
        2. Save location and exit map
        3. Place marker at current location
        4. Zoom to marker

location placement

2) New functions for date processing. [Provided by ODK1.4] Refer to the survey editing guideline and the example xls template.

3) Improved styling for phones with Android 3+. These will show an action bar with icons for some of the menu options. On smaller phones only one or two of the icons will be shown, the other menu options are accessible though the menu button.

icons

Android 2.2 phone menu showing all the icons. Android 3+ phones will show some icons on title bar, remaining menu options will be text when accessed through menu.

4) Access to question responses within repeating groups. [Provided by ODK 1.4].  This allows you to create one repeating group to get all the names of people in a household, then in a second repeating group you can ask detailed questions about each member of the household and refer to the name recorded for that householder. Refer to survey editing guideline and the example xls template.

5) Ability to defer constraint and “required” validation to the end of the survey. [Provided by ODK 1.4]. This can be enabled in the settings.

6) Sending partially completed forms back to the server.  If you have to save a survey half way through because it cannot be completed you can optionally send those results to the server.  After selecting “submit finished data” from the menu, press the menu button again and select “change view”. This will have two options “Show Complete Forms” and “Show Complete and Incomplete Forms”.  If you select the second option you will be able to see the saved incomplete forms and send them back to the server.

There will be an update to the server released by the 18th of October that will mark these incomplete surveys as “bad” with a reason of “incomplete”.  Hence they will not be included in exported spreadsheets or graphs unless the analyst marks them as “good”. If it is critical for you not to allow incomplete forms to be sent until the server is updated to mark them as bad by default, then defer use of fieldTask 3.0 until after the 18th of October.