Author Archives: Neil Penman

Coordinating response to a rapid onset emergency using Smap Server

Scenario

Multiple organisations are collaborating in responding to the emergency. Three of the tasks that might be required are illustrated in the following 3 videos (each less than a minute long).

Recording the plans for upcoming work including location and services to be provided
Coordinating actions with other organisations at a meeting
Assessing the effectiveness of interventions

Reporting on the plan

Letting other organisations, as well as people in your own organisation, know where you are operating and what service you will be providing at that location. So easy it can be done before you get your first coffee of the day. (well some people anyway!).

Note this video shows the use of an iPad application which hasn’t been released yet however the task can be performed just as easiliy in the iPad’s browser or any other browser for that matter.

Preparing for a meeting

Information from the plans of all collaborating organisations can be readily combined for reference at meetings.

Assessing Effectiveness

This is classic DME, however in the video the assessment data is combined with the planned interventions on a single map.

Version 15.04 – The real release

7 Replies

The April release of Smap is finally available. These release notes extend the previous notes for 15.04.

I had to add some more functionality which delayed its release a little, including:

Compressing select multiple questions when exporting to XLS
Authentication of emails sent as a notification of an event.

Compressing Select Multiple Questions

Smap Server has always exported the results of select multiple questions as 1 column per choice. Each column contains a “1” or a “0” to indicate whether or not the choice was selected. This is convenient for analysis but results in large unwieldily tables if your objective is to take the spreadsheet into a meeting and discuss its contents. Hence a new compress select multiple option has been added to the XLS export. This will put all the names of selected choices into a single column separated by a space.

Loading Select Multiple Values into a Survey from a CSV file

Having all the select multiple choices in a single column is also used to load select multiple values from a CSV file. This allows you to export a survey with compressed select multiple questions, modify the options in a CSV file and reload them back into the survey.

Authenticating Email Users

When setting up email notifications you can now specify the user name and password of the email user to send the emails. This should reduce the chance of email ending up in Spam and will allow you to store sent emails in the sent folder of that email account. Details are in the operations guide.

Bugs

Monitoring of notifications only showed notifications at the whole organisation level. These are now shown per project or per survey.

Update 15.04.02

Added more detail to the error message that is shown when you attempt to load tasks into a form that you have not explicitly marked as being allowed to get data from a CSV file. The error message now directs you to the settings dialog.
Added error message if there are no columns in the CSV file that match questions in the form. Previously this just resulted in a generic internal system error.
Added an error message if the CSV file to be loaded is not specified.
Fixed bug preventing loading of instance name data
Remove spaces at the beginning and end of values inside the CSV file. These spaces would cause image files in the ZIP file not to be found

Update 15.04.03

Fixed issue where tasks are not created from an existing survey if the locations of the existing survey are in a subform and the location is missing.

Update 15.04.04

Fixed issue with adding task addresses and filtering tasks.

version-15-04

2 Replies

Because the March version of Smap ran over time, its been a short month for the development team. Hence we have only added two functions in this release. These are:

the ability to pre-load survey results from a CSV file into a survey. These can then be used to assign tasks to users.
allowing users to complete their tasks using Web Forms as well as with fieldTask.

Tasks Refresher and Rationale for the new Features

Tasks can be created from a blank form or from existing survey results. In the second case, when the assigned user opens the task, it will be pre-populated with those results. Up until now you had to populate the initial data for these tasks by completing forms using either fieldTask or WebForms.

However you may have a list of assets that need to be inspected, or hospital clinics that need to be visited stored in a spreadsheet or an external system. In which case you can now load that spreadsheet directly into a form and then allocate tasks to people to go and complete those surveys. That is inspect the asset, visit and audit the hospital clinic etc.

Secondly using web forms you can assign tasks to people with laptops or iPhones as well as users with the Android app fieldTask.

Features

Data in the CSV can include text, integers, barcodes, GPS coordinates, images, video and audio.
Columns in the CSV file that do not match a question in the form are ignored
It is not necessary to include all of a forms questions in the CSV file
You can replace existing results or append.
Forms need to be marked, in the settings dialog of the editor, as accepting data uploaded from a file. This should reduce the chance of accidental loading of data to the wrong form.
You can load either a single csv file with just text. Alternatively you can load a zip file containing a CSV and images, video, audio files.
Tasks on Web Forms are shown as red buttons rather than the blue buttons for blank forms

Limitations

Polygon and line question types are not supported
The file must be CSV format and not excel format
The survey cannot be empty. That is you must have submitted at least one completed form prior to loading the tasks. You can use Web Forms for this and if this initial piece of data is not required it can be cleared when you load the tasks from the CSV file.
FieldTask does not show images from preloaded data. This is a high priority issue to fix in the next release.

Demonstration – Asset Inspection

Inspection Form

For this demonstration I created a form that included a text, image, select_one and geopoint question. I set the “name” question to be read only so it would not be updated during the survey. All of these except for the select_one question are going to be pre-populated from a CSV file.

This form was loaded to the server.

CSV File

The CSV file looks like the following image. Column headings should match the name of the question where you want to put the data. The exception is for geopoint (GPS coordinates) questions. If you have columns labelled “lat” for latitude and “lon” for longitude then the values from these columns will be loaded into the geopoint question.

The image files in the pic_asset column need to be added to a directory along with the CSV file. This directory should be zipped and then uploaded to the Server.

Uploading the CSV file

On the tasks page select “Load Tasks From File”. Then select the Form to load the data into. Indicate whether or not you want existing data replaced and select your CSV or ZIP file. Then press “Save”.

Assigning Tasks

You can then click on “Add Task Group” to assign the asset surveys to users.

Screen shot of tasks paging showing 3 tasks to inspect assets

Completing The Tasks

Using Web Forms

Web Forms Launcher screen showing tasks assigned to the user as red buttons

When you click on a task it will show in a web form.

Web form with data pre-populated from a CSV file

Note: the asset name question is a text question set to “read only”, hence the asset name from the CSV file cannot be edited. However the location and the picture can be replaced. The “health of asset” which was not in the CSV file can also be filled in.

Final results

After the user presses the submit button the updated information will be available on the server.

Results after one task completed

Note: You can see that the record loaded from CSV file has been replaced by the completed survey results. In this case the location and picture were not changed.

Bug Fixes

We only did one bug fix in this release which was a the request of Zarkman.

The menus on the Webforms page have been made consistent with other pages. That is the links to Admin, Analysis etc have been put into a “Modules” drop down menu.

version-15-03

2 Replies

This release is primarily about adding support for iPhones and iPads when they are using Web Forms. You can now take photos, and videos with your iPhones when completing smap surveys. I’d like to thank Zarkman for donating an iPhone to Smap Consulting in order to help with the testing of this new capability.

A user changing their settings on the Smap Server using the Zarkman donated iPhone

New Features

iPhone and iPad support

You can now complete surveys on your iPhone and iPad using WebForms. These surveys can included images, video and GPS coordinates.

If you conduct surveys that don’t take pictures then they can be submitted offline and will be stored until you have a network connection. However, when using an iPhone, if you take pictures as part of your survey then you will not be able to save the results as draft. Also if you are offline when you submit, then the results will not be stored to be submitted when you are back on line. You will need to stay in the web page until you get a network connection and then re-submit. You can open multiple pages in order to complete multiple surveys, with images and offline, while using an iPhone.

To complete a survey on your iPhone, use Safari to go to your server. Then select WebForms and finally select the big blue button for the survey you want to complete.

This capability was in large part enabled through upgrading to the latest version of enketo-core as well as being assisted by the sponsorship of Zarkman.

Grid layout for Web Forms

Thanks to the the new version of enketo-core you can layout your web forms in a grid.

An example grid layout from enketo running in web forms

To turn your survey into a web form grid you need to:

1. Add a settings sheet to your xlsform containing

style

theme-grid

2. Add w1, w2 etc to the appearance columns of questions that you don’t want to appear full width.

This example form from enketo has more detailed information.

Email Notifications

The email address to which to send a notification can now be obtained from the survey results. When creating a notification specify the question from which to get these addresses. This can be a select one, select multiple or text question. They question can be in a repeating group in which case all the emails entered will be used as the “to” address for the notification.

To specify multiple email addresses in a text question separate each email address with commas.

Getting email addresses from a select_multiple question

The names of choices in a select multiple question are used to create the data base table that stores the results. These database column names cannot contain ‘@’ or ‘.’. Hence for select multiple questions you have to modify the way the email address is specified.

Replace ‘@’ with ‘_amp_’
Replace ‘.’ with ‘_dot_’

Choices sheet to specify notification emails addresses chosen by select_multiple

In select_one and text questions you can simply write the email as normal.

Email Message Text

You can now specify the text to be included in an email notification. This can be specified for each notification. Alternatively a default email text can be specified in the organisation settings.

Specifying the email domain

The email domain to be used for emails can be specified at the whole of server level in the “server” table. Alternatively it can be specified for the organisation in the organisation settings.

PDF Generation

You can attach a PDF of the results to an email notification. A PDF will be automatically generated using the same grid layout as is available for WebForms. However in this release its still very much beta code.

More usefully a template for the PDF can be specified in the surveys settings. Go to the survey editor and click on the “File” menu and then “Settings”. Fields in the template that have the same name as a question will be populated with the results of the question.

PDF’s can also be downloaded from a table view of the survey results. Goto the analysis page and create a table containing the survey results. A menu button will be shown to the left of each record. If you click on that you have the option to download a PDF or open a WebForm containing the record. Note this feature is only enabled if the ability to edit surveys is specified in the organisation settings.

Miscelaneous

Files in the /smap/temp directory are now deleted each time you upgrade
If you rename a survey or delete it and then restore it then you can still download the XLS form that was used to create that survey.

Bug Fixes

The following issues have been fixed.

The default language could not be specified in the settings for the survey.
If you had two levels or more of repeating groups then when you exported results an unnecessary row containing data from the main form would have been added to the results.
Could not add media to questions using Firefox
Images of type “ico” were shown in the dashboard with an audio thumbnail
Filter values used in cascading selects were not allowed to have spaces.
When exporting surveys with more than one “/” in their name then the export failed.

Update (Version 15.03.02)

I have updated the released code to address some relatively minor bugs.

Notifications for soft deleted surveys were still being shown in the notifications list
The xls template for editing forms has been updated to include the settings tab and a column for the “style”. You can use this to format web forms in a grid.
The link to this blog from the server home page was pointing to the release notes for version 15.02

Update (Version 15.03.03)

Updated for a very significant bug. The web forms code was not included in the previous release! So if your web forms was looking a bit flat and not responsive to relevant conditions then this would be why. Installing this the .03 version should fix it.

Version 15-02

3 Replies

This release has the following new features

New Menu Bar
Export of media files
Improvements to Task Management interface
User management styled with Bootstrap
Form Builder Interface Modified
Ability to add media at an organisation level so that it can be attached to questions or choices in any form
Performance improvements
Added support for select multiple questions where the choices are sourced from an external CSV file.
Forwarding, which was broken in release 14.12, is now working again
The “from” address and “subject” in a notification email can be specified.
The form name will default to the file name when you upload a form
Addition of Spanish Translation
Updates to French Translation

Menus

The Smap server has been separated into 4 modules:

Home
Edit
Admin
Web Forms
Analysis

“Forms” page of “Admin” module with modules menu expanded

These modules can be accessed from any page by selecting the “Modules” menu. To the right of the modules menu will be the menu items to select a page within the current module.

Note there are still some pages that have not been updated with these new menus.

Export of Media Files

Simply select Export from the Analysis page then select your survey and the export type of “Media”. You then need to select the question that contains you media. By default each file will be named with the record number in which it was submitted. However you optional choice one or more other questions, the results of which will be combined to create the name of the media file.

Options for exporting images to a zip file

The media will be downloaded in a zip file.

Form Builder Interface Modified

The form builder now allows you to edit question text and media for any language in the form.

Adding media to choices using the new form builder

More details can be found in the training documentation: https://docs.google.com/file/d/0B5_SmpWlQYxvaDVYUGdtNU9MZmc/edit

Task Management

The user interface for the task management page was changed to use “Bootstrap”. Hence it looks different. However the behaviour of the page also changed quite significantly.

Task Management Page

These changes are:

Tasks located at location 0,0 are no longer shown on the map. They are assumed to be tasks without location.
You can add ad-hoc tasks to existing task groups.
Ad-hoc tasks can be created without specifying a location
Tasks selected on the map are also selected in the tables and vice versa

For more information on using the updated tasks interface refer to the training material in https://docs.google.com/file/d/0B5_SmpWlQYxvbHBCLW0ySkw2Z3M/edit.

Calculating Nepali Date and Anthro Z-Score

3 Replies

Aaron Mok has contributed an example form that uses Nepali dates to calculate the age of a child and then calculates the Z-Score for that child using their age, weight and height.

You can get the form and its attachments from here: https://drive.google.com/folderview?id=0B5_SmpWlQYxvTmFIcDkzSDIyaGc&usp=drive_web&pli=1&ddrp=1#

This form uses the pulldata() function to look up values in a CSV file. pulldata() is not supported on web forms / enketo so you will need to test it on an Android device.

I have loaded the form onto the demo server if you want to try it out without loading it onto your own server.

url: http://demo.smap.com.au
username: gplay
password: gplay

Let me know if you find any errors in the calculation so Aaron and myself can fix them.

Managing School Enrolments with Smap

4 Replies

Sometimes your forms need to change quickly. One such case was suggested by World Vision staff who are working in Syrian refugee camps. They wanted to record school attendance however new children are arriving daily and hence the attendance roll would be also changing daily.

Putting choices in a CSV file separate from the form can be the answer. The file name is specified in the “appearance” column. In the example below the choices will come from the “children.csv” file. The other parameters to the “search” function are filters so that only children in “class1” are included.

Select Question Getting its Options from a CSV File

Choices in external CSV files have been available for a while, however the January 2014 release of Smap will have the following enhancements:

Support for select_multiple. Previously only select_one worked,
You can now analyse select questions with external choices in graphs, tables and maps on the analysis dashboard,
and, CSV files can be attached to a single form or shared by all forms in the organisation.

Demonstration

This demonstration shows information on new school enrolments being collected using a form and then being added to a CSV file. This file is then used to record attendance in school and is also used by a separate form to record vaccinations given to the children.

The data can be used in multiple forms because it is loaded to the “Organisation” tab in the media page rather than the “Form” tab. If you loaded different files with the same name to “Form” and “Organisation” then the file at the “Form” level will take precedence.

The files for this example can be found here.

Process

A school teacher uses fieldTask and a form called “enrolments” to record school attendance. Any new children arriving that day are recorded in a sub form. (using “begin repeat”)
The back office team copy the details on the new children and put them into the “children.csv” file. They upload this file to the Smap server where it replaces the existing file.
The next day, before recording attendance, the Teacher presses “refresh” in fieldTask to get the latest form. They can then mark off the attendance of all the children including those that arrived the day before.
A medical specialist using a form called “vaccinate”, presses refresh in fieldTask before checking the children in a class. This will load the latest vaccination details into their phone.

CSV File

Children CSV file

This process is shown in the following video

Automation (Future)

The video showed a back office worker manually copying the daily enrolment data into the children CSV file. This task could be automated by implementing a custom web page. Apart from saving some effort for the back office administrators this would also apply updates faster resulting in more current data.

Other Applications of CSV files with Smap

CSV files storing data can be used in other applications, not just for school enrolments and vaccinations. For example asset management, managing workforce etc.

version 14-12 Notifications

Menu and Style Changes

The “bootstrap” javascript library is being implemented to replace “jquery-ui” and “foundation”. In this release the following pages have been updated:

Form Management
Notifications
Locations (User Trail)

Pages will look a bit different, here is the form management page:

Menus

Smap has been divided into 4 sections:

Home
Admin
Web Forms
Analysis

These are accessible from a drop down menu labelled “View”.

Drop down “View” menu

In the image above the “Admin” option is greyed out as the view option was selected from “Form Management” which is part of administration. In a future release another section called “Edit” will be added.

Within each section the menu options will appear to the right of the “view” menu. For administration these options are:

Form Management
User Management
Notifications
Monitoring
Locations
Tasks

Each section will also have common menu options

help
User Settings (an icon that looks like someone’s head and shoulders)

Unfortunately because this new menu layout is being implemented progressively you will find yourself jumping back and forth between the new and old menus as you navigate from page to page. If you find this unbearable let me know and I will try and make all the pages consistent faster.

Notifications

A notifications page has been added which allows you to request that an email be sent every time a new result is submitted to the server.

Notification Page

The notification page above shows a single notification that will send an email to two email addresses (separated by commas) when a result is submitted from the “modify stress test” survey. The email will contain a link to a web form view of the submission.

Web Form View of a Submission – Accessed by link in email

A log of all the notification events can be seen on the monitor page.

Application of email notifications

These could be used for quality assurance. A supervisor can set up a notification for each submitted result which they can then review. A future enhancement may allow filtering so that the emails are sent conditionally dependent on value of the data. Suggestions welcome.

Forwards

Previously you could set up a survey to be forwarded onto another server on the form management page. Now however, forwarding is considered a type of notification. Hence you can set up forwarding on the notifications page.

Miscellaneous Changes

The email host can be set per organisation. Previously an IT administrator had to set the address of your email server (smtp_host) in the tomcat web.xml file. Now it can be set in the users page by editing the organisation details. Note the value for smtp_host in web.xml is now ignored so you will need to update the organisation.
A default value of smtp_host can be set for every organisation on the server. This does have to be done by an it administrator in the survey_defintions database using a command like: insert into server (smtp_host) values (‘Your smtp host’);
When uploading a new form to the server from an XLS file, the name of the form will default to the name of the file. You can optionally change the form name.
Increased the width of the line showing the path taken by the user on the locations screen.
Previously you could not select a set of survey results for editing if that survey had been marked bad. This restriction has been lifted. You are now only prevented from editing results if that record has been replaced by another one.
Performance improvement to “data cleaning”. There were some issues in performance last month when multiple people were cleaning data in the same survey simultaneously. Database deadlocks were causing updates to hang which resulted in the server running out of database connections. These deadlocks have been fixed.

Version 14-11 GPS Trails

3 Replies

Automatic Deleting of Submitted Results

The server can be configured to request that the phone automatically delete successfully submitted surveys. This improves security on the phone as data is much less likely to fall into the wrong hands if, for instance, the phone is lost.

It is strongly recommended that automatic deletion be enabled. Currently it is off by default as I don’t want to surprise anyone with the changed behaviour. However in a future release automatic deletion may be made the default.

To enable automatic deletion edit your organisation and select “Delete submitted results from the phone”. Only an organisational administrator can do this. Automatic deletion currently will only happen when you use “Refresh” to submit your data.

Also if you configure the settings in fieldTask to send results automatically then the “refresh” method will be invoked, new form versions downloaded and submitted results deleted.

Recording GPS Trails

Recording of GPS trails for each user can be enabled by selecting a checkbox in the organisation dialog. Its the same dialog that can be used to specify automatic deletion of submitted results.

The path taken by the user will then be visible on the phone in the Maps tab, along with each location where they completed a survey. These survey locations are recorded even if there were no location questions in the survey form.

Map on Phone with user's GPS trail and survey locations

Map on Phone – One task has been rejected, one has been completed

When the fieldTask user presses the refresh button the GPS trail will be sent to the server along with any completed survey results.

The following video shows how the analyst or administrator can review the paths taken by the data collectors.

Privacy Issues

There are potential privacy issues with recording the path taken by the phone users. For this reason this function is off by default and has to be enabled on the server. It is advised that you contact the developers to discuss how you might use this feature and what additional privacy measures might need to be included prior to enabling. Current privacy controls are limited to:

The user can switch off GPS
The user can switch off recording by going to settings in fieldTask. However it will be re-enabled each time they do a refresh (assuming the server has be set to record the GPS trail)
Recording of GPS set to “on” is shown in fieldTask settings
The fieldTask map will show the trail of GPS points that will be sent to the server

Miscellaneous

The obsolete Nutiteq libraries used for showing maps on fieldTask have been replaced by the MapBox API. This is a lot more pleasant to look at and also will support future functionality such as offline and custom maps.
Added translations codes to the Analysis page. Thank you to Baba Sy and Telmo Peixinho for the French and Portugese translations.
Reduced each row on the task list in fieldTask to only have two lines. The appearance is now less crowded. The forms version number has been put on the same line as the form name and abbreviated to (v: NN)
Reporting of errors in template upload has been improved. These improvements include, checks for read only questions that are mandatory, checks for constraints that don’t refer to the current question, a reduction in unnecessary repetition when reporting on invalid functions.

Version 14.10.2 – Forgot Password

Adding a new user without specifying a password

This feature requires an email server to be specified in the Tomcat configuration. Refer to the operations manual for details if you are running your own Smap server. I am using Google smtp server for my hosted servers however if you are running your own Smap server you can set up your own email server.

When you select “Add User” in the user management screen the default setting will be to send an email to the user. The administrator will not need to specify the password.

Send email to user specified by default

The user will then get an email with a link that they can select to set their password. The email will also inform them of the URL of the server and of the user name they can use to log on with. The link will be valid for 48 hours. After that the user will need to request a new link by selecting “Forgot Password” on the home screen.

When creating the new user, you can select “Specify password” if you want to set the user’s password as in the case of a data collector. You should do this if the data collector’s ident is shared by several people.

Forgot Password

If you forget your password you can now reset it yourself, assuming that you have your correct email address recorded against your user account.

Select “Forgot Password” menu option.
Enter your email address and submit
When you get the email, click on the link and enter your new password

You should now be able to logon with your new password. The link sent in the email will expire after 1 hour.

Uploading multiple large attachments

This release fixes a significant bug whereby if the total size of attachments in a survey exceeded 10MB then some attachments were not processed correctly after being uploaded and could not be seen in the analysis pages.

odk splits large attachments into separate uploads. Smap has now been updated to process these and merge then into the main upload. You may see this on the monitor page.

In the above screen shot from the monitor you can see that 4 surveys have been uploaded for media2 and 13 uploads have been merged into these 4 surveys. In this template there were 4 video questions, the videos were recorded in high definition and each survey had around 60MB of data.

There were also problems with the server configuration which caused “unauthorised” responses to be sent if uploads of large attachments exceeded 5 minutes. This has also been fixed by removing the timeout on the digest authentication for submissions:

AuthDigestNonceLifetime -1

Miscellaneous

Added Translations for Form Management page. Thank you to Baba Sy for providing the French translations.
FieldTask has been updated so that it only shows the forms valid for the current server. If you change the server in the settings screen then the forms valid for that different server will be shown instead.
When getting a list of forms to download only those forms from your current organisation will now be shown. This change only affects users who have access to more than 1 organisation.
When editing survey results using web forms, the dialog with the “edit” button has been removed. If you select the button at the left of the survey record the web form will be opened immediately populated with the existing data.
Fixed a bug where existing data was not adding to a webform if the survey contained questions with names that are reserved for use by the database such as “end”