New Features

Adding notes to a survey

You can now record notes while you are completing a survey on fieldTask.  Notes can also be added after a survey has been saved as complete and before it it submitted to the server. You will need version 3.13 of FieldTask for this feature.

Adding notes while conducting a survey

Notes

• To add a note start the survey or task then select “menu” | “add comment”. Press the button “Save Changes” to save the comment or press the back button to discard it.
• The name of the question currently being answered is automatically added to the note
• When adding notes to a completed survey most of the questions will be marked as readonly and cannot be modified.  However, currently, some question types will still appear capable of modification. Despite this appearance, the changes to questions will not be saved, only the changes to the notes
• Also when adding notes to a completed survey make sure you save the survey as well as each individual note or the notes will not be updated

Important.  Only surveys added after version 16.03 of the server will have the notes saved.  If you need this feature applied retrospectively to existing surveys let us know and we can make a quick database change that will allow that.

The following video illustrates the functionality

Security Considerations

Some people may consider it a security issue that the answers to completed surveys can now be reviewed on the phone.  If this is an issue to anyone let us know and we will rush out a patch allowing the feature to be disabled on the server.  It would also be possible to add a password to the form on the server that needs to be entered to review the completed survey.

Security Controls that can be enabled on the phone now are:

• A PIN to prevent access to unauthorised users
• Encryption of the sd card using Android native tools
• Remote wipes of the sd card using third party Android management apps
• Ensure “auto-delete” of survey results after submission to the server is enabled

The above controls are more critical for ensuring security than preventing some people, who already are authorised to access the phone, from seeing the completed data.  However as stated if you have a requirement for this we will add it.

There are only a few changes in this release.

New Features

Times shown in local time

Previously all times were shown as GMT but now some times are shown in the local time of the user’s browser.  Where this is done the label will include the words “(Local Time)”.  These local times are shown in the following places:

• Monitor page:  Upload Date
• Analysis page on table views. Upload Date, Start Time and End Time
• Reports page.  The published time of the report.

On the analysis page, the start and end time of the survey is only shown in local time if you have not set your own start or end preload question.

Times in exported files remain GMT.  However if you export to XLS you can easily add a new column to show the time in another time zone.   You just need to add (offset in hours) / 24 to the date / time column.

Multi-Language Support

More of the web pages are available in other languages now.  However only the Spanish translations have been provided so far.   I’m waiting on the translators for French, Hindi and Portuguese to provide translations for these new pages.

Automatic Browser refresh when the version changes

Previously you had to press refresh on browser pages after a new version of Smap was released.  This is no longer required as the browser will refresh automatically to get the new code.  However there is a catch, for this feature to work it requires the browser to be refreshed!  So you you will need to do that again for this release but only once more (for every page!).

Bugs Fixed

• When setting up an email server for an organisation the email port was not being set.
• Various fixes to the editor approach to sub forms.

In this release the online editor is a little more stable, although still Beta.  We have also started to add some new features that make use of the potential of the online editor.

New Features

Multiple Forms updating a single table of data

With this feature you can have a single table of data and multiple forms contributing to it.   Each form can have unique questions so that it is the only form that provides answers to those questions but each form can also answer questions that are also answered by other forms.  To help you work out whats going on in your results there is a new “informational” column that shows the name of the form that contributed each row of data.

If seems a bit confusing then you probably don’t need it.  For assessments it is still good practice to only have one version of one form used by all data collectors.   However some potential uses for this new feature are:

• Validating or annotating survey results.   In the validation form you can set the questions that also appear in the data collection survey as “read only” and then add some additional questions for the reviewer to answer.
• Performing follow up surveys where an issue identified in the first survey is addressed.
• Having a different survey for each region where most of the questions are common but some are specific to the region.  You can achieve something similar by using “relevance” on questions however this requires the user to enter correctly the region they are using.  If there was a separate form then they would not be able to answer questions for the wrong region.

The following You Tube video demonstrates how to use this feature.  The scenario is for a medical clinic where the patient first enters some of their details and then the doctor expands on the patient record with a diagnosis.

Key steps to share a table are:

1. Create a form
2. On the admin page select “Create New Form”
3. Select the checkbox that says “Base on existing form”
4. Select the form to share with
5. Select the checkbox that says “Store results in a shared table”
6. You can then modify the second form, deleting questions, adding new ones etc.

Basing a new form on an existing form

When you create a new form in the editor you can select a checkbox to base it on an existing form.  By default this form’s data will go into its own table, unless you further select the checkbox to “share results”.

Kobotoolbox API

Some of the services from the KoboToolbox API are now supported.  More information is available form the Smap koboToolbox API help.  If you are developing applications that have to work with both Smap Server and kobo then this API may be useful.  Contact us if there are specific API endpoints that you need for your application.

This feature enhances Smap’s conformance to standards.  The main standard we use is JavaRosa which specifies the services that a mobile device uses to connect to the server.  This means that you can use any ODK compatible device with Smap including odkCollect and GeoOdk as well as fieldTask.  The kobo toolboxApi specifies services to access collected data and administer the server.  Hence you should be able to use software built with these services with the Smap server.

Bugs Fixed

• Scheduled Time was not included in tasks, only the date.
• If you had a quotation character in a form name then when you exported the results for that form it would not allow you to include the question text in the first row of a results export.
• If you had more than one “select” question with the same name then when loading from an XLS file the choices for one of the questions were dropped.
• The order in which questions were shown was not consistent between the dashboard and the export.

Version 15.12 of Smap is here.  It contains the long awaited Beta version of the online editor.  You are welcome to try out this editor both for the purpose of creating new surveys and also to edit a survey that has already been loaded.  However it is beta software and, due to its complexity, more than usually buggy.  We will be working on these bugs over the next few releases and also adding some more advanced functionality.   If you find any bugs yourself send me an email with the details.

Features

The new features are:

• Smap Online Editor
• Self registration of organisations on the https://sg.smap.com.au server
• Shared Resource page for files, pictures and maps used by multiple surveys
• Names of column headings are now shown un-modified

Smap Online Editor (Beta)

Yes finally an online editor for Smap.  This will allow you to:

• Create and test a quick survey
• Edit a survey after data collection has started
• Help get the questions translated into local languages
• Export to an xlsForm editor
• Import from an xlsForm editor

More details on the features are available here. The editor is a work in progress and there are still some things that either have to be done in the xlsForm editor or where its still simpler to use that editor. Using the export to xlsForm feature you can switch between editors depending on which is easier.

Self Registration on the Smap Shared server

You can now register your own organisation to use the Smap shared server. https://sg.smap.com.au.  Use of this server is free however it is restricted to “Not for Profit” organisations and individuals.  Support is available for $50 per month if you want it.  There is a “register” menu option on the home page.

Shared Resources

In previous versions of Smap you had to open the editor in order to add files that could be shared across organisations.  You still go into the editor to add files that are specific to a single survey but for all shared resources there is a menu in the admin module.

Here you can load:

• Media (images, video, audio) that are used in question and choice labels
• CSV files that contain data referenced from the survey
• New maps that are used during analysis.  Refer to here for details.

Column names in results tables shown un-modified

In previous versions of Smap if you named a question using characters that could not be used in the database column name they would be changed.  So “q1.4” would become “q1x4”.  This no longer happens so you will see the question names unmodified.

Bug Fixes

• Could not mark a record in a sub form as “bad”.  Thanks to Puskapa for finding this bug.
• If you opened a webform by clicking on a sub form record in the analysis page then the wrong record was opened. Opening a web form on a subform has now been disabled.  To edit a record you now need to open it from the main form. Again thank you to Puskapa for finding this bug.
• Exporting a PDF of a submission failed if the user had not changed their user settings, for example their title.

Mapbox allow you to create beautiful custom maps.  In the upcoming version 15.11 of Smap you will be able to add these maps as base maps during analysis.

You add the map using the new “Shared Resources” page in the admin module.  Details on how to do this is in the documentation for advanced assessment management slides 21 – 30.

Here are some sample mapbox base maps shown on an analysis panel.

Horn of Africa Areas of Food Insecurity

Maps that look like they have been drawn using a pencil

Features

Save edited forms in XLS

The Smap online editor is still under development however there are some edits you can currently do. These include:

• Adding images, audio and video to questions and choices
• Adding CSV files to look up values for choices
• Modify appearance
• Changing the question text
• Change the form’s display name
• Change the project that the form belongs to

It si not recommended that you add questions, as the online editor is a long way from being mature. It is recommended that you do any edits other than media, and appearance in the xlsForm editor.

If you make changes in the online editor you can now download this changed form as an xlsForm by selecting the download button next to the form name on the admin page.  Then select “Edited XLS Form”.   You can still get the original xlsForm that you uploaded, complete with any annotations and formatting, by selecting “XLS Form”.

The downloaded form will be in Microsoft Excel xlsx format.

Change the Instance Name

This has been added to the online editor.  When you complete a form in fieldTask it creates a new instance which is shown in the task list.  The name of this instance can be created from answers in the survey.  So for example you could see your instances in the fieldTask main screen with the name of the householder that you are interviewing.  This can be useful if you have to stop an interview part way through and then resume it later.  For this example the instance name might look like this:

${hh_name}

You can specify the instance name in the settings worksheet of your xlsForm, however if you want to change it after loading the form then you can now:

• Open the form in the online editor by clicking on its name
• Select “File” from the menut and then “Settings”
• Update the instance name and Save

Paging in Web Forms and Zarkman Mobile

Using the magic of Enketo you can now view your question on pages like in fieldTask. The difference is:

• With fieldTask, by default there is one question per page but if you group some questions using the appearance “field-list” then these will be shown on the one page
• With webForms and ZM, be default all the questions are on a single page but if you group some question, again using the appearance “field-list” then these will also be shown together on a separate page.

Another difference is that with WebForms and ZM this only works in the top level form and not in sub-forms.

If you specify pages in your form then “Prev” and “Next” buttons will be added to the bottom of each page.

Bug Fixes

• If you were loading choices from a CSV file and you specified a search in the appearance column of a form, but you also specified another appearance value first, then the csv file would have caused this problem: “quick search(‘locdetails’, ‘matches’, ‘domain’,’locTypes’)”. It sets an appearance of “quick” so that as soon as the user select a “choice” the next question will be shown.  It also specifies the search that will retrieve the choices from a CSV file.   This now works.  Thanks to Grable Services for finding this bug.

Installing your own server

The updated server is now available for download from http://www.smap.com.au/downloads.shtml

Updates

Version 15.10.02

• Fixed issue with duplicate instance names being created in the database if you specify an instance name in the settings dialog of an xlsForm.  This is a critical issue that will stop data being loaded into the results table and if you have version 15.10.01 you should upgrade to 15.10.02
• Back up smap.conf and smap-ssl.conf apache configuration files before overwriting them.  This overwriting happens when upgrading to version 15.09 so it could happen, which would be a little annoying, if you delete the “smap_version” text file in the home directory of your installation.  In version 15.10.02 you will be able to recover any customisations you may have made to these two files.

Version 15.10.03

• Fixed some issues with instance name creation.  This was failing if there were no spaces around a reference to a question.  Ie “concat(‘Householder: ‘, ${name})” used to fail because there is no space after the closing curly bracket.

Version 15.10.04

• Fixed issue where duplicates can be created under certain circumstances.

Version 15.10.05

• Fixed issue where child records in a form could not be marked as bad.

Finally version 15.09 is available.  The last release was 15.04 way back in April.  The reason for the long delay has been the development of the hybrid iOS / Android app Zarkman Mobile.  Hence the other changes to the system are relatively minor.

Zarkman Mobile

Zarkman Mobile is available for download from google play store and Apple app store.

Zarkman Mobile can only be used with Zarkman servers and hosted Smap servers. A “hosted Smap server” is one that is installed by Smap and has a domain name ending in smap.com.au.  Ie https://xxx.smap.com.au.

Other Features

Languages

Hindi has been added as a language on the server thanks to a contribution from SDRC.  Unfortunately I haven’t made available any more pages for translations so there is still a lot of English text.  In fact the server may have gone backwards a little in this respect.

Loading templates with data

More details in http://blog.zarkman.eu/using-templates-pre-loaded-data/. This requires the latest version of fieldTask which is included in this release and is in Google Play Store.  This feature allows you to do “red-lining” that is mark up an image by drawing on it as the answer to a question.

Data Cleaning – Set codes

You can now update any text question based on the results in another text question.   This allows you to set codes to correspond to arbitrary text. For example if you have 10 rows where the value of a text questions is:  “two wheeled vehicle”. Then you can set a code in another question for that answer.

Formatting of PDF export of templates

The formattng has been improved and there are a number of appearance options you can specify to change the columns, background colours and sizes of the table cells.

Bugs Fixed

• If you attempt to export a survey with a “#” in its name then the export will fail. The # will now be replaced by an underscore.
• There was no validation on “from” and “to” dates in the settings dialog in analysis.  These are now validated and you can select the date from a calendar.
• If you used placement-map as an appearance option for a geopoing question then the map was not being shown. The google maps now works.
• The word “null” will no longer appear as its name when you temporarily save a form.  This name can be set from the answers to questions in the survey, this feature is set up in the “settings” tab of the xls form.  However if you don’t specify the instance_name there then the name shown in fieldTask will now be set to the name of the form.
• Added filtering on “whole of survey” table panels.  Previously date filtering only worked when querying on a specific question.  Now you can view results for a date range or a data value.

Installing your own server

Version 15.09 is available for download from http://smaphs.com/downloads.shtml

Apache Web Server version 2.2 is no longer supported.  You should install Apache version 2.4 prior to upgrading to Smap 15.09.  There have also been changes to the way the Apache web server configuration is set up.  Please check the readme file included in the tar file.

Note initially version 15.09.03 was made available for download.  This has now been updated to 15.09.04.  The most significant issue was in the smap.conf which is the apache2 configuration for accessing the sever without encryption.  If you installed the older version and you arn’t using https then send an email to the smap-suite google group and I will send you the updated configuration file. This update also fixes a bug with creating new organisations.

For some time now its been possible to create a form pre-loaded with data as a task that can be completed once.  With Smap Version 15.09 you can specify a form pre-loaded with data as a template so that once you have filled it in then it will still be there to be completed again.

This capability has also been added to Zarkman Mobile.  However ZM also has the ability to create these templates on the phone.  So you can collect some initial data in the field. Then mark the results file as a template on the phone.  You can then continue collecting data using the new template.

There should be many applications for this capability.  One example is annotating a photo.  You could have an aerial photograph of an aggricultural field included in the template.  Then when completing the survey the user can mark on that photo the location of a crop or an area susceptible to flooding.

The following example is from the building industry where the plan of a building has been added to a template and the user marks the location of a fault during an inspection.

Example

1. Create a form and load the initial data

You can populate the form by completing it in web forms or on fieldTask.  Alternatively you can load the initial data from a CSV file.  In the form used in this example there were two questions:

• A multiple choice questions aasking what faults have been found during the inspection of a building
• An image question with appearance “annotate” that allows the user to mark on a plan the location of the faults

2. Create a task and mark it as repeating

A repeating task on the tasks page of the server

To set a task as repeating click on the button with the task name on it.  In the above example this is the button labelled “faults”.

3. Press “refresh” in fieldTask

fieldTask main screen showing the template with a blue icon

Templates pre-ploaded with data have a blue icon.  Forms, which don’t have any pre-loaded data, are still shown with yellow icons. Tasks that can only be completed once are shown with white icons.

4. Select the template to complete it

fieldTask main screen showing a completed survey and the template

After completing the template and saving you will see the completed results (the icon with the green tick), however the template with the blue icon will also still be shown and can be selected again. For this example scenario I completed the template multiple times and then viewed the results on the analysis databoard.

A table view of the inspection results showing the plan grouped by fault type. The user has marked the location of each fault on the plan with a red circle.

Note: The floor plan used in this post is from https://en.wikipedia.org/wiki/File:House_Plans_(Blueprints).pdf and is published under a Creative Commons 3.0 license.