Version 15.12 of Smap is here.  It contains the long awaited Beta version of the online editor.  You are welcome to try out this editor both for the purpose of creating new surveys and also to edit a survey that has already been loaded.  However it is beta software and, due to its complexity, more than usually buggy.  We will be working on these bugs over the next few releases and also adding some more advanced functionality.   If you find any bugs yourself send me an email with the details.

Features

The new features are:

• Smap Online Editor
• Self registration of organisations on the https://sg.smap.com.au server
• Shared Resource page for files, pictures and maps used by multiple surveys
• Names of column headings are now shown un-modified

Smap Online Editor (Beta)

Yes finally an online editor for Smap.  This will allow you to:

• Create and test a quick survey
• Edit a survey after data collection has started
• Help get the questions translated into local languages
• Export to an xlsForm editor
• Import from an xlsForm editor

More details on the features are available here. The editor is a work in progress and there are still some things that either have to be done in the xlsForm editor or where its still simpler to use that editor. Using the export to xlsForm feature you can switch between editors depending on which is easier.

Self Registration on the Smap Shared server

You can now register your own organisation to use the Smap shared server. https://sg.smap.com.au.  Use of this server is free however it is restricted to “Not for Profit” organisations and individuals.  Support is available for $50 per month if you want it.  There is a “register” menu option on the home page.

Shared Resources

In previous versions of Smap you had to open the editor in order to add files that could be shared across organisations.  You still go into the editor to add files that are specific to a single survey but for all shared resources there is a menu in the admin module.

Here you can load:

• Media (images, video, audio) that are used in question and choice labels
• CSV files that contain data referenced from the survey
• New maps that are used during analysis.  Refer to here for details.

Column names in results tables shown un-modified

In previous versions of Smap if you named a question using characters that could not be used in the database column name they would be changed.  So “q1.4” would become “q1x4”.  This no longer happens so you will see the question names unmodified.

Bug Fixes

• Could not mark a record in a sub form as “bad”.  Thanks to Puskapa for finding this bug.
• If you opened a webform by clicking on a sub form record in the analysis page then the wrong record was opened. Opening a web form on a subform has now been disabled.  To edit a record you now need to open it from the main form. Again thank you to Puskapa for finding this bug.
• Exporting a PDF of a submission failed if the user had not changed their user settings, for example their title.

Mapbox allow you to create beautiful custom maps.  In the upcoming version 15.11 of Smap you will be able to add these maps as base maps during analysis.

You add the map using the new “Shared Resources” page in the admin module.  Details on how to do this is in the documentation for advanced assessment management slides 21 – 30.

Here are some sample mapbox base maps shown on an analysis panel.

Horn of Africa Areas of Food Insecurity

Maps that look like they have been drawn using a pencil

Features

Save edited forms in XLS

The Smap online editor is still under development however there are some edits you can currently do. These include:

• Adding images, audio and video to questions and choices
• Adding CSV files to look up values for choices
• Modify appearance
• Changing the question text
• Change the form’s display name
• Change the project that the form belongs to

It si not recommended that you add questions, as the online editor is a long way from being mature. It is recommended that you do any edits other than media, and appearance in the xlsForm editor.

If you make changes in the online editor you can now download this changed form as an xlsForm by selecting the download button next to the form name on the admin page.  Then select “Edited XLS Form”.   You can still get the original xlsForm that you uploaded, complete with any annotations and formatting, by selecting “XLS Form”.

The downloaded form will be in Microsoft Excel xlsx format.

Change the Instance Name

This has been added to the online editor.  When you complete a form in fieldTask it creates a new instance which is shown in the task list.  The name of this instance can be created from answers in the survey.  So for example you could see your instances in the fieldTask main screen with the name of the householder that you are interviewing.  This can be useful if you have to stop an interview part way through and then resume it later.  For this example the instance name might look like this:

${hh_name}

You can specify the instance name in the settings worksheet of your xlsForm, however if you want to change it after loading the form then you can now:

• Open the form in the online editor by clicking on its name
• Select “File” from the menut and then “Settings”
• Update the instance name and Save

Paging in Web Forms and Zarkman Mobile

Using the magic of Enketo you can now view your question on pages like in fieldTask. The difference is:

• With fieldTask, by default there is one question per page but if you group some questions using the appearance “field-list” then these will be shown on the one page
• With webForms and ZM, be default all the questions are on a single page but if you group some question, again using the appearance “field-list” then these will also be shown together on a separate page.

Another difference is that with WebForms and ZM this only works in the top level form and not in sub-forms.

If you specify pages in your form then “Prev” and “Next” buttons will be added to the bottom of each page.

Bug Fixes

• If you were loading choices from a CSV file and you specified a search in the appearance column of a form, but you also specified another appearance value first, then the csv file would have caused this problem: “quick search(‘locdetails’, ‘matches’, ‘domain’,’locTypes’)”. It sets an appearance of “quick” so that as soon as the user select a “choice” the next question will be shown.  It also specifies the search that will retrieve the choices from a CSV file.   This now works.  Thanks to Grable Services for finding this bug.

Installing your own server

The updated server is now available for download from http://www.smap.com.au/downloads.shtml

Updates

Version 15.10.02

• Fixed issue with duplicate instance names being created in the database if you specify an instance name in the settings dialog of an xlsForm.  This is a critical issue that will stop data being loaded into the results table and if you have version 15.10.01 you should upgrade to 15.10.02
• Back up smap.conf and smap-ssl.conf apache configuration files before overwriting them.  This overwriting happens when upgrading to version 15.09 so it could happen, which would be a little annoying, if you delete the “smap_version” text file in the home directory of your installation.  In version 15.10.02 you will be able to recover any customisations you may have made to these two files.

Version 15.10.03

• Fixed some issues with instance name creation.  This was failing if there were no spaces around a reference to a question.  Ie “concat(‘Householder: ‘, ${name})” used to fail because there is no space after the closing curly bracket.

Version 15.10.04

• Fixed issue where duplicates can be created under certain circumstances.

Version 15.10.05

• Fixed issue where child records in a form could not be marked as bad.

Finally version 15.09 is available.  The last release was 15.04 way back in April.  The reason for the long delay has been the development of the hybrid iOS / Android app Zarkman Mobile.  Hence the other changes to the system are relatively minor.

Zarkman Mobile

Zarkman Mobile is available for download from google play store and Apple app store.

Zarkman Mobile can only be used with Zarkman servers and hosted Smap servers. A “hosted Smap server” is one that is installed by Smap and has a domain name ending in smap.com.au.  Ie https://xxx.smap.com.au.

Other Features

Languages

Hindi has been added as a language on the server thanks to a contribution from SDRC.  Unfortunately I haven’t made available any more pages for translations so there is still a lot of English text.  In fact the server may have gone backwards a little in this respect.

Loading templates with data

More details in http://blog.zarkman.eu/using-templates-pre-loaded-data/. This requires the latest version of fieldTask which is included in this release and is in Google Play Store.  This feature allows you to do “red-lining” that is mark up an image by drawing on it as the answer to a question.

Data Cleaning – Set codes

You can now update any text question based on the results in another text question.   This allows you to set codes to correspond to arbitrary text. For example if you have 10 rows where the value of a text questions is:  “two wheeled vehicle”. Then you can set a code in another question for that answer.

Formatting of PDF export of templates

The formattng has been improved and there are a number of appearance options you can specify to change the columns, background colours and sizes of the table cells.

Bugs Fixed

• If you attempt to export a survey with a “#” in its name then the export will fail. The # will now be replaced by an underscore.
• There was no validation on “from” and “to” dates in the settings dialog in analysis.  These are now validated and you can select the date from a calendar.
• If you used placement-map as an appearance option for a geopoing question then the map was not being shown. The google maps now works.
• The word “null” will no longer appear as its name when you temporarily save a form.  This name can be set from the answers to questions in the survey, this feature is set up in the “settings” tab of the xls form.  However if you don’t specify the instance_name there then the name shown in fieldTask will now be set to the name of the form.
• Added filtering on “whole of survey” table panels.  Previously date filtering only worked when querying on a specific question.  Now you can view results for a date range or a data value.

Installing your own server

Version 15.09 is available for download from http://smaphs.com/downloads.shtml

Apache Web Server version 2.2 is no longer supported.  You should install Apache version 2.4 prior to upgrading to Smap 15.09.  There have also been changes to the way the Apache web server configuration is set up.  Please check the readme file included in the tar file.

Note initially version 15.09.03 was made available for download.  This has now been updated to 15.09.04.  The most significant issue was in the smap.conf which is the apache2 configuration for accessing the sever without encryption.  If you installed the older version and you arn’t using https then send an email to the smap-suite google group and I will send you the updated configuration file. This update also fixes a bug with creating new organisations.

For some time now its been possible to create a form pre-loaded with data as a task that can be completed once.  With Smap Version 15.09 you can specify a form pre-loaded with data as a template so that once you have filled it in then it will still be there to be completed again.

This capability has also been added to Zarkman Mobile.  However ZM also has the ability to create these templates on the phone.  So you can collect some initial data in the field. Then mark the results file as a template on the phone.  You can then continue collecting data using the new template.

There should be many applications for this capability.  One example is annotating a photo.  You could have an aerial photograph of an aggricultural field included in the template.  Then when completing the survey the user can mark on that photo the location of a crop or an area susceptible to flooding.

The following example is from the building industry where the plan of a building has been added to a template and the user marks the location of a fault during an inspection.

Example

1. Create a form and load the initial data

You can populate the form by completing it in web forms or on fieldTask.  Alternatively you can load the initial data from a CSV file.  In the form used in this example there were two questions:

• A multiple choice questions aasking what faults have been found during the inspection of a building
• An image question with appearance “annotate” that allows the user to mark on a plan the location of the faults

2. Create a task and mark it as repeating

A repeating task on the tasks page of the server

To set a task as repeating click on the button with the task name on it.  In the above example this is the button labelled “faults”.

3. Press “refresh” in fieldTask

fieldTask main screen showing the template with a blue icon

Templates pre-ploaded with data have a blue icon.  Forms, which don’t have any pre-loaded data, are still shown with yellow icons. Tasks that can only be completed once are shown with white icons.

4. Select the template to complete it

fieldTask main screen showing a completed survey and the template

After completing the template and saving you will see the completed results (the icon with the green tick), however the template with the blue icon will also still be shown and can be selected again. For this example scenario I completed the template multiple times and then viewed the results on the analysis databoard.

A table view of the inspection results showing the plan grouped by fault type. The user has marked the location of each fault on the plan with a red circle.

Note: The floor plan used in this post is from https://en.wikipedia.org/wiki/File:House_Plans_(Blueprints).pdf and is published under a Creative Commons 3.0 license.

The April release of Smap is finally available.   These release notes extend the previous notes for 15.04.

I had to add some more functionality which delayed its release a little, including:

1. Compressing select multiple questions when exporting to XLS
2. Authentication of emails sent as a notification of an event.

Compressing Select Multiple Questions

Smap Server has always exported the results of select multiple questions as 1 column per choice.  Each column contains a “1” or a “0” to indicate whether or not the choice was selected.  This is convenient for analysis but results in large unwieldily tables if your objective is to take the spreadsheet into a meeting and discuss its contents.  Hence a new compress select multiple option has been added to the XLS export.   This will put all the names of selected choices into a single column separated by a space.

Loading Select Multiple Values into a Survey from a CSV file

Having all the select multiple choices in a single column is also used to load select multiple values from a CSV file.  This allows you to export a survey with compressed select multiple questions, modify the options in a CSV file and reload them back into the survey.

Authenticating Email Users

When setting up email notifications you can now specify the user name and password of the email user to send the emails.  This should reduce the chance of email ending up in Spam and will allow you to store sent emails in the sent folder of that email account.  Details are in the operations guide.

Bugs

1. Monitoring of notifications only showed notifications at the whole organisation level.  These are now shown per project or per survey.

Update 15.04.02

1. Added more detail to the error message that is shown when you attempt to load tasks into a form that you have not explicitly marked as being allowed to get data from a CSV file.  The error message now directs you to the settings dialog.
2. Added error message if there are no columns in the CSV file that match questions in the form.  Previously this just resulted in a generic internal system error.
3. Added an error message if the CSV file to be loaded is not specified.
4. Fixed bug preventing loading of instance name data
5. Remove spaces at the beginning and end of values inside the CSV file.  These spaces would cause image files in the ZIP file not to be found

Update 15.04.03

1. Fixed issue where tasks are not created from an existing survey if the locations of the existing survey are in a subform and the location is missing.

Update 15.04.04

1. Fixed issue with adding task addresses and filtering tasks.

Because the March version of Smap ran over time, its been a short month for the development team.  Hence we have only added two functions in this release. These are:

1. the ability to pre-load survey results from a CSV file into a survey. These can then be used to assign tasks to users.
2. allowing users to complete their tasks using Web Forms as well as with fieldTask.

 Tasks Refresher and Rationale for the new Features

Tasks can be created from a blank form or from existing survey results.  In the second case, when the assigned user opens the task, it will be pre-populated with those results. Up until now you had to populate the initial data for these tasks by  completing forms using either fieldTask or WebForms.

However you may have a list of assets that need to be inspected, or hospital clinics that need to be visited stored in a spreadsheet or an external system.  In which case you can now load that spreadsheet directly into a form and then allocate tasks to people to go and complete those surveys. That is inspect the asset, visit and audit the hospital clinic etc.

Secondly using web forms you can assign tasks to people with laptops or iPhones as well as users with the Android app fieldTask.

Features

• Data in the CSV can include text, integers, barcodes, GPS coordinates, images, video and audio.
• Columns in the CSV file that do not match a question in the form are ignored
• It is not necessary to include all of a forms questions in the CSV file
• You can replace existing results or append.
• Forms need to be marked, in the settings dialog of the editor, as accepting data uploaded from a file.  This should reduce the chance of accidental loading of data to the wrong form.
• You can load either a single csv file with just text.  Alternatively you can load a zip file containing a CSV and images, video, audio files.
• Tasks on Web Forms are shown as red buttons rather than the blue buttons for blank forms

Limitations

• Polygon and line question types are not supported
• The file must be CSV format and not excel format
• The survey cannot be empty.  That is you must have submitted at least one completed form prior to loading the tasks.  You can use Web Forms for this and if this initial piece of data is not required it can be cleared when you load the tasks from the CSV file.
• FieldTask does not show images from preloaded data.  This is a high priority issue to fix in the next release.

Demonstration – Asset Inspection

Inspection Form

For this demonstration I created a form that included a text, image, select_one and geopoint question.  I set the “name” question to be read only so it would not be updated during the survey. All of these except for the select_one question are going to be pre-populated from a CSV file.

This form was loaded to the server.

CSV File

The CSV file looks like the following image.  Column headings should match the name of the question where you want to put the data.  The exception is for geopoint (GPS coordinates) questions.  If you have columns labelled “lat” for latitude and “lon” for longitude then the values from these columns will be loaded into the geopoint question.

 The image files in the pic_asset column need to be added to a directory along with the CSV file. This directory should be zipped and then uploaded to the Server.

Uploading the CSV file

On the tasks page select “Load Tasks From File”.  Then select the Form to load the data into.  Indicate whether or not you want existing data replaced and select your CSV or ZIP file.  Then press “Save”.

Assigning Tasks

You can then click on “Add Task Group” to assign the asset surveys to users.

Screen shot of tasks paging showing 3 tasks to inspect assets

Completing The Tasks

Using Web Forms

Web Forms Launcher screen showing tasks assigned to the user as red buttons

When you click on a task it will show in a web form.

Web form with data pre-populated from a CSV file

Note: the asset name question is a text question set to “read only”, hence the asset name from the CSV file cannot be edited.  However the location and the picture can be replaced.  The “health of asset” which was not in the CSV file can also be filled in.

Final results

After the user presses the submit button the updated information will be available on the server.

Results after one task completed

Note: You can see that the record loaded from CSV file has been replaced by the completed survey results.  In this case the location and picture were not changed.

Bug Fixes

We only did one bug fix in this release which was a the request of Zarkman.

• The menus on the Webforms page have been made consistent with other pages.  That is the links to Admin, Analysis etc have been put into a “Modules” drop down menu.