Category Archives: Smap Release

Release announcements for Smap

Version 20.04

New Features

Artificial Intelligence Services

We are continuing to work on adding Artificial Intelligence to the Smap Platform and the following features are a step towards that goal. They add calls to generic AI services provided by AWS for translation, transcription and image recognition.

There is a cost for using these AWS services so by default monthly free limits have been added per organisation.

  • Image Labelling: 100 Images
  • Translation: 5,000 text characters
  • 250 seconds of Audio transcription

Note: These Quantities have been updated as the original post contained the free limits for an entire server. So the new quantity is about 10% of the previous value.

So feel free to use these services and you will get an error message when your monthly limit has been exceeded. If desired, you can then, contact Smap to extend the limit which will incur a monthly fee.

Automatic Translation of Surveys to other languages

The list of supported languages is available in the documentation: http://www.smap.com.au/docs/xls-languages.html

Languages can be translated when the survey is loaded from an XLS template. In this case only blank untranslated text will be modified. You can also initiate auto translation from the translate page of the online editor. In this second case you can specify that the existing language translations be overwritten.

It is important to note that there will be errors in the translations and the performance of AWS translate may vary with language. However this is normal in survey development even with manual translations. Ultimately, during data collector training, problems with the translation get resolved. The online editor translate page is a convenient place to make these last minute adjustments.

The following video shows how to automatically translate a survey in an spreadsheet template.

Automatic Translation of Text Data

Just as you can translate a survey into multiple languages automatically you can now translate text responses in submitted data. This is done by specifying the auto=yes parameter to a question in an oversight form.

The following describes the manual way to translate question responses which can be automated using the “auto” parameter. http://www.smap.com.au/docs/console.html#example-1-translate-a-text-answer

Automatic Conversion of Audio into Text

If you collect audio as part of your survey then if the audio is in one of the supported languages you can automatically convert it into text. This then allows you to search through your data for key words that were spoken in audio files. You can add automatic translation of the transcribed text by adding an additional translate question to the oversight form. Details: http://www.smap.com.au/docs/console.html#automatic-audio-transcription

Automatic Labelling of Images

This has been available for some time however it has been updated to work like the other services in the background without affecting the speed with which new submissions are processed.

Add “append” for repeat records when updating existing results

If you use a task to update an existing record then any repeating groups in that record will be replaced. The scenario might be that you are doing a follow up interview at a household where there were originally 3 household members. While completing the survey the enumerator might remove one household member, modifying a second and add 2 more. After submission there will be 4 household members with the updated data.

However sometimes you want to append new repeating groups. In particular if you set the parameter “instance_count” to zero then none of the existing repeats would have been downloaded with the task. The enumerator can then only add new repeating records and you would want these to be appended to the existing records.

This can now be done using the parameter: key_policy=append

The hospital example implementation has been updated to include this.

Other Changes

  • Add support for the “required” message in multiple languages
  • Bulk uploading of users, projects and roles from a spreadsheet

Bug Fixes

  • Issue 434. Support the use of apostrophes in text when filtering data on the server.
  • Issue 507. Setting the timezone on tasks exported to a spreadsheet

Version 19.02

Start a survey from within a survey

While completing a survey on a phone or tablet with fieldTask the user can start another survey / form and then when they have finished that they are returned to their original survey. The data collected by these two surveys are linked for analysis.

Two new question types have been added; “parent_form”, “child_form”. The difference between these is in how the collected data is linked.

Launching a Parent Form

There might be many records in a form that point to a parent form.  For example you may be reporting problems on pieces of equipment.  So you open the problem report form and select the equipment identifier, then record the problems for that one piece of equipment. However if equipment has not already been added to the system then you can launch a parent form, from within your problem form, to create an entry for that equipment. You can then continue filling in the problem report knowing that it will be linked to the new piece of equipment for analysis. All of this works offline.

In the above example the key Question would hold the identifier for the piece of equipment.

Launching a child form

There might be many records in the child form that point to the first form . For example you may be registering students for school attendance and you encounter a student for which there is no record.  You can then launch a child form to create an entry for that student who will be linked to the school.

In the above example the key question would hold the identifier for the school.

Other Changes

  • The online editor now greatly simplifies the setting of parameters and appearance. You can select these from options presented on the screen rather than having to type them in. This is particularly useful for cases like the search appearance where you can now select from a list of csv files or forms to reference and then select the columns from those that you want to include. No more awkward looking up of form identifiers! This is also useful when adding a new parent_form or child_form question type as you can simply select the form you want to launch using the parameters dialog.
  • Added support for range widgets in webforms
  • Allow backward navigation in a form to be prevented by using a setting on the server.

Release 18.7

Its been a bit slow over summer.  We are now announcing the july release and yet we are already into August.

Changes include:

  • rank question type
  • randomize appearance on select questions

Rank Question Type

This is just like a select multiple question except instead of choosing one or more options you can order them.

Adding to your survey

Specify in an XLS Form in the same way you specify a select multiple question except that you replace “select_multiple” with “rank”. Then you have a space and the list name that points to all the choices that you want to rank. So:
        type                          name                           label
rank my_list             question_name          Rank this list!

In the online editor select the question type called “rank” and then add the choices.

Using

The rank question works in FieldTask and WebForms.  In both devices you drag the choices into the order that you want.

 

Rank Question in WebForms

Analysis

If you export the data into an XLS report you will see that the ranked questions are shown in a similar way to select multiples with a column per choice.  However the column headings are the question name – the rank.  And the contents are the name of the choice.

Answers to a rank question in an XLS report.

Limitations

Not all exports support rank yet. The support in the API is also limited.  You also can’t currently use it in filters. For example get all data where london was marked as the first choice.

Randomize

If you add an appearance of randomize=true for a select or rank question then in fieldTask and WebForms the order in which the choices are shown will be randomised.  This can be useful to reduce the impact of bias in choice selection on the survey results.   Randomize currently only works on FieldTask and not on Webforms. No it works in webforms too.  Thanks to Alexander Torrado for picking this up.

Bugs Fixed

  • Choices pulled from another form were not showing in webforms.  If you looked up choices in a CSV file then these did show in webforms however if the choices were in another form they did not.

Update 8th August 2018 to add the bugs fixed section and to note that randomize works with webforms but you need to set it as a parameter and not in appearance.

Release 18.6

Email Tasks

The ability to send pre filled or blank forms as links in  emails out to people for completion in their browser.  Capabilities include:

  1. Forms can only be filled in once by each recipient (unless the repeat flag is set)
  2. The URL to access the form includes a key so that the user does not need to have an account on the server
  3. You can upload email address and form names from a file and then have the emails sent to all of those addresses
  4. There is an unsubscribe link in all emails
  5. People can resubscribe, if they change their mind by going to the subscriptions page (on the home page select subscriptions)
  6. Emails with a pre-filled in form can be sent automatically when a submission is received for any other form

Really what this does is enable you to extend you data collection beyond the professional data collectors who have a logon on the server.  Hence you can do email surveys, as per Survey Monkey and also integrate input from external people in a multi step process.  So in fact Smap goes way beyond the capabilities in Survey Monkey.

Note: Email tasks are disabled on the sg server by default. Contact us if you want to use this feature.

Auto sending email tasks

You can also assign tasks that include previously collected data to an email address .   As with other tasks, that are completed by logged in users,  you can create these from previously submitted results or results as they come in.  Here is a video of an example process but there are an infinite number of ways you can use this feature:

After completing the steps shown in the above video and then doing an XLS report I got the following output:

Loading Email Addresses from an Excel File 

Email tasks are managed from the tasks module,  From there you can create a task group and then click on the file menu to download an XLS file.  In this case for new task group it will be empty but you can specify the “tasks” here.

The columns with blue headings are where you put details about the form.  (See above). The columns with yellow heading contain details about the person being sent the email including of course their email address.  You can add multiple email addresses to be sent a single form hence the greyed out cells in the above example where we have one form but two email addresses.  Also you don’t need to fill in the URL column, this is only filled in when exporting in case you want to manually send the links out without using Smap’s email manager.

Once you have filled in your forms and email addresses.  Click on the Menu File > Load XLS.  Emails will not immediately be sent, you will then have the opportunity to review the loading emails and when you are ready you can select the menu “Email Unsent”.   This will send the emails with blank forms.

Unsubscribe From emails

This became urgently needed due to the addition of the “Email Tasks” functionality, however the unsubscribe link is now added to any email sent from the Smap server including for example an email adding you as a registered user.  If someone clicks on that link they are asked if they want to confirm the un-subscription.  If they say yes then they won’t be sent any more emails.   They only way to resume sending emails to an unsubscribed address is if you can persuade them to:

  • Goto to the server home page
  • Click on the subscriptions menu
  • And subscribe again

Updates to FieldTask version 5.5

Well this was released some time ago but I didn’t make an announcement on this news site so here it is.  Lots of new features here contributed by the folk from ODK as well as the fieldTask developers. Some of the main ones are:

  • Online lookup of data from a server while the user is completing the form. You can lookup reference data, choices for a select or query an AI web service.
  • Non printable characters in barcodes will now show as spaces rather than being removed completely
  • Includes ODKCollect 1.15
    – Can set a dark or light theme to change the screen appearance
    – Add a “file” type question that allows you to attach any file to the form

 

 

 

Version 18.05

Time for the May release of Smap.

Updated Webforms

Webforms has been upgraded to use the latest version of Enketo.  Some of the improvements are:

  1. Pulldata now works in web forms so you can reference data in a CSV file or from another survey inside your survey in webforms.   Try this out here.  This demo app checks the current status of piece of inventory prior to that status being updated.
  2. New and improved widgets.  You can see how these look here,  in webforms using the Enketo widgets.xls form.  You can try it here.

The XLS forms and all the images are available to upload from the google shared drive

Webform Widget Highlights

Collapsable Groups

You can now click on the group name to hide all the questions inside it.  This makes navigating complex forms simpler.

Recording distress etc on a scale of 1-10

This is not available in fieldTask.  If you add an appearance of “distress” to an integer a thermometer will be shown for you to select an answer in the range from 1 – 10.

Draw widget

This has been available in fieldTask for many years and you can now use it inside web forms.

Annotate Widget

As above, where you take a picture and then annotate it.

Location Widgets

Smap only allows one geospatial widget per form which makes it easier to use the data with GIS systems.  You can of course include multiple widgets within a single survey by putting them inside “begin repeat” sections.  However I have put the following widget examples from the original enketo survey into seperate surveys.

  1. Geoshape.  Record an area.  
  2. Geotrace. Record a line.
  3. Hide fields.

Specifying Task Start Dates

When specifying the rules for creating tasks from submitted data you can now specify when the task should start.  Previously the task was always set to start 7 days after the data that prompted it was received.

Now you can specify either the date and time the source results were received or the contents of a date/time question in the source survey as the base for the starting time.  You can then specify an interval in days, hours or minutes after that base time for the task to start.

Version 16.05

New Features

A better “download the form” dialog

When you select the download button next to a form name on the form management page you will see a dialog asking what you want to download.   You can often download either of two versions of the form template as an XLS file.

  • The original XLS file uploaded to the server
  • The latest version of the form including modifications that you have made using the editor on the server

These two choices are now a little clearer.

download

The first choice is now the latest version of the form.  Next to the choice is the current version number.  The second choice is now the original XLS file, however if no XLS file was loaded then this choice will not be shown.  The version number for the original XLS file will always be “1”.

Logon button on the home page

You can now login and also logout from the home page.  The menus that you will see on the home page change according to your role and whether or not you are logged in.

Improved Task Management (Hosted Servers Only)

A new improved task management page has been added.  This is only available on hosted Smap servers and won’t be available in the download version.  New features include:

  • Import / Export of tasks to an XLS file
  • Viewing tasks on a calendar

tasks

version 14-12 Notifications

Here is, depending upon your location in the world, the last release of Smap for 2014 or the first release for 2015. The major change in this release is the addition of email notifications when a survey is submitted.  However the menus at the top of each page are also being progressively updated which will probably be the most visible change for most people.

Menu and Style Changes

The “bootstrap” javascript library is being implemented to replace “jquery-ui” and “foundation”.  In this release the following pages have been updated:

  • Form Management
  • Notifications
  • Locations (User Trail)

Pages will look a bit different, here is the form management page:

Menus

Smap has been divided into 4 sections:

  1. Home
  2. Admin
  3. Web Forms
  4. Analysis

These are accessible from a drop down menu labelled “View”.

dd

Drop down “View” menu

In the image above the “Admin” option is greyed out as the view option was selected from “Form Management” which is part of administration.  In a future release another section called “Edit” will be added.

Within each section the menu options will appear to the right of the “view” menu.  For administration these options are:

  • Form Management
  • User Management
  • Notifications
  • Monitoring
  • Locations
  • Tasks

Each section will also have common menu options

  • help 
  • User Settings (an icon that looks like someone’s head and shoulders)

Unfortunately because this new menu layout is being implemented progressively you will find yourself jumping back and forth between the new and old menus as you navigate from page to page.  If you find this unbearable let me know and I will try and make all the pages consistent faster.

Notifications

A notifications page has been added which allows you to request that an email be sent every time a new result is submitted to the server.

Notification Page

The notification page above shows a single notification that will send an email to two email addresses (separated by commas) when a result is submitted from the “modify stress test” survey. The email will contain a link to a web form view of the submission.

Web Form View of a Submission – Accessed by link in email

A log of all the notification events can be seen on the monitor page.

Application of email notifications

These could be used for quality assurance.  A supervisor can set up a notification for each submitted result which they can then review.  A future enhancement may allow filtering so that the emails are sent conditionally dependent on value of the data. Suggestions welcome.

Forwards

Previously you could set up a survey to be forwarded onto another server on the form management page.  Now however, forwarding is considered a type of notification.  Hence you can set up forwarding on the notifications page.

Miscellaneous Changes

  • The email host can be set per organisation.  Previously an IT administrator had to set the address of your email server (smtp_host) in the tomcat web.xml file.  Now it can be set in the users page by editing the organisation details.  Note the value for smtp_host in web.xml is now ignored so you will need to update the organisation.  
  • A default value of smtp_host can be set for every organisation on the server.  This does have to be done by an it administrator in the survey_defintions database using a command like: insert into server (smtp_host) values (‘Your smtp host’);
  • When uploading a new form to the server from an XLS file,  the name of the form will default to the name of the file.  You can optionally change the form name.
  • Increased the width of the line showing the path taken by the user on the locations screen.
  • Previously you could not select a set of survey results for editing if that survey had been marked bad.  This restriction has been lifted. You are now only prevented from editing results if that record has been replaced by another one.
  • Performance improvement to “data cleaning”.  There were some issues in performance last month when multiple people were cleaning data in the same survey simultaneously. Database deadlocks were causing updates to hang which resulted in the server running out of database connections.  These deadlocks have been fixed.

 

 

Version 14.10 – Bug Fixes

This is a minor release with just some fairly minor bug fixes.  These include:

  • French translations have been added replacing the dummy words that I had previously used for testing.  Thank you to Baba Sy for providing these translations.
  • Show an error message if a template is uploaded when the user does not have access to any projects.  (Thank you to Janna Metzler for identifying this issue)
  • Prevent the editing of results that have been marked as “bad”.  For example if they have already been edited and replaced by another record.
  • Fixed several issues with roles.  It now works much better if a user is assigned a role of analyst but not administrator or vice versa.  A user with only enum rights can now also use webForms.
  • Replaced links to the old help pages with links to the google drive holding the documentation
  • Prevent a malformed template being created if you upload a completely empty template.  This now reports an error and does not create the template.  Previously it reported the error but still created a template that could not be run on fieldTask
  • Fix paging through results on the monitor page. Previously the page buttons were hidden.
  • Thumbnails were not being created for pictures with an extension of JPG (in capitals)
  • Support added for select multiple questions when exporting to Neo4J
  • Only allow a single select multiple question to be exported to a Neo4J node or relation
  • Prevent Neo4J labels from having spaces
  • Improve clarity of reporting errors in templates when using template upload.  Multiple redundant messages were previously created when calculate questions with lots of parameters had circular references.

The server can be downloaded from http://www.smap.com.au/downloads.shtml.

 

Version 14.09 – Form Refresh and Export to Neo4j

Version 14.09 of Smap is available for download.

Refresh – Synchronising phones with the server

The refresh button on fieldTask now does more than just clear the submitted results from the screen. It also now refreshes the forms that you have on your phone and submits any finalised results.  Note you will need to download fieldTask version 2.3.5 from your server on to your phone to use this new feature.

Pressing refresh will now:

        • Download new forms from the server that have been assigned to the user of the phone
        • Show available forms with a yellow clipboard icon
        • Update forms on the phone that have a new version
        • Delete forms from the phone for which the user no longer has access
        • Submit finalised results from the phone to the server
        • Notify the server of the list of forms and their version that the user has downloaded to their phone

You can select a form from the task list screen to complete a survey.  The name, version and project containing the form are shown next to the icon.

device-2014-09-20-090412

Phone with two forms

This new feature should simplify the training of the data collectors as the steps they need to download new forms, start a survey and submit results are all simpler.

Checking that data collectors have the correct version of a form

Sometimes it can happen that data collectors go out into the field with the wrong version of a form on  their phone.  At worst this may have an impact on quality of the collected data and at best you will need to do some messy combining of the data from the two forms prior to analysis. This new feature makes it much easier to check that all the data collectors have the correct version of the form.

From the monitor page select forms.  This will show the users, phones, downloaded forms and the versions of those forms that have been downloaded.  Any users / phones that have not downloaded a form or have an older version are highlighted.

monitor

Monitor Downloaded Forms

 Exporting results to Neo4J / Thingsat

You can now export results to the Neo4J graph database. This change was developed in collaboration with thingsat.  The following video shows how this can be done.

Other Changes

  1. You can edit collected data using web forms.  This capability is enabled by an administrator for your organisation.  By default it is off.  If you select a table showing data for the whole of survey then there will be a button next to each data record that will open the survey results in a web form. If the data is updated the old results will be marked as bad, and then the updated results added.
  2. When you attempt to submit results from a phone for a survey that has been deleted on the server then the error message will be “Not Found” rather than “Not Authorised”
  3. The error message shown, if you upload a survey template that has duplicate options, has been made clearer

Smap Version 14.03 Forwarding

Smap version 14.03 has finally been released.  It should have been available from the end of March. The next release should be a small one and hopefully we can get back to the end of month regular release date.

The major new feature is forwarding of surveys.  Other changes are improvements to monitoring, support for larger result sets (paging) and a more laptop friendly deployment.

Monitoring

Totals

When you select “Monitoring” by default you will be shown total counts for success, errors, duplicates and “not uploaded”.

monitor

Totals are only available in Table view.  In order to view locations of uploaded data you need to first select the source as “Instances” then select “show as map”.

Filtering

You can filter the uploads (or forwards) that you view in the monitor page by their status. Success, errors, duplicates and “not uploaded”.  Hence you can uncheck all statuses except for “Error” in order to see which uploads in your project or survey were reported as being in error.  When showing instances rather than totals the last 200 records are still shown.  However if you apply a filter to only show errors then it will be the last 200 errors that are shown.

Paging

Paging is now available in the Monitoring screen and for the survey table view in the dashboard.  200 records are shown by default but you can then select a forward / reverse button to move forward and back through the results. These buttons are located at the top of the table and are labelled “>>>” for forward and “<<<” for back.  They will be disabled unless there are additional pages available to view.

Total Result Records

In the dashboard, when you view all survey results as a table, then the total count of surveys will be shown as well as the count of “bad” records. This wil be a faster way of getting a count of all surveys loaded than having to export to a spreadsheet.

Deployment

Changing URL of the server

Previously it was assumed that the address of the server would not change.  This allowed the server to get away with storing the full url of all survey attachments in the database.  However this assumption is completely untenable if Smap Server is deployed on laptops.  The server URL can now be changed and all the attachments will still be accessible.

Simplified deployment

Almost all the editing of files that had to be done during the installation process is now not needed.  The one exception is that the IP address of the server still has to be set in the install script.   This is only required for webForms and will hopefully be removed soon.  So, assuming you are installing on a clean Ubuntu machine, the new installation process is:

  1. Download and extract the release
  2. change directory to smap/install
  3. Edit the install script to add the IP address (temporary)
  4. Type ./install.sh

This will also now run the deploy script so once it finishes you shoulld be able to log on and use the server.

This release will be added to the downloads page over the coming weekend.  There will be another blog post to announce its availability.

Bug Fixes

  1. Error in retrieving languages on the audit page fixed.  This bug prevented auditing.
  2. If the first record in survey had an invalid geometry then export to shape files would fail. An example of an invalid geometry is recording only one point when a polygon or line was requested.