Two Million Submitted Surveys

We have just gone past the 2 million mark in the number of surveys submitted to the Smap Consulting servers.   It took 6 years for the first million and 11 months for the second. So usage is accelerating.

Smap started as a project involving World Vision, RMIT and IBM in 2008 with the aim of making data collection faster, with better quality and lower cost.   Nowadays it is being used on a daily basis by organisations and individuals to achieve those goals for themselves.

Release 18.7

Its been a bit slow over summer.  We are now announcing the july release and yet we are already into August.

Changes include:

  • rank question type
  • randomize appearance on select questions

Rank Question Type

This is just like a select multiple question except instead of choosing one or more options you can order them.

Adding to your survey

Specify in an XLS Form in the same way you specify a select multiple question except that you replace “select_multiple” with “rank”. Then you have a space and the list name that points to all the choices that you want to rank. So:
        type                          name                           label
rank my_list             question_name          Rank this list!

In the online editor select the question type called “rank” and then add the choices.

Using

The rank question works in FieldTask and WebForms.  In both devices you drag the choices into the order that you want.

 

Rank Question in WebForms

Analysis

If you export the data into an XLS report you will see that the ranked questions are shown in a similar way to select multiples with a column per choice.  However the column headings are the question name – the rank.  And the contents are the name of the choice.

Answers to a rank question in an XLS report.

Limitations

Not all exports support rank yet. The support in the API is also limited.  You also can’t currently use it in filters. For example get all data where london was marked as the first choice.

Randomize

If you add an appearance of randomize=true for a select or rank question then in fieldTask and WebForms the order in which the choices are shown will be randomised.  This can be useful to reduce the impact of bias in choice selection on the survey results.   Randomize currently only works on FieldTask and not on Webforms. No it works in webforms too.  Thanks to Alexander Torrado for picking this up.

Bugs Fixed

  • Choices pulled from another form were not showing in webforms.  If you looked up choices in a CSV file then these did show in webforms however if the choices were in another form they did not.

Update 8th August 2018 to add the bugs fixed section and to note that randomize works with webforms but you need to set it as a parameter and not in appearance.

Release 18.6

Email Tasks

The ability to send pre filled or blank forms as links in  emails out to people for completion in their browser.  Capabilities include:

  1. Forms can only be filled in once by each recipient (unless the repeat flag is set)
  2. The URL to access the form includes a key so that the user does not need to have an account on the server
  3. You can upload email address and form names from a file and then have the emails sent to all of those addresses
  4. There is an unsubscribe link in all emails
  5. People can resubscribe, if they change their mind by going to the subscriptions page (on the home page select subscriptions)
  6. Emails with a pre-filled in form can be sent automatically when a submission is received for any other form

Really what this does is enable you to extend you data collection beyond the professional data collectors who have a logon on the server.  Hence you can do email surveys, as per Survey Monkey and also integrate input from external people in a multi step process.  So in fact Smap goes way beyond the capabilities in Survey Monkey.

Note: Email tasks are disabled on the sg server by default. Contact us if you want to use this feature.

Auto sending email tasks

You can also assign tasks that include previously collected data to an email address .   As with other tasks, that are completed by logged in users,  you can create these from previously submitted results or results as they come in.  Here is a video of an example process but there are an infinite number of ways you can use this feature:

After completing the steps shown in the above video and then doing an XLS report I got the following output:

Loading Email Addresses from an Excel File 

Email tasks are managed from the tasks module,  From there you can create a task group and then click on the file menu to download an XLS file.  In this case for new task group it will be empty but you can specify the “tasks” here.

The columns with blue headings are where you put details about the form.  (See above). The columns with yellow heading contain details about the person being sent the email including of course their email address.  You can add multiple email addresses to be sent a single form hence the greyed out cells in the above example where we have one form but two email addresses.  Also you don’t need to fill in the URL column, this is only filled in when exporting in case you want to manually send the links out without using Smap’s email manager.

Once you have filled in your forms and email addresses.  Click on the Menu File > Load XLS.  Emails will not immediately be sent, you will then have the opportunity to review the loading emails and when you are ready you can select the menu “Email Unsent”.   This will send the emails with blank forms.

Unsubscribe From emails

This became urgently needed due to the addition of the “Email Tasks” functionality, however the unsubscribe link is now added to any email sent from the Smap server including for example an email adding you as a registered user.  If someone clicks on that link they are asked if they want to confirm the un-subscription.  If they say yes then they won’t be sent any more emails.   They only way to resume sending emails to an unsubscribed address is if you can persuade them to:

  • Goto to the server home page
  • Click on the subscriptions menu
  • And subscribe again

Updates to FieldTask version 5.5

Well this was released some time ago but I didn’t make an announcement on this news site so here it is.  Lots of new features here contributed by the folk from ODK as well as the fieldTask developers. Some of the main ones are:

  • Online lookup of data from a server while the user is completing the form. You can lookup reference data, choices for a select or query an AI web service.
  • Non printable characters in barcodes will now show as spaces rather than being removed completely
  • Includes ODKCollect 1.15
    – Can set a dark or light theme to change the screen appearance
    – Add a “file” type question that allows you to attach any file to the form

 

 

 

Server Outage

I’d like to apologies for users of the sg server who experienced a 5 hour outage today. This should not have happened and was the result of deficiencies in my IT processes.

The number of surveys submitted to Smap hosted servers has been increasing over the last 3 years and is now averaging nearly 4,500 per day.

On the free sg server it is around 2,000 per day.  However on the 13th and 14th of June we received 11,897 submissions 80% of which included a high resolution image of a cocoa tree. This caused a big drop in the available disk space.

The IT process went:

  1. I received a text message that we were down to 4GB at 5pm UTC.
  2. Immediately ran some clean up scripts that freed up 9GB,
  3. I then added 60GB extra disk to the server.  However this disk is not made available until you reboot the server.
  4. The server was still being heavily utilised so I decided to wait until later in the evening to reboot when, given that most users are in Africa or Asia, the load should have been less.
  5. Then I forgot!

So instead of a 30 second outage we got 5 hours.  I will endeavour to ensure that this does not happen again.

Adding Artificial Intelligence to your forms

Single-layer feedforward artificial neural – By Akritasa [CC BY-SA 4.0 (https://creativecommons.org/licenses/by-sa/4.0)]

Smap Forms are a great way of implementing what used to be known as “Expert Systems”.   While you are collecting data with a Form questions can be displayed or hidden based on their perceived “relevance”.   In the same way recommendations and feedback can be shown based on the answers that have been provided.  If A and B then show C.

One reason that this is great with Smap is that the expert advice can be embedded easily in a system whose primary function is the humdrum capturing of information.  It is not necessary for the user to sit down for a dedicated session of receiving expert advice.  Another reason is that a domain expert can create these forms without needing input from a programmer.

However these rules are a long way from Artificial Intelligence.  Once your form has collected the answers to 20 plus questions as multiple choice, text, audio, images and video then there may be critical information that you want to feedback to the user immediately hidden in that data that you cannot extract with a few “if” statements.

Version 5.5 of fieldTask (available from the google play store) adds the ability to call a server passing it data you have collected and to get back a response that can be used to guide the next steps.  You need to be filling out the form on line of course and the service you call need not be an Artificial Intelligence web service, it may just look up reference data, however we see the use of AI services in this way as particularly compelling.

You can try this out in fieldTask by adding a “calculate” that calls a lookup_image_labels() function.  So in an xlsForm your questions might look like this:

type           name                      label / calculation
image        scene                      Take a photo
calculate   scene_objects        lookup_image_labels(${scene})
note           show_objects        Objects in photo are: ${scene_objects}

And this is what the above form looks like on the phone.

Firstly Take the Photo

And here are the objects that were identified

Well this is not actually very useful!

The AI service called by lookup_image_labels() is the AWS Rekognition service that can identify things in photographs such as cars, people, computers, desks etc.   Even that task is done far from perfectly, it did not identify the  bowl, unwashed or otherwise.  And it will probably identify nothing that the person filling in the form could not identify for themselves.   This information would be useful on the server in order to search for images  but you can already generate these labels automatically, using Smap, once the form has been submitted.  It hardly seems necessary to make the person completing the form wait for these labels to be returned from the server.

So why do it?

Well actually the Rekognition service is just an example.  We can add calls to your custom AI engines that can identify patterns in the data or images and that may be critical in identifying the advice to be provided or the further information that needs to be collected.

Smap can be used to collect the data required to train an AI engine and now it can use the decisions from that AI engine in collecting more data creating positive feedback to add value to your processes.

 

Version 18.05

Time for the May release of Smap.

Updated Webforms

Webforms has been upgraded to use the latest version of Enketo.  Some of the improvements are:

  1. Pulldata now works in web forms so you can reference data in a CSV file or from another survey inside your survey in webforms.   Try this out here.  This demo app checks the current status of piece of inventory prior to that status being updated.
  2. New and improved widgets.  You can see how these look here,  in webforms using the Enketo widgets.xls form.  You can try it here.

The XLS forms and all the images are available to upload from the google shared drive

Webform Widget Highlights

Collapsable Groups

You can now click on the group name to hide all the questions inside it.  This makes navigating complex forms simpler.

Recording distress etc on a scale of 1-10

This is not available in fieldTask.  If you add an appearance of “distress” to an integer a thermometer will be shown for you to select an answer in the range from 1 – 10.

Draw widget

This has been available in fieldTask for many years and you can now use it inside web forms.

Annotate Widget

As above, where you take a picture and then annotate it.

Location Widgets

Smap only allows one geospatial widget per form which makes it easier to use the data with GIS systems.  You can of course include multiple widgets within a single survey by putting them inside “begin repeat” sections.  However I have put the following widget examples from the original enketo survey into seperate surveys.

  1. Geoshape.  Record an area.  
  2. Geotrace. Record a line.
  3. Hide fields.

Specifying Task Start Dates

When specifying the rules for creating tasks from submitted data you can now specify when the task should start.  Previously the task was always set to start 7 days after the data that prompted it was received.

Now you can specify either the date and time the source results were received or the contents of a date/time question in the source survey as the base for the starting time.  You can then specify an interval in days, hours or minutes after that base time for the task to start.

Hint: Changing the question after you have started collected data

The replace button on the form management page lets you replace one survey with another at any stage of the data collection process.  There are no restrictions on what you can change and this can cause some issues if you change a question type.

For example a customer had a question in their form “How far away from home is your work?”.  They originally specified this question as “decimal” and had collected data where it was recorded as 3.0, 4.5 etc.  However they later changed the type of the question to “text” and the submitted results contained values like “5 minutes”.  Unfortunately these results were not being applied to the database so they could not see the later results.

Cause of the Problem

Once you start submitting data for a form, a table is created to store that data.  The answers to different types of questions are stored differently:

  • integer question answers are stored as Integer
  • decimal are stored as double precision
  • text, select_one, select_multiple, barcode, image, video, audio are stored as text
  • geopoint are stored as geometries

So if you then change a text question to a select there is no problem but if you change an integer to a text then the data will not load.

Detecting the Problem

The monitor page will show the the errors and list the results that were not loaded.  To see these:

  1. Select the “Admin” Module
  2. Select “Monitoring”
  3. Set the source to “Submitted”
  4. Select “Show Instances”
  5. Select your project and survey if there are a lot of submissions from other surveys shown

The errors indicating a problem with the question type will look like:

ERROR: invalid input syntax for type double precision: “5minutes” Position: 127

So a bit cryptic but a problem with question type.  Its trying to write some text into a decimal question’s database column.

Fixing the Problem

Usually the problem can be fixed simply from the Analysis module.  Open a table view of your survey.  This will show the results that have been loaded.  Click on the icon “<” at top right to open the aside.

There are two useful buttons here: “Delete Data” and “Restore Data”.  However note the “warnings” section below before using them.

Case 1: You are still testing the form

If you are still testing the form select “Delete Data”.  As well as deleting the test data this will also delete the tables that store the data so next time you submit some results it will recreate these tables but using the latest question types.

Case 2: You have some valuable data loaded (or submitted)

In this case select “Restore Data”.  This will also delete the tables but it will then reload all of the data from the original submissions.

Warnings

Using “Restore Data” will

  1. delete any modifications you may have made to data after it was collected.  For example fixing spelling mistakes using the modify page.
  2. resend any notifications attached to your survey.  For example you may be sending an email with a PDF containing details on each uploaded record.  If you don’t want these emails to be resent then you can disable notifications while you do the restore.  However any records that had failed to load due to the question type change will then also not trigger the notification. So tricky.

Taking note of these warnings it is a good idea to change the data types of questions with care.

Tutorial 1 – Data Management

There are several ways to manage collected data and the best approach depends on the problem that you are trying to solve.

  1. Using the Modify page.  You may be an administrator and you want to fix errors in the data such as misspelt locations, other text used to answer a select question when a choice should have been selected and so on.  This is sometimes called “data cleaning” and it can be done quickly and easily in the modify page.
  2. Oversight forms.  You may want to add notes to a data record or update a field such as a priority and this may be done as part of an ongoing process.  In this case you can add an oversight form to the data collection form and if if it has questions with the same name as the data collection form then the answers to these questions can be changed.
  3. Tasks.  You may want to send somebody out to update a previously submitted form and you want them to change existing data and add new data. You can assign a task to do this.
  4. Webforms.  You may want to completely go over a single submission and change many answers.  In this case you can edit the data using a webform.
  5. Export / Import.  Alternatively if you want to completely revamp all your data in a spreadsheet you can export it and then re-import the updated spreadsheet.

This tutorial will just cover the first of these which is the “Modify Page” approach.

New Feature added 18th April 2018.  A checkbox has been added to the main screen which is labelled “Select data to change using a primary key”.  This allows you to select a specific record in submission to change.  Its use is covered in a new section after “updating other question types”.

Getting Started

Modify is part of Analysis so after you logon select the Modules menu, then Analysis and finally the Modify menu.

Updating Text

Basic Text Questions

In my sample form I collected some data that included a text question asking for the country.  On the modify page I selected this “Sample Form” survey and then the country question.

Each distinct answer is shown along with a count of how many times in the survey that answer was given.  So you can see in the above table that “Uganda” was entered 3 times.  “France” was entered only once however there is also an entry for a country called “Franc”.  I’m going to assume that’s a typo and so I click on the “Update” button to the right of that text value.

A dialog box is shown.  At the top is written the action that is going to be performed: “Modify Franc To”. Then there is a text box labelled “Reason”.   I have entered “Spelling Mistake” in here.  This reason will be shown in the audit trail of changes.  Next there is a select box called “Existing Choice” which is disabled because there are no select choices associated with this question. Next is a select box called “Existing Text”.  This allows you to select any of the other answers that have been submitted for this question.  In this case I selected the answer “France”.  Finally there is another text field where you could have entered any new value for the text that you wished.

After you press the Save button you will see the updated results.

Now France is entered twice.  Exported results and graphs will use this updated data.

Replacing an “other” text question with a select choice

This time I selected the question “transport_other”

You can see there are 3 distinct values.  The first one is blank and it occured 4 times.  This is the number of times that the “other” text question was not relevant because a choice was selected and hence no answer was entered.  The values of “Scooter” and “Truck” were each entered once.  I am going to replace the answer “Scooter” with one of the choices in the select question and so I click on the “Update” button next to the “Scooter” text.

For the reason I have entered “Categorise Scooter as Motorbike”.  The “Existing Choice” select box is now enabled and I selected one of the choices from the select question which was “Motorbike”.  The system knows that the select question is related to the other question because there is a relevance on the other question that references the select question.  Rather than selecting an existing choice I could also have just changed the text in other to say “Moped”.

After clicking Save you will see that “Scooter” has gone and there are now 4 blank answers for other.

However if you export the data you will see that the transport select question now has an additional response for “Motorbike”.

Adding new select choices to replace other text

This is done exactly the same way as replacing text with an existing choice.  However you first need to add your additional choice to the select question, either using the online editor, or in XLSForm and then replace the existing form with the updated one.  You will then be able to select the new choice.

Updating Other Question Types

The previous sections covered updating text answers using only the existing answers to choose what will be updated.   This is great for changing all occurrences of a misspelt name however sometimes only some values need to be updated.  For example if you need to fix a wrongly entered age for a person then you don’t want to change all occurrences of that age.

Modify allows you to update text, select one, integer and decimal answers that have a specific value and also a text question with a specific value.  Note this might not always be discriminatory enough to get just the record you want to update in which case you will need to use a different method such as Webforms or an Oversight form.

I’m going to change an “age” question.  First I select the name question.  This has to be a text question. Then I check the checkbox “Update a different question” and then I selected the “age” question.

You can see in the results that there were 2 people called Richard one of whom has an age of 34 and the other 55.  I’m going to change the age of 55 to 57 by overtyping it and then pressing the update button at the bottom of the list of data.  That is all there is to it.

Updating Other Question Types For a Single Record

You may want to just change a value that is in error on a specific submission.  You can now do this by selecting the checkbox “Select data to change using a primary key”.

You will see that the drop down fields for selecting language and question will be replaced by a drop down to select the Form.  The checkbox “Update a different question” will also be automatically selected and disabled so you cannot change it.  In other words you are not allowed to change the primary key value only the value of another question.

You will also see the the “Text” value for each record on the left hand side of the form is a number which will correspond to the unique key of the record that you want to replace.

For the top level form discovering the primary key for the record that you want to change is easy.  This key is included in all exports of data.  If you want to update a specific value in a subform then you will need to use the table view in the analysis dashboard to find the unique key in the subform.

Reviewing and Reversing Changes

All changes to the data are audited and can be reversed.  To see the audit log select the “Review” Menu.

Above you can see the audit trail for the sample form where I recategorised scooter as a motorbike and changed the name of the country to France.  Each change includes the reason, a description of what happened which is created by the system and the name of the person who made the change.  If you click on the details button it will give more information about the change including listing the primary keys of all the records that were changed.  To reverse the change you simply click on the reverse button and the change will be undone.

There is a caveat here, you can reverse changes in any order however if one change depended on another and you don’t reverse them from last change to first then you may be left with some altered data in the system. If that were to happen then an error should be shown and you can fix the problem some other way.

Accountability Product

Announcing another “Product” from Smap.  These products consist  of form templates for and instructions on how to set them up on a Smap server to address a specific objective.  Or at least start to address that objective as you will want to customise the forms to suit your needs.

Overview

The accountability product includes 2 forms the first of which is a feedback form that can be completed on a mobile phone or in a web browser. The second is an oversight form that is attached to the feedback form and that can then be used to record actions taken in response to the feedback.

Feedback Form

The feedback form is just a standard form that can be completed in the fieldTask application on a phone or tablet as well as in web-forms in a browser. If you were taking feedback in a call centre then you would presumably use web-forms. Alternatively if you were in a remote location without data connectivity then you would probably use fieldTask.

Oversight Form

The oversight form has the following data entry fields:

  1. Responsible person. A text field to enter who is responsible for processing the feedback.
  2. This has the same name as the priority question in the feedback form and hence it shows the same value. The priority can however be changed from the value initially set in the feedback form.
  3. Translation of Feedback. This is a text question that is linked to the feedback description set in the feedback form.
  4. Transcription of the issue statement. The manager can play the audio statement and type the text into the text field.
  5. Image labels. The labels shown were generated automatically by Amazon Rekognition. However they can then be manually modified by the user.
  6. Date of any action taken

Getting Started

You can download a zip file containing this product and a PDF with instructions from the Smap Products web page.

 

Version 18.04

Improved Performance

Refreshing fieldTask will now be faster.  Note This requires fieldTask 5.33.  The changes are:

  1. If you change an image or other media file that is used in a question or choice then that new media file will be downloaded automatically to the mobile device using push notifications.
  2. A media file, including a CSV file, will now only be downloaded to a phone if it has changed.
  3. Media and CSV files are only downloaded once if they are used by more than one form on the device.  In other words as long as the media file is coming from “Shared Resources” it is also shared between forms on the device.
  4. If the media changes but the form has not changed then the form is not re-downloaded only the changed media files are.

You can expect substantial reductions in download time under the following circumstances:

  • You have a form that uses many images in questions and choices but you are making changes to that form by adding new questions or changing the question text.  Now after each form change only the new form will be downloaded and not all the images.
  • You have a csv file that is constantly being changed.  Now only the changed csv file will be downloaded and not the form and any other csv files or media.

However performance may be slightly worse if you have many forms that use media and only one is updated.  The reason being that during refresh the manifest will now be checked for all the forms that use media to see if the media file has been changed.  Previously the change in a media file would not have been picked up and the form would not have been updated on the device.  This is a quick operation and will only be noticed if the number of forms is quite large.

Other Features

Filtering on table view in dashboard

You can now use “advanced filtering” in these table views.  For example adding a filter such as

${district} = ‘capital’ and ${gender} = ‘female’

The number of records that match the filter as well as the total number of records will be shown just above the table of data.

Issues Fixed

  • Issue 63.  Labels are not shown in PDF exports if the label is for a select question that uses an external CSV file to get its choices
  • Issue 153.  When you download data and upload again in Analysis there is a warning message about “question name missing”
  • Issue. The Legacy XLS export does not allow select multiples to be split into separate columns.
  • Issue.  Replacing a survey that used uncompressed select multiples results in the select multiple questions being converted to compressed but the old data is not converted making it invisible.