Author Archives: Neil Penman

Conversational Surveys with Upinion

2 Replies

I have been looking at Upinion which seems to be a very nice tool for “Conversational Surveys”. Upinion define these as ongoing surveys conducted in a chat messaging tool such as facebook messenger or their own app. As a company Upinion are focussed on international development with the aim to “Amplify people’s voices in a crisis situation” and could be an excellent complementary information channel to use with Smap for getting a more complete view on a situation.

Upinion

To specify the questions you are going to ask you create “conversations” using a GUI drag and drop editor. The screenshot below is from a test conversation I created. A graph of responses is shown automatically beneath each question as they are received.

Secondly you can attach a group to the conversation. This group will contain the respondents to your survey.

Finally you need to invite respondents to join a group. They might do this by scanning a barcode or clicking on a link. A key point is that the respondent needs to sign up in order to be involved in a discussion and they can revoke their membership and delete their personal data at any time.

Using Smap and Upinion together

There are a vast number of ways to collect data from the field, IOT devices, satellites / drones, surveys using enumerators, emailed or IVR surveys direct to respondents, conversational surveys as per Upinion etc.

The core Smap solution is the classic DME approach to conduct a survey with enumerators (we also support emailed surveys and webforms). This is probably the most rigorous and comprehensive way to get a snapshot of a situation at a point in time. However it does have a number of disadvantages. For example preparing the survey and collecting the data can take some time and has a reasonable cost. (Much quicker and cheaper than paper surveys though!). This cost and time tends to mean that you need to ask a lot of questions as you may not be back for a while and you are not sure what you need when the survey is being designed. Of course large numbers of questions can make responding to a survey time consuming for the respondents as well.

In order to get ongoing continuous data after the survey completes, the survey could identify people who have a smartphone and access to a network, and who would be prepared to act as a community contact or sentinel. They can be given a barcode to scan which will sign them up to an Upinion group. From that point on you can conduct conversations with these contacts to get a real time view of their situation and the situation in their community. Based on the information received you may then conducted another more targeted survey of the broader community using Smap.

Smap has relocated to Brisbane

Leave a reply

The Smap Consulting headquarters has moved from Canberra to Brisbane. The primary reason is to focus on projects in the South Pacific, Brisbane being a base for a lot of this work. We will continue of course to support all of our customers anywhere in the world and travel will be quicker and easier from here.

The company address is now:

7/47 Herston Road
Kelvin Grove,
Queensland, 4059
Australia

[osm_map_v3 map_center=”-27.468,153.024″ zoom=”14″ width=”100%” height=”450″ post_markers=”1″]

Information Management

Leave a reply

Smap’s advanced capabilities in; linking data between forms, task management, unique key management, case management and notifications allows you to connect forms in order to create an information management system.   That is a set of collaborating forms that address one or more business processes in your organisation.

To illustrate this capability I have created a demonstration of a system that supports the registration and enrollment of students in education classes.   This demo is available on the Smap https://sg.smap.com.au server for you to try out.

The attached pdf provides detailed instructions on how to try the demo.  Hopefully this example will inspire you to create your own information management system using Smap.

Click to access studentEnrollment+minimal.pdf

Release 18.12

Leave a reply

Updates

Changes to the layout of the user administration page

The new feature to allow a user to belong to more than one organisation has prompted some changes to the user administration page.  The main one is the removal of the button to move a user to a different organisation.   Instead when you open the user for editing then, if you have organisation administration privilege, you will be able to set the users current organisation.

Set the users current organisation

Previously you would select one or more users by clicking on a checkbox next to their name and then click on the “Move to Organisation” or “Delete” button.  A delete button has now also been added next to each user so there is no longer a need to select users prior to taking an action.  The “Add User” button has also been moved above the list of users.

This is work in progress and their will be more changes to the user administration page layout in the next release.

Timezones

The management of timezones for reports and viewing data has been improved.

  1. You can set your timezone in your user profile.  (Available on all pages)
  2. The default timezone is the one your browser is set to however you can change this and it will be saved with your other settings.
  3. Exports to XLS include a new tab that shows the timezone used for all dates and times.
  4. You can set the timezone for a report independently of your personal time zone and this will be saved with the report

Legacy Reports

The old way of creating reports as snapshots of panels on the dashboard has been removed.

Administration Reports

Two reports have been added to the form management page:

  1. Usage.  How many submissions have been submitted per user for each project or form
  2. Form Access.  The users in your organisation have access to a form and why they have access or do not have access. This report considers the projects a user has access to, their security groups and their security roles as well as whether or not the form has been deleted or blocked.

Performance Improvements

As the number of people using Smap increases the size of data tables also increases and it becomes apparent that additional indexes are required.  2 new indexes were added in this release to improve the performance of messaging and submission processing.

Release 18.11

Leave a reply

There have been lots of mini updates over the last 4 months but no formal new releases until now. It is November and its time to get back to monthly releases.

New Features

1. A user can belong to more than one organisation. 

Many people find that they are working on multiple projects in different organisations.  Previously you would have needed more than one logon or would have had to keep asking the organisational administrator to move you from project to project.  Now however you can belong to multiple organisations, you can select the organisation you want to work in by clicking on your profile in any page.

All you projects, security groups and role are preserved for all the organisations that you belong to.  Hence you don’t need to set yourself up each time you switch organisations.  The security groups can be different between organisations so you might be an administrator in one and an enumerator in another.

To set this up go to the users admin page, select a user and then click on the organisations that they have access to.  Note you need to have “organisational admin” privilege to do this.

2. Set Max Image Size from Server

You can now specify the maximum image size in the fieldTask options on the user administration page.  This maximum value will be set on the phones when you refresh.

3. Referencing the survey key (HRK) and preloads in the advanced filter

You can now create filters such as: ${_hrk} = ‘abc 001’ and ${_device} = ‘1111111’

Two Million Submitted Surveys

2 Replies

We have just gone past the 2 million mark in the number of surveys submitted to the Smap Consulting servers.   It took 6 years for the first million and 11 months for the second. So usage is accelerating.

Smap started as a project involving World Vision, RMIT and IBM in 2008 with the aim of making data collection faster, with better quality and lower cost.   Nowadays it is being used on a daily basis by organisations and individuals to achieve those goals for themselves.

Release 18.7

2 Replies

Its been a bit slow over summer.  We are now announcing the july release and yet we are already into August.

Changes include:

  • rank question type
  • randomize appearance on select questions

Rank Question Type

This is just like a select multiple question except instead of choosing one or more options you can order them.

Adding to your survey

Specify in an XLS Form in the same way you specify a select multiple question except that you replace “select_multiple” with “rank”. Then you have a space and the list name that points to all the choices that you want to rank. So:
        type                          name                           label
rank my_list             question_name          Rank this list!

In the online editor select the question type called “rank” and then add the choices.

Using

The rank question works in FieldTask and WebForms.  In both devices you drag the choices into the order that you want.

 

Rank Question in WebForms

Analysis

If you export the data into an XLS report you will see that the ranked questions are shown in a similar way to select multiples with a column per choice.  However the column headings are the question name – the rank.  And the contents are the name of the choice.

Answers to a rank question in an XLS report.

Limitations

Not all exports support rank yet. The support in the API is also limited.  You also can’t currently use it in filters. For example get all data where london was marked as the first choice.

Randomize

If you add an appearance of randomize=true for a select or rank question then in fieldTask and WebForms the order in which the choices are shown will be randomised.  This can be useful to reduce the impact of bias in choice selection on the survey results.   Randomize currently only works on FieldTask and not on Webforms. No it works in webforms too.  Thanks to Alexander Torrado for picking this up.

Bugs Fixed

  • Choices pulled from another form were not showing in webforms.  If you looked up choices in a CSV file then these did show in webforms however if the choices were in another form they did not.

Update 8th August 2018 to add the bugs fixed section and to note that randomize works with webforms but you need to set it as a parameter and not in appearance.

Release 18.6

Leave a reply

Email Tasks

The ability to send pre filled or blank forms as links in  emails out to people for completion in their browser.  Capabilities include:

  1. Forms can only be filled in once by each recipient (unless the repeat flag is set)
  2. The URL to access the form includes a key so that the user does not need to have an account on the server
  3. You can upload email address and form names from a file and then have the emails sent to all of those addresses
  4. There is an unsubscribe link in all emails
  5. People can resubscribe, if they change their mind by going to the subscriptions page (on the home page select subscriptions)
  6. Emails with a pre-filled in form can be sent automatically when a submission is received for any other form

Really what this does is enable you to extend you data collection beyond the professional data collectors who have a logon on the server.  Hence you can do email surveys, as per Survey Monkey and also integrate input from external people in a multi step process.  So in fact Smap goes way beyond the capabilities in Survey Monkey.

Note: Email tasks are disabled on the sg server by default. Contact us if you want to use this feature.

Auto sending email tasks

You can also assign tasks that include previously collected data to an email address .   As with other tasks, that are completed by logged in users,  you can create these from previously submitted results or results as they come in.  Here is a video of an example process but there are an infinite number of ways you can use this feature:

After completing the steps shown in the above video and then doing an XLS report I got the following output:

Loading Email Addresses from an Excel File 

Email tasks are managed from the tasks module,  From there you can create a task group and then click on the file menu to download an XLS file.  In this case for new task group it will be empty but you can specify the “tasks” here.

The columns with blue headings are where you put details about the form.  (See above). The columns with yellow heading contain details about the person being sent the email including of course their email address.  You can add multiple email addresses to be sent a single form hence the greyed out cells in the above example where we have one form but two email addresses.  Also you don’t need to fill in the URL column, this is only filled in when exporting in case you want to manually send the links out without using Smap’s email manager.

Once you have filled in your forms and email addresses.  Click on the Menu File > Load XLS.  Emails will not immediately be sent, you will then have the opportunity to review the loading emails and when you are ready you can select the menu “Email Unsent”.   This will send the emails with blank forms.

Unsubscribe From emails

This became urgently needed due to the addition of the “Email Tasks” functionality, however the unsubscribe link is now added to any email sent from the Smap server including for example an email adding you as a registered user.  If someone clicks on that link they are asked if they want to confirm the un-subscription.  If they say yes then they won’t be sent any more emails.   They only way to resume sending emails to an unsubscribed address is if you can persuade them to:

  • Goto to the server home page
  • Click on the subscriptions menu
  • And subscribe again

Updates to FieldTask version 5.5

Well this was released some time ago but I didn’t make an announcement on this news site so here it is.  Lots of new features here contributed by the folk from ODK as well as the fieldTask developers. Some of the main ones are:

  • Online lookup of data from a server while the user is completing the form. You can lookup reference data, choices for a select or query an AI web service.
  • Non printable characters in barcodes will now show as spaces rather than being removed completely
  • Includes ODKCollect 1.15
    – Can set a dark or light theme to change the screen appearance
    – Add a “file” type question that allows you to attach any file to the form

 

 

 

Server Outage

Leave a reply

I’d like to apologies for users of the sg server who experienced a 5 hour outage today. This should not have happened and was the result of deficiencies in my IT processes.

The number of surveys submitted to Smap hosted servers has been increasing over the last 3 years and is now averaging nearly 4,500 per day.

On the free sg server it is around 2,000 per day.  However on the 13th and 14th of June we received 11,897 submissions 80% of which included a high resolution image of a cocoa tree. This caused a big drop in the available disk space.

The IT process went:

  1. I received a text message that we were down to 4GB at 5pm UTC.
  2. Immediately ran some clean up scripts that freed up 9GB,
  3. I then added 60GB extra disk to the server.  However this disk is not made available until you reboot the server.
  4. The server was still being heavily utilised so I decided to wait until later in the evening to reboot when, given that most users are in Africa or Asia, the load should have been less.
  5. Then I forgot!

So instead of a 30 second outage we got 5 hours.  I will endeavour to ensure that this does not happen again.

Adding Artificial Intelligence to your forms

Leave a reply

Single-layer feedforward artificial neural – By Akritasa [CC BY-SA 4.0 (https://creativecommons.org/licenses/by-sa/4.0)]

Smap Forms are a great way of implementing what used to be known as “Expert Systems”.   While you are collecting data with a Form questions can be displayed or hidden based on their perceived “relevance”.   In the same way recommendations and feedback can be shown based on the answers that have been provided.  If A and B then show C.

One reason that this is great with Smap is that the expert advice can be embedded easily in a system whose primary function is the humdrum capturing of information.  It is not necessary for the user to sit down for a dedicated session of receiving expert advice.  Another reason is that a domain expert can create these forms without needing input from a programmer.

However these rules are a long way from Artificial Intelligence.  Once your form has collected the answers to 20 plus questions as multiple choice, text, audio, images and video then there may be critical information that you want to feedback to the user immediately hidden in that data that you cannot extract with a few “if” statements.

Version 5.5 of fieldTask (available from the google play store) adds the ability to call a server passing it data you have collected and to get back a response that can be used to guide the next steps.  You need to be filling out the form on line of course and the service you call need not be an Artificial Intelligence web service, it may just look up reference data, however we see the use of AI services in this way as particularly compelling.

You can try this out in fieldTask by adding a “calculate” that calls a lookup_image_labels() function.  So in an xlsForm your questions might look like this:

type           name                      label / calculation
image        scene                      Take a photo
calculate   scene_objects        lookup_image_labels(${scene})
note           show_objects        Objects in photo are: ${scene_objects}

And this is what the above form looks like on the phone.

Firstly Take the Photo

And here are the objects that were identified

Well this is not actually very useful!

The AI service called by lookup_image_labels() is the AWS Rekognition service that can identify things in photographs such as cars, people, computers, desks etc.   Even that task is done far from perfectly, it did not identify the  bowl, unwashed or otherwise.  And it will probably identify nothing that the person filling in the form could not identify for themselves.   This information would be useful on the server in order to search for images  but you can already generate these labels automatically, using Smap, once the form has been submitted.  It hardly seems necessary to make the person completing the form wait for these labels to be returned from the server.

So why do it?

Well actually the Rekognition service is just an example.  We can add calls to your custom AI engines that can identify patterns in the data or images and that may be critical in identifying the advice to be provided or the further information that needs to be collected.

Smap can be used to collect the data required to train an AI engine and now it can use the decisions from that AI engine in collecting more data creating positive feedback to add value to your processes.