Category Archives: Uncategorized

Version 19.03

New Features

  1. Including data from linked surveys into a PDF
  2. New XLSX report on the task management page that will list all tasks in all task groups. Exports to XLSX for tasks in the current task groups are still available.
  3. Manage a password policy for fieldTask.
  4. Add support for the ODK setting “guidance_hint” when loading xlsForms
  5. Add support for the ODK setting “body::intent” when loading xlsForms
  6. Add a tasks API to return tasks in geojson format
  7. Add an audit API to return geoJson data on location and duration of each question asked in the survey
  8. Add the user identifier as a valid value for use in an advanced filter
  9. Record the PDF template for a survey in the surveys change log so that previous versions can be referenced
  10. Add ability to delete meta questions in the online editor as well as add the pdf_no appearance to them so that they are not included in PDF exports

Including data from linked surveys in PDFs

Smap allows you to combine Surveys to make “Systems“, where data in one survey can be linked to data in another. For example a survey recording the delivery of aid may include the identifier of the beneficiary. You can analyse this data taking account of the links between them by extracting the data into external tools like Power BI. However inside Smap itself we haven’t provided any capabilities to join data from different surveys during analysis. This new feature is the first step on that path.

When you create a PDF containing data from a single survey submission you can optionally specify:

  • Include reference data
  • Only include data from surveys launched from the current survey

The first option requires you to have specified a child subform as being launched from the survey you are exporting. You also need to specify a key question in the child survey.

If you include the second option you don’t need to specify a key question but only data collected by launching the child survey on the phone will be included.

Password Policy settings for FieldTask

FieldTask Version 6.02 has a logon page.  By default this is shown when the app is first run so the user can set the server, username and password without having to go to the settings menu.   After successful logon it is not shown again.  This corresponds to the default password policy of “Never show logon page”.  In the tab for fieldTask options on the usermanagement page you can also specify a policy of “Always show logon page” which will require the user to enter their password every time they start the app;  and “Periodically show logon page” which will require the user to enter their password again a specified number of days after their last successful logon.

Bugs Fixed

  1. Large API exports were causing out of memory errors. These data exports using /api/v1/data are now streamed so there should not be any size limits.

Version 19.01

New Features

User activity views have been added to the dashboard. If you open a table view or a map view then you can specify the subject as “User Activity”. Then select a user to show their activity.

The source of data, such as the form name that was used to collect it, or the file name that was imported is now shown in the table view on the dashboard.

Added display_name to choices. Display names have been available for questions for a while. These allow you to specify a name to replace the question name in reports but one that is not constrained by the limitations of the odk question name. For example you can include spaces. These are now available for choices. This is particularly useful for select_multiple questions as now you can replace default column headers such as “Question Name – choice_name” with something more meaningful such as “The selected choice 10 !!!!”.

Add the ability to restrict activity within an organisation. For example you can prevent the submitting of any new survey results, access to the data api or sending of notifications. If any change is made to these settings the organisation owner will be notified by email.

Added some initial customisation of the webform page appearance. For example you can now set background colours for some of the components.

Add a new data processing option to reports. Transformation of data from Long to Wide. hence if you have done a longitudinal survey you can get a report on things within that survey with one row per thing and all the time series data converted to additional columns.

Added some settings data into a settings workbook when creating an excel report. This now shows the timezone used in the report as well as the start and end time of any date filtering applied.

Bug Fixes

  • Replacing a select_one question with a select_multiple question after data collection had started caused further submissions to the form to fail. Now fixed.
  • Multiple bug fixes in webforms. Images in choices now work again as does the selection of drop down select questions.

Conversational Surveys with Upinion

I have been looking at Upinion which seems to be a very nice tool for “Conversational Surveys”. Upinion define these as ongoing surveys conducted in a chat messaging tool such as facebook messenger or their own app. As a company Upinion are focussed on international development with the aim to “Amplify people’s voices in a crisis situation” and could be an excellent complementary information channel to use with Smap for getting a more complete view on a situation.

Upinion

To specify the questions you are going to ask you create “conversations” using a GUI drag and drop editor. The screenshot below is from a test conversation I created. A graph of responses is shown automatically beneath each question as they are received.

Secondly you can attach a group to the conversation. This group will contain the respondents to your survey.

Finally you need to invite respondents to join a group. They might do this by scanning a barcode or clicking on a link. A key point is that the respondent needs to sign up in order to be involved in a discussion and they can revoke their membership and delete their personal data at any time.

Using Smap and Upinion together

There are a vast number of ways to collect data from the field, IOT devices, satellites / drones, surveys using enumerators, emailed or IVR surveys direct to respondents, conversational surveys as per Upinion etc.

The core Smap solution is the classic DME approach to conduct a survey with enumerators (we also support emailed surveys and webforms). This is probably the most rigorous and comprehensive way to get a snapshot of a situation at a point in time. However it does have a number of disadvantages. For example preparing the survey and collecting the data can take some time and has a reasonable cost. (Much quicker and cheaper than paper surveys though!). This cost and time tends to mean that you need to ask a lot of questions as you may not be back for a while and you are not sure what you need when the survey is being designed. Of course large numbers of questions can make responding to a survey time consuming for the respondents as well.

In order to get ongoing continuous data after the survey completes, the survey could identify people who have a smartphone and access to a network, and who would be prepared to act as a community contact or sentinel. They can be given a barcode to scan which will sign them up to an Upinion group. From that point on you can conduct conversations with these contacts to get a real time view of their situation and the situation in their community. Based on the information received you may then conducted another more targeted survey of the broader community using Smap.

Smap has relocated to Brisbane

The Smap Consulting headquarters has moved from Canberra to Brisbane. The primary reason is to focus on projects in the South Pacific, Brisbane being a base for a lot of this work. We will continue of course to support all of our customers anywhere in the world and travel will be quicker and easier from here.

The company address is now:

7/47 Herston Road
Kelvin Grove,
Queensland, 4059
Australia

Release 18.12

Updates

Changes to the layout of the user administration page

The new feature to allow a user to belong to more than one organisation has prompted some changes to the user administration page.  The main one is the removal of the button to move a user to a different organisation.   Instead when you open the user for editing then, if you have organisation administration privilege, you will be able to set the users current organisation.

Set the users current organisation

Previously you would select one or more users by clicking on a checkbox next to their name and then click on the “Move to Organisation” or “Delete” button.  A delete button has now also been added next to each user so there is no longer a need to select users prior to taking an action.  The “Add User” button has also been moved above the list of users.

This is work in progress and their will be more changes to the user administration page layout in the next release.

Timezones

The management of timezones for reports and viewing data has been improved.

  1. You can set your timezone in your user profile.  (Available on all pages)
  2. The default timezone is the one your browser is set to however you can change this and it will be saved with your other settings.
  3. Exports to XLS include a new tab that shows the timezone used for all dates and times.
  4. You can set the timezone for a report independently of your personal time zone and this will be saved with the report

Legacy Reports

The old way of creating reports as snapshots of panels on the dashboard has been removed.

Administration Reports

Two reports have been added to the form management page:

  1. Usage.  How many submissions have been submitted per user for each project or form
  2. Form Access.  The users in your organisation have access to a form and why they have access or do not have access. This report considers the projects a user has access to, their security groups and their security roles as well as whether or not the form has been deleted or blocked.

Performance Improvements

As the number of people using Smap increases the size of data tables also increases and it becomes apparent that additional indexes are required.  2 new indexes were added in this release to improve the performance of messaging and submission processing.

Release 18.11

There have been lots of mini updates over the last 4 months but no formal new releases until now. It is November and its time to get back to monthly releases.

New Features

1. A user can belong to more than one organisation. 

Many people find that they are working on multiple projects in different organisations.  Previously you would have needed more than one logon or would have had to keep asking the organisational administrator to move you from project to project.  Now however you can belong to multiple organisations, you can select the organisation you want to work in by clicking on your profile in any page.

All you projects, security groups and role are preserved for all the organisations that you belong to.  Hence you don’t need to set yourself up each time you switch organisations.  The security groups can be different between organisations so you might be an administrator in one and an enumerator in another.

To set this up go to the users admin page, select a user and then click on the organisations that they have access to.  Note you need to have “organisational admin” privilege to do this.

2. Set Max Image Size from Server

You can now specify the maximum image size in the fieldTask options on the user administration page.  This maximum value will be set on the phones when you refresh.

3. Referencing the survey key (HRK) and preloads in the advanced filter

You can now create filters such as: ${_hrk} = ‘abc 001’ and ${_device} = ‘1111111’

Release 18.7

Its been a bit slow over summer.  We are now announcing the july release and yet we are already into August.

Changes include:

  • rank question type
  • randomize appearance on select questions

Rank Question Type

This is just like a select multiple question except instead of choosing one or more options you can order them.

Adding to your survey

Specify in an XLS Form in the same way you specify a select multiple question except that you replace “select_multiple” with “rank”. Then you have a space and the list name that points to all the choices that you want to rank. So:
        type                          name                           label
rank my_list             question_name          Rank this list!

In the online editor select the question type called “rank” and then add the choices.

Using

The rank question works in FieldTask and WebForms.  In both devices you drag the choices into the order that you want.

 

Rank Question in WebForms

Analysis

If you export the data into an XLS report you will see that the ranked questions are shown in a similar way to select multiples with a column per choice.  However the column headings are the question name – the rank.  And the contents are the name of the choice.

Answers to a rank question in an XLS report.

Limitations

Not all exports support rank yet. The support in the API is also limited.  You also can’t currently use it in filters. For example get all data where london was marked as the first choice.

Randomize

If you add an appearance of randomize=true for a select or rank question then in fieldTask and WebForms the order in which the choices are shown will be randomised.  This can be useful to reduce the impact of bias in choice selection on the survey results.   Randomize currently only works on FieldTask and not on Webforms. No it works in webforms too.  Thanks to Alexander Torrado for picking this up.

Bugs Fixed

  • Choices pulled from another form were not showing in webforms.  If you looked up choices in a CSV file then these did show in webforms however if the choices were in another form they did not.

Update 8th August 2018 to add the bugs fixed section and to note that randomize works with webforms but you need to set it as a parameter and not in appearance.

Server Outage

I’d like to apologies for users of the sg server who experienced a 5 hour outage today. This should not have happened and was the result of deficiencies in my IT processes.

The number of surveys submitted to Smap hosted servers has been increasing over the last 3 years and is now averaging nearly 4,500 per day.

On the free sg server it is around 2,000 per day.  However on the 13th and 14th of June we received 11,897 submissions 80% of which included a high resolution image of a cocoa tree. This caused a big drop in the available disk space.

The IT process went:

  1. I received a text message that we were down to 4GB at 5pm UTC.
  2. Immediately ran some clean up scripts that freed up 9GB,
  3. I then added 60GB extra disk to the server.  However this disk is not made available until you reboot the server.
  4. The server was still being heavily utilised so I decided to wait until later in the evening to reboot when, given that most users are in Africa or Asia, the load should have been less.
  5. Then I forgot!

So instead of a 30 second outage we got 5 hours.  I will endeavour to ensure that this does not happen again.

Version 18.04

Improved Performance

Refreshing fieldTask will now be faster.  Note This requires fieldTask 5.33.  The changes are:

  1. If you change an image or other media file that is used in a question or choice then that new media file will be downloaded automatically to the mobile device using push notifications.
  2. A media file, including a CSV file, will now only be downloaded to a phone if it has changed.
  3. Media and CSV files are only downloaded once if they are used by more than one form on the device.  In other words as long as the media file is coming from “Shared Resources” it is also shared between forms on the device.
  4. If the media changes but the form has not changed then the form is not re-downloaded only the changed media files are.

You can expect substantial reductions in download time under the following circumstances:

  • You have a form that uses many images in questions and choices but you are making changes to that form by adding new questions or changing the question text.  Now after each form change only the new form will be downloaded and not all the images.
  • You have a csv file that is constantly being changed.  Now only the changed csv file will be downloaded and not the form and any other csv files or media.

However performance may be slightly worse if you have many forms that use media and only one is updated.  The reason being that during refresh the manifest will now be checked for all the forms that use media to see if the media file has been changed.  Previously the change in a media file would not have been picked up and the form would not have been updated on the device.  This is a quick operation and will only be noticed if the number of forms is quite large.

Other Features

Filtering on table view in dashboard

You can now use “advanced filtering” in these table views.  For example adding a filter such as

${district} = ‘capital’ and ${gender} = ‘female’

The number of records that match the filter as well as the total number of records will be shown just above the table of data.

Issues Fixed

  • Issue 63.  Labels are not shown in PDF exports if the label is for a select question that uses an external CSV file to get its choices
  • Issue 153.  When you download data and upload again in Analysis there is a warning message about “question name missing”
  • Issue. The Legacy XLS export does not allow select multiples to be split into separate columns.
  • Issue.  Replacing a survey that used uncompressed select multiples results in the select multiple questions being converted to compressed but the old data is not converted making it invisible.

Version 18.03

Version 18.03 is here with lots of bug fixes and the following two new features.

Using SVG images to Select Choices

Selecting Choices by clicking on an SVG image in fieldTask

You can use this feature in both webforms and in fieldTask.   The image needs to be a Scaler Vector Graphic (SVG) which is drawn using mathematical functions rather than being composed of pixels like in a photograph.  This means that the image can be enlarged to any size without becoming fuzzy.  To use an SVG image to select choices you need to:

  • Create your SVG file or get an existing one.
  • When you click on a region in your SVG editor it should show the “id” and allow you to edit that.  Set the ids to the name of the choices in your select list.
  • Attach the SVG file as an image on the select question
  • Set the appearance of the select question to “image-map”.

Example

To try this out you can download the sample survey from here.  Also download the sample SVG file (which I sourced from Wikipedia) and load both of them onto your server.

Reports

Yes Finally a new reporting page.  This can be accessed from the Analysis module by clicking on the menu item “reports”.  On this new reports page you will find a link to the old page (in case you are still using that).

How to use

Add reports by clicking the “Add Report” menu item.  You can specify a name for the report as well as parameters include filters, date range, split select multiples and languages.  This will create an entry for the report in the table containing:

  • The name of the survey
  • A name that you have given the report
  • An anonymous link.  You can send this link to anyone so that they can access the report.  No logon or account or password would be required.   This link does allow the recipient to change any aspect of the report.  Hence if you had set a data filter or a specific date range that cannot be changed.
  • An button that has available 3 actions:
    • Generate.  Runs the report to download the data.
    • Edit.  Allows you to change any aspect of the report.  Someone who has received an anonymous link for the report would see these changes the next time they accessed that link.
    • Delete. Will delete the report.

Limitations

  1. Only the new XLSX format report is supported.  Other types such as images, shape files will be added progressively and as requested.
  2. Support for security roles in the anonymous link have not been added yet

Contact me if you need any of this missing functionality urgently.