Version 19.05

The big change in this release is the addition of GPS coordinates to named locations.

Named Locations

When you add or edit a task you can set the GPS coordinates, an NFC identifier or both to specify where the task should take place. This is what we call an anonymous location since you are not using a “Named Location”.

However you can now also enter the name of a “location” that has the NFC UID and/or the GPS coordinates pre-defined. To do this Select the group name first and then the location name. (This change addresses issue #468).

Selecting a named location with group “STICKER” and location name “E1”

Named locations are shown in the “Shared Resources” page. Select the Admin module then “Shared Resources” , then select the “Locations” tab to view them. Click on the “Download” button to get the current locations in an excel file, or an empty template if you have not specified any locations.

Named locations on the shared resources page

Loading Named Locations from an Excel file

Locations are organised into groups and each group is shown in a separate tab of the excel file. Create new tabs giving each one a unique group name. Then for each group create locations. You can specify:

  • UID: The identifier of an NFC chip at the location
  • name: The name of the location
  • lat:  The latitude in GPS coordinates
  • lon: The longitude

The location name must be unique within the group. When the file is ready select “Add Files” and then “Upload”.

Updating a location when editing a task

After selecting a location you may adjust the GPS coordinates or change the NFC UID. You then have the option of:

  • Continuing with an anonymous location
  • Updating the location with the new coordinates / NFC UIC
  • Saving a new named location

Other changes

  • Added “auto” as a parameter option in the online editor to automatically start camera, barcodes and NFC reader when the user swipes to that question. The auto start is only applied when swiping forwards, there is not already a value recorded for the question and the question is the first one on the page. (This update implements issue #475)
  • Drastically reduced the memory required when importing data from an excel spreadsheet. This was tested importing 80,000 rows each with 500 columns without any problem. This new version is also much faster than the previous import.
  • API changes
  • In PDFs allow the colour of a marker showing position to be configured using an appearance setting for the geopoint question
  • Fixed issue #433. The advanced filter was not working correctly with wildcards.

Version 19.04

New Features

Its all about Tasks in this release.  Lots of great new features.

  • Auto start of external applications in fieldTask. (requires fieldTask 6.10). An external application will be automatically launched if you set the parameter “auto=yes”.
  • Geofencing for tasks. A “Show Distance” can be set on a task so that it will only be shown to the user if they are within that distance of the task location. In addition a “download distance” can be set on a task group. Tasks in that group will only be downloaded when the user is closer that that distance from the average center of all the tasks in the task group.
  • Set initial data for a task.
  • Tasks API.

Geofences

Show Distance

This is for tasks that have been downloaded to fieldTask. The app will show and hide them as the user moves in and out of the geofence.

You tube video showing creation and use of a geofenced task

There are two geofence capabilities added in this release.

  1. “Show Distance”. The tasks are downloaded to the phone but only shown to the user when they are close enough to the task location. This distance is set per task.
  2. “Download Distance”. Tasks are only downloaded to the device when the user is close enough. This distance is set on a task group.

Set Initial Data for a Task

Amazingly this was not previously possible.  You could set a task to update an existing record however now you can also create a brand new submission but initialise it with some arbitrary data.  When creating tasks from existing data you can select one of the following 3 options:

No initial data means that your form will open empty. “Pre-populate form with existing data” is the new option, the data will be taken from the existing record and mapped to compatible questions in the new form that you want completed and the original data will not be updated. The third choice “Update Existing Results” continues to allow you to update an existing record. The data for that record will be shown in the form that you are going to complete.

You can’t yet add initial data when you add an ad-hoc task via the user interface. However you can do this via the new tasks API (see below).

Tasks API

A tasks API is slowly emerging. This is documented on the Smap website at http://www.smap.com.au/apiTasks.shtml.

  • Get task groups for a project
  • Get tasks
  • Get task details
  • Get initial data for a task
  • Create a task

Version 19.03

New Features

  1. Including data from linked surveys into a PDF
  2. New XLSX report on the task management page that will list all tasks in all task groups. Exports to XLSX for tasks in the current task groups are still available.
  3. Manage a password policy for fieldTask.
  4. Add support for the ODK setting “guidance_hint” when loading xlsForms
  5. Add support for the ODK setting “body::intent” when loading xlsForms
  6. Add a tasks API to return tasks in geojson format
  7. Add an audit API to return geoJson data on location and duration of each question asked in the survey
  8. Add the user identifier as a valid value for use in an advanced filter
  9. Record the PDF template for a survey in the surveys change log so that previous versions can be referenced
  10. Add ability to delete meta questions in the online editor as well as add the pdf_no appearance to them so that they are not included in PDF exports

Including data from linked surveys in PDFs

Smap allows you to combine Surveys to make “Systems“, where data in one survey can be linked to data in another. For example a survey recording the delivery of aid may include the identifier of the beneficiary. You can analyse this data taking account of the links between them by extracting the data into external tools like Power BI. However inside Smap itself we haven’t provided any capabilities to join data from different surveys during analysis. This new feature is the first step on that path.

When you create a PDF containing data from a single survey submission you can optionally specify:

  • Include reference data
  • Only include data from surveys launched from the current survey

The first option requires you to have specified a child subform as being launched from the survey you are exporting. You also need to specify a key question in the child survey.

If you include the second option you don’t need to specify a key question but only data collected by launching the child survey on the phone will be included.

Password Policy settings for FieldTask

FieldTask Version 6.02 has a logon page.  By default this is shown when the app is first run so the user can set the server, username and password without having to go to the settings menu.   After successful logon it is not shown again.  This corresponds to the default password policy of “Never show logon page”.  In the tab for fieldTask options on the usermanagement page you can also specify a policy of “Always show logon page” which will require the user to enter their password every time they start the app;  and “Periodically show logon page” which will require the user to enter their password again a specified number of days after their last successful logon.

Bugs Fixed

  1. Large API exports were causing out of memory errors. These data exports using /api/v1/data are now streamed so there should not be any size limits.

Monitoring and making an immediate impact

Lets face it, collecting data can get a bit dull and there is often not an obvious connection between the data collected and any subsequent actions. However it is possible to take a more active approach to monitoring, to move beyond mobile data collection as a passive tool and start using it to have an immediate impact while still getting the data you need to assess performance.

Monitoring is an excellent application for mobile data collection tools which make it more timely, affordable and effective.   One way of collecting monitoring data is to embed the data collection into a delivery process so that when you do something, you also document what you have done.  In other words monitoring your own actions. This can be a useful alternative to more conventional longitudinal surveys.

You can put the emphasis on data collection but depending on the answers you are getting the tool can also prompt you to provide immediate advice to the interviewee or to deliver a service and then record that service delivery.  This approach can be called “Push Surveys” as you are doing a survey but are happy to change the state of what you are surveying.

Alternatively you may be focussed mainly on service delivery but incidentally recording data during that process.   For this you need a “Management System” such as Asset Management, Accountability or a Patient Management System. 

Smap has the following capabilities to support both “Push Survey” and “Management System” approaches to monitoring.

Push survey Capabilities

  1. Expert Systems.  You can use “relevance” in forms to create expert rules that can identify advice or further action to take.
  2. Artificial Intelligence.  From fieldTask you can call AI services in the cloud to answer more complex questions that might also involve the analysis of images, video or audio in order to identify required actions.  (Not available offline)
  3. Graphs.  FieldTask can present collected information in graphs to help in providing immediate feedback to an interviewee or community. (Works offline)
  4. Reference data.  Lookup data previously collected in the same survey or other surveys and use that in identifying the action or feedback to provide. (Works offline)

Management Systems

  1. Keys.  A unique “key” can be associated with each survey.  This key could be a beneficiary id or a part number for a piece of equipment.  Along with the key you can specify a “key policy” such as “merge”.    If you have a key policy of “merge” and you submit a survey with the same key as an existing record then the new data will be merged with the existing record.
  2. Reference Data.  These enable an identifier from another survey such as a beneficiary id to be added when completing a different survey, perhaps one that records delivery of aid.
  3. Managed Forms.  Allow you to review and modify data collected from the field.  One use of this is to transcribe audio recordings into text.
  4. Grouped Forms.  By grouping forms which share the same data you can present multiple views of a single data set.
  5. Launch a form from within a survey.  When completing a survey in the field it may be necessary to launch another Form and link the data between these forms.
  6. Tasks.  Assign a task to a user.
  7. Notifications.   Automatically email PDFs and tasks to people.

Version 19.02

Start a survey from within a survey

While completing a survey on a phone or tablet with fieldTask the user can start another survey / form and then when they have finished that they are returned to their original survey. The data collected by these two surveys are linked for analysis.

Two new question types have been added; “parent_form”, “child_form”. The difference between these is in how the collected data is linked.

Launching a Parent Form

There might be many records in a form that point to a parent form.  For example you may be reporting problems on pieces of equipment.  So you open the problem report form and select the equipment identifier, then record the problems for that one piece of equipment. However if equipment has not already been added to the system then you can launch a parent form, from within your problem form, to create an entry for that equipment. You can then continue filling in the problem report knowing that it will be linked to the new piece of equipment for analysis. All of this works offline.

In the above example the key Question would hold the identifier for the piece of equipment.

Launching a child form

There might be many records in the child form that point to the first form . For example you may be registering students for school attendance and you encounter a student for which there is no record.  You can then launch a child form to create an entry for that student who will be linked to the school.

In the above example the key question would hold the identifier for the school.

Other Changes

  • The online editor now greatly simplifies the setting of parameters and appearance. You can select these from options presented on the screen rather than having to type them in. This is particularly useful for cases like the search appearance where you can now select from a list of csv files or forms to reference and then select the columns from those that you want to include. No more awkward looking up of form identifiers! This is also useful when adding a new parent_form or child_form question type as you can simply select the form you want to launch using the parameters dialog.
  • Added support for range widgets in webforms
  • Allow backward navigation in a form to be prevented by using a setting on the server.

Version 19.01

New Features

User activity views have been added to the dashboard. If you open a table view or a map view then you can specify the subject as “User Activity”. Then select a user to show their activity.

The source of data, such as the form name that was used to collect it, or the file name that was imported is now shown in the table view on the dashboard.

Added display_name to choices. Display names have been available for questions for a while. These allow you to specify a name to replace the question name in reports but one that is not constrained by the limitations of the odk question name. For example you can include spaces. These are now available for choices. This is particularly useful for select_multiple questions as now you can replace default column headers such as “Question Name – choice_name” with something more meaningful such as “The selected choice 10 !!!!”.

Add the ability to restrict activity within an organisation. For example you can prevent the submitting of any new survey results, access to the data api or sending of notifications. If any change is made to these settings the organisation owner will be notified by email.

Added some initial customisation of the webform page appearance. For example you can now set background colours for some of the components.

Add a new data processing option to reports. Transformation of data from Long to Wide. hence if you have done a longitudinal survey you can get a report on things within that survey with one row per thing and all the time series data converted to additional columns.

Added some settings data into a settings workbook when creating an excel report. This now shows the timezone used in the report as well as the start and end time of any date filtering applied.

Bug Fixes

  • Replacing a select_one question with a select_multiple question after data collection had started caused further submissions to the form to fail. Now fixed.
  • Multiple bug fixes in webforms. Images in choices now work again as does the selection of drop down select questions.

Conversational Surveys with Upinion

I have been looking at Upinion which seems to be a very nice tool for “Conversational Surveys”. Upinion define these as ongoing surveys conducted in a chat messaging tool such as facebook messenger or their own app. As a company Upinion are focussed on international development with the aim to “Amplify people’s voices in a crisis situation” and could be an excellent complementary information channel to use with Smap for getting a more complete view on a situation.

Upinion

To specify the questions you are going to ask you create “conversations” using a GUI drag and drop editor. The screenshot below is from a test conversation I created. A graph of responses is shown automatically beneath each question as they are received.

Secondly you can attach a group to the conversation. This group will contain the respondents to your survey.

Finally you need to invite respondents to join a group. They might do this by scanning a barcode or clicking on a link. A key point is that the respondent needs to sign up in order to be involved in a discussion and they can revoke their membership and delete their personal data at any time.

Using Smap and Upinion together

There are a vast number of ways to collect data from the field, IOT devices, satellites / drones, surveys using enumerators, emailed or IVR surveys direct to respondents, conversational surveys as per Upinion etc.

The core Smap solution is the classic DME approach to conduct a survey with enumerators (we also support emailed surveys and webforms). This is probably the most rigorous and comprehensive way to get a snapshot of a situation at a point in time. However it does have a number of disadvantages. For example preparing the survey and collecting the data can take some time and has a reasonable cost. (Much quicker and cheaper than paper surveys though!). This cost and time tends to mean that you need to ask a lot of questions as you may not be back for a while and you are not sure what you need when the survey is being designed. Of course large numbers of questions can make responding to a survey time consuming for the respondents as well.

In order to get ongoing continuous data after the survey completes, the survey could identify people who have a smartphone and access to a network, and who would be prepared to act as a community contact or sentinel. They can be given a barcode to scan which will sign them up to an Upinion group. From that point on you can conduct conversations with these contacts to get a real time view of their situation and the situation in their community. Based on the information received you may then conducted another more targeted survey of the broader community using Smap.

Smap has relocated to Brisbane

The Smap Consulting headquarters has moved from Canberra to Brisbane. The primary reason is to focus on projects in the South Pacific, Brisbane being a base for a lot of this work. We will continue of course to support all of our customers anywhere in the world and travel will be quicker and easier from here.

The company address is now:

7/47 Herston Road
Kelvin Grove,
Queensland, 4059
Australia

Information Management

Smap’s advanced capabilities in; linking data between forms, task management, unique key management, case management and notifications allows you to connect forms in order to create an information management system.   That is a set of collaborating forms that address one or more business processes in your organisation.

To illustrate this capability I have created a demonstration of a system that supports the registration and enrollment of students in education classes.   This demo is available on the Smap https://sg.smap.com.au server for you to try out.

The attached pdf provides detailed instructions on how to try the demo.  Hopefully this example will inspire you to create your own information management system using Smap.

https://s3.amazonaws.com/smap-documents/studentEnrollment+minimal.pdf

Release 18.12

Updates

Changes to the layout of the user administration page

The new feature to allow a user to belong to more than one organisation has prompted some changes to the user administration page.  The main one is the removal of the button to move a user to a different organisation.   Instead when you open the user for editing then, if you have organisation administration privilege, you will be able to set the users current organisation.

Set the users current organisation

Previously you would select one or more users by clicking on a checkbox next to their name and then click on the “Move to Organisation” or “Delete” button.  A delete button has now also been added next to each user so there is no longer a need to select users prior to taking an action.  The “Add User” button has also been moved above the list of users.

This is work in progress and their will be more changes to the user administration page layout in the next release.

Timezones

The management of timezones for reports and viewing data has been improved.

  1. You can set your timezone in your user profile.  (Available on all pages)
  2. The default timezone is the one your browser is set to however you can change this and it will be saved with your other settings.
  3. Exports to XLS include a new tab that shows the timezone used for all dates and times.
  4. You can set the timezone for a report independently of your personal time zone and this will be saved with the report

Legacy Reports

The old way of creating reports as snapshots of panels on the dashboard has been removed.

Administration Reports

Two reports have been added to the form management page:

  1. Usage.  How many submissions have been submitted per user for each project or form
  2. Form Access.  The users in your organisation have access to a form and why they have access or do not have access. This report considers the projects a user has access to, their security groups and their security roles as well as whether or not the form has been deleted or blocked.

Performance Improvements

As the number of people using Smap increases the size of data tables also increases and it becomes apparent that additional indexes are required.  2 new indexes were added in this release to improve the performance of messaging and submission processing.