Version 20.04

New Features

Artificial Intelligence Services

We are continuing to work on adding Artificial Intelligence to the Smap Platform and the following features are a step towards that goal. They add calls to generic AI services provided by AWS for translation, transcription and image recognition.

There is a cost for using these AWS services so by default monthly free limits have been added per organisation.

  • Image Labelling: 100 Images
  • Translation: 5,000 text characters
  • 250 seconds of Audio transcription

Note: These Quantities have been updated as the original post contained the free limits for an entire server. So the new quantity is about 10% of the previous value.

So feel free to use these services and you will get an error message when your monthly limit has been exceeded. If desired, you can then, contact Smap to extend the limit which will incur a monthly fee.

Automatic Translation of Surveys to other languages

The list of supported languages is available in the documentation: http://www.smap.com.au/docs/xls-languages.html

Languages can be translated when the survey is loaded from an XLS template. In this case only blank untranslated text will be modified. You can also initiate auto translation from the translate page of the online editor. In this second case you can specify that the existing language translations be overwritten.

It is important to note that there will be errors in the translations and the performance of AWS translate may vary with language. However this is normal in survey development even with manual translations. Ultimately, during data collector training, problems with the translation get resolved. The online editor translate page is a convenient place to make these last minute adjustments.

The following video shows how to automatically translate a survey in an spreadsheet template.

Automatic Translation of Text Data

Just as you can translate a survey into multiple languages automatically you can now translate text responses in submitted data. This is done by specifying the auto=yes parameter to a question in an oversight form.

The following describes the manual way to translate question responses which can be automated using the “auto” parameter. http://www.smap.com.au/docs/console.html#example-1-translate-a-text-answer

Automatic Conversion of Audio into Text

If you collect audio as part of your survey then if the audio is in one of the supported languages you can automatically convert it into text. This then allows you to search through your data for key words that were spoken in audio files. You can add automatic translation of the transcribed text by adding an additional translate question to the oversight form. Details: http://www.smap.com.au/docs/console.html#automatic-audio-transcription

Automatic Labelling of Images

This has been available for some time however it has been updated to work like the other services in the background without affecting the speed with which new submissions are processed.

Add “append” for repeat records when updating existing results

If you use a task to update an existing record then any repeating groups in that record will be replaced. The scenario might be that you are doing a follow up interview at a household where there were originally 3 household members. While completing the survey the enumerator might remove one household member, modifying a second and add 2 more. After submission there will be 4 household members with the updated data.

However sometimes you want to append new repeating groups. In particular if you set the parameter “instance_count” to zero then none of the existing repeats would have been downloaded with the task. The enumerator can then only add new repeating records and you would want these to be appended to the existing records.

This can now be done using the parameter: key_policy=append

The hospital example implementation has been updated to include this.

Other Changes

  • Add support for the “required” message in multiple languages
  • Bulk uploading of users, projects and roles from a spreadsheet

Bug Fixes

  • Issue 434. Support the use of apostrophes in text when filtering data on the server.
  • Issue 507. Setting the timezone on tasks exported to a spreadsheet

Version 20.03

This release adds greatly improved support for conducting email surveys. Previously you could conduct mailouts of surveys using the task interface. Now there is a dedicated “Campaign” module. It can be accessed by selecting “Modules” then “Campaign”. Documentation.

Creating a Campaign

A campaign is associated with a survey. You can specify

  • A name for the campaign
  • The subject to be used in sent emails
  • The content of the email
  • A list of email addresses and names

Pre-populate Survey Forms with Data customised for each Recipient

You can include customised data specific to each recipient for any question in the survey.

Manage Contacts

Contacts can be added and edited on a new contacts page. However most commonly a contact will be added indirectly when they are included in a mailout. The contacts page is useful for changing the name of the a contact and also seeing quickly if an email address has unsubscribed.

Campaign API

The API allows you to:

  • Get a list of campaigns
  • Create campaigns
  • Add an email to a campaign

When adding an email you can specify that the URL with the survey should not be sent to the recipient via Smap. The API call will then return the URL to the calling program. This allows you to manage the distribution of survey links in your own application.

Version 20.02

New Features

Opt in for emails

There was existing functionality to include an unsubscribe link with every email sent. If the recipient selects that link and then unsubscribes they will no longer receive any emails. However previously, if they just ignored the email then they would keep getting more.

The opt in applies to emails sent by a notification, a task reminder or an email task. If it is the first time an email has been sent to that email address then an opt in email message asking if the person wants to opt in is sent instead and the email that triggered this is placed in a pending bucket. If the user opts in then the pending email and future emails are sent to them. Otherwise they will not receive any more emails from the above sources. They will still receive emails that allow them to set their password unless they unsubscribe.

You can review pending emails from the “Monitor” page. You can also resend an opt in message from their if the user has neither unsubscribed nor opted in.

Other Features

  • Changed the label for the button that creates an importable version of the data from “export” to “backup”. This differentiates it from the export menu which exports data for analysis.
  • Added support for the lookup_choices() survey functions in the online editor. This function is a replacement for search() that does an online search to get choices instead of downloading them in a CSV file.
  • Added support for default values in a survey that are calculated when the survey is opened. http://www.smap.com.au/docs/xls-default-values.html
  • Added a logs api. http://www.smap.com.au/docs/log-api.html

Bug Fixes

  • Issue where fieldTask used by recent users was not automatically synchronised with the server
  • Exports of CSV files using the api (/api/v1/data.csv) were not in unicode
  • Fix an error when changing a question from text type to note type in the online editor
  • Validate that names do not clash between surveys when surveys are uploaded into a group.

Version 20.01

New Features

The previous release was back in August 2019. (Version 19.08) so the new features listed below have been added since then.

  • Selection of sub forms (begin repeat) in the console
  • Drill down from parent form to child form in the console. For example If you have selected a parent record such as household #10, you can then select drill down to see the plots of land associated with that household.
  • Drill down from a form to the data that was collected in a form launched from the current form.
  • Online documentation. https://www.smap.com.au/docs. This has mostly replaced all the content in the old powerpoint files and added a lot more besides.
  • Ability for users to self assign tasks. Once a user has assigned themselves other users will be blocked from the task.
  • Late tasks are now shown in red on the console and in fieldTask. Cancelled tasks which were shown in red are now shown in brown.
  • Server calculations. These can be used to generate data on the server that is shown in the console.
  • Added a start-geopoint preload. This will automatically record a location for the survey when the enumerator answers their first question. A message will be shown that the location is being tracked but the enumerator will not need to do anything to record this location.
  • Simplified the XLSForm survey template
  • The projects, roles and organisation sections of the dialog that edits user settings have been made expandable. This caters for situations where there are very large numbers of projects that you are not interested in looking at.
  • Allow a users to be assigned to a project from the project editing dialog
  • Added support for Ubuntu 1804 in downloadable versions of the server
  • Show a warning to a fieldTask user if the time on their phone is more than 60 seconds different than the time on the server
  • Provide an API call to retrieve the times that fieldTask users have refreshed the forms on the their phones

The No-Code Digital Transformation

No-Code and Citizen Developers

The concept of no-code systems is that non-developers, sometimes referred to as Citizen Developers can create sophisticated IT systems without having to write any application code. This can result in IT systems being deployed faster at much, much lower cost.

This means that no-code has the potential to be hugely transformational to the IT industry. However according to this article in Forbes the no-code movement is being held up because it is “too disruptive”. The article goes on to say that “A traditional enterprise application that might require $2 million dollars to develop and deploy could be reduced to a cost of $50 thousand”, and that this impacts on traditional software development organisations and IT departments who are pushing back against the technology, limiting its adoption in established enterprises.

The opportunity in International Development

And herein is the opportunity for the use of no-code in international development where cost is critical and there is a dearth of existing IT systems. Also in international development there is already widespread usage of Mobile Data Collection (MDC) technology for M&E. This technology can be readily repurposed as a no-code implementation tool.

An Example – Developing an Approval System

The following video demonstrates building an approval system in a few minutes using Smap which is an MDC tool that also has strong capabilities in delivering Mobile Workforce Assistants and Case Management applications.

Implementing an approval system in a few minutes using MDC technology

Summary

We have the opportunity to change the way we use IT in development. It no longer needs to be seen as a major cost item developed by large teams working with head office. IT systems can be developed by citizen developers quickly and at low cost to make an immediate impact.

Of course this does not remove the need for traditional IT system development and of course IT governance. It would be a shame for example to create dozens of totally different approval systems. Also some core IT systems are likely to be more efficient if developed the traditional way however even in these cases a “no-code” pilot should be considered.

Upgrading from Oversight Forms to the Console

The new console release is not backwardly compatible with Oversight forms so once your server is upgraded you will need to start using the new approach. Here are some tips:

Using the console

Oversight forms used to be attached to a standard form. Each form could have only one oversight form. Hence when you went to the managed forms page and selected your form all the oversight form’s data was also immediately visible,

The console is more flexible, more options for the user and hence a little bit more work for them to set up what they want. When you go to the console the first time only the data for you current form will be shown. However you will see a drop down that allows you to select your “Oversight Survey”. Change this from “none” to the form that you want to use to edit your data. Once you have done that you will see the oversight questions as before. The selection you made should be remembered.

Selecting an oversight form / survey

Editing data

In the previous approach to oversight forms you could select a row and edit it. Now if you want to change a record of data you need to first lock it. So select the row then select the “Lock for Editing” button. This will assign you to the record so that you can make changes.

You can see who is assigned to a record in the _assigned column. This column is color coded:

  • Yellow if the record is not assigned
  • Blue if it is assigned to the current user
  • No color if it is assigned to someone else

Records assigned to someone else can only be viewed they cannot be changed. An administrator can reassign records to a different person. You can filter out records assigned to other people using the switches at the top right of the screen.

A view of the console showing the lock for editing button and the switches to filter by locked and unlocked records

Modifying the Oversight Form

Oversight forms are no longer anything special. They are just any old form that you want to use to edit data in the main form. To do this the questions in the each form would need to have the same name. The forms also need to be grouped.

Because oversight forms are like any other form you manage them from the form management page. When you upload the oversight form select the form that it will modify using the “Group” drop down.

A convenient way to create an oversight form might be to take a copy of the form you are going to manage and then add / delete questions as required.

Version 19.08

Well we Just got the August release out on the last day of the month, that was lucky. Mostly this release is about the console and has already been covered in the post https://blog.smap.com.au/a-console-for-case-management-release-1908/. However there is some additional functionality that I should describe:

Setting a Theme for Your Organisation

If you are an “Administrator” when you to to the user management page you will see a tab called “Appearance”. If you select that you can set the color of the menu bars and the logo that you see on the home page. In order to enable your new theme you will need to also check the checkbox labelled: “Set as Theme”

Oversight Forms have been Retired

Instead of having to create an oversight form using special syntax you can now group any other standard form with your primary form. There are some features of oversight forms that were not in the normal XLSForms that you use for data collection. Including Server Side Calculations and setting the background color of answers to questions. These have now been added to XLSForms (See Below)

Server Side Calculations

You can use a new question type “server_calculate”. Put the calculation in the “server_calculation” column.

Expressions are the same as for the advanced filter. https://sg.smap.com.au/sqlFilterHelp.html However as here we are creating new value and not filtering there is no equals sign. For example the following will show a date 7 days on from the time editing started on the survey:

  • ${_start} + {7_days}

These server calculations are evaluated at run time so if you change your calculation and replace the form then the server calculation will be updated for all the data you have already collected.

A new capability not available in the advanced filter is the if() function. Here your server_calculation will simply be:

  • if()

Then you specify the conditions in a new “conditions” work sheet. In the example below the value of the calcy question will be set to ‘Not neil again!’ if the value in the name question is ‘neil’. Note that there is a set of conditions for each server calculate question that has its calculation set to if(). Hence you put the question name in the first column.

There is only limited support in the online editor for server calculations. Specifically conditions are not supported.

Color Highlighting in the Console

You can use a new “styles” worksheet in the XLSForm editor to specify colors for cells in the console table that are dependent on the value of the data. The following example shows an example of a style list.

Note for styles we do use the list_name approach. Hence you can have a single set of styles that color multiple questions. Specify the name of the style list name in a new column “style list” in the survey work sheet.

Version 6.32 of FieldTask now in Play Store

This release includes a number of significant new features

History

If you select menu then history you can see a list of all survey instances that have been submitted to the server or deleted.

History view

The view is updated after you perform a refresh to synchronise with the server. In addition you can also delete an instance from the main screen by doing a long press on it. You will then select “reject” and will have to provide a reason.

Mapbox Tiles for use as offline maps

Both vector and raster tiles are supported.   A while ago offline map tiles were supported.  However this was when the maps used the google maps API.   ODKCollect have been progressively migrating to the MapBox API which supports tiles in a different format.  However now the feature is back.  Sample tiles are available here: https://github.com/klokantech/vector-tiles-sample just download them and copy the mbtiles file onto the phone’s sdcard under /sdcard/fieldTask/layers.  

Persian Calendar

If you have a date question and want to use the Persian calendar specify an appearance of persian.

A Console for Case Management – Release 1908

Organisations using Smap are increasingly managing work using data records that correspond to someone or something in the real world. This might be the maintenance of equipment, or the delivery of services such as education to a student, training to a teacher, health services to a patient or climate change resilience improvements to a community.

These applications represent an active form of Case Management where information is recorded against a case over time and used to guide the delivery of services. In the August release we have added a number of features to support this, including:

  • Find and view the record of data that represents a case
  • Update that data in a controlled way with logs and security
  • Request a task be sent via email, or to a mobile phone app, requesting an action be taken and data be collected
  • View all changes made to a case
  • View the status of all tasks
  • Send reminders if a task has not been completed via email or SMS or to fieldTask users

These activities can now be managed using a “Console” which allows a user to manage cases using the same forms that are used by the field workforce.

The role of the console in case management

What is a Record?

In the Smap universe every time you submit results from a survey you create or update a record.   You can group surveys so that they all share the same records.  This allows you to, for example, create a record with survey A and then update it with survey B.   Each case can be recorded in one or more of these records and so this feature is what provides a lot of the power of Smap as a case management tool.

Surveys in a group managing a record

When you select surveys on the records page, if they are in the same group then you will see the same records for each of them although the columns that you see in the table will be determined by the columns in the survey.

Actually this is a slight simplification in that you can also use security roles to restrict the records shown on a per form per user basis so not all forms in a group need show the same records to the same users.

New Features

Replacement of WebForms with a Console module

Here is some history to explain where the console came from. In the old Web Forms module, as well as having a list of forms you could launch in you browser, there were menu options for Browse Results and Managed Forms. Browse results was going to be the new analysis page but we never managed to make it better than the old Analysis page so it just lurked in Web Forms not being particularly useful. Managed Forms was similar to Browse results but it allowed you to attach another form and edit the submitted data using that form. This has been used in applications such as Feedback and Accountability.

Now Browse Results and Managed Forms have been merged and renamed to “Records”. The WebForms module itself has been renamed to “Console”. It still includes the page to launch forms and respond to tasks using your browser but it now has a focus on individual data records rather than analysis of all the collected data.

New Console Module Home Page

A scalable records page

By scalable we mean that the console can handle hundreds of thousands of records. Previously the browse results page had problems when customers had this many records as it tried to load all the data into the browser first and then filter.

Records page

If you look at the records page you will see that it is in two parts. The top half consists of filters that are applied on the database before data is retrieved. Underneath are the table view, map view and calendar view. For these the filters are applied in the browser. Hence you can select the survey, the from date, to date, an advanced filter and max records and a new query will be made on the database to get that data. The first time you open a survey the limit will be set to 1,000 so that you don’t have to wait to get the data. You can then change the limit of you need to see more, or less, records.

Modifying a Case

When managing a case you will need to be able to make changes to the data directly from the console. The first step is to lock a case for editing. Once you have done that it is assigned to you and other people cannot edit the same record, although an administrator could re-assign the case to someone else. In the diagram below the “Unassigned Records” switch is selected so that cases not assigned to anyone can be selected. An unassigned record has then been selected by clicking on it. The button “Lock for Editing” can then be clicked.

Locking a record for editing

Once you have locked a record the edit button will be shown. The questions you can change are determined by the Oversight survey that you have selected as well as any security roles that have been applied. These questions are shown on the right hand side of the page and are labelled as “Management Data”. Any remaining questions in the main survey that are not in the oversight survey are shown as read only on the left hand side for your reference.

Record opened for editing

In the above image there are 2 questions in the management form, a geopoint question and a text question labelled as “comment”. You can select a new location, change the comment and then click “Save”. The existing data for the record will be updated.

Reviewing changes to a Case

If you click on the events tab you can view all changes made as well as see all task and reminders that have been created for the record. You can also create new tasks that can be assigned to your field work force to update the case or send a reminder.

Events on a task

In the above diagram changes have been made to the record using the survey called “Map”. The location question and questions “q2” and “hi” were changed. For location questions; points, lines, areas the old location is shown in blue and the new location in red.

A task has also been created on this record but it has not yet been assigned to anyone.

Conclusion

There is a lot more functionality in this upcoming release that I will talkcover once the software is deployed to the servers. Hopefully this post will give you some ideas for the types of case management applications you will be able to create from about the middle of August.