Author Archives: Neil Penman

Version 21.11

Here is the November release of Smap. We did not release an update in October, although you may have noticed that the version number of some server installations said 21.10, but if you clicked on the link for changes it just took you back to the September changes. Unfortunately I never got around to creating the release notes so officially October never existed.

New Features

Specialised PDF components

There is a new question type “pdf_field” so you can specify specialised components to be included in a PDF report. Documentation is here. Currently two new components are available and they are intended for use in reporting faults in cables or pipelines. However now that the framework has been created, we can add additional components. Let the Smap team know if there is anything you would like.

Component 1 – Fault Image

This component is given the names of the questions that record, start of cable segment, end of cable segment and fault position. It then shows these in an image that includes the distance from the cable segment start to the first fault and the distance from the end of the cable segment to the second fault.

Component 2 – Fault Map

The input to this component is the same as the input to the image component but this time the information is shown on a map.

Background Report Generation

Just as the new PDF components are specialised, this new background report generation may not be useful for many organisations. However it has been added to help large organisations.

The first report that has been converted to background generation is the user usage report on the form management page, if your organisation has thousands of users and millions of submissions then this can take a long time and report generations for these large organisations have been failing.

Selecting a usage report on the form management page

Now once the report has been requested a message will be displayed saying that the report generation is in progress and the completed report can be found on the reports page.

To access reports select the menu “modules” and then “reports”. The reports generated for the last two weeks are shown here and can be downloaded by clicking on the report’s blue button. If the report is taking a while to complete you may see its status as “Pending” rather than “Complete”.

The previous functionality available on the reports page can be found under the tab “Public Reports”. This still allows you to define a report and then get the URL to that live data report that you can share with other people.

Allow webform links to tasks to be completed multiple times

Previously if you created a task as part of an email campaign then that task could only be completed once. However there are applications where you want a task, preloaded with some specific data, to be completed multiple times. For example if you have a barcode that links to a task and you want that task for that url to be preloaded with a venue location say. Documentation.

Other

  • The version of fieldTask used to synchronise with the server is now included in the user audit log. This also requires fieldTask version 6.515+. Documentation.

Bug Fixes

  • Survey forms that do not have any questions no longer cause an obscure error to be displayed when they are downloaded on to fieldTask. Instead when the user opens the survey on fieldTask they will see a helpful message stating that the form has no questions.
  • When getting an email address from survey data in order to send a notification, previously the name of the question was used in the data query instead of the column name of the question. If the question name contained characters that were not valid as table column names then this would not have worked.

Version 21.09

New Features

Lookup reference data using an expression

Using the pulldata() or lookup() function you can now specify an expression. For example the following pulldata calculation will get the average age of people older than the answer set in the “age” question. The reference data is taken from a csv file called “people”. More information can be found in the documentation.

if(${age} >= 0, pulldata('people', 'age', "cast(#{age} as integer) > ${age}", 'mean', 'eval'),'')

Consistent reference data support on FieldTask and WebForms

Well more consistent anyway, there are still a few idiosyncrasies as you move between fieldTask and webforms. So for example the new feature of using an expression to lookup reference data works in both fieldTask and WebForms.

Other

  • Links created for mailouts can now accept multiple submissions. To allow this select the checkbox “multiple submissions” when specifying your campaign.
  • User locations page
  • Template upload checks for missing end repeats

Version 21.08

New Features

Digest Function in Webforms

The digest function now works in both FieldTask and Webforms. One application of digest is to obfuscate sensitive data while still retaining a unique identifier for that data. For example instead of recording a person’s name you can record the digest. If you then complete another survey and the name of this new person is the same then the digests will also be the same. So you will know that the people in these two surveys had the same name you just won’t know what that name is.

Example Digest Application

In this example we are going to hide the persons name and use the digest of the name as the key for the record so we can update the data on that person.

1. Specify the survey.

typenamelabelappearancecalculation
textnameNamephoneonly
calculatedigest_namedigest(${name}, ‘SHA-1’)
intageAge
The survey

Note the appearance of “phoneonly” on the name will prevent the name from being stored in the server database. It will be replaced with “xxxx”.

2. Set the survey key to be ${digest_name}. This means that when we submit another record with the same digest as an existing record then the existing record will be updated.

3. Results as shown in the console

Submitted data

Note I have clicked the option to “show deleted data”, So you can see that record 1, which had an age of “23” was merged with record 3 that updated that age to “57”. Both of those records had the same digest_name. The actual name however is not recorded and only “xxxx” is stored in the database.

4. Final thoughts

Using the digest of a name as a key is probably not a good idea in the real world. This is particularly true where the recorded name is a transliteration from another language. Minor spelling mistake would result in a completely different digest so the value of the digest is probably low. However you could absolutely use the above approach with passport numbers or identity card numbers.

Also using digest does not in itself provide complete security for sensitive information. If you were using identity card numbers a determined attacker could generate a digest for every possible number in order to work out the original identity number. However if combined with other security procedures to limit access to data it does offer enhanced protection.

Other

  • You can hide / show blocked forms on the form management page by selecting a checkbox
  • Set a users timezone automatically to the timezone of their organisation unless they have specified their own specific timezone
  • Added a checkbox to the monitor page to ignore issues that are older than 100 days. This option is enabled by default.
  • More log event types older than 100 days are now moved into archive which should further speed up access to the logs.
  • Make duplicate choice names a warning in the online editor rather than an error.
  • Label the prime survey in a group of surveys as being part of that group. Previously it was only implied that the first survey in a group was part of the group since its name was used as the group name.
  • Make questions that are hidden in fieldTask also hidden in webforms
  • Add support for remote postgresql database installations, in particular AWS RDS.

Version 21.05

New Features

Local reference data

A great long standing feature of Smap is to allow surveys to lookup reference data that has been collected by another survey. This data can be used in calculations or as a list of choices in a select question and is downloaded onto your phone or tablet when you refresh fieldTask. The new “Local reference data” feature now allows you to also request that reference data be searched in survey instances that have been completed on your phone but not yet sent to the server. More details here.

One example of where this might be useful is; say you are working offline in a remote area with agriculture producers. Perhaps you have workers going to farms and recording details of a shipment using a survey. One of the questions in the survey could be to look up the farmer code from the list of registered farmers. However what if you find a farmer with a shipment ready to go who is not yet registered in the system? With this new feature you can register the farmer in the registration survey and then when you complete the shipment survey and select the farmer this new farmer’s identifier will be available along with all the other farmers in the system.

Searching of local reference data is completely transparent you just need to specify the normal pulldata or search function. You do however need to enable local reference lookups for each survey that uses it.

Get choice lists from repeats

Often in surveys you may collect repeating groups of data such as members of a household or faults in a piece of equipment. You can now refer to repeating data as the choice list in a “select one” or “select multiple” question.

So for example your select question might be “Select the household members that are attending school” and the list of choices would be the names of the household members. Details on how to do this including how to filter the choices is available here.

Webhooks

Web hooks are a way of calling other systems and passing them data when an event happens. For example you could go to your CRM system and ask it to create a “callback URL”. You can then go to Smap and on the notifications page add a notification that will call that URL whenever a completed survey is submitted. Data from that survey will then be immediately added to your CRM.

A great way to use these is by building a workflow in a tool like Zapier or one of the many other middleware tools out there. You can get a callback URL from zapier, add it to a smap notification and then specify a workflow in Zapier that will update your CRM, write to google sheets send an SMS or an email or all of these things. This can be done in a few minutes with no coding required. Details.

Specifying a Zapier Workflow action to write some answers from a Smap Survey into Google Sheets

Miscellaneous

Performance improvements

  • Archiving of log entries older that 100 days. Some of these are now being moved to an archive table so that queries of the log are faster. Access to this archive is not currently available from the Smap Server user interface so if you need to review old logs contact support.
  • The “all projects” option in monitor is no longer available as this could result in detailed data for dozens of projects and hundreds of surveys being searched which will probably be slow. Now you need to select the project that you want to monitor.
  • Additional information on submission and notification errors are now being written to the log file. This shows errors for the whole organisation so it is a more efficient way to find any errors than using the old “All Projects” search on the monitor page.
  • Prevent clashes when reference CSV files are being frequently updated. Previously under heavy load it had become common for a CSV file to be replaced while being read by another device downloading reference data causing the CSV read to fail.

Version 21.03

The March release of the Smap Server has been combined with a new version of FieldTask (v6.3000). For each new feature described below there is an indicator of whether that feature requires the updated SmapServer or FieldTask or both.

New Features

Change Organisation from within FieldTask

Requires FieldTask 6.300 as well as SmapServer 21.03.

Organisations allow you to define personal profiles for users, they contain other people, surveys and projects. The role you play in each organisation can be different as you can have different security groups making you an administrator in one organisation and an enumerator in another. Previously you have been able to switch organisations by selecting your user profile from the website menu however you can now do this from within fieldTask. Details.

Selecting a new organisation from within FieldTask

After you change organisation a refresh will automatically be initiated, replacing the surveys in FieldTask with those from the new organisation.

In App Audio Recording

Requires FieldTask 6.300 as well as SmapServer 21.03.

This feature is part of a new capability for recording audio within FieldTask without having to open an external audio app. This also allows you to continue recording audio while answering other questions. Details.

Recording audio while answering questions

Background Audio Recording

The new in-app audio recording also supports recording in the background. In this case background recording is specified as a meta question and is started as soon as the survey is opened. The first time it is used, permission is requested of the data collector. Then it is always turned on when the survey includes the meta question. However the data collector can turn off background recording using an option in the menu if necessary.

The audio recording is included with the survey results when the survey is saved as final. Details can be found here.

No More Device ID in FieldTask

Requires FieldTask 6.300

Returning the device ID was a privacy issue as it meant that different applications could correlate data submitted from the same device. Google no longer allow device device identifiers to be accessed so this has been replaced with the Firebase Installation Id. This ID is unique for each application on each device.

The impact of this is that if you are using the unique device identifier to determine which user submitted survey results then you will need to update this identifier after upgrading to FieldTask 6.300+. This might cause some inconvenience when monitoring submissions if it happens in the middle of an evaluation. However I would expect most organisations would update FieldTask between evaluations.

More secure storage of data on FieldTask

Requires FieldTask 6.300

Files and databases used by FieldTask will be automatically migrated to “scoped storage”. This will add additional restrictions on access to these files by other applications on the device.

When you first startup FieldTask after the upgrade you will see a message that the storage is being migrated.

Version 21.02

New Features

Making Web Forms a Web App

You can now install Web Forms as an application on to your iPhone, iPad, Android device or laptop as an application. This allows you to easily use all of the surveys that you have access to whether your are online or offline. If you have an internet connection opening a webform will get the survey from the network. (Although it may still be cached by your browser so its worth pressing the browser refresh button). Else, if you are offline, the last downloaded version of the survey will be downloaded.

Laptop

If you are using a laptop then this should work well with Chrome. Use Chrome to logon to your server and select Web Forms. An option to install the web app should appear in the browser bar as shown below.

Installing the Web Form app using chrome on a laptop

Notes

  • We are still working on automatically downloading surveys that are present on the webforms page. For the moment you should make sure that you open the survey in the webforms web app or in your browser before going offline.
  • Even when auto pre-downloading of surveys is available, this downloading will not include images. Hence if you use these in your survey you definitely have to open the survey itself before going offline.
  • Apple devices have to have iOS 13+ and you need to use Safari as your browser.

Personal Organisations

Organisations are one of the tools you can use to structure your surveys. Surveys are collected into projects and then projects are collected into organisations. Administrators can determine the enumerators or analysts that are given access to a project and the surveys it contains. But what if you want to have separate work areas where the administrators in one area cannot access and manage the second area? This is where organisations come in. A user with the “organisational administration” privilege can create organisations and give people access to them. Documentation on this existing feature is available here.

“Personal Organisations” now allow this feature to be used by administrators (that is an administrator who does not also have organisational administration privilege). An administrator who creates an organisation can control who has access to it. No other user can take it upon themselves to give themselves access with the exception of users who have the “organisational administration” privilege.

Creating a personal Organisations

As an administrator select the users menu in the admin module. Then click on the organisations tab. Click the “Add Organisation” button.

Adding a personal organisation

You need to give the organisation a name and if you like you can set some options for it. Normally though the default options are fine.

Note as an administrator you will only see your personal organisations on this tab. Someone with org admin privileges will see all organisations.

Giving other users access to your organisation

Click on the users tab and then edit the users you won’t to allow access to your organisation.

Adding sally to the organisation that I just created

You will see a section labelled organisations. This will have a list of your organisations and you can add and remove access for other users as you wish.

Switching to another organisation

Users can change organisations by selecting their profile. (The icon at the top right of every page that looks like a person’s head and shoulders).

Click on the organisation drop down and select the one you want to be. Then press save. The page will refresh and you will find yourself in the new organisation.

Bulk import and Export of users

User roles are now included in an export of users to a spreadsheet. They are also included when importing users from a spreadsheet.

Bug Fixes

  • Fixed issue with duplicate surveys being created when uploading a survey in XLSX format. This was caused by clicking multiple times on the upload button.
  • Fix colour of menu button in webforms when run on a mobile phone. Previously, if the background colour of the header bar had been changed then the menu button might not be visible.
  • Automatically return to the page you were trying to access after performing a logon
  • Problem with display of “Rank” widget in web forms when the appearance “minimal” was included
  • Add logging of changes to role settings to the application log

Performance Issues when looking up reference data

Yesterday I encountered a really good example of how using an online lookup of reference data from a survey can dramatically improve performance. The follow image shows the CPU usage on an 8 processor data collection server at the point when a single survey was modified to use online lookups instead of downloading all the reference data to the phones.

The heavy load prior to the switch was negatively impacting all users.

The cause of the problem

A large number of field workers were updating a survey with a hundred thousand records that was the source of reference data for other surveys. These people were working online with data sims in their phones. So as soon as a field worker completed a survey the results were sent to the server updating the reference data. That automatically triggered the downloading of the 100 thousand records, now including updated reference data, to all the other field workers. Each field worker would have the latest reference data each time they completed a survey but the load was high and response times were getting slow.

Choosing a solution for using reference data

The decision about how to access reference data depends on the type of work your field workers are doing and whether or not they have a network connection while they are doing it. Here are three scenarios to help you choose:

Scenario 1 – Working offline without a network connection

In this scenario your workforce might return to a central location that has wi-fi every evening where they will refresh their phones, uploading any data they have collected and downloading changes to surveys and reference data. The next day they head back out into the field and work offline.

The surveys can use the following functions to get reference data from other surveys and from csv files:

  • search. To get choices for a select question.
  • pulldata. To lookup reference values.

So what happens if the reference data gets really large and is updated continuously by the field worker. Well this is not actually a problem. The server will not be overwhelmed since the workers are only synchronizing their phones once per day. They should actually press the refresh button twice, maybe once in the evening and once in the morning. The first time it will upload all the new data and the second time they will download the updated reference information that includes all of the previous days work from all of the field workers.

Scenario 2 – working online with a permanent network connection

Now the field workers are submitting data as they complete surveys and getting updates to reference data whenever that changes. Because they are online you can use the following functions to get reference data:

  • lookup_choices. To get choice for a select question
  • lookup. To lookup reference data.

Now when completing a survey, if reference data is required, there will be a small delay while the network call is made and then the required reference data will be made available. It is no longer necessary to download large amounts of reference data, most of which will never be used, and store it on the phone. If the reference data is large you should find this a faster approach on the phone as you no longer need to wait for the reference data to be loaded into the survey when you open it.

If your reference data files are small, only a few thousand records, or they are not updated often then you can use either the online or the offline functions in this scenario.

Scenario 3 – A hybrid approach

In this scenario perhaps the workers are online most of the time and want immediate updates to reference data but occasionally they wander outside of the network and still need to lookup those references.

At the moment you would need to use the offline approach and look out for potential performance problems from large frequently updated sets of reference data. However we do have a change request pending to only synchronise the updates to the reference data and not to download all the data each time a single record changes. This change request will allow the hybrid scenario to scale without causing performance problems. Let us know if you need it and I will increase its priority.

Version 21.01

Features

Webforms works offline

Previously you could open a webform and then book mark it in order to be able to access that webform offline. Now, in version 21.01, the entire webforms page works offline. If you visit webforms while connected to the network all of the surveys that you have access to will be automatically downloaded and cached. Then if you go offline you will still be able to open these surveys.

Note there are some restrictions with this feature. If you have media files in your survey then these will not be automatically downloaded. If this is the case make sure that you open the survey while online. Once you have done this the media files will have been downloaded and cached and you will be able to access the survey though the webforms page while offline. In addition files required for language translation are not automatically downloaded. Hence you should open at least one survey before going offline to ensure that you have the language file for your language (English, Spanish etc).

A new look for the server

One of the challenges of maintaining a large server application over time is that the software frameworks that are used to manager the user interface change over time and it can be a lot of work to update old pages to match new ones. So the Smap server has web pages implemented using multiple technologies and with different appearances.

This release introduces a concerted effort to make these pages more consistent, simpler and faster. Of course that means introducing yet another technology version and style! In this case Bootstrap 4.5. However all the pages in the administration module along with the console have already been converted. Our aim is to convert and make consistent at least one other page per future release. Sol please bear with us while we work our way through this transition.

We have also standardised on horizontal menus across the top of the page. For no particular reason! Although they do use less screen space than vertical menus.

One advantage of the new pages is that they work much better in mobile phones.

Converted the user management page into two pages

The user management page now only has tabs for aspects of managing users:

  • Users
  • Projects
  • Organisations
  • Enterprises

All the other tabs that were also on that page are now on a Settings page:

Picture of settings page

Other

  • The font colour of the navigation menu can be specified in the appearance tab of the settings page
  • Appearance options for an organisation can no longer be set in the organisations tab. This tab is used by administrators to manage multiple organisations. Instead appearance is set in the appearance tab by administrators of an individual organisation.
  • Setting of the webform banner image has been moved to the webform options tab on the new settings page.
  • Users can be added to a security role in bulk. When creating or editing a role select the “Users” panel to add or remove users.
Bulk editing of users with a security role
  • Add a menu option to access Web Forms from the home page
  • Add an option to select the logo used in a reports for a form. This logo can be specified in the settings dialog in the online editor and is selected from images in shared resources.
  • Add support for embedding Quicksight dashboards into the server

Version 20.10

This month we have revisited a design decision made 8 years ago to restrict each form to having a single geometry (geopoint, geoshape, geotrace). By form I mean the main form within a survey or a subform that can be created using “begin repeat”.

The rationale for this was that shape files used in GIS analysis, and GeoJSON files, only allow a single geometry. All the other values are attributes of that one geometry. So the geometry might specify the location of a building with attributes such as name, number of stories, construction material. In a similar way in Smap each Form and Sub Form is stored in its own database table and this table was allowed at most one geometry with all the other questions being attributes of that geometry. In addition the location was always renamed to “the_geom” which simplified some of the early coding in Smap.

This restriction has now been lifted. You can specify multiple geopoints, geoshapes and geotraces in a single form. The relationship between these geometries and the other questions in the form is up to you and your from design. You may decide that a single geometry per form still makes sense for your project in which case you can create your forms that way.

Multiple geometries in a form

Uploading Surveys from a spreadsheet

No longer will you get an error if you have more than one location in your form. Also location questions will no longer be automatically renamed to “the_geom”. This has the potential to cause problems if you are updating an existing survey that has already collected location data. The issue can happen if:

  • You created a survey in a spreadsheet with a geopoint called say “my_location”
  • You uploaded that survey to Smap version 20.09 or less where its name would have been changed to “the_geom”
  • You then complete some surveys
  • Later you decide to make some change to the survey and you replace the existing survey. In the meantime Smap has been updated to version 20.10.
  • Now your location question will not be renamed and future surveys will record the location under “my_location”.

If the above happens you will see a warning in red text when you replace the survey saying that the location question will have a different name. If that is going to be an issue you can go back to your spreadsheet and change the name of the location question to “the_geom” so that it is compatible with the previously collected data. Then replace the survey again.

Analysis Page

When you create a map view for a survey you can now specify the geometry to be used for each form in the survey. A single geometry from each form will then be displayed on the map.

Dialog showing how the location questions are selected

Exporting Shape, KML and VRT files

When exporting these files you first have to select the form / sub form your are exporting. Now you can also select the location question to use.

Selecting location question for export, note only the questions in the selected form are enabled

Data API

If you specify that the data is returned as geoJson you can now optionally specify the question that will be set as the geometry using “geom_question” in the URL, for example:

http://demo.smap.com.au/api/v1/data/s1_34?links=true&geom_question=q3&geo_json=yes

Documentation

Importing Data

Previously when you backed up collect data to a spreadsheet, a geopoint question would be put under two columns “lat” and “lon”. Now each geopoint will still be put in two columns but the name will include the question name so for example “my_location_lat” and “my_location_lon”.

If you have an existing backup with just lat_lon then you can still import that as the system will just grab the first geopoint question that it can find in the form and of course there should only be one in that case.

Start Location

If you set start location in your survey then the location of the start of the survey will be automatically recorded without requiring input from the enumerator.

You can now select this preload to show on a map in the analysis page, export a shapefile based on the location or create follow up tasks at the location.

Note. When creating tasks the location question to be used is found automatically, it is chosen in the following order:

  • First. The first geopoint question answered by an enumerator that is found in the top level form
  • Second. A geopoint value automatically set using the “Start Location” preload
  • Third. The geometric center of the first geoshape or geotrace question that is found in the top level form

Other Changes

  • Users with “View Data” permission can now access reports
  • Fixed a bug where having server side calculations in a survey caused creation of a backup of the survey results in a spreadsheet to fail
  • Add a warning message in webforms if a browser is not supported. Previously some older browsers could just cause webforms to stop

Adding a Quicksight dashboard to Smap

Several customers use Power BI to show a dashboard from Smap data. They extract the data by periodically calling Smap’s data API. This works well, although it can add a significant CPU load to the Smap servers where there is a lot of data and Power BI updates frequently.

I thought I would have a go at using AWS Quicksight to do the same thing. This proved to be straightforward and within a few hours, including the learning curve, I had a working dashboard. which included the donut chart shown below.

Survey submissions by organisation.

Cool things about Quicksight

  • It is serverless so It can scale automatically from small implementations to very large. This scaling also applies to cost which should be small for small usage and large for heavy usage.
  • Dashboards can be embedded into your website
  • You are not restricted to just accessing Smap data you can include text documents, other databases, use artificial intelligence services to gain insights into the data etc.
  • Apps are available for iOS and Android to show the data.
  • It now supports maps which seems to be a relatively recent addition
  • When accessing a Smap server, which is also on AWS, the request does not need to go across the Internet
  • A built in data engine called “SPICE” which stands for Super-fast, Parallel, In-memory Calculation Engine offloads the work from the Smap server for your data crunching.
  • It is pretty easy to use.

How to get Started

  • Create an account on AWS
  • Ask Smap to provide a logon to access your data on the server. Specify to Smap the surveys you need to access as well as any table joins you want. We will set up secure read only access for you to just your data.
  • Start creating dashboards.
  • Embed the dashboards into your own applications

Summary

If you are already a PowerBI, Tableau or other BI user then you can use that tool to add a dashboard to visualise Smap data. Probably using the API but we can also consider a solution with a direct JDBC data connections.

If you are not currently using a BI dashboard then you can consider using AWS Quicksight.