Oh dear its 2023 already. A few holidays may have occasionally interrupted our development but we still have a January release.

New Features

Fingerprints

FieldTask can now capture fingerprints.

Setup

The fingerprints are captured using a fingerprint reader of which we currently support only one. The Mantra MFS100 from Mantratec which costs about USD80. This plugs into the USB port on an Android device. Assuming this port is USB-C then you will also need a USB-C to USB-A adapter.

It may be possible to support additional readers. Contact us with your requirements.

Some software is also needed to connect fieldTask to the physical reader. This is the Smap Fingerprint reader app. It can be downloaded from the server home page under the download menu. This app is an extension of the opensource Keppel app, the extension being to capture fingerprints as an image, Keppel only supports the ISO 19794-4 fingerprint template.

The lSO template can be used with Smap, it stores the fingerprint minutia in a text file, which is smaller than an image, however it is difficult to manually validate matches. Using images you can find matching records in the system that exceed a threshold of similarity and then visually inspect them to assess the accuracy of the match.

The form

Example form in Google Sheets.

For each fingerprint that you want to to capture create a question in your form. Set the question type and appearance as per the following table.

Fingerprint Type	Question Type	Appearance
Image	image	ex:uk.ac.lshtm.keppel.android.SCAN(type=’image’)
ISO 19794-4	text	ex:uk.ac.lshtm.keppel.android.SCAN(type=’image’)

The resulting fingerprint and ISO template are shown below. The black square was added using an image editor to mark the data as a sample only.

Finding Matches

Only a very limited matching capability has been added in this release. This functionality will be increased according to need. On the console, if you select a record, a new button labeled “Links” will be shown. Click on this button to open a page that shows the image fingerprints and any records that match. The default minimum score required for a match is 40.0. You can change this score and press the search button to search again. If you click on the link to a matching record it will open the record in a web form.

Algorithm to identify matches

Smap is using an open source fingerprint matching tool called SourceAFIS. Commercial alternatives offer more accurate matching however being open source results in a more sustainable system. Organisations collecting fingerprints should also investigate using this system to collect fingerprints that can be used to train an Artificial Intelligence system to find matches.

Download Menu on Home Page

Previously we had an image on the home page that you could click on to download the fieldTask app. This was a “big button” to make it easier for users of mobile phones to interact with the web server. However browsers on phones handle menus well, so in this release there is now a download menu.

This menu includes the fieldTask and the fingerprint readers apps which can be downloaded onto an Android device. Smap uploader is still available for download to a laptop.

Other Changes

• The “thread” value of a record is now included in the data API output. This contains the instanceid of the submission that created the record. Every time the record is updated it will get a new instanceid but the thread remains the same. It can be used to get the history of changes to a record.
• The online editor now allows multiple columns from a CSV file to be specified as the. choice label for choices obtained using a search function. Previously this could only be specified using the spreadsheet editor.

New Features

Highlighting of survey types on the survey admin page

An additional column, labeled “Type”, shows the survey type. This is left blank if the survey is a standard data collection survey. A colour highlight is also applied to help make the survey type stand out.

Survey Type	Highlight Colour	Description
Oversight	Orange	Survey used in the console to update a record
Read Only	Purple	Survey that is used by people with mobile devices to view data from the server, but does not submit any results.
Data	None	Standard data collection survey

Assign a case from FieldTask

A case consists of a record of data and a survey form to update that record. Once a case is assigned to a user it will appear in FieldTask and in WebForms. The case can be updated multiple times until it is complete and no longer assigned.

In this release you can now assign a person to a case by selecting their name in a survey in FieldTask. For example in a policing app you may want to assign a particular detective to follow up a case from the field without requiring that to done as a back office function.

The survey that contains the assignee and the survey to be completed must either be the same or be in the same bundle. There must also be a key so that the record to be used in the case can be identified

Keys apply to bundles of surveys

Previously a unique key had to be specified for each individual survey. Multiple surveys combined into a single bundle could end up having different keys and key policies. This could cause problems as new entries from surveys without key policies would create duplicate records.

The obligation was on the survey designer to make sure each survey in the bundle had the same key, However now a common key is used across all bundles and only needs to be specified once.

When you select “Keys” in the File menu in the online editor, the case management page will be shown, you can then specify the common key for all the surveys in the bundle.

Other Changes

• Cases are now shown on the WebForms page on the tasks tab.
• Cases can be rejected in FieldTask and WebForms. The case will be set to not assigned. In WebForms the reject button is to the right of the button that opens the case. In FieldTask there is a menu option to the right of the case on the tasks tab. Select this then select the reject option. This feature requires FieldTask version 6.808.
• WebForms will now mostly use the same colours as FieldTask to identify surveys and tasks. Yellow for data surveys, blue for tasks. The exception is for read only surveys which are coloured purple in fieldTask and grey in WebForms.
• WebForms now includes an icon in the task list that indicates whether the list item is a task or a case. Cases are identified using an open folder icon and tasks using a file icon.
• The “view” button in the console is now always enabled. This allows a person to view details of an unassigned record, including its history, without having to lock it first.
• XLSForms can now include formula. Hence you could set choice names or labels using a formula and the value of that formula will be used in the survey.

The November changes described in these release notes may also require fieldTask version 6.803 or later.

New Features

Read only surveys

It has become a common scenario for people to use survey forms to look up information on the server from fieldTask or WebForms and often no data needs to be recorded.

However every form, when it gets to the end, invites the user to “save and exit”. A record of data is then sent to the server taking up valuable communications bandwidth and space in the database.

To improve the user experience we have added the option of making a form “Read Only”. When you get to the end of these the button just says “Exit” and no data is saved or sent to the server. These forms also appear with a purple icon in fieldTask and WebForms so it is immediately obvious to users that these are forms for looking up data and not for create a new record.

Documentation: https://www.smap.com.au/docs/read-only-surveys.html

Case Management

When a case is created there could be multiple surveys that can be used to complete that case. In fact any survey that is in the same bundle. This is fine if the user is working with the console, they can just select the survey required to complete the case. However on fieldTask we only want to show a single icon which is associated with a single survey form.

One option to solve this problem is to allow the survey to be specified when the case is assigned to a user. Probably we will end up going this way. However in this release cases replicated to fieldTask are now associated with the survey that last updated the record of data. Notify us if you need to be able to specify the survey used to complete the case.

Other

• There is a new parameter (geotextlength) which can be used to specify the maximum number of rows to display of the text answer for a geotrace, or geoshape question in fieldTask. For example: geotextlength=3
• The locking and releasing of records in the console has been debounced. Previously it was easy to click multiple times on these buttons which might cause odd error messages to be displayed.
• When you access geotrace or geoshape data from another survey using the pulldata() function, this is now formatted so that it will show in a geotrace or geoshape question. Previously it would have only been visible as text data.
• The user assigned to a case is now returned in a data API call. Look for the key value of “_assigned”.

Bug Fixes

• There were issues with the sorting of tasks in the fieldTask display
• There was an error in deleting cases in fieldTask that become unassigned
• pdfmapsource appearance was being set for every question opened in the online editor
• The simple filter on the analysis page did not allow preloads to be specified
• Using a select multiple question to specify multiple email addresses to be sent a notification was not working
• There was a recently reintroduced issue with loading CSV files that have a leading BOM character.

New Features

Case Monitoring tab

A tab has been added to the monitoring page to show the number of new cases created per day and the number closed. The period defaults to 7 days but can be extended to 30 days. Select a survey that has case management enabled to show data on the chart.

Documentation is available here.

Archiving of Survey data

Commonly people are creating surveys to manage their day to day operations. These can become large with millions of rows and because they are accessed frquently this can also cause performance issues. Another common feature of this sort of usage is that only the last 12 months, or an even shorter period, of data might be frequently accessed.

To support this use case we have added the ability to archive data older than a specified date into a new survey. This can be done from the analysis page using the aside. Refer to documentation here.

Data older than the specified date will be moved into a new survey, identical to the current one with the archive date enclosed in square brackets and appended to its name. The survey with the archive data will be blocked by default.

That may be where you want to stop or you could move this survey to a project dedicated to backups so that it is out of the way or you could just delete it if the old data is of no use.

Minimum Password Strength

It has previously been possible to set minimum password strengths for the server or an organisation however to do this you would have needed to have organisational administration privilege. There is now a new tab on the users page called “Other Settings” that allows an administrator to set the minimum password strength for their organisation.

Other Changes

• The appearance of the monitor page has been changed so that tabs are now used to separate different types of data.
• Add a control to select “placement-map” as an appearance in the online editor for geotrace, geoshape and geocompound type questions.
• Modify the chart view in the console to allow users to add charts. In the future it will be possible to add charts on question data however for the moment these charts still need to be related to case management.
• Performance improvments to the console
• Allow ampersands ‘&’ to be used in a choice value
• Allow the source of a static map included in a PDF to be specified as an appearance setting on a question. The source can be google or mapbox.
• When a survey is tested on the admin page that survey is now set as the default survey show that its results will show automatically in the console.
• Removed the ability to forward a submission to another server.
• Add a monthly report on events such as static map views.

This release is focussed on improvements to case management. The changes have been requested specifically for an accountability system however they should be useful for any case management system that you implement.

New Features

Escalation / Assignment

Cases can be automatically assigned to a user or reassigned to a different user. The latter potentially being an “escalation”. The rules for an automatic escalation are specified in a notification and can be triggered by the updating of data in a case or a case alert (another new feature described below). For example you might create a notification that is triggered by the submission of a completed survey and that then “escalates” the new case to someone. In this case the escalation is just doing the initial assignment. However if an alert is triggered indicating that the case has not been closed within a specified time interval then this alert can also trigger a notification to escalate the case to a new user.

Completed Case

Completed cases are defined as a case that is not assigned to anyone and that has a status equal to the “final status” for the case. These will be hidden by default in the console. A case will enter the completed state automatically when its status is set to the specified value.

To enable “completed cases” use the online editor for a survey to select the tools menu and then Case Management Configuration. You can then specify the question that holds the status and the value that indicates that the case is complete.

Case Alerts

Alerts can be added to a survey bundle. These are triggered a specified period of time after the case is created if the case has not been closed. The alert can then be viewed in the console and used as a trigger to a notification to escalate, send an email or create a task.

Charts

There are currently 4 fixed charts that are related to case management.

• Status. Frequency of each status among the current cases.
• Assigned. Who has been assigned cases.
• Alert. Frequency of alerts triggered by the cases.
• Criticality. Frequency of each value in the criticality column.

The charts are on the chart tab of the console. They will use data from the current set of cases after filters have been applied. So if you have specified that only cases between two dates should be shown, the charts will only use that data.

Mobile Devices

A user who is assigned a case will see that case in the tasks tab of fieldTask. The case will remain on the device until either the user completes it or it is re-assigned to someone else.

Note the case is only downloaded if a final status has been specified for the bundle of surveys. That is if the case can be completed.

Cases are shown with a “folder” icon with a blue background if the case has not been modified on the device, once the case is modified it is shown with a purple background. However unlike tasks or other completed survey instances a modified case is always editable so the user can make multiple updates before synchronising.

The case will be labelled with the instance name of the record. If the instance name of the record is not set, then the key for the record will be used. Finally if that is not set the primary key of the record will be used to identify it.

Other Changes

• Allow calculations to be used to create media file names in the media export
• Allow setting of password strength requirements

New Features

Improved Performance

The number of submissions processed by hosted server has continued to increase with a corresponding increase in the size of datasets that are being analysed and reported on. The average submission rate is nearly 15,000 per day and 11 Million forms have now been completed on our hosted servers.

To address the increasing load we have made some minor performance improvements and scheduled additional improvements for the next release.

Multiple PDF Templates For One Survey

Previously you could add a single PDF template to a survey using the settings dialog in the online editor. That template would then be used anytime a PDF of submitted survey results was generated. The new functionality is much more flexible. You can have multiple templates that you can select when generating the PDF and you can also select not to use a template.

The settings dialog no longer has the PDF Template loading function, nor can you download the template from there.

Instead PDF templates are accessed in the online editor from the menu “Tools” and then “Templates”.

Templates loaded via the old settings approach will be found here along with any new templates that you load. Those old PDF templates cannot be disabled (set as not available), set as default or have a rule specified for when they should be automatically applied. If you want to use these new features you can download the template, delete the old one and reload on the templates page.

Templates can be set not available if you don’t want one to be used. One template can be set as the default.

Rules can be added to automatically select a template to be used in an email notification of submission results. These rules use the standard server expression syntax. For example:

${region} = 'region1'

Compound Question Type

This feature builds on the announcement in the previous release of a “pdf_field” question type that could be used to combine geopoint location questions into a single compound map or image.

Now by using a “geocompound” question type you can consolidate the data collection into a single question.

Geocompound works like the geotrace type in that a line can be sketched out over a map. However points on the line can also be tagged with a marker. In this release there are 3 marker types; none, pit and fault. In a future release these will be specified as parameters.

Geocompound questions are supported in webforms. They can be edited after collection and can be applied to PDF exports in the same way as the pdf_field type.

Other

• Added an option to webform settings to disable “Save as Draft” checkbox.
• Add an option to survey settings to compress PDFs
• Add support for an attribute “pdfshowimage”. This can be used with select questions that include an image with each choice. The image for the selected choice will then be shown in the exported PDF
• Add support for an attribute “pdfstretch”. This can be used with image type questions and will cause the image to be resized to fit into the space allocated for it in the PDF. The default action, without this attribute, is to fit the image into the space while keeping the height and length in their original proportions
• Make the default source of audio for an audio question be internal rather than an external audio app
• Allow traffic light symbols in PDF exports to be “crossed out”. Allow labels to be added to traffic light symbols.

Here is the November release of Smap. We did not release an update in October, although you may have noticed that the version number of some server installations said 21.10, but if you clicked on the link for changes it just took you back to the September changes. Unfortunately I never got around to creating the release notes so officially October never existed.

New Features

Specialised PDF components

There is a new question type “pdf_field” so you can specify specialised components to be included in a PDF report. Documentation is here. Currently two new components are available and they are intended for use in reporting faults in cables or pipelines. However now that the framework has been created, we can add additional components. Let the Smap team know if there is anything you would like.

Component 1 – Fault Image

This component is given the names of the questions that record, start of cable segment, end of cable segment and fault position. It then shows these in an image that includes the distance from the cable segment start to the first fault and the distance from the end of the cable segment to the second fault.

Component 2 – Fault Map

The input to this component is the same as the input to the image component but this time the information is shown on a map.

Background Report Generation

Just as the new PDF components are specialised, this new background report generation may not be useful for many organisations. However it has been added to help large organisations.

The first report that has been converted to background generation is the user usage report on the form management page, if your organisation has thousands of users and millions of submissions then this can take a long time and report generations for these large organisations have been failing.

Now once the report has been requested a message will be displayed saying that the report generation is in progress and the completed report can be found on the reports page.

To access reports select the menu “modules” and then “reports”. The reports generated for the last two weeks are shown here and can be downloaded by clicking on the report’s blue button. If the report is taking a while to complete you may see its status as “Pending” rather than “Complete”.

The previous functionality available on the reports page can be found under the tab “Public Reports”. This still allows you to define a report and then get the URL to that live data report that you can share with other people.

Allow webform links to tasks to be completed multiple times

Previously if you created a task as part of an email campaign then that task could only be completed once. However there are applications where you want a task, preloaded with some specific data, to be completed multiple times. For example if you have a barcode that links to a task and you want that task for that url to be preloaded with a venue location say. Documentation.

Other

• The version of fieldTask used to synchronise with the server is now included in the user audit log. This also requires fieldTask version 6.515+. Documentation.

Bug Fixes

• Survey forms that do not have any questions no longer cause an obscure error to be displayed when they are downloaded on to fieldTask. Instead when the user opens the survey on fieldTask they will see a helpful message stating that the form has no questions.
• When getting an email address from survey data in order to send a notification, previously the name of the question was used in the data query instead of the column name of the question. If the question name contained characters that were not valid as table column names then this would not have worked.

New Features

Lookup reference data using an expression

Using the pulldata() or lookup() function you can now specify an expression. For example the following pulldata calculation will get the average age of people older than the answer set in the “age” question. The reference data is taken from a csv file called “people”. More information can be found in the documentation.

if(${age} >= 0, pulldata('people', 'age', "cast(#{age} as integer) > ${age}", 'mean', 'eval'),'')

Consistent reference data support on FieldTask and WebForms

Well more consistent anyway, there are still a few idiosyncrasies as you move between fieldTask and webforms. So for example the new feature of using an expression to lookup reference data works in both fieldTask and WebForms.

Other

• Links created for mailouts can now accept multiple submissions. To allow this select the checkbox “multiple submissions” when specifying your campaign.
• User locations page
• Template upload checks for missing end repeats

New Features

Digest Function in Webforms

The digest function now works in both FieldTask and Webforms. One application of digest is to obfuscate sensitive data while still retaining a unique identifier for that data. For example instead of recording a person’s name you can record the digest. If you then complete another survey and the name of this new person is the same then the digests will also be the same. So you will know that the people in these two surveys had the same name you just won’t know what that name is.

Example Digest Application

In this example we are going to hide the persons name and use the digest of the name as the key for the record so we can update the data on that person.

1. Specify the survey.

type	name	label	appearance	calculation
text	name	Name	phoneonly
calculate	digest_name			digest(${name}, ‘SHA-1’)
int	age	Age

Note the appearance of “phoneonly” on the name will prevent the name from being stored in the server database. It will be replaced with “xxxx”.

2. Set the survey key to be ${digest_name}. This means that when we submit another record with the same digest as an existing record then the existing record will be updated.

3. Results as shown in the console

Note I have clicked the option to “show deleted data”, So you can see that record 1, which had an age of “23” was merged with record 3 that updated that age to “57”. Both of those records had the same digest_name. The actual name however is not recorded and only “xxxx” is stored in the database.

4. Final thoughts

Using the digest of a name as a key is probably not a good idea in the real world. This is particularly true where the recorded name is a transliteration from another language. Minor spelling mistake would result in a completely different digest so the value of the digest is probably low. However you could absolutely use the above approach with passport numbers or identity card numbers.

Also using digest does not in itself provide complete security for sensitive information. If you were using identity card numbers a determined attacker could generate a digest for every possible number in order to work out the original identity number. However if combined with other security procedures to limit access to data it does offer enhanced protection.

Other

• You can hide / show blocked forms on the form management page by selecting a checkbox
• Set a users timezone automatically to the timezone of their organisation unless they have specified their own specific timezone
• Added a checkbox to the monitor page to ignore issues that are older than 100 days. This option is enabled by default.
• More log event types older than 100 days are now moved into archive which should further speed up access to the logs.
• Make duplicate choice names a warning in the online editor rather than an error.
• Label the prime survey in a group of surveys as being part of that group. Previously it was only implied that the first survey in a group was part of the group since its name was used as the group name.
• Make questions that are hidden in fieldTask also hidden in webforms
• Add support for remote postgresql database installations, in particular AWS RDS.

New Features

Local reference data

A great long standing feature of Smap is to allow surveys to lookup reference data that has been collected by another survey. This data can be used in calculations or as a list of choices in a select question and is downloaded onto your phone or tablet when you refresh fieldTask. The new “Local reference data” feature now allows you to also request that reference data be searched in survey instances that have been completed on your phone but not yet sent to the server. More details here.

One example of where this might be useful is; say you are working offline in a remote area with agriculture producers. Perhaps you have workers going to farms and recording details of a shipment using a survey. One of the questions in the survey could be to look up the farmer code from the list of registered farmers. However what if you find a farmer with a shipment ready to go who is not yet registered in the system? With this new feature you can register the farmer in the registration survey and then when you complete the shipment survey and select the farmer this new farmer’s identifier will be available along with all the other farmers in the system.

Searching of local reference data is completely transparent you just need to specify the normal pulldata or search function. You do however need to enable local reference lookups for each survey that uses it.

Get choice lists from repeats

Often in surveys you may collect repeating groups of data such as members of a household or faults in a piece of equipment. You can now refer to repeating data as the choice list in a “select one” or “select multiple” question.

So for example your select question might be “Select the household members that are attending school” and the list of choices would be the names of the household members. Details on how to do this including how to filter the choices is available here.

Webhooks

Web hooks are a way of calling other systems and passing them data when an event happens. For example you could go to your CRM system and ask it to create a “callback URL”. You can then go to Smap and on the notifications page add a notification that will call that URL whenever a completed survey is submitted. Data from that survey will then be immediately added to your CRM.

A great way to use these is by building a workflow in a tool like Zapier or one of the many other middleware tools out there. You can get a callback URL from zapier, add it to a smap notification and then specify a workflow in Zapier that will update your CRM, write to google sheets send an SMS or an email or all of these things. This can be done in a few minutes with no coding required. Details.

Miscellaneous

• Allow option of including temporary users in usage report by user.
• Update appearance of duplicates page to use bootstrap 4.5 and horizontal menus.

Performance improvements

• Archiving of log entries older that 100 days. Some of these are now being moved to an archive table so that queries of the log are faster. Access to this archive is not currently available from the Smap Server user interface so if you need to review old logs contact support.
• The “all projects” option in monitor is no longer available as this could result in detailed data for dozens of projects and hundreds of surveys being searched which will probably be slow. Now you need to select the project that you want to monitor.
• Additional information on submission and notification errors are now being written to the log file. This shows errors for the whole organisation so it is a more efficient way to find any errors than using the old “All Projects” search on the monitor page.
• Prevent clashes when reference CSV files are being frequently updated. Previously under heavy load it had become common for a CSV file to be replaced while being read by another device downloading reference data causing the CSV read to fail.