Version 24.01

This release contains a simple little change to allow WebForms to launch another survey to update an existing record. For example you can have a survey that searches for a patient record and then launches a survey to edit that record. This feature only works online but will make it easier to implement fully functional systems without requiring any coding.

I am considering extending this capability to FieldTask however at the moment I am not sure whether the functionality in FieldTask should be offline, online or both so I will defer it till another release.

New Features

Launching a new survey from a Survey now available in WebForms

This works as per the existing functionality that allows you to launch a survey from inside a survey using FieldTask.

Selecting records to update from WebForms

When launching a survey form you can include a parameter with the instance id of the record to be updated. This will cause the launched survey to be pre-populated with the data from the record you want to update.

To do this create a survey that will look up records.

  1. Add a way of selecting the instance id of the record in another survey. This could use pulldata or the search() appearance in a select one question. Using search() is supported by the online survey editor.
  2. Add a “Child Form” or “Parent Form” question type. If you are not sure which one to use then either will be fine.
  3. Add a parameter that identifies the instanceid to use. For example “instance=${get_instance}”

The following video shows this functionality.

You can pass initial data to the started surveys that override the value in the instance.

“Restore Data” from Archive moved to background

The process that restores a table of data for a survey using the original submissions has always run in the background. However on larger installations the original submissions may be backed up to low cost storage. Copying these back to the server can take some time causing timeouts and reports of errors. However this retrieval of submissions from archive is now also done in the background. It is treated as a background report so an entry will be shown in the reports page with the result of the request.

Result of Background Request to Restore Data

In summary you may notice the following changes when requesting a restoration of data:

  1. No more error messages!
  2. There may be a lengthy delay of several minutes before the submissions start being re-applied
  3. You can check on the progress of the request in the reports module, once this shows the request is complete you can follow progress of the re-application of the submissions using the monitoring page.

Other

  • Allow a super user to reapply failed uploads from the monitoring page for a survey that requires a security role even when they do not have that role. A super user should not be constrained by RBAC.

Version 23.11

New Features

Dynamic Required Questions

Instead of entering a value of “yes” or “true” in the “required” column of the XLSForm, to indicate that a question is required, you can specify an expression. The question will then only be set required if that expression evaluates to true.

typenamelabelrequiredrequired_message
select_one yes_norequired_questionShould the next question be mandatory?
textsituationDescribe the situation${required_question} = ‘yes’You must answer
Making a question required during a survey

This feature is available for both FieldTask and WebForms.

You can also set the required expression in the online editor. First click the button to make the question required and then enter the expression.

Dynamic Read Only Questions

Like dynamic required questions, you can also make a question be read only or editable depending on an expression. For example you may pre-fill address questions with existing values and make these read-only but then use a question called “changed” to make them editable.

typenamelabelreadonlydefault
select_one yes_nochangedChanged?
textaddressAddressnot(${changed} = ‘yes’)Fleet Street
Making an address question read-only if the address has not changed

Note: Unlike the conditional required feature this is not available for WebForms.

Other

  • Dynamic Defaults inside of a repeat. Using both FieldTask and WebForms you can now set a default value for a question inside a repeat that is calculated from previous repeat instances. Documentation is here.
  • Performance Improvements. Hopefully you will find accessing the online editor and the console faster.
  • Updated some infrastructure libraries to the latest version including Jersey and the PostgreSQL JDBC driver.

Version 23.10

New Features

Filtering of case management alerts

Previously case management alerts were generated if a case was not closed and it was older than the specified interval. You can now also add a filter so that the alert can conditionally be created depending on it’s data. For example you could create an alert. “Serious issue is Late”, with a filter ${criticality} = ‘serious’.

Note: you can refer to any data in the case using the ${} syntax.

Extendable Initialised Repeats

Previously, when you added a repeat count to a sub form, the number of repeats was set by the value of that count and the person filling in the form would not be asked if they wanted to add another record. This is probably what you want if the repeat count is set inside the form or is fixed. However you may have the situation where you are filling out a sub form using initial data from another survey so the repeat count is set by the number of existing records but you still want to allow the user to add additional records.

FieldTask version 6.814 is required to use this feature in FieldTask.

The specific user scenario where this requirement arose was for a list of suspects in a police case. We created a form that allowed the existing suspect information to be updated but we also wanted to allow additional suspects to be added.

This can now be done in WebForms and FieldTask by setting the appearance “extendable” on the “begin repeat”.

Rate Limiting APIs

An improved rate limiter has been added. This now limits the number of requests per minute, per organisation. By default the rate per minute is set to 0 which means no limit. However you can change this in the settings for your server. You need to have server owner privileges to do this.

If the rate is exceeded then an error message is returned with the request. Currently if you set an API rate limit it will apply to the following services:

  • api/v1/data
  • api/v1/data.csv
  • surveyKPI/items (Called when you view a table of data for a survey on the analysis page)

Automatic Sentiment Analysis of Text Responses

A sentiment can be automatically calculated for text responses entered in any of the following languages; English, German, Spanish, Italian, Portuguese, French, Japanese, Korean, Hindi, Arabic and Chinese.  This is done using Amazon Web Services (AWS) Comprehend service.  The sentiment can have one of the following values; Positive, Negative, Neutral or Mixed. A confidence value in the sentiment can also be generated.

 To add the sentiment to a text question include the following parameters:

  • auto_annotate=true
  • sentiment=yes
  • from_lang={the language you are getting the sentiment from}
  • source={the question name that has the text – no curly brackets around the name}

Note the online editor assists you in setting most of the above. Although you will have to add sentiment=yes yourself as an “other” parameter. This is shown below.

Other Changes

  • Improved the performance of the shared resources page for sites with a lot of resources
  • Improved performance of submissions. Notifications and other events triggered by a submission are now processed in a separate queue so that the submission manager can get straight on to the next submission.
  • Server side validation to detect invalid characters in names for projects, users, roles and so on
  • Server side sanitation of html for labels to remove executable scripts now done when the text is entered as well as when it is used
  • Fixed issues with the script that does a fresh install of the server
  • Reduced the volume of log entries written by automatic scripts, now these will only write one entry per day, not every time they run
  • Allowed a notification that sends a periodic report to be disabled
  • Allowed dateTime type questions to be updated in an oversight form
  • Allowed dashes “-” in user names

Version 23.08

Increased Scalability for Logs

Previously the log page showed the last 10,000 entries. It wasn’t possible to go further back in time than this which was starting to become a problem for high volume users. 10,000 was also quite a lot of entries to retrieve in one hit and it could be quite slow.

The change included in this release is to require a month to be selected and then all log entries for that month will be shown. There are also now forward and back buttons so you can easily page through the data to find the log entries that you need.

Logs Page showing new Month Selector

Convert Date/Time values to FieldTask Format in CSV files

If you have been using the pulldata() or lookup() function to get a Date/Time value from one survey to use in another survey, then you will have seen some significant limitations:

  • The data is returned in a format that cannot be used in a Date/Time question! So you have to put it into a text question.
  • The timezone is UTC so if you are using it as a text value, and showing it to a user completing a different survey, then the value shown will be misleading.

Date/Time values are now returned in fieldTask format. They are still in the UTC timezone but if you load the value into a Date/Time question then it will be displayed in the timezone of the device. You can also now use date functions to convert the value to the local timezone and then format it for display as text.

Examples

typenamecalculation
dateTimetheDatepulldata(‘linked_s29_386’, ‘committal_date’, ‘case’, ${case})
1. Getting a value from another survey as the initial value of a Date/Time question
typenamecaclulation
textcommittalDateformat-date(date(pulldata(‘linked_s29_386’, ‘committal_date’, ‘case’, ${case})), ‘%Y-%m-%d %H:%M:%S’)
2. Getting the date value and displaying it as text in the local time zone

Note in example 2 above the date() function converts the UTC value into the local timezone and the format-date() function then converts this local date into text.

Always send an email to a person assigned to a case

This change builds on last months update which added the ability to select a checkbox to email a user who has been automatically assigned a case. Now if you manually assign a case to a user, and if they have an email address, then they will be automatically notified by email.

Version 23.07

People have previously approached me about adding the automatic generation of reports but I haven’t considered it a high priority until now. In the case of M&E surveys it is very important to monitor submissions so that you can identify interviews that didn’t happen or problems with the survey design but these surveys usually only last for a few weeks or months and you can do this monitoring using a web browser.

However as more organisations use Smap for case management, the management of new records is required on an ongoing basis and it would be useful to get a regular email containing new and updated cases.

Hence I have added this capability to Smap. You can of course also use this new feature to monitor your M&E surveys by generating a spreadsheet automatically each day containing the previous day’s completed surveys.

Periodic Reports

These consist of two parts:

  1. The report
  2. A notification that generates a report covering submissions for a period of time and emails it to recipients.

The specification of reports has not changed. You can create them in the reports module under the “Public Reports” tab. However the date question, start date and end date will all be ignored when the report is generated from a notification. The date used will be the date of submission of the record to the server. The start and end dates that determine which records will be included will be determined by the notification. There is also a restriction that only spreadsheet reports can be generated periodically, so no PDF reports.

A new trigger has been added to notifications called “Periodic”. The only target permitted is “Email” so an email will be sent with the attached report.

Notification dialog for a Monthly Report

You can then specify the period as one of:

  1. Daily
  2. Weekly
  3. Monthly
  4. Yearly

Depending on the selection additional fields will be shown allowing you to specify day of the week, day of the month and month of the year.

You can set the time when the report will be generated. The data included in the report will depend on the selected period:

  1. Daily. All submissions on the day before the report is generated.
  2. Weekly. All submissions for the week before the report is generated.
  3. Monthly. All submissions for the month before the report is generated. So if you have specified that the report is generated on the 9th day of the month then included submissions will be up until midnight on the 8th day.
  4. Yearly. All submissions for the year before the report is generated.

Other Changes

  • Improve the performance of the generation of usage reports.
  • Include deletion of a record, and restoration of a deleted record, in the history of that record

Version 23.06

This release addresses some long standing issues with shared resources. As usual it was during a training course when these shortcomings became obvious with a dozen people updating the same shared CSV file.

Shared Resources

The name of the shared resource can be specified on upload

Just like when loading a survey template you can now specify the name of the resource when you upload it. Previously the shared resource name was set to the same as the filename.

Uploading a CSV file called students (1).csv to use as a shared resource “students”

Shared CSV files are shown separately to other media files

A CSV reference file is quite different to a picture and these are now shown in their own tab on the shared resources page.

XLSX format spreadsheets can be loaded as reference CSV files

The file will be converted on upload into a CSV file that can be loaded into fieldTask and WebForms. No longer do you need to save your spreadsheet data as CSV format while being careful to specify the name that will be used to reference the CSV file from the survey form. You can load your spreadsheet directly and specify the name on upload.

Separation of upload and replace

Now if you try and upload a shared resource with the same name as an existing resource, the upload will be rejected. Instead you have to select the replace button next to the name of the resource in a similar manner to replacing a survey form. This will reduce the chance of accidentally changing an existing resource.

Shared Resource File showing replace button

Note the above picture also shows the new download and history buttons.

Shared Resource History

A history is kept of all changes to shared CSV and media files. You can look at this to see who made a change and when. You can also download the original file that was uploaded. So if an XLSX spreadsheet was uploaded you can download that spreadsheet from the history view whereas if you select the download button on the shared resource you will get the CSV version.

History of uploads to a CSV file

In a related change the uploading of a shared resource file and its deletion are both logged. Amazing to think that they weren’t.

Other Changes

  • An informative error is now shown when uploading a survey, as an XLSForm, that has an invalid choice filter name in the choices list. These names are restricted and cannot, for example, include spaces.
  • Allow filtering on meta data in a report even if the meta data is not included in the report. For example you may only want to report on records uploaded within the last two days but don’t want to include meta data such as _upload_time in the output.
  • Allow Map Tiler Maps in PDF Exports

Version 23.05

It has been a busy few months of training which has contributed to a delay of 3 months since the last monthly update. However here at last is a new release for May.

New Features

A new Change password page

Previously a user clicked on the user profile menu item and then, in the dialog that popped up, they would select “Reset Password”. Now when you click on the user profile menu item you will see three sub menus.

Sub Menus under User Profile
  • “User Profile” takes you to the user profile dialog where you can change your current organisation, timezone, name etc.
  • “Change Password” takes you to the new Change Password page
  • Logout, logs you out of the application. This was previously also in the user profile dialog.

The change password page includes a button that will generate a random password for you. After you change your password and move to another page you will be asked to logon again. After you have done that your browser can remember, if you agree, the new password that you have used.

Improved role security

Previously if a survey had a role then an administrator, who also had that role, could add a new survey to the same bundle. This could be a problem if the role on the survey had column or row restrictions as the administrator could then by pass these restrictions. Now only an administrator who also has security manager privilege, can bundle new surveys to a survey that has roles. The security manager can then set any required row or column filtering on the new bundled survey.

Add the ability to force password expiry

The date a password is set is now recorded and the server owner can specify that all passwords older than that must be updated.

Console escalations

When automatically escalating a case to a new user after a console alert or record update you can now also send an email to someone notifying them of the escalation.

Another new feature is that a checkbox can be selected to send an email to the person who has been assigned, notifying them of their new case. This only works if the assigned user has an email specified in their profile.

Allow a form to explicitly set the key value

A user can specify that a survey form has a unique key and a key policy. Then records submitted with that key will be treated according to the key policy, merge, replace, discard or none (append).

However some times it is not possible to identify the components of the key when completing a survey. For example the key could be defined as:

${surname}-seq(surname)

So in the above case submissions with the same surname will not be considered as duplicates instead a unique number will be append. For example:

Smith-1
Jones-1
Smith-2

So to update these records you can use tasks and case management. But what if you wanted to do an ad-hoc update. This might occur if you go to a location and realise that there is new information to add to the record.

You can use the search() function in your survey form to download the key, such as “Jones-2”. This search function can use multiple questions such as name, age, address, gender to uniquely identify the record to be updated. If you now store the results of the search function in a question with name “_hrk” then that value will be used as the unique key on submission.

Other changes and bug fixes

  • Remove the warning that used to be shown when a survey form was created that had a group filled only with calculations.
  • In the console you can filter directly on the user name who submitted a record and the name of the user that the record is assigned to.
  • Clear the saved advanced query in the console if it causes an error.
  • Improve the efficiency of the uploading of submissions so that is uses less CPU and is faster.
  • Fix an issue with the API call to get a submissions list which would fail if GPS coordinates were malformed
  • Fix an issue with the preservation of a record’s unique key between record updates if that key inlcuded a seq() or serial() function.
  • Added additional event logging
  • Prevent a new organisation name from being specified that differs only in case. So we won’t now have organisations like Acme and acme and aCme registered on the SG server.
  • Fixed an issue that prevented initiating tasks from the console.
  • Allow administrators to move projects to one of their private organisations.
  • Remove the ability to move a user to new organisation Users should instead be granted access to organisations in their user settings by an administrator. They can then move themselves using the user profile dialog.
  • Fix issues when uploading empty data into a survey form from a file. This empty data was being set to text without characters, or in the case of a geopoint question, to a location of 0.0 0.0. The values will now be set to undefined as they would be if submitted from fieldTask.
  • Fix an issue where an non valid period to wait for a console alert could be set. This non valid period would stop processing of console alerts.
  • Fixed an error that prevented polygons from being used as default areas when accessed by the pulldata() function from a WebForm.
  • Fixed an issue with generating the default link to a survey form in a campaign email when the location of the link was not specified in the email content.
  • On the analysis page allow charts to group data by “note” type questions. This change recognizes that people are commonly setting values to note questions using dynamic defaults.
  • Add improved validation of XLSForms that have non valid headings on the choices worksheet.
  • Group outputs in the administration usage report by the survey identifier rather than the survey id. This means that replaced surveys will be considered the same survey in the report.
  • Add an option to the administration usage report to not include “All time” usage.
  • Upgraded multiple software libraries that had become out of date and had security issues flagged against them.
  • Fix an issue where the cache of a WebForm was not being updated when the survey definition had changed. This was occurring if the WebForm opened an existing record.
    • Increase the maximum number of records shown in a map, on the analysis page, to 10,000 from 1,000.

Version 23.02

New Features

The main change in this release is the upgrade of all pages to JQuery 3.5.1 and Bootstrap 4.5. Most pages were updated some time ago but some of the older, more complicated ones, had remained with the old version of these libraries.

The change you will notice is that the appearance of pages is more consistent. Those pages that used the older libraries such as the online editor should now have the same theme as Form Management.

The one exception to this improved consistency is the analysis page. JQuery has been updated, but it still uses jQuery-UI and not Bootstrap to set its appearance. Probably this page will never be converted to Bootstrap, but one day we will write a new Analysis page starting from scratch to replace it.

Other Changes

  • Added the “today” meta item to the list of meta items supported in the online editor.
  • Support added for the “user” meta item in WebForms.
  • Choice names can now have spaces. This only works with select_one questions which is the reason it was previously not allowed. Be carefully using this if you are going to change the question type to a select multiple.
  • Allow the user currently assigned to a case to be used as a filter in a search() function
  • Allow read only questions to be added to exports in predefined reports
  • Sanitise the HTML used in WebForms to restrict it to a limited set of permitted HTML elements
  • Removed automatic conversion of text to uppercase when using the grid format in WebForms

Version 23.01

Oh dear its 2023 already. A few holidays may have occasionally interrupted our development but we still have a January release.

New Features

Fingerprints

FieldTask can now capture fingerprints.

Setup

The fingerprints are captured using a fingerprint reader of which we currently support only one. The Mantra MFS100 from Mantratec which costs about USD80. This plugs into the USB port on an Android device. Assuming this port is USB-C then you will also need a USB-C to USB-A adapter.

MFS100 attached to a phone

It may be possible to support additional readers. Contact us with your requirements.

Some software is also needed to connect fieldTask to the physical reader. This is the Smap Fingerprint reader app. It can be downloaded from the server home page under the download menu. This app is an extension of the opensource Keppel app, the extension being to capture fingerprints as an image, Keppel only supports the ISO 19794-4 fingerprint template.

The lSO template can be used with Smap, it stores the fingerprint minutia in a text file, which is smaller than an image, however it is difficult to manually validate matches. Using images you can find matching records in the system that exceed a threshold of similarity and then visually inspect them to assess the accuracy of the match.

The form

Example form in Google Sheets.

For each fingerprint that you want to to capture create a question in your form. Set the question type and appearance as per the following table.

Fingerprint TypeQuestion TypeAppearance
Imageimageex:uk.ac.lshtm.keppel.android.SCAN(type=’image’)
ISO 19794-4textex:uk.ac.lshtm.keppel.android.SCAN(type=’image’)
Specifying questions to collect fingerprints

The resulting fingerprint and ISO template are shown below. The black square was added using an image editor to mark the data as a sample only.

Image and ISO fingerprints in FieldTask

Finding Matches

Only a very limited matching capability has been added in this release. This functionality will be increased according to need. On the console, if you select a record, a new button labeled “Links” will be shown. Click on this button to open a page that shows the image fingerprints and any records that match. The default minimum score required for a match is 40.0. You can change this score and press the search button to search again. If you click on the link to a matching record it will open the record in a web form.

Algorithm to identify matches

Smap is using an open source fingerprint matching tool called SourceAFIS. Commercial alternatives offer more accurate matching however being open source results in a more sustainable system. Organisations collecting fingerprints should also investigate using this system to collect fingerprints that can be used to train an Artificial Intelligence system to find matches.

Download Menu on Home Page

Previously we had an image on the home page that you could click on to download the fieldTask app. This was a “big button” to make it easier for users of mobile phones to interact with the web server. However browsers on phones handle menus well, so in this release there is now a download menu.

Accessing the download menu from a mobile phone

This menu includes the fieldTask and the fingerprint readers apps which can be downloaded onto an Android device. Smap uploader is still available for download to a laptop.

Other Changes

  • The “thread” value of a record is now included in the data API output. This contains the instanceid of the submission that created the record. Every time the record is updated it will get a new instanceid but the thread remains the same. It can be used to get the history of changes to a record.
  • The online editor now allows multiple columns from a CSV file to be specified as the. choice label for choices obtained using a search function. Previously this could only be specified using the spreadsheet editor.

Version 22.12

New Features

Highlighting of survey types on the survey admin page

Form Management Page with multiple survey types

An additional column, labeled “Type”, shows the survey type. This is left blank if the survey is a standard data collection survey. A colour highlight is also applied to help make the survey type stand out.

Survey TypeHighlight ColourDescription
OversightOrangeSurvey used in the console to update a record
Read OnlyPurpleSurvey that is used by people with mobile devices to view data from the server, but does not submit any results.
DataNoneStandard data collection survey
Types of Survey and their colour on the survey admin page

Assign a case from FieldTask

A case consists of a record of data and a survey form to update that record. Once a case is assigned to a user it will appear in FieldTask and in WebForms. The case can be updated multiple times until it is complete and no longer assigned.

In this release you can now assign a person to a case by selecting their name in a survey in FieldTask. For example in a policing app you may want to assign a particular detective to follow up a case from the field without requiring that to done as a back office function.

Notification to assign a person to a case using data from another survey

The survey that contains the assignee and the survey to be completed must either be the same or be in the same bundle. There must also be a key so that the record to be used in the case can be identified

Keys apply to bundles of surveys

Previously a unique key had to be specified for each individual survey. Multiple surveys combined into a single bundle could end up having different keys and key policies. This could cause problems as new entries from surveys without key policies would create duplicate records.

The obligation was on the survey designer to make sure each survey in the bundle had the same key, However now a common key is used across all bundles and only needs to be specified once.

When you select “Keys” in the File menu in the online editor, the case management page will be shown, you can then specify the common key for all the surveys in the bundle.

Other Changes

  • Cases are now shown on the WebForms page on the tasks tab.
  • Cases can be rejected in FieldTask and WebForms. The case will be set to not assigned. In WebForms the reject button is to the right of the button that opens the case. In FieldTask there is a menu option to the right of the case on the tasks tab. Select this then select the reject option. This feature requires FieldTask version 6.808.
  • WebForms will now mostly use the same colours as FieldTask to identify surveys and tasks. Yellow for data surveys, blue for tasks. The exception is for read only surveys which are coloured purple in fieldTask and grey in WebForms.
  • WebForms now includes an icon in the task list that indicates whether the list item is a task or a case. Cases are identified using an open folder icon and tasks using a file icon.
  • The “view” button in the console is now always enabled. This allows a person to view details of an unassigned record, including its history, without having to lock it first.
  • XLSForms can now include formula. Hence you could set choice names or labels using a formula and the value of that formula will be used in the survey.