New Features

Direct WhatsApp Connections

This simplifies the use of WhatsApp conversations to open and resolve a case. You won’t need to use Vonage or other third-party gateways, although you still can if that adds value. So one less account to set up, one less bill to pay, and one less thing to break in your case management system.

This release also shows conversations in WebForms and FieldTask. This makes it easier to create a form that transfers information from a conversation into a record.

This feature is part of a long-running goal for Smap. We want case management systems that can engage directly with local communities, however remote they are. WhatsApp is often the channel people already use every day, including in places where other infrastructure is thin. Now community members can send reports, get updates, provide feedback and respond to follow-ups on WhatsApp, and the conversation is included in the case record.

Migration of Organisational Settings to Settings page

Many settings for an organisation used to be set in the organisations tab of the user page. This made some sense in that only an organisation owner or organisation administrator could change them and access to this tab was restricted to users with those permissions. However some of these settings were duplicated on the settings page.

The setup has now been simplified. All settings are now managed through the settings page. If you have organisational administration privilege then you will be able see an organisation tab for those settings. This means that if you are an organisational administrator and want to change the settings on some random organisation you will now need to switch to that organisation first.

Other

• Updates to the documentation to match the code. This had drifted over time but now the documentation should be fully up to date
• Data Subject Access and Right to be Forgotten need the Data Protection Officer(DPO) security group; Analyst access has been removed. Ask your security manager or organisational administrator to grant DPO to anyone who uses these tools.
• Importing campaign emails from a spreadsheet no longer fails when it has no settings sheet
• Rate limited API requests now get a 429 error (Too many requests) so they can respond appropriately
• WhatsApp messages are shown in the record history with a WhatsApp icon
• Webhook URLs with a port number now work

New Features

DHIS2 Integration

Smap and DHIS2 complement each other, together they provide a complete solution. Smap provides the operational layer, where individual work gets assigned, completed and closed in the field, often offline and on mobile devices. DHIS2 provides the information layer, where results are aggregated, indicators are computed and programs are managed nationally. Running both, with data flowing automatically from one to the other, is better than forcing either to do the other’s job.

Connecting to a DHIS2 server

A connection can be set up between a Smap organisation and a DHIS2 server. You will first need to create a user in DHIS2.

1. This user must not have two factor authentication enabled as DHIS2 refuses a personal access token that belongs to a 2FA account.
2. They must have “Data capture and maintenance” permission for organisation units that will be supported by the Smap instance
3. Give the user the “Data entry clerk” user role, or any role granting the Add/Update Data Value authority (F_DATAVALUE_ADD).
4. In the DHIS2 maintenance app, open the data set you need to update, edit it and select “Sharing Settings”. Add the user and specify that they can “Capture and View” data.
5. Get the user’s access token, allowing both GET and POST. Copy it when it is shown, as DHIS2 only displays it once.

You can then specify the connection in the DHIS2 tab on the Smap settings page. Paste the personal access token into the settings.

Lookup DHIS2 Reference Data

The Smap shared resources page has a tab to add DHIS2 reference data. Click on the “Add’ button. In the dialog you can select one of two reference types; “Organisation units” or “option set”. Organisation units will create a CSV file containing the organisation hierarchy from your DHIS2 server (you can specify a filter if not all organisations are relevant). Alternatively if you select “Option set” you can select the specific option list that you want to add as a CSV file. The resource name that you specify will be used in your search and pulldata functions to access the reference data.

The tab will then display the added dhis2 resource data sets.

The data will be refreshed on a daily basis, or you can manually press the refresh button for the resource. You can click on the list button to see a preview of the resource data.

Creating Data Items in a Survey to send to DHIS2

If you create a case management system for a clinic your focus will be on using this to manage individual patients. You don’t want to spend time entering data as required by DHIS2. Hence most commonly you would add these data elements as calculations. For example you might record patient age and whether they have been diagnosed with malaria. But the DHIS2 data set might want a count of children under 5 with malaria, total patients under the age of 5, people over 5 diagnosed with malaria and total patients over the age of 5. So 2 questions in your patient administration system need to be converted into 4 data values for DHIS2. You can do this with calculations:

under 5 patients:  if(${age_years} < 5, 1, 0)
over 5 patients: if(${age_years} >= 5, 1, 0)
under 5 with malaria:  if(${age_years} < 5 and ${malaria_confirmed} = 'yes', 1, 0)
over 5 with malaria: if(${age_years} >= 5 and ${malaria_confirmed} = 'yes', 1, 0)

The above calculations set the value to 1 or 0. This would be the most common approach as then each record will increment DHIS2 aggregate values by 1 or leave them unchanged.

Mapping Smap Data to DHIS2

You will need to set up a mapping between your questions and the DHIS2 elements in a dataset. This is set at the “Bundle” level. In the online editor select “Bundle and Case Management” from the Tools menu, then select the DHIS2 tab. The bundle can update multiple DHIS2 data sets and these will be listed in the tab. Select “Add” to add the mapping to a new dataset. Each dataset will have multiple data items that can be updated. Click on the “Add Row” button to add a mapping between a question and one of these data items.

Sending Smap Data to DHIS2

There are 3 ways to send data to DHIS2

1. Click on the “Send to DHIS2” button in Mapping Dialog shown above
2. Rely on automatic synchronisation at the refresh interval that you specify
3. Create a notification to send the data when the case is updated

It is recommended that you use both the notification and the automatic refresh. The notification will keep DHIS2 up to date in real time as the data changes, the automatic refresh will fix any dropped synchronisations due to network errors etc

Workflow Enhancements

Highlighting Workflow Items by Process

Previously this could be done by selecting the menu “Highlight” and then “Bundle”, however it has been improved in the following ways:

• The label “Bundle” has been replaced with “Process” as surveys grouped into a single Bundle can also be considered as collaborating on a process
• The highlighting is clearer with the whole element changing colour
• The menu option “Reset” orders the layout of the workflow elements vertically by the process that they are in

Viewing Who Does What

There is now a menu on the workflow page labelled “View”. If you select that you can choose “Who Does What” and see a table of which user or role is assigned to each stage of a process. This also shows an alert if there are stages of a process that do not have anyone assigned.

Other

• CSV Files are zipped before being downloaded to FieldTask. Previously dynamic compression was used but this was inefficient, especially for CSV files not affected by RBAC. (Requires FieldTask 5.4+)
• CSV File downloads are restarted after an error at the point at which they stopped which should improve connectivity when the connection is poor. (Requires FieldTask 5.4+)
• Values entered into the “search” control in the console now persist until removed or reset. “Search” is the control that filters records matching text.
• Downgraded iText from version 8 to version 5.4. This was due to multiple problems introduced by the later version of iText.
• Check for SMTP email servers responding with “limit reached” and put the email back in the queue. This email send is then not retried for 60 minutes. This reduces the load on email proxy servers.
• Refuse to delete SharePoint and DHIS2 shared resources if they are still in use by surveys
• Improve the efficiency and reliability of automatic synchronisation between a server and FieldTask
• Reduce the TTL for the Apache web server so that idle pool connections are discarded before they time out. This reduces the number of times a client has to re-request data after a network error.

New Functionality

Download Custom/Offline Maps from Shared Resources

Custom maps can show information such as boundaries, or geo infrastructure, that is not available in the normal base maps. These maps can also be downloaded to FieldTask allowing them to be used offline, hence you will usually see them referred to as “Offline Maps”.

Before this release offline maps could be manually added in FieldTask from the “maps” settings page. Now you can enable the management of maps from the server for your organisation. The user will still be able to add offline maps manually.

Managing Offline Maps in Shared Resource

Offline maps can be uploaded In shared resources and then tagged with the projects that will access them. You can also delete offline maps that are no longer needed.

Downloading the Server Managed Offline Maps to a device

The next time the user presses “refresh” on their device any missing offline maps will be downloaded from the server. This happens automatically if the user is on a Wi-Fi network. If the download is interrupted then next time the Wi-Fi is available, the download will continue from where it left off.

If the user is on a metered network then they will need to confirm that they want to download the map as these can be up to 500MB in size. To do this they select the menu “Settings” then “Maps” then “Layers” from the server. They will then be asked if they want to proceed with downloading the map.

Once the map is downloaded it will be displayed over the top of the base map that you selected.

Creating offline maps

Raster maps can be created in QGIS. If you are just creating an offline version of an existing maps it is simple to do. If you want to create a map from a photograph then you will need to georeference it first which will require some additional steps. Generally raster offline maps are more useful. Vector maps can be created but they don’t currently have any syling information.

Two Factor Authentication

A user can add two factor authentication to protect their account. Use the menu under user profile to open a dialog that includes a QR code. This QR code can be scanned using any standard authenticator to add 2FA.

Multiple Server Keys

A new API keys dialog allows you to create multiple named keys and specify the expiry of each. Keys can be revoked from the dialog and the date and time the key was last used is also now shown.

Other

• Added a report that lists users who have enterprise or organisational level permissions on the server. This report is available from the “Local Reports” menu on the form management page and requires the user to have at least organisational administration permissions to run it. Note this is new functionality but also a bug fix as this menu item previously existed but ran the same report as the “Organisational Structure” menu item.
• Performance Improvements. Replaced 4 database calls to get user information with 1. These were called at the beginning of every web service request so this change will reduce CPU and database resource usage. This change also adds caching for JDBC data source lookups.
• Deploy shared libraries to the tomcat shared class loader. This reduced the size of the Smap Server tar file by about 120MB after it had grown due to the larger libraries included in AWS SDK 2 and iText 8.
• Fixed a bug whereby the geo-point button was unresponsive in WebForms when it was run on a browser on a mobile device with a touch screen.
• Fixed a bug in which the open camera button was missing for an image question in WebForms run on a browser on a mobile device with a touch screen.
• Fixed a UX bug on the subscriptions page where after a successful subscription the button was still enabled to re-request that subscription.
• Fixed a bug in FieldTask where a long press on the maps tab would always select the nearest task even if the location of that task was not on the screen. The task has to be under the long press now.
• Performance improvements in Monitoring when the number of submissions is high (25 Million+).

New Functionality

Filter and Max Record Setting for CSV files generated from a Survey

In the online editor, while editing a survey that uses pulldata or search to get reference data from another CSV, you can select the menu File and then “Reference Filters” to set a maximum number of records or an arbitrary filter on the data.

Other

• Embedding images in spreadsheet reports did not always work if images were stored in AWS S3. This is now fixed.
• Fixed an issue with updating a field placeholder in a PDF template created by Libre Office.
• Fixed an issue with using exif tool to preserve exif data in thumbnails
• Upgraded AWS SDK from 1 to version 2
• Upgraded iText PDF generation library from 5.4 to version 8.05
• Fixed an Out of Memory era triggered by audit review on select questions with relevance.
• Made user selects searchable in the notifications dialog
• Added details to the log entries that report changes to notifications

New Features

New Operations Monitor 

A brand-new operations monitor gives supervisors an at-a-glance overview of work across the organisation including; open cases, at-risk records and alerts. You can drill down by unit or role, view individual cases, and open a dedicated case viewer.  This page is in the same module as the console. The Cases tab on the monitor page has been moved to the new operations page

New Pulldata / Lookup Function Builder

A new point-and-click tool in the online editor lets you build pulldata (lookup) expressions without writing them by hand. You can choose the CSV or Forms source, you can lookup unique values or repeating values. This feature complements the existing search function builder to make adding reference data to a survey easier.

The pulldata and lookup functions are set in the calculation of a question, so you will find there a button that will open the builder.

The button will only be enabled if there is currently no pulldata/lookup function in the calculation or there is only a pulldata/lookup function. So you can’t use this builder to edit expressions that combine pulldata functions with other elements.

Reference Records

A new reference type which is effectively a read only case has been created. This gives multiple users read-only access to records.  

FieldTask will show reference records in the task tab. If a reference record exists then there will be a select control to choose Tasks or References. If someone has been assigned to a record then that will be shown in the list. The user can select the reference to view its contents or swipe horizontally to remove it from their phone.

References are particularly useful when your people are mainly using mobile phones to access data offline. The references will be copied into FieldTask when the user refreshes and can then be reviewed and deleted. You can set up a notification to automatically create references from submissions. References can also be added and removed from the console.

You will need version 5.2.02 or later of FieldTask to see references.

More information is available in the documentation.

Operations Summary Email 

Periodic notifications can now send an operations summary digest by email, so managers receive a regular roundup without logging in. This complements the existing capability where a report on submissions can be sent periodically. To create this periodic report add a notification on the notifications or workflow page, set the trigger to “periodic” and select an operations report as the report.

Relative Date Ranges on Dashboards

The dashboard evolved to show the state of the system using all submitted data. It allowed you to add a fixed date range to individual panels but this could be laborious if you wanted the same date range applied to the entire dashboard.

To make the dashboard more useful for managing daily operations we have now added relative date ranges; last day, last week, last month or all. They apply to the entire dashboard. You can still add specific date ranges to a panel which will override the dashboard settings. Hence you could have a panel showing data from a specific week in the past to compare with the rest of the dashboard showing data from say, the last week.

Manage Dashboards on Behalf of Users

Each dashboard can be customised by the user. However sometimes it makes more sense for an administrator to set up a dashboard for somebody. This can now be done. Administrators will see a drop down list in the menu bar that allows them to select the user on whose behalf they will be acting.

Other

• Fixed an error that may be reported when logging in after a browser window is left connected for an extended period.
• Workflows now support a scheduled (periodic) start step that can send a report email on a recurring basis. These periodic items can be edited later to change their schedule, label or report.
• Workflow items for new forms are now positioned to avoid overlapping
• Workflow arrows no longer disappear when scrolling beyond the initial view
• Arrow-key navigation added to the WebForm form index panel
• The expanded notification dialog in WebForms can now be moved

New Features

GDPR Compliance Toolkit

A suite of privacy management tools has been added to help organisations meet GDPR obligations.

Individual questions can be marked as containing personally identifiable information (PII). The three settings are:

• Not Personal data (The default)
• Personal data (Will be checked during DSAR and RTBF requests)
• Anonymise on Submit (The contents will be anonymised in FieldTask or WebForms before being sent to the server)

Anonymised data is encrypted using SHA256, it can still be used for analysis as the encryption is consistent and the same input will create the same output, however the original value is not available. This is ideal for data such as identity card numbers.

A new Data Protection Officer (DPO) security group gives designated staff access to a privacy administration page. Only a user with “Security” or “Organisational Manager” privileges can give a user the DPO security group. This should not be given lightly as RTBF requests can result in the loss of data.

Data Subject Access Requests (DSAR) can now be exported into a spreadsheet from the privacy page. Data for questions that have their PII value set to “Personal Data” are searched. Searches include history and deleted rows as well as live records.

A Right to Be Forgotten (RTBF) form lets the DPO search for a subject’s data in questions that are defined as PII, review it, and redact any data found.

The final step “Redact Selected” will erase the data. This step is disabled by default unless “Enable Redactions” is set to yes in the organisation settings.

SharePoint Integration

Smap now integrates with SharePoint lists. This includes:

1. inserting or updating data in SharePoint lists when a new record is added or updated in Smap
2. Using SharePoint lists to provide choices for select questions.
3. Looking up reference data using the pulldata() function to provide guidance to the user or to manage the questions asked in the Smap form.

The image below shows configuration of a Workflow Item to insert an entry in a SharePoint list. This includes the mapping of survey questions to SharePoint columns.

Workflow Page

A Workflow page has been added. This is an umbrella for both tasks and notifications. Using the Workflow page you can create and manage tasks and notifications from a single place. If you have already set up tasks or notifications they will be displayed diagrammatically on the Workflow page, showing the connections between each triggering event and the task or notification it produces.

Notification Emails with Attachments

Email notifications sent from webforms and the workflow page now support file attachments. Attachments are saved to the record’s timeline so they appear in the history tab. Organisations can disable outbound webform notifications at the organisation level if needed.

WebForm Index

WebForms now has a collapsable index sidebar for navigating long forms. You can enabled WebForm indexes in the settings for a survey. They are particularly useful with cases where you are not working your way systematically through the form, instead you might open a case, quickly jump to a section you need to update and then submit the change.

Record History: Email Replies

Replies to emails, including attachments, sent from WebForms are now associated with the record and can be viewed in the History page in the console. This is currently only available if AWS SES is configured as the email server.

Record History: Attachments and Emails

The history tab on the console now shows attachments and inbound email responses alongside the submission timeline. History is now the default tab when viewing a record.

Ubuntu 26.04 and Tomcat 10 Support

The server-side stack has been upgraded to support Ubuntu 26.04 and Tomcat 10, including the migration from javax to jakarta namespaces and updated deploy scripts.

Other

• Console record filter “reset defaults” fixed, previously a user could not reset filters using the reset button in some circumstances
• Fix WebForm crash when a survey had multiple autocomplete selects in a repeat group
• IE11 support removed from WebForm
• Reduced the size of the WebForm client resulting in faster loading. This involved replacing node-forge with native hashes for digest calculation and downloading barcode libraries only if they are needed.

New Features

Dashboard updated

The dashboard now uses Bootstrap, like the rest of the system providing a more consistent user interface.

This was quite a difficult change to make due to the complexity of the code and how older user interface tools were fully embedded in it. I tried to do the update once before and gave up. However this time AI coding tools made the update possible.

You will notice that the Smap dashboard looks a lot nicer than it did before, however it still has the same function which is to provide a free and quick check on emerging issues in your surveys and to provide an operational overview where you can assess the state of your organisation’s processes at a glance.

As before it does not provide drill down, or in depth analysis capabilities. If you discover issues using the dashboard you can turn to your AI tools, a dedicated (and more expensive) dashboard, or analysis tools like R or spreadsheets to fully understand the situation.

Code is smaller and faster to load

Previously we used the older AMD approach to managing code modules. This was getting harder to maintain which resulted in code that was not always optimised.

Other Changes

1. Upgraded the entire client from Bootstrap 4 to Bootstrap 5 (not just the dashboard)
2. When you create a table view panel in the dashboard you can unselect the option “Show Meta Data” to hide the meta data. This allows you to focus on the answers to questions without having to scroll past start and end times etc.
3. Pie charts are available as a graph type in the dashboard now. Along with the previously supported histograms and line graphs for time-based data
4. Pie charts support “Group By”. A separate pie chart will be created in the panel for each distinct group by value
5. Accept / Reject tasks. You can now accept or reject a task from the WebForms launch page
6. Self assigned tasks are now shown in a different style in the WebForms launch page
7. Reliability fixes for sending campaign emails
8. WebP image support
9. Bundled Survey Names. Survey bundles now use the bundle name configured in the case management settings when displaying surveys to users
10. Highly visible “Reset Filters” option in the console. If you add filtering to your console data the “Reset Filters” menu will be shown in red and bolded. Just to remind you that you are filtering your data in case you wonder where your records went.
11. “Language” no longer shown as an option when exporting to CSV files. Language is only used on spreadsheet exports to add the labels to the data headers.
12. The project select and add panel controls are included in the menu bar in the dashboard. This leaves more room to show the dashboard panels.
13. Improvements in efficiency of the background processes that load submissions into the database and generate tasks and notifications. This will reduce the load on the database server and memory usage.
14. Include the availability of self assigned tasks in the tasks count shown for each user
15. Add a “hideinwebform” appearance so that questions can be hidden in WebForms but visible in FieldTask

New Features

Create histograms of text values including in calculate questions

When creating a histogram graph in the Analysis module you can select a text, calculate or server calculate question to display the frequency analysis. These can also not be used as “group” questions.

In the example above the age category is calculed from an integer age using:

if(${age} < 18, 'youth', if(${age} < 60, 'adult', 'elder'))

In the analysis module I then created a graph that showed the sex grouped by the calculated age category.

Added a bundle access report

This is an administration report that can be found on the survey management page under the menu “Local Reports”. It will show survey name, project name, survey type and which users can access the survey for each survey in a bundle.

Other

• Support has been added for the “no-collapse” appearance in the online editor. This appearance can be used to prevent groups from being collapsed to hide questions in WebForms.

New Features

Last Saved Support in WebForms

Last saved has been a feature in FieldTask for sometime, however it has just been added to WebForms in this release. Now, if you are using WebForms to enter new records, after pressing submit, any questions set to get their defalt value from the last submitted record will be populated. Details are here: https://www.smap.com.au/docs/xls-default-values.html#referring-to-the-last-saved-value

Form Access Report shows Row Filters

This report can be found on the admin page under the “Local Reports” menu. It shows for each user in the current project which forms they can access and why they can access it or not access it. It now also includes the row filter, created from the combination of roles that they have, that wll be applied when they view the data.

Greater Flexibility to Combine Roles into Row Filters

Another change to row filters is that they can now be combined with “and”. This discussion gets a bit boolean and complex but essentially the default behaviour is that when you give a user additional roles they get to see more data. So a user might have the roles to access Northern Region and Southern Region which means they can see data from both. You can now add roles that combine with “and” so that they reduce acccess. Hence the user might also have a role “nonsensitive” which means they can see only the nonsensitive data from Northern and Southern region. More details here: https://www.smap.com.au/docs/xls-default-values.html#referring-to-the-last-saved-value

Notifications Sent Once per Record

Each notification is now only sent once for each record. For example if you have two bundle level notifications “New Complaint” and “Set Critical” on a bundle with two surveys A and B you may see the following:

• Survey A submits new record. “New Complaint” is Triggered
• Survey B is used to update the record to change the assigned person. No notifications are triggered.
• Survey B is used to update the record to set it to “critical”. “Set Critical” is triggered.

Previously all of the above events would have triggered the “New Complaint” notification.

Add Barcode Reader to WebForms

I had a request from iOS users to be able to read barcodes using their apple devices. They are using WebForms on iOS devices. Hence support for the barcode question type in WebForms has been added.

Other Changes

• Set a limit of 30MB on uploaded file size
• Allow roles to be added to read only surveys