Create histograms of text values including in calculate questions
When creating a histogram graph in the Analysis module you can select a text, calculate or server calculate question to display the frequency analysis. These can also not be used as “group” questions.
A select question grouped by a calculate question
In the example above the age category is calculed from an integer age using:
In the analysis module I then created a graph that showed the sex grouped by the calculated age category.
Added a bundle access report
This is an administration report that can be found on the survey management page under the menu “Local Reports”. It will show survey name, project name, survey type and which users can access the survey for each survey in a bundle.
Example bundle access report
Other
Support has been added for the “no-collapse” appearance in the online editor. This appearance can be used to prevent groups from being collapsed to hide questions in WebForms.
Last saved has been a feature in FieldTask for sometime, however it has just been added to WebForms in this release. Now, if you are using WebForms to enter new records, after pressing submit, any questions set to get their defalt value from the last submitted record will be populated. Details are here: https://www.smap.com.au/docs/xls-default-values.html#referring-to-the-last-saved-value
Form Access Report shows Row Filters
This report can be found on the admin page under the “Local Reports” menu. It shows for each user in the current project which forms they can access and why they can access it or not access it. It now also includes the row filter, created from the combination of roles that they have, that wll be applied when they view the data.
Example Form Access Report
Greater Flexibility to Combine Roles into Row Filters
Another change to row filters is that they can now be combined with “and”. This discussion gets a bit boolean and complex but essentially the default behaviour is that when you give a user additional roles they get to see more data. So a user might have the roles to access Northern Region and Southern Region which means they can see data from both. You can now add roles that combine with “and” so that they reduce acccess. Hence the user might also have a role “nonsensitive” which means they can see only the nonsensitive data from Northern and Southern region. More details here: https://www.smap.com.au/docs/xls-default-values.html#referring-to-the-last-saved-value
Notifications Sent Once per Record
Each notification is now only sent once for each record. For example if you have two bundle level notifications “New Complaint” and “Set Critical” on a bundle with two surveys A and B you may see the following:
Survey A submits new record. “New Complaint” is Triggered
Survey B is used to update the record to change the assigned person. No notifications are triggered.
Survey B is used to update the record to set it to “critical”. “Set Critical” is triggered.
Previously all of the above events would have triggered the “New Complaint” notification.
Add Barcode Reader to WebForms
I had a request from iOS users to be able to read barcodes using their apple devices. They are using WebForms on iOS devices. Hence support for the barcode question type in WebForms has been added.
Reading a barcode (The pen is to block the reading so that I can take a picture)
Hi everyone. Welcome to the April 2025 release of the Smap Server. This release incorporates new features and fixes implemented for World Vision’s Accountability project and the Bougainville Police Electronic Occurrence Book.
Loloho Beach in Bougainville
New Features
Consistent lookup for images
Looking up an image, or an audio file, from a remote server to use in your form now works the same way for both WebForms and FieldTask. Hence you no longer need two separate forms for this. Example Survey Form.
Selecting an image from a server for editing in a form
Other Changes
Fix an issue with send email tasks and automatic notification when case is assigned. Previously these emails were not being sent.
Remove the dashboard security group. This security group is no longer used.
Trim values when loading an XLSX file as a CSV shared resource. It was possible to load values that had a trailing space which could cause problems when comparing what you thought was the value with another text value.
Fix issue with keys that use sequencing, not being correctly generated when their is pre-existing data. For example keys may use the seq() function to generate a unique number with each submission but if there were existing submissions when the key was added, each record would get the same sequence number.
Fix issue with question widths in a grid being set to 1 when opened in the online editor when they were previously set to 10, 11, or 12
Allow selection of a record in the console by clicking on the primary key as well as all of the other displayed columns
Debounce the Survey Save button in the online editor. this was previously debounced for the network request but not for the lengthy validation that happens before the web service is called
Disable the showing of the organisation name in the logon page. This caused confusion when users had access to multiple organisations and they were intending to logon to a different one than the one the used last time.
Update highlighting of errors in the online editor. Some errors were not being highlighted in red.
Columns can be reordered in the console
The order of console columns and the page length is saved to the server for each user
Quarterly reports can now be generated automatically each quarter as well as for day, week, month and year
Show the full server version on the home page including the minor version. For example 25.04.02
Fix issues with using token authentication in fieldTask with tasks. The tasks were not being downloaded if you had logged on with a token.
This release features more case management functionality ! It is designed to make creating new case management systems easier and quicker as well as to provide better indications to users when they have tasks to complete.
A count of tasks and cases is shown on the menu bar
The menu on the survey management page with the cases/task count next to the profile option
If you click on the profile menu you will see a menu option “My Tasks” which will take you to a page where you can complete your tasks in WebForms.
Notifications can be applied to a bundle of Surveys
When specifying the trigger for a notification you can now specify “Any survey in the bundle”. This will simplify management ot notifications when you have a large number of similar surveys for collecting data.
Roles can be synchronised to a bundle of Surveys
When you are editing the roles that apply to a survey you will now see a checkbox labelled “Apply these roles to all surveys in the bundle”.
Bundle Checkbox on Survey Roles page
If you select this checkbox then the settings for each role that is enabled in the current survey will be applied to all other surveys in the bundle. These settings include:
Row filters
Column filters
So in the above image the row filters and column filters in the CID role for the EOB survey will be applied to all other surveys in the bundle that also have the CID role. Furthermore if you subsequently change the row filters or column filters in any survey then other surveys that have that role enabled will be updated.
WebForms are now used when editing a record in the Console
The oversight form is now opened in a WebForm which allows you to use relevance, cascading selects et al when editing records. If you are annotating a question using the “source” parameter you can still do that on the View page.
Performance Improvements
Getting a list of tasks and cases is quite a slow operation and most users don’t have them tasks. Now the system tracks who has cases/tasks and when you refresh a device it only attempts to get the details for those users that have them.
Cache time zone requests. These requests to get available time zones are slow and the data will change rarely. Hence they are now cached.
New indexes on the database.
Other Changes
Fingerprint matching has been disabled. The algorithm was not accurate.
Added rounding of survey duration values
When disposing of a case, rather than a task, in fieldTask and in WebForms, the button labelled “Reject” changes to a label of “Release” .
Create cases on receipt of an SMS or WhatsApp Message
SMS or WhatsApp numbers can be registered with Vonage. Then messages sent to these numbers will create a new case. Subsequent messages from the same number will be added to the existing case creating a conversation. Once the case has been closed the next message will open a new case.
This features supports greater access by remote communities to services. For example a beneficiary of an aid program could submit a complaint via SMS without needing to see a representative of the Aid program in person. In a justice scenario a member of a remote community could submit a crime report via WhatsApp or SMS. The police could then respond to that report and ask follow up questions using the same channel. The case can then be closed or progressed through to the courts. For both of these scenarios, at every stage of the complaint process the community member can be kept informed and asked follow up questions.
Record conversations with third parties within a case
Conversations over SMS and WhatsApp with the person who opened a case using a message or anyone else involved in the case can be recorded inside a “conversation” question. There is also partial support for including email conversations which will be expanded on in the next release.
Conversation within a case – Note the green text bubbles indicate that the channel is WhatsApp
Other
Performance improvements to the processing of submissions. Multiple processors can now be started in parallel to apply updates to the database. This can deliver performance improvements on high volume servers.
Allow fieldTask to use tokens, scanned by a QR code, in order to logon to the server.
Added support for AWS SES email services. These are accessed using the AWS API and AWS credentials rather than SMTP.
Send email tasks at the time they are scheduled instead of immediately.
Fixed issues with login timing out just before an API request to the backend. These timeouts have to be explicitly handled whereas requests for a new HTML page are handled automatically be the session manager.
Removed blank choices from choice lists in oversight forms.
Removed feedback that a valid email address has been found when performing a password reset.
Added authorisation check to presentation of API page.
Removed compressed files from the list of valid shared resource files.
Converted export to spreadsheet requests from the console to use multipart mime formats. The previous form encoded approach was causing problems with Web Application Firewalls when there was a lot of data.
Changes in this release are almost entirely related to security.
New Features
Form/Session based authentication for the User Interface
This means you will no longer see a pop-up dialog when you logon. Instead you will be taken to a logon page.
Token based authentication for the API
A new version of the API has been created that uses token based authentication. This has only been implemented for the data API. Hence you can use the new URL /api/v2/data with security tokens.
A token can be obtained for a user by clicking on the user profile button and selecting “API Key”.
Creating a key
The key provides the same level of access to the server as the user who created it.
Example of using tokens with Power BI
In Power BI desktop select “Get data from another source”. Then click on “Other”
Power BI Step 1
2. Select “Web” and then click on “Connect”
Power BI Step 2
3. Get the version 2 API URL for the survey whose data you want to use. This can be found by selecting the API module then selecting version 2 and the survey you want.
4. Select “Advanced” in the Power BI dialog and paste the API URL into the first of the “URL parts”. Then get your API key by logging on to the server, selecting the user profile icon on the main menu and then selecting “API Key”. Add a header called x-api-key and paste the key as the value of this header. Then press “OK”.
Power BI Step 4
5. Then click on connect and you should see your data.
Support for Ubuntu 24.04
If you are installing your own server you should now use the latest 24.04 LTS version of Ubuntu to maximise the length of support that you will get.
Formatting of Dates and Times in Server calculations
Say you want to create a PDF with the format of a date or date/time looking like “22/05/2024”. If you just include your date in the PDF it will show as “2024-05-22”. Now you can create a server calculation using the “to_char” function:
to_char(${date_question},'DD/MM/YYYY')
Include this new server calculation in your PDF and set the date question to hidden.
Other Changes
Session timeout after 10 minutes of inactivity.
The service worker has been removed. This means you can no longer install WebForms as an app in a browser. This may be added back in a later release.
Logout now works properly in Firefox
DTD processing is disabled when parsing XML files
Error messages now do not include the version of the software that generated the error
This release includes 3 user requested changes. Hopefully they will prove useful to others
New Features
Notifications on Server Calculation Changes
Server Calculations can change anytime, for example if the calculation depends on the current date. Previously it was not possible to trigger a notification when the calculated value changed but now it is.
In notifications select a trigger of “Server Calculation”. You will also need to:
Specify the survey containing the server calculation
The Server Calculation question
The value that will trigger the notification. Note this value should be text and quotes are not required.
Adding a Server Calculation Notification
Web Form History
The history of submissions from WebForms is now stored in your browser. You can access this history from a menu on the server WebForms page or by opening the draw at the left of a WebForm and pressing the history button.
Open the drawer to view the history button while filling in a WebFormExample WebForm History
The history page shows the surveys that have been completed and submitted over the last 100 days. The action can be Submitted, Saved or Deleted. The instanceId is useful for connecting a Submission entry in the History to a record in the Console of the server.
Note the webform history is not stored on the server, so if you are reconciling somebody’s work you will have to ask them to talk through what they see in their history.
Download of shared CSV files in the format they were uploaded
Downloading a reference file from the shared resource page will now download the file that was last uploaded instead of downloading the data always in CSV format. This may be a spreadsheet in XLSX format rather than CSV format. The server will have converted the XLSX file to CSV for use in FieldTask but you don’t need to know that when you want to edit the reference data.
This avoids problems with the formatting of data loaded from a CSV file into a spreadsheet. For example I was working on a reference file that included assessments of distance vision such as 6/5 and 6/9. Every time I opened the CSV file, downloaded from the server, these values would be converted to dates. You can address this type of problem by working in Excel’s XLSX format and setting the format of the cells to text. You can then upload the XLSX file to the server so it can be used as a reference file and this XLSX file will be the one downloaded when you download the reference file for editing.
Other Changes
An integer representing the identifier of an email is now automatically added to the subject line of emails. This addresses a problem where email clients such as gmail do not effectively differentiate between multiple emails with the same subject. This can cause problems if you do multiple password resets in succession, you may end up clicking on the old link to reset your password.
This release contains a simple little change to allow WebForms to launch another survey to update an existing record. For example you can have a survey that searches for a patient record and then launches a survey to edit that record. This feature only works online but will make it easier to implement fully functional systems without requiring any coding.
I am considering extending this capability to FieldTask however at the moment I am not sure whether the functionality in FieldTask should be offline, online or both so I will defer it till another release.
New Features
Launching a new survey from a Survey now available in WebForms
When launching a survey form you can include a parameter with the instance id of the record to be updated. This will cause the launched survey to be pre-populated with the data from the record you want to update.
To do this create a survey that will look up records.
Add a way of selecting the instance id of the record in another survey. This could use pulldata or the search() appearance in a select one question. Using search() is supported by the online survey editor.
Add a “Child Form” or “Parent Form” question type. If you are not sure which one to use then either will be fine.
Add a parameter that identifies the instanceid to use. For example “instance=${get_instance}”
The following video shows this functionality.
You can pass initial data to the started surveys that override the value in the instance.
“Restore Data” from Archive moved to background
The process that restores a table of data for a survey using the original submissions has always run in the background. However on larger installations the original submissions may be backed up to low cost storage. Copying these back to the server can take some time causing timeouts and reports of errors. However this retrieval of submissions from archive is now also done in the background. It is treated as a background report so an entry will be shown in the reports page with the result of the request.
Result of Background Request to Restore Data
In summary you may notice the following changes when requesting a restoration of data:
No more error messages!
There may be a lengthy delay of several minutes before the submissions start being re-applied
You can check on the progress of the request in the reports module, once this shows the request is complete you can follow progress of the re-application of the submissions using the monitoring page.
Other
Allow a super user to reapply failed uploads from the monitoring page for a survey that requires a security role even when they do not have that role. A super user should not be constrained by RBAC.
Instead of entering a value of “yes” or “true” in the “required” column of the XLSForm, to indicate that a question is required, you can specify an expression. The question will then only be set required if that expression evaluates to true.
type
name
label
required
required_message
select_one yes_no
required_question
Should the next question be mandatory?
text
situation
Describe the situation
${required_question} = ‘yes’
You must answer
Making a question required during a survey
This feature is available for both FieldTask and WebForms.
You can also set the required expression in the online editor. First click the button to make the question required and then enter the expression.
Dynamic Read Only Questions
Like dynamic required questions, you can also make a question be read only or editable depending on an expression. For example you may pre-fill address questions with existing values and make these read-only but then use a question called “changed” to make them editable.
type
name
label
readonly
default
select_one yes_no
changed
Changed?
text
address
Address
not(${changed} = ‘yes’)
Fleet Street
Making an address question read-only if the address has not changed
Note: Unlike the conditional required feature this is not available for WebForms.
Other
Dynamic Defaults inside of a repeat. Using both FieldTask and WebForms you can now set a default value for a question inside a repeat that is calculated from previous repeat instances. Documentation is here.
Performance Improvements. Hopefully you will find accessing the online editor and the console faster.
Updated some infrastructure libraries to the latest version including Jersey and the PostgreSQL JDBC driver.
Previously case management alerts were generated if a case was not closed and it was older than the specified interval. You can now also add a filter so that the alert can conditionally be created depending on it’s data. For example you could create an alert. “Serious issue is Late”, with a filter ${criticality} = ‘serious’.
Note: you can refer to any data in the case using the ${} syntax.
Extendable Initialised Repeats
Previously, when you added a repeat count to a sub form, the number of repeats was set by the value of that count and the person filling in the form would not be asked if they wanted to add another record. This is probably what you want if the repeat count is set inside the form or is fixed. However you may have the situation where you are filling out a sub form using initial data from another survey so the repeat count is set by the number of existing records but you still want to allow the user to add additional records.
FieldTask version 6.814 is required to use this feature in FieldTask.
The specific user scenario where this requirement arose was for a list of suspects in a police case. We created a form that allowed the existing suspect information to be updated but we also wanted to allow additional suspects to be added.
This can now be done in WebForms and FieldTask by setting the appearance “extendable” on the “begin repeat”.
Rate Limiting APIs
An improved rate limiter has been added. This now limits the number of requests per minute, per organisation. By default the rate per minute is set to 0 which means no limit. However you can change this in the settings for your server. You need to have server owner privileges to do this.
If the rate is exceeded then an error message is returned with the request. Currently if you set an API rate limit it will apply to the following services:
api/v1/data
api/v1/data.csv
surveyKPI/items (Called when you view a table of data for a survey on the analysis page)
Automatic Sentiment Analysis of Text Responses
A sentiment can be automatically calculated for text responses entered in any of the following languages; English, German, Spanish, Italian, Portuguese, French, Japanese, Korean, Hindi, Arabic and Chinese. This is done using Amazon Web Services (AWS) Comprehend service. The sentiment can have one of the following values; Positive, Negative, Neutral or Mixed. A confidence value in the sentiment can also be generated.
To add the sentiment to a text question include the following parameters:
auto_annotate=true
sentiment=yes
from_lang={the language you are getting the sentiment from}
source={the question name that has the text – no curly brackets around the name}
Note the online editor assists you in setting most of the above. Although you will have to add sentiment=yes yourself as an “other” parameter. This is shown below.
Other Changes
Improved the performance of the shared resources page for sites with a lot of resources
Improved performance of submissions. Notifications and other events triggered by a submission are now processed in a separate queue so that the submission manager can get straight on to the next submission.
Server side validation to detect invalid characters in names for projects, users, roles and so on
Server side sanitation of html for labels to remove executable scripts now done when the text is entered as well as when it is used
Fixed issues with the script that does a fresh install of the server
Reduced the volume of log entries written by automatic scripts, now these will only write one entry per day, not every time they run
Allowed a notification that sends a periodic report to be disabled
Allowed dateTime type questions to be updated in an oversight form
