Improved Performance

Refreshing fieldTask will now be faster.  Note This requires fieldTask 5.33.  The changes are:

1. If you change an image or other media file that is used in a question or choice then that new media file will be downloaded automatically to the mobile device using push notifications.
2. A media file, including a CSV file, will now only be downloaded to a phone if it has changed.
3. Media and CSV files are only downloaded once if they are used by more than one form on the device.  In other words as long as the media file is coming from “Shared Resources” it is also shared between forms on the device.
4. If the media changes but the form has not changed then the form is not re-downloaded only the changed media files are.

You can expect substantial reductions in download time under the following circumstances:

• You have a form that uses many images in questions and choices but you are making changes to that form by adding new questions or changing the question text.  Now after each form change only the new form will be downloaded and not all the images.
• You have a csv file that is constantly being changed.  Now only the changed csv file will be downloaded and not the form and any other csv files or media.

However performance may be slightly worse if you have many forms that use media and only one is updated.  The reason being that during refresh the manifest will now be checked for all the forms that use media to see if the media file has been changed.  Previously the change in a media file would not have been picked up and the form would not have been updated on the device.  This is a quick operation and will only be noticed if the number of forms is quite large.

Other Features

Filtering on table view in dashboard

You can now use “advanced filtering” in these table views.  For example adding a filter such as

${district} = ‘capital’ and ${gender} = ‘female’

The number of records that match the filter as well as the total number of records will be shown just above the table of data.

Issues Fixed

• Issue 63.  Labels are not shown in PDF exports if the label is for a select question that uses an external CSV file to get its choices
• Issue 153.  When you download data and upload again in Analysis there is a warning message about “question name missing”
• Issue. The Legacy XLS export does not allow select multiples to be split into separate columns.
• Issue.  Replacing a survey that used uncompressed select multiples results in the select multiple questions being converted to compressed but the old data is not converted making it invisible.

Version 18.03 is here with lots of bug fixes and the following two new features.

Using SVG images to Select Choices

Selecting Choices by clicking on an SVG image in fieldTask

You can use this feature in both webforms and in fieldTask.   The image needs to be a Scaler Vector Graphic (SVG) which is drawn using mathematical functions rather than being composed of pixels like in a photograph.  This means that the image can be enlarged to any size without becoming fuzzy.  To use an SVG image to select choices you need to:

• Create your SVG file or get an existing one.
• When you click on a region in your SVG editor it should show the “id” and allow you to edit that.  Set the ids to the name of the choices in your select list.
• Attach the SVG file as an image on the select question
• Set the appearance of the select question to “image-map”.

Example

To try this out you can download the sample survey from here.  Also download the sample SVG file (which I sourced from Wikipedia) and load both of them onto your server.

Reports

Yes Finally a new reporting page.  This can be accessed from the Analysis module by clicking on the menu item “reports”.  On this new reports page you will find a link to the old page (in case you are still using that).

How to use

Add reports by clicking the “Add Report” menu item.  You can specify a name for the report as well as parameters include filters, date range, split select multiples and languages.  This will create an entry for the report in the table containing:

• The name of the survey
• A name that you have given the report
• An anonymous link.  You can send this link to anyone so that they can access the report.  No logon or account or password would be required.   This link does allow the recipient to change any aspect of the report.  Hence if you had set a data filter or a specific date range that cannot be changed.
• An button that has available 3 actions:
  • Generate.  Runs the report to download the data.
  • Edit.  Allows you to change any aspect of the report.  Someone who has received an anonymous link for the report would see these changes the next time they accessed that link.
  • Delete. Will delete the report.

Limitations

1. Only the new XLSX format report is supported.  Other types such as images, shape files will be added progressively and as requested.
2. Support for security roles in the anonymous link have not been added yet

Contact me if you need any of this missing functionality urgently.

New Form Management Page

The visual changes are minor however now when you click the “Upload Form” button it will use a new XLS Form uploader (see next section).  We no longer use the ODK pyxform loader.  Just in case this causes problems there is a link, under the upload button, back to the original page that still uses pyxform.  One reason you may want to use the old page is if you have to upload in XML file which the new upload function does not currently support.

There are 3 new columns in the forms list.

Group

The new XLS Form Uploader allows you to put forms into groups.  These will then share the same data tables.  Hence if you have a form with questions; name and age plus a second form in the same group with questions; name and gender.  Then the name answer from both forms will be put into the same table column and when you export results you will see data from both surveys in that column

Groups become really useful when you use tasks to progressively complete a survey.  So form one fills in some of the data then a user is assigned an instance of form 2 to get the rest of the data.  This instance can include the data collected by the first form.

You can also set a key policy of “merge” on surveys in a group so that records are updated rather than replaced.  This could allow you to create a records management system.

Replace

Yes you can now replace a form with another uploaded from an XLSForm template.  You can do this even after you have started collecting data.  If the form you are uploading does not have questions that were in the original form then any data collected for those questions will be hidden but not deleted.

Click the replace button next to the form to be replaced, specify the new template and click upload.

The only thing you should avoid changing is the “type” of a question.  This is only a problem if you have already collected data for the old type in which case future submissions will fail.  So if you need to change type, say from integer to text, then create a new question with a different name and delete the old question.  Alternatively if you are in a position to delete already collected data, perhaps it was test data, then you can do that before collecting new data.

Get Link

Click on this button to get a link to complete a webform that does not require a password.  You can copy this link and email it to someone who does not have an account on Smap and they can then fill in the web form.

If a form has a public link then the link button will be a darker shade of blue.  When you no longer want people without accounts to complete webforms then you should delete the links.

XLS Form Uploader

Choice Lists are Preserved

Previously if you had referred to a single choice list from multiple questions then after you uploaded that data this sharing of choices would all be lost.  If you then made changes to the form in the online editor and downloaded the resultant XLSForm you would have a mass of duplicate choices.  This no longer happens and you will get your original choice lists back.

Question Parameters

These have been added to odkCollect and fieldTask but previously could only be added to a survey using the online editor. Add them in the parameters column of the survey sheet.  Separate multiple parameters using a space. Examples include:

• rows=5                    Set the number of rows in a text question
• max-pixels=500     Set the maximum number of pixels on the longest edge of a photo to 500. The photo will be scaled accordingly.  This can be used to reduce the size of images.

And many others.  Check the Survey Edit Reference on the google help drive for a complete list.

Specify Security Roles

Roles you specify for a form will be created if they do not already exist.  If you add a role but do not specify columns or row filters then a person must have that role in order to be able to see the new form.

To add roles with column filters add a column  role::”role name” in the survey work sheet.  Specify any questions that should be visible to that role by putting “yes” in that role column for that question.

To add a role with a row filter add a column role::”role name” in the settings work sheet.  Specify the row filter underneath the role.

Other Uploader Changes

1. Simpler clearer error messages
2. Error messages are in multiple languages (well spanish at any rate, contributions for other languages are welcome)
3. Support for questions of type “chart”

Bug Fixes and other Changes

1. The api call that retrieved data in json format has been substantially optimised to reduce memory usage
2. Issue 188. Times in PDF exports are now set to local time.
3. The latest version of fieldTask is now included in the release so it can be downloaded from the web server rather than google play store.  The website will attempt to check if your Android version is greater than 4.0 and if so it will get the new fieldTask.  Otherwise it will get the old one.
4. Added a user security group of “view_data”.  People with this role will be able to view aggregated results like an analyst but will not be able to delete data or create new forms.

I’m still playing catch up on documenting the new features however its time to announce version 18.01.  Included is a summary report that helps with analysis of your collected data.

Summary Report

This is an excel file containing multiple worksheets including a worksheet for each question.  The data in these question worksheets will be formatted so that they summarise the responses to that question and can be be used directly to create charts. For Example the following chart was created directly from the data generated by the summary report.

How to find it

The Summary report is located on the browse results page which is in the web forms module.

• Login
• Select the menu modules then Webforms
• Select the menu Browse Results

You can then select the menu “Summary Report”.

Options

The following dialog allowing you to customise your report.

• Text Function.  This can be set to count or percent.  So if its “count” then the occurences of each different piece of text in an answer will be counted.  If you select “percent” and “Group By” then the percentage is calculated against the total count for each group value.  This was the setting used in the example chart shown above where the totals for male and female have been adjusted to each add up to 100.
• Numeric Function.  This can be set to Average, Min, Max or Sum.
• Group By.  A second question with which to group the results.  This will result in a two dimensional table as in the example above.

How to use it

The summary report is not intended to create your final report directly.  For example you may want the answers to some questions grouped by gender and the answers to other questions grouped by region.  Some results you may want as percentage, some as counts and some as both. Hence you would generate multiple summary reports using the above options and then copy and paste the tables, and charts you create from those tables, into your final report.

Filtering Data

Because the summary report is on the browse results page any filters you apply  will also be applied to the report.  Hence you may select that only data from a specific region where at least one text answer includes the word “drought” be included. Then generate the report.  The above filters will be shown on the “settings” tab of the summary report along with the numeric function, text function and group by that were selected.

Restrictions

Currently only data in the top level form is included.  If you have data in sub forms it won’t be included in the summary report.  This is because browse results only currently shows the top level form.  This restriction will be removed in a future release.

There are many more features in version 17.12 than are documented here.   Other release notes will follow.

Charts in FieldTask

Yes!  you can show charts inside the form on your tablet or phone.   You will need fieldTask version 5.23 as well as Smap Server 17.12 in order to use these.  This work was sponsored by InformEd International.

Here is an example form with basic charts.  If you load the form in the link then you should see the following charts among others:

Showing time series data from other forms

Using the pulldata() function you can get data from previously submitted forms.  We have now added a new pulldata() type that will retrieve data as a time series.   Hence if you are collecting data over time and you want to see a chart of that time series you would use this new function.

  pulldata(‘chart_xxxxx’, ‘data_column’, ‘key column’, key value)

The ‘chart_’  in the file name field tells the server to return data as a time series suitable for charting.

A detailed example form with comments can be found here.

Bugs Fixed

• Issue #156. Security roles page for a survey not being displayed
• Issue #157. Crash on export with language other than english
• Issue #158. Guidance not being added for tasks
• Issue #159. SMS URL deactivated
• Issue #160. Added a warning message on upload when a form references missing columns in a csv file

Time for November’s release!

The changes are

• Filtering of notifications and Exported Results
• Can now set default value for questions in the online editor.
• Can now specify the number of rows used by text questions in fieldTask
• Improvements to form monitoring (See below)
• Added option to do a full survey export without including its media.  Some surveys can have many gigabytes of images and other media submitted.  Just on the off change that you want to export the data and import it into a different survey but don’t need the images then you now have that option.
• Set a new notification to “enabled” by default.
• Show most recent data received first in table views in the analysis module
• More spanish translations

Filtering of Notifications and Exported Results

When creating a notification you can now specify a filter so that this notification is only sent when data is received that matched the filter.  You can also use the same sorts of filter when exporting data from the analysis module.

Examples:

• ${age} < 10      – Where the answer to the age question was less than 10
• now() – ${_upload_time} > {2_days}     – Export data uploaded more than 2 days ago
• ${name} not like ‘%om%’    – Any name that does not have the letters “om” somewhere within it

More information on specifying a filter can be found here: https://sg.smap.com.au/sqlFilterHelp.html

Improvements to Form Monitoring

Actually the main improvement is that this now works again!  The old approach relied on fieldTask notifying the server of the forms that the user had after they had done a “refresh”.  However this did not reflect changes when the user had downloaded a form using the ODK menus and without doing a fieldTask refresh.   Anyway recent releases of fieldTask no longer sent this notification.

The updated form download monitoring now records all form downloads whether as a refresh or when the form is selected explicitly.  One limitation of this new approach is that the device that did the download won’t be identified unless you are using version 5.14 of fieldTask.

Bug Fixes

• Error shown when a user registered a new organisation has been fixed.  This message did not stop the registration proceeding but would have been pretty disconcerting to new users
• Allow data to be imported when the question type is “range”

Its official!  the number of results submitted to servers managed by Smap has gone past 1 million.

It is fantastic to think that this tool has been useful to people and I hope it continues to be useful and to grow in usage.  Thankyou to everyone using Smap and everyone who has contributed over the years to making it all possible.

Two releases in one month!

The changes are:

1. Add ability to override fieldTask settings
2. More flexible task creation (Enhancement #118)
3. Add reviewing of task group settings
4. Add downloading of PDF Templates
5. Issue #126.  Dynamic choice names not shown with generated values in PDF exports
6. Issue #125.  Hyperlinks in PDF templates are all put on page 1.
7. Issue #82. User is permitted to try and create a task when there is no task group

The new features listed above are described in more detail below.

Overriding FieldTask Settings

There are a couple of changes to make this more useful

Firstly you no longer require organisational administrator privileges to set set fieldTask settings on the server.  Because of that they have been given their own tab in the user management page next to users and projects.

So now any administrator can specify the fieldTask settings but these changes still affect all phones that synchronise with the server.

Secondly, where there is a corresponding setting that you can specify on the phone then these will be overridden in a more consistent way.  This affects automatic synchronisation and the automatic deletion of instances from the phones after they are submitted.  For each of these you can now specify a value of “Set on phone”.  The phone user will then be able to configure these as they wish. Any other value will override the setting on the phone which will be disabled so the user cannot change it.

Hence for Automatically Synchronise the options are:

• Set on phone.
• Off.  The phone will not automatically synchronise.
• Wifi.  The phone will synchronise only over a wifi connection.
• Wifi and Cell.  The phone will synchronise over a cellular and wifi connection.

For deleting submitted results the options are:

• Set on phone
• Off.  Results are not deleted.
• On. Results are deleted.

For this to work as described above you will also need to be using fieldTask 5.11

More Flexible Task Creation

Previously you could create tasks from survey results and you could select which tasks to create by specifying a value for one question in that existing survey.   You can now do more sophisticated selection of tasks using the same approach that was added to notifications.

When creating a task group from existing results select the “Advanced Filter” option.  You can then specify the filter you need. For example in the following dialog I am organising a follow up survey for all results where the age was less than 25.

Reviewing Task Group Settings

You can now review the settings you used to create a task group.  In the task group menu select “Edit Task Group”.  The settings will be shown.  However as of 17.10 this feature is still in development.  Its main aim is to show the user what filters they used to create tasks from existing data.  The limitations are:

• You are not allowed to change any of the settings so at the moment it is really “View Task Group”
• Some of the settings, such as address parameters, are not shown

Downloading PDF Templates

If you have uploaded a PDF template for a form using the settings dialog then you can now download that template, also from the settings dialog.  To do this select the downward chevron next to the template name.

Filtering Notifications

When you add a notification, such as an email alert or forwarding of data to another server, you can now specify a filter so that only matching results generate the notification.

In the above example a PDF of the results will be emailed if the name contains the characters “smith” somewhere and the age is greater than 25.  You can get more details on the syntax of these filters by clicking on the (i) icon to the right.

Note that the name and the age has also been included in the body of the email along with the PDF for the results.  You can embed any question answer in either the subject or the content of the email.

Identifying Objects in Images

Artificial Intelligence has now been build into Smap!  In the parameters of an oversight form you can now add “auto=yes” to questions that refer to an image.  That image will then be processed by Amazon Recognition to identify objects automatically.  For an example of how to process an image in an oversight form refer to this article.

Bugs Fixed

• In the monitor there is a separate column for “Upload Errors”.  This just shows the number of errors that have been seen on phones or webforms when users attempted to upload results probably to a deleted or blocked survey.   Usually the administrator will fix the problem and then the user tries again and this time the data is sent.  You don’t need to worry about this number just make sure that all results end up being sent from all the phones successfully.
• Previously if you renamed a form part way through data collection you would see two rows of upload counts.  One for surveys uploaded to the old name and one for the new.  Now only the latest form name is shown along with the total of uploads for that form.

Translating and Transcribing

In this release we have improved support for translating answers to questions from one language to another.  For example from French to English.  You can use the same feature to facilitate transcribing to text from an audio recording or a video.

Specify the source parameter in an Oversight form

For example if you have an audio question called q2 and you want to write the translation into a column called audio_transcription, then you can write your oversight form as shown below.

Oversight Form

The parameter source=q2 has been added to the audio_transcription column.  Clicking on a record in managed forms then shows a page similar to the one below where the user can play the audio and write the translation.

You can also reference:

•  a text question in order to convert from one language to another.
• an image question to write commentary on an image
• and a video question in order to add commentary and a transcription.

Other Parameters

Parameters for oversight forms are new in version 17.09.  Other parameters are:

• rows=a number.   This sets the number of rows to show when entering data for a text column.   In the above example the number of rows were set to 3.  However you can click on the bottom right-hand corner of the text box to make it smaller or larger at any time.
• form_data=off  or form_data=on.   This is applied to a new row type called settings.  When set to off the form data for the selected record is not displayed which is useful when you are focussed on referencing specific questions.

Monitoring of Submitted Results

A new result type of “Upload Errors” has been added.  The background to this is that previously a change was made to the monitor to show errors that occur when the results of a survey are submitted and are reported back to the phone or webform.  Commonly these are due to a survey being blocked or deleted. However these errors are generally fixed directly by the users.  For example they will see an error on their phone because the survey is blocked.  The survey will then be unblocked and the user will re-submit.  All is good.  However the monitor page will then show 1 succesfull upload and 1 error.

It is still useful to see if there have been any errors in uploading results.  The new type of “Upload Errors” shows these.  These errors are just informational and should align with errors reported on the phone.   The “Errors” column will continue to show uploads that were accepted by the server but then  for some reason failed to get applied to the database.  These are the sort of errors where you might say “where is my data?” and are more significant.

The second change made to the monitor page is to always show the latest name of a survey.  Previously if you submitted 10 results to a survey then changed its name then submitted 10 more you would see two rows for the two survey names with 10 uploads each.  Now you will see one row and it will be for the latest survey name and have 20 uploads.

Other Changes

• Add hyper links to video and audio answers when generating PDFs of submitted data.
• When filtering by data values in field analysis the list of values to select in now sorted alphabetically
• Added “nochoose” to video and audio questions. Previously this only worked for image questions.  If set then when completing a form with image, audio or video questions the user will not be prompted to add media from the phones sdcard.
• Added another column called “upload Errors” to the monitor page that shows the status of submitted results.  In a previous change we had started showing