Author Archives: Neil Penman

Upgrading from Oversight Forms to the Console

The new console release is not backwardly compatible with Oversight forms so once your server is upgraded you will need to start using the new approach. Here are some tips:

Using the console

Oversight forms used to be attached to a standard form. Each form could have only one oversight form. Hence when you went to the managed forms page and selected your form all the oversight form’s data was also immediately visible,

The console is more flexible, more options for the user and hence a little bit more work for them to set up what they want. When you go to the console the first time only the data for you current form will be shown. However you will see a drop down that allows you to select your “Oversight Survey”. Change this from “none” to the form that you want to use to edit your data. Once you have done that you will see the oversight questions as before. The selection you made should be remembered.

Selecting an oversight form / survey

Editing data

In the previous approach to oversight forms you could select a row and edit it. Now if you want to change a record of data you need to first lock it. So select the row then select the “Lock for Editing” button. This will assign you to the record so that you can make changes.

You can see who is assigned to a record in the _assigned column. This column is color coded:

  • Yellow if the record is not assigned
  • Blue if it is assigned to the current user
  • No color if it is assigned to someone else

Records assigned to someone else can only be viewed they cannot be changed. An administrator can reassign records to a different person. You can filter out records assigned to other people using the switches at the top right of the screen.

A view of the console showing the lock for editing button and the switches to filter by locked and unlocked records

Modifying the Oversight Form

Oversight forms are no longer anything special. They are just any old form that you want to use to edit data in the main form. To do this the questions in the each form would need to have the same name. The forms also need to be grouped.

Because oversight forms are like any other form you manage them from the form management page. When you upload the oversight form select the form that it will modify using the “Group” drop down.

A convenient way to create an oversight form might be to take a copy of the form you are going to manage and then add / delete questions as required.

Version 19.08

Well we Just got the August release out on the last day of the month, that was lucky. Mostly this release is about the console and has already been covered in the post http://blog.smap.com.au/a-console-for-case-management-release-1908/. However there is some additional functionality that I should describe:

Setting a Theme for Your Organisation

If you are an “Administrator” when you to to the user management page you will see a tab called “Appearance”. If you select that you can set the color of the menu bars and the logo that you see on the home page. In order to enable your new theme you will need to also check the checkbox labelled: “Set as Theme”

Oversight Forms have been Retired

Instead of having to create an oversight form using special syntax you can now group any other standard form with your primary form. There are some features of oversight forms that were not in the normal XLSForms that you use for data collection. Including Server Side Calculations and setting the background color of answers to questions. These have now been added to XLSForms (See Below)

Server Side Calculations

You can use a new question type “server_calculate”. Put the calculation in the “server_calculation” column.

Expressions are the same as for the advanced filter. https://sg.smap.com.au/sqlFilterHelp.html However as here we are creating new value and not filtering there is no equals sign. For example the following will show a date 7 days on from the time editing started on the survey:

  • ${_start} + interval {7_days}

These server calculations are evaluated at run time so if you change your calculation and replace the form then the server calculation will be updated for all the data you have already collected.

A new capability not available in the advanced filter is the if() function. Here your server_calculation will simply be:

  • if()

Then you specify the conditions in a new “conditions” work sheet. In the example below the value of the calcy question will be set to ‘Not neil again!’ if the value in the name question is ‘neil’. Note that there is a set of conditions for each server calculate question that has its calculation set to if(). Hence you put the question name in the first column.

There is only limited support in the online editor for server calculations. Specifically conditions are not supported.

Color Highlighting in the Console

You can use a new “styles” worksheet in the XLSForm editor to specify colors for cells in the console table that are dependent on the value of the data. The following example shows an example of a style list.

Note for styles we do use the list_name approach. Hence you can have a single set of styles that color multiple questions. Specify the name of the style list name in a new column “style list” in the survey work sheet.

Version 6.32 of FieldTask now in Play Store

This release includes a number of significant new features

History

If you select menu then history you can see a list of all survey instances that have been submitted to the server or deleted.

History view

The view is updated after you perform a refresh to synchronise with the server. In addition you can also delete an instance from the main screen by doing a long press on it. You will then select “reject” and will have to provide a reason.

Mapbox Tiles for use as offline maps

Both vector and raster tiles are supported.   A while ago offline map tiles were supported.  However this was when the maps used the google maps API.   ODKCollect have been progressively migrating to the MapBox API which supports tiles in a different format.  However now the feature is back.  Sample tiles are available here: https://github.com/klokantech/vector-tiles-sample just download them and copy the mbtiles file onto the phone’s sdcard under /sdcard/fieldTask/layers.  

Persian Calendar

If you have a date question and want to use the Persian calendar specify an appearance of persian.

A Console for Case Management – Release 1908

Organisations using Smap are increasingly managing work using data records that correspond to someone or something in the real world. This might be the maintenance of equipment, or the delivery of services such as education to a student, training to a teacher, health services to a patient or climate change resilience improvements to a community.

These applications represent an active form of Case Management where information is recorded against a case over time and used to guide the delivery of services. In the August release we have added a number of features to support this, including:

  • Find and view the record of data that represents a case
  • Update that data in a controlled way with logs and security
  • Request a task be sent via email, or to a mobile phone app, requesting an action be taken and data be collected
  • View all changes made to a case
  • View the status of all tasks
  • Send reminders if a task has not been completed via email or SMS or to fieldTask users

These activities can now be managed using a “Console” which allows a user to manage cases using the same forms that are used by the field workforce.

The role of the console in case management

What is a Record?

In the Smap universe every time you submit results from a survey you create or update a record.   You can group surveys so that they all share the same records.  This allows you to, for example, create a record with survey A and then update it with survey B.   Each case can be recorded in one or more of these records and so this feature is what provides a lot of the power of Smap as a case management tool.

Surveys in a group managing a record

When you select surveys on the records page, if they are in the same group then you will see the same records for each of them although the columns that you see in the table will be determined by the columns in the survey.

Actually this is a slight simplification in that you can also use security roles to restrict the records shown on a per form per user basis so not all forms in a group need show the same records to the same users.

New Features

Replacement of WebForms with a Console module

Here is some history to explain where the console came from. In the old Web Forms module, as well as having a list of forms you could launch in you browser, there were menu options for Browse Results and Managed Forms. Browse results was going to be the new analysis page but we never managed to make it better than the old Analysis page so it just lurked in Web Forms not being particularly useful. Managed Forms was similar to Browse results but it allowed you to attach another form and edit the submitted data using that form. This has been used in applications such as Feedback and Accountability.

Now Browse Results and Managed Forms have been merged and renamed to “Records”. The WebForms module itself has been renamed to “Console”. It still includes the page to launch forms and respond to tasks using your browser but it now has a focus on individual data records rather than analysis of all the collected data.

New Console Module Home Page

A scalable records page

By scalable we mean that the console can handle hundreds of thousands of records. Previously the browse results page had problems when customers had this many records as it tried to load all the data into the browser first and then filter.

Records page

If you look at the records page you will see that it is in two parts. The top half consists of filters that are applied on the database before data is retrieved. Underneath are the table view, map view and calendar view. For these the filters are applied in the browser. Hence you can select the survey, the from date, to date, an advanced filter and max records and a new query will be made on the database to get that data. The first time you open a survey the limit will be set to 1,000 so that you don’t have to wait to get the data. You can then change the limit of you need to see more, or less, records.

Modifying a Case

When managing a case you will need to be able to make changes to the data directly from the console. The first step is to lock a case for editing. Once you have done that it is assigned to you and other people cannot edit the same record, although an administrator could re-assign the case to someone else. In the diagram below the “Unassigned Records” switch is selected so that cases not assigned to anyone can be selected. An unassigned record has then been selected by clicking on it. The button “Lock for Editing” can then be clicked.

Locking a record for editing

Once you have locked a record the edit button will be shown. The questions you can change are determined by the Oversight survey that you have selected as well as any security roles that have been applied. These questions are shown on the right hand side of the page and are labelled as “Management Data”. Any remaining questions in the main survey that are not in the oversight survey are shown as read only on the left hand side for your reference.

Record opened for editing

In the above image there are 2 questions in the management form, a geopoint question and a text question labelled as “comment”. You can select a new location, change the comment and then click “Save”. The existing data for the record will be updated.

Reviewing changes to a Case

If you click on the events tab you can view all changes made as well as see all task and reminders that have been created for the record. You can also create new tasks that can be assigned to your field work force to update the case or send a reminder.

Events on a task

In the above diagram changes have been made to the record using the survey called “Map”. The location question and questions “q2” and “hi” were changed. For location questions; points, lines, areas the old location is shown in blue and the new location in red.

A task has also been created on this record but it has not yet been assigned to anyone.

Conclusion

There is a lot more functionality in this upcoming release that I will talkcover once the software is deployed to the servers. Hopefully this post will give you some ideas for the types of case management applications you will be able to create from about the middle of August.

New FieldTask Features in Version 6.21

There is a new version of FieldTask available in the google play store. This has some cool new features.

Relevance and Constraint work inside a page

This long awaited feature has been added by the ODKCollect community. It means that you can now put multiple questions on one page of the screen using the “field-list” appearance and make some of the questions visible depending on answers to other questions on the same page.

A simple example is a text question for “other”. This can now appear automatically under the select question when the “other” choice is selected.

Track changes to answers while a survey is being completed

This takes auditing to a whole new level.   Changes are written to a csv file.

Example audit file

In the example above the user initially entered a value of “Tom” as the answer to question 1. They then went back and changed that value to “Neil”. Note the Duration column. This was added by taking the difference between the start (in milli seconds) and the finish and then dividing by 1000.

You can download the above CSV file using the API: https://sg.smap.com.au/api/v1/audit/log/s193_1778/uuid:9fcf9bf7-d780-4397-8781-b4ba0bbe0cb8

Logon with username “api” and password “api”

To enable logging of question changes

In the online editor select the “File” menu and then “Settings”. Select the checkboxes “Timing Data” and “Audit Changes to Questions”.

How to enable the recording of changes to answers within a survey

This can also be enabled from an xlsForm by specifying “yes” as the value in the timing_data and track_changes columns in the settings worksheet.

Viewing the Audit Log

The log has to be downloaded using the audit API. Here are some links to get you started:

Documentation: http://www.smap.com.au/apiAudit.shtml

Getting the URL that will retrieve your data:

  1. Get a list of data end points: https://sg.smap.com.au/api/v1/data/
  2. Select the URL for the survey for which you want the log file
  3. Select the URL labeled “audit_log” for the submission that you want to review

You should change the above URL to point to your Smap server and you can log in with your username and password to see your surveys. If you just want to see the demo csv file then log on to the sg server using api:api

Version 19.06

New Features

  1. Added reminders for tasks. These reminders are specified as a type of “notification”. You can set the time after a task has been accepted for a reminder email or SMS to be sent to the user who accepted it, or to someone else. So for example you could have a reminder after 2 days for the assigned user to “Get on with it!”. You could also set another reminder that if the task is not completed within 5 days a reminder is sent to a supervisor that includes a link to the task so that the supervisor can complete it. You can specify the reminder interval in hours or minutes as well as days, I’d like to think that “minutes” is there mainly so I can demonstrate the feature, however it may prove useful in fast moving problem domains.
  2. Allow URLs of images to be set as dynamic defaults for image questions. For example you could use cascading selects to select a schematic from thousands of available images. This schematic can then be annotated by the user and submitted with the completed form. This feature is now available in Webforms and in fieldTask 6.21. Here is an example form that gets the images from Wikipedia. Commonly however you will want to retrieve the images from the same Smap server that you are using to complete the form. By getting the images from the same server you won’t be blocked by CORS. Wikipedia does not enforce CORS but sites that require authentication, such as other Smap servers, would.
  3. Added dynamic defaults to webforms. A dynamic default is specified in the “calculation” attribute of a question. Hence you can set the initial answer to a question based on the answer to a preceding question. Allowing images to be used as dynamic default values as described above is a special case of this new feature. This feature is already available in fieldTask.
  4. Customise the appearance of the web site for an organisation by setting the colour of the menu bars and by specifying the main picture on the home page. You can specify these themes in the Organisations tab of the user management page.
  5. Add support for “track changes”. This is a feature from the ODK community that audits every change to a question answer while a form is being completed. So it adds to the existing audit capability that allows you to record the total time spent in each question and the GPS coordinates when it was finally answered. You can enable “track changes” in the settings dialog of the online editor. However you will need the soon to be released fieldTask 6.21 in order to use it.

Version 19.05

The big change in this release is the addition of GPS coordinates to named locations.

Named Locations

When you add or edit a task you can set the GPS coordinates, an NFC identifier or both to specify where the task should take place. This is what we call an anonymous location since you are not using a “Named Location”.

However you can now also enter the name of a “location” that has the NFC UID and/or the GPS coordinates pre-defined. To do this Select the group name first and then the location name. (This change addresses issue #468).

Selecting a named location with group “STICKER” and location name “E1”

Named locations are shown in the “Shared Resources” page. Select the Admin module then “Shared Resources” , then select the “Locations” tab to view them. Click on the “Download” button to get the current locations in an excel file, or an empty template if you have not specified any locations.

Named locations on the shared resources page

Loading Named Locations from an Excel file

Locations are organised into groups and each group is shown in a separate tab of the excel file. Create new tabs giving each one a unique group name. Then for each group create locations. You can specify:

  • UID: The identifier of an NFC chip at the location
  • name: The name of the location
  • lat:  The latitude in GPS coordinates
  • lon: The longitude

The location name must be unique within the group. When the file is ready select “Add Files” and then “Upload”.

Updating a location when editing a task

After selecting a location you may adjust the GPS coordinates or change the NFC UID. You then have the option of:

  • Continuing with an anonymous location
  • Updating the location with the new coordinates / NFC UIC
  • Saving a new named location

Other changes

  • Added “auto” as a parameter option in the online editor to automatically start camera, barcodes and NFC reader when the user swipes to that question. The auto start is only applied when swiping forwards, there is not already a value recorded for the question and the question is the first one on the page. (This update implements issue #475)
  • Drastically reduced the memory required when importing data from an excel spreadsheet. This was tested importing 80,000 rows each with 500 columns without any problem. This new version is also much faster than the previous import.
  • API changes
  • In PDFs allow the colour of a marker showing position to be configured using an appearance setting for the geopoint question
  • Fixed issue #433. The advanced filter was not working correctly with wildcards.

Version 19.04

New Features

Its all about Tasks in this release.  Lots of great new features.

  • Auto start of external applications in fieldTask. (requires fieldTask 6.10). An external application will be automatically launched if you set the parameter “auto=yes”.
  • Geofencing for tasks. A “Show Distance” can be set on a task so that it will only be shown to the user if they are within that distance of the task location. In addition a “download distance” can be set on a task group. Tasks in that group will only be downloaded when the user is closer that that distance from the average center of all the tasks in the task group.
  • Set initial data for a task.
  • Tasks API.

Geofences

Show Distance

This is for tasks that have been downloaded to fieldTask. The app will show and hide them as the user moves in and out of the geofence.

You tube video showing creation and use of a geofenced task

There are two geofence capabilities added in this release.

  1. “Show Distance”. The tasks are downloaded to the phone but only shown to the user when they are close enough to the task location. This distance is set per task.
  2. “Download Distance”. Tasks are only downloaded to the device when the user is close enough. This distance is set on a task group.

Set Initial Data for a Task

Amazingly this was not previously possible.  You could set a task to update an existing record however now you can also create a brand new submission but initialise it with some arbitrary data.  When creating tasks from existing data you can select one of the following 3 options:

No initial data means that your form will open empty. “Pre-populate form with existing data” is the new option, the data will be taken from the existing record and mapped to compatible questions in the new form that you want completed and the original data will not be updated. The third choice “Update Existing Results” continues to allow you to update an existing record. The data for that record will be shown in the form that you are going to complete.

You can’t yet add initial data when you add an ad-hoc task via the user interface. However you can do this via the new tasks API (see below).

Tasks API

A tasks API is slowly emerging. This is documented on the Smap website at http://www.smap.com.au/apiTasks.shtml.

  • Get task groups for a project
  • Get tasks
  • Get task details
  • Get initial data for a task
  • Create a task

Version 19.03

New Features

  1. Including data from linked surveys into a PDF
  2. New XLSX report on the task management page that will list all tasks in all task groups. Exports to XLSX for tasks in the current task groups are still available.
  3. Manage a password policy for fieldTask.
  4. Add support for the ODK setting “guidance_hint” when loading xlsForms
  5. Add support for the ODK setting “body::intent” when loading xlsForms
  6. Add a tasks API to return tasks in geojson format
  7. Add an audit API to return geoJson data on location and duration of each question asked in the survey
  8. Add the user identifier as a valid value for use in an advanced filter
  9. Record the PDF template for a survey in the surveys change log so that previous versions can be referenced
  10. Add ability to delete meta questions in the online editor as well as add the pdf_no appearance to them so that they are not included in PDF exports

Including data from linked surveys in PDFs

Smap allows you to combine Surveys to make “Systems“, where data in one survey can be linked to data in another. For example a survey recording the delivery of aid may include the identifier of the beneficiary. You can analyse this data taking account of the links between them by extracting the data into external tools like Power BI. However inside Smap itself we haven’t provided any capabilities to join data from different surveys during analysis. This new feature is the first step on that path.

When you create a PDF containing data from a single survey submission you can optionally specify:

  • Include reference data
  • Only include data from surveys launched from the current survey

The first option requires you to have specified a child subform as being launched from the survey you are exporting. You also need to specify a key question in the child survey.

If you include the second option you don’t need to specify a key question but only data collected by launching the child survey on the phone will be included.

Password Policy settings for FieldTask

FieldTask Version 6.02 has a logon page.  By default this is shown when the app is first run so the user can set the server, username and password without having to go to the settings menu.   After successful logon it is not shown again.  This corresponds to the default password policy of “Never show logon page”.  In the tab for fieldTask options on the usermanagement page you can also specify a policy of “Always show logon page” which will require the user to enter their password every time they start the app;  and “Periodically show logon page” which will require the user to enter their password again a specified number of days after their last successful logon.

Bugs Fixed

  1. Large API exports were causing out of memory errors. These data exports using /api/v1/data are now streamed so there should not be any size limits.