Category Archives: Smap Release

Release announcements for Smap

Release 18.6

Email Tasks

The ability to send pre filled or blank forms as links in  emails out to people for completion in their browser.  Capabilities include:

  1. Forms can only be filled in once by each recipient (unless the repeat flag is set)
  2. The URL to access the form includes a key so that the user does not need to have an account on the server
  3. You can upload email address and form names from a file and then have the emails sent to all of those addresses
  4. There is an unsubscribe link in all emails
  5. People can resubscribe, if they change their mind by going to the subscriptions page (on the home page select subscriptions)
  6. Emails with a pre-filled in form can be sent automatically when a submission is received for any other form

Really what this does is enable you to extend you data collection beyond the professional data collectors who have a logon on the server.  Hence you can do email surveys, as per Survey Monkey and also integrate input from external people in a multi step process.  So in fact Smap goes way beyond the capabilities in Survey Monkey.

Note: Email tasks are disabled on the sg server by default. Contact us if you want to use this feature.

Auto sending email tasks

You can also assign tasks that include previously collected data to an email address .   As with other tasks, that are completed by logged in users,  you can create these from previously submitted results or results as they come in.  Here is a video of an example process but there are an infinite number of ways you can use this feature:

After completing the steps shown in the above video and then doing an XLS report I got the following output:

Loading Email Addresses from an Excel File 

Email tasks are managed from the tasks module,  From there you can create a task group and then click on the file menu to download an XLS file.  In this case for new task group it will be empty but you can specify the “tasks” here.

The columns with blue headings are where you put details about the form.  (See above). The columns with yellow heading contain details about the person being sent the email including of course their email address.  You can add multiple email addresses to be sent a single form hence the greyed out cells in the above example where we have one form but two email addresses.  Also you don’t need to fill in the URL column, this is only filled in when exporting in case you want to manually send the links out without using Smap’s email manager.

Once you have filled in your forms and email addresses.  Click on the Menu File > Load XLS.  Emails will not immediately be sent, you will then have the opportunity to review the loading emails and when you are ready you can select the menu “Email Unsent”.   This will send the emails with blank forms.

Unsubscribe From emails

This became urgently needed due to the addition of the “Email Tasks” functionality, however the unsubscribe link is now added to any email sent from the Smap server including for example an email adding you as a registered user.  If someone clicks on that link they are asked if they want to confirm the un-subscription.  If they say yes then they won’t be sent any more emails.   They only way to resume sending emails to an unsubscribed address is if you can persuade them to:

  • Goto to the server home page
  • Click on the subscriptions menu
  • And subscribe again

Updates to FieldTask version 5.5

Well this was released some time ago but I didn’t make an announcement on this news site so here it is.  Lots of new features here contributed by the folk from ODK as well as the fieldTask developers. Some of the main ones are:

  • Online lookup of data from a server while the user is completing the form. You can lookup reference data, choices for a select or query an AI web service.
  • Non printable characters in barcodes will now show as spaces rather than being removed completely
  • Includes ODKCollect 1.15
    – Can set a dark or light theme to change the screen appearance
    – Add a “file” type question that allows you to attach any file to the form

 

 

 

Version 18.05

Time for the May release of Smap.

Updated Webforms

Webforms has been upgraded to use the latest version of Enketo.  Some of the improvements are:

  1. Pulldata now works in web forms so you can reference data in a CSV file or from another survey inside your survey in webforms.   Try this out here.  This demo app checks the current status of piece of inventory prior to that status being updated.
  2. New and improved widgets.  You can see how these look here,  in webforms using the Enketo widgets.xls form.  You can try it here.

The XLS forms and all the images are available to upload from the google shared drive

Webform Widget Highlights

Collapsable Groups

You can now click on the group name to hide all the questions inside it.  This makes navigating complex forms simpler.

Recording distress etc on a scale of 1-10

This is not available in fieldTask.  If you add an appearance of “distress” to an integer a thermometer will be shown for you to select an answer in the range from 1 – 10.

Draw widget

This has been available in fieldTask for many years and you can now use it inside web forms.

Annotate Widget

As above, where you take a picture and then annotate it.

Location Widgets

Smap only allows one geospatial widget per form which makes it easier to use the data with GIS systems.  You can of course include multiple widgets within a single survey by putting them inside “begin repeat” sections.  However I have put the following widget examples from the original enketo survey into seperate surveys.

  1. Geoshape.  Record an area.  
  2. Geotrace. Record a line.
  3. Hide fields.

Specifying Task Start Dates

When specifying the rules for creating tasks from submitted data you can now specify when the task should start.  Previously the task was always set to start 7 days after the data that prompted it was received.

Now you can specify either the date and time the source results were received or the contents of a date/time question in the source survey as the base for the starting time.  You can then specify an interval in days, hours or minutes after that base time for the task to start.

Version 16.05

New Features

A better “download the form” dialog

When you select the download button next to a form name on the form management page you will see a dialog asking what you want to download.   You can often download either of two versions of the form template as an XLS file.

  • The original XLS file uploaded to the server
  • The latest version of the form including modifications that you have made using the editor on the server

These two choices are now a little clearer.

download

The first choice is now the latest version of the form.  Next to the choice is the current version number.  The second choice is now the original XLS file, however if no XLS file was loaded then this choice will not be shown.  The version number for the original XLS file will always be “1”.

Logon button on the home page

You can now login and also logout from the home page.  The menus that you will see on the home page change according to your role and whether or not you are logged in.

Improved Task Management (Hosted Servers Only)

A new improved task management page has been added.  This is only available on hosted Smap servers and won’t be available in the download version.  New features include:

  • Import / Export of tasks to an XLS file
  • Viewing tasks on a calendar

tasks

version 14-12 Notifications

Here is, depending upon your location in the world, the last release of Smap for 2014 or the first release for 2015. The major change in this release is the addition of email notifications when a survey is submitted.  However the menus at the top of each page are also being progressively updated which will probably be the most visible change for most people.

Menu and Style Changes

The “bootstrap” javascript library is being implemented to replace “jquery-ui” and “foundation”.  In this release the following pages have been updated:

  • Form Management
  • Notifications
  • Locations (User Trail)

Pages will look a bit different, here is the form management page:

Menus

Smap has been divided into 4 sections:

  1. Home
  2. Admin
  3. Web Forms
  4. Analysis

These are accessible from a drop down menu labelled “View”.

dd

Drop down “View” menu

In the image above the “Admin” option is greyed out as the view option was selected from “Form Management” which is part of administration.  In a future release another section called “Edit” will be added.

Within each section the menu options will appear to the right of the “view” menu.  For administration these options are:

  • Form Management
  • User Management
  • Notifications
  • Monitoring
  • Locations
  • Tasks

Each section will also have common menu options

  • help 
  • User Settings (an icon that looks like someone’s head and shoulders)

Unfortunately because this new menu layout is being implemented progressively you will find yourself jumping back and forth between the new and old menus as you navigate from page to page.  If you find this unbearable let me know and I will try and make all the pages consistent faster.

Notifications

A notifications page has been added which allows you to request that an email be sent every time a new result is submitted to the server.

Notification Page

The notification page above shows a single notification that will send an email to two email addresses (separated by commas) when a result is submitted from the “modify stress test” survey. The email will contain a link to a web form view of the submission.

Web Form View of a Submission – Accessed by link in email

A log of all the notification events can be seen on the monitor page.

Application of email notifications

These could be used for quality assurance.  A supervisor can set up a notification for each submitted result which they can then review.  A future enhancement may allow filtering so that the emails are sent conditionally dependent on value of the data. Suggestions welcome.

Forwards

Previously you could set up a survey to be forwarded onto another server on the form management page.  Now however, forwarding is considered a type of notification.  Hence you can set up forwarding on the notifications page.

Miscellaneous Changes

  • The email host can be set per organisation.  Previously an IT administrator had to set the address of your email server (smtp_host) in the tomcat web.xml file.  Now it can be set in the users page by editing the organisation details.  Note the value for smtp_host in web.xml is now ignored so you will need to update the organisation.  
  • A default value of smtp_host can be set for every organisation on the server.  This does have to be done by an it administrator in the survey_defintions database using a command like: insert into server (smtp_host) values (‘Your smtp host’);
  • When uploading a new form to the server from an XLS file,  the name of the form will default to the name of the file.  You can optionally change the form name.
  • Increased the width of the line showing the path taken by the user on the locations screen.
  • Previously you could not select a set of survey results for editing if that survey had been marked bad.  This restriction has been lifted. You are now only prevented from editing results if that record has been replaced by another one.
  • Performance improvement to “data cleaning”.  There were some issues in performance last month when multiple people were cleaning data in the same survey simultaneously. Database deadlocks were causing updates to hang which resulted in the server running out of database connections.  These deadlocks have been fixed.

 

 

Version 14.10 – Bug Fixes

This is a minor release with just some fairly minor bug fixes.  These include:

  • French translations have been added replacing the dummy words that I had previously used for testing.  Thank you to Baba Sy for providing these translations.
  • Show an error message if a template is uploaded when the user does not have access to any projects.  (Thank you to Janna Metzler for identifying this issue)
  • Prevent the editing of results that have been marked as “bad”.  For example if they have already been edited and replaced by another record.
  • Fixed several issues with roles.  It now works much better if a user is assigned a role of analyst but not administrator or vice versa.  A user with only enum rights can now also use webForms.
  • Replaced links to the old help pages with links to the google drive holding the documentation
  • Prevent a malformed template being created if you upload a completely empty template.  This now reports an error and does not create the template.  Previously it reported the error but still created a template that could not be run on fieldTask
  • Fix paging through results on the monitor page. Previously the page buttons were hidden.
  • Thumbnails were not being created for pictures with an extension of JPG (in capitals)
  • Support added for select multiple questions when exporting to Neo4J
  • Only allow a single select multiple question to be exported to a Neo4J node or relation
  • Prevent Neo4J labels from having spaces
  • Improve clarity of reporting errors in templates when using template upload.  Multiple redundant messages were previously created when calculate questions with lots of parameters had circular references.

The server can be downloaded from http://www.smap.com.au/downloads.shtml.

 

Version 14.09 – Form Refresh and Export to Neo4j

Version 14.09 of Smap is available for download.

Refresh – Synchronising phones with the server

The refresh button on fieldTask now does more than just clear the submitted results from the screen. It also now refreshes the forms that you have on your phone and submits any finalised results.  Note you will need to download fieldTask version 2.3.5 from your server on to your phone to use this new feature.

Pressing refresh will now:

        • Download new forms from the server that have been assigned to the user of the phone
        • Show available forms with a yellow clipboard icon
        • Update forms on the phone that have a new version
        • Delete forms from the phone for which the user no longer has access
        • Submit finalised results from the phone to the server
        • Notify the server of the list of forms and their version that the user has downloaded to their phone

You can select a form from the task list screen to complete a survey.  The name, version and project containing the form are shown next to the icon.

device-2014-09-20-090412

Phone with two forms

This new feature should simplify the training of the data collectors as the steps they need to download new forms, start a survey and submit results are all simpler.

Checking that data collectors have the correct version of a form

Sometimes it can happen that data collectors go out into the field with the wrong version of a form on  their phone.  At worst this may have an impact on quality of the collected data and at best you will need to do some messy combining of the data from the two forms prior to analysis. This new feature makes it much easier to check that all the data collectors have the correct version of the form.

From the monitor page select forms.  This will show the users, phones, downloaded forms and the versions of those forms that have been downloaded.  Any users / phones that have not downloaded a form or have an older version are highlighted.

monitor

Monitor Downloaded Forms

 Exporting results to Neo4J / Thingsat

You can now export results to the Neo4J graph database. This change was developed in collaboration with thingsat.  The following video shows how this can be done.

Other Changes

  1. You can edit collected data using web forms.  This capability is enabled by an administrator for your organisation.  By default it is off.  If you select a table showing data for the whole of survey then there will be a button next to each data record that will open the survey results in a web form. If the data is updated the old results will be marked as bad, and then the updated results added.
  2. When you attempt to submit results from a phone for a survey that has been deleted on the server then the error message will be “Not Found” rather than “Not Authorised”
  3. The error message shown, if you upload a survey template that has duplicate options, has been made clearer

Smap Version 14.03 Forwarding

Smap version 14.03 has finally been released.  It should have been available from the end of March. The next release should be a small one and hopefully we can get back to the end of month regular release date.

The major new feature is forwarding of surveys.  Other changes are improvements to monitoring, support for larger result sets (paging) and a more laptop friendly deployment.

Monitoring

Totals

When you select “Monitoring” by default you will be shown total counts for success, errors, duplicates and “not uploaded”.

monitor

Totals are only available in Table view.  In order to view locations of uploaded data you need to first select the source as “Instances” then select “show as map”.

Filtering

You can filter the uploads (or forwards) that you view in the monitor page by their status. Success, errors, duplicates and “not uploaded”.  Hence you can uncheck all statuses except for “Error” in order to see which uploads in your project or survey were reported as being in error.  When showing instances rather than totals the last 200 records are still shown.  However if you apply a filter to only show errors then it will be the last 200 errors that are shown.

Paging

Paging is now available in the Monitoring screen and for the survey table view in the dashboard.  200 records are shown by default but you can then select a forward / reverse button to move forward and back through the results. These buttons are located at the top of the table and are labelled “>>>” for forward and “<<<” for back.  They will be disabled unless there are additional pages available to view.

Total Result Records

In the dashboard, when you view all survey results as a table, then the total count of surveys will be shown as well as the count of “bad” records. This wil be a faster way of getting a count of all surveys loaded than having to export to a spreadsheet.

Deployment

Changing URL of the server

Previously it was assumed that the address of the server would not change.  This allowed the server to get away with storing the full url of all survey attachments in the database.  However this assumption is completely untenable if Smap Server is deployed on laptops.  The server URL can now be changed and all the attachments will still be accessible.

Simplified deployment

Almost all the editing of files that had to be done during the installation process is now not needed.  The one exception is that the IP address of the server still has to be set in the install script.   This is only required for webForms and will hopefully be removed soon.  So, assuming you are installing on a clean Ubuntu machine, the new installation process is:

  1. Download and extract the release
  2. change directory to smap/install
  3. Edit the install script to add the IP address (temporary)
  4. Type ./install.sh

This will also now run the deploy script so once it finishes you shoulld be able to log on and use the server.

This release will be added to the downloads page over the coming weekend.  There will be another blog post to announce its availability.

Bug Fixes

  1. Error in retrieving languages on the audit page fixed.  This bug prevented auditing.
  2. If the first record in survey had an invalid geometry then export to shape files would fail. An example of an invalid geometry is recording only one point when a polygon or line was requested.

 

Smap Version 14.01 Task Management

Announcing the release of Smap Evaluation Tools Server version 14.01.  This release includes

  • Task management: the ability to filter the surveys that are used as locations for new tasks
  • Task management: Printing of task allocations across multiple pages
  • Support for Internet Explorer 10+ as the browser
  • Improved reporting of errors when uploading templates

Task Management

Task Filtering

When you create new tasks from an existing survey you can now specify a filter on one of the questions in the existing survey.  Only survey locations that match that filter will be added to the new task group. Select the checkbox called “Filter tasks” to enable filtering.

tm1

Task Filtering

 

In the following example the previous survey recorded the paths of various cable types including copper and fibre.  However for the new task I only wanted the surveyor to go to the cables of type copper so I enabled filtering and selected the option of “copper”

tm2

Tasks filtered by cable type of copper

Printing Task Sheets

When you select the “Print” button to print a task sheet the tasks are now shown on more than one page.  This attached PDF shows this. Note the address parameters that have been added to help locate the location of each task.  In this case these include the age, ethnicity and a picture.

 

 

 

Smap Server version 13.12 with Time Series Analysis

Announcing the release of Smap Evaluation Tools Server version 13.12. The time series analysis functionality in this release has been funded by the Speed Evidence project. This is to assist in the long term response to Typhoon Haiyan.

New functionality

  1. Time Series Analysis.  Showing trends over time using graphs, maps and tables in the analysis module.
  2. Server side calculations.  There are only two functions available at the moment; area of a polygon and length of a line or the perimeter of a polygon.  However other functions could be added such as the calculation of Z-score anthropometric indicators.
  3. Export of attributes from parent forms to shape files.  If you select a child form containing a geometry and export to a shape file then all the attributes of the parent (up to the maximum of 255) will also be exported.
  4. Specification of default language.  The default language to show on the phone can now be specified by selecting “view/edit” link next to the survey name on the template management page.
  5. Support for Internet Explorer has been improved however it is still not supported, that is I haven’t tested this release against IE.

Availability

Hosted servers will be progressively updated over the next week or so.  The download for this release will be available by the 19th of January.

 

 

Release of Smap Shapefiles Export

Announcing the release Smap Evaluation Tools Server version 13.10. New functionality includes:

  1. Exporting surveys to ESRI shapefiles.
  2. Administration and analysis screens are a bit more mobile phone friendly.  With the this release you should be able to create and view analysis panels on your mobile phone now.  However this is work in progress.
  3. Surveys that have been blocked will no longer be available for download.  Submission of results from these surveys will continue to be prevented until the survey is un-blocked.
  4. Incomplete surveys are no longer included in results automatically.  These can be uploaded from the phones but they will be marked as “bad” surveys with a reason of “incomplete” by default.

Exporting Shapefiles

Individual forms within a survey can be exported as shapefiles and then imported into a GIS such as ESRI.

The Smap Server now supports exporting to XLS, CSV, Openstreetmap as well as shapefiles.

Blocking Surveys

Installing

You can download the server tar file from here.