Version 16-10

The main change in this release is that managed forms have been improved and are now ready for production use.  Documentation is available here.  The Beta status has also been removed from the online editor.

Other Changes

Automatic “erase” of forms

A notice has been added to the home page stating that deleted forms will be automatically erased from the system after 3 months.  Deleting a form only makes it invisible.  You can restore it along with all of its data if you realise that you still need the data.  To permanently erase a survey you have to manually erase it.   However it seems that a lot of people are forgetting to do this which is a security issue as well as making the database larger than it needs to be.  Hence the automatic erase 3 months after deletion.  If you are deleting a survey which contains sensitive information you may still want to manually erase it straight away.  This new policy will take effect from the end of November 2016.

Include Maps in PDF Exports

When generating a PDF location, the answer to location questions are now shown on a map.

pdf

When location data is available then the map dimensions are generated automatically. If you are downloading a blank template then you will need to specify the map dimension in the appearance column of the form definition.  The appearance options are:

pdflocation_xxxx, where xxxx should be replaced by the longitude, latitude and zoom. For example:  pdflocation_121,14.6,16

pdfmap_xxxx, where xxxx should be replaced by the mapbox map to use.   The default is mapbox.streets

Filter by date when exporting results

Start and end dates can be selected when exporting results just as they can be for viewing results on the dashboard.

Filter results by the date they were uploaded to the server

In analysis you can now filter by the date the survey results were uploaded as well as start of survey, end of survey and date questions.  This is available when showing a graph, map or table on the dashboard. It has also been added to the new feature for exporting results.

HXL tags

HXL is the Humanitarian Exchange Language .  It consists of  number of tags that you can add to the second row of a spreadsheet of data that will help interpretation of that data when it is loaded into HDX. Tags look like Twitter hashtags, for example #sector to identify the column as containing the name of the cluster.

To specify the HXL for a column of data add the tag in the question’s appearance.  For choices in a select multiple the attribute “+label” is automatically added to the attribute.

The row of HXL tags will then be added to excel exports if you select the HXL option in the export dialog.

hxl

The major limitation of this current implementation is that the export data is raw data whereas HXL and HDX are mainly used with aggregated data.  The next release of Smap will include the use of HXL in aggregate reports.

Version 16-08

The major change in this release was the addition of “Roles” to control access to:

  • individual surveys
  • columns in a survey
  • rows in a survey

Only users who have “Security Management” group given to them on the user management page can set up roles.  The steps are:

  1. Create a “role” in the roles tab on the user management page.  This is just a name, any name.  For example we could create a role called “District 11”.
  2. Give 1 or more users that role.  To do this click on the user name in the “users” tab on the user management page. Then select the “District 11” role.
  3. Edit a survey to require this role.  To do this open the survey for editing and select the menu “Files” > “Roles”. Then enable the “District 11” role.

At this point only those users who have the “District 11” role will be able to access the survey.  This restriction also applies to administrators who will not be able to give themselves the role unless they are in the security management group.

To add column restrictions click on either “filter rows” or “filter columns” in the survey editor role management page (where you were in step 3 above).

The following video shows steps 1 to 3.

Accountability – Talk at GeOng 2016

Amos Doornbos and I gave a 5 minute lightning talk recently at the GeOng 2016 conference in Chambery.  The subject concerned using a Mobile Data Collection tool, such as Smap, as the starting point of a system to manage feedback and complaints.

These slides in google docs contain the content of the talk: https://docs.google.com/presentation/d/1qG_y_4DkZhmeM0B-X-Ibhtqume2jOyjdarOgV5yfdmI/edit?usp=sharing

 

A single Mobile Data Collection Tool vs adopting standards

I just left GeOng 2016 in Chambery which was a really good conference with lots of interesting workshops and presentations.  However I want to comment on one of the final sessions, the subject of which was “Response-effectiveness of new analysis tools for humanitarian professionals”.

In one segment of this session Anwar Mahfoudh from OCHA/Kobo argued for the complete elimination of the “proliferation” of Mobile Data Collection tools used by Humanitarian agencies.  He argued all not for profits and NGOs should use Kobo (maybe there might be some very small usage of “legacy” tools as well).    The rationale was to reap the benefits of standardisation such as lower costs since every data collector everywhere in the world would be familiar with the same tool.

OK I need to declare a small conflict of interest here; if this happened that would mean no more Smap for humanitarian agencies.

Anwar was arguing for standardisation, which is a good thing, he was just arguing for standardising on a tool rather than on the interfaces and processes.  Tools come and go; standards should last much longer.  After all, AOL and Netscape are not so common nowadays, and yet the underlying standards that they and the rest of the World Wide Web were based on have continued to evolve.  Fortunately two of the other panelists in the session,  from UNHCR and ICRC, both made this point and rejected the mono-tool approach.

Some useful standards could be:

  • xlsForm and the ODK XForm standard to allow forms to be moved between tools.  Even where an MDC tool does not support ODK natively they can implement these as an import/export.
  • Use of HXL to facilitate consistent interpretation and comparison of results gathered by different tools
  • The koboToolBox API.  Smap partially implements that so if you are pulling data automatically out of Kobo you can use the same interface to get it from Smap.

Hopefully OCHA/Kobo will soon switch to arguing the functional merits of their own tool and adoption of standards rather than seeming to argue that technical competition is a bad thing and won’t lead to better results for the community.

 

 

Version 16-07

New Features

Managed Forms

A managed form is created by adding an “oversight form” to a “survey form”.  Data in the oversight form can be updated from the user’s web browser to allow them to add additional information to the survey data.

mf

This feature should be useful in cases where individual records from a survey are significant and need a response.   In contrast where data is aggregated and analysed statistically managed forms do not add any value.

Demo

Setting up a managed form

  1. Create an oversight form
  2. Link the oversight form to a survey form
  3. Open the managed form

Create an oversight form

Select the menu modules > manage > setup managed forms

Then select the tab “Available Oversight Forms”.  You can then download the oversight template, define your form and upload it.

setupmf

Link the oversight form to the form

Select managed forms.  Then click on the edit button next to the name of the form you want to link.  Then specify the oversight form.

Open the oversight form

Select Modules > “Web Forms” > “Managed Forms”.

Export Results in xlsx and xls format

Previously when you exported results as an XLS file the format was actually HTML.  However this HTML could be read by Excel or LibreOffice.  Now you can export as an actual XLSX or XLS format as well as still exporting in an Excel compatible HTML.

excel

After selecting an output Format of “Excel” select the “Excel Format” you want.  The default is “HTML”, just because that is the format previously used however it probably makes more sense to choose xls or xlsx.

Other Changes

  • Export PDF templates in Arabic.
  • Set the threshold distance at which fieldTask will record a GPS point
  • Warnings are no longer shown after uploading an xlsForm template.  They added little value and were mostly confusing.
  • Removed the storage of survey “path” names from the database. This change should increase the robustness of the online editor.

Patches

v16-07-04

  • Management forms can now update data collected using a data form.  To do this specify the same name for a management form column as is used in your data form.
  • Fixed an issue that prevented “preload” question types from being downloaded when exporting an edited survey to an xlsForm.

v16-07-06

  • Added security roles.  You can now create roles in the user management page and assign them to users. This requires you to have the Security Manager or Organisational Administrator security group
  • In a managed form you can now specify that a select list is made up of users who have a role.  Multiple roles can be specified.
  • Fixed some security issues.

Smap is on the move

Smap Consulting now has a permanent company address in Canberra.  However one of the objectives of this move is to allow a more flexible physical location.  So although the company will be located in Canberra we may be working out of co-working space in Manila or Nairobi etc depending in the work we are doing and the location of our customers.

The company address is now:

PO Box 4644
Kingston,
Canberra, ACT, 2604
Australia

Our current physical location is at Brainspace a co working space in Kingston.

 

SPSS now available as an export option

You can now export data for import into SPSS.  This complements the existing formats of:

  1. xls
  2. csv
  3. kml
  4. shape
  5. Stata
  6. Openstreetmap

On the Smap server go to the analysis module and select “Export”, then select SPSS as the format. A Zip will be downloaded containing a CSV file for the data and an SPS file for the variable names.

Within SPSS import the CSV data file. Then load the variable names by selecting:

File > Open > Syntax

Then select:

“Run All”

Any suggestions on improvements to the information that is written to the SPS syntax file would be gratefully accepted.

Map Quest no longer support tile access

What is it with map providers! You may see something like this when you open a map view:

1468270992221

The changes Map Quest have put in place mean that they no longer support OpenLayers which is the mapping API that Smap uses in many places on the server. fieldTask is also affected.

More details on their announcement here: http://devblog.mapquest.com/2016/06/15/modernization-of-mapquest-results-in-changes-to-open-tile-access/

There is an updated version of the Smap Server to address this problem available on the downloads page.  http://www.smap.com.au/downloads.shtml

A fix for fieldTask should be available soon.

Google Map Keys Are Back

From the 22nd June 2016 you once again need an access key in order to use the google maps API.  This is a return to the situation as it was before version 3 of Google Maps.   However this change currently only applies to new web sites which is why I hadn’t noticed it up until now.

Smap does not use the Google Maps API directly to draw maps, instead we use Openlayers and Mapbox, however we do use it to retrieve google street maps and google satellite base maps.  If you have recently installed a new Smap server you may find these base maps no longer show up.

The problem has been addressed in version 16.06.11 of the Smap Server which is available from the downloads page of the Smap website.  This version has an extra tab on the users page labelled “Server”.  You will need the “Organisational Administrator” role to see this tab.

server

To get the google maps back you only need to set the google maps key. However you can also set the server level mapbox key and email configuration from this tab.

Version 16-06

New Features

Online Editor

1. Info Dialog

There was a problem with the editor in the previous release that it would not show that you had created a new form that shared a table with an existing form.  An “info” dialog has been added under the “file” menu to show this sort of information.

info

Info Dialog

The “Based On” text in the dialog shows the survey name and the form name that the new survey was based on.  These names are separated by double colons “::”.

2. Make questions read only

You can now make questions read only or convert them to writable when in the online editor.

Referencing data from other form

1. Specify a unique key for each record

Previously the only unique keys associated with a record of data were generated when the data was collected.  These are the “prikey” and the instanceId.  However if you then edit the previously collected data new versions of these keys are created.   Now you can specify a “Human Readable Key” as a combination of answers to questions in the form plus an internally generated number to make sure it is unique.   Its these questions from the form that make the key “Human Readable”, for example a key could be “Zambia-2016-11”.  This key will not be changed if the data is edited.

2. Link a question to data in another form

In the online editor you can specify that a question links to a “Human Readable Key” (HRK) in another form.  Then if you answer that question by entering an HRK the data can be linked in reports.  For example you may have a form that maintains assets and another form to report a fault.  If you enter the HRK for the asset into the form recording a fault then reports can be created on the faults per asset.

You can enter the HRK as a text question, using a Barcode question or a select_on question.

3. Referencing data collected by other forms

Previously it has been possible to load CSV files containing data that can be referenced, or used to create choice lists, while filling in a form.  It is possible to:

  1. extract previously collected data
  2. edit it into a suitable CSV file
  3. Then upload that CSV file for use by another survey

This new features simplifies that whole process by allowing you to reference other surveys instead of CSV files.  The latest data is then automatically extracted when you refresh fieldTask.

Browse Results

The results browser is available from the webforms module. Features include:

  1. Select the columns to show
  2. Filter on all “select one” type questions
  3. Search for text in the results

View Logs

It is now possible to view logs showing various events such as deleting or erasing forms and also viewing the data in forms.

Access the log by selecting the new module called “manage”.  There should be a high volume of logs generated, you can sort them and filter on survey name, user and event type.  You can also do a text search.

Bug Fixes

  1. Smap Uploader was failing to upload data using https.
  2. If no data had been submitted for a survey then a blank survey template could not be generated.